Generate a valid robots.txt file for your website. Control which pages search engine bots can crawl and index — no coding needed.
Googlebot, Bingbot, and all other major crawlers can access your entire site. Recommended for most websites.
Prevents all crawlers from indexing your site. Use only for staging/development environments.
Block GPTBot (OpenAI), Claude-Web (Anthropic), CCBot, and other AI scrapers from training on your content.
Hides admin and login pages from search engine indexing. Recommended for all sites.
Prevents login, register, and account management pages from appearing in search results.
Prevents internal search result pages from being indexed, which can cause duplicate content issues.
Prevents shopping cart and checkout pages from being indexed. Recommended for ecommerce sites.
Blocks /wp-admin/, /wp-includes/, xmlrpc.php — recommended if your site runs on WordPress.
The robots.txt file is a plain text file placed at the root of your website (e.g. yoursite.com/robots.txt) that tells search engine crawlers which pages they are allowed to access and which they should ignore. It follows the Robots Exclusion Standard — a protocol respected by all major crawlers including Googlebot, Bingbot, and Yandex.
A correctly configured robots.txt file does two important things: it protects pages you don't want indexed (admin panels, login pages, staging environments) and it conserves your crawl budget — the number of pages Google will crawl on your site per day. Wasting crawl budget on irrelevant pages means your important content gets crawled less frequently.
After setting up your robots.txt, run a full SEO Audit to confirm your important pages are being crawled and indexed correctly.
* for all crawlers, or a specific name like Googlebot for Google only.Disallow: / blocks everything. Disallow: /admin/ only blocks the admin directory.Disallow: / under User-agent: * blocks Google from crawling everything. This is a common mistake on WordPress sites during development that gets forgotten and left live.noindex meta tags for pages you want crawled but not indexed.Check if your robots.txt is blocking important pages, plus 50+ other technical SEO factors — free.
🚀 Run Free SEO Audit →robots.txt is a text file at your website's root that instructs search engine crawlers which pages they can access. It follows the Robots Exclusion Standard, respected by Googlebot, Bingbot, Yandex, and all major crawlers.
Yes — directly. Blocking important pages prevents them from being indexed and ranking. Accidentally blocking your homepage or key landing pages via robots.txt is one of the most damaging SEO mistakes possible.
No. robots.txt blocks crawling — Google never visits the page. A noindex meta tag allows crawling but prevents indexing. For maximum control over sensitive pages, use both: block crawling via robots.txt AND add noindex to the page itself.
Go to Google Search Console → Settings → robots.txt. You can test specific URLs to see if they're blocked. Alternatively, visit yoursite.com/robots.txt directly in your browser to see the current file.
Yes. Each User-agent block targets a specific crawler by name. Common ones to block: GPTBot (OpenAI), CCBot (Common Crawl), Claude-Web (Anthropic), Bytespider (TikTok). Our generator includes AI bot blocking as a one-click option.
The file must be at the root of your domain: yourwebsite.com/robots.txt — not in a subfolder. If your site is in a subdirectory (yoursite.com/blog/), a robots.txt in that folder will not be recognised by Google.