Robots.txt Checker & Validator — Free Online Tool

A robots.txt checker fetches your site's robots.txt file from the domain root, parses every User-agent, Allow, Disallow, and Sitemap directive, and flags syntax errors, accidental crawl blocks, and missing AI-crawler permissions. The most common mistake — a misplaced Disallow: / that hides the whole site from Google — removes pages from the index within 48 hours. Use the validator above to check any URL, or follow the step-by-step guide below to write a robots.txt from scratch.

What this free robots.txt validator checks

Enter any public URL above and the tool fetches the robots.txt file at /robots.txt on that domain. It then runs eight validation passes: (1) file accessibility — is the file returning HTTP 200 and served as text/plain; (2) UTF-8 encoding compliance per the RFC 9309 robots specification; (3) directive syntax — every Allow/Disallow line has a valid path; (4) User-agent grouping — no orphan rules; (5) wildcard correctness — * and $ used per Google's parser rules; (6) sitemap discoverability — at least one Sitemap directive present and reachable; (7) AI crawler access — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot explicitly handled; (8) accidental blocks on CSS, JavaScript, or image files Google needs for rendering.

Why robots.txt validation is critical for SEO in 2026

Google's crawler budget is finite. Every wasted crawl on an admin URL, faceted search parameter, or staging environment is a crawl you do not get on a real revenue page. A clean robots.txt directs that budget to the URLs that matter. Get it wrong and you see one of three failure modes: (a) full deindex — a stray Disallow: / removes the entire site within days; (b) partial blackout — an overly broad wildcard hides a section like /blog/ or /products/; (c) silent quality drop — Google cannot render your pages because CSS or JavaScript is blocked, lowering the page experience score and pushing you down the SERPs.

For Delhi-area B2B and SaaS clients I work with through SEO services in Delhi, the robots.txt is the first file I audit in every engagement. I have rescued sites that lost 60-80% of their organic traffic from a single deployment that pushed a staging-environment robots.txt to production. Validation takes thirty seconds and prevents months of recovery work.

The eight robots.txt rules every site should follow

(1) Place the file at the root of every host. example.com/robots.txt, blog.example.com/robots.txt, and m.example.com/robots.txt are three separate files for three separate hosts — subdomains do not inherit. (2) Keep it under 500 KB. Google ignores everything past that size. (3) Use absolute URLs in Sitemap directives. Sitemap: https://example.com/sitemap.xml, not /sitemap.xml. (4) Never block CSS, JS, or images Google needs to render the page. Modern Googlebot needs the same files a real browser needs. Block them and Google sees a broken page.

(5) Use the most specific User-agent rule that applies. When Google sees both User-agent: * and User-agent: Googlebot, only the Googlebot block applies — every other rule is ignored. (6) Test wildcards carefully. Disallow: /*.pdf$ blocks every PDF file. Disallow: /*.pdf (no $) blocks every URL containing .pdf anywhere in the path — usually not what you want. (7) Do not use robots.txt to hide private content. Disallow only prevents crawling, not indexing — Google can still index a URL it learns about from external links. Use noindex meta tags or HTTP auth for genuinely private pages. (8) Explicitly allow AI crawlers. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot are the crawlers that feed ChatGPT, Claude, Perplexity, Google AI Overviews, and the Common Crawl dataset most LLMs train on. Blocking them removes your brand from generative engine outputs.

How robots.txt affects AI search and GEO in 2026

Generative engines crawl the web with named bots. GPTBot feeds ChatGPT. ClaudeBot feeds Claude. PerplexityBot feeds Perplexity. Google-Extended controls whether your content trains future Gemini models — independent of regular Googlebot crawling for search. CCBot powers Common Crawl, the dataset used by most open-source LLMs.

The robots.txt is the only signal these crawlers respect. If you block them, you remove your brand from the corpus generative engines pull from when synthesizing answers — the foundation of Generative Engine Optimization (GEO). Some sites block them deliberately to prevent training data harvesting; that is a defensible business decision but it comes with a hard tradeoff: zero presence in AI engine outputs. For a brand investing in AEO and GEO, explicit Allow rules for these bots are non-negotiable. See our full SEO vs AEO vs GEO comparison for the strategic context.

Common robots.txt mistakes that destroy rankings

Mistake one: pushing the staging robots.txt to production. Staging environments typically have User-agent: * followed by Disallow: / to keep the site out of Google. Forget to swap it before launch and the production site goes dark. I have seen this kill 80% of organic traffic in two weeks. Always verify the live robots.txt within minutes of any deployment.

Mistake two: blocking faceted parameters too aggressively. Ecommerce sites often try to block filtered URLs like ?color=red&size=large using Disallow: /*?*, which also accidentally blocks every UTM-tagged URL, search result page, and pagination link. Use targeted parameter blocks instead, or better, use canonical tags and parameter handling in Google Search Console. Mistake three: blocking the sitemap. Listing Disallow: /sitemap.xml is more common than it sounds. Always validate that the sitemap URL itself is allowed. Mistake four: case mismatches. Disallow: /Admin/ does not block /admin/ — robots.txt rules are case-sensitive for paths.

Mistake five: relying on robots.txt for security. The file is publicly readable. Listing /secret-admin/ in Disallow advertises its existence to anyone who reads your robots.txt. Use HTTP authentication or IP allowlisting for sensitive areas. Mistake six: ignoring crawl-delay. Google ignores the Crawl-delay directive entirely. Bing, Yandex, and most other bots respect it. If you have server-load issues, configure crawl rates in each engine's webmaster tools rather than in robots.txt.

Robots.txt vs noindex vs canonical: when to use which

The three controls solve different problems and confusing them is the most common technical-SEO mistake I see. Robots.txt controls crawling. A URL blocked in robots.txt can still appear in Google's index if linked from elsewhere, but Google will show it without a title or snippet. Noindex meta tag controls indexing. The URL must be crawlable for Google to read the noindex directive — so noindex in robots.txt-blocked URLs is functionally useless. Canonical tags control which version Google indexes when duplicate content exists. They are advisory, not directive.

Rule of thumb: use robots.txt to save crawl budget on URLs Google should not bother fetching at all (admin pages, large media files, parameterized search results). Use noindex meta tags or HTTP x-robots-tag headers for URLs Google should crawl but not show in results (thank-you pages, internal search results, low-value tag archives). Use canonical tags for duplicate or near-duplicate content (paginated pages, sort orders, mobile/desktop variants).

How to fix a robots.txt blocking your site

Step one: confirm the block. Open the URL above in incognito and check that the file is actually returning. Step two: use Google's robots.txt report in Search Console to see what Googlebot last fetched. Step three: identify the offending rule — usually a stray Disallow: / or an overly broad wildcard. Step four: ship the fix and verify the live file matches what you intended. Step five: submit affected URLs through GSC's URL Inspection tool with "Request Indexing" to accelerate recovery. Recovery typically takes 7-21 days depending on how many URLs were blocked and your site's authority.

Combine this validator with the rest of the technical-SEO toolset: the schema generator to ship structured data once pages are crawlable, the llms.txt checker to verify your AI-engine manifest is reachable, and the page speed analyzer to confirm Core Web Vitals are not blocking rendering. For deeper context, read the long-form robots.txt optimization guide and the related crawl budget guide.