EN
Home Category Tool

robots.txt Checker

Parse robots.txt rules and test URL crawl permissions

FAQ

How do I check whether a page can be crawled?

Pick a crawler such as Googlebot, enter the URL, and the tool applies the longest-match rule with Allow winning over an equally long Disallow.

Are the wildcard and end-anchor supported?

Yes. An asterisk matches any sequence and a trailing dollar sign anchors to the end, matching how production crawlers behave.

Does Disallow keep a page out of the index?

Not reliably. It only blocks crawling; the URL can still be indexed from external links, shown without a snippet. Use noindex to truly keep it out.

About this tool

Parses robots.txt and decides whether a given URL may be crawled. Paste the content, or fetch it online - when a cross-origin read is blocked the tool tells you to open robots.txt in the browser and paste it back.

The matching rules are subtler than they look, and this tool follows production crawler behaviour:

  • Longest match wins: with both Disallow: /admin/ and Allow: /admin/public/, a request for /admin/public/a hits the longer Allow and is allowed.
  • Allow wins ties: at equal path length Allow beats Disallow, which is Google's convention.
  • Wildcards and end anchors: * matches any sequence, and a trailing $ anchors the end - Disallow: /*.pdf$ blocks only URLs ending in .pdf.
  • An empty value means no restriction: Disallow: with nothing after it allows everything; reading it as "block all" is a common mistake.
  • Multiple groups for the same User-agent are merged, and an exact UA match takes precedence over *.
  • No matching rule means allowed - that is the default.

Inline comments such as Disallow: /admin/ # backend are stripped correctly, so the comment does not become part of the path.

Two boundaries worth knowing: robots.txt only binds crawlers that choose to obey - scrapers routinely ignore it, and it cannot keep people out. It also does not prevent indexing; that needs noindex, which in turn requires the page to be crawlable. Combine them badly and you get the classic deadlock of blocking the crawl so noindex is never seen. Finally, this tool does not validate sitemaps and does not reproduce every UA prefix rule (Google matches Googlebot-* prefixes).