AI crawl checker

Enter your website or page URL and click Check crawl access. The tool fetches robots.txt from your site and tests the path against Googlebot, GPTBot, ClaudeBot, and other supported crawlers. Tap any crawler in the results matrix to see the matched user-agent group, winning Allow or Disallow rule, and line number.

No. robots.txt tells compliant crawlers which URLs not to fetch. A URL can still appear in search results from links, sitemaps, or historical data. Use noindex meta tags or HTTP headers when you need to manage indexing separately from crawl policy.

The * group applies to crawlers without a more specific matching token. When Googlebot has its own group, that named group replaces the wildcard for Googlebot only. Other crawlers still use * unless they have their own group.

The longest matching path pattern wins. If Allow and Disallow patterns match with equal length, Allow takes precedence. Empty Allow or Disallow values are ignored.

Compliant crawlers generally treat a missing robots.txt as no crawl restrictions. This tool evaluates a not-found live fetch as an empty policy that allows crawling.

Yes, when vendors publish separate user-agent tokens. For example, you can Disallow GPTBot for training while leaving OAI-SearchBot unrestricted. Always confirm current token names in each vendor's official documentation before publishing rules.

Google-Extended is a robots.txt token that controls whether Google may use your content for generative AI model training and related products. It is not a standalone HTTP crawler user-agent. Googlebot rules still govern standard Google Search crawling.

These are user-triggered fetchers, not autonomous crawlers. OpenAI states robots.txt may not apply to ChatGPT-User; Perplexity documents that Perplexity-User generally ignores robots.txt. This tool does not report definitive verdicts for them. Claude-User is also user-triggered but Anthropic documents that it honors robots.txt.

CCBot is Common Crawl's crawler — its archived datasets feed many AI training pipelines. meta-externalagent is Meta's crawler for AI training and product indexing. Bytespider is ByteDance's training crawler. All three support robots.txt tokens, though Bytespider compliance has been reported as inconsistent — verify in server logs if you block it.

Applebot-Extended is a robots.txt token that controls whether Apple may use content already crawled by Applebot for foundation model training. It does not crawl webpages on its own. Blocking Applebot-Extended does not block Applebot search crawling.

Yes. Expand View or edit robots.txt, paste or edit the file, and results update automatically. This lets you preview how rule changes affect each crawler without fetching again.

No. robots.txt is publicly readable and only influences compliant automated crawlers. Sensitive paths must be protected with authentication and server-side authorization, not robots.txt alone.