Free AI Crawler Checker
Can AI crawlers get into your site?
robots.txt says who is allowed. Your firewall decides who gets in. We check both: the rule for every AI crawler, then a live request as each one, compared with an ordinary browser.
- robots.txt tokens
- 24robots.txt tokens
- live crawler fetches
- 19live crawler fetches
- purpose groups
- 3purpose groups
- no sign-up
- $0no sign-up
Reference
Every AI crawler the checker tests
AI companies run separate crawlers for training, for their search index, and for fetching a page when a user asks. They have different consequences, so the report groups them the same way. Reviewed quarterly against each operator’s documentation.
Search crawlers
If you block them: You drop out of the index these assistants search when they answer. This is the group that decides whether you can be cited at all.
| robots.txt token | Operator | What it does | Live test |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search results | Fetched live |
| Claude-SearchBot | Anthropic | Indexes pages so Claude can cite them in search answers | Fetched live |
| PerplexityBot | Perplexity | Builds the index behind Perplexity answers | Fetched live |
| Googlebot | Google Search, including AI Overviews and AI Mode | Fetched live | |
| Bingbot | Microsoft | Bing Search, which also grounds Microsoft Copilot | Fetched live |
| Applebot | Apple | Siri and Spotlight search | Fetched live |
| Amazonbot | Amazon | Alexa and Amazon AI answers | Fetched live |
| DuckAssistBot | DuckDuckGo | DuckDuckGo AI-assisted answers | Fetched live |
User-triggered crawlers
If you block them: When someone pastes your link into an assistant, or it tries to open your page to answer them, the request fails.
| robots.txt token | Operator | What it does | Live test |
|---|---|---|---|
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks about it | Fetched live |
| Claude-User | Anthropic | Opens a page when a Claude user asks about it | Fetched live |
| Perplexity-User | Perplexity | Opens a page when a Perplexity user follows or asks about it | Fetched live |
| Google-CloudVertexBot | Fetches pages for agents built on Vertex AI | Fetched live | |
| meta-externalfetcher | Meta | Opens a page when a Meta AI user asks about it | Fetched live |
| MistralAI-User | Mistral | Opens a page when a Le Chat user asks about it | Fetched live |
Training crawlers
If you block them: Your pages stay out of future model training. Live AI answers are not affected, but models will know less about you when they answer without searching.
| robots.txt token | Operator | What it does | Live test |
|---|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI models | Fetched live |
| ClaudeBot | Anthropic | Collects content to train Claude models | Fetched live |
| anthropic-ai | Anthropic | Legacy Anthropic token, still honoured in robots.txt | robots.txt only |
| Claude-Web | Anthropic | Legacy Anthropic token, still honoured in robots.txt | robots.txt only |
| Google-Extended | Opt-out switch for Gemini training and grounding | robots.txt only | |
| Applebot-Extended | Apple | Opt-out switch for Apple Intelligence training | robots.txt only |
| meta-externalagent | Meta | Collects content to train Meta AI models | Fetched live |
| Bytespider | ByteDance | Collects content for ByteDance models | Fetched live |
| CCBot | Common Crawl | Open web dataset used to train many models | Fetched live |
| cohere-ai | Cohere | Opt-out switch for Cohere training | robots.txt only |
What we will not show you
AI visit counts, training-crawl counts and “mentions in conversations”
None of them can be measured from outside your website. Crawler visits live in your own server and CDN logs; conversations happen inside the assistants and are not published. A tool that shows you those numbers without access to your logs is estimating, and we would rather tell you what we actually measured.
Want the real numbers?
We can connect your server or CDN logs and show you which AI crawlers visit, how often, and what they read.
Talk to us about itQuestions
Questions about AI crawlers
Straight answers, including the ones where the honest answer is “nobody can measure that”.
How do I check if AI crawlers can access my website?
Run the AI Crawler Checker on your URL. It reads your robots.txt rule for each of 24 AI crawler tokens and names the exact line that decides it, then requests the page as 19 live crawlers and compares each response with a normal browser request. A crawler that robots.txt allows but your server refuses is reported as likely blocked.
Why does my robots.txt allow a bot but the checker says it is blocked?
Because robots.txt is only a request. Firewalls, CDNs and bot-protection services often block or challenge unfamiliar user agents regardless of what robots.txt says. Cloudflare, Vercel, Akamai and others can all do this by default. Check your bot-management settings for verified AI crawlers.
Can a spoofed user agent test really tell if a bot is blocked?
Not with certainty, which is why the checker says “likely blocked”. Our requests come from our servers, not from the operators’ published IP ranges, so a firewall that verifies IPs may treat us differently from the real crawler. The finding is based on the difference between our browser request and our crawler request, which is the most reliable signal available from outside.
Which AI crawlers matter most?
The search and user-triggered ones: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot and Perplexity-User for Perplexity, and Googlebot and Bingbot, which feed Google AI Overviews and Microsoft Copilot. Blocking any of them removes you from that assistant’s answers.
More free tools
- AI Visibility Checker
The full report: crawler access, readability, rendering gap, site-wide audit, Common Crawl, and what ChatGPT, Claude, Gemini and Perplexity say about you.
- llms.txt Checker
Find and validate your llms.txt and llms-full.txt against the llmstxt.org format.
- Guide: how to get your website on ChatGPT
How the assistants choose which businesses to name, why most sites never appear, and what to fix first.
Want this fixed? See what Devs Core builds.
Server-rendered sites AI can read, clean crawler access, structured data, and AI agents that work inside the systems you already run.