Every AI visibility problem has a boring version and an interesting version. The interesting version is about content, structure, and authority. The boring version is that the AI crawler tried to read your site, got a 403, and left. You cannot be cited by an engine that cannot fetch your pages — and this is the single easiest thing on the entire AI-visibility checklist to verify.
The bots that matter, and what each one does
People lump these together as "AI bots." They aren't the same, and blocking the wrong one costs you differently.
- GPTBot — OpenAI's crawler. Primarily gathers content for model training.
- OAI-SearchBot — OpenAI's search crawler. This is the one that feeds ChatGPT's browsing and citations. Blocking it removes you from ChatGPT's answers.
- ChatGPT-User — fetches a page live when a user's question requires it.
- ClaudeBot and Claude-SearchBot — Anthropic's crawlers, same split between training and search.
- PerplexityBot and Perplexity-User — indexing and live retrieval for Perplexity.
- Google-Extended — not a crawler at all. It's a permission token that controls whether Google may use content it has already crawled for Gemini and AI training. Blocking it does not remove you from Google Search. It may affect AI Overviews.
- Googlebot — regular Google indexing. AI Overviews depend on it.
The distinction that catches people out is training versus search. Blocking GPTBot is a legitimate editorial decision — plenty of publishers do it on principle. Blocking OAI-SearchBot is a visibility decision, and usually an accidental one. Many sites block both because a plugin offered a single checkbox labeled "block AI."
Why nobody notices
Here's the part that makes this a genuinely hidden problem: AI crawlers don't execute JavaScript. Google Analytics is JavaScript. So a bot can hit your site ten thousand times, get refused every time, and your analytics will show precisely nothing. There's no ranking to check. There's no missed call to count. The failure is completely silent.
Cloudflare's data shows how much traffic we're talking about. In June 2026, ClaudeBot accounted for roughly 19.8% of AI-bot traffic and GPTBot about 9.4% — and that ordering has flipped back and forth month to month as each lab runs training passes. The point isn't the ranking between them. It's that this is a substantial, continuous, invisible stream of requests to your server, and you have no idea how it's being answered.
How to check, in about ten minutes
Step one: read your robots.txt. Visit yoursite.com/robots.txt in a browser. Look for any Disallow: / under a user-agent you recognize from the list above. Also look for what isn't there — a missing bot is allowed by default, which is usually what you want.
A visibility-first posture looks roughly like this:
# Allow AI search & citation
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training — opt out here if you choose to
User-agent: GPTBot
Disallow: /That configuration says: cite me, but don't train on me. Reasonable people disagree about whether that's the right trade. What isn't defensible is making the choice by accident.
Step two: check the layer above robots.txt. This is where most real blocks live, and where nobody looks. Your CDN or firewall — Cloudflare, Sucuri, Wordfence, or a rule your host added — may be refusing AI crawlers regardless of what robots.txt says. Robots.txt is a request; a WAF rule is a wall.
We found exactly this on our own site. A single literal string match in an nginx configuration file — added by the host, not by us — was blocking Anthropic's crawler from reaching even robots.txt. The file said the bot was welcome. The server never let it read the file.
Step three: check your server logs. This is the only ground truth. Search your access logs for the user-agent strings above and look at the response codes. A wall of 403s tells you everything. If you can't get at logs, your host can.
What good looks like
Once the crawlers can reach you, three more things determine whether they can actually use what they find:
Your content must be in the HTML. AI crawlers don't run JavaScript. If your key text is injected client-side by a page builder or a framework, the bot sees an empty shell. View source — actual source, not the inspector — and search for a sentence from your page. If it isn't there, neither is your content, as far as these engines are concerned.
Your pages should be quotable. Answer-first structure: the direct answer near the top, the elaboration below. An engine lifting a passage wants a clean, self-contained one.
Your entity should be legible. Structured data telling the engine who you are, where you are, and what you do — connected to your off-site profiles. That's a longer subject, and we've written about it separately.
The uncomfortable part
We check this on every audit, and it is not rare to find a firm that has spent years and real money on content — while quietly serving a 403 to the engines they most wanted to reach. It costs nothing to fix. It costs a great deal to leave alone, and the cost never shows up on a report.
If you'd like us to check yours, that's what the GEO audit is for. Or read the whole list of what we look at in what we check.
Crawler share figures: Cloudflare Radar, June 2026 full-month data. Crawler names and their training-versus-search roles are published by OpenAI, Anthropic, Perplexity and Google; these change, so verify against current vendor documentation before editing robots.txt.