Start with robots.txt and look for the AI user agents by name. But a clean file proves little on its own. Blocks often live above it—in server configuration, a CDN, or a security rule—invisible to anyone without log access. Confirming access means reading server logs, not reading robots.txt.
First, the part you can check
Open yoursite.com/robots.txt in a browser. It's a plain text file; anyone can read it.
You're looking for user-agent names. As of mid-2026, the ones that matter most:
OpenAI runs three separate crawlers, each controlled independently:
GPTBot— model trainingOAI-SearchBot— indexing for ChatGPT searchChatGPT-User— fetches a page when a user's question requires it
Anthropic documented its own three-way split in February 2026:
ClaudeBot— model trainingClaude-SearchBot— indexing for Claude's search answersClaude-User— user-initiated fetches
Perplexity runs two:
PerplexityBot— indexingPerplexity-User— retrieval for a live query
Google separates its AI opt-out from search:
Google-Extended— controls use in Google's generative products. It does not affect normal Google Search ranking; blocking it doesn't hurt your blue-link rankings.
The distinction that trips people up
Blocking the training crawler does not block the search crawler.
This is the single most common misunderstanding, and it cuts both ways. A firm that added Disallow for GPTBot a year ago to opt out of AI training has not thereby removed itself from ChatGPT search — OAI-SearchBot is a separate agent. And a firm that blocked broadly, using a copied-and-pasted "block all AI" snippet from a blog post, may have quietly removed itself from the answer engines it most wanted to appear in.
Anthropic's own documentation is explicit that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." That's the vendor telling you the trade-off in plain terms.
So the check isn't just "are these names present." It's: are the right ones allowed, and does the pattern match what we actually intended? Many sites have a robots.txt reflecting a decision made two years ago under different assumptions, by someone no longer involved.
Why a clean robots.txt still proves almost nothing
Here is the part most guides leave out.
robots.txt is a request. It sits at the application layer and asks well-behaved crawlers to respect it. It is not the only place a request can be refused — and it's the only one you can read from outside.
A crawler can be stopped by:
- Server configuration. An nginx or Apache rule that rejects a user agent before the request ever reaches WordPress.
- A CDN or WAF. Cloudflare and similar services offer one-click AI-bot blocking, sometimes on by default, often enabled by a host without the site owner being told.
- A security plugin. Rate limiting or bot rules that catch legitimate crawlers alongside malicious ones.
- Hosting-level defaults. Some managed hosts ship with AI-crawler blocking enabled.
None of these appear in robots.txt. Your file can list every AI crawler with Allow: / while the server returns 403 to all of them.
We know this one from the inside: our own site had ClaudeBot blocked at the nginx level, in a Cloudways-managed configuration we hadn't written and couldn't see from the front end. The robots.txt was immaculate. The block was three layers above it. We found it in the logs, which is the only place it was visible.
The only check that actually confirms access
Server access logs. Specifically: has each crawler hit your site, when, what did it request, and what status code did you return?
If you have log access, a single grep tells you most of it:
grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|perplexitybot|google-extended" access.logWhat you're reading for:
- Present, returning 200 — that crawler is reaching you. Confirmed.
- Present, returning 403 or 429 — something is refusing it. Now you know to hunt.
- Absent entirely — ambiguous. Either it's blocked upstream, or it hasn't come by. Newer agents like
Claude-SearchBotmay simply not have appeared yet on a small site.
That last case is worth naming honestly, because it's the one that generates false alarms. Absence is not proof of blocking.
What you can and can't conclude
Verified — What robots.txt says. It's a public file and you read it yourself.
Unconfirmed — Whether crawlers can actually reach you. Without server logs, you're inferring access from the absence of a block in the one file you can see, and that inference is unreliable in both directions. This is the single most common gap between what a site owner believes about their AI visibility and what's true.
Pending — Whether an absent crawler is blocked or simply hasn't visited. Resolving that needs a longer log window, and sometimes a deliberate fetch test.
If you don't have log access
Ask your host. On most managed WordPress hosting, access logs are available through the control panel or on request, even if they're not obvious. If your host won't provide them, that's worth knowing for its own sake — you're operating a business asset you can't inspect.
Crawler access is the foundation everything else in AI visibility sits on: a site an engine can't reach can't be quoted, cited, or recommended, no matter how good the content is. The full symptom map is at how do I tell if my website is losing me clients.