AI search access: does robots.txt block ChatGPT?
ChatGPT, Claude and Perplexity have their own search crawlers that read websites so they can show and link them in their answers. One line in robots.txt can shut them out without anyone noticing. The free check reads robots.txt and tells you which services are blocked.

yourcompany.com
- UptimeResponds in 240 ms
- CertificateValid for 74 days
- DomainExpires in 211 days
- EmailMissing DMARC
- Links and images2 broken links
- Search accessHomepage can be indexed

What we check
Everything is checked from the outside, the way a visitor or a search engine sees your website.
Crawlers for AI answers
We check whether OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude) and PerplexityBot (Perplexity) may read the homepage under robots.txt.
Blanket blocks
A rule such as Disallow: / for all crawlers also shuts out AI search. It counts, unless the search crawlers have a rule of their own that lets them in.
Training does not count
Blocks for crawlers that only collect training data, such as GPTBot, ClaudeBot and Google-Extended, do not affect the result.
Example result
This is what it can look like when you run the free check. Each row says what we found.
yourcompany.com
- ChatGPTBlocked
- ClaudeBlocked
- PerplexityBlocked

How to fix it
You can do this yourself, or pass it on to whoever looks after your website.
Decide whether the block is intended
Some people do not want to appear in AI answers. Then the block should stay. If you do want to appear, read on.
Open robots.txt
Type your address followed by /robots.txt in the browser, for example yourcompany.com/robots.txt. Look for User-agent: * followed by Disallow: /, or lines that name OAI-SearchBot, Claude-SearchBot or PerplexityBot.
Let the search crawlers in
Add a group with the lines User-agent: OAI-SearchBot, User-agent: Claude-SearchBot and User-agent: PerplexityBot, followed by Allow: /. You can still block training crawlers one by one.
Change it where robots.txt is made
In WordPress, robots.txt is often generated by an SEO plugin; in Wix and Squarespace by the platform. If you cannot find the setting, ask whoever looks after the website or your web host.
Run the check again
Run the free check after the change. OpenAI says it can take about a day before its systems follow an updated robots.txt.
How the check works
How we check, and how to read the result.
Search crawlers and training crawlers
AI companies run separate crawlers for different purposes. OpenAI’s OAI-SearchBot is used to show websites in ChatGPT’s search features, while GPTBot collects data to train its models. Anthropic has Claude-SearchBot for search and ClaudeBot for training. Perplexity says PerplexityBot surfaces and links websites in search results and is not used for training. So you can block training and still allow search.
What a block does
OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says blocking Claude-SearchBot may make a site less visible and less accurately represented in search results.
Crawlers that fetch on request
When a user asks ChatGPT or Perplexity to open a specific page, ChatGPT-User and Perplexity-User are used. OpenAI says robots.txt rules may not apply to them, and Perplexity that its fetcher generally ignores robots.txt. That is why they are not part of the check.
Google’s AI Overviews are part of Google Search. Google-Extended only controls whether content may be used for Google’s AI models and, according to Google, does not affect inclusion in Google Search. Googlebot access is shown on the Web protection and search access row in the report.
How we read robots.txt
We fetch robots.txt for the homepage’s final address and follow the robots.txt standard, RFC 9309: a crawler follows the group with its own name, and otherwise the group for all crawlers. If there is no file, everything counts as allowed. If the file cannot be read, for example because a firewall stops us, the result is unknown.
With monitoring
With monitoring the check runs every day and the result shows in your account. We do not email about it, because a block is often a deliberate choice.
Frequently asked questions
Should I block AI crawlers?
That is your choice. If you do not want your content used for training, you can block the training crawlers and still let the search crawlers in, so the site can be shown and linked in answers.
Why is my site blocked when I never changed anything?
The block often comes from a plugin, a platform setting or a template that blocks every crawler except Google. It can also be a block left over from when the site was built.
Is it enough that Google can read the page?
For Google Search and Google’s AI Overviews, yes. ChatGPT, Claude and Perplexity use their own search crawlers, which only follow the rules for all crawlers or rules with their own names.
How long before a change takes effect?
OpenAI says about a day for search. The other services give no time. The free check reads the new file right away.

One price, everything included
The same monitoring for every website. Pay monthly or yearly.
First month free
$9per month
Per website, including tax
$0 for the first month, then the regular price. New customers, monthly plans. Card required; the code START is added for you.
- No lock-in
- Cancel anytime
- 14-day right of withdrawal
What's included
- Uptime and error pages checked every minute, around the clock
- Certificate, domain and security notices daily
- Links, images, email settings and cookies weekly
- Plain-English emails with a suggested next step
- History in your account and a monthly report
You see the full order before you pay. Or run a free check first
First billing month $0. Then $9 per website per month, including applicable taxes. Renews automatically monthly until canceled. Cancel before renewal in Manage billing in your account or via info@sitedolphin.com. Businesses can add a VAT ID at checkout.
Secure checkout via Stripe

Check your website for free
No account and nothing to install.