SiteDolphin

SiteDolphin / What we watch

AI search access: does robots.txt block ChatGPT?

ChatGPT, Claude and Perplexity have their own search crawlers that read websites so they can show and link them in their answers. One line in robots.txt can shut them out without anyone noticing. The free check reads robots.txt and tells you which services are blocked.

Free, no account needed. Nothing to install on your site.

yourcompany.com

Free check · done 9:41 am

2 things to fix
  • UptimeResponds in 240 ms
  • CertificateValid for 74 days
  • DomainExpires in 211 days
  • EmailMissing DMARC
  • Links and images2 broken links
  • Search accessHomepage can be indexed
Example result from the free check

Works with any public website, for example

WordPressWixSquarespaceJoomlaDrupaland more

What we check

Everything is checked from the outside, the way a visitor or a search engine sees your website.

  • Crawlers for AI answers

    We check whether OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude) and PerplexityBot (Perplexity) may read the homepage under robots.txt.

  • Blanket blocks

    A rule such as Disallow: / for all crawlers also shuts out AI search. It counts, unless the search crawlers have a rule of their own that lets them in.

  • Training does not count

    Blocks for crawlers that only collect training data, such as GPTBot, ClaudeBot and Google-Extended, do not affect the result.

Example result

This is what it can look like when you run the free check. Each row says what we found.

yourcompany.com

Free check · done 9:41 am

Blocked for ChatGPT, Claude and Perplexity
  • ChatGPTBlocked
  • ClaudeBlocked
  • PerplexityBlocked
A made-up example. Here robots.txt lets Googlebot in but blocks every other crawler, which includes the crawlers for AI answers.

How to fix it

You can do this yourself, or pass it on to whoever looks after your website.

  1. Decide whether the block is intended

    Some people do not want to appear in AI answers. Then the block should stay. If you do want to appear, read on.

  2. Open robots.txt

    Type your address followed by /robots.txt in the browser, for example yourcompany.com/robots.txt. Look for User-agent: * followed by Disallow: /, or lines that name OAI-SearchBot, Claude-SearchBot or PerplexityBot.

  3. Let the search crawlers in

    Add a group with the lines User-agent: OAI-SearchBot, User-agent: Claude-SearchBot and User-agent: PerplexityBot, followed by Allow: /. You can still block training crawlers one by one.

  4. Change it where robots.txt is made

    In WordPress, robots.txt is often generated by an SEO plugin; in Wix and Squarespace by the platform. If you cannot find the setting, ask whoever looks after the website or your web host.

  5. Run the check again

    Run the free check after the change. OpenAI says it can take about a day before its systems follow an updated robots.txt.

How the check works

How we check, and how to read the result.

Search crawlers and training crawlers

AI companies run separate crawlers for different purposes. OpenAI’s OAI-SearchBot is used to show websites in ChatGPT’s search features, while GPTBot collects data to train its models. Anthropic has Claude-SearchBot for search and ClaudeBot for training. Perplexity says PerplexityBot surfaces and links websites in search results and is not used for training. So you can block training and still allow search.

What a block does

OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says blocking Claude-SearchBot may make a site less visible and less accurately represented in search results.

Crawlers that fetch on request

When a user asks ChatGPT or Perplexity to open a specific page, ChatGPT-User and Perplexity-User are used. OpenAI says robots.txt rules may not apply to them, and Perplexity that its fetcher generally ignores robots.txt. That is why they are not part of the check.

Google

Google’s AI Overviews are part of Google Search. Google-Extended only controls whether content may be used for Google’s AI models and, according to Google, does not affect inclusion in Google Search. Googlebot access is shown on the Web protection and search access row in the report.

How we read robots.txt

We fetch robots.txt for the homepage’s final address and follow the robots.txt standard, RFC 9309: a crawler follows the group with its own name, and otherwise the group for all crawlers. If there is no file, everything counts as allowed. If the file cannot be read, for example because a firewall stops us, the result is unknown.

With monitoring

With monitoring the check runs every day and the result shows in your account. We do not email about it, because a block is often a deliberate choice.

Frequently asked questions

Should I block AI crawlers?

That is your choice. If you do not want your content used for training, you can block the training crawlers and still let the search crawlers in, so the site can be shown and linked in answers.

Why is my site blocked when I never changed anything?

The block often comes from a plugin, a platform setting or a template that blocks every crawler except Google. It can also be a block left over from when the site was built.

Is it enough that Google can read the page?

For Google Search and Google’s AI Overviews, yes. ChatGPT, Claude and Perplexity use their own search crawlers, which only follow the rules for all crawlers or rules with their own names.

How long before a change takes effect?

OpenAI says about a day for search. The other services give no time. The free check reads the new file right away.

One price, everything included

The same monitoring for every website. Pay monthly or yearly.

First month free

$9per month

Per website, including tax

$0 for the first month, then the regular price. New customers, monthly plans. Card required; the code START is added for you.

  • No lock-in
  • Cancel anytime
  • 14-day right of withdrawal

What's included

  • Uptime and error pages checked every minute, around the clock
  • Certificate, domain and security notices daily
  • Links, images, email settings and cookies weekly
  • Plain-English emails with a suggested next step
  • History in your account and a monthly report
Try it free for a month

You see the full order before you pay. Or run a free check first

First billing month $0. Then $9 per website per month, including applicable taxes. Renews automatically monthly until canceled. Cancel before renewal in Manage billing in your account or via info@sitedolphin.com. Businesses can add a VAT ID at checkout.

VisaMastercardAmerican ExpressSecure checkout via Stripe

Check your website for free

No account and nothing to install.

First month free Then $9/month · cancel anytime