AI crawlers blocked by robots.txt

This warning appears when your robots.txt file blocks access to well-known AI bots, including the crawlers used by ChatGPT, Claude, and Perplexity. Here's what that means and what you can do about it.

What this error means

Your robots.txt file is the first point of contact between your site and any automated crawler, whether a traditional search engine or an AI agent. It declares rules per user-agent that state which parts of the site each one is allowed to visit. We detected that one or more rules in your robots.txt block, at the root ("Disallow: /"), user-agents identified as AI bots: crawlers used to train language models or to power answer engines that cite sources in real time, such as GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, among others. This block can come from a rule specific to that bot, or simply be inherited from the generic "User-agent: *" group, which applies to any crawler without a rule of its own. It's not uncommon for these blocks to exist without anyone having put them there on purpose: many templates and robots.txt generators block everything by default, or someone added a broad rule years ago for a different reason and it was never revisited.

Why it matters to fix it

Whether these bots being blocked matters depends entirely on your goals, so it's worth understanding what's at stake before changing anything.

Visibility in AI answers

If an assistant like ChatGPT or Perplexity can't crawl your site, it can't cite it as a source when someone asks about topics related to your content.

Language model training

Blocking bots like GPTBot or CCBot prevents your content from being used to train future models, which may be exactly what you want if you're protecting original content.

Operation by AI agents

More and more automated agents browse the web on a user's behalf (comparing prices, filling forms, summarizing information). If these bots are blocked, your site is left out of that ecosystem.

Consistency with your actual strategy

The issue isn't the block itself, but that it happened without a deliberate decision: it can go unreviewed for years and no longer reflect what you actually want for your site.

How to fix it step by step

Fixing this isn't about blindly removing rules, but about deciding, on purpose, which bots you want in and which you don't.

Step 1

Identify which bots are blocked

Check the list of affected user-agents shown in this warning's detail.

Where to apply it: In this issue's table, "Blocked AI bots" column.

Step 2

Decide whether the block is intentional

If you want to protect your content from being used to train AI models, the block may be correct as it is.

Where to apply it: No action needed if this is a deliberate choice.

Step 3

Open your robots.txt and locate the rule

Look for the "User-agent" group matching each bot you want to unblock, or the "User-agent: *" group if it doesn't have its own.

Where to apply it: At https://yourdomain.com/robots.txt.

Step 4

Adjust or remove the "Disallow: /"

To allow access, remove the line blocking the root in that group, or add a specific exception for that user-agent.

Where to apply it: In your server's or CMS's robots.txt file.

Does your site have this problem?

Audit your website for free and discover if this and other SEO errors are affecting your ranking.

See all the errors we detect