How do I check if my website blocks AI crawlers?

Open yourdomain.com/robots.txt in a browser and look for lines like 'User-agent: GPTBot' or 'User-agent: ClaudeBot' followed by 'Disallow: /'. If you see those, you are telling the AI engines behind ChatGPT, Claude, and Google AI Mode not to read your site. Also check whether your CDN, like Cloudflare, has an AI-bot blocking feature switched on, because that can block crawlers even when your robots.txt looks fine.

Which AI crawlers matter for getting cited?

The crawlers that feed AI answers are GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), and Google-Extended (Gemini and Google AI Mode). If these are blocked, AI assistants cannot read your content, so they cannot recommend you when a customer asks them for a provider.

Does blocking AI training bots hurt my search visibility?

Blocking training is a separate choice from blocking citation. The damage happens when a blanket AI-bot block also stops the retrieval and search crawlers that decide who gets quoted in AI answers. You can refuse model training while still allowing the search and citation bots, so you stay quotable without donating your content to model training.

Why did my AI SEO efforts produce no results?

The most common silent cause is that the site blocks the AI crawlers it is trying to win over. Every optimization downstream, schema, content, structure, is wasted if the AI cannot fetch the page in the first place. Verify crawlability before anything else. It is the first check, not the last.

Is Your Website Blocking the AI That Should Recommend You?

Open your website's robots.txt right now and look for User-agent: GPTBot or User-agent: ClaudeBot followed by Disallow: /. If those lines are there, you are telling the exact AI engines behind ChatGPT, Claude, Perplexity, and Google AI Mode to stay off your site. They will not read you, so they cannot recommend you. I am writing this because I found that block on my own site, and I run an AI SEO system for a living.

Here is the honest version, because the lesson is worth more than the embarrassment.

What I found on my own site

I build AI revenue infrastructure for Las Vegas service businesses, and a core part of that is what the industry now calls Answer Engine Optimization: getting your business cited when someone asks an AI assistant "who's a good AI consultant in Las Vegas." My own automated SEO system runs that play on my own site daily.

Then the system flagged something I should have checked on day one: my site was returning a hard block to ClaudeBot, GPTBot, PerplexityBot, and Google-Extended. A managed security setting on my CDN had quietly switched on a blanket "block AI bots" rule. For weeks, I was optimizing content that the AI engines were never allowed to read.

I had built the whole strategy on a locked front door.

Why this happens to almost everyone

This is not a rare mistake. It is close to a default. Three forces line up to block AI crawlers without you ever deciding to:

CDN defaults. Cloudflare and similar networks now ship a one-click "Block AI bots" feature, and it is easy to enable during a security pass without realizing it also blocks citation and search crawlers, not just training bots.
Old robots.txt advice. A wave of 2024 guidance told everyone to add Disallow: / for GPTBot to "protect your content." That advice predates AI search being a real traffic source, and it now quietly removes you from AI answers.
Nobody checks crawlability first. Most SEO work starts with keywords and content. The question of whether the machine can even fetch the page gets assumed, not verified.

The result: businesses pour effort into being found by AI while their own configuration forbids it.

How to check yours in two minutes

Run this list:

Open yourdomain.com/robots.txt. Search for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot. Any Disallow: / under those is a block.
Check your CDN. In Cloudflare, look under Security for any "Block AI bots" or "Block AI Scrapers and Crawlers" setting, and the "Manage robots.txt" option.
Test enforcement, not just advice. A site can return a 403 to AI crawlers at the network layer even when robots.txt looks open. The robots file is a request; a firewall rule is a wall.
Decide training versus citation separately. It is reasonable to refuse model training. It is self-defeating to also block the search and retrieval bots that decide who gets quoted.

What this means for a Las Vegas business

When a potential customer asks ChatGPT or Google AI Mode "who should I hire for AI work in Las Vegas," the assistant answers from the sites it is allowed to read. If your competitors are readable and you are blocked, you are invisible at the exact moment of intent, no matter how good your site looks to a human.

The businesses that win the next year are not the ones with the most pages. They are the ones the AI is allowed to read, finds worth quoting, and names.

How I fixed it, and what I changed permanently

I removed the block so the citation crawlers return a clean 200, kept a deliberate stance on training, and reindexed the site so the engines recrawl. Then I did the part that matters more than the fix: I made crawlability the first check my system runs on any site, every day, with an alert the same day if a block ever returns. The failure that cost me weeks is now a check that runs before sunrise.

That is the difference between a tool and a system. A tool does the task. A system catches the thing you forgot to ask.

If you want to know whether your own site is quietly blocking the AI that should be recommending you, that is exactly the kind of gap my free AI Revenue Audit surfaces. Two minutes to check the front door is worth more than a year of optimizing behind a locked one.

Is Your Website Blocking the AI That Should Recommend You?