Are you blocking GPTBot without knowing it?

This one is a confession. We scanned our own founder's portfolio site during development, and the AI visibility report came back ugly: GPTBot blocked, ClaudeBot blocked, robots.txt full of AI disallow rules nobody remembered writing. The site looked perfect in a browser. To an AI crawler it did not exist.

The culprit was two Cloudflare toggles

Nobody had written those rules by hand. Cloudflare had, from two settings:

• Block AI bots. A security toggle that returns 403 to known AI crawler user agents on every page. It ships in an aggressive mode on many zones.
• Managed robots.txt. A feature that injects a Cloudflare-written robots.txt block, including disallow rules for GPTBot, ClaudeBot, and Google-Extended, over whatever your own file says.

Both are well-intentioned defaults for publishers who do not want AI training on their content. For a business that wants to be recommended by AI assistants, they are invisibility switches.

How to test your own site in two minutes

Fetch your homepage twice: once normally, once with an AI crawler user agent such as GPTBot. If the normal request returns 200 and the bot request returns 403 or a challenge page, you are blocking. Then read yourdomain.com/robots.txt and look for disallow rules naming GPTBot, ClaudeBot, PerplexityBot, or Google-Extended that you did not write.

The fix and the receipt

On the site in question we turned off the AI-bot block and the managed robots.txt in the Cloudflare dashboard. GPTBot went from 403 to 200 on the next request, the injected disallow rules disappeared, and the AI visibility score climbed once the access failures cleared, with the robots.txt fail dropping out of the top three issues entirely.

Our scan runs the crawler access test for all four major AI bots on every run, including the Cloudflare-challenge case. If you only check one AI visibility item this month, check this one. Nothing else matters while the door is locked.

Quick answers

How do I know if GPTBot is blocked on my site?

Request a page with GPTBot's user agent and compare it to a normal request, and read your robots.txt for GPTBot rules. A 403 or challenge page for the bot means it is blocked. RankMerlin's free scan runs this test for four AI crawlers.

Should I allow AI crawlers on my website?

For most businesses, yes. Blocked AI crawlers mean AI assistants cannot read or cite your site, so you disappear from AI answers. Block them only if you have a deliberate content-protection reason.