Cloudflare changes its defaults on September 15, 2026
Cloudflare is switching offAI crawlers on Sept 15.Are you still visible to AI?
Cloudflare fronts more than one in five domains. From September 15 its new default blocks the AI agents that fetch your pages the moment a buyer asks ChatGPT, Perplexity or Claude about you. There is no email when it happens. Your citations just stop. Test your site, it takes ten seconds.
Free · no email · we fetch your homepage as each AI crawler, live
What actually changes
Three kinds of AI traffic.Two get the door shut.
Cloudflare now sorts every AI bot into Search, Agent or Training. On any page that shows an ad, the new default lets Search through and blocks the other two. Cloudflare’s reasoning: an ad means a person was meant to land there.
Search
Still allowedCrawlers that collect and index your content so an engine can answer questions about it later. OAI-SearchBot, PerplexityBot, that family.
Agent
Blocked by defaultAutomated fetches made in real time on a person's behalf. When someone asks ChatGPT or Perplexity about your product and it goes to read your page, that is Agent traffic.
Training
Blocked by defaultCrawlers taking your content to train or fine-tune a model. GPTBot, ClaudeBot, and the rest of the training fleet.
The part most people will miss. Crawlers that do both Search and Training count as multi-purpose. Cloudflare names Googlebot, Applebot and Bingbot, and says they get blocked when Training is disabled unless you opt out in Security settings before the 15th. Who it applies to: new domains, new sites added by existing customers, and Free-plan sites. Paid plans with settings already saved keep them.
Why this hits brands harder
A default built for publishers.Backwards if you sell something.
Cloudflare is protecting sites that sell pageviews. By its own crawl data, as reported by TechCrunch, Anthropic’s crawler fetched about 11,000 pages for every visitor it sent back, and OpenAI’s about 1,700. For a publisher, blocking that is rational.
If you sell a product, the AI answer is the visit. The Agent fetch that just got blocked is the exact moment a buyer asked about you. The engine does not wait. It answers with whoever it could reach.
“Which platform should a 40-person team use to track AI visibility?”
“Which platform should a 40-person team use to track AI visibility?”
The quiet part
Nobody sends an email.You find out from a slow quarter.
A block at the edge leaves no trace in your analytics. Google Search Console does not show it. Your robots.txt still looks fine. The only signal is AI referrals and mentions tapering over weeks, and by then a competitor has been the answer for a while.
PageLens watches your citations across six engines every day and tells you the day they move. Cloudflare decides whether AI can read you. PageLens tells you whether AI still recommends you, and what to do when it stops.
Before the 15th
Four things to do this week.
Test your site, today
The check above fetches your homepage as seven AI crawlers. If any come back blocked, that is already happening to your buyers' questions.
Decide on purpose, not by default
In Cloudflare, allow the Agent and Search crawlers that send you buyers, and match it in robots.txt so both layers agree. Block training if you want. Just choose.
Measure what AI actually says
Reachable is not the same as recommended. The free audit asks the engines your buyers' real questions and shows who gets named.
Keep feeding the answer
PageLens tracks your citations across six engines, writes the pages AI wants to quote, publishes them on your own domain, and pushes your story onto the sources AI trusts.
Set it in Cloudflare
Five minutes in the console.Before the 15th.
Two screens matter. The category switches decide the default for whole classes of bots. The crawler list lets you override one bot at a time. Labels can differ a little by plan, and Cloudflare is still rolling the category controls out, so if a row is missing, the per-crawler list does the same job.
- 1
Open the AI traffic categories
In the Cloudflare dashboard pick your domain, then go to Security → Settings and find the AI traffic options: Search, Agent and Training. Cloudflare says every existing customer can set these now, ahead of the switch.
- 2
Allow Agent. Keep Search allowed.
Agent is the live fetch when a buyer asks ChatGPT, Perplexity or Claude about you. Set it to Allow. Training is your decision. If you block it, use the opt-out Cloudflare provides so multi-purpose search crawlers (Googlebot, Bingbot, Applebot) are not caught in the same net.
- 3
Check the crawler list
Open AI Crawl Control (direct link: dash.cloudflare.com/?to=/:account/:zone/ai), then the Crawlers tab. In the Actions column set Allow for ChatGPT-User, OAI-SearchBot, Perplexity-User, PerplexityBot and Claude-User. Block or Charge whatever you do not want.
- 4
Make robots.txt agree
The Directives tab shows what Cloudflare serves at the edge. If you turn on managed robots.txt, it prepends a Content Signals line that says search is fine and training is not. Whatever you choose, your own robots.txt should say the same thing. Generate a matching one.
- 5
Look above these settings too
Bot Fight Mode and WAF custom rules sit on top. If a crawler still gets a 403 after all this, that is where it is coming from. Re-run the test at the top of this page to confirm.
Steps follow Cloudflare’s July 1, 2026 announcement and its AI Crawl Control documentation. The panels on the right are an illustration of the layout, not screenshots of your account.
- defaultAllow
Search
indexes your pages to answer later
- change thisAllow
Agent
fetches live when a buyer asks
- your callBlock
Training
learns from your content
Keep multi-purpose search crawlers (Googlebot, Bingbot, Applebot) allowed while Training is blocked
- ChatGPT-UserAgentAllow
- OAI-SearchBotSearchAllow
- Perplexity-UserAgentAllow
- PerplexityBotSearchAllow
- GPTBotTrainingBlock
Questions
The details.
- What exactly changes on September 15, 2026?
- Cloudflare splits AI bot traffic into three categories: Search, Agent, and Training. On pages that display advertising, Agent and Training are blocked by default while Search stays allowed. Cloudflare announced this on July 1, 2026 and the new defaults take effect on September 15.
- Does it apply to my site?
- Cloudflare's announcement applies the new defaults to new domains onboarding to Cloudflare, and coverage of the change says it also covers sites added by existing customers and existing Free-plan sites. Paid customers who already configured their AI traffic settings keep them. The test at the top of this page tells you what is true for your domain right now, whichever plan you are on.
- It says defaults. Can I change it?
- Yes. Cloudflare says site owners can set these controls in their Security settings, including before September 15. The risk is not that you cannot change it. The risk is not knowing it happened.
- Which one actually hurts AI visibility, Agent or Training?
- Agent, immediately. That is the live fetch an engine makes when a buyer asks about you. Block it and the engine answers from whatever else it can reach, usually a competitor or a review site. Training is slower and affects what future models know about you at all.
- What about Googlebot and Bingbot?
- Cloudflare notes that multi-purpose crawlers which do both Search and Training, and it names Googlebot, Applebot and Bingbot, will be blocked when Training is disabled unless you opt out in your settings. That is worth checking twice.
- Does PageLens replace Cloudflare?
- No. Keep Cloudflare, and set it deliberately. PageLens is the layer that tells you whether AI can still read you, whether it still cites you across ChatGPT, Gemini, Perplexity, Grok, Copilot and Google AI Mode, and what to publish when it does not.
Don't find outfrom a slow quarter.
See what ChatGPT, Gemini, Perplexity, Grok and Copilot say about your brand today, and who they name instead. Free, about thirty seconds, no signup.
Free, takes about 30 seconds
Paste any domain and see how AI describes you today.