ChatGPT SEO: Check Crawler Access, Then Measure Recommendations

ChatGPT SEO covers two jobs: using ChatGPT as an assistant for ordinary search work, and optimizing so that ChatGPT's answers include your brand. The second job needs different checks and different measures of success, so decide which one you are doing before you spend a week on either.

ChatGPT cannot give you reliable search volume or competition figures. Treat it as a fast first draft and check every claim it writes. The rest of this page covers the second job, because that is where the checks are unfamiliar.

Eight of 47 well-known US-facing sites fully block OAI-SearchBot, the crawler that surfaces sites in ChatGPT search, and 15 of 47 fully block GPTBot, the crawler for model training. We read each site's robots.txt file, the plain text file at a domain's root that tells crawlers which paths they may fetch, on October 9, 2026.

OpenAI's crawler documentation separates three user agents (the names crawlers announce themselves with):

• OAI-SearchBot is used to surface websites in ChatGPT's search features. OpenAI's wording: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• GPTBot crawls content that may be used in training. "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
• ChatGPT-User visits pages when a user's request in ChatGPT triggers it.

Each row totals 47 sites. ChatGPT-User is shown for completeness. Because users trigger its visits, OpenAI says robots.txt rules may not apply, so those counts show intent, not enforcement.

This is our count, so here is how we made it. We attempted 52 well-known US-facing domains across publishers, retailers, software companies and SEO-industry sites. It is a convenience sample, not a random one. "Fully blocked" means a Disallow: / rule applies to the bot, either in a group naming it or through the User-agent: * fallback, with no Allow: /. "Path rules only" means some folders are off limits but the site is otherwise open. We could not see firewall or content delivery network (CDN) rules, only what each file declares.

We excluded 5 domains: Search Engine Land and Capterra returned a block page instead of a robots file, Best Buy returned no response, Stack Overflow returned an error status, and the Wikipedia domain's file belongs to its portal, not the encyclopedia. Stack Overflow's response was a block-all file, so counting it would add one to each "Fully blocked" figure.

Reuters blocks GPTBot through its wildcard fallback, not a rule naming it. The other 29 sites have path rules only on all three bots: Ahrefs, Asana, Backlinko, Business Insider, Etsy, G2, GitHub, The Home Depot, HubSpot, Investopedia, Moz, Neil Patel, NerdWallet, Notion, Salesforce, Semrush, Shopify, Slack, Stripe, Target, TechRadar, The Guardian, Trustpilot, Vercel, Walmart, The Washington Post, Wired, WordStream and The Wall Street Journal. You can see the pattern in files like The New York Times and Forbes.

Our reading is that the seven sites blocking GPTBot but not OAI-SearchBot (eBay, Forbes, Healthline, LinkedIn, Medium, Reuters, WebMD) opted out of training deliberately and stayed in search. Most businesses that want customers to find them should do the same: allow OAI-SearchBot, and treat GPTBot as a separate policy decision about training use. A blanket "block AI bots" rule written for one purpose can quietly remove you from the other.

Run two tests: read your robots.txt file for rules that apply to OAI-SearchBot, then confirm in your server or CDN logs that the crawler is being served real pages. Most sites can run both checks in a short session and settle the question of access. If you want the wider version of this exercise, our AI visibility audit guide takes the same idea from crawlability through to citation.

Does your robots.txt file allow OAI-SearchBot?

Open yourdomain.com/robots.txt, or run curl -s yourdomain.com/robots.txt | grep -i -A2 "oai-searchbot\|gptbot\|chatgpt-user\|user-agent: \*".

• Pass: no group naming OAI-SearchBot contains Disallow: /, and any User-agent: * group with Disallow: / is overridden by a group that names OAI-SearchBot with Allow: /.
• Fail: a group naming OAI-SearchBot contains Disallow: /, or a User-agent: * group contains Disallow: / and nothing names OAI-SearchBot.

Crawlers follow the group that names them and fall back to the * group only when none does. To fix a fail, add this and keep your other rules:

User-agent: OAI-SearchBot
Allow: /

OpenAI says its systems can take about 24 hours to adjust after a robots.txt change, so retest the next day.

Is your firewall or CDN turning OAI-SearchBot away?

Search your server or CDN logs for the last 30 days for requests whose user agent contains OAI-SearchBot. OpenAI's documented string includes OAI-SearchBot/1.4; +https://openai.com/searchbot.

• Pass: requests from that user agent receive 200 responses on your public pages.
• Fail: requests receive 403, 429 or a challenge page.
• Inconclusive: no requests at all. On a small or new site that can mean it has not been crawled yet, not that it is blocked. Retest after fixing robots.txt and after publishing new pages.

Passing both tests lets ChatGPT read your pages. It does not make ChatGPT choose them, which is what the next section measures.

Run a fixed panel of 10 buying questions, three runs each, and score the six questions that do not name your brand for mentions, citations and recommendations. That is 18 runs for your rates. The four questions that do name your brand are a separate check on what ChatGPT says about you.

Write each question as a full sentence, the way a buyer types it, not as a keyword string. Use a new or temporary chat each time so earlier conversation does not shape the answer. Keep the wording identical month to month and record the date. Record the ChatGPT model, logged-in status and whether search was on for every run, and keep them fixed. Citations come from web search, so run the panel with search on.

For each run, log: mentioned (yes or no), cited (yes or no), recommended (yes or no), competitors named, and domains cited. Then compute three rates, each out of 18 unbranded runs:

• Mention rate = runs with a mention ÷ 18
• Citation rate = runs with a citation ÷ 18
• Recommendation rate = runs with a recommendation ÷ 18

As an illustration of the arithmetic, 4 mentions, 1 citation and 3 recommendations give 22.2%, 5.6% and 16.7%. Answers vary between runs. Our rule of thumb: treat a change of one or two runs as noise and act on movement across several months.

The "domains cited" log matters most: it lists the pages ChatGPT links to for your category, which are your real competition for the answer. For the four branded questions, do not compute rates, because ChatGPT will repeat a name you typed. Read the answers instead for accuracy, tone and which competitors it pairs with you. Our guide to checking ChatGPT brand recommendations walks through reading those logs.

Is ChatGPT sending visits to your site?

In Google Analytics 4 (GA4), open Reports, then Acquisition, then Traffic acquisition, and look for sessions whose source contains chatgpt.

• Pass: you see sessions from a chatgpt.com source, and the count rises as your panel scores rise.
• Fail: none appear while your citation rate is above zero, which suggests visits are being filed elsewhere.

Some visits arrive without referrer information and land in Direct. Count the visible chatgpt.com sessions as a floor, not a total.

Yes, but mostly through ordinary SEO: the one study that tested this found GEO scores did not predict discovery, and the access and measurement checks on this page are the only ChatGPT-specific steps with direct support. GEO scores rate how well a page is written for AI answers.

The Discovery Gap study tested 112 startups drawn from Product Hunt's top 500 products across 2,240 queries. ChatGPT (the gpt-4o-mini model) recognized 99.4% of products by name but surfaced 3.32% in discovery-style queries. Perplexity managed 94.3% and 8.29%. GEO scores showed no correlation with discovery. For Perplexity, community presence correlated with discovery at r = +0.395 and referring domains, the number of other sites linking to you, at r = +0.319 (r runs from -1 to +1, where 0 means no relationship). The study reports no such correlation for ChatGPT.

Treat that as a hint, not a verdict. It is one preprint (a paper not yet peer reviewed), it covers young startups, and it tested a model, so results for established brands or for ChatGPT with live search may differ.

Google says the same about its own AI features, meaning AI Overviews (the AI-written summaries above some results) and AI Mode (its conversational search mode): "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." For that side of the problem, see how to rank in Google AI Overviews.

Our recommendation, in order:

1. Pass both access tests above.
2. Publish pages that answer your panel's questions with specifics a buyer can use, such as real prices, comparisons and original data, because those are the pages worth citing.
3. Build presence where your buyers compare options: review sites, communities and trade press. The study links this to Perplexity visibility; for ChatGPT it is our judgment that it helps, not a measured result.
4. Re-run the panel monthly and keep what moves it.

Be wary of anyone who guarantees a placement in ChatGPT. Answers vary run to run, and OpenAI documents crawler access as the lever for search appearance, not a ranking formula.

Move from the spreadsheet to a tool when you need more than a few dozen prompts, more than one AI engine, or weekly or daily runs. The method is the same as the panel above, at a scale that is tedious by hand.

Before paying for a tracker, you can run our free ChatGPT visibility checker, which asks ChatGPT live, with search, one buyer query and shows the answer, the sources it cites and whether you are in it (3 free checks). Our free AI visibility audit on pagelens.ai takes about 30 seconds: paste a domain and see how AI describes you today.

PageLens.ai tracks the prompts you choose and reports mentions, share of voice (your mentions as a percentage of all brand mentions across your prompts), sentiment (whether answers describe you favorably) and citation sources. Our citation analytics show which pages AI models cite instead of yours, and our content engine publishes pages for the gaps it finds to your own domain.

As of October 2026, our plans are priced in US dollars per month:

• Launch, $299: 100 tracked prompts run weekly, tracking ChatGPT, Google AI Mode and Perplexity.
• Growth, $699: 100 prompts run daily, adding Gemini and Grok.
• Enterprise, $1,499: 200 prompts run daily across seven engines: ChatGPT, Claude, Gemini, Perplexity, Grok, Copilot and Google AI Mode.
• Agency, custom pricing: custom prompts and cadence across the same seven engines.

If you want the reasoning behind choosing a tracker, read how to track ChatGPT visibility reliably. If you would rather see your own prompts scored, Book a demo.