--- title: "ChatGPT search can read your pages if OAI-SearchBot is allowed" description: "OpenAI's search crawler is OAI-SearchBot. Check robots.txt and Cloudflare settings so public pages stay readable. Access does not guarantee a citation." url: "https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/" lang: "en" published: "2026-10-20" modified: "2026-10-20" author: "Cal Hewitt" tags: ["ai-search","generative-engine-optimization","small-business"] image: "https://webleveling.com/og/blog-chatgpt-search-cloudflare-pilot-more-pages.jpg" alternates: es: "https://webleveling.com/es/blog/chatgpt-busqueda-oai-searchbot-robots-txt.md" --- # ChatGPT search can read your pages if OAI-SearchBot is allowed OpenAI's search crawler is OAI-SearchBot. Check robots.txt and Cloudflare settings so public pages stay readable. Access does not guarantee a citation. ![A closed laptop beside a small brass key and a stack of blank cards on a plain wooden desk.](https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/chatgpt-search-cloudflare-pilot-featured.jpg) The ChatGPT search Cloudflare pilot made headlines this week, and now you are wondering whether a setting on your own site could be quietly keeping customers from finding you. Usually the answer is simple. ChatGPT search can read a public page when OpenAI's search crawler, OAI-SearchBot, is allowed through your robots.txt file and through whatever your host or Cloudflare account puts in front of the site. The pilot is real, but it is a test run with a limited group of sites. It does not mean every site on Cloudflare is now being read, and allowing the crawler does not guarantee that ChatGPT will cite you. The good news is that you can run the first check yourself in a few minutes. Often one line in a file or one switch in a dashboard is the whole fix. **Key Takeaways** **Pilot, not a switch for everyone** Cloudflare and OpenAI began a research pilot in July 2026, and Cloudflare's October 15, 2026 update says a cohort of about 1% of its network took part. **One crawler decides search** OAI-SearchBot is the crawler OpenAI documents for ChatGPT search. GPTBot is a separate crawler for potential training use, so you can treat them differently. **Check four places** Your robots.txt file, your Cloudflare bot and AI settings, whether the page loads its main text without a login or script, and whether your sitemap lists the pages you care about. **Access is not a citation** OpenAI says allowing OAI-SearchBot helps a site be found and cited, but placement is not guaranteed. Watch for referrals tagged utm\_source=chatgpt.com. ## What did Cloudflare and OpenAI actually announce? Cloudflare and OpenAI launched a research pilot in July 2026, announced in [Cloudflare's July 8, 2026 press release](https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/). Cloudflare supplies network signals, such as how fresh content is, how good the traffic looks and when pages change. OpenAI's search crawler uses those signals to find useful pages faster, and OpenAI independently evaluates and ranks what it finds. On October 15, 2026, Cloudflare published [an update on the pilot's results](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/). It says a participating cohort of approximately 1% of Cloudflare's network helped OpenAI find and index hundreds of millions of new pages. Two limits matter for you. This is a pilot, not a universal ChatGPT indexing product. And being hosted on Cloudflare does not mean your site is part of it. ![A closed laptop, a small desk lamp and a blank paper calendar on a clean wooden desk.](https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/pilot-calendar-desk.jpg) A pilot has a start date and a limited group. Your own settings still apply either way. ## Do I have to opt in or opt out for ChatGPT search? The pilot does not replace your robots.txt file or your own access controls. Those stay yours. Allowing ChatGPT search to reach your pages is a choice you make through them, not an automatic opt-in to every way OpenAI might use a page. That choice is easier to see once the three OpenAI crawlers are separated. [OpenAI's crawler overview](https://developers.openai.com/api/docs/bots) describes each one, and [Cloudflare's bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/) sorts them into categories you can switch on or off. The three OpenAI crawlers and how Cloudflare labels them | Crawler | What it does | Cloudflare category | | --- | --- | --- | | OAI-SearchBot | Surfaces websites in ChatGPT search | AI Search | | GPTBot | Potential training use | AI Crawler | | ChatGPT-User | Fetches a page when a person asks ChatGPT to, and is not used for automatic search crawling | AI Assistant | OpenAI's documentation says: "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." It also says: "ChatGPT-User is not used to determine whether content may appear in Search." So, if you want to appear in ChatGPT search, OAI-SearchBot is the one to allow. If you want to keep GPTBot out, that is a separate rule. ## What should I check first so ChatGPT search can read my pages? Start with the file that decides most of this. Everything below takes a few minutes and costs nothing. 1. **Open your robots.txt file.** Go to yoursite.com/robots.txt in a browser and look for OAI-SearchBot. Confirm that the pages you want read are not disallowed for it, and that no broad rule such as \`User-agent: \*\` followed by \`Disallow: /\` applies to everything. 2. **Check Cloudflare, if your site uses it.** In your Cloudflare account, open the AI bot settings and confirm the Search category is set to Allow rather than Block. Cloudflare's [Block AI Bots page](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) lists the options, which are Allow, Block, and Block on pages with ads. 3. **Look at the other rules that can stop a visit.** Cloudflare's [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) sits alongside WAF rules, bot management, rate limits and CAPTCHA or script challenges. Any one of those can return a block even when robots.txt says yes. 4. **Load a key page in a normal browser.** Confirm the main text appears without logging in, filling out a form or clicking anything. 5. **Check your XML sitemap.** Confirm it exists and lists the important pages, and that old or redirected addresses are not still in it. If you do not know whether your site is behind Cloudflare, ask your host or web person. A host can put Cloudflare in front of your site without you ever opening a Cloudflare account. ![A small wooden box of index cards with a rubber stamp and a blank notepad beside it.](https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/index-card-checklist.jpg) A short checklist, worked in order, covers the settings that usually decide access. ## How long until a robots.txt change takes effect? OpenAI says changes to robots.txt can take about 24 hours to be picked up in its systems. So, after you edit the file or change a Cloudflare setting, give it a day before you decide it did not work. The wait is also why it helps to change one thing at a time. If you loosen three rules in one afternoon and something still blocks the crawler, you will not know which change mattered. OpenAI's guidance on [allowing its web crawlers](https://help.openai.com/en/articles/20001243-advertiser-guidance-for-allowing-openai-web-crawlers) covers the same crawler and the same rule. ## What usually keeps ChatGPT search from reading a page? When a public page cannot be read, the cause is nearly always one of a short list. This order is a sensible way to troubleshoot, not a measured ranking of how common each one is. Common blocks, and what fixing each one involves | What is in the way | What the fix involves | Rough size of the job | | --- | --- | --- | | A broad robots.txt rule blocks OAI-SearchBot | Edit one file, then wait for crawlers to refresh | A few minutes | | A Cloudflare AI policy, bot rule, WAF rule or challenge returns a block | Change the policy or add an exception, then test in the logs | Short to moderate, depending on the rules | | The page needs a login, cookie or form to show anything | Publish a public version of the information | Moderate | | The main text only appears after JavaScript runs | Serve the text as plain HTML from the server | Developer work | | Redirects, duplicate addresses or stale sitemap entries | Clean up URLs and canonical tags | Moderate | One more cause sits outside any setting. A page can be fully open to the crawler and still not answer a question clearly. If your service page says little about what you do, who it is for and where, there is nothing useful for a search tool to quote. That part is content work. A site with sloppy crawl settings and thin pages needs both, and [search engine optimization](https://webleveling.com/services/search-engine-optimization/) is the usual home for the crawl and page side of it. ![A magnifying glass resting on a stack of blank sheets next to a small brass key.](https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/key-and-blank-pages.jpg) A page that is open to the crawler still has to say something worth quoting. ## Does letting ChatGPT search read my site mean I am allowing AI training? No. OpenAI separates search access from potential training use, and [Cloudflare lets you set them separately](https://developers.cloudflare.com/ai-crawl-control/reference/bots/) because it labels them as different categories. Allowing OAI-SearchBot does not allow GPTBot. If you have already decided how you feel about training, you can keep that rule in place and still allow search. The post on [how Cloudflare can disallow AI training while Googlebot stays welcome](https://webleveling.com/blog/cloudflare-disallow-ai-training-googlebot/) covers that setup in more detail. Also, ChatGPT-User is not an automatic crawler. It fetches a page when a person's action in ChatGPT calls for it, and OpenAI says it is not used to decide whether content may appear in search. Blocking or allowing it does not change whether your pages are eligible for search results. ## Will allowing OAI-SearchBot get my business cited? It makes you eligible. It does not promise a result. OpenAI says allowing OAI-SearchBot helps a site be found, surfaced, and clearly cited and linked in ChatGPT search, and it also says [placement is not guaranteed](https://help.openai.com/en/articles/12627856). ChatGPT ranks results using several factors meant to identify relevant and reliable information. So, treat crawler access as the floor, not the finish line. A readable page that clearly answers a customer's question is what gives a citation something to point to, and the post on [how to get cited by ChatGPT](https://webleveling.com/blog/how-to-get-cited-by-chatgpt/) goes through what that looks like. No one can promise a citation, a ranking position, traffic or leads just because the crawler is allowed. A crawler setting can open the door; it cannot decide who walks through. ![A small stack of folded paper cards beside a single upright blank sign on a plain shelf.](https://webleveling.com/blog/chatgpt-search-cloudflare-pilot-more-pages/eligible-not-guaranteed.jpg) Being readable puts a page in the running. It does not pick the winner. ## How can I tell whether ChatGPT is sending visitors? OpenAI says visits that come from ChatGPT include the parameter \`utm\_source=chatgpt.com\`. In Google Analytics, look for that source in your traffic acquisition reports or filter landing pages by it. Zero visits does not by itself prove you are blocked. Check the access settings first, wait the 24 hours after any change, and then keep watching over several weeks. If your analytics cannot separate these visits, or your tracking is unreliable, fix that before drawing conclusions. The post on [what a sitemap and robots.txt do for a small site](https://webleveling.com/blog/does-your-website-need-a-sitemap-and-robots-txt/) is a useful companion while you tidy up the files that guide crawlers. ## When is it worth paying someone to look at this? You can do the first check alone. Often a robots.txt line or a Cloudflare setting is the whole fix, and you do not need a package of any kind. Paid help earns its cost when the problem crosses several layers. That includes WAF rules, a host plus Cloudflare plus a plugin all setting their own rules, pages that only show text after JavaScript runs, unclear permissions, broken redirects, or analytics you cannot trust. For ongoing upkeep, such as rechecking settings after a plugin update or a host change, [website maintenance](https://webleveling.com/services/website-maintenance/) keeps those configurations from drifting. If the server is slow to answer crawlers, the post on [slow server response and AI crawlers](https://webleveling.com/blog/slow-server-response-and-ai-crawlers/) shows what to look for. **Quiz: Can ChatGPT search read your pages?** 1. Which OpenAI crawler is documented for surfacing sites in ChatGPT search? - GPTBot - OAI-SearchBot **(correct answer)** - ChatGPT-User - Googlebot 2. Does the Cloudflare and OpenAI pilot mean every Cloudflare site is read by ChatGPT search? - Yes, every site on Cloudflare is included automatically - Yes, but only paid Cloudflare plans - No, it is a pilot with a participating cohort, and your robots.txt and access controls still apply **(correct answer)** - No, because Cloudflare blocks OAI-SearchBot by default 3. You change a robots.txt rule to allow OAI-SearchBot. What should you expect? - An instant ChatGPT citation - A change that can take about 24 hours to be picked up, with no guarantee of a citation **(correct answer)** - A ranking in the top three results - A change that only works if you also allow GPTBot ## Frequently Asked Questions About ChatGPT Search Cloudflare Settings **Does ChatGPT read every Cloudflare website?** No. The pilot covers a participating cohort, which Cloudflare's October 15, 2026 update puts at about 1% of its network. Being on Cloudflare does not put a site in it. **Which OpenAI bot controls ChatGPT search?** OAI-SearchBot. GPTBot is the crawler OpenAI documents for potential training use, and ChatGPT-User fetches pages when a person asks it to. **Can Cloudflare block OAI-SearchBot?** Yes. Cloudflare AI policies, WAF rules, bot controls and challenges can all affect access, so check each one even when your robots.txt file allows the crawler. **Does robots.txt matter for ChatGPT search?** Yes. OpenAI says its crawlers respect robots.txt, and the pilot does not replace your robots.txt file or your own access controls. **Can I allow ChatGPT search but block GPTBot?** Yes. They are separate crawlers with separate rules, and Cloudflare lists them in different categories, AI Search and AI Crawler. **Does allowing OAI-SearchBot guarantee a citation?** No. OpenAI says allowing it helps a site be found and cited, and also says placement is not guaranteed. ## Moving Forward ChatGPT search can read your public pages when OAI-SearchBot is allowed and nothing else in the way turns it back, whether that is a broad robots.txt rule, a Cloudflare setting, a firewall or challenge, a login, or text that only appears after a script runs. The Cloudflare and OpenAI pilot is a test with a limited group, and your own rules still decide who gets in. Spend ten minutes on the robots.txt file and the Cloudflare settings, change one thing at a time, give it 24 hours, and then watch your analytics for utm\_source=chatgpt.com. That tells you whether the door is open and whether anyone is walking through it, without guessing. If the checks turn up something you would rather not untangle alone, [Web Leveling](https://webleveling.com/) can look at the layers with you. Our [generative engine optimization](https://webleveling.com/services/generative-engine-optimization/) work covers how your pages show up in AI answers, and if a server-level block is behind it, our [web hosting](https://webleveling.com/services/web-hosting/) team can sort that out too. We work with small and medium businesses across the country and overseas. [Tell us what your settings and analytics are showing](https://webleveling.com/contact/), and we will help you read them. ## Terms Words used in this post Tap a term to see what it means. **OAI-SearchBot.** The crawler OpenAI documents for surfacing websites in ChatGPT search. **GPTBot.** The crawler OpenAI documents for potential training use, handled separately from search. **ChatGPT-User.** A fetcher that loads a page when a person asks ChatGPT to, and is not used for automatic search crawling. **robots.txt.** A text file at the top level of a site that tells crawlers which pages they may visit. **WAF.** A web application firewall, a set of rules that can block or challenge traffic before it reaches a site. **Sitemap.** An XML file that lists the pages you want search tools to find. **Cohort.** A group of sites chosen to take part in a test. - [Google AI answer links move up: what your site can control](https://webleveling.com/blog/google-ai-answers-links-and-official-site-section/): Google says it is showing inline links higher in AI Mode and AI Overviews. It is testing an official-site section, and no owner can claim a spot in it. - [See How Your Business Shows Up in Google AI Mode](https://webleveling.com/blog/see-how-your-business-shows-up-in-google-ai-mode/): Open Google's Generative AI performance report, run a repeatable Google AI Mode for business check, and fix what Google says matters. Steps for owners. - [What Makes AI Search Trust a Business? A 3-Month Plan](https://webleveling.com/blog/what-makes-ai-search-trust-a-business/): AI search trusts a business it can identify, whose facts match everywhere, and that others confirm. See what that means and a plan for your next 3 months.