You read that Google is adding one robots.txt setting for saying no to AI training, and now you want to know whether it is live, what it touches, and whether using it could cost you a place in Google Search. The Google AI training robots.txt control is planned, not switched on. On October 30, 2026, Google Search Central said it has "plans to support a shared AI training control in robots.txt."
Google puts the target at 2027. Training is also a separate decision from Search: Google says its existing AI training token, Google-Extended, does not impact a site's inclusion in Google Search.
Google says existing provider-specific controls will keep working, so the rules in your file today stay in force. Your next step takes about ten minutes: open your robots.txt and read what it already says.
Key Takeaways
In its October 30, 2026 post, Google says it aims to support the shared control along with other AI providers in 2027. Plan around the year, not a date.
Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal there.
Existing provider-specific controls keep working. Open robots.txt, save a copy, and leave the general Googlebot group alone.
What did Google announce about the AI training control?
Google's Search Central post, "Supporting a shared AI training control for websites," is dated October 30, 2026. It describes a shared "policy token" that lets a website express "AI training preferences across multiple AI providers in a single step." The control is based on work from the IETF AIPREF working group.
One detail matters for anyone who already blocks Google's AI training. If a site uses the token to opt out of training and does not explicitly specify Google-Extended, Google plans to treat that as disallowing Google-Extended. A site that names Google-Extended in its own group keeps whatever it wrote there.
On timing, Google writes: "We aim to support this new control along with other AI providers in 2027." Plan for the year, not a date, and expect nothing to be available to switch on today.
When does it start, and what do other companies say?
Google is not the only company that spoke on October 30, 2026. Microsoft Bing published its own announcement, "Your content, your choice," saying it is adopting the shared AI training preference while the IETF work continues. Its training preference takes effect as Microsoft crawls beginning February 1, 2027, and Microsoft recommends publishing the setting by March 1, 2027. Microsoft also says the setting does not change eligibility for Bing Search results.
Dates on the record
- 1
October 30, 2026
Google Search Central announces plans to support a shared AI training control in robots.txt. Microsoft Bing announces its training preference the same day.
- 2
2027
Google's stated aim for supporting the control along with other AI providers.
- 3
February 1, 2027
Microsoft's training preference becomes effective as Microsoft crawls.
- 4
March 1, 2027
Microsoft's recommended date for publishing the setting.
OpenAI and Anthropic each document their own crawler controls. OpenAI's crawler documentation lists separate controls for GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic's help article covers ClaudeBot, Claude-SearchBot and Claude-User. Keep those provider-specific rules in place, and check each company's page again when a shared signal is added.
Does the Google AI training robots.txt control affect Search?
Not as Google describes it. The shared control is a training preference. Google says Google-Extended governs whether content it crawled may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google's crawler documentation adds that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Search indexing and AI Overviews are separate again. Google's page on controlling the content you share on Search covers the controls for appearing in Search generative features such as AI Overviews and AI Mode. Allowing Google-Extended may make pages eligible for Gemini grounding, but it carries no promise of an AI Overview, an AI Mode appearance, a ranking, clicks or traffic.
| Control | What Google says it covers | Search inclusion |
|---|---|---|
| Google-Extended (live today) | Training of future Gemini models, and grounding in Gemini Apps and Vertex AI | Google says no impact, and no use as a ranking signal |
| Shared AI training control (planned, 2027) | A training preference across AI providers; an opt-out is treated as a Google-Extended disallow when Google-Extended is not specified | A training preference, separate from Search indexing |
| Search Console and page-level controls | Whether your content appears in Search features such as AI Overviews and AI Mode | Set separately from training |
The one place Search can get hurt is a rule written for the wrong crawler. A broad Disallow on the general Googlebot group, added to control AI training, affects ordinary Search crawling. Write training rules for training tokens and leave the Googlebot group as it is.
What should I check in my robots.txt today?
Open your site's robots.txt in a browser by adding /robots.txt to your domain. If the file loads, you can read every rule on your site in one sitting. Before you change anything, save a copy so you can put it back.
The ten-minute robots.txt check
Visit yourdomain.com/robots.txt and check that you see plain text, not an error page.
Look for Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot and Claude-SearchBot.
Note whether each group says Allow or Disallow, and which folders it covers.
Confirm it does not block pages you want found in Search.
Keep the original text before any edit, with the date.
A general rule under User-agent: * does not mean the same thing as a rule written for one named crawler, so read the two separately. Blocking GPTBot also leaves OAI-SearchBot and ChatGPT-User with their own rules, because OpenAI documents them as separate controls. A new rule works only after a provider fetches your updated file, and a crawler may keep a cached copy for a while.
If you are also wondering which AI crawlers have already visited, our posts on seeing which AI agents visit your website and on whether ChatGPT and Google AI can read your website cover that side. For the Cloudflare angle, see how Cloudflare's AI training block interacts with Googlebot.
Is the new robots.txt syntax ready to copy?
No. The IETF AIPREF work is a set of Internet-Drafts, not a finished standard, and drafts can change. The vocabulary draft defines preferences such as train-ai, and the attachment draft describes a Content-Usage line for robots.txt. In the draft's example, a line reading Content-Usage: train-ai=n expresses a preference against training.
The same attachment draft separates two ideas. Allow and Disallow govern whether a crawler may obtain a page, while Content-Usage expresses how an obtained page may be used. Microsoft's announcement uses a different-looking example built on a `__usage_ai-training` user-agent line. Because both are in motion, wait for Google's implementation documentation before adding any new line to your live file.
A published preference also depends on the receiver. The IETF AIPREF working group is still writing the vocabulary, and a robots.txt line is a request that each AI provider has to choose to follow. Think of it as a posted sign, not a lock on your pages.
When is it worth paying someone to handle this?
For a simple site, the answer today is no. You can read your own robots.txt in about ten minutes. Google's shared control is planned for 2027, the IETF syntax is still a draft, and the provider-specific rules you already have may express exactly what you want.
Help earns its fee when the situation has real moving parts:
- Overlapping rules: Several groups contradict each other, and you cannot tell which one wins.
- Several subdomains: Each one serves its own robots.txt.
- Conflicting layers: A CDN or firewall adds crawler controls that fight your written policy.
- Search traffic at risk: A mistaken block on the wrong crawler would cost you leads.
- A policy you need documented: Someone has to test the delivered file and watch for changes.
That work is an audit, implementation, testing and monitoring job, priced to the size of the file. A robots.txt setting is a request, so even a well-built policy cannot make every AI provider obey it, and it cannot promise rankings or AI citations.
Check what you know about Google's AI training control
Pick an answer to begin.
1. When does Google say it aims to support the shared AI training control?
2. What does Google say about Google-Extended and Google Search?
3. What is the safest first step today?
Frequently Asked Questions About Google AI Training Robots.txt Control
When will Google's shared AI training control start?
Google says it aims to support the control along with other AI providers in 2027. Plan around the year, not a date.
Will the control remove my website from Google Search?
Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal there. The shared control is a training preference.
What is Google-Extended?
It is a Google robots.txt control for whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
Is the IETF AIPREF syntax final?
No. The vocabulary and attachment documents are Internet-Drafts, and drafts remain subject to change. Wait for Google's documentation before copying any syntax.
Does blocking AI training block live AI answers?
Training, grounding, live retrieval and Search indexing are separate purposes with separate controls. Check each provider's documentation for the purpose you care about.
Do I need to change my robots.txt now?
Read it now and save a copy. Existing provider-specific rules keep working, so edit only if the file already blocks something you want found or allows something you want kept out.
What This Means for You
You have a say in AI training, and it is arriving in stages. Google announced on October 30, 2026 that it plans to support a shared control based on the IETF AIPREF work, with a 2027 target. Microsoft's version takes effect February 1, 2027. Google-Extended stays separate from Search, and your current provider rules keep working.
Knowing what each line in your robots.txt does protects you from the one costly mistake, a block aimed at AI that lands on Search. When the shared control is documented, you will be able to add it to a file you already understand.
If your file has overlapping rules, several subdomains or a CDN layer that conflicts with it, Web Leveling can help. Our generative engine optimization work covers auditing what AI crawlers can reach, implementing your policy, testing the delivered file and monitoring it as Google and the IETF publish updates. We work with small and medium businesses across the country and overseas. Send us your robots.txt and tell us what you want to allow and block, and we will tell you whether it needs work.
Terms
Words used in this post
Tap a term to see what it means.
robots.txt. A plain text file at the root of a website that tells crawlers which parts of the site they may fetch.
Google-Extended. A Google robots.txt token that governs whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
IETF AIPREF. An Internet Engineering Task Force working group writing a standard way to express how AI systems may use content.
Internet-Draft. A working document that is subject to change and is not yet a finished standard.
Grounding. When an AI system looks up outside content to support the answer it gives.
Content-Usage. A rule in the IETF draft for robots.txt that states how a fetched resource may be used, such as for AI training.
