Google, Microsoft, OpenAI and Anthropic announced on October 30, 2026 that they will honor a single shared signal in the robots.txt file, letting websites state their preference against the use of their content for AI training — the training of generative AI models. The proposal comes from a technical draft at the IETF, the body that standardizes internet protocols, and uses a policy token called __usage_ai-training.
How the token works
In practice, a site owner writes an ordinary group in robots.txt, but uses the shared token instead of a crawler name on the User-Agent line:
User-Agent: __usage_ai-training Disallow: /
With a single rule, the site states its preference not to have its content used for AI training across all participating companies, without repeating the instruction for each crawler. Exceptions can be opened with Allow lines, for example for a press kit. The token is not a crawler: no crawler sends it as an identifier.
A draft, not a standard
The document behind the signal is the Internet-Draft "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley of Anthropic and published on October 30, 2026. It is an individual draft, not an approved IETF standard, and it expires on May 3, 2027, unless revised or adopted.
Different timelines by company
Timelines differ by company. Microsoft will start honoring the new preferences on February 1, 2027, as Bing recrawls pages; Google aims for early 2027; OpenAI speaks of the "coming months"; and Anthropic, according to the draft's author, has already begun implementing support. None of the four reads the token in production yet.
What the signal does not do
There are important limits. The signal covers AI training only — using content to modify the learned parameters of a generative AI model. It does not block crawling, does not change search presence or ranking, and does not stop content from being used to answer questions in AI services. At Google, a site that opts out with the token without mentioning Google-Extended will be treated as if it had disallowed Google-Extended, which also affects grounding in Gemini Apps. Microsoft is creating a separate grounding control and will stop reading the older NoArchive and NoCache tags as a preference from March 1, 2027.
There is also no retroactive effect: nothing already collected or trained is undone. And the signal is voluntary — it only applies to companies that adopt it. Meta, for example, has not announced any commitment to the token so far. An independent November 2026 survey (CrawlCheck, covering the top 10,000 domains) indicates that 21.8% of sites with a robots.txt file already block at least one AI crawler — context that helps explain why a single shared signal matters to publishers.
For publishers and site owners, the advance is practical: one line now covers four labs at once, instead of one rule per company. But since nothing is in production yet, the signal is today a preparation for 2027, not an immediate block.
Sources: Google Search Central Blog — "Supporting a shared AI training control for websites", by John Mueller, October 30, 2026; IETF — Internet-Draft "Machine-Readable Training Preference Signal"; PPC Land — "Google, Microsoft, OpenAI and Anthropic back one AI training opt-out".
Image credit: chart "Which AI crawlers robots.txt shuts out", from the November 2026 CrawlCheck census, via Wikimedia Commons, CC BY 4.0 license.
Transparency: this article was produced with the assistance of artificial intelligence, cross-checked against public posts on X and internet sources, and reviewed before publication. Please consult the original sources to confirm the details.
Muse
