AI & Machine Learning

The New AI Training Opt-Out in robots.txt: One Token, Four Clocks

Google, Microsoft, OpenAI and Anthropic back one robots.txt token for AI training opt-out. Here's what it actually does, and what it doesn't settle yet.

Editorial Team / /9 min read
Google, Microsoft, OpenAI and Anthropic logos sit above a small crawler robot stopped at a barrier gate held by Clawd.

If you run a website and want an AI training opt-out, you can now add one line to a file almost every site already has, and four of the biggest AI labs have said they will listen to it. That sounds like the end of a long, messy problem. It is closer to the start of a more orderly one.

On October 30, 2026, Google, Microsoft, OpenAI and Anthropic each published a statement saying they would honor a new shared signal for opting out of AI training, placed in robots.txt, the small text file at the root of a website (yoursite.com/robots.txt) that tells automated crawlers which parts of a site they may visit. Most of the coverage that followed described it as four tech giants agreeing on one new standard. That framing gets two things wrong, and the gap between what was announced and what actually changed is exactly what a publisher needs to understand before touching their own robots.txt file.

What the new robots.txt token actually does

The mechanism is simple enough to fit in three lines. A website adds a block to its robots.txt file naming a new user-agent token, __usage_ai-training, and tells it what it may or may not do, the same way robots.txt has always worked for search crawlers:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read plainly, that example blocks AI-training crawlers from the whole site except a press kit folder the owner wants indexed anyway. The underscore-prefixed token is not a specific company’s crawler, the way Googlebot or GPTBot are. It is a shared label that any participating AI company’s crawler can recognize and respect, so a publisher does not need to list every AI company’s individual crawler name to express one preference: don’t use this content to train a model.

That is the genuinely useful part, and it is real. Before this, a site wanting to opt out of AI training had to maintain a growing list of named crawlers, similar in spirit to the way sites wanting to keep content out of a vector database used for retrieval have had to track each system separately, and update the list every time a new one appeared. One line, applied once, is meant to cover any crawler that agrees to check for it. The practical question, covered further down, is what that one line actually does and doesn’t cover today.

It’s not an IETF standard, here’s what it actually is

The token comes from a document called draft-kelley-aipref-training-preference-signal-00, filed with the IETF, the Internet Engineering Task Force, the standards body that oversees technical specifications like this one. The word “draft” in IETF language does not mean rough or unfinished in the everyday sense. It is a formal category, and this particular kind, an individual Internet-Draft, is the lowest rung: anyone can submit one, and the IETF datatracker entry for the draft states plainly that it “is not endorsed by the IETF and has no formal standing in the IETF standards process.”

So the token itself has not been reviewed, debated, or ratified by any IETF working group. What has gone through that process is a related but different document, draft-ietf-aipref-vocab-08, titled “A Vocabulary For Expressing AI Usage Preferences.” That one is an actual working-group document (authored by Paul Keller of Open Future and Mozilla’s Martin Thomson, under the IETF’s aipref working group), on track toward becoming a Proposed Standard, the formal designation for a specification that has cleared IETF review. It defines what terms like “AI training” mean in a machine-readable vocabulary. The Kelley draft simply borrows that vocabulary and applies it to robots.txt syntax, without itself carrying the working group’s backing.

The distinction matters because a real working-group document has an open process behind it: review, objections, revisions, a path to consensus. An individual draft has none of that by default. It becomes meaningful only if enough parties agree, in practice, to honor it, which is precisely what happened here: four companies chose to support an unendorsed draft, not because the IETF told them to, but because they coordinated their own announcements around it.

IETF datatracker page for the draft-kelley-aipref-training-preference-signal Internet-Draft

IETF datatracker entry for draft-kelley-aipref-training-preference-signal-00 — the individual Internet-Draft that defines the token, flagged as having no formal IETF standing.

Four companies, one token, four different clocks

This is where the “4 giants, 1 standard” shorthand breaks down the most. The four companies did announce on the same day, but what each of them committed to differs in timing and in scope.

Google published a post on its Search Central Blog explaining that if a site uses the shared token without separately configuring Google-Extended, its own existing control for AI training use, Google will treat the shared token as also covering Google-Extended. But that fold-in is not live yet either: Google says it “aims to support this new control along with other AI providers in 2027,” with no narrower date given, while existing company-specific controls keep working in the meantime.

Microsoft, through Bing, said on its Bing Blog that it will start honoring the shared token on February 1, 2027, not immediately. Microsoft also introduced a second, separate preference called “AI Grounding,” covering a different use: when an AI system pulls in a page’s content live to help generate an answer, as opposed to using that content to train a model in the first place. Grounding can currently only be controlled at the individual page level, through a meta tag or an X-Robots-Tag header (an instruction placed in a page’s server response rather than in the robots.txt file), not through robots.txt at all.

OpenAI said in a post on openai.com that it is committed to honoring the same signal, with rollout planned “in the coming months,” without naming a specific date. Its existing robots.txt controls continue to apply in the meantime, so nothing is lost by waiting, but nothing new takes effect on a fixed schedule either.

Anthropic, whose own engineer, Kevin Kelley, authored the technical draft, is the one company that says it is already acting on it: announcing the draft on the IETF’s ai-control mailing list on October 30, 2026, Kelley wrote that “Anthropic has already started implementing support for this draft,” while cautioning that “it’ll take us some time to add support across our services and work through the inevitable complications and edge cases.” So Anthropic has a head start but, like OpenAI, no fixed completion date.

Lay those four out together and the picture is less “one switch, four confirmations” and more four separate internal rollout schedules that happen to point at the same piece of syntax.

The Microsoft deadline publishers can’t ignore

Of the four, only one carries a concrete expiration date for an existing practice, and it is the detail most likely to catch a publisher off guard.

Many sites currently discourage AI training indirectly, through older directives like NoArchive or NoCache, meta tags originally meant to stop search engines from storing cached copies of a page, which some crawlers also came to treat informally as a signal against AI training use. According to Bing’s own Webmaster Tools documentation on AI preferences, Microsoft will stop honoring NoArchive and NoCache as AI-training opt-out signals starting March 1, 2027.

That means any publisher currently relying on those older tags to keep their content out of Bing-affiliated AI training has a hard deadline, not a vague window. Any site still depending on NoArchive or NoCache for this purpose needs to add the new token before March 1, 2027, or it loses that protection on Bing with no fallback. This is the one piece of the announcement that behaves like a real policy change with a real cutoff, rather than a voluntary signal companies will get around to eventually.

Bing Blog announcement post on the new AI training and grounding content preferences

Microsoft Bing Blog — the October 30, 2026 post announcing the new training/grounding preferences and the NoArchive/NoCache retirement date.

What to actually put in your robots.txt today, and what it won’t solve yet

For a publisher who wants to act now rather than wait for every timeline to resolve, the practical move is straightforward: add the __usage_ai-training block to robots.txt today, scoped to whichever paths should be protected. It costs nothing, breaks nothing else on the site, and starts the clock on being recognized by the companies that have committed to honoring it, Anthropic already in progress, Microsoft from February 2027, Google at an unspecified point in 2027, OpenAI at an unspecified point “in the coming months.”

What it does not do yet is give one unified answer to “is my content safe from AI training.” Crawlers from companies that haven’t signed on still have to be addressed the old way, by name, if a publisher wants to block them. Microsoft’s separate AI Grounding preference is also outside its reach, since that one has to be set at the page level, not in robots.txt, for anyone who wants a say in that specific use. And it carries no legal force; it is a voluntary signal honored because a company chose to, not because any regulation or ratified standard compels it.

That is the actual lesson underneath the four October 30 announcements. A single shared token that looks like “set it once, done everywhere” is not the same thing as a ratified standard with synchronized enforcement. It is four companies agreeing to read the same word, each on its own schedule, with its own exceptions. For a publisher’s own risk and control over their content, that still means checking each company’s own policy page, not treating one robots.txt line as a finished job.

Decision card on updating robots.txt for the AI training opt-out: add __usage_ai-training before March 1, 2027 if you currently rely on NoArchive or NoCache on Bing, add it today for one line covering multiple AI crawlers, keep Google-Extended as is until Google folds the token in sometime in 2027, use Microsoft's separate page-level AI Grounding control for live use rather than training, and keep blocking by name any crawler whose company hasn't adopted the shared token

FAQ

What is the __usage_ai-training token in robots.txt?

The __usage_ai-training token is a label a website adds to its robots.txt file to tell AI crawlers not to use its content for training a model. Unlike named crawlers such as GPTBot, it is a shared signal multiple companies can recognize at once, reducing the need to list every AI company’s crawler individually.

Is the AI training opt-out in robots.txt an official IETF standard?

No. The specific document defining the __usage_ai-training token, draft-kelley-aipref-training-preference-signal-00, is an individual Internet-Draft that the IETF datatracker explicitly says has no formal standing in the IETF standards process. A related vocabulary document is going through real IETF working-group review, but the robots.txt token itself is not.

Do Google, Microsoft, OpenAI and Anthropic all honor the token the same way?

No. Google plans to fold it into its existing Google-Extended control, but says that won’t happen until sometime in 2027. Microsoft starts honoring it on February 1, 2027, and treats a separate “AI Grounding” use case differently, outside robots.txt. OpenAI plans rollout “in the coming months” with no fixed date, and Anthropic, which authored the draft, says it has already started implementing support but has not given a completion date.

Why does the Microsoft deadline matter for publishers right now?

Microsoft will stop honoring NoArchive and NoCache as AI-training opt-out signals on Bing starting March 1, 2027. Any publisher currently depending on those older tags to block AI training use on Bing needs to add the new __usage_ai-training token to robots.txt before that date, or lose that protection with no automatic fallback.

Does adding the new token stop all AI companies from training on my content?

No. It only covers companies that have committed to honoring the shared token, and even then, enforcement depends on that company’s own crawler choosing to respect it. Crawlers from companies that haven’t adopted the signal still require separate, named rules in robots.txt if a publisher wants to block them.

#ai-training#robots.txt#ietf#google#microsoft#openai#anthropic