OpenAI, Google, Microsoft and Anthropic Back One robots.txt Signal for Opting Out of AI Training

OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 that they will support a single robots.txt signal that lets website operators opt out of having their content used to train generative AI models, SiliconANGLE reported. The signal is an individual Internet-Draft with no formal IETF standing, and only Anthropic is reported to have started implementing it.

What did the four companies announce?

According to SiliconANGLE, the four companies plan to back an upcoming crawling standard under which one robots.txt entry would block every scraper that collects training data for generative AI models. The report says OpenAI published the initial announcement on its own site and that the companies have submitted a draft to the Internet Engineering Task Force (IETF), the body that maintains the Robots Exclusion Protocol.

Today, a publisher who wants to keep content out of training sets has to list each provider's crawlers separately, and a provider that runs several crawlers needs several entries. The Tech Insider analysis names examples such as GPTBot, Google-Extended and ClaudeBot, and notes that operators must keep such lists current as companies rename or split their crawlers.

The new approach names a purpose rather than a company. The IETF datatracker entry for the draft, titled "Machine-Readable Training Preference Signal," defines a policy token, `__usage_ai-training`. An operator writes ordinary robots.txt rules keyed to that token in the User-Agent line. A disallowed path signals a preference against AI training, and an allowed path signals a preference in favour.

How does the signal work, and what status does it have?

The draft, `draft-kelley-aipref-training-preference-signal-00`, lists Kevin Kelley of Anthropic as its author. It is dated 30 October 2026 and expires on 3 May 2027. The datatracker describes it as an individual Internet-Draft that is not endorsed by the IETF, although its stated intended status is Standards Track.

The draft adds no new directive names or grammar. It leaves the path matching, longest-match precedence and group structure of RFC 9309, the 2022 robots.txt specification, unchanged. It also says that specific expressions should override general ones and that private agreements may override preferences expressed this way.

Two details matter in practice. First, wildcard groups (`User-Agent: *`) do not count as a match for the token, so an operator who has only a blanket rule has not opted out; the token must be named. Second, the draft separates content-usage preferences from traffic-management rules.

When will it take effect?

Timelines differ by company and by outlet. SiliconANGLE reported that OpenAI and Microsoft intend to add support "in the coming months" and that Google will follow "sometime next year." Meta Platforms expressed support for the idea behind the technology but has not committed to implementing it, according to the same report.

The two secondary articles add more specific, less firmly sourced claims. Tech Insider says Anthropic is already implementing the token and attributes a 1 February 2027 Microsoft date to a single report from PPC Land. Shattered.io repeats that date and says Microsoft will open temporary grounding preferences in Bing Webmaster Tools on 15 December 2026. Neither date appears in SiliconANGLE's account. Those dates should be treated as unconfirmed until the companies publish documentation.

Where do Google and Microsoft differ?

The most concrete disagreement concerns "grounding," in which an AI system fetches a live web page to answer a user's question rather than to collect training data.

SiliconANGLE reported that Microsoft, in a blog post, will offer a setting called "AI Grounding" for those cases. Operators could use it to stop Microsoft services from sending such requests, and it would extend to third-party AI tools built on Microsoft technologies. Tech Insider describes the setting as a page-level tag with Allow, Disallow and DisallowExtended values, and says the shared token is limited to training at Microsoft. On that reading, a publisher could refuse training while remaining eligible for cited answers.

Tech Insider says Google reads the token more broadly, as blocking both training and Gemini grounding. Shattered.io likewise says Google treats the token as a Google-Extended disallow, and that AI Overviews and AI Mode are excluded and remain under a separate Search Console control. The same piece flags that the exact reach of the token for grounding should be verified against Google's own documentation. The two companies have not publicly reconciled their approaches in the sources reviewed.

What does the signal not do?

Tech Insider lists what falls outside the scope: blocking crawling itself, search rankings, content collected before deployment, non-generative classification or ranking models, and retrieval at answer time, with some company-specific exceptions. SiliconANGLE likewise says the standard would not apply to models that "perform exclusively non-generative tasks."

Compliance is voluntary. Shattered.io calls the practical effect on announcement day "close to zero," and Tech Insider says the reports describe no penalty for a company that reads the token and trains anyway. OpenAI described the work as "a voluntary effort," according to Shattered.io. Smaller developers, open-weight builders and overseas labs have no stated obligation to honour the token, and third-party datasets could carry training data into models regardless.

Critics quoted in the coverage also point to the opt-out design. The signal arrives after leading models were trained on the open web, and it does not address deletion of existing training data or payment to publishers. Shattered.io also notes that Google and Microsoft both warn that opting out may reduce visibility in AI experiences, which it calls "a soft threat."

Why does it matter for publishers?

For a publisher, the change is administrative before it is legal. One named token could replace a growing list of crawler-specific rules, but not yet. Both explainers advise keeping existing GPTBot, Google-Extended and ClaudeBot rules until each company confirms it honours the token, and treating the signal as a request rather than a barrier.

What we don't know yet

Several material points are unconfirmed. The syntax OpenAI will publish, and its date, are not stated. Whether Meta will implement the token is open, and Shattered.io says some coverage describing Meta's support remains unconfirmed. The firm dates for Microsoft and Google rest on single or secondary reports. It is also unclear whether Google and Microsoft will converge on how the token applies to grounding, how a publisher could audit whether a model was trained on their content after setting the token, and whether the IETF will adopt the draft as a working-group document.

For now, the verifiable facts are narrow: a draft exists, four companies have said they will support it, and implementation, other than at Anthropic, is still ahead.