Cloudflare Gives Publishers a Search-Without-Training Switch
A new control separates search visibility from AI model training, shifting a major web-content dispute from informal requests toward enforceable infrastructure.
Key facts
- Cloudflare launched a Disallow AI Training setting and separate Search, Training and Agent controls on September 15, 2026.[1][2]
- The setting is intended to preserve search visibility while refusing AI training, including for mixed-use crawlers.[1][2]
- Cloudflare says mixed-use crawlers represent 36.6% of verified crawler traffic on its network.[1]
- The system depends on crawler identification and operator compliance rather than creating a universal legal restriction.[1][2]
Cloudflare has launched a control that lets website owners remain visible to search engines while refusing to let the same crawlers use their pages for AI training. The feature, introduced September 15, is designed to resolve a conflict that has become central to the economics of the generative-AI web: publishers want search traffic, but increasingly resist giving model developers unrestricted access to the material that generates that traffic.
What changed
The new “Disallow AI Training” setting distinguishes training from search and agent activity. Cloudflare says it applies even when a single crawler performs multiple jobs, a common arrangement involving major search platforms. Website owners can also block crawlers entirely, but that may remove their pages from search results as well. The company is replacing its older, broader “Block AI Bots” control with separate policies for Search, Training and Agent behavior. [0]
Cloudflare also introduced an “Accountable” designation for crawler operators that provide mechanisms for honoring publisher preferences. Apple, Google and Microsoft have either adopted or committed to support the setting, according to Cloudflare. The company says mixed-use crawlers account for 36.6% of verified crawler traffic on its network, making the distinction more than a technical edge case. [0]
Independent coverage of the rollout highlights the practical consequence: publishers can now opt out of training without automatically cutting off Google visibility. The change does not create a universal legal prohibition on data use, nor does it guarantee that every bot will comply. It is an infrastructure-level control that works only when operators identify themselves accurately and respect the preference. [1]
Why it matters
The announcement could establish a new bargaining position for publishers. Until now, a site owner often faced an uncomfortable choice: permit a crawler to collect content for both search and model development, or block it and sacrifice discovery. Separating those functions gives publishers leverage to preserve referral traffic while withholding one of the inputs that AI companies need most.
It also moves the debate from robots.txt files and voluntary statements toward enforcement at the network layer. That matters because robots.txt is advisory, while Cloudflare can inspect and block requests at the edge. If other hosting and security providers adopt comparable controls, access to high-quality web data may become more segmented, negotiated and measurable.
The unresolved question is whether the system can scale beyond participating platforms. Cloudflare’s controls cannot compel unidentified or noncompliant crawlers to honor preferences, and “training,” “search” and “agent” purposes may overlap in ways that are difficult to verify. The economics are unsettled too: refusing training may protect future licensing value, but it could also reduce exposure in AI-generated answers that increasingly function as discovery channels.
For now, Cloudflare has not ended the search-versus-training conflict. It has made the choice more granular—and turned a publisher preference into a setting that large parts of the web infrastructure can enforce.
