Major AI Firms Back Shared Opt-Out Standard for Scraping

Major AI Firms Adopt Shared Opt-Out Standard

OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a new technical standard that lets publishers block their content from being scraped for AI training across multiple companies at once. The development was reported earlier by Search Engine Roundtable and confirmed on October 30, 2026. Instead of configuring separate opt-out signals for each AI provider, site owners will soon be able to set a single preference that travels with their existing robots.txt file.

How the Policy Token Standard Works

The shared "policy token" was proposed by Kevin Kelley, who serves as a technical lead for standards at Anthropic, through the Internet Engineering Task Force. The signal sits inside the familiar robots.txt mechanism that has long governed which crawlers may access a site for search indexing, archiving, or other purposes. Until now, publishers who wanted to keep their material out of training datasets had to communicate that preference individually to every AI lab, a fragmented process that became more burdensome as the number of crawlers multiplied.

The announcement arrives as publishers contend with a rising tide of automated agents harvesting text, images, and code for model development. Some organizations have pursued alternative defenses. The Really Simple Licensing standard aims to give sites a way to charge crawlers for access, while Cloudflare has begun blocking known AI bots by default. The new token does not replace those approaches but offers a lighter-weight, unified method that requires no payment infrastructure or third-party service.

Anthropic says it has already implemented the signal. Google and Microsoft have indicated they will add support sometime in 2027, while OpenAI expects to roll it out in the coming months. Microsoft plans to go a step further by offering granular controls that distinguish between AI training, where content helps build the model itself, and AI grounding, where a model is pointed to a verified source to reduce hallucinations. The company also notes that a site's search visibility in Bing will remain unchanged regardless of the opt-out choice.

Google and Meta Join Publisher Controls Effort

Google previously introduced its own publisher controls that let sites decide whether their content appears in AI-powered search features or feeds training runs, without affecting ordinary search rankings. The new standard effectively extends that logic across the industry, reducing the operational overhead for sites that want consistent treatment.

Meta's communications head Andy Stone signaled support for the concept, calling a common signal an important step and expressing willingness to collaborate on standards and implementation. He stopped short of committing Meta's crawlers to honor the token, leaving the company's practical stance unclear.

What Developers and Site Owners Should Do

For developers and teams building with AI, the move signals a shift toward more predictable data-governance norms. A single, machine-readable declaration in robots.txt will eventually replace the patchwork of proprietary meta tags, API settings, and legal notices that currently govern scraping permissions. That simplification matters when you are auditing training data provenance, designing retrieval-augmented generation pipelines, or advising clients on content strategy. It also reduces the risk that a missed configuration on one platform undermines a publisher's intent across the board.

Anyone responsible for a public-facing site should start planning to add the policy token once the IETF specification stabilizes. Monitor the implementation timelines from each major crawler, test that the signal is respected in staging environments, and verify that search traffic remains unaffected. If you maintain internal tooling that generates or validates robots.txt, update those templates now so the new directive can be deployed with minimal friction when support goes live.

Frequently asked questions

What is the new shared opt-out standard for AI scraping announced by major AI firms?

OpenAI, Google, Microsoft, and Anthropic will adopt a common "policy token" inside robots.txt that lets publishers block their content from being scraped for AI training across all participating companies with a single declaration, replacing the previous need to configure separate opt-out signals for each provider.

When will each company implement the shared policy token for AI scraping opt-outs?

Anthropic has already implemented the signal. OpenAI expects to roll it out in the coming months. Google and Microsoft plan to add support sometime in 2027. Microsoft will also offer granular controls distinguishing AI training from AI grounding, while confirming Bing search visibility remains unaffected by the opt-out choice.

Who proposed the shared policy token standard and through which organization?

The shared policy token was proposed by Kevin Kelley, who serves as a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). The signal operates within the existing robots.txt mechanism that has long governed crawler access for search indexing and archiving.

How does the new standard differ from existing publisher controls like Google's previous system?

Google previously offered its own publisher controls for AI-powered search features and training runs without affecting ordinary search rankings. The new standard extends that logic industry-wide, reducing operational overhead by providing a single, machine-readable declaration in robots.txt that works across multiple AI companies instead of proprietary meta tags and API settings.

What should site owners do to prepare for the new AI scraping opt-out standard?

Site owners should plan to add the policy token once the IETF specification stabilizes, monitor implementation timelines from each major crawler, test that the signal is respected in staging environments, verify search traffic remains unaffected, and update internal tooling that generates or validates robots.txt for minimal-friction deployment.

Source: The Verge