OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a new technical standard that allows publishers to opt out of having their content scraped for artificial intelligence training. The move addresses a growing concern among media organizations, which have struggled to manage the influx of automated bots accessing their websites to feed data into large language models.
Under the new framework, publishers can apply their AI training preferences across multiple AI companies simultaneously. Previously, site owners had to configure settings individually for each provider, a process that was often complex and inconsistent. This shared control mechanism aims to simplify the process and give content creators more direct authority over how their work is used.
Shared Policy Token in Robots.txt
The technical foundation of this agreement is a shared policy token proposed by Kevin Kelley, a technical lead for standards at Anthropic. The signal will be embedded within a website’s robots.txt file, a standard protocol that has long been used to tell web crawlers which pages they can or cannot access. While robots.txt has traditionally governed search engine indexing and archiving, this update extends its scope to include AI training permissions.
By placing this token in the robots.txt file, a publisher can signal to all participating AI companies that their content should not be used for model training. The major tech firms have committed to restricting their AI crawlers from accessing sites that display this shared token. This approach creates a unified signal that conveys publisher preferences to multiple providers at once, reducing the administrative burden on content owners.
Andy Stone, communications head at Meta, signaled support for the new setting, stating that creating a common signal is an important step. However, Meta did not explicitly confirm whether it will honor the opt-out request in the same manner as its competitors. The company noted it looks forward to working with other AI firms on the standards and their implementation across the industry.
Implementation Timeline and Granular Controls
While Anthropic has already adopted the signal, the rollout for other major players will take time. Google and Microsoft plan to support the standard next year, while OpenAI stated it will implement the feature in the coming months. This staggered approach allows the companies to integrate the new protocol into their existing infrastructure without immediate disruption.
Microsoft has indicated it will offer granular controls that give publishers more specific options. These controls will allow site owners to opt out of AI training or AI grounding, which is the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft clarified that websites will still appear in Bing’s search results regardless of their AI training preferences, ensuring that search visibility remains unaffected by these privacy settings.
Google previously implemented similar controls that allow sites to manage whether their content appears in AI search features or is used for AI training. These existing measures demonstrate a growing industry recognition of the need for clearer boundaries between search indexing and model training data.
The announcement comes as publishers continue to grapple with unauthorized scraping. Some organizations have attempted to combat this by using the Really Simple Licensing standard, which allows publishers to charge crawlers for access to their content. Others, like Cloudflare, have blocked AI crawlers by default. The new standard offers a middle ground, providing a clear, technical method for publishers to express their preferences without resorting to blocking or charging mechanisms.
Source: The Verge
