Close Menu
iM.NewsiM.News

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Study: Escitalopram, Duloxetine Outperform Fluoxetine for Long-Term Depression

    November 21, 2026

    Curiosity Rover Captures Rare Twilight Clouds on Mars

    November 20, 2026

    US Lawmakers Seek Stricter AI Chip Export Controls

    November 20, 2026
    Facebook X (Twitter) Instagram
    iM.NewsiM.News
    Subscribe
    • Home
    • Lifestyle

      Study: Escitalopram, Duloxetine Outperform Fluoxetine for Long-Term Depression

      November 21, 2026

      US, Mexico, Bolivia Lose Measles-Free Status as Global Cases Rise

      November 20, 2026

      The Opera Coat Trend Takes Over Winter Fashion

      November 20, 2026

      New Weight-Loss Drug Cuts Knee Pain in Half for OA Patients

      November 20, 2026

      Robert Pattinson and Suki Waterhouse Redefine Hollywood Power Couples

      November 20, 2026
    • Relations

      US Lawmakers Seek Stricter AI Chip Export Controls

      November 20, 2026

      Trump Threatens Tax on US Tech Firms Building AI Data Centers Abroad

      November 20, 2026

      Former US National Security Adviser Jake Sullivan Confronted by Students at MIT

      November 20, 2026

      Xi Jinping’s Restraint: The Power of Not Acting in Shenzhen

      November 20, 2026

      EU and UK to Sign Reset Deal on Trade and Youth Mobility

      November 20, 2026
    • Technology

      Germany Proposes €25 Billion Buyout of TikTok Europe

      November 20, 2026

      AI Supply Chain Leaks 2 Million Secrets in Canary Test

      November 20, 2026

      Google to retire Nest app in January 2027

      November 20, 2026

      Microsoft Link to Windows Adds Second Screen Feature

      November 20, 2026

      Nothing Sets December 8 Date for AI Hardware Reveal

      November 20, 2026
    • Travel & Tourism

      TAP Air Portugal Adds Nonstop Porto Service from Washington Dulles

      November 20, 2026

      Oil Spills Threaten UAE and Oman Beaches Amid Hormuz Conflict

      November 20, 2026

      Aer Lingus to Retrofit A330 Fleet with Premium Economy and New Business Class

      November 20, 2026

      Sabre Targets AI Travel Bookings With New Connector

      November 20, 2026

      Riyadh Hotel Revenue Plummets 65% After Airport Attack

      November 20, 2026
    • Get in Touch
    iM.NewsiM.News
    Home»Technology»Gadgets»AI Giants Back New Standard for Publishers to Opt Out of Scraping
    Gadgets

    AI Giants Back New Standard for Publishers to Opt Out of Scraping

    im.newsBy im.newsOctober 31, 2026No Comments3 Mins Read14 Views
    Facebook Twitter Pinterest LinkedIn Tumblr Email
    Server room with AI scraping opt out robots.txt code on screen
    Photo: Coolcaesar / Wikimedia Commons, CC BY-SA 4.0
    Share
    Facebook Twitter LinkedIn Pinterest WhatsApp Email

    OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a new technical standard that allows publishers to opt out of having their content scraped for artificial intelligence training. The move addresses a growing concern among media organizations, which have struggled to manage the influx of automated bots accessing their websites to feed data into large language models.

    Under the new framework, publishers can apply their AI training preferences across multiple AI companies simultaneously. Previously, site owners had to configure settings individually for each provider, a process that was often complex and inconsistent. This shared control mechanism aims to simplify the process and give content creators more direct authority over how their work is used.

    Shared Policy Token in Robots.txt

    The technical foundation of this agreement is a shared policy token proposed by Kevin Kelley, a technical lead for standards at Anthropic. The signal will be embedded within a website’s robots.txt file, a standard protocol that has long been used to tell web crawlers which pages they can or cannot access. While robots.txt has traditionally governed search engine indexing and archiving, this update extends its scope to include AI training permissions.

    Aerial Microsoft West Campus August
    Photo: Jelson25 / Wikimedia Commons, Public domain

    By placing this token in the robots.txt file, a publisher can signal to all participating AI companies that their content should not be used for model training. The major tech firms have committed to restricting their AI crawlers from accessing sites that display this shared token. This approach creates a unified signal that conveys publisher preferences to multiple providers at once, reducing the administrative burden on content owners.

    Andy Stone, communications head at Meta, signaled support for the new setting, stating that creating a common signal is an important step. However, Meta did not explicitly confirm whether it will honor the opt-out request in the same manner as its competitors. The company noted it looks forward to working with other AI firms on the standards and their implementation across the industry.

    139 Server Room 01
    Photo: Indrajit Das / Wikimedia Commons, CC BY-SA 3.0

    Implementation Timeline and Granular Controls

    While Anthropic has already adopted the signal, the rollout for other major players will take time. Google and Microsoft plan to support the standard next year, while OpenAI stated it will implement the feature in the coming months. This staggered approach allows the companies to integrate the new protocol into their existing infrastructure without immediate disruption.

    Microsoft has indicated it will offer granular controls that give publishers more specific options. These controls will allow site owners to opt out of AI training or AI grounding, which is the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft clarified that websites will still appear in Bing’s search results regardless of their AI training preferences, ensuring that search visibility remains unaffected by these privacy settings.

    Google previously implemented similar controls that allow sites to manage whether their content appears in AI search features or is used for AI training. These existing measures demonstrate a growing industry recognition of the need for clearer boundaries between search indexing and model training data.

    PDC server room
    Photo: Esquilo / Wikimedia Commons, CC BY-SA 3.0

    The announcement comes as publishers continue to grapple with unauthorized scraping. Some organizations have attempted to combat this by using the Really Simple Licensing standard, which allows publishers to charge crawlers for access to their content. Others, like Cloudflare, have blocked AI crawlers by default. The new standard offers a middle ground, providing a clear, technical method for publishers to express their preferences without resorting to blocking or charging mechanisms.

    Source: The Verge

    AI Anthropic Gadgets Google Microsoft OpenAI Scraping
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Previous ArticleNorth Korea Suspected in $6.3bn Binance Hack
    Next Article 10 Eerie British Sites for Halloween
    im.news

    Related Posts

    Study: Escitalopram, Duloxetine Outperform Fluoxetine for Long-Term Depression

    November 21, 2026

    Curiosity Rover Captures Rare Twilight Clouds on Mars

    November 20, 2026

    US Lawmakers Seek Stricter AI Chip Export Controls

    November 20, 2026
    Leave A Reply Cancel Reply

    Latest Posts

    Study: Escitalopram, Duloxetine Outperform Fluoxetine for Long-Term Depression

    November 21, 20262 Views

    Curiosity Rover Captures Rare Twilight Clouds on Mars

    November 20, 20263 Views

    US Lawmakers Seek Stricter AI Chip Export Controls

    November 20, 20263 Views

    TAP Air Portugal Adds Nonstop Porto Service from Washington Dulles

    November 20, 20262 Views
    Stay In Touch
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss

    Epstein List Phase 1: Full Flight Logs, Client List Released

    February 28, 2025 News Focus 247 Views

    In a stunning development that has sent shockwaves through political and entertainment circles, Attorney General…

    Microsoft Stock Joins Exclusive $4 Trillion Club After Blockbuster Cloud + AI Earnings

    July 31, 2025

    DOJ & FBI Find No Epstein Client List, Suicide Confirmed

    July 7, 2025

    Subscribe to Updates

    Get the latest creative news from SmartMag about art & design.

    About Us
    About Us

    We provide the daily life news. You find latest trendy news at our portal, from entertainment to economy or politics.

    We're accepting new partnerships right now.

    Email Us: info@im.news

    Facebook X (Twitter) RSS
    Our Picks

    Study: Escitalopram, Duloxetine Outperform Fluoxetine for Long-Term Depression

    November 21, 2026

    Curiosity Rover Captures Rare Twilight Clouds on Mars

    November 20, 2026

    US Lawmakers Seek Stricter AI Chip Export Controls

    November 20, 2026
    Most Popular

    Epstein List Phase 1: Full Flight Logs, Client List Released

    February 28, 2025247 Views

    Microsoft Stock Joins Exclusive $4 Trillion Club After Blockbuster Cloud + AI Earnings

    July 31, 2025208 Views

    DOJ & FBI Find No Epstein Client List, Suicide Confirmed

    July 7, 2025165 Views
    © 2026 I'm News. Designed by I'm News.
    • Home
    • Lifestyle
    • Relations
    • Travel & Tourism
    • Technology

    Type above and press Enter to search. Press Esc to cancel.