Google plans support for shared AI crawler opt-out

OpenAI, Google, Microsoft and Anthropic say they will adopt a shared standard that lets publishers opt out of AI scraping across multiple companies.[1] The signal will be placed in a website’s robots.txt file, and The Verge reports participating companies will restrict their AI crawlers from accessing sites that use the shared token.[1] OpenAI says the shared control is intended to make it easier to communicate the same preference to multiple providers and reduce the work of maintaining separate rules.[2] The proposal came from Kevin Kelley, an Anthropic technical lead for standards, through the Internet Engineering Task Force.[1]

Google plans to support the signal next year, according to The Verge.[1] OpenAI says it aims to implement the signal in the coming months, and Anthropic had already adopted it when The Verge reported the plans.[1][2] The evidence does not give a specific launch date for Google or OpenAI, so publishers should treat these as plans rather than a single confirmed availability date.[1][2] The Verge reported the plans on October 30, 2026.[1]

OpenAI says the shared signal will not replace its existing robots.txt controls, which it says will continue to apply and be respected.[2] Microsoft says publishers can opt out of AI training or AI grounding, while their sites will remain in Bing search results regardless of their AI preferences.[1] Google has previously offered controls over AI training and AI search features without affecting a site’s search visibility, The Verge reports.[1] The shared signal therefore sits alongside provider-specific controls rather than, in OpenAI’s account, replacing them.[2]

The shared setting’s usefulness depends on how participating providers implement and honor it; the evidence identifies plans from four companies, not universal adoption across AI providers.[1] Meta communications head Andy Stone signaled support for the setting, but did not say whether Meta would honor it.[1] The Verge described the plans as arriving amid continued AI crawler activity on publisher sites.[1] It also cited Really Simple Licensing, which lets publishers charge crawlers visiting their sites, and Cloudflare’s default blocking of AI crawlers as alternative approaches.[1]