OpenAI to honor a shared AI-training preference signal in robots.txt

OpenAI says it will honor a shared signal in robots.txt for expressing preferences about using website content for foundational generative AI training.[1] The company describes the effort as voluntary and says it will let publishers communicate the same preference to multiple participating providers at once.[1] OpenAI says it is joining other participants in the IETF’s AI Preferences working group in announcing the shared signal, and encouraged other AI providers to honor it.[1] The announcement describes the effort as a practical step toward making website owners’ choices clearer and simpler.[1]

The IETF draft defines a cross-vendor policy token, `__usage_ai-training`, for the shared signal.[2] It introduces no new directive names or grammar, and says the token is not a directive for crawling behavior.[2] In the draft’s example, the preference is against AI training on every path except those under `/press-kit/`.[2] A robots.txt file without the shared policy token expresses no preference through this mechanism.[2] The draft also says parsers will continue to handle traffic-management signals as before and ignore policy-token groups they do not recognize.[2]

OpenAI says operators previously had to identify individual crawlers and maintain separate rules for each provider.[1] It says a shared signal reduces that work by allowing a preference to be communicated across multiple participating providers.[1] Operators should distinguish that policy preference from controls governing crawler access: OpenAI says the new signal will not replace its existing crawler controls in robots.txt.[1] The company says those existing controls will continue to apply and be respected.[1] The draft also notes that private agreements may override preferences expressed through the signal.[2]

The draft was published on October 30, 2026, but it is not endorsed by the IETF and has no formal standing in the organization’s standards process.[2] The Verge reported that OpenAI, Google, Microsoft and Anthropic said they would adopt the shared standard.[3] It also reported that Anthropic had already adopted the signal, while Google and Microsoft planned to support it the following year.[3] According to The Verge, Meta communications head Andy Stone supported a common signal but did not say whether Meta would honor it.[3]