OpenAI announced it will honor a shared robots.txt signal that lets websites express preferences about using their content for foundational generative AI training. The voluntary effort, coordinated with other participants in the Internet Engineering Task Force’s (IETF) AI Preferences working group, allows publishers and website operators to communicate the same preference to multiple participating providers at once. Previously, operators had to identify individual crawlers and maintain separate rules for each provider.

OpenAI says it aims to implement the signal in the coming months and will share another update with implementation details. The new signal will not replace existing controls: OpenAI‘s current robots.txt crawler controls, which give web publishers flexibility over their pages, will continue to apply and be respected. The announcement builds on ongoing IETF AI Preferences work; OpenAI says joining the shared signal strengthens its continued participation in IETF standards discussions, and that it will keep adapting its practices as conversations and the landscape evolve.

The company frames the step as practical support for an open web where people can publish and discover information and owners have clearer, simpler ways to communicate how their content is used. OpenAI also encourages other AI providers to honor the signal.