By Matt G. Southern
Publication Date: 2026-11-01 08:00:00
Google, Microsoft, and OpenAI announced plans on Friday to honor a shared robots.txt line for AI training preferences. A site could use it to state one training preference to every provider that takes part, instead of writing rules for each company’s crawler.
The line is from an Internet Draft that Anthropic’s Kevin Kelley submitted to the AI Preferences working group at the IETF (same date). This is listed in the IETF’s document tracker as an individual draft, which does not represent official IETF standardization activity. In a post to the working group’s email list, Kelley indicated that Anthropic has started development to provide this feature.
Based on their provided timelines, none of the three companies currently supports the line. Google and Microsoft described how each would handle it differently. According to Google, if a site sets up a shared opt-out line, Google plans to treat it like a Google-Extended block when a website doesn’t specify Google-Extended, and…
