Google, Microsoft, OpenAI Plan To Honor Shared robots.txt AI Training Opt-Out: how 8 outlets framed it | NewsCord

A shared robots.txt token

Google, Microsoft, and OpenAI announced plans to honor a shared robots.txt line for AI training preferences, and the line comes from an Internet Draft that Anthropic’s Kevin Kelley submitted to the AI Preferences working group at the IETF. Kelley’s draft uses an ordinary robots.txt rule group where a site writes a policy token named __usage_ai-training under a User-Agent line, with the example showing User-Agent: __usage_ai-training Disallow: / and Allow: /press-kit/. The draft defines AI Training as using content to modify a generative AI model’s learned parameters, and it says models that do only non-generative tasks like classification or ranking are exempted from that definition.

The draft also states that a wildcard group such as User-Agent: * can never match the token, and it says a file without a __usage_ai-training group indicates no preference, which Kelley states is true of every robots.txt file currently published. The draft further says the…