Google and Bing back one robots.txt rule to opt out of AI training

For three years, publishers who wanted to keep their work out of AI training data have had to learn a new crawler name every few months. Google-Extended, GPTBot, ClaudeBot, Applebot-Extended, CCBot: each company built its own switch, and each switch had its own rules about what it covered. On Friday, October 30, the two largest search engines in the Western world said they would honour a single one.

Google said it will voluntarily adopt a proposal under consideration at the Internet Engineering Task Force that lets sites set AI training preferences across several AI providers in a single robots.txt step, Barry Schwartz of Search Engine Roundtable reported. The mechanism is a shared policy token, written in robots.txt as a user-agent called “__usage_ai-training”. A site that disallows that token tells every participating company that its content should not be used to train models. “This will help websites express their AI training preferences across multiple AI providers in a…