Google and Microsoft join OpenAI and Anthropic in backing a shared AI-training signal for robots.txt

Google and Microsoft have joined OpenAI and Anthropic in pledging to honor a shared robots.txt preference about using website content to train AI models. The signal does not block bots from crawling websites, and the proposal has not been endorsed by the IETF.