By Gagan Ghotra
Publication Date: 2025-11-23 21:00:00
In recent years, the open web has felt like the Wild West. Creators have seen their work scraped, processed, and fed into large language models—most often without their consent.
It became a free-for-all, leaving site owners with almost no way to opt out or protect their work.
There have been efforts such as Jeremy Howard’s llms.txt initiative. Like robots.txt, which allows website owners to allow or block website crawlers, llms.txt provides rules that do the same for crawling bots from AI companies.
However, there is no clear evidence that AI companies follow llms.txt or adhere to its rules. Additionally, Google has specifically stated that it does not support llms.txt.
However, a new protocol is emerging that gives website owners control over how AI companies use their content. It can become part of robots.txt, allowing owners to set clear rules for how AI systems can access and use their websites.
IETF AI Preferences Working Group
To address this issue, the Internet Engineering Task Force (IETF)
