“Manners for Machines”: How new rules could prevent AI scrapers from destroying the Internet

Australians are among the most concerned people in the world about artificial intelligence (AI).

This fear is driven by fears that AI will be used to spread misinformation and defraud people, by fear of job losses, and by the fact that AI companies train their models on the expertise and creative work of others without compensation.

AI companies have used pirated books and articles and routinely send bots across the internet to systematically collect content for their models to learn from. This content can come from social media platforms such as Reddit, university repositories of academic papers, and authoritative publications such as news outlets.

There used to be a kind of détente policy when it came to online scraping. Although scraping was sometimes technically illegal, it was necessary for the Internet to function. For example, without scraping there would be no Google. Website owners embraced scraping because it made their content more available, in keeping with the “open web” vision.