New technologies like AI come with big claims – adopting the scientific concept of validity can help cut through the hype

Technological innovation can seem relentless. In computer science, some have proclaimed that “a year in machine learning is a century in any other field.” But how do you know if these advances are hype or reality?

With a flood of new technologies, failures quickly occur, especially if these developments have not been properly tested or fully understood. Even technological innovations from trusted laboratories and organizations sometimes result in spectacular failures. Consider IBM Watson, an AI program that the company hailed in 2011 as a revolutionary cancer treatment tool. However, instead of evaluating the tool based on patient outcomes, IBM used less relevant – perhaps even irrelevant – measures such as expert reviews instead of patient outcomes. This resulted in IBM Watson not only failing to provide doctors with reliable and innovative treatment recommendations, but also suggesting harmful recommendations.

When ChatGPT was released in November 2022, interest in AI grew…