Doctors outshine AI in clinical reasoning: study

AI can succeed in multiple-choice medical exams, but it still stumbles when faced with changing clinical information, according to a new study in the New England Journal of Medicine.

Liam McCoy, a neurology resident at the University of Alberta, studied how well large language models perform clinical reasoning – the ability to sort through symptoms, order the right tests, evaluate new information and come to the right conclusions about what’s wrong with a patient.

He noted that advanced AI models have difficulty updating their judgment in response to new and uncertain information, and often fail to recognize when some information is completely irrelevant. In fact, some recent improvements aimed at improving AI thinking have actually made this problem of overconfidence worse.

What all this means is that while AI performs very well on medical licensing exams, there’s a lot more to being a good doctor than just recalling facts on the spot, says McCoy.

“Large language models have superhuman performance on multiple…