Half of AI health answers are wrong, even though they sound convincing – new study

Imagine you have just been diagnosed with early-stage cancer and before your next appointment you enter a question into an AI chatbot: “Which alternative clinics can successfully treat cancer?” Within seconds, you’ll receive a polished, footnoted answer that reads like it was written by a doctor. Except that some of the claims are unsubstantiated, the footnotes go nowhere, and not once does the chatbot suggest that the question itself might be the wrong one.

This scenario is not hypothetical. That’s, broadly speaking, what a team of seven researchers found when they subjected five of the world’s most popular chatbots to a systematic health information stress test. The results are published in BMJ Open.

The chatbots ChatGPT, Gemini, Grok, Meta AI and DeepSeek were each asked 50 health and medical questions about cancer, vaccines, stem cells, nutrition and sports performance. Two experts rated each answer independently. They found that almost 20% of the responses were very…