
Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
Monica Munnangi, Saiph Savage
Working Paper
|
July 14, 2026
Patients seeking medical information often ask questions that embed incorrect assumptions or misconceptions. In such cases, safe medical communication requires not only answering the question, but identifying and correcting the underlying false belief. These interactions naturally unfold over multiple turns, a pattern now mirrored in interactions with LLMs. Yet current evaluation frameworks do not capture model behavior in these settings, where misconceptions can emerge, persist, or evolve over the course of a conversation. Whether LLMs can reliably correct such misconceptions over time remains largely unexamined. To study this, we introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 patient-physician conversation threads comprising 8,204 question-answer pairs, derived from real patient interactions on AskDocs. This dataset enables systematic evaluation of whether models can detect and correct misconceptions under a multi-turn context. We evaluate five LLMs using a rubric-based LLM-as-a-Judge framework that scores responses based on their ability to identify and correct misconceptions. Our experiments reveal a consistent pattern: even frontier models that can address misconceptions in a single interaction degrade substantially over subsequent turns. GPT-5 and Claude-Haiku correct these false presuppositions around 85% on initial questions but drop to roughly 50% within two follow-ups. An oracle analysis replacing prior model outputs with physician responses shows that much of the degradation is driven by error propagation, while performance remains imperfect even under correct context. Even when models tend to correct misconceptions initially, their performance degrades substantially over later turns, leading to inconsistent and potentially unsafe guidance in patient-facing settings and highlighting the need for evaluation frameworks that capture multi-turn behavior.
citation
Munnangi, M., & Savage, S. (2026). Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations [Preprint]. arXiv. https://doi.org/10.48550/arxiv.2607.12884
topics
Artificial Intelligence
Health & Wellbeing
Artificial Intelligence
Artificial Intelligence
related work

Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper


Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper


Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper

.png)