Performance of a large language model on the reasoning tasks of a physician
Summary
More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinion
Content
# Performance of a large language model on the reasoning tasks of a physician
*Published: 2026 Apr 30*
More than 65 years ago, complex clinical diagnostic reasoning cases were
introduced as the gold standard for the evaluation of expert medical computing
systems, a standard that has held ever since. In this study, we report the
results of a physician evaluation of a large language model (LLM) on challenging
clinical cases across five experiments with a baseline of hundreds of
physicians. We then report a real-world study comparing human expert and
artificial intelligence (AI) second opinions in randomly selected patients in
the emergency room of a major tertiary academic medical center. In all
experiments, the LLM outperformed physician baselines and displayed continued
improvement from prior generations of AI clinical decision support. Our study
suggests that LLMs have eclipsed most benchmarks of clinical reasoning,
motivating the urgent need for prospective trials.
DOI: 10.1126/science.adz4433