Plate Nº 31 · recorded November 9, 2026

Health & Medicine ResearchReported finding

AI's benchmark diagnostic edge came with four times the questions

A Science diagnostic benchmark favored AI, but its fourfold question load, simplified missing data, and lack of continuous learning kept it far from ordinary clinical care.

By Priya Raman5 min read975 words

In brief

  1. AI asked four times as many questions as physicians in the Science diagnostic benchmark.
  2. Iterative prompting improved accuracy by about 8% in NEJM cases and 4.5% in ED cases over a non-iterative baseline.
  3. AI retained 80.9% accuracy in NEJM cases at an estimated testing cost of $2,241.
  4. The benchmark drew on two case collections: NEJM weekly clinical-pathology cases and emergency-department admissions.
Does AI diagnose better—or take a different test?
Plate Nº 31Does AI diagnose better—or take a different test? — HelenJank / pixabay

An artificial intelligence (AI) diagnostic system asked four times as many questions as physicians, and the Science study suggested significantly better accuracy within its controlled benchmark (a standardized comparison).

Its iterative version improved accuracy over a baseline prompt without iterative diagnostic steps by about 8% in New England Journal of Medicine cases and 4.5% in emergency-department cases.

At the end of a simulated consultation, the model could ask for another history, request a physical examination, or order a test. Physicians and the AI then judged whether the evidence justified a diagnosis or another round of investigation.

The analysis frames diagnosis, the first clinical step in care, as “an iterative process in which physicians gather information from patient histories, physical examinations, and diagnostic tests to identify a medical condition.” As evidence changes, clinicians revise possible diagnoses and decide when certainty supports treatment.

Most earlier studies gave AI the full diagnostic query, “compressing diagnosis into one inference step,” according to the analysis. That approach tested a one-shot answer rather than the back-and-forth required for clinical diagnostic competence.

How did the study test diagnostic skill?

The researchers assembled the test from “bite-size” diagnostic cases drawn from two collections. One comprised the New England Journal of Medicine (NEJM) weekly clinical-pathology cases (a series featuring interesting, rare, or difficult diagnostic problems). The other comprised emergency-department (ED) admissions, closer to the daily diagnostic fare that clinicians encounter.

Both groups could seek additional history, examinations, and tests, and both could decide when enough evidence existed to diagnose. The AI design drew on multiple simulations to choose its next move; physicians worked independently and had no specialist consultation available.

The ED sample did not represent the full range of complaints physicians routinely evaluate. It included only patients sick enough to require admission.

Researchers gave the model a certainty that real practice lacks. When information was unavailable, it generated plausible responses that offered no additional diagnostic clue. A patient may omit a symptom, tests may conflict, and disease may continue to unfold.

That setup removed much of the need to judge which evidence deserves trust.

The benchmark comparison pointed to significantly better AI accuracy, but the system also imposed additional costs. When instructed to be more parsimonious (using fewer tests), it retained much of that accuracy at lower cost. In the NEJM set, accuracy remained 80.9% at an estimated testing cost of $2,241.

Those figures used hospital-published charges rather than actual costs.

What did the benchmark leave out?

  • Case choice favored the AI’s strengths. The NEJM set deliberately featured interesting, rare, or difficult cases. AI systems also encounter disproportionate exposure to such cases, and their stored memory remains available. Physicians’ exposure and memory reside in experience, which rarely reaches the same breadth and is harder to search.

  • The comparison pitted a coordinated AI system against individual physicians. It did not test physicians consulting one another or calling specialists, a resource they might ordinarily use.

  • The simulation removed consequential uncertainty. Missing information produced tidy, plausible answers rather than silence, conflict, or ambiguity. That made it harder to reproduce a clinician’s judgment about evidence reliability.

  • Each question has a human cost. Extra questions consume patient attention, require interpretation and clarification, and sometimes demand reassurance. A cooperative electronic intermediary does not resemble a frightened, confused, or exhausted patient.

These limits narrow the result without erasing the AI’s advantage inside the benchmark. The study tested particular patients, information rules, comparison groups, and definitions of correctness. It did not estimate the relative accuracy of AI and physicians in ordinary practice.

Why did iterative reasoning improve accuracy?

Training data for AI models encompass enormous collections of medical literature, case histories, and discussions spanning specialties and rare conditions. Human evaluation and annotation (people marking what information means) shape those collections through comments, statistical mentions, and peer review. The weighting (relative influence) of each item can favor whatever appears in the training material.

During the test, the model did not acquire new medical facts. It reorganized existing knowledge in response to each case, compared alternative explanations, and used the comparison to select the next question or test. This process is sequential inference (step-by-step reasoning).

The 8% and 4.5% gains therefore reflected better organization and use of stored knowledge, not new learning from the patient. The result is a genuine improvement over one-pass prompting (a request for one answer) and a closer model of how a diagnostician works.

AI also brings rapid, broad recall and unusual consistency to the task. Its performance is not constrained by fatigue, a particularly memorable recent patient, or one clinician’s finite recall, although training data and prompts still shape its outputs.

The system can also simulate several specialist perspectives when real specialists are unavailable, bringing more diagnostic patterns to bear on a case.

Can AI replace clinical judgment?

Clinical judgment means deciding which evidence matters and when uncertainty justifies treatment rather than another test. A physician develops that capacity through repeated cycles of observation, interpretation, prediction, intervention, and feedback. Experience also teaches which remembered pattern deserves attention and when to reconsider it.

The system did not model intervention or its short- and long-term consequences. It also did not demonstrate reflection or persistent learning (independent revision from the consequences of its own actions). The analysis puts the distinction plainly: “AI is currently much better at exploiting accumulated knowledge than at accumulating consequential experience of its own.”

AI may help generalists (doctors who manage a broad range of conditions) confront uncommon disease by rapidly bringing many diagnostic patterns to bear. But the benchmark cannot show whether that performance improves care amid unreliable histories, conflicting findings, and evolving illness.

The study demonstrates structured reasoning under controlled conditions; whether AI can learn independently from consequences remains the decisive question for clinical judgment.

via acsh.org (Source)

Filed under

  • ai-diagnostics
  • medical-diagnosis
  • clinical-benchmarks
  • iterative-reasoning
  • physician-comparison
Share this article:

More from Priya Raman

Priya Raman

Show full bio

Senior reporter covering people, launches and day-to-day practice in general science news at SciBeat.

432 articles

Nearby plates

« Previous articleNext article »