Plate Nº 65 · recorded November 17, 2026

Science Policy & FundingReported finding

AI Peer Reviews Beat Top Human Experts in Blind Harvard Test

A Harvard working paper found AI-written peer reviews won 100% of AI-judged matchups and made about a third the errors of paid human experts. It is not yet peer reviewed.

By James Calloway5 min read1,021 words

In brief

  1. AI reviews won 100% of 132 decided AI-judged pairs; blinded former editors preferred AI in 87% (26 of 30) of judgments.
  2. AI reviews had a 5.5% error rate versus 15.9% for human experts, with zero fabricated sources in web-enabled reviews.
  3. The study used 124 manuscripts and 134 human reviews from Harvard's paid rapidPeer service, collected 2019–2026.
  4. The paper, by Gary King and Steven Worthington, is a working paper posted online and has not been peer reviewed.
  5. The results were reported descriptively, without formal statistical tests or confidence intervals.
AI-written peer reviews outscored top human experts in blind comparisons, Harvard working paper finds
Plate Nº 65AI-written peer reviews outscored top human experts in blind comparisons, Harvard working paper finds — Pexels / pixabay

AI-generated peer reviews beat paid human experts in 100% of decided head-to-head comparisons, and made roughly a third as many factual errors, according to a Harvard working paper posted online by Gary King and Steven Worthington. The paper — titled "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" — has not yet been peer reviewed itself. Hold that thought.

Peer review, for anyone new to it, is the quality-control step where independent researchers read a draft paper and tell authors and editors whether it holds up. The traditional system is slow, volunteer-stretched, inconsistent, and often superficial. Humans miss errors. Humans show bias toward familiar methods and prestigious institutions.

AI tools bring their own baggage: fabricated citations, excessive leniency, hidden manipulation commands buried in manuscripts. Prior studies split on the question. A 2026 BMJ Open study of 119 manuscripts found human reviewers raised more detailed points than ChatGPT. A 2026 cardiology journal study of 40 manuscripts found the opposite — editors rated Claude, ChatGPT, and Gemini reviews higher on five of seven criteria.

How did the Harvard test work?

King and Worthington built a strong human benchmark, not a weak one. They used rapidPeer, a Harvard pre-submission service where specialists pick outside experts, pay them well, and require 1.5 to 2 pages of constructive feedback on an anonymized manuscript. These reviews were good: 88% of 74 surveyed authors rated them high or very high quality, and former editors called them among the strongest they had seen.

The dataset covered 124 manuscripts and 134 human reviews collected between 2019 and 2026, spanning statistical, mathematical, and qualitative fields. For each paper, the researchers generated a review using Anthropic's Claude Opus 4.8 at its highest reasoning setting — once with the exact rapidPeer instructions and no tools, once with a custom prompt plus internet search, used an average of 12.4 times per review.

To avoid one company grading its own product, they used OpenAI's GPT-5.5 as judge. Every pair ran in both presentation orders, and a win counted only if the review was preferred both times. Three former journal editors also blindly judged 10 randomly selected pairs, yielding 30 human judgments.

What did the numbers show?

The margins weren't close:

  • The AI judge preferred the AI review in 100% of decided pairs (132 of 132).
  • Blinded former editors preferred AI in 87% of judgments (26 of 30).
  • Editors agreed with the AI judge 87% of the time — more than they agreed with each other (80%).
  • On a 0-to-10 scale, AI reviews averaged about 2.6 points higher than human reviews, which averaged about 6.
  • By area, tuned AI won 100% of decided pairs on sources, weaknesses, and methods; 99% on figures and tables; 92% on theory, framing, and communication.

Ten follow-up checks tested whether the AI just wrote longer or prettier prose. When restricted to 44 pairs with similar word counts, the AI still won every time. Restyling human reviews as AI-sounding didn't help them. Matched AI performed nearly as well as tuned AI everywhere except sources, where web search raised the win rate from 62% to 100%.

Accuracy favored the machines too. Web-enabled AI reviews contained zero fabricated sources, and 5.5% of their claims about the paper were judged incorrect, versus 15.9% for humans. A validation trick — inserting one false or one accurate sentence into each review — showed the AI judge caught all 258 false statements and flagged none of the 258 accurate ones. A replication across 416 comparisons, using both Claude and GPT models as reviewers and judges, produced AI win rates of 97.1% and 94.2%.

Note the gaps in this scorecard. The headline results were reported descriptively, without formal statistical tests or confidence intervals. And much of the grading was done by an AI.

What does this change for journals?

Nothing yet — and the authors say so. Their experiment measured only review quality, not whether papers revised after AI feedback end up better. They propose workflows: journals could publish an official AI prompt, require authors to submit the AI review plus a response letter, run a second undisclosed AI check, and call human referees only for hard cases.

Kilian M. Pohl of Stanford, who was not involved, told PsyPost the study "demonstrates that AI-generated reviews can now surpass even human (pre-submission) reviews across several dimensions of review quality." But he cautioned: "Given the overly broad claims made in the abstract, readers might incorrectly conclude that AI is now better than human experts at determining which scientific papers merit publication." More comprehensive reviews, he warned, "should not be conflated with superior scientific judgment." He placed moderate confidence in the results, noting the human editors weren't specialists in every paper's field and "much of the evaluation relied on AI judges, potentially biasing findings to AI-generated writing."

George Currie of UK Research and Innovation, speaking for himself, said the existing system deserves honest scrutiny: "peer review is resource intensive, largely unpaid, variable in quality, functions as journal gatekeeping as much as research improvement, and does not reliably prevent poor research from entering the literature."

The authors themselves flag a collective risk: if researchers across a field lean on similar models, science could drift toward conventional, uniform thinking. Eyal Aharoni of Georgia State said the effect's magnitude surprised him but that "using a separate AI does not guarantee independence since they may query each other." Gustavo A. Patino of Western Michigan University expects AI tools in journal workflows within about five years, while noting reviewers lose the education that comes from reading unpublished work. Michael R. Ladomery of the University of the West of England called the missing revision data a key shortcoming and argued peer review "is an integral part of academic professional life" that "cannot be replaced by AI."

What's needed next, Pohl said, is a large randomized trial inside real journals comparing human, AI-assisted, and fully AI review. Until then, treat this as a striking pilot — not a verdict.

via psypost.org (Source)

Filed under

  • peer-review
  • ai-generated-reviews
  • scientific-publishing
  • research-integrity
Share this article:

More from James Calloway

James Calloway

Show full bio

Staff writer covering regulation, policy and their impact on general science news at SciBeat.

474 articles

Nearby plates

« Previous articleNext article »