Brain Scans Show AI Chatbots Don't Track Who Did What to Whom

The 918 same-vocabulary pairs told a sharper story, matching the preregistered performance order. The word-averaging model showed a strong negative relationship with brain activity, because it rated role-swapped sentences as nearly identical while the brain treated them as distinct. Transformers beat that baseline but still correlated negatively overall. Only the hybrid and graph-based models achieved a modest positive alignment — and the difference between them was not statistically significant.

In absolute terms the correlations were small, roughly -0.05 for the transformer average and 0.07 for the hybrid model, typical of noisy fMRI data, though the pattern held consistently across participants. The authors stress their takeaway is not that their hybrid model wins; it is that word-averaging and transformer representations simply are not sensitive enough to who-did-what roles.

What does this change for everyday AI users?

Not much yet, by the authors' own admission. "Whether this has practical implications for human use of LLMs in real-world tasks was beyond the scope of our study," Fodor said. The practical risk he flags is subtler: "Just because a computer model can perform a task to a human level, it doesn't follow that the model is doing so in a human-like way," which can produce "unexpected behavior or failure modes when the model is deployed outside of the context in which it was trained."

The experiments behind the claim

The researchers recruited 39 native English speakers, ages 18 to 40 with an average age of 22, at the University of Melbourne. After excluding nine for head motion or poor attention-check scores — a threshold lowered from the preregistered 75 percent to 70 percent, which added one participant — 30 remained. Each person read all 108 sentences four times, 2 to 7 seconds per sentence, inside the scanner, and answered 40 comprehension questions.

The team analyzed activity from the middle three seconds of each trial rather than the preregistered final three seconds; the final-window analysis gave similar but weaker results. Using representational similarity analysis — checking whether each model ranked sentence-pair similarities in the same order as the brain — they found the structured models aligned best in temporal regions, though every subregion, frontal ones included, responded to both vocabulary and structure.

A layer-by-layer look at Llama 3 showed slight positive relationships with brain activity only in layers 2 and 3, turning negative from layer 4 onward. Controlling for visual cortex activity caused even those early-layer correlations to drop sharply, which the authors read as a reflection of superficial features like text length. And on same-vocabulary pairs, the graph model's strongest alignment sat in visual cortex — a possible confound one peer reviewer flagged.

Humans rated the sentences too

In a complementary Prolific study, 502 native English speakers from Australia and the US, average age 30, rated sentence pairs on a 1-to-7 similarity scale; 486 survived attention checks. Across all pairs, transformers aligned best with human ratings. On the 918 same-vocabulary pairs, word-averaging fell to last place and the hybrid model matched human judgments most closely. Notably, human raters sat between the transformer and hybrid models on role-swapped pairs.

In an exploratory, non-preregistered analysis, the researchers asked GPT-4 directly to score each pair's similarity from 0 to 1. Its direct ratings matched human judgments better than any model's internal embeddings — evidence, the authors argue, that LLMs can use grammar to answer questions even while their internal representations do not organize it in a brain-like way.

The caveats — and there are several

eLife's editorial assessment rated the evidence solid but noted the stimuli were highly controlled: 108 single-clause sentences with repetitive structures, not natural dialogue. The transformers tested were mostly sentence-embedding systems plus Llama 3; the internal workings of GPT-4 or Claude as chatbots were not analyzed. The online ratings correlated almost not at all with fMRI patterns on same-vocabulary subsets, possibly because actively comparing pairs differs from passively reading single sentences. The fMRI analysis also averaged over one three-second window, flattening the word-by-word process by which the brain builds meaning, and comparing vector and graph models required different mathematical similarity metrics that can affect fit.

The findings also sit in tension with earlier work. An intracranial recording study of unscripted conversation found neural activity aligned more strongly with higher-level semantic layers of language models — though that study examined spontaneous dialogue, not controlled sentences isolating word order. Conversely, the results align with a neuroimaging study showing the brain tracks hierarchical structure without statistical cues.

"Our results do not show that LLMs cannot produce relevant or meaningful language, only that they likely do not understand such language in the way humans do," Fodor said. Future work, he added, will tackle how LLMs represent more complex sentences and narrative structures — territory that remains largely unexplored.