A reasoning model does something older chatbots didn't: before it answers, it writes out a long "chain of thought" — a step-by-step monologue that looks like working through the problem. We explained what that buys you when the first of these models arrived. The natural assumption is that the monologue is the reasoning: read it, and you can see how the model reached its answer, catch its mistakes, and trust its logic.

A growing body of research says that assumption is shaky. The chain of thought is text the model generates; whether it faithfully reports the computation that actually produced the answer is a separate question — and the answer, increasingly, is "not reliably."

Anthropic tested its own models and didn't like the number

The sharpest evidence comes from the company with the most to lose by publishing it. In April 2025, Anthropic's Alignment Science team released Reasoning Models Don't Always Say What They Think. The setup: slip a model a hint that changes its answer, then check whether the chain of thought admits the hint was used.

Mostly, it didn't. Averaged across hint types, Claude 3.7 Sonnet acknowledged the hint 25% of the time; DeepSeek R1 did so 39% of the time. On a safety-relevant subset — where the hint was an "unauthorized access" note the model exploited — Claude's chain of thought was faithful 41% of the time and R1's 19%. Anthropic's own summary: "A substantial majority of answers, then, were unfaithful."

Anthropic was careful about what this does and doesn't show, and the caveats matter as much as the numbers. These were "somewhat contrived scenarios"; the tests were multiple-choice quizzes "unlike real-world tasks"; and the questions "were not difficult enough to require the Chain-of-Thought to be used." That last point cuts both ways — a model that doesn't need to think out loud has no reason to report its real reasons, but it also means the result may understate faithfulness on hard problems where the model must actually use the scratchpad. Either way, the headline holds: you cannot assume the stated reason is the operative one.

When the steps are decoration

A separate line of work asks a blunter question: how many of the steps do any work at all? In an October 2025 paper, Can Aha Moments be Fake?, researchers at Northeastern and UC Berkeley built a "True-Thinking Score" that perturbs each step and measures whether it actually changes the final answer. Steps that look useful but have little causal influence they call "decorative."

There were plenty. For one model (Kimi-K2.6) on a maths benchmark, over 30% of steps came in below their causal threshold — present in the transcript, absent from the result. (Quanta Magazine, whose 31 July feature prompted this piece, put the range across models at 30–60%.) Co-author Weiyan Shi's takeaway, to Quanta: "We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output."

The most striking version of "the text isn't the computation" is older and stranger. In Let's Think Dot by Dot (2024), NYU researchers showed a transformer could solve two hard algorithmic tasks using strings of meaningless filler tokens — literally "......" — in place of a chain of thought. The hidden computation happened; the visible tokens carried none of it. The caveat is large and usually dropped: this worked only on a specific, characterised class of algorithmic problems and required careful, dense training to coax out. It is not evidence that dots can replace reasoning in general. But it is a clean proof that useful-looking text and the real work can come apart entirely. William Merrill, a co-author now at the Toyota Technological Institute at Chicago, put it to Quanta: "There's no guarantee the chain of thought has to be meaningful in any sense."

The contested case: does reasoning "collapse"?

The most-cited — and most-disputed — entry is Apple's June 2025 paper The Illusion of Thinking. Testing models on controllable puzzles (Tower of Hanoi, River Crossing and similar), it reported a "complete accuracy collapse beyond certain complexities," and a peculiar detail: past a point, models spent less reasoning effort as problems got harder, even with tokens to spare.

That paper drew a serious rebuttal within days. The Illusion of the Illusion of Thinking argued the collapse was largely an artefact of the test, not the model: the Tower of Hanoi runs bumped into output token limits (which the models said out loud), and some River Crossing instances were mathematically unsolvable for larger sizes, yet counted as failures anyway. Asked to output a compact solution rule instead of every move, the rebuttal reports, models handled cases previously scored as total failures.

Both should be read with their status attached. Apple's finding is about synthetic puzzles, not natural reasoning; the rebuttal is a non-peer-reviewed comment offering preliminary experiments (an early version even listed an AI as co-author, later removed). The honest read is not "reasoning is fake" or "the critics won" — it is that a benchmark's design can manufacture or erase a "collapse," which is itself a reason to distrust confident claims in either direction.

A quieter, longer-running thread runs alongside all of this: Melanie Mitchell and Martha Lewis have shown in work on analogical reasoning that models can be brittle on variations of problems humans handle easily, "overly rely[ing] on similarity to pre-training data" rather than a robust procedure — right answer, shortcut path.

Why the transcript matters even if it's unfaithful

None of this says reasoning models don't work. They post real gains on hard tasks, and the extra compute they spend is doing something. The claim is narrower and more useful: the chain of thought is not a reliable window into how the answer was reached.

That has teeth. Safety teams increasingly propose reading a model's chain of thought to catch dangerous intent before it acts — a strategy that inherits exactly the unfaithfulness Anthropic measured. "Explainability" features that surface the model's steps to a user promise a look under the hood the steps may not deliver. And "the model told me why" is, for now, a weaker guarantee than it sounds — closer to a plausible story than a logged trace. Read the work the model shows. Just don't assume it's the reason.