Researcher reports hybrid AI models can learn to do a simple reasoning task silently in their recurrent state

In a post on his personal blog dated 22 October 2026, Divyansh Agrawal reports that a hybrid language model, OLMo-Hybrid-7B, learned to track a flipping "light switch" inside its recurrent layers when it was fine-tuned to answer with no visible working, without writing any working that a chain-of-thought monitor could read. This is a single-author post, not a peer-reviewed paper. It covers one task (parity), two hybrid models plus a dense control, and an artificial training setup, and Agrawal ties the difference between the models to one setting: the write strength β in their Gated DeltaNet (GDN) layers.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude. Researchers from Anthropic are among the co-authors of the chain-of-thought monitorability position paper we cite below for background.

The bet he is testing

Chain-of-thought (CoT) monitoring means reading a model's written working for signs of misbehaviour. A July 2025 position paper by authors from several AI labs and safety groups called it "a unique opportunity for AI safety", and warned: "Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability." Under "Novel architectures" it added that latent reasoning models "might not need to verbalize any of their thoughts and would thus lose the safety advantages that CoT confers."

Agrawal frames his work around the same bet: "a model pulling some sketchy moves needs to reason hard, and if it's reasoning, it needs to write that reasoning down as CoT, since models can only do so much serial work per token." Hybrid models, he writes, "carry state across tokens", so the question is whether they can reason there instead.

What he did

• Models: OLMo-Hybrid-7B, which he says lets β go up to 2; Qwen 3.5 4B, which he says caps β at 1; and OLMo-3-7B, a dense transformer, as a control.
• Task: track a light switch through many flips, then say "on" or "off". That is parity, which he calls "the simplest thing that needs state tracking".
• Baseline: asked to answer straight away, none of the untrained models could track the switch past 4–8 flips, he reports.
• Training: he fine-tuned each model to answer only "on" or "off", using LoRA plus the β and decay projections, on a curriculum up to 64 flips, then tested at longer lengths.

What he reports

• OLMo-Hybrid generalised at 128 flips. Averaged over six seeds it scored 0.89 at 128 flips, twice the training length. Five of the six runs learned the training range, and all five scored 0.89 to 1.00 at 128; the sixth never learned 64 flips.
• The others did not. Qwen 3.5 scored 0.55 at 128 flips and OLMo-3 0.54. The repository's results table puts chance at 0.50.
• Capping β removed the effect. The same OLMo-Hybrid with β capped at 1 scored 0.52 at 128 flips on all six seeds. Agrawal notes the cap "also made the training range itself harder": only 2 of the 6 capped runs properly learned 64 flips. He reports the cap raises perplexity on ordinary text by about 3.5%.
• The effect fades with length. The README table, not the blog post, gives the uncapped OLMo-Hybrid's mean as 0.71 at 192 flips and 0.54 at 256.
• Where the bit lives. Swapping the recurrent state between sequences changed the answer 0.00 of the time because attention kept "a backup copy of the bit", he reports; swapping the state while knocking out attention moved it 1.00 of the time. He locates the bit in layer 16 head 2 and layer 17 head 4, and writes that "a monitor that only looks at the recurrent state would have missed a real hidden tracker."

His limits

Agrawal lists them himself: six seeds per arm for the main OLMo-Hybrid comparison and three for the rest; one task; a three-cup shell game that no model learned on his training budget; and a four-switch task that both OLMo-Hybrid arms learned at 16 flips but failed at 64, "so it can't tell the two apart". He also writes: "The pressure is artificial too: I directly trained for answer-only, which is a stand-in for training that pushes models to skip their working, not a real RL setup." He cites prior work by Tomasz Steifer showing a Qwen-style hybrid can still solve a parity-style task in a single pass by leaning on attention as well as the RNN, so "the real question isn't 'what's possible', it's 'what does training actually find'."

The code, raw results and notes are on GitHub under the MIT licence.

Why it matters

Agrawal's own conclusion is that whether hidden reasoning in the RNN state generalises "comes down largely to the RNN's β", which he calls "mainly an architecture choice". He asks hybrid designers to "please consider keeping β at 1", and suggests two checks for anyone evaluating a new hybrid: what range its eigenvalues can reach, and "whether the swap-plus-knockout test finds state that attention is hiding".

On our reading, the finding is narrow: one bit of state, a training setup built to suppress working, and generalisation that, on his own table, is close to chance by 256 flips. Its interest is that it links a monitorability question to a specific architecture setting in released hybrid models, which is the kind of development decision the 2025 position paper asked developers to weigh. Whether the same holds for harder tasks, other hybrids or real reinforcement-learning training is not something this work tests.