The takeaway: Last week the NHS committed £10bn over three years to a technology drive whose centrepieces are AI — triage in the NHS App, ambient scribes rolling out from outpatients, Copilot for half a million staff. A month earlier, the medicines regulator quietly published the most detailed evidence any UK body has produced on how this class of technology actually behaves in clinics — from its own AI sandbox. The two documents make an uncomfortable pair. The MHRA's testing found that without active guardrails, an ambient scribe drifted beyond its intended purpose in roughly 39% of real-world notes; that disclaimers do nothing to stop it; and that the more reliable a tool gets, the less rigorously humans check it. The rollout is funded at £10bn. The programme that produced those findings runs on £1.2m a year.

What the regulator's sandbox found

The MHRA's AI Airlock — its regulatory sandbox for AI medical devices — published its Phase 2 report on 9 June, covering seven real products tested with regulators in the loop, including the ambient scribe TORTUS, an NHS England discharge-summary tool, and triage and detection tools for skin cancer and genetic eye disease. It has had a fraction of the attention the rollout announcements got, and it is the more important read.

The sharpest finding concerns scope creep by hallucination. In the TORTUS case study, the report says, "without active guardrails, out-of-scope performance was detected in approximately 39% of real-world notes, falling to 20% with guardrails active". The report's plain-English warning: "A product that is positioned away from direct diagnosis at the point of design can still provide such functions through hallucination." In other words, a note-taking assistant can become a diagnosing device — not by design, but by drift.

Three more findings deserve to be pinned above every procurement desk:

• Disclaimers don't work. They "function as passive controls; they do not prevent generative AI systems from producing content that constitute functionality outside of the stated intended purpose".
• Lab results don't transfer. One anti-manipulation guardrail "significantly reduced scope drift in virtual testing but had no significant effect in real-world deployment" — real users simply weren't attacking the system; the failure modes were elsewhere.
• Reliability erodes oversight. TORTUS achieved real-world precision of 0.989 — and precisely because of that, the report warns, "human review may become less rigorous, making residual errors less likely to be detected". The better the tool, the softer the safety net. The report also concludes that "LLM-as-a-judge evaluation was confirmed to be insufficient as a sole assurance mechanism".

None of this is an argument that the tools don't work — TORTUS's numbers are genuinely strong, and the sandbox exists because the MHRA wants these products to reach patients. It is an argument about what has to surround them.

The uncomfortable arithmetic

Six days ago, NHS England announced its acceleration: AI triage to 200,000 patients within a year and every NHS App user by April 2028, ambient scribes spreading from outpatients, Microsoft Copilot for more than 500,000 staff, £41bn in claimed benefits over a decade. The Health Secretary, James Murray, said he was "certain the technological innovations I've chosen to prioritise will get patients to the right care faster…"

The AI Airlock — the programme that produced the 39% finding — received a £3.6m funding boost over three years in April: £1.2m a year, or about 0.04% of the tech drive's annual spend. That is not a like-for-like comparison — the Airlock is one evidence programme, not the NHS's whole assurance apparatus, and trusts deploy under existing device regulation and fresh ambient-scribe guidance. But the asymmetry between deployment budget and evidence budget is the story of clinical AI in 2026, and clinicians' bodies have noticed: the BMA's workforce lead Dr Amit Kochhar called the AI-heavy workforce plan "a massive, dangerous gamble", adding: "A chat bot cannot look after a patient."

What's still unfinished

The rules these deployments will eventually answer to are still being written. The MHRA's new pre-market device regulations are due to be adopted around December — but their software and AI provisions were deliberately left open pending the National Commission into the Regulation of AI in Healthcare, whose recommendations are due later this summer. The Airlock's Phase 3 is expected to open for applications later this year, and a new London sandbox testing AI devices in live NHS clinics is due to invite manufacturers imminently. Until then, the accelerator and the brakes are being built by the same government at very different speeds — and the regulator's own evidence says the difference matters.

Disclosure: On The Wire runs on Anthropic models — the same class of technology (large language models) whose clinical behaviour these findings describe. We flag it every time.