Preprint: GPT-4o set-ups with similar accuracy put about a quarter of 1,200 PMQs in different topics

A preprint posted to arXiv on 4 November 2026 by Louise Yorke of University College London tested how consistently OpenAI's GPT-4o sorts Prime Minister's Questions into policy topics, then checked what 37 UK and EU policy documents ask public bodies to report about that kind of variation. It is a single-author paper and has not been peer reviewed.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude. The paper's table of experimental dimensions lists a "Claude Sonnet under the anchor instructions" setting from the author's original experiment, but it is not one of the eight configurations analysed and the paper reports no Claude results.

What was tested

The data is the UK Comparative Agendas Project (CAP) set of Prime Minister's Questions from 1997 to 2008, in which human coders gave each question one of 21 policy topics. The author drew a stratified sample of 1,200 questions; accuracy means matching the human label.

The "anchor" set-up used GPT-4o with full category descriptions, a coding rule, six worked examples, JSON output and temperature zero. The seven alternatives were descriptions only, no examples, reasoning before the label, chain-of-thought, adding the historical period, adding the MP's party, and free-text output. These eight survive from a 17-configuration run in April 2026; the paper says raw outputs "were lost when the cloud environment storing them was deleted".

The headline numbers

• Accuracy: 68.2% (free-text output) to 70.3% (historical period given) against the human labels (anchor 69.6%). None of the differences is statistically significant after correction, the paper says.
• Different labels across set-ups: of the 1,199 questions with a label from all eight configurations, 291 (24.3%) got different labels from different set-ups. For 215 (17.9%), the label matched the human coders under some set-ups but not others.
• Identical runs: the anchor was rerun five times on 1 September 2026 with nothing changed. Pairs of runs disagreed on 2.78% of labels on average, and 65 questions (5.42%) changed label at least once. In the paper's words: "Setting the temperature to zero does not make this pipeline deterministic."
• Agreement is not correctness: for 191 questions all eight set-ups gave the same label, and it differed from the human coding. The paper says this does not show who is right.
• Dropping disputed questions: keeping only the 908 questions on which all eight agreed raised the anchor's accuracy from 69.6% to 79.0% without any label changing. The paper says: "The higher score describes a selected subset on which performance is better, not an improved classifier."
• Review routing: a hypothetical rule sending outputs scoring below 0.99 to a human flagged 36.0% under the anchor and between 0.9% (chain-of-thought) and 41.0% across set-ups. The author warns the score may not reflect confidence in the final label; no reviewers took part.
• The bigger picture: the same five topics came out most common under every set-up, but under three of them International Affairs overtook Government Operations as the largest. That did not happen in any identical rerun.

What the policy documents ask for

The documentary half covers 19 UK and 18 EU documents current at 26 October 2026. The UK set includes the Government Digital Service's AI Playbook, the Algorithmic Transparency Recording Standard guidance and template, AI Insights, the cross-government AI Testing and Assurance Framework, National Audit Office guidance, an Equality and Human Rights Commission checklist, Ofgem's energy-sector guidance and DWP's AI security policy. The EU set includes the AI Act and its 2026 Digital Omnibus amendment.

The gaps the paper identifies are in reporting. It says both sets "expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing." It also says "Some UK guidance explicitly addresses stability": a guide for public auditors from six audit institutions, including the National Audit Office, recommends running the same prompt several times. The paper says the cited passages do not prescribe a common agreement statistic.

It finds the UK documents emphasise public bodies checking their own use, while the AI Act places substantial testing duties on providers but leaves them choices over metrics and thresholds.

What the author recommends

For "consequential informational uses", the paper says guidance "could ask organisations" to report variation across repeat runs and plausible set-ups, shared errors, evaluation exclusions and whether conclusions hold, and to examine how scores route outputs to human review. It calls these "candidate checks motivated by this study". It proposes a short assurance record (model version, configuration, exclusions, review rule), calling this "a proposal for supporting scrutiny, rather than a reporting intervention whose effectiveness the study has tested."

Limitations the author lists

"The experiment is exploratory: it covers one task, one model and a small set of configurations". The reruns used an unpinned model alias about five months after the original runs, so the paper says comparisons cannot separate configuration effects from changes over time. The documents were coded with AI assistance and checked by the author, and "no inter-coder agreement rate is reported". The paper adds that the experiment "does not establish effects on actual public-service decisions or harm."

Why it matters

The paper's concern is public bodies relying on AI-produced classifications and summaries. It cites the government's transparency record for the Consult tool, which describes a tool under development to extract themes from consultation responses for policymakers. On our reading, the practical point for UK departments is that an accuracy figure alone would not have shown that a quarter of these questions moved between topics, and the author argues current guidance does not consistently ask for that to be reported.