What a 102-article test says about using AI tools for literature reviews

A preprint posted to arXiv on 3 November 2026 tests six general-purpose AI tools against a published, human-written review of the Spanish plume weather pattern and its 102 articles. The authors, from the University of Manchester and the University of Washington, report that "No single tool performed best across all tasks" and end with eight recommendations for researchers.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude. Two of the six tools tested here, Claude Code and Claude's web interface, are Anthropic products.

The set-up

• Online assistants: ChatGPT Deep Research, Gemini Deep Research and Claude's web interface, each in its research mode.
• Coding agents: Codex and Claude Code.
• Retrieval-grounded assistant: NotebookLM, which answers from uploaded documents.

All six were used on their provider's entry-level paid plan, with reasoning effort at maximum where offered. The benchmark review was withheld from every tool. The tasks were: find the articles, interpret each one, synthesise across them and propose future work.

Finding the articles

The search table has five tool rows and two summary rows. Codex found 42 of the 102 articles, ChatGPT Deep Research 40, Gemini Deep Research 16 (it reported 35 but named 16), Claude Code 15 and Claude's web interface 7. Together they found 57; 45 were found by none, although the authors say some "may be among Gemini's 19 unnamed articles".

The paper's search discussion is titled "Access, not capability, limited what the tools found", and its abstract reports that "Institutional library access increased discovery from 42 to 82 articles for Codex and from 15 to 79 for Claude Code." Even after login, the authors write, "Claude Code still found only 39% of those published before 2000", against 72% for Codex, though after login Claude Code found 93% of articles from 2020 onward against Codex's 74%. The American Meteorological Society's site "blocked Claude Code entirely but admitted Codex."

The authors caution that across the five initial runs, login and browser state "were not controlled uniformly", and that the login test "cannot separate" subscription access from getting past sites that had blocked the tools.

Reading each article

Codex, NotebookLM and Claude Code each built a 17-column database from the 102 PDFs. The scoring table has 16 measures. All three correctly identified the 18 articles that discuss the Spanish plume substantially. On the 84 passing mentions, Codex misclassified 66 (79%) as substantial, NotebookLM 65 (77%) and Claude Code 34 (40%).

On quotations, all 100 of Codex's and all 101 of Claude Code's were word for word (Claude Code's page number was wrong for 3); NotebookLM altered the wording of 7 of its 101. On spotting a prefrontal trough, recorded in seven articles, Codex got 1, Claude Code 4 and NotebookLM 6, with 0, 10 and 33 false positives respectively.

Writing the synthesis

Two experts scored the three syntheses blind. Reviewer A, lead author of the benchmark review, gave mean scores of 4.32 to Claude Code, 3.68 to NotebookLM and 3.34 to Codex. Reviewer B gave 2.92 to Codex, 2.82 to Claude Code and 2.78 to NotebookLM. The paper notes that Reviewer A "was potentially biased by being too close to the subject matter", and says the conclusions "rest on what both reviewers found, not on any ranking of the tools."

Reviewer B called the outputs "fine, but quite uninsightful". All three tools overstated agreement in the literature. NotebookLM's two direct quotations were both wrong; Codex used none; every Claude Code quotation checked (those in the first two of its ten themes) was accurate. Claude Code cited 79 of the 102 articles, covering all 18 substantial ones, and Reviewer A noted its length meant "more opportunities to get it wrong." Codex covered 13 of the 18 with 27 citations, and the paper describes its writing as "the most stilted and awkward of all three LLMs".

Proposing future work

Codex and Claude Code each reached versions of the benchmark review's own recommendations; NotebookLM reached one. All three proposed gaps broader than the Spanish plume.

What the paper recommends

In summary:

1. "Search with subscription access to improve the results", and audit where the tool was blocked.
2. Search publishers' websites and databases directly, as a gap "may therefore reflect lack of access rather than the lack of literature."
3. Search separately for older work.
4. "Judge the importance of each article yourself."
5. Write a detailed prompt: "The prompt shapes the answer".
6. Check a synthesis's claims and quotations against the original sources.
7. Remember that a tool "tends to sound confident even when it is wrong."
8. "Expert review by humans is still needed."

Limitations

The paper tested "one topic and one set of 102 articles", each tool ran once per task, and chance variation between runs "was not measured". It covers versions available in July and August 2026, and the authors "cannot know" whether the models were trained on the benchmark review. Each task used a single prompt and no other wording was tested; the synthesis prompt was written to prevent errors and did not ask for interpretation, and the authors say the low insight ratings "may therefore reflect the prompt as much as the tools". The reviewers agreed exactly on 24% of ratings and within one point on 67%, and specialised literature-review tools were not compared.

Why it matters

The authors conclude that the tools "could help with a literature review but cannot be left to write one alone, at least at this time". On our reading, the login result means a tool's search score here describes its access as much as the tool itself.