KNOWS benchmark: best web agent fully succeeds on fewer than 3% of office tasks

A new benchmark from researchers at the University of Utah, posted to arXiv on 24 September 2026, finds that the best agent it tested "fully succeeds in fewer than 3%" of its 110 tasks, which ask agents to search the live web and turn what they find into Google Docs, Sheets and Slides.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude, and Anthropic's Claude Opus 4.7 is one of the three named models in this paper's results table, including as the backbone of the top-scoring system.

What KNOWS is

The paper, The Hard Part Comes After Search, introduces KNOWS (Knowledge Navigation and Organized Web Synthesis). Its nine authors are Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen, Neha Bhat, Parker Henry DeYoung, Fateme Hashemi Chaleshtori, Nathan Stringham, Kenneth Marino and Ana Marasović; the paper lists the University of Utah as the affiliation. The arXiv listing says it has been accepted to Findings of EMNLP 2026.

• 110 tasks: 25 documents, 40 slide decks and 45 spreadsheets.
• Each agent gets a natural-language instruction (about 200 words on average), a live web browser and Google Workspace, and must produce a finished artifact.
• The authors estimate the tasks take humans "over 2 hours" on average. They did not run a controlled human baseline, but say every task was completed by an author during construction and passed all evaluator checks.
• The authors describe it as "the first benchmark spanning live web search, productivity tool use, and visual/spatial understanding".

How it is scored

Each task is split into checkpoints and 2,716 evaluation steps in total, checked by programs that mix deterministic checks with LLM judgements. The authors report four metrics, from strictest to loosest:

• SR (success rate): a task counts only if every step is correct.
• ASC: share of checkpoints fully completed.
• ACF: average fraction completed within each checkpoint.
• SF: fraction of correct steps.

Against an independent expert on 100 judgements, the evaluators agreed with a Cohen's kappa of 0.64 and 82% pairwise accuracy, the paper says.

The results

Table 3 has seven rows. All overall figures, in per cent:

Comet's 12.0% on Docs is the single non-zero success rate in any split; every system scored 0 on Sheets and Slides. The paper notes visual inputs were not available through the DeepSeek V4 Pro API at the time, so it has no screenshot row. The paper also says: "Among the general agents, GPT-5.5 performs relatively well compared to other models." The authors say they "deliberately exclude Google’s own models" "to partially mitigate the training confound" of models trained on Google Workspace.

What the authors draw from it

• They write that "the agent harness matters as much, if not more, than the underlying model": the same Opus 4.7 model scored 2.7 points higher on SR and 25.6 higher on SF inside Comet than in its text-plus-screenshot BrowserGym run.
• Adding screenshots "can significantly improve performance", they say.
• Partial scores can mislead: one Comet slide deck scored 0.54 ACF and 0.59 SF but, in the paper's words, "remains practically unusable".
• Their conclusion: current agents "can often find the right information, but frequently fail to synthesize, organize and display that information into coherent, usable artifacts." They call for "progress on tool use, visual understanding, and long-horizon reasoning".

The paper's claim is about agents "acting as end-to-end assistants" on these tasks; on our reading, it does not make a broader claim about general capability.

Limitations the authors list

• Domain coverage: 110 tasks "may not cover the full breadth of domains encountered in real-world web use".
• Live web: validity "is partially contingent on the current state of the web at evaluation time", and evaluation scripts "may break"; they commit to periodic re-testing.
• Artifact scope: only Google Sheets, Slides and Docs. A small five-task check of Comet on Microsoft 365 found it did better on two and worse on two; the authors conclude that "performance is not driven by optimization on the Google Workspace environment".

On evaluation reliability, the authors report that their evaluators agree with expert judgment "with a Cohen’s [kappa] of 0.64 and 82% pairwise accuracy, in line with related open-ended agent benchmarks", and, in a second check of 100 judgments on LLM/VLM-judged steps only, "agreement with a Cohen’s [kappa] of 0.69 and 89% pairwise accuracy". On our reading, that is reasonable but not perfect agreement.

The acknowledgements thank Google for funding the benchmark's construction, and Anthropic and OpenAI for donated API credits.

Why it matters

On our reading, KNOWS adds evidence that partial-credit scores on agent benchmarks can sit well above the rate at which an agent delivers a finished, usable file, and the authors conclude the harness matters as much as the choice of model. Its authors say the benchmark and evaluation code are being released for others to test against.