Epoch AI tested six models on its own research work and found they can’t yet produce Epoch-quality work unaided

The research organisation Epoch AI published a report on Thursday 8 October 2026 called "Can AI automate Epoch?". Its researchers Kelly Hong and Greg Burnham gave six AI models real tasks from their own jobs, and conclude that the models "can’t yet autonomously produce Epoch-quality work."

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude, and this report evaluates Anthropic's Claude Fable 5.1 alongside five models from other developers.

What Epoch tested

The suite has 11 tasks across five categories, all taken from work Epoch staff do:

• Graphic design: Epoch-style diagrams and charts, with access to the team's Figma file.
• Data Insight generation: the short data articles Epoch publishes, including one on AI use in physics papers on arXiv.
• Data Explorer generation: a dataset explorer page on AI mentions in the US Federal Reserve's Beige Book.
• AI data center research: researching two named data centres for Epoch's database.
• Research design: proposing a research project and running a pilot experiment.

Each model ran in its maker's own agent tool at its highest reasoning setting: GPT-6 Astra in Codex, Claude Fable 5.1 in Claude Code, Grok 4.6 in Grok Build, Gemini 3.8 Flash in Antigravity, Kimi K3 in Kimi Code and Qwen 3.8 Max in Qwen Code. Epoch says it followed the principle of providing "as much context to the model as we would to a new hire". A single human grader scored each output against rubrics built with Epoch staff. A score of 100% means the output meets the standard Epoch would expect from an employee.

The scores

Epoch's benchmark page gives a six-row table of mean rubric scores by category:

The report says "Fable 5.1 and GPT-6 Astra are broadly tied in the lead". Both average 65%. The category leads are spread around. GPT-6 Astra scores highest on graphic design and data centre research. Grok 4.6 scores highest on data insights (79%) and data explorers (74%). Fable 5.1 is one point ahead on research design.

What the models could do

Epoch says the two leaders were reliable on "well-defined parts of our tasks, such as coding and computational analysis". It adds that "Computer use was also far more capable than we expected." It dropped one graphic design task from the suite because Fable 5.1 "produced an Epoch-quality output". That task, which Epoch says "was more straightforward than others", was restyling a chart to match Epoch's house style, and Epoch says "We now largely automate this step in our own work."

Where they fell short

• House standards. The report says models "consistently missed implicit conventions that humans would pick up, such as our visual style or what our audience cares about". Its examples are a Fable 5.1 diagram that was "far more information-heavy than our typical graphics" and a GPT-6 Astra Data Insight topic it judged "too niche" for Epoch's readers.
• Too much detail, especially from Fable 5.1. Epoch gives a separate section to this, headed "Fable 5.1 produces outputs with extremely high information density". It says models "tend to introduce unnecessary complexity, especially Fable 5.1". It adds that "Fable 5.1 leans towards more verbose captions, which contrasts with our core design principle of communicating clearly with as little detail as possible."
• Experiment design. Models could suggest good research directions. But the report says "They struggle to design experiments that measure what they claim and treat flawed results as key findings." In one pilot, GPT-6 Astra set a token limit so low that it "cut off 61 of 280 responses". It then presented the result as "sensitivity to the acquisition budget", a framing Epoch calls "misleading".
• Open-weight models. Epoch says these "lag further behind" and struggle even on well-defined parts. Kimi K3 built a whole Data Insight on a filtering error, computing figures "across all arXiv papers, not just physics papers". The report says "None of the frontier closed-weight models made factual errors in their data insights."
• Similar ideas. Given Epoch's polling data, Fable 5.1, Gemini 3.8 Flash and Grok 4.6 all chose the same topic. Epoch says it "did not expect three out of six models to pick the same one".

Caveats Epoch lists

Epoch is open about the limits. Each model ran "only once per task", so "Scores are therefore noisy and may shift with more runs." It says the numbers "should not be treated as ground truth". The tasks only cover Epoch's work, and "Some observed differences may reflect the harness rather than the model itself." The authors say "the qualitative findings are more informative than these exact numbers."

Why it matters

Epoch argues that typical benchmarks "concentrate on easily verifiable tasks that capture only a subset of real work". It gives Kimi K3 as an example: it scores 158 on Epoch's Capabilities Index, "roughly tied" with Grok 4.6, yet it struggled on basic tasks that Grok 4.6 handled more reliably. The report concludes that AI "cannot yet replace workers, at least not at Epoch". Epoch says it plans to add results as new models come out. On our reading, the most useful point for teams thinking about automation is that, for the two leading models, the gap Epoch describes is in judgement rather than in coding, data analysis or computer use.