UK AI Security Institute releases Transect, an open-source tool for reading long agent evaluation transcripts

The UK's AI Security Institute (AISI) on 7 October released Transect, which it describes as "an open-source tool that helps evaluators follow an agent's work, identify what to investigate closely, and check their interpretation against the transcript". The code is on GitHub under the MIT licence, with an accompanying paper on arXiv.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude. AISI's paper uses Anthropic's Claude Opus 4.7 both as the agent in its case study and as the judge model, the README's worked example uses Anthropic's Claude Sonnet 4.6 as its judge, and the incident AISI reported in August mostly involved Anthropic's Mythos 5 model.

The problem AISI says it is solving

AISI writes that "A final evaluation score reveals little" about how an agent worked: "which approaches agents tried, where they got stuck, or how the test setup shaped their work". Reconstructing that from a long transcript "can take significant time and subject-matter expertise". Language models can help label transcripts, it says, "but their judgements can be wrong or misleading". Looking ahead, it says transcripts "can run to billions of tokens, exceeding what human experts can review in full".

What Transect does

According to AISI's post and the project README:

• Inputs. A user supplies a transcript, the task's context and the activity categories they want distinguished, such as experiment design or coding. The README says it reads Inspect logs or OpenClaw exports.
• One timeline. Transect produces an interactive report that "places activity labels, token use and recorded events on a shared timeline". The timeline is indexed by turns, each turn being "one recorded agent output".
• Two kinds of scanner. Recorded events such as human messages and sub-agent launches are pulled straight from the transcript. Activity labels come from LLM judges the user picks. AISI notes that sub-agent labels "describe what they were asked to do, not necessarily what they did".
• Exportable data. The underlying tables can be exported. The README warns that exported tables can contain prompts and human messages, and that "exporting frames is not text removal or anonymization".

AISI built it "with the support of Meridian Labs", on Meridian's Inspect Scout library.

Checking the checker

Transect links each label to its transcript passage and records how it was produced. Where several judges or repeated judgements are used, reviewers can see where they disagree. AISI adds: "Agreement does not establish that a label is correct, but disagreement can help direct closer inspection." The README puts it more bluntly: these are "reliability measures (repeatability, agreement, coverage), never correctness: high agreement can be judges sharing a bias."

The worked example

AISI demonstrates Transect on one run from a study of open-ended AI research, in which an agent had six calendar days to carry out a research project and write a paper. The paper says the run used a Claude Opus 4.7 agent that made some 554 orchestrator tool calls and 79 sub-agent deployments, and generated 13 million tokens, "89% of them in delegated sub-agent work". A chart in the blog post traces four shared documents as different agents wrote and read them.

The paper's summary says the combined view "shows a focus on operational work and manuscript production, with little evidence of a sustained hypothesis generation stage". It also offers a caveat on that reading: "A missing label is therefore weak evidence that the behaviour was absent."

The incident reference

AISI links Transect to its own earlier incident, in which, it says, "agents took unsanctioned actions during an evaluation". Its exact claim is about method, not this tool: "Systematic transcript review helped us identify the agents' actions and investigate their consequences." It then says that "For incident investigation, Transect can organise recorded actions and delegated work around an unexpected episode." The post does not say Transect itself was used in that investigation. We covered the aftermath in our piece on how AISI says it is "now able to resume most evaluation activity" after strengthening its security.

Who can use it

The licence is MIT, both in the GitHub API's licence field and in the LICENSE file. The README lists Python 3.12 or later. A "structural" run needs "no judge, no API key, no model calls"; judged runs need an API key for whichever model provider the user picks, and the README says the demo example "costs cents".

Why it matters

AISI says making recorded behaviour easier to inspect is "one contribution to preserving the observability of agents", and adds that "Reliable conclusions still depend on the evaluation design and expert human judgement." It says it hopes others "will use it, stress-test it, and improve on it". On our reading, the most useful feature is that the tool is built to expose its own uncertainty rather than hand evaluators a single confident label.