Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations

Source facts

The preprint presents Transect, an open-source package built on Inspect Scout for analyzing long agent runs whose transcripts can span hundreds of pages. Configurable evaluation families align recorded events, token use, sub-agent activity, and model-generated behavior labels on a common turn-based timeline, with traceable source turns and exportable tables. The authors demonstrate the workflow on an AI R&D evaluation generating almost 13 million tokens.

Analysis

Transect addresses observability loss as agent evaluations grow in length and complexity. Traceability from labels to source turns is an important control for model-assisted transcript analysis, where flexible judgments can otherwise weaken reproducibility.

Uncertainty/Doubt

• This is an arXiv preprint. The package's reproducibility, analysis quality, and generality beyond the demonstrated evaluation were not independently tested here.