systems
8 notes, newest first.
OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
The preprint proposes OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent. It evaluates full-reference…
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
The preprint proposes Reasoning Adaptation through Cross-Turn Estimation (RACE), which estimates when prior reasoning remains sufficient for subsequent actions. Its LoGiC procedure identifies reasoning turns whose…
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
The preprint presents Transect, an open-source package built on Inspect Scout for analyzing long agent runs whose transcripts can span hundreds of pages. Configurable evaluation families align recorded events, token…
SquidAgent: Parallelize Wisely, Coordinate Efficiently
The preprint identifies re-exploration and alignment as hidden costs that can make parallel multi-agent systems slower than a single-agent baseline. It proposes a token-based decision criterion, direct forking from the…
Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
The preprint proposes HEAR, a bidirectional Harness–Engine Pairing protocol that lets an agent harness communicate workflow intent and execution requirements while the inference engine returns runtime state…
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
The preprint evaluates agents as configurable systems whose task information, reasoning settings, self-verification, time budget, and backbone model all affect outcomes. On four scientific tasks, the abstract reports…
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
Researchers evaluated a multi-agent data analysis system called LAMBDA on numerical QRData tasks, using a three-layer human-AI grading cascade to assess its output. They found that both automated graders achieved high…
Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems
The paper presents Litmus, a system that creates evaluation metrics for AI pipelines by understanding the purpose of the code and asking questions about it. It was tested on three real AI pipelines and outperformed…
