OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

Source facts

The preprint proposes OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent. It evaluates full-reference, tool-schema-only, and log-only regimes. On SWE-bench trajectories, the authors report a +0.057 AUROC improvement over content-similarity approaches and about 18% compute savings under an abort policy, with 83% of interrupted runs reported as heading toward failure.

Analysis

OnTrack targets the timing gap between post-hoc evaluation and irreversible agent actions. Its results suggest that partial trajectory structure may support early intervention, though the value of reference traces and the cost of false positives remain central deployment questions.

Uncertainty/Doubt

• This is an arXiv preprint. The monitoring latency, trajectory selection, abort policy, and reported savings were not independently reproduced here.