Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Tracing Agentic Failure from the Flow of Success

Source

arXiv

arxiv.org/abs/2607.12747v1

Read the source

Source

Plain-Language Brief

The text discusses a new model called OAT for failure attribution in LLM-based agentic systems. The model aims to identify the steps causing failures in these systems, which is crucial for debugging and improvement. It's designed to be lightweight and trainable without requiring step-level supervision on failure data. Instead of relying on computationally expensive prompting-based pipelines or post-training on costly failure trajectories with error annotations, OAT uses one-class learning with neural controlled differential equations to model the patterns of successful trajectories in latent space. During inference, each step in a failure trajectory is given an anomaly score based on its deviation from learned successful trajectory patterns. Experiments show that OAT is faster and more effective than prompting-based baselines, with better performance on both in-domain and out-of-distribution datasets.

What the Document Says

  • The paper discusses a failure attribution model called OAT for LLM-based agentic systems.
  • OAT is designed to be lightweight and trainable without step-level supervision on failure data.
  • It models the patterns of successful trajectories in latent space using one-class learning with neural controlled differential equations.
  • During inference, each step in a failure trajectory is given an anomaly score based on its deviation from learned successful trajectory patterns.
  • Experiments show that OAT is faster and more effective than prompting-based baselines.

Key Concepts Mentioned

  • Failure attribution
  • LLM-based agentic systems
  • One-class learning
  • Neural controlled differential equations
  • Anomaly score

Claims Made by the Document

  • OAT is a practical failure attribution model.
  • OAT should be lightweight and trainable without step-level supervision on failure data.
  • OAT outperforms prompting-based baselines in both in-domain and out-of-distribution datasets.

Evidence Used

  • The paper describes the design of OAT as a lightweight, trainable model without requiring step-level supervision on failure data.
  • Experiments showing OAT's performance compared to prompting-based baselines are presented.

Methods / Approach

  • One-class learning with neural controlled differential equations

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.