Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Tracing Agentic Failure from the Flow of Success

Source

arXiv

arxiv.org/abs/2607.12747v1

Read the source

Source

Plain-Language Brief

The paper proposes a new method called OAT (One-Class Attribution for Tracing) for identifying errors in language models without relying on expensive or difficult-to-collect data. This method is faster and more accurate than existing prompting-based baselines, as demonstrated by experiments.

What the Document Says

  • The paper discusses failure attribution for LLM-based agentic systems.
  • Existing methods for failure attribution are either computationally expensive or require post-training on failure trajectories with step-level error annotations.
  • The authors propose a lightweight and trainable method called OAT that does not require step-level supervision on failure data.
  • OAT is based on one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space.
  • At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories.
  • Experiments show that OAT is faster and more accurate than prompting-based baselines.
  • The experiments were conducted on both in-domain and out-of-distribution datasets.

Key Concepts Mentioned

  • failure attribution
  • LLM-based agentic systems
  • prompting-based pipelines
  • one-class learning
  • neural controlled differential equations
  • anomaly score
  • inference time
  • in-domain and out-of-distribution datasets

Claims Made by the Document

  • OAT is a practical failure attribution model.
  • OAT does not require step-level supervision on failure data.
  • OAT is faster and more accurate than prompting-based baselines.

Evidence Used

  • The paper describes the methodology of OAT.
  • Experiments were conducted to compare the performance of OAT with prompting-based baselines.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.