Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents

Source

arXiv

arxiv.org/abs/2607.07405v1

Read the source

Source

Plain-Language Brief

Researchers found that LLM agents (Language Model-based tools) can violate policies without being detected because the resulting state changes are not reported by the tools or the agent. The researchers studied this issue in a specific airline domain and found that 78% of observed failures were silent wrong-state failures with no tool error. They then tested a lightweight intervention - deterministic, read-only pre-execution gates that check calls and current state before allowing a write. These gates improved the success rate of a benchmark from 29.6% to 42.0%. The gates also showed positive effects on gpt-5.2 at default reasoning but did not guarantee task success.

What the Document Says

  • Tool-using LLM agents can violate policies while appearing successful.
  • In the $τ^2$-bench airline domain, 78% of observed failures are silent wrong-state failures with no tool error and the failure rate is reproducible.
  • A lightweight intervention of deterministic, read-only pre-execution gates improves the success rate on gpt-4o-mini from 29.6% to 42.0%.
  • The effect of the gates is concentrated where they fire and have little impact elsewhere.
  • Two negative controls bound the mechanism: gates help when tools are policy-permissive and add little where tools already self-enforce.
  • The same failure mode persists at the frontier (gpt-5.2) and the gates improve success rate there as well.

Key Concepts Mentioned

  • LLM agents
  • Policy violation
  • Silent wrong state failures
  • Deterministic, read-only pre-execution gates
  • Airline domain
  • gpt-4o-mini
  • gpt-5.2

Claims Made by the Document

  • LLM agents can violate policies and not be detected.
  • Deterministic gates can prevent a known class of silent policy-violating writes.
  • The deterministic gates improve success rate on gpt-4o-mini and gpt-5.2.

Evidence Used

  • Observed failures in the $τ^2$-bench airline domain
  • Improved success rate after implementing deterministic gates
  • Reproducibility of failure rates across disjoint seeds
  • Positive effect of gates on gpt-5.2

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • The research did not provide information about the exact nature of the policies being violated.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • It is unclear whether the deterministic gates can prevent all types of silent policy-violating writes.
  • It is not stated if the deterministic gates have been tested in other domains besides the airline domain.

Open Questions

  • Can the deterministic gates prevent all types of silent policy-violating writes?
  • Have the deterministic gates been tested in other domains besides the airline domain?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The research does not provide information about the exact nature of the policies being violated.
  • It is unclear whether the deterministic gates can prevent all types of silent policy-violating writes.
  • It is not stated if the deterministic gates have been tested in other domains besides the airline domain.

Raw Abstract / Excerpt

Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully. In policy-permissive environments, a tool may execute any well-formed call even when the corresponding state transition is forbidden by domain policy. The result is a silent wrong state (a booking cancelled, a passenger count changed, a claim acted on without verification) that neither the tool nor the agent's self-report exposes. We study this failure mode in the $τ^2$-bench airline domain. On a budget agent, 78% of observed failures are silent wrong-state failures with no tool error, and the aggregate failure rate is reproducible across disjoint seeds, not sampling noise. We then evaluate a lightweight intervention: deterministic, read-only pre-execution gates that inspect the proposed call and current state before allowing a write. A four-gate suite raises full-benchmark success from 29.6% to 42.0% on gpt-4o-mini (+12.4pp; paired task-level bootstrap P=0.0012), and the lift reproduces on a disjoint 15-seed set (+12.3pp; P=0.0008). The effect is concentrated where the gates fire: on the 26/50 firing tasks, success rises by +19.2pp, while movement on the 24 non-firing tasks does not exclude zero. Two negative controls (a self-enforcing retail domain and BFCL) bound the mechanism: gates help when tools are policy-permissive and add little where tools already self-enforce. As suggestive evidence, not a central claim, the same failure mode persists at the frontier: gpt-5.2 at default reasoning still attempts policy-violating writes, and the same suite improves success from 61.2% to 71.6% (+10.4pp; P=0.020; n=5, no replication). The contribution is a bounded evaluation and reliability result: deterministic gates do not guarantee task success, but they can deterministically prevent a known class of silent policy-violating writes at the action boundary.