Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit

Source

arXiv

arxiv.org/abs/2608.04896v1

Read the source

Source

Plain-Language Brief

The study identifies a risk in NAVSIM v2.2 original scene single-stage scoring where reference-conditioned forgiveness can propagate shared reference failures into compliance credit, potentially leading to incorrect defensive driving evaluations. The researchers suggest an audit protocol to ensure score transparency and stability before using such scores for defensive driving claims.

What the Document Says

  • Defensive driving scores may be misleading if they don't distinguish between policies that observe surrounding actors and those that do not.
  • In re-simulation benchmarks, an agent can receive credit when a logged human reference fails a compliance channel under reference-conditioned forgiveness.
  • Shared unstable rollout transformations between agent and reference can cause this rule to propagate shared reference failures into broad compliance credit in NAVSIM v2.2 original scene single-stage scoring.
  • In the audited numerical backend, a route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed on the navtest split.
  • A fresh installation reproduces rollout divergence on a fixed 32-token diagnostic set.
  • Same-source dependency stack control and exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit.
  • Replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled.
  • The researchers conclude that the numerical instability is the direct trigger for this issue, and suggest an audit protocol for score transparency and stability.

Key Concepts Mentioned

  • defensive driving scores
  • re-simulation benchmarks
  • reference-conditioned forgiveness
  • NAVSIM v2.2
  • route-blind Ignore-All probe
  • PDM-Closed
  • rollout transformations
  • numerical instability
  • audit protocol

Claims Made by the Document

  • Defensive driving scores can be misleading if they don't distinguish between policies that observe surrounding actors and those that do not.
  • Shared unstable rollout transformations can cause reference-conditioned forgiveness to propagate shared reference failures into compliance credit in NAVSIM v2.2 original scene single-stage scoring.
  • The numerical instability is the direct trigger for this issue.

Evidence Used

  • Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete navtest split.
  • A fresh installation reproduces rollout divergence on a fixed 32-token diagnostic set.
  • Same-source dependency stack control and exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not. Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when the logged human reference fails a compliance channel. When agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit. We audit this risk in NAVSIM v2.2 original scene single-stage scoring. Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete 12,146-token navtest split. A fresh installation following the public specification reproduces rollout divergence on a fixed 32-token diagnostic set. A same-source dependency stack control and an exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit. On a 450-token control pool, replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled. Thus, the numerical instability is the direct trigger. Reference-conditioned forgiveness propagates the resulting shared reference failures into compliance credit. We contribute an audit protocol requiring score basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests before using such scores for defensive driving claims.