Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit

Source

arXiv

arxiv.org/abs/2608.04896v1

Read the source

Source

Plain-Language Brief

The paper discusses a flaw in the NAVSIM v2.2 defensive driving evaluation system where shared rollouts can lead to misleading compliance credit. This issue is demonstrated through a series of experiments and is resolved by replacing only the solver, thus highlighting the need for an audit protocol to ensure the reliability of such scores.

What the Document Says

  • Defensive driving scores are important in distinguishing between policies that consider surrounding actors and those that do not.
  • Re-simulation benchmarks may provide credit when a logged human reference fails a compliance channel, a practice known as reference-conditioned forgiveness.
  • When agent and reference share an unstable rollout transformation, this rule can lead to shared reference failures propagating into broad compliance credit.
  • The paper audits this risk in NAVSIM v2.2's original scene single-stage scoring.
  • Under a specific documented-stack condition on the numerical backend, certain probes outrank human replay and PDM-Closed over a large dataset.
  • A reproduction of rollout divergence occurs on a smaller diagnostic set in a fresh installation.
  • Dependency-sensitive numerical behavior is isolated in the shared velocity refit.
  • On a control pool, replacing only the solver eliminates rollout divergence and restores correct order while keeping forgiveness enabled.
  • The paper suggests an audit protocol for defensive driving scores which includes disclosure of score basis and stack, blind probes, overwrite reporting, and rollout stability tests.

Key Concepts Mentioned

  • defensive driving scores
  • re-simulation benchmarks
  • reference-conditioned forgiveness
  • NAVSIM v2.2
  • rollout transformations
  • compliance credit
  • route-blind probes
  • PDM-Closed
  • numerical instability
  • audit protocol

Claims Made by the Document

  • The NAVSIM v2.2 defensive driving evaluation system has a flaw where shared rollouts can lead to misleading compliance credit.
  • This issue is demonstrated through a series of experiments.
  • The issue is resolved by replacing only the solver.

Evidence Used

  • Under certain conditions, certain probes outrank human replay and PDM-Closed over a large dataset.
  • Rollout divergence occurs in a fresh installation on a smaller diagnostic set.
  • Dependency-sensitive numerical behavior is isolated in the shared velocity refit.
  • On a control pool, replacing only the solver eliminates rollout divergence.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not. Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when the logged human reference fails a compliance channel. When agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit. We audit this risk in NAVSIM v2.2 original scene single-stage scoring. Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete 12,146-token navtest split. A fresh installation following the public specification reproduces rollout divergence on a fixed 32-token diagnostic set. A same-source dependency stack control and an exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit. On a 450-token control pool, replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled. Thus, the numerical instability is the direct trigger. Reference-conditioned forgiveness propagates the resulting shared reference failures into compliance credit. We contribute an audit protocol requiring score basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests before using such scores for defensive driving claims.