Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Auditing the Risk Claims of Distributional Reinforcement Learning

Source

arXiv

arxiv.org/abs/2607.11607v1

Read the source

Source

Plain-Language Brief

The research paper 'Auditing the Risk Claims of Distributional Reinforcement Learning' by Hari Prasad questions the accuracy of risk claims made by distributional reinforcement learning agents, which are used for interpretability, risk-sensitive control, and safety monitoring. The study uses a decision-relevant screening metric, ground truth from snapshot-restart Monte Carlo, and statistical harness to audit three different distributional agents: QR-DQN, C51, and IQN on the MinAtar platform. The results show that 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence for these agents, indicating that the 'risk' learned by these agents is a training artifact rather than a reflection of environment stochasticity.

What the Document Says

  • The research examines the accuracy of risk claims made by distributional reinforcement learning (DRL) agents
  • Three DRL agents: QR-DQN, C51, and IQN are audited on MinAtar
  • 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence for these agents
  • The 'risk' learned by these agents is a training artifact rather than a reflection of environment stochasticity

Key Concepts Mentioned

  • Distributional Reinforcement Learning (DRL)
  • QR-DQN, C51, and IQN
  • MinAtar
  • Risk trade-offs
  • Training artifact

Claims Made by the Document

  • The risk claims of a trained distributional agent are true

Evidence Used

  • Excess Wasserstein gap between the top two actions, ground truth from snapshot-restart Monte Carlo, statistical harness (permutation nulls, bootstrap refutation, FDR control)

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned 'risk' reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.