Redistribution-based Cost Inference Improves Sparse Safe Offline RL

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Ebenezer Gelo, Geraud Nangue Tasse, Steven James, Benjamin Rosman
• Venue: arXiv: cs.LG
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12306v1
• PDF: https://arxiv.org/pdf/2608.12306v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

A new framework called Redistribution-based Cost Inference (RCI) is proposed to convert sparse stop-feedback in safe offline reinforcement learning into dense per-step costs. This conversion helps train a constrained offline policy and leads to lower violation rates compared to sparse and classifier-based baselines, as demonstrated in experiments on highway driving and robotic manipulation.

What the Document Says

• The paper discusses a framework called Redistribution-based Cost Inference (RCI).
• This framework aims to convert sparse stop-feedback into dense per-step costs for safe offline reinforcement learning.
• The RCI framework is used to train a constrained offline policy on the augmented dataset.
• The paper states that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, implying that the transformation is lossless theoretically and leads to better-conditioned cost critic learning in practice.
• Experiments on highway driving and robotic manipulation demonstrate lower violation rates for RCI compared to sparse and classifier-based baselines.
• The experiments also show robustness of RCI to heterogeneous dataset compositions and label noise.

Key Concepts Mentioned

• Redistribution-based Cost Inference (RCI)
• safe offline reinforcement learning
• temporal credit assignment problem
• CMDP
• feasible policy set
• optimal Lagrangian
• violation rates

Claims Made by the Document

• RCI framework can convert sparse stop-feedback into dense per-step costs.
• RCI leads to lower violation rates compared to sparse and classifier-baselines.
• Return-equivalent redistribution preserves the feasible policy set and optimal Lagrangian in a CMDP.

Evidence Used

• The paper describes the RCI framework and its conversion process.
• Experiments on highway driving and robotic manipulation demonstrate lower violation rates for RCI compared to baselines.
• The paper states that return-equivalent redistribution preserves the feasible policy set and optimal Lagrangian in a CMDP.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.