Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Language-Critique Imitation Learning from Suboptimal Demonstrations

Source

arXiv

arxiv.org/abs/2607.01225v1

Read the source

Source

Plain-Language Brief

A new approach called Language-Critique is proposed for imitation learning from suboptimal demonstrations. Instead of using compressed signals, it utilizes natural language to provide explicit feedback about task progress, failure modes, and corrective actions. The method constructs language labels from demonstrations, introduces a language-critique loss, and applies it to both behavior cloning and diffusion policies. It outperforms strong imitation learning and offline reinforcement learning baselines on diverse continuous control tasks.

What the Document Says

  • The paper presents a new method for imitation learning from suboptimal demonstrations using natural language as structured supervision signals.
  • The proposed method constructs language labels from demonstrations that describe current progress, identify suboptimal behaviors, and provide corrective guidance.
  • A language-critique loss is introduced to train policies directly using these structured signals without reducing them to scalars.
  • The method is applied to both behavior cloning and diffusion policies, resulting in LC-BC and LC-DP respectively.
  • The authors provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions.
  • Empirical evaluation shows that the methods outperform strong imitation learning and offline reinforcement learning baselines on diverse continuous control tasks.

Key Concepts Mentioned

  • imitation learning
  • suboptimal demonstrations
  • natural language
  • language-critique framework
  • behavior cloning
  • diffusion policies

Claims Made by the Document

  • The proposed method can provide explicit feedback about task progress, failure modes, and corrective actions.
  • The language-critique loss directly trains policies using structured signals without reducing them to scalars.
  • The methods outperform strong imitation learning and offline reinforcement learning baselines on diverse continuous control tasks.

Evidence Used

  • The paper describes the construction of language labels from demonstrations that explicitly provide feedback.
  • The introduction of a language-critique loss is detailed in the text.
  • Empirical evaluation results are presented showing outperformance of the proposed methods.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.