Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Source

arXiv

arxiv.org/abs/2606.27180v1

Read the source

Source

Plain-Language Brief

In this research, the authors present a new framework called VLM-guided PBRS, which learns potential functions for reward shaping directly from vision language model (VLM) feedback. The goal is to address the challenges of sparse rewards in reinforcement learning and prevent reward hacking. The method employs smaller, computationally efficient VLMs to obtain preference labels that can still accelerate learning, even if less accurate than those obtained from larger VLMs.

What the Document Says

  • The research introduces a new framework called VLM-guided PBRS for potential-based reward shaping.
  • VLM-PBRS learns the potential function directly from vision language model feedback.
  • The approach is based on potential-based reward shaping and preserves the original optimal policies.
  • It eliminates the need for expert-designed reward shaping terms.
  • Smaller, more computationally efficient VLMs are used to obtain preference labels.
  • Preference labels can still accelerate learning even if less accurate than those obtained from larger VLMs.
  • The research empirically validates the method in Meta-World and Franka Kitchen environments.
  • The authors claim improvements in sample efficiency and robustness to reward hacking.

Key Concepts Mentioned

  • reinforcement learning
  • sparse rewards
  • reward shaping
  • potential-based reward shaping (PBRS)
  • vision language model (VLM)
  • preference labels

Claims Made by the Document

  • VLM-guided PBRS learns a potential function directly from VLM feedback.
  • This approach preserves the original optimal policies.
  • It eliminates the need for expert-designed reward shaping terms.
  • Smaller, more computationally efficient VLMs are used to obtain preference labels.
  • Preference labels can still accelerate learning even if less accurate than those obtained from larger VLMs.
  • The research empirically validates the method in Meta-World and Franka Kitchen environments.
  • Improvements in sample efficiency and robustness to reward hacking are claimed.

Evidence Used

  • The text describes the introduction of a new framework called VLM-guided PBRS.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.