Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Source

arXiv

arxiv.org/abs/2606.27180v1

Read the source

Source

Plain-Language Brief

In this research, the authors propose a new method called VLM-PBRS for automating potential-based reward shaping using vision language model (VLM) feedback. The goal is to help reinforcement learning agents learn from sparse rewards more effectively by learning a potential function directly from VLM preferences over image pairs. This approach preserves the original optimal policies and reduces the need for expert-designed reward shaping terms. To make this method feasible, smaller, computationally efficient VLMs are employed instead of large ones.

What the Document Says

  • The authors introduce a new framework called VLM-PBRS which learns potential functions for potential-based reward shaping using vision language model (VLM) feedback.
  • The method queries a lightweight VLM to obtain preferences over image pairs and trains the potential function model using these preferences.
  • The approach is based on potential-based reward shaping, preserving the original optimal policies.
  • To avoid using large and expensive VLMs during policy learning, smaller, more computationally efficient VLMs are employed.

Key Concepts Mentioned

  • Vision language model (VLM)
  • Potential-based reward shaping
  • Reinforcement learning agents
  • Sparse rewards
  • Image pairs
  • Preference labels

Claims Made by the Document

  • The proposed method can accelerate learning.
  • The preference labels, even if less accurate, can still be used to improve sample efficiency.

Evidence Used

  • Empirical evidence from Meta-World and Franka Kitchen environments

Methods / Approach

  • VLM preference-based learning
  • Potential function model training

Limitations Stated by the Authors

The resulting preference labels are less accurate.

Limitations Noted by the Agent

Agent interpretation — not from the source.

Not stated in source

Open Questions

  • What is the specific algorithm used for learning the potential function from VLM preferences?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The research focuses on using a vision language model (VLM) to learn a potential function for potential-based reward shaping, which can help reinforcement learning agents learn from sparse rewards more effectively.

Raw Abstract / Excerpt

Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.