Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Source

arXiv

arxiv.org/abs/2606.27180v1

Read the source

Source

Plain-Language Brief

Researchers introduce a new method called VLM-PBRS for automating potential-based reward shaping using vision language model (VLM) feedback. The aim is to improve reinforcement learning agents' ability to learn from sparse rewards by learning the potential function directly from VLM preferences over image pairs. Smaller, more computationally efficient VLMs are used in this approach due to the high cost of invoking large VLMs repeatedly during policy learning.

What the Document Says

  • The research presents a new framework called VLM-guided Potential-based Reward Shaping (VLM-PBRS).
  • This framework learns the potential function directly from vision language model (VLM) feedback obtained through querying a lightweight VLM.
  • The potential-based reward shaping method preserves original optimal policies and eliminates the need for expert-designed reward shaping terms.

Key Concepts Mentioned

  • reinforcement learning
  • sparse rewards
  • potential-based reward shaping
  • vision language model
  • image pairs
  • policy learning

Claims Made by the Document

  • VLM preference-based learning can synthesize a potential function for PBRS.
  • A low-cost solution is provided that leverages small VLMs.

Evidence Used

  • The text states that the approach is based on potential-based reward shaping and learns the potential function from VLM preferences over image pairs.

Methods / Approach

  • Querying a lightweight VLM to obtain preferences over image pairs
  • Training a model of the potential function using these preferences

Limitations Stated by the Authors

  • Large vision language models are prohibitively expensive to invoke repeatedly during policy learning, so smaller, more computationally efficient VLMs are used.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The preference labels obtained from smaller VLMs may be less accurate compared to larger ones.

Open Questions

  • What is the specific algorithm or model used for training the potential function?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text mentions learning a model of the potential function, but it does not provide details about the specific algorithm or model used.

Raw Abstract / Excerpt

Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.