FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Source

Source name: arXiv
Source kind: paper
Authors / organization: Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Venue: arXiv: cs.RO
Published: 2026-06-24
Collected: 2026-06-25
URL: http://arxiv.org/abs/2606.26006v1
PDF: https://arxiv.org/pdf/2606.26006v1
DOI: UNKNOWN

Plain-Language Brief

The paper introduces FORCE, a three-stage framework for improving the efficiency of fine-tuning Vision-Language-Action (VLA) models via Reinforcement Learning (RL). The framework addresses two main issues: initial instability in Q-function and inefficient policy updates due to low-quality exploration data. By incorporating a Value-Calibrated Warm-Up phase and using the calibrated Q-function as a filter for action proposals, FORCE achieves better performance on various simulation and real-world tasks, with significant improvements in success rates, faster training times, and minimal human intervention.

What the Document Says

The paper presents FORCE, a framework for improving the efficiency of RL fine-tuning for VLA models.
FORCE addresses two main issues: catastrophic initial unlearning due to an unstable Q-function and inefficient policy updates caused by low-quality exploration data.
FORCE includes a Value-Calibrated Warm-Up phase using on-policy rollouts to mitigate the distributional shift of the Q-function.
During the online stage, the calibrated Q-function filters both the policy's action proposals and expert data for high-value actions.
Evaluation shows FORCE improves success rates by 79% absolutely, outperforms prior RL methods by 10%, and accelerates training by 32.5%.
FORCE mitigates common success rate drops and achieves robust performance without human intervention.

Key Concepts Mentioned

Vision-Language-Action (VLA) models
Reinforcement Learning (RL)
Q-function
Value-Calibrated Warm-Up
policy updates
human interventions

Claims Made by the Document

FORCE improves the efficiency of RL fine-tuning for VLA models.
FORCE addresses two core issues in RL fine-tuning: initial instability and inefficient policy updates.
Evaluation shows FORCE achieves significant improvements in success rates, faster training times, and robust performance.

Evidence Used

Improvements in success rates (79% absolute), outperformance of prior RL methods (10%), and accelerated training (32.5%) as a result of using FORCE.

Methods / Approach

Value-Calibrated Warm-Up
on-policy rollouts
Q-function filtering for high-value actions

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.