FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
• Venue: arXiv: cs.RO
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.26006v1
• PDF: https://arxiv.org/pdf/2606.26006v1
• DOI: UNKNOWN

Plain-Language Brief

The research introduces FORCE, a three-stage framework for fine-tuning Vision-Language-Action (VLA) models using Reinforcement Learning (RL). The framework addresses sample inefficiency issues by implementing a Value-Calibrated Warm-Up phase and self-distillation. The results show that FORCE improves success rates by 79%, outperforms previous RL methods by 10%, and accelerates training by 32.5%. It also achieves robust performance without the need for human intervention.

What the Document Says

• The text is about a new framework named FORCE for fine-tuning Vision-Language-Action (VLA) models with Reinforcement Learning (RL).
• The challenge in using RL for VLA models is sample inefficiency due to two core issues: catastrophic initial unlearning and inefficient policy updates.
• FORCE consists of three stages: Value-Calibrated Warm-Up, online stage, and self-distillation.
• The Value-Calibrated Warm-Up uses on-policy rollouts to mitigate the distributional shift of the Q-function.
• During the online stage, the calibrated Q-function filters both the policy's own action proposals and expert data.
• The results show that FORCE improves success rates by 79%, outperforms previous RL methods by 10%, and accelerates training by 32.5%.
• FORCE achieves robust performance without human intervention.

Key Concepts Mentioned

• VLA models
• Reinforcement Learning (RL)
• sample inefficiency
• Value-Calibrated Warm-Up
• online stage
• self-distillation
• success rates
• expert data

Claims Made by the Document

• FORCE improves success rates by 79%.
• FORCE outperforms previous RL methods by 10%.
• FORCE accelerates training by 32.5%.

Evidence Used

• The results show that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%.

Methods / Approach

• Value-Calibrated Warm-Up
• online stage
• self-distillation

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not mention any limitations of the introduced method from the agent's perspective.

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

• How does the Value-Calibrated Warm-Up phase mitigate the distributional shift of the Q-function?
• What is the role of self-distillation in FORCE?

Raw Abstract / Excerpt

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.