FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Source

Source name: arXiv
Source kind: paper
Authors / organization: Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Venue: arXiv: cs.RO
Published: 2026-06-24
Collected: 2026-06-25
URL: http://arxiv.org/abs/2606.26006v1
PDF: https://arxiv.org/pdf/2606.26006v1
DOI: UNKNOWN

Plain-Language Brief

The paper introduces FORCE, a framework designed to improve the efficiency of reinforcement learning fine-tuning for Vision-Language-Action (VLA) models. The framework consists of three stages: Value-Calibrated Warm-Up, Online Stage, and Policy Update. It addresses two core issues that hinder efficient RL fine-tuning: catastrophic initial unlearning and inefficient policy updates. The authors claim that FORCE achieves a 79% improvement in success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%. It also claims to mitigate the common success rate drop and achieve robust performance without human intervention.

What the Document Says

The paper presents FORCE, a framework for improving reinforcement learning fine-tuning for VLA models.
FORCE consists of three stages: Value-Calibrated Warm-Up, Online Stage, and Policy Update.
It aims to address catastrophic initial unlearning and inefficient policy updates that hinder efficient RL fine-tuning.
During the Value-Calibrated Warm-Up phase, on-policy rollouts are used to mitigate the distributional shift of the Q-function.
In the Online Stage, a calibrated Q-function filters both the policy's own action proposals and expert data.
The authors claim that FORCE achieves a 79% improvement in success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%.
It also claims to mitigate the common success rate drop and achieve robust performance without human intervention.

Key Concepts Mentioned

Vision-Language-Action (VLA) models
Reinforcement Learning (RL)
Q-function
Policy Update
Value-Calibrated Warm-Up
catastrophic initial unlearning
inefficient policy updates

Claims Made by the Document

FORCE achieves a 79% improvement in success rates.
FORCE outperforms prior RL methods by 10%.
FORCE accelerates training by 32.5%.
FORCE mitigates the common success rate drop.
FORCE achieves robust performance without human intervention.

Evidence Used

The result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.