Heavy-Ball Q-Learning with Residual Weighting Correction

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Donghwan Lee
• Venue: arXiv: cs.LG
• Published: 2026-06-25
• Collected: 2026-06-27
• URL: http://arxiv.org/abs/2606.27112v1
• PDF: https://arxiv.org/pdf/2606.27112v1
• DOI: UNKNOWN

Plain-Language Brief

The paper presents a corrected version of Heavy-Ball Q-Learning for reinforcement learning, which is guaranteed to converge faster than standard Q-Learning under certain conditions. The analysis uses a switched linear system (SLS) representation and joint spectral radius (JSR) of the associated switching families.

What the Document Says

• The paper introduces a corrected Heavy-Ball Q-Learning method for reinforcement learning.
• It provides a theoretical guarantee for the convergence of the method, and claims that it converges faster than standard Q-Learning under specific conditions.
• The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and joint spectral radius (JSR) of the associated switching families.
• The paper extends this construction to Q-learning with linear function approximation, deriving similar convergence and acceleration statements.

Key Concepts Mentioned

• Heavy-Ball Q-Learning
• Reinforcement Learning
• Convergence
• Switched Linear System (SLS)
• Joint Spectral Radius (JSR)

Claims Made by the Document

• Heavy-Ball Q-Learning converges faster than standard Q-Learning under certain conditions.
• The analysis provides a new insight into how heavy-ball momentum can accelerate Q-learning.

Evidence Used

• The paper presents a theoretical proof for the convergence of Heavy-Ball Q-Learning and its faster rate compared to standard Q-Learning under specific conditions.

Methods / Approach

• Switched Linear System (SLS) representation
• Joint Spectral Radius (JSR) analysis

Limitations Stated by the Authors

The paper does not provide empirical evidence or practical implementation details of the proposed method.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study is based on theoretical analysis and does not include real-world testing.

Open Questions

• What are the specific conditions under which Heavy-Ball Q-Learning converges faster than standard Q-Learning?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper discusses the theoretical advantages of the proposed method but does not provide empirical evidence or practical implementation details.

Raw Abstract / Excerpt

This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.