Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OPD-V: Visual On-Policy Self-Distillation with Modality Balance

Source

arXiv

arxiv.org/abs/2608.05131v1

Read the source

Source

Plain-Language Brief

The paper introduces a new method called On-Policy Self-Distillation with Visual Modality Balance (OPD-V) to improve the visual reasoning of multimodal large language models. The authors argue that previous methods overlook Modality Imbbalance, where textual information dominates generation, and propose using Positive Teacher and Negative Teacher to examine this limitation. OPD-V utilizes these teachers to create a Modality-Balance Trust Region for selecting on-policy tokens during self-distillation, resulting in improved reasoning performance across various benchmarks while reducing training cost.

What the Document Says

  • OPSD is a post-training approach for improving visual reasoning in multimodal large language models.
  • Existing OPSD methods overlook Modality Imbalance.
  • Modality Imbalance occurs when textual information dominates generation and underutilizes carefully designed privileged information.
  • The authors construct Positive Teacher with Zoom-In Image and Negative Teacher with Mask Image to examine this limitation.
  • Changes in their reasoning correctness and token logits reveal that Modality Balance can serve as privileged information.
  • OPD-V is a visual OPSD paradigm that utilizes Positive Teacher and Negative Teacher to create a Modality-Balance Trust Region for selecting on-policy tokens during self-distillation.
  • Experiments show that OPD-V improves reasoning performance across various benchmarks while reducing training cost.

Key Concepts Mentioned

  • On-Policy Self-Distillation (OPSD)
  • Multimodal large language models (MLLMs)
  • Modality Imbalance
  • Positive Teacher with Zoom-In Image
  • Negative Teacher with Mask Image
  • Modality-Balance Trust Region
  • Benchmarks
  • Training cost

Claims Made by the Document

  • Existing OPSD methods overlook Modality Imbalance.
  • Modality Balance can serve as privileged information.
  • OPD-V improves reasoning performance across various benchmarks while reducing training cost.

Evidence Used

  • The text describes the construction of Positive Teacher and Negative Teacher to examine Modality Imbalance, and the experiments showing improvements in reasoning performance and reduced training cost with OPD-V.

Methods / Approach

  • On-Policy Self-Distillation (OPSD)
  • Positive Teacher with Zoom-In Image
  • Negative Teacher with Mask Image
  • Modality-Balance Trust Region

Limitations Stated by the Authors

  • The paper does not provide specific details about the construction of the Positive Teacher and Negative Teacher, or how they exhibit different degrees of Modality Imbalance.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study only tests OPD-V on 6 benchmarks, 4 MLLM backbones, and 5 post-training methods. It is unclear whether the method will perform similarly on other datasets or models.
  • The study does not provide a detailed comparison with other OPSD methods to determine the exact improvement provided by OPD-V.

Open Questions

  • What specific steps are taken in the construction of the Positive Teacher and Negative Teacher?
  • How do these teachers exhibit different degrees of Modality Imbalance?
  • How does OPD-V compare to other OPSD methods?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper proposes a new method for improving the visual reasoning of multimodal large language models, but does not provide detailed information about the construction of the Positive Teacher and Negative Teacher or a comprehensive comparison with other OPSD methods.

Raw Abstract / Excerpt

On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM reasoning. When textual information dominates generation, the model cannot fully integrate its multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their reasoning correctness and token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy tokens used for self-distillation. Experiments across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves reasoning performance while reducing training cost.