CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Fengming Yu, Haiwei Pan, Kejia Zhang, Chunling Chen, Jian Guan, Baoying Ma
• Venue: arXiv: cs.LG
• Published: 2026-07-29
• Collected: 2026-07-30
• URL: http://arxiv.org/abs/2607.27054v1
• PDF: https://arxiv.org/pdf/2607.27054v1
• DOI: UNKNOWN

Plain-Language Brief

The research introduces Correlation Calibration-based Redundancy Suppression (CoCaRS), a method for knowledge distillation in heterogeneous settings where teacher and student models have different architectures. The proposed method aims to improve the effectiveness of direct knowledge transfer by calibrating feature decorrelation, capturing reliable semantic relations, preserving discriminative structure, and regulating contribution according to the relative loss scale.

What the Document Says

• Knowledge distillation enables a compact student model to learn from a powerful teacher.
• The research focuses on heterogeneous knowledge distillation with diverse model architectures.
• Representation discrepancies in heterogeneous settings limit the effectiveness of direct knowledge transfer.
• Redundancy suppression offers a new perspective by preserving cross-architecture invariance and reducing feature redundancy through decorrelation.
• Uniform decorrelation may weaken useful structural information.
• A fixed coefficient can make the effective contribution of redundancy suppression sensitive to teacher-student pairs and training stages.
• CoCaRS is proposed to better retain structural information while suppressing redundancy and reduce sensitivity to coefficient settings.

Key Concepts Mentioned

• knowledge distillation
• heterogeneous knowledge distillation
• redundancy suppression
• feature decorrelation
• semantic relations
• discriminative structure
• coefficient regulation

Claims Made by the Document

• CoCaRS can improve the effectiveness of direct knowledge transfer in heterogeneous settings.
• CoCaRS can reduce sensitivity to coefficient settings.

Evidence Used

• Extensive experiments on CIFAR-100 and ImageNet-1K validate the effectiveness of CoCaRS

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Knowledge distillation (KD) enables a compact student model to learn from a powerful teacher and has become an effective paradigm for model compression. The emergence of diverse model architectures has extended KD from homogeneous to heterogeneous settings. However, differences in architectural inductive biases between the teacher and student models often result in substantial representation discrepancies, limiting the effectiveness of direct knowledge transfer. Recently, redundancy suppression has offered a new perspective on heterogeneous KD by preserving cross-architecture invariance and reducing feature redundancy through decorrelation of teacher-student feature correlations. Nevertheless, this formulation may weaken useful structural information through uniform decorrelation, while a fixed coefficient may make the effective contribution of redundancy suppression sensitive to teacher-student pairs and training stages. To address these problems, Correlation Calibration-based Redundancy Suppression (CoCaRS) is proposed to better retain structural information while suppressing redundancy and reduce sensitivity to coefficient settings across teacher-student pairs and training stages. Specifically, CoCaRS calibrates feature decorrelation through Confusion Evidence Estimation (CEE) and Strength Allocation Control (SAC), which respectively capture reliable semantic relations for correlation estimation and preserve discriminative structure during decorrelation. Adaptive Coefficient Regulation (ACR) further regulates the contribution of the calibrated redundancy suppression objective according to its relative loss scale, reducing sensitivity to coefficient settings. Extensive experiments on CIFAR-100 and ImageNet-1K validate the effectiveness of CoCaRS in improving distillation performance and reducing sensitivity to coefficient settings. Code will be released soon.