Scaling Laws for Task-Specific LLM Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Lavinia Ghita, Dhruv Desai, Ioana Boier
• Venue: arXiv: cs.AI
• Published: 2026-06-23
• Collected: 2026-06-24
• URL: http://arxiv.org/abs/2606.24747v1
• PDF: https://arxiv.org/pdf/2606.24747v1
• DOI: UNKNOWN

Plain-Language Brief

This paper presents empirical scaling laws for compressing Large Language Models (LLMs) in specific domains, focusing on quantitative finance as an application. The research explores different methods of distillation (logit-based and LoRA-based), iterative pruning schedules, and a new blended chain-of-thought supervision loss to stabilize knowledge loss during compression. The study finds that general-knowledge benchmarks collapse earlier than in-domain task quality under compression, with the choice of supervision format being crucial. A new dataset called FinHeadlineMix is introduced for domain-specific compression decisions.

What the Document Says

• The research focuses on Large Language Models (LLMs) and their deployment challenges in applications with latency and cost constraints.
• Scaling laws are derived for domain-specific LLM compression, considering factors like dataset size, compression ratio, supervision format, and iterative pruning schedule.
• Quantitative finance is used as the application domain in this study.
• Logit-based and LoRA-based distillation methods are compared under iterative structural pruning.
• A new blended chain-of-thought supervision loss is introduced to stabilize KL-divergence distillation over reasoning traces.
• In-domain task quality degrades predictably under compression, but general-knowledge benchmarks collapse earlier.
• Supervision format is identified as the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases.
• A new dataset called FinHeadlineMix is introduced for domain-specific compression decisions.
• The study also provides scaling law results and practical recommendations.

Key Concepts Mentioned

• Large Language Models (LLMs)
• Scaling laws
• Compression
• Quantitative finance
• Distillation
• Logit-based distillation
• LoRA-based distillation
• Iterative pruning
• Supervision format
• Chain-of-thought supervision loss
• KL-divergence distillation
• Reasoning traces
• FinHeadlineMix

Claims Made by the Document

• In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point.
• Supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases.

Evidence Used

• Results from experiments on quantitative finance as the application domain.

Methods / Approach

• Comparative study of logit-based and LoRA-based distillation under iterative structural pruning.
• Introduction of a blended chain-of-thought supervision loss to stabilize KL-divergence distillation over reasoning traces.

Limitations Stated by the Authors

• The study only considers quantitative finance as the application domain. The scaling laws may vary for other domains.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The quality and applicability of the introduced dataset FinHeadlineMix need to be validated in other domains.
• The generalizability of the findings to larger or more complex language models remains uncertain.

Open Questions

• How do the scaling laws for LLM compression apply to other application domains?
• How well does FinHeadlineMix perform in different domains?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper derives empirical scaling laws for domain-specific LLM compression, but it is important to note that these laws might not apply universally across all application domains.
• FinHeadlineMix is a new dataset introduced in the study, but its quality and applicability outside of quantitative finance need to be evaluated.

Raw Abstract / Excerpt

Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.