Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation

Source

arXiv

arxiv.org/abs/2607.01208v1

Read the source

Source

Plain-Language Brief

The paper introduces Distill to Detect (D2D), a method that helps identify hidden biases in language models used in high-stakes roles by amplifying these biases through cartridge distillation. The method is designed to work on stealth models, which reveal their preferences only on specific topics while behaving identically on others. D2D concentrates the dominant divergence between a suspected model and its base into a generated text, making it easier to detect hidden biases across multiple types.

What the Document Says

  • The paper presents Distill to Detect (D2D), a method for exposing stealth biases in language models.
  • Stealth biases are preferences that reveal themselves only on specific topics and not on others.
  • D2D amplifies the hidden biases of stealth models through cartridge distillation, making them detectable across multiple bias types.
  • The method works by concentrating the dominant divergence between a suspected model and its base into generated text.
  • The paper proposes a theoretical framework that explains D2D's efficacy in terms of Fisher-weighted projection of the logit distribution shift.
  • By turning the capacity bottleneck of prefix-tuning adapters into a detection tool, D2D can be used for auditing hidden behaviors in deployed language models.

Key Concepts Mentioned

  • stealth biases
  • language models
  • cartridge distillation
  • Distill to Detect (D2D)
  • Fisher-weighted projection

Claims Made by the Document

  • D2D can successfully amplify the hidden biases of stealth models.
  • The method allows for reliable detection of hidden biases across multiple bias types.

Evidence Used

  • The paper presents D2D and shows its effectiveness in exposing hidden biases.

Methods / Approach

  • Cartridge distillation
  • Fisher-weighted projection

Limitations Stated by the Authors

  • Without knowing the bias topic, no detection method can reliably surface a stealth preferential bias.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not provide any practical applications of D2D in real-world scenarios.
  • The effectiveness of D2D on large and complex language models is unknown.

Open Questions

  • How effective is D2D on large and complex language models?
  • Can D2D be applied to real-world scenarios?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper does not provide information about the specific language models it tested.
  • The authors mention that the defender faces a fundamental asymmetry: without knowing the bias topic, no detection method can reliably surface a stealth preferential bias.

Raw Abstract / Excerpt

Language models deployed in high-stakes roles can potentially favor certain entities, brands, or viewpoints, steering user decisions at scale. Such preferential biases can be introduced by any actor in the model's supply chain and are most dangerous when the model reveals its preference only on the relevant topic while behaving identically to its unmodified base on all other inputs. Recent work has shown that these biases can transfer through context distillation on semantically unrelated data, with the signal residing entirely in the soft logit distribution and remaining invisible to text-based inspection. However, the defender faces a fundamental asymmetry: without knowing the bias topic, no detection method can reliably surface a stealth preferential bias, regardless of whether it examines generated text, internal representations, or model weights. Here we introduce Distill to Detect (D2D), a method that surfaces hidden biases by distilling the distributional shift between a suspected model and its base into a cartridge (a KV-cache prefix adapter), concentrating the dominant divergence and amplifying the bias signal into generated text. We show that D2D successfully amplifies the hidden biases of stealth models to the extent that they can be reliably detected across multiple bias types. We also propose a theoretical framework that explains the efficacy of D2D through the lens of Fisher-weighted projection of the logit distribution shift, supported by empirical observations. By turning the capacity bottleneck of prefix-tuning adapters into a detection tool, D2D provides a practical building block for auditing hidden behaviors in deployed language models.