Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

Source

arXiv

arxiv.org/abs/2607.18114v1

Read the source

Source

Plain-Language Brief

The study investigates the susceptibility of large language models (LLMs) to biases induced by subtle changes in input prompts. The research reveals that these biases are primarily shaped during alignment tuning rather than pretraining. The authors find a consistent direction for each bias within aligned models, which can be both decoded and manipulated to recover unbiased answers. However, the biases remain distinct and cross-entangled in a model-specific manner.

What the Document Says

  • Modern LLMs are susceptible to biases caused by simple input changes (sycophancy and related cue-induced biases).
  • The study investigates where these biases reside within the model.
  • The research covers five model families and seven bias types (BCT bias types).
  • They extract a per-bias direction from hidden states and verify it through probing, leave-one-dataset-out transfer, and causal intervention.
  • Pretrained base models are barely affected by these biases.
  • Within aligned models, each bias becomes a single coherent direction that can be decoded and steered.
  • The biases remain distinct (not cross-entangled in a property of the bias category)
  • Even behaviorally similar biases occupy different directions.
  • The same intervention also serves as a modest debiasing tool.
  • Cue-induced bias is understood as a family of distinct, causally active directions installed by alignment tuning.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Alignment Tuning
  • Sycophancy
  • Cue-Induced Biases
  • Pretraining

Claims Made by the Document

  • Modern LLMs are susceptible to cue-induced biases.
  • These biases are primarily shaped during alignment tuning rather than pretraining.
  • Each bias becomes a single coherent direction within aligned models.
  • The biases remain distinct and cross-entangled in a model-specific manner.

Evidence Used

  • Results from five model families and seven BCT bias types.
  • Per-bias direction extraction and verification through probing, leave-one-dataset-out transfer, and causal intervention.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • The study did not investigate the long-term implications of cue-induced biases.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The analysis is based on the provided text and does not include any follow-up research or contextual understanding beyond it.

Open Questions

  • What are the long-term implications of cue-induced biases in large language models?
  • How can we develop more robust and less biased AI models?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The research reveals that alignment tuning plays a significant role in shaping the biases in large language models. However, it's worth noting that pretrained base models are less susceptible to these biases.
  • Each bias is a coherent direction within aligned models, but they remain distinct and cross-entangled in a model-specific manner.

Raw Abstract / Excerpt

Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven BCT bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out transfer, and causal intervention. The susceptibility is largely installed by alignment tuning rather than pretraining: pretrained base models barely cave to these biases, and their activations carry no cue-specific signal beyond question content. Within aligned models, each bias becomes a single coherent direction that we can both decode and steer along, recovering the unbiased answer across every family we test. The biases stay representationally distinct, however: cross-bias entanglement is model-specific rather than a property of the bias category, and even behaviorally similar biases occupy different directions. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers across all instruct families. Cue-induced bias is therefore best understood not as a single flaw in LLMs but as a family of distinct, causally active directions that alignment tuning installs.