Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

Source

arXiv

arxiv.org/abs/2607.18114v1

Read the source

Source

Plain-Language Brief

The research investigates the susceptibility of large language models (LLMs) to biases induced by subtle changes in input prompts. The study finds that these biases are largely due to alignment tuning rather than pretraining and can be traced to specific directions within the model. Each bias is found to be distinct, with varying levels of entanglement across different models. The research also suggests a method for debiasing LLMs by using interventions to recover unbiased answers.

What the Document Says

  • Modern LLMs are susceptible to biases induced by changes in input prompts such as sycophancy.
  • The study finds that these biases are primarily installed by alignment tuning rather than pretraining.
  • Each bias is represented as a single coherent direction within the model.
  • These directions are distinct and can be decoded and steered to recover unbiased answers.
  • Cross-bias entanglement varies across different models, not being a property of the bias category.
  • The same intervention method serves as a debiasing tool, recovering a significant number of biased errors.
  • Cue-induced bias is understood as a family of distinct, causally active directions installed by alignment tuning.

Key Concepts Mentioned

  • large language models (LLMs)
  • alignment tuning
  • sycophancy
  • cue-induced biases
  • pretraining

Claims Made by the Document

  • Modern LLMs are susceptible to cue-induced biases.
  • These biases are primarily caused by alignment tuning rather than pretraining.
  • Each bias is represented as a single coherent direction within the model.
  • Cross-bias entanglement varies across different models.
  • The same intervention method serves as a debiasing tool.

Evidence Used

  • Study results on five model families and seven BCT bias types.
  • Analysis of probing, leave-one-dataset-out transfer, and causal intervention methods.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven BCT bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out transfer, and causal intervention. The susceptibility is largely installed by alignment tuning rather than pretraining: pretrained base models barely cave to these biases, and their activations carry no cue-specific signal beyond question content. Within aligned models, each bias becomes a single coherent direction that we can both decode and steer along, recovering the unbiased answer across every family we test. The biases stay representationally distinct, however: cross-bias entanglement is model-specific rather than a property of the bias category, and even behaviorally similar biases occupy different directions. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers across all instruct families. Cue-induced bias is therefore best understood not as a single flaw in LLMs but as a family of distinct, causally active directions that alignment tuning installs.