Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

Source

arXiv

arxiv.org/abs/2607.24645v1

Read the source

Source

Plain-Language Brief

A study explores the inconsistencies in linking sparse autoencoders (SAEs) with model behavior, focusing on the geometry of changes in model logits caused by feature interventions. The authors propose an unsupervised framework called Feature-Effect Geometry Analysis (FEGA). They find that consistent one-dimensional effects are rare, and introduce the concepts of value-like and pointer-like features to explain this variation.

What the Document Says

  • The study investigates the inconsistent links between sparse autoencoders (SAEs) and model behavior.
  • Inconsistencies occur in that features with clear activation descriptions may have weak or unexpected causal effects.
  • Steering can vary across prompts or oppose the intended direction.
  • Activation-based feature selection can miss features that produce the desired output change.
  • Prior work focuses on feature geometry inside the model, but this study looks at the geometry of changes in model logits caused by feature interventions.
  • The authors introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework for analyzing the resulting cloud of logit changes when the same active SAE feature is removed across contexts.
  • Across SAE variants, consistent one-dimensional effects are rare.
  • Value-like features are tied to static information such as factual attributes, while pointer-like features are associated with context-dependent operations.
  • Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions.
  • Pointer-like features predominantly exhibit diffuse effects.

Key Concepts Mentioned

  • sparse autoencoders (SAEs)
  • Feature-Effect Geometry Analysis (FEGA)
  • value-like features
  • pointer-like features

Claims Made by the Document

  • Inconsistent links between SAE features and model behavior limit their wide-scale use as interpretability tools.
  • Features with clear activation descriptions may have weak or unexpected causal effects.
  • Steering can vary across prompts or oppose the intended direction.
  • Activation-based feature selection can miss features that produce the desired output change.
  • Consistent one-dimensional effects are rare in SAE variants.
  • Value-like features more often exhibit structured, low-dimensional effects.
  • Pointer-like features predominantly exhibit diffuse effects.

Evidence Used

  • The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior.
  • Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change.

Methods / Approach

  • Feature-Effect Geometry Analysis (FEGA)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.