Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks
• Venue: arXiv: cs.LG
• Published: 2026-08-17
• Collected: 2026-08-18
• URL: http://arxiv.org/abs/2608.16747v1
• PDF: https://arxiv.org/pdf/2608.16747v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The study introduces a new method called CHIVE (Counterfactual Hypothesis Investigation Via Edits) to investigate and explain the behavior of language models in real-world scenarios by creating counterfactual prompt edits. The method is used to evaluate the effectiveness of common interpretability techniques for predicting model behaviors on related counterfactual inputs, but no improvement was found from these techniques. Additionally, CHIVE generates training data that can generalize to various out-of-distribution settings.

What the Document Says

• The paper presents CHIVE, a method for investigating the behavior of language models (LLMs) in real-world scenarios using counterfactual prompt edits.
• CHIVE identifies unexpected behaviors and generates high-quality explanations with supporting counterfactual evidence.
• The study evaluates common LLM interpretability techniques' ability to predict counterfactual model behaviors using CHIVE, but finds no improvement.
• Training models on CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings.

Key Concepts Mentioned

• language models
• counterfactual simulations
• interpretability techniques
• CHIVE
• explanations

Claims Made by the Document

• CHIVE can automatically discover explanations of naturally-occurring LLM behaviors.
• Common LLM interpretability techniques do not improve an agent's ability to predict counterfactual model behaviors.

Evidence Used

• The paper describes CHIVE as a method for investigating LLM behavior using counterfactual prompt edits and generating explanations.
• The study evaluates the effectiveness of interpretability techniques on predicting counterfactual model behaviors.
• Results from the evaluation show no improvement with the studied interpretability techniques.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a 'good' explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.