Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness

Source facts

The preprint tests whether legal models' named statutes or precedents causally support their verdicts. Holding case facts fixed, it substitutes the named authority and examines verdict changes and hidden-state dynamics across seven open-weight models and four benchmarks. The authors report that models often name the correct authority while verdict sensitivity to that authority is less consistent, and that adversarial instructions in case facts can remain effective.

Analysis

The work distinguishes citing an authority from relying on it. That distinction matters when generated legal explanations are treated as audit or compliance artifacts, but counterfactual verdict sensitivity is still only one test of legal reasoning faithfulness.

Uncertainty/Doubt

• This is an arXiv preprint. The model runs, hidden-state analysis, and robustness checks were not independently reproduced here.