Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

Source

arXiv

arxiv.org/abs/2606.23375v1

Read the source

Source

Plain-Language Brief

The authors present TF-RefusalBench, a multilingual benchmark for assessing over-alignment in legal language models used by the Swiss Federal Supreme Court. They find that over-alignment is complex and influenced by factors like the model, prompt, and text language being processed. The study also suggests that removing refusal directions improves task performance without triggering refusals.

What the Document Says

  • The use of LLMs in legal fields is debated due to potential errors and their gravity.
  • The Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across four official languages.
  • However, these models encounter challenges in criminal law due to refusals and disclaimers triggered by descriptions of violent or sexual offenses.
  • TF-RefusalBench is introduced as a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings.
  • The benchmark contains 5,200 prompts across four languages (French, German, Italian, English).
  • Over-alignment is found to be a complex phenomenon influenced by the model, prompt, and text language being processed.
  • Impact of over-alignment cannot be evaluated solely from an over-refusal perspective due to the disclaimer's impact on task faithfulness.
  • Study demonstrates that abliteration (removing refusal directions) improves task performance without triggering refusals.

Key Concepts Mentioned

  • LLMs
  • legal field
  • Swiss Federal Supreme Court
  • over-alignment
  • TF-RefusalBench
  • criminal law
  • multilingual
  • prompts
  • refusals
  • ablation

Claims Made by the Document

  • The use of LLMs in the legal field is debated.
  • The Swiss Federal Supreme Court uses small on-premises models for translation and summarization.
  • Criminal law poses challenges due to refusals and disclaimers triggered by violent or sexual offenses descriptions.
  • TF-RefusalBench is a multilingual benchmark for criminal-law translation and summarization.
  • Over-alignment is a complex phenomenon.
  • The impact of over-alignment cannot be evaluated solely from an over-refusal perspective.
  • Ablation improves task performance without triggering refusals.

Evidence Used

  • The text describes the debates about LLMs in the legal field, the use of models by the Swiss Federal Supreme Court, the challenges in criminal law, and the introduction of TF-RefusalBench.
  • The text mentions the complexity of over-alignment and its impact on task performance.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.