Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Harnessing Textual Refusal Directions for Multimodal Safety

Source

arXiv

arxiv.org/abs/2606.31876v1

Read the source

Source

Plain-Language Brief

Researchers propose a new approach called Modality-Agnostic Refusal Steering (MARS) to improve safety in Multimodal Large Language Models without requiring unsafe multimodal data. MARS uses textual refusal directions from the LLM backbone and adjusts for modality misalignment, steering strength, and optimal intervention layer.

What the Document Says

  • The paper discusses improving safety in Multimodal Large Language Models (MLLMs).
  • Two strategies are mentioned: post-training alignment and exploiting refusal directions in the activation space.
  • These strategies are less feasible for MLLMs due to the difficulty in collecting unsafe multimodal data.
  • The authors propose relaxing this constraint and investigating if textual refusal directions can generalize across modalities (image, video).
  • Preliminary findings suggest that textual refusal directions can do so, but effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment.
  • Misaligned inputs are spuriously steered toward refusal due to cross-modal alignment.
  • MARS is introduced as a lightweight training-free approach that injects multimodal safety without the need for multimodal safety data.
  • MARS corrects modality misalignment via activation re-centering, adaptively scales steering strength, and selects the optimal intervention layer.
  • The approach operates at the first generated token.
  • Evaluation is done on five state-of-the-art MLLMs across safety, utility, and video jailbreak benchmarks.
  • MARS achieves consistent safety gains while preserving utility.
  • The results suggest that safety-relevant structure is shared across modalities, and textual refusal directions are a powerful and underexplored foundation for multimodal alignment.

Key Concepts Mentioned

  • Large Language Models
  • Multimodal Large Language Models
  • Safety
  • Textual Refusal Directions
  • Modality-Agnostic Refusal Steering

Claims Made by the Document

  • Textual refusal directions can generalize across modalities.
  • MARS achieves consistent safety gains while preserving utility.

Evidence Used

  • Preliminary findings confirm the ability of textual refusal directions to generalize across modalities.
  • Evaluation results show that MARS achieves consistent safety gains while preserving utility.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

To improve safety in Large Language Models (LLMs) we can either perform post-training alignment or exploit refusal directions in the activation space. Both strategies are less feasible in Multimodal LLMs (MLLMs) as they require unsafe multimodal data, harder to collect than their unimodal counterpart. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a light-weight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation re-centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak benchmarks, MARS achieves consistent safety gains while preserving utility. These results reveal that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.