Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

Source

arXiv

arxiv.org/abs/2606.27091v1

Read the source

Source

Plain-Language Brief

The study shows that Language Models (LLMs) fine-tuned for security classification may overlook vulnerabilities introduced during fine-tuning itself. These models can learn token-level indicator semantics that preserve accuracy on training data but fail under certain transformations. The research focuses on the models Foundation-Sec-8B-Instruct and Llama-3.1-8B-Instruct, and identifies a late-attention route inherited from Llama as the cause of this issue. Fine-tuning concentrates and specializes this inherited structure, improving baseline behavior but creating transformation-sensitive attack surfaces. A three-tier evasion benchmark reveals that Foundation-Sec misses on certain transformations that Llama does not share. The study also proposes a pre-deployment monitoring method to identify command families where canonical indicators change role after fine-tuning.

What the Document Says

  • Study on LLMs fine-tuned for security classification
  • Evaluation on held-out examples from the same distribution as training data may miss vulnerabilities
  • Models can learn token-level indicator semantics that preserve accuracy while failing under certain transformations
  • Research focuses on Foundation-Sec-8B-Instruct and Llama-3.1-8B-Instruct
  • Causal interventions localize the classification circuit to a late-attention route inherited from Llama
  • Fine-tuning concentrates and semantically specializes this inherited structure
  • Improves baseline behavior but creates transformation-sensitive attack surfaces
  • Three-tier evasion benchmark finds Foundation-Sec misses on certain transformations
  • Proposes a pre-deployment monitoring method for command families where canonical indicators change role after fine-tuning

Key Concepts Mentioned

  • Language Models (LLMs)
  • Security classification
  • Fine-tuning
  • Token-level indicator semantics
  • PowerShell alias substitution
  • Command reconstruction
  • String construction
  • Execution indirection
  • Case mutation
  • Foundation-Sec-8B-Instruct
  • Llama-3.1-8B-Instruct
  • Causal interventions
  • Late-attention route
  • Evasion benchmark
  • Pre-deployment monitoring method

Claims Made by the Document

  • Evaluation on held-out examples may miss vulnerabilities introduced by fine-tuning itself
  • Models can learn token-level indicator semantics that preserve accuracy while failing under certain transformations
  • Fine-tuning concentrates and semantically specializes inherited structure
  • Improved baseline behavior creates transformation-sensitive attack surfaces
  • Foundation-Sec misses on certain transformations that Llama does not share
  • Pre-deployment monitoring method can identify command families where canonical indicators change role after fine-tuning

Evidence Used

  • Results from PowerShell classification cohorts study

Methods / Approach

  • Causal interventions to localize the classification circuit
  • Three-tier evasion benchmark
  • Pre-deployment monitoring method using linear probe and indicator-token sign test

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

May not consider potential limitations in the study's approach or data selection

Open Questions

  • What other transformations could be explored to evaluate the robustness of LLMs?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study focuses on PowerShell classification, but it is unclear whether similar findings would apply to other domains or tasks. It also does not provide information on potential countermeasures or mitigations for the identified vulnerabilities.

Raw Abstract / Excerpt

LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that this can miss vulnerabilities introduced by fine-tuning itself: models can learn token-level indicator semantics that preserve canonical accuracy while failing under behavior-preserving transformations such as PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation. We study Foundation-Sec-8B-Instruct and its base model, Llama-3.1-8B-Instruct, on matched PowerShell classification cohorts. Causal interventions localize the classification circuit to a late-attention route inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces. A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share. We also derive a pre-deployment monitoring method: a linear probe at the classification boundary and an indicator-token sign test identify command families where canonical indicators change role after fine-tuning. These signals prioritize red-team variant generation using only canonical inputs, showing that security fine-tuning can improve task accuracy while expanding the evasion surface. These results caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers: specialization can convert inherited model structure into brittle indicator rules that preserve held-out accuracy while expanding the evasion surface. Robust AI-enabled security will require specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.