Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

Source

arXiv

arxiv.org/abs/2606.27091v1

Read the source

Source

Plain-Language Brief

The study shows that Language Models (LLMs) fine-tuned for security classification can have vulnerabilities missed during standard evaluation because they learn token-level indicator semantics that work well on their training data but fail under certain transformations. The researchers found these vulnerabilities in a model called Foundation-Sec and its base model, Llama-3.1-8B-Instruct, when comparing them on PowerShell classification tasks. They also propose a method for pre-deployment monitoring that uses a linear probe at the classification boundary and an indicator-token sign test to identify command families where canonical indicators change role after fine-tuning.

What the Document Says

  • The study discusses vulnerabilities in Language Models (LLMs) fine-tuned for security classification.
  • These models can learn token-level indicator semantics that work well on their training data but fail under certain transformations like PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation.
  • The study compares Foundation-Sec and its base model Llama-3.1-8B-Instruct on PowerShell classification tasks.
  • Causal interventions suggest that the classification circuit in Foundation-Sec is inherited from Llama rather than created by fine-tuning.
  • Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces.
  • A three-tier evasion benchmark shows that Foundation-Sec misses on certain transformations that Llama does not share.
  • The study proposes a pre-deployment monitoring method using a linear probe and an indicator-token sign test.

Key Concepts Mentioned

  • Language Models (LLMs)
  • Security classification
  • PowerShell
  • Causal interventions
  • Token-level indicator semantics
  • Pre-deployment monitoring

Claims Made by the Document

  • Fine-tuning can introduce vulnerabilities in LLMs that are missed during standard evaluation.
  • Foundation-Sec misses on certain transformations that Llama does not share.
  • Pre-deployment monitoring using a linear probe and an indicator-token sign test can be effective.

Evidence Used

  • The study presents results from comparing Foundation-Sec and its base model Llama-3.1-8B-Instruct on PowerShell classification tasks.
  • The study presents results from a three-tier evasion benchmark showing that Foundation-Sec misses on certain transformations.

Methods / Approach

  • Causal interventions
  • Three-tier evasion benchmark
  • Pre-deployment monitoring using a linear probe and an indicator-token sign test

Limitations Stated by the Authors

The authors caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only considers PowerShell classification tasks. It is unknown if the findings extend to other types of transformations or languages.

Open Questions

  • Can these findings be applied to other types of transformations or languages?
  • What are the implications for other types of security classifiers?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study shows that fine-tuning can create vulnerabilities in LLMs that are not detected during standard evaluation. These vulnerabilities manifest as transformation-sensitive attack surfaces.
  • The authors suggest that robust AI-enabled security requires specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.

Raw Abstract / Excerpt

LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that this can miss vulnerabilities introduced by fine-tuning itself: models can learn token-level indicator semantics that preserve canonical accuracy while failing under behavior-preserving transformations such as PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation. We study Foundation-Sec-8B-Instruct and its base model, Llama-3.1-8B-Instruct, on matched PowerShell classification cohorts. Causal interventions localize the classification circuit to a late-attention route inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces. A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share. We also derive a pre-deployment monitoring method: a linear probe at the classification boundary and an indicator-token sign test identify command families where canonical indicators change role after fine-tuning. These signals prioritize red-team variant generation using only canonical inputs, showing that security fine-tuning can improve task accuracy while expanding the evasion surface. These results caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers: specialization can convert inherited model structure into brittle indicator rules that preserve held-out accuracy while expanding the evasion surface. Robust AI-enabled security will require specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.