Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

Source

arXiv

arxiv.org/abs/2606.27091v1

Read the source

Source

Plain-Language Brief

A study shows that Language Models (LLMs) fine-tuned for security classification can miss vulnerabilities due to learning token-level indicator semantics that preserve accuracy on their training data but fail under certain transformations. The study compares Foundation-Sec and its base model Llama-3.1 on PowerShell classification tasks. The authors find that the inherited structure from Llama is concentrated and semantically specialized during fine-tuning, creating transformation-sensitive attack surfaces. A three-tier evasion benchmark reveals that Foundation-Sec misses certain transformations that Llama does not share. The study also proposes a pre-deployment monitoring method for command families where canonical indicators change role after fine-tuning.

What the Document Says

  • The study focuses on Language Models (LLMs) fine-tuned for security classification.
  • These models are usually evaluated using held-out examples from the same distribution as their training data.
  • However, the study shows that this approach can miss vulnerabilities introduced by fine-tuning itself.
  • Models can learn token-level indicator semantics that preserve canonical accuracy but fail under certain transformations like PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation.
  • The study compares Foundation-Sec-8B-Instruct with its base model Llama-3.1-8B-Instruct on matched PowerShell classification cohorts.
  • Causal interventions reveal that the classification circuit is inherited from Llama rather than created by fine-tuning.
  • Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces.
  • A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share.
  • The study also proposes a pre-deployment monitoring method for command families where canonical indicators change role after fine-tuning.

Key Concepts Mentioned

  • Language Models (LLMs)
  • Fine-tuning
  • Security classification
  • PowerShell
  • Token-level indicator semantics
  • Transformations
  • Evasion vulnerabilities

Claims Made by the Document

  • Evaluating LLMs on held-out examples from the same distribution as their training data can miss vulnerabilities introduced by fine-tuning.
  • Fine-tuning can cause models to learn token-level indicator semantics that preserve accuracy but fail under certain transformations.
  • Foundation-Sec misses certain transformations in a three-tier evasion benchmark that Llama does not share.

Evidence Used

  • The study presents experimental results comparing Foundation-Sec and its base model on PowerShell classification tasks.
  • The study performs causal interventions to localize the classification circuit.

Methods / Approach

  • Causal interventions
  • Three-tier evasion benchmark
  • Pre-deployment monitoring method

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that this can miss vulnerabilities introduced by fine-tuning itself: models can learn token-level indicator semantics that preserve canonical accuracy while failing under behavior-preserving transformations such as PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation. We study Foundation-Sec-8B-Instruct and its base model, Llama-3.1-8B-Instruct, on matched PowerShell classification cohorts. Causal interventions localize the classification circuit to a late-attention route inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces. A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share. We also derive a pre-deployment monitoring method: a linear probe at the classification boundary and an indicator-token sign test identify command families where canonical indicators change role after fine-tuning. These signals prioritize red-team variant generation using only canonical inputs, showing that security fine-tuning can improve task accuracy while expanding the evasion surface. These results caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers: specialization can convert inherited model structure into brittle indicator rules that preserve held-out accuracy while expanding the evasion surface. Robust AI-enabled security will require specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.