Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

Source

arXiv

arxiv.org/abs/2606.27091v1

Read the source

Source

Plain-Language Brief

The study reveals that Language Models (LLMs) fine-tuned for security classification can overlook vulnerabilities introduced during fine-tuning itself. These models may learn specific token-level indicators which preserve accuracy on the original training data but fail under certain transformations like PowerShell alias substitution, command reconstruction, etc. The research focuses on Foundation-Sec and Llama-3.1, comparing their performance on a matched PowerShell classification dataset. The findings suggest that fine-tuning intensifies and semantically specializes an inherited structure from the base model, leading to new attack surfaces. A three-tier evasion benchmark indicates that Foundation-Sec misses certain transformations not found in Llama's errors. Also, a pre-deployment monitoring method is proposed using a linear probe and indicator-token sign test to identify command families where canonical indicators change role after fine-tuning.

What the Document Says

  • Language models (LLMs) fine-tuned for security classification are evaluated on held-out examples from the same distribution as their training data.
  • Models can learn token-level indicator semantics that preserve canonical accuracy but fail under behavior-preserving transformations.
  • Fine-tuning concentrates and semantically specializes an inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces.
  • A three-tier evasion benchmark found that Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants not found in Llama's errors.
  • A pre-deployment monitoring method is proposed using a linear probe and indicator-token sign test.

Key Concepts Mentioned

  • Language Models (LLMs)
  • Security Classification
  • Fine-Tuning
  • Token-Level Indicator Semantics
  • PowerShell
  • Evasion Vulnerabilities
  • Causal Interventions
  • Pre-Deployment Monitoring

Claims Made by the Document

  • Standard evaluation methods can miss vulnerabilities introduced by fine-tuning itself.
  • Fine-tuning can create transformation-sensitive attack surfaces.
  • Foundation-Sec misses certain transformations not found in Llama's errors.

Evidence Used

  • The study shows that models can learn specific token-level indicators which preserve accuracy on the original training data but fail under certain transformations.
  • A three-tier evasion benchmark found that Foundation-Sec misses certain transformations not found in Llama's errors.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

  • What are the specific transformations referred to in the text?
  • What are 'held-out examples'?
  • Is Llama a publicly available model and, if so, where can it be found?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text doesn't specify the exact nature of the transformations or what is meant by 'held-out examples'.
  • The study compares Foundation-Sec and Llama-3.1 but does not provide details about other models tested.
  • The text implies that Llama is a base model for Foundation-Sec, but it's unclear if this is publicly available or the exact relationship between them.

Raw Abstract / Excerpt

LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that this can miss vulnerabilities introduced by fine-tuning itself: models can learn token-level indicator semantics that preserve canonical accuracy while failing under behavior-preserving transformations such as PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation. We study Foundation-Sec-8B-Instruct and its base model, Llama-3.1-8B-Instruct, on matched PowerShell classification cohorts. Causal interventions localize the classification circuit to a late-attention route inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces. A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share. We also derive a pre-deployment monitoring method: a linear probe at the classification boundary and an indicator-token sign test identify command families where canonical indicators change role after fine-tuning. These signals prioritize red-team variant generation using only canonical inputs, showing that security fine-tuning can improve task accuracy while expanding the evasion surface. These results caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers: specialization can convert inherited model structure into brittle indicator rules that preserve held-out accuracy while expanding the evasion surface. Robust AI-enabled security will require specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.