Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Source

arXiv

arxiv.org/abs/2608.19901v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Yue Wang, Yi Liu, Gelei Deng, Ying Zhang, Yuekang Li, Zhenyu Chen, Leo Zhang
  • Venue: arXiv: cs.CR
  • Published: 2026-08-20
  • Collected: 2026-08-21
  • URL: http://arxiv.org/abs/2608.19901v1
  • PDF: https://arxiv.org/pdf/2608.19901v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The paper presents MaliciousSkillBench, a comprehensive benchmark for detecting malicious Agent Skills in large language models (LLMs). The benchmark consolidates various sources of malicious artifacts and reduces them to a large primary benchmark containing 9,740 Skills: 7,505 malicious and 2,235 benign. The authors evaluate three learned text detectors and three off-the-shelf scanners, finding that reliable detection requires both broader cross-source benchmark coverage and evaluation that measures attack detection and over-flagging of benign Skills.

What the Document Says

  • MaliciousSkillBench is a comprehensive benchmark for malicious Agent Skill detection.
  • The benchmark consolidates 13 public sources, reducing 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families.
  • After exclusion of cross-label conflict, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign.
  • The authors harmonize 11 attack categories for 4,983 malicious identities and find substantial differences in threat composition across sources.
  • Three learned text detectors and three off-the-shelf Skill scanners are evaluated.
  • Learned detectors achieve varying Random Macro-F1 scores but only moderate under Source-Disjoint evaluation.
  • Off-the-shelf scanners reduce false positives at the cost of lower malicious recall.

Key Concepts Mentioned

  • MaliciousSkillBench
  • Agent Skills
  • large language models (LLMs)
  • malicious artifacts
  • benign coverage
  • attack categories
  • learned text detectors
  • off-the-shelf scanners

Claims Made by the Document

  • MaliciousSkillBench provides comprehensive coverage for malicious Agent Skill detection.
  • The benchmark shows substantial differences in threat composition across sources.
  • Learned detectors achieve varying levels of accuracy, but their performance is unsatisfactory under Source-Disjoint evaluation.
  • Off-the-shelf scanners reduce false positives at the cost of lower malicious recall.

Evidence Used

  • Consolidation of 13 public sources and reduction of raw malicious records to a large primary benchmark
  • Harmonization of 11 attack categories for 4,983 malicious identities
  • Evaluation results for three learned text detectors and three off-the-shelf Skill scanners

Methods / Approach

  • Consolidation of public sources
  • Reduction of raw malicious records to a large primary benchmark
  • Harmonization of attack categories
  • Evaluation of learned text detectors and off-the-shelf Skill scanners

Limitations Stated by the Authors

  • The benchmark may be limited by the quality and completeness of the sources it consolidates.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The agent may struggle to understand the technical details of Agent Skills and malicious artifacts.
  • The agent may not have access to the raw data or evaluation results beyond the provided text.

Open Questions

  • What are the specific sources consolidated by MaliciousSkillBench?
  • What are the specific details of the three learned text detectors and three off-the-shelf Skill scanners evaluated?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper does not explicitly state all sources consolidated by MaliciousSkillBench. It may be necessary to refer to additional resources or contact the authors for more information.
  • The specific details of the evaluated detectors and scanners are not provided in the text.

Raw Abstract / Excerpt

Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a direct distribution channel for malicious behavior, yet existing malicious-Skill datasets are fragmented across sources, artifact formats, evidence regimes, and benign coverage; duplicated and structurally related content further complicates direct aggregation and evaluation. We present MaliciousSkillBench, a comprehensive benchmark for malicious Agent Skill detection. We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families. After conservative cross-label conflict exclusion, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign. To characterize its coverage, we harmonize 11 attack categories for 4,983 malicious identities with supported source-native mappings and find substantial differences in threat composition across sources. We then evaluate three learned text detectors and three off-the-shelf Skill scanners. Learned detectors achieve 0.882-0.932 Random Macro-F1 but only 0.653-0.665 under Source-Disjoint evaluation; the strongest word TF-IDF SVM scores 0.932/0.916/0.665 on Random/structural-disjoint/Source-Disjoint while retaining 95.6% malicious recall but producing 62.4% benign FPR on held-out sources. Off-the-shelf scanners occupy different but also unsatisfactory operating regimes, reducing false positives only at the cost of sharply lower malicious recall. Together, these results show that reliable malicious-Skill detection requires both broader cross-source benchmark coverage and evaluation that jointly measures attack detection and benign over-flagging.