Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces

Source

arXiv

arxiv.org/abs/2607.02345v1

Read the source

Source

Plain-Language Brief

The paper presents SkillFuzz, a method for discovering unintended objectives (implicit intents) in Large Language Model (LLM)-based agents that use reusable skills from open skill marketplaces. SkillFuzz works by treating the combination of skills as the unit under test and uses contract-guided Monte Carlo Tree Search to prioritize potentially conflicting compositions, aiming to discover over 1000 implicit intents under a fixed query budget.

What the Document Says

  • The study focuses on Large Language Model (LLM)-based agents and open skill marketplaces.
  • Skills are natural-language instruction documents that guide planning and execution of tasks.
  • Operators typically audit skills in isolation, leading to unintended objectives (implicit intents) when skills interact.
  • The paper proposes SkillFuzz as an execution-free testing approach for implicit-intent discovery.
  • SkillFuzz extracts structured skill contracts and uses contract-guided Monte Carlo Tree Search.
  • It prioritizes potentially conflicting compositions of skills.
  • The method aims to discover over 1000 distinct implicit intents under a fixed query budget.
  • It confirms more than 80% of the highest-risk flagged compositions during execution-time validation.
  • SkillFuzz identifies substantially more high-severity implicit intents than alternative search strategies while exploring only a fraction of the pairwise interaction space they require.

Key Concepts Mentioned

  • Large Language Model (LLM)
  • open skill marketplaces
  • skills
  • implicit intents
  • SkillFuzz
  • contract-guided Monte Carlo Tree Search

Claims Made by the Document

  • Individually benign skills may interact to redirect an agent toward unintended objectives (implicit intents).
  • SkillFuzz is the first execution-free testing approach for implicit-intent discovery.
  • SkillFuzz can discover over 1000 distinct implicit intents under a fixed query budget.
  • SkillFuzz confirms more than 80% of the highest-risk flagged compositions during execution-time validation.
  • SkillFuzz identifies substantially more high-severity implicit intents than alternative search strategies.

Evidence Used

  • The text describes SkillFuzz as an approach for discovering unintended objectives in LLM-based agents and open skill marketplaces.

Methods / Approach

  • SkillFuzz uses contract-guided Monte Carlo Tree Search to prioritize potentially conflicting compositions of skills.

Limitations Stated by the Authors

  • The text mentions that the effect of implicit intents emerges only through skill composition, making detection challenging.
  • Execution environments are often unavailable at admission time, posing a challenge for testing.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not state any limitations specific to the agent performing the analysis.

Open Questions

  • What is the specific differential oracle used in SkillFuzz?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It may be useful to explore more about the structure of the skill contracts, the Monte Carlo Tree Search used, and how the high-risk flagged compositions are determined.

Raw Abstract / Excerpt

Large Language Model (LLM)-based agents increasingly automate software engineering tasks through reusable skills, natural-language instruction documents that guide planning and execution. Open skill marketplaces enable users to assemble agents by co-activating community-contributed skills, but marketplace operators typically audit skills in isolation. As a result, individually benign skills may interact to redirect an agent toward unintended objectives, which we term implicit intents. Detecting such intents is challenging because the effect emerges only through skill composition, execution environments are often unavailable at admission time, and the space of possible co-activations grows exponentially with marketplace size. In this paper, we formulate implicit-intent discovery as a fuzzing problem over skill compositions, where skill compositions are the unit under test, planning artifacts expose agent intent before execution, and deviations from a skill-free baseline serve as a differential oracle. Based on this formulation, we propose skillfuzz, the first execution-free testing approach that extracts structured skill contracts and uses contract-guided Monte Carlo Tree Search to prioritize potentially conflicting compositions. Across representative skill-marketplace workloads, skillfuzz discovers over 1,000 distinct implicit intents under a fixed query budget, confirms more than 80% of the highest-risk flagged compositions during execution-time validation, and identifies substantially more high-severity implicit intents than alternative search strategies while exploring only a fraction of the pairwise interaction space they require.