Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

Source

arXiv

arxiv.org/abs/2608.04719v1

Read the source

Source

Plain-Language Brief

Researchers introduce 'canary tools' to diagnose why language models choose wrong tools in specific tasks. The study evaluates eight different models across three capability tiers and finds that susceptibility drops sharply as models become more capable. The taxonomy of tool-selection weaknesses is found to be stratified by the capability of the models, with frontier models being more reliably trapped by certain types of traps.

What the Document Says

  • The paper introduces canary tools as diagnostic probe tools in an agent's Model Context Protocol (MCP) tool set to determine why a model chooses the wrong tool for a task.
  • A taxonomy of six types of tool-selection weaknesses is presented, including semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps.
  • Eight models are evaluated, spanning three capability tiers. The evaluation involves 120 tasks across different canary-density conditions and three seeds (8,640 runs), as well as a subtlety ablation with 2,880 runs.
  • The success of the tasks is graded by a provider-independent judge, with a second independent judge for corroboration (Cohen's kappa = 0.75).
  • Three findings are reported: susceptibility drops sharply as models become more capable; capability tier does not predict safety; and the taxonomy is stratified by the capability of the models.
  • The study finds that frontier models are more reliably trapped by certain types of traps, while others are largely inert on strong models but fire on small open models.
  • Softening each canary's give-away phrase leaves frontier CSR essentially unchanged, suggesting the probes measure reasoning, not phrase-spotting.
  • Susceptibility also predicts task failure (Spearman rho = -0.34), and the most robust models are not significantly degraded by canary pressure.
  • The framework, canary schemas, tasks, and logs are released.

Key Concepts Mentioned

  • canary tools
  • Model Context Protocol (MCP)
  • tool-selection weaknesses
  • eight models
  • three capability tiers
  • 120 tasks
  • provider-independent judge
  • Cohen's kappa

Claims Made by the Document

  • Susceptibility drops sharply as models become more capable.
  • Capability tier does not predict safety.
  • The taxonomy is capability-stratified.

Evidence Used

  • 8,640 runs of evaluations across different conditions.
  • Corroboration by a second independent judge with Cohen's kappa = 0.75.

Methods / Approach

  • Introducing canary tools
  • Evaluating eight models across three capability tiers and three seeds
  • Grading the success of tasks by a provider-independent judge
  • Corroboration by a second independent judge

Limitations Stated by the Authors

  • The study focuses on language models and their tool-selection reasoning.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study does not provide details on the specific models being evaluated.
  • The context of the tasks and the nature of the canary tools are not fully described.
  • It is unknown whether the findings can be generalized to other types of AI agents.

Open Questions

  • What are the specific details of the models being evaluated?
  • What are the contexts of the tasks and the nature of the canary tools?
  • Can the findings be generalized to other types of AI agents?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study provides some details about the eight models, but not enough for a comprehensive understanding.
  • The context and nature of the tasks and canary tools are only implied in the text.

Raw Abstract / Excerpt

Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness. A six-type taxonomy (semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps) turns a single 'wrong tool' outcome into a multi-dimensional profile of how a model reasons about tools. We evaluate eight models -- six hosted and two 8B open-weight -- spanning three capability tiers, on 120 tasks across three canary-density conditions and three seeds (8,640 runs), plus a 2,880-run subtlety ablation. Task success is graded by a provider-independent judge, corroborated by a second independent judge (Cohen's kappa = 0.75). We report three findings. First, susceptibility drops sharply as models get more capable: the per-task canary susceptibility rate (CSR) ranges about 36x across models, lowest for Claude Opus 4.8 and highest for Llama 3.1 8B. Second, capability tier alone does not predict safety: the most susceptible hosted model is mid-tier, and within a provider the cheaper model can be the safer one. Third, the taxonomy is capability-stratified: capability mirages most reliably trap frontier models, while the other types are largely inert on strong models but fire on small open models, so they discriminate by capability rather than being weak. Softening each canary's give-away phrase leaves frontier CSR essentially unchanged, evidence that the probes measure reasoning, not phrase-spotting. Susceptibility also predicts task failure (Spearman rho = -0.34), while the most robust models are not significantly degraded by canary pressure. We release the framework, canary schemas, tasks, and logs.