Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

Source

arXiv

arxiv.org/abs/2607.27083v1

Read the source

Source

Plain-Language Brief

The paper presents a new approach, Cost-Aware Marginal Decision-Focused Stopping (CAM-DF), for tool selection in Large Language Model (LLM) agents that use various external services. The approach addresses the challenge of choosing the optimal number of tools to avoid both under-informing and overloading the agent with unnecessary tools. CAM-DF is trained on offline data and shown to perform better than existing methods, particularly under heterogeneous costs and high cost pressure. It reduces the number of tools exposed to the agent by 37% while maintaining comparable task success.

What the Document Says

  • The paper introduces CAM-DF for tool selection in LLM agents facing a challenge in choosing the right number of tools from diverse external services.
  • CAM-DF is formulated as a cost-aware decision over ranked tool prefixes and has a compact interpretable variant called CAM-DF-lite.
  • The approach trains directly on offline data, using the sign and magnitude of the gap between stopping now and the best continuation to label and weigh errors respectively.
  • CAM-DF outperforms other deployable methods on a benchmark called $τ$-bench Retail in terms of payoff.
  • The approach achieves higher gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes.
  • CAM-DF is shown to be effective under heterogeneous costs and high cost pressure, with larger gains under weaker rankings.
  • In live execution, CAM-DF exposes the agent to 37% fewer tools than full access while maintaining comparable task success.
  • The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.

Key Concepts Mentioned

  • Large Language Model (LLM) agents
  • Tool selection
  • Cost-aware stopping
  • CAM-DF
  • Offline training
  • Payoff
  • $τ$-bench Retail
  • Ranking sources
  • Cost regimes
  • Pre-execution plugin

Claims Made by the Document

  • Existing approaches leave acquisition under heterogeneous costs unaddressed.
  • CAM-DF attains the highest payoff among deployable methods on $τ$-bench Retail.
  • CAM-DF reduces the number of tools exposed to the agent by 37% while maintaining comparable task success.

Evidence Used

  • The paper presents CAM-DF and its evaluation results.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure. Routers and retrievers can rank candidate tools by relevance, but a ranking alone does not determine how many are worth selecting. Existing approaches leave acquisition under heterogeneous costs unaddressed. We formulate this decision as cost-aware marginal decision-focused stopping (CAM-DF) over ranked tool prefixes, with CAM-DF-lite as a compact interpretable variant. We train directly on the offline gap between stopping now and the best continuation: its sign labels the decision, its magnitude weights each error by the payoff at stake. We prove this objective is Bayes-aligned with the stopping target and that score-only rules are suboptimal under heterogeneous costs. We evaluate on 1,343 tasks across five tool-use domains. On $τ$-bench Retail, CAM-DF attains the highest payoff among deployable methods, with gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes. Our approach is state-of-the-art under heterogeneous costs and high cost pressure, with larger gains under weaker rankings. In live execution, CAM-DF exposes the agent to 37% fewer tools than full access while maintaining comparable task success. The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.