Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

Source

arXiv

arxiv.org/abs/2607.27083v1

Read the source

Source

Plain-Language Brief

A new approach, Cost-Aware Marginal Decision-Focused Stopping (CAM-DF), is proposed for tool selection in Large Language Model (LLM) agents that use external services. The method helps address the challenge of acquiring too many or too few tools by considering costs associated with each tool. The CAM-DF family is shown to perform well in various tasks and reduces the number of tools used while maintaining comparable task success, making it a lightweight solution for lowering costs.

What the Document Says

  • The text discusses a new approach called Cost-Aware Marginal Decision-Focused Stopping (CAM-DF) for tool selection in Large Language Model (LLM) agents.
  • The challenge addressed is the need to balance acquiring too few tools, which leaves tasks under-informed, and too many tools, which add cost, context load, and privacy exposure.
  • The CAM-DF approach trains directly on the offline gap between stopping now and the best continuation, with its sign labeling the decision and magnitude weighing each error by the payoff at stake.
  • CAM-DF is shown to outperform other methods in a benchmark called $τ$-bench Retail, reducing the number of tools used by 37% while maintaining comparable task success.
  • The CAM-DF family is described as a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.

Key Concepts Mentioned

  • Cost-Aware Marginal Decision-Focused Stopping (CAM-DF)
  • Large Language Model (LLM) agents
  • Tool selection
  • External services
  • Search engines
  • Databases
  • Connectors
  • Cost pressure
  • Rankings
  • Pre-execution plugin

Claims Made by the Document

  • CAM-DF outperforms other methods in certain tasks and cost regimes.
  • CAM-DF reduces the number of tools used while maintaining comparable task success.

Evidence Used

  • Results from evaluations on 1,343 tasks across five tool-use domains and live execution data.

Methods / Approach

  • Cost-Aware Marginal Decision-Focused Stopping (CAM-DF)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure. Routers and retrievers can rank candidate tools by relevance, but a ranking alone does not determine how many are worth selecting. Existing approaches leave acquisition under heterogeneous costs unaddressed. We formulate this decision as cost-aware marginal decision-focused stopping (CAM-DF) over ranked tool prefixes, with CAM-DF-lite as a compact interpretable variant. We train directly on the offline gap between stopping now and the best continuation: its sign labels the decision, its magnitude weights each error by the payoff at stake. We prove this objective is Bayes-aligned with the stopping target and that score-only rules are suboptimal under heterogeneous costs. We evaluate on 1,343 tasks across five tool-use domains. On $τ$-bench Retail, CAM-DF attains the highest payoff among deployable methods, with gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes. Our approach is state-of-the-art under heterogeneous costs and high cost pressure, with larger gains under weaker rankings. In live execution, CAM-DF exposes the agent to 37% fewer tools than full access while maintaining comparable task success. The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.