Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

Source

arXiv

arxiv.org/abs/2607.27083v1

Read the source

Source

Plain-Language Brief

The paper proposes a method called Cost-Aware Marginal Decision-Focused Stopping (CAM-DF) for selecting tools in large language model agents. This method aims to balance between acquiring enough tools and keeping costs low, as these agents rely on various external services. The approach involves training directly on the offline gap between stopping now and the best continuation, with the goal of making decisions that maximize payoff while minimizing cost and privacy exposure. The authors claim that their method outperforms other deployable methods in a retail task-benchmark and reduces tool usage by 37% in live execution without affecting task success.

What the Document Says

  • The paper discusses Cost-Aware Marginal Decision-Focused Stopping (CAM-DF) for tool acquisition in LLM agents.
  • LLM agents rely on various external services, and selecting the right tools is a challenge due to cost and privacy concerns.
  • Existing approaches do not address the issue of heterogeneous costs in tool selection.
  • The decision is formulated as a cost-aware marginal decision-focused stopping problem over ranked tool prefixes.
  • CAM-DF-lite is a compact interpretable variant of CAM-DF.
  • Training is done on the offline gap between stopping now and the best continuation, with the sign labeling the decision and the magnitude weighing errors by payoff at stake.
  • The objective is proven to be Bayes-aligned with the stopping target and score-only rules are suboptimal under heterogeneous costs.
  • Evaluation was done on 1343 tasks across five tool-use domains, showing CAM-DF attained the highest payoff among deployable methods in a retail task-benchmark.
  • CAM-DF reduces tool usage by 37% in live execution while maintaining comparable task success.
  • CAM-DF is a lightweight pre-execution plugin that can turn existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.

Key Concepts Mentioned

  • LLM agents
  • tool selection
  • heterogeneous costs
  • Cost-Aware Marginal Decision-Focused Stopping (CAM-DF)
  • payoff
  • privacy exposure

Claims Made by the Document

  • Existing approaches do not address the issue of heterogeneous costs in tool selection.
  • CAM-DF attained the highest payoff among deployable methods in a retail task-benchmark.
  • CAM-DF reduces tool usage by 37% in live execution while maintaining comparable task success.

Evidence Used

  • Training on the offline gap between stopping now and the best continuation.
  • Evaluation on 1,343 tasks across five tool-use domains.
  • Results from the evaluation showing CAM-DF's performance.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure. Routers and retrievers can rank candidate tools by relevance, but a ranking alone does not determine how many are worth selecting. Existing approaches leave acquisition under heterogeneous costs unaddressed. We formulate this decision as cost-aware marginal decision-focused stopping (CAM-DF) over ranked tool prefixes, with CAM-DF-lite as a compact interpretable variant. We train directly on the offline gap between stopping now and the best continuation: its sign labels the decision, its magnitude weights each error by the payoff at stake. We prove this objective is Bayes-aligned with the stopping target and that score-only rules are suboptimal under heterogeneous costs. We evaluate on 1,343 tasks across five tool-use domains. On $τ$-bench Retail, CAM-DF attains the highest payoff among deployable methods, with gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes. Our approach is state-of-the-art under heterogeneous costs and high cost pressure, with larger gains under weaker rankings. In live execution, CAM-DF exposes the agent to 37% fewer tools than full access while maintaining comparable task success. The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.