Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

Source

arXiv

arxiv.org/abs/2608.12282v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor
  • Venue: arXiv: cs.AI
  • Published: 2026-08-12
  • Collected: 2026-08-13
  • URL: http://arxiv.org/abs/2608.12282v1
  • PDF: https://arxiv.org/pdf/2608.12282v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The study introduces VAKRA, a benchmark for evaluating the ability of agents to reason across APIs and document collections in enterprise settings. The benchmark includes over 8000 executable APIs across 62 domains with tasks that increase in difficulty. The researchers find that even the best model only achieves around 70% accuracy on simple API tasks, with performance dropping significantly when reasoning depth increases or when policy-constrained questions are involved. Failures often occur during language-mediated reasoning such as entity disambiguation and cross-source grounding.

What the Document Says

  • The study presents VAKRA, a benchmark for evaluating agents in enterprise settings that reason across APIs and document collections.
  • VAKRA includes over 8000 executable APIs across 62 domains with tasks of increasing difficulty.
  • The tasks involve diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints.
  • Correctness is verified by re-executing predicted tool calls against live APIs.
  • The study evaluates frontier and open-weight models using a fixed ReAct harness.
  • The best model achieves 70.4% accuracy on single-hop endpoint-style tasks, but drops to 50-51% on compositional APIs.
  • Performance degrades by over 50% as reasoning depth increases.
  • Policy-constrained questions expose severe failures, with as low as 2.4% accuracy on unanswerable queries.
  • Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics.

Key Concepts Mentioned

  • VAKRA
  • benchmark
  • APIs
  • document collections
  • enterprise settings
  • multi-hop reasoning
  • multi-source reasoning
  • natural-language tool-use policy

Claims Made by the Document

  • Existing benchmarks evaluate capabilities in isolation.
  • The best model achieves around 70% accuracy on simple API tasks.
  • Performance drops significantly when reasoning depth increases or when policy-constrained questions are involved.
  • Failures often occur during language-mediated reasoning such as entity disambiguation and cross-source grounding.

Evidence Used

  • The study introduces VAKRA, a benchmark for evaluating agents in enterprise settings that reason across APIs and document collections.
  • The study finds that even the best model only achieves around 70% accuracy on simple API tasks, with performance dropping significantly when reasoning depth increases or when policy-constrained questions are involved.

Methods / Approach

  • Introduction of VAKRA benchmark
  • Evaluation of frontier and open-weight models using a fixed ReAct harness
  • Trace analysis

Limitations Stated by the Authors

  • The study does not provide specific details about the models used for evaluation.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The text lacks information about how the trace analysis was performed.

Open Questions

  • What models were used in the evaluation?
  • How was the trace analysis performed?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It is not stated in the source how the models used for evaluation were chosen or implemented.
  • The text does not provide details about how the trace analysis was conducted.

Raw Abstract / Excerpt

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4% on single-hop endpoint-style tasks and drops to 50--51% on compositional APIs; performance degrades by over 50% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA