Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

Source

arXiv

arxiv.org/abs/2608.12282v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor
  • Venue: arXiv: cs.AI
  • Published: 2026-08-12
  • Collected: 2026-08-13
  • URL: http://arxiv.org/abs/2608.12282v1
  • PDF: https://arxiv.org/pdf/2608.12282v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The researchers introduce VAKRA, a benchmark for evaluating the ability of agents to reason across structured APIs and document collections in enterprise settings. The benchmark includes over 8,000 executable APIs across 62 domains with tasks spanning three levels of difficulty. The best model achieves 70.4% accuracy on single-hop tasks but performs poorly on more complex tasks, dropping to around 50-51% on compositional APIs and experiencing significant degradation as reasoning depth increases. The researchers find that failures occur mainly in language-mediated reasoning processes such as entity disambiguation and cross-source grounding.

What the Document Says

  • The paper presents VAKRA, a benchmark for evaluating the performance of agents in enterprise settings that involve structured APIs and document collections.
  • VAKRA consists of over 8,000 executable APIs across 62 domains with tasks categorized into three difficulty levels: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints.
  • The authors use a fixed ReAct harness to evaluate frontier and open-weight models.
  • The best model achieves 70.4% accuracy on single-hop endpoint-style tasks but performs poorly on compositional APIs, dropping to around 50-51%.
  • Performance degrades significantly as reasoning depth increases, and policy-constrained questions reveal severe failures.
  • Trace analysis reveals that failures primarily occur during language-mediated reasoning processes such as entity disambiguation and cross-source grounding.

Key Concepts Mentioned

  • API
  • Enterprise setting
  • Benchmark
  • Agents
  • Reasoning
  • Document collections
  • Multi-hop reasoning
  • Structured APIs
  • Natural-language tool-use policy
  • Entity disambiguation
  • Cross-source grounding

Claims Made by the Document

  • VAKRA is a benchmark for evaluating agents in enterprise settings.
  • The best model achieves 70.4% accuracy on single-hop tasks but performs poorly on compositional APIs.
  • Performance degrades significantly as reasoning depth increases.
  • Policy-constrained questions reveal severe failures.

Evidence Used

  • The paper describes the creation and purpose of VAKRA.
  • The authors provide percentages to demonstrate the performance of the best model.
  • The text states that performance degrades as reasoning depth increases.
  • The authors mention failures on policy-constrained questions.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4% on single-hop endpoint-style tasks and drops to 50--51% on compositional APIs; performance degrades by over 50% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA