Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Source

arXiv

arxiv.org/abs/2608.06305v1

Read the source

Source

Plain-Language Brief

The paper proposes a new method for document search named Reliable Embedding-free Agentic Document-search (READ), which aims to improve the accuracy of retrieval-augmented generation on specific types of documents such as financial statements and audit reports. The proposed method uses deterministic operations to read the raw document, allowing for replayable audit trails instead of opaque similarity scores. The results show that READ answers 58.8% of verified questions compared to 15.7% answered by dense retrieval (p_Holm = 2 x 10^-5), and it outperforms a top-k tool by 23.5 points (p_Holm = 0.017). The paper also reports that BM25 is statistically indistinguishable from READ, suggesting that the result separates embedding-based from embedding-free retrieval, not agentic from lexical search.

What the Document Says

  • The paper proposes a method called READ for document search.
  • READ uses deterministic operations to read raw documents through normalized lexical search, structural navigation, and bounded span reads.
  • These operations expose the Model Context Protocol, creating replayable audit trails instead of opaque similarity scores.
  • The paper compares READ's performance with dense retrieval on 51 verified questions and finds that READ answers 58.8% compared to 15.7% by dense retrieval (p_Holm = 2 x 10^-5).
  • READ also outperforms a top-k tool by 23.5 points (p_Holm = 0.017).
  • BM25 is statistically indistinguishable from READ, suggesting the result separates embedding-based from embedding-free retrieval, not agentic from lexical search.
  • The text describes problems with current retrieval-augmented generation on financial statements and audit reports.

Key Concepts Mentioned

  • document search
  • Reliable Embedding-free Agentic Document-search (READ)
  • deterministic operations
  • Model Context Protocol
  • verified questions
  • dense retrieval
  • top-k tool
  • BM25

Claims Made by the Document

  • READ improves the accuracy of retrieval-augmented generation.
  • READ outperforms dense retrieval and a top-k tool.

Evidence Used

  • On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5)
  • READ still leads by 23.5 points (p_Holm = 0.017) when compared to a top-k tool.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.