Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Source

arXiv

arxiv.org/abs/2608.06305v1

Read the source

Source

Plain-Language Brief

The authors propose a new method for document search called Reliable Embedding-free Agentic Document-search (READ). In contrast to the common design of retrieval-augmented generation, READ uses an agent that reads the raw document through three deterministic operations. This method outperforms dense retrieval methods on financial reports and audit documents, answering 58.8% of verified questions correctly compared to 15.7% by the traditional approach.

What the Document Says

  • The authors present a new document search method called READ (Reliable Embedding-free Agentic Document-search).
  • READ uses an agent that reads documents through three deterministic operations: normalized lexical search, structural navigation, and bounded span reads.
  • This method outperforms traditional dense retrieval methods on financial reports and audit documents.
  • On 51 verified questions, READ answers correctly 58.8% of the time, compared to 15.7% by the traditional approach.
  • When given the same loop but a top-k tool, the agent reaches only 27.5%
  • The evidence does not support that BM25 (a dense retrieval method) is statistically distinguishable from READ.

Key Concepts Mentioned

  • document search
  • retrieval-augmented generation
  • READ
  • financial statements
  • audit reports
  • regulatory returns

Claims Made by the Document

  • READ outperforms traditional dense retrieval methods.
  • BM25 is not statistically distinguishable from READ.

Evidence Used

  • On 51 verified questions, READ answers correctly 58.8% of the time, compared to 15.7% by the traditional approach.
  • When given the same loop but a top-k tool, the agent reaches only 27.5%
  • BM25 is statistically indistinguishable from READ.

Methods / Approach

  • READ method
  • traditional dense retrieval methods
  • top-k tools

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The text does not specify any limitations of the agent used in READ.

Open Questions

  • What is the exact performance of READ on other types of documents outside financial reports and audit documents?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study focuses on a specific class of documents (financial statements, audit reports, regulatory returns), but it is unclear how well the proposed method would perform on other types of documents.

Raw Abstract / Excerpt

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.