Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Source

arXiv

arxiv.org/abs/2608.06305v1

Read the source

Source

Plain-Language Brief

The paper proposes a new approach for document search called 'READ (Reliable Embedding-free Agentic Document-search)', which aims to address the issues with current black-box retrieval designs in handling long documents like financial statements, audit reports, and regulatory returns. The proposed method uses an agent that reads the raw document through three deterministic operations: normalized lexical search, structural navigation, and bounded span reads. The agent's performance is compared to dense retrieval methods, showing improved results on 51 verified questions.

What the Document Says

  • The paper introduces READ (Reliable Embedding-free Agentic Document-search).
  • READ uses an agent that reads the raw document through three operations: normalized lexical search, structural navigation, and bounded span reads.
  • The performance of READ is compared to dense retrieval methods on 51 verified questions.
  • READ answers 58.8% of the questions correctly against dense retrieval's 15.7% (p_Holm = 2 x 10^-5).
  • When the same loop is given to an agent with a top-k tool, it reaches only 27.5%.
  • The paper also reports that BM25 is statistically indistinguishable from READ.

Key Concepts Mentioned

  • document search
  • READ
  • agentic operations
  • normalized lexical search
  • structural navigation
  • bounded span reads
  • financial statements
  • audit reports
  • regulatory returns

Claims Made by the Document

  • READ improves document search performance over dense retrieval methods.
  • BM25 is statistically indistinguishable from READ.

Evidence Used

  • The paper presents results comparing READ to dense retrieval on 51 verified questions.
  • The paper mentions the statistical significance of READ's performance against dense retrieval (p_Holm = 2 x 10^-5).

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.