Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Source

arXiv

arxiv.org/abs/2608.06305v1

Read the source

Source

Plain-Language Brief

A research paper proposes a new method for document search called Reliable Embedding-free Agentic Document-search (READ), which is designed to overcome limitations in current methods for long documents, particularly financial statements and reports. The authors argue that existing methods often fail due to issues like identical figures competing in the same embedding space and the separation of units from numbers by chunk boundaries. READ uses three deterministic operations - normalized lexical search, structural navigation, and bounded span reads - to provide a replayable audit trail instead of an opaque similarity score. The paper shows that READ outperforms existing methods on verified questions by a significant margin.

What the Document Says

  • The paper proposes Reliable Embedding-free Agentic Document-search (READ) for document search in long documents like financial statements and reports.
  • Existing methods have issues such as identical figures competing in the same embedding space and units being separated from numbers by chunk boundaries.
  • READ uses three deterministic operations - normalized lexical search, structural navigation, and bounded span reads.
  • These operations provide a replayable audit trail instead of an opaque similarity score.
  • The paper shows that READ outperforms existing methods on verified questions by a significant margin.
  • The paper also reports that BM25 is statistically indistinguishable from READ, implying the result separates embedding-based from embedding-free retrieval, not agentic from lexical search.

Key Concepts Mentioned

  • document search
  • financial statements
  • audit reports
  • retrieval-augmented generation
  • top-k nearest neighbours
  • embedding space
  • chunk boundary
  • unit problem
  • normalized lexical search
  • structural navigation
  • bounded span reads
  • Model Context Protocol
  • replayable audit trail
  • opaque similarity score
  • VERIFIED QUESTIONS
  • BM25

Claims Made by the Document

  • Existing methods are structurally unsound for a certain class of documents.
  • READ outperforms existing methods on verified questions by a significant margin.
  • BM25 is statistically indistinguishable from READ, implying the result separates embedding-based from embedding-free retrieval.

Evidence Used

  • 86.8% of content lines are table rows in a 780-page government financial report.
  • Thousands of near-identical figures compete in one embedding space.
  • A figure inherits its unit from a header a median of 13 lines above it.
  • READ answers 58.8% against dense retrieval's 15.7% on 51 verified questions.
  • BM25 is statistically indistinguishable from READ.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.