Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Source

arXiv

arxiv.org/abs/2608.06305v1

Read the source

Source

Plain-Language Brief

The paper introduces a new approach called READ (Reliable Embedding-free Agentic Document-search) for retrieving information from long documents such as financial statements and audit reports. The traditional method of embedding chunks of text and surface the top-k nearest neighbors of the query is criticized for being structurally unsound, leading to errors due to the inconsistency in separating numbers with different units. The new approach uses three deterministic operations performed by an agent on the raw document: normalized lexical search, structural navigation, and bounded span reads. This allows for a replayable audit trail instead of an opaque similarity score. The paper compares READ to a top-k tool and reports that READ answers 58.8% of verified questions correctly compared to 15.7% by dense retrieval (p_Holm = 2 x 10^-5).

What the Document Says

  • The paper presents an alternative approach called READ for document search.
  • Traditional method embeds chunks of text and surfaces top-k nearest neighbors.
  • Criticism: this method is structurally unsound for certain documents leading to errors in separating numbers with different units.
  • READ uses three deterministic operations performed by an agent on the raw document: normalized lexical search, structural navigation, and bounded span reads.
  • This allows for a replayable audit trail instead of an opaque similarity score.
  • The paper compares READ to a top-k tool.
  • READ answers 58.8% of verified questions correctly compared to 15.7% by dense retrieval (p_Holm = 2 x 10^-5).

Key Concepts Mentioned

  • Retrieval-augmented generation
  • chunk the text
  • embedding
  • nearest neighbours
  • financial statements
  • audit reports
  • regulatory returns
  • table rows
  • numerical chunks
  • fiscal-year header
  • Model Context Protocol
  • normalized lexical search
  • structural navigation
  • bounded span reads
  • replayable audit trail
  • dense retrieval
  • top-k tool

Claims Made by the Document

  • Traditional method of document retrieval is structurally unsound for certain documents.
  • READ improves the accuracy of answering verified questions compared to dense retrieval.
  • BM25 is statistically indistinguishable from READ.

Evidence Used

  • 86.8% of content lines are table rows in government financial report
  • Thousands of near-identical figures compete in one embedding space
  • A figure inherits its unit from a header a median of 13 lines above it
  • 27-30% of numeric chunks have no fiscal-year header at every chunk size tried
  • READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5)
  • BM25 is statistically indistinguishable from READ

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.