Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts

Source

OpenReview

openreview.net/forum

Read the source

Source

Plain-Language Brief

The paper proposes Retrieval-of-Thought (RoT), a method that reuses prior reasoning as 'thought' steps to solve new problems efficiently. This approach reduces the need for long reasoning traces, which inflate latency and cost. The method was tested on various reasoning benchmarks, showing significant efficiency gains in terms of reduced output tokens, inference latency, and cost while maintaining accuracy.

What the Document Says

  • The paper introduces Retrieval-of-Thought (RoT).
  • RoT reuses prior reasoning as 'thought' steps for new problems.
  • RoT organizes steps into a thought graph with sequential and semantic edges.
  • At inference, RoT retrieves query-relevant nodes and applies reward-guided traversal to assemble a problem-specific template.
  • This dynamic template reuse reduces redundant exploration and output tokens, inference latency, and cost.
  • The method was tested on reasoning benchmarks with multiple models.
  • RoT reduced output tokens by up to 40%, inference latency by 82%, and cost by 59% while maintaining accuracy.

Key Concepts Mentioned

  • Retrieval-of-Thought (RoT)
  • reasoning
  • thought graph
  • dynamic template construction
  • redundant exploration
  • output tokens
  • inference latency
  • cost

Claims Made by the Document

  • RoT reduces output tokens, inference latency, and cost while maintaining accuracy.
  • RoT establishes a scalable paradigm for efficient LRM reasoning.

Evidence Used

  • Findings show small prompt growth but substantial efficiency gains...

Methods / Approach

  • Retrieval-of-Thought (RoT) method
  • reusing prior reasoning as 'thought' steps
  • organizing steps into a thought graph
  • query-relevant nodes retrieval
  • reward-guided traversal
  • dynamic template assembly

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not provide information on potential limitations of the proposed method.

Open Questions

  • What are the specific details of the reward-guided traversal?
  • How does RoT handle situations where prior reasoning steps are not available or relevant?

Notes for Future Q&A

May include interpretation. Labeled where it does.

It's important to understand the details of the reward-guided traversal and how RoT handles scenarios where prior reasoning steps may not be applicable.

Raw Abstract / Excerpt

Large reasoning models improve accuracy by producing long reasoning traces, but this inflates latency and cost, motivating inference-time efficiency. We propose Retrieval-of-Thought (RoT), which reuses prior reasoning as composable ``thought' steps to guide new problems. RoT organizes steps into a thought graph with sequential and semantic edges to enable fast retrieval and flexible recombination. At inference, RoT retrieves query-relevant nodes and applies reward-guided traversal to assemble a problem-specific template that guides generation. This dynamic template reuse reduces redundant exploration and, therefore, reduces output tokens while preserving accuracy. We evaluate RoT on reasoning benchmarks with multiple models, measuring accuracy, token usage, latency, and memory overhead. Findings show small prompt growth but substantial efficiency gains, with RoT reducing output tokens by up to 40%, inference latency by 82%, and cost by 59% while maintaining accuracy. RoT establishes a scalable paradigm for efficient LRM reasoning via dynamic template construction through retrieval.