Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition

Source

• Source name: OpenReview
• Source kind: paper
• Authors / organization: Yifei Duan, Raphael Shang, Deng Liang, Yongqiang Cai
• Venue: ICLR.cc/2026/Conference
• Published: UNKNOWN
• Collected: 2026-06-29
• URL: https://openreview.net/forum?id=n2WDmnGJZ3
• PDF: https://openreview.net/pdf/5010e4d2c63cec8ef00b954bfcd057cb4b1f72b7.pdf
• DOI: UNKNOWN

Plain-Language Brief

Researchers propose a method called ReBA (Retrieval Backward Attention) to enhance the quality of word-level embeddings produced by pretrained language models without affecting sentence-level representations negatively. They claim that this method improves performance in multiple zero-shot tasks.

What the Document Says

• The paper discusses a method called ReBA (Retrieval Backward Attention) for improving the quality of word-level embeddings in pretrained language models without affecting sentence-level representations negatively.
• ReBA combines input repetition with a novel backward attention mechanism that allows tokens to incorporate future context.
• Across multiple zero-shot tasks, ReBA significantly improves word-level embeddings while preserving sentence-level gains.

Key Concepts Mentioned

• pretrained language models
• embeddings
• Retrieval Backward Attention (ReBA)
• zero-shot tasks

Claims Made by the Document

• Repeating input text can improve sentence-level representations.
• Such repetition may adversely affect word-level embeddings.
• ReBA significantly improves word-level embeddings while preserving sentence-level gains.

Evidence Used

• The paper mentions that repeating input text can improve sentence-level representations and that such repetition may adversely affect word-level embeddings.
• The paper states that across multiple zero-shot tasks, ReBA significantly improves word-level embeddings while preserving sentence-level gains.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The quality of embeddings produced by pretrained language models is critical to downstream performance. Prior work has shown that repeating input text can improve sentence-level representations, but such repetition may adversely affect word-level embeddings. To address this, we propose ReBA (Retrieval Backward Attention), a method combining input repetition with a novel backward attention mechanism that enables tokens to incorporate future context. Across multiple zero-shot tasks, ReBA significantly improves word-level embeddings while preserving sentence-level gains, offering new insights into enhancing representation quality in decoder-only language models.