RUMBA: Russian User Memory Benchmark

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko, Pavel Gulyaev, Alena Fenogenova
• Venue: arXiv: cs.CL
• Published: 2026-07-23
• Collected: 2026-07-24
• URL: http://arxiv.org/abs/2607.21447v1
• PDF: https://arxiv.org/pdf/2607.21447v1
• DOI: UNKNOWN

Plain-Language Brief

The authors introduce RUMBA (Russian User Memory Benchmark), a new long-term conversational memory benchmark for Russian. It offers a detailed taxonomy of memory-centric question types and a unified methodology that considers semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. The benchmark consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. An English subset is also provided.

What the Document Says

• The authors present RUMBA, a new long-term conversational memory benchmark for Russian.
• RUMBA provides a detailed taxonomy of memory-centric question types.
• The benchmark uses a unified methodology that accounts for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions.
• RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions.
• An English subset is also provided under the same methodology.

Key Concepts Mentioned

• long-term memory
• LLMs
• benchmark
• Russian User Memory Benchmark (RUMBA)
• QA pairs

Claims Made by the Document

• Existing benchmarks are English-centric and rely on aggregate retrieval metrics.
• RUMBA captures interactions between long-range context, temporal information, and reasoning.

Evidence Used

• The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric...

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.