RUMBA: Russian User Memory Benchmark

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko, Pavel Gulyaev, Alena Fenogenova
• Venue: arXiv: cs.CL
• Published: 2026-07-23
• Collected: 2026-07-24
• URL: http://arxiv.org/abs/2607.21447v1
• PDF: https://arxiv.org/pdf/2607.21447v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduce a new benchmark called RUMBA for evaluating the long-term conversational memory abilities of Language Models (LLMs), particularly focusing on Russian dialogues. The benchmark provides a detailed categorization of question types related to memory and includes temporal reasoning. It's designed to address the limitations of existing English-centric benchmarks by capturing interactions between long-range context, temporal information, and reasoning.

What the Document Says

• The study introduces RUMBA (Russian User Memory Benchmark).
• RUMBA is designed to evaluate the long-term conversational memory abilities of LLMs.
• It focuses on Russian dialogues but also provides an aligned English subset.
• RUMBA includes a fine-grained taxonomy of memory-centric question types.
• The benchmark accounts for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions.
• It consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions.

Key Concepts Mentioned

• Language Models (LLMs)
• RUMBA
• long-term conversational memory
• Russian dialogues
• benchmark
• QA pairs

Claims Made by the Document

• Existing benchmarks are English-centric and rely on aggregate retrieval metrics.
• RUMBA captures interactions between long-range context, temporal information, and reasoning.

Evidence Used

• The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric...

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.