Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

Source

arXiv

arxiv.org/abs/2607.06411v1

Read the source

Source

Plain-Language Brief

The paper introduces RuBench 1.0, a new benchmark for coding agents that uses tasks specified in Russian rather than English. The benchmark is designed to measure the performance of coding agents when they are given real maintenance work in a native language style. The authors evaluate several product configurations and find that the best configuration resolves 78.7% of the tasks, but there is a significant gap between the best and weakest models. They also find evidence that the deployed product may be substituting the model on some tasks.

What the Document Says

  • RuBench 1.0 is a benchmark for coding agents with 25 tasks mined from recent fix commits in five open-source repositories.
  • Each task is specified natively in Russian and judged by the upstream maintainer's regression tests.
  • The authors evaluate several product configurations, including Claude Code with Opus 4.8, Sonnet 5, Haiku 4.5, and Codex CLI with GPT-5.5.
  • The best configuration resolves 78.7% of tasks, but there is a significant gap between the best and weakest models.
  • Evidence was found that the deployed product may be substituting the model on some tasks.

Key Concepts Mentioned

  • RuBench
  • coding agents
  • open-source repositories
  • task statements
  • upstream maintainers
  • regression tests
  • product configurations
  • Claude Code
  • Opus
  • Sonnet
  • Haiku
  • Codex CLI
  • GPT-5.5

Claims Made by the Document

  • RuBench 1.0 measures the performance of coding agents in a native language setting.
  • The best configuration resolves 78.7% of tasks.
  • There is a significant gap between the best and weakest models.

Evidence Used

  • The authors describe RuBench 1.0 as a benchmark for coding agents that uses tasks specified in Russian.
  • The authors report on the evaluation of several product configurations and their results.
  • The authors provide statistics on the number of tasks resolved by each configuration.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify; Python, PHP, TypeScript, JavaScript), where each task is specified natively in Russian -- written from scratch in the style of an actual customer request, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All 25 fix commits postdate the training-data cutoffs of every evaluated model, giving a contamination argument that holds task-by-task. We evaluate deployed product configurations (CLI agent + model + reasoning effort) -- Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 -- with three independent runs each, reporting pass@1 with task-level confidence intervals, paired comparisons, dollar cost, and token usage. The best configuration resolves 78.7% of tasks; at N=25 only the gaps to the weakest model are statistically resolvable, which we state explicitly. Auditing full trajectories of a fifth, hors-concours configuration (Claude Code + Fable 5, July 2, 2026 release), we caught the product silently substituting the model: on 5 of 25 tasks (20%) an official safeguard fallback re-routed routine HTTP-protocol fixes to Opus 4.8 -- direct, reproducible evidence that the deployed product, not the model, is the unit actually measured. We release task statements, metadata, full agent trajectories, and diffs; grading oracles are withheld, with a SHA-256 manifest committed at publication time.