Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Source

arXiv

arxiv.org/abs/2607.01211v1

Read the source

Source

Plain-Language Brief

The paper evaluates three performance-optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to assess coding agents. The authors find that these benchmarks have issues with stability, scoring rules, and the number of tasks already solved by public submissions. They demonstrate that reference patches do not always replay well across different machines, leading to inconsistent results. The study also reveals that the ranking of public submissions depends on the benchmark's scoring rule, and a significant portion of the tasks are overly weighted. Moreover, at least one public submission often matches or beats the reference patch or unoptimized base code on most tasks.

What the Document Says

  • The paper examines three performance-optimization benchmarks: GSO, SWE-Perf, and SWE-efficiency.
  • These benchmarks evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches.
  • Most benchmark tasks can be replayed, but their reference patches are stable across different machines for only a small percentage of tasks (39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-efficiency tasks).
  • The study shows that public submission rankings depend on the benchmark's scoring rule.
  • Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise comparison.
  • SWE-efficiency's leaderboard scoring rule assigns overly high score weights to the worst ten tasks (58.5%-82.8%).
  • Looking across 10 public submissions for each task, at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks.
  • At least one public submission beats the unoptimized base code on 99.8% (449/450) of these tasks.

Key Concepts Mentioned

  • performance-optimization benchmarks
  • GSO
  • SWE-Perf
  • SWE-efficiency
  • coding agents
  • real repositories
  • runtime
  • unoptimized baselines
  • official reference patches
  • public submissions
  • scoring rules

Claims Made by the Document

  • Most benchmark tasks can be replayed, but their reference patches are stable across different machines for only a small percentage of tasks.
  • Public submission rankings depend on the benchmark's scoring rule.
  • Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise comparison.
  • SWE-efficiency's leaderboard scoring rule assigns overly high score weights to the worst ten tasks (58.5%-82.8%).
  • At least one public submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-efficiency tasks.
  • At least one public submission beats the unoptimized base code on 99.8% (449/450) of these tasks.

Evidence Used

  • The paper provides data showing the stability of reference patches across different machines for each benchmark.
  • The study compares official rankings and finds disagreement on 9 out of 28 pairwise submissions.
  • The authors demonstrate that SWE-efficiency's leaderboard scoring rule assigns high score weights to the worst ten tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.