Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Source

arXiv

arxiv.org/abs/2607.01211v1

Read the source

Source

Plain-Language Brief

The paper analyzes three benchmark tools used to evaluate coding agents: GSO, SWE-Perf, and SWE-fficiency. The study finds that many of the reference patches don't pass validity rules when replayed across different machine types. Additionally, the ranking of public submissions depends on the specific scoring rule used by each benchmark. Lastly, it is found that at least one submission often matches or outperforms the reference patch for most tasks.

What the Document Says

  • The paper discusses performance-optimization benchmarks GSO, SWE-Perf, and SWE-fficiency.
  • These benchmarks evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches.
  • The study shows that many reference patches do not pass validity rules when replayed across different machine types for these benchmarks.
  • The ranking of public submissions depends on the specific scoring rule used by each benchmark.
  • It is found that at least one submission often matches or outperforms the reference patch for most tasks.

Key Concepts Mentioned

  • Performance-optimization benchmarks
  • Coding agents
  • GSO
  • SWE-Perf
  • SWE-fficiency
  • Public submissions
  • Scoring rules
  • Reference patches
  • Machine types

Claims Made by the Document

  • Many reference patches do not pass validity rules when replayed across different machine types.
  • The ranking of public submissions depends on the specific scoring rule used by each benchmark.
  • At least one submission often matches or outperforms the reference patch for most tasks.

Evidence Used

  • Replay of official reference patches for 740 code optimization tasks across four common types of Google Cloud machines.
  • Analysis of 8 public submissions shared by GSO and SWE-fficiency.
  • Comparison of at least one submission with the reference patch on 450 replay-valid GSO and SWE-fficiency tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.