Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Source

arXiv

arxiv.org/abs/2607.01211v1

Read the source

Source

Plain-Language Brief

The paper evaluates three repository-level performance optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to measure coding agents. The authors find issues with the stability of runtime for these benchmarks, dependence on scoring rules, and overlap between solved tasks by public submissions and reference patches. They propose a method to identify tasks with more reliable performance signals and expose remaining performance gaps.

What the Document Says

  • The paper analyzes three benchmark tools: GSO, SWE-Perf, and SWE-efficiency, which measure coding agents by applying patches to real repositories.
  • The study finds that most reference patches don't satisfy the original benchmark validity rules in every cross-machine replay for a majority of tasks.
  • Public submission rankings depend on the benchmark scoring rule, with disagreements between official rankings and SWE-efficiency assigning overly high score weights to certain tasks.
  • At least one public submission matches or beats the reference patch on most replay-valid GSO and SWE-efficiency tasks.
  • The paper concludes by suggesting a method to identify tasks with more reliable performance signals and expose remaining performance gaps.

Key Concepts Mentioned

  • repository-level performance-optimization benchmarks
  • GSO
  • SWE-Perf
  • SWE-efficiency
  • coding agents
  • runtime
  • public submissions
  • benchmark scoring rule
  • Google Cloud machines

Claims Made by the Document

  • Most reference patches don't meet the original benchmark validity rules in every cross-machine replay for a majority of tasks.
  • Public submission rankings depend on the benchmark scoring rule.
  • At least one public submission matches or beats the reference patch on most replay-valid GSO and SWE-efficiency tasks.

Evidence Used

  • The authors replayed the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines.
  • They compared public submissions shared by GSO and SWE-fficiency and analyzed the disagreements between official rankings.
  • They looked at 10 public submissions for each task to determine if at least one submission matches or beats the reference patch.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.