FIELD NOTES TOPIC
scoring rules
1 note, newest first.
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper analyzes three benchmark tools used to evaluate coding agents: GSO, SWE-Perf, and SWE-fficiency. The study finds that many of the reference patches don't pass validity rules when replayed across different…
