SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

Source facts

SWE-Serve contains 53 repository-grounded tasks derived from production changes to SGLang, spanning six inference-engineering families. Tasks use hidden functional and regression tests, end-to-end serving tests, calibrated performance gates, and adversarial verifier review. The preprint reports a gap between patches that pass local tests and production correctness under end-to-end evaluation.

Analysis

The benchmark moves coding-agent evaluation closer to the deployment boundary: a patch is not successful merely because it works locally. Production inference systems need functional, regression, end-to-end, and performance evidence in the same evaluation loop.

Uncertainty/Doubt

• This is an arXiv preprint; task representativeness, benchmark results, and model comparisons were not independently reproduced here.