SWE-Serve Measures the Test-to-Serving Gap for Coding Agents

SWE-Serve evaluates whether AI coding-agent patches to inference-serving software work through the full serving path, including loading a real model, handling requests, and returning correct results through the public interface. [1] Developed with input from the SGLang team, the benchmark contains 53 executable tasks derived from 83 merged pull requests to the open-source SGLang project. [1] Across 19 tasks with live-serving checks, patches passed 69.4% of the time without those checks and 45.9% with the complete verifier. [1]

Existing repository-level benchmarks cover general software-engineering tasks, while inference benchmarks often focus on kernel generation or performance optimization. [1] The 53 tasks span six engineering families: advanced decoding, model and backend enablement, kernels and quantization, serving APIs, caching and runtime state, and distributed execution and scheduling. [1] Twelve tasks run on CPU, while 41 use a single NVIDIA H100. [1] NVIDIA says this first release does not evaluate other inference engines, multi-GPU execution, or multi-node serving. [1]

SWE-Serve’s central lesson is that repository-level correctness is not enough for inference-serving changes when a patch must also survive model loading and public-interface requests. Its scope differs from general repository benchmarks and inference benchmarks focused on kernel generation or performance optimization because it tests repository-scale work across the serving stack. The decline from 69.4% to 45.9% makes live-serving verification a material part of the benchmark rather than an optional add-on. What the sources don't address: whether the result generalizes beyond SGLang to other inference engines or to multi-GPU and multi-node serving.