How SWE-Serve Exposes the Gap Between Local Tests and Live Serving | NVIDIA Technical Blog

An AI coding agent’s patch can pass tests yet fail when the server loads a real model and handles requests. Evaluating changes to inference-serving software therefore requires checking the full serving path, including whether the system returns correct results through its public interface.

Developed with input from the SGLang team, SWE-Serve evaluates this gap with 53 tasks derived from merged changes to SGLang, an open-source system for serving large language models. Across 19 tasks with live-serving checks, the same patches passed 69.4% of the time when those checks were excluded, but only 45.9% with the complete verifier. About one in three patches that passed the other checks failed live-serving tests.

Quick links: Read the paper | Explore the leaderboard | Run SWE-Serve on GitHub

What SWE-Serve tests

Existing repository-level benchmarks evaluate coding agents across general software-engineering tasks, while inference benchmarks often concentrate on kernel generation or performance optimization. SWE-Serve instead tests repository-scale changes across the inference-serving stack, including model enablement, decoding, caching, scheduling, serving APIs, and runtime performance.

To evaluate this broader engineering work, SWE-Serve turns 83 merged SGLang pull requests into 53 executable tasks across six inference-engineering families.