llm-d 1.0 adds production features for large-scale inference

The release adds priority- and fairness-aware flow control, intended to let latency-sensitive requests and bursty agent workloads share a fleet without starving one another.[1] It also adds router high availability, including a hot standby and rolling updates intended to keep traffic flowing during upgrades and maintenance.[1] The configuration API is now stable at v1, and the request path has been hardened for production deployments.[1] For agent workloads, session-aware routing, turn-priority scheduling and native Responses API state are designed to help reuse context across turns.[1]

The project supports inference across model servers including vLLM and SGLang.[1] IBM’s newsroom says managing context often consumes more resources than generating the final output itself.[1] That makes routing and cache behavior relevant to the cost and responsiveness of serving workloads, particularly when agents make repeated requests. The release’s session-aware routing and turn-priority scheduling address this operational challenge, but the announcement does not establish that every workload will see the same performance or cost results.[1]

Researchers used llm-d to deploy the approximately 753-billion-parameter open-weight GLM 5.2 model on widely deployed NVIDIA GPUs.[1] They reported peak output above 6.6 million tokens per minute, serving up to 3,000 concurrent coding agents with zero preemptions.[1] At current cloud rental rates, the deployment’s cost per token would be five to 10 times lower than equivalent commercial API pricing, according to the project’s report.[1] Researchers also reported an approximately 98% prefix-cache hit rate and a serving cost of about $0.11 per million tokens.[1] These are reported results for this deployment, not a guarantee for other configurations.

Google Cloud says it is integrating llm-d 1.0 into Google Kubernetes Engine across its TPU and NVIDIA GPU portfolio.[1] The company also reports that prefix-aware routing and host-memory offload reduced median time to first token by more than 85% and raised total throughput 2.4 times versus a plain Kubernetes Service.[2] JetBrains says llm-d runs in production behind its free Junie Lite plan, while DigitalOcean says it underpins its Serverless Inference and Managed Agents services.[1] TechTarget analyst Jason Andersen said packaged versions from cloud providers or software vendors are the likeliest enterprise route to adoption, and cautioned that vendor support does not necessarily mean commitment.[4]