By By Simon Yoon
Publication Date: 2026-10-20 19:18:00
The llm-d project released version 1.0 on Oct. 20, adding workload controls and routing features for organizations running large AI models on Kubernetes infrastructure they manage.
IBM, Red Hat and Google are lead maintainers of llm-d, an open-source framework that distributes inference requests among model servers such as vLLM and SGLang. The project is hosted by the Cloud Native Computing Foundation as a sandbox project.
The new controls let operators prioritize requests, share capacity among users and workloads, and shed traffic when demand exceeds capacity. Existing deployments keep their current scheduling behavior unless operators enable the new policies. Version 1.0 also adds a configuration API.
The release includes a standby router. Active-active routing and failover for individual inference requests remain planned features. llm-d also supports coordinated upgrades where separate machine groups handle prompt processing and response generation.
Session-aware routing can send a…
