DeepSeek open-sources its DeepInfer inference framework

DeepSeek announced that it was open-sourcing DeepInfer, describing it as its in-house inference framework.[1] The company says DeepInfer powers its large-scale production services, and its GitHub release describes offline inference with core features needed for serving at scale.[3] The release is published under the Apache 2.0 license.[2] DeepSeek’s X post announcing the framework is timestamped October 20, 2026.[1]

DeepSeek lists optimized kernels, a model and sampler runtime, and parallelism features for attention and mixture-of-experts workloads.[3] Its listed hardware support includes NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.[3] The framework also supports structured generation constrained by a regex, JSON schema or EBNF grammar.[3] DeepSeek says its asynchronous scheduler overlaps CPU batch scheduling with device execution and is enabled by default.[3] It also says confidence estimates schedule verification to accelerate decoding with draft tokens.[3] These capabilities describe what the release offers; the evidence does not establish performance on other hardware or models.

MarkTechPost reported that DeepSeek’s charts show decode throughput above 25,000 output tokens per second per GPU at 256 requests per rank.[4] DeepSeek’s published benchmark measurements used four SM100 GPUs for prefill and 64 for decode.[3] MarkTechPost also reported that a single cluster serves 155 trillion tokens per day.[4] These figures should be read as reported results, not as independently verified performance: The Next Web said DeepInfer’s figures had not yet been tested independently outside DeepSeek.[2] The evidence does not establish how results would transfer to other workloads or deployments.

DeepSeek also announced that it was open-sourcing deepseek-recipe, a production component for token encoding and decoding.[1] DeepSeek says DeepInfer was built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.[3] The vLLM project thanked DeepSeek for crediting the open-source community and said it would improve its V4.1-Flash support using ideas from DeepInfer.[2] DeepSeek says code for future models and further optimizations will be added to the same repository.[1] The Next Web reported that the repository had more than 8,000 stars by Tuesday afternoon.[2]