DeepSeek has published the source code of DeepInfer, the in-house inference framework the company uses to serve requests to its models. For now it targets DeepSeek V4.1 Flash, with support for future models promised later.

DeepSeek weights could already be downloaded and run with tools like vLLM. But speed and cost depend heavily on how the serving software schedules requests and spreads work across chips. Developers now get the same optimizations that sit behind the official API.

What is inside

The project ships under the Apache 2.0 license. It bundles optimized compute kernels and communication libraries, including the previously released FlashMLA, DeepGEMM and DeepEP, plus mechanisms for spreading inference across multiple chips. It supports speculative decoding, asynchronous scheduling and structured outputs, where the response format is defined up front.

Alongside the engine, the company updated deepseek-recipe, a Rust and Python library that turns API requests into model inputs and back again. It helps reproduce the behavior of the official API in a self-hosted deployment.

What is still missing

In 2025 DeepSeek said releasing the full engine was impractical because it was too tightly tied to internal infrastructure. This release works around that in part. The open version focuses on offline inference and does not yet include KV cache management, splitting inference stages across machines, or load balancing between replicas. Teams will have to build those pieces themselves to run a full service.

Nvidia and Huawei side by side

Only two platforms are supported from day one: Nvidia Blackwell GPUs (SM100) and Huawei Ascend 950 NPUs. Older cards, including Hopper, are not on the list. For DeepSeek this continues its shift toward Chinese hardware amid US export restrictions.

Our take

For anyone hosting models on their own servers, this is a useful reference: you can see how a large provider squeezes performance out of hardware and borrow individual techniques. The bar is high, though: without Blackwell or Ascend you cannot run DeepInfer, so most teams are better off staying with vLLM or an API for now. Details are in The New Stack report.