NVIDIA Details EPD Disaggregation to Accelerate Multimodal Inference

Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill and decode stages. [1] The technique is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts (MoE) models. [1] By using EPD disaggregation with the open source NVIDIA Dynamo inference framework, systems can achieve up to 5x faster time to first token (TTFT) and 7x faster end-to-end response time. [1] Dynamo supports this by separating the phases into independently scalable stages, allowing dedicated encoder workers to improve batching, memory efficiency, and overall throughput. [1]

In standard aggregated serving, vision encoding, LLM prefill, and decode all share a single worker and scheduling domain. [1] This shared design can become a bottleneck because vision encoding for requests with multiple images or video can take hundreds of milliseconds or longer. [1] Because the encoder and LLM work share the same GPU, a media-heavy request can delay its own prefill and contend with concurrent prefill. [1]

Separating vision encoding from LLM processing allows infrastructure teams to scale components independently, removing a significant bottleneck for video and image-heavy workloads. This shift from aggregated serving addresses the resource contention that occurs when preprocessing media and running vision transformers compete with standard prefill operations on a shared GPU. The use of an open-source framework like Dynamo provides a scalable blueprint for managing these complex pipelines, particularly for increasingly common multimodal applications. What the sources don't address: How the networking overhead of passing data between the disaggregated vision encoder and the LLM workers impacts performance at massive concurrent scale.