NVIDIA describes co-design across AI infrastructure

The post, authored by Eduardo Alvarez, Jamie Li and Ronil Prasad, is dated Nov. 5.[1] NVIDIA frames the engineering challenge as co-design across GPU architecture, scale-up interconnect, serving software, CPU and storage.[1] It identifies HBM as the fastest, most expensive and most contested memory tier, shared by model weights and KV cache.[2] The framing matters because a serving system’s usable capacity depends not just on the GPU, but also on how the surrounding system stores and moves context.[1][2]

NVIDIA describes NVFP4 as 4-bit E2M1 values with shared FP8 scaling for each 16-value block and a per-tensor FP32 scale.[1] It says the format cuts model-weight footprint by about 3.5 times versus FP16 and 1.8 times versus FP8, and has become the default precision for several open agentic benchmarks with strict accuracy requirements.[1] NVIDIA also says NVFP4 can deliver up to four times FP8 throughput in cited benchmarks at the same interactivity.[1] Its Wide Expert Parallelism approach spreads experts across eight or more GPUs, which the company says can free HBM for cache and concurrency.[1]

NVIDIA says GPU HBM capacity stayed at 288 GB from GB300 to Rubin while bandwidth rose to 19.2 TB/s.[1] It reports GB200 NVL72’s aggregate NVLink bandwidth at 130 TB/s and says EP32 can deliver up to 1.8 times EP8’s output tokens per second per GPU at 100 tokens per second per user.[1] For KV cache, NVIDIA describes Dynamo’s router as tracking cache-block locations and weighing cached-prefix overlap and worker load when assigning requests.[1] Decode GPUs can then dedicate HBM to cache and batch size rather than holding headroom for bursty prefill, according to the company.[1]

NVIDIA says its KV Cache Runner tracks cache blocks across GPU, pinned CPU and storage tiers, including local SSDs and mounted network filesystems.[1] In a separate design, Dynamo can stage inactive-session cache in Vera CPU memory during tool calls and return it over NVLink-C2C when an agent resumes.[1] NVIDIA also describes CMX as shared context memory with lower cost per bit than HBM, and says Dynamo moves inactive context there and retrieves it before reuse.[2] NVIDIA lists Cloudian, DDN, Dell Technologies and Everpure as CMX partners.[2]