NVIDIA tutorial serves GLM-5.3 across two DGX Stations
NVIDIA published a tutorial for serving GLM-5.3 locally across two NVIDIA DGX Stations using vLLM. The setup keeps the model weights in GPU memory and serves the model through one API endpoint.
Key takeaways · 4
- 01
The tutorial shows a local GLM-5.3 deployment across two DGX Stations with vLLM.
- 02
NVIDIA says the roughly 464 GB checkpoint is kept in GPU memory across two Stations, with CPU weight offload disabled.
- 03
The setup uses TP=2 and connects the Stations over ConnectX-8 RDMA to serve one model through a single endpoint.
- 04
The reported AIPerf peak was 607.8 output tokens per second at concurrency 16; throughput was 385.0 at concurrency 32.
A two-Station serving recipe
13, 2026, and credits Fatima Fall as its author.[1] It shows how to serve GLM-5.3 locally across two NVIDIA DGX Stations with vLLM.[1] The tutorial uses GLM-5.3 NVFP4, which NVIDIA describes as a quantized version of Z.ai’s mixture-of-experts model.[1] NVIDIA says the model has 753 billion parameters and activates 40 billion parameters per token.[1] Sparse activation reduces computation, but the weights still need to be available because tokens can route to different experts.[1]
Why the model spans two GPUs
NVIDIA puts the checkpoint at about 464 GB, while one GB300 GPU has 252 GB of HBM3e.[1] Its tutorial uses two DGX Stations to keep the model weights in GPU memory, with CPU weight offload disabled.[1] The Stations connect over ConnectX-8 RDMA and serve one model through a single endpoint.[1] NVIDIA sets tensor parallelism to TP=2, splitting model tensors across the two GB300 GPUs.[1] In this arrangement, both Stations compute each request, and Station A returns the answer through an OpenAI-compatible API.[1]
One API server, one worker
The two Stations have different roles in NVIDIA’s deployment. Station A, designated rank 0, hosts the API server.[1] Station B, rank 1, runs only a worker and has no API server.[1] This division means the tutorial’s single endpoint is associated with Station A even though both Stations participate in computation.[1] For teams adapting the configuration, the distinction between the API-serving rank and the worker rank is part of the documented setup, not an indication that each Station exposes a separate endpoint.[1]
Benchmark results and terms
NVIDIA benchmarked the system with AIPerf on the SPEED-Bench Coding dataset.[2] The test recorded peak overall output throughput of 607.8 tokens per second at concurrency 16; at concurrency 32, reported output throughput was 385.0 tokens per second.[1][2] NVIDIA’s test also reported mean request latency of 453.5 seconds.[2] The deployment enables multi-token prediction, FP8 KV-cache storage, and prefix caching.[2] The listed terms are the NVIDIA Open Model Agreement and the original GLM-5.3 license terms.[1] The published figures therefore describe this specific benchmark and configuration.[2]
For infrastructure and engineering teams, the tutorial offers a concrete way to think about distributing a large model across two machines while keeping weights in GPU memory. Its benchmark figures can inform evaluation planning, but teams should compare them against their own workload and configuration.
Why it matters
AI training for your team
kju.ai is built for teams: daily sessions, team analytics, and rollout support for your organization.
How this developed
13 November 2026
NVIDIA tutorial serves GLM-5.3 across two DGX Stations
