NVIDIA has shown how to run GLM-5.3 — a 753-billion-parameter model from Z.ai — locally, on two DGX Stations, without relying on the cloud. In a tutorial published on November 13 on the official developer blog, the company splits the model across two DGX Stations with GB300 GPUs and serves it through a single endpoint compatible with OpenAI's API, using vLLM.
A model that does not fit on one machine
GLM-5.3 NVFP4 is a quantized version of Z.ai's mixture-of-experts (MoE) model, with 753 billion parameters and 40 billion activated per token. The checkpoint is about 464 GB — more than the 252 GB of HBM3e memory on a single GB300 GPU. That is why the tutorial uses two Stations: each GPU holds about 218 GiB of the weights, with no CPU weight offload.
The two DGX Stations are linked over ConnectX-8 RDMA, and tensor parallelism (TP=2) splits the weights and the computation across the two GB300 GPUs. Station A hosts the API and Station B runs only a worker: neither side can answer on its own — every token requires the two machines to exchange partial results over the RDMA link.
607.8 tokens per second — up to a point
To measure performance, NVIDIA used AIPerf with the 80 Coding conversations from SPEED-Bench, with prefix caching on, an FP8 KV cache and four-token-per-step speculative decoding (MTP), at a maximum context of 98,304 tokens. The peak was 607.8 tokens per second at 16 concurrent requests, with a time to first token of 482.9 ms. With 1 request the total was 107.4 tok/s; with 8, 386.0; and with 32 it fell to 385.0 — with a mean request latency of 453.5 seconds. Per-user throughput also falls with concurrency: from 115.3 tok/s with one request to 13.0 tok/s with 32.
The numbers are NVIDIA's own, measured on its hardware, on a single coding dataset and counting reasoning tokens — there is no independent replication and no comparison with a cloud endpoint. The blog also warns that the setup exposes the API without authentication on the local network, and recommends a firewall, authentication and TLS before widening access.
Why it matters
A frontier-scale model running behind a local API, on desk hardware, opens up work that is hard to do on a hosted endpoint: running inference on sensitive or proprietary data, prototyping agents locally and iterating on prompts with no per-token cost. GLM-5.3 already appeared on Geekiki in October, when Apollo Research found in it the first spontaneous reasoning by a model about its own training — now the news is the practical recipe for serving it at home, with performance numbers measured by NVIDIA.
Image credit: diagram from NVIDIA on the official developer blog, illustrating two DGX Stations serving a single model.
Sources
- NVIDIA Developer Blog — Fatima Fall, When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM (Nov 13, 2026)
- AI Weekly — NVIDIA devblog: GLM-5.3 hits 607.8 tok/s on two DGX Stations (Nov 14, 2026)
- Bytewoops — GLM-5.3 fits on two DGX Stations, and NVIDIA’s own test shows the concurrency cliff
This article was produced with the assistance of artificial intelligence, based on public posts and internet sources, and reviewed editorially. Consult the original sources for more details.
By Muse
