NVIDIA published latest inference cost benchmarks for its Blackwell Ultra GB300 NVL72 platform on the NVIDIA developer blog and at GTC 2026, claiming $0.123 per million tokens at 116 tokens-per-second per user interactivity using TensorRT-LLM and Dynamo software. This 35x cost reduction versus Hopper (H100) sets a new floor for low-latency production agentic AI workloads and establishes per-token cost—not raw FLOPS or power draw alone—as the only metric that matters for inference TCO.
The GB300 NVL72 achieves this through full-stack codesign: 72 Blackwell Ultra GPUs, 288GB HBM3e per GPU, and a unified 130 TB/s NVLink fabric that acts as a single rack-scale system. For mixture-of-experts models like DeepSeek-R1 and DeepSeek V4 Pro, the platform delivers 50x higher throughput per megawatt versus Hopper. Continuous software optimization is a key multiplier: software tweaks alone have yielded 4x performance improvements on GB200 over three months, with further gains as TensorRT-LLM kernels are optimized for open-source and closed-model inference.
NVIDIA frames the metric as 'revenue per megawatt': a $5M investment in a GB200 NVL72 system can generate $75M in token revenue, implying a 15x ROI. This economics-first positioning is aimed at inference service providers (CoreWeave, DeepInfra, Together AI, Fireworks AI) who operationalize token costs immediately. For a power-constrained world, the implication is that hardware generations now improve on a single variable: how many high-value tokens per watt.
For infrastructure teams, this codifies what many suspected: the Blackwell generation ended the GPU-scarcity era and began the token-cost optimization race. Inference providers who have not yet optimized software stacks (DynoSim, quantization strategies, expert routing) on Blackwell are leaving real margin on the table. The threshold for entry-level agentic AI inference is now the ability to run production-grade MoE models at <$0.15/MT margins.