![\"Line](\"https:\/\/www.newsbeep.com\/il\/wp-content\/uploads\/2026\/04\/MLPerf-9x.webp.webp\")
<\/p>\n
\t\tFigure 1. NVIDIA Delivers 9x More Cumulative MLPerf Training and Inference wins<\/p>\n
This post takes a closer look at the latest benchmark updates, the industry-leading performance achieved on the NVIDIA platform, and the full-stack engineering that makes it possible.\u00a0<\/p>\n
New benchmarks, new performance records<\/a><\/p>\n The MLPerf Inference benchmark suite is routinely updated to ensure that it reflects models, modalities, use cases, and deployment scenarios that matter to the community. Only the NVIDIA platform submitted results on all newly added models and scenarios this round, and delivered the highest performance across all of them.<\/p>\n This round of MLPerf Inference added several new tests, including:<\/p>\n DeepSeek-R1 Interactive: Following the addition of DeepSeek-R1 reasoning LLM based on a sparse mixture-of-experts (MoE) architecture in MLPerf Inference v5.1, MLCommons added a new Interactive scenario with 5x faster minimum token rate and 1.3x shorter time to first token compared to the server scenario, representing higher-interactivity deployments<\/p>\n Qwen3-VL-235B-A22B<\/a>: Vision-language model with a total of 235B parameters. This represents the first multi-modal model in the MLPerf Inference suite. Two scenarios are tested: Offline and Server.\u00a0<\/p>\n GPT-OSS-120B<\/a>: 120B-parameter MoE reasoning LLM, developed by OpenAI. This benchmark includes three scenarios: Offline, Server, and Interactive<\/p>\n WAN-2.2-T2V-A14B<\/a>: 4B-parameter text-to-video generative AI model. Two scenarios tested: single-stream, which measures the latency to process a single video generation request, and offline, which measures the number of samples processed per second in a batch-processing scenario.\u00a0<\/p>\n DLRMv3 <\/a>\u2013 A generative recommendation benchmark that replaces the DLRM-DCNv2 test. It uses a transformer-based architecture that increases model size and compute intensity compared to the prior benchmark. It tests offline and server scenarios.\u00a0<\/p>\n BenchmarkDeepSeek-R1GPT-OSS-120BQwen3-VLWan 2.2DLRMv3Offline2,494,310 tokens\/sec*1,046,150 tokens\/sec79 samples\/sec0.059 samples\/sec104,637 samples\/secServer1,555,110 tokens\/sec*1,096,770 tokens\/sec68 queries\/sec21 secs**(Single Stream)99,997 queries\/secInteractive250,634 tokens\/sec677,199 tokens\/sec*********Table 1. NVIDIA platform throughput on newly added workloads and scenarios in MLPerf Inference v6.0\u00a0<\/p>\n * Not a new scenario in MLPerf Inference v6.0 MLPerf Inference v6.0, Closed Division. Results retrieved from www.mlcommons.org on April 1, 2026. NVIDIA platform results from the following entries: 6.0-0039, 6.0-0073, 6.0-0075, 6.0-0076, 6.0-0078, 6.0-0081, 6.0-0094. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.<\/p>\n \t\tFigure 2. NVIDIA achieves 2.7x performance gain and 2.5M token\/s on DeepSeek-R1<\/p>\n NVIDIA TensorRT-LLM software updates unlock up to 2.7X performance gains on the same Blackwell Ultra GPUs<\/a><\/p>\n NVIDIA continually optimizes the performance of its software stack to increase delivered token throughput from existing platforms. This delivers reductions in token production cost and enables AI factory operators to serve more users to generate more revenue with a given infrastructure footprint.<\/p>\n The additional performance also provides headroom to run future AI models and serve existing models in demanding scenarios, such as higher token rates and longer contexts. This continual improvement makes it possible for NVIDIA GPUs introduced years ago to remain productive, at high utilization rates, in the cloud.\u00a0<\/p>\n This round, NVIDIA GB300 NVL72\u2014launched last year\u2014delivered up to 2.7x higher token throughput compared to its debut submissions just six months ago on the server scenario of the DeepSeek-R1 benchmark1. This means 2.7x more tokens from the same GB300 NVL72-based infrastructure and power footprint, reducing the cost to manufacture each token by more than 60%. This speedup, achieved by NVIDIA partner Nebius<\/a>, showcases a core advantage of the NVIDIA platform: an open, expansive ecosystem where customers and partners can uniquely optimize and innovate on top of our software stack.\u00a0<\/p>\n 1MLPerf Inference v5.1 and v6.0, Closed Division. Results retrieved from www.mlcommons.org<\/a> on April 1, 2026. NVIDIA platform results from the following entries: 5.1-0072,\u00a0 6.0-0081. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org<\/a> for more information.<\/p>\n Powering the DeepSeek R1 performance improvements in the server and offline scenarios were several software enhancements, including:<\/p>\n Faster kernels\u2014this included a combination of higher-performance kernels and the use of fewer kernels because of kernel fusions.<\/p>\n Optimized Attention Data Parallel\u2014Better balancing of context requests between different ranks, enabling significant speedups in end-to-end performance.<\/p>\n The latest features of the open source NVIDIA TensorRT-LLM inference serving software and the NVIDIA Dynamo<\/a> open source distributed inference serving framework were used to support the newly added and more challenging DeepSeek-R1 Interactive scenario. This includes:<\/p>\n Disaggregated serving: This capability in Dynamo separates and individually optimizes the configurations of each inference phase (prefill and decode), respectively, enabling optimal overall throughput.\u00a0<\/p>\n Wide Expert Parallel (WideEP): For higher-interactivity scenarios, execution time for MoE models is bound by expert weight load time. By splitting, or sharding, the experts across multiple GPUs across NVL72 nodes, this bottleneck is reduced, improving end-to-end performance.<\/p>\n Multi-Token Prediction (MTP): At higher interactivity levels, batch sizes are smaller, and performance is dominated by how quickly weights can load into memory, leaving compute performance underutilized. By applying compute otherwise that goes unutilized to predict and verify additional tokens in parallel (up to three in this implementation), throughput at high interactivity is increased.\u00a0<\/p>\n KV-aware routing: This capability of Dynamo routes inference requests by evaluating their compute costs across different workers.<\/p>\n NVIDIA was the first and only platform to submit DeepSeek-R1 results on MLPerf Inference when the benchmark debuted last year. This round, NVIDIA not only increased performance on returning scenarios for DeepSeek-R1 but \u200cwas once again the only platform to submit on the newly added interactive scenario.<\/p>\n And even on Llama 3.1 405B\u2014a very large, dense LLM launched almost two years ago\u2014 GB300 NVL72 performance increased by 1.5x in the server scenario.<\/p>\n BenchmarkGB300 NVL72\u00a0v5.1GB300 NVL72v6.0SpeedupDeepSeek-R1(Server)2,907 tokens\/sec\/gpu8,064 tokens\/sec\/gpu2.77xDeepSeek-R1(Offline)5,842 tokens\/sec\/gpu9,821 tokens\/sec\/gpu1.68xLlama 3.1 405B(Server)170 tokens\/sec\/gpu259 tokens\/sec\/gpu1.52xLlama 3.1 405B(Offline)224 tokens\/sec\/gpu271 tokens\/sec\/gpu1.21xTable 2. Performance improvements, normalized on a per-GPU basis, on DeepSeek-R1 and Llama 3.1 405B server and offline scenarios in v6.0 compared to v5.1<\/p>\n MLPerf Inference v5.1 and v6.0, Closed Division. Results retrieved from www.mlcommons.org on April 1, 2026. NVIDIA platform results from the following entries: 5.1-0072, 6.0-0017, 6.0-0078, 6.0-0082. Per chip performance is derived by dividing total throughput by the number of reported chips. Per-chip performance is not a primary metric of MLPerf Inference v5.1 or v6.0. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.<\/p>\n Additionally, NVIDIA submissions on the newly added multimodal, video generation, and recommendation benchmarks were powered by open source software frameworks optimized for the NVIDIA platform. The Qwen3-VL vision-language submission used the vLLM open source framework, showing how the community is rapidly building advanced multimodal optimizations to accelerate image-heavy inference workloads on the latest GPUs like NVIDIA Blackwell Ultra. The WAN-2.2 text-to-video submission used the TensorRT-LLM VisualGen<\/a>, which accelerates diffusion-based video generation pipelines on NVIDIA GPUs.\u00a0<\/p>\n For DLRMv3, the submission was built on two open-source projects: the NVIDIA recsys-example<\/a> for high-performance transformer-based recommendation inference, and NV Embedding Cache<\/a> for GPU-accelerated embedding table lookups. Both were critical to achieving record throughput on this more demanding generative recommendation benchmark.<\/p>\n Through extensive and ongoing engineering, NVIDIA is continually increasing performance on existing hardware on existing models, as evidenced by these results. At the same time, NVIDIA collaborates closely with model builders and open source inference frameworks to ensure that the latest models run on the NVIDIA platform on the day of launch.\u00a0<\/p>\n Scale-out inference with NVIDIA Quantum-X800 InfiniBand platform enables millions of tokens per second<\/a><\/p>\n NVIDIA also set new throughput records at scale on the DeepSeek-R1 model in the offline and server scenarios by submitting results using four GB300 NVL72 systems interconnected with NVIDIA Quantum-X800 InfiniBand<\/a> scale-out networking.\u00a0<\/p>\n DeepSeek-R1 | 4x GB300 NVL72Tokens\/SecondOffline2,494,310Server1,555,110Table 3. DeepSeek-R1 throughput on four GB300 NVL72 systems scaled up with NVLink and scaled out with NVIDIA Quantum-X800 InfiniBand<\/p>\n MLPerf Inference v6.0, Closed Division. Results retrieved from www.mlcommons.org on April 1, 2026. NVIDIA platform results from the following entries: 6.0-0076. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.<\/p>\n With 288 Blackwell Ultra GPUs\u2014the largest scale ever submitted to any benchmark in MLPerf Inference\u2014submissions set new system-level throughput records, enabling millions of tokens processed per second.\u00a0<\/p>\n
** Wan 2.2 features a single stream scenario, which measures end-to-end request latency, instead of a server scenario. Lower is better.
*** Not tested in MLPerf Inference v6.0<\/p>\n![\"Bar](\"https:\/\/www.newsbeep.com\/il\/wp-content\/uploads\/2026\/04\/MLPerf-2.7x.webp.webp\")
<\/p>\n