Meanwhile, for longer context lengths, the company says that its platform is able to sustain more than 450 tokens a second.\u00a0<\/p>\n
![\"Benchmark](\"https:\/\/www.newsbeep.com\/il\/wp-content\/uploads\/2026\/07\/5268748.webp\")
<\/p>\n
\n At short context lengths (10,000 tokens) Artificial Analysis found SambaNova’s heterogeneous compute platform, which combined four H200 GPUs with 16 of its SN50 RDUs was able to achieve decode speed of 763 tok\/s in MiniMax M2.7.
\n Artificial Analysis\n <\/p>\n
This feat was accomplished by combining Nvidia GPUs with SambaNova Reconfigurable Dataflow Units (RDUs) to form a heterogeneous inference platform.<\/p>\n
Specifically, the computationally intensive prefill phase of the inference pipeline, during which prompts are processed and key value caches are generated, was handled by four Nvidia H200 GPUs.<\/p>\n
Meanwhile, memory-bandwidth-bound decode operations, where output tokens are generated, were done on a single SambaNova rack containing 16 SN50 accelerators.<\/p>\n
Disaggregating prefill from decode has become a key lever for reducing token costs for long-running AI agents, like code assistants. Nvidia initially demonstrated<\/a> this with its NVL72 rack systems, by varying the ratio of GPUs used for prefill versus decode. The company further disaggregated this with its Groq-based LPX racks<\/a> revealed at GTC this spring. Since then, just about everyone from AMD to AWS and Cerebras has announced some kind of disaggregated or heterogeneous inference platform using one or more accelerators.<\/p>\n With SambaNova’s latest performance figures, the startup hopes to demonstrate how customers can breathe new life into their aging GPU fleets by using its systems as decode accelerators. And because its systems are air-cooled, they can be deployed in existing datacenters \u2014 something that can’t be said of Nvidia’s latest generation of Rubin GPUs<\/a>, which absolutely need liquid cooling.<\/p>\n SambaNova plans to show off even more powerful inference configs, with 128 and eventually 256 accelerators to demonstrate its ability to maintain high token generation rates at high throughput. As we\u2019ve previously explored<\/a>, this is something that GPUs alone have historically struggled with and one of the key drivers behind Nvidia\u2019s Groq acquihire<\/a> late last year.<\/p>\n The results come just a month after SambaNova and Intel announced Vector Core Compute would be among the first to deploy the combined GPU + RDU offering with TogetherAI as their first large-scale customer.<\/p>\n