{"id":230767,"date":"2025-10-17T05:20:20","date_gmt":"2025-10-17T05:20:20","guid":{"rendered":"https:\/\/www.newsbeep.com\/us\/230767\/"},"modified":"2025-10-17T05:20:20","modified_gmt":"2025-10-17T05:20:20","slug":"two-nvidia-dgx-spark-systems-fused-with-m3-ultra-mac-studio-to-deliver-2-8x-gain-in-ai-benchmarks-exo-labs-demonstrates-disaggregated-ai-inference-serving","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us\/230767\/","title":{"rendered":"Two Nvidia DGX Spark systems fused with M3 Ultra Mac Studio to deliver 2.8x gain in AI benchmarks \u2014 EXO Labs demonstrates disaggregated AI inference serving"},"content":{"rendered":"

EXO Labs’ primary project is EXO, an open-source framework for running large language models (LLMs) efficiently across mixed hardware setups. Rather than treating inference as a task bound to a single GPU or accelerator, EXO automatically spreads workloads across whatever devices you have\u2014turning a cluster of desktops, laptops, workstations, servers, tablets, or even smartphones into a cooperative AI mesh. EXO’s newest demo combines two of NVIDIA’s DGX Spark systems<\/a> with Apple’s M3 Ultra\u2013powered Mac Studio to make use of the disparate strengths of each machine: Spark has more raw compute muscle, while the Mac Studio can move data around much faster. EXO 1.0, currently in early access, blends the two into a single inference pipeline, and it apparently works shockingly well<\/a>.<\/p>\n

\n

\"Two
\n<\/p>\n

Two NVIDIA DGX Spark systems linked up with a Mac Studio using 10-Gigabit Ethernet for disaggregated AI inference. (Image credit: EXO Labs)<\/p>\n

To understand how, you need to know that LLM inference has two distinct phases: first, the prefill stage, when the model reads and processes the user’s prompt. This part is compute-bound, meaning it benefits from powerful GPUs like the Blackwell part in the DGX Spark. The decode phase follows, generating tokens one by one. That step is heavily bandwidth-bound, making the M3 Ultra’s super-wide memory bus ideal. EXO’s trick is to divide those phases between machines, streaming the model’s internal data (called the KV cache) layer by layer so the two systems can work simultaneously instead of waiting on each other.<\/p>\n

\n

\"A\n<\/p>\n

The performance results from EXO Labs’ blog post<\/a>. (Image credit: EXO Labs)<\/p>\n

In EXO’s benchmark with Meta’s Llama-3.1 8B model, the hybrid setup achieved nearly a threefold speedup over the Mac Studio alone \u2014 matching the DGX Spark’s prefill speed while keeping the M3 Ultra’s quick generation time. The result is a 2.8\u00d7 overall gain, and that was with an 8K-token prompt on a relatively modest 8B model. Longer prompts or larger models<\/a> should see even greater gains.<\/p>\n

You may like<\/p>\n

This kind of “disaggregated inference” isn’t exactly a novelty, but it’s still very clever. It hints at a future where AI performance scales not by buying one massive accelerator, but instead by more intelligently orchestrating the hardware you already have. NVIDIA seems to agree: its upcoming Rubin CPX platform<\/a> will use compute-dense Rubin CPX processors for the context-building prefill stage while standard Rubin chips with huge HBM3e memory bandwidth handle the decode stage\u2014he same principle EXO is already demonstrating on off-the-shelf hardware.<\/p>\n

\n

\"A\n<\/p>\n

NVIDIA’s Dynamo framework<\/a> serves a similar purpose, but without the automatic device subscription feature of EXO. (Image credit: NVIDIA)<\/p>\n

EXO’s early-access release is still in the experimental stages. The current open-source version<\/a> (0.0.15-alpha) dates back to March 2025, and the full 1.0 build \u2014 with automated scheduling, KV streaming, and heterogeneous optimizations \u2014 isn’t public yet. It’s not plug-and-play software for consumers, at least not yet; for now, it’s a research-grade tool proving that disaggregated inference can deliver real gains.<\/p>\n

Even so, it’s an exciting proof of concept. By making intelligent use of mixed hardware, EXO shows that high-performance AI doesn’t have to be monopolized by data centers<\/a>. It’s enough to make you wonder about the potential of the devices sitting around your office.<\/p>\n

<\/p>\n

\n

\"Google\n<\/p>\n

<\/a><\/p>\n

Follow Tom’s Hardware on Google News<\/a>, or add us as a preferred source<\/a>, to get our latest news, analysis, & reviews in your feeds.<\/p>\n

Get Tom’s Hardware’s best news and in-depth reviews, straight to your inbox.<\/p>\n

<\/p>\n","protected":false},"excerpt":{"rendered":"EXO Labs’ primary project is EXO, an open-source framework for running large language models (LLMs) efficiently across mixed…\n","protected":false},"author":2,"featured_media":230768,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,74],"class_list":{"0":"post-230767","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/230767","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=230767"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/230767\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/230768"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=230767"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=230767"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=230767"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}