{"id":98375,"date":"2025-08-27T00:15:17","date_gmt":"2025-08-27T00:15:17","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/98375\/"},"modified":"2025-08-27T00:15:17","modified_gmt":"2025-08-27T00:15:17","slug":"d-matrix-corsair-in-memory-computing-for-ai-inference-at-hot-chips-2025","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/98375\/","title":{"rendered":"d-Matrix Corsair In-Memory Computing For AI Inference at Hot Chips 2025"},"content":{"rendered":"

\"Card<\/a>Card Level Scaleup: 16 Chiplet Hierarchical All-to-All<\/p>\n

The second machine learning presentation of the afternoon comes from d-Matrix. The company specializes in hardware for AI inference, and as of late has been tackling the matter of how to improve inference performance by using in-memory computing. Along those lines, the company is presenting its Corsair in-memory computing chiplet architecture at Hot Chips. As a quick note: we covered\u00a0d-Matrix Pavehawk Brings 3DIMC to Challenge HBM for AI Inference<\/a> a few days ago.<\/p>\n

Not to be confused with that Corsair<\/a>, d-Matrix claims that Corsair is the most efficient inference platform on the market, thanks to its combination of in-memory computing and low-latency interconnects.<\/p>\n

\"\u201cRethinking\u201d\u201cRethinking\u201d AI Inference
\n\"LLMLLM Token-generation is Memory Bound<\/p>\n

Each token in an LLM is memory bound. All the weights need to be read. Batching allows for these weight fetches to be amortized.<\/p>\n

d-Matrix\u2019s goal is to reach saturation at moderate batch sizes in order to hit specific latency targets.<\/p>\n

\"VoiceVoice is latency-critical and even more Memory Bound<\/p>\n

Real-time voice requires very low latency. Making it a good target for d-Matrix\u2019s technology.<\/p>\n

\"AIAI Agents: SLM & Rise of Inference-Time Compute<\/p>\n

AI agents fall into the same boat. Multiple small models being executed to accomplish the desired task.<\/p>\n

\"d-Matrixd-Matrix Corsair: Chiplet-based Inference Acceleration Platform<\/p>\n

And here is Corair, d-Matrix\u2019s accelerator. Two chips, each with 4 chiplets. Built on TSMC 6nm. 2GB of SRAM between all of the chiplets. This is a PCIe 5.0 x16 card, so it can be easily added to standard servers.<\/p>\n

Meanwhile at the top of the card are bridge connectors to tie together multiple cards.<\/p>\n

Each chiplet interfaces with LPDDR5X, with 256GB of L5X per card.<\/p>\n

\"CorsairCorsair Chiplet<\/p>\n

And here is how the chiplets are organized into slices. Around the edge are LPDDR and D2D connections. As well as 16 lanes of PCIe.<\/p>\n

\"CardCard Level Scaleup: 16 Chiplet Hierarchical All-to-All<\/p>\n

Two cards can be passively bridged together, making for a 16 chiplet cluster, with all-to-all connectivity.<\/p>\n

\"SystemSystem and Scaleup Architecture<\/p>\n

8 cards in turn can go into a standard server, such as a Supermicro X14. In this example there are also 4 NIC cards to offer scale-up capabilities.<\/p>\n

\"CorsairCorsair Key Pillars \u2013 Low Latency, Batched Throughput<\/p>\n

Corsair was built for low-latency batch throughput inference.<\/p>\n

They support block floating point number formats. Energy efficiency is 38 TOPS per Watt.<\/p>\n

\"CorsairCorsair Chiplet Built Using Modular Hardware Blocks<\/p>\n

The dispatch engine within each chiplet is based on RISC-V. 1 chiplet is split up into 4 quads. About 1TB\/sec of D2D bandwidth.<\/p>\n

\"EnergyEnergy Efficient DIMC Architecture<\/p>\n

Diving deeper, the matrix multiplier inside Corsair can perform a 64\u00d764 matmul with INT8. Or 64\u00d7128 with INT4.<\/p>\n

\"CorsairCorsair supports 5x Weight Compression<\/p>\n

Corsair also supports FP formats with scale factors. As well as structured sparsity \u2013 though it\u2019s only used for compression. Overall, it gets d-Matrix to 5x compression.<\/p>\n

\"CoreCore Architecture<\/p>\n

All 8 matrix units can be tied together.<\/p>\n

\"DataflowDataflow with Block Floating-Point Numerical Formats<\/p>\n

Dataflow. Accumulated on the fly and then converted to the desired output format.<\/p>\n

\"MemoryMemory System: Global Memory, Stash, and LPDDR<\/p>\n

As for memory, there is a stash memory that feeds the cores. Each stash is 6MB. There are 2 LPDDR channels per chiplet.<\/p>\n

\"ScalingScaling Challenges for Large-Model Inference<\/p>\n

When you have high memory bandwidth, the collective latency becomes increasingly critical.<\/p>\n

\"CorsairCorsair Scaleup \u2013 Hardware-Software Codesign<\/p>\n

So in order to do a 16 chiplet all-to-all connection, d-Matrix got latency down to 115ns D2D. Even going through PCIe switches, they can still hold latency to 650ns.<\/p>\n

\"PackagePackage Level Scaleup: 4 Chiplet All-to-All<\/p>\n

Another shot of Corsair chiplets on an organic package.<\/p>\n

\"TransparentTransparent NIC \u2013 Ethernet Scale-Out<\/p>\n

And here is the NIC that d-Matrix uses for scale-out fabrics. 2us of latency.<\/p>\n

\"RackRack Level Scale-Out: Multi-node and Multi-Rack<\/p>\n

Using this, d-Matrix can rack and stack many servers.<\/p>\n

\"AviatorAviator Software: Easy to use and optimized for Corsair
\n\"AviatorAviator Software: Codesigned for LLM Acceleration<\/p>\n

And no inference accelerator would be complete without a matching software stack to enable the hardware and its features.<\/p>\n

\"PowerPower Efficiency (TOPS\/W)<\/p>\n

And here\u2019s a look at power consumption. 275W @ 800MHz. Meanwhile 1.2GHz chugs 550W. Higher clockspeeds are worse for overall efficiency, but not immensely so.<\/p>\n

\"PerformancePerformance and Flexibility for Use Cases<\/p>\n

And here are some Llama3 performance figures. The time per output token is just 2ms even for the larger Llama3-70B.<\/p>\n

\"StackingStacking of logic on DRAM interposer in 3D<\/p>\n

Underneath the chip, d-Matrix uses a silicon interposer with capacitor for power reliabiltiy reasons. d-Matrix goes one further and 3D stacks DRAM beneath their Corsair chiplets, keeping the local memory very, very close.<\/p>\n

\"3D3D DRAM Test Vehicle<\/p>\n

And they have a prototype 3D DRAM test vehicle that\u2019s been built. 36 micron D2D stacking. The logic die sits on top, while the DRAM sits underneath.<\/p>\n

How does d-Matrix make stacked DRAM + logic work? Keep the heat density to under 0.3W\/mm2, which keeps from heating up the DRAM too much.<\/p>\n","protected":false},"excerpt":{"rendered":"Card Level Scaleup: 16 Chiplet Hierarchical All-to-All The second machine learning presentation of the afternoon comes from d-Matrix.…\n","protected":false},"author":2,"featured_media":98376,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[64,63,257,72057,72058,105],"class_list":{"0":"post-98375","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-computing","8":"tag-au","9":"tag-australia","10":"tag-computing","11":"tag-d-matrix","12":"tag-hot-chips-2025","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/98375","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=98375"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/98375\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/98376"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=98375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=98375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=98375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}