AI chip and infrastructure orders continue to flood inBut memory bandwidth could be a limiting factoring in putting all that new compute power to efficient useTools can help improve available memory bandwidth, but there’s no cure all yet<\/p>\n
Orders for AI gear have been pouring in and not just at chip giant Nvidia either. AMD bagged deals with Meta, Microsoft, OpenAI and Oracle. Cisco already has<\/a> $2 billion worth of AI orders this year. And just this week Broadcom revealed it landed<\/a> a $10 billion contract for AI racks based on its XPU chips. But the AI boom is about to hit a serious speed bump.<\/p>\n We\u2019ve already written about the power, cooling and networking challenges AI data centers face. But there\u2019s another major problem that could hinder AI growth: memory.<\/p>\n While GPUs get all the love, AI requires more than pure processing power. As we recently pointed out<\/a>, CPUs will play a vital role in orchestrating workloads and data pipelines for AI applications. But memory bandwidth allows data to flow, and right now there\u2019s not nearly enough of it.<\/p>\n \u201cMemory is very much a limiting factor in AI scale-out and performance,\u201d J. Gold Associates Founder Jack Gold explained. \u201cGPUs are often restricted in performance by the need to connect to external memory over interconnects that slow things down. So, anything that can bring memory closer\/faster to the GPU has a big performance improvement.\u201d<\/p>\n How did this happen?<\/p>\n JB Baker, VP of Products at ScaleFlux, told Fierce that while both processor and memory capabilities have grown exponentially in recent years, the latter hasn\u2019t grown at the same multiplier as the former.<\/p>\n The result is a gap between how many calculations a chip can process in one second and how much memory bandwidth is available to send data to the chips to be processed. In other words, memory has become a bottleneck. (There’s a nifty little chart you can check out here<\/a>, and an IEEE paper on the Memory Wall here<\/a> if you’re feeling extra nerdy.)<\/p>\n Baker said it\u2019s a bit like having a sprawling crop field next to a massive lake full of water but only having a tiny garden hose to get the water to the field.\u00a0<\/p>\n \u201cThere\u2019s a lot of potential compute capacity that is going unutilized and at the same time it\u2019s burning power,\u201d he said. \u201cSo, it\u2019s not only that I lost out on things that those processors could have done, but they burn power sitting idle.\u201d<\/p>\n Gold noted the issue isn\u2019t unique to GPUs \u2013 CPUs have run up against this wall as well. But the issue is coming to a head in the AI era, particularly in light of power constraints and a desire to massively scale AI deployments.<\/p>\n Can it be fixed?<\/p>\n Baker said right now there\u2019s no cure-all solution, but things are moving in the right direction.<\/p>\n He pointed to compute express link (CXL) technology and solid state drive (SSD) advancements as tools that can help scale memory bandwidth. (Of course, both are products ScaleFlux offers, but others are working on the problem as well.<\/p>\n