Chinese AI darling DeepSeek unveiled an updated version of its cost-and-latency-optimized Flash model on Thursday, with a new version 4.1 that includes\u00a0architectural improvements more significant than you would expect in a point release because the changes might open the door to larger, smarter, and less resource-intensive models.<\/p>\n
At 763 billion parameters, the point release<\/a> is more than 2.5x the size of the model it replaces. In fact, the model is larger than the V3 and R1 models that put DeepSeek on the map back in early 2025.<\/p>\n Despite its ginormous parameter count, DeepSeek V4.1 Flash\u2019s memory requirements aren\u2019t nearly as high as you\u2019d expect for a model of its size. Under the hood, DeepSeek’s devs have made numerous architectural changes that see the LLM become smarter while dramatically reducing the resources necessary to serve it.<\/p>\n DeepSeek has managed this through two key improvements. First, it made significant changes to how the model handles the key-value (KV) caches used to track model state across multiple sessions. These so-called KV caches can be quite memory-hungry, particularly in high-throughput applications like chatbots.<\/p>\n Updates to the model\u2019s various attention mechanisms and the introduction of a new causal encoder-decoder (CED) enabled the devs to improve prompt processing performance while cutting KV cache consumption to between 13 percent and 25 percent of DeepSeek V4 Flash’s requirements.<\/p>\n In other words, the V4.1 release can support four to eight times as many users in the same KV cache footprint.\u00a0<\/p>\n DeepSeek\u2019s technical report<\/a> goes into far greater detail on the architectural changes, but arguably the most interesting change is the introduction of a different kind of model weight.<\/p>\n Of its 763 billion parameters, 196 billion are N-gram parameters that form what DeepSeek’s developers refer to as a \u201cconditional memory module.\u201d The idea is that by decoupling memory from computation, DeepSeek can make its models smarter while also reducing the compute and memory resources required to serve them.<\/p>\n What the heck is an N-gram?<\/p>\n The big idea behind DeepSeek\u2019s V4.1 Flash\u2019s memory module is similar in many respects to Per-Layer Embedding (PLE) tech originally developed by Google<\/a>\u2019s Gemma team. The goal with PLE was to get LLMs to be smart enough to run usefully on devices with constrained bandwidth, memory, and compute \u2013 like smartphones.<\/p>\n DeepSeek\u2019s implementation, first detailed in a January research paper,<\/a> trades PLE embeddings for N-grams. At a high level, N-grams are just groups of tokens. A three-gram would be three tokens in a row, a two-gram would be two, and so forth.\u00a0<\/p>\n As complicated as that might sound, it actually works a bit like word or phrase association.<\/p>\n If you were asked: “Find the parameter of a right triangle when only two sides are known.” For those of you for whom geometry isn’t too distant of a memory, the phrases “use the pythagorean theorem” or “A2\u00a0+ B2\u00a0= C2” or perhaps “the perimeter would be the sum of its sides” might immediately pop to mind.<\/p>\n The N-gram parameters found in models like DeepSeek V4.1 Flash are similar in concept. The weights are a source of implicit knowledge or ingrained memory. Rather than just calculating which combination of tokens have the highest probability of answering the question, as LLMs have traditionally done, the N-gram weights supplement this by quickly surfacing relevant information through a cheap lookup.<\/p>\n This is a gross oversimplification of what’s going on under the hood. In fact, the model isn’t looking up the prompt so much as a series of hashes. These are numbers representing “Find the parameter,” “right triangle,” and so on.\u00a0 Similarly, the contents of the lookup table aren’t raw responses. They’re another mathematical representation, called vectors, which get fed into the inference pipeline.\u00a0<\/p>\n However, the end result is the same: The model can provide smarter, more nuanced answers without the performance penalty normally associated with additional parameters.<\/p>\n What makes N-grams so cheap<\/p>\n The relationship between model size and intelligence is well established at this point. The reason DeepSeek\u2019s n-gram parameters are so interesting is actually related to how the data is accessed.<\/p>\n As a general rule, modern LLMs are autoregressive during decode. That means for every token a chatbot or agent generates, the entirety of the model\u2019s active weights have to be read from memory, making bandwidth the limiting factor.<\/p>\n As we\u2019ve previously discussed<\/a>, architectural changes, like the rise of mixture of expert models or ultra-low precision block-floating point datatypes<\/a>, have helped to minimize this bottleneck. But the N-gram weights found in DeepSeek V4.1 Flash work a bit differently.<\/p>\n They offer a way of effectively increasing the number of parameters available to the model during inference without a proportionate increase in memory pressure.<\/p>\n These N-gram weights are essentially enormous look up tables (LUTs). This makes them fast and cheap to query, since unlike the rest of the model\u2019s active parameters, they don\u2019t need to be read in their entirety from memory each time a token is generated. It\u2019s just a few dozen table lookups per token.<\/p>\n This has a couple of implications for memory access, but the big one is that those n-gram weights don\u2019t have to be crammed into GPU memory to maintain performance. They can be offloaded to system RAM or, possibly even a sufficiently speedy storage array.<\/p>\n So what does that mean in practice? If you look at DeepSeek V4.1 Flash, the model would normally need a minimum of 763 GB of GPU memory to hold the weights at FP8.<\/p>\n However, since those n-gram weights can be offloaded to cheaper system memory, we can get away with around 567 GB of GPU memory.\u00a0<\/p>\n We emphasize the minimum here, because in production those numbers are going to be substantially higher since we also need to take into account key-value caches, which scale with context length and concurrent users.\u00a0<\/p>\n During inference, those N-gram weights supplement the eight billion active parameters DeepSeek uses to\u00a0process a prompt, in theory increasing the accuracy and quality of the output in the process. <\/p>\n It\u2019s important to note at this point that the n-gram weights don\u2019t actually increase the active parameter count. Instead, they function more like an oddly specific encyclopedia that almost instantly opens relevant pages as the model processes prompts.<\/p>\n The future of open LLMs<\/p>\n While DeepSeek\u2019s latest model may have one of the largest pools of N-gram parameters yet, it\u2019s not the only model developer betting on tech to make deploying larger models more efficient.\u00a0<\/p>\n As mentioned earlier, Google is already employing a similar approach using PLE to offload less bandwidth-sensitive weights to local storage. So far, it’s only been applied to tiny models \u2014 at least that we know of (it’s not like Google is particularly transparent about its proprietary models). <\/p>\n