{"id":654915,"date":"2026-05-07T06:09:11","date_gmt":"2026-05-07T06:09:11","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/654915\/"},"modified":"2026-05-07T06:09:11","modified_gmt":"2026-05-07T06:09:11","slug":"googles-gemma-4-ai-models-get-3x-speed-boost-by-predicting-future-tokens","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/654915\/","title":{"rendered":"Google&#8217;s Gemma 4 AI models get 3x speed boost by predicting future tokens"},"content":{"rendered":"<p>Google launched its <a href=\"https:\/\/arstechnica.com\/ai\/2026\/04\/google-announces-gemma-4-open-ai-models-switches-to-apache-2-0-license\/\" rel=\"nofollow noopener\" target=\"_blank\">Gemma 4 open models<\/a> this spring, promising a new level of power and performance for local AI. Google\u2019s take on edge AI could be getting even faster already with the release of <a href=\"https:\/\/blog.google\/innovation-and-ai\/technology\/developers-tools\/multi-token-prediction-gemma-4\/\" rel=\"nofollow noopener\" target=\"_blank\">Multi-Token Prediction<\/a> (MTP) drafters for Gemma. Google says these experimental models leverage a form of speculative decoding to take a guess at future tokens, which can speed up generation compared to the way models generate tokens on their own.<\/p>\n<p>The latest Gemma models are built on the same underlying technology that powers Google\u2019s frontier Gemini AI, but they\u2019re tuned to run locally. Gemini is optimized to run on Google\u2019s <a href=\"https:\/\/arstechnica.com\/ai\/2026\/04\/google-unveils-two-new-tpus-designed-for-the-agentic-era\/\" rel=\"nofollow noopener\" target=\"_blank\">custom TPU chips<\/a>, which operate in enormous clusters with super-fast interconnects and memory. A single high-power AI accelerator can run the largest Gemma 4 model at full precision, and quantizing will let it run on a consumer GPU.<\/p>\n<p>Gemma allows users to tinker with AI on their hardware rather than sharing all their data with a cloud AI system from Google or someone else. Google also changed the license for Gemma 4 to Apache 2.0, which is much more permissive than the custom Gemma license Google employed for previous releases. However, there are inherent limitations in the hardware most people have to run local AI models. That\u2019s where MTP comes in.<\/p>\n<p>LLMs like Gemma (or Gemini) generate tokens autoregressively\u2014that is, they produce one token at a time based on the previous token. Each one takes just as much computing work as the last one, regardless of whether the token is just a filler word in an output or a key piece of information in a complex logical problem.<\/p>\n<p>The problem with rolling your own AI is that your system memory probably isn\u2019t very fast compared to the high bandwidth memory (HBM) used in enterprise hardware. As a result, the processor spends a lot of time moving parameters from VRAM to compute units for each token, and compute cycles are going unused during this process.<\/p>\n<p>\n      Gemma 4 26B on a NVIDIA RTX PRO 6000. Standard Inference (left) vs. MTP Drafter (right) in tokens per second. Same output quality, half the wait time.\n    <\/p>\n<p>\n      Gemma 4 26B on a NVIDIA RTX PRO 6000. Standard Inference (left) vs. MTP Drafter (right) in tokens per second. Same output quality, half the wait time.<\/p>\n<p>MTP uses that time to bypass the heavy model and generate speculative tokens with the lightweight drafter. While the draft models are smaller (just 74 million parameters in Gemma 4 E2B), they\u2019re also optimized in several ways to speed up speculative token generation. For example, the drafter shares the key value cache (essentially the LLM\u2019s active memory) so it doesn\u2019t need to recalculate context the main model has already worked out. The E2B and E4B drafters also use a sparse decoding technique to narrow down clusters of likely tokens.<\/p>\n","protected":false},"excerpt":{"rendered":"Google launched its Gemma 4 open models this spring, promising a new level of power and performance for&hellip;\n","protected":false},"author":2,"featured_media":654916,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-654915","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/654915","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=654915"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/654915\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/654916"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=654915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=654915"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=654915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}