A team of researchers from Intel Labs and the Israel\u2019s Weizmann Institute of Science has presented a new approach they say could significantly boost the performance of large language models (LLMs), the generative AI systems behind tools like ChatGPT and other chatbots, while helping make AI development more open and cost-efficient.<\/p>\n
The technique, presented at this year\u2019s International Conference on Machine Learning (ICML) in Vancouver, focuses on a method known as speculative decoding, an inference optimization that pairs a smaller, faster \u201cdraft\u201d model with a larger, more precise one. The draft model quickly generates a probable chunk of text in response to a user\u2019s prompt, which the larger model then checks for accuracy. By offloading much of the heavy computational work to the faster model, the combined system can deliver responses up to 2.8 times faster than conventional approaches, according to the research team.<\/p>\n
The advance addresses one of the core technical challenges in generative AI: the large computing demands required to run powerful models at scale. \u201cWe have solved a core inefficiency in generative AI,\u201d said Oren Pereg, a senior researcher in Intel Labs\u2019 Natural Language Processing Group. \u201cOur research shows how to turn speculative acceleration into a universal tool. This isn\u2019t just a theoretical improvement; these are practical tools already helping developers build faster and smarter applications today.\u201d<\/p>\n
While speculative decoding is not a new idea in itself, past approaches have typically required the small and large models to share the same vocabulary or to be trained together. This limited its practical use, as developers had to build custom small models compatible with each specific large model.<\/p>\n
The researchers\u2019 new method removes that constraint. By introducing three new algorithms that decouple speculative decoding from vocabulary alignment, the team says developers can now combine models from entirely different sources, or even different vendors, without needing to retrain them together. In an AI market increasingly fragmented by proprietary ecosystems, the ability to mix and match models could make it easier to deploy generative AI systems across a wider range of hardware, from cloud data centers to edge devices.<\/p>\n
The technique is already integrated into the widely used open-source Hugging Face Transformers library, making it accessible to millions of developers without the need for custom implementation. <\/p>\n
\u201cThis work removes a major technical barrier to making generative AI faster and cheaper,\u201d said Nadav Timor, a Ph.D. student working under Prof. David Harel at the Weizmann Institute. \u201cOur algorithms unlock state-of-the-art speedups that were previously available only to organizations that train their own small draft models.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"A team of researchers from Intel Labs and the Israel\u2019s Weizmann Institute of Science has presented a new…\n","protected":false},"author":2,"featured_media":10050,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,276,277,49,48,61],"class_list":["post-10049","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-ca","tag-canada","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/10049","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=10049"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/10049\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/10050"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=10049"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=10049"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=10049"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}