ai and ml<\/p>\n
NeMo Switchyard brings GPT-5-style model routing to the mainstream<\/p>\n
Soaring AI infrastructure costs and model pricing, combined with uncertain returns on investment, threaten to stall enterprise adoption.<\/p>\n
To make enterprise AI spend a bit more manageable, Nvidia this week unveiled a new software platform that blurs the line between expensive proprietary models and open weights alternatives.\u00a0\u00a0<\/p>\n
Announced alongside Nemotron 3.5-30B-A3B-Lightning<\/a>, Nvidia\u2019s latest open weights model, NeMo Switchyard<\/a> is the GPU giant\u2019s latest overture to enterprise. So what exactly is it? Well, it\u2019s a router.<\/p>\n The idea is simple. Switchyard essentially functions as a proxy that sits between the inference server\u2019s API endpoint and the models. But rather than sending every request to the same model, Switchyard can be configured to route prompts to different models in order to optimize for cost, latency, or output quality. <\/p>\n By routing some requests to smaller, cheaper, and potentially locally hosted AI models, Nvidia claims Switchyard can cut job completion costs by 74 percent relative to using Claude Opus 4.8 alone, albeit with an approximately six-point accuracy tradeoff.<\/p>\n The right tool for the job<\/p>\n The key metric in all of this is completion cost rather than price per token. A model might cost one-tenth as much as OpenAI\u2019s or Anthropic\u2019s top model, but if it requires 10x the tokens to complete the request, it isn’t actually cheaper.<\/p>\n Certain elements of an AI workload may benefit from a larger, smarter model, but not all do.\u00a0<\/p>\n For example, it\u2019d be overkill to ask Claude Opus to generate a title card or summarize a website. It\u2019ll certainly work, but it\u2019ll also cost a fortune compared to Haiku or a locally hosted model that\u2019s been fine tuned just for that purpose. The fewer tokens you burn on the big smart model, the less expensive your API bill is going to be.<\/p>\n Nvidia software teams have spent the last several years developing models for this reason. The Lightning model announced this week is only its latest. The 30 billion-parameter MoE model is positioned as a low-latency, general purpose model that can either be used on its own or in conjunction with a larger, smarter model via a router like Switchyard.<\/p>\n The company has also developed several application-specific models. Nemotron Parse<\/a> is one such example. \u201cIt\u2019s a small model, one billion parameters, and it\u2019s really good at one task, which is taking a PDF in and then explaining the context inside that PDF whether it\u2019s charts or graphs or tables,\u201d Joey Conway, senior director of AI software and models at Nvidia, explained in a recent interview with The Reg.<\/p>\n Many frontier models struggle with this task because PDFs are designed by humans for humans, so by offloading that work to task-specific models, enterprises can not only improve the accuracy of their AI apps, but also reduce costs in the process.<\/p>\n This all might sound familiar: It’s not the first time we\u2019ve seen model routers employed as a cost-saving measure.<\/p>\n Back when OpenAI launched GPT-5, ChatGPT would dynamically route prompts to different versions of the model based on their complexity.\u00a0<\/p>\n As we wrote at the time, OpenAI\u2019s router was likely implemented<\/a> to reduce the number of compute cycles spent on mundane tasks like rewording emails to sound more professional (“not only \u2026 but also”).<\/p>\n