{"id":786862,"date":"2026-09-29T23:38:09","date_gmt":"2026-09-29T23:38:09","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/786862\/"},"modified":"2026-09-29T23:38:09","modified_gmt":"2026-09-29T23:38:09","slug":"open-source-tool-distills-jev-so-you-can-run-it-locally","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/786862\/","title":{"rendered":"Open source tool distills Jev so you can run it locally"},"content":{"rendered":"

<\/p>\n

\n ai and ml\n <\/p>\n

<\/a><\/p>\n

\n Jevstiller targets 98% agreement by learning familiar requests on your hardware while sending uncertain and audited queries upstream\n <\/p>\n

Sure, Jev is cheap and quick to respond to queries, but you know what would be even better? How about a local student model that learns from your Jev requests and runs familiar ones on your hardware instead?<\/p>\n

Jevstiller is an open source project that, as its name suggests, distills Jev\u2019s outputs into a local model that can do some of Jev\u2019s arguably fast work even more quickly and efficiently – no need to spend $42 per billion input tokens or wait around 300 ms for a response on requests it can handle locally, say Jevstiller\u2019s creators.<\/p>\n

Answering locally means no Jev tokens get spent, however cheap they may be, and answers can come in as little as 15 ms directly from a device\u2019s CPU. The model that Jevstiller constructs on a user\u2019s machine is small and fast, and, if its developers\u2019\u00a0claims<\/a>\u00a0are to be believed, helps the system achieve 98 percent overall agreement with Jev by handling familiar requests locally and forwarding others to Jev.<\/p>\n

Jev, as we\u2019ve detailed<\/a> in several<\/a> stories<\/a> recently, is a new paradigm of AI design that foregoes all the chattiness and free-form text generation that come with the typical modern LLM in favor of being able to answer just a few basic types of questions: Choice, Score, and Noul. Because it doesn\u2019t generate a bunch of text and need to ruminate on the structure of human language, Jev is quick, cheap, and rapidly attracting developer interest as a practical application of AI for structured decision workloads where a full-blown generative LLM may be overkill.<\/p>\n

The Jevstiller team likens it to a cache that sits in front of Jev itself, scooping up requests and determining whether they need to be routed to the actual Jev servers when the local version isn\u2019t confident it can answer a question.\u00a0<\/p>\n

\u201cA request is answered locally only if every question in it can be; otherwise the whole request goes to Jev with the caller\u2019s own key, Jev\u2019s response is returned unchanged, and each choice answer becomes a training row,\u201d the Jevstiller team explains in a help document<\/a>.\u00a0<\/p>\n

Spinning up a new Jevstiller basically means you\u2019re setting up a model that doesn\u2019t know anything about Jev responses, so it needs to learn how Jev responds to queries by passing all of them to Jev and gathering both questions and answers.\u00a0<\/p>\n

\u201cOnce there are enough samples per class, a background worker fits a student and a routing policy,\u201d Jevstiller\u2019s documentation explains. \u201cConfident, familiar inputs are answered locally. Everything else goes to Jev.\u201d\u00a0<\/p>\n

Say you use Jev to filter incoming emails to a particular team, or to determine priority of a particular task, for example. Once Jevstiller sees enough examples of those queries and responses, it can confidently answer the same way Jev would.\u00a0<\/p>\n

Maintaining agreement with Jev,\u00a0as the Jevstiller team notes, is the key design problem they were seeking to address. They seem to think they\u2019ve arrived at a solution in the form of regular auditing that constantly assesses Jevstiller\u2019s responses against a random selection of queries.\u00a0<\/p>\n

\u201cA fixed 2 percent of all requests goes to Jev regardless of what the local model thinks, with the local model\u2019s answer recorded alongside,\u201d the developers say, noting that such a solution is the only way to get an unbiased view of Jevstiller\u2019s routing since it automatically declines to answer difficult questions. Jevstiller continually retrains itself based on all Jev responses, and the target agreement rate can be configured at the local level.\u00a0<\/p>\n

If the audit suggests at any time that Jevstiller\u2019s agreement rate with Jev has fallen below the configured target, the audit rate goes up so Jevstiller\u2019s retraining happens more quickly. If the worst happens and Jevstiller thinks its ability to agree with Jev has collapsed, “every request goes back to Jev and training restarts from that point.”\u00a0<\/p>\n

The team\u2019s own experimentation demonstrated that the concept works, as they explained.\u00a0<\/p>\n

\u201cIn the 24-hour soak, a stand-in Jev silently changed every answer at hour twelve,\u201d Jevstiller\u2019s devs said of an experimental run. \u201cThe local share fell from 90% to 9% within four minutes as the audits caught it, and was back at 90% within 49 minutes, trained on post-change answers only, with nobody touching anything.\u201d<\/p>\n

Agreement with Jev doesn\u2019t mean a response is correct, though, which the Jevstiller team admits. \u201cAgreement is not accuracy,\u201d they note.\u00a0<\/p>\n

In other words, whether you go this route to even further reduce your AI bill or choose some other method, Jev is still just another AI model that can make mistakes, no matter how good it claims to be.\u00a0<\/p>\n

Those who want to give Jevstiller a try can learn more about getting started here<\/a>.\u00ae<\/p>\n","protected":false},"excerpt":{"rendered":"ai and ml Jevstiller targets 98% agreement by learning familiar requests on your hardware while sending uncertain and…\n","protected":false},"author":2,"featured_media":786863,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-786862","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/786862","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=786862"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/786862\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/786863"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=786862"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=786862"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=786862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}