{"id":15081,"date":"2025-07-22T09:33:05","date_gmt":"2025-07-22T09:33:05","guid":{"rendered":"https:\/\/www.newsbeep.com\/ca\/15081\/"},"modified":"2025-07-22T09:33:05","modified_gmt":"2025-07-22T09:33:05","slug":"apple-details-how-it-trained-its-new-ai-models-see-highlights","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ca\/15081\/","title":{"rendered":"Apple details how it trained its new AI models, see highlights"},"content":{"rendered":"

\t\"Apple<\/p>\n

During WWDC25, Apple announced new versions of its on-device and cloud-based foundation models. Now, they have published a tech report detailing how those models were trained, optimized, and evaluated. And the report includes some genuinely interesting under-the-hood tidbits.<\/p>\n

In a comprehensive document called \u201cApple Intelligence Foundation Language Models \u2013 Tech Report 2025<\/a>\u201c, the company walks through multiple aspects of the new models, including their architecture, data sources, pre-training, post-training, tool use development, optimizations, and benchmarks.<\/p>\n

\"\"Modeling overview for the Apple foundation models. Image: Apple<\/a><\/p>\n

It is a very technical, but very worthwhile read if you like to get into the nuts and bolts of this sort of stuff. Here are a few particularly interesting highlights.<\/p>\n

The local model was split into two blocks<\/p>\n

We already knew that Apple\u2019s on-device model (the one developers will get to tap into<\/a>) has around 3 billion parameters. Now, the company has detailed that this model is actually divided into two blocks:<\/p>\n

\u201cBlock 1 contains 62.5% of the total transformer layers, while Block 2 contains the remaining 37.5% of the transformer layers, but had the key and value projections removed.\u201d<\/p>\n

In practice, this means that the local model requires 37.5% less memory for caching, and the time it takes to output the first token (basically, a fragment of a word) was also cut by about 37.5%. Still, Apple structured the split in a way that it says preserves the model\u2019s overall performance and output quality.<\/p>\n

\"\"Apple\u2019s on-device vs external models on representative benchmarks. Image: Apple<\/a><\/p>\n

As a side note, a few years ago, Apple published this study<\/a>, which looked at swapping parts of an LLM between RAM and flash storage as needed, in order to pack a local model that was bigger than what would otherwise fit on the device\u2019s memory.<\/p>\n

While Apple ultimately took a different route, it is interesting to note the different ways the company has been experimenting to offer good local performance, even on memory-constrained devices.<\/p>\n

The cloud-based model has a creative architecture<\/p>\n

For its server model, Apple built a custom architecture that was tailor-made for its Private Cloud Compute <\/a>platform. It\u2019s called Parallel-Track Mixture-of-Experts (PT-MoE), and the way it works is pretty neat.<\/p>\n

In a nutshell (and at the risk of oversimplifying things), Mixture of Experts is when, instead of relying on one huge AI model, it\u2019s split into smaller subnetworks (or experts) that are only activated when the task is related to something they\u2019re\u2026 well, an expert in.<\/p>\n

So if your prompt is about cooking, only cooking-related experts are activated, while others remain dormant. The result is still a massive overall model, but its modular design allows it to respond faster (and often more accurately) than if everything were running through the huge, unified model, for every prompt.<\/p>\n

Here is an IBM Mixture of Experts explainer, in case you have 8 minutes to spare:<\/p>\n<\/p>\n

Apple built a new kind of Transformer called the Parallel Track Transformer, then scaled it up with Mixture of Experts (MoE) layers. That sounds way too complicated, but the gist of it is:<\/p>\n

Traditional Transformers process tokens through a single stack of layers, one after the other. But rather than using this single-track approach to calculate every token, Apple\u2019s design splits the model into multiple, parallel tracks. Each track processes tokens independently, and only syncs up at certain points.<\/p>\n

Then, inside each of those tracks, Apple replaced every other regular transformer layer with an MoE layer, which activates just a few experts for each token, while the rest stay idle. And because each track has its own local experts, the model avoids the processing bottlenecks that happen when everything has to coordinate across the entire system.<\/p>\n

\"\"<\/p>\n

Add to that a clever setup that balances local context with big-picture understanding (called Interleaving Global and Local Attention Layers), and the result is a very modular, efficient, and scalable model that\u2019s faster and leaner, but still pretty smart.<\/p>\n

Apple increased multilingual representation by 275%<\/p>\n

One of the biggest knocks against the initial rollout of Apple Intelligence was (and still is) limited language support beyond English. With its new models, Apple has expanded language support, and the document details the steps it took in order to do that.<\/p>\n

According to the document, Apple increased the amount of multilingual data used during training from 8% to 30%. This includes both organic and synthetic content.<\/p>\n

Apple also increased its tokenizer (which is basically the model\u2019s token vocabulary) by 50%. This means that its model now knows 150K different tokens, up from the previous 100K.<\/p>\n

The company says that these changes led to \u201csignificant gains\u201d in performance across non-English benchmarks, especially after reinforcement learning fine-tuning.<\/p>\n

In the deocument, Apple explains that evaluations were conducted using prompts written by native speakers (rather than translations), and the model was tested on both accuracy and how natural its responses sounded in local contexts. If this sounds familiar, you probably read our recent coverage of this Apple Research study<\/a>.<\/p>\n

In practice, all of this means that features like Writing Tools should work more reliably in the supported languages.<\/p>\n

\"\"<\/p>\n

From where did Apple source its data?<\/p>\n

Like with its first models, most of the training data came from crawling the web. But Apple says that its Applebot crawler respects robots.txt exclusions, meaning that if a website doesn\u2019t want Apple to scrape its content, it can say so, and Applebot will leave it alone.<\/p>\n

That said, here is how Apple says it sourced the data for its new models:<\/p>\n

Publicly available web data:\u00a0Although Apple doesn\u2019t specify quantities or ratios, it does say that the largest portion of its training data came from Applebot crawling web pages. Apple applied multiple layers of filtering to remove low-quality, unsafe, or irrelevant content, including spammy pages, shallow or templated text, and broken formatting.<\/p>\n

Licensed data:\u00a0Apple doesn\u2019t go into much detail here, but does confirm that some of the training data was licensed from publishers. Earlier reports had suggested that Apple had been negotiating<\/a> with Cond\u00e9 Nast (The New Yorker, Vogue, Wired, etc.), NBC News, and IAC (People Magazine, The Daily Beast, and Better Homes and Gardens, etc.), so it\u2019s likely that at least some of that material made it in.<\/p>\n

Synthetic data:\u00a0Apple generated synthetic data using smaller models and custom pipelines, particularly for math, code, instruction tuning, and vision-language tasks. While the company also doesn\u2019t specify how much of the dataset this represented, it notes that synthetic data played a large role in key training steps like fine-tuning, reinforcement learning, and improving multilingual support. And If you\u2019re wondering what synthetic data just means \u201cmade-up stuff,\u201d we\u2019ve got an explainer on why that\u2019s not the case<\/a>.<\/p>\n

Visual data:\u00a0To support image understanding, Apple collected over 10 billion image\u2013caption pairs, including screenshots with OCR, and handwritten notes. It also used its own models to generate additional, richer captions. In the past, it was reported that Apple had held licensing talks with Shutterstock<\/a>, so it\u2019s possible some of that material also made it in.<\/p>\n

9to5Mac\u2019s take<\/p>\n

There has been no shortage of news on Apple\u2019s internal drama<\/a>, technical struggles, and overall inability to gain the momentum it needs to bridge the gap (which some might call a chasm) between its AI offerings, and the competition. All of those are true.<\/p>\n

Yet, the fact that Apple is largely perceived as being behind on AI doesn\u2019t mean the company is standing still. This report offers an interesting insight into the under-the-hood improvements (and shortcomings) of Apple\u2019s newest models, along with extensive details on a privacy-conscious approach that few companies are even attempting.<\/p>\n

Accessory deals on Amazon<\/p>\n

FTC: We use income earning auto affiliate links. More.<\/a><\/p>\n

\"\"<\/a>\t\t\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"During WWDC25, Apple announced new versions of its on-device and cloud-based foundation models. Now, they have published a…\n","protected":false},"author":2,"featured_media":15082,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,276,277,49,48,61],"class_list":["post-15081","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-ca","tag-canada","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/15081","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=15081"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/15081\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/15082"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=15081"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=15081"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=15081"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}