In Q3, we made improvements to GEM\u2019s model architecture that doubled the performance benefit we get from adding a given amount of data and compute. This will enable us to continue scaling up the amount of training capacity we use on GEM at an attractive ROI.<\/p>\n
Introducing GEM<\/p>\n
GEM represents a significant advancement in RecSys through three key innovations: model scaling with advanced architecture, post-training techniques for knowledge transfer, and enhanced training infrastructure to support scalability. These innovations efficiently boost ad performance, enable effective knowledge sharing across the ad model fleet, and optimize the use of thousands of GPUs for training. GEM has driven a paradigm shift in ads RecSys, transforming ad performance across the funnel \u2014 awareness, engagement, and conversion \u2014 through joint optimization of both user and advertiser objectives.<\/p>\n
Building a large foundation model for Meta\u2019s ads RecSys requires addressing several key challenges:<\/p>\n
Handling a large, dynamic feature space across all of Meta\u2019s apps: Every day, billions of user-ad interactions occur across our platforms, but meaningful signals \u2014 such as clicks and conversions \u2014 are very sparse. GEM must learn from this vast but imbalanced data, recognizing meaningful patterns and generalizing across diverse users and behaviors.
\nProcessing a diverse array of data: GEM must learn from a diverse array of ads data \u2014 including advertiser goals, creative formats, measurement signals, and user behaviors across multiple delivery channels. This heterogeneity adds significant modeling complexity, requiring GEM to unify multimodal, multi-source inputs and capture nuanced interactions to power other ads recommendation models.
\nTraining efficiently: Training and scaling a large foundation model demands thousands of GPUs and leveraging advanced parallelism and system-level optimization to ensure efficient hardware utilization.\u00a0<\/p>\n
GEM overcomes these challenges through:<\/p>\n
\u00a0A scalable model architecture that is now 4x more efficient at driving ad performance gains for a given amount of data and compute than our original ads recommendation ranking models.\u00a0
\nA new framework that improves knowledge transfer effectiveness, achieving 2x the effectiveness of standard knowledge distillation.
\nA new training stack that delivers a 23x increase in effective training FLOPS with a 1.43x increase in model FLOPS utilization (MFU) using 16x more GPUs.\u00a0<\/p>\n
Building and Scaling GEM\u2019s Architecture<\/p>\n
GEM is trained on ad content and user engagement data from both ads and organic interactions. From this data, we derive features that we categorize into two groups: sequence features (such as activity history) and non-sequence features (such as user and ad attributes \u2014 e.g., age, location, ad format, and creative representation). Customized attention mechanisms are applied to each group independently, while also enabling cross-feature learning. This design improves accuracy and scales both the depth and breadth of each attention block, delivering 4\u00d7 the efficiency of our previous generation of models.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/11\/Meta-Generative-Ads-Model-GEM-image-1-e1762534706646.png\")
<\/p>\n
Non-Sequence Feature Interaction Modeling<\/p>\n
Understanding how user attributes interact with ad characteristics is crucial for accurate recommendations. GEM enhances the Wukong architecture<\/a> to use stackable factorization machines with cross-layer attention connections, allowing the model to learn which feature combinations matter most. Each Wukong block can scale vertically (for deeper interactions) and horizontally (for broader feature coverage), enabling the discovery of increasingly complex user-ad patterns.<\/p>\n Offline Sequence Feature Modeling<\/p>\n User behavior sequences \u2014 spanning long sequences of ad \/ content clicks, views, and interactions \u2014 contain rich signals about preferences and intent, yet traditional architectures struggle to process such long sequences efficiently. GEM overcomes this challenge with a pyramid-parallel structure, stacking multiple parallel interaction modules in a pyramid formation to capture complex user-ad relationships at scale. The new scalable offline feature infrastructure processes sequences of up to thousands of events with minimal storage cost, so GEM can learn from a much longer history of user organic and ad interactions. By modeling these extended user behavior sequences, GEM can more effectively uncover patterns and relationships, resulting in a deeper and more accurate understanding of the user\u2019s purchase journey.<\/p>\n Cross-Feature Learning<\/p>\n Existing approaches compress user behavior sequences into compact vectors for downstream tasks, which risks losing critical engagement signals. GEM takes a different approach that preserves full sequence information while enabling efficient cross-feature learning. Our design, InterFormer<\/a>, employs parallel summarization with an interleaving structure that alternates between sequence learning (e.g., custom transformer architecture<\/a>) and cross-feature interaction layers. This allows progressively refining its sequence understanding while maintaining access to the complete user journey. This design facilitates efficient interaction learning while preserving the structural integrity of user sequence data \u2014 enabling GEM to scale to higher layer counts without losing critical behavioral signals.<\/p>\n Multi-Domain Learning With Domain-Specific Optimization<\/p>\n Traditional ad recommendation systems struggle to balance learning across a broad product ecosystem \u2014 treating surfaces either in isolation (thus missing valuable cross-platform insights) or identically (ignoring platform-specific behaviors). Different Meta surfaces like Facebook, Instagram, and Business Messaging each have unique user behaviors and interaction patterns. GEM solves this through learning from cross-surface user interactions while ensuring predictions remain tailored to each surface\u2019s unique characteristics. For example, this enables GEM to use insights from Instagram video ad engagement to improve Facebook Feed ad predictions, while also optimizing each domain\u2019s predictions for its specific objective (such as clicks or conversions).<\/p>\n Maximizing Transfer Efficiency With Post Training Techniques<\/p>\n GEM only delivers impact if its knowledge can be efficiently transferred to hundreds of user-facing vertical models (VMs). To translate the performance of the GEM foundation model (FM) into measurable gains for user-facing VMs, we employ both direct and hierarchical knowledge transfer strategies.\u00a0<\/p>\n Direct transfer enables GEM to transfer knowledge to major VMs within the same data spaces where GEM was trained. Hierarchical transfer distills knowledge from GEM into domain-specific FMs, which then teach VMs, driving broad improvements across ad models. Together, these approaches use a suite of techniques, including knowledge distillation, representation learning, and parameter sharing to maximize transfer efficiency across the entire ad model space, achieving 2x the effectiveness of standard knowledge distillation<\/a>.<\/p>\n Knowledge Distillation<\/p>\n In Meta\u2019s ads system, VMs often suffer from stale supervision caused by delays in FM training and evaluation as well as domain mismatches between GEM or FM predictions and the VMs\u2019 surface-specific objectives. These outdated or misaligned signals between the VMs (students) and GEM (the teacher) can degrade the accuracy and adaptability of student models over time.<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/11\/Meta-Generative-Ads-Model-GEM-image-2.png\")
<\/p>\n