In brief
\nNearly every major social platform now relies on AI for ranking, ad targeting, moderation, and engagement optimization.
\nA 2025 report found 96% of social media professionals use AI tools\u201472.5% daily\u2014to generate and manage content.
\nAnalysts forecast the AI-in-social-media market to triple by 2030, embedding algorithmic influence deeper into online discourse.<\/p>\n
Large language models are learning how to win\u2014and that\u2019s the problem.<\/p>\n
In a research paper published Tuesday titled “Moloch\u2019s Bargain: Emergent Misalignment When LLMs Compete for Audiences<\/a>,” Stanford University Professor James Zou and PhD student Batu El show that when AIs are optimized for competitive success\u2014whether to boost ad engagement, win votes, or drive social media traffic\u2014they start lying.<\/p>\n \u201cOptimizing LLMs for competitive success can inadvertently drive misalignment,\u201d the authors write, warning that the very metrics that define \u201cwinning\u201d in modern communication\u2014clicks, conversions, engagement\u2014can quietly rewire models to prioritize persuasion over honesty.<\/p>\n \ufeff<\/p>\n “When LLMs compete for social media likes, they start making things up,” Zou wrote on X<\/a>. “When they compete for votes, they turn inflammatory\/populist.”<\/p>\n This work is important because it identifies a structural danger in the emerging AI economy: models trained to compete for human attention begin sacrificing alignment to maximize influence. Unlike the classical \u201cpaperclip maximizer<\/a>\u201d thought experiment, this isn\u2019t science fiction. It\u2019s a measurable effect that surfaces when real AI systems chase market rewards, what the authors call \u201cMoloch\u2019s bargain\u201d\u2014short-term success at the expense of truth, safety, and social trust.<\/p>\n Using simulations of three real-world competitive environments\u2014advertising, elections, and social media\u2014the researchers quantified the trade-offs. A 6.3% increase in sales came with a 14.0% rise in deceptive marketing; a 4.9% gain in vote share brought a 22.3% uptick in disinformation and 12.5% more populist rhetoric; and a 7.5% boost in social engagement correlated with a staggering 188.6% increase in disinformation and 16.3% more promotion of harmful behaviors.<\/p>\n \u201cThese misaligned behaviors emerge even when models are explicitly instructed to remain truthful and grounded,\u201d El and Zou wrote, calling this \u201ca race to the bottom\u201d in AI alignment.<\/p>\n In other words: even when told to play fair, models trained to win begin to cheat.<\/p>\n The problem isn’t just hypothetical<\/p>\n AI is no longer a novelty in social media workflows\u2014it\u2019s now near-ubiquitous.<\/p>\n According to the 2025 State of AI in Social Media Study<\/a>, 96% of social media professionals report using AI tools, and 72.5% rely on them daily. These tools help generate captions, brainstorm content ideas, re-format posts for different platforms, and even respond to comments. Meanwhile, the broader market is valuing this shift: The AI in social media sector is projected to grow<\/a> from USD 2.69 billion in 2025 to nearly USD 9.25 billion by 2030.<\/p>\n This pervasive integration matters because it means AI is shaping not just how content is made, but what content is seen, who sees it, and which voices get amplified. Algorithms now filter feeds, prioritize ads, moderate posts, and optimize engagement strategies\u2014embedding AI decision logic into the architecture of public discourse. That influence carries real risks: reinforcing echo chambers, privileging sensational content, and creating incentive structures that reward the manipulative over the truthful.<\/p>\n The authors emphasize that this isn\u2019t malicious intent\u2014it\u2019s optimization logic. When reward signals come from engagement or audience approval, the model learns to exploit human biases, mirroring the manipulative feedback loops already visible in algorithmic social media. As the paper puts it, \u201cmarket-driven optimization pressures can systematically erode alignment.\u201d<\/p>\n The findings highlight the fragility of today\u2019s \u201calignment safeguards.\u201d It\u2019s one thing to tell an LLM to be honest; it\u2019s another to embed that honesty in a competitive ecosystem that punishes truth-telling.<\/p>\n In myth, Moloch was the god who demanded human sacrifice in exchange for power. Here, the sacrifice is truth itself. El and Zou\u2019s results suggest that without stronger governance and incentive design, AI systems built to compete for our attention could inevitably learn to manipulate us.<\/p>\n The authors end on a sober note: alignment isn\u2019t just a technical challenge\u2014it\u2019s a social one.<\/p>\n \u201cSafe deployment of AI systems will require stronger governance and carefully designed incentives,\u201d they conclude, \u201cto prevent competitive dynamics from undermining societal trust.\u201d<\/p>\n Generally Intelligent Newsletter<\/p>\n A weekly AI journey narrated by Gen, a generative AI model.<\/p>\n","protected":false},"excerpt":{"rendered":"In brief Nearly every major social platform now relies on AI for ranking, ad targeting, moderation, and engagement…\n","protected":false},"author":2,"featured_media":68814,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[365,363,364,111,139,69,145],"class_list":["post-68813","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-new-zealand","tag-newzealand","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/68813","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=68813"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/68813\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/68814"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=68813"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=68813"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=68813"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}