{"id":527989,"date":"2026-04-13T05:30:24","date_gmt":"2026-04-13T05:30:24","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/527989\/"},"modified":"2026-04-13T05:30:24","modified_gmt":"2026-04-13T05:30:24","slug":"ai-polls-are-fake-polls","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/527989\/","title":{"rendered":"\u201cAI polls\u201d are fake polls"},"content":{"rendered":"

\"\"<\/a>Cover image<\/a> from Isaac Asimov\u2019s short story \u201cFranchise\u201d.<\/p>\n

A few weeks after Donald Trump\u2019s second presidential win, I took the train up from London (where I was living at the time<\/a>) to Oxford to attend a conference on polls and forecasts of the 2024 election<\/a>. Most of the attendees were pollsters or academics, but I also watched presentations from Aaru<\/a> and Electric Twin<\/a>, two companies that do what is interchangeably called synthetic sampling, silicon sampling, or creating synthetic audiences. Sans startup jargon, that means they use large language models (LLMs) to simulate responses to public opinion polls by having AI agents take on the role of survey respondents.<\/p>\n

I had already heard of Aaru thanks to some articles with eye-catching headlines like \u201cNo people, no problem: AI chatbots predict elections better than humans\u201d<\/a> in the months leading up to Election Day. The guys behind the company were making some big, some might even say far-fetched claims, such as: <\/a> \u201cwithin two years, we will simulate the entire globe \u2014 from the way crops are grown in Ukraine to how that impacts production of oil in Iraq, trade through the strait of Malacca, and elections for the mayor of Baltimore.\u201d When Semafor asked Aaru\u2019s cofounders \u2014 Cameron Fink and Ned Koh \u2014 about my boss, they said \u201cwe respect all those who came before us.\u201d Nate (as he so often does<\/a>) shared his thoughts on Twitter:<\/p>\n

\"X<\/p>\n

Nate Silver@NateSilver538<\/p>\n

LOL I wish there were a way to short this business this is maybe the single worst use case for AI I’ve ever heard.<\/p>\n

\"X<\/p>\n

Ben Smith @semaforben<\/p>\n

Sorry @NateSilver538 https:\/\/t.co\/xNTCJwSMmh<\/p>\n

8:57 PM \u00b7 Sep 20, 2024 \u00b7 539K Views<\/p>\n

87 Replies \u00b7 52 Reposts \u00b7 1.5K Likes<\/p>\n

<\/a><\/p>\n

Fink and Koh were relatively good-natured about this back-and-forth when we spoke at Oxford. They even offered to mail me one of the t-shirts featuring Nate\u2019s quote they apparently had made. I never took them up on the offer, which I now somewhat regret.<\/p>\n

These synthetic sampling companies fell off my radar for a while, but they do still exist. In fact, Aaru recently received a $1 billion valuation<\/a>. Is what they\u2019re doing anywhere close to the most important frontier in AI development? Not by a longshot, especially when Anthropic just developed a model so adept at exploiting software vulnerabilities that it\u2019s only being released to 40 companies<\/a>.<\/p>\n

Still, silicon sampling is increasingly finding its way into public polling. Axios reported in March<\/a> that \u201ca majority of people trust their own doctors and nurses\u201d based on findings from Aaru \u2014 without mentioning<\/a> that the \u201cpeople\u201d in that sentence were actually LLMs. Around the same time, the Public Sentiment Institute \u201cboosted\u201d their online sample of 373 real survey respondents with 114 AI agents<\/a>. (Spoiler alert: even the co-founder of Electric Twin doesn\u2019t think that\u2019s a particularly defensible approach.) Polling companies like Qualtrics<\/a> and Ipsos<\/a> are also developing synthetic data panels.<\/p>\n

So, what should we make of these \u2026 \u201cpolls\u201d? Let\u2019s get one thing out of the way: whatever they are, they\u2019re not polls in the way that term is usually defined.<\/p>\n

On one hand, using LLMs to essentially make up fake survey respondents sort of sounds like the dumbest idea ever, one that will only imperfectly replicate real polls while introducing all sorts of biases. On the other hand, with LLMs improving at a remarkable, perhaps even alarming rate<\/a>, maybe I\u2019m a dinosaur at the ripe old age of 24 because I still want to rely on polls that talk to actual people.<\/p>\n

I\u2019m not going to argue that synthetic samples are completely useless. In fact, as I\u2019ll return to later, there is evidence that some techniques can replicate topline survey results quickly and cheaply. But the marketing from certain companies can be slightly optimistic. \u201cNo traditional poll will exist by the time the next general election occurs,\u201d said Fink in 2024<\/a>. We\u2019re just 206 days away from the midterms, and based on the fact that I still have to collect a bunch of polls every day, I\u2019d say he should have run that prediction by a sample of AI agents before the interview.<\/p>\n

To see why synthetic samples can\u2019t replace polls, here\u2019s a quick primer on how they work. The simplest version of these models involves taking a LLM (like ChatGPT or Claude), giving it a demographic profile<\/a> (e.g., a white, college-educated woman who lives in Utah and makes $70k a year), and then asking it to respond to a survey question. You repeat that process a few thousand times using different demographic profiles and end up with a sample of synthetic survey responses.<\/p>\n

The actual models used by private companies are more sophisticated than this, usually because they incorporate more demographic characteristics for each agent and provide them with extra information. Aaru, for example, feeds agents a diet of news and information they\u2019d be likely to consume<\/a>, while Electric Twin incorporates their customers\u2019 proprietary data<\/a> about the audience they\u2019re trying to replicate. The way Ben Warner, the co-founder of Electric Twin, explained it to me was \u201cwe have a large amount of data on\u2026 for instance, 5,000 people. Can we make an accurate prediction of how they would respond to another question?\u201d<\/p>\n

Still, without any reference to cost, speed, or accuracy, it should be obvious why synthetic samples can\u2019t replace polls. Polling is fundamentally a data collection process. We might use surveys to make predictions by feeding them into election forecasts<\/a>, but the main purpose of a poll isn\u2019t prediction, it\u2019s gathering new data about what people think and how they feel<\/a>. Silicon sampling, on the other hand, produces no new data. It\u2019s simply a model: you input LLM training data, demographic prompts, and a bunch of other information, and it spits out a prediction for what a poll would say.<\/p>\n

We love models here<\/a>, but models aren\u2019t polls. That difference is an important philosophical sticking point<\/a> for most pollsters I talk to. \u201cI think politics should stay away from [synthetic sampling], because we\u2019re trying to\u2026 represent the voice of the people,\u201d said Natalie Jackson, a vice president at GQR Insights. Democratic pollster John Hagner told me \u201cI think I\u2019m just incredibly skeptical of this idea. I don\u2019t think it\u2019s research. At that point, you\u2019re asking the machine to tell you what you already believe.\u201d Hagner has seen some presentations of early synthetic sampling experiments, but so far, \u201cif it\u2019s being used in a campaign, people are keeping it incredibly quiet.\u201d<\/p>\n

But Eli, I hear you saying, aren\u2019t polls themselves increasingly governed by modeling decisions<\/a>? Indeed they are: pollsters\u2019 choices on which sampling method to use, how to define their likely voter models<\/a>, and how to weight their samples can and do lead to dramatic differences in the results they publish<\/a>. Aaru even referenced these limitations in the methodology statement included with that maternal mortality \u201cpoll\u201d<\/a> \u2014 although I\u2019m using the term \u201cmethodology statement\u201d loosely here, because it doesn\u2019t really explain how the model works at all.<\/p>\n

\"\"<\/a><\/p>\n

We can ignore the (frankly preposterous) implication that synthetic sampling isn\u2019t subject to a separate set of biases<\/a>. The important point is that there\u2019s still a meaningful difference between using weighting and other statistical techniques on actual polling data and using a model to predict what a poll would say. The latter is far closer to election forecasts or techniques like MRP<\/a> \u2014 potentially useful models, but not a replacement for polls.<\/p>\n

To be fair, other synthetic sampling companies are perfectly happy with the distinction between polls and models. Warner compared polling and synthetic sampling to different tools in a toolbox. \u201cThe mistake I think we make is we think that these new tools should either work in exactly the same way or somehow replace these old tools,\u201d he said. \u201cRather than thinking of it as, okay, so we\u2019ve always had the hammer, we\u2019ve always had the screwdriver, now we\u2019ve got a saw. But don\u2019t use a saw to try [to] do the job of a hammer.\u201d<\/p>\n

A quick comment from Nate<\/p>\n

Eli didn\u2019t ask me for a comment \u2014 rather rude of him, don\u2019t you think? But since I\u2019m editing this story, I figured I\u2019d add a few quick thoughts rather than putting words in his mouth. <\/p>\n

Beyond the frequently misleading marketing, what bothers me the most about the AI \u201cpoll\u201d hype is that as AI tools make statistical inference cheaper and\/or better (note that these are not synonyms) that actually increases the comparative value of collecting original data. You might be able to train a model to make a reasonable estimate of what some hard-to-reach poll respondent would say \u2014 say, a young Black man who voted for Trump. (Such a person checks a number of boxes for a voter who is usually hard to reach in surveys.) Indeed, this is closely related to what models like the Silver Bulletin forecast already do. They essentially smooth out the kinks in noisy survey data by making inferences based on past voting patterns or national polls or surveys of other states.<\/p>\n

But you don\u2019t actually know what these voters think unless you\u2019re reaching them directly. If there\u2019s a shift in opinion among this subgroup, you\u2019re not going to detect it. So if I were running a campaign, I\u2019d invest more in going the extra mile to find a representative sample of these voters. And then I\u2019d hire some smart quants \u2014 perhaps with help from Claude et. al. \u2014 to figure out the implications for campaign strategy based on that proprietary data that my competitors didn\u2019t have access to. -Nate Silver<\/p>\n

If synthetic surveys are just a new type of model, the next obvious question is whether the models are at least accurate. The answer to that question depends very much on who you ask.<\/p>\n

On one end of the spectrum, you have the maximalist argument that synthetic sampling is better and more accurate than actual polls. \u201cIt\u2019s an incredibly challenging problem to go to someone and say \u2018hey, we\u2019re going to be more accurate at predicting human behavior than you, even when you talk to your customers directly,\u201d Koh recently told CNBC<\/a>. In his view, synthetic sampling isn\u2019t a saw to polling\u2019s hammer, it\u2019s \u201cmagic.\u201d<\/p>\n

There\u2019s certainly evidence that synthetic samples can replicate certain survey toplines<\/a>. But if Aaru does have any examples of their approach outperforming the polls, they\u2019re keeping those to themselves. Aaru\u2019s 2024 election model<\/a>, for example, had Kamala Harris leading in Michigan, Nevada, Pennsylvania, and Wisconsin on November 4th. And although they\u2019ve since taken down their forecast page, they gave Harris a 50.5 percent chance of winning the race on November 2nd<\/a>.<\/p>\n

After the election, Fink told Semafor he was happy enough with those results because they were \u201cwithin margin of error,\u201d a term that is meaningless when applied to a \u201csample\u201d of AI agents. And of course, Aaru says their models have improved since 2024<\/a>, so supposedly now they\u2019d be more accurate than the polls? Still, their stronger argument is on cost: \u201cWe are significantly faster and cheaper than traditional polling, and still more accurate,\u201d said Fink<\/a>. The first two claims there are undeniably true, but the third brings us to the opposite end of the spectrum.<\/p>\n

Both Jackson and Hagner are skeptical that these models are reliable for anything far beyond replicating common survey toplines. \u201cI just\u2026 don\u2019t think the machines are what we want when we\u2019re looking for nuanced views. My example on this is people in Arizona and Nevada in 2024 who voted for Trump and voted for expanding abortion in their states on ballot initiatives,\u201d said Jackson. Hagner identified a similar issue<\/a>: \u201cthe reports that have come through at the meetings that I\u2019ve been at are that the early experiments on this, they cannot get respondents to be as racist or sexist or, frankly, as negative as human respondents.\u201d<\/p>\n

Academic research mostly agrees on this point. While there are some papers that show promising results when using LLMs to replicate polling data<\/a>, most show that LLMs suffer from various quirks like producing too few \u201cdon\u2019t know\u201d responses<\/a> and can seriously overpredict the favorability<\/a> of politicians like Donald Trump and Kamala Harris. They also seem to struggle with too little variation between demographic subgroups<\/a>, so the difference in predicted opinion between Democrats and Republicans, for example, is too small.<\/p>\n

When I asked Warner about these studies, his response to these papers was that just because academics can\u2019t get synthetic sampling to work doesn\u2019t mean that the technique doesn\u2019t work in general. \u201cActually, the argument is, okay, yours does not [work]. That does not mean [\u2026] for this complex set of machinery, which uses a lot of investment, a lot of time, a lot of money, you can\u2019t get it to work.\u201d<\/p>\n

Cards on the table, I\u2019m somewhat sympathetic to this argument because academics aren\u2019t exactly great at making election forecasts<\/a>. Usually, the people with skin in the game are the most accurate. Warner\u2019s argument is that the approach Electric Twin takes \u2014 which includes, for example, making multiple predictions for each synthetic respondent using different models and prompts and subsequently averaging those to get a final prediction like a sort of ensemble forecast<\/a> \u2014 produces better results than the simpler academic models.<\/p>\n

Warner shared a comparison<\/a> between his method and the method from a recent academic paper<\/a> with me, and Electric Twin was indeed able to get more accurate replication. But even still, he acknowledged that synthetic sampling \u201cis not a crystal ball.\u201d \u201cIf you asked me, do I think using other data sources will be more accurate than asking somebody who they will vote for, I would probably say no. But if you asked me \u2018would your system be useful for our turnout modeling today?\u2019 I would say yes.\u201d<\/p>\n

But for better or worse, it looks like the method is already getting more popular in the market research world. Most of the clients Aaru talks about these days are businesses like EY and McDonald\u2019s<\/a>. <\/p>\n

That\u2019s not to say AI won\u2019t pop up in other parts of the political polling process. Pollsters are already using it to code open-ended survey responses, and some firms, like YouGov, are testing using LLMs to ask survey respondents questions<\/a>.<\/p>\n

More worryingly, one danger to actual polls is that AI agents can be used to infiltrate online surveys. Most online polls use various checks to prevent that from happening, but there\u2019s conflicting evidence on how effective those filters<\/a> are and how prevalent AI agents currently are in online panels<\/a>. If those agents ever become impossible to detect, it might spell the end of online polling, but the solution isn\u2019t to replace all of your respondents with ChatGPT.<\/p>\n","protected":false},"excerpt":{"rendered":"Cover image from Isaac Asimov\u2019s short story \u201cFranchise\u201d. A few weeks after Donald Trump\u2019s second presidential win, I…\n","protected":false},"author":2,"featured_media":527990,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":{"0":"post-527989","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-technology","12":"tag-uk","13":"tag-united-kingdom","14":"tag-unitedkingdom"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/527989","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=527989"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/527989\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/527990"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=527989"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=527989"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=527989"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}