{"id":176819,"date":"2026-03-27T03:47:32","date_gmt":"2026-03-27T03:47:32","guid":{"rendered":"https:\/\/www.newsbeep.com\/us-ny\/176819\/"},"modified":"2026-03-27T03:47:32","modified_gmt":"2026-03-27T03:47:32","slug":"can-artificial-intelligence-fix-social-science","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us-ny\/176819\/","title":{"rendered":"Can Artificial Intelligence Fix Social Science?"},"content":{"rendered":"

Artificial intelligence can do many of the things that social scientists do. It can analyze data, write and review code, identify appropriate statistical methods, and offer suggestions on study drafts. It can even take a dataset and a research question and produce an entire paper on its own.\u00a0Given that human-led social science is often\u00a0marred<\/a>\u00a0by mistakes, dubious methods, ideological bias, and even outright fraud, one can hope that AI will improve the field in the years ahead.<\/p>\n

Some recent studies, though, highlight the limitations of current models. For now, AI is a productivity- and quality-enhancing tool, but not a panacea for what ails social science, nor a reason to let one\u2019s guard down.<\/p>\n

Finally, a reason to check your email.<\/p>\n

Sign up for our free newsletter today.<\/p>\n

Perhaps the biggest social-science story of the week centers on a 2021 study of tech clusters\u2014cities with large numbers of inventors working in a given field\u2014and their impact on innovation. A charity found this study helpful in its funding decisions and\u00a0hired economist Michael Wiebe<\/a>\u00a0to extend it. But when Wiebe dug into the materials, he uncovered a series of technical and coding errors that, once corrected, undermined the study\u2019s conclusions. The\u00a0American Economic Review\u00a0(which also published the original study) has accepted Wiebe\u2019s\u00a0comment detailing these issues<\/a>.<\/p>\n

As it happened, Wiebe also\u00a0tried feeding the materials<\/a>\u00a0into AI chatbots\u2014two versions of ChatGPT and also Refine, a special tool calibrated to improving academic work\u2014asking them to scrutinize each key result to see if they\u2019d detect these problems as well.<\/p>\n

On the one hand, the bots were helpful. They caught several issues, including a key coding error. Social scientists should run their papers and code through AI and investigate any problems it flags; doing so takes very little time. On the other hand, the AIs missed many problems. And Wiebe notes that he did not test for false positives\u2014cases where the AI identified a problem that didn\u2019t exist. Another\u00a0new study<\/a>\u00a0finds that AI \u201cediting\u201d of text often distorts the meaning. Personally, I can attest to having been falsely accused of a coding error by ChatGPT.<\/p>\n

The lesson here: AI \u201cpeer review\u201d can improve papers, yet you can\u2019t necessarily trust a paper that\u2019s been AI-vetted.<\/p>\n

But what if we take the human-written paper and code out of the equation entirely, and simply give AIs the data and a research question? If they can consistently identify the best methods, apply them, and reach the correct conclusions, that would be a huge advantage. After all, human research teams can do\u00a0markedly different things even when applying the same data to the same question<\/a>, a reality that stems from both ideological bias and differences in methodological choices.<\/p>\n

Unfortunately, a\u00a0new study<\/a>\u00a0finds the same problem in AIs. Working with 150 Claude Code agents from the Sonnet 4.6 and Opus 4.6 families, researchers provided New York Stock Exchange data and asked them to answer questions, such as whether daily trading volume, intraday volatility, and the price impact of trades changed over time.<\/p>\n

Some of these questions produced little variation in results. But others produced huge differences, driven by subtle choices. For example, \u201ctrading volume\u201d can be interpreted as\u00a0dollar\u00a0volume or\u00a0share\u00a0volume, which produced results in the opposite direction. Similarly, the change in volatility depends heavily on whether raw or proportional changes are measured. Strikingly, different versions of Claude even displayed distinct \u201cempirical styles,\u201d favoring particular modeling approaches and ways of measuring variables (such as daily versus monthly).<\/p>\n

Offering the AIs a peer review from another AI prompted some revisions, but did not lead to convergence on similar results. When researchers instead provided examples of top papers on similar questions, the AIs often imitated those methods and converged.<\/p>\n

In other words, like humans, AIs will branch out and do things differently, unless nudged onto the same path. That can be useful if you already know the right path, but it\u2019s a clear limitation if you want reliable results without extensive human steering. After all, it\u2019s precisely human fallibility and bias that make AI appealing in the first place.<\/p>\n

And speaking of human bias,\u00a0yet more new research<\/a>, by the Manhattan Institute\u2019s Jim Manzi,\u00a0traces the ideological orientation of academic work since 1960 (using AI, naturally, to classify articles\u2019 political valence). It finds that \u201croughly 90 percent of politically relevant social-science articles leaned left\u201d over this period; that every discipline leaned left on average; and that each has moved further left since 1990.<\/p>\n

Such findings underscore the need for social science to draw on a wider range of perspectives, and, with careful prompting, AI might help serve that purpose. Yet they also point to a constraint: AI is trained on the existing body of human writing, including biased research, to begin with. As Manhattan Institute reports from David Rozado have demonstrated, AI models often have\u00a0left-leaning ideological priors<\/a>\u00a0and\u00a0exhibit other biases as well<\/a>, such as tending to pick the first of two options given.<\/p>\n

AI can spot errors in human work and generate passable code and prose at remarkable speed, advantages we should not understate. But for now, the technology still makes frequent mistakes, carries its own ideological baggage, and fails to converge on consistent results when different models tackle the same question\u2014unless heavily steered by the very humans whose foibles we hope to escape.<\/p>\n

Robert VerBruggen<\/a>\u00a0is a senior fellow at the Manhattan Institute. He writes the \u201cConcrete Evidence\u201d newsletter for City Journal\u2019s Substack, where this column<\/a> first appeared.<\/p>\n

Photo: Wasan Tita \/ iStock \/ Getty Images Plus<\/p>\n

Donate<\/a><\/p>\n

City Journal is a publication of the Manhattan Institute for Policy Research (MI), a leading free-market think tank. Are you interested in supporting the magazine? As a 501(c)(3) nonprofit, donations in support of MI and City Journal are fully tax-deductible as provided by law (EIN #13-2912529).<\/p>\n","protected":false},"excerpt":{"rendered":"Artificial intelligence can do many of the things that social scientists do. It can analyze data, write and…\n","protected":false},"author":2,"featured_media":176820,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35],"tags":[75,84,83,9,24,63],"class_list":["post-176819","post","type-post","status-publish","format-standard","has-post-thumbnail","category-manhattan","tag-manhattan","tag-manhattan-headlines","tag-manhattan-news","tag-new-york","tag-new-york-city","tag-nyc"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/posts\/176819","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/comments?post=176819"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/posts\/176819\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/media\/176820"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/media?parent=176819"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/categories?post=176819"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ny\/wp-json\/wp\/v2\/tags?post=176819"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}