Penn researchers found that overstating cause-and-effect relationships is a growing problem in the social sciences.<\/p>\n
In an August study that analyzed<\/a> nearly 200,000 papers, researchers at Penn\u2019s Computational Social Science Lab found that nearly half of the surveyed articles made causal overstatements. Their research showed that these overstatements have tripled since the 1980s, and that artificial intelligence models tend to amplify them.<\/p>\n Lead author and fourth-year Ph.D. student at the Annenberg School for Communication Calvin Isch said in an interview that correlation refers to when \u201ctwo measures go together,\u201d but one might not directly cause another.<\/p>\n For example, he explained, an increase in shark attacks might be correlated with an increase in ice cream sales, but shark attacks do not cause ice cream sales, or vice versa. \u00a0<\/p>\n \u201cThese are two things that might go together because whenever it\u2019s warm out, people go to the beach and they buy more ice cream, but they also are at the beach and there can be more shark attacks,\u201d he added. \u201cThese are two things that might be correlated, but they\u2019re not in any way causally related.\u201d\u00a0<\/p>\n Get the DP delivered straight to your inbox.<\/p>\n Sign Up<\/a><\/p>\n The research team at the Computational Social Science Lab, which is directed by Annenberg School professor Duncan Watts, looked at 194,631 observational, cross-sectional studies \u2014 research that collects data only at a single time point.<\/p>\n The group showed that 46% of these studies had some form of causal overstatement in their abstracts or titles. According to the study, the percentage rose from roughly 20% in the 1980s to 60% in 2024.\u00a0<\/p>\n Isch argued that overstating causal relationships in scientific articles is a form of \u201cnarrative license\u201d that departs<\/a> from \u201cevidence to create a more compelling story.\u201d<\/p>\n \u201cIf I put a lot of money into a thing that I think is causally connected, but it\u2019s not, it\u2019s just wasted money, and often it can come with other consequences,\u201d he said, adding that \u201cthat\u2019s really why it matters in the social sciences.\u201d<\/p>\n \u201cMost of these domains are investigating these things with societal relevance, and people will be looking to them for practical advice about how we can do better policies,\u201d Isch added.<\/p>\n The researchers also experimented with AI large language model summaries of articles using various chatbots, and found that specific prompting strategies can increase causal overstatements.<\/p>\n According to Isch, when researchers asked an LLM to explain hard-to-understand concepts \u201cat an eighth-grade level,\u201d the model was \u201cmore likely to make a causal claim by a great deal.\u201d\u00a0<\/p>\n He explained that adding a \u201cguardrail prompt,\u201d such as \u201cmake sure that all the claims you have are proportional to the evidence,\u201d can help reduce this issue.<\/p>\n While Isch acknowledged that the researchers could not determine exactly what drove the increase in causal overstatements over time, he speculated that it may be explained by how \u201cin the late 1990s, early 2000s, the NSF and the NIH changed the way that they rewarded funding to include impact-oriented science.\u201d<\/p>\n \u201cIf you want to improve the world, it\u2019s good that we\u2019re doing science that is going to improve the world,\u201d he added. \u201cIt adds an incentive for scholars to state that their findings are causing impact, even if they might not be, right?\u201d<\/p>\n In its most recent strategic plan<\/a>, the National Science Foundation also stated that it \u201cwill review and revise its funding opportunities to enhance impact\u201d and phase out lower-priority opportunities.<\/p>\n Isch added that University incentives based on \u201cthe number of publications that you get and the prestige of the journals they get in\u201d might also contribute to this increase in overclaiming.\u00a0<\/p>\n Now, Isch\u2019s team plans to broaden their investigations to encompasses other study designs and research fields such as medicine, where it might be that an \u201carticle itself looked at rats and mice, but the abstract does not say that \u2014 it looks like it\u2019s applying to humans.\u201d<\/p>\n Based on his findings, Isch emphasized that readers of social research should always \u201ccheck their own priors.\u201d<\/p>\n \u201cI would just say as you\u2019re approaching scholarship, be really willing to interrogate the methods as best you can, or even ask language models to interrogate the methods for you,\u201d he said.<\/p>\n Staff reporter Can Doga Bolukbasi covers science and health and can be reached at bolukbasi@thedp.com<\/a>. At Penn, he studies biochemistry. Follow him on X @cdbolukbasi<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"Penn researchers found that overstating cause-and-effect relationships is a growing problem in the social sciences. In an August…\n","protected":false},"author":2,"featured_media":285522,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[69,71,70],"class_list":["post-285521","post","type-post","status-publish","format-standard","has-post-thumbnail","category-philadelphia","tag-philadelphia","tag-philadelphia-headlines","tag-philadelphia-news"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts\/285521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/comments?post=285521"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts\/285521\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/media\/285522"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/media?parent=285521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/categories?post=285521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/tags?post=285521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}