\n\tAs the fall semester came to a close, Andrew Heiss, an assistant professor in the\u00a0Department of Public Management and Policy\u00a0at the\u00a0Andrew Young School of Policy Studies\u00a0at\u00a0Georgia State University, was grading coursework from his students when he noticed something alarming. <\/p>\n
\n\tAs is typical for educators these days, Heiss was following up on citations in papers to make sure that they led to real sources \u2014 and weren\u2019t fake references supplied by an AI <\/a>chatbot. Naturally, he caught some of his pupils using generative artificial intelligence<\/a> to cheat: not only can the bots help write the text, they can supply alleged supporting evidence if asked to back up claims, attributing findings to previously published articles. But, as with attorneys who have been caught generating briefs with AI because a model offered false legal precedents<\/a>, students can end up with plausible-sounding footnotes pointing to academic articles and journals that don\u2019t exist.<\/p>\n \n\tThat in itself wasn\u2019t unusual, however. What Heiss came to realize in the course of vetting these papers was that AI-generated citations have now infested the world of professional scholarship, too. Each time he attempted to track down a bogus source in Google Scholar, he saw that dozens of other published articles had relied on findings from slight variations of the same made-up studies and journals. \u00a0<\/p>\n \n\t\u201cThere have been lots of AI-generated articles, and those typically get noticed and retracted quickly,\u201d Heiss tells Rolling Stone. He mentions a paper<\/a> retracted earlier this month, which discussed the potential to improve autism diagnoses with an AI model and included a nonsensical infographic that was itself created with a text-to-image model. \u201cBut this hallucinated journal issue is slightly different,\u201d he says. <\/p>\n \n\tThat\u2019s because articles which include references to nonexistent research material \u2014 the papers that don\u2019t get flagged and retracted for this use of AI, that is \u2014 are themselves being cited in other papers, which effectively launders their erroneous citations. This leads to students and academics (and any large language models they may ask for help) identifying those \u201csources\u201d as reliable without ever confirming their veracity. The more these false citations are unquestioningly repeated from one article to the next, the more the illusion of their authenticity is reinforced. Fake citations have turned into a nightmare for research librarians, who by some estimates are wasting up to 15 percent of their work hours<\/a> responding to requests for nonexistent records that ChatGPT<\/a> or Google Gemini alluded to. <\/p>\n \t\tEditor\u2019s picks<\/p>\n \n\tHeiss also noticed that the AI-generated notes could be convincing to a reader because they included the names of living academics and titles that closely resemble existing literature. In some cases, he found, the citation led him to an actual author, yet the heading of the article and the journal were both fabricated \u2014 they just sounded similar to work the author has published in the past and a real periodical that covers such topics. \u201cThe AI-generated things get propagated into other real things, so students see them cited in real things and assume they\u2019re real, and get confused as to why they lose points for using fake sources when other real sources use them,\u201d he says. \u201cEverything looks real and above-board.\u201d<\/p>\n \n\tSince LLMs have become commonplace tools, academics have warned that they threaten to undermine our grasp on data by flooding the zone with fraudulent content. The psychologist and cognitive scientist Iris van Rooij has argued that the emergence of AI \u201cslop\u201d across scholarly resources portends nothing less than \u201cthe destruction of knowledge<\/a>.\u201d In July, she and others in related fields signed an open letter<\/a> calling on universities to resist the hype and marketing in order to \u201csafeguard higher education<\/a>, critical thinking, expertise, academic freedom, and scientific integrity.\u201d The authors claimed that schools have \u201ccoerced\u201d faculty into using AI or allowing it in their classes, and they asked for a more rigorous, comprehensive analysis of whether it can have any useful role in education at all.<\/p>\n \t\tRelated Content<\/p>\n \n\tAnthony Moser, a software engineer and technologist, was among those who foresaw how chatbots could eventually hollow out educational institutions. \u201cI\u2019m imagining an instructor somewhere making a syllabus with ChatGPT, assigning reading from books that don\u2019t exist,\u201d he wrote in a post<\/a> on Bluesky<\/a> in 2023, less than a year after the AI model first came out. \u201cBut the students don\u2019t notice, because they are asking ChatGPT to summarize the book or write the essay.\u201d This month, Moser reshared<\/a> that post, commenting: \u201cI wish it had taken longer for this to become literally true.\u201d<\/p>\n \n\tMoser tells Rolling Stone that to even claim LLMs \u201challucinate\u201d fictional publications misunderstands the threat they pose to our comprehension of the world, because the term \u201cimplies that it\u2019s different from the normal, correct perception of reality.\u201d But the chatbots are \u201calways\u00a0hallucinating,\u201d he says. \u201cIt\u2019s not a malfunction. A predictive model predicts some text, and maybe it\u2019s accurate, maybe it isn\u2019t, but the process is the same either way. To put it another way: LLMs are\u00a0structurally indifferent to truth.\u201d\u00a0<\/p>\n \n\t\u201cLLMs are pernicious because they\u2019re essentially polluting the information ecosystem upstream,\u201d Moser adds. \u201cNonexistent citations show up in research that\u2019s sloppy or dishonest, and from there get into other papers and articles that cite them, and papers that cite those, and then it\u2019s in the water,\u201d he says, likening this content to like harmful, long-lasting chemicals: \u201chard to trace and hard to filter out, even when you\u2019re trying to avoid it.\u201d Moser calls the problem \u201cthe entirely foreseeable outcome of deliberate choices,\u201d with those who raised objections \u201cignored or overruled.\u201d<\/p>\n \n\tBut AI can\u2019t take all the blame. \u201cBad research isn\u2019t new,\u201d Moser points out. \u201cLLMs have amplified the problem dramatically, but there was already tremendous pressure to publish and produce, and there were many bad papers using questionable or fake data, because higher education has been organized around the production of knowledge-shaped objects, measured in citations, conferences, and grants.\u201d <\/p>\n \n\tCraig Callender, a philosophy professor at the University of California San Diego and president of the Philosophy of Science Association, agrees with that assessment, observing that \u201cthe appearance of legitimacy to non-existent journals is like the logical end product of existing trends.\u201d There are already journals, he explains, that accept spurious articles for profit, or biased ghost-written research meant to benefit the industry that produced it. \u201cThe \u2018swamp\u2019 in scientific publishing is growing,\u201d he says. \u201cMany practices make existing journals [or] articles that aren\u2019t legitimate look legitimate. So the next step to\u00a0non-existent\u00a0journals is horrifying but not too surprising.\u201d<\/p>\n \n\tAdding AI to the mix means that \u201cswamp\u201d is growing fast, Callender says. \u201cFor instance, all of this gets compounded in a nearly irreversible way with AI-assisted Google searches. These searches will only reinforce the appearance that these journals exist, just as they currently reinforce a lot of disinformation.\u201d\u00a0\u00a0<\/p>\n \t\tTrending Stories<\/p>\n \n\tAll of which contributes to a feeling among researchers that they\u2019re being buried in an avalanche of slop, with limited capacity to sift through it. \u201cIt\u2019s been incredibly disheartening for faculty, I think fairly universally, especially as fake content gets accidentally enshrined in public research databases,\u201d says Heiss. \u201cIt\u2019s hard to work back up the citation chain to see where claims originated.\u201d<\/p>\n \n\tOf course, many aren\u2019t even trying to do that \u2014 which is why the phony stuff has been so widely disseminated. It\u2019s almost as if the uncritical and naive adoption of AI has made us more credulous and sapped our critical thinking at the precise moment we should be on guard against its evolving harms. In fact, someone may be toiling away on a (real) study of that phenomenon right now.<\/p>\n","protected":false},"excerpt":{"rendered":"As the fall semester came to a close, Andrew Heiss, an assistant professor in the\u00a0Department of Public Management…\n","protected":false},"author":2,"featured_media":327669,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,4327,733,4308,27409,1398,86,56,54,55],"class_list":{"0":"post-327668","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-ai-slop","10":"tag-artificial-intelligence","11":"tag-artificialintelligence","12":"tag-college","13":"tag-education","14":"tag-technology","15":"tag-uk","16":"tag-united-kingdom","17":"tag-unitedkingdom"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/327668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=327668"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/327668\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/327669"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=327668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=327668"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=327668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}