Sign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
There are a litany of reasons to take the output of generative AI with a heavy dose of salt. Its answers are known to sound authoritative, convincing, and elaborate \u2014 while simultaneously being completely and utterly wrong<\/a>.<\/p>\n It\u2019s a lesson astrophysicist and science communicator Paul Sutter learned the hard way, and was brave enough to share with the world. In a article for the publication Nautilus<\/a> earlier this month, Sutter retold a truly harrowing story that feels like something yanked straight out of a stress-induced nightmare.<\/p>\n In February, he gave a presentation in front of a room full of \u201ccollaborators,\u201d revealing a new update for his algorithm designed to spot voids<\/a>, or empty regions between galaxies.<\/p>\n The new version was \u201cten times faster than the old one\u201d had a \u201cmore sophisticated way of dealing with the ugly realities of an actual data set\u201d and could \u201chandle surveys a hundred times larger,\u201d Sutter recalled.<\/p>\n It was also in part the result of extensive sessions consulting an AI to write the update\u2019s code \u2014 \u201cvibe coding,\u201d in the parlance \u2014 which he recalled as \u201cindispensable.\u201d<\/p>\n But just ten minutes into his presentation, a collaborator interjected telling that \u201csomething seemed off to them.\u201d As it turns out, the way the new algorithm handled the edges of the surveys was \u201cwrong,\u201d Sutter admitted.<\/p>\n \u201cIt wasn\u2019t a typo, and it wasn\u2019t a missing citation or a factor of two,\u201d he wrote. \u201cIt was subtle, but it was very wrong, and everything downstream of it was also wrong, and I had shared the whole thing in a room full of people who trusted me.\u201d<\/p>\n Sutter\u2019s cringe-inducing tale perfectly illustrates how many users of AI tools are putting themselves at risk without knowing it, including in the most rarefied echelons of academia. The science world has been inundated with poorly-researched and often unedited AI slop, a major reckoning forcing academics to be responsible for everything they publish<\/a> \u2014 including any potentially embarrassing hallucinations.<\/p>\n The AI coding tool Sutter was using \u201csounded like it understood,\u201d he wrote, despite being \u201clittle more than a sophisticated next-word predictor.\u201d<\/p>\n \u201cAn LLM\u2019s fluency is not an accident, and it is not an emergent mystery,\u201d Sutter wrote. \u201cIt\u2019s a trait we bred, the way we bred wolves into dogs that watch our faces when we open the treat bag.\u201d<\/p>\n \u201cSo how do we deploy a tool that is sometimes wrong but always pleasing? How do we trust AI?\u201d he concluded. \u201cThe simple answer is: don\u2019t.\u201d<\/p>\n Sutter likened our extensive use of AI tools to alchemy, an ancient practice that was devised by what he calls \u201cpre-scientists.\u201d <\/p>\n \u201cWe are now in the pre-chemistry era of AI,\u201d the astrophysicist declared. \u201cThe crucible is closed, and like the alchemists we are not going to stop using it.\u201d<\/p>\n To get to the truth, he says, we must carefully trace an AI\u2019s chain of reasoning and audit its every output. Following his embarrassing slip-up in February, Sutter vowed that he works \u201cdifferently now\u201d and is ready to immediately \u201cdistrust\u201d anything an AI says.<\/p>\n It\u2019s a cautionary tale that even some of the most gifted thinkers can easily be tempted by the allure of AI.<\/p>\n As such, when we pasted his latest piece for Nautilus into AI detecting tool Pangram \u2014\u00a0which is far from perfect \u2014 it informed us that 56 percent of the text appeared to be written by an AI.<\/p>\n