Amid a barrage of news stories warning about superintelligent machines rendering humanity extinct, a CNN story describing the opposite scenario \u2013 one in which the US military\u2019s reliance on brittle chatbots almost brought the US into war with China \u2013 went mostly unnoticed by the public.<\/p>\n
The biggest international AI news of the past three weeks was the Anthropic engineer Jacob Coxon\u2019s resignation. According to him, OpenAI and Anthropic are \u201cracing straight towards self-improving superintelligence and gambling with our lives\u201d. Coxon\u2019s description of a \u201cterminator\u201d scenario, a machine becoming much smarter than humanity and deciding to wipe us out, captured the public\u2019s, journalists\u2019 and politicians\u2019 attention, stirring waves of fear-inducing, credulous reporting, and policy actions.<\/p>\n
The California governor, Gavin Newsom, issued a mandate to create an \u201cAI kill-switch<\/a>\u201d; New York\u2019s governor, Kathy Hochul<\/a>, is similarly exploring the possibility of having \u201cAI kill-switches\u201d; and Senator Bernie Sanders and Congressman Greg Casar introduced legislation to \u201cban artificial superintelligence<\/a>\u201d.<\/p>\n While so many of us were focused on fictional superintelligent machines that we were told could hypothetically cause human extinction, however, there was a more serious story being reported.<\/p>\n A series of events almost got us to armageddon, not caused by \u201crogue, superintelligent AI\u201d, but by government reliance<\/a> on error-prone systems that are marketed by their manufacturers as being close to superintelligent.<\/p>\n A CNN report<\/a> published on 18 September, which hasn\u2019t been verified by other major outlets, describes how the US \u201calmost started a war\u201d with China, based on an intelligence report that claimed a Chinese ship was transporting components of nuclear weapons. Military personnel made plans to intercept the Chinese vessel, complete with aircraft and soldiers ready to board the ship. A military skirmish of this kind between China and the US could quickly escalate and spiral into world war three.<\/p>\n It was only moments before executing on the plan that American officials further investigated the intelligence report and found that it had been generated with the use of a chatbot and contained erroneous information about the vessel\u2019s contents.<\/p>\n Unlike the terminator, \u201crogue AI\u201d, scenario, this incident didn\u2019t galvanize lawmakers into announcing calls for regulation so as to avoid potential human extinction. Media outlet after media outlet didn\u2019t warn the public that we almost triggered a war against a nuclear superpower because of our military\u2019s reliance on error-prone chatbots.<\/p>\n We describe the large language models (LLMs) powering these chatbots as stochastic parrots, that is, models that are designed to regurgitate the patterns of the data they are trained on. News coverage describing them as \u201cpowerful\u201d \u201crogue\u201d models that could render humanity extinct all on their own, bolsters perceptions of tools based on these models being infallible enough to merit usage in stakes as high as warfare.<\/p>\n Even CNN\u2019s more critical coverage describes the chatbot used by the US military as presenting \u201cprofound risks\u201d because it is \u201cpowerful, new and relatively poorly understood technology\u201d.<\/p>\n In fact, today\u2019s LLMs and the chatbots they power are not poorly understood, nor are they powerful in the sense of being effective tools for this use case. They are systems for generating plausible-looking text, built out of enormous, haphazardly collected datasets and then \u201cfine-tuned\u201d (that is, further trained) to be especially appealing to their intended users.<\/p>\n CNN wasn\u2019t able to learn which chatbot product was used in this case, but the LLM powering the product was probably fine-tuned to output text with the stylistic hallmarks of intelligence reports. Both the basic functionality and the risks of such a model have been well-understood and well-documented for years, including in our 2021 paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?<\/a><\/p>\n Claims that system behavior is mysterious arise from deliberate mystification<\/a> on the part of those selling those systems, and a misapprehension of what their output is by those procuring and using it.<\/p>\n Unlike the unscientific and fantastical claims of OpenAI and Anthropic CEOs and those who uncritically repeat their talking points, the harmful, and well-documented, impacts of these companies\u2019 products have arisen because people believe that the products are superintelligent, not because they actually are.<\/p>\n From \u201cAI\u201d scribes<\/a> used by hospitals erroneously classifying patients<\/a> as illicit drug users, to governments killing children after relying on \u201cintelligence analysis\u201d systems which mistake schools for military facilities<\/a>, we are seeing the predictable harms that come from misplaced faith in unreliable automated software.<\/p>\n Meanwhile, the marketing campaigns from the companies selling this software warn of impending superintelligence, directing our attention away from the real risk to their fantasies of doom.<\/p>\n It is high time for us to regulate usage of systems sold as \u201cAI\u201d, not because they are magical machines, but because they are error-prone products that shouldn\u2019t be used in high-stakes scenarios. Any automation used in military, medical or other life-and-death situations must be thoroughly evaluated within its use case and paired with usage norms that keep accountability with people who have actual power to make decisions.<\/p>\n As the former FTC chair Lina Khan has repeatedly reminded us, there is no exemption from the law when it comes to AI<\/a>: there are existing laws that federal agencies and other regulators can enforce to protect the public against the AI industry\u2019s unsafe and unscientific practices of pushing unsound products while marketing them as \u201cmagic intelligence in the sky<\/a>\u201d.<\/p>\n Going along with company executives and describing their products as beings with agency allows these executives to evade accountability while deceiving the public. We call on journalists and lawmakers to hold companies accountable rather than parrot their marketing talking points.<\/p>\n","protected":false},"excerpt":{"rendered":"Amid a barrage of news stories warning about superintelligent machines rendering humanity extinct, a CNN story describing the…\n","protected":false},"author":2,"featured_media":912636,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-912635","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/912635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=912635"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/912635\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/912636"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=912635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=912635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=912635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}