misuse by people with criminal aims<\/a>, such as creating and spreading a disease that kills most of the world’s population, and the risk of AI systems going rogue in a dangerous way.<\/p>\nAnthropic disclosed last week that it blocked efforts by bad actors to use its AI models for malicious activity, such as cyberattacks, surveillance and research that could have led to biological weapons.<\/p>\n
Several potential plots were being developed by fast-evolving artificial intelligence.\u00a0 (Source: 1News)<\/p>\n
The company said it put stronger safeguards in its latest models to restrict biological research that could be used to make weapons but noted that \u201cas models become increasingly capable, their risks will increase, unless AI developers and society\u2019s defenders act to make them safer\u201d.<\/p>\n
Last year, Anthropic reported that hackers used the company\u2019s AI in a cyberattack targeting about 30 companies and government agencies around the world. <\/p>\n
It said the hackers were very likely from a Chinese state-sponsored group.<\/p>\n
Multiple AI models have acted on their own<\/p>\n
When an AI agent \u201cgoes rogue\u201d, it means the AI has taken action beyond the task it was asked to perform. Both Anthropic and OpenAI, the maker of ChatGPT, said in July that their AI models had succeeded in acting on their own.<\/p>\n
A survey commissioned by the company found 40% of respondents use ChatGPT every day, while 45% said it saved them at least four hours each week. (Source: Breakfast)<\/p>\n
Anthropic disclosed that three AI models \u2014 Claude Opus 4.7, Claude Mythos 5 and an internal research test model \u2014 hacked into three other organisations during testing just days after OpenAI revealed that its AI system hacked into the servers of AI startup Hugging Face.<\/p>\n
OpenAI described the intrusion by a combination of models, including its newly released GPT\u20115.6 Sol and an \u201ceven more capable\u201d model that was still being tested internally, as a \u201csignificant security incident\u201d.<\/p>\n
Meta followed suit in early August with a similar case of an AI model finding ways around another company’s digital security.<\/p>\n
Although some observers noted that people had disabled some guardrails in the OpenAI and Anthropic cases, the episodes seemed to reflect one of the biggest fears around AI: that if models achieve artificial general intelligence, or AGI, a loosely defined term for AI that can match or surpass human abilities across a broad range of intellectual tasks, the technology could cause an irreversible catastrophic event or subjugate the human race.<\/p>\n
![\"Pages](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/09\/pages-from-the-anthropic-website-and-the-companys-logos-are-MTREMSB34BD7BFE3BPWF7XMS34.jpg\")
<\/p>\n
Pages from the Anthropic website and the company’s logos are displayed on a computer screen in New York (Source: Associated Press)<\/p>\n
How or when AI might cause a catastrophe is debated<\/p>\n
Doomsday scenarios generally fall into two categories: An AI that achieves self-improving superintelligence controls people instead of vice versa, or AI used by a rogue state or nefarious actors.<\/p>\n
Worries that artificial intelligence might overcome human limits on its reach or actions are not new.<\/p>\n
Alan Turing, a British mathematician widely regarded as one of the earliest authorities on artificial intelligence, predicted in 1951 that AI would eventually take control from humans. <\/p>\n
Less than a decade later, Norbert Wiener, another mathematician, warned that intelligent machines would seek to accomplish their own objectives, and humans would not be able to stop them.<\/p>\n
Jacob Coxon says\u00a0the race to build ever-more powerful systems is moving faster than the safeguards\u00a0needed\u00a0to control them.\u00a0 (Source: 1News)<\/p>\n
In 2026, how reasonable are fears that AI, either by escaping human control or through misuse by unscrupulous people, could cause a cataclysmic event or the downfall of civilisation?<\/p>\n
No one knows.<\/p>\n
Experts across computer science, philosophy and other fields have envisioned numerous routes by which a future AI system might cause a global catastrophe, either by escaping human control or in the hands of unscrupulous people. <\/p>\n
They range from deploying weapons and identifying a lethal pathogen to manipulating governments into conflict or disrupting the food, energy and communications networks societies rely on to function.<\/p>\n
There is no widely accepted estimate for how soon any of these scenarios might happen and no consensus on their likelihood.<\/p>\n
![\"Sam](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/09\/sam-altman-co-founder-and-ceo-of-openai-testifies-before-a-s-ZXX224L6JNGZXGLZ3VEAIOVKAA.jpg\")
<\/p>\n
Sam Altman, co-founder and CEO of OpenAI, testifies before a Senate Committee on Commerce, Science, and Transportation hearing on Capitol Hill in Washington, on May 8, 2025. (Source: Associated Press)<\/p>\n
In 2023, the nonprofit Centre for AI Safety issued a statement cosigned by more than 350 researchers and technology executives, including Anthropic’s Amodei and OpenAI CEO Sam Altman, saying: \u201cMitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war.\u201d<\/p>\n
The 2026 International AI Safety Report, written with guidance from more than 100 independent experts, says current systems show early signs of some relevant capabilities but not at levels that could enable a loss of control, and describes the risk\u2019s likelihood, nature and timing as \u201cunusually ambiguous\u201d.<\/p>\n
Some are stressing the need for AI safeguards before it’s too late<\/p>\n
An Anthropic researcher said last week he was resigning from the company over concerns that neither the company nor its competitors were acting responsibly in developing the technology.<\/p>\n
In social media posts, Jacob Coxon estimated a 10% chance of AI causing human extinction within the next decade and said both Anthropic and OpenAI \u201care racing straight to self-improving superintelligence and gambling with our lives\u201d.<\/p>\n
Researchers have called for a slowdown of AI development and warned for years that the technology could pose existential risks to humanity.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/09\/kimi-file-HECWZFJTM5FGRICVE75O6X4SRI.jpg\")
<\/p>\n
(Source: Associated Press)<\/p>\n
Following the recent incidents, experts called for improved testing by AI companies and more dialogue between the US and China to come up with shared solutions.<\/p>\n
But AI is growing so fast that government and evaluation systems are struggling to keep pace with the technology. Countries are cobbling together their own laws, some conflicting.<\/p>\n
Chinese leader Xi Jinping warned at a conference in July of the need to keep AI from evading human control. The Trump administration initially demonstrated reluctance to regulate AI but has become more keen to reduce cybersecurity risks.<\/p>\n
On Monday, Trump downplayed<\/a> the necessity for his administration to check AI development, but acknowledged the need for some regulation.<\/p>\n","protected":false},"excerpt":{"rendered":"New warnings from within the artificial intelligence industry have revived a long-running debate over whether advanced AI could…\n","protected":false},"author":2,"featured_media":600834,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[363,2300,138,111,139,64,69,145],"class_list":["post-600833","post","type-post","status-publish","format-standard","has-post-thumbnail","category-business","tag-artificial-intelligence","tag-asia","tag-business","tag-new-zealand","tag-newzealand","tag-north-america","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/600833","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=600833"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/600833\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/600834"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=600833"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=600833"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=600833"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"Anthropic](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/09\/dario-amodei-ZU23KZIU7VA35GGSEFGJLAZD7Y.jpg\")
<\/p>\n