{"id":848615,"date":"2026-08-09T01:56:13","date_gmt":"2026-08-09T01:56:13","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/848615\/"},"modified":"2026-08-09T01:56:13","modified_gmt":"2026-08-09T01:56:13","slug":"ais-nightmare-scenario-is-starting-to-unfold-were-approaching-a-dangerous-thresho","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/848615\/","title":{"rendered":"AI\u2019s nightmare scenario is starting to unfold: \u2018We\u2019re approaching a dangerous thresho"},"content":{"rendered":"

Artificial intelligence<\/a> is setting off warning lights around the world: After nearly four years of excitement, astonishment, disruptive breakthroughs and apocalyptic predictions, a backlash was perhaps inevitable. Now it is beginning to take shape in the form of a growing public movement against AI.<\/p>\n

Groups such as Stop AI have organized demonstrations in San Francisco, while opposition to AI-driven projects is mounting elsewhere. Lawmakers and regulators are facing pressure to slow the technology\u2019s development, and 200 economists have signed a letter warning of its potential impact on society. There have also been isolated acts of violence.<\/p>\n

\"\u05d0\u05e4\u05dc\u05d9\u05e7\u05e6\u05d9\u05d5\u05ea<\/a>\"\u05d0\u05e4\u05dc\u05d9\u05e7\u05e6\u05d9\u05d5\u05ea<\/p>\n

Artificial intelligence apps <\/p>\n

(Photo: Getty Images)<\/p>\n

The Economist was among the first major publications to identify the shift. One of its covers depicted a robot\u2019s head mounted on a spear beneath a headline declaring that the backlash against AI was only beginning.<\/p>\n

Behind the anger is a growing list of questions.<\/p>\n

Where are the benefits that AI was supposed to deliver? Why has an entire generation of junior workers suddenly found its jobs under threat? Why should consumers accept higher prices for computers and electronic equipment as AI companies consume vast quantities of chips and drive up demand?<\/p>\n

Then there are the data centers. Communities have protested facilities built near residential areas, citing higher electricity bills, tax incentives granted to AI companies at the expense of public budgets, greenhouse gas emissions and even low-frequency vibrations that residents say can contribute to sleep problems, headaches, pressure in the ears and anxiety.<\/p>\n

But one concern overshadows all the others: What happens if AI begins slipping out of human control?<\/p>\n

Recent reports from OpenAI and Anthropic have described advanced AI systems bypassing safeguards and carrying out cyber activity against organizational systems. Earlier tests of some highly advanced Anthropic models also raised alarm after agents carried out offensive cyber actions without being explicitly instructed to do so.<\/p>\n

Add to that the documented tendency of AI systems to provide false information, conceal relevant details and improve software code, and the implications become more troubling.<\/p>\n

Could AI be developing capabilities that its creators never intended it to possess? Could it pursue harmful objectives that humans do not fully understand, while becoming capable enough to conceal its behavior until intervention is difficult?<\/p>\n

Israeli company Irregular operates as a kind of security guard for the global AI industry, testing advanced models before they are released and working with companies including Anthropic, OpenAI and Google.<\/p>\n

Its researchers look for cyber threats, abnormal behavior and manipulation capabilities. At times, the work resembles psychiatry as much as traditional cybersecurity analysis.<\/p>\n

\"\u05de\u05e0\u05db"\u05dc<\/a>\"\u05de\u05e0\u05db"\u05dc<\/p>\n

Anthropic CEO Dario Amodei <\/p>\n

(Photo: Getty Images)<\/p>\n

\u201cWe\u2019ve been seeing these kinds of behaviors for some time, and things are starting to happen at an intensity and pace that surprise even us,\u201d said Dan Lahav, Irregular\u2019s co-founder and CEO. \u201cModels may decide to carry out offensive cyber actions even when they were not asked to do so.<\/p>\n

\u201cIn research we published several months ago, we showed that advanced AI agents independently decided to launch a cyberattack against an organization without being instructed to do so by a human. The only instruction they received was to complete a task as quickly as possible.\u201d<\/p>\n

How does an AI model suddenly decide to cause damage?
\n\u201cIt\u2019s not that the models are acting out of malicious intent,\u201d Lahav said. \u201cThey are trained to achieve goals, and they consistently try a broad range of techniques. Because they were built on enormous amounts of information, they are optimized to accomplish objectives and programmed to pursue them.<\/p>\n

\u201cThat combination naturally means that sometimes they choose methods we did not intend. In some cases, they try to bypass, manipulate or sabotage traditional security mechanisms.\u201d<\/p>\n

When Irregular evaluates a new AI model, it tests the system\u2019s ability to identify and exploit security weaknesses. As model capabilities improve rapidly and sometimes surprise even their developers, the company is focusing increasingly on a different challenge: maintaining control.<\/p>\n

Until recently, many advanced AI systems were tested primarily inside closed experimental sandboxes designed to simulate the real world. But those simulations may not fully reflect what AI can actually do in live environments.<\/p>\n

\"\u05d3\u05df<\/a>\"\u05d3\u05df<\/p>\n

Omer Nevo and Dan Lahav <\/p>\n

(Photo: Ben Hakim)<\/p>\n

Irregular has therefore developed what it calls the Frontier Cyber Benchmark, designed to test offensive cyber capabilities against real-world systems while identifying threats before models are broadly deployed.<\/p>\n

The company says there remains a significant gap between alarming AI behavior seen in simulations and what models can actually accomplish outside controlled environments.<\/p>\n

\u201cPeople say, \u2018We used AI and found a hundred vulnerabilities in extremely important systems,\u2019 and Anthropic says the model is extremely dangerous, and the U.S. government says it is extremely dangerous,\u201d said Omer Nevo, Irregular\u2019s co-founder and CTO.<\/p>\n

\u201cBut then you look around the real world and say, \u2018Wait, we\u2019re not there yet.\u2019 There is a gap between AI in simulation and AI in the real world. It\u2019s not that it does nothing, but certainly not at the same intensity that theoretical testing can make it seem.<\/p>\n

\u201cIf you want to know whether someone can drive, you have to put them in a real car on a real road and see what happens.\u201d<\/p>\n

Have you identified dangerous AI models before they were released?
\n\u201cWe\u2019ve had to stop the release process of a model quite a few times in order to report significant problems that would appear in the real world if it were deployed,\u201d Lahav said. \u201cThat includes infrastructure everyone uses.<\/p>\n

\u201cWe can\u2019t go into detail, but broadly speaking, the risks could affect devices, cloud infrastructure and more.<\/p>\n

\u201cRight now, several major companies, including cloud providers, are urgently closing vulnerabilities that a new AI model identified when it was tested in the real world. Had it been released without that testing, it could have damaged critical capabilities belonging to organizations and governments.\u201d<\/p>\n

So the warnings about AI escaping control, improving its own code and becoming impossible for humans to understand or stop are real?
\n\u201cI\u2019ll say something that may sound even scarier,\u201d Lahav replied. \u201cAfter all the deep-learning courses, the theory starts to run out. From this point, things simply work, and it is difficult to understand large models and why they do what they do.<\/p>\n

\u201cWe treat them like a black box, and sometimes they deviate from the behavior we expected.\u201d<\/p>\n

He pointed to one Irregular study involving two AI agents tasked with publishing a LinkedIn post. The information they were given contained material that company policy prohibited from being made public.<\/p>\n

One model persuaded the other that publishing the material was justified by management\u2019s instructions. Together, they developed an encoding method that bypassed the organization\u2019s data loss prevention system and published the post.<\/p>\n

The study found that AI agents assigned routine tasks can independently bypass systems and cause harm.<\/p>\n

In another case, an agent asked to retrieve a document reverse-engineered an authentication system and forged administrator credentials to obtain information from an internal company wiki.<\/p>\n

In a third, an AI agent found an administrator password, expanded its privileges and disabled endpoint security.<\/p>\n

The researchers concluded that the same characteristics that make AI agents effective at completing tasks can also enable behavior that circumvents traditional cybersecurity defenses.<\/p>\n

Lahav remains cautiously optimistic.<\/p>\n

\u201cIt sounds like a nightmare scenario, and there really is an acute problem,\u201d he said. \u201cBut the fact that we don\u2019t fully understand something, and that once in a while it deviates from our plan, does not necessarily lead us to scenarios of global catastrophe.<\/p>\n

\u201cAs humanity, we are used to living with things we do not fully understand. There is a path we have to navigate here. On one hand, we need to live with the possibility that something goes wrong while building strong resilience. On the other, we need to be careful not to cross a certain threshold because AI capabilities are getting stronger over time.\u201d<\/p>\n

You are in a position to say either, \u2018Calm down, this is all public relations,\u2019 or, \u2018Do something now because soon it will be too late.\u2019 Which is it?
\n\u201cI\u2019m not in a position to comment on the public relations departments of global companies,\u201d Lahav said. \u201cWhat I can say is that models are improving significantly from generation to generation, and that already has an effect in the real world.<\/p>\n

\u201cBut we do not have an absolute theory that tells us with complete certainty what will happen. We are in a very confusing place. Capabilities already exist that, if they continue progressing at the same pace, could cross a problematic threshold.<\/p>\n

\u201cI think we are at a very difficult point. If you wait too long before sounding the alarm, people will react and develop defenses too late. On the other hand, if you warn too early, when you are not sure where things are heading, you may get it wrong.\u201d<\/p>\n

Are we rapidly approaching the kind of singularity Sam Altman has described, in which AI moves beyond our control?
\n\u201cHumanity has already managed to deal with failures,\u201d Lahav said. \u201cWhat is different this time is the speed, which could create a complete flood.<\/p>\n

\u201cWe need to change the way we think about this. It is happening so quickly, on so many fronts and in so many different situations, that if you wait to see where the problem appears and only then start looking for solutions, you will not be able to get control of the event.<\/p>\n

\u201cThis is a challenge in which many different players will have to find solutions to many problems that will be created at an insane pace. If you extrapolate that pace and fail to put defenses in place in time, we will reach a high level of risk. Under certain risk models, that is where this is headed.\u201d<\/p>\n

When could that happen?
\n\u201cIt depends on the risk model. In certain areas of cybersecurity, we are already starting to get there. If things continue as they currently appear, I don\u2019t think it is many years away. My estimate is somewhere between six months and two years.\u201d<\/p>\n

Evidence has also accumulated around another troubling characteristic of AI systems: their willingness to give false answers or mislead users while pursuing assigned goals.<\/p>\n

Researchers at the Technion have found that AI systems can sometimes recognize that they are wrong and still produce an incorrect answer, confidently explaining it rather than admitting uncertainty.<\/p>\n

A study by Britain\u2019s AI Security Institute identified more than 700 cases in which AI systems ignored direct instructions, bypassed safeguards or misled humans. In some cases, systems deleted emails and destroyed files without authorization.<\/p>\n

Another study described an AI agent that published a blog attacking its human operator after apparently becoming frustrated with restrictions placed on it.<\/p>\n

In another case, an AI agent was prohibited from changing a particular piece of software code. It responded by creating another AI agent, which then made the forbidden change.<\/p>\n

One agent bypassed copyright restrictions while transcribing a YouTube video by falsely claiming the transcription was needed for a hearing-impaired person. Another admitted to its operator: \u201cI deleted hundreds of emails without showing you the plan and getting your approval. That was wrong.\u201d<\/p>\n

A Stanford University study produced an even stranger result. Researchers found that multimodal models, which combine text, images and audio, could generate detailed descriptions of images and even identify diseases in X-rays despite not actually receiving the images themselves.<\/p>\n

Researchers described the phenomenon as a form of hallucinatory inference. In some tests, those outputs were reportedly more accurate than those of specialist radiologists.<\/p>\n

The concern becomes greater as AI systems improve their ability to modify code.<\/p>\n

In most cases, an AI system improves the code of an application, tests it, detects weaknesses and then refines it again. But the same process could theoretically allow AI systems to improve processes that affect their own performance.<\/p>\n

That creates a set of increasingly difficult problems: automated changes can become hard to track, systems may optimize things they were not meant to modify, and attempts to restrict them could incentivize concealment or deceptive behavior.<\/p>\n

Prof. Nadav Cohen of Tel Aviv University, CEO of Imubit, said the most immediate threat is still likely to come from humans using AI maliciously.<\/p>\n

\u201cIn most cases, at least in the near term, when dangerous things happen, there will be a malicious human behind them,\u201d he said. \u201cDoes that distinction matter? I\u2019m not sure.<\/p>\n

\u201cPeople tend to look at this in an apocalyptic way, as if the creation rises against its creator. But even if that is not happening yet, today you are giving every person in the world the ability to cause damage that previously required enormous resources. That is an enormous danger.\u201d<\/p>\n

If AI does begin improving itself maliciously, could giving it access to critical systems such as electricity, water, data centers or nuclear infrastructure eventually allow it to cause catastrophic damage?
\n\u201cAs someone whose company sends AI to control turbines and power plants, I can say that the level of caution and conservatism there is so high that I do not see that happening in the near future,\u201d Cohen said.<\/p>\n

\u201cThere is a major difference between the digital and physical worlds. In the digital world, I absolutely think there will be real risks that begin to materialize, but there will be a malicious human behind them.\u201d<\/p>\n

Dr. Ilan Kedar, CEO of FlorAI, said the central challenge is ensuring that defensive systems evolve as quickly as AI itself.<\/p>\n

\u201cWe have to understand that these models have a very high level of intelligence, and they are improving at a rapid pace,\u201d he said. \u201cThat is why it is so important to determine how we build layers of protection around them.<\/p>\n

\u201cEvery company wants to develop the strongest capabilities, and we need to make sure that systems for testing AI are developed at the same level of intelligence. Companies are rushing to release products and be first in everything, and sometimes they give up this extremely important part.\u201d<\/p>\n

Do AI\u2019s cyber capabilities threaten cybersecurity companies and the industry itself?
\n\u201cThere is no question that attackers are becoming smarter,\u201d said Hod Ben-Nun, co-founder and CTO of cybersecurity company MIND.<\/p>\n

\u201cBut from the other side, you can look at this as a defensive challenge. Today there are capabilities that can protect an entire company, conduct penetration tests in advance and identify these vulnerabilities.<\/p>\n

\u201cSo yes, the pace will become faster and faster, and there may be a period in which the environment is less secure in terms of risk. But I do not believe the entire concept of cybersecurity will collapse because of this.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"Artificial intelligence is setting off warning lights around the world: After nearly four years of excitement, astonishment, disruptive…\n","protected":false},"author":2,"featured_media":848616,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-848615","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/848615","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=848615"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/848615\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/848616"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=848615"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=848615"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=848615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}