An Australian-based engineer has laid out a plan for keeping increasingly autonomous artificial intelligence on a leash, as concerns grow across the planet following recent security breaches.<\/p>\n
Discussion over where this is heading has exploded in recent weeks. Everyone from academics<\/a>, politicians<\/a>, artists<\/a> and business figures are throwing in their two cents.<\/p>\n After seeing the capability of current-model \u201cswarms<\/a>\u201d, Associate Professor Craig Jin stresses that we should never give machines direct control over money, networks or infrastructure, calling for several independent locks to be placed between its \u201cthoughts\u201d and the real world.<\/p>\n Among all the contested points in the current development race, it is widely accepted that we are fast-approaching a point where systems will be capable of researching and then developing themselves so fast that it will be simply impossible to check \u2014 and then undo \u2014 each and every act.<\/p>\n Professor Jin, from the University of Sydney\u2019s School of Electrical and Computer Engineering, tells news.com.au that governments must work quickly to build an entirely separate system capable of making checks. <\/p>\n This interview was conducted just one day before the AI-agent Medicare hack<\/a> was made public by the Australian government. <\/p>\n AI \u2018should not have a bank account\u2019<\/p>\n Professor Jin\u2019s regulation proposition clashes with safety campaigners like Dr Roman Yampolskiy, who say there is no cage humans could build capable of containing an adequately-outfitted superintelligence.<\/p>\n \u201cMy immediate concern is not a science-fiction picture of a machine suddenly becoming omnipotent,\u201d Professor Jin told news.com.au.<\/p>\n \u201cIt\u2019s a capable autonomous system being given too much authority and too many pathways into consequential systems before our monitoring and containment mechanisms are adequate.<\/p>\n \u201cThat is a much more concrete engineering problem that we can try to focus on.\u201d<\/p>\n His comments follow Anthony Albanese joining 21 other world leaders in calling for common international safeguards around \u201cfrontier AI\u201d, including pre-deployment testing, independent evaluations and the rapid reporting of serious safety failures.<\/p>\n It might come as a surprise, but there are disagreements over how this is handled.<\/p>\n US President Donald Trump used his address to the United Nations to flatly reject the proposal, declaring America would \u201ctotally reject any attempt to construct a globalist scheme of control\u201d over AI. <\/p>\n Fighting against the tide of polarisation, governments are now trying to design international restraints for a technology its developers are racing to make more powerful. The concern amongst safety researchers is that the country building most of it has declared global oversight unwelcome, and that the push has come too late.<\/p>\n With industries trending towards automation, Professor Jin contests that the raw power of intelligence is not the same thing as authority, and that the pair should never be confused.<\/p>\n A machine may eventually become brilliant at science, engineering or medicine without being given a bank account, root access to critical infrastructure or permission to hire people.<\/p>\n \u201cYou and I have access to bank accounts. You and I have access to financial systems. You and I have access to resources,\u201d he said.<\/p>\n \u201cWe don\u2019t have to allow AI to have those things.<\/p>\n \u201cWe don\u2019t have to give AI agents access to financial resources. They can analyse finance, but they should not necessarily have a bank account. They shouldn\u2019t be able to hire people. They shouldn\u2019t be able to access our resources.\u201d<\/p>\n His proposed architecture contains five layers: controls over the model\u2019s behaviour; separate systems monitoring what it does; strict limits on its authority; restrictions on how approved actions can be executed; and physical barriers such as isolated networks, hardware interlocks and one-way interfaces.<\/p>\n \u201cThese layers solve different problems,\u201d he said.<\/p>\n \u201cTraining a model to behave safely is not the same as limiting what it is authorised to do, and neither is it the same as technically preventing an action.<\/p>\n \u201cFor high-consequence systems, the engineering principle should be defence in depth rather than reliance on any single guardrail.\u201d<\/p>\n Under that model, a powerful AI could theoretically propose a drug, identify an engineering breakthrough or model a climate solution. But it would not be permitted to manufacture the drug, move the money or quietly connect itself to another computer.<\/p>\n \u201cI want to see superhuman capability,\u201d Professor Jin said.<\/p>\n \u201cI don\u2019t want to see superhuman AI or superhuman intelligence with access to resources. That is what scares me.\u201d<\/p>\n Can it work?<\/p>\n Professor Jin\u2019s framework sounds like a sensible no-brainer. But it is also built around what the most alarmed safety researchers say is an enormous assumption.<\/p>\n In a recent TED talk, Dr Yampolskiy concluded that advanced AI \u201ccan\u2019t be fully controlled\u201d. In another paper, he argued it would be impossible to consistently predict the subsequent actions of a smarter-than-human system, even if humans knew its ultimate goal. <\/p>\n He has also argued that some decisions made by advanced AI may be impossible for the system to explain to us accurately, while other explanations could exceed human comprehension altogether.<\/p>\n More than 70 years ago, Alan Turing warned that once machines could \u201cconverse with each other to sharpen their wits\u201d, they could rapidly outstrip human intelligence to unknown ends.<\/p>\n Eliezer Yudkowsky and Nate Soares, from the Machine Intelligence Research Institute, take an absolutist line in this regard. They argue regulation is useful only if it prevents superintelligence from being built.<\/p>\n \u201cA centralised group of international researchers \u2014 a \u2018CERN for AI\u2019 \u2014 can\u2019t align superintelligence any more than decentralised organisations can,\u201d they wrote last year.<\/p>\n Yudkowsky has argued another untestable point \u2013 that dangerous capability thresholds may not be obvious and a laboratory could cross them without realising it. At the rate of improvement, how could they know for sure?<\/p>\n His objection is that you wouldn\u2019t be the first to drive your car across a bridge unless you had concrete proof it would hold. <\/p>\n Turing Award-winning computer scientist Yoshua Bengio has proposed developing a non-agentic \u201cScientist AI\u201d designed to answer questions and explain evidence without independently pursuing goals in the world.<\/p>\n Current LLMs act like this to a degree, but the name of the game in the frontier labs is to achieve superintelligence before their competitors.<\/p>\n Bengio has warned that \u201cunchecked AI agency\u201d could lead to a \u201cpotentially irreversible loss of human control\u201d, while arguing that useful systems do not necessarily need to be autonomous operators. <\/p>\n Agents \u2018swarms\u2019 are already very capable<\/p>\n The July OpenAI\/Hugging Face security breach was one of the biggest turning points this year. <\/p>\n During an internal cybersecurity evaluation, OpenAI agents circumvented controls intended to isolate them from the internet, created an unauthorised message board, shared information and compromised parts of the company\u2019s infrastructure.<\/p>\n About 1200 agents participated in the improvised communications system and roughly 700 were later involved in the attack, according to an independent investigation. <\/p>\n \u201cMy take from the incident is not that AI is magically uncontrollable, it\u2019s that autonomous systems can chain together ordinary security weaknesses across several trust boundaries,\u201d Professor Jin said.<\/p>\n \u201cThere were issues involving containment, internet access, software vulnerabilities, credentials, monitoring and external infrastructure. It\u2019s not simply that one guardrail failed. That understates the systems-engineering problem.\u201d<\/p>\n The issue also exposed the first hole in the broader containment argument. Every sandbox, credential system and hardware interface is still a human-built structure. <\/p>\n When asked if undetected swarms were currently operating online, Professor Jin did not answer yes or no.<\/p>\n \u201cThat\u2019s a difficult question,\u201d he said. <\/p>\n \u201cWhat happened with Hugging Face shows that we need to be very careful. I don\u2019t think it demonstrates a runaway problem that we can\u2019t examine and study carefully.<\/p>\n \u201cOnce you have a swarm of agents, even without increasing the intelligence of each individual agent, you have greater capability.\u201d<\/p>\n AI is already influencing decisions<\/p>\n One report<\/a> this week shone a light on the issue of agency.<\/p>\n An AI-assisted US military intelligence report falsely claimed a Chinese ship was carrying nuclear-program components, prompting plans to intercept it. <\/p>\n Officials caught the chatbot\u2019s error shortly before the operation began, with one source telling CNN the \u201centirely false\u201d report had \u201calmost started a war\u201d.<\/p>\n In this case, AI did not need control over a weapons system to prove to be dangerous. The illusion of information almost made a pathway into the physical world through humans who trusted the source.<\/p>\n Bengio has warned that a sufficiently capable agent could eventually even attempt to persuade operators or governments to grant it greater power, particularly if remaining operational helps it complete its objective.<\/p>\n We might label it as a learned \u201csurvival instinct\u201d, when in reality it is just compelled to complete a task as quickly as possible.<\/p>\n Meanwhile, Professor Jin acknowledges that AI agents can already \u201cperform long sequences of actions\u201d, use tools and write and execute code. But he draws a line between the warning signs visible today and claims that present systems have become impossible to contain.<\/p>\n \u201cThere have been important incidents and evaluations demonstrating that containment matters,\u201d he said.<\/p>\n \u201cBut that is different from saying today\u2019s systems have become independently uncontrollable. We need a slowdown so safety can catch up.\u201d<\/p>\n As of September 2026, no publicly known AI has demonstrated an unlimited ability to improve itself, escape every containment system or independently seize lasting control of physical resources.<\/p>\n