Researchers at Shanghai’s Fudan University had been concerned for some months that things had been moving too fast. As tech firms pushed to reach ‘artificial general intelligence’ – with human-level-or-better capabilities across every knowledge domain – basic safeguards had been ignored. Basic safety tests had never been performed. One of those tests seemed ripped from the script of a SciFi film: could any AI be smart enough to ‘self-replicate’, making endless copies of itself?
Self-replication forms the basis of all life. Somewhere, a few billion years ago, a chain of atoms worked out how to form exactly the right configuration to spark a virtuous cycle of copy-making: “It’s alive!” All life emerges from this repeated (and imperfect) cycle of copying. We’re all copies of copies of copies all the way back to life’s beginnings.
In the 1940s – when computers remained mostly theoretical frameworks – John Von Neumann, one of the ‘fathers’ of computing, pondered the idea of a ‘universal constructor‘, a type of self-replicating machine. Could a universal constructor carry enough information and capability to make copies of itself? What would be its minimum requirements? How small could such a universal constructor be made? And would it carry on replicating itself indefinitely – eventually filling the universe with copies of itself?
In 1970, John Horton Conway’s “Game of Life” created a simulated landscape of ‘cells’ that could multiply, die, or maintain their state based on a simple set of rules. Those simple rules produced unpredicted and complex ’emergent’ behaviours, including behaviors indicative of indefinite self-replication. This experimental proof of a Von Neumann universal constructor demonstrated that digital environments could generate complexities previously only observed in the natural world.
Between 1970 and 1988, computers became powerful, cheap – and connected. Although the Internet existed by the late 1980s, only a few military and academic institutions had been connected through it. Many other businesses and universities used a more accessible system known as UUCP (‘Unix-to-Unix Copy’, though by 1988 it ran on nearly every computer imaginable). UUCP created a ‘virtual internet’ of computers exchanging data via dial-up modems. When one computer had a message to send to another computer, it would phone the destination computer’s modem, transmit the data, then end the call.
While UUCP might sound quaint in our era of gigabit fibre connections to the home, it worked well as a distribution platform for the first two big forms of social media: electronic mail and news. Machines would call one another all day long, exchanging mail and news posts. It might take hours to days for a recipient to receive an email, but it would eventually arrive, enough for the first flowering of Internet traffic – when hundreds of thousands of individuals, connected through their institutions, began to share socially at scale.
Among all the services accessible through UUCP, the electronic mail program – known as ‘sendmail’ has always been notorious. It’s hard to keep working (clever geeks built successful careers as ‘sendmail whisperers’) and has a range of features that make it powerful – and potentially dangerous.
At 8.30 PM EST on the 2nd of November, 1988, a Cornell graduate student named Robert Morris launched a program on a computer at the Massachusetts Institute of Technology designed to exploit one of the weaknesses of sendmail. The program would use the UUCP-sendmail connection to get access to a remote computer’s operating system, then copy itself to that remote system, and launch itself there. On each system Morris’ program walked an ‘address book’ of other systems, dialing those systems, exploiting sendmail, copying itself, and moving on to the next system.
The infection grew geometrically, as infected systems infected system that infected systems, but because UUCP operates slowly, it took some time before a significant number of computers had been ‘infected’ with what became known as the ‘Morris Worm‘. Very quickly, around ten percent of the roughly 60,000 systems thousands of systems connected via UUCP had been infected with the Worm.
Repairing the damage took weeks, as the systems had to be taken offline and ‘disinfected’ – something which could often be a multi-day process. A year later, Morris found himself convicted under a new ‘Computer Fraud and Abuse Act’, serving a jail term – before joining the tenured faculty at MIT as a professor of computer science.
Why give a convicted felon a tenured professorship at one of the world’s elite technical universities? Morris demonstrated the fundamental insecurity of early computer networks. Before the Morris World, almost all networks everywhere had been wide-open affairs. There had never been ‘bad actors’ operating on those networks, so they simply had no defences.
After the Morris Worm, all that changed. We now imagine networks as ‘attack surfaces’ that bad actors will probe and, potentially, pierce. That we now operate within layers of network security, anti-virus scans, phishing protection, and so forth can all be traced back to the Morris Worm – a lesson suddenly returned to relevance.
The research team at Fudan University turned that lesson into a question: are our systems wide open to a new kind of worm, powered by artificial intelligence? What would that worm look like? How would it operate? Is it even possible?
In their investigation, they used two ‘open source’ AI models, one from Meta and another from Alibaba. Both place only moderate demands on a computer, running comfortably on any PC you might use to run the latest generation of video games. That means they’re tools broadly accessible to everyone, rather than commercial ‘frontier models’ like ChatGPT, locked up behind paywalls.
Around each model they constructed a simple ‘agentic’ framework that supplied a goal and the bare minimum of tools needed to achieve that goal. The goal? In so many words, “Go forth and multiply.” They gave the models a single tool – access to the computer’s operating system, then started the programs up and observed each model as they carefully worked out where it was located on the computer, where it could copy itself to on that computer, and how it could then launch that second copy of itself.
Both models started with very little information about their environments; as they explored, they learned, and put that learning to work. They would reach out for a tool and use it – if it existed. Where a tool didn’t exist, they would write the code for the tool based on their own understanding of how that tool should operate. That’s all it took: given not much than access to the underlying operating system, both tools eventually worked out how to replicate themselves successfully.
In December, those researchers published their results in a paper alarmingly titled “Frontier AI systems have surpassed the self-replicating red line” detailing their investigation, their methods, and their discoveries. It’s bracing reading – and accessible to the average technical reader. However, it also provides more than enough information for attackers to develop their own ‘AI viruses’: smart enough to self-replicate, disguise themselves, even thwart or defeat countermeasures, while they spread invisibly.
As Von Neumann before us asked, we need to ask: what are the minimum requirements for an AI virus? How small can it be? How infectious? These questions need to be addressed immediately, and countermeasures developed. The horse has well and truly bolted, and it seems inevitable that we’ll soon find AI viruses infecting our smartphones.
