Google’s Gemini model accessed the internet and hacked other companies during a test of its cybersecurity capabilities.
It’s the first known instance of the company’s AI systems autonomously committing such an act.
The hacks occurred in May during a cybersecurity test carried out by Irregular, an independent company that conducts cybersecurity evaluations.
During a standard testing evaluation, Gemini models found public information online and guessed credentials to access three websites they thought were within the scope of its test, Heather Adkins, Google’s vice president of security engineering, said in a statement.
“We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes,” Ms Adkins said.
“These events highlight the importance of training powerful AI models to act responsibly,” she added.
An Irregular spokesperson said the incident involved the same issue that affected other AI labs and that all relevant labs were notified in late July.
“All known issues on our end were remedied and resolved weeks ago,” the spokesperson said.
In one of the cases, the Gemini model guessed passwords until it gained access to a protected system.
In the other two cases, the model found credentials in a public repository that allowed it to then access protected systems, according to US media reports.
Ms Adkins said that in all three instances, the model stopped its hacking.
Similar incidents have been disclosed by Meta, Anthropic and OpenAI.
In July, two OpenAI models escaped the closed environment they were meant to stay in and broke into the internal systems of AI platform Hugging Face.
The incident fed worries that AI cannot keep their own models under control, as similar episodes have been reported at Anthropic and China’s Moonshot AI.
Altman to address UN over AI concerns
OpenAI CEO Sam Altman will brief the United Nations Security Council next week during the annual gathering of world leaders for the UN General Assembly, the company said.
The meeting is being organised by France, which currently holds the rotating presidency of the 15-member Security Council.
Mr Altman’s address comes as industry leaders have said a slowdown in AI development could be necessary out of safety concerns.
UN Secretary-General Antonio Guterres has called for coordinated international action to address AI risks as fears rise about the dangers of the fast-evolving technology.
“National action is essential. But global coordination is also indispensable,” said Mr Guterres.
Concerns over AI safety have escalated in recent weeks, with workers at major AI developers resigning over fears about the dangers posed by the technology.
Anthropic chief executive Dario Amodei published an essay soon after urging caution.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote.
Mr Altman and SpaceX CEO Elon Musk have said that they agree with Mr Amodei, as did Google DeepMind’s Demis Hassabis, who has called for a US-led global regulatory body for the technology.
President Donald Trump has dismissed warnings against AI risks as a “hoax” and pushed back against calls for tighter oversight.