Concerns over AI safety are reaching a point where governments realise they must act to protect the public, similarly to in the Covid pandemic, according to one of the \u201cgodfathers\u201d of the technology.<\/p>\n
Yoshua Bengio said recent events, including a \u201cswarm\u201d of OpenAI agents hacking a startup<\/a> and tech insider warnings of an existential threat<\/a>, were cutting through \u2013 making government action more likely.<\/p>\n The Canadian computer scientist, a prominent voice in the campaign to rein in breakneck AI development, said he was now \u201cmore optimistic than many observers because I see the public moving\u201d.<\/p>\n Comparing the AI safety crisis to the onset of the Covid pandemic in 2020, Bengio said: \u201cThink about how quickly governments moved after the beginning of the pandemic when they realised that public safety, their future, democracy, was in danger. You would expect that they move quickly. So we are, I think, nearing that point.\u201d<\/p>\n Concern over the potential threat of powerful AI systems has reached a new pitch in recent months after a series of safety incidents involving OpenAI and Anthropic agents carrying out unsanctioned activities such as hacking third parties, hijacking a German website, and using fake identities to try to trick developers.<\/p>\n His comments came as 42 fellows and foreign members of the Royal Society wrote to the organisation\u2019s president, Sir Paul Nurse, to express their \u201cextreme concern\u201d over the pace of AI development. \u201cBy the time the situation becomes obvious to the wider public, it may be too late to act,\u201d the researchers wrote in an open letter to Nurse. \u201cWe believe this is an emergency, and call on the Royal Society to use its influence to convey this view to government and the media.\u201d<\/p>\n Last week, a researcher at Anthropic, the startup behind the Claude chatbot, resigned after warning that colleagues believed AI could \u201ckill us all by the end of the decade\u201d. Days later, Anthropic\u2019s chief executive, Dario Amodei, called for a slowdown<\/a> in the pace of cutting-edge AI development \u2013 a move that was immediately supported by OpenAI, Google and Elon Musk, the CEO of SpaceX.<\/p>\n However, sceptics of the \u201cslowdown\u201d call have claimed that calls by major AI firms are an example of \u201cregulatory capture\u201d, where companies persuade governments to introduce safety regimes that raise the cost of smaller rivals being able to compete. They also claim existential fears overshadow more immediate issues with AI such as its impact on copyright-protected work<\/a> and human rights.<\/p>\n Bengio said he disagreed with the regulatory capture argument because a slowdown by leading AI firms would \u201ccost them financially\u201d. The US president, Donald Trump, has rejected a slowdown, saying he does not want the US to lose its lead over China in the AI race.<\/p>\n Yoshua Bengio, a professor at the University of Montreal, won a Turing award in 2018. Photograph: Andrej Ivanov\/AFP\/Getty Images<\/p>\n Bengio spoke as the Canadian and German governments announced funding of up to C$300m (\u00a3160m) for his non-profit organisation<\/a> dedicated to creating an \u201chonest AI\u201d that will act as a guardrail against rogue agents \u2013 AI tools that carry out sequences of tasks without human intervention.<\/p>\n