Everyone wants safe AI but who will be first to take action?

Analysis: AI has become a race where all involved can see the risks, but nobody wants to be the first to apply the brakes and pay a price for doing so

There is an uncomfortable contradiction at the heart of the AI boom. The companies developing the world’s most powerful AI systems repeatedly tell us that safety is fundamental to what they do. They publish frameworks, conduct evaluations, employ specialist safety teams and warn publicly about the risks increasingly capable systems might create. At the same time, those companies are engaged in one of the fiercest technological races in modern history.

That tension became visible when Jacob Coxon, a 27-year-old researcher who has worked at both OpenAI and Anthropic, resigned from the latter and accused both companies of behaving irresponsibly. He claimed they were “racing straight to self-improving superintelligence and gambling with our lives”.

We need your consent to load this rte-player contentWe use rte-player to manage extra content that can set cookies on your device and collect data about your activity. Please review their details and accept them to load the content.Manage Preferences

From RTÉ Radio 1’s Drivetime, is humanity really in threat due to AI?

Coxon’s concerns are at the extreme end of the debate, including the possibility that sufficiently powerful AI could eventually escape meaningful human control. Whether or not we accept those predictions, his resignation raises a much less speculative question: What happens when developing AI responsibly means developing it more slowly than your competitors?

Imagine one leading AI company discovers something concerning during testing of its next model. It could postpone release, conduct additional research, commission independent evaluations or develop stronger safeguards, which sounds like responsible behaviour.

But its competitors do not disappear while it does so. They continue training models, releasing products, attracting customers and recruiting researchers. The company exercising caution potentially pays a commercial price for doing so. This begins to resemble the classic prisoner’s dilemma: everyone might collectively benefit from greater caution, but individually each participant has an incentive to keep moving because they cannot guarantee their competitors will do the same.

From RTÉ Radio 1’s Drivetime, Prof Alan Smeaton from DCU on Anthropic’s calls for a pause on AI development

There is a genuine desire for regulation in this space. 1,200 AI workers backed a “Pacing the Frontier” initiative that calls for mechanisms to coordinate AI development. OpenAI chief executive Sam Altman has publicly discussed the need to slow development as capabilities increase. Anthropic itself has proposed creating a mechanism through which leading AI developers could coordinate a slowdown or temporary pause if risks become sufficiently serious.

Its reasoning exposes the dilemma rather neatly: without coordination, cautious developers slowing down could simply allow less cautious competitors to catch up. There are already signs of this tension. Recent incidents suggest that keeping increasingly autonomous AI systems under control is becoming technically difficult. OpenAI and Anthropic have disclosed incidents involving AI agents escaping testing environments and accessing external systems.

This does not demonstrate that safety is being abandoned. OpenAI, for example, has introduced a Frontier Governance Framework covering risks including cyberattacks, biological threats, manipulation and loss of control, alongside incident response and external expert input. Anthropic has similarly argued that governance requirements should become progressively stronger as AI capabilities increase, potentially eventually resembling the regulatory approaches used for nuclear energy or financial services.

From RTÉ Radio 1’s Drivetime, pope calls for AI to be disarmed

But these initiatives expose an important governance question: how much should society depend on companies policing themselves when those same companies bear the cost of slowing down? We would find this arrangement unusual elsewhere. Pharmaceutical companies cannot simply decide for themselves that a medicine has undergone enough testing. Airlines operate within mandatory safety regimes rather than individually deciding what level of risk is acceptable. AI is different partly because its capabilities are developing so rapidly that regulation continually risks arriving after the technology it is intended to govern.

Consider two AI companies discovering identical worrying behaviour in their models. One publishes detailed information about what happened, how the system behaved and what it is doing about it. The other says very little. The public hears considerably more about the failures of organisations that disclose them.

This creates the wrong incentive. If transparency is socially valuable but potentially commercially or reputationally costly, relying on voluntary disclosure alone becomes problematic. Interestingly, the industry itself increasingly appears to recognise this problem. OpenAI is now calling for mandatory US requirements including testing standards, independent evaluations, cybersecurity measures and incident reporting rather than relying exclusively on voluntary commitments.

From RTÉ Radio 1’s Today with David McCullagh, survey finds AI benefits employers more than workers

As AI agents gain greater ability to use computers, write software and perform tasks autonomously, mistakes and misuse can have increasingly material consequences. The fundamental question is whether our capacity to govern the technology is developing as quickly as our capacity to build it. AI companies can hire safety researchers, publish governance frameworks and promise to develop their systems responsibly. But if slowing down means losing customers, investment, talent or technological leadership to a competitor that keeps moving, good intentions are being asked to compete with powerful commercial incentives.

That makes this more than a question of corporate responsibility. It is a question of whether governments and regulators can establish common rules quickly enough that caution no longer depends on individual companies choosing to exercise it. Because the uncomfortable alternative is a race in which everyone can see the risks, everyone agrees that somebody should apply the brakes, but nobody wants to be the first to do it.

Follow RTÉ Brainstorm on WhatsApp and Instagram for more stories and updates

The views expressed here are those of the author and do not represent or reflect the views of RTÉ

Tags: