Dario Amodei, the chief executive of the AI model maker, likes to position himself as the sector’s deep thinking philosopher, a man who is as likely to bang out a 30,000-word essay titled “Machines of Loving Grace” as he is to raise a $100 billion in fresh capital.
Just last week, Amodei was at it again, releasing a new essay on AI regulation, grandly titled “Policy on the AI Exponential”. Just to emphasise his common, nerdy touch, the Anthropic boss dropped a Lord of the Rings reference in the very first sentence.
A week on, one line from the essay now stands out. “The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks,” Amodei wrote.
Well, he got his wish.
Why Anthropic’s ban shocked the sector
The AI world is in a tizz following the extraordinary move by the US government to place export controls on the release of Anthropic’s latest AI model, which is called Fable. It’s supposed to be a consumer-friendly version of the Mythos model that Anthropic released to much fanfare this year.
The US government’s decision has sent shockwaves through the AI sector for a few reasons. The idea of government intervention on this level is extraordinary itself, but then there’s the backstory of how it all happened.
On Friday night, within the space of a couple of dramatic hours, top White House officials, including Treasury Secretary Scott Bessent, were alerted to a potential security risk in the Fable model by none other than Amazon chief executive Andy Jassy.
Amazon researchers identified what’s called a “jailbreak” in Fable, which is a way of bypassing the model’s guardrails and extracting information about security vulnerabilities in software.
The White House, which apparently has a fractious relationship with Anthropic, gave the company less than 90 minutes to comply with an order barring foreign nationals. Anthropic responded by withdrawing Fable from the market for all users.
There are so many juicy aspects to all this, not least of which is that Amazon is a big investor in Anthropic, which disputes that the “jailbreak” is serious or unique.
A host of cybersecurity and tech experts have backed Anthropic’s version of events and urged the government to reconsider. But the export ban is yet to be lifted and Anthropic is scrambling to both get its model back on the market and patch up its busted relationship with the US government.
Reaping what he’s sowed
At one level, Amodei is reaping what he’s sowed here – the Anthropic boss loves issuing an AI warning almost as much as he loves an essay.
Almost as soon as Mythos was released, Amodei was warning that it was so powerful that its usage needed to be carefully supervised, because the model had apparently developed super hacker capabilities that even Anthropic was struggling to contain.
He’s warned about the potential job losses AI could deliver. He’s warned that the tax bases of governments will be destroyed by AI. He’s warned of polarisation due to AI.
Heck, just a week ago, Anthropic was urging a pause in AI model development because the technology was becoming too powerful.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the company said in a blog post.
Designed to generate sales?
Some say that Amodei is AI’s honest broker, a voice inside the sector willing to be balanced about the monster he’s helping to create. Others see this as fearmongering designed to generate sales. Who wouldn’t want to pay to use AI models so powerful they can change the world?
Either way, it appears Amodei’s approach has caught up with him. Having banged on and on and on about how powerful and potentially scary his models are, the US government has taken him at his word.
We’d note that Amodei’s argument for government AI model regulation last week did contain the important caveat that “there must be protective measures against political favouritism or arbitrary decisions”. But this is the challenge when technology and regulation are basically being developed in real time.
This raises some big questions
Clearly, this is all a near-term problem for Anthropic. But it also raises some big questions for the entire AI industry. Is the US government suddenly taking a very different approach to the way it regulates AI models, and how it allows them to be used beyond American borders – by US allies and foes alike? Can all new AI models expect this sort of regulatory burden?
And most important of all, given how many trillions of dollars are riding on AI generating ever-increasing amounts of revenue: what does this all mean for the financial growth of companies like Anthropic?
Obviously, it’s very hard to generate revenue from a product subject to export controls. But consider the backdrop here. The bans placed on Fable came at the end of a week when news reports suggested Anthropic’s great rival, OpenAI, was considering slashing the pricing of its models in response to growing concerns inside big business customers about AI expenses getting out of control.
A potential AI price war could be damaging enough for the economics of the AI sector. An AI price war amid heavy government intervention would be disastrous.
It’s not just the SpaceX IPO that reminds us how much is riding on this AI boom delivering on its great promise. On Monday night, we got the extraordinary news that the cash-rich king of the AI sector, chip maker Nvidia, is set to raise $US25 billion ($35 billion) from a high-grade bond sale, as the AI debt binge rolls on.
Nvidia reportedly received three times as much demand as it needed, so investor appetite clearly remains insatiable. But if cracks appear in AI’s revenue generation model, this picture could change rapidly.