David Pocock has criticised the government’s AI committee, of which he is a member, for giving too little time to interview AI executives and allowing that evidence to be given in camera. The independent senator says the government is “starstruck with the frontier”.
This week, executives from OpenAI and Anthropic appeared before the Joint Select Committee on Artificial Intelligence, with OpenAI chief strategy officer Jason Kwon on a PR mission to deliver an in-person mea culpa over the Medicare statistics breach.
“The whole thing is so deferential,” Pocock tells The Saturday Paper. “The way we’re treating these so-called frontier labs, basically two US companies, very new, that I don’t think any Australians really trust, and yet we treat them with such deference.
“I think we should have seen a lot more scrutiny and expected more in terms of details and what was going to actually change in the future.
“It very much seemed, ‘Well, we’re sorry, but this is basically the price of what we’re doing.’ ”
OpenAI’s chief executive, Sam Altman, brought that thought out into the open this week, telling Politico: “We believe that the world should accept some bad things happening for the benefits of this technology.”
In a relatively non-combative committee engagement, Kwon said OpenAI had made operational changes in the wake of the Medicare breach. It won’t email generic government addresses to notify them about “novel incidents”. Nor will it wait until it has all the facts before engaging impacted parties – in this case, the Australian government.
“I agree that the process by which people became aware of this incident inside our company could have been much better, and we want to make sure that something like that does not happen again,” Kwon told the committee, while also saying the Medicare portal’s cyber defences were “not … sophisticated”.
“I think people were thinking about this as a technical situation and they wanted to contact the technical counterparties, but it’s not good enough,” he said.
Terry Flew, co-director of the Centre for AI, Trust and Governance at the University of Sydney, told the committee that was an unsatisfactory response. “You certainly can’t have governance by public apology. We have seen that in the history of social media. ‘The bot did it’ is not really a satisfactory answer.”
Labor MP Zaneta Mascarenhas asked Kwon if AI models could currently break into an oil and gas facility and operate it.
“The answer here is we don’t know,” he replied. “It’s really going to depend on what sorts of precautions they already have in place.”
How can critical infrastructure be safeguarded from AI attack?
The only way to tell, according to Kwon, is to seek permission to collaborate with AI as “we cannot actually run that test just by ourselves”.
Liberal MP Andrew Hastie asked Kwon for the 60-second “elevator pitch” to Aussie mums, dads, workers and seniors for AI investment. The response took three minutes and involved diagnoses of disease, scientific problem-solving and economic growth and opportunity.
“Something that you can expect to happen if you help the technology generate investment in Australia,” Kwon said.
OpenAI and Anthropic, in competition in business, research and outreach to Australia, appeared in one-hour sessions before the joint select committee on Tuesday in Sydney, with half of Anthropic’s time spent in camera due to the sensitive nature of the discussion.
“I raised concerns with the committee,” Pocock says. “I think that there should be public scrutiny, and I think even just the amount of time that we got with this whole thing is basically about two US AI companies and yet we just got an hour with each. I think there’s a lot more that we need to be asking of them.
“Obviously, I can’t talk about what happened in the in camera session, but I personally disagree with that happening.”
“You certainly can’t have governance by public apology. We have seen that in the history of social media. ‘The bot did it’ is not really a satisfactory answer.”
The Labor chair of the committee says the work of the committee is bipartisan and in the national interest. She says the in camera component is a normal part of all parliamentary hearings and it is important for the members to hear evidence in confidence. Anthropic is not the only witness to request in camera.
“I would say we are pro-AI for Australians, not the companies building the tech,” committee chair Jo Briskey told The Saturday Paper. “We recognise where we need to hold those tech companies accountable to deliver something that is in the Australian interest.
“We’ve already demonstrated that as a government … and I think that’s what Australians want. That’s what Australians want to see their government doing, acting in their interest, not in the interests of big tech companies on the other side of the world.”
Other groups that gave evidence to the committee were more forthcoming about what they saw as an increasing, catastrophic-level risk. Bioweapons are at the forefront of that concern.
“As these models’ capability increases, we increase the risk that someone’s going to use these models to do something pretty terrible, particularly in the biological domain,” Devon Whittle, the director of Global Shield Australia, told the committee.
“The broader that these capabilities are diffused through society, the more chance you’ll find that one disaffected person, or that one rogue state who suddenly has the capability to put in place actions that they have in their head, wants to cause harm.
“We know that in the past, for example, Al Qaeda had a program to try to develop a biological weapon. The only thing that stopped them is they couldn’t get the PhDs required to put this program into place. Now you can have a PhD in your pocket.”
Anthropic released a threat intelligence report in September showing five cases where scientists were caught bypassing safety on Claude models to produce work that could support biological weapon or pathogen development. Google has also reported an attempt to use its AI tool, Gemini, to obtain a step-by-step guide to creating a biological weapon.
“We need to be ready and prepared to respond when things go wrong, and there are things in that space Australia can be doing,” Whittle warns. “AI models are giving new capabilities to malicious actors to develop biological agents and weapons, but there’s a really simple way for Australia to have one little safeguard in place, which is to screen imports of synthesised DNA products when they are at the border. At the moment, that doesn’t happen.
“California has just put in place a law that will require screening of synthesised DNA … and in Australia this change could happen basically through the stroke of the pen of the secretary of the Department of Agriculture.”
Pocock says this appears a sensible and urgent measure.
“This is something that could actually just be done through the department. They could just be changing the regulations around import of genetic materials. Seems very sensible,” he says. “Why is government so slow to act, and why do they keep caving to industry pressure to shelve something like an AI safety Act? And then when experts say, ‘Hey, these are the risks’, yeah, they just don’t seem to act.
“For me, it feeds the distrust in politics, where we always seem to have to end up in a crisis for the government to make some kneejerk reaction to try and solve things when we could have actually avoided that in the first place.”
The joint select committee has been tasked with interrogating the Australian AI sector as the government readies to introduce new national AI standards, focusing on energy sourcing, water supply, security, safety, transparency and local community engagement. The standards, due early next year, will regulate frontier AI and large-scale infrastructure.
The Gradient Institute, an AI safety group, says three actions are needed as a matter of urgency: substantially increasing funding to the AI Safety Institute; introducing mandatory AI incident reporting; and creating a specialised investment fund to build AI, AI safety and AI security technologies.
“We care about the safety. If we have good regulation in place, good requirements and constraints on model development training in Australia, we can actually build a more safety infrastructure technology industry,” Gradient Institute chief executive Bill Simpson-Young told the committee.
“If you think about what the counterfactual is, the training is not being done here, it’s being done elsewhere, and we have very little control of it. If it’s being done here, at least we can actually say, ‘Put conditions and constraints on how it’s done here.’ We should be encouraging all the training to be done in Australia because the alternative is much worse.”
The OpenAI Hugging Face swarming hack and Medicare breach show how easy it is for problems to arise.
“We’ve run some simulations just recently internally, where we put some agents together. We have a fake market. We have a simulated market. The agents are told to behave within that market. They will just naturally want to collude,” Simpson-Young says. “We then give them rules, instructions as part of their prompt, saying it is illegal to collude, you must not collude. They will still collude. They will still find a way.
“It’ll be subtle. There’ll be stuff in their language where they’re just giving hints to each other, and they’re actually ending up colluding. I think issues like this are going to be really problematic as you start to get more agentic commerce. You’re going to have a lot of behaviours like that happening at scale in the market that are going to be really hard to detect.”
The Albanese government says it is acting.
“People will only adopt technologies they trust. Businesses will only invest in what they believe is reliable. Nobody shares the benefits of something they are afraid of,” assistant minister for technology Andrew Charlton said in a speech on Thursday.
“Safety is not the brake on the AI opportunity. If anything, it is the condition on which the benefits can be fully realised.”
Charlton says the government is “deeply” committed to enabling AI training onshore, developing AI talent in Australia and shaping the technology. He argues a special regulatory case for AI due to it being hard to recall once digitally released, the rapid pace of change, poorer government AI literacy compared with developers, the need for global agreement over safety and the huge risk of unintended consequences from the technology.
“AI needs regulating because its harms are severe, hard to undo, borne by people who never chose them, and sometimes invisible until they arrive,” Charlton says.
“The market will not fix this alone, because the incentives reward speed and capability, and even the people at the top of the industry cannot slow down by themselves. Australia’s answer is to put the burden where the risk is created, to build accountability into the whole system, and to build our own capability alongside it.”
An increasing number of Australian companies are using AI through the open-source marketplace to interact with AI and create their own bespoke models.
Rupert Taylor-Price, the founder of cloud computing supplier Vault Cloud, said up to 90 per cent of tasks such as software development could be handled without frontier models.
So, what do we need frontier models for then, asked the independent member for Curtin, Kate Chaney, who quickly added, “other than national security?”
“Curing cancer,” Dan Navfield from the Canberra-based company Nexrupta told the committee. “Really, really complex, world-changing problems are where frontier models are best suited. So, deep research, very strong development kind of ideas is where they do their best work. As we said, you don’t need them for most work that happens every day, and so it is important to have them available.”
Pocock argues open weighted models, AI-trained models released publicly for anyone to use and modify, are where Australia should be focusing as a country.
“That’s where the real opportunity is,” he says. “We hear a lot about productivity. That’s where the productivity gains are, and it seems like we’re so sort of starstruck with the frontier that we’re potentially going to throw creators under the bus for them when most likely the main game is actually not the frontier labs unless they hit, you know, AGI [artificial general intelligence] or recursive self-improvement, then who knows? Then it’s a different game.”
As for the frontier labs, the senator sees them as talking to Australia like “we are some sort of diplomatic mission” and that we should accept “there’s going to be collateral damage”.
“I think we should absolutely be treating these companies like any other company that has incredibly powerful, but also potentially very dangerous, technology,” Pocock says. “They seem to want regulation, but President Trump won’t regulate them. So, we should at least put some safeguards in place here.
“I think we’re rightly asking questions: What’s all this for? What’s in it for us? What’s the long-term benefit? And how do we ensure that we don’t make the mistakes of the past, where we see big investment for a government to bump up GDP, and then the future consequences are borne by some other government? And we’re not thinking about that.”