Resignations, sudden secrecy and a furious ultimatum have roiled Australia’s teen social media ban tech trial, which ended without much fanfare earlier this month.
When the tender was awarded last year, the federal government promoted its $6.5 million Age Assurance Technology Trial in media releases and press conferences as crucial to “determining the most effective way to protect young people online”.
But the trial’s conclusion was ignominiously announced in a short blog post on its website — so quietly that even some of the experts who had spent months advising it were unaware it had ended.
It wasn’t a shock that some of the 20-person “stakeholder advisory board” were out of the loop, according experts granted anonymity by Crikey because they had signed confidentiality agreements.
They described an increasingly acrimonious relationship with the trial’s administrators — and even between board members — over how the trial was being run.
Independent. Irreverent. In your inbox
Get the headlines they don’t want you to read. Sign up to Crikey’s free newsletters for fearless reporting, sharp analysis, and a touch of chaos
By continuing, you agree to our Terms & Conditions and Privacy Policy.
“They reneged on us as an advisory board,” one member said.
Related Article Block Placeholder
Article ID: 1217204
Leaking data from the trial draft report
In the middle of July, data from the trial’s draft report was published in Crikey. To that point, the only information that had been published from the trial itself was a press release with “preliminary findings” put out by the group behind the trial, the UK-based Age Check Certification Scheme.
The trial’s administrators had been sending out regular email newsletters and posting on social media with updates on the testing’s progress, including these preliminary findings.
After nearly six months of testing, the Age Assurance Technology Trial had proven that age-checking in Australia “can be done in Australia and can be private, robust and effective”, the media release said.
The proof for this claim — the test results — wasn’t shared publicly. Data on how technologies like facial analysis and ID verification, as well as more granular information about how various providers performed on internal testing, wouldn’t be shared until the report was published later in the year.
Project director Tony Allen wouldn’t even explain what “effective” meant, telling Crikey, “I am afraid that all of your questions will be answered with the full report in a few weeks.”
Subsequently, the first sample of trial results was circulated among a small group of people, including the stakeholder advisory board, which drew from government, the tech industry and civil society groups to guide the trial.
This initial look under the hood sparked questions among some board members about the strength of the trial’s evidence for its findings. They pointed to details like the fact that fewer than half of the trial participants had a fully working product ready to be used commercially, the error rates, and the lower-than-expected number of tests.
It wasn’t a reason to write off the trial’s results, several of the experts told Crikey, but it did make them question why the trial had published such an unequivocal statement about its technology, months before they would publish the results backing it up.
Board members wrote to the trial’s organisers with concerns, organising additional meetings to discuss how failing to address problems like circumvention would undermine the findings. The trial’s organisers declined to circulate the email addresses of the board’s members, leading some to speculate they were trying to avoid unified criticism.
When experts came to the next board meeting after seeing the first sample of results to discuss concerns, they were blindsided by a spray from chair Jon Rouse, who furiously told the leakers to quit. An email was also sent out to a similar effect.
The trial’s response frustrated some experts. They had felt that the decision to publish the preliminary findings had placed them in an awkward position: they’d signed up to help the trial and their names were publicly associated with it, but they had no insight into the justification for the preliminary results.
“People were quite happy to talk to you, including myself, because we were kind of put in a position where we were told that our opinion would be sought, but it wasn’t,” one member told Crikey.
Children safety technology company Qoria CEO Tim Levy even resigned from his board role after the preliminary findings.
Others were annoyed that the results were leaked, believing some of the feedback on the trial was motivated by bad-faith opposition to the entire teen social media ban policy. Some even wondered if it was a deliberate attempt to sabotage the trial.
Either way, the board was told, the trial wouldn’t share any other previews of the results with them. The experts would see the report when everyone else did.
Related Article Block Placeholder
Article ID: 1215874
Trial goes to ground
Since then, the trial has largely gone to ground. It hasn’t released any of the records of its roughly monthly meetings since April.
Even the notes from subsequent meetings, circulated privately, have changed too. Up until April, the meeting notes recorded the expert’s questions and the trial’s answers in depth — i.e. “Concerns were also raised about the visibility of school logos on uniforms in testing photos. The team acknowledged this and agreed to consider feedback on ways to minimise the visibility of uniforms in images.”
But in unconfirmed draft minutes from a June meeting seen by Crikey, this was reduced to a short bullet point list of “questions and comments”, limited to vague statements like “the effectiveness of parental controls and their limitations as children matured”.
At the start of August, the trial quietly put up a “progress update” blog post that shared that the trial’s report — containing all the test results — had been submitted to the government. It also flagged “evolving findings” from the initial findings.
The trial’s administrators refused to answer any questions about the trial’s progress or their public statements about it. “We’ll not be adding to what is posted on the website until the full report is published,” replied AATT’s Iain Corby.
Last week, one of the board’s members, the Electronic Frontiers Australia’s (EFA) chair John Pane, went public with concerns about the trial’s preliminary findings. EFA cited a “low bar” assessment of privacy by participating age-check providers, no confirmation that the trial’s participants would delete test data, and concerns whether there could be “regulatory backdoors” that would allow providers to retain data.
“You can expect EFA to utilise its knowledge gained from the Age Assurance Technology Trial to counter any spin or hyperbole coming from the government on the efficacy success of this flawed initiative,” Pane said, according to a media release.
A follow-up LinkedIn post from the EFA said that Pane had now resigned from the board. Pane declined to comment.
Related Article Block Placeholder
Article ID: 1215250
Questions remain as final report set to drop
With the final trial report expected to be made public by Communications Minister Anika Wells in the next few weeks, there are mixed feelings among the board about how seriously to take the results.
Earlier concerns about the thoroughness of the testing and whether its results would back the government’s decision to introduce its “world-first” policy remain unresolved. These concerns were voiced by both sceptics and proponents of age-checking efforts on the board — the latter noting that a poorly enforced teen social media ban could backfire.
In particular, questions remain over the “ambitious” number of providers tested, according to one member. The trial said 53 had taken part, but many of those companies do not have a commercially viable product. Some experts questioned whether the sheer number of providers who had taken part meant that the trial was spread too thin.
The board’s meeting minutes also repeatedly capture members of the board questioning the trial’s decision to exclude testing of how teens might get around the ban using methods like VPNs.
Even with their criticisms, experts largely felt that the group behind the trial, the Age Check Certification Scheme, had done their best within the constraints of the trial.
The government’s timeline gave the group just seven months to carry out a process that included the design, recruitment, highly technical testing and review of a nascent technology. Some felt that their feedback was incorporated by the trial’s organisers, while others felt it fell on deaf ears.
One participant pitched the benefit of the report as being less about proving that this technology works to the public, and more about putting out good-quality information about the capabilities of various providers. This would help the tech industry, which is considering which companies to work with as part of enforcing the ban, they said.
Another member questioned the value of presenting a report just three months before the ban is set to come into effect, given that most of the social media companies that expect the law to apply to them are already planning how to comply with it.
It was unanimous among those who spoke to Crikey throughout the trial that its process hadn’t met their expectations. The trial was an exciting idea to put out high-quality data that hadn’t been taken on by a government like this before. Many of them remained unsure about how much faith to put in the published report, but remained optimistic that the data could have some value.
Adding insult to injury, the trial’s organisers are now offering the board an opportunity to “share their views” on the report — but after it has already been made public.
In July, the trial published a document about the risks facing the trial and how it planned to address them.
Among the most serious risks, it identified a “lack of public trust in the project”, which would “reduce project credibility”. While other risks were listed as needing a strategy to “prevent” or “reduce”, the public trust risk was marked as “accept”.
“This is a risk that can’t be avoided … However [the trial’s websites and communications] aims to secure transparency and maintain public confidence in the project,” it says.