When Caroline Flack died in February 2020, social media was filled with an outpouring of grief from celebrities and fans.
As well as posts expressing sympathy after the TV presenter took her own life, was a reckoning with the online abuse and extreme scrutiny that she had faced on a daily basis. It spurred a hashtag, #BeKind, that encouraged trolls to stop and think about the impact of their comments.
One of the people watching that hashtag was David Cole, a coder who was working at the UK’s cybersecurity agency GCHQ in Cheltenham.
Cole said: “When Caroline Flack died, I remember thinking how upsetting it was that someone could be bullied like that. It was going around in my head while I was working in AI at the time for the government and I knew we had the ability to detect negative comments, hide them from users, and do that so rapidly that it would stop people seeing this sort of content.”
Cole, who declined to describe his previous work in detail, decided to quit his job to found a company called Arwen AI with the IT director Matt McGrory. Now their technology is being used by Flack’s former employer, ITV, as well as Sky, Mercedes, the Six Nations and the dating app Hinge, among dozens of other celebrities and companies.
Arwen has teamed up with Flack’s friends and family, who have organised a festival in the Love Island presenter’s honour every year since Flack’s death, to offer greater protection to celebrities from online abuse.
Flackstock will be held on Friday August 8 at Crystal Palace in south London with performances from Mel C, S Club 7 and Pixie Lott, with all proceeds donated to mental health charities. All the acts, none of whom are being paid to appear, will receive free access to Arwen.
Natalie Pinkham with Christine Flack, right, at Flackstock
COURTESY OF NATALIE PINKHAM
The late presenter’s mother, Christine Flack, and her close friend Natalie Pinkham, who fronts Formula 1 coverage for Sky Sports, said the collaboration was part of their effort to secure a lasting legacy for the star.
Pinkham said: “Suicide always brings with it a lot of toxicity, blame, shame, anger and guilt and we were determined to right a few wrongs and build a positive legacy for her.”
Arwen uses a three-step process to assess and put a stop to any online abuse. First, the AI detects that a comment has been posted on the user’s timeline. It then uses an algorithm to analyse the language and detect hate speech, misogyny, racism or profanity.
Finally, the AI designates the comment safe, severe or suspect — which could result in a report to police. “Severe” comments are almost immediately hidden from the user’s timeline or deleted.
Pinkham with Caroline Flack
NATALIE PINKHAM
The AI is programmed to flag comments based on the celebrity’s instructions and personal identity. Cole gave an example of a female footballer who wants to filter out misogynistic comments: “In women’s football, abusive users could say ‘she shouldn’t be playing’. It wouldn’t be misogynistic in another context, so we build a database of these examples, label them manually and build our own classifier using a large language model.”
McGrory said: “A lot of these comments sit in what we call the ‘lawful but awful’ world — they probably aren’t breaching any laws but they’re causing lots of harm to people.”
A negative comment is usually hidden within two or three seconds of it being posted and detected by the algorithm — meaning that it’s gone before almost anyone has the chance to see it.
When comments are flagged as severe, such as death or rape threats, Arwen uses another AI-powered monitoring organisation called Sport Radar to track down the human user behind the abuse. At a recent tennis tournament in Stuttgart, where a man had posted threatening comments towards a player from inside a venue, Arwen alerted officials in this way and German police swooped to arrest him on Centre Court. He has since received a suspended sentence, Arwen said.
Mercedes F1 was Arwen’s first client. Its communications team requested the app to protect its gay digital chief from the homophobia which was prevalent on its online channels. From there Arwen picked up Red Bull, the Association of Professional Tennis, and ITV shows such as I’m a Celebrity … Get Me Out of Here and Love Island, among others.
MPs can claim Arwen’s services on expenses as a security product, Cole said, and the start-up has also worked one-on-one with celebrities such as Rosie Jones, a comedian with cerebral palsy who often receives abuse.
Christine Flack said that five years after her daughter’s death, she still remembers some of the vile comments that were posted about her. “The online stuff was very hard for Carrie and it still is for lots of people,” she said. “She’d be on her phone all the time and if she didn’t see it, someone would tell her about it.
“A message that you send online might take you two seconds, but it can have a much more long-lasting impact than that. You don’t forget it.”
Flack died in 2020 and had regularly faced online abuse
SUZANNE PLUNKETT/REUTERS
Pinkham has experienced the abuse Arwen hopes to tackle online. “I’ve had a couple of incidents that I’ve reported to Sky and it’s been sorted, which is amazing. F1 used to be a really small community and everyone protected each other. But now it’s a bit like the Wild West and I find, as a woman, that when I ask difficult questions of the drivers, it goes beyond criticism.”
Flackstock and the AI collaboration are about the positives that can come from Flack’s death, her mother said: “It’s a day where we just think about Caroline, but what we want out of it is more than just a day to remember her. We want a legacy for Caroline and if we can eliminate some of what she experienced, that’s our goal.”