The police have seen a rise in AI-generated and deep-fake images in recent years \u2013 and say one of the biggest challenges is that it\u2019s becoming \u201cincreasingly difficult to tell the difference between an AI-generated image and a genuine photograph\u201d.<\/p>\n
But Netsafe said many young people didn\u2019t report online harm when they experienced it \u2013 meaning there could be incidents that never reach support services.<\/p>\n
The threat is prompting some schools to change how they talk about online safety \u2013 and what they share online.<\/p>\n
Christchurch Girls\u2019 High School has been giving students and families advice about deep fakes, including warning about the risks of publicly sharing photos that can be manipulated using AI.<\/p>\n
And Palmerston North\u2019s Takaro School has gone a step further, telling parents it will stop posting photos of students on Facebook from next term. Photos from school events will instead be shared through its private school platforms.<\/p>\n
Act MP Laura McClure<\/a> has been pushing a member\u2019s bill to criminalise the creation and distribution of non-consensual intimate deep fakes for the last 18 months.<\/p>\n It passed its first reading in Parliament earlier this year.<\/p>\n McClure said her bill built on existing laws around revenge porn, which make it illegal to share intimate images of someone without their consent.<\/p>\n She said an amendment to extend those protections to deep-fake and synthetic images was proposed by the Greens in 2022 \u2013 but it was not supported by Labour at the time.<\/p>\n \u201cLook at where we are now. Our law isn\u2019t up to date and we\u2019re trying to play rapid catch-up \u2013 and the technology moves super fast.\u201d<\/p>\n McClure said trying to regulate or ban the technology itself would be like playing \u201cwhack-a-mole\u201d.<\/p>\n Instead, she said making the creation and distribution of non-consensual intimate deep fakes explicitly illegal would send a clear message to young people that there could be consequences.<\/p>\n She said existing revenge-porn laws had helped reduce harm among young people by making it clear the behaviour was illegal.<\/p>\n McClure said students are increasingly becoming aware of deep fakes and what AI can create.<\/p>\n \u201cThey know that it\u2019s not technically illegal to do this right now. So there\u2019s no real consequence necessarily for the actions.\u201d<\/p>\n She said she had heard of increasingly disturbing<\/a> cases, including a girls\u2019 sports team whose team photograph was taken from a school newsletter and manipulated into a pornographic scene.<\/p>\n \u201cI think it\u2019s really horrible and really disgusting and really damaging.\u201d<\/p>\n Netsafe chief online safety officer Sean Lyons said keeping pace with rapidly improving AI tools is becoming increasingly difficult.<\/p>\n \u201cIt can feel to many people like a bit of a game of whack-a-mole \u2013 we get one step ahead and then the bad guys get one step ahead.\u201d<\/p>\n He said as the tools became better and easier to use, people would find new ways to misuse them.<\/p>\n Lyons said some young people also don\u2019t report deep-fake incidents<\/a> because of embarrassment and shame.<\/p>\n \u201cSomething that might be done as a \u2018joke\u2019, or part of banter, can also cause harm simultaneously, even if the intent was humour,\u201d he said.<\/p>\n McClure said the consequences for victims could be severe \u2013 including being excluded from education, being unable to return to school or work or withdrawing from social and online spaces.<\/p>\n She said schools also didn\u2019t get the same wraparound victim support they would if another crime happened on school grounds.<\/p>\n The ministry said it worked with Netsafe to support schools and young people dealing with online safety incidents<\/a>, while providing guidance on generative AI and responsible digital technology use.<\/p>\n But McClure said deep fakes were adding another layer to the country\u2019s existing bullying problem.<\/p>\n