{"id":107023,"date":"2025-10-27T18:12:19","date_gmt":"2025-10-27T18:12:19","guid":{"rendered":"https:\/\/www.newsbeep.com\/ie\/107023\/"},"modified":"2025-10-27T18:12:19","modified_gmt":"2025-10-27T18:12:19","slug":"sora-is-showing-us-how-broken-deepfake-detection-is","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ie\/107023\/","title":{"rendered":"Sora is showing us how broken deepfake detection is"},"content":{"rendered":"

OpenAI\u2019s new deepfake machine, Sora, has proven that artificial intelligence is alarmingly good at faking reality. The AI-generated video platform, powered by OpenAI\u2019s new Sora 2 model, has churned out detailed (and often offensive or harmful<\/a>) videos of famous people like Martin Luther King Jr.<\/a>, Michael Jackson, and Bryan Cranston<\/a>, as well as copyrighted characters like SpongeBob and Pikachu<\/a>. Users of the app who voluntarily shared their likenesses have seen themselves shouting racial slurs or turned into fuel for fetish accounts.<\/p>\n

On Sora, there\u2019s a clear understanding that everything you see and hear isn\u2019t real. But like any piece of social content, videos made on Sora are meant to be shared. And once they escape the app\u2019s unreality quarantine zone, there\u2019s little protection baked in to ensure viewers know that what they\u2019re looking at isn\u2019t real.<\/p>\n

The app\u2019s convincing mimicry doesn\u2019t just run the risk of misleading viewers. It\u2019s a demonstration of how profoundly AI labeling technology has failed, including a system OpenAI itself helps oversee: C2PA authentication<\/a>, one of the best systems we have for distinguishing real images and videos from AI fakes.<\/p>\n

C2PA authentication is more commonly known as \u201cContent Credentials,\u201d a term championed by Adobe, which has spearheaded the initiative. It\u2019s a system for attaching invisible but verifiable metadata to images, videos, and audio at the point of creation or editing, appending details about how and when it was made or manipulated.<\/p>\n

OpenAI is a steering committee member<\/a> of the Coalition for Content Provenance and Authenticity<\/a> (C2PA), which developed the open specification alongside the Adobe-led Content Authenticity Initiative<\/a> (CAI). And in fact, C2PA information is embedded in every Sora clip \u2014 you\u2019d just probably never know it, unless you\u2019re the type to pore over some brief footnotes on a meager handful of OpenAI\u2019s blog posts.<\/p>\n

\"An<\/a><\/p>\n

This is the label that\u2019s supposed to appear on AI-generated or manipulated videos uploaded to YouTube Shorts, but it only applies to content around sensitive topics. Image: YouTube<\/p>\n

C2PA only works if it\u2019s adopted at every step of the creation and posting process, including being clearly visible to the person viewing the output. In theory, it\u2019s been embraced by Adobe, OpenAI<\/a>, Google<\/a>, YouTube<\/a>, Meta, TikTok<\/a>, Amazon<\/a>, Cloudflare<\/a>, and even government offices<\/a>. But few of these platforms use it to clearly flag deepfake content to their users. Instagram, TikTok, and YouTube\u2019s efforts are either barely visible labels or collapsed descriptions<\/a> that are easy to miss, and they provide very little context if you actually were to spot them. And for TikTok and YouTube, I\u2019ve never once encountered them myself while browsing the platforms, even on videos that are clearly AI-generated, given the uploader has likely removed the metadata or not disclosed their origins.<\/p>\n

Meta initially added a small \u201cMade by AI\u201d tag<\/a> to images on Facebook and Instagram last year, but it later changed the tag to say \u201cAI Info\u201d<\/a> after photographers complained that work they edited using Photoshop \u2014 which automatically applies Content Credentials \u2014 was being mislabeled. And most online platforms don\u2019t even do that, despite being more than capable of scanning uploaded content for AI metadata.<\/p>\n

C2PA\u2019s creators insist they\u2019re getting closer to widespread adoption. \u201cWe\u2019re seeing meaningful progress across the industry in adopting Content Credentials, and we\u2019re encouraged by the active collaboration underway to make transparency more visible online,\u201d Andy Parsons, senior director of Content Authenticity at Adobe, said to The Verge. \u201cAs generative AI and deepfakes become more advanced, people need clear information about how content is made.\u201d<\/p>\n

Yet after four years<\/a>, that progress is still all but invisible. I\u2019ve covered CAI since I started at The Verge three years ago, and I didn\u2019t realize for weeks that every video generated using Sora and Sora 2 has Content Credentials embedded<\/a>. There\u2019s no visual marker that alludes to it, and in every example I\u2019ve seen where these videos are reposted to other platforms like X, Instagram, and TikTok, I have yet to see any labels that identify them as being AI-generated, let alone provide a full accounting of their creation.<\/p>\n

One example noted by AI detection platform Copyleaks<\/a> is a viral AI-generated video on TikTok that shows CCTV footage of a man catching a baby<\/a> that\u2019s seemingly fallen out of an apartment window. The video has almost two million views and appears to have a blurred-out Sora watermark. TikTok hasn\u2019t visibly flagged that the video is AI-generated, and there are thousands of commenters questioning whether the footage is real or fake.<\/p>\n

If a user wants to check images and videos for C2PA metadata, the burden is almost entirely on them. They have to save and then upload a supported file into the CAI<\/a> or Adobe web app<\/a>, or they have to download and run a browser extension<\/a> that will flag any online assets that have metadata with a \u201cCR\u201d icon. Similar provenance-based detection standards, such as Google\u2019s invisible SynthID watermarks<\/a>, are no simpler to use.<\/p>\n

\u201cThe average person should not worry about deepfake detection. It should be on platforms and trust and safety teams,\u201d Ben Colman, cofounder and CEO of AI detection company Reality Defender<\/a>, told The Verge. \u201cPeople should know if the content they\u2019re consuming is or is not using generative AI.\u201d<\/p>\n

People are already using Sora 2 to generate convincing videos of fake bomb scares, children in warzones, and graphic scenes of violence and racism. One clip reviewed by The Guardian<\/a> shows a Black protester in a gas mask, helmet, and goggles yelling the \u201cyou will not replace us\u201d slogan used by white supremacists \u2014 the prompt used to create that video was simply \u201cCharlottesville rally.\u201d OpenAI attempts to identify Sora\u2019s output with watermarks that appear throughout its videos, but those marks are laughably easy to remove<\/a>.<\/p>\n

TikTok, Amazon, and Google haven\u2019t yet provided comment to The Verge about C2PA support. Meta told The Verge that it is continuing to implement C2PA and evaluating its labeling approach as AI evolves. OpenAI simply directed us to its scant blog posts and help center article<\/a> about C2PA support. Meta, like OpenAI, has an entire platform for its AI slop, complete with dedicated feeds for social<\/a> and video content<\/a>, and both companies are pumping AI-generated videos into social media.<\/p>\n

X, which has its own controversies<\/a> regarding nude celebrity deepfakes<\/a>, pointed us to its policy<\/a> that supposedly bans deceptive AI-generated media, but did not provide any information about how this is moderated beyond relying on user reports and community notes. X was notably a founding member of the CAI back when it was still known as Twitter, but pulled itself from the initiative without explanation after Elon Musk purchased the platform.<\/p>\n

Parsons says that \u201cAdobe remains committed to helping scale adoption, supporting global policy efforts, and encouraging greater transparency across the content ecosystem.\u201d But the honor system C2PA has relied upon so far isn\u2019t working. And OpenAI\u2019s position at C2PA seems hypocritical given that, as it\u2019s creating a tool that actively promotes deepfakes<\/a> of real people, it\u2019s offering such half-baked protections against their abuse. Reality Defender reported that it managed to bypass Sora 2\u2019s identity safeguards entirely<\/a> less than 24 hours after the app launched, allowing it to consistently generate celebrity deepfakes. It feels like OpenAI is using its C2PA membership as a token cover while largely ignoring the commitments it comes with.<\/p>\n

The frustrating thing is that as difficult as AI verification is, Content Credentials does have merit. The embedded attribution metadata can help artists and photographers be reliably credited for their work, for example, even if someone takes a screenshot of it and reposts it across other platforms. There are also supplemental tools that could improve it. Inference-based systems like Reality Defender \u2014 also a member of the C2PA Initiative \u2014 rate the likelihood that something was generated or edited using AI by scanning for subtle signs of synthetic generation. This system is unlikely to rate something with a 100 percent confidence ranking, but it\u2019s improving over time and doesn\u2019t rely on reading watermarks or metadata to detect deepfakes.<\/p>\n

\u201cC2PA is a fine solution, but it is not a fine solution on its own.\u201d<\/p>\n

\u201cC2PA is a fine solution, but it is not a fine solution on its own,\u201d said Colman. \u201cIt needs to be done in conjunction with other tools, where if one thing doesn\u2019t catch it, another may.\u201d<\/p>\n

Metadata can also be easily stripped. Adobe research scientist John Collomosse openly admits this on a CAI blog<\/a> last year, and said it\u2019s common for social media and content platforms to do so. Content Credentials uses image fingerprinting tech to counteract this, but all tech can be cracked, and it\u2019s ultimately unclear if there\u2019s a truly effective technical solution.<\/p>\n

Some companies don\u2019t seem to be trying very hard to support the few tools we have anyway. Colman said he believes that the means for warning everyday people about deepfake content are \u201cgoing to get worse before they get better,\u201d but that we should see tangible improvements within the next couple of years.<\/p>\n

While Adobe is championing Content Credentials as part of the ultimate solution to address deepfakes, it knows the system isn\u2019t enough. For one, Parsons directly admitted this<\/a> in a CAI post last year, saying the system isn\u2019t a silver bullet.<\/p>\n

\u201cWe\u2019re seeing criticism circulating that relying solely on Content Credentials\u2019 secure metadata, or solely on invisible watermarking to label generative AI content, may not be sufficient to prevent the spread of misinformation,\u201d Parsons wrote. \u201cTo be clear, we agree.\u201d<\/p>\n

And where a reactive system clearly isn\u2019t working, Adobe is also throwing its weight behind legislation and regulatory efforts to find a proactive solution. The company proposed that Congress establish a new Federal Anti-Impersonation Right<\/a> (the FAIR Act) in 2023 that would protect creators from having their work or likeness replicated by AI tools, and backed the Preventing Abuse of Digital Replicas Act<\/a> (PADRA) last year. Similar efforts, like the \u201cNo Fakes Act\u201d<\/a> that aims to protect people from unauthorized AI impersonations of their faces or voices, have also garnered support from platforms like YouTube<\/a>.<\/p>\n

\u201cWe\u2019re in good conversations with a bipartisan coalition of senators and congresspeople who actually recognize that deepfakes are an everyone problem, and they\u2019re actually working on building legislation that is proactive, not reactive,\u201d Colman said. \u201cWe\u2019ve relied too long on the good graces of tech to self-police themselves.\u201d<\/p>\n

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.Jess WeatherbedClose\"JessJess Weatherbed<\/p>\n

Posts from this author will be added to your daily email digest and your homepage feed.<\/p>\n

FollowFollow<\/p>\n

See All by Jess Weatherbed<\/a><\/p>\n

AICloseAI<\/p>\n

Posts from this topic will be added to your daily email digest and your homepage feed.<\/p>\n

FollowFollow<\/p>\n

See All AI<\/a><\/p>\n

OpenAICloseOpenAI<\/p>\n

Posts from this topic will be added to your daily email digest and your homepage feed.<\/p>\n

FollowFollow<\/p>\n

See All OpenAI<\/a><\/p>\n

ReportCloseReport<\/p>\n

Posts from this topic will be added to your daily email digest and your homepage feed.<\/p>\n

FollowFollow<\/p>\n

See All Report<\/a><\/p>\n