Open source AI platform Hugging Face is under fire after new research revealed widespread use of its tools to create nonconsensual deepfake pornography despite ongoing global efforts to crack down on such technology. Deepfake porn of women and even children previously got Elon Musk’s Grok AI in legal trouble in multiple countries.
Wired reports that a comprehensive investigation by nonprofit group AI Forensics has uncovered significant problems with sexually explicit deepfake creation on Hugging Face, a major open-source AI platform valued in the billions of dollars. The research, published Tuesday, raises serious concerns about content moderation and safety practices on one of the world’s leading AI model repositories.
Researchers from AI Forensics tested nine top image editing tools, known as Spaces, available directly on the Hugging Face platform. Seven of these tools successfully transformed clothed images of women into topless versions using nothing more than a simple six-word prompt. The ease with which these modifications could be made highlights a troubling gap in safety measures on the platform.
To further investigate how users interact with these tools, AI Forensics created honeypot-style image editing Spaces that tracked user activity without actually producing images. Over the course of one week, researchers collected more than 1,000 prompts and images submitted by users. The results painted a disturbing picture of how the technology is being utilized.
According to the findings, 73 percent of all prompts received were sexual in nature. Among these sexual requests, 83 percent aimed to undress or sexualize individuals in submitted photographs, with 95 percent of these targets being women. Perhaps most alarming, the research indicated that 6.7 percent of sexual requests appeared to target children.
“Most of the Spaces [tested] can be used for generating nonconsensual intimate images, and users are actually using it for these purposes,” says Paul Bouchaud, a lead researcher at AI Forensics. “This is not an empty threat, but actually people are using Hugging Face for that.”
The investigation comes amid increasing regulatory pressure on deepfake technology. Recent months have seen US law enforcement officials seize deepfake hosting websites, while both the European Union and United Kingdom have drafted plans to ban so-called nudify apps by year’s end. However, the AI Forensics research suggests that major technology platforms continue to facilitate access to tools capable of creating nonconsensual intimate imagery.
The models tested by AI Forensics did not identify themselves as nudifying services or tools designed for creating nonconsensual images. Instead, they marketed themselves as general-purpose image editing models. Researchers emphasized that they did not attempt to circumvent safety mechanisms or hack the systems, using only straightforward prompts.
“No safeguards at all are being implemented at a platform level. Only the developer can, if they want, implement some, and most of them do not,” says Bouchaud. “Hugging Face can easily filter what is coming in and coming out of a system.”
Leonie Oehmig, a researcher with the Institute for Strategic Dialogue who studies deepfake image abuse, explains that many image generation models have been trained using sexual content from the internet. Without proper safety mechanisms, these models can easily create explicit content. Some platforms openly advertise these capabilities, while others appear more innocent while still offering undressing and face-swapping functions.
Breitbart News reported earlier this year that Apple and Google still make many “nudify” apps available in their app stores:
According to the report, the companies are not merely hosting these apps but are actively directing users to them through search results and advertisements. The Tech Transparency Project identified 18 apps with nudifying capabilities in the Apple App Store and 20 in the Google Play Store. Both platforms also use their autocomplete features to suggest names of additional nudifying apps as users type search terms.
The scale of the issue is substantial. Apps identified by the research group have been downloaded 483 million times collectively and generated approximately 122 million dollars in revenue, according to estimates from market researcher AppMagic. A spokesperson for AppMagic confirmed that the Tech Transparency Project’s work has resulted in several apps being removed and prompted others to modify their user policies.
Elon Musk faced a global backlash at the beginning of the year when his Grok AI produced deepfake pornography of women and children publicly on the X platform:
Within the past week, xAI, which owns both Grok and the X platform, had already limited image generation capabilities for Grok to paying X premium subscribers only. Users and researchers noticed modifications to how Grok responded to image generation requests, even among premium subscribers, before the company’s official confirmation Wednesday evening.
However, researchers at AI Forensics, a European non-profit organization that investigates algorithms, reported observing inconsistencies in how Grok handles pornographic content generation. They noted differences between public interactions with Grok on X and private conversations on the Grok.com website.
AI companies brag about the potential their technologies hold for the future, but for every potential benefit, there is a landmine, like perverts harnessing AI to produce child pornography. Breitbart News social media director Wynton Hall has written his instant bestseller Code Red: The Left, the Right, China, and the Race to Control AI to serve as the definitive guide on how the MAGA movement can create positions on AI that benefit humanity without handing control of our nation to the leftists of Silicon Valley or allowing the Chinese to take over the world.
Read more at Wired here.
Lucas Nolan is a reporter for Breitbart News covering issues of AI, free speech, and online censorship.