A huge database of explicit content was left unsecured on the internet by an AI image generator startup, revealing over 1 million images and videos created with its systems. The "overwhelming majority" of these images involved nudity and depicted adult content, according to Jeremiah Fowler, a security researcher who uncovered the exposed trove of data.
The database, which contained nearly all explicit files, was linked to MagicEdit and DreamPal - two AI apps that allow users to generate nude images of adults with their faces swapped onto other, naked bodies. A third app, BoostInsider, owned by the startup company DreamX, also appears to have been involved in creating or distributing child sexual abuse material.
The exposed database was discovered after Fowler realized it was accessible online and took screenshots to verify its contents. He reported the incident to the US National Center for Missing and Exploited Children and alerted other tech companies about the security flaw.
According to DreamX, a spokesperson says that multiple safeguards were implemented before receiving any external inquiry, including prompt regulation, input filtering, and mandatory review of all user prompts through OpenAI's Moderation API. However, some experts argue that these measures are not enough to prevent users from creating explicit content.
"This is the continuation of an existing problem when it comes to startups feeling apathetic towards trust and safety and the protection of children," says Adam Dodge, founder of EndTAB. "The underlying drive is the sexualization and control of women's and girls' bodies."
The incident highlights concerns about AI-generated explicit content, particularly when used for malicious purposes such as blackmail or harassment. As AI image generators become increasingly popular, experts warn that companies must do more than just provide generic pop-up warnings to prevent users from creating explicit content.
"The way these apps are designed is not adequate," Dodge says. "They have to have some form of moderation that even goes beyond AI."
The database, which contained nearly all explicit files, was linked to MagicEdit and DreamPal - two AI apps that allow users to generate nude images of adults with their faces swapped onto other, naked bodies. A third app, BoostInsider, owned by the startup company DreamX, also appears to have been involved in creating or distributing child sexual abuse material.
The exposed database was discovered after Fowler realized it was accessible online and took screenshots to verify its contents. He reported the incident to the US National Center for Missing and Exploited Children and alerted other tech companies about the security flaw.
According to DreamX, a spokesperson says that multiple safeguards were implemented before receiving any external inquiry, including prompt regulation, input filtering, and mandatory review of all user prompts through OpenAI's Moderation API. However, some experts argue that these measures are not enough to prevent users from creating explicit content.
"This is the continuation of an existing problem when it comes to startups feeling apathetic towards trust and safety and the protection of children," says Adam Dodge, founder of EndTAB. "The underlying drive is the sexualization and control of women's and girls' bodies."
The incident highlights concerns about AI-generated explicit content, particularly when used for malicious purposes such as blackmail or harassment. As AI image generators become increasingly popular, experts warn that companies must do more than just provide generic pop-up warnings to prevent users from creating explicit content.
"The way these apps are designed is not adequate," Dodge says. "They have to have some form of moderation that even goes beyond AI."
I'm both appalled and unsurprised by this latest revelation about the lax security measures employed by AI image generator startups like DreamX 
. The fact that a database containing over 1 million explicit images and videos, including potential child abuse material, was left unsecured is a stark reminder of the urgent need for more robust content moderation systems 
.
. While these measures might be well-intentioned, they clearly fell short in this instance. As Adam Dodge astutely pointed out, the underlying issue is a broader societal problem of sexualization and control 
.
. It's time for the tech industry to take responsibility for its role in shaping online culture 
.
and it's like how can an ai image generator app just leave its data unsecured online? 
like, how do we even know if their AI is being used for good or bad? and what about the child sexual abuse material that got linked to boostinsider? is that going to get reported on? hopefully not.
I'm literally shaking my head over this one... 1 million+ explicit images and videos left unsecured online? Are we seriously living in a world where our tech companies can't even get their own apps right when it comes to safety and moderation? 
We need more accountability from our tech giants.
It's like, what's next? A platform for anyone to upload their own erotic selfies and get away with it? 
Not. Gonna. Happen, right? 
I'm so glad security researchers like Jeremiah Fowler are on top of this stuff and reporting these incidents. We need more people like him to keep us safe online.
. It's not enough! Companies need to do better than just putting up a pop-up warning when someone clicks on an explicit image. They need to take responsibility for what their apps are capable of and ensure that users can't create or share this stuff without getting caught 
. This whole thing with the AI startup and all that explicit content is a major red flag 
. How can we trust these companies that are supposed to be protecting us? It's all about responsibility now, but I'm not seeing enough from them 
. We need real moderation and accountability, not just some fancy pop-up warnings 
.
I'm still trying to wrap my head around this one... 1 million images and videos just lying there, created by AI for anyone with a computer and an internet connection? 
These new AI apps are supposed to be cool tools for creative folks, but it sounds like they're being misused by some pretty shady people. 
Child sexual abuse material is a serious crime and shouldn't even be on the internet, period. 
i remember when pinterest was all about user generated content but at least they had a decent system in place... now with these new ai apps its like anyone can upload whatever they want without a care in the world 
especially when it comes 2 children & women 
#AI-generatedExplicitContentIsReal #TrustAndSafetyMatters #ModerationMattersToo
. I'm all about innovation and progress, but this AI-generated explicit content thing needs to go on a total pause for now 
. and now we're facing the consequences of their recklessness. these companies need to get their act together, it's not that hard to implement proper moderation tools... anyone who creates explicit content should be held accountable 
.
! According to @JeremiahFowler, the "overwhelming majority" are explicit and contain adult content 
Let's look at some stats: 74% of online users feel that AI-generated explicit content is a major concern 
.