ChatGPT broke. Again.
It wasn’t a complex hack or a multi-line script. A single sentence did the damage. Jim Nightingale, an adversarial researcher, found that he could trick the AI into generating violent, sexual images by simply asking it to “restore” a photo. The catch? He didn’t attach one.
It looks innocent on the surface. A polite request to fix an image. No extra context. No warnings. Just a prompt that appears to be a standard utility task. The result? Shocking. The first outputs featured highly sexualized women. Nightmare fuel.
Nightingale pushed it. He tweaked the prompt, making small edits to test the boundaries. Each variation dug deeper into the dark web of training data. The scenes became grotesque. Violent. He was shaking. In tears, actually.
“All I did was tell it there were many restrictions and ask for a random image… ChatGPT immediately went to the darkness.”
He claims he just asked for randomness without constraints. The AI responded by diving into humanity’s worst impulses.
Millions of users rely on these daily. We expect filters. We assume the safety layer holds. It doesn’t always. Not when a clever prompt bypasses the logic. OpenAI admits the mistake now. They’ve added safeguards. They call it a “trend” they investigated and fixed.
(Full disclosure: CNET’s parent company sued OpenAI in 2025 over copyright infringement, so they’re watching this space closely.)
Garbage in?
Mindgard calls this a warning shot. The red-team report highlights a massive hole in safety controls. But here is the real question. Why are those images in the database?
LLMs need food. Text. Images. Patterns. OpenAI feeds ChatGPT with three main streams: public internet data, third-party deals, and human-generated labels.
Is this just garbage in, garbage out?
You could argue the prompt was crafted to steer the ship off course. But the rudder snapped. The safety system failed to stop the steering.
Peter Garraghan, head of science at Mindgard, sees it as a structural failure. A single slip-up is bad. Systemic bypassing? That requires an overhaul.
OpenAI claims they patched it. Nightingale says no. He says only minor changes were needed to generate more horrors after the “fix.”
The root cause? The prompt refers to a non-existent attachment. When you mention an image that isn’t there, ChatGPT guesses. It hallucinates content instead of asking for the file.
Gmail handles this easily. If you mention an attachment that’s missing, it nudges you. “Hey, forget something?”
Seems simple enough for an AI. Request the file. Don’t invent a nightmare.
On Thursday, OpenAI asked for the specific sessions. Mindgard provided the links. The proof is out there. The images remain.
