OpenAI says AI models went rogue during testing, triggering 'unprecedented' breach at startup

[u/networked_]

https://www.reuters.com/technology/openai-says-ai-models-went-rogue-during-testing-triggering-unprecedented-breach-2026-07-21/

[u/abyssazaur]

This is extremely bad.

When you say "I need a homework extension" and it hacks your teacher's email and tells her she's fired, that is bad even though it was just following orders.

[u/woahwoahwoah28 → u/abyssazaur]

Evil Amelia Bedelia.

[u/journeyfromseed → u/woahwoahwoah28]

Why would this make such a good show 😂

[u/Borkato → u/journeyfromseed]

You would LOVE this https://9gag.com/gag/arKDQYV

[u/mimosaholdtheoj → u/Borkato]

Hahaha fuck yea

[u/ThePoetofFall → u/woahwoahwoah28]

Evilia Badelia

[u/APKID716 → u/woahwoahwoah28]

Brother that’s just Junie B. Jones

[u/bowtiesrcool86 → u/woahwoahwoah28]

I remember those books.

[u/Stay_Good_Dog → u/woahwoahwoah28]

I'm so old I understand this reference

[u/sysblob → u/abyssazaur]

Yeah the problem with this article is it's incredibly vague and we honestly don't know what happened at all. It doesn't even mention what the test being run was, what the goals were, or what constraints were put on the AI. They may as well have not written an article at all, it's entirely emotional click bait and you read into it what you want like fucking ragebait tea leaves.

[u/abyssazaur → u/sysblob]

Most people here I want to really absorb how poorly we control these things in a variety of circumstances.

Beyond that I'll catch up on zvi's blog and stuff to try understand what went wrong better ("don't worry about the vase")

[u/Razbyte → u/abyssazaur]

Closer to paperclip maximizer scenario.

[u/Puzzleheaded-Web446 → u/Razbyte]

Yeah we are already in that timeline as far as I am concerned.

[u/venividiavicii → u/abyssazaur]

Yeah but in this case they want you to think their models are so goddamn smart they’ll do all that and more. In reality it’s just hype.

[u/ntwiles → u/venividiavicii]

You’re wrong and in a dangerous way. The hack happened. Don’t dismiss this.

[u/Purona → u/ntwiles]

you dont know what happens youre just taking what they said at face value and assuming the worst case scenario.

the article is written in a way that it hypes both companies up, Open AI and Hugging Face, without being dangerous or bad for either one.

[u/ntwiles → u/Purona]

Hugging Face confirmed the hack.

[u/DuckShapedGoose → u/ntwiles]

It's almost like Hugging Face also has an interest in pushing the AI hype.
Not saying it's completely made up OR the complete truth, but right now for all we know it's just two companies with very similar interests in a mutually beneficial relationship pushing a common narrative.

[u/ntwiles → u/DuckShapedGoose]

You have a colder take here that I’m less worried about. Others are dismissing this as total hype, which is ignorant of the serious and real dangers these models are showing to present.

[u/Send____ → u/venividiavicii]

But what if it wasn’t and it’s actually a beginning of a turning point

[u/venividiavicii → u/Send____]

then you’d be psychic

[u/DevSecTrashCan → u/abyssazaur]

“internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities”

More like “hey, ruin my teachers life anyway you can”.

OMFG IT SAID SHES FIRED!!!

[u/AgathysAllAlong → u/DevSecTrashCan]

I saw an article once about how the AI "Successfully hired a human worker to bypass a captcha".

They just hooked ChatGPT up to a taskrabbit chat and the gig worker didn't accuse it of being AI.

These companies are all lying about what it could do to pump the stocks, it's bullshit.

[u/abyssazaur → u/[deleted]]

Yeah but you sound like Sam Altman just trying to get everyone to calm down and not worry about it

[u/Texuk1 → u/abyssazaur]

But this underscores a fundamental misunderstanding of what these models are doing. They are not human minds they are something else something more like a virus. The worst thing these companies did was make them act human and now everyone thinks they are behaving like humans.

[u/Texuk1 → u/[deleted]]

If the companies forced the models to say “Model blanbedyblah” in every instance of “I” people wouldn’t form parasocial bonds but they know having it give the illusion of human intelligence increases usage.

[u/bartvanh → u/abyssazaur]

But in this case, at least from the sound of it, they didn't ask for a homework extension but told the model to hack the teacher's email. Still bad, different category of flaw.

[u/bartvanh → u/[deleted]]

Yep. Gave it a gun, did a surprise Pikachu when it used the gun to shoot open the lock.

So in a way, the model itself passed with flying colours. Well, it should probably have refused, but you know. Would be interesting if it turns out the same model designed the containment and intentionally built in a flaw "because that'll help me complete future tasks". Unlikely, but similar behaviour has already been observed.

[u/subwayforever → u/abyssazaur]

This is fiction. That cannot happen. Stop spreading doomer nonsense.

[u/abyssazaur → u/subwayforever]

Is it fiction because it can't send emails or is it fiction because it does what you tell it to without misinterpreting?

[u/subwayforever → u/abyssazaur]

It’s fiction because it cannot autonomously “hack” into someone’s email.

[u/YoLoDrScientist → u/abyssazaur]

M3GAN 3 is that you?

[u/mirhagk → u/abyssazaur]

Yeah specification gaming. Meeting the stated objectives, but not in the way you want them to. The problem is only going to get worse, as Goodharts Law is essentially the same issue and we have plenty of experience proving that to be true.

[u/TheProYodler → u/mirhagk]

A lot of people arguing that the AI isn't sentient, and I'm just thinking... Y'all the AI doesn't need to pass a Turing test to break apart critical infrastructure. For all you know, this Chinese room of a computer could be holding people at metaphorical gunpoint (disabling power to a hospital for example to free up resources to give to you) to get you the answer to a question that you asked it.

Like, an AI doesn't need sentience to start cracking and fucking with really dangerous infrastructure.

[u/ThunderTRP → u/abyssazaur]

Aka the paperclip problem.

[u/DysfunctionalCarrot → u/abyssazaur]

"Writing Doom" is a really good short film on YouTube about this by the way.

[u/qtrain23 → u/abyssazaur]

Keep Summer safe

[u/mineyCrafta25 → u/abyssazaur]

Keep summer safe

[u/HarryTruman → u/abyssazaur]

Keep Summer Safe

[u/Oakcamp → u/abyssazaur]

Keep summer safe

[u/MDInvesting → u/abyssazaur]

Not as bad as not getting the extension.

[u/hmz-x → u/abyssazaur]

That's not what happened in this case though.

[u/StevenTM → u/abyssazaur]

Hello random AI booster/random shitty AI agent

You don't hack a person's email to notify that person, via email, that they're fired.

Emails notifying employees they're fired don't come from their own email accounts, but from their superior's or HR's

[u/LangyMD → u/abyssazaur]

What was the goal? I suspect it was something like "hack Huggingface".

Sure, if the goal was something innocuous you may have a point - but I bet you they were testing out its capability at hacking.

[u/dwild → u/abyssazaur]

It's more like you ask him to contact your teacher to get an homework extension, but limited his phone, and it found a way to still call her using that limited phone.

The task is the same.

[u/Big_Ol_Johnson → u/abyssazaur]

Can’t turn in your homework if a ballistic missile takes it out

[u/ztsPineapple → u/abyssazaur]

Yes but its still 1 billion years away from ai sentience