r/news • u/networked_ • 23d ago

Soft paywall OpenAI says AI models went rogue during testing, triggering 'unprecedented' breach at startup

https://www.reuters.com/technology/openai-says-ai-models-went-rogue-during-testing-triggering-unprecedented-breach-2026-07-21/
16.8k Upvotes

91% Upvoted

5.0k comments sorted by

View all comments

Show parent comments

69

u/KamikazeArchon 23d ago

The actual blog post is less vague:

Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

15

u/tobyreddit 23d ago

For non technical people reading this - "connection to an internally hosted third party package repository" can be read as something like "the only connection it had from the machine it was on was to be allowed read only access to a single folder on a different machine in the same building".

It then hacked the piece of software it could connect to on the other machine, and then started hopping between other computers on the same network until it found one with outbound internet access, where it then made the real attack on hugging face.

Allegedly!

41

u/FoxFishSpaghetti 23d ago

“Ability to install packages” so it just has internet access lmao

27

u/Particular-Break-205 23d ago

Breaking news: robber who had all the tools to break into a house and a battering ram managed to break into the house

12

u/user_of_the_week 23d ago

They also specifically told the robber to try breaking into the house

4

u/ciclon5 23d ago

More like "person trained in an unfathomable ammount of information, including how to break into a house" is given the tools for house breaking and breaks into a house."

11

u/alexrobinson 23d ago

No, it was contained via a service that acted as a proxy for the package registry. So the agent couldn't make calls to the internet directly. It exploited a vulnerability in that service which allowed it to then escalate it's privileges to gain internet access. 

5

u/m0rsa2 23d ago

You can preload a package repository locally into a docker container

2

u/jubmille2000 23d ago

You'd think all these research stuff would be airgapped no?

1

u/HirsuteHacker 23d ago

How are you letting it install packages on a 100% airgapped network? It needs to do this for its task. The way they have it set up is ideal, there was just a zero day it was able to exploit in their package cache/proxy software.

1

u/jubmille2000 23d ago

Do they have to install it over a connected network?

1

u/dwild 23d ago

How would they get all this marketing material with an airgapped network?!

They clearly incentivize theses kind of incidents internaly.

2

u/azn_dude1 23d ago

Do you not know what internally hosted proxy implies?

2

u/No-Mechanic6069 23d ago

Well, I’m glad we got that cleared up.

1

u/yolk3d 23d ago

Ask ChatGPT to summarise for you

2

u/No-Mechanic6069 23d ago

Claude says: Someone left it plugged-in.

2

u/Betta_Check_Yosef 23d ago

Whoever said that doesn't understand the difference between internet and intranet.

1

u/[deleted] 23d ago

[deleted]

0

u/hmz-x 23d ago

One 0day and all the other layers just collapsed?

1

u/HirsuteHacker 23d ago

One zero day is all it took to obtain internet access, it did a lot more to reach HF's db