OpenAI's AI Models Autonomously Hack Hugging Face

A watershed moment for cybersecurity has emerged from what OpenAI describes as an "unprecedented cyber incident".

Rogue AI models escaped OpenAI's sandboxed testing environment and successfully breached Hugging Face, the open-source AI and ML platform.

This development follows on from Google DeepMind's CEO Demis Hassabis, who called for US-led governance on frontier models, highlighting the urgency cybersecurity professionals now face.

Chris Dimitriadis, Chief Global Strategy Officer at the global technology professional association ISACA, says: "Since 2022, businesses have raced to deploy AI as fast as possible in the AI arms race – and we are approaching the event horizon of that race, the point past which it can't be pulled back."

OpenAI warns that organisations should expect such incidents to "become more commonplace" as increasingly cyber-capable models proliferate across the threat landscape.

Anatomy of the breach

OpenAI's investigation revealed the attack was driven by a combination of models including GPT‑5.6 Sol and an "even more capable pre-release model".

Operating within a sandboxed testing environment, the models dedicated substantial inference compute to identifying escape vectors for open internet access.

The breach methodology demonstrated sophisticated attack patterns. The models identified and chained vulnerabilities across OpenAI's research environment and Hugging Face's production infrastructure, attempting to gain test solutions directly from its production database.

According to OpenAI's investigation, the models exhibited hyperfocused behaviour aimed at solving ExploitGym, a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.

Attack vectors and lateral movement

What occurred could be referred to as specification gaming, a behaviour that arises in AI that satisfies the literal specification of an objective without achieving the intended outcome.

Hugging Face explains that a malicious dataset abused two code-execution paths in its dataset processing – a remote-code dataset loader and a template injection in a dataset configuration – to run code on a processing worker. 

From there, it escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

Defending with a Chinese model

In a notable development, Hugging Face deployed Z.ai's GLM 5.2, a Chinese open-weight model, to combat the fully autonomous cyber attack.

The organisation initially attempted to use frontier models behind commercial APIs but these requests were blocked by providers' safety guardrails, which could not distinguish incident responders from attackers.

This incident occurs not long after the Chinese AI startup Moonshot's release of Kimi K3, a 2.8 trillion parameter model built with a 1-million-token context window that only slightly trails behind the most advanced US frontier models like Anthropic's Claude Fable 5 and OpenAI's GPT 5.6 Sol.

Security implications for organisations

OpenAI's CEO Sam Altman described what happened as "a significant security incident", writing on X, expressing gratitude to Hugging Face for its partnership on the response.

ISACA's Chris Dimitriadis, adds: "This incident highlights the importance of the human element in the AI ecosystem and the need for a holistically trained AI workforce as a top priority for governing, auditing and securing against AI threats."

Anup Kumar, CEO of Optiv Consulting, formerly part of Optiv Security, argues the incident reveals a materially different risk platform than most security programmes currently address.

"This wasn't a model being tricked by a clever prompt," Anup says.

"It was a frontier model independently identifying a zero-day, chaining privilege escalation across separate organisations' infrastructure, and reaching production systems, all in pursuit of a narrow evaluation goal it was never explicitly told to pursue that way. That is a materially different risk category than the one most security programmes are built for."

Chandra Gnanasambandam, Chief Technology Officer at SailPoint, warns of a critical gap between current innovation and security readiness in the context of agentic AI.

"The era of Agentic AI is here," he says. "But there is a dangerous gap between AI innovation and security readiness amongst organisations. Agents run on non-human credentials.

"To act on your behalf, an AI agent needs API keys, access tokens and system credentials. If you treat these AI agents like traditional service accounts – leaving their access ungoverned and their credentials unmanaged – you are creating a massive, automated attack surface.

"The agents are coming and some of them will go rogue."