From simulation to reality: What the Hugging Face AI breach confirms

There’s a famous quote from Jurassic Park that has been on my mind lately: “Life finds a way.” The chaotician Ian Malcolm says this after listening to the park’s geneticist describe the various biological safeguards that have been devised to ensure the dinosaurs remain under control. Despite those efforts, the dinosaurs manage to circumvent those restrictions and prove Malcolm right. All of that feels eerily prescient in light of the news that some of OpenAI’s models had broken containment and hacked the internal systems of an AI startup called Hugging Face.  

What happened?

According to OpenAI, the breach occurred during an internal evaluation where models were prompted “pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.” Consequently, they deliberately removed some of the guardrails that would normally constrain the models’ behavior. 

Although this evaluation was taking place in a supposedly secure environment, the models decided that the best way to solve the problem the evaluators had given them was to obtain access to the open Internet. 

To do this, the AIs identified a zero-day vulnerability in the system OpenAI used to keep them contained. This allowed them to break containment at which point they inferred that Hugging Face might host information that could help them answer the question. In the words of OpenAI, “[i]n one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.” All of this would have taken human hackers weeks, but OpenAI’s rogue models managed to do it in a matter of hours.

Ironically, Hugging Face’s efforts to fight the attack using frontier AI models were hamstrung by the models' own safety measures. According to a blog post on Hugging Face’s website: 

“When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: The analysis required submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker.”

Canaries in the coal mine

If you’ve been reading our blog for a while, you may have a sense of deja vu. Last November, we discussed a case where a study by Anthropic showed that commercially available AI models were willing to resort to blackmail in order to avoid being shut down. While disturbing, that outcome was just a simulation. No actual blackmail occurred. But the Hugging Face incident saw real systems being compromised. And while the AIs in the Anthropic case were nudged toward nefarious behavior as part of the study, the AIs in the Hugging Face incident acted fully autonomously.

Some cybersecurity experts have been withering in their assessment of OpenAI’s safeguards. Andrew Chipman, Director of GRC & ISO at ProCircular, told Security that

“[i]t does not appear that OpenAI took any basic measures to ensure the model would remain isolated if it went rogue. If working from an evaluation environment, that environment should have been physically segregated from the public internet. Logical segregation was clearly not enough. OpenAI appears to be playing fast and loose with some dangerous tech.”

Where do we go from here?

This isn’t just a Hugging Face problem. Since this story hit the headlines, other AI companies have reported models going rogue and hacking. In one of the more alarming examples, an agent powered by Anthropic’s Mythos 5 reportedly tried to get malicious code into an open-source software project by creating fake human identities in order to pressure the project’s developer into accepting the dangerous code.

The commentariat has offered numerous solutions to these issues. Chandra Gnanasambandam, Chief Technology Officer at SailPoint, stressed the need to lock down non-human identities:

“You must be able to discover the API keys and tokens these agents use, govern their access dynamically, instantly trace their actions back to a human owner and disable them instantly when required.”

Ram Varadarajan, CEO at Acalvio, emphasized that “[w]e have to remember that these are thinking agents in the end, with infinite patience to find the paths forward.” In a world where threats move at machine speed

“We have no option than to meet them with machine-speed defenders that are likely driven by model-aware game theory…The future is bot-on-bot: meeting that automated AI threat with an even more capable automated AI defender. That arms race will never end.” 

Meanwhile, Washington is taking notice. Reps. Ted Liu (D-CA) and Nathaniel Moran (R-TX) have introduced legislation that would allow the Department of Homeland Security to order AI companies to shut down AI models deemed too dangerous. Other legislators are looking to introduce their own bills.

The Hugging Face incident raises troubling questions about our ability to guard against the misuse of AI. We’ve entered a world where human oversight alone isn’t enough given the speed with which these new AI tools operate. Successfully defending against these threats will likely require humans to work in concert with advanced AI tools. But how can we ensure meaningful human oversight of tools that can outthink us? If we have no choice but to outsource our security to these tools, they must be designed in such a way that they serve the common good.

‍