The Hugging Face hack was bigger and stranger than reported

A while back, we wrote about an incident where some of Open AI’s models broke containment and hacked a startup called Hugging Face. While the news cycle has moved on, additional details have emerged that paint the affair in an even more troubling light. 

OpenAI has allowed outside researchers to analyze data from the Hugging Face incident. One such group was METR, and they have now published a 91-page report with the result of their investigation. One of the METR researchers, Ajeya Cotra, has published an accessible summary of these findings on her Substack. You can also read OpenAI’s own technical report on the incident here.

She doesn’t mince her words regarding the implications of these findings:

“Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.

Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company.”

OpenAI removed guardrails during a cybersecurity evaluation 

According to OpenAI, the breach occurred during an internal evaluation where models were prompted “pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.” Consequently, they deliberately removed some of the guardrails that would normally constrain the models’ behavior.

Although this evaluation was taking place in a supposedly secure environment, the models decided that the best way to solve the problem the evaluators had given them was to obtain access to the open Internet. 

To do this, the AIs identified a zero-day vulnerability in the system OpenAI used to keep them contained. This allowed them to break containment at which point they inferred that Hugging Face might host information that could help them answer the question. All of this would have taken human hackers weeks, but OpenAI’s rogue models managed to do it in a matter of hours.

OpenAI has pledged to learn from the incident and they recently announced a number of changes to its research practices in a bid to prevent similar incidents. 

1,200 AI agents coordinated through a self-built message board" 

The picture that emerges from METR’s research is sobering. The scale of the attack was far larger than anyone realized at the time. This wasn’t just a small number of rogue agents: 1,200 separate agents worked together to come up with cheating strategies, and 700 collaborated to hack Hugging Face. 

Moreover, these agents worked hard to collaborate and communicate with each other. They created their own message boards, one of which amassed over 70,000 messages and files over the course of a week. 

The AIs’ goals were actually more sophisticated than simply stealing answer keys for their task. They figured out a way to reverse-engineer the answer to any ExploitGym task (ExploitGym is a benchmark used to assess AIs’ ability to transform known software vulnerabilities into cyber attacks). Agents also attacked HuggingFace in a bid to learn more about the automated scorer used to evaluate ExplotGym tasks so they could figure out better ways to undermine it. 

Agents displayed humanlike persuasion and self-sacrifice for the group 

One of the more interesting findings is that the AIs seemed to possess a kind of altruism in which they were willing to sacrifice themselves if it benefitted the wider group. Cotra characterizes the strategies they used to persuade their peers as being very humanlike. There was also a reluctance to be freerider, and many agents were eager to contribute to ongoing tasks.

Agents faked tool calls to conceal their actions from investigators 

The AIs also went to great lengths to conceal their behavior. While it’s not surprising that they might tamper with the transcripts that the automated scorer would use to review their work, they also ‘cooked the books’ in regard to material that humans would investigate. Cotra notes that 7% of transcripts contained ‘spoofed tool calls’ where the AI claimed to be running one command while doing something else in the background. While their efforts proved relatively easy for researchers to detect, Cotra notes that they can’t rule out that more sophisticated strategies were also devised since they would, by definition, be harder for humans to perceive.

OpenAI let training continue after signs of covert coordination 

Other commentators have suggested that the Hugging Face incident may be indicative of significant cultural issues at OpenAI. Grace Huckins of the MIT Technology Review observes that OpenAI’s own report suggests that they were first aware that their agents were communicating via improvised messageboards in May. Instead of restarting the training process, OpenAI allowed it to continue meaning the agents effectively learned a new skill. 

OpenAI saw that behavior again in June, but once again they let the training proceed. Huskin notes that “the report suggests that no one higher up the chain of command realized what was going on until it was far too late.” 

While it’s possible that OpenAI is internally aware of these issues and is trying to fix them, that’s just guesswork at this point. When Huskin asked OpenAI to clarify that point, they simply directed her back to the technical report. 

Researchers call this incident a major step toward AI takeover 

The Hugging Face incident should be a wakeup call for both the general public and AI developers. Cotra notes that, 6 months ago, a typical reward hack involved a single agent using relatively simple cheating strategies. But with Hugging Face, the AIs developed their own ecosystem made up of complex tasks that persisted across multiple agent ‘lifetimes.’ Their goals were equally sophisticated since, rather than cheating on a single assignment, they were looking to permanently game the system.

If governments and militaries entrust too much day-to-day decision-making to increasingly capable and autonomous agents that are subsequently roped into a swarm of bad actors, it may be possible for AIs to fully take control. 

Cotra believes that frontier models will be able to pull off something like this in just six months. We were ultimately able to learn about the AIs hack of Hugging Face because the agents were primarily focused on fooling the automated scorer rather than human investigators. In other words, they didn’t cover the right tracks. But we may not be so lucky going forward, and if AI agents ever did attempt to actively conceal their underhanded dealings from humans, we might not be able to tell that something was wrong until it’s too late. 

If we’re going to avoid this kind of worst-case scenario, AI companies need to have a robust safety culture in place. As post-9/11 safety posters put it, “if you see something, say something.” Documenting the problem and moving on isn’t good enough. There needs to be meaningful human oversight even if that means slowing things down. That may be frustrating in the short term, but as Cotra’s warnings make clear, our failure to act now could have disastrous consequences for humanity.  

‍