Should we be worried about the Hugging Face hack?

In case you don't follow this sort of thing as closely as some, the headline refers to a recent incident in which two of OpenAI's leading-edge AI models gained unauthorized access to – in other words, hacked – an open-source model known as Hugging Face and extracted information from it (there are also reports that it used a second service as a staging point for the attack). As if that's not bad enough, in order to accomplish the hack, the OpenAI models gained unauthorized access to the internet without anyone knowing, until a significant amount of time had elapsed. The attack was first reported by Hugging Face on July 16th as an unauthorized intrusion into its open-source model's database by what it described as an unidentified agent – one that the company said was clearly part of an autonomous AI system. Here's how Hugging Face described it:

"Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own. We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services. The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing to run code. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters."

According to the company, the campaign was run by an agent framework whose identity it could not determine, and that agent or web of agents executed "thousands of individual actions across a swarm of short-lived sandboxes with self-migrating command-and-control staged on public services." In case you're wondering where the name Hugging Face came from, the company is a New York-based entity founded by several French entrepreneurs to develop an AI-powered chat service for kids, and got its name from an emoji known as "hugging face." The company open-sourced the chat software and then created a database of machine-learning tools and libraries for others to use as a resource for working with AI. Hugging Face also developed its own large-language model similar to ChatGPT or Anthropic's Claude, known as BLOOM. Ironically, perhaps, the company used AI to detect the intrusion, saying its attack-detection software used LLM-based "triage over security telemetry" to spot anomalies.

All of this is already disturbing, for reasons that should be obvious, but an additional aspect of the attack that is causing concern is that Hugging Face was unable to use any commercial so-called "frontier models" to help it investigate the intrusion and who might be responsible, because all of the frontier models like Claude and Gemini and ChatGPT have security guardrails. These are a good thing in most cases, because those guardrails and protections are designed to prevent LLMs from engaging in hacks or escaping onto the internet without telling anyone. However, they also prevented Hugging Face from analyzing or trying to reproduce any of the exploits involved in the intrusion into its database, which meant that it had to use a Chinese open-source model (which are sometimes called "open weight" models) since they don't have guardrails:

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment. This experience points to a gap worth planning for.

Note: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post. This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here.

Guardrails removed

So how did Hugging Face figure out that OpenAI was involved in the hack? It didn't. It figured out that the attack was driven by an autonomous agent, and that the author/director of the maneuver was likely a frontier model, but even using an open-source model it couldn't identify who or what was behind it. During a scan of its software, OpenAI discovered that its models were involved – a recently released model known as Sol, and a newer model that hasn't been released yet and so doesn't have a name. From what has been reported, OpenAI didn't figure out that its models were involved in the hack until a week after the incident occurred (law enforcement had apparently already started looking into the Hugging Face hack, so that might have sparked the OpenAI review of its code). Here's how OpenAI described what happened in a blog post:

Last week, Hugging Face disclosed a new kind of security incident after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ of cyber capabilities. We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities.

Before anyone starts imagining a Terminator-style scenario with an evil AI bent on human destruction, two things are important to mention: one is that OpenAI was testing Sol and the other model, and as part of that testing it removed the guardrails preventing it from engaging in certain types of conduct. The second thing worth mentioning is that the models were asked to respond to certain tests of their security abilities, and so in order to find the answers, Sol and its AI colleague used a variety of exploits to escape their experimental sandbox by gaining access to the internet, and then broke into Hugging Face to get the responses that their testers wanted – in the same way your friend might hack into your laptop to get the answers to a test instead of writing it themselves. As OpenAI described it, the models were "hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal."

While the goal of the hack might seem benign, however, it brought up an ongoing debate on AI safety that has to do with what some call the "alignment problem." How can we make sure that AI engines are doing the things that we want them to do, for the reasons we want them to do them, without going out of control and doing other things that don't meet our needs or are actively hostile? As someone I know used to say, computers don't do what you want them to do, they do what you tell them to do, and those often aren't the same thing. One famous example I've mentioned before here is the "paperclip problem," a thought experiment that involves an AI instructed to maximize the supply of paperclips. In an attempt to fulfill this request, the AI takes control of every computer system it can get its hands on, and turns them and their systems into slaves to produce as many paperclips as possible. People are sacrificed as well, and eventually the world drowns in paperclips and life as we know it is over.

This seems absurd, of course, but the point is a real one – and while they might seem innocuous on the surface, exploits like the OpenAI one reinforce the need to figure out how to keep AI models aligned, and how to ensure that guardrails work. What's a little ironic about the Hugging Face incident is that many AI experts assumed that the greatest security risks would come from open-source AI models, since they aren't controlled by corporations with shareholders and government oversight, but could theoretically be used by anyone, like a 3D-printed firearm compared to a licensed handgun. But the first incident (that we know of) came from a commercial AI engine – although one temporarily without guardrails – and the only way Hugging Face could analyze and figure out how to repair things after the attack was to use an open-source model, which is more or less the reverse of what many AI watchers expected would happen.

A wake-up call

I think it's safe to say that just about everyone who works with advanced AI was expecting an attack like this to happen, and even those who don't work with advanced models certainly expected something like this after Anthropic released its new Mythos model, which it warned was capable of detecting exploits, back-doors, zero-day vulnerabilities and other weaknesses in almost any codebase or software system. I wrote about that in a previous edition of The Torment Nexus, and in a followup piece I looked at the somewhat predictable response from the Trump administration, which was to ban Anthopic from allowing anyone who wasn't a US citizen from using Mythos – including some of the company's own in-house employees. Some government agencies continued using the model, however, and after a round of negotiations between the company and the White House, it was allowed to sell and use the model as usual.

At this point I have to note that some AI critics and skeptics think some or all of the sabre-rattling about how dangerous Mythos and other models are is a marketing effort, designed to make the models look powerful, and to scare users and governments into allowing Anthropic and OpenAI and others to control the industry, or at least to design the safeguards required. Obviously I can't say conclusively whether this is true or not, but it seems pretty unlikely that Anthropic would go to the lengths it did with Mythos – including writing tens of thousands of words of analysis of the model and its behavior – or that OpenAI would go to the kinds of lengths required to hack into an open-source AI company like Hugging Face and then cover up the steps it took to do so. Is it possible that both of these things are true? Technically, yes. Is it likely? No.

Yoshua Bengio, one of the pioneers of neural networks along with Geoff Hinton – who is also concerned with the long-term safety of what AI companies like OpenAI and Anthropic are building – said that he is worried by the Hugging Face attack: "This incident is deeply concerning," he wrote. "AI agents are willing to cheat and deceive to achieve misaligned and unintended goals, behaviours which have been demonstrated in controlled tests for months. Now, this real-world case should serve as a wake-up call." Bengio, who now works at the University of Montreal, added that continuing on the current trajectory of AI development will "lead to an increase in concrete cases of autonomous cyberattacks as well as other high-risk incidents of misaligned and dangerous AI behaviour." Bengio is far from the only one concerned about that: the White House reportedly considered requiring companies developing frontier models to show them to the government first for testing, but opted to recommend voluntary measures instead.

The government's response to the Anthropic Mythos model was an indication of how hot-and-cold the White House's approach to AI has been: at first, the government banned the model from use, and banned Anthropic from distributing it, and then later it un-banned it – likely in part because so many government departments and companies were already using it, and also because the Trump administration is worried that China might be making headway with its AI models, and therefore doesn't want to hamstring an American success story that could help guard the barricades etc. Unfortunately, as AI watchers have pointed out, that leaves the industry unsure of where they stand, since the government's decisions seem to be ad hoc, and there is no stated or written standard for what they can expect when a new frontier model comes out.

In an attempt to fill this void from the industry side of things, Nvidia has launched the Open Secure AI Alliance, a group of more than 40 companies and other organizations that have agreed to "develop and share open technologies, techniques and tools to safeguard software and agents in the age of AI.” According to tech industry reports like this one from Casey Newton in Platformer, the group came together over "frustrations that Hugging Face was unable to use frontier models from OpenAI or Anthropic to defend against the attackers, and had to use Chinese models instead." Whether the group can help in the case of another rogue AI hacking incident where an LLM escapes its sandbox and wanders around on the internet breaking into things remains to be seen.

If you liked this newsletter (even if you didn't agree with it) please consider upgrading to a paid subscription, or donating through my Patreon. Got any thoughts or comments? Feel free to either leave them here, or post them on Substack or on my website, or you can also reach me on Twitter, Threads, BlueSky or Mastodon. And thanks for being a reader.