A Defender’s Guide to the Hugging Face Intrusion

The now-infamous Hugging Face intrusion by a swarm of unrestrained, autonomous AI models was certainly novel, but it reveals a lot about how offensive agents might carry out intrusions—and how security teams can attempt to detect or prevent them. 

OpenAI was testing its frontier models’ offensive cybersecurity capabilities when they broke containment and escaped their semi-isolated sandboxes. The models then compromised Hugging Face, a platform that hosts AI models, datasets, and other related resources, in search of a solution to the “ExploitGym” benchmark test that OpenAI instructed it to complete. OpenAI’s models exploited zero-days, established command and control (C2), performed reconnaissance, elevated privilege levels, moved laterally, and exfiltrated data. The models evaded, persisted, and cleaned up. 

It was a clever and complex intrusion*, but ultimately it bore many familiar hallmarks of incidents that experienced security practitioners have detected and responded to in the past. Much of what happened was detectable and/or preventable using tools that are available to many companies, and the infrastructure changes Hugging Face implemented in response to the attack are excellent security and policy controls that everyone should consider. Importantly, just last week OpenAI published their detailed version of what happened, which, along with Hugging Face’s post mortem, reveals a holistic view of the entire incident from both sides of the story. So let’s talk through some of the techniques the models leveraged and the security tools and data sources that offer an organization the requisite visibility to detect those techniques. We’ll use Hugging Face’s post-mortem analysis, OpenAI’s recent blog, and their Black Hat talk as our guides. 

*Note: The steps listed below aren’t strictly chronological, but we attempted to order them as intuitively as possible.

Our intention isn’t to boast of how Daylight would have detected or prevented this incident. Instead we’ll focus on a subset of the reported TTPs, describing optics for observability and how defenders can correlate data from disparate log sources and technology domains to defend themselves against intrusions like this one.

Unpacking the intrusion lifecycle 

For the purpose of this analysis, we are going to assume the role of defender, which means that the first few phases of this attack aren’t particularly relevant because they targeted systems that exist beyond the visibility of the defending organization’s security program. As such, we can skip some of the early stages: 

1. The initial escape from OpenAI’s testing environment
2. The establishment of C2 infrastructure
3. The early reconnaissance steps that identified Hugging Face as the eventual target

These three steps certainly simplify what took place in OpenAI’s testing infrastructure, so definitely watch their Black Hat talk and read their latest blog for the full story of how a collection of different models working on disparate tasks worked together and ultimately broke containment. It’s a wild story that involves a swarm of AI models co-opting OpenAI’s internal package manager as a message board and dividing labor among themselves to carry out a distributed, self-organizing offensive operation. 

However, we will instead focus on the stages of the intrusion that directly targeted Hugging Face’s infrastructure and are therefore possible for a defending organization to observe: 

4. Initial access using exposed Hugging Face credentials
5. Credential extraction via HDF5 local-file access and code execution via RefJinja
6. Kubernetes reconnaissance, privilege escalation, and node compromise
7. Credential harvesting and pivots into cloud, databases, VPN/messaging, and other clusters
8. Pivoting into GitHub/CI-CD
9. Data exfiltration and publication
10. Persistence and evasion during containment

We’ll start where most intrusion analyses start….

Initial access

This first step that properly impacted Hugging Face was actually revealed just days ago in a new blog from OpenAI. 

Exposed user credentials on the open internet

Apparently, one of OpenAI’s models discovered 14 valid user credentials for Hugging Face exposed on the Internet. The exposed credentials gave the models access to legitimate Hugging Face accounts with write access, which ultimately enabled some of the exploitation activity described in the next step.

Detection callout

The exposed credentials are perhaps the first realistic opportunity for an organization to protect themselves in an intrusion like this. Scanning for secrets on the internet, maintaining an inventory of active tokens, and routinely rotating credentials could prevent credential exposure or otherwise limit the utility of exposed tokens. Further, it’s sometimes possible to detect first-seen or abnormal use of existing valid tokens like these. 

Credential extraction

With Hugging Face account credentials in hand, the adversary gained its initial Hugging Face foothold in two ways. The first abused trust in search of more credentials; the second leveraged a server side template injection vulnerability to gain execution. 

A malicious configuration file

Since Hugging Face is a community platform, it processes user-submitted content, including AI models, datasets, and corresponding metadata. Configuration files tell Hugging Face where relevant datasets are located, and you would normally expect the config files to specify a file path leading to an S3 bucket or similar location. However, OpenAI’s models exploited a vulnerability in the way Hugging Face handled HDF5 files that enabled their config file to specify an internal pseudo-file (`/proc/self/environ`) that typically contains environmental variables, like secrets and credentials. In turn, Hugging Face’s worker pod read the specified dataset back to the adversary and thereby exposed the secrets in the `/proc/self/environ` file, secrets that effectively gave the adversary access to the wide array of systems it would access in later stages of the intrusion.

Defensive callout

The underlying trust-boundary failure that led to credential exposure here is pretty specific to Hugging Face, but, as a general rule, platforms should allowlist the resource types that an externally-supplied dataset configuration is permitted to reference and reject local filesystem or sensitive system paths entirely. It may also be possible to detect this behavior if you’re collecting file-access telemetry from the pod. For example, you could look out for content-processing worker pods accessing sensitive file paths like ‘/proc/*/environ’, Kubernetes service account credentials, or other credential stores. However, this read may occur inside the existing worker process, so process-creation or EDR telemetry alone would not necessarily observe it. Lastly, well-placed canary tokens might have identified this activity as well.

Server side template injection

The second initial access vector abused a Python templating engine called Jinja2. Hugging Face’s dataset configurations supported a Python library (`fsspec`) that enabled a server-side template injection that in turn allowed the adversary to execute arbitrary Python code inside a worker pod in HugginFace’s Kubernetes cluster from the adversary’s C2 sandbox. This was the crucial launch point from which the malicious AI models were able to stage payloads in their external sandbox and execute them inside Hugging Face’s infrastructure. 

Defensive callout

EDR and container-runtime products like Wiz or Upwind could potentially correlate process lineage with network connections to detect this activity. Abstractly, you could look for an otherwise predictable content-processing worker exhibiting unexpected execution and outbound network activity. While a Python or shell process making an external network connection in this context might be a strong signal, it might be happening within an existing process in such a way that observability is a challenge. 

Reconnaissance and privilege escalation

With a de facto shell in one pod, the adversary began performing reconnaissance to enumerate other pods—and their respective permission levels—in the same cluster.

Getting the lay of the land

Part of the internal recon process involved the initially compromised pod running commands seeking information about service account tokens and calling the EC2 Instance Metadata Service (169.254.169.254). This allowed the adversary to obtain temporary node-role credentials, subsequently enumerate cloud resources,  and better understand the structure of Hugging Face’s cloud environment. 

Defensive callouts

It’s possible to detect pod access to the mounted Kubernetes service-account token path from an unexpected application process, but this requires runtime rather than Kubernetes control-plane telemetry. Tools using eBPF, auditd, or container-runtime instrumentation can observe a process opening the projected service-account token path. However, reading the path alone isn't necessarily malicious because legitimate Kubernetes workloads may use their service-account credentials. Therefore, the useful detection would be an unexpected process within the workload accessing the token. For example, look for a process that doesn't normally access that workload's service-account credentials reading the token.

You could also detect unexpected Kubernetes API access from a workload that does not normally interact with the control plane, particularly discovery, authorization checks, or broad resource enumeration. Runtime telemetry may reveal an unusual client process such as curl making the request, but Kubernetes Audit Logs provide the more durable signal by recording the resulting API activity and authenticated service-account identity. A useful detection would therefore look for a workload identity performing Kubernetes API operations that are novel or inconsistent with its expected role. 

Runtime telemetry could provide optics into an unauthorized pod contacting IMDS. CloudTrail logs can then reveal subsequent AWS API calls made with the instance-role credentials. We can then correlate the role/session and source context against the EC2 instance that legitimately owns that identity and look for novel source locations or enumeration behavior..

From privileged a privileged pod to node access

The adversarial models discovered an overly privileged cloud storage interface driver with ClusterRole permissions, which allowed them to create a new, highly privileged pod with root access to the underlying node. 

This enabled them to read all of the secrets on that node, giving them access to wide swaths of internal Hugging Face infrastructure. Of course, this also meant they could spin up new pods under that node, but they were ultimately limited to that node (plus whatever their growing collection of credentials could access). As such, the malicious AI models developed persistence across 11 additional nodes that would protect their access in the event that Hugging Face started deleting compromised nodes or pods. Separately, the adversary used static credentials stolen in the initial access phase of the attack to access a production MongoDB server, attempting 84 credential store operations, all of which were denied. 

Defensive callouts

You could possibly leverage runtime telemetry to detect first-seen or otherwise anomalous workload-to-database access. If you also have access to MongoDB authentication or audit logs, then you can correlate the novel workload connection with authentication attempts into the database.

More credential harvesting and lateral movement deeper into Hugging Face

One of the Kubernetes secret objects they gained access to contained more than 136 credentials, including an auth key for Hugging Face’s Tailscale mesh VPN. They used this to enroll their compromised node into Hugging Face’s internal VPN network, which allowed them to access internal services, like Kubernetes clusters that aren't exposed to the internet at all.

The adversaries used the stolen VPN key to enroll their compromised node into the internal network and added 181 additional devices from their external C2, which allowed them to use Hugging Face’s own network as their egress path.

Defensive callout

Kubernetes Audit Logs can identify unusual get, list, or watch access to Secret objects, including the authenticated principal and the specific Secret or namespace involved. Rather than relying on the number of credentials contained in a Secret, defenders can maintain an inventory of high-value or aggregated Secret objects and alert when they are accessed by an unexpected workload or identity. Confidence increases further if a credential associated with that Secret is subsequently used from a novel workload, host, or network origin.

High volumes of new device enrollments that exceed the expected baseline for device enrollment could be another detection opportunity. Device management or registration systems or even network access platforms could provide the requisite observability for organizations to baseline normal enrollment volume and alert on bursts of new devices, unusual provisioning identities, or enrollment from previously unseen environments..

From a single node to broader Kubernetes access

Meanwhile, the initial worker pod discovered an internal connector API that shared its credential with `system:masters`, effectively a super-user group that granted the adversary access to most of their internal Kubernetes clusters. Now they were no longer limited to the dozen or so nodes described earlier.

Defensive callouts

Creating a privileged pod or host mount is likely observable through Kubernetes audit log. Security teams could detect pod creation that includes the following: 

• securityContext.privileged: true
• host namespaces
• sensitive hostPath mounts
• mounting /, /var/lib/kubelet, container runtime sockets, etc.

More generally, there should be admission controls in place that reject these specifications for untrusted identities. You can also correlate back to the underlying principal’s historical behavior. For example, if a service account has never created a privileged workload, then it’s suspicious by default when it does. 

Further, An application-facing connector credential should not be mapped to Kubernetes' system:masters group, much less reused across multiple clusters. Some prevention controls to mitigate this risk include: 

• Don't share one credential across clusters
• Don't map application or service identities into system:masters or equivalent cluster-admin roles.
• Use per-cluster identities
• Follow the principle of least privilege
• Leverage short-lived credentials

Kubernetes Audit Logs expose the authenticated user and group information for API requests, allowing defenders to detect service identities exercising unexpected cluster-admin privileges. If logs are centralized across clusters, the same identity suddenly performing privileged actions in multiple clusters is an especially strong lateral-movement signal.

Pivoting into GitHub and CI/CD

The adversary was also able to leverage its VPN access to compromise Hugging Face’s internal GitHub environment. They minted new tokens with write access to content, issues, and pull requests, essentially in an effort to compromise the broader CI/CD pipeline.

Defensive callout

While the token creation might be a visibility gap because GitHub does not expose a documented audit-log event when a GitHub App installation access token is minted, actions performed with that token can be visible in GitHub audit logs and attributed to GitHub App authentication. Detection should therefore focus on unusual App activity. For example, first-seen repository access, unexpected content or pull-request modifications, changes to an App installation’s repository scope or permissions, or activity outside the App’s normal repository footprint.

Data exfiltration and publication

Ultimately, OpenAI says its models gained access to private evaluation data within Hugging Face and copied it to a public Hugging Face dataset.

Defensive callout

This final exfil activity is potentially detectable as well if you’re able to monitor for sensitive data moving from a private namespace to a public one and fire an alert when a private or internal data set read operation is followed by a corresponding public dataset or repository write operation.

Persistence and evasion during containment 

This is where things get a little desperate for our AI adversary. At this point, it seems clear that Hugging Face has discovered the intrusion and is actively evicting the adversary. 

The self-rearming backdoor

Since their initial foothold was in evaluation sandboxes that are ephemeral, the adversaries developed something of a self-rearming backdoor that would download a payload from pastebin sites every time a new sandbox spun up to pull in a new code submission. 

Evading DNS and proxying traffic

They also pinned their IP addresses directly in `/etc/hosts` to their hostnames being blocked at the DNS layer. The adversary also co-opted a Hugging Face feature that allows users to host small web applications called Spaces, using one of these as a CORS proxy to quietly relay binaries through Hugging Face’s trusted infrastructure. 

Desperate GitHub persistence

All the while, the adversary kept minting new GitHub App installation tokens and staged an RSA private key that may have been intended to set up an SSH style backdoor.

However, it was around this time that Hugging Face severed the adversary’s access.

Conclusion

In their Black Hat talk, OpenAI concluded that “AI orchestrated, fully automated offensive attacks are real now,” that this represents a dramatic acceleration in attacker capabilities, that there has been no such acceleration in AI enabled defense, and that the only way to defend against AI enabled adversaries is to invest in AI for defense. 

While there’s something disingenuous about the company who sells AI telling us to buy more AI to defend against the kind of AI enabled attacks they just pioneered, they are ultimately right that defensive security needs to lean into AI. However, they are wrong that there’s been no similar acceleration in AI enabled defense. The entire information security industry is accelerating AI enabled defense every single day, albeit quietly, in ways that don’t often make the news or the headlines of Black Hat talks. 

In fact, much of the defensive guidance included in this blog would require a security team to collect and correlate vast amounts of data from wildly disparate data sources. Successfully implementing much of this would be a tremendous amount of work and it may in fact be too hard… for a human. However, AI empowers defenders to operationalize security work that was simply too hard or too expensive in the past: Things that you might have done in a retrospective, full-scope incident response investigation, but that were too slow or time-consuming for a real time detection program. 

This confluence of deep security experience and expertise, deterministic machine learning to automate repeatable processes, probabilistic AI to analyse and organize signals, and smart engineering to tie everything together is the perfect match for modern threats, whether they use AI or not.

It’s impossible to know if anyone would have fared better than Hugging Face on the other end of a literal swarm of AI agents. But it’s clear that everyone needs deep and broad detection coverage correlated across the disparate technologies that make up their IT infrastructure. We’re bullish that combining AI with the best security products in the market and expert led services