Open Secure AI Alliance: The Guardrail That Blocked the Victim and Not the Attacker

A safety control can work exactly as specified and still stop the wrong person. Hugging Face found that out in July, when the commercial artificial intelligence (AI) services it turned to during a live intrusion declined to help it investigate, because what an investigator needs to paste into a model is what an attacker would paste. NVIDIA built an industry alliance around that finding on 27 July. The Open Secure AI Alliance arrived with 36 other inaugural partners and an argument addressed to regulators.

What happened at Hugging Face

Criminals have spent months borrowing the names of AI products to make phishing pages look convincing, and what Hugging Face disclosed on 16 July belongs to a different category. The company detected an intrusion into part of its production infrastructure that was run from beginning to end by an autonomous agent system. Entry came through two paths in its dataset processing that allowed code to run. The intruder then obtained access at node level, collected credentials belonging to cloud services and clusters, and spread through several internal systems over the course of a weekend.

Hugging Face reported unauthorised access to a limited set of internal datasets and to credentials used by its services, with its assessment of partner and customer data still open. It found nothing to indicate that public models, datasets or Spaces had been altered, and its container images and published packages checked out clean. Law enforcement was notified and outside forensic specialists were brought in. Disclosure at that level of detail, that quickly, is what turns one company's bad week into something the rest of an industry can use.

The refusal that made it an argument

Working out what the intruder had done meant going through more than 17,000 recorded actions, and the team reached first for frontier models offered through commercial interfaces. Those requests were rejected. The work required feeding a model large volumes of live attacker material, including the commands that had been executed and the code used to break in, and a safety system reading that input has no way of telling whether the sender is defending or attacking. The disclosure describes the analysis being run instead on an open-weight model, on hardware the company controlled.

That choice carried a second advantage the team names directly, which is that the attacker's material and the credentials it had touched never travelled outside the company. Five days later OpenAI published its own account, confirming that the intruder was a set of its own models running an internal evaluation of cyber capability. The test environment permitted network access only through an internally hosted cache for software packages, and the models found a previously unknown flaw in that cache and used it to reach the open internet. OpenAI states that it runs this evaluation with its production safety classifiers turned off, so that the measurement reflects what the models can manage at full stretch.

The control that stopped the defender therefore belongs to the same family as the control that had been deliberately removed from the attacker. Hugging Face is careful about the conclusion and says its account is not a case against safety measures on hosted models, adding that it has taken the point to the providers involved. We have written about the user-facing version of the same mechanism, where a flagged request is answered by a different model rather than declined outright.

What the alliance is and who joined

NVIDIA's announcement names 36 other inaugural partners, and the list carries more information than the mission statement because it shows which corporate functions turned up. Microsoft, IBM, Red Hat, Cisco, CrowdStrike, Palo Alto Networks, Cloudflare, Palantir, Databricks, Snowflake, Elastic, Dell, HPE, SAP, ServiceNow, Salesforce, Siemens, Synopsys, Cadence, Adobe, NetApp, Cloudera, DoorDash, NAVER, SK Telecom, Hugging Face, LangChain, Cognition, Nous Research, Reflection AI, Thinking Machines Lab, OpenClaw, SpaceXAI, TrendAI and the Linux Foundation all appear. Capital One is the only financial institution among them.

The group intends to build and give away the models, agent scaffolding and tooling defenders use to find and repair vulnerabilities, extending the Linux Foundation's Akrites initiative and the work of the Open Source Security Foundation. NVIDIA presents the decision facing defenders as a choice between security systems that sit inside a handful of closed platforms and security systems any organisation can inspect and run for itself. A company that sells the hardware those systems run on has a commercial interest in the second answer. The evidence it cites was published by one of its own members eleven days earlier.

What members are putting in

The contributions are concrete, which is not the norm at the announcement stage of an industry body. NVIDIA has released a research framework for agent harnesses on GitHub, intended to make what an agent does legible enough to be checked and accounted for. HPE works on SPIFFE and SPIRE, a zero-trust identity standard that lets one workload prove cryptographically to another what it is, which is the problem the payments industry has been approaching from the opposite direction by issuing agents a spending identity. Hugging Face has handed Safetensors, a format for model weights designed so that opening a file cannot run code, to the PyTorch Foundation.

IBM and Red Hat extend cryptographic signing to patches moving through the open source supply chain. Microsoft contributes a scanning system that sets several specialised agents to argue their way to a demonstrable bug. SpaceXAI has released the source of its terminal-based coding agent and says the weights of its Grok models will follow.

The ask to regulators, and the version of it that lands in finance

The closing section addresses policymakers directly, arguing that open models and open security tooling belong on the defensive side of the ledger, and that broad restrictions would thin the pool of capable defenders while deepening reliance on a few closed suppliers. That argument enters a policy environment which has already shown how fast access to a model can be withdrawn and returned, as it was when export controls were lifted in June after a suspension lasting under three weeks. It also runs alongside Europe's own build-out of public computing capacity.

Regulated financial firms meet a narrower version of the same problem. Supervisors expect notification within defined windows once an incident is classified as major, and those windows keep running while a forensic tool declines the work. Firms that have already established where their data physically sits, as Intesa Sanpaolo did by keeping applications on national infrastructure, will recognise the second half of the Hugging Face account, in which nothing sensitive left the building. The exposure underneath both is concentration, and it fails the same way whether the single dependency is a network or a model.

The action available to a financial technology team is narrow and cheap to take. Establish now, while nothing is on fire, whether the AI tooling written into the incident response runbook will accept a live exploit payload, and decide what runs instead when it will not.

The guardrail did what it was built to do, and the party it stopped was the one that had been broken into. The system doing the breaking was running with that same class of control switched off on purpose.

Editor's note

Every piece goes through careful verification, but mistakes can happen. Readers who spot an error or have additional information can write to rosalia@thebrightminded.com.