UK AI Verification: Narayan Tells Frontier Labs Not to Release Models They Cannot Control

UK AI verification, the checking of what developers claim about how their models were built and tested, is one of the measures in an artificial intelligence (AI) risk package that minister Kanishka Narayan presented on 12 October at Imperial College London’s Base One incubator. His message to the companies building the most powerful systems was direct: act on what their scientists warn about, look into behaviour nobody expected and hold back any model whose risks they cannot adequately control. The package also covers cyber defence, a review of how serious AI incidents and close calls should be reported, and new work on how to respond when autonomous AI agents go wrong.

The labs and their researchers have spent the months since July describing frontier AI as dangerous, and those descriptions have mostly circulated as headlines. London’s package starts treating them as claims that can be checked and acted on.

The labs have been selling the danger

Pacing the Frontier, a statement published in July and signed by 1,386 employees of frontier AI companies, among them Anthropic chief executive Dario Amodei and OpenAI chief scientist Jakub Pachocki, asked the US government to back an international effort to build the tools needed to pace AI development. Anthropic researcher Jacob Coxon resigned in September with a public warning that went viral. OpenAI chief executive Sam Altman then told Fortune that OpenAI would not go public in 2026, calling the moment ill-advised given the safety work still to do. Anthropic’s IPO prospectus, obtained by Reuters and reported by CNN, describes its models as a possible catastrophic or existential risk to humanity.

Each of those moments put responsibility at the centre of how the labs present themselves, while they compete for users, talent and investors. The Bright Minded has examined safety as marketing and the timing of these warnings in detail. A warning that strengthens a company’s reputation keeps working for as long as nobody asks to check it.

What London announced

The Cabinet Office announcement was issued jointly with the AI Security Institute (AISI), the National Cyber Security Centre (NCSC) and the Department for Digital, Culture, Media and Sport. It leads with four commitments.

The first extends a programme that puts frontier AI to work on government cyber defence, after two pilots that helped uncover and repair over a thousand weak points in government systems. The second explores stronger protections inside the Cyber Security and Resilience Bill, along with the rules around it, so that hospitals, utilities and other essential services are covered against AI-related risk, including loss of control, the scenario in which autonomous systems slip out of human oversight. The same commitment includes a study of how serious incidents involving AI, and narrowly avoided ones, should be spotted and reported and how lessons can travel quickly, with the detail settled together with Parliament, specialists and the operators expected to deliver it.

The third brings the NCSC and AISI together to prepare a response for incidents caused by AI agents, meaning software that carries out multi-step jobs with minimal human oversight. The government’s reasoning is that an agent incident could spread faster and further than the emergencies its response systems were built for. The fourth is the confirmation that AISI scientists are now evaluating the newest advanced models, after work with leading labs and the US government.

The minister borrows the labs’ own words

Narayan’s instruction to the labs carries no legal force yet, and it is phrased in the language of responsibility the labs themselves have used all year. Heeding researchers, investigating unexpected behaviour and withholding a system that cannot be controlled are standards the industry already claims for itself.

The line about scientists lands a month after Coxon’s resignation, which carried one researcher’s warning about the race to build ever more capable models to a mass audience. Narayan’s framing moves that conversation from social media into a government press release. A lab that tells investors its models could pose an existential risk and then ships them has now heard from the government of the country that will hold next year’s G20 presidency, with AI at its centre, which half of that sentence it is expected to act on.

What £13,000 bought

The most useful number behind the package sits in a government case study published in June by the Department for Science, Innovation and Technology and the NCSC. It describes weekly in-person hackathons run by the Government Cyber Coordination Centre, a partnership between the two bodies, in which teams pointed frontier models at public government code to find weaknesses before attackers did.

Participants logged 407 findings across nine government organisations in one month, among them critical weaknesses that could have let outsiders get past logins, read data or run their own code on government services. Every critical weakness was fixed, no finding showed evidence of exploitation, and the bill for model usage came to £13,000. One team built a six-step agent chain in which each stage challenged the one before it, and a team member still checked every line by hand.

The case study’s own conclusion is the least dramatic sentence in the whole debate: the structure built around a model mattered more than which frontier model sat inside it. Frontier models of the kind their makers describe to investors as a possible risk to humanity spent that month hunting bugs in government code. The next phase extends the pilot to more departments, more models and closed-source code.

Verification puts the warnings to the test

The government calls AI verification an emerging field that tests whether what developers say about building, testing and assessing their AI systems holds up under scrutiny. It says Britain’s specialist expertise and organisations put the country in a position to lead the field, and promises details of its support in the coming months.

Verification is audit applied to AI. A lab’s statement that a model was red-teamed, meaning attacked on purpose to expose its flaws, or that it passed a set of safety evaluations becomes something a third party can test, in the same way an auditor tests a balance sheet. The safety story the labs tell turns into a set of assertions with someone checking them.

Finance already runs on this logic. Financial firms in the European Union have had to report major technology incidents to their supervisors under the Digital Operational Resilience Act (DORA) since January 2025. UK banks, insurers and payment and e-money companies had until March 2025 to show they could keep their important business services within set limits of disruption, under Financial Conduct Authority (FCA) and Bank of England rules. Those payment and e-money firms sit at the core of financial technology, which means regulated finance and its fintech suppliers already practise the incident reporting and resilience testing that the UK is now examining for AI.

Where the UK sits

The UK regulates AI through existing law and sector regulators, with no single horizontal statute. Katharine Wooller of Softcat argued in May that the UK’s patchwork approach leaves many financial firms holding back on AI while they wait for regulatory clarity.

The new package keeps that model and adds machinery to it: possible incident reporting duties through the Cyber Security and Resilience Bill, verification as a field to build and model evaluation through AISI. Narayan said the government would legislate wherever it has to and would lean on British scientific and technical institutions alongside the law. The detail of any binding duties will be agreed with Parliament and the operators set to carry them.

The warnings the labs and their researchers issued this year were written to be believed. London took them at their word, and a warning taken at its word becomes a liability its author has to manage.

Editor’s note

Every piece published on The Bright Minded goes through careful verification, but mistakes can happen. If you spot an error, have additional information, or want to flag anything, write to rosalia@thebrightminded.com.