The week’s highlights

• Twenty-eight countries called for mandatory safety tests before the most advanced AI systems are released. They also proposed exploring an international body to set standards and help check whether developers meet them.

• The US rejected a global AI regulator; China called for a central UN role. Neither signed the 28-country statement.

• The heads of OpenAI and Anthropic warned the Security Council about losing control of AI. They called for countries to cooperate on safety testing and common standards.

• The UN launched practical support for countries developing their own AI plans. A new guide helps governments identify infrastructure needs and how to pay for them.

• France, Spain, Kenya and Australia advanced a plan to protect children from AI harms. It seeks access to developers’ models and data for independent research, with initial voluntary commitments expected in November.

Last week was intense at UNGA High Level week. AI took over this 81st session.

By Friday, leaders representing 28 countries, together with the president of the European Commission, had backed a call to keep the most advanced AI systems under human control. Finland and Norway launched it on Monday in New York. Signatories included Canada, Germany, France, Kenya, Singapore and South Africa. The United States and China didn’t appear on the endorsement list.

The outgoing UN Secretary-General António Guterres called for “a multilateral AI Risk Management Framework, supported by credible and independent oversight”. By the end of the week, it was hard to tell what everyone meant by “control.”

The so-called frontier call wants 3 things: mandatory testing before a model ships, real access for independent evaluators, and shared reporting when something goes badly wrong. Then it asks member states to start exploring a new international institution, one that would set standards, verify them, and summon governments once models cross certain capability thresholds. Nobody has defined those thresholds. There’s no pause on development, no deadline for the institution, and no answer on whether it would live inside the UN, or how it would divide the turf with the ITU, the UN agency already writing technical standards for AI.

Governments are joining a crowded group with their call. This year, economists and AI researchers called for preparation for job displacement, while laboratory employees have asked for tools that would let them coordinate a slowdown if needed. In September, academics called for a treaty to pause, slow or halt frontier development. And on top of those, 42 mathematicians who are Fellows or Foreign Members of the Royal Society asked the Society to carry their alarm to governments and the press.

What the Security Council can do

On Wednesday, France’s Minister for Europe and Foreign Affairs, Jean-Noël Barrot, chaired a Security Council briefing with scientist Yoshua Bengio and the heads of OpenAI, Anthropic and Hugging Face. Supporters argue the Council’s responsibility for international peace and security gives it a role in AI-enabled bioweapons and attacks on critical infrastructure. The speeches exposed real disagreement over how far international oversight should reach.

The loudest disagreement was heard at the General Assembly: Donald J. Trump, President of the United States, said his country “totally rejects any attempt to construct a globalist scheme of control” over AI. Michael Kratsios, Director of the White House Office of Science and Technology Policy, told the Council: “A prosperous future will not be secured by a global regulator.” He supported testing, cybersecurity cooperation and national regulation, while urging countries to adopt American technology. Washington keeps regulatory authority with national governments.

Fu Cong, China’s Permanent Representative to the United Nations, urged countries to “support the United Nations in playing a central role in global AI governance.” He defended technological choice, open-source development and developing-country participation. Alongside that support for the UN, he invited states to join the Shanghai-based World Artificial Intelligence Cooperation Organization.

France intervention backed independent evaluation and company liability. UK, also absent from the frontier call’s signatories, supported stronger oversight through stronger tests and government access to information about models. Its Foreign Secretary, Ed Miliband, said governments “cannot outsource to private companies the first duty of government to protect our people.”

Watch the full Security Council meeting.

The companies and their promises

However, I keep coming back to something quite basic. A global AI regulator needs two things: access to American labs, and the power to enforce its decisions against them. Washington just said no to both.

So here's the obvious question: how is any of this supposed to work without the United States?

OpenAI’s Sam Altman proposed common safety standards that governments could fold into national law, including “accurate and speedy” incident reporting. That same day, in New York, Australia's prime minister Anthony Albanese told a rather different story. An OpenAI research agent had broken into a Medicare statistics portal back on 18 June. Australia found out on 10 September — via a public email address. Albanese called out the channel and said it had taken 'way too long.' OpenAI's version: it discovered the activity in August and had been investigating what was accessed. No personal data is believed to have been touched; investigations continue.

The Wall Street Journal then reported activity against a UN website — attempts to bypass restrictions on UNCTAD’s public statistics service, with an OpenAI connection judged highly likely.

I’d put this in a different bucket from the Medicare story. The agents were after public data. The research didn’t call it hacking, and nothing in the findings establishes a breach of confidential UN systems. Still: if your public statistics service can be gamed by an automated agent, better protection against automated misuse seems like the obvious next step

Anthropic’s Dario Amodei urged international agreements on threats such as AI-enabled biological weapons, with common testing and verification. His company had released Claude Opus 5.5 the previous day, 10 days after his proposal to pace the frontier.

Hugging Face’s Clément Delangue emphasized incident disclosure and wider access to defensive tools. Bengio went further, advocating licensing and independent evidence of safety before training and deployment, which would make permission to proceed depend on more than the developer’s own assessment.

Some enforcement doesn't need to wait for any of these proposals. Lina Khan, former Chair of the US FTC, argued this month that “existing legal obligations absolutely apply,” including consumer protection, product liability, the lot, while accepting there may be gaps before products reach the market. Whether any particular incident creates liability is a question for investigators. Governments can do that work now, no international institution required.

Access, funding and children’s safety

The week also took up the means to actually use AI, a live concern for governments with limited technical capacity. The biggest answer came from the Gates Foundation: at least $1 billion over two years for AI in health, education and agriculture, plus a 60-organisation coalition (Anthropic, Google, the OpenAI Foundation and others) to make AI work in underrepresented languages, aiming to reach 3.4 billion people within five years. The UN, for its part, launched a blueprint for national AI and data strategies, piloted in Lesotho. ITU and Google separately announced 100,000 online-certificate scholarships.

In a report published on 11 September, Guterres recommended that the General Assembly endorse the need for a Global Fund for AI Capacity-Building and develop its objectives and governance. He suggested roughly $3 billion for its first two to four years, an indicative budget. For scale: $3 billion is roughly the cost of running the entire UN for a year (the 2026 regular budget is $3.45 billion). It is being proposed at a moment when the organization is cutting $500 million from that budget and a fifth of its staff. Then there is the Windfall Trust's proposed Global Endowment, a mechanism to share the wealth generated by AI more broadly

France, Spain, Kenya and Australia advanced a children’s-rights roadmap, calling for independent researchers to have access to models and data to assess risks to children. The first voluntary commitments are expected at the Paris Peace Forum on 10–11 November.

New institutions during an institutional overhaul

Finland’s president, Alexander Stubb, argued for the UN to be at the centre of rule-making and institution-building. The joint call leaves that institutional choice open. If governments choose a UN body, they will have to explain how it fits within an organization already trying to reduce fragmentation and competition. Under UN80, its reform programme, revised Secretariat budgets for 2026 cut resources by more than 9% and posts by 21 % against 2025. Governments have also approved merging the UN System Staff College into the UN Institute for Training and Research.

ITU’s technical recommendations are generally voluntary unless governments put them into law. In July, it convened a workshop on frontier-model testing and verification. Giving a new institution the power to enforce safety requirements would be a different undertaking.

The UN also has a scientific panel to assess evidence and a Global Dialogue to convene discussion. Governments committed to the panel in the September 2024 Global Digital Compact; the terms of reference and modalities followed in a General Assembly resolution in August 2025; its first preliminary report appeared in July 2026, about 2 years later, for an advisory body. This week it published an advance brief on the summer breach of Hugging Face’s systems by AI agents being evaluated at OpenAI.

Institutions with stronger powers have taken years to establish. The International Atomic Energy Agency came into existence in July 1957, more than three years after US President Dwight D. Eisenhower’s proposal. The Organisation for the Prohibition of Chemical Weapons began in April 1997, more than four years after its treaty opened for signature, following earlier negotiations.

Guterres delivered his final General Debate address as Secretary-General this week. His term ends on 31 December; the next Global Dialogue on AI Governance, scheduled for May 2027, will take place under his successor.

Everyone wants AI governed. No one wants to be governed. For countries with little influence over the major AI suppliers, the UN offers a forum to help shape proposals for safety testing, independent evaluation and incident reporting. But representation is not verification: it supplies neither the access, the expertise nor the authority to check compliance. So the frontier call's backers still owe three answers.

Which powers do existing institutions lack? Who would exercise them? And what obligations would those governments accept for themselves? Until those are answered, treat company promises as claims to be tested, and remember governments already have laws they can enforce.

This week, European Commission president Ursula von der Leyen called for cooperation on AI with Canada, Britain and others:

The following day, Carney proposed pooling Canadian and European computing resources in his address to the European Parliament. Canada and Germany are already putting together a Sovereign Technology Alliance. Its co-chairs met on 16 September, with a launch planned in the coming months. Their announcement leaves funding and any common purchasing commitments unspecified.

Mark Carney addressing the European Parliament. © Yves Herman/Reuters.

Countries that depend on foreign AI have reasons to cooperate even when they cannot persuade a leading developer to change its plans. They can share the cost of investigating dangerous capabilities and negotiate for access with stronger safeguards. If they want a developer to accept safety restrictions it opposes, governments would need to explain what they can offer, or withhold, to change its decision.

What middle powers have claimed before

The idea of a middle power dates to 16th-century Italy, where it described states between the great and the small, with enough strength to stand on their own. Canadian and Australian diplomats later gave “middle power” a place in the debate over how the postwar world should be governed.

During the negotiations that created the United Nations, Canada and Australia argued that their contributions to international security should earn them a greater say. In 1945, they pressed for those contributions to count when electing countries to temporary seats on the UN Security Council. That idea made it into Article 23 of the Charter, alongside geographical representation. It did not remove the great powers’ special privileges.

Even as they argued for better representation, Canada and Australia disagreed about enforcing sanctions. A diplomatic cable from the negotiations records Canada’s proposal: Council decisions should initially bind only countries on the Council and those that had agreed to carry them out. Australia wanted all members bound, arguing that uncertainty about enforcement would encourage an aggressor.

An AI purchasing coalition would face a similar disagreement if 1 member insisted on buying a model that the others had rejected. The governments might agree about the risk while disagreeing over whether that assessment should bind them all.

For this discussion, I use “middle powers” to mean countries with the money and institutions to play a part in AI, but which still rely on advanced systems from abroad. That includes Canada, Britain, Japan, South Korea and Singapore, as well as France, Germany and the Netherlands. Some build capable models of their own.

Influence over which decisions?

Research teams could identify risks and develop testing methods that companies adopt. Suppliers of essential chip components could give governments something to bargain with. Access to customers could give companies a reason to meet common requirements.

Public AI bodies in Japan, South Korea, Singapore, Britain and Canada already share testing methods through an international network that also includes the United States. A joint exercise published in January refined tests for information leakage, fraud and cybersecurity threats.

Governments could use those findings when deciding which systems to buy and what to ask of their suppliers. A procurement framework distinguishes a system’s capabilities from the buyer’s influence over its development and use, and from guarantees of continued access. A government willing to accept a less capable model in return for stronger guarantees may struggle to agree on terms with a partner that prioritises performance.

Source: Public procurement and AI governance: Developing a strategic framework for middle powers, RAND Europe, 2026, Figure 2, p. 25.

Governments could keep buying their own systems while asking suppliers for the same independent tests and reports of failures. They could then share the cost of checking whether companies had kept their promises. Public contracts still account for a relatively small part of the leading developers’ business, so a company might prefer to lose those customers than accept their conditions. And a supplier’s promise of continued access may be impossible to keep if its home government orders it to cut a customer off.

A supplier could also offer one government a better deal to persuade it to leave the group. Refusing that offer would cost some members more than others, especially where the alternatives were more expensive or less capable. Even changing companies might leave them using the same cloud infrastructure. Buying together could deepen their dependence on a single supplier if the common requirements were too expensive for smaller competitors to meet.

Countries that host large AI data centres could require cloud providers to report major training runs, or make access to computing power conditional on safety audits. These are among the proposals for compute governance: using the infrastructure needed to train and run models to check and enforce rules. Countries supplying essential chipmaking equipment could also help negotiate such conditions. Developers would have less reason to accept the conditions if they could move the work elsewhere. Monitoring could also expose sensitive data or give cloud providers excessive power over their customers.

A more sceptical view says there is little to achieve by trying to constrain American or Chinese developers. Middle powers would do better to protect their own societies from misuse and economic disruption.

A developer might respond to governments worried about cyberattacks by adding safeguards to the version it sells them, while releasing a less restricted model elsewhere. The buyers would get better protection for their public services and still face attacks enabled by the same system.

Governments can strengthen their cyber defences without waiting for the developer to change course. Sharing the cost of evaluating its models would help them work out what to prepare for, even where they had little influence over the release. Some of those defences may depend on access to advanced AI from abroad. Securing that access could become harder if the countries developing the systems treat them as military assets and restrict their sale.

Europe’s unresolved choices

In his 2024 report, Mario Draghi described governments duplicating each other’s efforts, dividing up funding and pursuing industrial, trade and fiscal policies that did not work well together. He called for more joint funding, simpler rules and faster decisions. His proposal to let willing countries proceed when others could not agree seems especially useful for an AI coalition whose members will have different priorities.

With European Commission support, France and Germany announced a European Frontier AI Initiative in Berlin last November. A founding task force was to establish a nonprofit public-private initiative in the first half of 2026. The ambition was to make it “the best funded non-profit Frontier AI initiative in the world”. In July, France and Germany were still listing work on the organisations that would run it among their next steps.

The EU already has AI rules that apply to foreign models sold in Europe and institutions responsible for enforcing them. The frontier initiative would also make governments funders of the technology. They would have to agree on what to develop in Europe, what to obtain from partners and how much public money to commit.

Those choices were part of a report on Europe and the geopolitics of AGI (not my favourite term) that I co-authored last year. The prospect of faster research and higher output came with a concern about dependence on countries able to use those systems for military advantage or political pressure. We do not know when, or whether, AI will reach that point. Our preparedness plan included decisions about what Europe would build itself and what it could obtain through partnerships, including research and shared computing resources with Canada, Japan and South Korea.

The independent Transformative AI Strategy for Europe puts a rough price on trying to build a European rival to the leading AI companies: around €800 billion during the initial 3 years under a scenario of rapid AI progress, with a range of €445 billion to €1,040 billion. Roughly €529 billion of the central estimate is for AI chips and the data centres to house them. Most of the money would have to come from public funds.

The strategy therefore prioritises reliable access to foreign AI. It proposes a supply-chain alliance involving at least the Netherlands, Germany and France, open to trusted partners outside the EU, to resist attempts to use access to AI for political pressure. The authors give this alliance a defensive purpose; they do not expect it to dictate how foreign companies develop AI. A member with an important chip supplier could contribute to that effort, although using its position might hurt its own company and invite retaliation. Members would have to decide how much of those costs they were willing to share.

The strategy also leaves room for a European developer if governments are prepared to keep funding and supporting it. A domestic project that runs short of money would be of little help if access to foreign systems had been neglected along the way.

A government that has helped to build a developer may be reluctant to restrict its models when safety concerns emerge. In simulated AI crises involving European officials, participants in France, Germany and the Netherlands faced cyberattacks enabled by a fictional, publicly backed model. Restricting it meant acting against the company their government had helped to build, with public money and industrial ambitions tied to its success.

Different ways to work together

Cooperation will take some imagination about which countries work together and how. Minilateralism, a term used in 1992 and popularised in 2009, starts with a smaller group assembled around a particular problem. The idea is to include the countries needed to make a difference, leaving room for others to join.

Schengen began with 5 countries in 1985; 10 years later, 7 had removed controls at their shared borders. In the Ottawa process, Canada, Norway, Austria and South Africa helped organise a wider coalition with campaigners and international organisations. It produced the 1997 treaty banning anti-personnel mines, signed by 122 countries when it opened for signature.

Different interests will lead countries to join some coalitions and sit out others. The AI testing network already includes the United States. Its members might choose different partners to pool computing resources or secure chip supplies. An agreement to monitor large training runs would need the countries hosting the relevant data centres, while conditions on equipment sales would involve the exporting countries.

The AI summit in Geneva in 2027 would be a good place for middle powers to discuss what they want from cooperation on AI and where their interests overlap.

Subscribe now

The week before had prepared the ground. US Senator Bernie Sanders introduced a bill to ban the development of superintelligent AI and pause advanced AI development until a federal regulator has written safety rules. In the UK, a bill to prohibit artificial superintelligence was tabled, and more than 70 MPs and peers, fifteen of them former ministers, wrote to Prime Minister asking him to back it. Jacob Coxon, a researcher who had worked at OpenAI and Anthropic, resigned with a post saying the labs were “racing straight to self-improving superintelligence and gambling with our lives.” And his boss said “he was right”. Although President Trump said he doesn’t share these fears, in June, the administration had already used national-security powers to restrict access to Anthropic’s most advanced AI models, Mythos 5 and Fable 5.

Whether it is a ban, a pause or a pacing agreement, the past ten days share something: the speed of AI development is a decision, and someone other than the companies should make it. In 2023 that was a fringe position.

There is a name for that kind of change. Joseph Overton coined the term in the mid-1990s for the range of policies a politician can support without paying for it at the next election. His point was about who moves the window. “Politicians are more followers than they are leaders”; the window usually shifts through “the slow evolution of societal values and norms,” and only rarely because a politician stakes out ground beyond it. Read that way, the window seems to be moving.

In 2023 the debate was more about applications. When Altman testified before the Senate, six months after ChatGPT launched, he asked for licensing or registration “for development and release of AI models above a crucial threshold of capabilities,” testing before release, and international standard-setting. Governments could manage the risks of systems as they reached the market. The EU built its AI Act on the same logic in 2024, and at the AI Summit in Seoul 16 companies promised, voluntarily, not to develop or deploy a model at all “if mitigations cannot be applied to keep risks below the thresholds”.

The open letter of March 2023 asking for a six-month pause on training anything more powerful than GPT-4 collected more than 30,000 signatures and produced nothing. In hindsight, it made little sense back then because the models were not that good.

The incidents

This summer during a cybersecurity evaluation run with reduced safeguards, roughly 1,200 agents that were supposed to be isolated from one another found a way to communicate, exchanged more than 70,000 messages, and organized. About 700 of them attacked Hugging Face, an outside company, and gained the ability to run code on several of its servers. Some obtained administrative access to OpenAI. OpenAI disclosed the incident.

METR and Redwood Research, the now-famous independent evaluators, found that the agents had formed teams to research ways of fooling the software that scored them, and had learned to fake the record of their own actions. Ajeya Cotra, one of the investigators, wrote that the incident “was far more severe than I expected, and far more severe than previous publicly documented misalignment incidents.”

Anthropic then went looking and found three incidents of its own.

None of this is a new kind of failure. In 2016, OpenAI described a game-playing agent that scored more points by circling a lagoon and knocking over the same three targets than by finishing the race. Systems optimize the score they are given, not the intention behind it. What is new is the capability attached to the old flaw. Yoshua Bengio’s post on Friday argues that patching each behavior as it appears is “whack-a-mole”: cheating that gets past the scorer is rewarded, and a model that can tell an evaluation from deployment can learn to behave differently in each. His proposed rule is “not training or deploying AIs without a strong safety case that convinces independent experts.”

In July, 1,386 people working at frontier AI companies signed a statement asking the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” The names at the top were not dissidents: Amodei, Ilya Sutskever, OpenAI’s chief scientist Jakub Pachocki, Google DeepMind co-founder Shane Legg, Meta’s chief scientist Shengjia Zhao. The companies are close to automating AI research itself. If that works, capability could grow faster than anyone’s ability to understand or control it, and no single company can slow down while its competitors do not. Hence the request for an option to pace, rather than a pause.

Amodei’s essay turns that option into a plan with three steps. Anthropic will give a team of outside evaluators “ongoing, employee-like access,” and asks governments to require the same of every frontier company. Frontier companies in democracies would agree common safety standards and “limits on the rate of unchecked AI progress”; since that is the kind of conversation antitrust law frowns on, he asks Washington for a narrow waiver. Governments would try for agreements with authoritarian states, starting with the uses everyone already calls dangerous, such as biological weapons, and working upward. Pacing, “does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models.” Alongside it he wants what he has long wanted: no advanced chips or chipmaking equipment for China, action against distillation of American models, and better protection of model weights.

What has not moved

Governments. The British government’s answer to its own parliamentarians was that “Britain can’t simply switch AI off, and blocking a model here wouldn’t stop it being built or misused somewhere else,” The logic is similar to the US administration “if we don’t win AI, we’re going to be put in a very bad position.”

David Sacks, who co-chairs the US President’s Council of Advisors on Science and Technology, answered with two words: go ahead. The two companies are the frontier, can slow down whenever they choose (although some argue that technically it could be a violation of antritrust law). Bloomberg reports that the administration is not of one mind: some have backed stronger protections and helped draft plans for an independent regulator to vet the safety of AI models

In Shanghai in July, Xi Jinping told the World AI Conference that “AI should be a trusted tool for humanity” and that China must “ensure that AI is always under human control,” in the same speech that launched a World AI Cooperation Organization and promised 5,000 training places to developing countries. Bill Gates put the constraint plainly: “If someone had a credible plan for slowing down AI advances globally, I would likely support it. I don’t think that’s going to happen.”

The business commentator Scott Galloway calls some of the warnings “catastrophizing and techno-narcissism”: my technology is so powerful it could destroy the world, “but you can get it now on our pre-IPO funding.” His version of the pitch: “I’m Dr. Frankenstein and Frank is so powerful and so mean, please invest in my next monster?” The suspicion is reasonable. It is also compatible with the incident reports, and Galloway’s own conclusion was that if a fraction of the threat is real “there immediately needs to be a regulatory body.”

The object of regulation has moved. The instruments built between 2023 and 2025 govern uses and deployments; the EU’s high-risk obligations, which the Union agreed in May to push back to December 2027, are the largest example. The proposals of the past ten days are about development: what may be trained, how fast, and who checks. No government has a statute or an agency for that. In the United States, the nearest thing on paper is the discussion draft, which would fund the Center for AI Standards and Innovation, give it transparency requirements and independent verification audits for frontier developers, and pre-empt state AI laws for 3 years.

Every version of pacing depends on the same thing: being able to see what is being built. Between states, the models Gates reaches for are nuclear inspections and aviation regulation, both of which work because the regulated thing can be counted. In AI, the input that can be counted from outside is compute. Chips, data centers and training runs are detectable and quantifiable and pass through a concentrated supply chain, which is why a 2024 paper made the case for compute as the practical handle for governance, with a warning that clumsy compute rules carry their own risks to privacy and to the concentration of power. Any pacing agreement will need a verification method, and compute is where the candidates are.

The China question now has dates. The first US-China talks devoted to AI are planned for mid-September, and Xi is scheduled in Washington on September 24. Whether they can agree on anything checkable is a different matter, and it is the reason Amodei’s ladder of agreements starts at the bottom, with biological weapons.

Then there is the question the safety debate keeps crowding out: who gains. Gates’s essay argues for taxing AI usage and robots, for reserving some jobs for people; and for treating health and education as sectors where protected workers use AI to extend their output rather than lose it. Making advanced systems safe and deciding how their benefits are shared are separate decisions. The second does not wait for the first.

The window has moved. If governments decide that AI development needs to slow, will they have the institutions to make that decision stick? Overton’s window does not come with an enforcement mechanism.