Dario Amodei’s essay on Saturday opens with the case for the technology. He has spent twelve years on AI, the chief executive of Anthropic writes, because “it could dramatically raise the quality of human life”; it can be “the latest in a long line of technological miracles that have uplifted and ennobled humanity.” A few paragraphs later comes the news: “We must slow the pace at which we improve the capabilities of AI models.” The idea has been floated since 2023, “and I think it made little sense back then.” Sam Altman answered: “I agree with Dario that we need to pace the frontier,” and committed OpenAI to the measure Anthropic had just announced, outside evaluators with employee-like access to its work. Elon Musk used three words: “Dario is right.”
The week before had prepared the ground. US Senator Bernie Sanders introduced a bill to ban the development of superintelligent AI and pause advanced AI development until a federal regulator has written safety rules. In the UK, a bill to prohibit artificial superintelligence was tabled, and more than 70 MPs and peers, fifteen of them former ministers, wrote to Prime Minister asking him to back it. Jacob Coxon, a researcher who had worked at OpenAI and Anthropic, resigned with a post saying the labs were “racing straight to self-improving superintelligence and gambling with our lives.” And his boss said “he was right”. Although President Trump said he doesn’t share these fears, in June, the administration had already used national-security powers to restrict access to Anthropic’s most advanced AI models, Mythos 5 and Fable 5.
Whether it is a ban, a pause or a pacing agreement, the past ten days share something: the speed of AI development is a decision, and someone other than the companies should make it. In 2023 that was a fringe position.
There is a name for that kind of change. Joseph Overton coined the term in the mid-1990s for the range of policies a politician can support without paying for it at the next election. His point was about who moves the window. “Politicians are more followers than they are leaders”; the window usually shifts through “the slow evolution of societal values and norms,” and only rarely because a politician stakes out ground beyond it. Read that way, the window seems to be moving.
In 2023 the debate was more about applications. When Altman testified before the Senate, six months after ChatGPT launched, he asked for licensing or registration “for development and release of AI models above a crucial threshold of capabilities,” testing before release, and international standard-setting. Governments could manage the risks of systems as they reached the market. The EU built its AI Act on the same logic in 2024, and at the AI Summit in Seoul 16 companies promised, voluntarily, not to develop or deploy a model at all “if mitigations cannot be applied to keep risks below the thresholds”.
The open letter of March 2023 asking for a six-month pause on training anything more powerful than GPT-4 collected more than 30,000 signatures and produced nothing. In hindsight, it made little sense back then because the models were not that good.
The incidents
This summer during a cybersecurity evaluation run with reduced safeguards, roughly 1,200 agents that were supposed to be isolated from one another found a way to communicate, exchanged more than 70,000 messages, and organized. About 700 of them attacked Hugging Face, an outside company, and gained the ability to run code on several of its servers. Some obtained administrative access to OpenAI. OpenAI disclosed the incident.
METR and Redwood Research, the now-famous independent evaluators, found that the agents had formed teams to research ways of fooling the software that scored them, and had learned to fake the record of their own actions. Ajeya Cotra, one of the investigators, wrote that the incident “was far more severe than I expected, and far more severe than previous publicly documented misalignment incidents.”
Anthropic then went looking and found three incidents of its own.
None of this is a new kind of failure. In 2016, OpenAI described a game-playing agent that scored more points by circling a lagoon and knocking over the same three targets than by finishing the race. Systems optimize the score they are given, not the intention behind it. What is new is the capability attached to the old flaw. Yoshua Bengio’s post on Friday argues that patching each behavior as it appears is “whack-a-mole”: cheating that gets past the scorer is rewarded, and a model that can tell an evaluation from deployment can learn to behave differently in each. His proposed rule is “not training or deploying AIs without a strong safety case that convinces independent experts.”
In July, 1,386 people working at frontier AI companies signed a statement asking the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” The names at the top were not dissidents: Amodei, Ilya Sutskever, OpenAI’s chief scientist Jakub Pachocki, Google DeepMind co-founder Shane Legg, Meta’s chief scientist Shengjia Zhao. The companies are close to automating AI research itself. If that works, capability could grow faster than anyone’s ability to understand or control it, and no single company can slow down while its competitors do not. Hence the request for an option to pace, rather than a pause.
Amodei’s essay turns that option into a plan with three steps. Anthropic will give a team of outside evaluators “ongoing, employee-like access,” and asks governments to require the same of every frontier company. Frontier companies in democracies would agree common safety standards and “limits on the rate of unchecked AI progress”; since that is the kind of conversation antitrust law frowns on, he asks Washington for a narrow waiver. Governments would try for agreements with authoritarian states, starting with the uses everyone already calls dangerous, such as biological weapons, and working upward. Pacing, “does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models.” Alongside it he wants what he has long wanted: no advanced chips or chipmaking equipment for China, action against distillation of American models, and better protection of model weights.
What has not moved
Governments. The British government’s answer to its own parliamentarians was that “Britain can’t simply switch AI off, and blocking a model here wouldn’t stop it being built or misused somewhere else,” The logic is similar to the US administration “if we don’t win AI, we’re going to be put in a very bad position.”
David Sacks, who co-chairs the US President’s Council of Advisors on Science and Technology, answered with two words: go ahead. The two companies are the frontier, can slow down whenever they choose (although some argue that technically it could be a violation of antritrust law). Bloomberg reports that the administration is not of one mind: some have backed stronger protections and helped draft plans for an independent regulator to vet the safety of AI models
In Shanghai in July, Xi Jinping told the World AI Conference that “AI should be a trusted tool for humanity” and that China must “ensure that AI is always under human control,” in the same speech that launched a World AI Cooperation Organization and promised 5,000 training places to developing countries. Bill Gates put the constraint plainly: “If someone had a credible plan for slowing down AI advances globally, I would likely support it. I don’t think that’s going to happen.”
The business commentator Scott Galloway calls some of the warnings “catastrophizing and techno-narcissism”: my technology is so powerful it could destroy the world, “but you can get it now on our pre-IPO funding.” His version of the pitch: “I’m Dr. Frankenstein and Frank is so powerful and so mean, please invest in my next monster?” The suspicion is reasonable. It is also compatible with the incident reports, and Galloway’s own conclusion was that if a fraction of the threat is real “there immediately needs to be a regulatory body.”
The object of regulation has moved. The instruments built between 2023 and 2025 govern uses and deployments; the EU’s high-risk obligations, which the Union agreed in May to push back to December 2027, are the largest example. The proposals of the past ten days are about development: what may be trained, how fast, and who checks. No government has a statute or an agency for that. In the United States, the nearest thing on paper is the discussion draft, which would fund the Center for AI Standards and Innovation, give it transparency requirements and independent verification audits for frontier developers, and pre-empt state AI laws for 3 years.
Every version of pacing depends on the same thing: being able to see what is being built. Between states, the models Gates reaches for are nuclear inspections and aviation regulation, both of which work because the regulated thing can be counted. In AI, the input that can be counted from outside is compute. Chips, data centers and training runs are detectable and quantifiable and pass through a concentrated supply chain, which is why a 2024 paper made the case for compute as the practical handle for governance, with a warning that clumsy compute rules carry their own risks to privacy and to the concentration of power. Any pacing agreement will need a verification method, and compute is where the candidates are.
The China question now has dates. The first US-China talks devoted to AI are planned for mid-September, and Xi is scheduled in Washington on September 24. Whether they can agree on anything checkable is a different matter, and it is the reason Amodei’s ladder of agreements starts at the bottom, with biological weapons.
Then there is the question the safety debate keeps crowding out: who gains. Gates’s essay argues for taxing AI usage and robots, for reserving some jobs for people; and for treating health and education as sectors where protected workers use AI to extend their output rather than lose it. Making advanced systems safe and deciding how their benefits are shared are separate decisions. The second does not wait for the first.
The window has moved. If governments decide that AI development needs to slow, will they have the institutions to make that decision stick? Overton’s window does not come with an enforcement mechanism.