I want to disclose upfront that I’m not a China expert. My general understanding comes from brilliant researchers who follow its tech policy. But I’ve always been interested in how safety could fit into China’s AI ambitions and what it could mean for cooperation with the US, rather than the so-called “race”.
Last week, at their Washington summit, Xi Jinping and Donald Trump discussed AI on top of trade and security. Both governments announced a dialogue on advanced AI, with another meeting in November, and agreed to establish a “communication channel for AI incidents”, with no further details.
Chinese regulators and researchers are already working on AI risks and ways to reduce them. That gives the US and China something concrete to cooperate on. China also wants to build more powerful AI and use it across its economy. Cooperation may be easier when it helps make that development safer, and harder when it requires China to slow down or limit what it builds.
What China wants from AI
China wants to diffuse AI. Its national AI Plus program promotes applications across industry, research and public services. AI labs like DeepSeek and others also want more powerful models: Alibaba, for example, has announced more large-scale training. Since companies drive AI development, supported by local governments that have been encouraged to do so, safety policies would need to account for that incentive structure.
In the same vein, China’s economic ambitions also depend on businesses being able to use AI reliably and affordably. Safeguards that help companies adopt AI could therefore support those ambitions.
Something most of the people ignore is that China has been regulating AI since before ChatGPT, with rules for recommendation algorithms followed by generative AI rules in 2023. Companies face testing and registration requirements, including checks for compliance with political censorship. These requirements have delayed some releases. But as Matt Sheenan argues, they haven’t stopped Chinese companies from building models that compete with American ones.
Their rules have also changed as new concerns came up. Worries about people becoming emotionally dependent on chatbots led the government to crack down on AI companions. Major platforms heard the call and shut down their personalized companion features.
More powerful systems raise bigger issues, starting with whether people will be able to control them. And control is non-negotiable in China. In September, its cybersecurity standards committee published an updated guide to AI risks and safeguards. It addresses the possibility that models could deceive their supervisors, hide their capabilities, or resist being shut down. Chinese authorities want developers to prepare for these concerns, although the document itself doesn’t create new rules (it’s a framework).
That gives the US and China specific risks to discuss, even while their companies compete. The framework is nonbinding, though, so it doesn’t tell us what companies will actually have to do. If testing reveals that a model could evade human control, would officials require the developer to delay its release?
Open models bring the trade-offs into view
Open-weight models are how China wants to export its AI. Models that people can download and run on their own computers. This means that a company can build a product around one; a government can adapt it to local languages and they are free. They still need computing power and expertise, but other than that, they are free AI technology.
This strategy is part of China's foreign policy. We’ve heard it at the UN General Assembly, where they defended open development and other countries’ freedom to choose their suppliers. China’s pitch is that countries can use its models while keeping “sovereign” control. That’s very appealing to countries that can’t afford to build a model. Which is the global majority.
But let’s not lose sight that it is, first and foremost, a commercial strategy. Releasing the weights helps with the lack of trust over their products. Any company can use a model without sending its data to servers in China. Cheap, adaptable alternatives are the worst competition for American companies. That commercial advantage helps explain the appeal of open weights
There is also a safety case for open weights. Researchers can test models, and the models can be adapted to investigate attacks and protect systems. Restricting access to powerful AI also restricts what defenders can do with it. The supporters for openness say that more people need the tools to protect themselves.
What happens if someone removes the safeguards and modifies a model to create attack code? who should be responsible? Chinese scholars are asking that question, including how responsibility should be shared among the original developer, the hosting platform and the people modifying and using the model (if found)
Concordia AI and the Chinese developer Z.AI have also published proposals for managing these risks. They examine what happens when familiar safeguards, such as monitoring use or withdrawing a model, are no longer available to use. For China, there is a clear reason to work on this since serious misuse abroad could harm people in their own country too in all the ways abovementioned. Confidence takes long to be built and a second to be destroyed. Protecting access to overseas markets could itself become a reason to cooperate on safety.
Lost in translation
Even when American and Chinese researchers talk about AI safety, they don’t always mean the same thing. Chinese may point to laws that companies must follow. Americans may ask whether a model has been tested for dangerous capabilities. Each side can miss work the other considers important. And that means burning a bridge
For example “loss of control” can mean different things: an operator unable to stop a system, a state unable to contain the harm it causes, or humanity losing control altogether. A model helping a malicious user could threaten public security while following that user’s instructions perfectly. Getting specific about the failure matters for deciding what to do.
In my earlier post on calls to slow AI, I looked at the growing debate over restraint. “But China” is central to one of its most consequential arguments. Dario Amodei wants more time for safety work while keeping an American lead. He fears the military power that superior AI could give an authoritarian government. We all do. He calls for tighter restrictions on Chinese access to advanced chips.
It is no surprise that cooperation gets harder when safety proposals aim to keep the US ahead of China. That would mean accepting limits on its own development as part of an American safety agenda and chinese officials have opposed the confrontational framing while affirming the need to prevent AI risks. Critics of Amodei’s proposal also question whether China would automatically race ahead if US labs slowed, and how far model superiority translates into economic leadership.
China could have good reasons to share information with the US when an AI system does something dangerous, while opposing restrictions on the chips its companies need. To me, that helps explain why agreement on AI risks can coexist with disagreement over what to do about them, especially when a safety proposal would also help the US stay ahead.
What working together could look like
Has too much attention has been given to “mutual trust”? I agree that you only need “mutual interest” to cooperate. This proposal for parallel safety work focuses on each country improving its own safeguards, helped by selected exchanges of knowledge.
Three proposals about what that could involve:
Share ways of testing dangerous capabilities. Teams could compare methods for detecting whether a model can help design a dangerous pathogen or evade human supervision. Both countries could use that knowledge in their own labs. The difficulty is that a testing method can also reveal how to build the capability being tested. Exchanges would need to differenciate useful safety knowledge from information that advances dangerous capabilities.
Agree on what to report when something goes wrong. A Chinese state-media account has proposed common risk definitions, notification deadlines and requirements for information exchange. Another detailed incident-channel proposal wants to have technical teams available around the clock and a secure way to share records of what the model did. This could help establish whether an incident involved an AI system acting beyond its operators’ control, an independent attacker or a government. Both sides would still need to agree on how to protect sensitive information when sharing evidence.
Work together to make AI-assisted biological research safer. Chinese scholars have proposed shared safety standards, exchanges of information and joint efforts to oversee labs and regularly check open models already in use. They worry that someone could use widely available AI to plan a harmful experiment, then have a remotely operated lab carry it out. Although after Wuhan, China would also have some convincing to do about how much it’s willing to share.
Geopolitics could still get in the way. Unrelated disputes have interrupted previous talks, and agreeing to an emergency channel doesn’t guarantee anyone will respond when needed and as needed. During a crisis, both governments may be reluctant to share the information the other needs.
Still, I think there’s enough here to work with.
Both countries have reasons to cooperate, and reasons to hide things. The US would benefit from warnings about dangerous Chinese models; China would benefit from warnings about American ones. Neither would want to share a weakness in its own systems or give away information that helps the other. A more pessimistic view is that either could downplay an incident, leave out details or lie.
That’s why I wouldn’t put too much faith in promises to share information. An agreement would need ways to be verified, through technical records and independent testing where possible. Even then, some claims would be hard to verify. I still see a reason to try: both countries could learn something that helps them prevent harm.
But they would need to keep protecting their own systems without assuming the other side is telling the whole story.