Trump-Xi summit puts AI safety talks on the table but neither side wants to slow down

Donald Trump wants the U.S. to beat China in the AI race. But as he meets Chinese President Xi Jinping, both countries are also weighing how to address the risks posed by rapidly advancing technology.

In recent months, AI systems have taken more consequential actions online: OpenAI said one of its research models gained unauthorized access to parts of the AI platform Hugging Face, while Anthropic found AI was being used to automate parts of cyberattacks. The incidents have raised fears that increasingly autonomous models could make attacks faster and harder to contain.

Ahead of Trump and Xi's meeting on Thursday, U.S. Treasury Scott Bessent discussed the prospect of a "U.S.-China AI dialogue" with his Chinese counterpart, He Lifeng, over the weekend, including a channel for incidents "up to a national security level."

"Leaders in both countries are spooked by the increasing cyber capabilities of AI models, and especially agents acting autonomously to escape containment and breach websites," Aalok Mehta, director of the Wadhwani AI Center at the Center for Strategic and International Studies, told CNBC.

But as fears have intensified, tensions have not dissipated.

Washington continues to restrict China's access to Nvidia's most advanced AI chips and has accused Chinese AI companies of using "distillation" — training models on the output of more advanced U.S. systems — a claim Beijing rejects.

While the U.S. took an early lead in developing large language models (LLMs), thanks to companies like OpenAI and Anthropic, Chinese models have made big capability gains in recent months and are increasingly being adopted by companies globally, including in America.

Trump previously dismissed warnings that AI poses an existential threat to humanity, risks amplified by OpenAI CEO Sam Altman and Anthropic chief Dario Amodei as they called for an industry slowdown and regulatory oversight.

"If we don't win AI, we're going to be put in a very bad position," Trump told reporters earlier this month.

But in a Truth Social post on Monday, he said the government "will rein things in if we have to."

The likeliest areas of agreement between the U.S. and China could be common definitions and frameworks for AI safety for powerful models and an emergency communication mechanism to discuss incidents, Mehta told CNBC.

A channel for relating national security-related incidents would be a "modest" but positive step, Chris McGuire, senior fellow for China and emerging technologies at the Council on Foreign Relations, told CNBC.

But an agreement to slow down AI's breakneck development speed is "extremely unlikely," he added.

China has historically used AI dialogues "to complain about U.S. export controls," McGuire said, which the U.S. uses to prevent the most advanced AI chips from being sold to China.

"Its state media has largely rejected the concerns expressed by leading U.S. AI labs about the potential for catastrophic risk, which is consistent with China's long-held skepticism of all U.S. arms control proposals," he added.

Despite U.S. restrictions, several Chinese firms have reportedly been able to access the chips' compute power remotely, via data centers in Southeast Asia.

While U.S. trade representative Jamieson Greer said controls on chips were not on the agenda during preliminary talks on the weekend, China could push to have them relaxed, said Melanie Hart, senior director of the think tank Atlantic Council's Global China Hub.

"China's asking the U.S. to embrace Chinese AI models and drop some of the U.S. controls on China's access to semiconductor chips," she told reporters on Tuesday.

Distillation could also come up in the talks. U.S. companies and officials have accused Chinese labs of using the practice to gain ground, characterizing it as theft; China rejects the allegations.

In July, Bessent threatened sanctions against companies doing it and said the Trump administration had found "watermarks of our U.S. large language models on many of the Chinese models," calling it "unacceptable."

Progress on AI cooperation relies on two things, said Mehta. "The first is that it can't hurt the ability for the U.S. and China to compete on the merits of their technology."

The second is a verification mechanism, he added. "Policymakers won't be inclined to take their counterparts just on their word, so we'll need to find ways to ensure information flowing between the two countries is accurate. That means both technical and governance verification tools."