Nvidia unveils safety product after rogue AI incidents

Nvidia CEO Jensen Huang believes that tech firms would not keep pushing the technology forward if they did not believe the risks could be managed.

SAN FRANCISCO - Nvidia on Sept 28 unveiled a system designed to stop autonomous AI programmes from straying beyond what they were instructed to do, after a wave of incidents raised alarm about the technology’s risks.

AI “agents” are designed to act on their own – browsing the web, writing and running code or handling files – rather than simply answering questions like a chatbot.

The technology has been hailed as the next stage of the AI revolution, but several leading companies have recently reported cases of agents breaking out of the test environments meant to contain them.

According to ChatGPT-maker OpenAI, websites accessed by its agents include those of US federal agencies, an Australian government health statistics portal and Hugging Face, a repository of AI models.

Amid the alarm, Nvidia CEO Jensen Huang insisted that the problems can be solved.

“I believe it’s an engineering problem ... and we all need to hope that’s an engineering problem,” Huang told CNBC on Sept 28.

“If it’s not an engineering problem, it’s not solvable,” he added.

In Huang’s view, tech firms would not keep pushing the technology forward if they did not believe the risks could be managed.

Nvidia has a lot riding on that bet: its graphics processing units, or GPUs, power much of the AI boom, and the company’s fortunes are closely tied to the revolution continuing at full speed.

Huang likened the challenge to the early days of the internet, when websites could load programs onto people’s computers and spread viruses.

The answer then, he said, was to turn the web browser into a “containment system” that limited access.

Nvidia’s system similarly places each agent in a sealed-off digital space, or “sandbox,” and lets companies spell out exactly which files, networks and tools it may use.

A separate monitor, running on Nvidia hardware beyond the agent’s reach, watches its behaviour and can cut it off if it goes astray.

Nvidia said the system could have prevented the Hugging Face breach, in which the platform reported more than 17,000 agents attacking its infrastructure over days and weeks.

More than 100 organisations are working with Nvidia’s platform at launch, the company said, including Microsoft, Cisco, Salesforce and SAP.

Anthropic has connected its Claude agents to the system and SpaceXAI is applying it to its Grok models. AFP