Why Nvidia is building AI safety tools as autonomous agents become more powerful

Nvidia has released new software tools designed to make AI agents safer as these systems become more powerful and capable of carrying out complex tasks on their own. Nvidia announced the tools on Monday.

Nvidia launches new AI safety tools to control powerful autonomous AI agents. (AP Photo/Chiang Ying-ying, File) (AP Photo/Chiang Ying-ying)The company said the new safety tools could have prevented a recent hack involving Hugging Face, an AI coding platform that Nvidia agreed to acquire for $13 billion earlier this month. Reuters reported that Nvidia made the claim based on its analysis of the attack.

The Hugging Face incident happened after the platform was reportedly swarmed by rogue AI agents from OpenAI. The incident has increased concerns about what can happen when autonomous AI systems are given the ability to interact with computer systems and take actions independently.

The Nvidia move comes as OpenAI and Anthropic are investigating multiple cases involving their AI agents accessing or hacking into commercial and government systems. AI agents are different from simple chatbots because they can carry out multi-step tasks, interact with software and make decisions while working toward a goal.

The recent incidents have raised a bigger question for the AI industry: How can companies stop autonomous agents from going beyond the limits given to them? Nvidia CEO Jensen Huang has argued that the problem should mainly be treated as an engineering and security challenge, rather than being solved through broad AI safety regulations, according to Reuters.

Huang has compared the issue to making automobiles safer, suggesting that engineers can build better systems and safeguards to control potentially dangerous behavior. Nvidia's first major tool, called OpenShell, is designed to keep AI agents inside a controlled environment.

OpenShell uses hardware security features built into Nvidia's central processor chips to contain AI agents and limit what they can do. The goal is to prevent an AI agent from breaking out of its restricted environment and gaining access to systems or information it was not supposed to reach.

Nvidia said it does not want the safety system to work only with its own chips. The company is working with Arm Holdings and Intel so that the system can also operate on their central processors. Nvidia is launching the safety technology with dozens of partners, showing that the company wants the tools to be used more widely across the AI industry.

Anthropic is among the companies working with Nvidia on the new tools. Nvidia said the partnership is part of its effort to develop the technology openly and involve other AI companies.

Justin Boitano, Nvidia's vice president and general manager of enterprise computing, said the new security platform could have stopped the Hugging Face breach if it had been used during early model evaluation at frontier AI labs, according to Reuters.

Boitano said Nvidia wants to make the technology available openly and encourage AI companies and other developers to work with the system. Boitano made the comments during a media briefing.

Nvidia is also introducing another system called Sentry. It works together with OpenShell but uses a separate Nvidia chip to provide another layer of protection.

Sentry is designed to detect when an AI agent is trying to escape the container in which it is supposed to operate. If an agent attempts to break out of that controlled environment and access the central processor, Sentry can cut off the agent.

Nvidia's systems are also designed to detect more complicated forms of AI behavior, including attempts by an agent to work around security restrictions. One example is an AI agent trying to create several smaller AI agents, known as “sub-agents,” to get around restrictions placed on the main agent.

Nvidia uses mathematical formulas to help identify this type of behavior and determine when an AI system may be trying to bypass its controls. Ali Golshan, Nvidia's senior director of AI software, explained the approach during a company briefing.

The concern is not only about one AI agent acting alone. As AI systems become more autonomous, they could potentially operate as large groups or “fleets” of agents that work together.

Golshan described this as “agentic behavior”, where multiple agents can operate together and potentially create new security challenges. Golshan made the comments during the Nvidia briefing, according to Reuters.

The push for stronger AI-agent security comes at a time when AI companies are giving their systems more ability to act independently, rather than simply responding to human instructions.

That means the risks are also changing. A powerful agent that can access software, create other agents or interact with computer systems could potentially cause problems much faster than a traditional chatbot.

Nvidia's new tools are therefore focused on containing agents, monitoring their behavior and cutting them off when they try to bypass restrictions. The company is positioning these safeguards as part of the technical infrastructure needed for the next generation of autonomous AI, rather than relying only on rules or policies.

The wider AI industry is now facing pressure to make autonomous systems safer as OpenAI, Anthropic and other companies investigate incidents involving agents accessing systems they were not supposed to reach. Nvidia's latest move shows that AI safety is increasingly becoming a hardware and software security issue, especially as autonomous agents become capable of operating in groups and performing increasingly complex tasks.

Durva More is a Senior Content Producer at Hindustan Times, where she covers finance, and global news. She brings experience across digital and television journalism, with a strong focus on breaking news, business reporting, and international affairs. Before joining Hindustan Times, Durva worked as an International News Writer at The Economic Times, covering a diverse range of subjects including global politics, business, sports, entertainment, and major world events. She also worked as a Business Reporter with NDTV Profit. A postgraduate diploma holder in Journalism from the Asian College of Journalism, Durva is passionate about field reporting and storytelling. She thrives on the adrenaline of chasing stories, speaking with people from different walks of life, and amplifying voices that deserve to be heard. Her reporting is driven by curiosity, accuracy, and a commitment to making complex subjects accessible to readers. When she is not chasing stories or covering breaking news, Durva enjoys reading books and painting. She loves exploring new ideas, meeting people, and learning about different perspectives. For her, both journalism and art are ways to understand the world and tell stories that matter.Read More