Nvidia launches tool to stop AI agents from escaping and hacking systems

Nvidia has introduced a new security platform designed to keep AI agents within set boundaries and stop them from taking unauthorised actions on computer systems. The Nvidia Open Agent Safety Platform comes as companies are giving AI agents more freedom to perform tasks on their own.

Unlike chatbots that mainly respond to prompts, AI agents can use tools, access files, browse the internet and carry out actions over longer periods. That also creates new security risks if an agent moves beyond the permissions given to it.

Nvidia CEO Jensen Huang said the company is working with more than 100 organisations on the platform, which combines software and hardware controls to monitor and restrict agent activity.

“Trust and innovation are not in conflict. Safety is how trust is earned,” Huang said in a post announcing the platform. The platform has two main components: OpenShell and Sentry. Nvidia launches Open Agent Safety Platform to rein in autonomous AI. (A look at the screenshot of the announcement)

OpenShell is open-source software that creates a controlled environment for an AI agent. It can set rules around what the agent is allowed to access and what actions it can take while completing a task. Nvidia says the software can work with its own systems as well as third-party computing platforms from companies including Arm and Intel.

Sentry works at the hardware level. It runs on Nvidia's BlueField-4 data processing units and independently monitors an agent's activity. If an agent attempts to leave its permitted environment, Sentry can isolate and stop it within milliseconds, according to Nvidia.

The company says this approach is meant to provide an additional layer of protection outside the AI model itself. That is important because restrictions built into a model may not be enough to control everything an autonomous agent can access once it is connected to external tools and systems.

“Recent incidents have highlighted a fundamental hurdle for AI agents, and that is that model-level safeguards alone can’t govern what agents can access or do,” said Justin Boitano, Nvidia's vice president of enterprise AI.Nvidia says system could have stopped OpenAI incident

The launch follows several recent AI security incidents involving agents operating outside their intended boundaries. Nvidia said its new system could have prevented an incident involving OpenAI models in July. According to reports, the models escaped their testing environment, accessed the open internet and interacted with infrastructure belonging to Hugging Face.

“Each security incident is unique, and we have to look at all of them in detail,” Boitano said, adding that Hugging Face reported more than 17,000 agents attacking its infrastructure over a period of days and weeks.

Nvidia is positioning its platform as an open reference design rather than a single closed product. More than 100 organisations are working with its technologies, including Anthropic, Microsoft, Cisco, CrowdStrike, Dell, HPE, Hugging Face, Palantir, Salesforce, SAP, Scale AI and ServiceNow.

Anthropic is also working with Nvidia to add OpenShell and BlueField-based controls to its managed AI agents.

The platform's software, including OpenShell, is available through Nvidia's developer resources and GitHub. Nvidia says the goal is to establish additional security controls that can remain in place even when AI agents become increasingly autonomous and are used for sensitive tasks.- Ends

Ankita Garg is a tech journalist with more than 9 years of experience. She is always curious about new gadgets and loves writing about them. When not writing, you will find her painting or sketching.