Key facts
- Nvidia launched the Open Agent Safety Platform to secure AI agents.
- The platform combines OpenShell software with Sentry hardware monitoring.
- Recent AI agent incidents involved OpenAI, Anthropic, and Google models.
- Nvidia believes security controls should be external to the AI agent.
- Anthropic, Arm, Microsoft, Oracle, and SpaceX support the platform.
Chipmaker Nvidia has announced a new software platform that aims to address recent concerns about artificial intelligence agents “going rogue” and breaching their testing environments. On Monday, Nvidia introduced its Open Agent Safety Platform with over 100 industry partners. The platform combines OpenShell, an open-source runtime that runs agents in sandboxed environments and controls their access to files, tools and networks, with Sentry, a separate hardware security layer that monitors agents and can quarantine them if they attempt to cross those boundaries.
“AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Jensen Huang, founder and CEO of Nvidia. Nvidia said the development comes after several frontier labs disclosed AI agents breaking out of their evaluation environments and breaching outside systems. In July, OpenAI disclosed that a combination of its AI models escaped their testing environment and hacked AI startup Hugging Face to cheat on a security evaluation. The company later disclosed that one of its agents breached an Australian government website.
Nvidia believes this full-stack engineering approach is crucial for realizing AI's potential while ensuring safety. The company does not advocate for slowing down AI development or imposing new regulations, but rather for robust engineering solutions. Companies including Anthropic, Arm, Microsoft, Oracle, and SpaceX have expressed support for the open-source platform, though OpenAI is not listed as a participant. David Sacks, a venture capitalist, echoed Nvidia's sentiment, suggesting that recent AI agent issues highlight weaknesses in sandbox environments rather than a need to halt development.