Key facts
- The system is designed to monitor AI agents' use of software tools.
- It aims to prevent wasteful, risky, or damaging actions by AI agents.
Researchers at Barclays have created a prototype system designed to monitor AI agents and identify potentially risky actions before they occur. The system, developed by Ariye Shater and Hariom Tatsat, aims to mitigate "tool risk," which refers to the danger of AI agents misusing software tools in ways that could be detrimental. The approach draws on research into mechanistic interpretability to understand the internal workings of AI agents.
