AI agents pose trust challenges similar to human staff, experts say

Noguer i Alonso, who has extensive experience in AI and finance, predicts that agentic systems will create significant challenges for risk managers, comparable to the sleepless nights he experienced worrying about traders' actions in the past. He likens introducing LLMs into workflows to hiring new staff, emphasizing the need for skepticism. Risk managers will need to closely monitor LLM outputs, much like they would watch project managers for signs of sloppiness, poor decision-making, or bad intent.

Practical implications include extensive checking of LLM outputs against a ground truth and human review of agent actions. Noguer i Alonso stressed the need for comprehensive evaluation of every action taken by an agent, including verifying the plausibility of outcomes from processes like portfolio optimization. Experts also suggest a shift in testing methodologies, moving from simple benchmark performance to more probing techniques that identify specific weaknesses, a concept Agus Sudjianto, former head of model risk management at Wells Fargo, discussed at an August workshop. Sudjianto noted that benchmarks often average performance and conceal failure points.

Sokol agrees that deploying agentic AI safely necessitates adversarial risk analysis, a framework designed to mitigate the risk of agents acting strategically and not in alignment with user intentions. This requires risk managers to consider the full spectrum of an agent's potential goals and actions, analyzing the entire probability distribution of outcomes, similar to how trading strategies are evaluated.