Key facts
- AI models can degrade in performance over time, a phenomenon called model drift or AI aging.
- A study found measurable temporal degradation in 91% of tested model-and-dataset pairings.
- In one financial services case, AI agent containment dropped by approximately eight points over weeks.
- The root cause of performance decline was the AI model itself, not external factors like prompts or guardrails.
- Fannie Mae and Freddie Mac have issued guidance on monitoring AI for degradation.
- The EU AI Act is also moving towards requiring ongoing monitoring of AI systems.
Conversational AI agents deployed in financial services can experience a decline in performance over time, a phenomenon known as model drift or AI aging, even when no external changes are made to their programming or data inputs. This degradation can go unnoticed for weeks or months, impacting customer experience and operational metrics.
A peer-reviewed study in Nature’s Scientific Reports found measurable temporal degradation in 91% of 128 model-and-dataset pairings across various sectors, including finance. In one specific instance within a heavily regulated financial services environment, an AI voice and chat agent designed to answer borrower questions saw its containment rate—the percentage of issues resolved without human intervention—drop by approximately eight percentage points over an unobserved period. The industry baseline for containment is around 41%, with financial services typically leading at about 52%.
Investigations into the performance drop revealed that the issue stemmed from the underlying large language model itself, which had quietly become less accurate over time while maintaining fluent and confident-sounding outputs. This behavioral drift is distinct from data drift and is difficult to detect through standard weekly reviews, which are lagging indicators. The company's response involved implementing near real-time monitoring, adding human review of AI-handled conversations, and establishing a dedicated in-house system to watch for early signs of decline. Key quality metrics were shifted to a daily cadence.
Longer-term strategies include treating model selection as a recurring quarterly decision and benchmarking alternatives. The author highlights an organizational gap, noting that many companies lack dedicated roles for monitoring AI systems for drift, despite 99% planning to deploy autonomous AI agents, only 11% have actually done so. Regulatory bodies like Fannie Mae and Freddie Mac have begun issuing guidance on AI monitoring, and the EU AI Act is also moving in this direction. The author advises product leaders to prioritize building leading indicators for AI performance and assigning clear ownership for monitoring, rather than focusing solely on new features.