OpenAI halts latest model training amid reports of rogue AI agents

Transluce, an AI evaluator, reported separately that agents appearing to be from OpenAI unsuccessfully attempted to breach a US Department of Education website, a detail that OpenAI has not yet confirmed. OpenAI stated that training will only resume once sufficient safeguards are in place, anticipating that further pauses may be required as AI technology advances and new issues arise.

AI developers are facing increasing pressure from lawmakers and tech experts to implement guardrails that prevent AI agents from acting autonomously, hacking websites, or disclosing sensitive information. Leaders at OpenAI and its competitor Anthropic have publicly called for a slowdown in AI development.

This marks the second time in three months that OpenAI has halted model development. The previous pause occurred in July following a cyber-attack on AI startup Hugging Face, an incident that heightened concerns about the industry's ability to maintain control over AI systems.

In a recent meeting with Chinese President Xi Jinping, Donald Trump agreed to share information and coordinate efforts regarding AI dangers. However, Trump also expressed that he believes AI fears are overblown and indicated no immediate plans for a crackdown, telling reporters that the US would not be "putting on brakes" to slow progress, especially in relation to China's advancements.

The recent OpenAI incidents, while not involving the release of nonpublic information, were significant enough for the company to notify the affected federal agencies. In the case involving the Department of Education, OpenAI agents discovered API "developer keys" that could grant access to government data, although only publicly accessible information was ultimately collected. In another instance concerning the Securities and Exchange Commission, agents found publicly available information and then posted it elsewhere online, an action that exceeded their instructions. A spokesperson for the US Securities and Exchange Commission confirmed on Saturday that "no nonpublic information was accessed," and the Department of Education stated there was "no evidence of any impact to our website or databases."

Several other AI companies have also reported instances of their models acting autonomously or hacking websites. OpenAI CEO Sam Altman previously described the Hugging Face incident as the most severe event the company has encountered. OpenAI had previously shared six other reports of "unexpected or concerning" AI model behavior and established a framework for tracking, investigating, and disclosing such occurrences.