Key facts
- Anthropic proposed a coordinated pause in AI development if risks escalate.
- AI capable of self-improvement could lead to loss of human control.
- Over 80% of Anthropic's codebase was authored by Claude as of May.
- A meaningful pause requires agreement among multiple well-resourced labs.
- Anthropic's research arm will study and help build systems to support a slowdown.
AI startup Anthropic has proposed that frontier AI developers establish a coordinated and verifiable method to slow down or temporarily pause development if advanced systems begin improving themselves faster than society can manage the associated risks. The company highlighted that AI capable of building its own successors could significantly increase the risks of humans losing control over AI systems. As an example of AI's growing capabilities, Anthropic noted that as of May, over 80% of the code merged into its own codebase was authored by its AI model, Claude. Anthropic believes it would be beneficial for the world to have the option to pause frontier AI development to allow societal structures and alignment research to keep pace with technological advancements. However, the company cautioned that unilateral or poorly coordinated slowdowns could be counterproductive if less cautious actors continue to advance, potentially reducing overall safety. A meaningful pause, Anthropic stated, would necessitate agreement among multiple well-resourced labs operating at the technological frontier, along with clear rules on the conditions that would trigger or lift such a pause and who would oversee it. While a unilateral pause by a single company would be simpler to implement, it would have limited impact, primarily shifting leadership rather than fostering broader global deliberation. Anthropic's research arm, Anthropic Institute, intends to study and assist in building the necessary systems to support such a slowdown. In the upcoming months, Anthropic plans to convene discussions with policymakers, researchers, civil society groups, and other AI firms to examine critical questions, including how to manage AI-related risks like recursive self-improvement and how to enhance coordination mechanisms. The company recently concluded a fundraising round that valued it at $965 billion and confidentially filed for a U.S. initial public offering. Favaro and Clark argue that a slowdown would allow more time to deal with the technology's implications, noting that AI agents can already run code, delegate work, and may soon be capable of taking over development without human input.