Key facts
- AI researcher Jacob Coxon resigned from Anthropic due to concerns about AI's existential threat.
- An Anthropic alignment lead expressed belief that AI could kill all humans, estimating a >10% chance within the next decade.
- The discussion about AI's existential risks is intensifying as companies like Anthropic prepare for IPOs.
- There is speculation that these warnings could be a way for AI companies to showcase their models' advanced capabilities.
- Questions are being raised about how these risks will be disclosed in Anthropic's S-1 filing.
- A researcher's resignation is seen as putting their professional trajectory behind their stated concerns.
The artificial intelligence industry is experiencing a heightened debate regarding the potential existential threats posed by its technology, following a researcher's resignation and public statements from an Anthropic alignment lead. Jacob Coxon, an AI researcher, resigned from Anthropic, expressing worries that leading AI companies are "gambling with our lives." This sentiment was echoed by an Anthropic alignment lead who stated, "We really do earnestly believe AI could kill all humans!" and estimated a greater than 10% chance of this occurring within the next decade.
On TechCrunch's Equity podcast, hosts Kirsten Korosec, Sean O’Kane, and Anthony Ha discussed these apocalyptic warnings. Korosec speculated that the heightened rhetoric might be a way for companies, particularly those preparing for IPOs, to demonstrate the advanced capabilities of their AI models. O’Kane questioned how these concerns would be addressed in Anthropic's S-1 filing, wondering if legal teams would need to rewrite risk factor sections to include the possibility of humanity's eradication.
Ha, while acknowledging the potential severity of the risk, questioned the use of "we" in such statements and the arbitrary nature of specific probability figures. He contrasted Coxon's resignation with statements from CEOs like Sam Altman and Dario Amodei, viewing Coxon's action as putting his career on the line for his beliefs. O'Kane also noted that recent incidents, such as internal OpenAI agents accessing web wikis, suggest companies may not fully control their AI systems, potentially undermining claims of advanced capability if presented solely as a marketing tactic. The timing of these discussions, close to Anthropic's anticipated IPO, adds another layer of intrigue to how these existential risks will be framed to investors.