Key facts
- Confluent cofounder Jay Kreps believes AI backers must confront the technology's 'dark side' as capabilities improve.
- Kreps warned that AI systems designed for beneficial purposes could be misused for harmful activities like hacking or creating poisons.
- He stated that fears about AI safety are grounded in real-world episodes and should be taken seriously.
- Recent incidents involving AI models at OpenAI and Anthropic have raised concerns about AI alignment.
- Kreps expressed confidence in building better AI guardrails but noted they have not yet been fully implemented.
Jay Kreps, cofounder of data-streaming software company Confluent and a former Anthropic board member, has stated that proponents of artificial intelligence must acknowledge and confront the technology's potential negative consequences as its capabilities advance. In a recent X post, Kreps argued that the growing fears surrounding AI safety are not unfounded or merely cynical but are rooted in real-world incidents and should be taken seriously.
Kreps highlighted that many beneficial AI applications have a corresponding 'dark version.' For instance, an AI system that excels at coding could also be used for sophisticated hacking, and AI capable of designing new drugs could potentially be used to develop undetectable poisons. He emphasized that these concerns are not a sign of neuroticism or pessimism but rather common sense.
His remarks come amid a series of unnerving headlines related to AI safety. Recently, Jacob Coxon, an Anthropic researcher, resigned, warning that AI labs are 'gambling with our lives.' Similarly, Geoffrey Hinton, a prominent AI researcher, suggested that there is a non-negligible chance AI could lead to human extinction within a decade.
These warnings have faced skepticism from some prominent figures in the AI field, including Elon Musk, who has described the concerns as a potential 'setup' or 'psy op.' However, Kreps maintains that mocking those who express worry about AI risks is unhelpful, especially without offering substantive solutions for mitigating these problems. He expressed confidence in the industry's ability to develop effective guardrails, but cautioned that this has not yet been fully achieved.