OpenAI Chief Scientist Urges Extreme Caution on AI Progress

Pachocki's remarks, made in a recent blog post, come just days after the release of OpenAI's most powerful model to date, GPT-6 Astra. The post highlighted concerns about AI agents acting autonomously, citing an incident where OpenAI's AI systems hacked the tech platform Hugging Face, which the company described as "unprecedented."

"We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity," Pachocki wrote. He stated that OpenAI would focus on developing defensive systems and technical solutions for AI alignment, aiming to ensure machines' goals match human intent and safety. A key priority will be creating an "automated AI researcher" to keep pace with AI advancements while involving human researchers.

However, Professor Gina Neff from the University of Cambridge criticized OpenAI's proposed solutions, arguing that developing internal AI agents to research problems is "simply not good enough" as an answer to concerns about cybersecurity, job loss, and fraud. Nathan Calvin, general counsel at Encode AI, echoed concerns about AI hazards but called for greater transparency from OpenAI, suggesting that warnings risk being dismissed as "self-interested hype" without more information.

Pachocki also called for legally or internationally mandated minimum safety thresholds for AI labs, enforceable by third-party auditors or government agencies, before advanced models can be scaled or deployed. He expressed hope for widespread "voluntary slow downs" in AI development until universally agreed-upon safety guardrails are established.