The Shadow of Self-Improving AI: A Warning from Within

The recent resignation of Jacob Coxon, a 27-year-old researcher at Anthropic, has shed light on the disturbing trend within the artificial intelligence industry. Coxon’s public post exposed the internal debate among engineers and researchers about the catastrophic risks associated with superintelligent AI. The issue is no longer just about hypothetical dangers but an unsettling reality: influential players are racing towards self-improving intelligence without sufficient safeguards.

Coxon’s departure has sparked a wave of resignations, with two more researchers leaving Anthropic and Google AI safety teams joining METR, a nonprofit evaluating catastrophic AI risk. Their concerns echo Coxon’s: that the choices determining humanity’s fate are being made in private Slack channels by engineers working on “the MacBooks of some engineers living in San Francisco.” This is not just alarmist speculation but a warning from within the industry itself.

The internal debate within AI labs has been ongoing, but rarely spoken out loud. Coxon’s candid comments highlight the discrepancy between what senior people believe privately and what they say publicly. At OpenAI, where Coxon worked on GPT-4o, the civilizational stakes have not sunk in. In contrast, at Anthropic, they have, but the company believes it must get there first before a rival acts responsibly.

Evan Hubinger’s concurrence with Coxon’s concerns is telling. As an alignment stress testing lead at Anthropic, he agrees that AI could kill all humans and puts his odds above 10 percent within the next decade. His candid admission underscores the gravity of the situation: even those closest to the issue recognize its severity.

High-profile figures like Dario Amodei, Demis Hassabis, Sam Altman, and Elon Musk are backing plans to slow down model capabilities, although these efforts feel more like Band-Aid solutions than a genuine effort to address root issues. Their intentions may be well-meaning, but they do little to alter the trajectory of AI development.

Anthropic’s CEO Amodei has proposed a three-step plan: embedded third-party evaluators with employee-like access, coordination among labs in democracies, and eventually an agreement with China. However, this plan does little to address the lack of transparency and regulation within the industry.

The timing of Coxon’s resignation is striking, given Anthropic’s plans for listing next month and selling investors on responsible development as a central part of its pitch. The $100 billion fundraising effort at a $2 trillion valuation raises questions about the motivations behind this push towards self-improving AI.

Decisions like these remain on the laptops of engineers in private companies due to the absence of federal regulations in the US and a lack of coordination among lawmakers. Until there’s a fundamental shift in how AI development is approached – one that prioritizes transparency, regulation, and genuinely collaborative efforts – the risks associated with superintelligent AI will continue to loom large.

As researchers like Coxon speak out, it’s time for policymakers to take notice and act. The warnings have started collecting evidence; now it’s up to governments to build a mechanism for slowing development if models begin improving themselves. Anything less would be a dereliction of duty in the face of an unfolding catastrophe.