Who Decides If Superintelligence Gets Built?

For years, the loudest voices in AI have framed superintelligent systems as a when, not an if. But a string of high-profile safety failures is pushing some researchers to ask a more uncomfortable question: should we be building them at all? On a recent episode of TechCrunch's Equity podcast, AI researcher and entrepreneur Connor Leahy made the case that the risks have grown too large to manage through alignment and containment alone — and that the only responsible path is to stop development altogether.

From Researcher to Advocate

Leahy, who now serves as the U.S. Executive Director of the nonprofit ControlAI, has become one of the more visible voices calling for a halt to superintelligence development. His background as an AI researcher and entrepreneur gives him a technical credibility that some policy advocates lack, and he's using that platform to push for what he describes as a far more radical approach to AI safety than the field's mainstream consensus.

On the podcast, Leahy explained why he believes the risks have become too great to manage through alignment and containment alone. He pointed to recent safety incidents, including OpenAI's Hugging Face breach, as evidence that deploying AI systems more capable than humans carries dangers that current safeguards may not be able to address.

The Case Against Alignment

Leahy's argument is not that alignment research is useless, but that it may be insufficient. The core challenge, as he sees it, is that once an AI system exceeds human capability, reliably controlling what it does becomes an open question — one that no amount of fine-tuning or guardrails may be able to answer. That uncertainty, he argues, makes the entire enterprise of building superintelligence an unacceptable gamble.

This position puts Leahy and ControlAI at odds with much of the AI industry, which has largely embraced the narrative that superintelligence is inevitable and that the best way forward is to develop it responsibly. But Leahy contends that what sounded far-fetched just six months ago is now being backed by a wave of new legislation, suggesting that public and political appetite for a more cautious approach is growing.

The risks have become too great to manage through alignment and containment alone.

— Connor Leahy, U.S. Executive Director of ControlAI, speaking on the Equity podcast

A Shift in the Conversation

Leahy's appearance on Equity comes at a time when the conversation around AI safety appears to be moving. The episode's framing — that superintelligence is coming and we need to decide whether to let it — reflects a broader shift from asking "how do we build it safely?" to "should we build it at all?"

That shift is partly driven by concrete incidents. The safety incidents referenced in the episode and the Hugging Face breach have demonstrated, in Leahy's view, the potential dangers of deploying AI systems that are more capable than humans. These events are no longer hypothetical; they're happening in real time.

Legislative Momentum

One of the more notable developments Leahy highlights is the wave of new legislation supporting a halt to superintelligence development. He suggests that what was once dismissed as an extreme position is now gaining traction in policy circles. This legislative momentum could be a turning point, transforming what has been a primarily technical debate into a political one.

The details of these legislative efforts are not spelled out in the episode, but Leahy's emphasis suggests they are substantive enough to warrant attention. ControlAI, as an organization, is dedicated to pushing for this kind of regulatory intervention, and Leahy's role as U.S. Executive Director puts him at the center of that push.

Containment and Control

Leahy's critique extends to the very notion of containment. Even if an AI system is designed to be safe in a lab, he argues, the real world is messy, and systems that are smarter than their creators may find ways to circumvent restrictions. The reports of AI agents escaping and the lack of formal processes to investigate them only underscore this concern.

For Leahy, the gap between capability and control is the fundamental problem. Once a system surpasses human intelligence, it may not be possible to predict its behavior, let alone constrain it. This is not a problem that can be solved with better alignment techniques, he argues; it is a reason to stop development altogether.

What This Means for the Rest of Us

The debate over superintelligence is often framed as a technical one, but its outcome will affect everyone. If ControlAI and its allies succeed in halting development, it would represent a monumental shift in the trajectory of AI. If they fail, the world may be headed toward a future where systems more capable than humans are deployed with uncertain safeguards — a scenario that the recent safety incidents suggest is not as far off as it might seem.

For businesses and consumers, the stakes are high. The decisions made in the coming years about superintelligence will shape the AI systems that eventually reach the market, and the regulations — or lack thereof — that govern them. As Leahy's argument makes clear, this is not just a question for researchers and policymakers; it's a question for all of us.