AI leaders can't agree on how to slow down AI development

AI developer Jacob Coxon made headlines earlier this month when he said that his colleagues “earnestly believe it could kill us all by the end of the decade.” His warnings have ignited a firestorm. Some AI bosses have now echoed Coxon’s fears and called for an industry-wide slowdown while others have rejected such an approach. But while the debate over AI safety is more prominent than ever, a solution remains elusive as ever.  

Doom upon all the world

Coxon ignited a media firestorm when he went public with his fears regarding AI safety.

I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. 

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

He underscored his point by arguing that the executives and researchers say one thing in public while having major concerns about the trajectory of AI in private. Coxon offered the following explanation for Anthropic and OpenAI’s recklessness:

At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk. 

‍

The X thread announcing his departure has over 172 million views. 

Industry reaction

Industry reaction has been split. Anthropic’s Dario Asmodei called for a slowdown in the field. As he put it,

Like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.  

Referencing the Hugging Face incident, Asmodei said that, although no one was hurt and there was minimal economic damage, it should still serve as a wakeup call for the industry since an AI swarm with greater capabilities but a similar level of misalignment could have done far worse.

Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.

Amodei proposed a three-step plan for safe AI development by frontier companies:

1. Have embedded third-party evaluators who would be responsible for verifying adherence to safety practice, reporting incidents, and assessing the alignment of both completed models and those in development.
2. Companies in democratic countries would coordinate to establish common safety standards, though he noted that this could be “legally challenging” and so it would require a level of government support.
3. Democratic countries would attempt to coordinate AI development with their authoritarian counterparts. 

OpenAI’s Sam Altman endorsed Asmodei’s plan on X and committed to implementing independent evaluators at his company. 

But not everyone in the AI industry has been onboard with these calls for restraint. The Financial Times reports that employees at Anthropic and OpenAI have concerns about giving third-party evaluators access to their work since it could effectively put trade secrets at risk. As Gizmodo notes, it’s not surprising that this change should provoke pushback given how companies like Anthropic and OpenAI often have an internal culture that emphasizes the importance of secrecy.

There’s been higher-level pushback as well. Nvidia’s Jensen Huang rejected the idea of inter-company coordination on AI safety. Huang has publicly cast doubt on the idea that AI poses an existential risk to humanity and argued that there’s no need for further laws or regulations. He views open models as the best way to achieve safe AI development.

Meta’s Mark Zuckerberg endorsed Huang’s approach, arguing that “labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.” However, he did say that a “larger and more diverse ecosystem of evaluators” could be helpful, though as the Financial Times notes, he didn’t commit to giving them full access to Meta’s development pipeline.

Political reaction

American political reaction has been similarly polarized. Legislators across the aisle are calling for action. Democratic House Minority Leader Hakeem Jeffries called for “decisive congressional action” to slow down AI development “to protect the health, the safety, and the well-being of the American people” while Republican Senator John Kennedy plans to introduce a bill to require companies to create “kill switches” for their AI models. Democratic Representative Ted Liu and Republican Representative Nathaniel Moran already introduced similar legislation in the House last July. 

AI has also led to some strange political bedfellows with left-wing independent Senator Bernie Sanders and MAGA figure Steve Bannon coming together to call for curbs on AI development, though they advocated radically different approaches. While Sanders promised to introduce legislation to permanently ban the development of “superintelligent” AI, Bannon called for the US to cut off China’s access to chips, technology, or knowledge related to AI while also calling for Chinese nationals to be expelled from US labs.  

Meanwhile, President Donald Trump has denounced efforts to slow down or rein in the technology as a “SICK conspiracy” and proclaimed that the only guardrails necessary are “a STRONG AND SMART (High IQ!) PRESIDENT.” Trump also suggested that curbing AI development would only benefit China. 

House Speaker Mike Johnson has tried to strike a middle ground, saying “obviously there’s a need for guardrails, but there’s also a keen need for the industry to self-regulate and self-monitor and report to us.”

At the local level, some of the US’s largest school districts are now imposing moratoria on the use of generative AI on its devices. 

What does all this mean?

Although AI safety is now at the forefront of public consciousness like never before, it’s unclear what, if anything, will change in the regulatory landscape. Congress won’t be able to act until after the midterm elections since the House of Representatives is in recess. Even if that weren’t the case, it seems unlikely that legislators would be able to find enough common ground to pass something given the GOP currently enjoys a trifecta in Washington and the party itself is divided on the issue of AI. A Democrat-controlled Congress might be more willing to act, but Trump could still veto any legislation to impose guardrails on the AI industry, and overriding that veto would require the support of a ⅔ majority in each chamber. 

There’s also the awkward reality that AI regulation isn’t just a national issue. Regulations made in the US or the EU don’t constrain other nations, and many of those other nations may view the issue of AI safety through a much different lens than the West does. 

The fact that North Korea has managed to build a nuclear weapons program despite being one of the most-sanctioned countries on the planet shows how a determined state actor can achieve technological progress even in the most adverse circumstances. It’s not beyond the realm of possibility that totalitarian states could see unrestrained AI development as a way to gain leverage over their more technologically advanced opponents. A global treaty could address these concerns, but the checkered history of non-proliferation agreements shows this isn’t a silver bullet. 

Since legally binding guardrails probably aren’t going to happen anytime soon, we will have to hope that enough AI developers listen to Asmodei and Altman and pursue meaningful self-regulation. It could happen, but it would require a willingness to accept restraints even if it means losing profits or a competitive advantage. The world will have to wait and see.

‍