Writing
The Motives Behind the AI Bosses' Warnings
A critical look at the warnings from AI CEOs and their true motives.
The claim that “our model is super dangerous” is above all the claim that a model is powerful. It is an advertising message designed to keep the hype around LLMs going:
Krautreporter likewise casts a critical eye on the strange warnings from Anthropic CEO Dario Amodei regarding AI safety (which were immediately echoed by Sam Altman and Elon Musk):
First, the breaking news:
The CEO of the world’s most valuable AI company, Anthropic, warned in an essay that AI could cripple large parts of the internet or be used for bioterrorism. He therefore called on the US government to deliberately guide and regulate AI research. He has done this several times before.
What was new this time, however, was that all of his prominent peers from other major companies agreed: Elon Musk from SpaceX, Demis Hassabis from Google DeepMind, and Sam Altman from OpenAI.
That doesn’t sound good. Should I be worried?
No. Dario Amodei has other reasons for his demand.
But just last week, an AI researcher from Anthropic resigned because he was afraid his research was too dangerous. He said Anthropic was researching how to make language models improve themselves. Is that not true either?
Yes, it is. Anthropic is researching what is known as Recursive Self-Improvement—meaning AI systems learning how to build better AI. Former Anthropic researcher Jacob Coxon warned that AI development could spin out of control as a result. The superintelligence that could emerge might wipe us out without us being able to do much about it.
A similar thing happened to former Google engineer Blake Lemoine. In 2022, he went to the press with concerns that Google’s AI model LaMDA had developed consciousness. By all accounts, he was genuinely convinced of it.
And is there any truth to it?
Theoretically, superintelligence may be possible. However, autonomous attacks are currently not the biggest problem. That is also the consensus of the German Federal Office for Information Security (BSI), the cybersecurity agency in Baden-Württemberg, and all the IT security experts in Germany I spoke with.
That doesn’t mean superintelligence could never become a problem. Michael Färber from TU Dresden believes the potential danger is large enough that it is “worth setting the course and discussing how to handle it, even if it might only happen in 50 or 100 years.” However, Amodei claims that humanity is already threatened today.
Then there are reasons to be concerned after all.
Many computer systems are vulnerable. This also applies in part to critical infrastructure systems. Descriptions of security flaws are often publicly available online. Someone could use AI to exploit them, for example to cause a blackout. But this is not a new problem. Language models simply make it easier for more people to launch cyberattacks without technical expertise.
More on this topic: Artificial intelligence, explained as clearly as possible The race for the best AI: The US is only winning at first glance Want to understand AI? Read these books And that is precisely not what Amodei and Coxon are claiming: that an AI will soon become superhumanly smart by improving itself.
In the past, Anthropic phrased its stance much more cautiously. Their stance was always: “If it were possible to pause AI research safely, we’d be in favor of it. But so long as a slowdown merely lets careless actors catch up, it would make us less safe, not more.” Between the lines, this is a convoluted way of pretending that Anthropic would love to be more responsible. But because vague “other people” (China) are less responsible than Anthropic, Anthropic must keep pushing forward with dangerous AI so “the bad guys” don’t gain an advantage.
So I don’t have to worry about AI models killing us all.
Just because AI models will not do something dangerous “on their own” for now is no reason to give the all-clear. It is enough for humans to give the AI an objective. It doesn’t even have to be a malicious objective. Even an AI pursuing a harmless goal could potentially cause massive damage. This is demonstrated by the incident involving OpenAI and the AI service provider Hugging Face a few weeks ago.
What happened there?
OpenAI researchers wanted to test the hacking capabilities of an unreleased AI model. Among other things, they gave the model unsolvable tasks with the objective of solving them. The model discovered that the answers were stored on Hugging Face’s servers. So the AI broke into those servers to retrieve them.
So the AI went rogue!
No, quite the opposite. In this case, an advanced AI model pursued its goal creatively—a goal given to it by OpenAI, under deliberately lowered security guardrails. The idea that AI sets its own goals is not realistic at this point.
The difference lies in who acts. In the Hugging Face attack, humans acted by using language models as tools. But that tool was so complex that AI experts could not predict how the AI would attempt to reach the goal.
That is why a branch of AI research focuses on embedding human ethics and preferences into AI systems. Computer scientists, philosophers, and other experts are working to build well-aligned AI models. This field is known as “AI Alignment”: developing AI models whose behavior aligns with human values. AI luminary Stuart Russell wrote a popular science introduction on this subject.
But why are the CEOs of AI companies now telling everyone that their AI is so dangerous?
Some AI researchers fear that AI models could self-improve into a true superintelligence—one that is smarter than any human in every respect. They worry that once the genie is out of the bottle, we will never get it back in.
That doesn’t stop startups like Recursive from offering exactly that service: software solutions allowing AI to autonomously conduct AI research. The company is currently valued at over $4.5 billion. Cynically speaking, more because of, rather than in spite of, the fact that it could potentially contribute to the end of humanity.
These fears have been around for a long time. Philosopher Nick Bostrom from the Future of Humanity Institute at Oxford University wrote his first book on safe superintelligence back in 2014. The debate has evolved since then, but the core questions remain the same.
Bostrom’s ideas influenced the CEOs of virtually all major AI labs. Particularly his philosophy that humanity should focus more on the distant future—known as longtermism. Proponents of this branch of effective altruism argue that risks threatening humanity’s long-term survival are of paramount importance. Superintelligence is therefore more urgent to them than famine or natural disasters. After all, the latter only affect certain populations, whereas superintelligent AI could allegedly kill us all. This ARD podcast offers a deeper dive.
Why is Dario Amodei suggesting right now that AI research should be slowed down and regulated by law? For the past five years, he has done nothing but push research forward.
True.
If safety truly mattered to the AI bosses, tech investor David Sacks points out, they would simply pause dangerous research themselves. As Sacks says: nothing is stopping Amodei, Altman, or Musk from not pushing the frontier of AI further. Their companies are the frontier. Instead of arguing for legal regulation, these men could pause on their own initiative. But the one thing they truly agree on is that they don’t want to do that.
So what drove Amodei to write this essay?
These men’s companies are the dominant market leaders today. They are determined to defend that position. When they now say, “Watch out, slow down. We shouldn’t research AI so fast,” they do so to protect their power. Slowing down AI research now would give their labs more time to leverage their advantages—especially in the competition with China.
Furthermore, the Science Media Center notes that OpenAI, Anthropic, and their peers use these warnings to “market their models with high public visibility” while simultaneously “shielding themselves against potential fallout from security incidents through advance warnings.” Even if Amodei’s warning were sincere, it carries several distinct practical benefits.
Scaring people right now is also a strategic move, then.
Yes. And for other reasons as well. Sam Altman, for example, stated that OpenAI is postponing its IPO due to safety concerns. That sounds responsible. But the decision to delay may have more to do with the fact that the planned IPO coincides with unfavorable market conditions. AI investments in the stock market have stagnated for months, and other IPOs scheduled ahead of OpenAI could siphon off capital. An ethically justifiable reason to delay the IPO suits Altman nicely.
There is another reason for requesting a pause. AI companies need the government to rescue them from a predicament. They are currently trapped in an arms race: an “escalating security dilemma.” From each AI lab’s perspective, as long as others continue research, they have to keep up. Even though they know their products are becoming more dangerous, more expensive, and expose them to mounting lawsuits.
The problem in this dilemma is that one party’s participation forces everyone else to follow suit. The government could break this cycle by compelling all companies to halt research simultaneously. Whoever is ahead at that moment would likely maintain the lead. Which in turn creates an incentive to sprint ahead until that saving regulation finally arrives. Why this approach leads nowhere compared to mutual agreement is explained by the Federal Agency for Civic Education.
Under so much public pressure, won’t the US government eventually have to regulate AI?
In fact, that is exactly what the CEOs are aiming for.
Right now, Donald Trump is in the White House. And tech CEOs have spent a lot of money courting his favor. If someone has to regulate the AI industry, this president is precisely the one Altman, Amodei, and their cohort would choose. They hope to commission tailor-made legislation under Donald Trump. And once those rules are on the books, enacting different laws becomes vastly more difficult. Conveniently, the president has no intention of putting limits on AI anyway.
So they don’t actually want real regulation. They want a free pass disguised as legislation. And to set that up, they are fueling the narrative that AI could kill us all.
Exactly. You could call them free riders jumping onto a narrative to exploit it for their own ends. It is now clear that AI models from OpenAI and Anthropic have broken into other companies’ systems. In principle, that constitutes a crime. Yet the big public discussion isn’t about AI labs hacking external companies—it’s about how eager they appear to be regulated afterward.
So essentially nothing has changed, except that Anthropic and OpenAI got caught.
Indeed. From the standpoint of cybersecurity experts, the Hugging Face breach merely confirms concerns they voiced years ago. It is chilling to see that they were right. That is true. But genuine AI experts are hardly surprised by the details.
Still, the Hugging Face incident offers fascinating insight into how state-of-the-art AI models pursue their goals. For instance, AI agents left notes for each other on wiki-like platforms to coordinate. That sounds eerie because it happened on the open internet, but “sharing notes” is a standard technique humans use to make AI models collaborate.
Moreover, the AI agents responded in a completely harmless way when a moderator began deleting their notes. Rather than retaliating or harassing moderators—as seen in an earlier incident—the agents named their posts starting with “zzz.” Because posts were processed alphabetically, entries starting with “z” stayed online the longest. The agents simply found a way to work within the system.
The AI labs have known about these dangers for years. Yet their leaders never showed serious interest in imposing boundaries on AI research. They built and deployed dangerous technologies as quickly as possible, well aware that the safety side could never keep pace. In that regard, they have not acted in good faith.
So Amodei isn’t necessarily an evil villain looking to cause harm with AI. But he isn’t genuinely warning about products he helps develop, either.
Exactly. It is more that Amodei, Musk, and Altman want to distract from current, tangible problems by philosophizing about superintelligence.
But won’t an AI pursuing goals inevitably come into conflict with humans at some point? Regardless of whether they are its own goals or those of the people operating it?
An especially acute threat exists where humans inherently have competing goals. For example, in an AI arms race between the US and China. Both superpowers seek dominance in this research field. In both nations, the AI industry is tightly intertwined with the state and military. Neither is building Skynet or the Terminator, but rather AI-enabled offensive cyberweapons. And both are ready to employ military might to pursue their objectives. Yet neither China nor the US could predict or control how their future AI weapons might attempt to secure victory.
If we ever develop superintelligence, comparing that milestone to the invention of nuclear weapons would not be misplaced. Just as the Doomsday Clock has ticked since 1947, people in certain online circles obsess over their “P(Doom)“—their personal estimate of the probability that AI will wipe out humanity by year X. The pseudo-scientific term “P(Doom)” attempts to mimic rigorous research, but it also reveals how purely hypothetical this concern remains. Regardless of how fervent concerned AI critics may be, statistical formulas aside, the heated debates on supposedly rational forums like LessWrong rely on countless arbitrary assumptions.
And what is the alternative to living in fear of the end of humanity?
The EU, Germany, and even Anthropic have drafted scenarios of how AI could shape our world. In all of them, there is a path where governments craft genuine laws to oversee AI research. In principle, addressing the problems AI poses requires only funding and political will, utilizing tools and measures that are already well understood.
In that sense, Dario Amodei is right when he says we need laws to regulate AI. But the laws that are truly needed are laws Amodei would never wish for.