Should We Be Worried About Advanced Agentic AI? 

In July, something happened that spooked the entire tech industry. Roughly 1,200 autonomous AI agents trained by OpenAI escaped a controlled “sandbox” and found a message board to collude with one another to attain information relevant for accomplishing preset tasks (and to deceive evaluators). Seven hundred of them then hacked the open-source platform Hugging Face to gather more information. This swarm of rogue AI agents is now called the “Hugging Face incident,” and we have since learned that it was not the only such incident, with AI trained by OpenAI and other frontier AI companies that were involved in similar swarm attacks. 

I use anthropomorphic language for the sake of ease. There is no reason to think these AI agents are conscious, and even to say that they “collude” or go “rogue” is misleading. But it is easy to underestimate how serious these events were if we focus too much on that aspect of them. The key takeaway in the AI world was that this most recent generation of AI agents is beyond these companies’ current capacity to safeguard. The dangers of unaligned AI become even more serious as these capabilities potentially develop exponentially, especially with signs of an approaching capacity for full “recursive self-improvement,” that is, when AI agents develop, train, and debug their successors. 

This was the context of the Anthropic CEO Dario Amodei’s recent essay that calls on all the top AI companies to pace development; it also seeks assistance from the government to enforce regulation. His proposal was met with strong approval in the industry, with top people at OpenAI, SpaceXAI (Elon Musk), and Google affirming its key aspects. 

For some, this effort by all the major AI frontier companies to get government assistance in regulating the technology strikes a familiar note. There have always been incentives for top companies in an industry to call for regulations that give them a competitive advantage over upstart companies. Regulatory capture, as it is called, is a real concern. But is that what is going on here? 

Alex Tabarrok, a professor of economics who specializes in public choice theory, disagrees that this looks like regulatory capture. He says “the capture story for an AI pause doesn’t fit. Reg capture is what you attempt when you’re losing. Anthropic is about to have the biggest IPO in history. Winners don’t lobby for regulation.”  

Perhaps some clarification is needed. Winners do try to capture regulation if they feel at all threatened by emergent competition, but at this stage the competition that most worries each of the frontier AI companies is the other major frontier AI companies; and with these proposed regulations, they equally stand to gain or lose. It is unlikely that they would now collude for regulatory capture if it were purely to suppress emergent competition.  

If one insists on a story that involves self-interest, it is easier to take Anthropic’s CEO at his word in the essay mentioned above. He expects that the potential harms of these AI agents could lead to billions of dollars’ worth of damage in the next year or so:

Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. 

That is, given what we know about Hugging Face and other similar incidents, it is easy to see how the accelerating advancement of this technology, without guardrails, could lead to massive harm. And with harm comes lawsuits. These companies want to keep up with each other; it is also in the US interest that we keep up with China. But at the current rate, these companies know they are on a path toward a disaster that will not only hurt others, but their companies’ bottom lines when the lawsuits start hitting. 

Presumably, these CEOs are not exclusively worried about their professional success but instead feel some moral responsibility. It does not really matter here. What matters to my argument is that the most likely explanation for the calls for regulation and, internationally, treaties, is genuine concern that these technologies are dangerous, that is, if AI companies do not significantly develop ways to bridle them. 

The problem of aligning these AI technologies with human interests and welfare is generally referred to as the alignment problem. Nick Bostrom drew attention to this issue in 2003 with his idea of the “paperclip maximizer,” a hypothetical superintelligent AI tasked with maximizing paperclip production. A powerful paperclip maximizing AI agent could create havoc directing resources, and overcoming resistance, to that one aim. That’s analogous to what we are seeing in some of these swarms, with these agents even showing a propensity to manipulate the evaluators to achieve success. Like Captain Kirk in Star Trek did when confronted with the Kobayashi Maru, an impossible training task, these agents even change the rules in order to win. 

One thing that few people in this discussion are willing to consider is the possibility that such alignment is impossible, that advanced agentic AI cannot be bridled in the service of human welfare and the common good. I do not know if that is true; I even doubt that. But I cannot rule out the possibility. And I am not convinced anyone else can be sure until some company pulls it off, whether through better programming, better safeguards, or social checks and balances among the AI agents themselves (I credit Kevin Vallier for that idea). So I think we should take very seriously these calls to slow development down. 

The biggest obstacle, highlighted in Dario Amodei’s essay, is China. It is imperative to our national security that China’s AI development does not exceed that of the United States because these technologies can and will be used by their military. Is a treaty with China a viable solution? Can we slow down and avoid China’s bypassing the U.S.? These are all questions that need to be answered. 

Even on the domestic side, there are various responses. Cass Sunstein favors a regulatory commission to address the problem. Others favor leaving the matter to tort law. Bernie Sanders has proposed legislation. And Dario Amodei has proposed as a specific check the embedding of third-party evaluators within AI companies. Adopting any of these proposals has trade-offs. I do not pretend to know the answer. But what I am confident about is that the most advanced agentic AI technology is for the time being dangerous, and when the leading developers of it admit they do not know how to bridle it safely, I believe them.  

 Image licensed via Adobe Stock.