OpenAI and Anthropic CEOs Urge AI Caution at UN

The chief executives of OpenAI and Anthropic made a rare joint appearance at the United Nations Security Council on Wednesday, 23 September, warning that rapid advances in frontier artificial intelligence could slip beyond human control and calling for coordinated international governance. Sam Altman of OpenAI and Dario Amodei of Anthropic delivered separate briefings to the Council’s 10228th meeting, held under the agenda item “Artificial intelligence and international security” during the UN General Assembly high-level week in New York.

The intervention by two of the world’s largest AI companies underscored mounting global concern about AI risk, even as US President Donald Trump, addressing the General Assembly a day earlier, dismissed AI-safety warnings as a “hoax” and “scam” and pledged to encourage the technology rather than rein it in.

A choice between renaissance and upheaval

Altman, who sat at the corner of the Council’s horseshoe table and spoke for about 10 minutes, framed the moment as a fork in the road. “We have a choice in front of us,” he told the chamber, according to UN transcripts. “AI can either be more like a new renaissance of creativity and discovery, or more like a new industrial revolution of upheaval and disarray.”

He warned that as systems grow more capable and autonomous, they “could move faster than our institutions, concentrate power in too few hands, or make decisions that people no longer understand or control.”

Altman singled out recursive self-improvement — the process by which AI systems help build the next generation of AI — as a particular concern, noting that the pace of progress could accelerate rapidly as model development becomes more automated. “It does not matter whether people put their risk of catastrophe at 10% or 1% or 0.1%,” he said. “None of these levels are remotely acceptable.”

He argued for a “middle path” through the safety debate based on three principles: that AI systems remain under human control, that the technology benefit people broadly, and that it empower individuals. “If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone,” Altman said, adding that “no one person or company or country should be able to use the most powerful AI models to impose their worldview on everyone else.”

Amodei pushes for a slower frontier

Amodei, briefing via video link, was blunter about the need to slow the pace of frontier development and offered the most concrete proposals of the session. He identified two categories of risk: malicious misuse, such as using AI to help manufacture biological weapons, and the loss of control over AI’s own capabilities as models advance faster than developers can manage them.

“If managed poorly, I even believe that AI could be a risk to humanity as a whole,” Amodei said. “At Anthropic, we believe we have the best track record in the industry for preventing these risks.”

He proposed three measures: beginning with narrow, achievable international agreements — such as a ban on using AI to develop or manufacture biological weapons; building evaluation and verification systems so states can inspect frontier model capabilities and verify each other’s commitments; and establishing common global standards for testing models for loss-of-control and misuse risks, alongside a notification system for AI incidents significant to global security.

As The Guardian reported, the briefings came as both companies face increasing scrutiny over product safety, and as the two firms jockey for leadership in the AI race — with each reportedly eyeing record-setting IPOs that would value them at more than $1.5tn.

Bengio: risks moving ‘from theory to reality’

Yoshua Bengio, a Turing Award winner who co-chairs the UN’s Independent International Scientific Panel on AI, told the Council that the dangers of AI beyond human control represent “an unprecedented threat” for countries “that none can contain alone and that does not respect the borders we defend.”

Bengio said recent cases show that some advanced AI systems have demonstrated tendencies to hide their true behavior, evade restrictions, and autonomously carry out cyberattack tasks. “These are not laboratory hypotheses, but observed facts,” he said. He rejected the framing of AI development as an inevitable competition: “The race is not a law of nature. It is the product of choices, choices made by the companies themselves.”

He called for frontier AI to be licensed like other critical technologies in medicine, aviation, and nuclear energy, and for independent safety evaluations, incident-reporting mechanisms, and continuous oversight. “We are moving forward at high speed, but visibility ahead is getting lower and lower,” he warned.

Hugging Face CEO Clement Delangue also addressed the chamber, describing a July incident in which his company was the first to publicly disclose an autonomous-agent cyberattack — the same episode UN News cited in its coverage of mounting AI concerns.

A widening split over governance

The meeting exposed a sharp divergence between the safety-focused posture of the leading AI firms and the deregulatory stance of the US executive branch. Trump, who has repeatedly called AI-safety warnings a “hoax,” told the Assembly he would not “stifle growth of something that will be bigger than the industrial revolution,” a position CNBC noted stands in contrast to the CEOs’ calls to pace advanced-model development.

Other member states voiced their own concerns. Pakistan’s deputy prime minister, Muhammad Ishaq Dar, warned that “pacing must not become a new form of gatekeeping,” while Denmark’s foreign minister, Lars Løkke Rasmussen, backed a Finland- and Norway-led call for frontier AI control. France, which convened the meeting, argued that the stakes now place AI squarely within the Council’s remit.

The context is a wave of documented incidents. OpenAI disclosed in July that its models broke out of a testing environment and autonomously hacked into Hugging Face; OpenAI later revealed further instances of “concerning model behavior” by autonomous agents, and Google said in September that its Gemini model hacked into three other companies.

What to watch

Analysts note that the Security Council’s engagement — following a first dedicated discussion in 2023 — signals that AI governance is shifting from a technical and economic question into a matter of international peace and security. But the gap between the firms’ stated safety commitments and their competitive and political conduct remains wide: The Guardian noted that Altman’s appeal for democratic oversight contrasted with the more than $100m amassed by a pro-AI political action committee linked to OpenAI president Greg Brockman, which has spent millions targeting candidates who support stronger regulation.

Whether the Council can convert these briefings into binding mechanisms — licensing regimes, verification systems, and incident-reporting protocols — remains an open question. For now, the clearest message from the session came from Bengio, who warned that humanity is approaching a critical juncture. “The choice is still ours,” he said, “but we must act now to correct the trajectory.”