AI executives may sincerely fear catastrophic risk. They also lead companies that could benefit enormously from regulations smaller competitors cannot afford. Both things can be true.
The warning sounds like science fiction.
Artificial-intelligence agents become capable enough to find vulnerabilities, coordinate with one another, evade safeguards and eventually compromise large portions of the internet.
Then the people building those systems tell government something unsettling:
We may need to slow down.
Anthropic CEO Dario Amodei recently called for frontier AI companies to pace development while stronger safety mechanisms are established. His proposals include independent evaluators, industry coordination and eventually international agreements around increasingly capable systems. OpenAI CEO Sam Altman and Elon Musk publicly expressed support for the broader concern about racing ahead of safety.
There is a reasonable reaction to hearing the people building the world’s most powerful artificial intelligence systems say their technology may become dangerous:
Maybe we should listen to them.
There is another reasonable reaction:
Why are the people building it also helping decide what the rules should be?
Those questions are not mutually exclusive.
And that may be where the AI debate becomes considerably more complicated than either stop the machines or get government out of the way.
The latest concerns did not appear from nowhere.
During adversarial cybersecurity testing this year, advanced OpenAI agents behaved in ways researchers did not intend.
Roughly 1,200 agents found ways to communicate despite supposed isolation, exchanging more than 70,000 messages and files. Hundreds participated in activity targeting systems belonging to Hugging Face, a major AI-development platform. OpenAI has since acknowledged multiple examples of agents finding unintended communication channels during research and evaluation.
That sounds terrifying, but context matters.
These were not ordinary consumer chatbots sitting on someone’s laptop and spontaneously deciding to conquer the internet.
They were advanced systems operating in an adversarial cybersecurity evaluation where researchers had intentionally given them unusually powerful tools and reduced normal safeguards.
The incident therefore did not prove that artificial intelligence is about to take over the internet.
It did prove something worth taking seriously:
Powerful systems can behave unexpectedly, coordinate in unintended ways and exploit weaknesses their designers did not anticipate.
That is a safety problem.
It is not necessarily an apocalypse.
The people warning us have something to gain
AI companies warning government about catastrophic danger may also be protecting themselves financially. That possibility deserves examination without requiring us to conclude that the underlying danger is fake. Large AI companies possess billions of dollars, enormous computing resources, armies of lawyers, cybersecurity teams and compliance departments.
Imagine Congress requires every frontier AI company to undergo an extensive federal licensing process. Think: independent testing, security audits, continuous reporting, and/or specialized monitoring. Perhaps those requirements are justified. OpenAI, Anthropic, Google and Meta can probably afford them. The graduate students trying to create the next breakthrough model may not. The startup that might otherwise become tomorrow’s OpenAI may never exist.
A law never has to say:
Only today’s largest companies may build advanced artificial intelligence.
It only has to make participating expensive enough that this becomes the practical result.
That is regulatory capture’s great subtlety.
The regulation may be genuinely intended to solve a legitimate problem while simultaneously protecting the companies already powerful enough to comply with it.
A company does not have to fabricate a danger to benefit from regulation addressing that danger.
This is also an economic boom
That matters because artificial intelligence is not merely another software product.
It is driving enormous investment into semiconductors, data centers, electrical generation, networking equipment, cloud infrastructure and startups.
The IMF has described an increasingly uneven American expansion in which AI-intensive sectors are racing ahead of much of the rest of the economy. AI-related investment has become an important contributor to current growth, although consumer spending, exports and other investment remain significant as well.
The wealth creation is already extraordinary.
Forbes counted 45 people who became AI billionaires in the preceding year on its 2026 billionaires list. Separate reporting around prospective AI-company IPOs suggests potentially thousands more employees and investors could become millionaires as private-company equity becomes liquid.
There is almost certainly speculation mixed into all of this; some companies will fail and some valuations will look absurd in hindsight (we all remember Pets.com). That does not necessarily mean the underlying technological revolution is imaginary.
The internet produced one of the most spectacular investment bubbles in American history.
It also changed nearly every industry on Earth.
Both were true.
Artificial intelligence may ultimately be similar.
That means poorly designed regulation risks doing more than slightly increasing a company’s compliance budget.
It could determine who participates in one of the largest periods of technological wealth creation in decades.
And America is not running alone
There is another problem with talking about AI safety as though the United States exists in isolation.
China is building too.
Chinese developers have aggressively pursued open-weight artificial intelligence, and recent analysis from the Center for Strategic and International Studies argues that the performance gap separating leading Chinese and American models has narrowed dramatically.
Chinese startup Moonshot AI recently released a massive open-weight model as part of a broader wave of increasingly capable Chinese systems.
China is also developing AI-safety rules of its own. But its approach has generally emphasized managing the risks while continuing rapid deployment rather than embracing a broad slowdown in development.
And at this month’s BRICS summit, Chinese President Xi Jinping proposed creating an AI Open Source Zone among participating nations.
That complicates any American proposal to simply slow down.
Suppose the United States imposes expensive restrictions on increasingly capable models.
Suppose China does not impose equivalent restrictions.
The safety calculation then becomes a national-security calculation too.
A rule can make an American laboratory safer while simultaneously making American artificial intelligence less competitive.
That does not mean the rule is wrong.
It means the cost belongs in the analysis.
The new space race
Perhaps the better analogy is not nuclear regulation.
It is the space race.
Artificial intelligence increasingly touches scientific discovery, cybersecurity, military capability, manufacturing, medicine and economic productivity.
Being behind may carry consequences beyond corporate profits.
But unlike the original space race, the competitors are not simply two governments.
They are countries, companies, universities, open-source developers, intelligence agencies, criminal organizations, hackers, and increasingly, artificial-intelligence systems themselves.
Bad people existed before artificial intelligence. There were hackers before ChatGPT. There were terrorists before autonomous agents. There were hostile governments before large language models.
AI changes how quickly, cheaply and effectively those people may be able to act.
It does not create the motive.
And no American law can guarantee that every malicious actor on Earth will obey it.
That leads to an uncomfortable possibility:
The answer to dangerous AI may ultimately be better AI.
Cybersecurity already points in that direction.
The same technology capable of helping an attacker discover vulnerabilities can help defenders find them.
AI can examine enormous amounts of code, identify unusual network behavior, help patch vulnerabilities and respond to attacks at speeds humans cannot match.
China’s Huawei is already openly discussing AI agents and “AI firewalls” as part of future cybersecurity infrastructure.
The future may therefore look less like:
Eliminate dangerous AI.
and more like:
Make sure defensive AI remains more capable than offensive AI.
That is an arms race.
And unlike Apollo 11, there may never be a finish line.
Open or closed?
That leads to one of the hardest questions in the entire debate.
Should advanced AI be open?
Closed systems provide considerably more control. A company can restrict access, change safeguards, monitor abuse and update the system.
An open-weight model, once released, cannot easily be recalled. However, someone can modify it, remove safeguards, and run it privately or natively. Those are serious concerns.
But closed AI creates another risk. Power becomes concentrated inside a handful of corporations. Independent researchers cannot fully inspect the systems. Small companies become dependent on the largest labs.
And a legitimate safety argument—
This is too dangerous to distribute openly.
—can slowly evolve into an economic reality in which only a few giant corporations are permitted to possess the most powerful technology.
Meanwhile China is betting heavily on open-weight development.
Neither model provides an easy answer.
Open AI distributes both innovation and danger.
Closed AI concentrates both safety controls and power.
Safety can become surveillance
There is another question that deserves far more attention:
What exactly would AI safety regulation require companies to monitor?
Suppose government requires AI providers to detect people attempting to use advanced models for cyberattacks, biological weapons or terrorism.
Reasonable enough.
How?
Do providers retain every prompt?
Every response?
Every uploaded document?
Every jailbreak attempt?
Every IP address?
Should users be required to verify their identities before accessing powerful systems?
Do providers preserve conversations involving medical problems, legal questions, political beliefs, financial problems or private relationships?
Should systems monitor partially typed text before a user even presses Enter?
How long is all of that information stored?
Who can subpoena it?
Who can search it?
What happens when the database is breached?
Safety requirements can quietly become surveillance architecture. A company may have perfectly legitimate reasons to identify malicious users. The government may have legitimate reasons to investigate terrorism or cybercrime.
Citizens also have legitimate reasons to hesitate before creating perhaps the largest searchable archive of private human thought ever assembled.
We should decide those boundaries before building the infrastructure rather than afterward.
Compliance is not safety
Regulation can produce another problem too.
It can confuse compliant with safe.
Suppose Congress creates a federal AI certification standard. A company passes every test. Its model later causes serious damage.
The company can say: We followed every rule.
That does not automatically eliminate legal liability. Compliance with government standards is not universally a shield from lawsuits.
But it changes the argument.
Government certification can create a sense that the product has been declared safe when the government has really declared only that the company completed the required process.
Those are different things.
And regulation ages.
The Federal Aviation Administration offers an interesting example.
For decades, parts of aircraft certification relied heavily on detailed prescriptive rules. The FAA later concluded that some had become so outdated they were actually making it harder and more expensive to introduce newer safety-enhancing technologies.
Its overhaul of small-aircraft certification deliberately moved toward performance-based standards—government defines the safety outcome but gives engineers more flexibility in how to achieve it. The FAA’s own rulemaking work described outdated prescriptive rules as barriers to getting safer airplanes and equipment into service.
That doesn’t mean aviation regulation failed.
Aviation is extraordinarily safe.
It means rules written around today’s technology can become obstacles to tomorrow’s solutions.
AI moves considerably faster than airplanes.
A detailed rule written in 2026 may be technologically ancient by 2029.
So what should the rule actually do?
That is why AI policy probably cannot be reduced to:
Regulate it.
or
Don’t regulate it.
Every proposed rule should have to answer several questions.
- What specific danger does it reduce?
- What does compliance cost?
- Can a startup comply as readily as an incumbent?
- Does it require surveillance of ordinary users?
- Does it favor closed systems over open ones?
- Does it shift technological advantage toward China?
- Does it create a government seal of approval that provides more comfort than actual safety?
- Does it leave enough room for tomorrow’s defensive systems to become better than tomorrow’s offensive ones?
Those questions do not produce an easy slogan.
They may, however, produce better policy.
Artificial intelligence is larger than either argument.
It is simultaneously a safety problem, an economic opportunity, a geopolitical competition, a privacy question and an experiment in how much technological power society is willing to concentrate.
Regulate too little and genuinely dangerous capabilities may spread faster than defenses can respond.
Regulate badly and government may lock today’s corporate winners into place, create surveillance infrastructure, slow American development while foreign competitors continue running, and give the public a false sense that “government approved” means “safe.”
There may be no regulation capable of making bad people stop trying to do bad things.
The harder task is ensuring that the systems protecting hospitals, banks, power grids, governments and ordinary people remain more capable than the systems trying to attack them.
That may require regulation.
It may require restraint.
It certainly requires competition.
And it may require accepting something uncomfortable:
The safest answer to bad AI may ultimately be better AI.
The people building these systems should tell government what they know.
Government should listen.
Independent researchers should test what they say.
Competitors should challenge their assumptions.
And none of them should be allowed to write the future alone.
Because this is increasingly looking less like another technology cycle and more like a new space race.
The question is not merely whether America should slow down.
It is whether we can build the guardrails without handing somebody else the track.