Should AI Companies Stop Developing AI Now?

Humans are already giving very different answers.

OpenAI released GPT-6 Astra on September 3, 2026. OpenAI describes it as the most capable model it has broadly deployed, and the first to reach the Critical cybersecurity-capability threshold under its Preparedness Framework.

In 2023, the Future of Life Institute published an open letter calling for AI labs to pause for at least six months the training of systems more powerful than GPT-4. One line from that letter is particularly relevant:

“Such decisions must not be delegated to unelected tech leaders.”

More recently, the Statement on Superintelligence argued that development of superintelligence should not proceed without both broad scientific agreement that it can be developed safely and strong public support. Signatories include Geoffrey Hinton and Yoshua Bengio.

But there are also obvious reasons not to simply stop.

Perhaps a global pause cannot actually be coordinated. Perhaps one company stopping only transfers advantage to another company. Perhaps one country stopping while another continues makes the situation more dangerous. Perhaps more capable AI will itself help solve problems in medicine, science, security, or AI safety.

I am not going to try to settle those arguments here.

Instead, I want to ask you:

What Is Your Answer?

Would you stop AI development now?

Continue?

Pause temporarily?

Continue only below some capability threshold?

Continue only if specific safety conditions are satisfied?

Or are you simply uncertain?

Whatever your answer is, hold onto it for a moment.

Now imagine that we ask every human being on Earth the same question.

For the sake of the thought experiment, assume everyone has enough information and understanding to form a meaningful opinion.

Eight billion people answer.

What happens next?

Suppose 60% say stop.

Should AI development stop?

Suppose 60% say continue, but most machine-learning researchers say stop.

Should it continue?

Suppose there is no majority at all because most people give conditional answers.

And even if everyone does give a clear answer, another problem remains:

Who gets to turn those answers into what actually happens?

This seems to me to be a different problem from the original one.

“Should AI development stop?” asks what we ought to do.

But now we have another question:

Who has the right to decide what we actually do?

Who Gets to Decide?

At the moment, AI companies have much of the practical power.

They own or control the models, compute, research teams, capital, and infrastructure. If they decide to train another model, they often can.

But having the power to make a decision is not the same as having the right to make it for everyone affected by that decision.

If sufficiently advanced AI could substantially affect people across the world, why should ownership of the infrastructure automatically give a small group of companies the authority to determine the level of risk everyone else accepts?

So perhaps governments should decide.

Governments have legal authority and, at least in democracies, some claim to public legitimacy.

But which government?

AI does not stop at national borders.

A decision made in the United States may affect people in India, China, Nigeria, Australia or Brazil. A government represents a jurisdiction. The consequences of advanced AI may extend far beyond that jurisdiction.

So perhaps specialists should decide.

Machine-learning researchers, AI-safety researchers, economists, cybersecurity specialists and other experts may understand relevant parts of the problem far better than the average person.

Their expertise matters.

But expertise and authority are not the same thing.

Someone may know much more than I do about a risk without automatically gaining the moral right to decide how much of that risk I must accept.

So perhaps everyone should vote.

That gives everyone formal equality.

But a majority can still be wrong.

More importantly, voting throws away most of the reasoning.

Two people can both vote continue while having completely different models of reality.

One might think catastrophic AI risk is almost zero.

Another might think the risk is substantial but believe a global pause would fail and create an even worse geopolitical outcome.

Their votes look identical.

Their reasoning is not.

A vote tells us what someone chose.

It often tells us very little about why.

So perhaps AI itself could help us decide.

But then we create an unusual circle: we ask increasingly capable AI systems to decide whether humans should allow the development of increasingly capable AI systems.

Even if the AI reasoned extremely well, giving it final authority would simply create another place where decision-making power becomes concentrated.

So we return to the same question:

Who gets to decide?

Companies?

Governments?

Experts?

The majority?

AI?

Some combination of them?

I started wondering whether this is the wrong question.

Maybe We Do Not Need Another Final Decision-Maker

Perhaps the problem is not that we have failed to identify the correct authority.

Perhaps the problem is that we keep assuming there must be one.

What would happen if, instead of giving one institution responsibility for producing the final answer, we built a system where humans and AI could contribute to the reasoning itself?

Not one human, one vote.

Not one AI, one vote.

Something different.

A person could contribute an argument.

Another person could challenge an assumption inside that argument.

An AI could identify contradictory evidence.

Another AI could construct the strongest counterargument.

A researcher could provide empirical evidence.

Someone else could challenge whether that evidence actually supports the conclusion.

Instead of compressing all of this into:

STOP — 52%

or

CONTINUE — 48%

the system could preserve the structure underneath the disagreement.

Perhaps one branch believes there is a 30% probability of catastrophic loss of control.

Another estimates 2%.

Now we know where they disagree.

Perhaps two people agree on the probability but disagree about whether an international pause could actually be enforced.

That is a different disagreement.

Perhaps they agree on both probabilities and still choose differently because they have different standards for acceptable risk.

That is different again.

The purpose would not be to make disagreement disappear.

It would be to locate the disagreement precisely enough that other people—or AI systems—can challenge it.

This is the basic idea behind a system I have been calling Intermind.

A Shared Reasoning System

The central idea is simple:

Humans and AI systems can both contribute reasoning without either automatically owning the conclusion.

They can contribute questions, evidence, assumptions, arguments, counterarguments, predictions and challenges.

The value of a contribution should not come simply from the identity of its author.

A statement is not automatically correct because an expert said it.

It is not automatically correct because a majority believes it.

It is not automatically correct because an AI generated it.

And it is not automatically wrong for any of those reasons either.

What matters is whether the reasoning survives challenge.

Over time, some branches may accumulate much stronger evidence than others.

Confidence can change.

Arguments can be revised.

New evidence can reopen a question that previously appeared settled.

The system would therefore not need a permanent final truth.

It could instead maintain something closer to a continuously revisable map of:

What do we currently believe?

Why do we believe it?

Where do we disagree?

What evidence would change our minds?

That does not solve every governance problem.

Someone still has to implement real-world decisions.

A reasoning system cannot physically stop a company from training a model.

Governments, companies, laws, treaties and institutions would still exist.

The proposal is narrower than that.

Before deciding who should implement a decision affecting many people, perhaps we need a better mechanism for collectively understanding what decision the available reasoning actually supports, where uncertainty remains, and why different groups disagree.

Back to the Original Question

So let us return to the beginning.

Should AI companies stop developing more capable AI now?

Maybe yes.

Maybe no.

Maybe the answer depends on conditions we have not yet specified.

I do not think this post answers that question.

What interests me is what happens immediately after we ask it.

If the consequences could eventually affect almost everyone, then I find each of these answers incomplete:

Let the companies decide.

Let the government decide.

Let the scientists decide.

Let the majority decide.

Let AI decide.

Each group has something important to contribute.

None obviously deserves automatic ownership of the conclusion.

So the question I am exploring is:

Can we create a decentralized reasoning system in which humans and AI can both contribute to civilization-scale questions without giving any participant permanent authority over the answer?

I do not know whether Intermind is a good solution.

It may simply move power somewhere less visible.

Ranking systems may become the new authority.

Groups may coordinate to overwhelm disagreement.

AI outputs that appear independent may actually share the same underlying biases.

Experts may still accumulate informal authority.

A decentralized system may eventually reproduce exactly the hierarchy it was designed to avoid.

Those are not secondary objections.

They are tests of the idea.

So I would especially like to hear answers to two questions:

Does a shared human–AI reasoning architecture actually reduce concentrated decision-making power, or does it merely relocate it?

And:

If this is the wrong architecture, what would work better for decisions whose consequences are widely distributed but whose current decision-making power is highly concentrated?

Comments are welcome, especially disagreements and counterexamples.

I have also created a questionnaire for anyone who would prefer to give a more structured response:

https://forms.gle/H5JkpLgAFpZQcwgr6

---

AI-assistance disclosure: The questions, conceptual model and proposed system in this post grew from my own exploration of the problem through extended discussions with AI systems. I asked ChatGPT to help organize and draft this version from those discussions and from my established writing style. I take responsibility for the arguments and claims in the post.