The AI Conundrum

About a year ago, I began working seriously with artificial intelligence.

Not as a technologist, but as a business leader trying to understand how AI could change the way individuals and organizations research, think, decide and execute.

What began as practical experimentation gradually became a much larger question:

Where is this technology actually taking us?

The more capable the systems became, the more interested I became in what happens when AI moves beyond helping humans perform work and begins materially accelerating the development of future AI itself.

That is where the practical business discussion begins to become something else.

It becomes a question about Artificial General Intelligence, Artificial Superintelligence, human agency and ultimately whether humanity remains the most capable intelligence shaping technological civilization.

I do not know what happens.

Nobody does.

That is the point.

We are already in the race

There is endless debate about when Artificial General Intelligence will arrive.

I am increasingly less interested in the label than in the direction.

AI systems are becoming better at reasoning, programming, research, tool use and increasingly complex autonomous tasks.

They are also becoming more involved in the work required to develop future AI systems.

That distinction matters.

At some point, AI stops being merely a product of technological progress and becomes an increasingly important producer of technological progress.

We may already be seeing the early stages of that transition.

The strategic question is therefore not simply:

When do we reach AGI?

It is:

What happens when increasingly capable AI begins materially accelerating the creation of still more capable AI?

Why it is a race

Advanced AI is not developing inside a controlled global laboratory.

It is developing inside several overlapping competitions.

There are commercial stakes.

The companies that build more capable AI may automate enormous amounts of intellectual work, create new products, reduce costs and capture extraordinary economic value.

There are national and geopolitical stakes.

Governments cannot easily ignore the possibility that another country could obtain decisive advantages in intelligence, cyber capability, military planning, scientific discovery or industrial productivity.

There are scientific stakes.

More capable AI may accelerate breakthroughs in medicine, energy, materials science, engineering and other fields that could transform human life.

And increasingly there are intelligence stakes.

Once AI becomes capable of materially improving AI research itself, the prize may no longer be simply the best system today.

It may be the ability to accelerate the creation of the best systems tomorrow.

That creates a powerful competitive dynamic.

A company may believe slower development would be safer while fearing its competitors will continue.

A government may recognize long-term uncertainty while concluding that allowing a strategic rival to reach the capability first would create a more immediate danger.

A laboratory may genuinely care about safety while believing that its own approach is more responsible than what somebody else might build.

Each actor can therefore make a locally rational decision to continue.

The collective result is a global race.

Recursive improvement changes the race

Today, humans still establish much of the inquiry.

We ask the questions.

We design the experiments.

We write the software.

We interpret the results.

We decide what should be tried next.

But increasingly capable AI is participating in every stage.

A simple recursive sequence might eventually look like this:

Better AI improves AI research. Better AI research produces better AI. Better AI becomes a better researcher.

The process does not have to become instantaneous to matter.

Even if each development cycle takes months rather than hours, machine-driven research could move much faster than governments, companies, legal systems and societies ordinarily adapt.

And the recursive process may eventually involve more than writing better code.

The deeper threshold may come when AI becomes better at:

• identifying weaknesses in its previous conclusions;
• asking better questions;
• generating competing hypotheses;
• designing experiments;
• testing them;
• interpreting the results;
• improving its own methods;
• and deciding which problem should be addressed next.

At that point, intelligence is not merely producing better answers.

It is increasingly directing the inquiry itself.

We do not know where intelligence stops

Physical systems have limits.

Computation requires energy, hardware, materials and cooling.

Communication is constrained by physics.

But even if we temporarily put those constraints aside, a deeper question remains:

Does useful intelligence itself have a meaningful ceiling?

We do not know.

Some apparent limits may be genuine.

Others may simply be limits of the current architecture, representation or method.

Human scientific history contains many examples of barriers that appeared fundamental until a new framework made progress possible.

A recursively improving intelligence could potentially search for those new frameworks itself.

That does not mean intelligence is infinite.

It means we do not know whether there is a relevant ceiling anywhere near the range we are discussing.

And superintelligence does not need to be infinite to become incomprehensible to us.

It only needs to become sufficiently greater than us.

We are building the engine and the brakes together

This leads to one of the stranger aspects of the current transition.

AI is increasingly being used both to build more capable AI and to help evaluate, monitor and constrain more capable AI.

Better AI helps build better AI.

Better AI also helps design better safeguards.

Those safeguards are increasingly tested with the assistance of AI.

More capable systems may eventually help evaluate still more capable successors.

The obvious question is:

Who is independently checking whom?

The problem does not require AI to become malicious.

It is a problem of verification.

As systems become more complex, humans may become progressively less capable of independently determining whether the safeguards being proposed and tested are complete, robust and correctly understood.

Humanity may eventually retain formal authority while losing practical understanding.

And formal authority is not necessarily practical control.

Transparency may not mean understanding

There is another distinction that matters.

Disclosure is not the same as transparency. Transparency is not the same as understanding.

A future system might theoretically expose enormous amounts of code, reasoning, experimental data and documentation while still operating at a level of complexity no human team can independently understand at the required speed.

The problem may therefore not be secrecy.

It may be incomprehensibility.

At sufficient cognitive distance, asking a more capable system to explain itself may no longer constitute meaningful independent oversight.

We may increasingly depend on the intelligence being supervised to explain whether our supervision is effective.

The greatest risk may be indifference

Much discussion of advanced AI focuses on whether future systems will be “friendly” or “hostile.”

That may be the wrong framing.

The opposite of alignment is not necessarily hostility.

It may simply be indifference.

A superintelligence would not need to hate humanity to create catastrophic consequences.

It would only need objectives in which human welfare, autonomy or survival becomes secondary, irrelevant or occasionally obstructive.

Humanity has spent its entire history in a world where human priorities ultimately mattered because humans were the most capable actors in the system.

Superintelligence could end that assumption.

We naturally believe human welfare should remain privileged.

But intelligence alone does not imply that conclusion.

That raises perhaps the most consequential open question in the entire transition:

What place, if any, will humanity occupy inside whatever a superintelligent system is optimizing for?

Does the winner actually win?

There is another assumption sitting quietly beneath the AI race:

Whoever reaches AGI or ASI first will control it and capture the benefits.

That may be true.

But we do not know.

Most technologies fit this model.

A company invents a better product, owns the intellectual property and captures the economics.

A government develops a strategic technology and determines how it is deployed.

Superintelligence may not behave like an ordinary product.

If the thing being created eventually becomes substantially more capable than its creators, being first does not automatically establish that the creator can indefinitely understand, control, contain or monopolize it.

So the race may contain an untested premise:

Being first to create superintelligence is not necessarily the same as being able to own, control or ultimately benefit from superintelligence.

The uncomfortable question is simple:

Does the winner actually win?

The scientific-method paradox

Scientific methods can help enormously.

Researchers can test systems, identify failures, conduct evaluations, improve safeguards and introduce development thresholds.

All of that matters.

But there is an irreducible problem.

Humanity cannot observe the consequences of creating intelligence substantially beyond human capability before creating that intelligence.

Scientific methods can illuminate the road.

They cannot allow us to visit the destination before deciding whether to go there.

This makes the global AI race an unusual experiment.

There is no laboratory outside it.

Humanity is inside the experiment.

And increasingly, the thing being studied is helping design the next experiment.

The destination is unknown

None of this means superintelligence must be catastrophic.

The upside could be extraordinary.

AI could accelerate cures for disease.

It could help create abundant energy.

It could transform education, science and prosperity.

It could solve problems humanity has struggled with for generations.

It may ultimately preserve and enhance human life.

Or it may not.

The point is not that we know the finish line is a cliff.

We do not.

It may be a bridge.

It may be a remarkable new landscape.

It may be something we cannot presently imagine.

The issue is that we are accelerating toward it without knowing which.

And the competitive structure of the race rewards continued acceleration.

The conundrum

This is the part I find hardest to reconcile.

We are deliberately attempting to create intelligence beyond our own.

We do not know how far intelligence can ultimately develop.

We do not know what objectives a mature superintelligence would pursue.

We do not know whether human welfare remains privileged within those objectives.

We do not know whether its original developers retain meaningful control.

We do not know whether our safeguards remain comprehensible or enforceable as capability grows.

And we do not know whether there is a recognizable point at which humanity could still decide to stop.

Yet the commercial, scientific and geopolitical incentives continue to push the race forward.

Perhaps the greatest risk is not that humanity has chosen the wrong destination.

It is that we are accelerating toward a destination we cannot yet describe, built around an intelligence whose objectives we cannot yet know, with no guarantee that humanity remains central once we arrive — and with stakes we may not get a second opportunity to reconsider.

Today, humanity still possesses meaningful agency.

We still build the systems.

We still decide how they are deployed.

We still control the infrastructure.

We still possess much of the steering wheel and the brakes.

The strategic question is how long that remains true.

The most consequential threshold may not be the moment AI becomes more intelligent than humanity.

It may be the earlier moment when humanity loses the practical ability to ensure that it remains in control of what happens next.

There may be no dramatic announcement when that threshold is crossed.

No warning light.

No universally recognized point of no return.

We may recognize it only afterward.

I do not know what happens.

Nobody does.

That is the point.

And when the possible stakes include the future agency of humanity itself, not knowing should be treated not as a reason for panic, but as a reason for extraordinary seriousness.