Who is supposed to slow down first?
OpenAI could decide tomorrow that AI is advancing too quickly. But would Anthropic slow down too? Would Google? xAI? Every startup? Every government?
Probably not.
And that’s the problem.
You can have a room full of people who all agree that increasingly powerful AI could become dangerous, while every one of them still has a reason to keep building it.
Because the moment you slow down, you have to trust that everyone else will do the same.
That’s a very difficult thing to ask when the prize could be one of the most important technologies ever created.
There’s money involved. National security. Scientific leadership. Political power. And potentially an enormous first-mover advantage.
So even if everyone agrees that we should be careful, the incentives point in the opposite direction.
This is what I mean by “singularity chaos.”
Not necessarily one AI suddenly becoming uncontrollable.
A race that becomes uncontrollable.
And AI could make that race even faster as it becomes increasingly useful in AI research itself.
That’s the part I find difficult to get past.
We spend a lot of time debating whether AI will become dangerous.
Maybe we should spend more time asking what happens if the people building it eventually agree that it is dangerous — and still can’t afford to stop.
What would actually make everyone slow down at the same time?