The people warning us most seriously about advanced AI are often the same people racing to build it.
Anthropic warns about powerful autonomous AI. OpenAI is actively monitoring things like AI self-improvement, long-range autonomy and autonomous replication. Demis Hassabis talks about both the enormous potential of AGI and the need to develop it safely. Elon Musk has spent years warning about AI risk.
But none of them are stopping.
And maybe they can’t.
If Anthropic decides to slow down, that doesn’t mean OpenAI will. If OpenAI slows down, Google might not. If all of them slow down, there are labs and governments elsewhere that may keep going.
So you end up with a pretty bizarre situation where everyone can understand the risk and still have a perfectly rational reason to keep accelerating.
Basically: “I don’t know if we should be building this, but I definitely don’t want someone else to build it first.”
What makes this even more interesting is that AI is starting to help with AI research itself. Better models can help researchers code, test ideas, analyze results and potentially automate more of the work involved in creating the next generation of models.
So what happens if that feedback loop really takes off?
I don’t think that means we’ve reached the singularity. But I wonder if we’ve been imagining the singularity the wrong way.
Maybe there isn’t some obvious moment where everyone says, “Well, this is it.”
Maybe it starts with hundreds of companies and governments competing, AI systems helping develop better AI systems, huge amounts of money being spent, regulators trying to keep up, and everyone continuing because nobody trusts everyone else to stop.
Not a singularity exactly. More like singularity chaos.
I’m curious what people think: is there actually a realistic way out of this race dynamic, or does competition make continued acceleration almost inevitable?