A subplot from The Hitchhikers Guide to the Galaxy may actually come true!

[Towlie]

The Hitchhiker's Guide to the Galaxy by Douglas Adams includes the story of Deep Thought, the supercomputer tasked with finding the answer to The Ultimate Question of Life, the Universe, and Everything. When it ultimately failed to produce a useful answer it set out to build another computer more powerful than itself!

Although Douglas Adams didn't elaborate on the implications of this, it follows that if a computer can create another computer more powerful than itself then that more powerful computer can in turn create another computer more powerful than itself, and so on, until some sort of intelligence limit is eventually reached.

I've believed for years that this could eventually happen but I didn't expect to see it in my lifetime until I spotted this article in the Guardian:

Anthropic says the world should have option to ‘pause’ on AI

US firm says it will convene policymakers for discussion of dangers, in post detailing progress of its Claude model

Anthropic has floated the idea of a worldwide “temporary pause” on AI development – and said it was going to convene “policymakers” to discuss the dangers of advanced AI – in its latest release touting the capabilities of its products.

In a long post on Thursday, Anthropic detailed the progress of its AI model, Claude, towards “recursive self-improvement” – that is, being able to make better and more powerful versions of itself. Recursive self-improvement is a bugbear of AI safety researchers, viewed as the key step for AI to become superintelligent and therefore unleash widespread consequences on humanity.

The idea features heavily in the widely read AI 2027 doomsday scenario of last year, which imagines AI agents designing more and more intelligent versions of themselves, one of which eventually kills all of humanity with a bioweapon in order to make room for more datacentres and solar panels.

more...

One thing I can say for sure is that nobody, not Anthropic nor anyone else in the world, will be able to stop the progress of technology. There will be no "worldwide temporary pause”. It'll be like the Manhattan Project, where once it's found to be possible then there follows a desperate race to be the first.