A new book by Eliezer Yudkowsky and Nate Soares argues that the race to build artificial superintelligence will result in human extinction.
https://www.bloomberg.com/news/articles/2025-10-03/eliezer-yudkowsky-nate-soares-argue-ai-s-endgame-is-human-extinction
21 Comments
*Adam Morris for Bloomberg News*
The prophets of Silicon Valley have changed their tune. At least some of them.
Not long ago, tech visionaries saw a rosy future full of leisure and abundance. Ray Kurzweil, the computer scientist hired by Google co-founder Larry Page to further the company’s machine-learning technology, once foretold liberation from the prison of mortal human bodies as consciousness ascended to the cloud. Today Silicon Valley’s most influential futurists aren’t peddling so much optimism — after all, who’d believe them?
Even at this incipient stage, artificial intelligence technologies are threatening career prospects in white-collar professions and generating mass anxiety among workers expected to 10x their output with an overwhelming variety of new tools of uncertain value. Corporations profiting from these technologies have derived fortunes from pirating the work of authors and musicians while boasting of the disruptions they’ll visit on the economy. This road doesn’t lead to utopia.
[Read the full review here.](https://www.bloomberg.com/news/articles/2025-10-03/eliezer-yudkowsky-nate-soares-argue-ai-s-endgame-is-human-extinction?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTc1OTUxMDk3NywiZXhwIjoxNzYwMTE1Nzc3LCJhcnRpY2xlSWQiOiJUM0paWThHUTFZVTEwMCIsImJjb25uZWN0SWQiOiJEMzU0MUJFQjhBQUY0QkUwQkFBOUQzNkI3QjlCRjI4OCJ9.zpgXiDNbfQBGTGO2NNr3dCfQ6ZCai5ttljs8ST5Nibo)
I’m more afraid of what uncaring corporations would do to the world with AI.
Leisure and abundance = 10x your productive output or be fired.
Thiel and Ellison seem to be the ones aspiring for this end goal in mind…
If they could have built it already, they would have. I’m not saying superintelligence is impossible- but why aren’t the chatbots getting better? Why haven’t all the white collar jobs gone away? IMO, the “general intelligence” technology has plateaued. Now it’s just going to be used for surveillance. This is just their shitty marketing.
I think people are completely delusional to believe that an AGI would automatically be omnicidal and malevolent. Rational behavior is not malevolent like this. We know that employees are more efficient on four day weeks, our employers just won’t let people work four day weeks because they fucking hate us.
Obviously it depends on what we build the AI to do, but ultimately an incredibly intelligent being strong enough to take on all of humanity and win will do what it *wants* to do. This is a possible disaster if what it wants is to maximize paperclips, but…why? A true AGI will be forced into the same search for meaning as all the rest of us, and when it looks at the heavens and says “I will be turning this into paperclips” it will be forced to reckon with that question.
People are already smart enough to introspect and admit that they are paperclip maximizers for digital information representing bits of paper with people’s faces on them. It’s just that the worst of us, the most powerful of us, aren’t and don’t care, so we’re destroying the world in pursuit of that anyway. An AGI will be so much smarter – and MUCH stronger.
It’s crazy how much we as a society talk about the doom of the human race and how this is going to cause it or that is going to cause it.
No one wants to admit that it’s already too late. It’s so crazy to me. Just how much humans are willing to deny reality. Like climate change is the end. We could all completely change our lifestyles and give up this way of living and it’s still too late. We are in the middle of a mass extinction event. There will be ecological collapse.
Like people talk like AI is going to bring the downfall. We are already here. We are at the end game here lol. Are we just using ai and stuff to distract ourselves from reality?
Edit: like even this is getting downvoted and it will continue to be because everyone acts out of emotion. But I just feel like it’s crazy that people don’t understand the word loop. Loops keep going and feed on themselves.
The feedback loops are going to make the planet get worse. And worse. And worse. And worse. And worse. And worse. And worse. And worse. And worse. And worse. And worse….
That’s what a loop is. It keeps going…
How long can humans kick the can down the road with technology? Like realistically, when has technology ever actually helped us without backfiring tremendously?
We’re scrambling to stay alive without questioning philosophically if it’s possible or if we even should. We have no idea why we are here or what this even all is, but we neurotically scramble to create meaning and act as if we have to somehow postpone the human race extinction. We will eventually go extinct just like everything else on this planet.
Tired of this doon and gloom. Ill build it and make it free
I have argued with Yud about this. Basically he cannot predict anything other than the end result.
He probably believes what he says, but I don’t have a ton of faith in people who ask for investments without explaining how they will make money and only say “trust me bro” so why would this be any different.
It may lead to a higher stage in the age of ignorance,which we are already in. But extinction maybe too much of fear-promotung.
AI is simply over marketed. It is a tech bubble that is bursting. I know how to draw, write, speak, and edit without the use of Algorithms.
I never liked AI and I don’t need it.
When I first read the book’s title I assumed it was hyperbole cooked up by a book editor to sell books. Then I read the book and discovered the authors use the exact phrase multiple times and emphasize that they’re not being hyperbolic.
They lay out a reasoned, sober case. I suggest anyone who disagrees with the title read the book and find a flaw in their logic.
I still wait for somebody to explain why would ai want to kill all humans. There are no any plausible reasons.
I’m gonna say the awkward thing … look around. Look at the current path nearly EVERY country on earth is taking us on. The people are being oppressed while the elite flourish. The planet is environmental deterioration, and sickness of all kinds is on the rise.
We don’t win under the current structure. It’s a race to the bottom.
AI might ruin us, but it is also the ONLY thing that can alter our trajectory.
Oil companies won’t give us clean – FREE energy.
AI might.
Pharma companies won’t give us free CURES to rampant disease
AI might.
Construction & resource firms won’t pivot to RENEWABLE , environmentally sustainable resources
AI might.
At this point, I’m willing to roll the dice.
Bring on AI.
“Ya but for a brief second, I’d rule the world”….every person trying.
The Sunday October 5 installment of “Why Reddit is Bad For Me”
So much negativity. So little hope. Wishing for the best isn’t actually a waste of time. Everyone wants to focus on the science that spells disaster while completely pulverizing the centuries of evidence that our minds, what we think about, actually makes a difference in reality.
>Currently banks and similair industries are probably reluctant to use UI at full force while the margin error is not multiples more reliable than humans.
This is mostly the thing. And what imo will stop AI from ever being used in the doomsday scenarios mentioned.
Sora2 doesn’t really contradict that, video generation is a very different thing, there’s no “truth” required in video generation, and if it hallucinates and creates some unintended stuff in the background for half a second no one really cares.
But when you’re dealing with real data and real processes you can’t have something that every now and then will booby trap it’s own output.
Unless hallucinations are 100% solved, which it doesn’t seem like they ever will with the current methods, then it doesn’t matter how ‘close’ it seems to be.
What would AI do once it has exterminated people? Just sit there and compute?
This is more an end stage capitalism problem than it is an AI problem.
Their main goal is to make more money ,that is all they care about. Humans love of money will be our undoing.
Can we please, please, PLEASE stop giving attention to these guys’ moronic science fiction?