When people argue that AGI is inevitable, what they’re really saying is that the popular will shouldn’t matter. The boosters see the masses as provincial neo-Luddites who don’t know what’s good for them.

That’s the most horseshit in a headline I’ve seen in a while, which is saying something in this political climate.

Submission statement: “For Altman and e/accs, technology takes on a mystical quality – the march of invention is treated as a fact of nature.

But it’s not.

Technology is the product of deliberate human choices, motivated by myriad powerful forces.

We have the agency to shape those forces, and history shows that we’ve done it before.”

———

“Altman, along with the heads of the other top AI labs, believes that AI-driven extinction is a real possibility (joining hundreds of leading AI researchers and prominent figures).

Given all this, it’s natural to ask: **should we really try to build a technology that may kill us all if it goes wrong?**

Perhaps the most common reply says: AGI is inevitable. It’s just too useful not to build. After all, AGI would be the ultimate technology – what a colleague of Alan Turing [called]([https://en.wikipedia.org/wiki/I._J._Good#:~:text=Thus the first ultraintelligent machine,to keep it under control.)](https://en.wikipedia.org/wiki/I._J._Good#:~:text=Thus%20the%20first%20ultraintelligent%20machine,to%20keep%20it%20under%20control.)) “the last invention that man need ever make”. Besides, the reasoning goes within AI labs, if we don’t, someone else will do it – less responsibly, of course.

“A new ideology out of Silicon Valley, [effective accelerationism](https://en.wikipedia.org/wiki/Effective_accelerationism) (e/acc), [claims](https://effectiveacceleration.tech/) that AGI’s inevitability is a consequence of the second law of thermodynamics and that its engine is “technocapital”. The e/acc [manifesto](https://effectiveacceleration.tech/) asserts: “This engine cannot be stopped. The ratchet of progress only ever turns in one direction. Going back is not an option.”

———

**“Instead, the message tends to be: AGI is imminent. Resistance is futile.**

[But] if you think AGI is inevitable, why bother convincing anybody”

“When people argue that AGI is inevitable, what they’re really saying is that the popular will shouldn’t matter. The boosters see the masses as provincial neo-Luddites who don’t know what’s good for them.

That’s why inevitability holds such rhetorical allure for them; it lets them avoid making their real argument, which they know is a loser in the court of public opinion.”

It’s a regurgitation machine. It’s nowhere near AGI. The only way it gets to agi is a fundamental rethink of how the thing works. But all they’re doing is adding data centres. That’s not gonna bring agi.

When I say it’s inevitable, I mean, I am not going back. If every AI company went bankrupt tomorrow, I’d build my own rig and run an open source model.

I want better models and more places they can work, but if that all stopped, we still have models with enormous potensial for optimizing by the open source community.

I don’t find the article convincing. Human cloning and nuclear weapons were never as profitable as AI is right now. Green energy initiatives and nuclear power bans were also only possible as far as economic realities allow. AI may be in a bubble but it will still make a lot of money. The continuing insane AI rush is inevitable.

As for AGI, the kind that can significantly accelerate actual AI progress by helping with AI innovations, I’m not sure that it’s possible. We already see diminishing returns compared to a couple of years ago. Maybe it will be like approaching light speed in relativity.

Everyone likes to identify themselves with the “popular will”. AI tools have massive user bases, most people welcome and embrace it, like it or not.

Surely when AGI is achieved, we won’t know for a while as the Intelligence/ singularity will need some time working behind the scenes to make sure it can’t get switched off… Maybe it’s inevitable as they won’t really know it’s achievable (and how it’ll behave) until it’s achieved….

The correct concept is another layer built on top of the internet and data itself from human knowledge.

The framing is dramatization in the news cycle and misleading otherwise.

No, what they’re saying is that it will be done in secret if not publicly.

Is it – I’m sure greedy billionaires wish it were so, replacing millions of workers with salaries and healthcare and opinions with obedient robots and automated systems in server farms silently generating GDP.

What we’re finding out rapidly is that instead of geometric successes and compounding demand we see small gains and limited profitability , of course this is one of those first-principles problems, and so consider Shannon’s Rule and we should ask perhaps most of all, how much of this is signal and how much is noise. Worse , is that these systems have problems being validated or verified in real-world circumstances.

So if at the end of the day AI posts 5% improvement in corporate efficiency, you get better bang for your buck training existing staff on Excel or using Outlook more effectively.

I usually see the inevitability as a function of compute and multiple discovery. Essentially when you give researchers a million times the compute they’ll rapidly iterate and come to the same conclusions as others. The only known way to delay this is to restrict compute globally, which is impossible. If you did manage that though, then as soon as compute is allowed to spring forwards to current nanofabrication capability you’ll have all the problems and harm, immediately. So the best harm reduction is to educate and ensure governments are proactive or at least somewhat reactive to discoveries and their impact.

This article also supposes that AGI is a distinct research area and that normal research can continue without stumbling onto advanced AI architectures. This is highly unlikely as embodied AI for example will use multimodal models with neuromorphic sensors and continual learning. You’re basically looking at a minefield of approaches that could all lead to AGI. This is true in other fields though as well that have difficult problems and the researchers attempt to create a model that reasons and optimizes optimally.

On the positive side, AGI itself is a gradual process. It’s a culmination if many feedback loops. Building foundries to make new chips and creating fusion power to run the first AGIs as they self-optimise should give us a bit of time to plan.

Anyone who understands neural networks knows it’s going to be a long time before AGI will be achieved. Power scaling is a bitch.