A few secretive AI companies could crush free society, researchers warn | What happens when AI automates R&D and starts to run amok? An intelligence explosion, power accumulation, disruption of democratic institutions, and more

“What can happen is that a virtuous cycle develops, where the AI that runs R&D keeps replacing itself with better and better versions, becoming a “self-reinforcing loop” that is beyond oversight.

The danger arises when the rapid development cycle of AI building AI escapes human ability to monitor and intervene, if necessary.

“Importantly, if an AI system develops consistent scheming tendencies, it would, by definition, become hard to detect — since the AI system will actively work to conceal its intentions, possibly until it is powerful enough that human operators can no longer rein it in,” they write.

The authors foresee a few possible outcomes. One is an AI model or models that run amok, taking control of everything inside a company:

>

A second scenario returns to those malicious human actors. It is a scenario they call an “intelligence explosion,” where humans in an organization gain an advantage over the rest of society by virtue of the rising capabilities of AI.

The most dramatic “spillover scenario,” they write, is one in which such companies rival society itself and defy government oversight.”

Nah, that ain’t happening.

AI cant yet reason and it does not and we don’t know how to do that. We cannot even precisely define what that concept means.

Gotta be careful with that stuff regardless. A lot of people have been conflating democratic with “democratic©️™️“ for quite a long time