View / How the AI apocalypse story escaped containment - Semafor

The “AI is going to kill us all” news cycle continued into the weekend, as OpenAI’s Sam Altman and XAI’s Elon Musk joined Anthropic CEO Dario Amodei’s call to invite independent reviewers into AI labs to ensure the world’s smartest computer scientists don’t accidentally cause the extinction of the human race.

A calm read of Amodei’s proposal may lower your blood pressure: He’s proposing things every AI lab needs to do anyway, mostly for commercial reasons. Here they are: Operational excellence, alignment, interpretability, testing and evaluation.

Imagine if an auto company CEO wrote a blog post suggesting that the industry should try to make reliable cars that react in predictable ways when you turn the steering wheel and hit the accelerator and brakes. Oh, and maybe try to figure out how these piston thingies make the wheels turn. Of course AI companies need to do these things. Who would put AI in charge of anything important if they didn’t know what it was going to do, or why?

Why wouldn’t AI companies invite independent evaluators in to help them with some of this? If philanthropists want to provide free labor, why put up a fight? What Amodei didn’t address is what happens when one of these independent evaluators decides that a frontier lab they are monitoring should simply shut down operations, leaving investors holding the bag. There is a risk that Anthropic, or any other AI company that gets this feedback will simply find a different independent evaluator.

Amodei proposes evaluating the danger level of a model by looking at “checkpoints. “If models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z,” he writes.

But model capabilities aren’t the most instructive measure of whether they pose a danger to humanity. If this were the case, today’s panic would have begun 29 years ago, when Deep Blue beat Garry Kasparov at chess. Imagine, a billion AI agents as smart as a chess champion, plotting to take over the world. Humanity would have no chance.

Thankfully, artificial intelligence doesn’t work like human intelligence. Just because a model can solve an extremely complex math problem doesn’t mean it could plot and execute check mate against humankind.

Even the ability of an AI model to improve the creation of new AI models (called recursive self-improvement) is not the thing that endangers humanity. To do that, AI models would need to reach a level of sophistication that allows them to operate independently on tiny computers, and then to continually learn in secret. Without that ability, humans can simply unplug them.

Frontier labs are certainly not building for that scenario: They’re spending hundreds of billions of dollars on AI data centers with the assumption that AI will continue to require massive, powerful computers for many years into the future. If they build a superintelligent AI model that can run on any computer and continuously learn, they will be putting themselves out of business.

There are plenty of real, even immediate AI risks, but human extinction can’t really happen as long as there’s an off switch.

There’s no mention of the doomiest scenarios in Amodei’s blog, or by people like the former Anthropic researcher Jacob Coxon, because even the most concerned people inside these companies don’t see it as an immediate threat.

The subject of AI safety, however, may have escaped containment in the laboratory that is San Francisco, spread by the wires of a breathless media for whom mass extinction is — let’s face it — an incredibly fun story. Now that it’s infected the networks of Washington and Brussels, and the biggest risk may be that hysteria replaces thoughtful debate.

Room for Disagreement: A new list on the rationalist site LessWrong, long a hub of worries about AI, offers “some ways AI could kill us,” including deadly viruses, killer drones, and blocking out the sun.

The “AI freakout” has reached a tipping point, the Wall Street Journal declares.

Altman told Fortune the company had delayed its IPO to settle safety concerns.

Momentum toward a bipartisan AI safety bill stalled in Washington Friday as some Democrats worried it wasn’t tough enough, Semafor’s Ashley Gold scooped.

The politics of AI are changing fast, and a conservative leaders last week described the technology as unleashing a horde of unwelcome “algorithmic immigrants.”