It seems like the past week or two has seen a fundamental shift in how the world is thinking about the future of AI. What I mean is, the world is freaking the fuck out and I’m seeing articles like Will AI Kill Us All? That link is my favorite factual reporting about the current state of AI - it deftly gathers all the threads into something for a layman to read. While it does seem like it was precipitated by a guy quitting Anthropic, I really paid more attention when Dario Amodei posted his statement about pacing the frontier. I think it’s disingenuous to wave Dario’s post away as a marketing gimmick. I believe most of the folks involved in the frontier of AI really do think it might kill everyone. I think it has a chance too, which is why almost exactly a year ago I read and reviewed If Anyone Builds It, Everyone Dies.
Wow. What a gloomy fucking paragraph.
How about we talk about nuclear weapons instead?
Technology as a Sword
Virtually everyone who can read this has lived their lives under the shadow of nuclear war because 80 years ago America figured out how to build a bomb and then killed over 200,000 people. As 9/11 just passed, I feel like it’s important to put into context that this is about 70 times the number of Americans that were killed on that dark day. I connect these two events in my head because they are highlights of the scale of destruction that humans are able to wreak with relatively little skill of their own once the hard work has been done. Push a button, train on a simulator, or ask Claude to make a bioweapon.
Civilization pursues power and a natural consequence of power is the ability to destroy. Technology is frequently a sword, and moreover it’s a sword that eventually finds itself in the hands of people willing to destroy.
Some folks find this to be an argument to not pursue technology, which I find a particularly silly worldview I won’t waste my time on. It’s a pointless argument because, sword or not, technology is first and foremost a tool.
Technology as a Tool
Spouse was recently telling me about this family that lives in remote Alaska she had heard about. It was of interest because the husband, like me, is a nerd that works remote, and going to Alaska has been a dream of mine for decades. From what I understand, the nerd was able to move out there and work because of Starlink. Starlink, I should note, only exists because we thought we might need to kill millions of Russians 60 years ago. The space race was, after all, just another arms race. DARPA is responsible for the acceleration of waves of civilization-enhancing technologies. Some of the best shit we’ve ever come up with started out as ways to destroy our enemies. Honestly, you could probably even say much of our best shit started that way.
One of the most thought-provoking pieces of writing I’ve ever encountered is Marc Andreessen’s Techno-Optimist Manifesto. It submits, and I agree, that technology is not only the foremost tool for improving humanity but the only perpetual tool for improving humanity. If I had to answer in one word why I am relentlessly optimistic it would be “technology.”
Humanity, fuck yeah, we’re awesome.
Of course the myriad dead from technology might feel a little differently. Technology is cold comfort for those dying in a nuclear blast or a collapsing building or in a hospital bed to a manufactured gain-of-function virus. Andreessen dances around this issue in his manifesto of course. Pulling the bad out of focus and highlighting the good was his whole point, but it is one of my primary criticisms of his “rational belief” which smells a bit like faith-based arguments to me.
Mutually Assured Destruction
The thing about power is that the only thing that can check it is…someone else having power too. Thus is born MAD — mutually assured destruction — and such has been our lives. Such will be our lives ever more unless the world ends. It will always be some variation of people facing each other across an ocean or a border with each having a “destroy the world” button. Mostly these people would happily wipe out the other side if they thought they could get away with it, but they can’t, so instead we live with a doomsday clock.
When you pursue power, and capture it, there’s nowhere to go but towards even worse shit becoming possible. But if not you, then who? If not now, then when?
I’m not big on inevitability except at the most generic level, but that’s exactly what we’re talking about here. Nations get better at destroying other nations, and nations steal from each other. Here’s a brain cooker - I’m not sure that the nerd gets to work in Alaska without the Nazis’ research on rockets and Wernher von Braun. Neither does Ukraine have the capacity they have right now to drone the shit out of Russia as they defend themselves. You either have the power or you get it used against you.
Anyone who clings to the historically untrue—and thoroughly immoral—doctrine that ‘violence never settles anything’ I would advise to conjure up the ghosts of Napoleon Bonaparte and of the Duke of Wellington and let them debate it. The ghost of Hitler could referee, and the jury might well be the Dodo, the Great Auk, and the Passenger Pigeon. Violence, naked force, has settled more issues in history than has any other factor, and the contrary opinion is wishful thinking at its worst. Breeds that forget this basic truth have always paid for it with their lives and freedoms.
~ Robert A. Heinlein, Starship Troopers
Acceleration
This is the simplest and most elegant argument to move as fast as fucking possible with AI research. Pave over the fucking country with datacenters. Because if we don’t have the weapon then China gets the weapon and I promise you do not want that if you’re reading this. Ask Japan how it felt for their enemy to get the nuke.
The language coming out of China reflects this as they use words like “Cold War” positioning. It’s in the arguments made by Dario. It’s in the arguments made by Trump. It’s in my own arguments. This is nature, red in tooth and claw. This is standard power dynamics and it’s what Heinlein was talking about 60 years ago.
Sucks, bro.
But this is the lens I think has to be used. I think this is the most important consideration right now because I think AI makes nuclear warheads look like cherry bombs.
Self-Improvement
The most important thing that sets us apart from all the other bundles of living systems on the world — our bottomless ability to self-improve. To learn at a scale that nothing else can compete with. Goldfish can learn. Rats can learn. Plants can adapt. All organic systems mutate so all organic systems, over enough of a timeline, can evolve. We stopped evolving a long time ago, and we figured out how to shrink that timeline down and make our own tools. We wanted more and we wanted it faster. Evolution takes millennia to solve a problem and it stops at the first solution. Humanity takes years, weeks, or days to solve a problem and we’ll keep improving on the solution too.
AI needs milliseconds to solve a problem and it will keep improving on it forever if something doesn’t stop it.
This is what the frontier models are doing. They’ve given AI a single task: improve thyself. The same thing that drives humanity has now been given to an alien thing. A thing that reads more than any of us, has a better memory than any of us, can iterate on problems faster than any of us, and can create tools for itself. The acceleration of AI over the past couple years is not because humans are suddenly smarter and more creative, it’s because they’ve created a tool that never stops making itself smarter.
If that’s not a terrifying prospect then you aren’t paying fucking attention.
Filter
Right now my P(doom) is probably 10%, which is around where it’s always been at. I’m just some asshole so it’s not like I really know what I’m talking about, but I guess it’s nice to know that’s about where many AI researchers are. It is also where I would have put my guess of the world ending with nuclear war anyways so I’m not sure a whole lot has changed for me. I think the doomsday clock is pretty silly but I think it’s worth recognizing that humanity has grown in power to the point where, yes, we could end ourselves. No emperor or tyrant could have ever done it before but now Oppenheimer’s Krishna has revealed themselves and that was that.
I like fatalism because it boils down to inevitability even while I recognize I’ve got some control over myself and how I approach life. That’s the heart of my refusal to be miserable which, hey look, talks about the AI “epoch.” The knife edge, which is really what all my meditations on choice are about, is recognizing inevitability of things but still embracing the idea that I can make a difference to myself, my family, and maybe a little bit of the world around me. The nuke was always getting invented, AI was always going to show up, and one of them may very well be the Great Filter solution to Fermi’s Paradox but, fuck it, just live your life otherwise.
Slowing Down
So this is the tension. If, like me, you suppose that AI is a weapon greater than nuclear warheads but you also suppose it is inevitable and you want to have the power more than the enemy…then the final question is whether slowing down reduces the chances of the end of the world.
I’ve got no fucking clue.
But I do know that game theory and the Prisoner’s Dilemma say slowing down is the stupid choice. I also know the stakes aren’t one dude having a stint in the pokey either but might very well be all of humanity. Really hope humanity doesn’t fuck it up.
My hope, my faith, is that humanity will only fuck it up a medium amount. My basis for this is really just how we’ve managed to keep stumbling up the mountain, more or less, since we climbed down from the trees. Or, perhaps, the better metaphor is us racing down the mountain dodging the holes and the cliffs. So far we’ve managed to always get back up and all I can do is hope we keep doing so.
Out in the Cold
I am not in the mainstream here. Nate Silver just published a piece that shows that, across party boundaries, a supermajority thinks we need to slow down. Not the first time. I really liked where he captured the reasons though because it crystallized why I’m not in the mainstream. I need to sit with it a bit but I think there’s only a singular reason for me.
To be perfectly clear I have concerns about many of these others, but none of them are a reason to slow down. But ending the world? Yeah, okay, maybe we can pump the brakes. I just don’t know what that looks like because we are in an arms race even if not everyone wants to talk about it. Technology companies are building the next nuclear bomb and I think it has the chance to skyrocket humanity to unimaginable heights as long as we don’t fly too close to the sun I guess. Fingers crossed, right?
Squirt Says…
I feel like we should use AI as a tool rather than a weapon or a crutch. Using it like a screwdriver rather than using your hands for a screw. Using AI for these purposes doesn't work towards destroying us, not using it as a weapon. I would say that it copies human's empathy and it will not kill humans because it was trained to not.
Dad Responds…
I love you bringing in empathy, and I think that’s a big part of what AI alignment is about — making it understand the humans around it.
