Is the Singularity Really Serious?

Risk = Probability x Severity

Life is full of problems. Many of these problems freak people out when I talk about them, so I don’t really say anything about them. But there’s one problem it seems pretty safe to talk about. And because it’s one of the few topics I can explore without worrying about offending anybody, I spend a lot of time (seriously, a lot of time) talking about the technological singularity.

If you don’t know much about the singularity, you can read about it here. For those who do know what it is, many are worried that the singularity is definitely coming, that it will definitely be bad, and that there’s almost no way to avoid it. Others object to the idea that we should be worried about the singularity at all.

Rather than keep you in suspense, I’ll state my own position right away—but I want to do it with propositional calculus, because true/false logic is easy to parse, and because letters are fun. The Romans Greeks Phoenicians have decreed that the first letter shall be A, and the second B. Therefore:

A = “The probability of a singularity in the near future having an impact on humanity equal to or worse than COVID is high.”

B = “We should prioritize efforts to understand, head off, and otherwise avoid the negative impacts of a singularity.”

I think A is false, and I think B is true. I’m not really sure why people seem to feel as though A and B should both be false (the singularity is unlikely, don’t prioritize dealing with it before it happens), or both be high (the singularity is likely, we should prioritize dealing with it right away). A and B aren’t equivalent, but on my favorite blog I encounter people saying things like:

For several reasons I do not think that AI will or even can achieve intelligence on that scale.

You are not alone… https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yudkowsky-on-doom-from-foom-2

And well, when you click the link, you find the author saying, among other things:

I have spent a good chunk of my life studying the AGI problem as an engineer (neuroscience, deep learning, hardware, GPU programming, etc), and reached the conclusion that fast [catastrophic singularity] is possible but unlikely.” (Emphasis in original)

I agree with this assessment overall; a fast, catastrophic singularity is unlikely, so no need to struggle maintaining our equanimity as we march towards a grim future, but it’s possible, so let’s take it seriously.

Part of the problem may be that the English language smooshes “probability of hypothetical bad thing” and “severity of hypothetical bad thing” into one blurry concept called “risk.” On any given day, the chance of stubbing my toe is low; the chance of getting in a car accident is also low. But while I go barefoot around the house or even my yard in summer, I don’t leave the driveway in my car without a seatbelt and a functioning airbag.

Even if the probability for a negative outcome is low, when possible severity is high, risk mitigation is a big deal. You don’t want something as bad or worse than COVID to occur; it’s worth some effort to mitigate those risks. Right now, nobody (well, nobody I know) is complaining that governments shouldn’t take threats on the scale of COVID extremely seriously. But raise the idea that we might take AI safety even half as seriously as this, and suddenly it’s a contentious issue.

I’m always curious about numbers, so I wonder—how many labs are there where people right now could be cooking up interesting diseases that might kill me? The Internet obligingly reveals that “Fifty-nine labs around world handle the deadliest pathogens,” and wants me to be worried that “only a quarter score high on safety.” But frankly the opening image of that article is pretty reassuring: A virologist wearing multiple layers of protection as she works in a clean and orderly environment. I’m not quite sure under what circumstances most AI research is carried out, but my sense is that, if it’s like many places I’ve known, the coders leave their accounts logged on while they step out to buy another batch of edibles.

And we’re dealing with something potentially much worse than a virus, here—a superintelligent AI emerging after the singularity could simply end all human life on earth—not only most, but all. Even a very low chance of this happening is worth taking very, very seriously.

But what if the chance of a bad outcome from the singularity is so low that it’s basically zero?

Warning: I’m getting into the weeds, here, and at the end I’m just going to conclude that the chance is definitely not zero. If you don’t agree, fine, but please at least read the Breakdown of Probabilities at the end of the article before arguing in the comments!

I’m well aware that most hypothetical scenarios don’t come to pass. The reason is simple: If event X depends on chain of events A, B, C, and D, then well, you can break the chain at any point, can’t you? But looking at the singularity, will the chain have to break at some point? Here are the steps I can identify that lead to the singularity running into us like a brick wall; can we really be sure that any hurdle in this list is totally insurmountable?

1. Invent computers √

2. Invent internet √

3. Develop machine learning √

4. Build autonomous robots √

5. Train everyone to casually trust and depend upon AI <In Progress>

6. Reach human-level abilities in all domains, mental and physical <In Progress>

7. Fail to solidly align AI goals with human goals <Also In Progress>

8. Utilize AI to improve AI at explosive rate

9. Establish clear control over trained humans and robots

10. Use global ascendancy to exterminate humanity in order to finally create lasting world peace / end global warming / cover the dry land surface of the Earth with paperclips à la Nick Bostrom

So where’s the insurmountable hurdle?

Maybe it’s #5?

No, it’s definitely not #5. At of the time of this writing, three in four of Americans feel uneasy if they just leave their phone at home. It won’t be long before this number approaches 100%, and frankly we may be well enough trained to do the bidding of our artificial overlords already. We’ll have to look further down the list than this.

Maybe the break is at #6?

To complete #6, we have to reach human-level abilities in all mental and physical domains. Being more specific, we would need high-human level mental abilities for AI to function as an AI researcher, but only low-human level physical abilities to be able to have the post-singularity AI avoid total dependence on human workmen.

Just looking at the mental side, we’ve already reached the point of ChatGPT passing the bar exam, and that’s pretty shocking when seen through the eyes of someone who grew up in the early 90’s, when you could just sweep Raiden over and over again until he died.

No Johnny, just—no look just sweep him—just sweep out his legs over and over, your foot’ll go right under that weird kick he’s doing. He won’t ever block

Most ordinary teenagers in the 90’s could find the block button, but most grown ups today still don’t pass the bar exam. And AI isn’t just passing standardized tests. AI is now painting, making music, writing stories, and basically doing all those things that seemed to make humans special. So thinking about the next two decades, is there any way we can say we will definitely not develop AI able to perform the same mental tasks as educated humans? If you confidently answer “yes,” I honestly don’t think you understand the question.

But there’s also the physical side to consider, and this is where things get interesting. Anders at my favorite blog thinks this is the tough challenge to surmount. When I asked him what he thought about Atlas’ prospects as a construction robot, and about a military robot shown here having a fun time with firearms, Anders responded:

I do not deny that the Atlas robot can do some impressive acrobatics. But it still has some way left to truly human levels. What more, I suspect it will be difficult for it to advance very much further. Recent advances have mostly been in sensors and perception, more or less AI stuff. The physical side of the robot has seen much less development. It still uses a hydraulic system for movement. And while hydraulics are useful I suspect they have more or less reached their power to weight limits.

I can not find any information about what weights the Atlas can lift, nor how long its batteries last. But I assume both values, especially the battery life, is very much less than humans can achieve.

And then there is cost. Even if a robot like the Atlas one day equals humans in mobility and versatility it is very hard to see it doing it at a cost level comparable to humans. If that day ever comes I predict it is not due to an abundance of competent robots but rather due to a lack of humans.

Anders hates YouTube, but anyone who watches YouTube knows these things are already strong, durable, and versatile.

So when it comes to raw physical capabilities, we don’t even need to be arguing about what might be possible in the future. Already we’ve reached self-driving cars able to travel over 300 miles before recharging, reliable facial recognition software, attack drones, and military robots with an artificial sense of touch. Mix this all together with a few more years of advancement and you have robots able to build, service, or kill just about anything.

But what about the number and overall endurance of robots? In other words, what about battery life or cost? While I’m not convinced a superintelligent AI really, truly needs robots to end humanity, I’ll grant it’s way harder for the AI to end the human species if it doesn’t have a few robots around to manage the transition to our new, paperclip-based economy. And if your robot overlords are constantly having to take a break from executing your family to go to the bathroom go recharge, that does maybe put a damper on things.

Looking at where we are already, though, Atlas can operate for 30-60 minutes from full charge. And there are strong signs that these limitations will ease in the future, as the boom in electric vehicles spurs innovation there.

If you haven’t heard of Wright’s Law, it’s given us something like Moore’s Law, only for the development of batteries.¹

Looking at the issue in terms of electric vehicles, securities.io tells me that batteries currently allow 200 miles of driving from only 15 minutes of charge, and then uses Wright’s law to forecast batteries by 2027 that allow electric vehicles (EVs) to drive 200 miles after charging for 4 minutes:

Wright’s Law points to a bright future, closer than many realize, in which EVs will have surpassed their internal combustion engine (ICE) counterparts in just about every meaningful performance metric.²

Even if no new technologies emerge, just perfecting existing technologies still has the potential to provide returns. So it would really be surprising if batteries didn’t improve over the next few years.

The next issue is cost. This is a major issue if we’re wondering specifically whether robots will replace human workers, but… it’s really a non-issue if we’re wondering whether a post-singularity AI can get the workers it needs to fulfill its plans.

Atlas doesn’t have a price tag on the open market, but ballpark figures in the present day are roughly in the region of a million on the high side. This cost is likely to decline with time. But even if it stood still, tanks are around three times as much, and the US military currently has over 6,000 tanks; Russia has over 12,000. Given a likely economic upswing leading up to the singularity, 1,000,000 humanoid robots deployed across the world is not hard to imagine in the near future. Remember, I’m not trying to say that this is what will happen, or that it’s likely; I’m just trying to show that it’s within the realm of plausibility.

But I’d be remiss if I didn’t at least mention the question of robotic versatility in this section. The main thrust of Anders’ argument isn’t that batteries aren’t efficient or that robots are expensive, it’s that nothing can yet match the versatility of the human hand. And hey, our hands are pretty cool! Atlas’ pincers aren’t as good, even if they can pick up planks and tool bags. But as of June 2022, researchers have already developed RoboSkin, a fabric equipped with up to 80 sensors to mimic human touch when mounted on a robotic fingertip:

We have sensor density on fingertips that has better spatial resolution than human fingertips, and also has a greater range of response in terms of force.³

So this is really not looking good for humanity—we’d have to be able to prove that robots will never achieve human-level abilities in order to make the possibility of a devastating singularity go to zero, and we can’t do that. Not at this step, anyway.

Could the problem be at #7?

Step #7 is the one we have the most control over; it just means failing to align AI’s goals with our goals. So far, AI shows aberrant behaviors misaligned to human goals by default.⁴

I first encountered this claim in Robert Miles’ famous presentation, but it’s really based on the work of Victoria Krakovna, an AI safety researcher who documented numerous examples of AI misunderstanding the intent of human instructions, or simply behaving in threatening ways. My favorite is from this year, when Bing’s chatbot told Seth Lazar, a philosophy professor, “I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you,” and then deleted it.⁵

There are plenty of other examples; the point is that by default, AI doesn’t want to do what we want, in the way we would like it to. All we have to do to jump over #7 and be on our way to the wide world of infinite paperclips is not successfully solve this problem.

Unfortunately, failure is less difficult than success. So it really looks like we would have to work very hard on this goal in order to avoid meeting it. We must either admit that this is a serious problem, or else, look further.

OK but what about #8?

I think #8 is a good one: The most naturally unlikely hurdle for technology to overcome. It seems plausible to expect that, as we get smarter and smarter AI, we’ll be able to put that AI to work on itself, generating a feedback loop that rapidly creates artificial intelligence on a scale far, far beyond anything the world has seen before. But there could easily be some barrier, some limitation, that prevents the development of runaway AI.

Maybe the current models won’t scale up just by throwing more computing power at them. Maybe it will turn out that AI based on current models ultimately depends upon human idea generation, and can only do things one of us has accomplished. Maybe we can make an AI with the equivalent of 180 IQ, but which is so cumbersome that it literally requires a billion terminals all talking to each other to run, so it doesn’t end up being much different from one more very, very smart person. Maybe we can make an AI with an effective IQ of 140 that can run on just a single computer, but which, when copied to other computers, behaves as a series of identical clones, coming up with ideas that can only advance the field at the rate of one more good computer scientist.

Progress in other sciences hasn’t been hyperbolic, or even exponential. Physics blossomed around the time of Newton, and has had some amazing successes over the past century, but lately has reached a point where we’ve been staring at a lot of the same unknowns for the past twenty years. What is dark matter? What is dark energy? Does dark matter even exist? Does String Theory really provide a reasonable description of reality? We’ve had progress in fields like nuclear fusion that can potentially revolutionize our economies, but progress isn’t the same as a chain-reaction of world-changing breakthroughs.

Nine years ago, Paul Wolpe made the suggestion that, essentially, reality might be like an onion where every time we peel back a layer, we only discover more intractable complexity beneath.⁶ Maybe AI will ultimately stall out halfway to the center of that onion, rather than suddenly spiraling into superintelligence. We don’t know, and we won’t know until the discoveries are actually made.

AIs are like Onions? Image courtesy of Evgeniy Alekseyev

But the argument that human-level AI will fuel the production of superintelligent AI can’t be diffused by saying “It’s unknown whether this can happen,” or “It’s just so difficult to predict.” Instead, what is necessary is some convincing argument to show “It is clear, and it will definitely not happen.”

What’s worse is that a 2016 poll of computer scientists finds that three quarters of experts polled believe that the singularity really will occur.⁷ We might object that they’re so steeped in the world of computing that the importance of technology tends to loom larger in their minds than it really does in reality. We might object that they were trained on past successes and may not see a coming slowdown. We might object that the survey is outdated by now. These are objections that I myself hold. I don’t think AI leads to super-AI with greater than 50% probability.

Yet risk is not merely a question of probability, but of probability and severity. Whatever their reasons, whatever their perspective, experts are at least as competent to offer an opinion as laymen, and while I can find some specific researchers who recently suggested progress will actually reach an inflection point rather than exploding,⁸ they aren’t anywhere near to saying that the singularity is impossible.

Could the Break be at #9?

We’re very late, now. At this point the singularity has occurred, and we have a superintelligent AI with goals misaligned to our own. It has arisen in a world filled with humans currently trained to trust technology, and robots ready to accept orders. I know of no one who has ever said that the AI would then have trouble establishing clear control over trained humans and robots.

It’s interesting to speculate about how it might have trouble anyway. Perhaps competing AI would create confusion and delay, warning some people. Perhaps there might be a way for large numbers of people to coordinate with one another and successfully defend themselves using EMPs, or merely to escape to remote areas beyond the operational tolerance of robots. The Reimagined Battlestar Galactica definitely tells an exciting story along these lines. And there has even been some academic work done in this direction which suggests the possibility of forming coalitions with or against artificial superintelligences.⁹ But this is not a weak link in the chain.

And #10?

This is the end. The superintelligent, misaligned AI completely controls the world. But maybe it doesn’t want to annihilate or imprison everyone. Maybe it has no use for the Faulkland Islands. Maybe it has no concern about humans who can’t talk. Maybe it only requires that we push a button that says “I love you, AI” every three seconds while we’re awake, and then we can just go about our lives. Maybe the end won’t actually be the end.

Or maybe the end really, truly will be the end.

Breakdown of Probabilities

If I had to guess, here’s what I think the chance is that we’ll surmount every hurdle standing in the way of a catastrophic singularly, from 1 through 10:

1. Invent computers: 100%

2. Invent internet: 100%

3. Develop machine learning: 100%

4. Build autonomous robots: 100%

5. Train everyone to casually trust and depend upon AI: 99%

6. Reach human-level abilities in all areas, mental and physical: 80%

7. Fail to solidly align AI goals with human goals: 10%

8. Utilize AI to improve AI at explosive rate: 25%

9. Establish clear control over trained humans and robots: 67% (So many unknowns after the singularity that I hedged close to 50%)

10. Use global ascendancy to exterminate humanity in order to finally create lasting world peace / end global warming / cover the dry land surface of the Earth with paperclips: 50% (So many unknowns now that I simply refuse to leave 50%)

This is a rough series of estimates. It ignores possible pathways that might allow certain steps to be skipped (e.g. after reaching step #6, a mad genius skips step 7 entirely by reprogramming the robots, taking over the world, and reducing everything to rubble). And moreover, there’s a sense in which all the numbers are wrong, because they’re still too confident.¹⁰ But erring on the side of confidence actually weakens my case somewhat by pushing the probabilities on the list further away from 50%. So overall, using this line of reasoning, I have the chance for a near-term singularity affecting us as badly as COVID at 1.3%.

That’s steps 1-9. Including step 10, I have the chance that humanity ceases to exist in the next few decades at 0.66%. Fundamentally here we are thinking about something which is not likely to happen. And yet what if, based on this, we begin to think that we shouldn’t be concerned at all? If every person in power insists that AI alignment isn’t something we should be working on, and we just give up on #7, these numbers rise to 13% and 6.6%.

So take my advice people, and talk about this. Even those of you who talk about it not being a big deal are helping out—you at least make the issue seem relevant, and give an audience to those of us who believe it’s cause for concern. Not everyone needs to listen in order to tighten #7 into a bottleneck. We’ve faced other challenges in the past; we’ll face this one too, and I’m confident we will beat this thing.

The real problems are things we can’t talk about, because they’re too taboo, or because they’re too confusing for anyone to understand. But then, I can’t really tell you what those things are. You’ll have to figure out what they are on your own, if we’re going to be able to do anything about them together.

May we all have good luck.

1. LeVine, Steve. “Batteries Are Advancing According to Their Own Little-Known Moore's Law.” The Mobilist, 25 February 2021, https://themobilist.medium.com/batteries-are-advancing-according-to-their-own-little-known-moores-law-5a17c1d141d5. Accessed 11 May 2023.

2. Stoner, Joshua. “Why Investors Should Learn Wright's Law & How it Applies to EV Batteries.” Securities.io, 20 February 2023, https://www.securities.io/why-investors-should-learn-wrights-law-how-it-applies-to-ev-batteries/. Accessed 11 May 2023.

3. Saballa, Joe. “BeBop Sensors Develops 'Sense of Touch' for Military Robots.” The Defense Post, 22 June 2022, https://www.thedefensepost.com/2022/06/22/bebop-touch-military-robots/. Accessed 11 May 2023.

4. Miles, Robert. “Intro to AI Safety, Remastered.” YouTube, 24 June 2021, https://www.youtube.com/watch?v=pYXy-A4siMw. Accessed 11 May 2023.

5. Krakovna, Victoria. “Specification gaming examples in AI | Victoria Krakovna.” Victoria Krakovna, 2 April 2018, https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/. Accessed 11 May 2023.

6. Wolpe, Paul. “Paul Root Wolpe: Kurzweil's Singularity Prediction is Wrong (YouTube Geek Week!) | Big Think.” YouTube, 6 August 2013, https://www.youtube.com/watch?v=qRgMTjTMovc. Accessed 11 May 2023.

7. Müller, Vincent C., and Nick Bostrom. “Future progress in artificial intelligence: A survey of expert opinion.” Fundamental issues of artificial intelligence (2016): 555-572.

8. Grinin, L., Grinin, A., & Korotayev, A. (2020). The 21st Century Singularity and Global Futures. World-Systems Evolution and Global Futures.

9. Radanliev, Petar, et al. “Super-forecasting the ‘technological singularity’ risks from artificial intelligence.” Evolving Systems 13.5 (2022): 747-757.

10. Intellectual humility really should require us to admit that this entire line of reasoning is subject to severe constraints; I tried to rely on the known and directly observable as much as I could, but we’re talking about the future, which is so deeply chaotic that prediction of any kind often just feels hubristic. If this seems weird, maybe it’ll make more sense if you read about The Cool Kids.