its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us.”) Apparently, the original dissidents promulgating the AI-risk message were the AI pioneers themselves! Evolution’s Fatal Mistake There have been many arguments, some sophisticated and some less so, for why the Control Problem is real and not some science-fiction fantasy. Allow me to offer one that illustrates the magnitude of the problem: For the last hundred thousand years, the world (meaning the Earth, but the argument extends to the solar system and possibly even to the entire universe) has been in the human-brain regime. In this regime, the brains of Homo sapiens have been the most sophisticated future-shaping mechanisms (indeed, some have called them the most complicated objects in the universe). Initially, we didn’t use them for much beyond survival and tribal politics in a band of foragers, but now their effects are surpassing those of natural evolution. The planet has gone from producing forests to producing cities. As predicted by Turing, once we have superhuman AI (“the machine thinking method”), the human-brain regime will end. Look around you—you’ re witnessing the final decades of a hundred-thousand-year regime. This thought alone should give people some pause before they dismiss AI as just another tool. One of the world’s leading AI researchers recently confessed to me that he would be greatly relieved to learn that human-level AI was impossible for us to create. Of course, it might still take us a long time to develop human-level AI. But we have reason to suspect that this is not the case. After all, it didn’t take long, in relative terms, for evolution—the blind and clumsy optimization process—to create human-level intelligence once it had animals to work with. Or multicellular life, for that matter: Getting cells to stick together seems to have been much harder for evolution to accomplish than creating humans once there were mult