Calibrating the AlI-Risk Message While uncannily prescient, the AlI-risk message from the original dissidents has a giant flaw—as does the version dominating current public discourse: Both considerably understate the magnitude of the problem as well as AI’s potential upside. The message, in other words, does not adequately convey the stakes of the game. Wiener primarily warned of the social risks—trisks stemming from careless integration of machine-generated decisions with governance processes and misuse (by humans) of such automated decision making. Likewise, the current “serious” debate about AI risks focuses mostly on things like technological unemployment or biases in machine learning. While such discussions can be valuable and address pressing short- term problems, they are also stunningly parochial. I’m reminded of Yudkowsky’s quip in a blog post: “[A]sking about the effect of machine superintelligence on the conventional human labor market is like asking how US—Chinese trade patterns would be affected by the Moon crashing into the Earth. There would indeed be effects, but you’d be missing the point.” In my view, the central point of the AI risk is that superintelligent AI is an environmental risk. Allow me to explain. In his “Parable of the Sentient Puddle,” Douglas Adams describes a puddle that wakes up in the morning and finds himself in a hole that fits him “staggeringly well.” From that observation, the puddle concludes that the world must have been made for him. Therefore, writes Adams, “the moment he disappears catches him rather by surprise.” To assume that AI risks are limited to adverse social developments is to make a similar mistake. The harsh reality is that the universe was not made for us; instead, we are fine- tuned by evolution to a very narrow range of environmental parameters. For instance, we need the atmosphere at ground level to be roughly at room temperature, at about 100 kPa pressure, and have a sufficient concentration of oxygen. A