concept of “Pareto-topia”: the idea that AI, if done right, can bring about a future in which everyone ’s lives are hugely improved, a future where there are no losers. A key realization here is that what chiefly prevents humanity from achieving its full potential might be our instinctive sense that we’re in a zero-sum game—a game in which players are supposed to eke out small wins at the expense of others. Such an instinct is seriously misguided and destructive in a “game” where everything is at stake and the payoff is literally astronomical. There are many more star systems in our galaxy alone than there are people on Earth. Hope As of this writing, I’m cautiously optimistic that the AI-risk message can save humanity from extinction, just as the Soviet-occupation message ended up liberating hundreds of millions of people. As of 2015, it had reached and converted 40 percent of AI researchers. It wouldn’t surprise me if a new survey now would show that the majority of AI researchers believe AI safety to be an important issue. I’m delighted to see the first technical Al-safety papers coming out of DeepMind, OpenAI, and Google Brain and the collaborative problem-solving spirit flourishing between the Al-safety research teams in these otherwise very competitive organizations. The world’s political and business elite are also slowly waking up: AI safety has been covered in reports and presentations by the Institute of Electrical and Electronics Engineers (IEEE), the World Economic Forum, and the Organization for Economic Cooperation and Development (OECD). Even the recent (July 2017) Chinese AI manifesto contained dedicated sections on “AI safety supervision” and “Develop[ing] laws, regulations, and ethical norms” and establishing “an AI security and evaluation system” to, among other things, “[e]nhance the awareness of risk.” I very much hope that a new generation of leaders who understand the AI Control Problem and AI as the ultimate environmental risk can rise ab