whereas the iron-eating bactertum Thiobacillus ferrooxidans 1s thrilled. Who’s to say the bacterium is wrong? The fact that a machine has been given a fixed objective by humans doesn’t mean that it will automatically recognize the importance to humans of things that aren’t part of the objective. Maximizing the objective may well cause problems for humans, but, by definition, the machine will not recognize those problems as problematic. e Intelligence is multidimensional, “so ‘smarter than humans’ is a meaningless concept.”® It is a staple of modern psychology that IQ doesn’t do justice to the full range of cognitive skills that humans possess to varying degrees. IQ is indeed a crude measure of human intelligence, but it is utterly meaningless for current AI systems, because their capabilities across different areas are uncorrelated. How do we compare the IQ of Google’s search engine, which cannot play chess, with that of DeepBlue, which cannot answer search queries? None of this supports the argument that because intelligence is multifaceted, we can ignore the risk from superintelligent machines. If “smarter than humans” is a meaningless concept, then “smarter than gorillas” is also meaningless, and gorillas therefore have nothing to fear from humans; clearly, that argument doesn’t hold water. Not only is it logically possible for one entity to be more capable than another across all the relevant dimensions of intelligence, it is also possible for one species to represent an existential threat to another even if the former lacks an appreciation for music and literature. Solutions Can we tackle Wiener’s warning head-on? Can we design AI systems whose purposes don’t conflict with ours, so that we’re sure to be happy with how they behave? On the face of it, this seems hopeless, because it will doubtless prove infeasible to write down our purposes correctly or imagine all the counterintuitive ways a superintelligent entity might fulfill them. If we treat superint