When it comes to real robots helping real people, the standard definition of AI fails us, for two fundamental reasons: First, optimizing the robot’s reward function in isolation is different from optimizing it when the robot acts around people, because people take actions too. We make decisions in service of our own interests, and these decisions dictate what actions we execute. Moreover, we reason about the robot—that is, we respond to what we think it’s doing or will do and what we think its capabilities are. Whatever actions the robot decides on need to mesh well with ours. This is the coordination problem. Second, it is ultimately a human who determines what the robot’s reward function should be in the first place. And they are meant to incentivize robot behavior that matches what the end-user wants, what the designer wants, or what society as a whole wants. I believe that capable robots that go beyond very narrowly defined tasks will need to understand this to achieve compatibility with humans. This is the va/ue-alignment problem. The Coordination Problem: People are more than objects in the environment. When we design robots for a particular task, it’s tempting to abstract people away. A robotic personal assistant, for example, needs to know how to move to pick up objects, so we define that problem in isolation from the people for whom the robot is picking these objects up. Still, as the robot moves around, we don’t want it bumping into anything, and that includes people, so we might include the physical location of the person in the definition of the robot’s state. Same for cars: We don’t want them colliding with other cars, so we enable them to track the positions of those other cars and assume that they’ ll be moving consistently in the same direction in the future. A human being, in this sense, is no different to a robot from a ball rolling on a flat surface. The ball will behave in the next few seconds the same way it behaved in the past few; it keeps