might alter its motion when carrying something heavy, to emphasize the difficulty it has in maneuvering heavy objects. The more that people know about the robot, the easier it is to coordinate with it. Achieving action compatibility will require robots to anticipate human actions, account for how those actions will influence their own, and enable people to anticipate robot actions. Research has ,ade a degree of progress in meeting these challenges, but we still have a long way to go. The Value Alignment Problem: People hold the key to the robot’s reward function. Progress on enabling robots to optimize reward puts more burden on us, the designers, to give them the right reward to optimize in the first place. The original thought was that for any task we wanted the robot to do, we could write down a reward function that incentivizes the right behavior. Unfortunately, what often happens is that we specify some reward function and the behavior that emerges out of optimizing it isn’t what we want. Intuitive reward functions, when combined with unusual instances of a task, can lead to unintuitive behavior. You reward an agent in a racing game with a score in the game, and in some cases it finds a loophole that it exploits to gain infinitely many points without actually winning the race. Stuart Russell and Peter Norvig give a beautiful example in their book Artificial Intelligence: A Modern Approach: rewarding a vacuuming robot for how much dust it sucks in results in the robot deciding to dump out dust so that it can suck it in again and get more reward. In general, humans have had a notoriously difficult time specifying exactly what they want, as exemplified by all those genie legends. An AI paradigm in which robots get some externally specified reward fails when that reward is not perfectly well thought out. It may incentivize the robot to behave in the wrong way and even resist our attempts to correct its behavior, as that would lead to a lower specified reward. A