ways we don’t understand and certainly can’t follow in real time, we face a problem: We don’t know what to tell the machine not to do. So many of the things we’d hope to teach it - be compassionate, fight for liberty, follow a moral code - far transcend what might be achieved by us in math. We haven’t after all, even solved the problem of how program ourselves reliably with these values. If Bostrom’s paper clip machine seems fantastic, it is easy enough to conjure other and more real dangers lingering at the edge of disappeared human control. Think of health care. To begin, you need to know about an important “game” from the world of research into how humans interact with each other known as the “The Ultimatum Problem.” It runs like this: I tell you that you can have a million dollars, but you have to split it with someone else. How you split it is up to you, but if your partner rejects the formula you propose, neither of you gets a cent. Offer to split the pot with a dollar to your pal and the rest to you. Insulting. But where to settle? You might expect that the smartest offer would be a 50/50 split, but humans are greedy. You want more and can probably get it; your partner does not want to end up with zero. Generally when scientists shake this cocktail of greed and fear they find an offer of $300,000 is nearly always accepted. However, there’s a surprising way to change the outcome: Match the human against a computer in the negotiation. A pal suggesting an 80/20 split to a friend will be rejected. Too greedy. But a computer? Somehow the impersonality, the beeping digital charmlessness of the machine lures biological players to compromise. An offer of $200,000 is usually happily accepted. It may be, scientists think, that our competitive instinct is muted when we interact with a machine. But researchers have also discovered they can manipulate the split other ways: Sad movies, war chants, hard rock - each bends the emotions of players and changes the result. Incre