February 24 – 26, 2017 An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes ARTIFICIAL INTELLIGENCE ADVERSE OUTCOMES & TEAMS 1) FINANCIAL MARKETS AI, Trading Systems, and Market Manipulation Chair: Michael Wellman Red Team: Miles Brundage, Randy Bryant, Gary Marchant, Jaan Tallinn Blue Team: Michael Littman, Frank Wilczek, Greg Cooper, Yan Shoshitaishvili 2) DEMOCRACY, INFORMATION, AND IDENTITY AI, Information, and Democracy Chair: Shahar Avin Red Team: Miles Brundage, Seán Ó hÉigeartaigh, Andrew Maynard, Eric Horvitz Blue Team: Gary Marchant, Gireeja Ranade, Michael Littman, Subbarao Kambhampati 3) WAR & PEACE AI, Military Systems, and Stability Chair: Bart Selman Red Team: Richard Mallah, Eric Horvitz, Michael Wellman, Frank Wilczek Blue Team: Vinh Nguyen, Kathleen Fisher, Lawrence Krauss, John Launchbury, Rachel Bronson 4) AI, CYBERSECURITY, AND AI ATTACK SURFACES AI Attacks on Computing Systems, Devices, Infrastructure Manipulation & Disruption of AI Systems Chair: Kathleen Fisher Red Team: Jeffrey Coleman, John Launchbury, Vinh Nguyen, Mauno Pihelgas Blue Team: Ashish Kapoor, Randy Bryant, Yan Shoshitaishvili, Ben Zorn 5) AI, GOALS, AND INADVERTENT SIDE EFFECTS Runaway Resource Monopoly Self-Improvement, Shift of Objectives Chair: Seán Ó hÉigeartaigh Red Team: Jaan Tallinn, Nate Soares, Jeff Coleman, Bart Selman Blue Team: Dario Amodei, Greg Cooper, Shahar Avin, Ben Zorn 6) DEEP LONG-TERM SOCIETAL INFLUENCES AI, Agency, and Disempowerment Chair: Gireeja Ranade Red Team: Richard Mallah, Andrew Maynard, Nate Soars, Mauno Pihelgas Blue Team: Subbarao Kambhampati, Frank Wilczek, Lawrence Krauss, Dario Amodei 1 1) FINANCIAL MARKETS AI, Trading Systems, and Market Manipulation (Incorporating contributions by Michael Wellman and others) February 24 – 26, 2017 An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes There has been advan