KEsu Origins 7 February 24-26, 2017 PROJECT An Origins Project Scientific Workshop ARIZONA STATE UNIVERSITY Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes 5) Al, GOALS, AND INADVERTENT SIDE EFFECTS Runaway Resource Monopoly (Contributions from Shahar Avin, Sean O hEigeartaigh, Greg Cooper, and others) An important result from theoretical consideration of risks from advanced autonomous systems is the combination of two theses: orthogonality, that states that the goal an autonomous system is trying to achieve can be entirely unrelated to its optimization power; and the notion of instrumental goals, that for a large class of goals there is a set of convergent sub-goals (for an agent advanced enough to discover them) that include self- and goal-preservation, resource- and capacity-increase, etc. (e.g., as discussed in Bostrom, 2014). One suggestion for maintaining control over advanced systems that pose risks from the combination of the above considerations is to limit the system's ability to access increasing resources. To make this situation concrete, consider an installation of a reinforcement-learning task scheduler for a group of distributed data centres (e.g. Amazon Web Services). The goal of the algorithm is to minimize time-to-execution of the tasks sent to the system by users. As part of its general scheduling remit, it is also responsible for scheduling its own optimization sub-processes. The system has a clear incentive to control an increasing set of compute resources, both for increasing its optimization power and for achieving its main goal of reducing time-to-execution. Aware of these considerations, the engineers of the system put in place various hard-coded limits on the amount of resources the system can access, but these limits can be subverted through privilege escalation, masquerading as other tasks, manipulation of users, physical control, etc. POSSIBLE TRAJECTORY e Ateam within a large tech corporatio