Origins February 24 — 26, 2017 PROJECT An Origins Project Scientific Workshop ARIZONA STATE UNIVERSITY Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes in the past with Chess and Go. Computer systems are initially inferior to their human counterparts but quickly come to dominate the space. The purpose of ACWs means they will be equipped with strategies for replication, persistence, and stealth, all attributes that will make it hard to defend against them were they to “go rogue.” Because of this concern, it is likely a good idea for designers to add built-in “kill switches”, lifetimes, or other safety limitations. Figuring out how to effectively limit the actions of an ACW while maintaining its usefulness is likely a very hard problem. Current practices of cyber defense (especially against advanced threats) continue to be heavily reliant on manual analysis, detection and risk mitigation. Unfortunately, human-driven analysis does not scale well with the increasing speed and data amounts traversing modern networks. There is a growing recognition that the future cyber defense should involve extensive use of autonomous agents that actively patrol the friendly network, and detect and react to hostile activities rapidly (faster than human reaction time), before the hostile malware can inflict major damage, or evade elimination, or destroy the friendly agent. This requires cyber defense agents with a significant degree of intelligence, autonomy, self-learning and adaptability. Autonomy, however, comes with difficult challenges of trust and control by humans. The scenario considers intelligent autonomous agents in both defensive and offensive cyber operations. Their autonomous reasoning and cyber actions for prevention, detection and active response to cyber threats will become critical enablers for both industry and military in protecting large networks. Cyber weapons (e.g., malware) rapidly grow in their sophistication, and in the