KEsu Origins February 24 — 26, 2017 PROJECT An Origins Project Scientific Workshop ARIZONA STATE UNIVERSITY Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes the agent must rely on the data gathered from the sensors (there is no human in the loop to decide this), there can be unexpected situations where the agent would stop some human interaction with the system or interrupt maintenance activities, because it deemed that these actions could harm the system. For example, the system administrator stopping some services during system maintenance, or upgrading to a newer software version. e Replication to third-party systems and collateral damage — Building on the first problem of the agent not having the correct information. If the term friendly network gets misconfigured and the agents have the capability to self-transfer to new friendly hosts, it can happen that the agent would distribute to external networks, start defending it and take responsive actions on third party hosts. Such incidents would make the agents very difficult to halt. e Friendly fire — One agent might consider another agent as an adversary and start trying to eliminate/evade each other. e Silent compromise — If the adversary manages to get access or reverse engineer the agents (without the agent self-destructing), they could potentially trick or reconfigure the agents to turn on themselves. CYBER-OFFENSE Cybercrime is a growth industry, from stolen credit cards to ransomware. Very crudely, it's a two tier system, with a "spray and pray" approach at the low-skill end that targets millions of system in the hope some of them would be vulnerable (through technical or human failing); at the other end are tailor-made attacks that rely on slow progression of escalation and compromise, often requiring advanced technical skills for discovering zero-day vulnerabilities and intimate knowledge of the target. Advanced artificial intelligence may be used to automate some