Contents xvii 12 The Engineering and Development of Ethics.............................. 205 1221 JGPOdUMCtIOM, «225 sa2ms cote: Case) EMDeE CG 8@EBE eEws ewEes eMews emte|s cases eas ZOO 12.2 Review of Current Thinking on the Risks of AGI .......................--.. 206 12.3 The Value of an Explicit Goal System. ......... 0.0... c ce eee ee 209 12.4 Ethical Synergy ........ 0... eee etter tence eee eee 21D 12.4.1 Stages of Development of Declarative Ethics ......................... 211 12.4.2 Stages of Development of Empathic Ethics .......................... 214 12.4.3 An Integrative Approach to Ethical Development..................... 215 12.4.4 Integrative Ethics and Integrative AGI ......................-....-+. 216 12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model ..............0000 0002 ccc ee eee eee 219 12.5.1 The Golden Rule and the Stages of Ethical Development ............. 222 12.5.2 The Need for Context-Sensitivity and Adaptiveness in Deploying Ethical Principles .......0..0.00. 000.0 ce eee eee 223 12.6 The Ethical Treatment of AGIs .............0 0000 cece eee eee es 226 12.6.1 Possible Consequences of Depriving AGIs of Freedom ................. 228 12.6.2 AGI Ethics as Boundaries Between Humans and AGIs Become Blurred . 229 12.7 Possible Benefits of Closely Linking AGIs to the Global Brain................ 280 12.7.1 The Importance of Fostering Deep, Consensus-Building Interactions Between People with Divergent Views............ 00.000: eevee eee eee QOL 12.8 Possible Benefits of Creating Societies of AGIS ......... 0.00. e cee eee ee 283 12.9 AGI Ethics As Related to Various Future Scenarios ........................ 284 12.9.1 Capped Intelligence Scenarios ...........0...0 0.0 cece eee eee eee 284 12.9.2 Superintelligent AI: Soft-Takeoff Scenarios ...................2.0..-.. 285 12.9.3 Superintelligent AI: Hard-Takeoff Scenarios ......................-.. 285 12.9.4 Global Brain Mindplex Scenarios .