226 12 The Engineering and Development of Ethics The implications of these ideas for ethical instruction are complex and won’t be fully elabo- rated here, but a few of them are compact and obvious: 1. The teacher(s) must be observed to follow their own ethical principles, in a variety of contexts that are meaningful to the AGI 2. The system of ethics must be relevant to the recipient’s life context, and embedded within their understanding of the world. 3. Ethical principles must be grounded in both theory-of-mind thought experiments (empha- sizing logical coherence), and in real life situations in which the ethical trainee is required to make a moral judgment and is rewarded or reproached by the teacher(s), including the imparting of explanatory augmentations to the teachings regarding the reason for the par- ticular decision on the part of the teacher. Finally, harking forward to the next section which emphasizes the importance of respecting the freedom of AGIs, we note that it is implicit in our approach to AGI ethics instruction that we consider the student, the AGI system, as an autonomous agent with its own “will” and its own capability to flexibly adapt to its environment and experience. We contend that the creation of ethical formations obeying the above imperatives is not antithetical to the possession of a high degree of autonomy on the part of AGI systems. On the contrary, to have any chance of succeeding, it requires fairly cognitively autonomous AGI systems. When we discuss the idea of ethical formulations that are unlikely to be undermined by the ongoing self-revision of an AGI mind, we are talking about those which are sufficiently believable that a volitional intelligence with the capacity to revise its knowledge (“change its mind”) will find the formulations sufficiently convincing that there will be little incentive to experiment with potentially disastrous ethical alternatives. The best hope of achieving this is via the human mentors and train