7.3 Toward a Formal Characterization of Real-World General Intelligence 137 7.3.2 Connecting Legg and Hutter’s Model of Intelligent Agents to the Real World A notable aspect of the Legg and Hutter formalism is the separation of the reward mechanism from the cognitive mechanisms of the agent. While commonplace in the reinforcement learning literature, this seems psychologically unrealistic in the context of biological intelligences and many types of machine intelligences. Not all human intelligent activity is specifically reward- seeking in nature; and even when it is, humans often pursue complexly constructed rewards, that are defined in terms of their own cognitions rather than separately given. Suppose a certain human’s goals are true love, or world peace, and the proving of interesting theorems — then these goals are defined by the human herself, and only she knows if she’s achieved them. An externally- provided reward signal doesn’t capture the nature of this kind of goal-seeking behavior, which characterizes much human goal-seeking activity (and will presumably characterize much of the goal-seeking activity of advanced engineered intelligences also) ... let alone human behavior that is spontaneous and unrelated to explicit goals, yet may still appear commonsensically intelligent. One could seek to bypass this complaint about the reward mechanisms via a sort of “neo- Freudian” argument, via ® associating the reward signal, not with the “external environment” as typically conceived, but rather with a portion of the intelligent agent’s brain that is separate from the cognitive component ® viewing complex goals like true love, world peace and proving interesting theorems as in- direct ways of achieving the agent’s “basic goals”, created within the agent’s memory via subgoaling mechanisms but it seems to us that a general formalization of intelligence should not rely on such strong assumptions about agents’ cognitive architectures. So below, after introducing the