John McCarthy, Making robots conscious of their mental states, Machine Intelligence 15, 1996, revised ca 2002 at http://www-formal.stanford.edu/jmc/consciousness.html [Comments in square brackets are my own thoughts.] McCARTHY, MAKING ROBOTS CONSCIOUS OF THEIR MENTAL STATES ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The primary importance of this paper is that it @ lists many specific introspective capabilities that robots could and should have; @ advocates that these capabilities should be based on explicit logical representations of knowledge about what is known and how knowledge is acquired, and on deductive and nonmonotonic reasoning with such intro- spective knowledge; and @ provides formal sketches illustrating the proposed kinds of representation, with appeal to "contexts" as a new kind of entity in the domain of discourse, "concepts" of individuals, properties, and functions, distinct from those individuals, properties, and functions, and an extension of the situation calculus to the mental realm. He briefly dismisses qualia (p.2), saying that robots won't need them -- what we care about is how robots function. But he says that consciousness in the sense of being able to form sentences about mental processes and contents is crucial for human-level intelligence (p.4-5). [I think that besides the ability to *form* sentences we need the ability to *use* them in general reasoning. McCarthy probably takes this for granted most of the time, but it is a point worth emphasizing. When he talks about thermostats and such, he seems not to have a reasoning requirement in mind.] As an example of the usefulness of introspection he mentions the problem of answering the question, "Is Clinton sitting, standing, or lying down right now?". Introspection tells us we can't answer this (perhaps because we find that we have no relevant knowledge). He comes back to this more than once. He also mentions the importance of knowing what we don't know. In general, he suggests that robots could be made much more thoroughly self-aware than humans, since they could inspect their entire code, and monitor their own thinking in detail (p.7-8). Their reasoning could improve based on taking account of facts about their own intellectual history (just on the principle that knowing more leads to more conclusions -- and better ones, if the reasoning is nonmonotonic). They could also use this to change their own thinking behavior, but McCarthy sets aside this topic (p.8). A limit on robot introspection is that the thinking can't get ahead of itself, i.e., predict what the robot will think, ahead of thinking it [cf. McDermott on free will]. Serious introspection (p.8-13) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here he introduces contexts, in the sense of entities with respect to which certain sentences are true. Contexts seem to form a partial ordering under a "subcontext" relation. So a sentence that is true in a context c is also true in the subcontexts of c, but the converse is not in general the case. Ist(c,p) (or c: p) says that proposition p is true in context c. Note that p needs to be a term, not a sentence, if this is to be FOL. E.g., in 'Sitting(Clinton)', 'Sitting' is a function (not a predicate), and 'Clinton' is a constant (representing the *concept* of Clinton, not the person himself). The main uses McCarthy seems to have in mind for contexts is to provide a "Here, Now, I" context (or just "Now, I"), for all the propositions that are true now for the robot, e.g., that its battery is low; and to allow hypothetical (assumption-based) reasoning (see p.10 and p.18). To say that the robot doesn't presently know whether Clinton is sitting, McCarthy writes C(Now,I): ~Know-whether(Sitting(Clinton)). [Here it's not clear to me how to interpret Now and I in a standard way, since the denotation of Now is presumably not fixed. Thus the context term C(Now,I) does not seem to be interpretable in a standard way, and so it's not clear what Ist(C(Now,I),p) would mean.] [Also McCarthy worries that proving that one can't infer something amounts to proving one's own consistency -- which is impossible by Goedel's 2nd theorem in any consistent, sufficiently general logic. But I think the flaw lies in equating "I can't infer" with "It is not deducible from my axioms". For example, I can't infer whether the first player in a game of chess can force a win; but in principle a positive or negative answer is deducible from the rules of chess, which I know! So being able to infer and deducibility just aren't the same.] Anyway, he enumerates some important kinds of self-knowledge and self-observation: - not knowing the value of something (e.g., someone's phone number); here he introduces upper-case versions of all ordinary constants to denote the corresponding concepts; this enables him to say that a person may know Mike's phone number, and not know Mary's phone number, even if the phone numbers are the same, because the knowledge is expressed as knowledge of the complex concept formed from the concepts of phone-number and of Mary and Mike, and those complex concepts are different for Mary and Mike. - observing the state of one's physical body and internal parameters; - observing that one knows nothing about a particular subject (rather than just lacking some particular fact); here he mentions "Clinton sitting" again; - reasoning about one's abilities [he doesn't say how; I think this requires planning abilities]; - keeping "a journal of physical and intellectual events" [episodic memory]; - observing one's own goal structure, not just obeying it; - observing one's intentions [my example: to answer the question, "If the phone rings in the next 10 minutes, will you hear it?" affirmatively, one needs to know that one intends to stay in the vicinity of the phone, and keep it turned on, among other things]; - observing one's mental actions in solving a problem or arriving at a belief, thus being able to report on it, and perhaps revise a belief in light of new evidence; - knowing what general constraints one is complying with in one's goal-directed reasoning (e.g., not harming people); - knowing everything one believed up until now; - knowing what one's current choices are for achieving one's goals, and knowing that one will do whatever action, among those choices, promises the best result (free will, again) ; he gives a formula for this; - knowledge of general facts about mental processes; - a desire to extend one's knowledge (curiosity); p. 13-14 he makes a point of distinguishing understanding of some skill from mere awareness that one has the skill (swimming and gymnastics examples). Formalized self-knowledge and mental events (p. 14-19) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here he introduces a "mental situation calculus". This is essentially the usual sit. calc., i.e., variable properties have a situation (state-of-the-world) argument, and actions map a given situation into a new one; however, many of these variable properties (fluents) are now knowledge properties. He uses Holds(p,s) to say that proposition p holds in situation s, where -- as in the case of Ist(c,p) -- p is a term representing a proposition, not a sentence. Thus Holds(Know(Not Know(p)),s) means that it is known (by me) in situation s that p is not known (by me). Here Not is just a 1-place function, which we are applying to the term Know(p). We might have written Not(Know(p)), but McCarthy skips the brackets. He also elaborates on the example of "knowing Mike's phone number" here, introducing an s-argument into the Holds assertion. Kinds of mental events ~~~~~~~~~~~~~~~~~~~~~~ McCarthy then enumerates kinds of mental events -- the sequential arrival of sentences in working memory (stream of consciousness), learning something, forgetting, deciding, assuming, inferring, and perceiving. He formulates a few axioms in the mental sit. calc., such as Holds(Know(p),Result(Learn(p),s), i.e., knowing p holds in the situation that is the result of the action of learning p. He is justly dubious that learning p can always be viewed as an action by the robot -- e.g., it may be the result of someone else's telling-action. So he reformulates it as Occurs(Learn(p),s) -> Holds(F Know(p),s), i.e., if learning p occurs in situation s, then Know(p) holds at some time in the future, in situation s. [This seems too weak, because nothing ensures that knowing p holds *right after* learning p. He tries to fix this with a Next(p,s) function (the next state in which p holds), but I'm not convinced this works -- Next(p,s) could still be a thousand time steps after s. For deciding, he writes an axiom that deciding to do something leads to intending to do it (which is a persistent state). For perceiving, he posits that if an event of seeing object o occurs, then the robot will know that it saw o. Finally he briefly mentions, as hard problems, inferring that one can't achieve a goal, and inferring that some bad event won't occur, no matter what I do. [The former seems approachable through planning; if I fail to find a plan to achieve a goal, I can't achieve it.] Logical paradoxes and inferring non-knowledge (p.19-27) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [These lengthy sections seem to me of limited interest, as they presuppose (to me) unrealistic notions of "(not) knowing" and "(not) being able to infer". Goedel's theorems are relevant if you think these notions involve proofs of consistency (because as mentioned such proofs are ruled out by Goedel's 2nd incompleteness theorem), but I don't.] Humans and robots (p. 28-31) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ He hypothesizes that the main difference between us and our ape ancestors is that we are better at introspection (in the sense of forming sentences about our mental states and actions, which McCarthy equates with consciousness; i.e., we are more conscious than they (cf. e.g., Arbib, MsDermott). He urges making robots explicitly conscious of contexts, which seem largely unconscious in us. [I'm not convinced of the utility of his notion of contexts.] He suggests that emotions in humans serve to "gate" the beliefs that enter into conscious working memory; e.g., hatred favors negative thoughts about the object of hate, and ways of attacking that object. He argues against this sort of "gating" for robots -- they should just be rational at all times, not succumbing to hatred or panic [cf. the various positions taken in Fellous & Arbib; however, McCarthy seems to be neglecting issues of computational efficiency]. He also doesn't want us to make robots likable or dislikable, because people will anthropomorphize, and "human society is complicated enough already". Remarks (p.32-35) ~~~~~~~~~~~~~~~~ Some of the more interesting ones are that - Turing already pointed to the potential usefulness of self-observation for self-improvement; - This is not a theory of human thought and consciousness, but of how we could implement robot thought and consciousness; - The "theater of consciousness" rejected by Dennett may be appropriate for robots [in the sense of having a single place where all the conscious sentences/thoughts are located]; - He likes Francis Crick's work, but thinks AI will get there first; - There's no need for a Freudian subconscious, with its own purposes; - Robots should hope, so that they'll learn from (non)fulfilment of their hopes; similarly for wishing (but not wishing that its goals were different); - Associative links are good for retrieval, but this needs to be followed up with reasoning; we probably need deduction, NMR, model-finding, 2nd-order logic, and case-based reasoning [same as reasoning by analogy??]