From cs240@cs.rochester.edu Tue Nov 4 18:00:13 2003 Date: Tue, 4 Nov 2003 17:58:27 -0500 From: cs240 To: tetreaul@vienna.cs.rochester.edu David Lu: Assignment #5 David Lu!!
cs240: Assignment #5
dl002m@mail.rochester.edu
10/17/2003

I Feel Good! I Knew That I Would!1

The way in which animals interact with the world around them is governed by their brains. Input signals are brought to the brain through various nerves, and then somehow they are processed, resulting in some sort of output, perhaps of the motor variety [this ignores reflexes, which do not get processed by the brain]. The inner process of evaluation is what concerns us though. What makes one result better than another, or one input better than another for that matter? The answer is rewards. Just like your Visa giving you extra sky-miles because you used your card, animals (including humans) get rewards for things that are judged beneficial by means of dopamine. In fact, it could even be said that getting sky-miles is a reward in the neurological sense of the word, because it is judged to have positive effects at some later date. Rewards tell us a lot about the brain and how it functions. However, there are other factors and we shall eventually realize there is more to the brain than just rewards.

The basic theory behind rewards is this. In order to survive, the brain needs to predict the future. Not in the science fiction-y sense, but in the sense that given certain conditions, other events will probably result. The way in which the brain can predict rewards is crucial to its functioning. A reward is defined as ^Óan operational concept for describing the positive value that a creature ascribes to an object, a behavioral act, or an internal physical state.^Ô2 What this often translates to is dopamine. In the brain, dopamine feels good, as evidenced by the effects of addictive drugs on the use of dopamine and self-stimulation tests. However, in order for the rewards to mean something they must be associated with the proper events that caused them, or at least seem to. For instance, if I see food, my body starts preparing to digest it since it knows that seeing food often results in eating food, which is a good feeling. However, at some point in my life, I had to make that connection, which is what prediction is all about.

Prediction links a reward to a conditioned stimulus. This link is observed through the output of dopamine neurons. A higher output translates to a better feeling and more sparse output is worse. The training for the prediction starts with just a reward. The reward triggers the release of dopamine into the brain. Okay. We now have a positive reinforcement. When the conditioned stimulus is added in, it changes the chemistry of the mix. Now the expected values are changed. Conditioning training repeatedly starts to associate the reward with something else, so now the dopamine release does not come when the reward is administered, but before it. Eventually, the dopamine is released immediately following the triggering event. Now the association is complete. When the conditioned stimulus occurs, the brain anticipates the coming reward (i.e. predicts), and thus releases the dopamine early.

On a computational level this works using expected value and error when compared to the real value. The value is computed using V(t) = E[yr(t) + y1r(t+1) + y2r(t+2) + ^Å] where V is the ^Ódiscounted sum of all rewards and 0<=y<=1.2 What this means is that for any time t, there is going to be some amount of reward in the future. It takes into account all future events, and assigns a weight to them so that the further they are in the future, the less they are worth. However, since this is impractical because it is quite hard to calculate everything that will happen in the future. Thus it is often rewritten recursively V(t) = E[r(t)+yV(t+1)). The end result is a calculation of error, which is used to adjust the pathways. This system of weights, errors and adjustments can easily be construed to refer to the plasticity of the connections between neurons.

The conclusions drawn from this are a bit startling. However, they are not always accurate. Taken to the extreme, the result is that our entire lives end up being drives to get dopamine. This in effect brings into the whole meaning of life^Åis our only purpose to get as much dopamine as we can? Also, if choices are driven by a quest to get the most dopamine, doesn^Òt the prediction eliminate all decision from our conscious life? On the surface, the reward and prediction scheme also seems to answer the question of nature vs. nurture. In truth, all this is only part true. It turns out there is more to the system.

We cannot accept that the only thing governing our bodies is this dopamine drive. To start, there seems to be some other planning that happens within humans that would be rather difficult to incorporate into this system. It is said that ^Órewards that arrive sooner [are] more important than rewards that arrive later.^Ô3 The equation tells us that this happens exponentially. Consider this thought experiment. I give you the choice of receiving five dollars now, or ten dollars in half an hour. External factors such as trust aside, the above equation would tell us that we would rather have the five dollars because the reward garnered by the ten would be so discounted by that point that it would be pointless to wait. Meanwhile, I know I would rather have the ten. You might say that the reward of having ten dollars is exponentially greater than that of five, but that seems unlikely. This shows that there are some higher planning mechanisms in the brain (or mind) that have yet to be accounted for.

There is also the sense that I make conscious decisions. If my body runs solely on dopamine, then how am I supposed to make choices? How, then, can I be held responsible for my actions? ^ÓI^Òm sorry I ran that stop sign officer, my dopamine made me do it. It^Òs not my fault.^Ô I can consciously lift my hand from the keyboard. I get no pleasure from this, other than the fact that I^Òm proving a point. I would be greatly surprised if there was a dopamine spike in my head just then. It therefore makes sense that one of two things is true: either a) my lifting my hand was caused by a higher non-computational entity (i.e. free will) or b) something in my brain allowed me to alter the weights set upon certain actions, thus sticking with the dopamine driven system, but while still maintaining some idea of free will.

If you believe this dopamine concept, then the question of nature vs. nurture seems to be answered. Of course it^Òs nurture, they would say, obviously every aspect of your personality if governed by the complex dopamine rewards, not your genes. Believers would say that the only thing that your genes give you the ^Ófour f drives^Ô (flight, fight, feeding and reproduction). The rest of your existence is driving towards those plights. This of course is absurd. Clearly there are other qualities that are instilled in us from our genes, behaviorally speaking. I don^Òt like eggs, for no logical reason. However, other people enjoy them and receive dopamine for eating them. Why shouldn^Òt I? Because clearly there are other factors affecting me. Also, there are qualities that don^Òt factor into the four f drives. Moral self worth, integrity, intelligence. Unless you say that these qualities^Ò purpose is to get you the fourth f, then their worth lies somewhere else. The appreciation of art, for example, most likely gives dopamine rewards as well, yet there is something higher that allows for it. The existence of inherited qualities and the higher satisfactions in life seem to point to a more advanced concept in the brain that transcends dopamine rewards.

There are also a couple of logistical issues I^Òd like to point out here. The equations described above require storage space to store the expected result and the error. I seriously doubt that there is a mechanism in the brain that computes and stores the information precisely how it was described. It could be true that that is the way it works, but the logistics of it don^Òt seem to map out. There is also the issue of not feeling good. Going back to the original experiment, after training is done, the dopamine will release immediately after the stimulus. However, if no reward is given at the expected time, then there is a lull in the dopamine. It is released at a slower rate. We are led to believe that the absence of good feeling is what we perceive as bad feeling. However, studies have shown that actual pain is registered not in dopamine releasing areas, but in the thalamus.4 This points again to a higher function that is not simple dopamine anticipation.

I do not try to refute the fact that dopamine, the reward system and the anticipation thereof do not exist. However, clearly there are some aspects to the brain that transcend the endless pursuit of dopamine. The result is something that could be said to be non-computational. The higher function allows for proper goal assessment, decision making and cohesive brain activity. To end with an example, you right now are under the influence of these factors. You are currently receiving dopamine since you are almost done reading this essay. You did a good job reading it too, and your satisfaction results in a dopamine burst, because good things follow your completion. However, you are also able to admire it^Òs wit and intellectual argument, which are appreciated on a different level. You also know that finishing this essay is helping your career along, and thus, your long term planning is falling into place. All in all, reading this essay made you feel good.

References:

  1. http://www.funky-stuff.com/jamesbrown/Lyrics/IFeelGood.htm
  2. Wolfram Schultz, Peter Dayan, P. Read Montague ^ÓA Neural Substrate of Prediction and Reward^Ô Science Volume 275 (March 14, 1997) 1593.
  3. Schultz, et al. ^ÓA Neural Substrate of Prediction and Reward^Ô 1595.
  4. Wikipedia.com


Please note an element of sarcasm in the last paragraph. I'm not really that egotistical.