**Readings:
Manning and Schutze, Section 2.1**

Intuitively, probability is a measure of certainty about a certain outcome. For instance, if we toss a coin, we expect it to end up heads half the time. When we roll a die with 6 numbers, we expect to get a 6 one time out of six. The probability of a coin coming up heads is .5 and the probability of a die coming up 4 is 1/6. Something that it certain has a probability 1, whereas something that is impossible has a probability 0.

There are two philosophical ways of viewing probability: In the frequentist view probability is a ratio between two sets of events. For instance, given the set of coin tosses, we can consider the subset of tosses that come up heads. Thus the probability of getting heads is

(# heads) / (# tosses).

Given a set of observations, we can compute probabilities.

In the subjective view probability is a degree of belief that a certain proposition is true (or event will occur). This might not be based on any prior observations - it’s an indication of strength of belief. We might believe the probability that the next coin toss will come up heads is .5 (even through we obviously have never seen the next coin toss before!). For instance, a scientist might assert that the probability of a large comet hitting the earth in the next 100 years is .001 (even though the scientist has no record of such a comet ever hitting the earth).

It took more than three centuries to develop a mathematical theory of probability that did a reasonable job of capturing peoples different views and intuitions. Modern probability theory started in the 1930's with work by Kolmogorov. There are more thorough treatments of the following concepts available on the web.

To develop a theory of probability we need to define the types of events that can occur and what it means to talk about the chance of some event occurring. Here we will be developing the model for discrete probability distributions, which are the most intuitive and useful for natural language applications.

We can think of probability theory as formalizing the notion of performing
experiments. Say we have an experiment that has a certain number of possible
outcomes. We can perform the experiment repeatedly and note what the outcomes
are. For instance, we might do an experiment involving tossing a coin, and the
possible outcomes would be heads of tails. This can be formalized by
introducing the notion of a **random variable**, which can range over a set of outcomes (called the **sample
space**). For example, we could represent
coin tossing by a random variable TOSS, which has as its sample space the set {*heads,
tails*}. Note that we are currently assuming that each toss of the
coin is independent of the tosses that occurred before. Later on we will look
at models where such independence assumptions do not hold.

We can now introduce the notion of a **discrete probability distribution**. For any random variable X with finite (or
countable) sample space {x_{1}, ..., x_{n}}, a discrete
probability distribution P is a function that satisfies three key properties:

(1) 0 <=
P(X = x_{i}) <= 1 for all i

(2) S_{i} P(X=x_{i}) = 1

(3) P(X in {y_{1}, .., y_{m}}) = S_{i} P(X=y_{i})

The first two properties simply constrain values to be between 0 and 1, where the total probability over the entire sample space equals 1. The third property, called the countable additivity property, is the fundamental property that makes the mathematics work. These three properties are called the Kolmogorov axioms.

Note we have not said exactly how to compute the probability distribution.
There are many possible functions that could work. For instance, we could
assume that each outcome is equally likely and have the **uniform distribution**. In a sample space of size N, each outcome would
have a probability of 1/N. A very useful distribution is called the **Bernoulli
distribution**, which is used for sample
spaces of size 2. There is one parameter to specify in this distribution, p,
and the probability of one outcome is p and the other is (1 - p). In natural
language processing, however, we generally don't know the distributions as a
mathematical function, but must estimate them from a set of observations. In
this case, the distributions are constructed by counting the number of times
each element of the sample space occur in a large series of experiments. We
will talk about estimation procedures at some length later in the course.
Today, however, lets consider a simple case where we have a finite set of
experiments that occurred and there will be no more. Specifically, consider a
particular horse Harry, who ran 100 races in his career (these are the experiments)
and has now passed away. We have developed a time machine that can take us back
to a random race where we can bet. There is no way to identify the particular
date of the race, however, so we never know which particular race will be “run”
today.

The results of the race can be captured by a random variable R, with sample
space {*win,lose}*. Say Harry won 20 of
the races, then we could say

P(R=win) =
20/100 = .2

P(R=lose) = 80/100 = .8

Note that these values satisfy the requirements of a probability function. If this is all we know, when we go to bet, we should bet on the assumption that there is only a 20% chance that Harry will win.

Of course, to represent richer situations, we will need many different
random variables. For instance, in modeling horse races we might care what the
weather was like at each race (with a random variable W with values rain &
shine). W is a probability distribution in its own right. But of more interest
to us is how the two interact. To model this we talk of a **joint probability
distribution**, which is a new probability
distribution that combines the two random variables R and W. This joint
probability would have four values, corresponding to

<win, rain>, <lose, rain>, <win, shine>, <lose, shine>

Say that it rained in 30 of Harry’s races, and he won 15 of them. With this information and the above information about his overall wins and losses, we can figure out the following joint probabilities:

P(R=win, W=rain)
= 15/100 = .15

P(R=win, W=shine) = 5/100 = .05

P(R=lose, W=rain) = 15/100 = .15

P(R=lose, W=shine) = 65/100 = .65

Note that the joint probability distribution obeys the three conditions for a probability distribution.

If you are given a joint distribution, then you can derive the probability
distributions for each single random variable (now called the **marginal
probability**) by the following formulas,
which should fit your intuition.

P(R=win) =
P(R=win, W in {rain, shine}) = P(R=win, W=rain) + P(R=win, W=shine)

P(R=lose) = P(R=lose, W in {rain, shine}) = P(R=lose, W=rain) + P(R=lose,
W=shine).

If we are betting on Harry, we are more interested in how we expect him to do based on certain conditions that we know to be the case. For instance, if it’s raining, what is the probability that Harry will win? Intuitively, it 15/30, or .5 (a big change from .2, his overall chance of winning). Conditional probability is a theory of how to model these intuitions.

Technically, the **conditional probability** is written as P(A | B), and defined as follows:

P(A | B) = P(A, B) / P(B)

Note that P(R=win |
W=rain) = P(R=win, W=rain)/P(W=rain) = 15/30 = .5 as we expect. Note also that
when we write an equation just mentioning the random variables, this is really
an abbreviation for a universal quantification over all the values of the
random variables. Thus, if random variable A has values a_{i}, and B
has values b_{j}, then saying P(A | B) = P(A, B) / P(B) really means

P(A=a_{i}
| B=b_{j}) = P(A=a_{i}, B=b_{j})/P(B=b_{j}) for
all i, j

Almost all our
knowledge about the world is in the form of conditional probabilities, i.e., it
is relative to a certain context. If the above examples were more realistic,
they would all be conditional on as many other variables as we use to represent
the world. For instance, we might represent what Harry had for lunch, or what
Harry's current health is, or how Harry did in his last race. One the key
issues in formulating any problem is selecting what random variables to use,
and when we formulate knowledge, knowing what the dependencies are. For
example, say we also represent when we got up in the morning, and how we got to
the racetrack. Except for a few people who believe in lucky omens, most of us
believe that when got up and how we got to the racetrack doesn't affect how
Harry performs in the race. This is the notion of **independence**. Two random variables are independent if the
probability distribution of one isn't affected by the values of the other. For
example, using R as the random variable for Harry winning or not, and G the
random variable that states whether we drove or walked to the racetrack, we
believe

P(R | G) = P(R)

i.e., the probability of Harry winning is independent of how we got to the racetrack. Making independence assumptions is critical in building probabilistic models for it allows us to ignore certain information that is not relevant.

An equivalent way to define independence relates the joint distribution to the marginal distributions. In particular, random variables X and Y are independent if

P(X, Y) = P(X) * P(Y)

In other words, the probability of X=a and Y=b together is simply the product of the probabilities of X=a and Y=b. Either one of these definitions of independence can be used to derive the other, so they are formally equivalent.

Clearly in our racing example, R and W above are not independent, as it is much more likely for Harry to win if its is raining. But lets introduce another random variable A which captures whether I attend the race (track) or not (home). Say I go to the track 60 times, and Harry wins 12 races. This means that the probability that Harry wins if I attend the race is 12/60 = .2. The same as if I don’t attend, and the same probability over all races. thus we can prove that the random variable R is independent of the random variable A

P(R=win |
A=track) = 12/60 = .2 = P(win)

P(R=win | A=home) = 8/40 = .2 = P(win)

P(R=lose | A=track) = 48/60 = .8 = P(lose)

P(R=lose | A=home) = 32/40 = .8 = P(lose)

i.e., we have shown that

P(R | A) = P(R).

Note that just because A and R are independent doesn’t mean that going to the races doesn’t effect how you bet! It might combine with other random variables the weather. For instance, it might be that I went to the track twelve times when it rained, and Harry won every race! This would indicate that Harry was certain to win when I attended and it was raining. Now we might not believe this correspondence, but computing from the data, it is true. Thus, the random variable R is independent of A, but not independent of A and W.

Since conditional probabilities are so important, we will spent some time developing some simple rules for manipulating them.

One simple theorem
that is very useful allows us to compute certain conditional probabilities when
we know other ones. This is called **Bayes Rule** and is written as

P(A | B) = (P(B | A) * P(A)) / P(B)

As an example, say we don’t know all the information about Harry’s races, but we do know the probability that he wins any races (.2), the probability that he wins when its raining (.5), and the probability that it rains (.3). We can calculate the probability that it rained on a day when Harry won a race:

P(rain | win) =
(P(win | rain) * P(rain)) / P(win)

= (.5 * .3) / .2

= .75

Looking back at the
full joint distribution, we can see that this is the right result. In
particular, if we fill in the counts for each event in a table as follows

Joint Distribution count |
R=win |
R=lose |
Marginal counts for W |

W=rain |
15 |
15 |
30 |

W=shine |
5 |
65 |
70 |

Marginal counts for R |
20 |
80 |
100 |

we see that we can get P(rain | win) = .75 by dividing 15 (the count for <win, rain>) by 20 (the count for the marginal <win>). In situations where we can fill in a table like this one, we can compute the conditional, marginal and joint probabilities directly. But in most cases, the number of random variables and/or the number of values for these variables makes building such a table impossible. Bayes rule is used extensively in a wide range of applications to compute the probabilities we want to have from probabilities that we can obtain more easily.

Another very useful
rule, which computes joint probabilities from conditional probabilities, is
called the **chain rule **(Shown for a set
of random variables A to Z):

P(A ... Z) = P(A | B ...Z) * P(B | C ... Z) ... P(Y | Z) * P(Z)

To see why this theorem holds, consider the three variable case, which would claim

P(A, B, C) = P(A | B, C) * P(B | C) * P(C)

If we expand out the conditional probabilities with their definitions, we get

`P(A, B, C) = P(A, B, C) * P(B, C) *
P(C)`

When written this
way, we see that each terms numerator cancels the previous terms denominator,
leaving us with a simple expression that P(A, B, C) equals itself. The Chain
rule we be important when we start to try to estimate the probability
distributions of data.

For typical
applications in natural language processing, we are interested in sequences of
the events, such as the parts of speech of a series of words, or the sequence
of words that best accounts for a series of acoustic events. To model such
phenomena, we use sequences of random variables, with an index indicating the
time. For instance, if we wanted to model three tosses of a coin in a row, we
would have three random variables T_{1} T_{2} T_{3}.
When the probability distribution for T_{i} does not change over time,
it is called a **Stationary Stochastic Process**. These are the very useful for many applications. For instance, the coin
tossing example can be modeled as a stationary stochastic process because we
know the probability distributions of each t_{i} is independent of all
the others, namely P(T_{i} = h) = .5, for all i. Thus, it is easy to
compute the probability of a sequence, say of H H H. We simply multiply the
individual probabilities together:

P(H_{1}=h, H_{2}=h, H_{3}=h)
= P(H_{1}=h) * P(H_{2}=h) * P(H_{3}=h) = .5^{3}
= .125

In general, however, the probability distribution of the i’th event might depend on many of the events that precede it.

There is a very useful class of
Stochastic Processes that falls between these complete independence case and
the last example. In this class, the probability distribution for U_{i}
depends only on what happened at one previous case, i.e., on U_{i-1}.
Such models are have the **Markov**
property, and they will be one of the main tools for a host of NLP
applications. Markov models are useful because they are powerful enough to
often produce reasonable models of behavior, but simple enough to allow for
efficient algorithms for finding probabilities of sequences. While you might
not think so at first, Markov models are also stationary processes as the
probability distribution at each time step can be modeled by the conditional
probability distribution P(U_{i} | U_{i-1}).

Note we can define a hierarchy of Markov models where the current distribution depends only on k previous events. As k increases, however, the computational burden of computing over them increases correspondingly.

Not all stochastic processes are
stationary, there’s a traditional among mathematicians to use examples of
taking balls from urns. Let’s say we have an urn containing one red ball and
one blue. We choose a ball from the urn, and then we put it back with another
one of the same color. We model this process with a series of random variables
U_{i}, but they are not independent of each other. To see this,
consider the first two turns. For the first selection, clearly P(U_{1}=red)
= P(U_{2}=blue)=.5. But now consider the probability distribution for U_{2}.
If we picked a red and step one, we now have a probability of 2/3 of picking
red at step two. Similarly, if we picked a blue ball at step one, there is only
a 1/3 probability of selecting red at step two. In other words,

P(U_{2}=red | P(U_{1}=red) = 2/3

P(U_{2}=red | P(U_{1}=blue) = 1/3

The probabilities for P(U_{2}=blue)
are easily derived from the above. For the third drawing, the probability that
P(U_{3}=red) depends on what happened in the previous two tosses, and
in general, for the n’th turn, the probability depends on the n-1 previous
turns. Clearly, it is non trivial to compute the probability of a sequence such
as red, red, blue, red.

To use probability theory for natural language applications, we need to know what the probability distributions are for language. Unfortunately, we don't know what these distributions are. Rather, we have to estimate the distributions from looking at data, which we will discuss next lecture.

The other big issue, however, is what form of probability distribution works best for particular applications. In general, we will always be trying to balance a tradeoff between the accuracy to which a probability distribution can model the phenomena versus the amount of data we would require to adequately estimate it. Today, let us consider one simple example. Say we want to build a program using probability distributions that identifies the part of speech for words in a corpus - a part-of-speech tagger.

The simplest
probability model for this problem would introduce one random variable C with
its set of outcomes being the part of speech tags. In this case, we assume that
the probability distribution, C_{i}, is simply equal to C, i.e.,

P(C_{i}) = P(C)

Given this model, a the best tagger we could build would simply pick the most likely tag, namely a noun, for each word, and we could expect to get about at 33% accuracy.

We can do much
better by using a conditional probability model, where the probability of the
tag is conditional on what the word is. If we let W_{i} be a series of
random variables ranging over all the words, the we need to estimate the
distribution for P(C_{i} | W_{i}). Again, we simplify the model by assuming a these
probabilities are independent of their position in the text, i.e., there are
two random variables C and W such that

P(C_{i} | W_{i}) = P(C
| W) for all i

Given this model, the tagger would pick
the most likely part of speech for each word (e.g., *eat* is most likely a V, *hat* is most likely a N, etc). Surprisingly, such models
can obtain about a 90% accuracy in tagging tests. As a result, this model
serves as a good **baseline model**, one which we can compare new
models to.

An even better model can be obtained with a model that depends not only on
the current word, but also the tag of the preceding word, i.e., P(C_{i}
| W_{i}, C_{i-1}). These are often called **bigram** models. For part of speech tagging, approaches
based on this model can obtain about 95% accuracy for English.