so we've been talking about information
how you measure information
and of course, you measure information in bits
how you can use information to label things
for instance, with bar codes
so you can use information
to label things
and then we talked about
probability and information
and if I have probabilities for events P_i
then it says that the amount of information
that's associated with this event occurring is
minus the sum over i
p_i log to the base 2
of p_i
this beautiful formula that was developed
by Maxwell, Boltzmann, and Gibbs
back in the middle of nineteenth century
to talk about the amount of entropy
and atoms or molecules
this is often called S
for entropy, as well
and then was rediscovered
by Claude Shannon
in the 1940's
to talk about information theory in the abstract
and the mathematical theory of communication
in fact, there is a funny story
about this that
Shannon, when he came up with this formula
minus sum over i p_i log to the base 2 of p_i
he went to John Von Neumann
the famous mathematician
and he said, "what sould I call this quantity?"
and Von Neumann says
"you should call it H, because that's what Boltzmann called it "
but Von Neumann, who had a famous memory
apparently forgot
of Boltzmann's H
of his famous "H theorem"
was the same thing but without the minus sign
so it's a negative quantity
and it gets more and more negative
as opposed to entropy
which is a positive quantiy
and gets more and more positive
so, actually these fundamental formulas
about information theory
go back to the mid 19th century
a hundred and fifty years
so now we'd like to apply them
to ideas about communication
and to do that, I'd like to tell you
a little bit more about
probability
so, we talked about
probabilities for events
probability x
you know x equals
"it's sunny"
probability of y
y is "it's raining"
and we could look at the probability
of x
and I'm gonna use the notation
I introduced for Boolean logic before
is the probabiity of this thing right here
means "AND"
"probability of X AND Y"
or we can also just call
this the probability of X Y simultaneously
keep on streamlining our notation
this is the probability
that it's raining... it's sunny and it's raining
now, mostly in the world
this is a pretty small probability
but here in Santa Fe
it happens all the time
and as a result, you get rather beautiful
rainbows...single, double, triple
on a daily basis
so, we have...
this is what is called
the joint probability
the joint probability that it's sunny and its raining
the joint probability of X and Y
and what do we expect of this joint probability?
so, we have the probability of X and Y
and this tells you the probability that it's sunny and it's raining
we can also look at the probability X AND NOT Y
so X AND (NOT Y)
again using our notation
introduced to us by the famous husband
of the daughter of the severe general of the British ___(?)
George Boole, married to Mary Everest
and we have a relationship which says
that the probability of X on its own
should be equal to the probability of X AND Y plus the
probability of X AND (NOT Y)
and the probability of X on its own
is called the "marginal probability"
so, it's just the probability that
it's sunny on its own
so the probability that it's sunny on its own
is the probability that it's sunny and it's raning
plus the probability that it's sunny and it's not raining
I think this makes some kind of sense
why is called the "marginal probability"?
I have no idea
so let's not even worry about it
there's a very nice picture of probabilities
in terms of set theory
I don't know about you
but I grew up in the age of "new math"
where they tried to teach us
about set theory
and unions of sets
and intersections of sets and things like that
from starting at a very early age
which means people of my generation
are completely unable to do their tax returns
but for me, dealing a lot with math
it actually has been quite helpful
for my career to learn about set theory at the age of 3 or 4
or whatever it was
so, we have a picture like this
this is the space or the set of all events
here is the set X
which is the set of events X, where it's sunny
here is the set of events Y, where is the set of events where it's raining
this thing right here is called
"X intersection Y"
which is the set of events
where it's both sunny and it's raining
but in contrast, if I look at
this right here
this is "X union Y"
which is the set of events
where it's either sunny or raining
and now you can kind of see
where George Boole got his funny "cap" and "cup" notation
we can pair this with X AND Y
X AND Y, from a logical standpoint
is essentially the same as this union of these sets
and similarly, X intersection Y is X OR Y --translator's note: professor Lloyd meant "union" when referring to OR and "intersection" when referring to AND http://www.onlinemathlearning.com/intersection-of-two-sets.html--
so when I take the logical statement corresponding to the set of events
that I write it as X AND Y
the set of events is the intersection of it's sunny and it's raining
X OR Y is the intersection of events where it's sunny or it's raining --translator's note: professor Lloyd meant "union" when referring to OR, "intersection" refers to AND--
and you can have all kinds of you know nice pictures
here's Z where let's say it's snowy at the same time it's sunny
which is something that I've seen happen here in Santa Fe
this is not so strange in here
where we have X intersection Y intersection Z
which is not the empty when in terms of Santa Fe
ok, so now let's actually look
at the kinds of information that are associated with this
suppose that I have a set of possible events, I'll call one set labeled by i
the other set, labeled by j
and now I can look at p of i and j
so this is a case where the first type of event
is i and the second type of event is j
and I can define
you know, I'm gonna do this slightly different
let's call this... we'll be slightly fancier
we'll call these event x_i and event y_j
so, i labels the different events of x
and j labels the different events of y
so, for instance x_i could be two events either it's sunny or it's not sunny
so i could be zero, and it would be 'it's not sunny'
and 1 could be it's sunny
and j could be it's either raining
or it's not raining
so there are two possible value of y
I'm just trying to make my life easier
so we have a joint probability distribution x_i and y_j
this is our joint probability, as before
and now we have a joint information
which we shall call I of X and Y
this is the information
that's inherent in the joint set of events
X and Y
in our case, it being sunny and not sunny, raining and not raining
and this just takes the same form as before
we sum over all different possibilities
sunny-raining, not sunny-raining, sunny-not raining, not sunny-not raining
this is why one shouldn't try to enumerate these things
p of x_i y_j logarithm of p of x_i y_j
so this is the amount of information that's
inherent with these two sets of events together
and of course, we still have this, if you like the
marginal information, the information of X on its own
which is now just the sum over events x on its own
of the marginal distribution
why it's called "marginal" I don't know
it's just the probability for X on its own
p of X_i log base two of X_i
and similarly we can talk about
I of Y is minus the sum over j p of Y_j log to the base 2 of
p of Y_j
this is the amount of information
inherent whether it's sunny or not sunny
it could be up to a bit of information
if it's probability one half of being sunny or not sunny
then there's a bit of information let me tell you in Santa Fe
there's far less than a bit of information
on whether it's sunny or not
because it's sunny most of the time
similarly, raining or not raining
could be up to a bit of information
if each of these probabilities is 1/2
again we're in the high desert here
it's normally not raining
so, you've far less than a bit of information
on the question whether it's raining or not raining
so, we have joint information
constructed out of joint probabilities
marginal information, or information on the original variables on their own,
constructed out of marginal probabilities
and let me end this little section by defining
a very useful quantity which is called the mutual information
the mutual information, which is defined to be
I( X ...I normally define it with this little colon
right in the middle, because it looks nice and symmetrical
and we'll see that this isn't symmetrical
it's the information in X plus the information in Y
minus the information in X and Y taken together
it's possible to show that this is always greater or equal to zero
and this mutual information can be thought of as the amount of information
the variable X has about Y
if X and Y are completely uncorrelated, so it's completely uncorrelated whether it's sunny
or not sunny or raining or not raining
then this will be zero
however, in the case of sunny and not sunny
raning and not raining, they are very correlated
in the sense that once you know that it's sunny
it's probabiy not raining, even though
sometimes that does happen here in Santa Fe
and so in that case, you'd expect
to find a large amount of mutual information
in most places in fact, you'll find that knowing
whether it's sunny or not sunny
gives you a very good prediction
about whether it's raining or it's not raining
mutual information measures the amount of information that X can tell us about Y
it's symmetric, so it tells us the amount of information that Y can tell us about X
and another way of thinking about it
is that it's the amount of information
that X and Y hold in common
which is why it's called "mutual information"