So now instead considering to masses of books
we are gonna consider something much more
interesting and subtle I think surprising .
The frequency of words inside of books
So here is the book.
And different words appear in it
“were” and “move” and “positions” and so on.
And some of these words are very common
and appear quite frequently.
And other words are much rarer.
So what would the distribution of
these frequencies look like.
So I wanna illustrate this
by thinking not about entire book,
but just a short quotation first.
So here is a passage
from the writing of Henri Poincare,
mathematician who helped found
chaos and dynamics.
And he writes,
and I should mention that
I have a pdf of this slide in the next one
in case this is not readable or
you wanna print it out
and work through this on your own
So here’s the passage,
The scientist does not study nature
because it is useful;
he studies it because he delights in it,
and he delights in it because it is beautiful.
If nature were not beautiful
it would not be worth knowing,
and if nature were not worth knowing,
life would not be worth living.
So let’s think about
how often certain words appear here.
So the word “it” appears 6 different times
in this passage. Let’s see.
if I can find them all 1,2,3,4,5,6
So there is 6 occurrences of the word “it”
The word “not” appears 5 different times.
And then there 4 words that appear 3 times.
So the “nature” appears 3 times. Let’s see.
1,2,3 there are the 3 instances of “nature”.
There are a bunch of words
10 of them that appear twice.
And then there are 8 words that appears once.
So I got a bunch of different words
with different frequencies
So what I am gonna do now
is considered not the word themselves
but just the frequencies.
So there is one 6, one 5
4 threes
8 twos
excuse me 10 twos
and 8 once
So the world frequencies then are these.
one 6 and 5
4 threes and so on
And so I could take these data here.
And I could turn that into a histogram.
Like just count the frequency of occurrences
of these different numbers.
So let’s see what happens if I do that.
So I wanna make a histogram.
There is one occurrence of 6
So the histogram for 6
is gonna be 1
There is one occurrence of 5
And 0 fours
So this is gonna look like this
0 fours
There are 4 threes 1,2,3,4
Then we had 10 twos
And 8 once
So this histogram has just arrived from these data
which is derived from counting words frequencies.
So what this means
is that in this passage
There are 8 words that appear once.
There are 10 words that appear twice.
4 words that appear 3 times.
0 words that appear 4 times.
And there is one word that appears 5 times .
And one word that appears 6 times.
So what we’ll do next
is we consider word frequencies
not for couple sentences
but for much much longer piece of text.
So we’ll consider the word frequencies
from the novel Moby Dick.
So this is a novel by Herman Melville.
I have to confess I’ve never read it
and to be honest I probably never will.
But I have made plots of the word frequencies of it.
may be that’s little bit more fun.
But any way
So Moby Dick
total number of words is about 210.000
And the different number,
the number of different words is about 18.800
So let’s think about what we might expect
Are we gonna see that may be
most words are used 3 or 4 times?
or most words are gonna be
used only once or twice.
Well, let’s see.
So here’s a result of making a histogram
of word frequencies for Moby Dick.
The source for this data
I’ll put out information here
and put a link also on this,
on the page for this video.
So here’s what this says.
So this is the frequency.
And then this is the number of occurrences.
So this says
There are
Let’s see 2, 6
something like more then 8000 words
that appear only once.
There are around 3000 words that appear twice.
There are few words still that appear
3 times and 4 times and 5 times.
So most words in this novel
are the most common frequency is 1.
And then it drops off very quickly.
So when you make a histogram
you can count the number of occurrences
you can also think in terms of probability.
So here’s the same data.
And all I’ve done is I’ve just plotted
the y axes in terms of probability.
So here’s what this says
of the 18.000 or so unique words different words
that appear in Moby Dick
almost half of them appear only once.
So I think that’s surprising and really interesting
And then about 18 percent of them,
all these words appear only twice and so on.
So we’re seeing something
that drops off very quickly.
And has a very big peak for the small value 1
What does this look like if I go out further still
Well, let’s see.
So here I am plotting not the first 30, not 30
but out of 300
so how many words appear 300 times in the novel,
or 250 times, or 260 times
Well, clearly not very many.
May be it's 0, it’s hard to tell.
Because the scale of this plot,
things get scrunch down a lot.
So what we’ll do is we’ll look at log-log plot of this.
So what I am gonna do
is just take the logarithm of the frequency
the logarithm of the probability
and I am gonna plot those.
So here’s what happens if I do that
So this is the log of the frequency.
And this is the log of the probability.
And we can see that there is a straight line here
to a pretty good approximation
and so we learned in the last unit that
when we see a straight line like this
we expect, well I say that is
we saw the straight line a lot
and we were doing box counting.
So we would expect that
this would be described by a formula
similar to that box counting formula.
And we’ll take a look at that in a second.
But first let me just point out
few more things about this plot.
So again down here this says that there are
a lot of words that appear only once.
But what about this word.
This is the most common word.
It has the largest frequency
and let’s see this is for Moby Dick the word “the”
And it appears 14086 times.
So we have a lot of words that appear exactly once
And then we have one word “the”
that appears 14000 some odd times.
That’s a really big range of frequencies
of a lot bigger than the range of book masses
that I had before.
Let’s see.
The second most common word
just in case your curios is “of”
Then what we’ve got “and”
and “a”
and the word “of” appears 6414 times.
And you could look the data file
and get the rest of these.
So there are few words in Moby Dick and
in English in general,
And this is a general feature of languages
that appear very very often.
And then there are lots and lots and
lots and lots of words that appear
very rarely in this particular novel only once.