In the previous unit we showed that

a naive maximum entropy model

for the distribution
of languages abundances

for disturbing how popular a language is

where popularity is

how many times that language
is used in the open source archive

that model, which takes
the following functional form

and constraint the speculation value
of language abundance

that constraints only

the average number of
times a language is used

in the archive over all languages

this maximum entropy model
is completely unsuccessful

in reproducing the data we see

it under predicts the popular languages

and over predicts
the less popular languages

this is the best fit model, which is what

happens when I solve lambda and z

to get the correct average constraint

that we see in the data, ok?

so, what I'm gonna do is ask you

to consider a richer model

and again

by model I mean probability distribution

and this distribution is
gonna try to explain not only

the popularity of languages, but also,

the amount of time devoted to them

the amount of programmer time devoted

so n here is again, as before

the number of projects

that appear in the language

and this here

epsilon is the amount of
time devoted per project

you should think of epsilon as modelling

some kind of language efficiency

and this n is modelling
language popularity

some languages are for example

extremely popular

and extremely efficient, so they have

n is large and epsilon is small

other languages perhaps,
are not quite has popular

and are perhaps less efficient

or require more programmer time

and of course there might be
extremely efficient languages

that are profoundly unpopular,
and vise versa,

the intuition here is that we are tying to

describe two things at once, previously

we where building a model
that only included

one variable, popularity, now we are gonna

try to build a model that
has two variables

popularity and efficiency

or amount of time devoted per project

so, what we're gonna do

is we're gonna constraint

two quantities, as before

we're gonna constraint,
average number of projects

and we're also gonna constraint

the average amount of programer time

devoted to a language

so the idea is that there is some

intrinsic force that keeps the average
number of projects per language constant

there is some kind of

large scale social factors

or set of social factors

that keeps the average
number of projects constant

and also constraints

the average number of programmer time

devoted to a particular language

so

let's put in a
maximum entropy distribution

for this two parameter model

we're gonna constraint n

so we know

that we have a term that looks like

e to the lambda one times n

as before

e to the negative lambda one times n

and we're also going to have a term

in the exponential that looks like

lambda two, that's
the second Lagrange multiplier

times n times epsilon

and of course there will be
an over all normalization factor of Z

so the only thing that should
be mysterious to you

is where we got this second term from

we know where this comes from

but why do we have this?

and so what I'll do is

I'll very quickly remind you

of how we got this functional
forms in the first place

we did the derivative

of the function we wish
to maximize the entropy

and now we will have three constraints

we'll have a constraint

on the average number of projects

that constraint g1

g1 of p is equals c

we'll have a constraint g2

on this term here

g2 of p is equal to some c prime

and finally we have that
over all normalization constraint

g3 of p i

and so

I'll write this out here

g1 of p is equal

and g2 of p

so, g1 of p is the average
number of languages

or the average number of projects
for particular language

that's P(n) times n

and we can write that
out in terms of the joint distribution

as P(n,epsilon) times n

now we not only sum over all values of n

but we integrate over all
of values of programmer time epsilon

g2(p) looks very similar, except now

we're constraining not just n

but n times epsilon

so we have to integrate over epsilon

and we have to sum over all values of p

and that's what our equation looks like

that's what our constraint looks like

for this second quantity here

so now, when we take the derivative
of g1 with respect of p1

we know what that looks like

that looks like lambda 1 times n

and when we take the derivative
of this here with respect of p i

everything comes out, except

for lambda2 n epsilon

and finally g 3

is just a constant on
the over all normalization

is says that

when I integrate over all epsilon

and sum over all n that's equal to 1

so when I take the derivative of this here

with respect to p

we get lambda 3 times unity

on this other side here

the derivative of entropy
with respect to p i

is equal to log p i

negative log p i, plus, or rather
minus one

and when we arrange this I get p i

is proportional to e to the
negative lambda 1 n

minus lambda 2 n epsilon

everything else can be factored into some

underlying normalization

so that's where this
functional form comes from

and is a general principle

a general rule of thumb

that if you wanna see

what the maximum entropy
distribution is constraining

look term by term

look term by term in the exponential

to find it

what we have now

is the following functional form

for the joint distribution
of language popularity

and language efficiency

and I'm gonna do one more thing

which is I'm gonna try to recover

the original marginal distribution

Pn, by integrating

the joint distribution

with respect of epsilon

so what I did was

I build a more sophisticated model

that had two constraints

it constraint the average
number of projects

and the average amount of programmer time

devoted to a particular language

and now what I'm gonna do is

I'm gonna integrate out
this hidden variable

so if I do that, if I do this integral

here's what I get

I still get this factor of e to
the negative lambda 1 n

I'll call this lambda prime

I still get the factor of 1 over Z

and the I simply have to do

the following little integral

negative lambda 2 n epsilon d-epsilon

and that equal to 1 over
n negative lambda 2

times e to the negative lambda 2 n epsilon

my integral range is from zero to infinity

and so when I put this terms in here

my final functional form, and again

I'm not keeping track of
all the multiplications

so I'm turning Z into Z'
to absorb this factors here

the final functional form

looks like an exponential on top

divided by n

so my new functional form

for the language abundance

is modified

previously, just to remind you

previously, when I did Max. Ent.

constraining only the average
number of projects

my distribution of language popularity

looked like an exponential

and we realized this is a really bad fit

this is the fit you see in the plot here

but now, what we've done

is produce a joint distribution
with two constraints

one of the constraints is the same

the other is a new constraint

that involves that hidden variable

programmer efficiency

and when I integrate out
that hidden variable

I get a new functional form

for language popularity

in particular, we find
that is proportional

to e to the negative lambda n divided by n

this distribution is called

the Fisher log series

so, here's the functional form

which we got by integrating out

this hidden programmer efficiency variable

and what you can see now is that

that distribution is a far better model
for language popularity

and what we've done here

just to be clear

is that we postulated
this additional constraint

involving a hidden variable

that we can't see

and we've integrated out
that hidden variable

what we should be impressed by

is the fact not only

that we are able to get a good fit

but also by the underlying mechanisms

that the Max. Ent. model
is suggesting to you

it's suggesting that two
things are constraint

languages tend not to get too popular

on average

there is some kind of limit to how
popular a language can get

on average

that's the source of this constraint, here

but there's also a limit

to how much time or
how much effort can be devoted

to projects in a particular language

and that's this constraint here

so essentially, what's happening

is this languages up here

are allowed to become more popular

because they don't take up too
much programmer time per project

this in the language of an ecologist

this are the really abundant species

with really low metabolic needs

so we're now able to reproduce

this really popular languages

we're able to accurately explain, or

we're able to accurately predict

that there should be
really run away popular languages

like C and C++ and Java

and also we're able to explain

why the popularity in
the languages in the tali here

is lower than you would expect

from an exponential model

in particular presumably

a lot of this languages are
associated with lower efficiencies

or rather epsilon higher

meaning that there is a
higher amount of time

devoted per project to those languages

and intuitively, we're programming in Java

this is a really high efficiency language

compared to programming in Haskell

this is a profoundly
beautiful language, perhaps

but is one that you would
not associate with efficiency

writing fast libraries

for parsing html pages

so, by postulating the existence
of this hidden variable

and using a maximum entropy argument

we can actually start reproducing the data

and it feels like we starting
to talk about

important constraints in the system

so you should be impressed by this

in part because it's deeper

than the result we've got
in the taxi cab case

in the taxi cab case we
picked a particular constraint

and it just so happened to correspond

to a really simple mechanistic model

and you might say

well I could have come up

with that mechanistic model all on my own

thank you very much

here what we've done

it's postulate a set of constraints

that are scientifically substantive

the first constraint says that

we're gonna constraint
the average number of projects

so the open source movement
can do what ever it wants

as long as it keeps the average
number of projects per language fixed

people are making decisions

in fact, they are making decisions

that are in a maximally disordered way

that's the maximum entropy
part of the assumption

so that they preserve

this quantity here

on average

and further more

the open source community

by what ever method

or by what ever group cognitive method

that's able to do this

and presumably by a hole lot
of different interacting process

also preservers the average amount

of programmer time devoted
to a particular language