1
Lecture 3
•
HW2 solutions
•
Reviewing Lecture 2+ some comments
•
About entropy and disorder
•
Introduction to Mathematica.
•
Random and pseudo-random numbers.
•
Our first probabilistic experiments with Mathematica
•
Student’s presentation: paradoxes of probability
1. Reminder
2
Multiplication Rule
If N experiments are performed in order and the number of outcomes for the k-th
experiment is nk then the total number of all possible outcomes is n1n2…nN
•Number of all possible permutations between N different objects is N! where
N! = 1 *2 *3 …. (N-2)* (N-1) *N is called “N factorial”. If there are some groups of
“identical” * (or indistinguishable) objects (with m, s, p, etc. elements) the the total
number of distinct permutations is N!/(m!s!p!…)
•Number of permutations of N objects taken m in a time is PN,m = N!/(N-m)!
Note 1: These formulas are used when the order of objects matters
Note 2: The convention is that 0!=1
Number of combinations of n objects taken m in a time: is
Cn.m=Pn,m/m!= n!/[m!(n-m)!] (used when order of objects does not matter)
* The
“identity” is defined by the context. The objects should not be physically identical to fit
this requirement. (in fact, “physically identical” objects can only exist on the sub-atomic level).
For instance, if we only care about the eye color, then all the blue-eyed people are identical in
the context defined by this experiment.
3
Examples:
1.1 Counting sequences of distinguishable and indistinguishable objects.
How many different arrangements are there of the letters comprising the
word “ALL”. The answer depends on whether or not you can tell the L’s
apart. Suppose first that one L has subscript “1” and the other has
subscript “2”. Then all the letter are distinguishable and there is W= 3!
different arrangements:
AL1L2, AL2L1, L1AL2, L2AL1, L1L2A, L2L1A
However, if two A’s are indistinguishable from each other and have no
subscripts, then there are only W=3 distinct sequences:
ALL, LAL, LLA. More generally, W = N!/NL!. The nominator is the number
of permutations if all the characters were distinguishable and the
denominator NL! corrects for “overcounting”.
4
Examples:
1.2 Bose-Einstein statistics. This example is discussed in Molecular
Driving Forces by Ken A. Dill and Sarina Bromberg © 2003, p.
13(click here)
How many ways can n indistinguishable particles put into M boxes, with any
number of particles per box. This problem is related to the properties of the
microscopic particles called “bosons”, such as photons and He4 atoms.
(These bosons have integer spin, while the fermions, such as protons,
electrons, have half-integer spin).
The problems has its own value and we won’t discuss its relation to B-E in
more details. Consider first 3 identical balls and two boxes.
5
There are 4 ways to distribute 3 balls between 2 boxes (Fig.
(a)) Fig (b) hints that this problem can be solved by
considering a linear array of n particles and m-1 movable
walls.
W(3,2)= 4!/3! = 4. For n particles and M boxes the
answer is W(n,M) = (n+M-1)!/[n!(M-1)!].
6
In fact, this problem can be described in terms of
permutations in the linear array of n indistinguishable
particles mixed with M-1 movable walls that partition the
system into M boxes (spaced between the walls). There are
n+M-1 positions in the array; each can be occupied either by
a wall or by a particle.
The total number of the arrangements is (n+M-1)!. Given
that there are two groups of identical objects, n and M-1
each, we derive the aforementioned result:
W(n,M) = (n+M-1)!/[n!(M-1)!].
Try at home to understand it better.
7
Some distributions for the repetitive independent
experiments.
1. The binomial distribution
Conditions: Two outcomes, with probabilities p (success) and q=1-p (failure); the
number of trials (experiments) is fixed
Question : The probability of k successes in n experiments
Answer: p(n,m)=Cn,k pkqn-k
Try it:
Try proving that in 3 flips of a coin the total probability of getting either 0 or 1 or 2 or 3
heads equals 1.
Use Matematica and the Binomial distribution. Can you generalize it for n flips?
8
Some distributions for the repetitive independent
experiments.
2.
The multinomial distribution
Conditions: m possible outcomes with probabilities p1, p2 ,.., pm . The number
of trials n , is fixed.
Question: What is the probability of getting exactly ni outcomes of type i in n=
n1+n2+..+nm experiments?
Answer:
n
n
n
n
!
1
2
p ( n , n ,...n ) 
( p ) ( p ) ( p ) k
1 2
2
k
k
n ! n !...n ! 1
1 2
k
9
Reminder
3.
The geometric distribution
Conditions: p and q=1-p are the success and failure probabilities.
Question : what is the probability that the first success occurs in the experiment
number k?
Answer:
p(n=k)=qk-1p
4.
The negative binomial distribution
Conditions: two alternative outcomes, probabilities p and q. The number of trials, Tn
, is not fixed.
Question: What is the probability that Tn trials will be required for getting exactly n
successful outcomes?
Answer:
p (T  n  k )  C
p n (1  p )k
n
k  n 1 .k
10
Let’s discuss a question
What is the probability of getting 5 Heads tossing 10
coins.
Make your guess.
Is it 1?
0.5?
0.25?
0.1?
11
The solution is given by the Binomial distribution.
The probability of n heads out of 10 is
P(10,n)= C10,n(1/2)n(1/2)10-n=C10,n /210
Let’s solve it with Mathematica
12
13
As you can see, the probability function p(H) reaches its
maximum at H= 5. However, p(H=5) does not equal 0.5, but
rather ~0.25.
Why is that?
Do it yourself at home:
How p(n/2) depends on n?
Check it for n=20,50,100, 1000. What is the tendency?
Plot p(H) for these cases, with the number of heads H
changing from 0 to n.
14
In addition to Lect 2:
Hypergeometric distribution
In the Bernoulli process that lead to Binomial and Geometric
distribution, it was assumed that the trials were independent.
Sometimes this is not the case.
For example, consider urn containing 5 white and 10 black marbles.
What is the probability of obtaining exactly 2 white and 3 black marbles
in drawing 5 marbles without replacement from the urn.
It is clear that if we draw with replacement, the trials are independent
and we can use the binomial distribution (what will be the answer?)
However, when we draw without replacement, the probability changes
after each draw. We already solved this kind of problems before. Let’s
now formalize the solution.
Consider urn containing 5 white and 10 black marbles. What is15
the
probability of obtaining exactly 2 white and 3 black marbles in
drawing 5 marbles without replacement from the urn.
According to its definition, the probability can be found as
P = N success / N total
(3.1)
where N success is the total number of the “successful” outcomes (2 white and 3
black ) N total is the total number of all possible outcomes.
We can obtain this probability as follows:
1. Find total number of ways we can draw 5 marbles out of 15: N total
2. Find the total ways we can draw 2 white marbles out of 5 (C5,2)
=C15,5.
3. The same for 3 black marbles out of 10 (C10,3).
4. Multiply (2) by (3) to find the total number of ways that we can be successful :
N
success
= C5,2* C10,3.
16
Using (3.1) we find
P(2 white and 3 black)= C5,2* C10,3/ C15,5~0.4
In a general case, if
A= number of outcomes having a given characteristic
B = the same with another characteristic
A+B is the total number of all possible outcomes
a= number of elements of A found in N trials
b = number of elements of B found in N trials,
So that N= a + b.
then
P (a, b) 
C A , a C B ,b
C A  B ,a  b

C A,a C B , N  a
C A B,N
This is called hypergeometric distribution
17
Problem:
On a board of directors composed of 10 members, 6 is in favor of
certain decision and 4 oppose it.
What is the probability that a randomly chosen subcommittee of 5
will contain exactly 3 who favor the change?
What is the probability that a subcommittee of 5 will vote against
the change?
(This sort of problems will be included in the test).
18
Class presentations
Starting with this week you can make ~15 min in-class presentations.
These are the extra-credit volunteer projects. They are not
obligatory, but everyone is encouraged and will be given a chance
to make one.
The first topic is the “Paradoxes of probability 1” with focus on three
paradoxes:
(1) The “two children” paradox
(2) The prisoner dilemma
(3) “Intransitive spinners”
Their discussion can be found in the Chapter 10 of Bennett’s
“Randomness”.
The second one is the “Paradoxes of probability 2 “:
The “Buffon needle”.
The references will be provided.
19
A short insert about entropy and disorder
Why does the atmosphere get thinner with elevation?
There are two major “forces” in this problem. One is a force of gravity. It is
responsible for the potential energy of the molecules,
Ep(h)=gmn(h) h
(3.1)
relative to the ground level (here m is the molecular mass for some of the
atmospheric gases and n(h) is its concentration at the elevation h) . This force is
trying to collect all the molecules at the ground level.
Various Natural phenomena are often described in terms of optimization, or
minimization of different “energy functions”. If the Nature just minimized the
potential energy, all the atmosphere would’ve condensed on the surface, and life
would be impossible.
In fact, the Nature is trying to minimize a different function, F, which includes
Ep(h) as just one of its components. In our problem, where the gas is considered
ideal (the interaction between the molecules is neglected), F can be presented as
20
F=Ep-T S
(3.2)
T is the temperature. The function S is called “entropy”. It
plays the most important role in statistical physics and
thermodynamics. In fact, it’s a gateway for the probability.
Entropy describes a universal tendency towards disorder.
For the ideal gas model,
S=-k n(h)ln(n(h))
(3.3)
Inserting Eqs. (3.1) and (3.3) in (3.3) and using the conditions
F=minn(F(n)), one can find
n(h)=n(0) Exp(- μ gh/RT)
(3.4)
* This equation also describes decrease in atmospheric pressure with elevation, the so called
“barometric formula”. You will probably like the following story related to this formula.
μ is the molar mass, and R is the gas constant.
It is clear from the Eq. 3.2 that the role of S increases with
temperature T. That’s why it’s instructive to analyze how the n(h)
depends on T. Let’s consider the exponential distribution
function (3.4) and draw it for three different temperatures
T1<T2<T3.
21
22
What lesson do we learn from this picture?
Each of the curves represents a probability of finding a gas molecule at
elevation h. In fact, n(h)/n0 is the probability distribution function. When
T grows, the distribution becomes more uniform. At high
temperatures,e.g. T=9T1 , the probability of finding a molecule at h=4 is
almost as big as at h=0. The distribution becomes shallower and wider.
This behavior can be characterized as an increasing disorder.
The tendency is quite universal. An example with a crystal of sugar
placed in water is illustrative. The crystal, which symbolizes the order
in Nature, turns into a solute, an absolutely disordered state, similar to
the gas phase.
Note: the generalized distribution (3.4),
n(r)=n(0) Exp(-EP(r)/kT)*
is known as “Boltzmann distributions”.
(3.5)
23
n(r)=n(0) Exp(-EP(r)/kT)
(3.5)
The Boltzmann distribution is quite universal and has numerous applications.
It basically indicates that the probability of occupying a certain state with
energy En is proportional to the exponential factor Exp(-En/kT).
In other words, the higher the energy of a configuration, the less
probable it is.
This is the key factor in computer modeling of biological molecules such as
proteins. They possess multiple “conformations” with close energies and
thus having close probabilities. This fact is responsible for the structural
mobility of the proteins. On the other hand, the protein folding into its native
(functional) conformation occurs almost instantly when the appropriate
physiological conditions are applied. Coexistence of multiple conformational
states with rapid transition to a unique functional conformation is one of the
major challenges in computational biophysics.
Equations (3.5) directly describes “ionic atmospheres” surrounding charges
in solvents, and accounting for electrostatic interactions in chemistry and
biology. In this case EP(r) is due to electrostatic interaction between the
24
Principles of Monte-Carlo molecular
modeling : statistical algorithms for
simulating equilibrium properties
(Boltzmann distribution) - (to be added)
[This is a place-holder  ]
25
A few more words about relation between the entropy
and probability
“When you drop an egg on the floor, it breaks. But dropping a broken egg on
the floor doesn't cause it to become whole.
A new deck of cards comes with all the cards in order. Shuffling them mixes
up the cards. Now take a deck of cards that's mixed up and shuffle them. The
cards do not come back to their original order.
Pick up a can of air freshener and push down the button. The air freshener
spews out of the can and spreads out around the room. Now try gathering up
the air freshener and putting it back in the can. Doesn't work, does it? “
“The shuffling of a deck of cards, the spray of an aerosol
can, the flow of heat from warm things to cold things, the
breaking of an egg; these are all examples of what
physicists call " irreversible processes" . They occur very
naturally, but all the king's horses and all the king's men
can't undo them”.
26
A few more words about relation between the entropy
and probability
In the isolate system entropy never decreases:
All processes in the isolated system are directed towards
the macroscopic state with higher probability.
Two examples: (1) Energy flow from a hot to a cold object
(2) Gas always flow from the the higher
density to the lower
Let’s consider a second case in more details to illustrate a
relation between the entropy and probability
27
The probability of each “macro-state” is determined by the # of
possible configurations (microstates) corresponding to this macrostate.
Events (states) with higher entropy have higher probability.
Example: connected containers with N distinguishable particles.
L
R
N= nL + nR. The “macro state” can be described by (N, nL)
28
D
A
B
C
For (4,1) there are 4 microstates:
(A,BCD), (B,ACD),{C,ABD),{D,ABC)
Calculate how many states are there in configuration
(4,2): 2? 4? 8?
A
B
D
C
29
The answer is 6.
You probably noticed that the number of microstates
with nL particles in the left box is
 ( N , n )  C N ,nL 
N!
n L ! ( N  n L )!
The total number of microstates is 2N.
Then we find for the probability:
P(N,n)=CN,n/2N=N!/(2N n!(N-n)!)
Analyzing this expression you will find (similar to N
coins) that n=N/2 corresponds to the most probable
state.
30
The entropy is defined as
  ln(  )
Thus, it is proportional to ln(P):
It increases when the macroscopic state becomes more
probable.
This statement describes one of the major links
between the Nature and Probability.
31
The fight between the Chaos and the Order can be discovered in every
aspect of our life.
If we do not take special actions, the disorder around us will always
increase: desk, clothing, home will become messier and messier. They
never come back to the order by themselves. But does it have anything
to do with Nature? Sure it does.
Why the disorder always increases if you give it a chance? The answer
is quite simple: while the ordered state can be achieved by only a few
arrangements (choices), the disorder is represented by practically
infinite number of them.
32
A child carefully builds a tower from N blocks, placing them in the
alphabetical order. There is only one proper arrangement.
The task is not an easy one. It takes a lot of time and efforts.
Another child simply touches it, and disorder is achieved in
a moment. No time or efforts are required.
A few manifestations of the entropy laws.
• Easier to destroy than build
• The information is costly and vulnerable.
• Energy can not be used completely. Whenever there is a
transfer of energy, some of the potential energy is lost as heat,
the random movement of molecules.
A possible project: “Entropy, Information and probability”.
33
Read Claude Shannon and related links
The Information is often described as “negentropy” (although sometimes
this term can be misleading). It is associated with rear occurrences, unlikely
events. In our example with Encyclopedia Britannica, the “information” can
be associated with the correct order of the volumes on the shelf.
As we have shown before, the probability of finding it at random is much
less that a probability of pulling one particular atom from the body of our
planet.
Scientific discoveries, elaborate architectural design and masterpieces of arts
are so valuable exactly because they exist against the odds of random
sampling. The probability that monkey hitting randomly a keyboard will
type just “To be or not to be” is ~10-15 (check this estimate).
The project can run in any direction, including possible applications of the
information theory in molecular biology, molecular computations, etc. (click
here for references).
34
Random and pseudorandom numbers
The experiments with real dice are very slow. To overcome this
shortcomings, the computer experiments can be used. To deal with
randomness, computers are equipped with so called "Random number
generators".
They use the sequences of numbers that, on one hand, are fixed, and in this
sense are completely deterministic, but, on the other hand, they display
every sign of randomness.
Suggested reading: “Randomness” by Deborah J. Bennett, Ch. 8.
35
One of the most popular methods for generating random numbers is the linear
congruential generator. These generators use a method similar to the folding
schemes in chaotic maps. The general formula is, I k  ( a I k 1  c ) m o d m
The values a, c and m are pre-selected constants. a is known as the multiplier, c is
the increment, and m is the modulus.
The quality of the generator is strongly dependent upon the choice of these
constants The method is appealing however, because once a good set of the
parameters is found, it is very easy to program. One fairly obvious goal is to make
the period (the time before the generator repeats the sequence) long, this implies
that m should be as large as possible. This means that 16 bit random numbers
generated by this method have at most a period of 216= 65,536, which is not nearly
long enough for serious applications.
The choice of a = 1277, c = 0, m = 131072 looks okay for a while but eventually
has problems. A scatter plot for this for 2000 pairs from this generator reveals linear
bands emerging from the plot. A random walk plot shows that after a while the slope
changes (it should be a constant slope).
36
The choice of a = 16807, c = 0, m = 2147483647 is a very good set of parameters for
this generator Park & Miller (1988). This generator often known as the minimal
standard random number generator, it is often (but not always) the generator that
used for the built in random number function in compilers and other software
packages.
To learn about the more advanced methods, and about the differences between the
pseudo-random and Quasi-random numbers, read E,F. Carter.
In the Mathematica program that we will be using, the random numbers are
generated by the function called Random[ ]. For instance,
Random[Integer,{1,6}] produces a result of single through of die. To imitate
n consecutive tests, one has to fill with random numbers a table:
Table[Random[Integer,{1,6}], {n}].
37
Monte-Carlo estimate for π
Let’s apply our new skills to calculate the magic number π.
Here is the idea behind. Suppose that we draw a circle of radius R=1.The
numeric value of its area is π. Now we insert this circle in a square 3*3.
Suppose than we started shooting randomly in the square, and count the
holes. Let Nin the
number of bullets that hit inside the circle,
while N is the total number of holes.
At large N the ratio Nin / N should become
proportions to the π /9. This provides a
an opportunity of estimating π using the
random number generator.
Here is how it can be done with
Mathematica.
38
The rest of this class we will be
practicing with Mathematica.
Please, follow steps 1-4 from the
Mathematica Lessons link.
39
Home assignment.
Practice for the test
1.
Read the first three lectures, and the home assignments.
2.
Solve the Typical Test Problems, compare with the solutions
3.
Solve the additional problems for practice.
4.
Read the assignment column in the “lecture materials” page.
Read the assignment at the web site
Driving Forces
The combinatorics of disulfide bond formation. A
protein may contain several cysteines, which may pair together
to form disulfide bonds as shown in Figure 1.17.
If there is an even number n of cysteines, n/2 disulfide
bonds can form. How many different disulfide pairing arrangements
are possible?
40