LECTURE 2
PROBABILITY REVIEW AND
RANDOM PROCESS
1
REVIEW OF LAST LECTURE

The point worth noting are :





The source coding algorithm plays an important
role in higher code rate (compressing data)
The channel encoder introduce redundancy in data
The modulation scheme plays important role in
deciding the data rate and immunity of signal
towards the errors introduced by the channel
Channel can introduce many types of errors due to
thermal noise etc.
The demodulator and decoder should provide high
Bit Error Rate (BER).
2
REVIEW:
LAYERING OF SOURCE CODING

Source coding includes
Sampling
 Quantization
 Symbols to bits
 Compression


Decoding includes
Decompression
 Bits to symbols
 Symbols to sequence of numbers
 Sequence to waveform (Reconstruction)

3
REVIEW:
LAYERING OF SOURCE CODING
4
REVIEW:
LAYERING OF CHANNEL CODING

Channel Coding is divided into

Discrete encoder\Decoder


Used to correct channel Errors
Modulation\Demodulation

Used to map bits to waveform for transmission
5
REVIEW:
LAYERING OF CHANNEL CODING
6
REVIEW:
RESOURCES OF A COMMUNICATION
SYSTEM

Transmitted Power


Bandwidth (spectrum)


Average power of the transmitted signal
Band of frequencies allocated for the signal
Type of Communication system

Power limited System


Space communication links
Band limited Systems

Telephone systems
7
REVIEW:
DIGITAL COMMUNICATION SYSTEM

Important features of a DCS:
Transmitter sends a waveform from a finite set of
possible waveforms during a limited time
 Channel distorts, attenuates the transmitted signal and
adds noise to it.
 Receiver decides which waveform was transmitted from
the noisy received signal
 Probability of erroneous decision is an important
measure for the system performance

8
REVIEW OF PROBABILITY
9
SAMPLE SPACE AND PROBABILITY

Random experiment: its outcome, for some
reason, cannot be predicted with certainty.

Examples: throwing a die, flipping a coin and
drawing a card from a deck.
Sample space: the set of all possible outcomes,
denoted by S. Outcomes are denoted by E’s and
each E lies in S, i.e., E ∈ S.
 A sample space can be discrete or continuous.
 Events are subsets of the sample space for which
measures of their occurrences, called
probabilities, can be defined or determined.

10
THREE AXIOMS OF PROBABILITY
For a discrete sample space S, define a
probability measure P on as a set function that
assigns nonnegative values to all events, denoted
by E, in such that the following conditions are
satisfied
 Axiom 1: 0 ≤ P(E) ≤ 1 for all E ∈ S
 Axiom 2: P(S) = 1 (when an experiment is
conducted there has to be an outcome).
 Axiom 3: For mutually exclusive events E1, E2,
E3,. . . we have

11
CONDITIONAL PROBABILITY





We observe or are told that event E1 has occurred but are
actually interested in event E2: Knowledge that of E1 has
occurred changes the probability of E2 occurring.
If it was P(E2) before, it now becomes P(E2|E1), the
probability of E2 occurring given that event E1 has
occurred.
This conditional probability is given by
If P(E2|E1) = P(E2), or P(E2 ∩ E1) = P(E1)P(E2), then E1
and E2 are said to be statistically independent.
Bayes’ rule

P(E2|E1) = P(E1|E2)P(E2)/P(E1)
12
MATHEMATICAL MODEL FOR SIGNALS

Mathematical models for representing signals



Deterministic signal: No uncertainty with respect to the
signal value at any time.



Deterministic
Stochastic
Deterministic signals or waveforms are modeled by explicit
mathematical expressions, such as
x(t) = 5 cos(10*t).
Inappropriate for real-world problems???
Stochastic/Random signal: Some degree of uncertainty in
signal values before it actually occurs.
For a random waveform it is not possible to write such an explicit
expression.
 Random waveform/ random process, may exhibit certain
regularities that can be described in terms of probabilities and
statistical averages.
 e.g. thermal noise in electronic circuits due to the random
movement of electrons
13

ENERGY AND POWER SIGNALS


The performance of a communication system depends on the
received signal energy: higher energy signals are detected
more reliably (with fewer errors) than are lower energy
signals.
An electrical signal can be represented as a voltage v(t) or a
current i(t) with instantaneous power p(t) across a resistor
2
defined by
v (t )
p (t ) 

OR
p (t )  i
2
(t )
14
ENERGY AND POWER SIGNALS


In communication systems, power is often normalized by assuming
R to be 1.
The normalization convention allows us to express the
instantaneous power as
2
p (t )  x (t )
where x(t) is either a voltage or a current signal.


The energy dissipated during the time interval (-T/2, T/2) by a real
signal with instantaneous power expressed by Equation (1.4) can
then be written as:
The average power dissipated by the signal during the interval is:
15
ENERGY AND POWER SIGNALS



We classify x(t) as an energy signal if, and only if, it has nonzero
but finite energy (0 < Ex < ∞) for all time, where
An energy signal has finite energy but zero average power
Signals that are both deterministic and non-periodic are termed
as Energy Signals
16
ENERGY AND POWER SIGNALS

Power is the rate at which the energy is delivered

We classify x(t) as an power signal if, and only if, it has nonzero
but finite energy (0 < Px < ∞) for all time, where

A power signal has finite power but infinite energy

Signals that are random or periodic termed as Power Signals
17
RANDOM VARIABLE
Functions whose domain is a sample space and
whose range is a some set of real numbers is
called random variables.
 Type of RV’s


Discrete


E.g. outcomes of flipping a coin etc
Continuous

E.g. amplitude of a noise voltage at a particular instant of
time
18
RANDOM VARIABLES
Random Variables



All useful signals are random, i.e. the receiver does not know
a priori what wave form is going to be sent by the transmitter
Let a random variable X(A) represent the functional
relationship between a random event A and a real number.
The distribution function Fx(x) of the random variable X is
given by
19
RANDOM VARIABLE
A random variable is a mapping from the sample
space to the set of real numbers.
 We shall denote random variables by boldface,
i.e., x, y, etc., while individual or specific values
of the mapping x are denoted by x(w).

20
RANDOM PROCESS
A random process is a collection of time functions, or signals,
corresponding to various outcomes of a random experiment.
For each outcome, there exists a deterministic function, which
is called a sample function or a realization.
Random
variables
Real number

Sample functions
or realizations
(deterministic
function)
time (t)
21
RANDOM PROCESS

A mapping from a sample space to a set of time functions.
22
RANDOM PROCESS CONTD
Ensemble: The set of possible time functions
that one sees.
 Denote this set by x(t), where the time functions
x1(t, w1), x2(t, w2), x3(t, w3), . . . are specific
members of the ensemble.
 At any time instant, t = tk, we have random
variable x(tk).
 At any two time instants, say t1 and t2, we have
two different random variables x(t1) and x(t2).
 Any realationship b/w any two random variables
is called Joint PDF

23
CLASSIFICATION OF RANDOM PROCESSES
Based on whether its statistics change with time:
the process is non-stationary or stationary.
 Different levels of stationary:

Strictly stationary: the joint pdf of any order is
independent of a shift in time.
 Nth-order stationary: the joint pdf does not depend
on the time shift, but depends on time spacing

24
CUMULATIVE DISTRIBUTION FUNCTION
(CDF)
cdf gives a complete description of the random
variable. It is defined as:
FX(x) = P(E ∈ S : X(E) ≤ x) = P(X ≤ x).
 The cdf has the following properties:

0 ≤ FX(x) ≤ 1 (this follows from Axiom 1 of the
probability measure).
 Fx(x) is non-decreasing: Fx(x1) ≤ Fx(x2) if x1 ≤ x2
(this is because event x(E) ≤ x1 is contained in event
x(E) ≤ x2).
 Fx(−∞) = 0 and Fx(+∞) = 1 (x(E) ≤ −∞ is the empty
set, hence an impossible event, while x(E) ≤ ∞ is the
whole sample space, i.e., a certain event).
 P(a < x ≤ b) = Fx(b) − Fx(a).

25
PROBABILITY DENSITY FUNCTION

The pdf is defined as the derivative of the cdf:
fx(x) = d/dx Fx(x)

It follows that:

Note that, for all i, one has pi ≥ 0 and ∑pi = 1.
26
CUMULATIVE JOINT PDF JOINT PDF
Often encountered when dealing with
combined experiments or repeated trials of a
single experiment.
 Multiple random variables are basically
multidimensional functions defined on a
sample space of a combined experiment.




Experiment 1

S1 = {x1, x2, …,xm}

S2 = {y1, y2 , …, yn}

0 <= P(xi, yj) <= 1 (Joint Probability of two or more
outcomes)
Marginal probabilty distributions
Experiment 2
If we take any one element from S1 and S2



Sum all j P(xi, yj) = P(xi)
Sum all i P(xi, yj) = P(yi)
27
EXPECTATION OF RANDOM VARIABLES
(STATISTICAL AVERAGES)





Statistical averages, or
moments, play an important role
in the characterization of the
random variable.
The first moment of the
probability distribution of a
random variable X is called
mean value mx or expected value
of a random variable X
The second moment of a
probability distribution is meansquare value of X
Central moments are the
moments of the difference
between X and mx, and second
central moment is the variance
of x.
Variance is equal to the
difference between the meansquare value and the square of
the mean
28
Contd
The variance provides a measure of the variable’s
“randomness”.
 The mean and variance of a random variable give
a partial description of its pdf.

29
TIME AVERAGING AND ERGODICITY
A process where any member of the ensemble
exhibits the same statistical behavior as that of
the whole ensemble.
 For an ergodic process: To measure various
statistical averages, it is sufficient to look at only
one realization of the process and find the
corresponding time average.
 For a process to be ergodic it must be stationary.
The converse is not true.

30
GAUSSIAN (OR NORMAL) RANDOM
VARIABLE (PROCESS)

A continuous random variable whose pdf is:
μ and are parameters. Usually denoted as
N(μ, ) .
 Most important and frequently encountered
random variable in communications.
31
CENTRAL LIMIT THEOREM

CLT provides justification for using Gaussian
Process as a model based if
The random variables are statistically independent
 The random variables have probability with same
mean and variance

32
CLT

The central limit theorem states that


“The probability distribution of Vn approaches a
normalized Gaussian Distribution N(0, 1) in the limit
as the number of random variables approach
infinity”
At times when N is finite it may provide a poor
approximation of for the actual probability
distribution
33
AUTOCORRELATION
Autocorrelation of Energy Signals




Correlation is a matching process; autocorrelation refers to
the matching of a signal with a delayed version of itself
The autocorrelation function of a real-valued energy signal
x(t) is defined as:
The autocorrelation function Rx() provides a measure of how
closely the signal matches a copy of itself as the copy is
shifted  units in time.
Rx() is not a function of time; it is only a function of the time
difference  between the waveform and its shifted copy.
34
AUTOCORRELATION




symmetrical in  about
zero
maximum value occurs at
the origin
autocorrelation and ESD
form a Fourier transform
pair, as designated by the
double-headed arrows
value at the origin is
equal to the energy of the
signal
35
AUTOCORRELATION OF A PERIODIC (POWER)
SIGNAL


The autocorrelation function of a real-valued power
signal x(t) is defined as:
When the power signal x(t) is periodic with period T0,
the autocorrelation function can be expressed as:
36
AUTOCORRELATION
OF POWER
SIGNALS
The autocorrelation function of a real-valued periodic signal
has properties similar to those of an energy signal:




symmetrical in  about
zero
maximum value occurs at
the origin
autocorrelation and PSD
form a Fourier transform
pair, as designated by the
double-headed arrows
value at the origin is
equal to the average
power of the signal
37
38
39
SPECTRAL DENSITY
40
SPECTRAL DENSITY




The spectral density of a signal characterizes the
distribution of the signal’s energy or power, in the
frequency domain
This concept is particularly important when considering
filtering in communication systems while evaluating the
signal and noise at the filter output.
The energy spectral density (ESD) or the power spectral
density (PSD) is used in the evaluation.
Need to determine how the average power or energy of
the process is distributed in frequency.
41
SPECTRAL DENSITY

Taking the Fourier transform of the random
process does not work
42
ENERGY SPECTRAL DENSITY

Energy spectral density describes the energy per unit
bandwidth measured in joules/hertz

Represented as x(t), the squared magnitude spectrum
x(t) =|x(f)|2

According to Parseval’s Relation

Therefore

The Energy spectral density is symmetrical in frequency
about origin and total energy of the signal x(t) can be
expressed as
43
POWER SPECTRAL DENSITY

The power spectral density (PSD) function Gx(f) of the
periodic signal x(t) is a real, even ad nonnegative function of
frequency that gives the distribution of the power of x(t) in
the frequency domain.

PSD is represented as (Fourier Series):

PSD of non-periodic signals:

Whereas the average power of a periodic signal x(t) is
represented as:
44
NOISE
45
NOISE IN THE COMMUNICATION SYSTEM



The term noise refers to unwanted electrical signals that are
always present in electrical systems: e.g. spark-plug ignition
noise, switching transients and other electro-magnetic signals
or atmosphere: the sun and other galactic sources
Can describe thermal noise as zero-mean Gaussian random
process
A Gaussian process n(t) is a random function whose value n at
any arbitrary time t is statistically characterized by the
Gaussian probability density function
46
WHITE NOISE


The primary spectral characteristic of thermal noise is that its
power spectral density is the same for all frequencies of
interest in most communication systems
A thermal noise source emanates an equal amount of noise
power per unit bandwidth at all frequencies—from dc to about
1012 Hz.

Power spectral density G(f)

Autocorrelation function of white noise is

The average power P of white noise if infinite
47
WHITE NOISE
48
WHITE NOISE
Since Rw( T) = 0 for T = 0, any two different
samples of white noise, no matter how close in
time they are taken, are uncorrelated.
 Since the noise samples of white noise are
uncorrelated, if the noise is both white and
Gaussian (for example, thermal noise) then the
noise samples are also independent.

49
ADDITIVE WHITE GAUSSIAN NOISE
(AWGN)

The effect on the detection process of a channel with
Additive White Gaussian Noise (AWGN) is that the
noise affects each transmitted symbol independently

Such a channel is called a memoryless channel

The term “additive” means that the noise is simply
superimposed or added to the signal—that there are
no multiplicative mechanisms at work
50
RANDOM PROCESSES
SYSTEMS
AND
LINEAR

If a random
process forms
the input to a
time-invariant
linear system,
the output will
also be a
random process
51
DISTORTION LESS
TRANSMISSION
52
Remember linear and non-linear group delays in
DSP
DISTORTION LESS TRANSMISSION
What is required of a network for it to behave like an
ideal transmission line?
 The output signal from an ideal transmission line
may have some time delay and different amplitude
as compared with the input
 It must have no distortion—it must have the same
shape as the input
 For idea distortion less transmission
53
IDEAL DISTORTION LESS TRANSMISSION

The overall system response must have a constant magnitude
response

The phase shift must be linear with frequency

All of the signal’s frequency components must also arrive with
identical time delay in order to add up correctly

The time delay t0 is related to the phase shift and the radian
frequency  = 2f by

A characteristic often used to measure delay distortion of a
signal is called envelope delay or group delay, which is defined
as
54
BANDWIDTH OF DIGITAL DATA
 Baseband

Signals containing frequencies ranging from 0
to some frequency fs
 Bandpass

signals
or Passband Signals
Signals containing frequencies ranging from
fs1 to some frequency fs2
55
NOTE
Chapters/Topics from
different books
Chapter 1 from
Bernard Sklar
 Chapter 1 from Simon
Haykin
 Appendix 1 from
Digital
Communication,
Simon Haykin for
Probability

Topics to be covered on
your own
Periodic, Non-periodic
Signals
 Analog and Digital
Signals
 Ideal Filters
 Realizable filters

56
REFERENCES
Bernard Sklar
 University of Saskatchewan
 Communication System, Simon Haykin
 MIT open source lectures (Robert Gallager)

57