4-1 Introduction
The Sampling Problem
Unbiased Estimators
4-2&3 Sampling Theory --The Sample Mean and Variance
Sampling Theorem
4-4 Sampling Distributions and Confidence Intervals
Student’s T-Distribution
4-6 Curve Fitting and Linear Regression
4-7 Correlation Between Two Sets of Data
Concepts
 Sample means and sample variance relation to pdf mean and variance
 Biased estimates of means and variances
 How close are the sample values to the underlying pdf values ?
 Practical curve fitting, using an NTC resistor to measure temperature.
Statistics Definition: The science of assembling, classifying, tabulating, and analyzing data or facts:
Descriptive statistics
– the collecting, grouping and presenting data in a way that can be easily understood or assimilated.
Inductive statistics
or statistical inference
– use data to draw conclusions about or estimate parameters of, the environment from which the data came from.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 1 of 19 ECE 3800

Sample Mean
X
ˆ

1 n i n 
 1
X i
E
 
ˆ

1 n i n 
 1
E
 

1 n i n 
 1
X

X
Var

E




1 n
2 i n n 
 1 j
 1
Var
 

1 n

 

  n
2

 
2
X i


X j





 
2
 

 
2 n

 2 n where  2
is the true variance of the random variable, X.
Destructive testing or sampling without replacement in a finite population results in another expression:
Var
 

 n
2

N
N

 n
1
S
2

1 n i n 
 1

X i

E S
 
 n
 n
1
  2
or for small populations where  is the true variance of the random variable.
2

E

N
N
 1
 n
 1
  n
2
To create an unbiased estimator, scale by the biasing factor to compute:
~
S
2  n n
 1

S
2  n n
 1

1 n i n 
 1

X i

2

 n
1
 1 i n 
 1

X i

X
ˆ 2
 for small populations
~
S
2 
N
N
 1
 n n
 1

S
2
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 2 of 19 ECE 3800

What is the probability that our estimates are within specified bounds … by measuring samples, can you prove that what you built or did is what was specified or promised?
When in doubt … assume Gaussian. Then, the normalized random variable becomes
(the sample mean with the mean removed, divided by the variance of the sample mean)
Z

X


X n if the true population mean is not known, it can be replaced by the sample variance
T

S
 n
X
 1

X
ˆ
~
S

X n
This distribution is defined as a Student’s t distribution with v
 n
 1 degrees of freedom.
The Student’s t probability density function (letting as f
T
 v

 v
2
 
1
 
 v
2 v
 n


1 
 t
2 v
1 , the degrees of freedom) is defined
 v
 1
2
We can then define a q% confidence interval as the interval in which the estimate will lie with a probability of q. The limits of the interval are defined as the confidence limits and q is also defined to be the confidence level.
Thus we are interested in
X
 k
  n
 
X
 k
  n where k is a constant defined as
Using the estimated sample mean and the variance of the sample mean:
X
 t n

S
 1
 
X
 t n

S
 1
X
 t

~
S n
 
X
 t

~
S n
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 3 of 19 ECE 3800

Fitting lines/curves to scatter plots.
Data provided as (x,y) pairs. Is there a function that goes through all the points? Yes …
If you want to use a polynomial of degree n-1 for n pairs! But we usually want simple curves to represent the data, like lines or parabolas, etc. where y
 a
 bx
or y
 a
 bx
 cx
2
To fit the curve we want to minimize the following function (the squared error): i n 
 1 y
 i
 a

 b
 x i
 c
 x i
2

 
2

For a linear regression (a line), we have err
 i n 
 1 y
 i

 a
 b
 x i
 2
2

To minimize for the values a and b, take the derivatives and set them equal to zero. Then solve for a and b: d da
 i n 
 1
2 
 y i

 a
 b
 x i
 
 0 d db
 i n 
 1
2 
 y i

 a
 b
 x i
 
 x i
 0
Solving results in a
 i n 
 1 y i n
 b i n 
 1 x i
and b
 n i n 
 1 y i n i n 
 1
 x i x i
2

 i n 
 1


 i n 
 1 x i x i i n 
 1


 y i
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 4 of 19 ECE 3800

d i n 
 1 y i da
 i n 
 1
2 
 y i

 a
 b
 x i
 
 0
 i n 
 1
 a
 b
 x i
  n
 a
 b
 i n 
 1 x i a
 i n 
 1 y i
 b i n 
 1 x i n

1 n


 i n 
 1 y i
 b i n 
 1 x i


Working on b d db
 i n 
 1
2 
 y i

 a
 b
 x i
 
 x i
 0 i n 
 1 y i
 x i
 a
 i n 
 1 x i
 b
 i n 
 1 x i
2
Substituting for the computation for a i n 
 1 y i
 x i



1 n
 i n 
 1 y i
 x i

1 n
 i n 
 1 y i
 i n 
 1 x i i n 
 1 y i
 b n 
 1 i x i
  b

1 n



 i n 
 1 x i



2
 b


  i n 
 1 x i
 b
 i n 
 1 x i
2 i n 
 1 x i
2  b



 i n 
 1 x i
2 
1 n



 i n 
 1 x i



2 


Isolating b b
 i n 
 1 y i i n 
 1
 x i x i
2


1 n
1 n




 i n 
 1 i n 
 1 y i
 x i


 i n 
 1
2 x i
 n
 n i n 
 1
 y i i n 
 1
 x i x i
2




 i n 
 1 i n 
 1 y i
 x i


 i n 
 1
2 x i
Now that b is determined based on the values, return to a
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 5 of 19 ECE 3800

Substituting for the computation for b into a a

1 n


 i n 
 1 y i
 b i n 
 1 x i



1 n
 






 i n 
 1 y i
 n
 i n 
 1 n
 y i i n 
 1
 x i x i
2




 i n 
 1 i n 
 1 y i
 x i



2 i n 
 1 x i
 i n 
 1 x i







 a
 i n 
 1 y i
 i n 
 1 x i
2 
1 n
 i n 
 1 y i



 i n 
 1 x i



2 n
 i n 
 1 x i
2
 i n 
 1



 i n 
 1 y i
 x i x i



2
 i n 
 1 x i

1 n
 i n 
 1 y i



 i n 
 1 x i



2
Therefore a becomes a
 i n 
 1 y i
 i n 
 1 x i
2 n
 i n 
 1 x i
2
 i n 
 1



 i n 
 1 x i
 x i i n 
 1



2 y i
 x i
Alternate formulation using the computed sample means of x and y a
 b

Y
ˆ
 

1 n
1 n

 i n 
 1 y i
 x i

Y
ˆ
1 n
 i n 
 1 x i
2 
 
2 i n 
 1
1
 n x i
2 i n 
 1
 x i
2
 


1 n
 

2 i n 
 1 y i
 x i
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 6 of 19 ECE 3800

%%
% Figure 4_5
% clear; close all ; x=(0:0.5:10)';
% Linear Curve values y=a*c+b a=2; b=4; yref = a+b*x;
% Random noise added to the line ydata = yref + 5*randn(size(x)); figure plot(x,ydata, 'x' ,x,yref) legend( 'Data' , 'Ref Line' ) meanx=mean(x); meany=mean(ydata); meanxsq = mean(x.^2); meanysq = mean(ydata.^2); meancorr = mean(x.*ydata);
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 7 of 19 ECE 3800

aest_equ = (meany*meanxsq-meanx*meancorr)/(meanxsq-meanx^2); best_equ = (meancorr-meany*meanx)/(meanxsq-meanx^2); yest_equ = aest_equ + best_equ*x; p=polyfit(x,ydata,1); aest = p(2); best = p(1); yest = polyval(p,x); figure plot(x,ydata, 'bo' ,x,yref, 'k' ,x,yest, 'r' ,x,yest_equ, 'm' ); legend( 'Data' , 'Ref Line' , 'Polyfit Line' , 'Equ Line' ) fprintf( 'Computation error\n' ) max(abs(yest_equ-yest)) rxy = meancorr/sqrt(meanxsq*meanysq)
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 8 of 19 ECE 3800

i 1 2 3 4 5 6 7 8 9 10
Figure 4-6
450
400
350
300
250
200
150
100
50
V=426.05+-0.654015*x+-0.0333712*x
2
0
10 20 30 40 50 60
Temperature (in C)
70 p=polyfit(x,y,2); a = p(3); b = p(2); c = p(1); z = a + b*x + c*x.^2; figure plot(x,y, 'bo' ,x,z, 'r' ); xlabel( 'Temperature (in C)' ) ylabel( 'Breakdown Voltage' ) title( 'Figure 4-6' ) grid atxt=sprintf( 'V=%g+%g*x+%g*x^2' ,a,b,c); text(50,375,atxt);
80 90 100
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 9 of 19 ECE 3800

For a single random variable, we have defined measures of the relationship of one sample or event and the next. These are the means and moments and the variance.
Mean or 1 st
Moment 2 nd
Moment
E
 
E





 x


 
 x
 i n
 1
 f
 dx


  x
 n x i



 dx E
E
 




 x
2
 f
 



 
 x
2
 i n
 1
 dx


  x
 n x i



 dx

X

E

1 n
 i n 
 1 x i
E

R
XX

1 n
 i n 
 1 x i
2
2 nd
Central Moment
E
E



X

X

2 






 x

X

2
 f
 



X

X

2 






 x

X

2
 i n 
 1
 dx


  x
 n x i



 dx
E



X

X

2 


1 n
 i n 
 1
 x i

X

2

1 n
 i n 
 1 x i
2
 2  x i

X

X
2
E



X

X

2 


1 n
 i n 
 1 x i
2

2 n

X
 i n 
 1 x i

1 n
 i n 
 1
X
2
E



X

X

2 


1 n
 i n 
 1 x i
2  2 
X
2

X
2

1 n
 i n 
 1 x i
2 
X
2

X
2 
C
XX

E



X

X

2 


1 n
 i n 
 1 x i
2 
X
2

1 n
 i n 
 1 x i
2 



1 n
 i n 
 1 x i



2
The variance is a measure of the similarity of successive samples or events with each other. How close or correlated with the others would an event be expected to be?

X
2 
C
XX

1 n
 i n 
 1 x i
2 



1 n
 i n 
 1 x i



2

R
XX
 
X
2
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 10 of 19 ECE 3800

For two sequences or paired groupings (x,y).
If we assume that every (x,y) pair is equally likely, the pmf of the functions has the same value for every pair. Repeated pairs simply sum the probability at the point. So, for correlation,
E

X

Y




 
 

 x
 y
 f x
,  dx
 dy for
 x i
, y i

pairs, i
 1 to n
we can define a pmf for each sample point as 1/n. Therefore,
E

X

Y

 n  i
 1 x
 y



  x
 x i
  n y
 y i



R
XY

E

X

Y


1 n
 i n 
 1 x i
 y i
Defining the cross correlation
E
E
 
X

X

 
Y

Y
 


 

 x

X

  
 
X

X

 
Y

Y
 
 n  i
 1


 x



X

  




 f
 
 dx
 dy
 x
 x i
  n y
 y i



E
 
X

X

 
Y

Y
 

1 n
 i n 
 1
 x i

X

  y i

Y

E
 
X

X

 
Y

Y
 

1 n
 i n 
 1
 x i
 y i
 x i

Y
 y i

X

X

Y

E
 
X

X

 
Y

Y

C
XY



E

1 n


X i n 
 1

X
 x i

 

Y y i



Y

1 n


 n
1 n


X i n 
 1

Y
 x i


1 n y i

 n


X

Y
X

Y

1 n
 n

X

Y
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 11 of 19 ECE 3800

For two sequences or paired groupings (x,y).
If we assume that every (x,y) pair is equally likely, the pmf of the functions has the same value for every pair, 1/n. Repeated pairs simply sum the probability at the point. So,
E



E


E



E




X


X


X

X
 
Y

Y

 


X Y

X

X
 
Y

Y


X

Y



 r
X

X
X


X
 
E




 

Y
X

X
1

Y



Y


Y

Y
X
Y

Y

X




1 n







 

1 n
 i n 
 1
Y




X i



Y n
 

 x

Y
1
 1







X



Y
X


 x
 i


1 n
X n 

 x i
 y i


X y
1
 i


 n
  

Y
 1 x i
Y
1 n




 x i


X y i n



Y


X

X x x i



Y
 f
 
Y x i
 x
, y i


1 n
  n


Y y i n

Y

 dx
 y
X
X


 dy
 y
Y i
X





Y
1 n

 n

X

Y

 r
 
XY

E




X


X
X

 
Y


Y
Y





1 n
 i n 
 1

 x i
 y i

X
 
Y
 
X

Y


C
X
XY
 
Y or making it fully data driven r
 
XY

E



X

X
 
Y

Y

 


X Y

1 n
 n  i
 1
1 n
 i n 
 1
 x i
 y i

 


1 n
 i n 
 1 x i





1 n
 n  i
 1 y i

 x i
2 


1 n
 i n 
 1 x i


2

1 n
 i n 
 1 y i
2 


1 n
 i n 
 1 y i


2
The text defines this as the Pearson’s r statistical measure, the linear correlation coefficient between two sets of data! from Wikipedia
Pearson product-moment correlation coefficient: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 12 of 19 ECE 3800

Then, a

Y
ˆ
 

R
R
XX
XX


 
 

2
R
XY 
Y
ˆ
 

R
XX

 

R
XY
C
XX and b

R
R
XY
XX


Y
ˆ
   
X
ˆ
2

C
C
XY  
XX
Pavlovian conditioning for sampled data … always compute the following with data x: Mean, 2 nd moment, variance ( 
X
, R
XX
, and

X
) y: Mean, 2 nd moment, variance ( 
Y
,
R
YY
, and

Y
) x and y:
R
XY
,
C
XY
, and

XY

X
2

C
XX

X

E

1 n
 i n 
 1 x i

E
1 n

 i n 
 1 x i
2
R
XX





1 n
1 n
 i
 n 
 1 i n 
 1 x i
2 x i



2

R
XX
C
XY
 
X
2
R
XY

E

X

Y


1 n
 i n 
 1 x i
 y i

E
 
X

X

 
Y

Y
 

1 n
 i n 
 1
 x i
 y i
  
X

Y

XY


C
X
XY
 
Y
For more information:
Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering,
3rd ed.”, Pearson Prentice Hall, Upper Saddle River, NJ, 2008. Chap. 8.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 13 of 19 ECE 3800

Based on Vishay BCComponents, Resistor Products Application Note, Document Number:
29053, 24 May, 2012.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 14 of 19 ECE 3800

Note: The B constant will be called a K constant in the following material.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 15 of 19 ECE 3800

The data is an exponential curve with respect to temperature.
NTC Resistors typically referenced to 25° C or 298.15° K.
For the 1 st order approximation, assume
R
 

R
 
 exp
K



1
T
2

1
T
1


Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 16 of 19 ECE 3800

Plotting the resist versus temperature based on the data and some approximations, we have.
4.00E+05
3.50E+05
3.00E+05
2.50E+05
2.00E+05
1.50E+05
1.00E+05
5.00E+04
0.00E+00
0
20 40 60
Temperture   (deg   C)
80 100
Data   Sheet
K25/85
K25/60
See Excel Spread Sheet for values
Typically, data sheets provide K values based on 25° C and 85° C.
K
 ln


R
R
 
 




1
T
2

1
T
1


 4190
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 17 of 19 ECE 3800

For better accuracy within a critical region, the K can be computed to bound desired temperature operating points.
For Sunseeker, key temperatures for battery operation are 45° C and 60° C. Therefore a K based on 25° C and 60° C is sufficient for operation. This resulted in a portion of the spread sheet analysis.
EPCOS NTC Thermistor Application Notes, Feb. 2009.
A reference current or voltage is required. In this case a known voltage is provided to a resistor divider and the output voltage is indicative of the temperature.
The resulting curve is highly non-linear due to the exponential nature of the device. To “linearize the curve” and reduce the steepest part of the curve, place the NTC in parallel with a large resistor.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 18 of 19 ECE 3800

For Sunseeker:
A 2.5 Vref drives the resistor divider. The Upper value used is 100k Ω and the resistor in parallel with the NTC thermistor is 330k Ω . An inverting op-amp is not used, we are directly connected to a 24-bit ADC.
The resulting voltage to temperature curve is
1.8000
1.6000
1.4000
1.2000
1.0000
0.8000
0.6000
0.4000
0.2000
0.0000
0 20 40 60 80 100
See the spread sheet for the expected ADC outputs and hexadecimal digital values.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System
Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
B.J. Bazuin, Spring 2015 19 of 19 ECE 3800