MEDICAL IMAGING INFORMATICS:
Lecture # 1
Basics of Medical Imaging Informatics:
Estimation Theory
Norbert Schuff
Professor of Radiology
VA Medical Center and UCSF
Norbert.schuff@ucsf.edu
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 1/31
UCSF VA
Department of
Radiology & Biomedical Imaging
What Is Medical Imaging Informatics?
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Picture Archiving and Communication System (PACS)
Imaging Informatics for the Enterprise
Image-Enabled Electronic Medical Records
Radiology Information Systems (RIS) and Hospital Information Systems (HIS)
Digital Image Acquisition
Image Processing and Enhancement
Image Data Compression
3D, Visualization and Multi-media
Speech Recognition
Computer-Aided Detection and Diagnosis (CAD).
Imaging Facilities Design
Imaging Vocabularies and Ontologies
Data-mining from medical image databases
Transforming the Radiological Interpretation Process (TRIP)[2]
DICOM, HL7 and other Standards
Workflow and Process Modeling and Simulation
Quality Assurance
Archive Integrity and Security
Teleradiology
Radiology Informatics Education
Etc.
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 2/31
UCSF VA
Department of
Radiology & Biomedical Imaging
What Is Our Focus?
Learn using computation tools to maximize information and
gain knowledge
Pro-active
Improve
Data
collection
Measurements
Imaging
Extract
information
Re-active
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 3/31
Refine
Model
knowledge
Model
Compare
with
model
UCSF VA
Department of
Radiology & Biomedical Imaging
Challenge:
Extract Maximum Information
1.
Q: How can we estimate quantities of interest from a
given set of uncertain (noise) measurements?
A: Apply estimation theory (1st lecture by Norbert)
2.
Q: How can we code the quantities?
A: Apply information theory (2nd lecture by Wang)
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 4/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Estimation Theory: Motivation Example I
Gray/White Matter Segmentation
Hypothetical Histogram
1.0
0.8
0.6
0.4
0.2
0.0
Intensity
GM/WM overlap 50:50;
Can we do better than flipping a coin?
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 5/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Estimation Theory: Motivation Example II
Goal:
Capture dynamic signal
on a static background
Courtesy of Dr. D. Feinberg Advanced MRI Technologies, Sebastopol, CA
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 6/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Basic Concepts
Suppose we have N scalar measurements : xN   x 1 , x  2  ,...x  N  
We want to determine M quantities (parameters):
Definition: An estimator is:
Error estimator:
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 7/31
T
θM  1 , 2 ,... M 
T
θ̂ j  h j  x N  ; j  M
θ  θˆ  θ; for N >M
UCSF VA
Department of
Radiology & Biomedical Imaging
Examples of Estimators
N
1
Mean Value: ˆ     x  j 
N j 1
150
Intensities (Y)
100
50
Variance
0
ˆ  ˆ 2 
N
1
x  j   ˆ 


N  1 j 1
-50
100
300
500
700
900
Measurements (x)
Amplitude:
200
Frequency:
Intensity
100
Phase:
0
Decay:
-100
100
300
500
700
Measurements
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 8/31
ˆ1
ˆ2
ˆ3
ˆ4
900
UCSF VA
Department of
Radiology & Biomedical Imaging
2
Some Desirable Properties of Estimators I:
Unbiased: Mean value of the error should be zero
E θˆ - θ = E θ  0  E θˆ  E θ
Consistent: Error estimator should decrease asymptotically as number of
measurements increase. (Mean Square Error (MSE))
MSE  E θ̂ - θ
2
 0 for large N
If estimator is biased
MSE  E θ̂ - θ - b
variance
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 9/31
2
E b
2
bias
UCSF VA
Department of
Radiology & Biomedical Imaging
Some Desirable Properties of Estimators II:
Efficient: Co-variance matrix of error should decrease asymptotically to its
minimal value, i.e. inverse of Fisher’s information matrix) for large N
  
ˆ
ˆ
Cθ  E θ-θ
θ-θ
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 10/31
T
1
 J max
UCSF VA
Department of
Radiology & Biomedical Imaging
Example:
Properties Of Estimators Mean and Variance
Mean:
1 N
1
ˆ
E    E x j   N  
N j 1
N
The sample mean is an unbiased estimator of the true mean
Variance:
E  ˆ   
2
1 N
 2 E
N j 1
x j  
2
1
2
2
 2  N 
N
N
The variance is a consistent estimator because
It approaches zero for large number of measurements.
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 11/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Least-Squares Estimation
Linear Model:
xN  HθM  v N
vN  0
Generally:
x1  h111  h12 2  v1
100
Y2
where N > M
x2  h211  h22 2  v2
80
...  ....
60
xN  hN 11  hN 2 2  vN
40
0
50
100
150
200
250
Y1
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 12/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Least-Squares Estimation
The best what we can do:
ELSE  arg min
1
vN
2
2

1
T
x

H

x

H

 N
M  N
M 
2
Minimizing ELSE with regard to  leads to
H T x N   H T H  θˆ LSE  0
θLSE   HT H  H T xn
1
•LSE is popular choice for model fitting
•Useful for obtaining a descriptive measure
But
•Makes no assumptions about distributions of data or parameters
•Has no basis for statistics
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 13/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Maximum Likelihood (ML) Estimator
We have: x N   x 1 , x  2  ,...x T  
T
Xn is random sample from a pool with certain probability distribution.
Goal: Find  that gives the most likely probability distribution underlying xN.
ˆ
ML
 arg max p  xN | θ 
Max likelihood function
ML can be found by
dy
ln p  x N | θ 
0
dθ
θ θ ML
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 14/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Example I: ML of Normal Distribution
ML function of a normal distribution
p  xN |  , 
log ML function
1st
log ML equation
MLE of the mean
2nd log ML equation
MLE of the variance
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 15/31
2
   2 
2 N /2
 1 N
2
exp  2   x  j     
 2 j 1

2
N
1 N
2
ln p  xN |  ,     ln  2   2   x  j    
2
2 j 1
2
d
1
2
ln p  xN | ˆ ML , ˆ ML


 ˆ 2
d
ML
ˆ ML
N
  x  j   ˆ   0
j 1
1 N
  x j
N j 1
d
N
1
2
ˆ
ˆ
ln
p
x
|

,






N
ML
ML
2
4
dˆ 2
2ˆ ML
4ˆ ML
ˆ
2
ML
ML
N
  x  j   ˆ   0
j 1
ML
1 N
   x  j   ˆ ML 
N j 1
UCSF VA
Department of
Radiology & Biomedical Imaging
Example II: Binominal Distribution
(Coin Toss)
Probability density function:
n= number of tosses
w= probability of success
f  y | n, w  
n!
n y
 w y 1  w 
 y ! n  y !
0.7
0.2
y
0.1
f(y|n=10,w)
0.0
0.3
0.2
0.1
0.0
1
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 16/31
2
3
4
5
N
y
6
7
8
9
10
UCSF VA
Department of
Radiology & Biomedical Imaging
Likelihood Function Of Coin Tosses
Given the observed data f (y|w=0.7, n=10) (and model), find the
parameter w that most likely produced the data.
L(w | y  7, n  10)  f  y | w  0.7, n  10
Likelihood
0.25
0.20
0.15
0.10
0.05
0.00
0.1
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 17/31
0.3
0.5
W
0.7
0.9
UCSF VA
Department of
Radiology & Biomedical Imaging
ML Estimation Of Coin Toss
Compute log likelihood function
n!
ln L  w | y   ln
 y ln w   n  y  ln 1  w
 y ! n  y !
Evaluate ML equation
d ln L  w | y  y  n  y 
 
0
dw
w 1 w
y  nw
y

 0  wMLE 
w(1  w)
n
According to the MLE principle, the PDF f(y|w=y/n) for a given n
is the distribution that is most likely to have generated the observed data
of y.
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 18/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Connection between ML and LSE
Assume:
 are independent of vN
ML and vN have the same distribution
pθ  xN | θ   pv  xN  Hθ | θ 
vN is zero mean and gaussian
ML function of a normal distribution
2
T
 1
 1
p  x N | θ   exp   2  x N  Hθ  x N  Hθ    exp   2 x N  Hθ 
 2

 2

Clearly, p(x|) is maximized when LSE is minimized
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 19/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Properties Of The ML Estimator
• is consistent: the MLE recovers asymptotically the true
parameter values that generated the data for N  inf;
• Is efficient: The MLE achieves asymptotically the
minimum error (= max. information)
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 20/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Maximum A-Posteriori (MAP) Estimator
We have random sample: x N   x 1 , x  2  ,...x T  
We also have random parameters:
T
θ  p θ
Goal: Find the most likely  (max. posterior density of ) given xN.
ˆ
MAP
 arg max p  xN | θ  p  θ 
Maximize joint density
MAO can be found by
dy


ln p  x N | θ  p  θ 
 ln p  x N | θ   ln p  θ   0
dθ
θ
θ
θ θ MAP
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 21/31
UCSF VA
Department of
Radiology & Biomedical Imaging
MAP Of Normal Distribution
We have random sample:

1 N
1 N
2
2
ln p  xN | ˆ , ˆ  p  ˆ , ˆ    2   x  j   ˆ x   2  p  ˆ ML   0
uˆ
ˆ x j 1
ˆ μ j 1
The sample mean of MAP is:
μˆ MAP 
 2
N
  T 
2
x
2
 x j
j 1
If we do not have prior information on ,  inf or T inf
μˆ MAP  μˆ ML , μˆ LSE
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 22/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Posterior Density and Estimators
p(|x)
MSE
X
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 23/31
MAP

UCSF VA
Department of
Radiology & Biomedical Imaging
Summary
• LSE is a descriptive method to accurately fit data to a
model.
• MLE is a method to seek the probability distribution that
makes the observed data most likely.
• MAP is a method to seek the most probably parameter
value given prior information about the parameters and
the observed data.
• If the influence of prior information decreases, i.e. many
any measurements, MAP approaches MLE
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 24/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Some Priors in Imaging
•
•
•
•
•
Smoothness of the brain
Anatomical boundaries
Intensity distributions
Anatomical shapes
Physical models
– Point spread function
– Bandwidth limits
• Etc.
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 25/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Imaging Software Using MLE And MAP
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 26/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Segmentation Using MLE
A: Raw MRI
B: SPM2
C: EMS
D: HBSA
from
Habib Zaidi, et al,
NeuroImage 32
(2006) 1591 – 1607
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 27/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Brain Parcellation Using MLE
Manual
EM-Affine
EM-NonRigid
EM-Simultaneous
Kilian Maria Pohl, Disseration 1999
Prior information for Brain parcellation
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 28/31
UCSF VA
Department of
Radiology & Biomedical Imaging
MAP Estimation In Image Reconstruction
Human brain MRI. (a) The original LR data. (b) Zero-padding
interpolation. (c) SR with box-PSF. (d) SR with Gaussian-PSF.
From: A. Greenspan in
The Computer Journal Advance Access published February 19, 2008
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 29/31
UCSF VA
Department of
Radiology & Biomedical Imaging
Literature
Mathematical
•
H. Sorenson. Parameter Estimation – Principles and Problems.
Marcel Dekker (pub)1980.
Signal Processing
•
S. Kay. Fundamentals of Signal Processing – Estimation Theory.
Prentice Hall 1993.
•
L. Scharf. Statistical Signal Processing: Detection, Estimation, and
Time Series Analysis. Addison-Wesley 1991.
Statistics:
•
A. Hyvarinen. Independent Component Analysis. John Wileys &
Sons. 2001.
•
New Directions in Statistical Signal Processing. From Systems to
Brain. Ed. S. Haykin. MIT Press 2007.
Medical Imaging Informatics 2009, Nschuff
Course # 170.03
Slide 30/31
UCSF VA
Department of
Radiology & Biomedical Imaging