Multiplexed
Fluorescence Unmixing
Marina Alterman, Yoav Schechner
Technion, Israel
Aryeh Weiss
Bar-Ilan, Israel
2
Natural Linear Mixing
i = Xc
i
c
c
i
Raskar et al. 2006.
i
c
ImageJ image sample
collection.
3
Natural Linear Mixing
i = Xc
?
i
c
c
i
Raskar et al. 2006.
i
How do you
measure i?
c
ImageJ image sample
collection.
Single Source
Excitation
demultiplex
i1
1
1
22
4
Multiplexed
Excitation
3
a1
1 1 0
Beam
combiner 1
1
i2
2
33
2
1 1 0
a2
1 0 1
1 0 1
a3
0 1 1
1
1
3
2
i3
3
3
1 1 0
2
Why Multiplexing?
i
+ noise
Intensity vector
Trivial
Measurements
Multiplexed
Measurements
SNR
SNR
Same
acquisition
time
5
6
i – single source
intensities
η - noise
Multiplexing - Look closer
  
  
acquisition  a   
  
  
estimation
 

 

 ˆi  

  
  
 
W
 
 i 

Xc
 
 
 
W
Estimate c
not i
var (h )





1
 
 
a
 
 
 
V ar 
ˆi

2
N sources
1
trace   W W  


T
Minimum  W=?
Multiplexing: a=Wi,
Mixing: i=Xc
Common Approach
Acquired
multiplexed
intensities
7
This Work
Single
source
intensities Concentrations
ˆi
a
Wi
size(i)=7
cˆ
a
Wi≠Wc
Ndyes=3
Nsources=7
ˆi
cˆ
Wc
efficient acquisition
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
8
Fluorescence
Cell structure and
processes
Fluorescent
Specimen
Horse Dermal Fibroblast Cells
Intestine Tissue
Corn Grain
Flea
http://www.microscopyu.com/galleries/fluorescence, http://www.microscopy.fsu.edu/primer/techniques/fluorescence/fluorogallery.html
9
Linear Mixing
i
c
More
molecules
per pixel
Brighter
pixel
Molecules
per pixel
c
i i=µx∙c
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
10
Linear Mixing
i
{cd}
For each pixel:
i = x1
x 2 ∙ ∙ ∙ xN
dyes
c1
c2
∙∙
∙
c Ndyes
vector of
concentrations
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
(spatial distribution)
11
Linear Mixing
i1
s=1
s=2
{cd}
For each pixel:
i2
{cd}
i 1= x 1,1 x 1,2∙ ∙ ∙ x1,N
dyes
i 2= x 2,1 x 2,2∙ ∙ ∙ x2,N
∙∙
∙
dyes
i s= x s,1 x s,2 ∙ ∙ ∙ x s,N
c1
c2
∙∙
∙
c Ndyes
dyes
vector of
intensities
Mixing matrix
vector of
concentrations
(spatial distribution)
12
Linear Mixing
i1
s=1
{cd}
i2
s=2
{cd}
For each pixel:
i = Xc
vector of
intensities
Mixing matrix
vector of
concentrations
(spatial distribution)
Fluorescent Microscope
13
unmix1
Intensity image
e(λ)
300 400 500 600 700
Emission
Filter
λ
s= 1
s= 2
s= 3
e(λ)
300 400 500 600 700
s= 4
Dichroic
Mirror
λ
L2(λ)
s= 5
Excitation
Filter
Fluorescent
Specimen
unmix1
Excitation
Sources
s: illumination sources
Blue
α(λ)
300 400 500 600 700
d=1
λ
Fluorescent Microscope
14
unmix1
Intensity image
(mixed)
e(λ)
300 400 500 600 700
Emission
Filter
λ
s= 1
s= 2
Cross-talk
s= 3
e(λ)
s= 4
300 400 500 600 700
unmix2
Dichroic
Mirror
λ
L2(λ)
Excitation
Filter
Fluorescent
Specimen
unmix1
Green
s= 5
s: illumination sources
Blue
α(λ)
Cross-talk
300 400 500 600 700
d=2
Excitation
Sources
d=1
λ
Problem Definition
Fluorescent
specimen
unmix1
mix
Intensity image
(mixed)
Unmix
How to multiplex for
least noisy unmixing?
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
unmix2
15
16
Sum up the concepts
Acquired multiplexed
image intensities
Single source
Image intensities
W
a
X
multiplexing
demultiplexing
-1
W
i
Concentrations
mixing
c
unmixing
X-1
multiplexed unmixing
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
17
Look closer - again
  
  
a 
  
  
  
W
i – single source intensities
η - noise
 
 
i 
Xc
 
 
 
Estimate c not i
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
Multiplexed Unmixing
acquisition
acquired
measurements
For
each
pixel
i
noise
W
a =
18
multiplexing
matrix
X
c + η
mixing
matrix
estimation
  
  
cˆ 
  
  
  
W X





1
WX is not square
 
 
a
 
 
 
OR Weighted Least OR
Squares
Minimum Variance in c  W=?
Other
estimators
19
Generalizations
Minimum Var  W=?
Image intensities
i =?

c =?
2
V ar 
ˆi
N sources
var(η) = constant
1
T

trace  W W  


var (h )
concentrations
V ar 
var(η) = constant
η - noise
var (h )
cˆ


2
N dyes
trace  ( W X ) W X 
T
V ar 
cˆ
N dyes

trace   W X  

Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

co v (h )
c =? var(η) ≠ constant
1
1
T
1
noise
 W X  
1

20
Generalized Multiplex Gain
What is the SNR gain for unmixing?
Only
Unmixing
VS.
Unmixing
+
Multiplexing
trivial
G A IN c =
V arˆ
c
V arˆ
c
W = I
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
21
Significance of the Model
GAINc
2.2
a
2
1.8
ˆi
Wc
Wi≠Wc
cˆ
VS.
1.6
1.4
ˆi
a
1.2
ˆc
Wi
1
3
4
5
6
7
Nmeasure=Nsources
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
22
Significance of the Model
GAINc
2.2
2
1.8
1.6
1.4
1.2
1
3
4
5
6
Nmeasure=Nsources
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
7
23
Significance of the Model
GAINc
2.2
2
1.8
ˆi
a
1.6
cˆ
Wi
1.4
1.2
1
3
GAIN < 1
4
5
6
Nmeasure=Nsources
For specific 3 dyes, camera and filter characteristics
7
24
Natural Linear Mixing
i = Xc
?
i
c
c
i
Raskar et al. 2006.
i
c
a = Wi
ImageJ image sample
collection.
25
Multiplexed
Unmixing
Generalization of multiplexing theory
  
  
a 
  
  
  
 
 

Xc
i
 
 
 
W
The goal is unmixing
SNR improvement
Efficient Acquisition
Exploit all available sources
a =
W
Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing
X
c
+ η