3D Optical Flow Method Implementation for Mapping of 3D
Anatomical Structure Contours across 4D CT Data
Geoffrey Zhang1, Thomas Guerrero2, Tzung-Chi Huang1,3, Mathew J Fitzpatrick1, Geoffrey Ibbott1, Kang-Ping
Lin3, George Starkschall1
1
Department of Radiation Physics, The University of Texas M. D. Anderson Cancer Center, Houston, TX 77030, USA , 2 Department of
Radiation Oncology, The University of Texas M. D. Anderson Cancer Center, Houston, TX 77030, USA, 3 Department of Electrical
Engineering, Chung-Yuan University, Taiwan, ROC
Abstract
A 3D optical flow program that includes a multi-resolution feature has been developed and applied to 3D anatomical structure
contour mapping for 4D CT data. The study includes contour mapping for real patient CT data sets, and also for a thoracic
phantom in which the displacement for each voxel is known. The optical flow calculation agrees to within 1 mm with the known
displacement.
Keywords
3D Optical flow, deformable image registration, 3D treatment planning systems, radiotherapy
Introduction
Normal anatomical structures change during radiation
treatment due to respiratory motion. Manually contouring
these structures for treatment planning is a time consuming job,
especially for multiple phases of 4D CT images acquired at
different respiratory levels [1]. We propose the use of a
deformable image registration algorithm based on an optical
flow method to assist in the contouring process.
The basic principle of the method is that a 3D CT image data
set acquired at one phase of the respiratory cycle is deformed
to the data set acquired at another phase. The deformation
matrix is then applied to contours delineated on the original
image data set to yield contours on the second data set. The
optical flow method elastically maps images on a voxel-tovoxel basis. It differs from other methods by its ease of use
and precision in mapping structures of interest. There is no
user intervention required to select matching control points and
the entire image volume is mapped in one step.
Many different optical flow algorithms have been described [210]. Theoretical differences aside, most can be classified
through their trade-off in speed and accuracy of
implementation [11]. A 3D optical flow program has been
developed and validated at The University of Texas M. D.
Anderson Cancer Center in collaboration with Chung-Yuan
University based on an extended Horn and Schunck’s
gradient-based algorithm [2,12].
Material and methods
3D Optical Flow
Like most motion estimation algorithms based on image
intensities, the basic assumption in this 3D optical flow
algorithm is that the intensity of any infinitesimal volume
changes little with time, indicating that the material is
incompressible. This is expressed mathematically as [2,3,810,13],
f x  dx, y  dy, z  dz, t  dt   f x, y, z, t  ,
(1)
where f(x,y,z,t) denotes the image intensity at a point (x,y,z) at
time t. Expanding the left hand side in a Taylor series
f  x  dx, y  dy, z  dz , t  dt 
 f  x, y , z , t  
f  x, y, z , t 
x
f  x, y, z , t 
t
dx 
f  x, y, z , t 
y
dy 
f  x, y , z , t 
z
dz 
(2)
dt  higher order terms
and substituting equation(1) into (2) yields the optical flow
equation:
f x, y, z, t 
f x, y, z, t 
f x, y, z, t 
f x, y, z, t 
vx 
vy 
vz 
0
x
y
z
t
(3)
where vx = dx/dt, vy = dy/dt, vz = dz/dt, are originally defined
as the three components of the velocity that describe the spatial
change rate of the voxel with respect to time. They are actually
the three components of the spatial displacement of the voxel
between the two image sets involved in an optical flow
calculation. The optical flow calculation determines these
three components for each voxel.
To solve equation (3) for the three components of the velocity,
Horn & Schunck’s velocity smoothness constraint is extended
to 3D to minimize the Laplacians of the three components:
2
 vx   vx   vx 
 x    y    z  

 

 
2
2
 v y   v y   v y 

 
 
 

x

y

z

 
 

2
2
2
2
(4)
 vz   vz   vz 
2
 

 
  s
 x   y   z 
2
2
multi-resolution technique, with larger voxels at lower
resolution. The registration starts at a user given resolution
level that is a 2 to the nth power multiple of the original
resolution, and increases hierarchically until the finest
resolution is achieved. Figure 1 shows the flow chart of the
multi-resolution feature. With the multi-resolution feature,
optical flow is more suitable to radiotherapy image registration
applications where relatively large image changes may often
occur.
To allow some intensity variation between images, another
non-zero term is introduced:
f  x, y, z, t 
x
vx 
f  x, y, z, t 
y
vy 
f  x, y, z, t 
z
vz 
f  x, y, z , t 
t
  of
(5)
The weighted contribution of the errors,  of2 and  s2 , over the
image volume, V, is the total error,  , to be minimized. Thus
to obtain the velocity solution for each voxel the quantity


2
    of
  2 s2 dx dy dz
(6)
Figure 1: Multi-resolution flow chart. In the figure, restart optical
V
where  is interpreted as a weighting factor, is minimized
through variation.
Applying variational calculus, the three velocity components
are calculated using three Gauss-Seidel iterations:
vx n 1
  n  f
f 
 n  f
 n  f
 vx x  v y y  vz z  t 
f 

 vx n  
2
2
2
x
 f   f   f 
2      
 x   y   z 
v yn 1
  n  f
f 
 n  f
 n  f
 vx x  v y y  vz z  t 
f 

 v yn  
2
2
2
y
 f   f   f 
2      
 x   y   z 
vz n 1
  n  f
f 
 n  f
 n  f
 vx x  v y y  vz z  t 
f 

 vz n  
2
2
2
z
 f   f   f 
2      
 x   y   z 
(7)
0
0
 0
The initial vx  , v y and vz  are set to 0. The intensity
gradient components in equation (7) are calculated beforehand
by averaging a forward difference in a finite volume.
An optical flow calculation requires input of a source image
and a target image. The output from the calculation includes
the 3D velocity matrix data file equal to each voxel’s
displacement and an estimated image to assemble the target
image..
Originally, the optical flow method could only handle very
small displacements, less than one voxel differece,, limiting its
applications. This problem was solved by implementing a
flow means starting the registration with the velocity matrix resulted
from previous resolution level
4D Image Data
To test this algorithm, two sets of 4D CT image data sets were
used. One set was acquired from a patient involved in an
I.R.B.-approved clinical protocol, while the second set was
acquired of a thoracic phantom developed at the M.D.
Anderson Radiological Physics Center (RPC) [14] placed on a
table that allowed programmable 1D motion. All data sets
were acquired on a commercial multislice helical CT scanner
(MX8000 IDT, Philips Medical Systems, Cleveland, OH).
A physician contoured anatomical structures on one of the 4D
data sets, usually the set obtained at end expiration. The
original contours were then elastically mapped to all the image
sets of other respiratory levels using the velocity matrices
calculated from the image registration between the contoured
image set to other sets using optical flow.
Results and discussion
Optical flow has been applied in structure contour mapping for
the 4D gated CT scan data of the RPC thoracic phantom on a
motion table that moved in one direction repetitively. The
motion distance was set to be 17.5 mm in the SI direction. In
this case, only 1D translation was involved. Contours of lungs,
heart and spinal cord were mapped from the contoured image
to all other images. Figure 2 shows an example of heart contour
mapping in coronal view. The displacement of each voxel
inside the contours was calculated using the velocity matrix.
Figure 3 shows the histogram of the right lung displacement.
The mean superior-inferior (SI) displacement of the contoured
volume agrees with the known displacement very well. The
estimated RMS SI displacement for the right lung was 16.8
mm while the SI displacement of the two CT scans is 17.5 mm.
Due to lack of intensity variation inside the phantom lung, the
aperture effect [15] introduces some displacement errors for
some voxels. The 0.7 mm difference between the calculation
and reality is quite reasonable considering the CT slice
thickness is 3 mm.
References
[1] Keall, P.J., Starkschall, G., Shukla, H., Forster, K.M., Ortiz,
V., Stevens, C., Vedam, S.S., George, R., Guerrero, T.,
Mohan, R. 2003 Acquiring 4D thoracic CT scans using a
multislice helical method. Phys Med Biol submitted for
publication.
[2] Horn, B.K.P. & Schunck, B.G. 1981 Determining optical
flow. Artif. Intell. 17, 185-203
[3] Nagel, H. & Enkelmann, W. 1986 An investigation of
smoothness constraints for the estimation of displacement
vector fields from image sequences. IEEE Trans. Pattern
Analysis Machine Intell. 8, 565-593
Figure 2: Overlaid coronal view of original and mapped heart
contour images of the RPC phantom. The contour follows the
phantom motion precisely.
[4] Fleet, D.J. & Jepson, A.L. 1990 Computation of component
image velocity form local phase information, Int. J.
Computer Vision 5, No.1, 77-104
[5] Camus, T. 1997 Real-time quantized optical flow, J. RealTime Imageing 3, 71-86
[6] Lucas , B.D. & Kanade, T. 1981 An iterative image
registration technique with an application to stereo vision,
Proceedings of the DARPA Image Understanding Workshop,
121-130
[7] Uras, S.; Girosi, F.; Verri, A. & Torre, V. 1988 A
computational approach to motion perception, Biol.
Cybernetics 60, 79-97
[8] Fitzpatrick, J.M. 1988 The existence of geometrical
density-image transformations corresponding to object
motion. Computer Vision Graphics Image Processing 44,
155-174
[9] Kearney, J.K. & Thompson, W.B. 1987 Optical flow
estimation: an error analysis of gradient-based methods with
local optimisation. IEEE Trans. Pattern Analysis Machine
Intell.9, 229-244
[10] Song, S.M. & Leahy, R.M. 1991 Computation of 3D
velocity fields from 3D Cine CT images of a human heart,
IEEE Trans. Med. Imaging 10, 295-306
Figure 3: Histogram of calculated displacement in contoured right
lung in the RPC phantom.
A set of real patient data is also included in the study. Contours
of lungs, heart and esophagus are mapped from the end
expiration image to the end inspiration image. Figure 4 shows
an example of the esophagus contour mapping, and Figure 5,
the right lung. Note, in particular, the accuracy of the lung
contour obtained without the use of thresholding methods.
[11] Liu, H.; Hong, T.; Herman, M.; Camus, T. & Chellappa, R.
1998 Accuracy vs efficiency trade-offs in optical flow
algorithms, Computer Vision and Image Understanding 72
No. 3, 271-286
[12] Zhang, G.; Guerrero, T.; Huang, T.C; Lin, K.P.;
Starkschall, G. 2003 Semi-automated CT segmentation
using deformable image registration, submitted to Int. J. of
Rad. Oncology Biology Phys.
Conclusion
[13] Zhou, Z.; Synolakis, C.E.; Leahy, R.M. & Song, S.M.
1995 Calculation of 3D internal displacement fields from
3D X-ray computer tomographic images, Proc. R. Soc.Lond.
A 449 537-554
A deformable image registration matrix, describing the
deformation of a 3D CT image data set from one phase of the
respiratory cycle to the other, obtained by use of an optical
flow algorithm can be used to generate a set of contours of
normal anatomic structures in all phases of a 4D CT image data
set.
[14] Cherry, C.P.D., et. al., 2000. Design of a heterogeneous
thorax phantom for remote verification of three-dimensional
conformal radiotherapy, Med Phys., 27, p1400
[15] Beauchemin, S.S. and Barron, J.L. 1995 The computation
of optical flow. ACM Computing Surveys, 27(5): p. 433 466.
Figure 4: Esophagus contour mapping from end expiration to end inspiration CT images using optical flow. Both sets of the images shown in
this figure are picked from the same location in image coordinate system. They are not exactly the same location anatomically
Figure 5: Right lung contour mapping from end expiration to end inspiration CT images using optical flow. The image sets shown in this figure
are at the same location in image coordinate system, but not exactly the same location anatomically