V/SLAN:Combining Intra-Operative Video and
Pre-Operative Images for Surgical Guidance
From: AAAI Technical Report SS-94-05. Compilation copyright © 1994, AAAI (www.aaai.org). All rights reserved.
A.C.F. Colchester,t R.L. Evans,3 J. Zhao,t DJ. Hawkes,2C.Studholme,t D.L.G.Hill, 2 3P. Roberts,
M. Stephens,3 N. Maitland,3 AJ. Strong,4 Mj. Gleeson,s D.G. Thomas,6 2and T.C.S. Cox.
Departmentsof 1Neurology, 2Radiological Sciences, & 5Otolaryngology, UMDS,
Guy’s Hospital, LondonBridge SEI
9RT;
3Roke ManorResearch Ltd, Romsey, Hants SO510ZN;
4Departmentof Neurosurgery, MaudsleyHospital, De Crespigny Park, LondonSE5 8AF;
6Department Neurosurgery, National Hospital for Neurology and Neurosurgery, QueenSquare, LondonWC1.
Introduction
In order to approachhis target safely and accurately a
surgeon needs to relate information derived from preoperative imagesto the live patient in theatre, and he
also needs to integrate intra-operative imaging
information (video, ultrasound etc) to update his
surgical plan. Weare developing a novel frameless
stereotactic system (P’/$LAN) which uses v~ible
landmarks,for registering data obtainedpre-operatively
(the pre-operative patient representation or POPR)to
data obtained from stereo video imagingin theatre (the
intra-operative patient representation or IOPR).
An important componentof the system is a hand-held
pointing device (locater), without mechanical
electrical linkages, whichcan be tracked by the stereo
video system.
Method
Thescalp andface of a volunteerwith frontal and vertex
balding was segmented from an MRI scan using
interactive techniques and wasrepresented as a binary
volumeone voxel thick. This is referred to as the
POPRscalp. The scalp surface was also reconstructed
from a calibrated stereo video systemdesigned for use
in theatre. The head was illuminated by a structured
light projector, and correspondingpoints in the paired
7ideo images were matched. The positions of a series
3f3-Dskin surface points werethen calculated, andthis
point set is referred to as the IOPRscalp.
l’he POPRand IOPRscalp representations were then
registered using a chamfermatchingproceduret in the
~oilowingway. A distance transform was calculated for
the POPR
scalp surface and was held as a set of voxels
shose value represented the shortest distance to the
POPR
scalp surface. A series of trial registrations was
:hen attempted, by projecting IOPRpoints into the
?OPRdistance transform. For each trial registration,
235
the distance values of the voxel addresses of the
projected points was summedto produce the ’cost’ of
the trial. The final POPR
and IOPRscalp registration
was taken as the minimum
cost pose, found by using a
genetic algorithmto refine an initial coarse estimate.
A prototype hand-held locater has been designed with
a series of distinctive marksalong the handle whichcan
be recognised and tracked by the stereo video system
under conventional illumination 2,3. Rapid movementof
the locater is detected by the systemand a ’fast-tracking’
modeis initiated. Whenthe locater is stationary, a
slower ’fine-positioning mode’is activated, whichhas
improvedaccuracy (within _+.0.7ramat the tip for the
current prototype).
Results
Goodquality scalp and face reconstructions, both from
MRimages (the POPRscalp) and from stereo video
(the IOPRscalp) were obtained (figures I and 4).
achieved excellent registration between the two data
sets, and have illustrated this by projecting the POPR
scalp (the meshin figure 2) onto one of the stereo pair
of the video images from which the IOFRscalp was
derived. The registration is a true 3-D match, and the
superimposition of figure 2 takes account of the
perspective transformation as well as optical and
electronic distortions in the video chain which were
established during the cameracalibration.
The prototype locater has been successfully tracked by
the same stereo video system. Figure 3 shows a live
video frame with detected features on the handle
overlaid with a computer-generated bright dot: this
allows rapid visual checkingthat the tracking systemis
operating correctly. A symbolic representation of the
pointer can be showncombinedwith reconstruction of
scalp, brain, tumouretc from the IOPR(figure 4). The
displays can be generated from any chosen angle.
independentlyof the video views.
patient’s skin interactively while viewingthe display, in
the exact position plannedpre-operatively.
Discussion
Wehave producedthe first full 3-Dregistration of data
from stereo video image, with pre-operative scan
reconstructions. Optical imaging of the scalp maybe
limited by the patient’s hair, but pre-operativelyat least
part of the scalp is shaved, and ,this, combinedwith
facial features for anterior and lateral surgical
approaches, is likely to provide adequate exposurefor
video registration.
This initial registration pr~edure will be used before
the mainsurgical incisions have been carried out. The
registration will then be updated by tracking features
whichremainreliably fixed in relation to the patient’s
head. The locator is not neededfor this registration.
Otherffamelessstereotactic methods4’5,6,7’8,Tcurrently
rely on the surgeonto touch fiducials with the locator
for proper registration to be achieved.Withour system,
this strategy is also available as an alternative or
addition to the use of visible landmarks. The same
video systemthat is used in the scalp extraction is also
used to track a hand-held locator or instrument, and
this potentially has muchgreater flexibility than is
possible with a mechanical device connected by an
articulated arm4’5,6. Our iocator is illuminated
9 are
passively, and no cables7, powersuppliess, or LEDs
involved.
With good quality magnetic resonance imaging the
majority of the POPR face and scalp can be
reconstructed accurately. Someproblems can arise
locally, for exampleadjacent to the ethmoid sinuses
(medialto the eyeball) wheresignal voids can arise with
certain MRsequences. The stereo video surface
reconstruction also provedreliable for the majority of
the surface, although in areas where the depth is
changing rapidly and discontinuities can arise, for
example under the eyebrow, some errors do occur.
Interestingly, this is near the area wherethe errors in
the MRdata can occur. Our system presents the
surgeon with combined displays which allow him to
checkand if necessarycorrect the registration if errors
are detected.
Given registration of the POPRand IOPR, various
combinations of objects from the POPRor the IOPR
can be displayed with each other or superimposedon
sensor data from pre- or intra-operative imaging
modalities. This is exemplified by the use of the
prototype system for craniotomyplanning and guidance.
Pre-operatively the surgeon interacts with the POPR
to
plan his surgical approach and positions the cranial
windowappropriately. The windowis handled as an
object in the POPR
whichcan be overlaid onto the live
video. This allows the surgeon to draw a line on the
236
References
1 Hill DGIO Hawkes DJ (1994) Medical image
registration using knowledgeof adjacencyof anatomical
structures. Image and Vision Computing 12(3) (in
press)
2 Evans RJ (1990) KalmanFiltering of Pose Estimates
in Applications of the RAPIDVideo Rate Tracker Proc
British MachineVision Conference, Univ. of Sheffield
Press, pp79-84.
3 Harris CG, Stennett C (1990) RAPID- A Video Rate
Object Tracker. Proc. British Machine Vision
Conference, Univ. of Sheffield Press, pp73.77
4 Guthrie BIGKaplanR, Kelly PJ (1990) Neurosurgical
stereotactic
operating arm. Stereotact. Funct.
Neurosurg. 54+55:497-500.
5 Watanabe E, Mayanagi Y, Kosugi Y, Manaka S,
Takakura K (1991) Open surgery assisted by the
neuronavigator,a stereotactic, articulated, sensitive arm.
Neurosur. 28:792-800.
6 Galloway Rio Maciunas RJ, Edwards CA (1992)
Interactive image guided neurosurgery. IEEETrans
BiomedEnging 39:1226-1231.
7 Pelizzari CA, Tan KK, Levin DN, Chen GTY’,Baiter
J (1991)Interactive 3DPatient-linage registration. Proc
IPMI’91, Colchester ACF,HawkesDJ (eds). SpringerVerlag, Heidelberg, p132-141.
8 Reinhardt HF, Gerhard A. Horstmann, Otmar Gratzl
(1993) Sonic Stereometry in Microsurgical Procedures
for Deep-Seated Brain Tumors and Vascular
Malformations. Neurosurg. 32:51-57.
9 Krybus W, Knepper A" Adamsio Ruger R, MeyerEbrecht D (1991) Navigation support for surgery
meansof optical position detection. Comput.Assist.
Radiology. Lemke HURhodes MLJaffee CC Felix R
(eds), Springer -Verlag, Berlin p362.
Figure 3 A frame of live video showingthe hand held locator being tracked. The small white
dots are overlaid by the computerto indicate the recognised features.
~ll, ,
all"
Figur’e 1 Surface rendered image of scalp segmented from MR(part of the POPR).
Figure 2 Projection of the POPRscalp shownas a wire-frame meshonto one of the
stereo pair of the video images. The good registration is demonstratedby the close
match between the mesh and the video image.
238