Methods for elastic adaptation of segmented volumes of interest for adaptive radiotherapy planning Urban Malsch, Christian Frühling, Rolf Bendl Department of Medical Phyiscs, German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany Abstract In this paper we present an approach to calculate displacement vector fields between planning CT image series and verification CT image series acquired during fractionated radiotherapy. After successful calculation these vector fields can be applied to all volumes of interest segmented for initial treatment planning. This way it is possible to deduce volumes of interest which conform to the verification image without conventional time consuming segmentation. Fast generation of a precise model of the patient’s anatomy is considered to be an inevitable pre-requisite for adaptive radiotherapy, which should allow evaluation and adaptation of irradiation parameters based on verification images immediately before of each fraction. Keywords Template Matcher, Optical Flow, adaptive Radiotherapy Introduction Conformal radiotherapy together with three-dimensional inverse and forward planning has shown that precise adaptation of dose to the shape of the target volume can allow dose escalations in the tumor and in this way tumor control probability can be increased while severe side effects can be reduced simultaneously. Even if it is technically possible to adjust dose distributions very precisely to a given shape, it is still necessary to add security margins around the tumor to compensate for deviations in size, shape and position of organs during therapy. Those deviations are the result of positioning errors, organ movements due to variations in filling of hollow organs (bladder, rectum), tissue dependent reaction on radiation and motion artifacts e.g. caused by respiration during therapy. The goal of adaptive radiotherapy is to consider those deviations which are quite common during fractionated radiotherapy. Several levels of adaptivity were proposed so far. One strategy is to calculate more individualized security margins by measuring and estimating organ motions based on a couple of different image series acquired before therapy or during the first fractions. The methods described in this paper are designed to support a more complex level of adaptivity. The goal of our approach is to measure deviations before each fraction and to compensate them directly by patient repositioning or by modifying the beam setup. That means, after having acquired a verification image series major planning steps must be repeated before each fraction. Since delineation of anatomy is a time consuming task - but always necessary to enable plan evaluation based on dose-volumehistograms - we have focused our efforts on methods to speed up this crucial preprocessing step. To be able to solve this segmentation task in a very short time frame, our strategy is to calculate deviations between verification and planning CTs and to apply the calculated displacement vector fields to the already segmented volumes of interest. The hypotheses is, that this approach of adjusting the already generated three-dimensional patient model is faster than repeated conventional delineation of the anatomy based on the verification images. To allow adjacent plan evaluation and modification before applying the daily fraction the adjustment of volumes of interest should not exceed more than 10 minutes. To solve that difficult task we have examined several concurrent strategies and we are presenting an approach which is based on a combination of two promising ones. Material and methods The first approach which we have investigated is the estimation of a motion or displacement vector field between planning and verification CTs by calculating the optical flow between the different image series [1]. While the optical flow approach tries to track all image gradient information over the time, the second approach tries to identify a restricted number of small corresponding sub-volumes (“templates”) in both data sets [2]. After calculating displacement vectors between these corresponding sub-volumes a global displacement vector field can be derived by applying some interpolation strategies. Since both methods have inherent advantages and disadvantages a combination of both should produce more reliable results. To optimize start conditions for both strategies we first apply a rigid transformation either calculated based on evaluation of stereotactic markers or on the maximization of Mutual Information [3]. Optical Flow Optical flow is an elegant method for calculating deviations between consecutive images by solving a system of partial differential equations. The equation system describes the motion by analyzing the changes in image gradient information. That means image intensities are differentiated in 3D space and time. If f(k,0) is a image at time t0 we are looking for a time dependent vector field u(k,t) which solves the equation f(k - u(k,t), 0) = f(k, t) The displacement vector field u(k,t) can be determined exactly under several conditions: 1. Image structure is linear: f (k u ) f (k ) u f 2. locally constant: 0 with u(l ) constl Be (k ) 3. sufficient image structure: Rank ( M ( f , )) 3 , with M ( f , ) f (f ) T Be ( k ) If these conditions hold, u can be estimated by: u2 f (k ) g (k ) f g 2 (f g ) with =g(k) = f(k, t) While the first two conditions are normally fulfilled, the main problem is condition three, which can also be deduced from the equation above. To get a solution we need sufficient gradient information, that means optical flow cannot be calculated in homogenous or nearly homogenous regions and only the spatial components in direction of the gradients can be determined locally. Motion estimations with optical flow are especially reliable for small movements, if movements exceed the size of moving objects corresponding gradients are not mapped properly and the calculated displacement or motion vectors are wrong. Hierarchical optical flow To overcome these problems, we have implemented a hierarchical approach. In a preprocessing step we are calculating Gaussian pyramids for both data sets [4], that means we resample the image series with lower resolutions. In images with lower resolution homogenous regions are reduced resp. eliminated and this way an coarse estimation for the displacement vectors can be established. This estimation can be used to fill in gaps where no results can be obtained at a higher resolution. Of course, flow calculations on low resolution images will introduce errors in the size as well as in the orientation of the calculated vectors but this information is used only, if it is not possible to calculate more precise values at finer resolutions. The Template Matcher The global strategy of this approach is quite different. A global displacement vector field is estimated (interpolated) based on a small number of displacement vectors which are determined by identifying small corresponding sub-volumes in both data sets. The general procedure can be divided into four individual steps: In the first step, a small number of suitable regions in the planning dataset will be identified. These regions are called “templates” [2]. In the second step, they will be continuously relocated and/or rotated, to find the best correspondence in the verification dataset. As a result of this iterative search a motion vector is assigned to each template. The pool of all motion vectors describes an elastic transformation between both datasets. Based on this incomplete vector set a displacement vector for each single voxel is interpolated in the third step. After the complete vector field is determined the volumes of interest, segmented on the planning data set are adjusted according to that vector field and this way they should fit to the new situation. The four steps will be explained in detail: Selection of suitable template positions: The quality of the outcome of this approach and its performance depends strongly on the number and position of the selected templates. A low number of templates cannot describe complex deformations. A large number of templates will result in excessive computation time. Unsuitable regions cannot be reliably matched and will result in wrong movement vectors. Our strategy is to prefer positions on the edge of organs or tissues, since their movements can be observed more easily. To avoid clustering of templates, a minimal distance between the templates is required. Templates inside organs or inside regions with homogenous intensity are difficult to match and are rejected. Since it can be assumed that segmentations on the initial planning data set was based on perceivable image structures, the corresponding points are promising candidates. But of course a restriction to only those points will often not sufficiently describe a complex transformation. A raw identification and localization of edges of organs or tissues borders is a prerequisite for template positioning. For that purpose we have developed a fast algorithm which identifies edges between tissue classes. Different types of tissue (fat, muscle, bone, etc) are classified according to the associated Hounsfield-units in the CT image series using either static or dynamic thresholds, calculated on a global intensity histogram. To generalize this registration method and to use it to register series acquired with other image modalities corresponding intensity levels must be determined. After the classification of tissues, edges are easily identified by evaluating the proximity of each voxel. An edge voxel is characterized by less than four (or eight) neighbors within the same tissue class. Upon identification of edges template positions are determined. To get a reasonable number of templates and to avoid accumulations, the condition of the minimal distance between templates was introduced. That means a candidate position is added only to the list of templates, if no other template in a given neighborhood was already selected. To reduce the number of templates further, possible template positions are neglected, where only minimal motion is observable. Two methods for a fast estimation of movements were implemented: The first method compares the perpendicular intensity profiles of corresponding edges in both image sequences and identifies that position where the correlation between both profiles becomes maximal. The necessary translation is used as an estimation of motion. The second method simply identifies the distance to the next edge with the same tissue class in the second dataset. By using this information the number of templates can be reduced significantly, but due to reasons during final interpolation which are described below, it is recommended to keep some of the templates in regions where no movements can be observed. The last criterion in selecting template positions are the local contrast of possible candidates. To be able to identify them easily in the second image the candidates with a high local contrast should be preferred. Bladder Left femur Right femur Rectum Figure 1: Template-matcher. A original planning dataset, B original verification dataset (one slice of a 256×256×52 CT-Volume, which shows bladder, prostate, rectum and both femurs). C a raw segmentation of edges from A. The bright dots highlight template positions. D the segmented edges from B. E is the unmatched superposition of the contours displayed in D onto the image displayed in A. F shows the edges of the verification dataset D projected onto the transformed planning data set. Bladder and muscles fit quite well, rectum was adjusted but shows still some deviations, largest deviations at femurs due to missing placement of templates. Search for corresponding template positions: After the initial template selection, the corresponding regions in the verification dataset must be identified. We use a local correlation coefficient as a similarity measure. Ax A B x B T CC i xi TA , B i T Ax A B x B 2 xi TA , B i T xi TA , B i T 2 The correlation coefficient will reach its maximum in case of positive linear dependency (dark/bright regions in image BT are projected to dark/bright regions in image A). In an iterative optimization procedure each template is translated and rotated until the similarity measure reaches a maximum. Therefore an optimization in six variables (tx, ty, tz, and rx, ry, rz) must be performed. In our implementation we use Powell’s approach to determine that point where our optimization function will reach its maximum [5]. Transformation vectors for each voxel: After the motion vectors for all templates are found, for all other voxels interpolated vectors can be determined with the help of Bookstein’s thin-plate spline algorithm [6, 7] and based on the resulting vector field the planning cube can be elastically deformed to match the verification image. N predetermined anchor points p i (the template positions in the planning data set) and N corresponding target points v i (verification data set) are given. Due to the interpolation, each voxel r of the resultant cube will receive a new gray value F (r ) ( Fx (r ), Fy (r ), Fz (r )) . Where each component (k = x, y, z) is calculated as Fk (r ) ai ,k u r pi . N Since our first results have identified a series of starting points for further enhancements a more detailed evaluation on various patient data sets is deferred until this work is done. i 1 u is a specific base function R3R1 describing the nature of the thin-plate splines and ai,k are the coefficients, which are calculated by solving the three equation systems Rk (with k = x,y,z): Rk ( pi ) a j ,k u pi p j vi ,k . N j 1 Where p i are the i=1...N anchor points and vi,k the three components of the N target points. The base function u( t ) is defined according to [6]: u( t ) = || t || which describes the minimization of the bending energy of thin-plate splines in the 3D case. The thin-plate spline interpolation is global in nature and one can not avoid the global effect of each template. Among others, this means that the transformation of each voxel is affected of each anchor point. This is easy to understand, since the number of coefficients ai is equal to the number of templates. To avoid that image structures, which haven’t changed their position in both data sets, are moved due to the influence of remote templates, it is necessary to add templates in regions with no observable motion. This way a fixation of these structures at their initial positions can be assured. Results The result of the template matcher depends largely on the number of templates. The required number of templates depends on the size of the data set and on the occurred deformations. With our test data sets (~ 50 slices, each with 256×256 pixels) we have observed acceptable correction of the occurred motions when using about 1000 or more templates. A sample of achievable results is displayed in figure 1. The necessary processing time varies from ranges from 8 to 10 minutes. The necessary processing time is at the upper tolerance level. If we reduce the number of templates, larger deviations between the transformed planning data set and the verification image are observed. Therefore current work is focused on less time consuming interpolation methods which should allow us to use more templates. We expect a more noticeable gain in performance from synergy effects by using motion estimations when searching corresponding template positions. The current implementation does not consider any a priori information about occurred movements. By using results of optical flow calculations on resampled images with coarse resolution a first hint about possible motion direction can be introduced which should improve the starting point for the optimization step. The hierarchical optical flow algorithm as stand-alone method doesn’t always seem be able to generate a reliable result. Here the quality depends heavily on the frequency of which it is necessary to fall back to the motion calculations in low resolution images. 1000 900 800 700 600 500 400 300 200 100 0 Correspondence Bookstein Coefficients Transformation 100 33 1 15 Sum 50 320 121 1 40 161 620 211 30 83 294 1350 394 79 162 635 1720 497 86 193 863 Figure 2: Calculation time (in s) for selected steps of the template matcher in dependency of the number of templates. The calculation time of the Bookstein coefficients ai,k increase superproportional with the number of templates Discussion In this contribution we present an approach to calculate displacement vector fields between planning and verification CTs based on optical flow calculations and based on a template matching strategy. After successful calculation these vector fields can be applied to deform already segmented anatomical structures. This way it is not necessary to perform conventional segmentation on verification image series in fractionated radiotherapy. The run time of the approach is currently at the upper limit of the tolerated range. Therefore current work is focused on methods resulting in an increase of performance to get additional possibilities for enhancements in quality, too. References [1] Horn B K P; Schunk B G, 1981 Determining optical flow Artificial Intelligent 17 185-204 [2] Rösch, P 2001 Template Selection and Rejection for Robust Non-Rigid 3D Registration in the Presence of large deformations Proceedings of the SPIE 545–556 [3] Maes et al 1997 Multimodality Image Registration by Maximization of Mutual Information. IEEE Transactions on Medical Imaging, Vol. 16, No. 2: 187 - 198 [4] Likar B, Pernus F, 1999 A Hierarchical Approach to Elastic Registration Based on Mutual Information. Image and Vision Computing, pages 33-44. [5] Press W H, Teukolsky S A, 1995 Numerical Recipies in C Cambridge University Press 545–556 [6] Bookstein F, 1989 Principal Warps: Thin-Plate Splines and the Decomposition of Deformations IEEE Transaction on Pattern Analysis and Machine Intelligence 11 567–585 [7] Arad N, Resifeld D, 1995 Image Warping Using few Anchor Points and Radial Basis Functions Computer Graphics Forum 14 1:35–46