Image Processing and Image Analysis for Medical Visualization (II) Bernhard Preim 1 Classification • Segmentation procedures • • • • • • • • • Introduction Point-oriented Procedures Edge-oriented Procedures Region-oriented Procedures Applications: Segmentation of organs, lesions and vessels Postprocessing with morphologic operators Skeletonization Quantitative analysis of medical image data Implementation of image processing algorithms Bernhard Preim 2 Introduction Bernhard Preim 3 Segmentation: Introduction • Goal: Identification of pixels that meet one homogeneity criterion • Why structures are segmented? • Selective highlighting and fading of structures, high-quality visualization • Basis for a classification (muscles, bones, …) • Basis for further processing (e.g. skeletonization) • Basis for a quantitative analysis (volumetry, form-describing parameters, …) Bernhard Preim 4 Segmentation: Introduction Bernhard Preim 5 Segmentation: Introduction Quality criteria: • Precision (comparison with manual segmentation by human experts) • Speed determines application areas (emergency cases, intraoperative use, …) • Robustness against noise • impacts of disruptions, such as patient movement, metal artifacts, influence of variable pathological changes • Fast feedback • Reproducibility (example: follow-up during MS lesions) Bernhard Preim 6 Segmentation: Introduction • Examples: Identification of pathological changes (tumors), different organs, vessels, bones, catheters, … • Problem: Suitability of segmentation methods strongly depends on: • the applied modalities (CT, MRT, ultrasound, …) • the specific acquisition parameters (contrast agent administration, resolution, noise, sequence, …) • the objects to be detected (thin hierarchical vessels, compact large organs) Bernhard Preim 7 Segmentation: Introduction Introduction • Point-oriented Procedures • Edge-oriented Procedures • Region-oriented Procedures • Model-based Procedures Homogeneity Critera: • grey values, gradients, textures Bernhard Preim 8 Segmentation: Point-oriented Procedures • Pixels are attributed to objects based on the gray value (thresholding) • Histogram analysis to identify suitable thresholds (minima) • Advantage: fast, simple method • Disadvantage: Neglect of spatial relations → postprocessing • Variations: local threshold methods (threshold based on a local histogram analysis) Bernhard Preim 9 Segmentation: Point-oriented Procedures • Definition of optimal thresholds by adaptation of probabiliy density distributions to the histogramm • Incorporation of model assumptions • Example: Assumption: three structures are searched for. Intensity values are normally distributed. • Approach: • 1. Estimation of parameters of the normal distribution (mean, standard deviation) based on the histogram. • 2. Estimation of thresholds to minimize the segmentation error. Bernhard Preim 10 Segmentation: Point-oriented Procedures Example: Threshold definition for the segmentation of brain structures in MRI (From: [Hahn, 2001]). Bernhard Preim 11 Segmentation: Point-oriented Procedures Adaptation of 3 Gaussian distributions to the histogram of an MRI brain dataset. Criterion: Minimal quadratic error between histogram and model function (Least Square Fit). Bernhard Preim 12 Segmentation: Point-oriented Procedures • Intensity-based segmentation and definition of regions (connected component analysis, CCA) • Application: segmentation of the kidney from CT data © Pommert [2004] Bernhard Preim 13 Segmentation: Point-oriented Procedures Another example: Bone segmentation by combining thresholding (above) and CCA (below) Bernhard Preim 14 Segmentation: Region-oriented Procedures Frequent Approaches: • Threshold-based region growing based on seed points (Region Growing, Volume Growing) • Watershed transform Bernhard Preim 15 Segmentation: Region Growing • Starting from a seed pixel (white), adjacent pixels with similar intensity are looked for • Problem: Holes in the contour Bernhard Preim 16 Segmentation: Region Growing Region growing with interactive threshold selection. Threshold-volume diagram supports the threshold detection. Strong discontinuities in the diagram are “candidates” for thresholds. (From: Boskamp et al. [2004]) Bernhard Preim 17 Segmentation: Region Growing Problem: Leakage Refinement: Masking of “disturbing” regions (pre-masking) by setting marks. (From: Boskamp et al. [2004]) Bernhard Preim 18 Segmentation: Region Growing Pre-maskings for a subsequent region growing. The target structure (left) is separated from the disturbing structure (right). Source: Boskamp et al. [2004] Bernhard Preim 19 Segmentation: Region Growing Results of vessel segmentation → Video: Hepatic_Vasculature_1, Hepatic_Vasculature_2 Bernhard Preim 20 Segmentation: Region Growing Results of vessel segmentation → Video VesselSeg3D Movie Bernhard Preim 21 Segmentation: Region Growing • Advanced approaches: • User specifies a direction for the growing process. • Thresholds are adapted to the local intensity distribution. • Consideration of gradient strength. Adding voxels terminates at voxels with high gradient magnitude. • Combination with knowledge representation. Bernhard Preim 22 Segmentation: Region Growing Combination with knowledge representation. Fitting cross sections of an ellipse to avoid leakage. Segmentation of the coronary artery. Source: Hennemuth et al. [2005] Bernhard Preim 23 Segmentation: Region Growing Result of the modified region growing segmentation (From: Hennemuth et al. [2005]) Bernhard Preim 24 Segmentation: Watershed Transform • Idea: Consider the gradient image as mountains that are flooded. Associated catchment basins correspond with one object. • Gray values as elevations; water creates a catchment basin around local minima. Watersheds are created between the catchment basins. • Each catchment basin is assigned a representative value (mean value) –> creation of a mosaic picture. • Smaller (lower) basins are merged via “flooding”. Bernhard Preim 25 Segmentation: Watershed Transform Source: Hahn et al. [2000] Bernhard Preim 26 Segmentation: Watershed Transform Example • Starting point: strong oversegmentation (due to noise in CT data) • stepwise region merging via “flooding” • image as hierarchy of regions (Schindewolf et al. [2000]) Bernhard Preim 27 Segmentation: Watershed Transform Region Merging (Merge Tree) Bernhard Preim 28 Segmentation: Watershed Transform • Exclusion of sub-areas through “exclude” (creation of a new watershed) • Inclusion of sub-areas through “include” (removal of a watershed) Source: Hahn et al. [2004] Bernhard Preim 29 Segmentation: Watershed Transform Segmentation of brain and ventricles Source: Hahn et al. [2000] Bernhard Preim 30 Segmentation: Watershed Transform Source: Hahn et al. (2003) Bernhard Preim 31 Segmentation: LiveWire • Edge-based procedure is based on a path search (Dijkstra‘s algorithm, http://www.youtube.com/watch?v=87_1K2GQFdU) • Searching of paths with minimum costs • Choose the cost function for the path from p to q such that the edge represents the object boundary • Elements of the function: • expected gray values of the target object fz • expected gradient strength fg • edge direction or changes of the direction fD • Optionally: values of the second order derivative Bernhard Preim 32 Segmentation: LiveWire • Example for a cost graph Bernhard Preim 33 Segmentation: LiveWire • Suitability of a cost function is illustrated in the cost image (dark means low costs) Bernhard Preim 34 Segmentation: LiveWire LiveWire segmentation of the liver. Will be shown in the exercise! Source: Schenk et al. (2003) Bernhard Preim 35 Segmentation: LiveWire Summary: LiveWire is an effective approach • if the target structure is clearly delineated, • if the properties of the target structure and its boundary are rather homogenous, and • If the target structure does not exhibit many identations, such as brain folds. LiveWire is an intuitive method which is controlled precisely and easily. Organ segmentation in CT is an example where it works well, the segmentation of brain structures in MRI is an example where it is not appropriate. Bernhard Preim 36/62 Segmentation: Model-based Approaches • Idea: Problem-specific know-how is incorporated as constraint • Examples: • • • • Number of objects (brain: gray and white substance, brain ventricle) Rough size of regions Intensity values and gradient strength Location of objects (in relation to other (presegmented) objects) • Parallel: voice recognition, separation of words • Knowledge about possible words (dictionaries and combinations of words are used) Bernhard Preim 37 Segmentation: Model-based Approaches • Example: atlas-based brain segmentation • Approach: • The brain of a patient is mapped to a segmented (normal) brain • Segmentation of anatomical structures is adopted • Pathological changes are structures that can not be assigned (e.g. lesions in MS patients) Bernhard Preim 38 Segmentation: Model-based Approaches • Segmentation • • • • Masses are sensors that analyze the image intensity and gradient strength For stability: Constraints for angles, springs with high stiffness Configuration of masses and springs represents the expected form Stable mass-spring models for segmentation of lymph nodes (-> video), vessels, thyroid cartilage and M. sternoclaidomastoideus J. Dornheim et al., Proc. of MICCAI 2006 and Journal of Academic Radiology 2007 Bernhard Preim 39 Segmentation: Model-based Approaches → Video: Lymph node segmentation → Video: Complete lymph node segmentation → Video: Thyroid cartilage adjustment Bernhard Preim 37 Segmentation: Model-based Approaches Segmentation of blood vessels of the neck J. Dornheim et al., Proc. of EG Workshop on VCBM Bernhard Preim → Video: NeckVesselSegmentation 41 Segmentation: Interaction • Fully automatic segmentation is rarely performed → manual input before/after application of a segmentation procedure Forms of interaction: • Correction of thresholds • Limitation of the segmentation to one region of interest (ROI) • Line drawings • Stepwise refinement (include, exclude) Postprocessing may be required! Bernhard Preim 42 3D Object Manipulation • Automatic model-based segmentation often causes partly incorrect results • Incremental and intuitive manipulation of the corresponding 3D objects • Inspiration: - general 3D object - manipulation techniques for bulging and bending of 3D objects (Igarashi‘s Teddy system, Laplacian Modelling) (Bending Tool, Master thesis, D. Proksch [2009], BVM [2010]) Bernhard Preim 43 3D Object Manipulation Bulging Tool Sketch Tool (Master thesis, D. Proksch [2009], Proksch et al. BVM [2010]) → Videos: BendingTool, BulgeTool, SketchTool, TractionTool Bernhard Preim 44 3D Object Manipulation Further improvements: • Model information shall be used during the correction → • According to the model the contour is attracted by strong gradients and other characteristics Bernhard Preim 45 Segmentation: Representation of Results Result: • Assignment of voxels to 1 or more objects • Objects may: • overlap • be hierarchically structured (lung tumor) Representation: • 1 bit per object • Segmentation image (1 bit combination per voxel that specifies to which objects the voxel belongs) Bernhard Preim 46 Overlapping Structures • Tumor voxels are also voxels of the liver parenchyma • Infiltrations Bernhard Preim 47 Overlapping Structures • Problem: no explicit voxel assignment any longer • 1st solution: • Explicit assignment of each structure to 1 bit of an 8 byte value • Limited to 64 structures Bernhard Preim 48 Multicoded Segmentation Masks Refined solution: • Serial creation of a new volume by adding a segmentation mask • Storage of voxel values in a Lookup table Bernhard Preim 49 Multicoded Segmentation Masks Small example. Classification of voxels to 4 objects. Reduction of theoretically 16 possibilities to 7 (reduction of 4 bit to 3). The reduction effect is larger for (realistically) larger examples. (Source: Mühler et al., 2010) Bernhard Preim 50 Multicoded Segmentation Masks Advantages: • Only 1 volume left • According to the number of overlappings "many" structures are storable in 1 data set • Table: 1-3 bytes required HLA = neck dissection Bernhard Preim 51 Visualization of Segmentation Results • Standard approach: use of a binary mask (0 background, 1 object) • Visualization via Marching Cubes with threshold value 0.5 • Problem: Quality, especially in case of large slice distances • Possible solutions: Interpolation of inter layers (e.g. Cubic Interpolation); high storage requirements • Instead of Marching Cubes: Cubic Interpolation Bernhard Preim 52 Visualization of Segmentation Results Arteria carotis. Slice distance: 4 mm. Middle: cubic interpolation of inter layers, right: shape-based interpolation. After interpolation: isotropic data. (Images: Tobias Mönch, Vis. group) Bernhard Preim 53 Visualization of Segmentation Results Visualization of segmentation results as contours. Quelle: Zachow [2004] Bernhard Preim 54 Visualization of Segmentation Results Brain vessels extracted from MRI data. “Normal” and enhanced interpolation. Transfer functions not fully identical. Thus, details are missing (right). Screenshot: Fernando Vega, Universität Erlangen Bernhard Preim 55 Visualization of Segmentation Results • Surface relaxation. Small deformation, which smoothes and minimizes the surface. The binary volume defines constraints (this minimizes the error). Eye with optic nerve. Source: Insight Toolkit Bernhard Preim 56 Visualization of Segmentation Results Left: Marching Cubes Right: Reduction of aliasing artifacts Brain illustration Source: Insight Toolkit Bernhard Preim 57 Morphological Operators: Erosion Processing of ultrasound data with erosion, [Sakas, 1995] Bernhard Preim 62/62 Morphological Operators: Erosion Advanced morphological processing to close holes in a bone Bernhard Preim 63 Skeletonization and Analysis of Image Data • Skeleton: form-describing structure • Discrete skeleton: 1 voxel-wide line that proceeds through the middle of the object. • Motivation: • Basis for measurement (e.g. vessel radii) • Basis for analyzing branchings • Path planning for virtual endoscopy (VL) • Basis for vessel visualization (VL 9) • Skeletonization: • Centers of maximum circular disks • Continued erosion (thinning) Bernhard Preim 64 Skeletoinzation: Illustration • Connection of the centers of maximum circular disks • Grass fire: the outer areas, which remain after burning down the grass fire, form the skeleton Bernhard Preim 65 Skeletoinzation: Approaches • Thinning: gradual erosion of the skeletonized object until a 1 voxel-wide line results. • Search for centers of maximum circular disks • Approach: • Distance transformation calculation for all object points • Determination of the centers of maximum circular disks • Connection of the located centers Bernhard Preim 66 Skeletoinzation: Problems • • • • Irrelevant side branches (b) Lines that are 2 pixels wide (c) Topology is not preserved after rotation! Many procedures are not (hardly) transferable to 3d Bernhard Preim 67 Filtering of Irrelevant Side Branches • Criteria: • Absolute length of the side branches • Ratio: length side branch/main branch • Hierarchy position (only lowest level branches can be irrelevant) Bernhard Preim 68 Quantitative Analysis of Medical Image Data Why is a quantitative analysis carried out? • In addition to a visual review • Objektivization of diagnoses and courses of disease • Decision support for continuation/termination of a therapy (chemo, hormone, drug therapy) • Therapy planning (security edges) • Geometrical modeling of implants • Visualization of the quantitative analysis (e.g. bone density) Bernhard Preim 69 Quantitative Analysis of Medical Image Data What is analyzed? • Number, dimension and volume of pathological structures (tumors, infarct areas, herds in MS patients) • Angle between anatomical structures (e.g. to assess malpositions) • Distance between pathological structures and land marks (description of localization) • Distance between pathological structures and organs at risk (for therapy planning) • Gray values of objects (mean value, deviation) Bernhard Preim 70 Quantitative Analysis: Volumetry • Volumetry of compact objects: Segmentation and subsequent multiplication of voxel number and voxel size • Problem: processing of boundary voxels, especially in case of filigree structures with many edge voxels (partial volume effect) • Goal: Classification to objects on a sub-voxel level (Voxel vi: 10 % Obj1, 90 % Obj2) Bernhard Preim 71 Quantitative Analysis: Volumetry • Histogram-based analysis of the partial volume effect. If there is a “high” plateau between maxima in the histogram (intensity values that cannot be clearly assigned to a structure), the volumetric uncertainty is huge. • Example: Volumetry of brain ventricles (Schindewolf et al. [1999]) Bernhard Preim 72 Quantitative Analysis: Measurement of the Extent • Goal: Identification of the prolongation of an object o • First idea: Identification of the axis-oriented Bounding Box • Approach: Search through all coordinates and identify minima and maxima in x, y and z direction. • Problem: The calculated prolongation changes as soon as the object is rotated. Bernhard Preim 73 Quantitative Analysis: Measurement of the Extent • Goal: Identification of the maximum diameter of an object o • Idea: Application of a main axis transformation • Approach: Identification of the barycenter and the main axes (mechanical interpretation: axes of inertia) that characterize the orientation of o. • Result: Local, rectangular coordinate system with barycenter as origin and length of the diameter in these new axes (Oriented Bounding Box). Bernhard Preim 74 Quantitative Analysis: Measurement of the Extent 1. Identification of the barycenter vector m 2. Subtraction of the barycenter: pi from P: pĩ = pi – m 3. Identification of covariance matrix A: 1 (pĩx)2 ((pĩx) (pĩy)) A = – ((pĩx) (pĩy)) (pĩy)2 N ((pĩx) (pĩz)) ((pĩy) (pĩz)) ((pĩx) (pĩz) ((pĩy) (pĩz)) (pĩz)2 4. Identification of the Jacobian matrix C, such that A = C-1AC (these are the normalized eigenvectors of covariance matrix A) 5. Virtual object rotation according to matrix C 6. Calculation of the Bounding Box of the rotated object (OBB) to define the length of the main axes 7. Back-transformation according to C-1 through transposition Bernhard Preim 75 Quantitative Analysis: Measurement of the Extent Calculation principle Example of use (3d) Bernhard Preim 76 Quantitative Analysis: Distance Transform • Goal: Emanating from a structure (feature points), all pixels shall be assigned the integral distance to this structure. • Result: Distance Map (Source: Lohmann [1998]) • Applications: • Assessment: Which structures are located within a certain edge and another structure? • Question to determine influence areas (e.g. of vessels): Is a voxel closer to structure1 than to structure2? • Metrices • Euclidean Metrics: DE [(i,j), (h,k)] = sqrt ((i-h)2 + (j-k)2) • City-Block Distance: D4 [(i,j), (h,k)] = |i-h| + |j-k| • Chessboard metrics: D8 [(i,j),(h,k)] = max{|i-h|, |j-k|} Bernhard Preim 77 Quantitative Analysis: Distance Transform Identification of the supply areas of blood vessels MevisLab EuclideanDTF Bernhard Preim 78 Implementation of Image Processing Algorithms Concepts for efficient image processing: • Division of an image page by page or tile by tile • Processing only after request of a display (Lazy Evaluation) • Caching strategies • Processing of single tiles/pages in separate threads (all such concepts are supported in MeVisLab; to some extent, the usage requires dedicated programming, e.g. in case of threads) Bernhard Preim 79 Summary • Numerous segmentation methods are necessary to cope with the variety of biological structures and imaging procedures • Generally applicable vs. special procedures • Segmentation often as core of a pipeline with preprocessing of the images and postprocessing of the segmentation results • Skeletonization is an image analysis step that is based on segmentation • Often, the quantitative analysis is a decisive advantage of digital image analysis Application: Specification of diagnoses and courses of disease. Bernhard Preim 80 Literature W. Abmayr. Einführung in die digitale Bildverarbeitung, Teubner-Verlag Stuttgart 1992 T. Boskamp, D. Rink, F. Link et al. New Vessel Analysis Tool for Morphometric Quantification and Visualization of Vessels in CT and MR Imaging Data Sets, Radiographics, 2004, Heft 2 J. Dornheim, D. Lehmann, L. Dornheim, B. Preim and G. Strauß. „Reconstruction of Blood Vessels from Neck CT Datasets using Stable 3D Mass-Spring Models“, Proc. of EG Workshop on Visual Computing in Biology and Medicine, S. 77-82, 2008 J. Dornheim, H. Seim, B. Preim, I. Hertel and G. Strauss. „Segmentation of Neck Lymph Nodes in CT Datasets with Stable 3D Mass-Spring Models“, Proc. of Medical Image Computing and Computer-Assisted Intervention (MICCAI), S. 904-911, 2006 J. Dornheim, H. Seim, B. Preim, I. Hertel and G. Strauß. „Segmentation of Neck Lymph Nodes in CT Datasets with Stable 3D Mass-Spring Models“, Special Issue of Academic Radiology, S. 1389-1399, 2007 H. K. Hahn, H.-O. Peitgen. “The Skull Stripping Problem In MRI Solved By A Single 3D Watershed Transform”, Medical Image Computing and Computer-Assi-sted Intervention (MICCAI'2000), Pittsburgh, Springer, 2000, S. 134-143 H. K. Hahn and H.-O. Peitgen (2003)., “IWT – Interactive Watershed Transform: A hierarchical method for efficient interactive and automated segmentation of multidimensional grayscale images”, Proc. SPIE Medical Imaging, SPIE Band 5032, Bernhard Preim 81 Literature A. Hennemuth, S. Bock, T. Boskamp, D. Fritz, C. Kuehnel, D. Rinck, M. Scheuering, “One-click coronary tree segmentation in CT angiographic images,” in Proc. of Comput. Assist. Radiol. Surg. (CARS’05), 2005, S. 317–321 A. Hennemuth. Computer-Assisted Diagnosis and Therapy Planning Coronary Artery Disease Based on Cardiac CT and MRI. PhD thesis, Department for Mathematics and Computer Science, University Bremen, 2012 H. Lamecker, T. Lange, and M. Seebass. A Statistical Shape Model for the Liver. Proc. of Medical Image Computing and Computer-Assisted Intervention (MICCAI),volume 2488 of Lecture Notes in Computer Science, pages 422–427, Springer, 2002 Th Lehmann, W Oberschelp, E Pelikan und R Repges. Bildverarbeitung für die Medizin, Springer-Verlag, Heidelberg, 1997 G. Lohmann. Volumetric Image Analysis, Wiley und Teubner, 1998. K. Mühler, C. Tietjen, F. Ritter and B. Preim (2010). „The Medical Exploration Toolkit: An Efficient Support for Visual Computing in Surgical Planning and Training, IEEE Transactions on Visualization and Computer Graphics, S. 133-146 D. Proksch, J. Dornheim, B. Preim. „Interaktionstechniken zur Korrektur medizinischer 3DSegmentierungen“, Bildverarbeitung für die Medizin (BVM) (420-424), 2010 Bernhard Preim 82 Literature G. Sakas and S.Walter. Extracting surfaces from fuzzy 3d-ultrasound data. In Proc. of ACM SIGGRAPH, pages 465–474, 1995 A. Schenk, G. Prause, H. O. Peitgen (2000). Efficient Semiautomatic Segmentation of 3D Objects in Medical Images. Proc. of MICCAI 2000: 186-195 D. Selle. Analyse von Gefäßstrukturen in medizinischen Schichtdatensätzen für die computergestützte Operationsplanung, Dissertation, Universität Bremen, 1999 T. Schindewolf, U. Freese, J. Meißner. “Segmentierung und Volumetrie der Hinrventrikel mit MRT-Datensätzen”, Bildverarbeitung für die Medizin, Springer-Verlag, 1999, S. 92-96 T. Schindewolf, H.-O. Peitgen. “Interaktive Bildsegmentierung von CT- und MR-Daten auf Basis einer modifizierten Wasserscheidentransformation”, Bildverarbeitung für die Medizin, 2000 J. G. Sled, A. P. Zijdenbos, and A. C. Evans. A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Transactions on Medical Imaging, 17(1):87–97, 1998 F. Vega, N. Sauber, B. Tomandl, C. Nimsky, G. Greiner, and P. Hastreiter. Automatic Adjustment of Bidimensional Transfer Functions for Direct Volume Visualization of Intracranial Aneurysms. In Proc. of SPIE Medical Imaging, volume 5367 of LNCS, pages 275–284, 2004 S. Zachow, H.-C. Hege, and P. Deuflhard. Computer assisted planning in cranio-maxillofacial surgery. Journal of Computing and Information Technology, 14(1):53–64, 2004 Bernhard Preim 83