Kapitel 13 Interactive Segmentation Live-wire approach Random walker segmentation Kapitel 13 “Interactive Segmentation” – p. 1 Live-wire approach Live-wire approach (intelligent scissors): The user interactively picks a seed point on the boundary. Then, a live-wire is displayed in real time from the initial point to any subsequent position taken by the cursor. The entire 2D boundary is specified by means of a set of live-wire segments in this manner. The detection of segments is formulated as a graph search problem, which finds the globally optimal (minimum-weight) path between an initial start pixel and an end pixel. Kapitel 13 “Interactive Segmentation” – p. 2 Live-wire approach E.N. Mortensen and W.A. Barrett: Intelligent scissors for image composition. SIGGRAPH, 191-198, 1995. Kapitel 13 “Interactive Segmentation” – p. 3 Exkurs: Dijkstra‘s algorithm Problem: finding shortest paths from a source vertex to all other vertices in a graph. Dijkstra's algorithm: Greedy solution to the single-source shortest path problem. It works on both directed and undirected graphs. All edges must have nonnegative weights. Input: Weighted graph G = {E, V} and start vertex s∈V, such that all edge weights are nonnegative. Output: Lengths of shortest paths (or the shortest paths themselves) from the given start vertex s∈V to an end node t∈V (or all other vertices). Kapitel 13 “Interactive Segmentation” – p. 4 Exkurs: Dijkstra‘s algorithm Step 1: Assign the start node with distance zero and mark it as visited. Assign all other vertices with infinity and mark them unvisited. Step 2: Consider the most recently visited node X with distance Dx. For each of its unvisited neighbors Y with distance Dy, replace Dy by min(Dx+ weight(X,Y), Dy) Step 3: Choose the unvisited vertex with the smallest distance and mark it visited. Step 4: Repeat Steps 2 and 3 until the end node is marked visited. Step 5: Go backwards through the graph, retracing the minimum-weight path from the end node to the start node. Kapitel 13 “Interactive Segmentation” – p. 5 Exkurs: Dijkstra‘s algorithm Example: Find the minimum-weight path from node s to node t. s 2 1 5 2 3 1 2 3 4 t Kapitel 13 “Interactive Segmentation” – p. 6 Exkurs: Dijkstra‘s algorithm 2 1 5 2 1 2 3 1 2 2 3 4 5 5 2 1 3 5 6 2 4 1 2 3 2 1 2 3 3 4 1 4 4 6 5 3 2 2 1 3 1 3 1 7 4 2 5 2 3 3 ∞ 3 4 2 3 4 3 5 4 3 1 32 3 2 1 ∞ 4 1 23 5 2 3 3 2 1 3 5 1 1 ∞ 2 3 1 1 2 2 3 2 4 5 Kapitel 13 “Interactive Segmentation” – p. 7 Exkurs: Search algorithm from AI Search is a classical AI problem and many solutions exists (e.g. A*) Kapitel 13 “Interactive Segmentation” – p. 8 Live-wire approach Example: Kapitel 13 “Interactive Segmentation” – p. 9 Live-wire approach Example: Composition E.N. Mortensen and W.A. Barrett: Intelligent scissors for image composition. SIGGRAPH, 191-198, 1995. Kapitel 13 “Interactive Segmentation” – p.10 Live-wire approach Example: A cursor snap mechanism forces the mouse point to the pixel of maximum gradient magnitude within a user-specified neighborhood. Live-wire segment snaps to a boundary as the free point moves (via cursor movement). The path of the free point is shown in white. Livewire segments from previous free point positions are shown in green. Kapitel 13 “Interactive Segmentation” – p.11 Live-wire approach Example: Continuous snap–drag of a live-wire to a coronary edge (the boundary is completed in about 2s) E.N. Mortensen and W.A. Barrett: Interactive live-wire boundary extraction. Medical Image Analysis, 1(4): 331-341. Kapitel 13 “Interactive Segmentation” – p.12 Live-wire approach Reproducibility for live-wire and manual tracing tool. Eight users extracted five object boundaries five times with the livewire tool and the same object boundaries three times with the manually tracing tool. Manual tracing takes 2-3 times longer. The boundaries are virtually identical regardless of which user is performing the task! Kapitel 13 “Interactive Segmentation” – p.13 Live-wire approach Extension to 3D: segment 3D volume data or time sequences of 2D images. A.X. Falcao and J.K. Udupa: A 3D generalization of user-steered live-wire segmentation. Medical Image Analysis, 4(4), 389-402, 2000. The user specifies contours via live-wiring on a few slices that are orthogonal to the natural slices of the original data. If these slices are selected strategically, then one obtains a sufficient number of seed points in each natural slice which enable a subsequent automatic optimal boundary detection therein. Kapitel 13 “Interactive Segmentation” – p.14 Random walker segmentation Given labeled pixels, for each pixel: What is the probability that a random walker starting from this pixel first reaches each set of labels? L.Grady: Random walks for image segmentation. IEEE-TPAMI, 28: 1768–1783, 2006. unseeded (unlabeled) pixel (a) A two-region image edge weight: similarity between two nodes, based on e.g., intensity gradient, color changes (c) A 4-connected lattice topology seeded (labeled) pixels (b) Use-defined seeds for each region low-weight edge (sharp color gradient) (d) An undirected weighted graph Kapitel 13 “Interactive Segmentation” – p.15 Random walker segmentation The algorithm labels an unseeded pixel in following steps: Step 1. Calculate the probability that a random walker starting at an unseeded pixel x first reaches a seed with label s 0.97 0.90 0.97 0.97 0.90 0.85 0.15 0.85 0.15 0.85 0.15 0.10 0.10 0.03 0.03 0.03 0.03 0.03 0.03 Probability that a random walker starting from each unseeded node first reaches red seed 0.10 0.10 0.15 0.85 0.15 0.85 0.15 0.85 0.90 0.97 0.97 0.90 0.97 Probability that a random walker starting from each unseeded node first reaches blue seed Kapitel 13 “Interactive Segmentation” – p.16 Random walker segmentation Step 2. Label each pixel with the most probable seed destination (0.97,0.03) (0.90,0.10) (0.97,0.03) (0.97,0.03) (0.90,0.10) (0.85,0.15) (0.15,0.85) (0.85,0.15) (0.15,0.85) (0.85,0.15) (0.15,0.85) (0.10,0.90) (0.03,0.97) (0.03,0.97) (0.10,0.90) (0.03,0.97) A segmentation corresponding to region boundary is obtained by biasing the random walker to avoid crossing sharp color gradients Kapitel 13 “Interactive Segmentation” – p.17 Random walker segmentation partially labeled image green segmented image Probabilities red yellow blue Kapitel 13 “Interactive Segmentation” – p.18 Random walker segmentation Algorithm summary: 1. Generate weights based on image intensities. 2. Build Laplacian matrix. 3. Solve system of linear equations for each label. 4. Assign pixel (voxel) to label for which it has the highest probability. Kapitel 13 “Interactive Segmentation” – p.19 Random walker segmentation Weight generation: The random walker is governed by edge weights (e.g., probabilities of reaching a pixel pj from its neighboring pixel pi). f(): intensity function β: influences how quickly the probability decreases The probability is higher the more likely the two pixels belong to the same region. Kapitel 13 “Interactive Segmentation” – p.20 Random walker segmentation Laplacian matrix L: di : degree of pixel pi It is shown that finding the probabilities for a walk starting at some pixel to arrive at some seed pixel is related to minmizing: Kapitel 13 “Interactive Segmentation” – p.21 Random walker segmentation Solution of system of linear equations: Partition the pixels into two sets, VM (all marked/seed pixels, regardless of their label) and VU (unseeded). The pixels in L and x are ordered such that seed pixels are first and unseeded pixels are second. Then, the optimization problem becomes: LM: edge weighted among the marked pixels LU: edge weighted among the unmarked pixels Kapitel 13 “Interactive Segmentation” – p.22 Random walker segmentation To minimize it, the equation is differentiated with respect to unknown XU: This is set to zero for finding the minumum of D(XU), resulting a system of |VU| linear equations: XM: 1 for the label under consideration, and 0 for all other labels (remember: we need to solve XU for each label) Kapitel 13 “Interactive Segmentation” – p.23 Random walker segmentation Cardiac segmentation across modalities Kapitel 13 “Interactive Segmentation” – p.24 Random walker segmentation Segmentation of objects with varying size, shape and texture Kapitel 13 “Interactive Segmentation” – p.25 Random walker segmentation Segmentation of natural images Kapitel 13 “Interactive Segmentation” – p.26 Random walker segmentation Author webpage: http://cns.bu.edu/~lgrady Random walker paper: http://cns.bu.edu/~lgrady/grady2006random.pdf Random walker MATLAB code: http://cns.bu.edu/~lgrady/random_walker_matlab_code.zip Random walker demo page: http://cns.bu.edu/~lgrady/Random_Walker_Image_Segmentation.html Kapitel 13 “Interactive Segmentation” – p.27