advertisement

Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello ([email protected]) -- DIEGM University of Udine Roberto Toldo, Filippo Fantini, Simone Fantoni, Luca Giona -- 3Dflow srl www.3dflow.net 3DF Zephyr Software per la ricostruzione 3D da fotografie Sviluppato da 3Dflow srl spinoff UniUD (ex UniVR) Gestisce blocchi non strutturati Gestisce camere eterogenee con calibrazione ignota Adatto sia per la ricostruzione di piccoli oggetti che per rilievi fotogrammetrici UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 3DF Zephyr pipeline Image acquisition Image orientation 3DF Samantha (structure and motion) 3d photo-consistency 3d surface from from images 3d photo-consistency 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006] Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 3DF Zephyr pipeline Image acquisition Image orientation 3DF Samantha (structure from motion) 3d photo-consistency 3d surface from from images 3d photo-consistency 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006] Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Structure from motion Credits to R. Toldo (3Dflow s.r.l.) , R. Gherardi and M. Farenzena (U. Verona) Structure from motion 2-­‐d %e-­‐points matching UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 object 3-­‐d points Structure from motion Motion = exterior orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Orientation of a block ¤ Block: set of overlapping photographs ¤ Independent models block adjustment ¤ Create independent stereo-models with relative orientation for each pair of overlapping photographs ¤ Simultaneous transform the models into the ground coordinate system with absolute orientation (given known GCP). ¤ Bundle block adjustment ¤ Compute directly the relations between image coordinates and object coordinates, without introducing model coordinates as intermediate step. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 n of: and orientation of the image points arameters Requirement: initial values Bundle block adjustment ds astical i l model d l uation (RNE) on (TEP) ation (LEP) 011 19 20-21 Febbraio 2014 UAV / RPAS in Italia - Modena Resection-intersection cycle ¤ 1. Initialize: ¤ (a) Select two suitable photographs; ¤ (b) Solve relative orientation and form the stereo-model (intersection) ¤ 2. For every additional photo (following a suitable order), ¤ (a) Solve exterior orientation (resection) ¤ (c) Refine the existing model (intersection); ¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ This is also known in CV as sequential structure from motion (à la Bundler) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Hierarchical variation ¤ The previous method can be generalized by organizing the photographs on a tree instead of a chain. ¤ The tree is produced by hierarchical clustering the photographs according to their overlap (the overlapping relationship is indeed not a chain) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Hierarchical variation (Samantha) ¤ 1. Form many independent stereomodels from photo pairs at the leaves of the tree ¤ 2. Traverse the tree; in each node one of these operations takes place: ¤ Grow one model by adding one photo with resection followed by intersection; ¤ Merge two independent model with absolute orientation; ¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ Note: ¤ If the tree reduces to a chain, the algorithm is a sequential SfM; ¤ If the tree is perfectly balanced, only the Merge step is taken, and the resulting procedure resembles the IMBA. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Tree traversal Relative Orientation Exterior Orientation Absolute Orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Comments ¤ CV methods are designed for irregular and unknown block structure. Recovering block structure and tie-points matching is fundamental (see forward) ¤ CV methods do not make use of GCP but at the end (free solution) because they are compulsory. ¤ The gold standard is bundle adjustment, the other methods are seen as tools to get an approximate solution. ¤ The hierarchical method is more effective than the sequential one in drift containment. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Pre-processing summary ¤ Keypoint extraction ¤ Matching - broad phase: select O(n) views to be matched ¤ Matching – narrow phase: match keypoints between pair ¤ Clustering: determine processing order (can be on-line) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Keypoints ¤ Detector: scale-space extrema of the scale-normalized Laplacian [T. Lindeberg, 1994]. SIFT is an variation on this theme. ¤ We use a 8-level scale-space and in each level the Laplacian is computed by convolution (in CUDA) with a 3 × 3 kernel. ¤ Key: multiresolution pyramid based on derivative operator. ¤ Descriptor: 128- dimensional radial descriptor based on the accumulated response of steerable derivative filters. ¤ Key: derivatives, directions histogram, UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Which images are to be matched? ¤ Recover the image graph, i.e., the graph that tells which image overlaps (or can be matched) with which other. ¤ For each key-point descriptor its approximate k nearest neighbours in feature space are computed (via ANN) ¤ A 2D histogram is then built: increment bin(i,j) whenever a keypoint of image i has a keypoint of image j in its k-neighbourhood; UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Which images are to be matched? ¤ Image graph G = (V,E) where V are views and the weighted adjacency matrix is the 2D histogram. ¤ This graph has | V | = O(n2). The objective is to extract a subgraph G’ with a number of edges that is linear in n. ¤ In graph theory, a graph is k-edge-connected if it remains connected whenever fewer than k edges are removed. ¤ We devised a strategy that builds a subgraph G’ of G which is k-edge-connected by construction and has has (n-1)k = O(n) edges UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Matching ¤ Match keypoints (images connected in the graph) ¤ Validate matching with relative orientation (RANSAC) ¤ Non linear refinement of relative orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Tracks ¤ From pairwise matches to tracks ¤ just a matter of bookkeeping… UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Clustering ¤ Agglomerative, hierarchical clustering problem ¤ Simple (complete, average, ward’s) linkage ¤ Distance needed: ¤ Common matches ¤ Good coverage UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 3DF Zephyr pipeline Image acquisition Image orientation 3DF Samantha (structure and motion) 3d photo-consistency 3d surface from from images 3d photo-consistency 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006] Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Multi-view stereo Credits to R. Toldo (3Dflow s.r.l.), and G. Vogiatzis (Aston U.) Depth-map fusion 1. Compute depth hypotheses 2. Volumetrically fuse the depth-maps 3. Extract a 3d surface Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Computing depth maps ¤ For each pixel in the current image ¤ March along the back-projected ray of the pixel inside the bounding volume ¤ For each depth value d, project the 3D point in all the neighbouring views and compute NCC ¤ Equivalently, we work in image space on rectified images (easy to code on GPU) ¤ The mapping between epipolar lines is a 1-d homography UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Computing depth maps UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Aggregating NCC scores ¤ Several NCC profiles must be aggregated to extract the depth of the given pixel. ¤ All local maxima could be good hypotheses. ¤ How do we pick right one? Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 MRF estimate ¤ Voting: each local peak casts a vote (weighted by its score value) on a discrete histogram along the optical ray. ¤ Winner take-all: the most voted value is the depth of the point à can do better ¤ MRF relaxation: ¤ the k bins of the histograms with the highest score are as the candidate depths for the pixel. ¤ minimization selects the best depth among several hypotesis taking neighbourhoods into account. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 MRF result Winner-take-all UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Depth map after MRF Merge depth maps ¤ Depth maps are lifted in 3D space to produce a photoconsistency volume, represented by an octree that accumulates the scores coming from each depth map. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Image from Vogiatzis, 2010 Photo-consistency of a 3d point B A is photoconsistent B is NOT photoconsistent A UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Visibility ¤ The photoconsistency volume at this stage contains spurious points, which do not belong to a real surface. ¤ They are characterized by two features: ¤ their photoconsistency is generally lower than actual surface points ¤ they usually occludes actual surface points. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Visibility ¤ This observation leads to an iterative strategy where the photoconsistency of an occlusor is decreased by a fraction of the photoconsistency of the occluded point. ¤ Points with negative photoconsistency are eventually removed. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 3DF Zephyr pipeline Image acquisition Image orientation 3DF Samantha (structure and motion) 3d photo-consistency 3d surface from from images 3d photo-consistency 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006] Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 3D surface from points ¤ Compute an indicator function from oriented points by solving a Poisson problem [Kazhdan et. al] UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Alcuni risultati: 3DF Zephyr con foto da APR Dati gentilmente forniti da F. Remondino, FBK Nettuno Fotografie nadirali (15) e oblique (28), con due fotocamere diverse (compatte). Tutto il blocco di 43 fotografie viene orientato da Zephyr. Punti di legame: 13K Reference variance: 1.58 pix Nettuno Dopo multiple-view stereo si ottengono 4.498M punti (vertici mesh). [Video] Cerere Fotografie nadirali (52), oblique (24) e terrestri interno/esterno (212), con tre fotocamere diverse (compatta su APR, reflex digitale a terra). Tutto il blocco di 288 fotografie viene orientato da Zephyr. Punti di legame: 127K Reference variance: 0.73 pix Cerere Dopo multiple-view stereo si ottengono 3.521M punti (vertici mesh) [Video] Ventimiglia Fotografie nadirali (253) e oblique (224), con stessa fotocamera (reflex digitale). Tutto il blocco di 447 fotografie viene orientato da Zephyr. Punti di legame: 329K Reference variance: 0.59 pix Ventimiglia Dopo multiple-view stereo si ottengono 2.715M punti (vertici mesh) [Video] Ventimiglia Due vedute della maglia poligonale risultante Ventimiglia Ortofoto Impronta del pixel (GSD): 0.0018m (quota inferiore) Accuracy analysis Distance to ground control points 0.06 [m] 0.04 0.02 0 2 4 3 5 6 7 8 9 1011121314151617181921222324 Difference with ground control points on each dimension x [m] 0.04 0.02 0 −0.02 2 4 3 5 6 7 8 9 1011121314151617181921222324 y [m] 0.02 0 −0.02 −0.04 2 4 3 5 6 7 8 9 1011121314151617181921222324 0.1 z [m] 0.05 0 −0.05 −0.1 2 4 3 5 6 7 8 9 1011121314151617181921222324 Cross validation mean distance: 0.0358 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014 Grazie per l’attenzione. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014