Impiego di 3DF Zephyr
nel rilievo da APR
Andrea Fusiello (andrea.fusiello@uniud.it) -- DIEGM University of Udine
Roberto Toldo, Filippo Fantini, Simone Fantoni, Luca Giona -- 3Dflow srl
www.3dflow.net
3DF Zephyr
Software per la ricostruzione 3D da
fotografie
Sviluppato da 3Dflow srl
spinoff UniUD (ex UniVR)
Gestisce blocchi non strutturati
Gestisce camere eterogenee con
calibrazione ignota
Adatto sia per la ricostruzione di
piccoli oggetti che per rilievi
fotogrammetrici
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
3DF Zephyr pipeline
Image
acquisition
Image
orientation
3DF Samantha
(structure and
motion)
3d photo-consistency
3d surface from
from images
3d photo-consistency
3DF Stasia
(multiple view
stereo)
Poisson [Kazhdan,
Bolitho, and Hoppe,
2006]
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
3DF Zephyr pipeline
Image
acquisition
Image
orientation
3DF Samantha
(structure from
motion)
3d photo-consistency
3d surface from
from images
3d photo-consistency
3DF Stasia
(multiple view
stereo)
Poisson [Kazhdan,
Bolitho, and Hoppe,
2006]
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Structure from motion
Credits to R. Toldo (3Dflow s.r.l.) , R. Gherardi and M. Farenzena (U. Verona)
Structure from motion
2-­‐d %e-­‐points matching UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
object 3-­‐d points Structure from motion
Motion = exterior orientation
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Orientation of a block
¤  Block: set of overlapping photographs
¤  Independent models block adjustment
¤  Create independent stereo-models with relative orientation for each
pair of overlapping photographs
¤  Simultaneous transform the models into the ground coordinate system
with absolute orientation (given known GCP).
¤  Bundle block adjustment
¤  Compute directly the relations between image coordinates and object
coordinates, without introducing model coordinates as intermediate
step.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
n of:
and orientation
of the image points
arameters
Requirement: initial values
Bundle block adjustment
ds
astical
i l model
d l
uation (RNE)
on (TEP)
ation (LEP)
011
19 20-21 Febbraio 2014
UAV / RPAS in Italia - Modena
Resection-intersection cycle
¤  1. Initialize:
¤  (a) Select two suitable photographs;
¤  (b) Solve relative orientation and form the stereo-model (intersection)
¤  2. For every additional photo (following a suitable order),
¤  (a) Solve exterior orientation (resection)
¤  (c) Refine the existing model (intersection);
¤  3. Transform the final model into the ground coordinate system with
absolute orientation (given known GCP).
¤  This is also known in CV as sequential structure from motion (à la
Bundler)
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Hierarchical variation
¤  The previous method can be generalized by organizing the
photographs on a tree instead of a chain.
¤  The tree is produced by hierarchical clustering the photographs
according to their overlap (the overlapping relationship is indeed
not a chain)
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Hierarchical variation (Samantha)
¤  1. Form many independent stereomodels from photo pairs at the
leaves of the tree
¤  2. Traverse the tree; in each node one of these operations takes place:
¤  Grow one model by adding one photo with resection followed by
intersection;
¤  Merge two independent model with absolute orientation;
¤  3. Transform the final model into the ground coordinate system with
absolute orientation (given known GCP).
¤  Note:
¤  If the tree reduces to a chain, the algorithm is a sequential SfM;
¤  If the tree is perfectly balanced, only the Merge step is taken, and the
resulting procedure resembles the IMBA.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Tree traversal
Relative Orientation
Exterior Orientation
Absolute Orientation
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Comments
¤  CV methods are designed for irregular and unknown block
structure. Recovering block structure and tie-points matching is
fundamental (see forward)
¤  CV methods do not make use of GCP but at the end (free solution)
because they are compulsory.
¤  The gold standard is bundle adjustment, the other methods are
seen as tools to get an approximate solution.
¤  The hierarchical method is more effective than the sequential one
in drift containment.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Pre-processing summary
¤  Keypoint extraction
¤  Matching - broad phase: select O(n) views to be matched
¤  Matching – narrow phase: match keypoints between pair
¤  Clustering: determine processing order (can be on-line)
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Keypoints
¤  Detector: scale-space extrema of the scale-normalized Laplacian
[T. Lindeberg, 1994]. SIFT is an variation on this theme.
¤  We use a 8-level scale-space and in each level the Laplacian is
computed by convolution (in CUDA) with a 3 × 3 kernel.
¤  Key: multiresolution pyramid based on derivative operator.
¤  Descriptor: 128- dimensional radial descriptor based on the
accumulated response of steerable derivative filters.
¤  Key: derivatives, directions histogram,
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Which images are to be matched?
¤  Recover the image graph, i.e., the graph that tells which image
overlaps (or can be matched) with which other.
¤  For each key-point descriptor its approximate k nearest neighbours
in feature space are computed (via ANN)
¤  A 2D histogram is then built:
increment bin(i,j) whenever a
keypoint of image i has a keypoint
of image j in its k-neighbourhood;
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Which images are to be matched?
¤  Image graph G = (V,E) where V are views and the weighted
adjacency matrix is the 2D histogram.
¤  This graph has | V | = O(n2). The objective is to extract a subgraph
G’ with a number of edges that is linear in n.
¤  In graph theory, a graph is k-edge-connected if it remains
connected whenever fewer than k edges are removed.
¤  We devised a strategy that builds a subgraph G’ of G which is
k-edge-connected by construction and has has (n-1)k = O(n) edges
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Matching
¤  Match keypoints (images connected in the graph)
¤  Validate matching with relative orientation (RANSAC)
¤  Non linear refinement of relative orientation
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Tracks
¤  From pairwise matches to tracks
¤  just a matter of bookkeeping…
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Clustering
¤  Agglomerative, hierarchical clustering problem
¤  Simple (complete, average, ward’s) linkage
¤  Distance needed:
¤  Common matches
¤  Good coverage
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
3DF Zephyr pipeline
Image
acquisition
Image
orientation
3DF Samantha
(structure and
motion)
3d photo-consistency
3d surface from
from images
3d photo-consistency
3DF Stasia
(multiple view
stereo)
Poisson [Kazhdan,
Bolitho, and Hoppe,
2006]
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Multi-view stereo
Credits to R. Toldo (3Dflow s.r.l.), and G. Vogiatzis (Aston U.)
Depth-map fusion
1.
Compute depth hypotheses
2.
Volumetrically fuse
the depth-maps
3.
Extract a 3d surface
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Computing depth maps
¤ For each pixel in the current image
¤ March along the back-projected ray of the pixel inside
the bounding volume
¤ For each depth value d, project the 3D point in all the
neighbouring views and compute NCC
¤ Equivalently, we work in image space on rectified
images (easy to code on GPU)
¤ The mapping between epipolar lines is a 1-d
homography
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Computing depth maps
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Aggregating NCC scores
¤  Several NCC profiles must be aggregated to extract the
depth of the given pixel.
¤  All local maxima could be good hypotheses.
¤  How do we pick right one?
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
MRF estimate
¤ Voting: each local peak casts a vote (weighted by its
score value) on a discrete histogram along the optical
ray.
¤ Winner take-all: the most voted value is the depth of the
point à can do better
¤ MRF relaxation:
¤  the k bins of the histograms with the highest score are as the
candidate depths for the pixel.
¤  minimization selects the best depth among several hypotesis
taking neighbourhoods into account.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
MRF result
Winner-take-all
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Depth map after MRF
Merge depth maps
¤ Depth maps are lifted in 3D space to produce a
photoconsistency volume, represented by an
octree that accumulates the scores coming from
each depth map.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Image from Vogiatzis, 2010
Photo-consistency of a 3d point
B
A is photoconsistent
B is NOT photoconsistent
A
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Visibility
¤ The photoconsistency volume at
this stage contains spurious
points, which do not belong to a
real surface.
¤  They are characterized by two
features:
¤  their photoconsistency is generally
lower than actual surface points
¤  they usually occludes actual
surface points.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Visibility
¤ This observation leads to an iterative strategy where
the photoconsistency of an occlusor is decreased by a
fraction of the photoconsistency of the occluded point.
¤  Points with negative photoconsistency are eventually
removed.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
3DF Zephyr pipeline
Image
acquisition
Image
orientation
3DF Samantha
(structure and
motion)
3d photo-consistency
3d surface from
from images
3d photo-consistency
3DF Stasia
(multiple view
stereo)
Poisson [Kazhdan,
Bolitho, and Hoppe,
2006]
Images from Vogiatzis, 2010
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
3D surface from points
¤ Compute an indicator function from oriented
points by solving a Poisson problem [Kazhdan et. al]
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Alcuni risultati: 3DF Zephyr con foto da APR
Dati gentilmente forniti da F. Remondino, FBK
Nettuno
Fotografie nadirali (15) e oblique (28), con due fotocamere diverse (compatte).
Tutto il blocco di 43 fotografie viene orientato da Zephyr.
Punti di legame: 13K
Reference variance: 1.58 pix
Nettuno
Dopo multiple-view stereo si ottengono 4.498M punti (vertici mesh).
[Video]
Cerere
Fotografie nadirali (52), oblique (24) e terrestri interno/esterno (212), con tre
fotocamere diverse (compatta su APR, reflex digitale a terra). Tutto il blocco di
288 fotografie viene orientato da Zephyr.
Punti di legame: 127K
Reference variance: 0.73 pix
Cerere
Dopo multiple-view stereo si ottengono 3.521M punti (vertici mesh)
[Video]
Ventimiglia
Fotografie nadirali (253) e oblique (224), con stessa fotocamera (reflex digitale).
Tutto il blocco di 447 fotografie viene orientato da Zephyr.
Punti di legame: 329K
Reference variance: 0.59 pix
Ventimiglia
Dopo multiple-view stereo si ottengono 2.715M punti (vertici mesh)
[Video]
Ventimiglia
Due vedute della maglia poligonale risultante
Ventimiglia
Ortofoto
Impronta del pixel (GSD): 0.0018m (quota inferiore)
Accuracy analysis
Distance to ground control points
0.06
[m]
0.04
0.02
0
2 4 3 5 6 7 8 9 1011121314151617181921222324
Difference with ground control points on each dimension
x [m]
0.04
0.02
0
−0.02
2 4 3 5 6 7 8 9 1011121314151617181921222324
y [m]
0.02
0
−0.02
−0.04
2 4 3 5 6 7 8 9 1011121314151617181921222324
0.1
z [m]
0.05
0
−0.05
−0.1
2 4 3 5 6 7 8 9 1011121314151617181921222324
Cross validation mean distance: 0.0358
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
Grazie per l’attenzione.
UAV / RPAS in Italia - Modena 20-21 Febbraio 2014