Real-Time Exemplar-Based Face Sketch Synthesis Pipeline illustration
advertisement
Real-Time Exemplar-Based Face Sketch Synthesis Pipeline illustration Yibing Song1 Linchao Bao1 1City Qingxiong Yang1 Ming-Hsuan Yang2 University of Hong Kong 2University of California at Merced Note: containing animations Our assumption: a database containing photo-sketch pairs 1. photo database 2. sketch database Aligned Coarse Sketch Generation Step 1: KNN search Test photo Test photo patch π»π p Relative Relative Similarly βππ βπ = [ βππ position position βπ² π ] Training photo dataset π»π π»π π»π Matched photo patch π°ππ· Matched photo patch π°ππ· Coarse Sketch Generation Step 2: Linear Estimation from Photos Test photo patch π»π βππ πππ β βππ βπ² π +πππ β Matched Matched photo patch π°ππ· photo patch π°ππ· +ππ² π β = Matched photo patch π°π² π· 2. Compute linear mapping function defined by πππ , πππ , β― , ππ² π Coarse Sketch Generation Step 3: Apply Linear Mapping to Sketches Test photo Coarse sketch Repeat for every pixel p βππ πππ β βππ βπ² π +ππ² π β +πππ β Matched sketch pixel Matched πΊ π sketch pixel πΊ π·+βπ π·+βππ = π¬π Estimation on pixel p Matched sketch pixel πΊ π·+βπ² π Denoising: State-of-the-art Image Denoising Algorithms Coarse sketch Nonlocal Means (NLM) q π€(π, π) r p After NLM ππππΏπ = πΈπ + π€(π, π) πΈπ +β― For all pixels in the neighbor of p: Ψπ Little improvement Because: coarse sketch image is not natural. π€(π, π) is not a good similarity measurement between p and r. [NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005. Motivation – BM3D BM3D groups correlated patches in the noisy image to create multiple estimations. How BM3D works Our idea for sketch denoising: group highly similar sketch estimations. [BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transformdomain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007. Proposed Spatial Sketch Denoising Algorithm (SSD) Estimations from pixels in local region Ψπ Test photo r π¬ππ q p βππ , πππ Matched sketch πππ β Averaging estimations to generate output sketch value. π² β π , ππ² π βππ , πππ Similarly πΊπ+βπ +ππ β π π πΊπ+βπ π πΊπ+βπ π πΊπ+βπ π Nonlocal Means (NLM): ππππΏπ = π€(π, π) β πΈπ +π€(π, π) β πΈπ +β― Proposed SSD: πππππ· = π 1 β π¬π + p 1 β π¬ππ +β― +ππ² π β πΊ= π+βπ² π πΊπ+βπ² π π π¬π Robustness to the region size Ψπ - the only parameter involved p Proposed SSD is robust to Ψπ Input 5x5 local region π³π = ππ 11x11 local region π³π = πππ 17x17 local region π³π = πππ 23x23 local region π³π = πππ Note: When πΉπ is sufficient large (i.e., πΉπ >100), the proposed SSD can effectively suppress noise while preserving facial details like the tiny eye reflections (see close-ups).
