Feature-preserving Artifact Removal
from Dermoscopy Images
Howard Zhou1, Mei Chen2,
Richard Gass2, James M. Rehg1,
Laura Ferris3, Jonhan Ho3, Laura Drogowski3
1School
of Interactive Computing, Georgia Tech
2Intel Research Pittsburgh
3Department of Dermatology, University of Pittsburgh
Skin cancer and melanoma

Skin cancer : most common of all cancers
Skin cancer and melanoma

Skin cancer : most common of all cancers
Hemangioma
Basal Cell
Carcinoma
Compound nevus
Seborrheic keratosis
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Skin cancer and melanoma


Skin cancer : most common of all cancers
Melanoma : leading cause of mortality
Hemangioma
Melanoma
Basal Cell
Carcinoma
Compound nevus
Seborrheic keratosis
Melanoma
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Skin cancer and melanoma



Skin cancer : most common of all cancers
Melanoma : leading cause of mortality
Early detection significantly reduces mortality
Hemangioma
Melanoma
Basal Cell
Carcinoma
Compound nevus
Seborrheic keratosis
Melanoma
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Dermoscopy
Clinical View view
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Dermoscopy




Skin surface microscopy
Improve diagnostic accuracy by 30% for trained,
experienced physicians
Requires 5 or more years of experience
Computer-aided diagnosis (CAD) to assist less
experienced physicians
Clinical view
Dermoscopy view
Artifacts in dermoscopy images


Hair, air-bubbles,…
Interfering with computer-aided diagnosis
[ Image courtesy of Grana et al. 2006]
Artifacts in dermoscopy images


Hair, air-bubbles,…
Interfering with computer-aided diagnosis
[ Image courtesy of Grana et al. 2006]
Artifacts in dermoscopy images


Hair, air-bubbles,…
Interfering with computer-aided diagnosis
Hair  lesion boundary
[ Image courtesy of Grana et al. 2006]
Artifacts in dermoscopy images


Hair, air-bubbles,…
Interfering with computer-aided diagnosis
Hair  lesion boundary
[ Image courtesy of Grana et al. 2006]
Artifacts in dermoscopy images


Hair, air-bubbles,…
Interfering with computer-aided diagnosis
Hair  lesion boundary
Hair  pigmented network
[ Image courtesy of Grana et al. 2006]
Previous work

Hair detection and tracing


Thresholding and averaging



Fleming et al. 1998
“DullRazor”, Tim K. Lee et al. 1997
Schmid et al. 2003
Thresholding and inpainting

Paul Wighton et al. 2008 (right here in the
conference)
Schmid et al.


Detection: thresholding
Removal: morphological
operations
Schmid et al.


Thresholding  false
detection
Accidental removal of
diagnostic features
Thresholding
Schmid et al. 2003
Schmid et al.

Morphological operation
(neighbors’ average)
blurring
Morphological operation
Schmid et al. 2003
Feature-preserving
artifact removal
(FAR)


Detection: Explicit curve
modeling
Removal: Exemplarbased inpainting
Schmid et al. 2003
Our method (FAR)
FAR

Curve modeling 
more accurate hair
detection
Thresholding
Schmid et al. 2003
Curve modeling
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Morphological
Thresholding operation
Schmid et al. 2003
Exemplar-based
Curve modeling inpainting
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Morphological
Thresholding operation
Schmid et al. 2003
Exemplar-based
Curve modeling inpainting
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Schmid et al. 2003
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Schmid et al. 2003
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Schmid et al. 2003
Our method (FAR)
System overview
Dermoscopy
image
Threholding
Luminance
difference 
dark thin
structure
Line points
Detection
Line segments
Line points
Line points linking
Exemplar patches
Curve fitting &
intersection analysis
Hair removed
Mask
Exemplar-based
inpainting
Parameterized
curves
Input dermoscopy image
Enhancing dark-thin structure


Luminosity channel in CIE L*u*v*
Difference b/a morphological closing
[ Schmid-Saugeona et al. 2003, “Towards a computer-aided diagnosis system for pigmented skin lesions” ]
Detecting line points
Curve B(t)
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
Cross section
Curve B(t)
n(t)
f(x)
n(t)
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
Cross section
Curve B(t)
n(t)
f(x)
n(t)
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
Cross section
f’ = 0
|f’’| large
Curve B(t)
n(t)
f(x)
n(t)
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
Cross section
f’ = 0
|f’’| large
Curve B(t)
n(t)
f(x)
n(t)
n(t) : direction ┴ curve B(t)
eigenvector corresponding to the maximum
absolute eigenvalue of the local Hessian
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
n(t)
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Detecting line points
[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]
Linking line points

Link the neighboring points to get line
segments (sets of ordered line points)
Fitting polynomial curves

P
A set of ordered points Pi s
Fitting polynomial curves

P

A set of ordered points Pi s
Parametric curve
Fitting polynomial curves


P
B(t)
A set of ordered points Pi s
Parametric curve
Fitting polynomial curves

A set of ordered points Pi s
Parametric curve

Minimize sum of squared distance

P
B(t)
Fitting polynomial curves

A set of ordered points Pi s
Parametric curve

Minimize sum of squared distance

Linear system (can be solved by
Gaussian elimination)

P
B(t)
Handling hair intersection
Hair intersection
Line segments
Intersection analysis
Link Line segment
Configurations:
……
Before curve fitting and linking
Line segments
After curve fitting and linking
Parameterized curves
After curve fitting and linking
Parameterized curves
After curve fitting and linking
Hair mask
After curve fitting and linking
Hair mask
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Image courtesy of Criminisi et al. 2003 ]
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Exemplar-based inpainting


Fill in with patches from the image itself
Patch ordering structure propagation.
[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]
Before FAR
After FAR
More results


Explicit curve
modeling
Exemplar-based
inpainting
Schmid et al. 2003
Our method (FAR)
More results


Explicit curve
modeling
Exemplar-based
inpainting
Schmid et al. 2003
Our method (FAR)
FAR

Exemplar-based
inpainting 
preserving features
Schmid et al. 2003
Our method (FAR)
When is FAR not suitable ?

Oops, too much hair!
When is FAR not suitable ?


Too much hair
Makes explicit
modeling
difficult
Schemid et al. 2003 (DullRazor)
Our method (FAR)
Conclusion


Automatic system that
detects and removes
curvilinear artifacts
Feature-preserving
artifact removal:


Explicit curve modeling
Exemplar-based
inpainting
Future work

Speed up exemplar-based inpainting
Future work


Speed up exemplar-based inpainting
Handle hair with arbitrary intensity
Future work



Speed up exemplar-based inpainting
Handle hair with arbitrary intensity
Extend to removing air bubbles
Questions ?
Additional results
Original Dermoscopy image
Our method (FAR)