3
Introduction
„
Color Science,
Color Management, and
Color Image Quality
You process images
„
„
„
„
Professor Jon Yngve Hardeberg
„
The Norwegian Color Research Laboratory
Gjøvik University College, Gjøvik, Norway
jon.hardeberg@hig.no, http://www.colorlab.no,
and
Hardeberg Image and
Color Technology
Maybe capturing them
Maybe trying to
improve their visual
appearance
Maybe you want to
compress them without
sacrificing quality
Maybe you try to
recognize objects,
something humans
do easily
Maybe you want to
visualize complex
image data using color?
http://color.hardeberg.com
4
Introduction
„
Outline
You need to
„
„
„
5
„
Understand color
perception
Make sure
What You See Is
What Others See
Be able to judge
image quality
„
„
„
Introduction
Color Science
Color Management
Color Image Quality
6
What is Color?
Well, my favorite
color is red!
7
What is Color?
“Color consists of the characteristics of light
other than spatial and temporal inhomogeneities;
light being that aspect of radiant energy of
which a human observer is aware through the
visual sensations which arise from the
stimulation of the retina of the eye.”
[OSA 1940]
Well, my favorite
color is red!
(t-t-t togduan?)
1
8
„
What is Color?
“Color consists of the characteristics of light
other than spatial and temporal inhomogeneities;
light being that aspect of radiant energy of
which a human observer is aware through the
visual sensations which arise from the
stimulation of the retina of the eye.”
[OSA 1940]
„
„
„
Color is a sensation
„
„
Interaction between the
physical world and our senses
Psychophysics
Is the tree green if
no one’s there?
"I want you to realize
that there exists no
color in the natural
world, and no sound nothing of this kind;
no textures, no
patterns, no beauty,
no scent." (Sir John
Eccles)
10
Color: Light, surface, eye
Light’s Spectral Distribution, l(λ)
11
Stare at this flag for 30 seconds...
Sensitivity, si(λ)
Eye’s Spectral Sensitivities si(λ), i =1,2,3
Reflectance, r(λ)
Radiance, l(λ)
Surface’s Spectral Reflectance, r(λ)
9
When a tree falls in
the forest, and no one
is around to hear it, is
there a sound?
Wavelength, λ
Wavelength, λ
S
M
L
Wavelength, λ
12
13
Complementary after-images
„
„
What did you see?
„
#1: A white screen only
„
#2: The Norwegian flag
„
#3: The Danish flag
„
#4: First #3 then #2
Conclusions
„
„
The eye is quite different from a CCD sensor
Our linear model is limited (but very useful)
2
14
15
Retinal cone distribution vs. CMOS sensor
Colorimetry
„
International Lighting
Commission – CIE
„
„
„
1931 Standard
Colorimetric Observer
Determined based on color
matching experiments
Provides a standardized
way of quantifying color
„
„
CIEXYZ
Foundation for consistent
communication of color
16
17
Color Spaces
„
Color spaces for images and video
3 different photoreceptors, L, M, S
„
„
„
3-output linear model
Color can be specified
by three numbers
Color space - analogy to
geometrical 3D space
C3
„
C
RGB space is most common
„
„
Native to cameras, scanners, displays
Based on additive color mixing
C1
„
CMY(K) for printing
„
Subtractive color mixing
C2
„
Luminance-chrominance
spaces
„
„
Important to separate, for instance for ”Chroma Subsampling”
HSV, HSL, YIQ, YUV, YCbCR, Lab, Luv, …
18
YCbCr Color Space
„
A luminance-chrominance color space:
YCbCr
„
„
„
„
Color Spaces
„
3 different photoreceptors, L, M, S
„
Standardized for digital video by
Recommendation ITU-R BT.601.4
Used in JPEG image compression and MPEG
video compression.
Closely related to the YUV transform.
„
„
Luminance Y’ (a.k.a. “Luma”)
„
„
„
„
19
Y’ = 0.299R’ + 0.587 G’ + 0.114B’
Calculated from gamma-corrected signals
R’, G’, B’
Note difference from colorimetric luminance Y
Chrominance Cb and Cr
„
Cb = ((B’−Y’) / 1.772) + 0.5
Cr = ((R’−Y’) / 1,402) + 0.5
„
C3
C
Device-independent color spaces
„
„
3-output linear model
Color can be specified
by three numbers
Color space - analogy to
geometrical 3D space
C1
C2
LMS, XYZ, CIELUV, CIELAB, sRGB …
Device-dependent color spaces
„
Monitor RGB, Printer CMYK, Scanner RGB ...
3
20
CIELAB color space (a.k.a. L*a*b*)
Device independence
„
Device-independent color spaces
„
„
„
„
„
Defined according to colorimetry
LMS, CIEXYZ, CIELUV, CIELAB, sRGB …
Device independent
Pseudo-uniformly related to human perception:
„
Monitor RGB, Printer CMYK, Scanner RGB ...
Defined according to a standard observer
„
International Standard (CIE)
Used in Color Fax and CMS
„
Lightness L*
„
Device-dependent color spaces
„
21
„
„
„
L* = 100 = white
L* = 0 = black
Chromaticity a* and b*
„
„
a* = green (-) to red (+)
b* = blue (-) to yellow (+)
22
„
„
„
„
Standardized RGB
Device independent
Definition based on a
typical PC monitor
Advantages:
„
„
„
„
„
„
Euclidean distance in CIELAB space
„
ΔE*ab “Delta E”
A good measure of perceptual color difference
„
Much used as color quality metric:
„
Ease of use
International Standard (IEC)
Endorsed by major business
players (HP, MSFT)
A common color language - Scan to sRGB - Print from sRGB - - More and more consumer imaging devices “speak” sRGB
„
Disadvantages:
„
http://www.srgb.com
„
Color Difference
From http://www.srgb.com
sRGB Color Space
23
Average ΔE*ab = ?
Max ΔE*ab = ?
Yes
Original
24
Reproduction
25
Perceived color depends on...
„
„
„
Spectral distribution of light
Spectral reflectance of surface
Spectral sensitivity of eye
… but also on …
„
„
„
„
„
Size of area
Viewing direction
Viewer adaptation
Surrounding color
Local contrast …
Brain
4
26
Assimilation
27
Simultaneous contrast
28
29
30
31
Local vs. global color changes
1
3
Adding a cyan filter over the
cushion
Adding a cyan filter over the
whole image
1
2
1
„
„
„
2
3
Images courtesy of Dr. R.G.W Hunt, Univ. of Derby, UK
2
Original image
Illustrates adaptation to white point color constancy
3
White balance for cameras
Color appearance modeling - CAM
5
32
Outline
„
„
„
„
33
The first law of color management
Introduction
Color Science
Color Management
Color Image Quality
„
Different imaging devices
never produce equal color!
34
Scanners ‘speak’ color differently
Scanner A
File A
35
Printers ‘speak’ color differently
≠
Printer A
Original
Image
≠
Digital file
Scanner B
File B
Printer B
36
We speak color differently
37
Complex imaging systems
I wonder if this skirt matches my
beautiful “Barbie Doll Pink” blouse?
Let’s see - they write it’s “Hot Pink,”
and in the image it says
(R,G,B) = (245, 174, 203) ??!?
6
38
39
Principle of Color Management
„
Image
interchange
through
DeviceIndependent
Color Space
„
„
Esperanto
of colors
Imaging
devices
characterized
by Profiles
„
Principle of Color Management
„
Image
Profile
Camera
Profile
Color conversion between devices by
“coupling” two profiles - source and
destination
Monitor
Profile
Device-Independent
Color Space
Scanner
Profile
Printer
Profile
Scanner
Profile
Printer
Profile
Dictionaries
40
41
LUT-based colorspace conversion
„
One look-up table
(3DLUT) per conversion
„
„
„
Pre-calculated conversion
for a subset of all possible
input colors
Colorspace conversions in an MFP
B
PC
Application
•
Local color copy
•
Scan to PC
•
Print from PC
•
ITU Color Fax
•
G
R
Scan
Device
•
YCbCr
sRGB
Interpolation for inbetween values
Content of LUTs depends
on the actual devices, and
is determined by device
profiling
„
Internet
Color Fax
Colorspace
Conversions
Scanner
RGB
CIELAB
•
CMYK
•
PSTN
Color Fax
Print
Device
A.k.a colorimetric
characterization
Scanner RGB
Printer CMYK
Scanner RGB
sRGB
sRGB
Printer CMYK
•
Scanner RGB
•
CIELAB
CIELAB
Printer CMYK
Internet Color Fax
•
Scanner RGB
•
YCbCr
YCbCr
Printer CMYK
42
International Color Consortium - ICC
„
43
Input Device Profiling (scanner)
International standard for Color Management
„
„
„
„
Architecture of Color Management Systems
ICC Profile file format
Now also ISO standard
See www.color.org
Previously known or
measured
colorimetric data
Color Target
Scanned
Image
File
Device-independent
color data (CIELAB)
Device-dependent
RGB data
Mean values
extraction
Scanner
Profiling
Algorithm
ICC
Profile
Scanner
PC
Device
Independent
Color
Image
7
44
Output Device Profiling (printer)
Approaches to device profiling
Device-dependent
CMYK or RGB data
CMYK or RGB
printer target
digital file
45
„
„
A.k.a colorimetric device characterization
Typical algorithms based on
„
Device-independent
color data (CIELAB)
Printer
Profiling
Physical models
„
„
„
„
Data-driven models
„
ICC Profile
„
„
„
Image File
from Fax or
Scan
Printer
Controller
„
Printing
Engine
GOG model for CRT displays
Neugebauer model for halftone print
++
3DLUT Interpolation
Polynomial regression
Neural networks
Thin-plate splines
++
46
Color Gamut Mapping
Some active research areas
„
Improved algorithms for device
profiling / colorimetric
characterization
„
„
„
Compensation for varying
viewing conditions
„
„
„
„
LCD and DLP projection displays
Hue-preserving camera
characterization model
Color appearance modeling
Color gamut mapping
Spectral reproduction
Workflow issues
„
„
„
Ease of use
Get people to do the right things
New areas such as digital film
production
47
L. Seime and J.Y. Hardeberg, Colorimetric
Characterisation of LCD and DLP
Projection Displays, Journal of the Society of
Information Display, 11(2): 349-358, 2003
„
E.B. Mikalsen; J.Y. Hardeberg & J.-B. Thomas.
Verification and extension of a camerabased end-user calibration method for
projection displays. CGIV, 2008
„
Imaging devices have only a limited number of reproducible
colors: Color Gamut
What to do if your image contains “out-of-gamut” colors?
„
C.F. Andersen and J.Y. Hardeberg,
Colorimetric characterization of digital
cameras preserving hue planes, 13th Color
Imaging Conference, pp. 141-146, 2005
„
J.Y. Hardeberg, Colorimetric Scanner
Characterization, Acta Graphica, 155, 2005
Need for Gamut Mapping
„
A. Alsam; J. Gerhardt & J.Y. Hardeberg.
Inversion of the Spectral Neugebauer
Printer model. AIC Colour 05, 44-62, 2005
Can’t print anything “more yellow” than the yellow ink!
„
„
“Do the best you can
with the colors you have”
Huge unsolved problem
in color imaging R&D
Clipping out of gamut colors
is not good enough
J.Y. Hardeberg, I. Farup, Ø. Kolås, and G.
Stjernvang, Color management in digital
video: Color Correction in the Editing
Phase, Proc. IARIGAI Conference, pp. 166-179,
Lucerne, Switzerland, September 2002
48
Gamut Mapping and Visualization
„
Current research topics
„
„
Image-dependent and/or
interactive new gamut
mapping algorithms
Importance of gamut
boundary descriptors:
„
„
Spatial gamut mapping
„
„
Segment-maxima, convex hull,
alpha-shapes, regular
structure, …
With University of Milano
Popular freeware tool
developed by us
„
„
ICC3D
http://www.colorlab.no/icc3d/
Ivar Farup and Jon Y. Hardeberg, Interactive
Color Gamut Mapping, Proceedings of the
11th International Printing and Graphic Arts
Conference, Bordeaux, France, October 2002
Ivar Farup, Jon Y. Hardeberg, and Morten
Amsrud, Enhancing the SGCK Colour Gamut
Mapping Algorithm, Proc. CGIV 2004, pp.
520-524, Aachen, Germany, April 2004
Arne Magnus Bakke; Jon Yngve Hardeberg &
Ivar Farup. Evaluation of Gamut Boundary
Descriptors. IS&T and SID's 14th Color
Imaging Conference 50-55, 2006
49
Outline
„
„
„
„
Introduction
Color Science
Color Management
Color Image Quality
Ivar Farup; Carlo Gatta & Alessandro Rizzi. A
Multiscale Framework for Spatial Gamut
Mapping. IEEE Transactions on Image
Processing, 16(10):2423-2435, 2007
Øyvind Kolås & Ivar Farup. Efficient Huepreserving and Edge-preserving Spatial
Color Gamut Mapping. IS&T and SID's 15th
Color Imaging Conference, 207-212, 2007.
I. Farup, J. Y. Hardeberg, A. M. Bakke, S.
Kopperud, and A.Rindal, Visualization and
Interactive Manipulation of Color Gamuts,
Proc. 10th Color Imaging Conference, pages
250-255, Scottsdale, Arizona, November 2002
8
50
What is color image quality?
51
What is color image quality?
52
What is color
image quality?
53
What is color
image quality?
„
„
„
What is color?
What is an image?
What is quality?
„
„
Dictionary: “Degree
of excellence”
ISO9000: “The
features of a product
(or service) which
are required by a
customer”
54
55
Customer-driven quality definition
Quality
Quality
Quality
Quality
Quality
Quality
Quality
Quality
is
is
is
is
is
is
is
is
what
what
what
what
what
what
what
what
the
the
the
the
the
the
the
the
customer
customer
customer
customer
customer
customer
customer
customer
wants!
wants!
wants!
wants!
wants!
wants!
wants!
wants!
„
„
Norwegian governmental purchase of digital
print equipment
E.g. letterhead quality of primordial
importance
P. Nussbaum and J.Y. Hardeberg,
Print Quality Evaluation for
Governmental Purchase Decisions,
Proc. IARIGAI Conference,
Copenhagen, September 2004
9
NOT
56
How to evaluate color image quality?
How to evaluate color image quality?
„
From the web site of a major French
consumer electronics retailer:
„
„
NOT
How to evaluate color image quality?
Qualité d'image = taille du point x
nuances de couleurs
Image quality = point size x color depth
NOT
58
61
Psychophysical experiments
„
„
Measurements
„
„
„
Colorimetric and planimetric
Pixel-by pixel comparisons
Perceptual models etc.
Subjective evaluations
„
„
„
”Expert opinions”
Customer’s choices
Different protocols
„
„
„
„
„
„
59
How to evaluate color image quality?
60
How to evaluate color image quality?
57
Category judgment
Rank order
Pair comparison
Time consuming
Thorough statistical
analysis
„
But some questionable
assumptions?
Psychophysical experiments
„
„
”Panel tests”
Statistical analyses
10
62
Measuring color image quality
By color measurements
„
Match of specific colors
„
Match monitor/print
„
Match original-reproduction
„
Match to a reference print
„
„
„
„
Preferred color reproduction
Original
„
E.g. Company logo
Typical consumer user scenario
Generally it is not
desired to
reproduce the
colorimetry of the
original scene
Color copy, fax
Do not use numbers blindly!
(Giorgianni&Madden, 1997)
„
63
Mean ΔE*ab = ?
Max ΔE*ab = ?
Reproduction
64
Colorimetric match ≠ visual match
„
65
Inadequacy of RGB color difference
Basic colorimetry is inadequate for crossmedia color reproduction
ΔRGB=100
ΔRGB=100
ΔEab=22
ΔEab=63
CIEDE2000=4,8
CIEDE2000=32
66
Inadequacy of pixel-by-pixel difference
67
Perceptual image difference metrics
„
Reproduction 1
„
„
„
Original
Reproduction 1 and 2 have the same average
ΔEab of 0.6
CIE ΔEab colour difference equation is
designed to quantify the colour difference for
single pair of colour patches
Don’t ever use RMS or PSNR to evaluate your
image compression algorithm again!
Exactly how big is the difference between
these two images?
Reproduction 2
ΔE ab = 35
11
68
Perceptual image difference metrics
„
„
„
Perceptual image difference metrics
Metrics taking into account visual CSFs have been
proposed
„
69
Reproduction
Original
s-CIELAB
iCAM
Many other
metrics
„
SSIM
S-CIELAB
iCAM
CIELAB
Image difference maps
70
Perceptual image difference metrics
„
„
„
We found no correlation
between current image
difference metrics and
perceptually evaluated
image differences
Higher-level cognitive
processes come into play
Current active research
area
71
Perceptual image difference metrics
„
E. Bando, J.Y. Hardeberg, D. Connah,
Can Gamut Mapping Quality be
predicted using Colour Image
Difference Formulae?, SPIE Proc.
5666, 2005
Research project funded by the
Research Council of Norway
„
„
Ca 1.4 MNOK over 4 years 20072011
New metrics development/evaluation
„
„
„
Implement and evaluate existing
methods
New metrics based on improved
colour difference metrics
New metrics based on perceptual
predictors
„
„
„
”Most studies on image quality employ subjective
assessment with only one goal – to avoid it in the future”
„
„
„
„
„
Image compression
High dynamic range tone mapping
Colour gamut mapping
Halftoning
Eye tracking experiments
New metrics based on multilevel
methods
Outline
„
„
„
CIQ
machine
Applications
73
„
Yendrikhovskij
„
„
Standard optimization methods
Heuristic methods
Variational methods
Such as Retinex
72
Perspectives on image quality research
„
New metrics based on saliency maps
„
„
Using metrics to improve
reproduction&representation
„
Related project funded by Océ Print
Logic Technologies (Paris, France)
„
„
„
Ca 5.4 MNOK over 4 years 20072011
Introduction
Color Science
Color Management
Color Image Quality
CIQ=100!
12
74
75
The Norwegian Color Research Laboratory
„
Research group at the Faculty of Computer
Science and Media Technology at Gjøvik
University College
„
„
Research in
color science and image processing
„
Rooted in graphic arts industry needs but now serving
the converging media technology industry
„
„
Great people
„
„
„
„
„
„
„
„
3 permanent faculty
3 associated faculty
4 doctoral students and 1 postdoctoral researcher
1 color management engineer
Bachelor and master students
„
„
„
„
Color and spectral measurement, Color management
software, viewing booths, image acquisition and
reproduction devices, monochromator, etc.
>200m2 lab space
Strong ties to the faculty’s media related
bachelor and master programs
„
Well equipped laboratory facilites
„
„
„
Color Management and Device Characterization
Color Image Quality
Color Gamut Mapping and Visualization
Multispectral Color Image Acquisition and
Reproduction
Fundamental Colorimetry and Psychophysics
http://www.hig.no/media
http://www.master-erasmusmundus-color.eu/
Rich national and international network
with industry and academia
http://www.colorlab.no
76
Colorlab Research Overview
„
Applied research
„
„
„
„
„
„
Graphic arts industry
Color management
Print quality evaluation
PSO color certification
„
„
„
„
„
Display technology
Movie production
Photo management
Teleconferencing (?)
„
Computer vision
„
Security applications
„
„
„
„
„
„
Recycling sorting
„
„
People counting
Intelligent video
surveillance
Document and image
forensics
„
EU- funded through Marie Curie program
„
Partners:
„
„
Color memory
Eye movements
„
„
In-between…
„
„
„
„
Print-oriented courses in the UK, fall 2008
Large final conference in Gjøvik in 2010
„
„
NFR/SHP 2007-2011
5.51 MNOK
http://www.create.uwe.ac.uk/
Series of conferences and research courses
„
Color and perception
„
Color Research for European Advanced
Technology Employment
„
„
NFR/SHP 2003-2007
4.8 MNOK
Perceptual image
difference metrics
„
„
„
Multispectral color
imaging
„
Media technology industry
CREATE
Fundamental research
„
77
„
Gamut mapping
Print and image quality
Device modeling
Face detection
„
„
Free participation (including travel) for selected
young researchers
Università degli Studi di Milano, Italy;
Gjøvik University College, Norway;
University of Leeds, UK
University of Ulster in Belfast, Northern Ireland,
Universidad Autónoma de Madrid, Spain;
Université de Reims Champagne-Ardenne,
France;
University of Pannonia, Hungary.
78
CIMET key facts
„
„
MSc programme: Color in Informatics and
Media Technology
Collaborative effort supported by the EU
„
„
„
Prestigeous for institutions
„
„
In Master Mundus programme: bringing together the best
Europan forces to attract the best brains to Europe
University of Saint-Etienne (France, coordinator), the
University of Granada (Spain), the University of Joensuu
(Finland), and Gjøvik University College (Norway)
Only three selected programmes in Norway in 2007
Questions?
„
„
Generous grants from EU
„
Highly competitive selection
„
To cover tuition fees (EUR 10.060 per year)
E-mail:
„
„
„
„
„
http://www.hig.no
http://color.hardeberg.com
http://www.colorlab.no
„
Thanks to:
„
Upcoming events:
„
„
Current and past Colorlab faculty and students
Open Colorlab Workshops
„
Only 30 students per year
http://www.master-erasmusmundus-color.eu/
jon.hardeberg@hig.no
jon@hardeberg.com
Web:
„
Prestigeous for students
„
„
79
„
„
4.6.08: Psychophysical experiments and statistics
19.6.08: Face detection and recognition
Gjøvik Color Imaging Symposium 2009
„
Immediately following SCIA09
13