NO-REFERENCE SYNTHETIC
IMAGE QUALITY ASSESSMENT
USING SCENE STATISTICS
Debarati Kundu and Brian L. Evans
Embedded Signal Processing Laboratory (ESPL)
Department of Electrical and Computer Engineering
The University of Texas at Austin
49th Asilomar Conference on Signals, Systems and Computers
November 11, 2015
2
Outline
• Introduction
• Subjective Quality Evaluation of Synthetic Scenes
• No-reference Objective Quality Evaluation of Synthetic Scenes
• Classification of Distortion Type in Synthetic Scenes
• Conclusion
3
Motivation
• Natural images from optical cameras
~350M photos/day uploaded to Facebook
• Computer graphics in video games
Massively multiplayer online (23.4M subscribers)
Cloud-based (e.g. Nvidia Grid servers)
Livestreaming (e.g. YouTube Gaming)
• Animated movies and synthetic images
• Visual quality assessment
Give immediate feedback to content creators
Maximize visual quality during compression
Reduce rendering complexity
Source: Statista.com
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
4
Image Quality Assessment (IQA) Algorithms
“Full-Reference”
(FR) IQA require
an “original”high
quality image for
comparison.
Reference
Test
“No-Reference”
(NR) IQA
assume that no
“original image”
is available for
comparison.
Supplemental
Information
Full Reference
“Reduced- IQA
Reference” (NR)
IQA require
supplementary
information about
the original image
Introduction |
ESPL Database|
Reduced
Reference IQA
Subjective
Opinion Scores
No-Reference
IQA
Correlation
Performance
[Dr. Alan C. Bovik,
EE381V Digital Video,
Spring 2015]
ESPL Objective Image Quality| Distortion Classification| Conclusions
5
ESPL Synthetic Image Database
• 25 reference images + 500 distorted images
• 5 distortion categories
• Annotated with differential mean opinion scores (DMOS)
• 52 observers: 12 among 64 subjects removed as outliers
• Single Stimulus Continuous Quality Evaluation with hidden reference
• Website: http://signal.ece.utexas.edu/~bevans/synthetic/
Reference (DMOS = 0)
Introduction |
ESPL Database|
Distorted (DMOS = 74.68)
ESPL Objective Image Quality| Distortion Classification| Conclusions
6
ESPL Database: Distortions
Interpolation
(DMOS = 63.23)
Additive Noise
(DMOS = 60.33)
Introduction |
ESPL Database|
Original
(DMOS = 0)
Blur
(DMOS = 50.89)
JPEG Compression
(DMOS = 74.68)
Fast Fading
(DMOS = 60.26)
ESPL Objective Image Quality| Distortion Classification| Conclusions
7
No-Reference IQA Algorithms
• Many use Natural Scene Statistics (NSS)
Distribution of pixel values, transform coefficients or MSCN pixels
Natural pristine images have same distribution regardless of content
Distortion causes deviation from statistical distribution
• Mean Subtracted Contrast Normalized (MSCN) pixels [Ruderman1993]
Models normalization in primary visual cortex
MSCN uses K x L neighborhood around pixel I(i, j) and Gaussian weights wk,l
I(i, j) - m (i, j)
Iˆ(i, j) =
s (i, j) +1
m (i, j) =
K
L
å åw
k.l
I(i + k, j + l)
Mean
k=-K l=-L
s (i, j) =
K
L
å å w [ I(i + k, j + l) - m (i, j)]
k.l
2
Standard
deviation
k=-K l=-L
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
8
Mean Subtracted Contrast Normalization
Original Image
MSCN Image
Introduction |
ESPL Database|
Standard Deviation Image
ESPL Objective Image Quality| Distortion Classification| Conclusions
9
Distorted Image Statistics in Different Domains
• Different distortions affect scene statistics characteristically
• Can be used for distortion classification and blind quality prediction
MSCN Coefficients
Steerable Pyramid Wavelet
Coefficients
Curvelet Coefficients
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
10
Learning-Based NR-IQA on ESPL Database
Model based feature
extraction
Support Vector
Machine Regressor
Input Image
Correlation
Algorithms
Features
BRISQUE
MSCN coefficients
BLIINDS-II
DCT coefficients
DIIVINE
Steerable Pyramid
GM-LOG
Gradient Magnitude +
Laplacian of Gaussian
CORNIA
Dictionary learning
Introduction |
Predicted
Quality Score
ESPL Database|
Pretrained model
Human
Scores
Distorted images split into training & test
sets in the ratio of 4:1
Split randomized over multiple trials
Median of correlation values over 1000
trials considered here.
ESPL Objective Image Quality| Distortion Classification| Conclusions
11
Learning based
methods
Correlation of NR-IQA and Human Scores
Spearman’s Rank Ordered Correlation Coefficient for different distortions
Algorithms
Interpolatio
n
Blur
Additive
Noise
JPEG
Blocking
Fast
Fading
Overall
DESIQUE
0.701
0.872
0.917
0.932
0.819
0.888
GM-LOG
0.798
0.800
0.881
0.902
0.796
0.881
CORNIA
0.800
0.870
0.858
0.884
0.777
0.847
BLIINDS-II
0.806
0.838
0.879
0.885
0.726
0.837
NIQE
0.364
0.354
0.835
0.385
0.392
0.470
LPCM
0.415
0.834
0.623
0.211
0.108
-
CPBDM
0.676
0.757
0.746
0.765
0.347
-
FNVE
0.320
0.463
0.863
0.517
0.461
-
JPEG-NR
0.540
0.593
0.748
0.928
0.464
-
Introduction |
Noise
evaluation
ESPL
Database|
metrics
Blockiness
evaluation
ESPL Objective Image Quality|
metric Distortion Classification|
Blur
evaluation
metrics
Conclusions
12
SROCC
Training Robustness in Learning-Based Methods
• For each box, red central line is median of SROCC over 1000 trials
• Bottom/top of each box represents 25th/75th percentiles
• Whiskers span most extreme non-outlier data points
• Outliers plotted individually
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
13
Classification of Distortions in ESPL Database
Algorithms
Interpolation
Blur
Additive
Noise
JPEG
Blocking
Fast
Fading
Overall
GM-LOG
100.0
96.0
100.0
96.5
96.6
97.8
C-DIIVINE
94.5
96.2
100.0
94.5
93.8
95.8
BRISQUE
94.4
96.6
100.0
91.8
89.8
94.4
BLIINDS-II
91.6
87.7
100.0
81.3
74.6
85.9
CurveletQA
88.4
85.8
100.0
81.3
74.6
85.9
Singular value decomposition of NR-IQA features in 2-D
GM-LOG features
Introduction |
ESPL Database|
BLIINDS-II features
ESPL Objective Image Quality| Distortion Classification| Conclusions
14
Conclusions
• Distortions change scene statistics of synthetic scenes similar to the
case for natural scenes
• Learning based generic models perform better than models meant for
individual distortions
• Multiple domain feature combining models perform best
Future Work
• Create an image database with shading and color artifacts
• Design NR-IQA to correlate better with interpolation artifacts
• Use NSS models to evaluate photorealism of rendered scenes
ESPL Database: http://signal.ece.utexas.edu/~bevans/synthetic/
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
15
Questions?
16
Algorithm Acronyms
• BLIINDS-II: Blind Image Quality Assessment DCT domain (BLIINDS-II)
•
•
•
•
•
•
•
•
[Saad2012]
CORNIA: Codebook Rep for No-Ref Image Assessment (CORNIA) [Ye2012]
DESIQUE: DErivative Statistics-based Quality Evalulator [Zhang2013]
DIIVINE: Distortion Identification-based Image Verity and INtegrity
Evaluation [Moorthy2011]
GM-LOG: Gradient Mag-Laplacian of Gaussian Index [Xue2014]
NIQE: Natural Image Quality Evaluator [Mittal2013]
LPCM: Local Phase Coherence Measurement [Hassen2013]
CPBDM: Cumulative Probability of Blur Detection Metric [Narvekar2011]
FNVE: Fast Noise Variance Estimation [Immerkar96]
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
References
17
• [Bremond2010] J. Petit, R. Bremond, and J.-P. Tarel, “Saliency maps of high dynamic range images," in Proceedings of the
•
•
•
•
•
•
•
•
•
•
6th Symposium on Applied Perception in Graphics and Visualization , ser. APGV '09. New York, NY, USA: ACM, 2009.
[Ferwerda2007] G. Ramanarayanan, J. Ferwerda, B. Walter, and K. Bala, “Visual equivalence: Towards a new standard for
image fidelity,” in SIGGRAPH ACM, 2007
[Kundu 2014] D. Kundu and B. L. Evans, ``Spatial Domain Synthetic Scene Statistics'', Pro c. Asilomar Conf. on Signals,
Systems, and Computers, Nov. 2-5, 2014, Pacifi c Grove, CA USA.
[Larson1997] G. W. Larson, H. Rushmeier, and C. Piatko. 1997. A Visibility Matching Tone Reproduction Operator for High
Dynamic Range Scenes. IEEE Transactions on Visualization and Computer Graphics 3, 4 (October 1997), 291-306.
[Larson2010] E. C. Larson and D. M. Chandler, “Most apparent distortion: fullreference image quality assessment and the
role of strategy,” J Electronic Imaging, vol. 1 9, no. 1, p. 011006, 2010.
[Ma2015] Kede Ma; Yeganeh, H.; Kai Zeng; Zhou Wang, "High Dynamic Range Image Compression by Optimizing Tone
Mapped Image Quality Index," in Image Processing, IEEE Transactions on , vol.24, no.10, pp.3086-3097, Oct. 2015
[McGuire2012] M. McGuire, P. Hennessy, M. Bukowski, and B. Osman. 2012. A reconstruction filter for plausible motion
blur. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D '12), ACM, New
York, NY, USA, 135-142.
[Moorthy2011] A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From natural scene statistics to perceptual
quality,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3350– 3364, Dec 2011.
[Nafchi2014] H. Ziaei Nafchi, A. Shahkolaei, R. Farrahi Moghaddam, and M. Cheriet, “Fsitm: A feature similarity index for
tone-mapped images," Signal Processing Letters, IEEE , vol. 22, no. 8, pp. 1026-1029, Aug2015.
[Narwaria2013] M. Narwaria, M. Perreira Da Silva, P. Le Callet, and R. Ppion, “Tone mapping-based high-dynamic-range
image compression: study of optimization criterion and perceptual quality," Optical Engineering , vol. 52, no. 10, pp. pp. 102
008-1 102 008-15, Oct 2013.
[Ruderman1993] D. L. Ruderman and W. Bialek, “Statistics of natural images: Scaling in the woods,” in Neural
Information Processing Systems Conference and Workshops, 1 993, pp. 551–558.
18
References (cont’d)
• [Saad2012] M. A. Saad, A. C. Bovik, and C. Charrier, “Blind image quality assessment: A natural scene statistics
•
•
•
•
•
•
•
•
•
approach in the DCT domain.” IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3339–3352, 2012
[Wang2003] Z.Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality
assessment,” in Proc. Asilomar Conference on Signals, Systems and Computers, vol. 2, Nov 2003, pp. 1398–1402
Vol.2.
[Winkler2012] Winkler, S., "Analysis of Public Image and Video Databases for Quality Asses sment," in Selected
Topics in Signal Processing, IEEE Journal of , vol.6, no .6, pp.616-625, Oct. 2012
[Xue2014] W. Xue, X. Mou, L. Zhang, A. C. Bovik, and X. Feng, “Blind image quality assessment using joint
statistics of gradient magnitude and laplacian features,” IEEE Transactions on Image Processing, vol. 23, no. 11,
pp. 4850–4862, Nov 2014
[Ye2012] Peng Ye, Jayant Kumar, Le Kang, and David Doermann, “Unsupervised Feature Learning Framework for
No-reference Image Quality Assessment,” in Proc. Intl. Conf. on Computer Vision and Pattern Recognition, June
2012, pp. 1098–1105
[Yeganeh2013] H. Yeganeh and Z. Wang, “Objective quality assessment of tone-mapped images," Image
Processing, IEEE Transactions on , vol. 22, no. 2, pp. 657-667, Feb 2013.
[Zhang2011] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality
assessment,” IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378–2386, Aug 2011
[Zhang2012] L. Zhang and H. Li, “SR-SIM: A fast and high performance IQA index based on spectral residual,” in
Proc. IEEE International Conference on Image Processing, Sept 2012, pp. 1473–1476
[Zhang2013] Y. Zhang and D. M. Chandler, “No-reference image quality assessment based on log-derivative
statistics of natural scenes,”Journal of Electronic Imaging, vol. 22, no. 4, 2013
[Zhang2014] L. Zhang, Y. Shen, and H. Li, “VSI: A visual saliency-induced index for perceptual image quality
assessment,” IEEE Transactions on Image Processing, vol. 23, no. 10, pp. 4270–4281, Oct 2014
19
Backup
20
ESPL Database: Distortion Parameters
• Interpolation
• Images downsampled by factors ranging from 3 to 6
• Upsampled back using nearest neighbor interpolation
• Blur
• RGB color channels filtered with circularly symmetric 2D Gaussian kernel
• Standard deviation ranging from 1.25 to 3.5 pixels
• Same kernel employed for each color channels
• Additive Noise:
• Same noise variance used for all color channels
• Noise standard deviation ranged from 0.071 to 0.316 pixels
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
21
ESPL Database: Distortion Parameters (cont’d)
• JPEG compression
• MATLAB imwrite function was used
• Bits per-pixel (bpp) ranged from 0.0445 to 0.1843.
• Higher bpp images were not considered to better simulate playing a cloud
video game under restricted bandwidth conditions.
• Simulated Fast Fading Channel
• Original images compressed into JPEG2000 bitstreams
• Wireless error resilience features enabled and 64 x 64 tiles
• Transmitted over a simulated Rayleigh-fading channel
• Signal-to-noise ratio (SNR) was varied at the receiver from 14 to 17 dB
Back
• SNRs greater than 17 dB did not introduce perceptible distortions due to
the error resilience feature of the JPEG2000 codec.
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
22
ESPL Database: Methodology
• Single Stimulus Continuous Quality Evaluation (SSCQE) method
• Each subject evaluated each image
• Three sessions, of one hour each, separated by at least 24 hours.
• Each session divided into two sub-sessions of 25 minutes
• Separated by a break of five minutes.
• 64 subjects
• Age range : 18 – 30 years
• Mostly without prior experience in participation of subjective tests
• Verbal confirmation of 20/20 (corrected) vision was obtained
• Viewed roughly 175 test images during each session
• Randomly ordered using a random number generator
• Testing sessions were preceded by training session of 10 images
Introduction |
ESPL Database|
Back
ESPL Objective Image Quality| Distortion Classification| Conclusions
23
ESPL Database : Methodology (cont’d)
• User interface programmed on MATLAB using Psychology Toolbox
• NVIDIA Quadro NVS 285
• Dell 24 inches U2412M display
• Normal office illumination
• Each image displayed for 12 seconds
• Viewing distance: 2-2.25 times display height
• Scores between 0-100 was entered
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
24
ESPL Database: Processing of raw scores
• Raw scores were first converted to raw quality difference scores:
dij = riref ( j ) - rij
• rij: Score assigned to j-th image by the i-th subject
• riref(j): Score assigned by same subject to corresponding reference
• Difference scores normalized for each subject and averaged
Histogram of DMOS scores
Introduction |
ESPL Database|
Scatter plot of DMOS scores
Back
ESPL Objective Image Quality| Distortion Classification| Conclusions
25
ESPL Database: Processing of raw scores
• Raw scores were first converted to raw quality difference scores:
dijk = riref ( j)k - rijk
• rijk: Score assigned to j-th image by the i-th subject in k-th session
• riref(j)k: Score assigned by same subject to corresponding reference
• DMOS score is zero for reference images
• DMOS scores converted to Z-scores per session
N
1 ik
mik =
å dijk
N ik j=1
zij' =
N
ik
1
s ik =
(dijk - mik )2
å
N ik -1 j=1
zijk =
dijk - mik
100(zij + 3)
6
1 M '
DMOS j = å zij
M i=1
s ik
Nik: Number of videos seen by i-th subject in k-th session
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
26
ESPL Database: Outlier rejection
• Done as per ITU-R BT 500.11 recommendation
• Compute kurtosis of scores per subject to check Gaussianity
• If kurtosis falls between the values of 2 and 4 (Gaussian)
• Subject rejected if more than 5% of his scores falls outside ±2σ from mean.
• For non Gaussian distributions
• Subject rejected if more than 5% of his scores falls outside ±4.47σ from mean.
• 12 out of 64 subjects rejected
• Testing degree of consensus among subjects:
• Subjects divided into two groups randomly
• DMOS scores for all the image calculated individually from each group
• Pearson’s linear correlation coefficient was 0.9813 between the groups
• Shows a high level of consensus among the subjects
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
27
ESPL Database: Calculating correlations
• Let Qj be the quality predicted by the IQA algorithm for the j-th image.
• Four parameter monotonic logistic function fit IQA predictions to quality
scores:
b1 - b2
Q'j = b2 +
1+ exp[-
Q j - b3
]
| b4 |
N
• Spearman’s Rank-order correlation coefficient: SRCC = 1-
6å di2
i=1
2
N(N -1)
• where di is the difference between the i-th image’s ranks is subjective and
objective evaluations.
• Kendall’s correlation coefficient: KCC = N c - N d
0.5N(N -1)
• Nc and Nd are the number of concordant (of consistent rank order) and
discordant (of inconsistent rank order) pairs in the data set respectively.
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
28
ESPL IQA: Modeling of MSCN Gradients
Distribution of the gradient magnitude of MSCN coefficients
•
•
Modeled using Rayleigh, Weibull and Nakagami distributions
Mean square error, J-Divergence, and Pearson’s Chi-squared values for distributions fitted to
histograms of MSCN coefficients, averaged over 221 pristine synthetic images [Kundu 2014]
MSE
J
Chi Square
Rayleigh
0.00891
4.730
0.769
Weibill
0.0251
5.00432
0.663
Nakagami
0.00916
5.304
0.892
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
29
ESPL IQA: Comparison with Natural Scenes
Histogram of scale parameter
(JS divergence = 1.5655)
Kurtosis
Histogram of shape parameter
(JS divergence = 1.0503)
Skewness
Skewness-kurtosis scatter plot
of MSCN coefficients of synthetic images[Kundu2014] (221 images) and natural images
Back
[Martin2001] (500 images from Berkeley Segmentation Dataset)
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
30
Generalized Gaussian Density
• The GGD
pg (r)=
b
( )
2sG b
-1
(
exp r / s
)
b
r ÎÂ, s,b>0
includes the special cases
b = 1 (laplacian density)
b = 2 (gaussian density)
b = ¥ (uniform density)
• Many authors have observed the GGD behavior of BP image
signals.
[Ref: Dr. Alan C. Bovik,
EE381V Digital Video,
Spring 2015]
• Wavelet coefficients
• DCT coefficients
• Usually reported that b » 1 but varies (0.8 < b < 1.4).
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
31
Symmetric Alpha Stable
• A random variable X is called stable if its characteristic function can be
expressed as:
• sgn(t): sign of t
•
Back
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
32
Steerable Filter
• Create directional derivative of gaussian in arbitary direction
by a coordinate rotation by q:


  x 2  y2 
1

h θ (x) =
 cos θ  x   sin θ  y  exp 
4 
2


2πσ
2σ


2
 j   cos θ  U   sin θ  V   exp  2  πσ  U 2  V 2 




• In particular:
h θ (x) =  cos θ  h 0 (x)   sin θ  h π (x) 
 cos θ  H 0 (u)  sin θ  H π (u) = Hθ (u)
2
2
• A derivative-of-gaussian filter of any orientation (derivative
direction ) is exactly a linear combination of two orthogonal
derivative-of-gaussian filters
[Ref: Dr. Alan C. Bovik, EE381V Digital Video, Spring 2015]
Introduction |
ESPL Database|
Back
ESPL Objective Image Quality| Distortion Classification| Conclusions
33
ESPL IQA: Steerable Pyramid comparison
(a) Orientation = 0 degree
(b) Orientation = 60 degrees
(c) Orientation = 0 degree
(d) Orientation = 60 degrees
Histogram of scale (a)(b) and shape parameters(c)(d) of the steerable pyramid decomposition
of synthetic images (221 images) and natural images
Back
[Martin2001] (500 images from Berkeley Segmentation Dataset)
Introduction |
ESPL Database|
ESPL Objective Image Quality| Distortion Classification| Conclusions
34
ESPL IQA: Performance-Time Complexity
Tradeoff
• Some NR-IQA metrics comparable with the best FR-IQA
algorithms.
Introduction |
ESPL Database|
Back
ESPL Objective Image Quality| Distortion Classification| Conclusions