2012 International Conference on Networks and Information (ICNI 2012)
IPCSIT vol. 57 (2012) © (2012) IACSIT Press, Singapore
DOI: 10.7763/IPCSIT.2012.V57.07
1
1
1
1 +
1 Digital Scientific Imaging Lab, Graduate School of Advanced Imaging Science, Chung-Ang University
We need a practical evaluation method for image quality that can measure consumers’ recognition of changed image quality. In this study, we build a subjective image quality assessment method based on objective image quality factors in order to evaluate consumers’ preference objectively. To measure consumers’ recognition of image quality, the close relationships between objective image quality measurement and the subjective image quality assessment must be understood. In this approach, we broaden previous research based on adjective word evaluation. Also, we extend objective image quality measurement into general words that can be easily understood by both manufacturers and consumers.
image quality, preference, subjective assessment.
Since Kodak introduced the world’s first digital camera in 1975, the existing film camera markets have been transferring to the new paradigm of digital cameras. Digital cameras have spread out with low cost and high quality, and they remain daily necessaries that can photograph easily and allow people to enjoy images
[1]. Therefore, digital camera markets extensively advertise image quality advancement, and the objective image quality evaluations meet the needs and interests of consumers. In the process of consumers evaluating image quality, there are two attributes: the objective properties and psychophysical recognitions. The consumer’s subjective response to image quality is presented as the final determination of image quality.
Therefore, we need a practical evaluation method for image quality that can measure consumers’ recognition of changed image quality. In this study, we build a subjective image quality assessment method based on objective image quality factors in order to evaluate consumers’ preference objectively. As objective image quality factors, we select the factors set by the International Standards Organization: dynamic range, resolution, noise, and color reproduction. For the image stimuli, we chose a portrait image for its popularity.
Portrait image stimuli in this test contain about 50% skin tone in the whole image. Therefore, we establish a guideline to evaluate image quality based on skin tone, color and texture. For the general subjects, we use easy words in the subjective image quality assessment. After the survey, we apply the most frequently selected words and build a subjective image quality assessment in order to measure image quality preference.
In this approach, we broaden previous research based on adjective word evaluation. Also, we extend objective image quality measurement into general words that can be easily understood by both manufacturers and consumers. According to our new subjective image quality assessment, we can measure the subject’s preference and analyze the relationship between preference and objective attributes affecting to preference.
Engeldrum(2004) suggested a “Complete Image Quality Circle” that could be used to evaluate final consumer preference based on customer quality preference, technology variables, system models, physical
+ Corresponding author. Tel.: +82-2-820-5749; fax: +82-2-817-0979.
E-mail address : dhhar@cau.ac.kr.
34

image parameters, visual algorithms, customer perceptions, and image quality models. Each attribute factor affects a consumers’ image quality preference and are strongly related [3]. Throughout The Complete Image
Quality Circle, Engeldrum suggested that preferred images should be applied to product development.
However, it shows a limitation in the concept model in that it does not contain a specific method of evaluating subjective or objective image quality measurement factors. Based on Engeldrum’s research, we produce a synthesized subjective image quality evaluation model. We adopt objective image quality factors as measurement metrics and analyze the results based on preference correlation. According to Park (2011), it is impossible to draw a meaningful correlation between the rank of objective image quality factors and subjective image preference. Due to the differences of scales among objective image quality attributes and preference ranks, it is hard to analyze correlations statistically. Therefore, non-parametic statistics
(Spearman's rank order correlation) were used to analyze correlation, but the p-value was insignificant [refer
Table 1], [4].
Table 1. Correlation between the rank of objective image quality factors and preference
Canon 5dmk2
Panasonic FX65
Sony W290
Canon 500D
Olympus PEN
Samsung NX10
Nikon D300 p-value <0.1
1
2
3
4
5
6
7
-
5
2
3
4
6
7
1
0.456
Resolution(LW/PH)
3
4
2
5
6
1
7
0.517
Noise
1
6
7
3
5
4
2
0.671
Color reproduction( ∆ E)
2
4
5
1
7
6
3
0.584
However, there are corresponding factors throughout the rank order analysis between objective image quality factors and consumer preference. Image quality is the total impression of objective image quality and the physiological recognition process. Therefore, evaluation items are needed to contain the meaning of objective image quality factors. A new subjective image quality assessment is presented, which has image quality questions based on objective image quality factors that are expressed with general words. The assessment method can be used to analyze the correlation between factors affecting the final image quality preference evaluation. In this research, we build a new subjective image quality assessment based on objective image quality factors.
For the scene selection used in subjective evaluation, we refer to the most frequently photographed scenes, which consist of portrait (87%) and non-portrait (13%) images, based on analyzing 1300 photos from the web [4]. In this study, we select portraits containing over 50% skin tone as stimuli. To test digital cameras, 7 cameras with diverse sensor size are used, using Full-Frame, APS-C (Advanced Photo System type-C), and Hybrid sensor formats. Each scene was shot with the same object distance and angle without the use of auto scene-recognition functions [refer table 2].
Table 2. Test cameras
Canon 5dmk2 Canon 500D
21.1 Mega pixel 15.1 Mega pixel
Nikon D300
12.3 Mega pixel
Samsung NX10
14.6 Mega pixel
Olympus PEN
36x24mmCMOS 22.3x14.9mm CMOS 23.6x15.8mm CMOS 23.4x15.6mm CMOS 4/3"CCD
Sony W290
12.3 Mega pixel 12.0 Mega pixel 10.0 Mega pixel
1/2.3"CCD
Panasonic FX65
1/2.3"CCD
Park(2012) surveyed subjects who expressed their recognition of images with adjectives. The images presented were made by controlling objective image quality factors. The most frequently used words were ‘sharp’, ‘clear’, ‘bright’, and ‘distinct’. These words can be used in subjective image quality assessment questions. The survey questions were composed as follows: The first question, which pertained to resolution, asks the subject to assess the image quality on a
5 tier rating scale in terms of the words ‘sharp’ and ‘unsharp,’ which were the most selected adjectives,. For the second
35

question, the word ‘graininess’ was used to represent noise, the qualities ‘clear’ and ‘unclear’ were rated on a
5 tier rating scale. The third question pertains to color reproduction evaluation. It consists of a 5 tier rating scale of image quality from worse to better. The fourth question, regarding dynamic range, asks the subject to evaluate the range from highlight to the shadow in an image. The words ‘sufficient’ and ‘insufficient’ were evaluated on a 5 tier rating scale. For the last question, subjects were asked to evaluate the image in terms of the words ‘good’ and ‘bad’ on another 5 tier rating scale. Subjects were asked to answer the questions based on intuition. The actual survey questions are shown in figure 1.
Fig. 1: Image quality assessment survey sample
According to ISO 20462 (2004), untrained subjects are desirable to decide the image quality with specific standard stimuli [5]. Therefore, this survey uses objective image quality evaluation factors which are based on easy words from previous research. Subjects in the survey were asked to select answers within 5~6 seconds in order to prevent contrived answers and to obtain intuitive evaluations. Stimuli are shown using the triple-comparison method, which has high connectivity and repeatability according to ISO 20462. In other words, for digital cameras numbered 1-7, one scene would be photographed using a sequence of cameras such as (1, 2, 4), (2, 3, 5), (3, 4, 6), (4, 5, 7), (5, 6, 1), (6, 7, 2), (7, 1, 3), without repetition. The numbers represent camera models. By using three stimuli comparison, we try to reduce the subjects’ stress and survey time.
We employ a softcopy test method using monitors, and 100 undergraduate students as subjects who are familiar with using digital cameras. The students have normal eyesight and have passed a color blindness test.
The test monitors used are Samsung Syncmaster 400 Pn models calibrated by X-rite Eyeone Display 2. We follow the ISO 3664 standard, which provides test environmental conditions for observing images, as well as the ISO 20462 recommendations for subjective image quality assessment. According to ISO recommendations, for considerations of stress of the subjects and for test efficiency, they suggest that the optimum number of stimuli is fewer than 27 sets, presented within 45 minutes to 1 hour. Also, for the reliability of the test, the minimum number of subjects should be 10, but over 20 are recommended. In this test, we present 7 stimuli within 25 minutes to 100 subjects.
In the subjective image quality assessment questions, we set 4 questions for objective image quality measurement factors, and add a preference question to determine the subjects’ preferred image. These 5 factors in the assessment model are named the Complete Image Quality Evaluation (CIQE): CIQE dynamic range, CIQE noise, CIQE resolution, CIQE color reproduction, and CIQE preference. We use descriptive statistics and correlation analysis in order to analyze the subjective image quality assessment results. The results are verified by significance levels of p<.05, p<.01, and p<.001 by PASW Statistics 18. We analyze the correlation between CIQE dynamic range, CIQE noise, CIQE resolution, CIQE color reproduction, and
36

CIQE preference using 7 sets of stimuli shown using a three-comparison method. Subjects answer the first question which asks for their perception of differences in resolution. The resolution answer choices are
‘unsharp’, ‘a little unsharp’, ‘normal’, ‘a little sharp’, and ‘sharp’. The second question for noise asks about the grain of an image, and subjects can decide between ‘unclear’, ‘a little unclear’, ‘normal’, ‘a little clear’, and ‘clear’. The third color reproduction question evaluates subjects’ color preferences, with ‘worse’, bad’,
‘normal’, ‘good’, and ‘better’ as possible answers. The fourth dynamic range asks about the sufficiency of details from highlight to shadow, with possible answers of ‘insufficient’, ‘a little insufficient’, ‘normal’, ‘a little sufficient’, and ‘sufficient’. The final total preference question asks about the subjects’ preference for an image, with possible answers of ‘worse’, bad’, ‘normal’, ‘good’, and ‘better’. The total descriptive statistics results are shown in table 3. The mean values of CIQE resolution, CIQE preference, CIQE color reproduction, CIQE dynamic range, and CIQE noise are 3.36, 3.17, 3.13, 3.13, and 3.10, respectively.
Table 3. Statistical results of survey
CIQE resolution
CIQE noise
CIQE color reproduction
CIQE dynamic range
CIQE preference
N
100
100
Min.
3
2
Max.
4
4
Mean value
3.36
3.10
Standard deviation
.414
.372
100 2 4 3.13 .392
100
100
2
1
4
4
3.13
3.17
.379
.484
In other words, subjects answered the 5 questions as a ‘normal’ decision due to the matter of answer method. We use a Likert-type scale, which has the same interval scale for an item, and measures values and calculates the total scores. In this process, subjects usually select a medium value and avoid the extreme scales [7]. As a result, subjects have a tendency not to select a perceived adjective, but to choose normal.
This means that it is hard to analyze the difference between each question. Therefore, we analyze the Pearson correlation between the total preference and CIQE dynamic range, CIQE noise, CIQE resolution, and CIQE color reproduction. Table 4 shows each item correlation. For example, CIQE resolution and CIQE preference have a correlation coefficient of r=.637 (p<.01), which means a positive correlation. The maximum correlation is 1.0 and the minimum correlation is 0. Once subjects recognize that the resolution is sharp in an image, they prefer that image.
Table 4. Pearson correlation between CIQE factors
CIQE
Resolution
CIQE Noise
Pearson CC 1
P-value
N 100
Pearson CC .494(**) 1
P-value .000
CIQE color reproduction
CIQE dynamic range
CIQE preference
CIQE color reproduction
CIQE dynamic range
CIQE preference
Pearson CC .669(**) .735(**)
P-value .000 .000
Pearson CC .605(**) .658(**)
P-value .000 .000
N 100 100
Pearson CC .637(**) .618(**)
P-value .000 .000
N 100 100
1
100
.860(**)
.000
100
.830(**)
.000
100
1
100 100
.806(**) 1
.000
100 100
** p < .01, CC=Correlation Coefficient
CIQE noise and CIQE preference show an r=.618 (p<.01) correlation coefficient, meaning that if the grains in an image look soft, the image is preferred by subjects. Also, CIQE color reproduction and CIQE preference have an r=.830 (p<.01) correlation coefficient, which is the highest correlation in this study. If subjects perceive good color reproduction in an image, it is preferred. CIQE dynamic range and CIQE preference have a correlation coefficient of r=.806(p<.01). If an image describes details sufficiently from the highlight to shadow, it is preferred. Therefore, the various CIQE factors listed in order of descending
37

correlation with CIQE preference are CIQE color reproduction, CIQE dynamic range, CIQE resolution, and
CIQE noise. In particular, CIQE color reproduction and CIQE preference have a high level of correlation, because subjects focus on skin color when they evaluate portrait image quality.
Currently, objective image quality measurement is provided by manufacturers via numerical values, and it is considered as professional area that is hard to understand by general consumers. Therefore, subjects have difficulty evaluating image quality efficiently with respect to objective image quality factors. Previous research has focused on statistical factorial analysis in subjective image quality assessment. Therefore, in this research, we introduce a new subjective image quality assessment based on explanations of objective image quality factors. The method can be used to analyze the final image quality preference. The proposed
Complete Image Quality Evaluation (CIQE) method uses generally worded questions that are analyzed by correlation. The results of application of the method revealed that the stimuli in descending order of correlation with preference for an image are: CIQE color reproduction, CIQE dynamic range, CIQE noise, and CIQE resolution. Moreover, with regard to portrait stimulus characteristics, CIQE color reproduction and CIQE preference have the strongest correlation. Throughout this research, we tried to determine what constitutes preferred image quality not only to pursue high quality image performance, but also to analyze factors which influence image quality. The quantificational results could be used by manufacturers to produce practical products.
This work was supported by National Research Foundation of Korea-Grant funded by the Korean
Government (NRF)-2012-32A-G00024.
[1] S. H. Lee. Never stopping development of digital camera . SERI business note, no.22, 2009.
[2] H. J. Park. Research on language of digital camera image quality assessment based on objective image quality evaluation factors. The society of Korean photography.
2012, no.26.
[3] P. G. Engeldrum. A Theory of Image Quality: The Image Quality Circle. Journal of Imaging Science and
Technology.
2004, 48(5): 446-456.
[4] H. J. Park. Correlation research between objective and subjective image quality assessment. Korea Contents
Association , 2011, vol. 11, no.8.
[5] ISO 20462 Photography - Psychophysical experimental methods to estimate image quality . International
Organization for Standardization. 2004.
[6] ISO 3664 Viewing conditions - Graphic technology and photography . International Organization for
Standardization. 2009.
[7] B. J. Kim. Marketing research method . Dae Kyung Publish, 2005.
[8] ISO 14524 PhotographyElectronic Still-picture Cameras-methods for measuring Opto- Electronic Conversion
Fuctions (OECFs) , International Standard Organization, 2005.
[9] ITU-R Rec. BT. 500-11. Methodology for the subjective assessment of the quality of television pictures .
International Telecommunication Union, 2002.
[10] J. Radun, T. Leisti, J. Häkkinen, H. Ojanen, J-L. Ilives, T. Vuori, G. Nyman. Content and Quality: Interpretation -
Based Estimation of Image Quality. ACM Transactions on Applied Perception . 2008. 4(4), 21.
[11] E. H. Jang, S. S. Choi, K. H. Lee, J. H. Sohn. Development of Emotion Assessment Scale in Evaluation of
Television Picture Quality. Korean Journal of the science of Emotion & sensibility . 2009, vol.12, no.1:121-12.
[12] H. J. Park, D.H. Har. Subjective Image Quality Assessment based on Objective Image Quality Measurement
Factors. IEEE Transactions on Consumer Electronics , August 2011, Vol. 57, No. 3.
38