Open Profiling of Quality as a mixed method approach to
study multimodal experienced quality
Dominik Strohmeier
Institute for Media Technology
Ilmenau University of Technology
P.O. Box 10 05 65, 98684 Ilmenau
+49 (0) 3677 / 69 2671
dominik.strohmeier@tu-ilmenau.de
Measurement, Human Factors, Experimentation
levels of human perception processes [3]. Multimodal quality
assessment has tried to measure the perceived quality on a
quantitative basis [1]. Results of these methods present quality in
terms of its hedonistic excellence. However, quality is a
“multidimensional construct of user perceptions and behaviors”
[15] and only few methods exist that offer possibilities to study
and understand this construct. Bech et al.’s RaPID method [1] was
a first profiling approach, but it is limited through extensive
training with test participants. Focusing on naïve assessors,
Jumisko-Pyykkö et al. [4] and Radun et al. [7] use interviewbased methodologies to combine quantitative evaluation with an
elicitation of experienced quality factors. In contrast to interviews,
Lorho uses methods of sensory profiling in his Individual
Vocabulary Profiling [5] approach. It allows assessors to develop
and to apply their own quality attributes in the evaluation task.
However, this approach has not been applied in mixed method
research.
Keywords
3. OPEN PROFILING OF QUALITY
Open Profiling of Quality, subjective quality, quality evaluation,
Quality of Experience, mixed method
OPQ [12] is a mixed method that combines evaluation of quality
preferences and the elicitation of idiosyncratic experienced quality
factors. It therefore uses quantitative psychoperceptual evaluation
and, subsequently, an adaption of Free Choice Profiling. The
goals of the method are a) to define the excellence of overall
quality, b) to understand the characteristics of quality perception,
c) to construct a link between preferences and quality attributes,
and d) to provide a test methodology that is applicable to use with
naïve assessors. OPQ does not limit assessors in the development
of quality attributes to certain predefined parameters. It is open for
assessors to elicit whatever they perceive as quality factors.
ABSTRACT
To reach users’ overall quality acceptance is a very important
aspect for the success of new technologies. Therefore, user studies
following a psychoperceptual evaluation approach have been used
for a long time. However, these studies neglect the dualistic nature
of quality. While they are able to capture quality in its holistic
excellence, methods to understand underlying quality factors are
still missing. Open Profiling of Quality (OPQ) is a mixed method
approach that combines quantitative and descriptive quality
analysis with naïve participants. OPQ is meant as a tool to create
deeper understanding of perceived multimodal quality.
Categories and Subject Descriptors
H.1.2 [User/Machine Systems]: Human information processing
General Terms
1. INTRODUCTION
The optimization of quality according to users’ perception has
become a very important part in the development of new
multimedia technologies. To provide better experience and higher
quality than existing systems is a crucial point for the success of
new technologies. Modern quality research needs to take into
account the different aspects of quality. Beside the evaluation of
quality in the actual context of use, understanding of underlying
quality factors is a main goal of user-centered Quality of
Experience research [1]. Recommendations on quality evaluation
methods have targeted a quantitative evaluation of critical
components to measure their quality excellence [8]. But quality in
terms of its dualistic nature is not only related to the degree of
excellence of a product. Quality is also characterized by
distinctive attributes or characteristics that are possessed by
products. Only few research approaches exist that try to capture
the dualistic nature in subjective quality evaluation. In this paper,
we present Open Profiling of Quality (OPQ) as a mixed method
research approach to study perceived multimodal quality.
2. RELATED WORK
While the produced quality of multimedia application relates to
the quality provided by the system under its constraints related to
technical factors, perceived quality describes the users’ perception
of quality. Perceived quality is characterized by an impact of all
3.1 Procedure
The procedure of OPQ is described in detail in [10] and [12].
First, a psychoperceptual evaluation is conducted which follows
the recommendations of the ITU [8]. Test participants rate the
acceptance of overall quality and the satisfaction with overall
quality. Then, a sensory profiling evaluation follows in a second
task. In our research approach, we adapt the method of FreeChoice Profiling [14] which allows users to evaluate overall
quality using their own vocabulary. After an introduction, test
participants develop their own quality attributes, preferably
adjectives, in the attribute elicitation task. In the following
attribute refinement task test participants select their final
vocabulary by identifying those attributes that a) are unique and b)
can be described precisely by the users. At the end of the attribute
refinement task, each attribute is attached to a 10cm long line
labeled with ‘min’ and ‘max’ in its extremes. It results in an
individual score card which the test participants will use for
stimuli evaluation. In the sensory evaluation task, test stimuli are
presented one after another. The test participants mark the
sensation of the attributes on their score card for each test item.
‘Min’ means that the attribute is not perceived at all while ‘max’
refers to its maximum sensation. Stimuli can be repeated several
times during the evaluation.
6. ACKNOWLEDGMENTS
3.2 Analysis
[1] Bech, S., Hamberg, R., Nijenhuis, M., Teunissen, C., de
Jong, H., Houben, P., and Pramanik, S. 1996. “The RaPID
perceptual image description method (RaPID)”. In Proc.
SPIE. Vol. 2657, pp.317-328.
[2] Jumisko-Pyykkö, S. Strohmeier, D. “Report on research
methodologies for the experiments”, Technical report Project
Mobile3DTV, November 2008
[3] Jumisko-Pyykkö, S., Reiter, U., and Weigel, C. 2007.
„Produced quality is not perceived quality - a qualitative
approach to overall audiovisual quality”. In Proceedings of
the 3DTV Conference.
[4] Jumisko-Pyykkö, S., Häkkinen, J., and Nyman, G. 2007.
”Experienced quality factors qualitative evaluation approach
to audiovisual quality”. Proceedings of the IS&T/SPIE 19th
Annual Symposium of Electronic Imaging, 2007
[5] Lorho, G. 2005. “Individual Vocabulary Profiling of Spatial
Enhancement Systems for Stereo Headphone Reproduction”.
Proceedings of Audio Engineering Society 119th Convention,
New York (NY), USA
[6] McEwan, J.A. 1996 “Preference Mapping for Product
Optimization”, in Multivariate Analysis of Data in Sensory
Science, ed. Naes. T and Risvik, E., Amsterdam: Elsevier
[7] Radun, J., Leisti, T., Häkkinen, J., Ojanen, H., Olives, J.-L.,
Vuori, T., and Nyman, G. 2008. Content and quality:
Interpretation-based estimation of image quality. ACM
Trans. Appl. Percept. 4, 4, 1-15
[8] Recommendation ITU-R BT.500-11. 2002. Methodology for
the Subjective Assessment of the Quality of Television
Pictures, ITU Telecom. Standardization Sector of ITU.
[9] Seuntiens, P.J.H. 2006. “Visual Experience of 3D TV”, PhD
thesis, Eindhoven: Technische Universiteit Eindhoven
[10] Strohmeier, D. Tech, G. 2010. “Sharp, bright, threedimensional: open profiling of quality for mobile 3DTV
coding methods,” in Proc. SPIE Electronic Imaging Conf.
2010, San Jose, California, USA
[11] Strohmeier, D., Jumisko-Pyykkö, S., Kunze, K. “New,
lively, and exciting or just artificial, straining, and
distracting? A sensory profiling approach to understand
mobile 3D audiovisual quality,” in Proc. 4th VPQM
Workshop, Scottsdale, USA, Jan. 2010.
[12] Strohmeier, D., Jumisko-Pyykkö, S., Kunze, K. 2010. “Open
Profiling of Quality: A mixed method approach to
understanding multimodal quality perception,” submitted to
Hindawi Advances in Multimedia
[13] Strohmeier, D., Jumisko-Pyykkö, S., Reiter, U. 2010
“Profiling Experienced Quality Factors of Audiovisual 3D
Perception”, accepted at Qomex 2010, Trondheim, Norway.
[14] Williams, A.A., Langron, S.P. 1984. “The use of Free-choice
Profiling for the Evaluation of Commercial Ports.” In:
Journal of the Science of Food and Agriculture 35
[15] Wu, W., Arefin, A., Rivas, R., Nahrstedt, K., Sheppard, R.,
and Yang, Z. 2009. Quality of experience in distributed
interactive multimedia environments: toward a theoretical
framework. In Proceedings of the 17th ACM International
Conference
on
Multimedia,
Beijing,
China
The application of Open Profiling of Quality results in two
independent data sets. The quantitative data set can be analyzed
using Analysis of Variance to get a preference ranking of the
items under assessment. The sensory data can be analyzed using
Generalized Procrustes Analysis (GPA). The result of GPA is a
low-dimensional model of the users’ perception. This model
shows the separation of the test items along its principle
components as well as the correlation of principle components
and idiosyncratic attributes. By identifying their meaning the
researcher gets deeper insight into underlying quality rationale to
explain possible quality preferences in the quantitative data. The
final step of OPQ is the combination of quantitative and sensory
data. Using External Preference Mapping [6] allows the
researcher to map preference data into the perceptual model. This
finally connects the users’ preferences to their quality attributes.
4. APPLICATION OF OPQ IN 3DTV
RESEARCH
Open Profiling of Quality has been developed in constructive
research in three studies related to 3DTV research. For detailed
results of each study I refer to the respective full papers
[10][11][12][13]. The first study targeted the evaluation of
different audiovisual 3D presentation setups [13]. The application
of OPQ has helped to see that participants were qualitatively able
to differentiate variables, although no quantitative excellence of
stimuli was identified. Further, different sensorial preferences as a
part of any multimodal quality evaluation were identified. The
second and third studies were both conducted in the context of
mobile 3DTV research. The second study [11] examined
influences of mono and stereo audio and visual presentation
format on perceived overall quality and perceived depth. The third
study finally targeted the selection of an optimum coding method
for mobile 3D television and video [10]. The application of Open
Profiling of Quality has helped to understand the quality rationale
of autostereoscopic perception. The perceptual models of the
second and third study both show that video quality is the
determining component in 3D quality perception. The expected
added value of depth perception only was visible when the
perception of artifacts was low. These results confirm Seuntiens’
3D Quality of Experience model [9] and extend it to a hierarchical
interaction of video quality and depth perception.
5. CONCLUSION
In this paper, I present Open Profiling of Quality as a mixed
method approach in multimodal quality assessment. OPQ closes
the shortcomings of commonly applied quantitative research
methods. It combines the evaluation of the quality excellence as
well as the characteristics of quality for the items under
assessment and so allows capturing quality in terms of its dualistic
nature. Although further work needs to study the application of
OPQ in audiovisual quality research, OPQ is a valuable tool in the
context of multimodal quality evaluation.
MOBILE3DTV project has received funding from the European
Community's ICT programme in the context of the Seventh
Framework Programme (FP7/2007-2011) under grant agreement
no 216503.
7. REFERENCES