PART I
Experimental
considerations
Perceptua l Audio Eva lua tio n – Theo ry, M etho d a nd Applica tio n Søren Bech and Nick Zacharov
~c 2006 John Wiley & Sons, Ltd. ISBN: 0-470-86923-2
CHAPTER
2
Definition of
research question
and hypothesis
Perceptua l Audio Eva lua tio n – Theo ry, M etho d a nd Applica tio n Søren Bech and Nick Zacharov
~c 2006 John Wiley & Sons, Ltd. ISBN: 0-470-86923-2
17
hIS chapter discusses the formulation of research questions and the
principles behind establishing a testable statement based on a premise
Tor hypothesis and a number of initial conditions. The formulation of
the hypothesis, the initial conditions and the testable statement is
perhaps the most important part of any scientific experiment. Firstly
because they determine the scientific quality of the experimental
results, secondly because the truthfulness of all or some of the initial
conditions are determined by the design of the experimental set-up and
thirdly because the statistical analysis of the observed data depends on
the form of the testable statement.
The research question is a broad formulation of the problem that
is to be investigated. An example would be the research question that
Perceptua l Audio Eva lua tio n – Theo ry, M etho d a nd Applica tio n Søren Bech and Nick Zacharov ~c
2006 John Wiley & Sons, Ltd. ISBN: 0-470-86923-2
17
5
Chapter 2 Definition of research question and hypothesis
was formulated for the Archimedes project3: ‘Why is the perceived
sound quality of a loudspeaker different in different domestic sized
rooms and in different positions in the same room?’.
This question involves a large number of physical (see discussion
in Chapter 5) and psycho-acoustical (see discussion in Chapter 4)
variables that need to be carefully considered in order to obtain
meaningful experimental results. The next logical step would thus
be to divide this general question into a number of experimentally
manageable questions where subsets of these variables are examined
in more detail. However, the first immediate question one could ask
is if it has been proven that loudspeakers actually do sound different
in different rooms and in different positions.
Some researchers would argue that their practical experiences have
shown this to be the case and thus conclude that it must be a general
problem for all loudspeakers in all rooms. Others would argue that
though experience indicates that this might be true it is not possible to
draw conclusions for all loudspeakers and all rooms on the basis of a
few observations and therefore a more rigourous validation is needed
before a major research project is started.
The above dilemma, empirical versus rational knowledge, is one
of the basic questions when discussing how scientific knowledge may
be established. The following discussion will be an introduction to the
problem and provide some guidance on how to tackle this and other
related questions.
The purpose of this chapter is not to give a complete overview of the
history of scientific argumentation, but to introduce the concepts and
to provide an understanding of the general principles in formulating
a research hypothesis.
The discussion will be based on two main references, Fjelland and
Gjengedal [123] and Johansson and Lynoee [248], so only additional
references will be noted in the following text.
Scientific knowledge is traditionally characterised by three fundamental aspects:
It is theoretical Scientific knowledge is theoretical in the sense that
science does not solve practical problems per se. This is not to
3The Archimedes project was a five-year collaborative research effort funded
under the European research scheme Eureka. The partners were the Acoustics
Laboratory (now Acoustic Technology) of The Technical University of Denmark, KEF
Electronics of England, and Bang & Olufsen of Denmark. Results of the project have
been published in, for example, Bech [29–31,34]
Chapter 2 Definition of research question and hypothesis
6
say that scientific questions cannot originate from practical problems or that scientific knowledge cannot be used for practical
purposes, but if the work is aimed at solving a particular practical
situation and the gained knowledge cannot be generalised, then it
is not considered science in the traditional sense.
It is the truth The basic difference between knowledge and belief is
that knowledge can only be the truth whereas belief can be either
true or false. In addition, it is often required that knowledge must
be documented using established scientific rules and measures.
It is systematical The systematical aspect is based on the fact that
new knowledge must be based on systemised knowledge from
previous experiments.
These requirements have led to formal systems representative
of how scientific knowledge is established–the so-called axiomatic or
deductive systems. Platon (427–347 B.C.) was one of the first to develop
such a system and later Galileo Galilei (1564–1642) and Isaac Newton
(1642–1727) used such a system to organise the development of their
new theories. An axiomatic system is based on a set of definitions upon
which it is possible to establish axioms or proven hypotheses, and based
on the definitions and the axioms it is possible to establish theories.
The knowledge represented by the axioms or hypotheses must be
true knowledge for the developed theories to be the building blocks
for new knowledge. However, the definition of true knowledge and
how it is proven to be true have been debated since Galilei, who
used rational arguments (using common sense) in some cases and
empiricism (using his senses to gather experience) in others. This
fundamental division between how scientific knowledge is created is
represented by two major principles of scientific argumentation. One
is the principle of empiricism, which claims that only observations and
experiments can determine which theories or hypotheses are true. The
other, the principle of rationalism, claims that deduction using common
sense is the correct way to obtain true knowledge.
Most scientists, especially those working in areas involving both
exact sciences (for example mathematics) and social sciences (for
example psychology), will use a mixture of the two principles,
depending on the experimental situation. Perceptual experiments
such as evaluation of reproduced sound quality typically involves this
mixture, so a short description of both principles will be given in the
following text.
7
Chapter 2 Definition of research question and hypothesis
2.1 Principle of empiricism
The principle of empiricism is based on the fact that it is only observations and experiments that form the basis for the generalisation
of observations: reasoned from the particular to the general. This is
expressed in the principle of induction: ‘(the premiss:) If a certain
observation A is observed together with another observation B and
further that A has never been found without B then it follows: (the
conclusion) the more often A and B are observed together, the greater
is the likelihood that they will appear together in another situation
where one of them is observed’.
An argument based on induction for the premiss of the Archimedes
project, that the same loudspeaker sounds different in different positions in the same room, would be as follows:
Premiss 1: It has been observed through a number of listening tests,
including a range of different loudspeaker types, that the same
loudspeaker positioned in different positions (observation A) in
the same room sounds different (observation B).
Conclusion: It is true for all loudspeaker types that the same loudspeaker will sound different in all positions in the same room.
According to this approach, there would be no reason to start
a series of listening tests to prove the assumption, but instead the
experimenter should start a series of experiments to examine various
reasons for this observed difference between positions.
The problem, according to critics of the principle of empiricism, is
that the conclusion can be flawed but the premiss can still be true. This
violates another fundamental law in scientific argumentation: The law
of contradiction, which states that a statement and its negation cannot
be true at the same time.
When empiricism is the foundation for generalisation, an often
applied research strategy is to test the observed principle or law under
a range of different conditions and, if it is found to be valid in all the
tested situations, it is accepted as a general law. Examples of laws
based on empiricism can be found both in physics where Boyle’s law
states that the relation between the pressure p and the volume V of a gas
is always constant (p × V = constant), and in psychophysics where
Stevens, [403, 406] using a direct magnitude estimation procedure,
2.2 P rincip le o f r ati on a li sm
8
found that the relationship between sensation S and stimulus intensity
I can be described by a power law (S = constant × ( I I 0 )k), where I0
is a reference intensity and the exponent k depends on the modality
and the stimulus. The ITU-R recommendation BS.1387 [213], which is
intended for prediction of the perceived impairment in sound quality
of a low bit-rate codec as a function of bit rate, is also an example
of a ‘law’ or relationship that has been verified for a large number of
situations and has been accepted as an international standard.
The principle of empiricism has often been criticised, especially the
fact that the main focus is on confirmative results rather than deviating
results, which according to the critics is where scientific progress is
often made. One of the main opponents of the empirical principle
was Popper (see, for example, [350]), who in 1934 published the book
‘The logic of scientific discovery’. Popper argues that, no matter
how many supportive observations there are, there will always be a
finite probability that there are cases that do not support the previous
observations and such cases will violate the law of contradiction as
discussed earlier.
Popper’s alternative is the so-called ‘hypothetico-deductive’
method, which is discussed in the next section.
2.2 Principle of rationalism
Popper [350] suggested the ‘hypothetico-deductive’ as an alternative
(in fact, Popper argued that it is the only method) to the principle of
empiricism or inductive method discussed in the previous section.
Before discussing the hypothetico-deductive method, it is useful
to examine the principle behind deductive arguments. A deductive
argument is based on a number of premisses and a conclusion. A
simple deductive argument could be as follows:
Premiss 1: When it rains the street will be wet.
Premiss 2: It rains.
Conclusion: The street will be wet.
The deductive principle means that if the premisses are true the
conclusion must be true. The opposite, the premisses were true and
the conclusion false, would violate the law of contradiction.
9
Chapter 2 Definition of research question and hypothesis
The ‘conclusion’ is typically of a form that can be tested in an
experiment (in the example above, you would look out of the window)
and if the conclusion can be proven to be false (the street is not wet)
it follows that one of the premisses must be false (probably that it is
not raining). However, if the conclusion is found to be true (the street
is wet), it does not finally prove that the premisses are true (the street
cleaning truck could just have passed by and sprinkled water on the
street). This principle is called the asymmetry between falsification
and verification, and Popper argued that only falsification leads to true
knowledge and progress: A falsification means that the experimenter
must re-think or reformulate the premisses, which is equivalent to
scientific progress.
The hypothetico-deductive method is based on a set of general
observations that lead to the formulation of a hypothesis (see premiss
1 in the example above). The hypothesis together with a set of
initial conditions (premiss 2) leads, based on a deductive argument,
to a testable statement (the conclusion). The hypothetico-deductive
method thus reasons from the general to the particular, which is the
opposite of the inductive method discussed above.
Because the test statement is based on deductive arguments, it
follows that if the initial conditions are true but the test statement
is false then the hypothesis must be false. If the test statement was
found to be true it could be assumed, but not formally verified, that
the hypothesis was true and this would not lead to any real scientific
progress according to Popper. So the experimenter’s primary aim is, if
the hypothetico-deductive method is used, to formulate one or several
hypotheses that together with the initial conditions lead to a testable
statement that must be rejected.
If the hypothetico-deductive method is used to test the assumption
of the Archimedes project, the rationale would be as follows: The experience shows that for some loudspeaker types the same loudspeaker
will sound different in different positions in the same room. However,
it has also been found that the sound quality of certain other types are
less influenced by their positions. So, in order to follow the principle
of falsification, the focus is on the loudspeaker types that are not
influenced by their positions in the room:
Hypothesis (premiss 1): Physically identical loudspeakers positioned
in different positions in the same room sound similar.
2.2 P rincip le o f r ati on a li sm
10
Initial conditions (premiss 2): Two physically identical loudspeakers
in two different positions in the same room are compared in a
listening test and the results of the test are true representations
of the perceived sound quality of the individual loudspeakers.
Testable statement (conclusion): The perceived sound quality of the
two loudspeakers will be identical.
This strategy was used in the Archimedes project, and the results
have been reported in Bech [29], where the test statement was proven
statistically false for some of the tested loudspeakers, and true for others.
This was the experimental basis for concluding that the hypothesis was
false for at least some of the combinations of loudspeaker types and
positions, and it was decided to perform further experiments to examine
the influence of position on sound quality.
Now the basic assumption of the deductive principle is that if the
premisses are true the conclusion must be true, and further that if
the conclusion is false and the initial condition is true the hypothesis
is false. The principle also states that the testable statement should
be formulated with the purpose of falsification. The truthfulness of
the initial condition thus becomes very important and this is often
related to the quality of the experimental part of the investigation. The
above example from the Archimedes project is a good illustration of
why this can be quite complicated and difficult to achieve in the real
experimental situation.
The initial condition was that the results of the listening tests
represent the true magnitude of the perceived sound quality of the
tested loudspeakers. However, this will depend on a number of
variables such as the selected programmes (musical genre and number
of excerpts), physical properties of the loudspeaker, number and type
of subjects, physical identity of the two loudspeakers, and so on. So the
truthfulness of the initial condition is related not to the truthfulness of
a single condition but instead to a sum of conditions.
The scientific quality of the inferences of an experiment thus relies
very much on the truthfulness of each individual part constituting
the initial condition. The truthfulness of these individual parts is
determined by the variables that define the experimental conditions.
These variables are identified and discussed in the remainder of
this book.
11
Chapter 2 Definition of research question and hypothesis
2.3 Other principles of scientific argum entation
The principles of empiricism and rationalism discussed above are,
in most scientific papers, the two basic principles for establishing a
testable statement and for the subsequent discussion of the results.
However, there are a number of other relevant types of scientific argumentation that are especially important when considering practical
experiments often used in, for example, engineering in contrast to
theoretical disciplines like mathematics.
2.3.1 Probabilistic reasoning
Probabilistic reasoning is used when the truthfulness of the hypothesis
is based on a limited number of observations from a given population.
This introduces a finite probability that a rejected hypothesis is not
representative or correct for the whole population. This is a very
typical situation in medical experiments where only a limited number
of persons can be tested and likewise in listening tests where in
addition to a limited number of subjects only a limited number of
programmes, listening rooms, loudspeaker positions, and so on, can
be tested. This makes the probabilistic reasoning approach the most
typical for assessments of sound quality using subjects.
Probabilistic hypotheses can be tested using either a deductive
principle or an inductive principle, but in addition a probabilistic
hypothesis arguments can be
mathematical.
objective.
subjective.
The classical mathematical principle is always deductive, and
theoretical probabilistic theory follows the standard mathematical
rules. However, when probability calculations are used on an observed
data set rather than theory, the truth of the hypothesis becomes
testable. However, the deductive line of reasoning can still be applied
and a typical example could be as follows:
Hypothesis (premiss 1): For a standard dice, the relative frequency
of observing any one number in the range 1–6 is 1 6.
2.3 Other principles of scientific argumentation
12
Initial condition (premiss 2): A standard dice is thrown n times.
Testable statement (conclusion): Approximately 6 of the n observations will be ‘one’.
The hypothesis is now based on an observed probability and the
truthfulness is now finite depending on the number of observations
in the observed sample. A hypothesis originating from a calculated
probability (based on a number of samples) is called an ‘objective’
hypothesis.
A ‘subjective’ hypothesis refers to the fact that some probabilities,
although being objectively determined (based on a number of samples), have other conditions attached to them that must be fulfilled
before the objective probability is relevant. The evaluation of these
other conditions then becomes the subjective part. An example would
be that it is possible to calculate the objective probability that an
otologically normal person will hear a given pure tone at a given frequency and sound pressure level, but there might be other conditions
(for example, the person is tired or has a cold) that could influence
the ability to hear and thus need to be evaluated by the experimenter
before that subject is used in the listening test where the objective
probability is used.
The argument including the dice is strictly deductive even if it
includes an objective hypothesis; however, it is often the case that the
objective hypothesis is not tested and instead the hypothesis is based
on an inductive argument:
Observation (premiss 1): Approximately 6 of the numbers observed
for this dice were ‘one’.
Conclusion: The probability of getting a ‘one’ with this dice is 6.
It is important to note that if the hypothesis is based on an inductive
argument then it is impossible for the argument that includes the
hypothesis to be anything but inductive.
The above discussion hopefully illustrates that in some cases it is
very difficult to determine if an argument is deductive or inductive. In
situations where the hypothesis is based on a limited sample, which
is the case in statistics, the argument will always be inductive as
the premisses describes the sample, but often (but not always) the
conclusion refers to the whole population.
13
Chapter 2 Definition of research question and hypothesis
This means that the design of most listening tests, theoretically
speaking, will be following the inductive principle, but it is still important to apply the hypothetico-deductive principle when designing
the experiment to ensure the truthfulness of the initial conditions.
However, when the general validity of the conclusions are discussed,
it is important to consider and keep in mind the principle the hypothesis was based on. This requires a careful assessment of the statistical
assumptions and the subsequent analysis. This is discussed in more
detail in Section 6.2.
2.3.2 Argumentum ad hominem
There is another principle called ‘argumentum ad hominem’. It refers
to the principle of using personal statements from authorities within
the field as proven facts without checking them formally. This type of
arguments should be avoided, in the authors’ view, as they carry no
scientific weight in themselves. This is not to say that the authorities
being referred to are not producing scientifically valid results, but then
these should be referenced, and not personal, statements. This line of
reasoning is often applied in Hi-Fi magazines, where the reviewers
will have or gain status as authorities in their areas and quite often the
reader will use the arguments of the reviewer for buying a particular
product.
2.3.3 Conclusion by analogy
Yet another principle is ‘conclusions by analogy’, which means that
the conclusion is assumed to be valid because it is similar to the
premisses. An example would be the doctor who concluded that cold
water should be used to treat burnt skin. He drew that conclusion
by analogy from the boiled egg he used to have every morning. One
morning the egg was more hard boiled than usual and he found the
reason to be that his wife had not rinsed the egg in cold water as
she used to. He concluded that cold water could stop the process
of coagulation in protein and hence that cold water could stop the
coagulation process when used on burnt skin.
This line of reasoning is rarely used (i.e. the authors have never
seen it being applied) in auditory evaluations, but the principle should
not be excluded as not being useful.
2.4 Summary
27
2.4 Summary
The above sections have discussed the major principle for establishing
a hypothesis and based on a set of initial conditions, a testable statement. The most often applied principle in audio evaluations, based
on the experience of the authors, is that of probabilistic reasoning.
This, however, is also one of the most complicated as the line between
the principles of empiricism and rationalism is not always crystal
clear in such experiments. However, it is still the authors’ view that
rationalism and the hypothetico-deductive principle should be aimed
for although it is clear that this is not trivial and requires careful
consideration of a large number of factors. Especially, experimental
planning and control are important to ensure the truthfulness of the
initial conditions. The next chapter will discuss the fundamentals of
experimentation and how this relates to the hypothesis and testable
statement.