Chapters 2, 3, 4, 10, 11, 13, 14, 15
Chapter 2: Experimental research

Most common research methods: observations, field studies, surveys, usability
studies, interviews, focus groups, controlled experiments
o Descriptive investigation: observation, survey, focus groups – construct an
accurate description of what is happening (first step- identify interesting
phenomena that establish cornerstone of research)
o Relational investigation- enable researcher to identify relations between
multiple factors. The value of factor X changes as the value of factor Y
changes (Discovers connections between multiple variables)
o Experimental research- enables the identification of casual relationships
between entities or events. It can tell how something happens, and in some
cases why (Explore the fundamental casual relations)

Research Hypotheses
o A precise problem statement that can be directly tested through empirical
investigation
 Hypothesis- smaller, more focused, can be examined by 1
experiment VS Theory- Covers larger scope, requires a sequence of
empirical studies (page 23)
o Null hypothesis- States that there is no difference between experimental
treatments
o Alternative hypothesis- always a statement that is mutually exclusive with
the null hypothesis
 H0- There is no difference between the pull-down menu and the
pop-up menu in the time spent locating pages
 H1- There is a difference between the pull-down menu and the popup menu in time spent locating pages
 Independent variable- type of menu (pull up or pop up)
 Depend variable- time spent in locating web pages
o Goal of experiment is to test the null vs the alternative and decide which one
should be accepted and which one should be rejected
o Dependent variable- Suggests that the variable is dependent on a
participants behavior or the changes in the independent variable (usually
outcomes that they need to measure)
 Efficiency (time to complete task), accuracy (error rate), subjective
satisfaction (likert scale usually), ease of learning & retention rate
(how quick and easily someone learsn new app or task), physical or
cognitive demand (how long without fatigue)
o Independent variable- The factors that the researchers are interest in
studying or the possible “cause” of the change in the dep variable (usually
treatments or conditions that researcher can control)
 Typical: typing vs speech based, mouse vs joystick or touch pad, pull
down vs pop up, font sizes, background colors, age, gender,
computer experience, education, mood, lighting, temp,
sitting/walking



Basics of experimental research (example page 27)
o Treatments/conditions- refer to the different research techniques, devices
or procedures we want to compare
o Units- objects in which we apply the experiment treatments. Normally
human subjects with specific characteristics
o Assignment method- The way in which the experimental units are assigned
different treatments
o Randomization- the random assignment of treatments to the experimental
units or participants (toss coin, random digital table, software driven
randomization). Randomize both assignment and other factors
Significance tests
o A process in which a null hypothesis is contrasted with an alternative
hypothesis to determine the likelihood that the null hypothesis is true
 Allow us to determine how confident we are that the results
observed from the sampling population can be generalized to the
entire population
o Sampling- Collect a smaller group from the large population and use that
smaller group to represent the entire population
o Type I error- aka false positive. The mistake of rejecting the null hypothesis
when it is true and should not be rejected (example page 32) (worst) (alpha
– 0.05)
o Type II error-aka false negative. The mistake of not rejecting the null
hypothesis when it is false and should be rejected (beta)
Limitations of experimental research
o Difficult to construct a well-defined and testable hypothesis
o Difficult to follow the strict control of factors that may influence the
dependent variables
o Lab-based experiments not good representation of users typical interaction
/ Hawthorne effect
Chapter 3: Experimental design




True Experiments- study involves multiple conditions and the participants are
randomly assigned to each condition
o Based on at least one testable hypothesis and aims to validate it
o Usually at least two conditions or groups
o Dependent variables usually measured through quantitative measurments
o Should be designed + conducated with goal or removing bias
o Should be replicable with dif participant samples, at dif times, at dif
locations, by dif experiments
Quasi-experiments- study involves multiple groups or measures, but the
participants are not randomly assigned to different conditions
Non-experiments- only one observation group or only one measure involved
Example: There is no difference between the target selection speed when using a
mouse, a joystick or a trackball to select icons of different size (s, m, large)
o Two independent variables: pointing device + icon size
o Three conditions under this: mouse, joystick, trackball
o Three more conditions under this: small, medium, large
o Total of 9 conditions in this experiment
o Dependent variable: speed







Decide structure of experiment
o How many independent variables do we want to investigate?
 More than 1? Factorial design
 Just 1? Basic design
o How many different values does each independent variable have?
 Between-group, within-group or split-plot
Between-group design- Each participant is only exposed to ONE condition
(Keyboard: one person tries QWERTY, one DVORAK, one Alphabetic)
o + Avoids learning effect, only exposed to one condition
o + Takes shorter time, fatigue and frustration controlled
o – Results are subject to substantial impacts from individual differences
o – Larger number of participants needed (4 conditions x 16 people = 64)
o Should be used when experiment has simple tasks with limited individual
differences, tasks that would be greatly influence by learning effect. Novice
& experienced ex, cannot be both
Within-group design- Each participant is exposed to all conditions (Keyboard: Three
people try all three)
o + Requires smaller sample size (16 people)
o + Less cost
o – Possible impact of learning effects
o – Fatigue, takes more time
o Should be used when experiment investigages tasks with large individual
differences, less susceptible to learning effect, or small participant pool
(complicated tasks like reading / writing)
Factorial design- Divide experiment groups or conditions into multiple subsets
according to the indepdent variables.
Split-plot design- One or more indep variables are investigated through betweengroup and other variables are investigated through within-group
Interaction effect- The differing effect of one independent variable on the
dependent variable, depending on the particular level of another independent
variable. Performance may be affected by multiple factors jointly
Reliability- Reliable experiments can be replicated by other research teams in other
locations and yield results that are consistent, dependable and stable.
o Random Errors- can reduce by enlarging observed sample size, closer to
observed value (typing speed example)
o Systematic errors- aka biases. Always push the observed values in the same
direction = too high or too low
 Caused by measurement instruments- Stop watch reacts slow
 Caused by experimental procedures- non randomized task
conditions = learning effect, not consistent instructions and
wording. Should: randomize order, prepare written document with
instructions for participant + experimenters, run pilot studies
 Caused by participants- should: recruit participants carefully
(representative of target user pop), create an environment with
little stress, reassure participants you are testing interface- not
them, reschedule session if participant is tired or nervous
 Caused by experimenter behavior- Experimenter may untentionally
or intentionally influence results (wording, body languaging, late)
 Caused by environmental factors- Physical: noise, temperature,
lighting, humidity. Social: number of people surrounding,
relationaship between people and participant