Introduction to Research Methods
Q560: Experimental Methods in Cognitive Science
Lecture 2
A CogSci major named White, an Informatics major named
Black, and a Psyc major named Red meet for a coffee.
One of the three says “I’m wearing a black hat, and you
two are wearing a red and a white hat, respectively, but
none of us has a hat color that matches his name.” White
responds “you are quite correct.”
What color is the Psyc major’s hat?
Harvard Medical Review (1971):
Hypothesis: Coffee consumption causes urinary tract cancer

Data were collected from a group of patients, some with urinary
tract cancer, some with other diseases. Also, family history, age,
gender, etc.

Not everyone who drinks coffee gets urinary tract cancer, and not
everyone with urinary tract cancer is a coffee drinker…so it would
seem that the evidence contradicts the hypothesis

However, 25% of cancer patients habitually drink coffee,
compared to only 10% of those without cancer…so there would
also seem to be some evidence in favor of the hypothesis
Experimental Methods in CogSci

“Experimental methods” is a bit overly-restrictive

Exploratory research is a fundamental component of theory
building; experiments test theory

Some early experiments:

Very early: statisticum collegium and statista

1747: James Lind’s scurvy “experiment”

1920: Fisher’s classic “lady tasting tea”
Lind’s Scurvy “Experiment”
 1747: as surgeon on HMS Salisbury, Lind conducted a
controlled experiment to develop a cure for scurvy
 At the time, the concept of vitamins was unknown
 Selected 12 men “as similar as I could have them”

A quart of cider every day

25 drops of sulphuric acid 3x day

½ pint of seawater every day

Mixture of garlic, mustard, and horseradish

2 spoons of vinegar 3x a day

2 oranges + 1 lemon every day
Fisher’s Lady Tasting Tea

1920: Hanging out on the weekend at Cambridge bored…

The famous “lady” in question was Dr. Muriel Bristol

Can she tell, by taste alone, whether the tea or the milk was
added first to the cup?
↳
↳
↳
↳
4 cups of each type….how to present them?
Blindfolded
Need to control portions
Could 8/8 be due to guessing alone?
↳
↳
How many need to be correct to reject this assumption?
Using Fisher’s exact test, 1/70 chance of guessing
● Led to Statistical Methods for Research Workers (1925) and The
Design of Experiments (1935)
● Salsburg, D. (2001). The lady tasting tea: How statistics
revolutionized science in the twentieth century. Owl Books
Research in Cognitive Science
1. The role of theory
2. The role of exploratory research
3. The role of hypotheses
4. The role of experiments
5. The role of statistics
6. Iterative science
1. The Role of Theory



Theories have two roles in CogSci:
1.
As a framework to tie together isolated findings and to understand
cognitive phenomena
2.
They allow us to generate new hypotheses to test
Armchair vs. verbal conceptual vs. formal theories:

CogSci focuses on formal models

One is not “better” …they should be different stages
Formal models:

Descriptive

Predictive

Explanatory
1. The Role of Theory

“But I have observed over the years that there is a tendency for
even the best cognitive scientists to lose sight of large issues in
their devotion to particular methodologies, their pursuit of the null
hypothesis, and their rigorous efforts to reduce anything that
seems interesting to something else that is not. An occasional
reminder of why we flash those stimuli and measure those
reaction times is sometimes useful.”
--George Miller, Psychological Science, 1990.

Estes (1975):
2. The Role of Exploratory Research

There is nothing wrong with:

Introspection /self report

Naturalistic observation/case studies

Surveys

Correlational Designs
…as long as you don’t stop there

Exploratory methods are a necessary component to theory
building, and in designing contrastive experiments
3. The Role of Hypotheses

Inductive method:



In the past, theories were tested by gathering data that supported
them; now, we try to prove ourselves wrong.
NSHT:

Hypothesis is not a hunch…it is a testable prediction from theory

Probabilistic statement tied to a model of chance

Mommy seems to call everything with four legs a doggie:

H1: All four-legged things are doggies

H0: At least one four-legged thing is not a doggie
Note that most Bayesians hate NSHT!
4. The Role of Experiments

Experiment is designed to test the hypothesis:

Requires manipulation, control, and causality

The manipulated factor is the only free variable that could produce
the change in the outcome variable

Observed outcome is either due to chance (H0) or the manipulation
(H1)

Cf. Our shark beacon researcher…
5. The Role of Statistics

Statistical models allow us to efficiently summarize outcomes,
and determine whether the outcome is due to chance

Two main branches of Statistics:
1.
Descriptive Statistics: Summarizing and communicating information
about a group of numbers (data)
2.
Inferential Statistics: Drawing conclusions based on the data
collected, and making predictions that go beyond the immediate data
6. Iterative Science

The results of the experiment are used to reject, prune, or refine
the theories used to generate the hypotheses

i.e. ....back to step 1
Research Approaches

Field vs. Laboratory Studies


Nomothetic vs. Ideographic approaches:


Criticisms: Frankfurt school, Estes, etc.
Hypothesis is not a hunch…it is a testable prediction from theory
Non-experimental (desc), quasi, experimental

The quest for causation
Descriptive Techniques

Non-experimental methods describe behavior, but do not let us
identify the causes or reasons for the behavior

They are useful when you can’t ethically manipulate a variable, or
want to simply describe/predict behavior

Descriptive techniques are the backbone of many fields (e.g.,
medical science, meteorology, political science, etc.)
Descriptive Techniques
1. Case Studies
2. Observational Methods:
I observed squirrels masturbating for 2000h with 10x50 binoculars
from trees orObservation
a vehicle situated
within 40 m of the perimeter of a
Naturalistic
(Qualitative)
burrow cluster. Activities and masturbation frequency among
• males
Requires
wereimmersion
recorded using scan sampling at 5-min intervals…
•
Goal is to provide complete + accurate picture, not to test hypotheses
•
Issues of reactivity
Systematic Observation (Quantitative)
•
Requires a predefined coding system
Descriptive Techniques
3. Self-Report Measures:

What people say rather than what they do

If interested in opinions or social perceptions

Demand characteristics (cues) + biases

Note: We often don’t know how/why we do things

Interviews vs. questionnaires + surveys: only as good as questions
4. Correlational Research:

Correlation is not causation?

Direction and 3rd variable problems
Quasi-Experimental Techniques
 A true experiment requires random assignment
 Often, this is impossible or unethical
 Some examples:

Non-equivalent control groups design

NECG-pretest-posttest design

Interrupted time-series design

Control series design
Experimental Techniques
 Goal is to establish causal relation between two variables
 Requires manipulation, randomization, and control
 Experimental method:

A constant is a characteristic that is fixed across conditions

A variable is a characteristic that changes across conditions

To make inferences, we manipulate a variable of interest, and
observe the effect on an outcome variable, holding all other variables
constant (cf. shark beacon study)

Independent variable is manipulated, dependent variable is
measured
 Types of IVs:

Between-subjects (independent measures)

Within-subjects (repeated-measures)
Components of the Classic Experiment
 Control condition (no treatment)
 Experimental condition (receives treatment)
 Random assignment of subjects to conditions
 Extraneous variables:

Random and systematic

Confounding variables (cf. shark beacon)….source of error in
interpretation

Goal: reduce randoms and eliminate confounds via systematic
control, design, and randomization
An example w/ Gordon Bower’s classic imagery experiment
Extraneous Variables
 Situation and participant variables:

E.g., timing, systematic selection, etc.

Systematic and random (cf. shark beacon study)
 Hawthorne Effects:

You behave differently when being observed

Dangers of hypothesis guessing

Transparency and single-blind conditions
 Experimenter Effects (Rosenthal):

Confirmation bias, self-fulfilling prophecy

Double-blind studies
Between-Subjects Designs
 Independent groups of participants are assigned to the different
levels of the IV
 The variable takes on different values across individuals
 Pros:

Controls for timing variables, can test everyone at once

Do not need equivalent sized groups
 Cons:

Variability due to individual differences

Need more subjects (more than twice as many!)
Within-Subjects Designs
 Each participant is measured under all levels of the IV
 The variable takes on different values within individuals
 Pros:

More sensitive, less random noise

Fewer subjects are needed
 Cons:

Timing confounds: History and maturation

Carryover effects

Practice, sensitization, fatigue

Solutions: Counterbalancing, randomized blocks, Latin squares
Fundamentals of Measurement
 Qualitative (categorical) and quantitative (numerical)
 Discrete vs. continuous variables
 Stevens’ scales of measurement:

Nominal

Ordinal

Interval

Ratio
Fundamentals of Measurement
A discrete variable consists of separate, indivisible categories.
No values can exist between two adjacent categories
Example: political party, dog breed, number of children in
household, etc.
A continuous variable has an infinite number of possible values
between two adjacent values
Example: time, weight, pupil diameter, etc.
Scales of Measurement
1. Nominal Scale: Values are be categories that differ only in
name
Example: Gender, ethnicity, eye color, numbers on race cars
Values on a nominal scale only represent different
categories, and may not be averaged
Scales of Measurement
2. Ordinal Scale: Values are categories organized in an
ordered sequence (ranks)
Example: Places in a race, Olympic medals, etc.
Values on an ordinal scale are nominal, but also contain a
greater than/less than relationship between values on the
scale. However, you cannot determine the magnitude of the
relationships
Scales of Measurement
3. Interval Scale: Ordered categories that are all intervals of
exactly the same size
Example: Temperature in Fahrenheit, IQ scores
Differences between intervals are meaningful, but ratios are
not (because there is no absolute zero)
Scales of Measurement
4. Ratio Scale: An interval scale with an absolute zero point
Example: Reaction time, height, errors on a test,
temperature in Kelvin
All the qualities of an interval scale, but ratios of numbers
reflect ratios of magnitude (because the zero reflects a true
absence of the variable being measured)
Scales of Measurement
1. Do different values denote different categories?
2. Are larger values really “more” of the variable than
smaller values?
3. Are there equal intervals between values?
4. Is zero on the scale really an absence of the variable?
NOIR
Scales of Measurement
1. Numbers on basketball jerseys
Nominal
Discrete
2. Sizes of McDonald’s drinks/fries
Ordinal
Discrete
3. Weight
Ratio
Continuous
4. Scores in Jeopardy
Interval
Continuous
5. Golf scores
Interval
Discrete
Bar Graphs and Histograms
Frequency
Frequency
40
0
1
2 3
4 5
6 7
8 9 10
30
20
10
0
Seconds
blue grey green brown
Eye color
Histogram (left) for continuous variables
Bar graph (right) for discrete variables
Populations and Samples
Observations are usually made on individuals.
A population is the set of all the individuals of
interest.
Populations are often so large that it is impossible
to obtain measurements from all the individuals
A sample is a set of individuals selected from a
population – we usually want samples to be
representative (not biased)
Chris H Praful
Justin
Trinity
Mary
Ellen
Nicole
Kim
Sean
Frank
George
Sarah
John S
Greg
Erin
Jim
James
Sorab
Ruben
Trevor
Ruben
Tank
Rich
Ji
David
Will
Cory
Steve
Sean
Justin
John L Ricky
Art
Matt
Chris J
Royce
June
Ruben
Tom
Pete
Alex T
Dennis
Alex K
Grant Sue
Jhung
Ruben
Ruben
Chen
Jullian
Bubbles
Nathan Gillian
Brad
Chuck
Vera
Amanda
Tessa
Sample
Sampling
Dennis
James
Sue
Erin
June
Xiangen
Brenda
Xiangen
Hillary
Sample should be
All CogSci Students (Population)
• Representative
• Generalizable
The Population
All Individuals of Interest
Sampling
Results from the
sample are generalized
back to the population
The Sample
Individuals
selected for study
Parameters and Statistics
A parameter describes a population
A statistic describes a sample
Parameter
• Average GPA for all
U.S. university students
• Average height of all
CogSci students
Statistic
• Average GPA for IU
students
• Average height for this
class
• We use a statistic to estimate a parameter
• Generally, Greek letters denote parameters, and
Roman letter denote statistics
The Population
Average Height = 5’9’’
Inferential Statistics:
Sampling
How good an estimate
of the parameter is the
statistic?
The Sample (n=60)
Average Height =
5’6’’
The Population
Average Height = 5’9’’
Inferential Statistics:
Sampling
How good an estimate
of the parameter is the
statistic?
The Sample (n=120)
Average Height =
5’10’’
Sampling Error: The discrepancy between the sample
statistic and the true population parameter it is
estimating
Sampling Error
Sampling Error: The discrepancy between the sample
statistic and the true population parameter it is
estimating
To reduce sampling error:
• Use a sufficiently large sample
• Use random selection: selecting individuals from
the population at random for your sample to create
an unbiased sample (sometimes bias is subtle—
telephone survey example)
Statistical Truth: We can only “prove” something if we
can measure the population. As a result of sampling
error, we can only ever determine “beyond a
reasonable doubt”
Sampling Error

Random Sampling

Systematic Sampling

Quota Sampling

Opportunity Sampling

Volunteer Sampling

Snowball Sampling