Today:
•
•
•
•
•
Assignment 2 misconceptions
Our process
True Experiments: Single-Factor Design
Assignment 3 Q&A
Mid-term: format, coverage
Assignment 2 problems
• Still issues with IV/DV identification
• Not enough detail to really answer the questions
• Not fully operationalizing the concept of the DV into
the actual data that will be measured
• Discrete/continuous (for the measurement)
• Scaling types
• What are descriptive statistics
• Which descriptive statistics are appropriate for the
different types of data
• Estimation of reliability/validity of measurement (not
the experiment)
Where we are in the process
• Have defined broad research question and
identified hypotheses or very refined research
questions that must be evaluated to answer
the question (Ass 1)
• Have identified the variable that need to be
manipulated/studied in order to evaluate the
hypotheses and have thought about what kind
of data will need to be measured (Ass 2)
Assignment 3
• Start thinking about threats to validity – what will you have
to justify in order to convince readers that you have a valid
study that addresses the research question
– Note: as of yet, you don’t have a study designed. But you
should be thinking about possible study designs that might be
appropriate
• Ass. 2 was descriptive stats (individual variables) – in Ass. 3,
you need to think about which comparisons you need to
make
– Different measurements, single variable?
• Satisfaction before/after treatment?
– Correlations between different variables?
• Satisfaction as it relates to a participant’s age?
– Focus on describing the data – what tables/graphs would you
use to show the relationships
Next:
• Assignment 4: Hypothesis testing
– Starting to think about statistical evaluation of the
hypotheses:
• Stating alternate and null hypotheses
• Is each hypothesis one tailed or two tailed?
• If there was a type I error, what would be the
conclusion statement based on the error? What should
it be if you knew it was an error?
• If you had a type II error, what would be the conclusion
statement based on the error? What should it be if you
knew it was a error?
Next…
• Assignment 5: Control
– By now, you should be able to focus in on your
study design
– For each hypothesis, identify:
• Where you will get the data
• How you will rule out alternate explanations for the
results (give an alternate & show why it’s not viable)
• The limitations to your study
Ongoing action items (1)
• Look at related work:
– What papers will you cite when writing your
research paper
– Papers to motivate the problem
– Papers that have used similar methodologies to
answer similar types of questions
– Papers that establish baselines in your area
– Papers about methodologies
– Papers that you should compare your results with
(if you actually end up running the study)
Ongoing action items (2)
• Think about what kind of study is needed to
be able to answer the research
questions/hypothesis that you have identified
– How many factors are you looking at?
– How much control do you need?
– What are your biggest validity concerns that need
to be addressed?
– What is the usual approach in your discipline?
–…
The research paper
• Full details on Tuesday
• Proposing a study to evaluate your research
questions
• The assignments are the setup to get you thinking
about the details
• In the paper, you will be writing them formally
• In the presentation, you will be presenting them
• Your ability to discuss alternatives and justify your
chosen approach will be key to a good mark
Mid-term
• 2:35-4:05, Friday, October 28th
• 90 minutes (will aim to have most of you done
in 60 minutes)
• At 4:20 we will have in-class discussion of the
mid-term answers
– note: this is your chance to discuss what would be
a good answer and I have not budgeted time to do
this on an individual basis, so stick around. After
this session, it should be self-evident why you
receive the marks you get on your mid-term
Mid-term
• 2:35-4:05, Friday, October 28th
• 90 minutes (will aim to have most of you done
in 60 minutes)
• At 4:20 we will have in-class discussion of the
mid-term answers
– note: this is your chance to discuss what would be
a good answer and I have not budgeted time to do
this on an individual basis, so stick around. After
this session, it should be self-evident why you
receive the marks you get on your mid-term
What will be covered
• Concepts covered in class (including the
material from today)
• Concepts covered in tutorials (up to the
material covered on Monday, October 17)
• Anything is fair game – if we talked about it,
you may be evaluated on it
Question types
• Multiple choice or T/F with explanation
• Short answer
– Define/contrast/explain/describe concepts
covered
• Analysis (apply your knowledge)
– Given a research question or a study description,
answer questions about the study design, validity,
data analysis, etc.
True Experiments: Single-Factor
Design
Research Methods & Statistics
Oct 21, 2011
True Experiment
• Experimental control
• Control as many potential threats to validity as
possible
• Random assignment of participants/data to
conditions
• Could be within-subjects or between-subjects
Control
• True experiment = complete control over the
subject assignment to conditions and the
presentation of conditions to subjects
– Control over the who, what, when, where, how
• Control of the who => random assignment to
conditions
– Only by chance can other variables be confounded
with IV
• Control of the what/when/where/how => control
over the way the experiment is conducted
Quasi-Experiment
• When you can’t achieve complete control
– Lock of complete control over conditions
– Subjects for different conditions come from
potentially non-random pre-existing groups
Terminology
• Factors: IVs of an experiment
• Level: particular value of an IV
• Condition: a group or treatment (technique)
– e.g., Condition 1: old system, Condition 2: new
system
• Treatment: a condition of an experiment
• Subject: participant (can also think more
broadly of data sets that are ‘subjected’ to a
treatment)
Factors to Treatments
• At least 1 Factor (IV) has to vary to have an
experiment
– Effect of screen size and input technique on
performance (speed, accuracy)
• An IV must always have at least 2 levels
• Condition refers to a particular way that subjects
are treated
– Between subject: experimental conditions are the
same as the groups
– Within subjects: only 1 group, that experiences every
condition (can be many conditions in an experiment)
Experimental Design: spot the flaw
• One-Group Post-Test-Only Design
– Group of subjects are given a treatment
– Then tested on the dependent variables
• What’s the problem?
Experimental Design: spot the flaw
• Post-Test-Only, non-equivalent control groups
– Non-random allocation of subjects into groups
– One group is given the treatment, one doesn’t
receive it (different levels to each group)
– Post-test: measure the DV
• What’s the problem?
Experimental Design: spot the flaw
• One-Group Pre-Test-Post-Test Design
– Single group (within subjects)
– Pre-test: measure the DVs
– Give the treatment
– Post-test: Re-measure the DVs
• What’s the problem?
Good Experimental Design
• Two-Group, Post-Test Design
• Two conditions
• Two groups:
– Between subjects: random allocation
• Treatment
• Post-test: measure the DV
• What’s really important?
Within-subjects
• Similar to the one-group pre-test-post-test design
• It solves the individual differences issues
• But raises other problems:
– Need to look at the impact of experiencing the two
conditions
– Will they get tired? Gain practice? Learn what is
expected?
• Need to control for order and sequence effects?
Order Effects
• Changes in performance resulting from
(ordinal) position in which a condition appears
in an experiment (always first?)
• Arises from warm-up, learning, fatigue, etc.
• Effect can be averaged and removed if all
possible orders are presented in the
experiment and there has been random
assignment to orders
Sequence effects
• Changes in performance resulting from
interactions among conditions (e.g., if done
first, condition 1 has an impact on
performance in condition 2)
• Effects viewed may not be main effects of the
IV, but interaction effects
• Can be controlled by arranging each condition
to follow every other condition equally often
Counterbalancing
• Controlling order and sequence effects by arranging
subjects to experience the various conditions (levels of the
IV) in different orders
• Self-directed learning: investigate the different
counterbalancing methods
–
–
–
–
Randomization
Block Randomization
Reverse counter-balancing
Latin squares and Greco squares (when you can’t fully
counterbalance)
– http://www.experiment-resources.com/counterbalancedmeasures-design.html
Discussion
• Pros and cons of within versus between
subjects designs
Assignment 3: Q&A
• For five of the threats to validity covered in class, give an
example of a possible threat to validity for your research
question. Note: as you have not yet designed your
experiment, this exercise is to identify possible threats
under a possible experimental design. You can use different
possible experimental designs for each threat.
• What comparisons should you make to explore the answers
to your research question? That is, which of the variables
you identified in Assignment 2 should be analyzed
individually and which variables should be compared
against each other (which do you think might have
interactions)? For each variable, specify what tabular and
graphical methods you would use to describe the data.