Current Developments in
Quantitative Research
Methods
LOT Winter School
January 2014
Luke Plonsky
Welcome & Introductions
Course Introduction
 Methodological reform (revolution?) taking place
 Goal: more accurately inform theory, practice, and future research
Theory
Research
Practice
 Content objectives: conceptual and practical (but mostly
conceptual)
 Inform participants’ current and future research efforts
 Motivate future inquiry with a methodological focus
 Not stats-heavy, technical; assumed: basic knowledge of
descriptive and inferential statistics (e.g., M, SD, t test, ANOVA)
 Examples mostly from second language (L2) research
 Lecture, all-group discussion, and small-group discussion  ask
Qs at any time!
Course Overview
 Monday/today: Statistical power, effect sizes, and
fallacies of statistical significance
 Tuesday: Meta-analysis and the synthetic approach
 Wednesday: Assessing methodological quality
 Thursday: Replication research
 Friday: Data transparency, reporting practices, and
visualization techniques
Statistical power, effect sizes,
and fallacies of statistical
significance
Luke Plonsky
Current Developments in
Quantitative Research Methods
Day 1
Review of Common Stats: Comparing Means
ANOV
A
Mean
scores
(DV)
t test
Groups (IV)
from Bialystok &
Miller (1999)
Review of Common Stats : Correlations
 Question: What is the relationship between two
(continuous) variables?
 Positive, negative, curvilinear
 Strong, weak, moderate, none
from DeKeyser
(2000)
A Model of Research
Conduct a study
(e.g., the effects of A on B)
p < 0.05
p > 0.05
Important finding / Get published!
Modify relevant theory, research, practice
Trash
p Values
(Another quick review)
Q: Wait, real quick: What’s a p value?
A1: The probability of similar results (e.g., differences between groups;
relationship between variables) given NO difference between groups / no
relationship between variables
A2: NOT an indication of the magnitude, importance, direction, or
replicability of an effect/relationship
WHAT WE REALLY WANT TO KNOW!
 Also: Observed p values vary as a function of sample size (N), effect size
(e.g., Cohen’s d), and variance.
OK, on to the Controversy
 60+ years and 400+ articles (e.g,.,
Schmidt, 1996; Thompson, 2001)
“The almost universal reliance on merely
refuting the null hypothesis as the standard
method for corroborating substantive theories …
is a terrible mistake, is basically unsound, poor
scientific strategy, and one of the worst things
that ever happened in the history of psychology”
(Meehl, 1967, p. 72).
 APA Task Force on Statistical
Inference (Wilkinson & TFSI, 1999)
 AL: strict (dogmatic?) adherence to
NHST; very little discussion until
recently (Crookes, 1991; Ellis, 2006; Larson-Hall, 2010; Lazaraton,
1991; Nassaji, 2012; Norris, 2013; Norris & Ortega, 2000, 2006; Oswald &
Plonsky, 2010; Plonsky, 2011, 2012, 2013; Plonsky & Gass, 2011)
http://oak.ucc.nau.edu/ldp3/bib_nhst.html
(Anderson et al., 2000)
Wilkinson & TFSI (1999)
 Purpose: “to initiate discussion in the field about changes in
current practices of data analysis and reporting”
 General recommendations: be transparent; calculate power
a priori; inspect data descriptively and visually; simpler
analyses are best
 Specifics: report exact p values; report (contextualized) ESs
for all tests; CIs
Main arguments against NHST?
NHST is Unreliable
 The effects of A and B are always different—in some decimal
place—for any A and B. Thus asking ‘are the effects different?’
is foolish (Tukey, 1991, p. 100).
Study
N1
N2
M1 (SD1)
M2 (SD2)
p
d
1
5
5
15 (3)
18 (4)
.2265
0.85
↑15
↑45
↑15
↑45
15 (3)
18 (4)
15 (3)
18 (4)
2
3
↓.0276
↓.0001
0.85
0.85
The (nil) hypothesis that
d = 0 is (almost) always
false! (Cohen, 1994)
NHST is Unreliable (Cont’d)
 Same goes for p values based on correlations
 Remember: r = .30 = 0.30 = 0.30
p = .05
NHST is Unreliable (Cont’d)
“[with NHST] … tired researchers, having collected data
on hundreds of subjects, then conduct a statistical test to
evaluate whether there were a lot of subjects, which the
researchers already know, because they collected the
data and know they are tired.”
Thompson, 1992, p. 436
NHST is Crude and Uninformative
 Continuous data  yes/no dichotomy
 p values say nothing about:
 Replicability
 Theoretical or practical importance
 Magnitude of effects
 p > .05 ≠ zero effect size: The absence of evidence for differences is not
evidence for equivalence (Kline, 2004, p. 67)
 Large p values can correspond to large effects and vice versa
 Other explanations for p > .05?
 small sample/low power/high sampling error; small (i.e., hard-todetect effect size; unreliable instruments; weak treatment; other
hidden variables; …
 Appropriate for a limited period of exploratory research
 (Should be an) inverse relationship between theoretical maturity
and reliance on p
NHST is Crude and Uninformative
From Papi & Abdollahzadeh (2012)
What could these t test and resulting p values
possibly contribute here?
Key
p >.05
Do
you but
see
sizeable
d
any
similar
patterns
p here?
<.05 but
not large
d
(Hint:
look
at the p
p >.05
values
and
w/neg.
ESs) d
Taylor et al. (2006)
NHST is Arbitrary
 …surely, God loves the .06 nearly as much as the .05
(Rosnow & Rosenthal, 1989, p. 1277)
 How much more (or less) would we know if the conventional
alpha level was .03 (or .15)?
 What if tests of statistical significance never existed? (Harlow et
al., 1997)
NHST is Counter-productive
 Adherence to NHST (and p values) constrains progress of theory
 inefficient research efforts
 NHST & publication bias (Rothstein,
et al., 2005)
Conduct a study
(e.g., the effects of A on B)
 Scenario: 100 intervention studies; H0 is true (i.e., no difference
between treatments A and B with alpha .05)
p < 0.05
 (At least) 5 studies will find p < .05
p > 0.05
 95 studies will sit unpublished, or be re-run until p < .05
Important finding / Get published!
(jelly beans cause acne)
 Type 1 error (false positive) in published studies = 100%
relevant theory, research, practice
 Treatment effects Modify
(which
are nil) become grossly overestimated
Trash
Summary
 (Quantitative) linguistics research relies heavily on NHST, which
is…





highly controversial at best and possibly dangerous and to-be-avoided;
unreliable;
crude and uninformative;
arbitrary; and
counter-productive
OK, but what we can do to improve?
Power
(Or: a possible solution to our obsession with p values?)
Statistical Power
 What is it?
 Why does it matter?
 How many participants do I need? (A very practical and
common question)
What kind of power is needed vs. typical?
 Table 2 in Cohen (1992)
d=0.2
d=0.5
d=0.8
Are these Ns typical in linguistics research?
What kind of power is needed vs. typical?
 Plonsky & Gass (2011)
 2% conducted a power analysis
 Median d = 0.65 + median n = 22
 Overall post hoc power = .56
 Plonsky (2013)
 1% (6/606 studies) conducted a power analysis
 median d = .71 (inflated?) + median n = 19
 Overall post hoc power = .57
 What does this mean for





Internal validity (and, hence, external validity/generalizability)?
Past research?
Theory-building?
Practical implications?
Availability bias in meta-analyses?
The “Power Problem” in L2 Research
(Plonsky, 2013, in press)
 Rarely analyze power
 Small samples
 Heavy reliance on NHST (median = 18)
 Effects not generally very large
 Omission of non-statistical results
 Rarely check assumptions
 Rarely use multivariate statistics
Tools for Power Analysis
 Cohen’s (1988, 1992) power tables
 A priori
 Conceptually?
 Practically: http://danielsoper.com/statcalc3/calc.aspx?id=47
 Post hoc
 Conceptually?
 Practically: http://danielsoper.com/statcalc3/calc.aspx?id=49
Quick Review
What if you can’t get enough
power?
 This may be the case when, for example…
 You’re studying a very small or hard-to-find population (L3
learners of Swahili with L1 Korean)
 You have limited funding for running participants
 Your phenomenon/relationship/effect of interest is small (i.e.,
hard to detect)
 Your advisor says you can’t use the PSY participant pool
 Avoid or limit inferential stats
 Form less (sub)groups  less contrasts
 Focus on descriptives (including effect sizes and CIs)
 ‘Bootstrap’ the data?
Bootstrapping
 Random re-sampling from observed data to produce a simulated but
more stable outcome (see Larson-Hall & Herrington, 2010)
 (More) robust to: outliers, non-normal data  common
 Larson-Hall & Herrington (2010)
 ANOVA: p>.05 between NSs (n=15) and 3 learner groups (n=14, 15, 15)
 Tukey post hocs: p < .05 ONLY between NSs and Group A (p = .002); pb =
.407; pc = .834
 Bootstrapped post hoc tests  p < .05 for all three groups
 p values non-statistical due to a lack of power; Type II error
 Plonsky et al. (in press)




Re-analyzed raw data from 26 primary L2 studies
4 (of 16) Type I ‘misfits’ (i.e., 25% Type I ‘misfit’ rate)
0 Type II ‘misfits’
Too much power (via large N)  inflated findings?
BUT EVEN WITH GREATER POWER
VIA BOOTSRTAPPING, OUR
RESULTS ARE STILL BASED ON THE
FLAWED NOTION OF STATISTICAL
SIGNIFICANCE
EFFECT SIZES!
(Or: a MUCH BETTER solution to our obsession with p values)
Effect Sizes
Questions we’ll address
 What are they? How do we calculate them?
 What advantages do ESs provide over p values?
 How can we interpret ESs?
What is an effect size?
 A quantitative indication of the strength of a relationship
or an effect
 Common effect sizes
 Standardized mean differences (Cohen’s d)




 M1-M2 / SDpooled (see Excel macro for calculating d)
Correlation coefficients (e.g., r)
Shared variance (R2, eta2)
Odds Ratios (likelihood of A given B)
Percentages
Why Effect Sizes?
- An alternative to NHST (p) Null Hypothesis Significance Testing (p) vs. Effect Sizes (d)
 Unreliable: result dependent on sample size (e.g., Kline, 2009)
ESs: not dependent on N
 Crude and uninformative: a) forces continuous data into a yes/no
dichotomy; b) tells us nothing about practical significance or magnitude (e.g.,
Cohen, 1994)
ESs: Express magnitude/size of relationship
(i.e., WHAT WE REALLY WANT TO KNOW)
 Arbitrary: …surely, God loves the .06 nearly as much as the .05 (Rosnow &
Rosenthal, 1989, p. 1277)
ESs: Continuous and can be compared/combined
across studies
36
Research Questions and Their
Answers Using NHST vs. ESs
 Think of a study you read recently or one that you’re
working on.
 What were the RQs?
 Where they phrased dichotomously (Do …? Is there a
difference …?)?
 If so, what kind of answer can come from such a RQ?
 How might the findings differ with an emphasis on magnitude
rather than presence/absence of a relationship or effect?
Why Effect Sizes?
- Journal Requirements  APA Publication Manual, 6th Edition
 Three major L2 Journals: Language Learning, TESOL
Quarterly, Modern Language Journal
• Plonsky & Gass (2011): 0% (1980s)  0% (1990s) 
27% (2000s)
• Plonsky (2013): 3% (1990s)  42% (2000s)
So now effect sizes get reported more often…

?
…but very rarely do we interpret them
What do they
mean
anyway?
What does d =
0.50 (or 0.10, or
1.00…) mean?
How big is ‘big’?
And how small
is ‘small’?
What implications do
these effect have for
future research,
theory, and practice?
ESs: Summary
 ESs are best understood in relation to other, field-specific effects
 d ≈ 0.40 (small)
 d ≈ 0.70 (medium)
 d ≈ 1.00 (large)
Empirically-based, field-specific scale for
d values in L2 research
 …if people interpreted effect sizes [using fixed benchmarks] with the same
rigidity that .05 has been used in statistical testing, we would merely be being
stupid in another metric (Thompson, 2001, pp. 82–83).
 Additional considerations:








Theoretical and methodological maturity (over time)
SD units
Research setting (lab vs. classroom; SL vs. FL)
Length/intensity of treatment
Manipulation of IVs
Publication bias
Sample size / sampling error
Instrument reliability
A Revised Model of Research
Conduct a study
(e.g., the effects of A on B)
p < 0.05
d=?
p > 0.05
d=?
Accumulation of results (via meta-analysis)
More precise and reliable estimate of effects
Modify relevant theory, research, practice
Trash
Based on our discussion today,
what changes would you suggest
to the field?
10 Suggestions for Reform
1.
2.
3.
4.
A diminished reliance on NHST / p-values
Drop the “significant” from “statistically significant”
Focus on the practical and theoretical importance of results
Better educate ourselves and future generations of researchers 
Emphasize: ESs, alternatives to NHST, synthetic-mindedness in
primary research  De-emphasize NHST
5.
6.
ESs (for all findings, not only when p < .05)
CIs (for all findings, not only when p < .05)  “a quiet but insistent
reminder that no knowledge is complete or perfect” (Sagan, 1996)
7.
8.
9.
10.
Replication (to mitigate effects of low power)
Examine data visually
Meta-analysis / a synthetic approach
Initiative from the top down
Further Reading
 Beyond significance testing (Kline, 2013)
 The cult of statistical significance (McCloskey, 2008)
 Understanding the new statistics (Cumming, 2012)
 Effect sizes for research (Grissom & Kim, 2012, 2nd
ed.)
 Statistical power analysis for the behavioral sciences
(Cohen, 1988, 2nd ed.)
Connections to Other Topics
to be Discussed this Week
 Meta-analysis (relies on ES) rather than p values
(TUESDAY)
 Replication (THURSDAY)
 Reporting practices (full descriptives including ES, always;
data transparency, etc.) (FRIDAY)
Tomorrow: Meta-analysis
 Motivation for and benefits of (conceptual
understanding)
 Procedures/techniques (practical understanding)