chapter
7
Statistical Issues
in Research
Planning and
Evaluation
Chapter Outline
• Probability
• Meaningfulness
• Power
• Using information in context of the study
• Reporting statistical data
Probability
• What are the odds that a certain thing(s) will
happen?
– Equally likely events
– Relative frequency
• Alpha (α):
– Level of chance occurrence set by the researcher
prior to the study
– Typical: p < .05 or p < .01
• Odds that findings are due to chance are either 5 or 1 in
100
– Controls for Type I error
Truth Table for
the Null Hypothesis
Ho true
Ho false
Accept
Correct decision
Type II error
(beta)
Reject
Type I error
(alpha)
Correct decision
Probability
• Beta (β):
– Magnitude of a Type II error
– Acceptance of null hypothesis when it is FALSE
• As alpha is set increasingly smaller, beta
becomes larger
Sampling for Null Hypothesis
From Experimental procedures for behavioral science, 3rd ed., by R.E. Kirk © 1995. Reprinted with
permission of Brooks/Cole, an imprint of the Wadsworth Group, a division of Thomson Learning. Fax
800-730-2215.
Meaningfulness
• The practical significance of an effect or
relationship
• Effect Size (ES)
– Standardized value that is the difference between
the means divided by the standard deviation
• Formula:
– ES = (M1 – M2)/s
Estimating Effect Size
• ES allows comparison between studies using
different dependent variables because it puts
data in standard deviation units.
• Scale:
–
–
–
–
ES of 0 is no difference,
ES of 0.2 or less is small,
ES of ~0.5 is medium, and
ES of 0.8 or more is large
Power
• The probability of rejecting the null
hypothesis when it is false (detecting a real
difference), or
• The probability of making a correct decision
• This concepts answers the following
questions:
– How large a difference is important in theory or
practice?
– How many participants are needed to declare an
important difference as significant?
Effect Size Curve to Estimate
Sample Size When p = .05
Effect Size Curve to Estimate
Sample Size When p = .01
Using Information in the
Context of the Study
• How do findings from the study fit within
the context of
– Theory
– Practice
• Context is what matters with regard to
“meaningfulness”
– Estimates of significance are driven by sample size
– Estimates of meaningfulness are driven by the size
of the difference
– Context is driven by how the findings will be used
Summary
• Information needed when planning research
1. Alpha
•
•
Establishes the acceptable magnitude of Type I error
Usually .05 or .01
2. Effect size
•
The outcome of a study expressed in standard deviation
units
3. Power
•
•
The chance of rejecting a false null hypothesis
1 - beta
4. Sample size
•
Number of participants in the study
Using the Power Calculator When
Reading a Research Study
When reading research, often sample size,
means, and standard deviations are
supplied. You can calculate the effect size
by the formula in chapter 7. Using this data
and the Power Calculator at the Web site
below, you can estimate the power to detect
a difference or relationship.
http://calculators.stat.ucla.edu/powercalc/
Using the Power Calculator
to Plan Research
If you are planning your own research, you
can often estimate the effect size from other
studies. By setting your alpha (say .05) and
power (say .8), you can use the Power
Calculator at the website below to estimate
the sample size you need.
http://calculators.stat.ucla.edu/powercalc/