Measuring confidence to teach
statistics to
middle & high school grades:
The development & validation
of the SETS instruments
The Research Team
Supported in part by CAUSE (under NSF DUE #0618790)
Leigh M. Harrell-Williams
Virginia Tech
Georgia State University
Rebecca L. Pierce
Ball State University
M. Alejandra Sorto
Texas State
University
Lawrence M. Lesser
The University of Texas
at El Paso
Teri J. Murphy
Northern Kentucky
University
Why measure middle and high school
grades pre-service teachers’
Self-Efficacy to Teach Statistics?
• Teachers who are prepared to teach mathematics are
expected to teach statistics as outlined by the PreK-12
Guidelines for Assessment and Instruction in Statistics
Education (GAISE) (Franklin et al., 2007) and the Common
Core State Standards for Mathematics (CCSSM) (National
Governors’ Association, 2010).
• Self-efficacy is task specific; any instrument that assesses
self-efficacy needs to be task specific as well.
Self-Efficacy, Attitudes, and Beliefs
Teacher efficacy affects:
- teacher motivation
Existing Instruments:
- willingness to use more - attitude towards
innovative techniques
statistics (SATS, ATS)
- student achievement
- efficacy for
learning/doing statistics
- time spent teaching
(CSSE, SELS)
certain concepts
- statistical knowledge
(Czerniak, 1990;
(SCI)
Riggs & Enochs, 1990;
Wenta, 2000).
No prior instrument
measures self-efficacy
for future teachers.
GAISE PreK-12 Curriculum Framework
Process Components
1.
2.
3.
4.
A
Five
Objectives
Formulate Question
Collect Data
Analyze Data
Interpret Results
“Although these three levels may parallel
grade levels, they are based on
development in statistical literacy, not
age. Thus, a middle-school student who
has had no prior experience with
statistics will need to begin with Level A
concepts and activities before moving to
Level B.” (p. 13)
Three
Developmental
Levels
C
B
Nine
Objectives
Six
Objectives
CCSSM Framework
Grade-Specific
Standards for Each
Grade Level
VERSUS
Topic-Specific Standards
Similar to the 4 processes in the GAISE,
there are 8 mathematical practices that
are threaded throughout the CCSSM.
Development Process for
Middle Grades SETS Instrument
Identified
representative
behaviors from GAISE
items
Determined
alignment of GAISE
report to state
standards
Spring/Summer 2008
Draft items created
for instrument using
language aligned with
GAISE and state
standards
Development Process for
Middle Grades SETS Instrument
Revised item wording
based on input from
practicing elementary &
middle school teachers
Fall 2008/Spring 2009
Pilot Study
2009
Data Collection Study
for Validation Purposes
2010 - 2011
The Middle Grades SETS Instrument
26 Likert scale items in this format:
Please rate your confidence in teaching middle grades
students the skills necessary to complete the following
tasks successfully:
Scale of 1 to 6
14 Demographic items
Statistics Standards in the
Common Core, Grades 2 - 6
Selected SETS Items based
on GAISE Level-A
Grades 2 - 5:
Represent and interpret data
Collecting measurements and creating line plot
and bar graphs
12 items
•
Collect data to answer a posed statistical
question in contexts of interest to middle
school students.
Grade 6:
Develop understanding of statistical
variability:
• Recognize statistical question
• Data has distribution with specific
center/variation/shape
Summarize and describe distributions
• Create boxplots/histograms
• Summarize data numerically
•
Recognize that there will be natural
variability between observations for
individuals.
•
Select appropriate graphical displays and
numerical summaries to compare
individuals to each other and an
individual to a group.
Statistics Standards in the
Common Core, Grades 7 - 8
Grades 7:
Use random sampling to draw inferences about a
population.
• Inferences can be made from sample about
population if sample is representative.
• Generate multiple samples to gauge accuracy.
Draw informal comparative inferences about two
populations.
• Use graphs to estimate differences.
• Use numerical values to assess differences.
Grade 8:
Investigate patterns of association in bivariate
data.
• Construct and interpret scatterplots.
• Informally fit, assess and interpret a linear
relationship.
• Use contingency table to evaluate relationship.
Selected SETS Items based
on GAISE Level-B
15 items
•
Recognize the role of sampling error
when making conclusions based on a
random sample taken from a population.
•
Recognize that a sample may or may not
be representative of a larger population.
•
Recognize sampling variability in
summary statistics such as the sample
mean and the sample proportion.
•
Use interquartile range, five-number
summaries, and boxplots for comparing
distributions.
•
Interpret measures of association.
2010 – 2011 Validation Study
•
Four US public institutions of higher
education with significant proportion of
students pursuing degrees in education
•
309 participants enrolled in either an intro
statistics course or a math education course
Validation Study - Methods
•
Confirmatory Factor Analysis
•
Item Analysis
•
Rating Scale Analysis
•
Reliability
Validation Study - CFA
•
Compared unidimensional and two-dimensional
factor structure using Multidimensional Random
Coefficient Multinomial Logit Model as
implemented in Conquest software
Two dimensions: (Friel, Curcio, & Bright, 2001)
•
–
–
•
•
Efficacy to Teach “Reading the Data”
Efficacy to Teach “Reading Between the Data”
AIC and BIC confirmed two dimension structure
0.85 between-dimension correlation
Validation Study – Item Analysis
Index
Statistic
Composite
Reading the Data
Reading Between
The Data
rpoint-polyserial
Mean
0.66
0.67
0.69
MSweighted
SD
Min
Max
Nitems
Mean
SD
Min
Max
NiMS > 1.5
0.05
0.55
0.73
26
-
0.06
0.59
0.73
11
1.00
0.13
0.71
1.17
0
0.05
0.62
0.77
15
1.00
0.23
0.71
1.52
1
MSunweighted
Mean
-
1
0.98
SD
Min
Max
NiMS > 1.5
-
0.12
0.78
1.18
0
0.21
0.72
1.47
0
Validation Study – Rating Scale Analysis
Criterion
Met? Essential?
N > 10 for each response category
Yes
Yes
Unimodal distribution for each response category
Yes
Yes
Average measures increase with category
Yes
Yes
Outfit MNSQ < 2 for each category
Yes
Yes
Category thresholds increase with category
Yes
No
Measure implies Category & Category implies Measure
(Coherence)
No
No
Category thresholds increase by 0.81 logits
No
No
Category thresholds don’t increase by more than 5 logits
Yes
No
Validation Study - Reliability
Reliability of Separation - Analogous to Cronbach’s Alpha
 Composite Score: 0.94
Subscales:
 “Reading the Data” (Level A): 0.87
 “Reading Between the Data” (Level B): 0.91
The High School Grades SETS
Instrument
• Items completed Spring 2012
• Based on both GAISE (all levels) and two of the four strands
for CCSSM for High School Statistics & Probability
• Interpreting Categorical & Quantitative Data
• Making Inferences & Justifying Conclusions
• Data collection
• In-service: Summer 2012
• Pre-service: Fall 2012 & Spring 2013
• Analysis during Spring/Summer 2013
The High School Grades SETS
Instrument
44 Total items:
• 26 items from Middle Grades SETS instrument
• 18 “new” items based on level C of GAISE and the two
strands of the CCSSM
Item format:
• Please rate your confidence in teaching high school
students the skills necessary to complete the following
tasks successfully:
• Scale of 1 to 6
14 Demographic items
CCSSM “Making Inferences & Justifying
Conclusions” Strand
Selected SETS High School
Items Based on
GAISE Level C and CCSSM
Understand and evaluate random processes
underlying statistical experiments
•
•
S-IC.1. Understand statistics as a process for making
inferences about population parameters based on a
random sample from that population.
S-IC.2. Decide if a specified model is consistent with
results from a given data-generating process, e.g.,
using simulation.
Make inferences and justify conclusions from
sample surveys, experiments, and observational
studies
•
•
•
•
S-IC.3. Recognize the purposes of and differences
among sample surveys, experiments, and
observational studies; explain how randomization
relates to each.
S-IC.4. Use data from a sample survey to estimate a
population mean or proportion; develop a margin of
error through the use of simulation models for
random sampling.
S-IC.5. Use data from a randomized experiment to
compare two treatments; use simulations to decide if
differences between parameters are significant.
S-IC.6. Evaluate reports based on data.
•
Evaluate whether a specified model is
consistent with data generated from a
simulation.
•
Explain the role of randomization in
surveys, experiments and observational
studies.
•
Estimate a population mean or
proportion using data from a sample
survey.
•
Evaluate how well the conclusions of a
study are supported by the study design
and the data collected.
Potential Uses
• Research
• Assessment of Teacher Preparation Programs
• Analysis of Need for In-Service Professional
Development Programs
Current Projects
Using SETS Instruments
Jean Linner
Lassiter High School & GaDOE/GCTM Academy
“Recognizing that low teacher efficacy can inhibit effective
teaching as well as student learning, we hope to use the SETS
instrument for high school in a longitudinal study to identify and
target those professional learning experiences that increase
teacher efficacy for teaching statistics.”
Current Projects (continued)
Stephanie Casey & Andrew Ross
Eastern Michigan University
Scores from the SETS for pre-service secondary mathematics
teachers enrolled in a statistical methods course were used to
assess the pre-service teachers' confidence in teaching statistical
concepts in general. Additionally, scores were used to compare
the treatment (reform-oriented instruction) section and the
control (traditional instruction) section. The results will be used
to inform the instruction of the course and assess whether there
was a significant difference between the treatment and control
groups with respect to their self-efficacy for teaching statistics.
Current Projects (continued)
Salome Martínez & Eugenio Chandía
Universidad de Chile & Pontificia Universidad
Católica de Chile
The government of Chile is funding a curriculum project for the
preparation of future elementary teachers. The curriculum
consists of a series of textbooks for all the content areas,
including the Data and Chance strand of the national standards.
The SETS instrument will be used to evaluate the impact of the
implementation of the textbooks in 10 Chilean institutions of
teacher preparation.
Information on Using SETS Instruments
E-mail Rebecca Pierce
at rpierce@bsu.edu
to request “Terms of Use” form
Thank you for attending our webinar!
We’d like to open the floor to discuss the
following:
• How would you envision using one of the SETS
instruments?
• What else would you like to know about the
SETS instruments?
Related Papers
 Hilton, S., Kaplan, J., Hooks, T., Harrell, L. M., Fisher, D., & Sorto, M. A. (2008).
Collaborative projects in statistics education. In Proceedings of the 2008 Joint
Statistical Meetings, Section on Statistical Education (pp. 752-756). Alexandria,
VA: American Statistical Association.
 Harrell, L. M., Pierce, R. L., Sorto, M. A., Murphy, T. J., Lesser, L. M., & Enders, F.
B. (2009). On the importance and measurement of pre-service teachers’
efficacy to teach statistics: Results and lessons learned from the development
and testing of a GAISE-based instrument. In Proceedings of the 2009 Joint
Statistical Meetings, Section on Statistical Education (pp. 3396-3403).
Alexandria, VA: American Statistical Association.
 Sorto, M. A., Harrell, L. M., Pierce, R. L., Murphy, T. J., Enders, F. B., & Lesser, L.
M., (2010). Experts’ perceptions in linking GAISE guidelines to the self-efficacy
to teach statistics instrument. In Proceedings of the 2010 Joint Statistical
Meetings, Section on Statistical Education (pp. 4289-4294). Alexandria, VA:
American Statistical Association.
 Harrell-Williams, L. M., Sorto, M. A., Pierce, R. L., Lesser, L. M., & Murphy, T. J.
(Under Review). Validation of Scores from a New Measure of Pre-service
Teachers’ Self-efficacy to Teach Statistics in the Middle Grades.
REFERENCES
 Czerniak, C. M. (1990). A study of self-efficacy, anxiety, and science knowledge in
preservice elementary teachers. Paper presented at the National Association for
Research in Science Teaching, Atlanta, GA.
 Enochs, L.G., Smith, P.L., Huinker, D. (2000). Establishing factorial validity of the
mathematics teaching efficacy beliefs instrument. School Science and
Mathematics, 100, 194-202.
 Finney, S. J., & Schraw, G. (2003). Self-efficacy beliefs in college statistics courses.
Contemporary Educational Psychology, 28, 161-186.
 Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M., & Scheaffer,
R. (2007). Guidelines for Assessment and Instruction in Statistics Education
(GAISE) Report: A Pre-K-12 Curriculum Framework. Alexandria, VA: American
Statistical Association. (Also available at
http://www.amstat.org/education/gaise/)
 Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical
factors influencing comprehension and instructional implications. Journal for
Research in Mathematics Education, 32(2), 124-158.
REFERENCES (continued)
 Lutzer, D., Rodi, S., Kirkman, E., & Maxwell, J. Statistical Abstract of
Undergraduate Programs in the Mathematical Sciences in the United States, Fall
2005, CBMS Survey, American Mathematical Society, Providence, R.I., 2007.
 National Governors Association (2010). Common Core State Standards for
Mathematics.
http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf
 Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary
teacher's science teaching efficacy belief instrument. Science Education, 74(6),
625-637.
 Watson, J. (2001). Profiling Teachers’ Competence and Confidence to Teach
Particular Mathematics Topics: The Case of Chance and Data. Journal of
Mathematics Teacher Education, 4, 305–337.
 Wenta, R. G. (2000). Efficacy of preservice elementary mathematics teachers.
Unpublished doctoral dissertation, Indiana University.