Common Metrics Surveys
Use of Data to Improve Educator
Preparation Programs
CAEPCon 2015
September 18, 2015
Mark Baron, University of South Dakota
Stacy Duffield, North Dakota State University
Daria Paul-Dona, Minnesota State University, Mankato
Jon Pedersen, Winona State University
Moderator: Robin White, FHI 360
Agenda
•
•
•
•
•
•
•
Overview of NExT initiative
Overview of common metrics framework
Use of common metrics data for accreditation
Common metrics validity and reliability
Technical aspects of survey administration
Use of data for program improvement
Questions and discussion
NExT Overview
• 14 colleges and universities working together
to transform how teachers are recruited,
prepared, placed, and supported, using data
to drive continuous improvement.
• NExT institutions have program specific goals
in each of these areas.
• NExT sites have formal relationships with P-12
partners and work with them to meet the
established goals.
Higher Education Partners
Valley Partnership
• Minnesota State University, Moorhead
• North Dakota State University
• Valley City State University
Minnesota State
University, Mankato
University of
South Dakota, Vermillion
Saint Cloud
State University
University
of Minnesota,
Twin Cities
Winona State
University
Twin Cities Private
College Consortium (TC2)
• Augsburg College
• Bethel University
• Concordia University–St. Paul
• Hamline University
• St. Catherine University
• University of St. Thomas
The NExT Common Metrics Framework
• Development of a valid and reliable set of
common survey instruments
• Decision-making by consensus across 14 IHEs
• Alignment of items across four surveys
administered at different points in time
• Use of psychometric analyses to guide survey
revisions
Surveys
Entry Survey
Exit Survey
• Who are the teacher candidates?
• What encouraged them to become
teachers?
• How do candidates feel about their
preparation?
• How will we contact them after
graduation?
Transition to
Teaching
• What are graduates’ perceptions about
their preparation and effectiveness
after the first year of teaching?
Supervisor Survey
• What are the supervisors’ perceptions
of the graduates’ effectiveness as firstyear teachers?
Data Governance Recommendations
• Designed to guide responsible use and sharing
of common metrics data with various internal
and external audiences
• Created by subcommittee of IHE
representatives and approved by entire
common metrics work group
Addressing Accreditation Expectations
Relationship to CAEP Standard Four – Program Impact
Satisfaction of Employers
4.3. The provider demonstrates, using measures that result in
valid and reliable data and including employment milestones
such as promotion and retention, that employers are satisfied
with the completers’ preparation for their assigned
responsibilities in working with P-12 students.
Supervisor Survey (after first year of teaching)
• Focuses on InTASC and Minnesota Standards of Effective Practice
University of MN – Twin Cities Campus
• Developing a Survey in the 4th or 5th Year of Employment
Addressing Accreditation Expectations
Relationship to CAEP Standard Four – Program Impact
Satisfaction of Completers
4.4 The provider demonstrates, using measures that
result in valid and reliable data, that program
completers perceive their preparation as relevant to
the responsibilities they confront on the job, and that
the preparation was effective.
Factor Analysis
• Conducted by Hezel Associates; all four surveys
• Revisions based on factor analytic data
– (i.e., strong vs weak items, issues with collinearity, number of items
needed to support a factor structure, etc.)
• Principal axis factor analysis (exploratory) with Varimax
rotation conducted to evaluate underlying structure of items
for each part
• Assumptions (determinant, KMO, Bartlett) tested to ensure
that factor analyses were appropriate for these data.
• Kaiser criterion was used to determine how many factors to
retain in each analysis
Reliability Analyses
Exit Survey
– 9 sub-scales identified
– All α levels > .79; for 6 of 9 subscales, α > .90
Transition to Teaching Survey (TTS)
– 9 sub-scales identified
– All α levels > .76; for 3 of 9 subscales, α > .90
Supervisor Survey
– 6 sub-scales identified
– All α levels > .87; for 4 of 6 subscales, α > .90
Technical Aspects
• Administration
– Survey Tools
• Qualtrics, Survey Monkey, Others, Paper, etc.
– Methods
• Discussing Surveys
• Course Lists, Special Events, etc.
– Target Populations
• Names, Cohorts, Demographic Information
Technical Aspects
• Qualtrics
– Look and Feel (logos)
– Collaboration
•
•
•
•
Relatively easy
Shared effort
IHEs retain autonomy
Clean data!
– Panels and Embedded/Stored Data
– Targeted Reminders
Technical Aspects
• Messaging
– Mention Smart Phones
– Use Panels for Sophisticated Mailings
– Schedule Multiple Mailings/Reminders
– Conduct Targeted Follow-up
Outcomes on Response Rates
Among the Institutions
•
•
•
•
Recruiting Diverse Candidates
Supporting Graduates
Strengthening Field Experiences
Other Common Instruments
Program Improvements
Institutional Example of Data Use
Winona State University
• Local analysis
• Data visualization
• Change from previous
administration
• Summary points
• Long-term trends
over time
• Consistent design
Studying Results Across Instruments –
NDSU
Disagree
Exit Survey
Tend to
Agree
Tend to Disagree
Agree
n
%
n
%
n
%
n
%
Effectively teach the subject matter in
my licensure area.
NDSU; n=81
NExT Aggregate; n=1661
0
22
1.3%
2
94
2.5%
5.7%
41
640
50.6%
38.5%
38
899
46.9%
54.1%
Select instructional strategies to align
with learning goals.
NDSU; n=81
0
-
4
4.9%
36
44.4%
41
50.6%
NExT Aggregate; n=1659
13
0.8%
97
5.8%
674
40.6%
869
52.4%
Effectively teach the subject matter in
my licensure area.
NDSU; n=42
0
-
1
2.4%
13
31.0%
28
66.7%
NExT Aggregate; n=671
6
0.9%
38
5.7%
231
34.4%
396
59.0%
Select instructional strategies to align
with learning goals and standards.
NDSU; n=42
0
-
1
2.4%
20
47.6%
21
50.0%
NExT Aggregate; n=671
9
1.3%
44
6.6%
268
39.9%
350
52.2%
Effectively teach the subject matter in
my licensure area.
NDSU; n=31
0
-
1
3.2%
10
32.3%
20
64.5%
NExT Aggregate; n=318
1
0.3%
6
1.9%
84
26.4%
227
71.4%
Select instructional strategies to align
with learning goals and standards.
NDSU; n=31
0
-
4
12.9%
8
25.8%
19
61.3%
NExT Aggregate; n=316
0
-
16
5.1%
89
28.2%
211
66.8%
Transition to Teaching Survey
Supervisory Survey
Studying Results Over Time
North Dakota Common Metric Pilot
Buy-In
State
Support
• Driven by new CAEP Requirements
• Share resources across institutions
• Unanimous agreement by NDACTE institutions
• ND Education Standards and Practices Board Support
• Leadership from NDSU and VCSU
Resources
• Permission from NExT and Bush Foundation for use of surveys
• Access to pre-programmed surveys and data templates
State
Aggregate
• Each institution administered surveys
• Data entry using templates
• Secure transfer to NDSU of de-identified data for state-level, aggregate-only report
USD : Systemic Data Improvements
• CM results shared with SOE faculty across all
disciplines – not just teacher preparation
• Enhanced analysis of CM results – comparison
& disaggregation of data
• SOE Research Symposium – CM data-driven
research shared among SOE faculty & P-12
partner school districts
• Greater feedback from P-12 partner school
districts using CM data
USD : Program Improvements
• Enhanced integration of coursework and fieldbased experiences
• Adopted quality assurance measures with student
teachers & field-based supervisors
• Encouraged use of common language and vision
regarding effective teaching behaviors
• Made classroom management a required course
for all education majors
• Added an educational technologist to the faculty
USD : Sample Comparative Results
Studying Results Across Instruments –
USD
Tend to
Disagree
Disagree
Exit Survey
Tend to
Agree
Agree
n
%
n
%
n
%
n
%
Effectively teach the subject matter in
my licensure area.
USD; n=105
NExT Aggregate; n=1661
1
22
1.0%
1.3%
9
94
8.6%
5.7%
49
640
46.7%
38.5%
46
899
43.8%
54.1%
Select instructional strategies to align
with learning goals.
USD; n=106
3
2.8%
5
4.7%
61
57.5%
37
34.9%
NExT Aggregate; n=1659
13
0.8%
97
5.8%
674
40.6%
869
52.4%
Effectively teach the subject matter in
my licensure area.
USD; n=30
1
3.3%
0
-
13
43.3%
16
53.3%
NExT Aggregate; n=671
6
0.9%
38
5.7%
231
34.4%
396
59.0%
Select instructional strategies to align
with learning goals and standards.
USD; n=31
0
-
1
3.2%
22
71.0%
8
25.8%
NExT Aggregate; n=671
9
1.3%
44
6.6%
268
39.9%
350
52.2%
Effectively teach the subject matter in
my licensure area.
USD; n=16
0
-
0
-
6
37.5%
10
62.5%
NExT Aggregate; n=318
1
0.3%
6
1.9%
84
26.4%
227
71.4%
Select instructional strategies to align
with learning goals and standards.
USD; n=16
0
-
1
6.3%
4
25.0%
11
68.8%
NExT Aggregate; n=316
0
-
16
5.1%
89
28.2%
211
66.8%
Transition to Teaching Survey
Supervisory Survey
Institutional Example of Data Use
Minnesota State University,
Mankato
•
Formal Data Sharing with eight P12 Partner Districts: Establishing
trust, mutuality & transparency
•
Establishing shared Human Capital
Model
•
Annual Data Sharing Days to
review common metrics
outcomes, P-12 growth measures,
candidate/graduate performance
measures
Discussions of performance of
graduates hired performance
Source: Milanowski, Heneman, Kimball, 2009. “Review of Teaching
(with consent).
•
Performance Assessments for Use in Human Capital Management”
(www.smhc-cpre.org/resources)
Minnesota State University, Mankato
All in the Spirit of Partnership
2009-2014
National Professional Development School
(PDS) Exemplary Achievement Award, 2012
PDS Mission:
To interconnect,
sustain, and enrich
learning communities
of students, staff,
faculty and citizens
from the University
and P-12 schools who
foster human
development,
professional growth,
and optimal learning
opportunities through
research reflection
and practice.
PDS Member Districts:
Bloomington, SibleyEast, St. Peter,
Faribault, Le SueurHenderson, Owatonna,
Waseca, and Mankato
Area Public Schools
Metrics for Measuring Candidate Success
Established, 1988. Enhanced, 2009-2014, in
partnership with Archibald Bush Foundation, Network
for Excellence in Teaching
Co-Teaching
Signature Initiatives
in Teacher Preparation:
• Professional Development School (PDS)
Partnerships
• Teachers of Tomorrow
• Co-Teaching
• Mentoring & Induction
• AVID Teacher Preparation Initiative
• Global Experience
• edTPA
RESEARCH POSTER PRESENTATION DESIGN © 2012
www.PosterPresentations.com
Maverick Recruitment Accomplishments
Candidate/graduate
competencies and program
effectiveness are measured
using a continuum of multiple
measures. Measures include
those unique to Minnesota
State Mankato as well as the
NExT Common Metrics. The
Common Metrics Surveys,
aligned to INTASC and MnSEPT
standards, have demonstrated
reliability and validity
established through factor
analytic studies.
The graphs shown at left
represent trends from the
Common Metric Exit Survey
plotted across time and
cohorts and serve as a sample
of what our faculty,
administrators and P-12
partners regularly review,
individually and together.
Exit Surveys are completed at
the end of the student
teaching semester, and reveal
the teacher candidates’
perceptions about the
effectiveness of their
preparation programs.
These results are always
interpreted in combination
with results from our other
content, performance and
observational measures
acquired at critical program
transition points.
We are continuously striving to
increase the reliability and
validity of all measures used.
A sample of our battery is seen
below:
• Core assessments in content
and pedagogical areas
• Field experience
observations and formative
assessments
• Summative performance
evaluations
• edTPA
• Common Metrics
• MTLE
Data collected and analyzed,
such as the graphs depicted at
left, provide faculty with
evidence that guides their
thinking on how to build on
current program strengths and
address specific areas
requiring additional
development.
Advancement via Individual Determination
(AVID)
The AVID/PDS Alliance increases school-wide learning
and performance through utilization of researchsupported methods of effective instruction, meaningful
and motivational professional development, and acts as
a catalyst for systemic reform and change.
AVID was named a “Promising Practice” by the
Minnesota State Colleges & Universities system as a
newly developed intervention to address college
readiness and developmental education innovations.
AVID Professional Learning opportunities serve teacher
candidates, Minnesota State Mankato faculty, P-12
teachers, administrators, and counselors. Each
semester approximately 30 Minnesota State Mankato
students serve as tutors in AVID elective classes across
the PDS districts. Tutors are trained by AVID Elective
Teachers on questioning strategies, levels of thinking,
and organizational strategies to use with students.
SUPPORT
Mentors are….
• Teachers on Special Assignment
(TOSAs)*
• University Supervisors
• P-12 Mentor Teachers
• Clinical Faculty
Challenges and Opportunities
NExT Steps
Roll out surveys in MN, ND, and SD—and beyond
Continue to build the aggregate
Continue longitudinal data tracking
Continue to ensure reliability and validity of surveys
Develop data agreements and/or legislation to allow
for data sharing with K-12 partners
• Find resources to support this work
•
•
•
•
•
Questions, Comments & Discussion
Mark Baron - mark.baron@usd.edu
University of South Dakota
Stacy Duffield - stacy.duffield@ndsu.edu
North Dakota State University
Daria Paul-Dona - daria.dona@mnsu.edu
Minnesota State University, Mankato
Jon Pedersen - jpedersen@winona.edu
Winona State University