Office
of
Research
2014
RESEARCH DIGEST
2014 HIGHLIGHTS
Formative &
Informative
Research
Page 2
Evaluation
Studies
Page 6
Our Staff
Page 13
Milestones
Page 13
The Office of Research underwent many changes in 2014, and from a
historical perspective one could say the office came full circle. The idea
for having an independent internal research unit within the West Virginia
Department of Education (WVDE) originated in 2008 as part of Global 21
restructuring efforts. The original internal research unit was established within
the Office of Assessment and Accountability—renaming the office to the
Office of Assessment, Accountability and Research—to provide a spectrum
of research and evaluation expertise; serve as a clearinghouse for all WVDE
internal, external, and joint research and evaluation projects; and provide cost
savings by building a strong internal evaluation team.
In 2010, as the unit evolved it was moved out of its original assessment office
home to become a separate Office of Research, although it still maintained
close ties with the assessment and accountability functions of the WVDE.
During this time the office continued to mature by articulating a mission
statement and goals, developing an office logic model, and formalizing
processes to ensure high quality internal research and evaluation. It also
established an Institutional Review Board (IRB) to ensure the protection
of subjects involved in WVDE research and evaluation projects. To date,
the IRB has reviewed and approved nearly thirty internal research and
evaluation studies. In collaboration with the WVDE Office of Legal Services
and Accountability and Office of Data Management and Analysis, a Research
Review Committee also was established to advance the Department’s data
governance and data access and use policies. This committee has reviewed
approximately twenty external research projects to determine whether to allow
university and other external researchers access to WVDE-collected data.
Through a series of organizational and staffing changes in 2013 and 2014,
the assessment and research functions of the WVDE were reunited once
again under a new Office of Assessment and Research within the Division of
Teaching and Learning. While the assessment unit of the office led the way in
transitioning to new high-quality assessments aligned with West Virginia’s Next
Generation Content Standard and Objectives, the research unit continued
to advance its mission of providing scientifically rigorous, independent, and
objective research and program evaluation services to the WVDE and state
education policy makers.
Throughout these organizational changes, 2014 proved to be a busy and
productive research year. We completed a number of projects that we believe
will provide relevant, timely, and useful information to education stakeholders
throughout the state, whether they represent WVDE staff, the state board,
the legislature, educators, or citizens at large. This 2014 Research Digest
chronicles that work.
FORMATIVE AND
INFORMATIVE RESEARCH
The Association Between School
Discipline and Academic Performance:
A Case for Positive Discipline
Approaches
•
•
Andy Whisman and Patricia Cahape Hammer,
September 2014
This study examined the impact on student academic
performance of referrals for disciplinary intervention in
West Virginia. The study also examined differences in
these impacts among various student subgroups.
LESS THAN 30% OF WEST
VIRGINIA STUDENTS WERE
REFERRED FOR DISCIPLINE IN
2012–2013; ONLY 11% HAD MORE
THAN ONE REFERRAL.
Method of study. Using discipline referral data entered
into the West Virginia Education Information System
(WVEIS) for the 2012-2013 school year and employing
cross tabulations and binary logistic regression
procedures, the odds of scoring below proficiency on
WESTEST 2 based on disciplinary involvement and
student characteristics were determined. Findings were
essentially the same for math and reading/language arts,
so results focus only on math.
Findings. Of 160,480 students in the analysis, about
29.6% had one or more referrals for inappropriate
behaviors. Overall about 12% had only a single referral,
10.4% had 2 to 4 referrals, and 6.7% had 5 or more.
Results include the following:
• Students with one or more discipline referrals were
2.4 times more likely to score below proficiency in
math than those with no discipline referrals; math
proficiency among these students exhibited a 40
percentage point deficit (29.7% vs. 70.3%).
• As the number of discipline referrals increased
so did the odds of poor academic performance.
Students with 2 to 4 referrals were 2.7 more likely
to score below proficiency; students with 5 or
more were 4.6 more likely. Students with a single
discipline referral saw a 25 percentage point
proficiency gap, while 2 to 4 referrals added
another 20 points and 5 or more referrals added
yet another 20 points.
2
•
•
When the disciplinary consequences take the
form of in-school or out-of-school suspension, the
risk of scoring below proficiency increases and
proficiency gaps widen.
Students with disabilities who had a single
discipline referral were no more likely to score
below proficiency than students with disabilities
without discipline referrals. However, when they
received 2 to 4 referrals they were 3.7 more
likely to score below proficiency; with 5 or more
discipline referrals they were 12 times more likely.
Low-income students with a single discipline
referral were 1.48 times more likely to score below
proficiency than low income students with no
referrals; those with 5 or more were 3.25 more
likely.
While previous studies showed Black students to
be at greater risk of receiving discipline referrals
and suspensions, no interaction was found
between disciplinary involvement and race relative
to academic performance.
Limitations of study. 2012–2013 was a transition year as
West Virginia deployed a new discipline management
system. It is not clear what effect this transition had on
the completeness or accuracy of data summarized in this
report.
Recommendations include (a) encourage diligence
in accurately reporting discipline behaviors as
required by Policy 4373; (b) provide training/technical
assistance specific to positive discipline approaches
and alternatives to suspension; (c) build district and
school staff capacity to provide appropriate behavioral
interventions via the Support for Personalized Learning
three-tiered framework; and (d) further investigate
subgroup disparities and deliver professional
development and technical assistance to minimize them.
AS THE NUMBER OF DISCIPLINE
REFERRALS INCREASED SO DID
THE ODDS OF POOR ACADEMIC
PERFORMANCE. WHEN
STUDENTS RECEIVED AN OUTOF-SCHOOL SUSPENSION, THEIR
PROFICIENCY DECREASED EVEN
MORE.
For more information, contact coauthor, Andy Whisman, Office of Research
(swhisman@k12.wv.us), or download the full report at http://wvde.state.wv.us/
research/reports2014.html.
A Descriptive Analysis of Harassment, Intimidation,
and Bullying Student Behaviors: 2012–2013
Andy Whisman, March 2014
This report describes the occurrence of discipline referrals and corresponding
interventions and consequences used by schools for harassment, intimidation,
and bullying behaviors during the 2012–2013 school year.
Method of study. Using data entered into the West Virginia Education
Information System (WVEIS), we conducted two sets of analyses—one
focusing on discipline referrals to examine the number, seriousness, and types
of harassment, intimidation, and bullying behaviors and interventions used
by schools; and a second addressing questions about the characteristics of
students reported for these behaviors.
OF THE TOTAL STUDENT DISCIPLINE
REFERRALS REPORTED IN WVEIS, 2.5% WERE
FOR HARASSMENT, INTIMIDATION, AND
BULLYING BEHAVIORS.
Findings. Of the 220,656 student discipline referrals reported in WVEIS, 5,474
(2.5%) were for harassment, intimidation, and bullying behaviors. Most of
these referrals were at the middle school level (49%) followed by high school
and elementary school at 25% each. Students referred for harassment,
intimidation, and bullying behaviors also were referred for other categories
of inappropriate behaviors, including disruptive/disrespectful conduct
(43%), failure to obey rules/authority (31%), aggressive conduct (12%), and
tardiness or truancy (11%). Unspecified or district-defined disciplinary actions
accounted for 23% of interventions or consequences used by schools in
response to harassment, intimidation, or bullying behaviors. Of the remaining,
31% were out-of-school suspensions, followed by in-school suspensions
(21%), detentions (9%), and administrator/teacher and student conferences
(6%). All other interventions or consequences were used at lower rates.
Of the 4,409 students referred for disciplinary
action for harassment, intimidation, or bullying
behaviors, most (83%) were referred for
a single offense. Nearly three quarters
(74%) of the students were male. White
students were present at a slightly
lower rate than their respective
statewide representation (87% vs.
91%), while Black students were
present at a higher rate (10% vs. 5%).
Other races were present in roughly the
same proportions as their representation
in the overall student population. A quarter
(25%) of students referred for these
behaviors were identified as eligible
for special education services. Risk
ratio calculations indicate Black
students were twice as likely to
experience discipline referrals for
these behaviors compared to White
students, and multiple race students
were 1.5 times more likely. Similarly,
students with disabilities were twice
as likely to be referred compared
with students without disabilities.
Limitations of study. 2012–2013 was
a transition year as a new discipline
management system was deployed.
The effect this transition had on the
completeness or accuracy of data
is not clear. Other limitations were
noted.
MOST OF THESE
REFERRALS WERE
AT THE MIDDLE
SCHOOL LEVEL
(49%) FOLLOWED
BY HIGH SCHOOL
AND ELEMENTARY
SCHOOL AT 25%
EACH.
Recommendations include (a)
address harassment, intimidation,
and bullying behaviors with
evidence-based interventions
integrated into a school-wide
approach aimed at improving
behaviors and overall conditions
for learning; (b) build staff
capacity to provide appropriate
behavioral interventions in the
context of the three-tiered Support
for Personalized Learning (SPL)
framework and integrate SPL as
part of school-wide approaches to
promote appropriate behavior;
(c) minimize the use of out-of-school
suspensions, and couple in-school
suspensions with interventions to
avoid depriving students of needed
3
supports; and (d) investigate the
issue of subgroup disparity in
discipline practices, and deliver
professional development and
technical assistance to schools to
help minimize disparities.
For more information, contact author,
Andy Whisman, Office of Research
(swhisman@k12.wv.us), or download
the full report at http://wvde.state.
wv.us/research/reports2014.html.
Results of a Statewide
Professional Learning
Survey of West Virginia
School Principals and
Teachers
Patricia Cahape Hammer and Nate
Hixson, April 2014
As a result of legislation in 2013
calling for multiple education
reforms, the West Virginia Board of
Education called for about a dozen
initiatives affecting various aspects
of the state education system,
including one that would push
decision making about professional
learning to the school and district
level, with its implementation taking
place primarily within professional
learning communities (PLCs). To
get a clearer picture of teachers’
and principals’ current views about
their professional learning needs
and various types of professional
development afforded to them in
recent years, the Office of Research
conducted a statewide survey of a
representative sample of teachers
and principals. The survey focused
on the following overarching
questions: What are teachers’ and
principals’ views about (a) their
professional learning experiences
overall and (b) their engagement in
PLCs?
4
Professional Learning Experiences Overall
Current preferences. Both teachers and principals rank face-to-face
professional learning higher than either blended or online learning. They also
tended to rank in-school as their preferred location for professional learning,
followed by in-district. Teachers believed that workshops (i.e., one-time events
on a focused topic), academies/institutes (i.e., series of events on a focused
topic over time), and conferences, in descending order, had the greatest
impact on improving their instruction to support greater student learning. For
teachers, PLCs ranked sixth out of the list of seven options, while for principals
PLCs ranked fourth.
Takeaway: If we want to redirect professional learning to school-based
PLCs, there is work to do with both groups of educators, but especially
with teachers, regarding their expectations for PLCs as important settings
for improving practice.
Anticipated time spent in formal professional development. When asked how
much time they expected to spend in formal professional development (e.g.,
workshops, academies, institutes, or graduate courses) during the 20132014 academic year, teachers reported expecting to spend an average of 23
hours, while principals anticipated spending an average of 33 hours. Teachers
are required by code to participate in at least 18 hours of professional
development, which is tracked and often offered by districts. Principals, on the
other hand, have a much lower requirement of 45 hours every 6 years, once
they have completed their new principal training their first year.
Takeaway: Teachers are spending slightly more than the required
minimum amount of time in professional development, while principals
are participating at levels much higher than required.
Perceptions about which experiences were most effective for improving
student learning. The great majority of principals (83%) rate the professional
development that happens at their own schools as being of the most value
for improving student learning. Teachers were a good deal less likely to
characterize professional development provided by their school or district as
good or excellent (about 60%). The great majority (80%) of teachers, on the
other hand, rated graduate courses as being the most beneficial for improving
their instruction and student learning. The Center for Professional Development
(CPD) was the second-highest rated provider for both groups and vendors
were the lowest rated for both groups. WVDE and RESAs scored in the middle,
with about two thirds of teachers and principals rating their offerings as good
or excellent.
Takeaway: There is a substantial mismatch in perceptions about the
effectiveness of professional development offered by schools, with
principals much more likely to consider it good or excellent than
teachers. There was general agreement, however, about vendors —
with only about half of both respondent groups rating professional
development that vendors provide as good or excellent and the other
half rating it poor or fair.
Needs for the future. Teachers seemed to sense less
urgency than principals about the need for additional
professional development on any of the seven topics
listed in the survey. The only topics that rose above
a neutral rating were related to implementation of the
NxGen CSOs. “Establishing and maintaining effective
PLCs” was among the bottom three needed topics
according to principals and the bottom two according to
teachers.
Takeaway: Most teachers and principals recognized
an ongoing need for NxGen-related professional
learning, but the need for capacity building in the
area of PLCs was not recognized as an urgent need
by either group.
Professional Learning Communities
The balance of this report focuses on
responses of educators who reported
they currently participate in at least one
professional learning community—
which was about three quarters of
principals in our total respondent
group and a little less than two
thirds of teachers. We asked these
respondents to answer questions
based on their experience with the
PLC in which they are most engaged
for their own professional learning.
Frequency, duration, and scheduling of
meetings. More than half of all respondents
indicated that their PLC met only once a month
or less. About a third of respondents are meeting at least
once a week, however. Principals and teachers report
similar durations for their PLC meetings, with about half
reporting meetings lasting up to 45 minutes and the
other half, less than 45 minutes. About half of principals
meet with their PLCs during nonschool hours—that is,
before school (14%), after school (32%), or during their
lunch time (2.4%). About a third of teachers meet during
nonschool hours, either before or after school (34%) and
another 21% meeting during their planning periods. A
small group of both teachers and principals mentioned
meeting weekly during scheduled 1-hour delays for
students, which may be an innovative way to schedule
more frequent meetings.
Takeaway: Large portions of the state’s principals
and teachers who currently participate in a PLC do
not have time reserved during the school day for
their meetings or, in the case of teachers, must meet
during periods reserved for planning. If PLCs are to
be the focus of innovation and change, especially in
the implementation of the NxGen standards, finding
ways to reserve time for them during the regular
workday, every week, will be essential.
Leadership, organization, and use of time. Looking first
at principals, it appears that their PLCs are facilitated by
peers. The same holds true for teachers, with only a small
percent being facilitated by principals. The great majority
of teachers (about 80%) and nearly all principals (over
95%) reported using agendas or protocols to
guide their work.
More than two thirds of teachers
are involved in PLCs that focus on
their grade level, content area, or
specialization—as recommended in
the research literature. About half
of administrators were focused
on leadership and administration.
Substantial proportions of both
groups were involved in wholeschool focused PLCs. For both
groups, the four top-ranking activities
requiring the most PLC time were in
descending order, (a) discussing student
data and learning needs; (b) addressing
district-/state-mandated requirements; (c) discussing
problems at the school; and (d) sharing information about
curricular or instructional resources and tools. Only two of
these (a and d) fall within best practices for the work of a
PLC (Cobb & Jackson, 2011).
Takeaway: Both teachers and principals report
spending relatively large amounts of time focused
on addressing district and state mandated
requirements and discussing problems at
school, which may more appropriately be topics
for collaborative planning periods rather than
professional learning.
Tone, success, and progress. Nearly two thirds of
principals characterize the tone of their meetings as
trusting, compared with a little less than half of teachers.
More than half of administrators reported a large or very
large impact for their PLCs on their own learning, and in
turn, student learning. Teachers did not give such high
5
ratings for impacts. About three times more teachers thought the PLC they
participated in had no impact or a small one (about 28%). The remaining
teachers divided evenly between moderate impacts and large/very large
impacts.
Takeaway: While a sizable majority of principals seem to have
established interactions based on trust in their PLCs, a minority of
teachers have done so. Generally, administrators have a higher
estimation of both the impacts of their PLC on student learning and their
progress as a group overall. Teachers seem to be the group most in
need of a more positive experience as members of PLCs.
References
Cobb, P. & Jackson, K. (2011). Towards an empirically grounded theory of
action for improving the quality of mathematics teaching at scale. Mathematics
Teacher Education & Development, 13(2), 6-33.
Hammer, P. C. (2013). Creating the context and employing best practices
for teacher professional development: A brief review of recent research.
Charleston, WV: West Virginia Department of Education, Division of Teaching
and Learning, Office of Research.
This report was published by the WVDE Office of Assessment, Accountability,
and Research. For more information, contact Patricia Cahape Hammer
(phammer@k12.wv.us). To view a PowerPoint presentation with notes and
charts displaying the results of the survey, visit http://wvde.state.wv.us/
research/reports2014.html.
EVALUATION STUDIES
21st Century Community Learning Centers: A
Descriptive Evaluation for 2012-2013
Larry J. White, March 2014
TOPICS FOR WHICH PROGRAM DIRECTORS
REPORTED NEEDING ADDITIONAL
PROFESSIONAL DEVELOPMENT INCLUDE
PROGRAMMING, STAFF DEVELOPMENT, AND
STEM/STEAM.
This evaluation study provides descriptive information about the
implementation and outcomes of the 21st Century Community Learning
Centers (CCLC) program in West Virginia, from September 2012 through May
2013. The report draws on information from online surveys of directors of 30
CCLC programs and from school teachers for 4,102 participating students.
Method of study. The report draws on information from online surveys
of directors of 30 CCLC programs and from school teachers for 4,102
participating students.
6
Findings. Most participating
students attended elementary
grades. Depending on the
program, the average days
students attended ranged from
about 2 to 93. Teachers perceived
the greatest improvements in
behaviors related to promptness
and quality of homework turned
in, overall academic performance,
and participation in class. Sources
of the most volunteers included
K-12 service learning programs,
parents and faculty members, local
businesses, and postsecondary
service learning. Program directors
reported the greatest level of
success with “other” (100%) and
service learning K-12 (99.7%)
volunteer groups. The two most
frequent types of support received
from partners were programming
and resources. Partnerships
engaged in funding, programming,
resources, and training were
reported to be the most effective.
The professional development
topics best attended by program
directors were programming, STEM/
STEAM, and program evaluation.
As for parent and community
involvement, more than half of
program directors indicated they
either had no family components in
their programs or that they were well
below target goals. Of those who
reported success in this area, three
main themes emerged: (a) the right
types of activities, (b) ongoing, even
daily contact with parents, and (c) a
shared commitment to the program,
which involved engaging parents
in meaningful work toward program
goals. Program directors reported
offering more than 600 substance
abuse prevention activities, involving
more than 7,000 students and nearly
1,600 adults. Nearly three quarters
of program directors found the
continuous improvement process
for after school moderately or very
helpful. Likewise, the great majority
found the WVDE monitoring visits
moderately or very helpful.
REQUESTED
TECHNICAL
ASSISTANCE
TOPICS INCLUDE
PROGRAM
EVALUATION,
PROGRAM
SUSTAINABILITY,
AND PROJECT
MANAGEMENT.
PARENT
INVOLVEMENT,
TOO, SEEMS TO
NEED ATTENTION.
Limitations of study. We cannot
assume that the CCLC attendance
was a key factor in the improvement
of behaviors perceived by teachers.
We did not hear from all program
directors, so we lack information
about at least two of the programs.
Recommendations. Topics for
which program directors reported
needing additional professional
development include programming,
staff development, and STEM/
STEAM. Requested technical
assistance topics include program
evaluation, program sustainability,
and project management.
Parent involvement, too, seems
to need attention. Additional
recommendations include, (a)
provide more opportunities for
networking among program
staff to encourage the sharing
of best practices; (b) improve
both the frequency and quality
of communication with program
staff to improve understanding of
program requirements; (c) to the
extent possible, provide technical
assistance and professional
development support to grantees to maximize their capacity to successfully
implement their programs; (d) make reasonable efforts to reduce the reporting
burden and other compliance-related tasks in order to reserve time for
program implementation, while keeping grantees accountable and focused
on improving program delivery; and (e) consider revising state evaluation
instruments to be less cumbersome and redundant.
For more information, contact the Office of Research (phammer@k12.wv.us),
or download the full report from the WVDE Office of Research website at http://
wvde.state.wv.us/research/reports2014.html.
21st Century Community Learning Centers 2013:
A Quasi-Experimental Investigation of Program
Impacts on Student Achievement in Mathematics
and Reading/Language Arts
Larry White and Nate Hixson, January 2014
This report summarizes an evaluation study investigating the effects of
participation in the 21st Century Community Learning Centers (CCLC)
program on student achievement in mathematics and reading/language arts,
for the cohort of students who participated during the 2012-2013 school year.
Method of study. We conducted a quasi-experimental examination of withinand between-group differences in student assessment outcomes (WESTEST
2) in both mathematics and reading/language arts. The treatment group
consisted of students who participated in at least 30 days of CCLC during the
2012–2013 school year. A control group consisting of demographically similar
students who did not participate in CCLC was selected using propensity score
matching (PSM).
ONLY IN READING/LANGUAGE ARTS DID THE
INTERACTION EFFECTS BETWEEN GROUPS
AND TIME APPROACH SIGNIFICANCE IN
GRADES 8 AND 10. IN BOTH CASES, CCLC
STUDENTS OUTPERFORMED THE NON-CCLC
COMPARISON GROUP.
Findings. Using a series of independent samples t-tests, we found no
significant differences between CCLC and non-CCLC students’ year-to-year
gains in mathematics or reading/language arts nor in their 2012–2013 end-ofyear scores in mathematics or reading/language arts. Using a series of paired
t tests, we found CCLC students exhibited statistically significant mathematics
gains from 2011-2012 to 2012-2013 in Grades 4, 5, 6, 7, 9 and 11; the
students also exhibited statistically significant gains during this time period
in reading/language arts in Grades 4, 5, 6, 7, 8, and 10. We ran similar tests
for non-CCLC students and found statistically significant mathematics gains
in several grades (i.e., Grades 4, 5, 6, 7, 8, 9 and 11), and reading/language
arts gains in Grades 4, 5, 6, and 7. Using repeated measures analysis of
variance tests, we found significant main effects for time, but there were no
statistically significant interaction effects between groups and time. However,
in reading/language arts, the interaction effects approached significance in
Grades 8 and 10. In both cases, CCLC students outperformed the non-CCLC
7
comparison group.
Limitations of study. This study only encompassed a single year of CCLC intervention. It is likely that academic
achievement gains on standardized assessments would not be realized until more time has elapsed. Second, we were
able to examine only Grades 4–11 in the study due to a lack of available achievement data for Grades K-3. This is a
significant limitation when one considers the fact that approximately 50% of the 2012–2013 CCLC cohort was enrolled in
these grades.
Recommendations. To the extent possible, we will attempt to prepare next year’s edition of this report at the outset of
the 2014–2015 school year, a time when the data are more actionable for CCLC program staff. Further, we will work
with CCLC program staff to plan additional research to be conducted at the conclusion of the 2013-2014 school year
to examine the impact of longer-term participation in CCLC on student academic achievement outcomes. The study
will examine outcomes for students who participated in CCLC for at least 2 academic years to determine if prolonged
participation in the program produces statistically significant gains in achievement.
For more information, contact Larry White, Office of Research (lwhite@k12.wv.us), or download the full report from the
WVDE Office of Research website at http://wvde.state.wv.us/research/reports2014.html.
Educator Enhancement Academies Evaluation Study: Phase 1-Preparation of
RESA-Based, Next Generation CSO Trainers
Patricia Cahape Hammer and Nate Hixson, March 2014
This report focuses on six regional
Educator Enhancement Academies
(EEAs) hosted by the eight regional
education service agencies (RESAs) in
the spring of 2013. The EEAs prepared
RESA-based NxGen trainers to provide
professional development for educatorsprimarily teachers-in schools across the
state.
FINDINGS ALSO SHOWED DIFFERENCES
BETWEEN THE FOUR ACADEMIES LED BY
WVDE TRAINERS COMPARED WITH THE TWO
ACADEMIES LED BY CORWIN PRESS, WITH
WVDE GENERALLY OUTPERFORMING CORWIN.
Method of study. Two surveys of all 953 EEA participants were conducted: the first was conducted after the spring
academies; the second was conducted online in September 2013 to collect participants’ views once they had led their
own training during the summer.
Findings. The EEAs provided important components of a coherent instructional system by focusing on the new NxGen
standards and instructional shifts needed to teach to the standards, and by introducing participants to materials and
tools at their disposal in their own trainings and classrooms. The design of the EEAs reflected three of five elements of
research-based professional development, including having a strong content and content pedagogy focus; alignment
with school, district, and state goals; and active learning. Later phases of the study will examine the remaining two
elements (collective participation and duration/time span). Findings also showed differences between the four EEAs led
by content experts from the West Virginia Department of Education (WVDE) compared with the two led by Corwin Press.
For example, participants at the WVDE-led trainings were much more likely than those at Corwin trainings to indicate
that the training had been a good start and they were looking forward to training others or that the training had provided
everything they needed to train—by a margin of about 2.5 to 1. Conversely, attendees at Corwin-led events were about
THE ACADEMIES PROVIDED IMPORTANT COMPONENTS OF A COHERENT
INSTRUCTIONAL SYSTEM BY FOCUSING ON THE NEW NXGEN STANDARDS
AND INSTRUCTIONAL SHIFTS, AND BY INTRODUCING PARTICIPANTS
TO MATERIALS AND TOOLS FOR USE IN THEIR OWN TRAININGS AND
CLASSROOMS.
8
12 times more likely to indicate they did not feel ready
to train others. Further, the Corwin-led participants were
neutral about the provided materials being relevant and
useful for their own trainings, while WVDE-led participants
quite strongly agreed they were relevant and useful.
Although just under 40% of both groups thought the
quality of their experiences had been about the same as
in previous regional events, about 56% of WVDE-led EEA
attendees thought their EEA experience was more useful,
compared with 16% for Corwin—a 40-point difference.
Limitations of study. Slightly different methodologies were
used for soliciting participation in the first survey, which
may have introduced temporal and response bias due to
a time lapse before surveying participants in the Corwinled EEAs.
Recommendations include (a) develop additional
materials and tools to support teachers’ use of NxGenassociated instructional practices; (b) continue to offer
NxGen professional learning opportunities to the RESAbased trainers; (c) adopt a common set of standards
for professional development—and for train-the-trainer
events—to guide providers statewide; (d) include
standards and clear objectives in contracts with vendors
and other providers; (e) evaluate the success of the
training based on the trainers’ effectiveness in meeting
the standards and objectives; and publish the results.
For more information, contact Patricia Cahape Hammer,
Office of Research (phammer@k12.wv.us), or download
the full report at http://wvde.state.wv.us/research/
reports2014.html.
Implementation of the Master Plan
for Statewide Professional Staff
Development for 2012-2013: An
Evaluation Study
Patricia Cahape Hammer, April 2014
This report examines four main aspects of the
implementation of the West Virginia Board of Education’s
Master Plan for Statewide Professional Development: (a)
basic information about the size and scope of the effort,
including attendance, duration, and adherence to the
newly adopted standards for professional development;
(b) quality of the sessions; (c) alignment of sessions
to Board goals; and (d) the impacts of the sessions on
participants’ knowledge, practice, and attitudes/beliefs.
Method of study. The following results are based on
1,018 session reports submitted by the Center for
Professional Development (CPD), one institution of higher
education (IHE, Marshall University), all eight regional
education service agencies (RESAs), and 14 offices in
the West Virginia Department of Education (WVDE); and
nearly 6,000 survey responses (64% response rate) from
a random sample of participants in these PD sessions
spanning the period June 1, 2012–May 31, 2013.
TAKEN TOGETHER, RESULTS
SHOW GENERAL PARTICIPANT
SATISFACTION, BUT DO NOT
SHOW MUCH MOVEMENT IN
IMPROVING THE QUALITY AND
IMPACT.
Findings. The most notable 3-year implementation trend
was the reduction by nearly two thirds of PD sessions
submitted by the RESAs, while CPD increased its slate
of sessions more than fivefold; IHEs held steady; and
the WVDE more than doubled its offerings. The WVDE
was responsible for more than three quarters of all
participants in sessions offered through the PD Master
Plan. Other notable implementation trends include (a)
about a third of sessions offered were brief (up to 4
hours), another third were half-day to slightly less than
2-day technical training, and the remaining third lasted
2 or more days; (b) nearly 7,400 educators participated
in sessions of 30 hours or more duration; and (c) overall,
there was less than a 60% level of compliance with the
new Board standards for professional development.
As for perceived quality, goal alignment, and impacts
trends during this time, we note the following: (a) at 3.9
on a 5-point scale, there
has been only a slight
gain (0.1 point) over
the 3 years; (b)
there has been
a rise to 76.1%
in agreement
among
participants
that the
sessions they
attended were
aligned to the
Board goal they were
intended to support,
9
the highest rate so far; and (c) although perceived effect sizes ranged from
moderate to very large, there was only a slight gain for perceived impacts on
knowledge, and slight decreases for impacts on practice and attitudes/beliefs
during the three years. Taken together, these results show general participant
satisfaction, but do not show much movement in improving the quality and
impact.
Limitations of study. Implementation and other findings in this report are based
on self-reports by providers and participants, which may be subject to bias.
Recommendations include (a) find ways to increase the participation of IHEs;
(b) develop goals for professional development with a longer view, commit to
those goals for a sustained period of time and publicize them broadly, to help
providers align their efforts into a more coherent statewide system; and (c)
provide information about the Board standards for professional learning to all
professional development providers working in the state, and develop training
and incentives that will motivate providers to craft their offerings to meet those
standards.
For more information, contact Patricia Cahape Hammer, WVDE Office of
Research (phammer@k12.wv.us), or download the full report from the Office of
Research website at http://wvde.state.wv.us/research/reports2014.html.
Implementation of the Master Plan for Statewide
Professional Staff Development for 2013-2014: An
Evaluation Study
Patricia Cahape Hammer, December 2014
RECOMMENDATIONS INCLUDE (A) ADDRESS
RESEARCH-BASED PD PRACTICES AND
BOARD STANDARDS FOR PROFESSIONAL
LEARNING THAT LAGGED BEHIND OTHERS AS
DESCRIBED HERE; (B) BRING THE LAST OF
THE PUBLIC IHES INTO THE PD MASTER PLAN;
(C) PROMOTE THE BOARD’S STANDARDS FOR
PROFESSIONAL LEARNING AT THE SCHOOL
AND DISTRICT LEVELS; (D) ADOPT A MODEL
FOR PD PROVIDERS THAT ALIGNS WITH AND
SUPPORTS RESEARCH-BASED PD PRACTICES
AND LOCAL LEARNING COMMUNITIES; (E)
REVISIT THE PURPOSES AND POSSIBLE
USES OF THE PD MASTER PLAN TO GUIDE
PD, ALIGNING IT TO THE TRANSFORMING
PROFESSIONAL DEVELOPMENT INITIATIVE;
AND (F) CONSIDER WAYS TO REDESIGN THE
EVALUATION OF THE PD MASTER PLAN AND
PUT IT TO BETTER USE.
10
The Master Plan for Statewide
Professional Development (PD
Master Plan) includes major
providers from four PD provider
groups: the Center for Professional
Development (CPD), institutions of
higher education (IHEs), regional
education service agencies
(RESAs), and the West Virginia
Department of Education (WVDE).
The goal of this evaluation was to
study the effectiveness, efficiency,
and impact of the plan.
Method of study. Most data were
collected via more than a thousand
reports submitted by providers and
more than 4,500 responses to a
survey from a random sample of
participants in the sessions.
Findings. Regarding effectiveness,
the PD Master Plan included more
topics (479), sessions (1,056),
and participants (33,196) than
in previous years. Providers’
session reports showed that of
the total participants, about 5,700
educators participated in PD lasting
at least 30 hours, which is the
duration shown by research to be
the minimum needed to change
teacher practice and impact
student learning. The two Board
Standards for Professional Learning
that providers least often reported
meeting were Standard 4 (Use of
data to plan, assess, and evaluate
professional learning) and Standard
6 (Applies research on change and
sustains support for implementation
for long-term change). Of five
research-based PD practices,
active learning scored lowest;
only slightly higher was sufficient
duration and timespan. Regarding
efficiency, decentralization of PD
appears to be working, with the
WVDE decreasing its offerings,
while the RESAs’ rose dramatically;
still, the WVDE reported the greatest
numbers for both sessions held
and participants in attendance. RESAs and WVDE
offices operated very collaboratively, partnering with
each other and IHEs. CPD, on the other hand operated
independently with few collaborations according to their
session reports. Regarding impact of the Master Plan, all
four of the Board’s Goals for Professional Development
received coverage, and the participant surveys showed
a high level of general agreement—at least 75%—that
the sessions they attended had been helpful in meeting
the Board goal with which it was aligned. According to
retrospective self-reports, the PD had large effects on
educators’ knowledge of the PD topic, and moderate
effects on their practice and attitudes/beliefs.
Limitations of study. The response rate for this study was
lower than usual, and the use of a retrospective pretest/
posttest methodology to assess changes in knowledge,
behavior and skills, and attitudes and beliefs poses some
concerns.
Recommendations include (a) address research-based
PD practices and Board Standards for Professional
Learning that lagged behind others as described
above; (b) bring the last of the public IHEs into the PD
Master Plan; (c) promote the Board’s Standards for
Professional Learning at the school and district levels;
(d) adopt a model for PD providers that aligns with and
supports research-based PD practices and local learning
communities; (e) revisit the purposes and possible uses
of the PD Master Plan to guide PD, aligning it to the
Transforming Professional Development Initiative; and
(f) consider ways to redesign the evaluation of the PD
Master Plan and put it to better use.
For more information, contact Patricia Cahape Hammer,
Office of Research (phammer@k12.wv.us), or download
the full report from the Office of Research website at
http://wvde.state.wv.us/research/reports2014.html.
Mathematics Academies 2011-2013:
Cohort 1 Evaluation Study
Amber D. Stohr, March 2014
We examine the quality and impacts of professional
development (PD) provided to the first cohort of
participants in the Mathematics Academies Initiative, a
collaboration of the Office of Special Programs, RESAs,
and Carnegie Learning, Inc. Cohort 1, which initially
included 119 members (primarily special educators)
focused on proportional reasoning in 2011-2012. In 20122013 the 77 returning participants worked on developing
algebraic thinking.
PARTICIPANTS IMPROVED THEIR
ATTITUDES AND DISPOSITIONS
TOWARD MATHEMATICS AND
MATHEMATICS TEACHING—
ESPECIALLY THEIR ENJOYMENT
OF IT, THEIR SENSE OF
POSSESSING THE COMPETENCE
AND KNOWLEDGE THEY NEED TO
TEACH THEIR STUDENTS, AND
THEIR CONFIDENCE.
Method of study. This mixed methods study used
three instruments each year. Post-PD and end-of-year
surveys collected perceptual information regarding
the quality of various aspects of the academies; and
changes in knowledge, attitudes, and practices related to
mathematics and mathematical instruction. The Learning
Mathematics for Teaching assessment (LMT) measured
changes in mathematics content knowledge and
instructional capacity.
Findings. For both years of Cohort 1, the Carnegie
Learning trainers received the highest ratings of all the
quality measures. Reaffirming these findings, the vast
majority of comments about trainers were positive.
Survey participants also gave excellent ratings for
the overall quality of the PD, and high ratings for
the academy materials—although not quite as
high as the other ratings. Participants improved
their attitudes and dispositions toward
mathematics and mathematics teaching—
especially their enjoyment of it, their
sense of possessing the competence
and knowledge they need to teach
11
their students, and their confidence. Further, results also suggest many of the attendees implemented the skills and
knowledge they gained as a direct result of the math academies. The Year 1 pre/post LMT data, revealed statistically
and practically significant mathematics content and pedagogical knowledge gains for proportional reasoning, which
substantiated self-reported gains. However, Year 2’s algebraic thinking LMT did not yield statistically significant results
and as a result, increases in knowledge or pedagogy reported by teachers could not be verified. An analysis of the
design of the Mathematics Academies Initiative provided additional credence for its potential effectiveness in improving
instructional practice—that is, the academies were structured to include four of five recommended research-based
elements: (a) content and pedagogy focus; (b) coherence; (c) active learning; and (d) sufficient duration and timespan.
A fifth element, collective participation, was not part of Cohort 1’s experience.
Limitations of study. The study was based on self-reported information, which has an innate risk of response bias. The
study lacked a knowledge/pedagogy test for the high school teachers. There was no method available for linking gains
made by Cohort 1 participants to the mathematics performance of their students.
THE YEAR 1 PRE/POST LEARNING MATHEMATICS FOR TEACHING
ASSESSMENT DATA, REVEALED STATISTICALLY AND PRACTICALLY
SIGNIFICANT MATHEMATICS CONTENT AND PEDAGOGICAL KNOWLEDGE
GAINS FOR PROPORTIONAL REASONING, WHICH SUBSTANTIATED SELFREPORTED GAINS. HOWEVER, THE YEAR 2 ALGEBRAIC THINKING LMT DID
NOT YIELD STATISTICALLY SIGNIFICANT RESULTS.
Recommendations include (a) continue offering academy activities that have elements of effective PD; (b) encourage
special education directors to recruit participants from counties unrepresented in Cohort 1; (c) review academy content
and classroom examples to ensure their appropriateness for special education students and environments; (d) promote
collective participation by recruiting coteaching pairs or teams from the same school or district; (e) continue to develop
the algebra content knowledge and instructional skills of middle school special educators; and (f) explore ways to
investigate if any correlation exists between the Mathematics Academies Initiative and gains in math among special
education students.
For more information, contact Amber D. Stohr, Office of Research (astohr@k12.wv.us), or download the full report at
http://wvde.state.wv.us/research/reports2014.html.
12
OUR STAFF
Our expert staff is trained and experienced in utilizing state-of-the-art qualitative and quantitative social science and
assessment methodologies. As 2014 closed, the following professionals made up our staff:
Larry J. White, Interim Director, Office of Assessment and Research
Andy Whisman, Ph.D., Assistant Director, Research
Patricia Cahape Hammer, M.A., Coordinator, Research and Evaluation
Chad Morrison, M.A., Coordinator, Research and Evaluation
Jason E. Perdue, M.A., Manager, Technology and Data
Amber D. Stohr, M.A., Coordinator, Research and Evaluation
We were supported by two staff associates:
Jennifer Kozak, Secretary II
Kristina Smith, Secretary II
MILESTONES
Juan D’Brot, former executive director of the Office of Assessment, Accountability, and Research took a position as
director of Strategic Research Solutions at CTB/McGraw-Hill. Office of Research founder Larry J. White returned to serve
as interim director of the reconfigured Office of Assessment and Research.
Former assistant director, Nate Hixson became a senior manager at ICF International, and Andy Whisman, former
coordinator of research and evaluation, moved into the assistant director position.
We welcomed Chad Morrison to our staff as a coordinator of research and evaluation. Chad’s current projects focus
primarily on student behaviors and school health. Prior to joining the Department, he worked in the nonprofit sector in
hunger relief for Mountaineer Food Bank. Additionally, he worked with RESA 4 as a program site coordinator for 21st
CCLC programs. Chad received his bachelor’s degree in sociology and
anthropology from West Virginia University. He holds a master’s degree
in sociology with emphasis on applied social research from West
Virginia University. He has also completed
a master’s certificate in cultural resource
management.
Cathy Moles advanced to become
the lead secretary for the Division of
Technology.
13
NOTES
14
West Virginia Board of Education
2014-2015
Gayle C. Manchin, President
Michael I. Green, Vice President
Tina H. Combs, Secretary
Thomas W. Campbell, Member
Lloyd G. Jackson II, Member
Beverly E. Kingery, Member
L. Wade Linger, Jr., Member
William M. White, Member
James S. Wilson, Member
Paul L. Hill, Ex Officio
Chancellor
West Virginia Higher Education Policy Commission
James L. Skidmore, Ex Officio
Chancellor
West Virginia Council for Community and Technical College Education
Michael J. Martirano, Ex Officio
State Superintendent of Schools
West Virginia Department of Education
15
Michael J. Martirano, Ed.D.
State Superintendent of Schools