Virginia Middle School Research Alliance (VMSRA)
Using the early warning system process to identify and
meet middle schoolers needs
Aimee Evan
VMSRA Lead, REL Appalachia
Virginia School-University Partnership
Summer Leadership Conference
Charlottesville, VA
June 25, 2014
Introductions
• Name
• Role
• Division
• Previous involvement with a Regional Educational Laboratory (REL)
• What brought you to this session
2
Background on REL Appalachia
Regional Educational Laboratory (REL) Program
• U.S. Department of Education, Institute of Education Sciences.
• RELs provide regional support for:
– Applied research and evaluation.
– Technical support and information sharing to build capacity to use data for
improved education outcomes.
4
REL Appalachia’s Mission
• Meet the applied research and technical support needs of Kentucky,
Tennessee, Virginia, and West Virginia.
• Bring evidence-based information to policymakers and practitioners:
– Provide support for a more evidence-reliant education system.
– Inform policy and practice for states, districts, schools, and other
stakeholders.
– Focus on high-priority, discrete issues and build a body of knowledge over
time.
5
What Does REL Appalachia Do?
• Assess regional research needs by monitoring emerging education issues
and challenges.
• Maintain and refine research alliances through ongoing dialogue between
educators in each region and researchers.
• Provide analytic technical support to increase use of data and analysis to
understand policies and programs, make decisions, and support effective
practice.
• Conduct research and evaluation studies of rigor and method appropriate
to the questions the studies attempt to answer.
• Distribute results of REL research across the region.
• Coordinate and partner with other RELs and federal, state, and local
education research and technical assistance organizations.
6
Research Alliances
• What is a research alliance?
– A partnership between education stakeholders and REL Appalachia.
• What is the purpose of a research alliance?
– As partners, REL Appalachia and alliance members develop and carry out a
research and analytic technical assistance agenda on priority topics.
• Who are the education stakeholders in an alliance?
– May include representatives from one or more schools, divisions, state
education agencies, and other organizations (e.g., colleges and universities).
7
VMSRA Goals
What Is the VMSRA?
• Superintendents, assistant superintendents, directors of curriculum
and instruction, directors of assessment and testing, principals,
teachers from the following divisions:
–
–
–
–
–
Louisa County
Norton City
Page County
Pulaski County
Russell County
–
–
–
–
–
Rockingham County
Smyth County
Staunton City
Tazewell County
Wythe County
• Executive directors of SURN (School-University Research Network) and
VSUP (Virginia School-University Partnership).
• Virginia Department of Education.
• REL Appalachia researchers in rural education, early warning systems
(EWSs), data use.
9
VMSRA Goals
• Assist middle school practitioners in using data to inform instructional
decisionmaking and improve student outcomes by:
– Identifying struggling students who need additional support.
– Selecting, implementing, and monitoring interventions to support students.
• REL Appalachia’s work with the alliance will focus on expanding the state’s
EWS to middle schools.
• EWS implementation is taking place beginning with four interrelated
projects in 2014/15.
10
What Is an EWS?
• Provides a systematic way to identify (“flag”) students early who are at
risk of failure.
• Grounded in research.
• Relies on readily available (and familiar) data: ABCs – Attendance,
Behavior, and Course grades/assessment results.
• Provides information that is actionable by educators and other adults in
schools and divisions.
• Requires educators to diagnose further student needs, and to use
professional judgment to support at-risk students.
• Targets resources to support at-risk students while they are still in school,
before they go too far down the road of academic failure and drop out.
• Examines patterns and identifies school climate issues.
11
Why Middle School Matters
• A student’s decision to drop out of school is a gradual process that starts
well before high school.*
– The habits and patterns set in middle school follow students to high school.
– Only 50% of students entering high school two or more years behind grade
level in math/literacy are promoted to grade 10.
– Students repeating grade 9 have a 20% likelihood of graduating from high
school.
• Students at risk of dropping out can be identified as early as grade 6 and
share characteristics of low attendance rates, course failure, and low
assessment scores.**
• An EWS helps catch these students before it’s too late.
* Allensworth & Easton (2005, 2007)
** Balfanz (2009); Balfanz, R., & Herzog, L. (2005)
12
How Does an EWS Work?
The EWS Continuous Improvement Cycle
Step 1: Establish
roles and
responsibilities
Step 7: Evaluate
and refine the
EWIMS process
Step 6: Monitor
students and
interventions
Step 5: Assign and
provide
interventions
Step 2: Use the
EWS Middle Grades
Tool
Step 3: Review the
EWS data
Step 4: Interpret
the EWS data
* Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013) 14
Infrastructure and Support is Just as
Important as the Process
EWS – A Systems Perspective
A. Data and Computer
Data Systems
Step 1: Establish
roles and
responsibilities
Step 7: Evaluate
and refine the
EWIMS process
Step 6: Monitor
students and
interventions
B. Educator Knowledge &
Skills for Data Use
Step 5: Assign
and provide
interventions
Step 2: Use the
EWS Middle
Grades Tool
Step 3: Review
the EWS data
Step 4: Interpret
the EWS data
C. School Organization
for Data Use
* Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013) 16
Projects Toward EWS Implementation
VMSRA Projects
• Data catalog of assessments and interventions currently being collected in
schools.
• Document how data are currently being used in schools, and what
supports and barriers are in place to help or that hinder use.
• Identify most powerful data to track.*
• Provide workshops on using data efficiently and effectively to:*
– Identify and target resources to struggling students.
– Monitor each student’s progress toward mastery.
* These are proposed activities and not yet approved by IES.
18
What Have We Learned So Far?
• Most data are available for an EWS; using data is difficult.
– Extracting or “triangulating” multiple data points is complex.
– Data are not always centrally located in one program.
– Not all pertinent data are electronic; some are kept in hard copy only.
• Analyzing historical patterns for students who are struggling may not be
possible because some data do not “save” from year to year.
• Sharing information about students across school levels—from
elementary to middle to high school—is not common.
• Assessment data is not always teacher-friendly: working with Alliance to
balance assessments FOR learning with assessments OF learning.
19
Difference Between Assessments OF and FOR Learning
Criteria
Assessments OF Learning
Assessments FOR Learning
Purpose
Check achievement status for
public reporting
Help students learn more
Informs
Others about students: teachers,
school leaders, policy makers, etc.
Students about themselves
Focus on
State or local standards
Achievement targets that underpin
standards
Answers
questions
Have students MET standards?
Have students mastered the
learning blocks/foundations of the
learning standards?
Timing
After learning is supposed to have
occurred
During learning, while there is still
time to help
Uses
Certification; public reporting;
grading; gatekeeper decisions
Understand success; diagnose
needs; see progress
Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg
20
Difference Between Assessments OF and FOR Learning
Criteria
Assessments OF Learning
Assessments FOR Learning
Teachers
role
Timekeeper
Partner; coach
Students
role
Get the highest score; avoid failure Understand success; do better next
time
Motivation Reward and punishment
Belief that success is within reach if
they keep trying
Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg
21
Importance of a Balanced Assessment Portfolio to an EWS
• An EWS uses historical patterns to identify students in need of further
assistance.
• It’s up to educators to get students in need of assistance back on track.
• Educators need the right data.
– To meet student’s needs, educators have to know what those needs are.
– To determine whether interventions are working, educators need to know
what to monitor the progress of, and they need data to be able to do the
monitoring.
• Assessments OF learning help determine whether a student is struggling.
• Assessments FOR learning help determine how to help the student.
22
Discussion
• What is the one thing we must get right to make an EWS worth
undertaking?
• What are current hurdles or barriers that would make implementing an
EWS difficult? Acting on data difficult?
• Where does an EWS help meet your goals in your division? Where does it
fall short?
23
Future Work with the VMSRA
• Data-use interviews: How are you using data? What supports exist? What
barriers hinder?
• Validation study*: What are the most powerful data to identify struggling
students?
• Workshop for data leaders* (July 23): superintendents, assistant
superintendents, curriculum specialists, directors of
assessment/accountability, building principals.
– What does effective data practice look like?
– How do you create an entrenched strategy around using data, from the
division to the classroom? What foundational elements have been found to
support effective data use? What additional/different elements are needed
to use data effectively?
• Workshops* to assist in creating or building on these foundational
elements.
* These are proposed activities and not yet approved by IES.
24
Alliance Membership Review
• Open to any Virginia middle school, district, or state officer:
– Principals and central office staff are greatly encouraged to join.
• Choose a membership level (core or general) at your discretion; time
commitment varies:
– Core group will participate in monthly phone conferences with REL Appalachia
to develop the research agenda and project implementation plans for
2013/14. Thereafter, calls will be bi-monthly to discuss project progress,
receive information, or offer feedback.
– General members will participate in quarterly meetings to receive information
and offer feedback on alliance projects.
• All members may choose to participate on alliance project teams or
advisory committees.
• REL Appalachia staff will coordinate and manage all REL work.
References and Materials
•
Allensworth, E. M., & Easton, J. Q. (2005, June). The on-track indicator as a predictor of high school
graduation. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from
http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf
•
Allensworth, E. M., & Easton, J. Q. (2007, July). What matters for staying on-track and graduating in
Chicago public high schools: A close look at course grades, failures, and attendance in the freshman
year. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from
http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf
•
Balfanz, R. (2009). Putting middle grades students on the graduation path: A policy and practice brief.
Baltimore: Johns Hopkins University, Everyone Graduates Center. Retrieved from
http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policyand-practice-brief/
•
Balfanz, R., & Herzog, L. (2005). Keeping middle grades students on-track to graduation: Initial analysis
and implications. Presentation at the second Regional Middle Grades Symposium, Philadelphia, PA.
•
Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M. (2013). Middle grades
early warning intervention monitoring system implementation guide. Washington, DC: National High
School Center at American Institutes for Research. Retrieved from
http://betterhighschools.org/EWS_middle.asp
26
Connect with Us!
www.relappalachia.org
@REL_Appalachia
Aimee Evan
aevan@QRA-LLC.net
703-655-3695
27