Evaluation Plan 1
“RTI Made EZ” Evaluation Plan
Prepared for
Chris Nichols
Assistant Principal
Paulding County High School
By
Jeff Wallace
University of West Georgia
Introduction
Paulding County High School is located in Dallas, Georgia which is approximately 30 minutes
west of Atlanta. The school serves 1350 students ranging from ninth to twelfth grades. It is a
part of the Paulding School District which includes 33 schools and has a total of 28,500
students. PCHS opened in 1969 and was the only school in the county until 1992. The school
has 96 certified teachers and is currently receiving a state grant for low income and low
achieving schools. Low achievement is not new in education, but for Paulding County High
Evaluation Plan 2
School it has come swiftly, as they were a “Georgia School of Excellence” in 2003. Changing
demographics coupled with changing district attendance lines brought change to the school
quickly. Just as all other schools are mandated, PCHS, and the district as a whole have tried to
use Response to Intervention (RTI) as a means of correcting the issues at hand. “Interventions,
formal and informal, have been common in schools for many years. More recently Response to
Intervention (RTI) has been implemented to ensure that all students receive any needed
instruction and interventions to achieve academic success” (Martinez, 2011). Because of this,
PCHS is working within RTI to find ways to better serve the students and the challenges they
face academically. “RTI cannot be characterized by one educational program or curriculum, but
rather a transformation in the way that systems, schools, and professionals operate” (Sansosti,
2008). For this reason, the Paulding School district decided to go beyond the minimum
requirements set forth by state and federal laws and purchased software that is meant to
enhance the RTI process.
“RTI Made EZ” is a program that enables administrators and teachers alike to track students’
progress and needs in math and english through an information collecting spread sheet. The
county, to this point, has trained the RTI administrator at each school on the software, and it is
the job of that administrator to train the teachers. The software uses grades entered by
teachers to alert the proper administrator and teachers that the student is falling behind or
needs assistance in a certain area. Both administrators and teachers have access to the
information and can use it to proceed in the best interest of the student as it pertains to
interventions. It is the hopes of the District that this approach will be superior to the “wait till
failing” approach that has been used in the past.
I, Jeff Wallace, am the evaluator. I am a US History teacher at PCHS where I am entering my
10th year. This evaluation is part of my Program Evaluation class for my EDS in instructional
technology. The client for this evaluation is PCHS assistant principal, Chris Nichols. Mr. Nichols
would like for me to explore the overall use, understanding, and effectiveness of the “RTI Made
EZ” program. It is his concern that though the software can be useful if it is not understood or
implemented effectively, then its purpose will not be served.
The basic format of this evaluation will be in case study form. “A case study program
evaluation’s main purpose is to provide stakeholders and their audiences with an authoritative,
in-depth, well-documented explication of the program” (Stufflebeam, 2000). To this end, there
will be an examination of what is being done within the district and the school in order to
implement “RTI Made EZ” and accomplish stated goals. Given that “almost all words in the
English language that end in “ing” refer to process or means” (Kaufman, 2006), this evaluation
will examine words such as using, accomplishing, failing, meeting, and implementing as they
relate to the program. This however should not and will not be the only focus of the evaluation.
Evaluation Plan 3
Because “useful evaluations focus on ends and not just means” (Kaufman, 2006), the evaluation
will also identify what is actually being accomplished by the program. In order to complete a
well-rounded evaluation, it is imperative that each level of the program and each stakeholder
within the program be included.
Purpose
The Paulding school district purchased the software “RTI Made EZ” in an attempt to better
help those students who are deficient in certain areas of Math and English. The county believes
that the “application of RTI principles significantly enhances the likelihood of resolving common
academic and behavioral problems and improving system outcomes” (Reschly, 2009). If this is
true, then the faster the areas of concern are identified, the faster we can start solving the
problem(s).
The purpose of this evaluation is to determine the overall effectiveness of the program as a
whole. From a school standpoint is information delivered in an easy way to understand format
and is proper time given to master the software? From a classroom standpoint, is information
delivered in a way that teachers can see the benefit and ease of using this software for student
achievement? From the student standpoint, is the information used by teachers and
administrators actually increasing achievement? What barriers are there, and how can those be
eliminated? The ultimate goal of the evaluation is to streamline the process so that the purpose
of the program can be better achieved.
Objectives
1) Explore the level of understanding that school administrators and teachers have for “RTI
Made EZ” and evaluate the effectiveness on student achievement to this point.
2) Explore the barriers that are standing in the way of this program becoming more
effective.
Evaluation Questions
1)
2)
3)
4)
5)
What impact has this program had on student success at PCHS?
How is the program being diffused to teachers and administrators?
In what ways is the program currently in use?
What support is available for teachers and administrators with use of the program?
What barriers are being faced with the implementation of this program?
Evaluation Plan 4
Participants
The participants for this evaluation will be PCHS teachers that are responsible for the
tracking of students in Math and English classes. Administrators will also be involved, as they
oversee the program within the school. The exact number will not be known until the data
collection begins.
Design and Procedures
At the end of the first nine weeks program teachers and the RTI administrator will be asked
to participate in a qualitative survey. This survey will be delivered through e-mail, or can be
taken online through Google docs. The survey will help determine the demographics as well as
the rate of use that each participant has in his or her classroom. After the survey, select
teachers will be asked to participate in an interview process. The interview will consist of more
detailed questions that cannot be answered in a survey. Those asked to participate in the
interview will be those that have the most experience with the program.
Instruments
Evaluation
Questions
1) Impact on
success.
2) Program
Diffusion
3) Program
Use
4) Support
available
5) Barriers
Faced
Survey
District
Grade
Reports
Interview
X
x
X
X
X
X
X
X
X
Survey- The survey is designed to collect qualitative data in various ways. Stakeholders will
answer a series of questions that will give the demographics of those questioned. They will then
answer a series of multiple choice and yes no answers that will give a an indication as to their
feelings about how the program was taught to them, their experience and usefulness of the
Evaluation Plan 5
program, the support they are given, as well as the amount of time they devote to the program
on a weekly basis.
Interview- The interview is designed to get more in depth qualitative information. Teachers
asked to participate in the interview process will be asked questions of more depth. These will
include but not be limited to things such as, how has this program aided your students, how do
you use this for increased student success, how have your students reacted to this program,
what barriers hinder this program from being more successful?
District Reports- Each year the district puts out the EOCT and Graduation Test scores for the
entire county and school by school. These results will be used to obtain quantitative data by
comparing yearly test scores for students with disabilities.
Data Analysis
In an effort to determine the overall effectiveness of “RTI Made EZ” the following steps will
be taken.
1) Analyze the results of the individual Surveys as well as the Interview responses.
2) Analyze surveys as a whole and create a chart to indicate the averages as it relates to the
answers.
3) Analyze the interviews as a whole and look for patterns within the answers. Document these
patterns within a chart.
4) Create a list based on the interview answers showing the barriers that teachers face, and a
list of possible solutions to those problems.
5) Determine based on answers from interview and surveys whether there has been sufficient
training and continued professional learning in order to allow teachers to be successful.
6) Use county test score results to determine if the program is having a positive effect on the
students of PCHS and if so how much.
Conclusion
Following the findings of the surveys, interviews, and county testing results, the evaluator
will prepare a report of the findings in type form. The evaluator will use the findings of the
evaluation to create a list of recommendations for the client as well any other pieces of
Evaluation Plan 6
information that could allow for the programs improvement. Following completion of the
report, a draft will be delivered to the client, Chris Nichols. The client will review the report and
be given a few days to make any requests of the evaluator. Once that time has expired and
each side has completed their obligations a report will be delivered to Dr. Baylen of The
University of West Georgia.
References
Martinez, R. (2011). Response to intervention: how is it practiced and perceived? International
Journal of Special Education, 26(1), 44-52. Retrieved from
http://www.internationaljournalofspecialeducation.com
Sansosti, F. J. (2008). Viewing response-to-intervention through an educational change
paradigm: what can we learn? California School Psychologist, 13, 55-66. Retrieved from
http://www.casponline.org
Reschly, D. J. (2009, August). Teacher preparation for response to intervention in middle and
high schools. tq research & policy brief. National Comprehensive Center for Teacher
Quality, 28. Retrieved from http://www.tqsource.org
Stufflebeam, D. L. (2000). Evaluation models viewpoints on educational and human services
evaluation. (2 ed., p. 54). Norwell: Kluwer Academic Publishing.
Kaufman, R. (2006). Practice evaluation for educators. (p. 29-30). Thousand Oaks: Corwin
Press. Retrieved from http://books.google.com/books/feeds/volumes?q=0-76193197-x
Evaluation Plan 7
Appendix 1
Teacher Survey
Thank you for taking the time to answer this short survey!
1) Male or Female? *
A) Male
B) Female
2) Age? *
A) 25-35
B) 36-45
C) 46-55
D) 56-65
3) Years teaching, including this one? *
A) 0-5
B) 6-10
C) 11-15
D) 16-20
E) 21-25
F) 26-30
G) 31+
4) Highest degree earned? *
A) Bachelors
B) Masters
C) Specialist
D) Doctorate
5) Subject you teach?
Evaluation Plan 8
A) Math
B) English
C) Neither
6) How well would you say you have been trained on "RTI Made EZ"? *
0 – None
1 - Not enough
2 – Enough
3 - More than enough
7) How much time to you devote to "RTI Made EZ" on a weekly basis *
0- None
1- 0-15 minutes
2 - 15-30 minutes
3 - 30-60 minutes
4 - 60+ minutes
8) What percentage of your students with open RTI's do you utilize this software with? *
0 – none
1 - 0-10%
2 - 10-25%
3 - 25-50%
4 - 50%+
9) How easy is it to get help if you do not understand a part of the software? *
0 – Impossible
1 – Hard
2 – Easy
3 - Very easy
10) How many people would you say really understand the program at your school? *
0
1
2
3
4
Evaluation Plan 9
5
6+
11) What keeps you from using this software more than you do? *
A) Other classroom responsibilities.
B) Don't understand exactly what they want me to do with it.
C) I don’t feel it helps the students
D) No pressure to use it
12) In your opinion what impact has the program had on student success? *
0 – none
1 – little
2 – some
3 - a lot
13) How much time have you had to spend at "RTI Made EZ" training this year?
1- none
2- 0-30 mins
3- 30-60 mins
4- 60+ mins
14) Where do you go for help if you have a question about the program? Try to figure it out
myself
A) Counselor in the building
B) Administrator in the building
C) County level official
15) In your opinion, is the program attaining its goals? *
A) yes
B) no
16) On a scale of 0-5 with 5 being the best, how would you rate this program for helping
students?
17) On a scale of 0-5 with 5 being the easiest, how easy is this program to use?
Evaluation Plan 10
18) On a scale from 0-5 with 5 being the most, how much change needs to occur with the
program overall?
19) On a scale from 0-5 with 5 being the most, how serious does your school take this program?
20) On a scale from 0-5 with 5 being the most, what are the chances you would use this
program if not made to?
Appendix 2
Stakeholder Interview
1) How were you trained in “RTI Made EZ”?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
2) What do you feel could have made the training experience better?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
3) How do you use the program in the classroom?
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
4) What would lead you to use it more often in the classroom?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
5) What barriers get in the way of this program being more successful in your opinion?
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
Evaluation Plan 11
6) How do you feel the program has impacted your students’ success?
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
7) What is the sense of teachers as to the importance of this program?
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
8) What could be done from the counties standpoint to improve the program?
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
9) What could be done from the schools standpoint to improve the program?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
10) What could be done from the students’ standpoint to improve the program?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
11) What resources are at your disposal if you have questions or concerns about the program?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
12) What is the logical next step(s) for this program in order to continue student success?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
Data Collection Organization Matrix
Question
Related
Level
Required
Data
Data collection
Data
Primary
Evaluation Plan 12
Results
1) What impact
has this program
had on student
success at PCHS?
Survey
Q12
and 16
Micro
Data/Measura
ble Indicators
Survey/Intervie
w/County test
results
Source
tools/Procedures
Survey
And
Intervie
w
Teachers and
other stakeholders
will respond to an
online survey and
participate in an
interview. County
Test scores will be
looked at from
years past and
present to find
impact on student
success
Teachers and
other stakeholders
will respond to an
online survey and
participate in an
interview
Intervie
w Q4
County
Test
Report
2) How is the
program being
diffused to
teachers and
administrators?
3) In what ways is
the program
currently in use?
4) What support is
available for
teachers and
administrators
with use of the
program?
5) What barriers
are being faced
with the
implementation of
this program?
Survey
Q6, 13,
14
Macro
Survey/Intervie
w Results
Survey
And
Intervie
w
Macro
Survey/Intervie
w Results
Survey
And
Intervie
w
Macro
Interview
Results
Survey
And
Intervie
w
Macro
/Micro
Survey/Intervie
w Results
Survey
And
Intervie
w
Intervie
w Q1
Survey
Q7 and
8
Intervie
w Q2
Survey
Q14
Intervie
w Q9
Survey
Q11,
18, 20
Intervie
w Q3
and 6
Teachers and
other stakeholders
will respond to an
online survey and
participate in an
interview
Teachers and
other stakeholders
will respond to an
online survey and
participate in an
interview
Teachers and
other stakeholders
will respond to an
online survey and
participate in an
interview
collection by
date
10/21
Responsibility
10/21
Evaluator
10/21
Evaluator
10/21
Evaluator
10/21
Evaluator
Evaluator
Evaluation Plan 13