Eagle County Schools TIF Grant:
Proposed Program Evaluation Plan
August 2008
Eagle County Schools (ECS) is moving to identify a third-party vendor to provide evaluation
services for the TIF grant. ECS has identified a number of areas where additional evaluation
support is needed and an overall evaluation plan is described below. ECS has budgeted
$27,000 per year for third-party evaluation services and has requested to carry-over $27,000
from the 2007-2008 TIF budget to 2008-2009. Once a third-party vendor is selected during
this next quarter, evaluation goals from the plan will be prioritized based on cost and
importance (with regard to understanding the effectiveness and impact of the grant) and
then a more detailed scope of work will be developed in partnership with the vendor.
The following evaluation plan reflects the TIF grant design, and will be constructed in such a
way as to collect process data (which teachers and building administrators are receiving
professional development and performance-based incentives and in what amounts),
outcome data (immediate and intermediate successes and long-term goal achievement), and
to assess relationships between these measurement areas. The following plan is offered as a
starting point from which the final evaluation plan would be developed in consultation with
the selected vendor. In addition to meeting the evaluation’s process and outcome
measurement and analysis objectives, collected data would be used for formative purposes
which would be summarized in progress reports and used by TIF grant staff to support
critical program reflection and drive ongoing improvements.
The larger evaluation plan and measures will be designed to be responsive to the inherent
logic of the TIF grant. The primary research questions to be answered by the evaluation
include:
1. What is the perceived effectiveness of the district-level teaching support structures
developed under TAP and further supported by TIF (i.e., master/mentor teachers,
Cluster meetings, professional development time and cycles, data-based decision
making)? What is the relationship between these structures and student outcomes as
measured by standardized achievement tests?
2. What are the relationships between TIF supported professional development,
teacher and administrator job satisfaction, teacher and administrator job
performance ratings, and student outcomes as measured by standardized
achievement tests?
3. What are the relationships between performance-based incentives, teacher and
administrator job performance ratings, and student outcomes as measured by
standardized achievement tests?
4. How does teacher quality vary by teacher type (master, mentor, and career)? How is
teacher quality at ECS changing over the life of the TIF grant? How do different
elements of quality affect student outcomes?
5. To what degree are teachers and building administrators satisfied with and
supportive of the elements of the TAP/TIF?
Process Data Collection
Process measures are divided into three distinct areas: 1) those related to teacher and
building administrator characteristics, 2) others measuring provided inputs (professional
development and performance-based incentives) and 3) those related to products that
support data-based decision making, such as EDmin (the ECS data dashboard). Firstly,
during the hiring process, each teacher completes a survey that assesses his or her anticipated
level of effectiveness. This initial screening tool will then be related to subsequent job
performance evaluation scores. Building principals and assistant principals will be surveyed
periodically to understand the extent to which there is “buy-in” for the underlying principles
of TAP/TIF and how the buy-in changes over time. Secondly, measures for professional
development provided and amounts and timing of performance-based incentives will be
integrated into the process evaluation. Thirdly, process measures around products that
support data-based decision making will be created to ensure that information related to all
elements of the TIF grant are collected. For example, user statistics and usability
assessments for EDmin will be conducted both for formative work as well as for
understanding the role of this product on the overall goals of the TIF grants. To the greatest
extent possible, these process data will be used as independent variables for later analyses
(i.e., in relation to immediate, intermediate, and overall goal attainment).
Specific questions addressed by the process evaluation will include:
1. What is the impact of master and mentor teachers on career teacher practices?
2. What does the allocation of teacher incentives, whether TIF or other district funds,
look like at ECS, including distribution details, award amounts, and timing?
3. How does the quality of the teacher hiring pool at ECS change over time?
4. How do the teacher and building administrator retention rates at ECS vary over
time? How do perceived attitudes and impacts of TAP/TIF affect retention rates?
5. How do teacher job performance ratings vary over time?
6. How has the district communicated about TIF to ECS staff?
7. To what extent are teachers and administrators supportive of the elements of
TAP/TIF?
8. What is ECS doing to support improvement in teacher practice within the context of
TIF?
9. Who uses data products such as EDMIN and for what purposes?
10. What does data-based decision making look like at ECS and how is it changing over
the life of the TIF grant?
Immediate and Intermediate Outcome Data Collection
Immediate and intermediate outcome data will be collected for all professional development
(PD) provided under the TIF grant. To the extent possible, measurements will be
conducted on a pre-post basis using existing valid and reliable measures. Where these are
not available, the program would have measures developed in collaboration with the
evaluation vendor. While performance on immediate and intermediate outcomes can be
analyzed as an independent assessment of the impact of PD on teacher practice and
effectiveness, they will also be used in more complex statistical models to assess their
relationship to longer-term program goal attainment (e.g., student outcomes).
The following reflects a preliminary list of areas for which professional development
measurement would be conducted. For the immediate outcome area, this includes:
 Content knowledge in the area(s) addressed by the PD
 Perceived value of the PD for teaching and student learning
 Satisfaction with the PD provided
Intermediate outcomes will be defined separately for PD, performance-based incentives, and
data products. For example, intermediate outcomes for incentives will include a teacher
survey of the perceived impact on teaching and job satisfaction. For PD, master and
mentors may be asked to periodically conduct short observations in areas related to the
training. For data products, intermediate outcomes would examine the amount of time
spent discussing data in administrative and cluster team meetings.
A complete specification of all immediate and intermediate outcome measurement areas will
occur in the planning period once a vendor is identified.
Goal Assessment
Several goal areas have been identified for the TIF grant. The ultimate successes of the
grant would be reflected in teacher and building administrator job performance ratings and
student achievement results including:
 Year-to-year gains in student achievement on NWEA
 Year-to-year gains in student achievement on CSAP (Colorado’s high-stakes
achievement tests)
 Year-to-year building level growth on CSAP
 Increased teacher effectiveness as measured by the SAS EVAAS’ value-added
model
 ECS progress in closing the achievement gap among English Language
Learners and students that come from economically disadvantaged families
 Parent and public perception of ECS
These data would be assessed yearly over the life of the grant to gauge changes over time. In
addition, teachers and building administrators will be asked to rate their satisfaction with the
grant and to offer suggestions for improvement.
.
Data Uses and Analyses
Collected data will be used for a variety of purposes including:
 Supporting formative programmatic improvement
 Improving management of grant elements and fidelity to the inherent logic of
TAP/TIF
 Reporting process and outcome information through the use of descriptive and
inferential statistics
Various analytic models would be developed in relation to the underlying logic of TAP/TIF
that seek to test the broader research questions posed above. Descriptive statistics will be
used to describe the teacher, administrator and student populations; the amounts and types
of PD and incentives provided; and the extent of other district-wide supports provided.
These data areas will be combined with immediate and intermediate outcome measurements
to examine various relationships that are assumed in the underlying logic of TAP/TIF. It is
envisioned that, at a minimum, factorial analysis of variance (ANOVA) and regression
models would be developed to test and isolate various program effects to the greatest extent
possible. As with all design decisions, specific analytic options will be developed in
collaboration with the selected evaluation vendor.