2014
College Club Program
Evaluation
PREPARED FOR: JOSHUA WRIGHT
EDUCATION COORDINATOR, COLLEGE CLUB
BOYS & GIRLS CLUB OF DANE COUNTY
LA FOLLETTE SCHOOL OF PUBLIC AFFAIRS
KATE AUSTIN
ISAAC HEDTKE
Table of Contents
Executive Summary .................................................................................................................................... ii I: Introduction ............................................................................................................................................ 1 Purpose of Evaluation ............................................................................................................................... 1 Literature Review ...................................................................................................................................... 1 College Club ............................................................................................................................................. 3 Research Question and Hypotheses .......................................................................................................... 5 II: Evaluation Design: ................................................................................................................................ 5 Logic Model .............................................................................................................................................. 5 Research methodology ............................................................................................................................. 8 Target population and sampling plan ..................................................................................................... 13 Data sources and collection procedures ................................................................................................. 14 Data analysis procedures ....................................................................................................................... 16 III: Quality Control and Human Subjects Protection ....................................................................... 1817 IV: Evaluation Implementation .............................................................................................................. 18 Timeline .................................................................................................................................................. 18 Implementation Factors ...................................................................................................................... 2019 Proposed Budget ..................................................................................................................................... 20 V: Anticipated Results ......................................................................................................................... 2221 Remaining Questions .......................................................................................................................... 2322 Stakeholders ............................................................................................................................................ 23 Appendix A: Logic Model ........................................................................................................................ 25 ............................................................................................................................................................ 25 Outcomes -- Impact ................................................................................................................................... 25 Appendix B: Power Calculation .............................................................................................................. 26 Appendix C: BGCDC Contract ............................................................................................................... 27 Appendix D: Parent/Guardian Survey ................................................................................................... 29 Appendix E: Student Survey ................................................................................................................ 3739 Appendix F: Data Collection Table ..................................................................................................... 4344 Appendix G: Timeline .............................................................................................................................. 45 Appendix H: College Club Evaluation Budget ...................................................................................... 47 References:................................................................................................................................................. 48 i Executive Summary
Achievement gaps in educational attainment and college attendance across racial, ethnic,
and socio-economic groups are well documented across the country and in the state of
Wisconsin. In Dane County, educational disparities between students are even more pronounced.
To locally address this growing problem, the Boys & Girls Club of Dane County (BGCDC) has
developed the College Club program to support students in elementary and middle school and to
prepare them for high school success and beyond. This multi-component program has been in
existence for five years. To demonstrate programmatic impacts, we have developed an impact
evaluation plan with the following research question: does participation in the College Club
program improve academic performance, increase successful entrance into a high school college
preparatory program, and improve college readiness when compared to students with comparable
demographics who have not participated in the program.
We have proposed a difference-in-difference, quasi-experimental methodology for this
evaluation. Sixth to eighth grade members of the College Club afterschool programs at the Taft
and Allied BGCDC Centers will be followed longitudinally for 10 years, along with a
comparison group of their peers who participate at BGCDC Centers, but are not involved with
College Club. Data will be collected using mixed methods, at the outset of the evaluation,
quarterly and yearly while the students are in middle-school, and upon entry and exit of high
school.
With this evaluation we expect BGCDC and College Club Coordinators to be able
articulate short, intermediate and long-term impacts for those students who participated in
College Club compared to those who did not. We also recommend collection of implementation
data to allow for an improved utilization of services.
ii I: Introduction
Purpose of Evaluation
The achievement gap in Dane County is growing. The Race to Equity Report for Dane
County illustrates this problem and its effect on education. In 2010-2011, 50 percent of nonHispanic Black students at the Madison Metropolitan School District (MMSD) did not graduate
with a regular diploma in four years of high school. The same is true for 16 percent of white
students. Both numbers are higher than the Wisconsin rates of 36 and 9 percent, respectively, and
there exist additional disparities beyond the black/white gap (WCCF 2013). Helping improve
students’ chances of successfully completing high school and college can begin by providing the
structure and support for changing learning habits and behaviors. One way to target these skills is
through afterschool programs. The Boys & Girls Club of Dane County (BGCDC) provides
afterschool services to youth in the area, including the College Club program. Conducting an
evaluation of College Club will illustrate if the program is effective, and highlight potential areas
for improvement. Given the current need to improve the education of youth in Dane County, this
evaluation could not be more timely nor relevant.
Literature Review
College Club is based on the reasoning that time dedicated to academic learning outside
of the classroom will help improve a student’s academic performance and educational
attainment. Afterschool programs gained acceptance for two main factors. First, students have
unsupervised time after the school day ends and before parents return home from work, which
needs to be structured. Second, the opportunity exists for academic learning and social
development during this time (Power 2008). Research has found that students who attend
afterschool programs most often perform better on standardized tests, and have higher self1 efficacy scores and educational aspirations than those students who attend less frequently
(Cosden et al. 2001).
Literature often separates afterschool programs into two categories: extended learning
and enriched learning. Extended learning is aligned with a student’s schoolwork, directly helping
them with tutoring, homework, and furthering their understanding of the key topics from class.
Enriched learning may be aligned with the schoolwork, but focuses more on outside projects to
develop further learning (Britsch et al. 2005). Both types of academic afterschool programs
provide indirect support towards academic achievement. Participation in these programs
increases the amount of engaged learning, educational equity, and key skill development for
academic success (Britsch et al. 2005).
A review of relevant literature conducted by the Northwest Regional Education
Laboratory showed a positive effect on reading scores for low achieving students who completed
out of school learning. The five studies focusing on tutoring all showed a positive effect, with
greater results for students who received one on one tutoring and those who had the lowest
reading scores (Britsch et al. 2005). Additionally, a study in California used regression
discontinuity design to evaluate the effectiveness of using computer-based programs to improve
literacy rates. This study found statistically significant effects of using computer based programs
outside of the school day to improve reading comprehension and a student’s overall academic
performance (Yancsurak 2013).
Additional studies have found similar results. While Granger (2008) did not find positive
results in all programs reviewed, he concluded that afterschool programs can have a positive
effect on student achievement in reading and math. Other studies have found greater
participation in afterschool programs and longer duration increase the benefits received by the
2 students (Afterschool Alliance 2006). And Hollister (2003) found mentoring and tutoring
afterschool programs are associated with positive outcomes in school activities and a reduction
in negative out of school outcomes.
College Club
College Club is the afterschool college prep program of the Boys & Girls Club of Dane
County, and it utilizes the successful techniques discussed above. It serves 300 students from 1st
through 12th grades in the Madison area and works to increase college readiness through
academic monitoring, tutoring and case management (BGCDC 2014). There are currently five
College Club sites located across the Madison area. These are at the Allied and Taft Boys &
Girls Clubs, at Wright and Cherokee Middle Schools, and at Verona High School. College Club
primarily targets low-income students, students of color, and students who will be first
generation college attendees with their program. The program is in its fifth year, with the cohort
size growing each year.
This proposed evaluation focuses on the programs at the Allied and Taft Boys & Girls
Clubs, specifically on middle school age children, grades six to eight. The Allied and Taft Clubs
were chosen because these are similar programs; the participant demographics are similar
between Allied and Taft, and both are after-school centered programs. While College Club
members and Coordinators at these centers interact with the schools and teachers, they are not
based in the school like the other three College Club programs.
All College Club participants must be Boys & Girls Club members, and there are three
ways to be selected to participate in the program. The first is through student interest. A student
who shows initiative through regular attendance at the Boys & Girls club and seeks help on their
homework and requests entrance can be accepted. The second path to selection is through a
3 parent or guardian request. Upon request the staff will assess the student’s interest and needs for
acceptance into the program. The third way a student is identified for participation is through a
teacher or school request for participation. Upon this request, the student is verified as a Boys &
Girls Club member, and his/her interest is assessed. Academic performance is not considered for
entry into College Club. The club attempts to retain current participants first, but has a goal of
filling every available slot and currently has a waiting list for participation.
College Club uses eight components to achieve its goal of increasing academic
performance and increasing participants’ chances of completing higher education. The program
is overseen by the Education Director, and Education Coordinators are at each site to help
orchestrate the eight components. The components are tutoring, case management, college
exposure, mentoring, school/teacher engagement, family engagement, summer academic
programs and eliminating barriers to academic achievement. Certified volunteers conduct the
tutoring. Education coordinators conduct the case management, work to eliminate barriers, and
coordinate with the schools and teachers for their involvement. The Education Coordinator also
works with the tutors to help focus tutoring sessions. Students are matched with a professional
with college experience for their mentoring. The college exposure, family engagement and
summer programming are planned at the program level. Additionally, College Club students
have access to Stride Academy, a computer-based training program focused on math and reading
skills. While this service is available to all members at the BGCDC, it is more structured for
College Club students when paired with the other components of the program.
The College Club program has received national recognition for its innovation and
improvement in reading and math scores; however, there has not been a formal study evaluating
its impact (BGCDC 2014). Conducting the evaluation now will help identify the areas of the
4 program that are succeeding in meeting goals and those that are struggling. The evaluation
results will allow the program coordinator to improve the program. Additionally, the evaluation
will help strengthen future grant applications to increase funds for College Club.
Research Question and Hypotheses
Our evaluation of the College Club program at the Boys & Girls Club of Dane County
(BCGDC) will attempt to answer the following research questions: does participation in the
College Club program improve academic performance, increase successful entrance into a high
school college preparatory program, and improve college readiness when compared to students
with comparable demographics who have not participated in the program? Additionally, we
recommend collection and analysis of implementation data to assess whether students are
reaching program benchmarks, and to improve utilization of services.
There are two potential results from this evaluation: an effect of participating in College
Club or no discernable difference between those who participated and those who did not. Results
showing an increase in academic performance, entrance into a college preparatory program,
and/or improved college readiness are signs of an effective program. However, no effect, or a
negative effect is also possible. There could be multiple factors attributing to a result of no effect.
The design of this evaluation will help illustrate how the services are utilized by participants.
Utilization results could help identify an implementation issue with the program services.
II: Evaluation Design:
Logic Model
College Club as implemented at the Taft and Allied BGCs is comprised of eight key
components (please see Appendix A for detailed logic model).
5 1. Tutoring: College Club members receive weekly, individualized tutoring sessions
from certified Wisconsin teachers or graduate students in education at the University
of Wisconsin-Madison. The Education Coordinator creates an individual
development plan for each member, based on communication with the member, their
family, and teachers. The tutor then works with the member to improve in core
content areas identified in the development plan and tracks progress using tutor logs.
2. Case Management: The Education Coordinator at each BGCDC site is responsible
for the case management of the College Club members. Coordinators work with
members to set yearly academic goals and individual development plans. They are
also responsible for identifying social, psychological and emotional barriers that may
impede academic success, good attendance and healthy behavior. Case management
is documented and progress is tracked by Education Coordinators through case
management logs.
3. Addressing and Eliminating Barriers to Academic Achievement: After the
Education Coordinator identifies psychological and socioeconomic barriers to
academic achievement, individualized attention is given to each member to attempt to
address and remove the barriers. Types of barriers members encounter can vary
widely, with some more easily resolved than others.
4. College Exposure: Education Coordinators plan at least two college visits for all
members each school year. Trips are designed to give members a better
understanding of the college environment and show attending college is attainable.
5. Mentoring: At each site, 10 members are matched with an adult mentor from the
community. For those members with mentors, they meet for a minimum of 60
6 minutes of individualized time per week, which will be tracked and measured through
mentoring logs. Mentors focus on improved academic performance through
homework completion and problem-solving, but may spend time in recreational
settings as well.
6. School and Teacher Engagement: Education Coordinators tailor individual
development plans and tutoring to align student needs with school and teacher
curriculum. They also arrange for teacher, principal and counselor visitors to the
BGCDC to create a strong support network for members.
7. Family Engagement: Educational Coordinators maintain communication with the
families of College Club members and host at least two academic conferences or
recognition events throughout the year. They work to establish sustained relationships
with families, and keep families engaged in the academic achievement of their child.
8. Summer Academic Programming: Academic programming is planned by
Education Coordinators for members during the summer at the BGCDC sites.
Reading and math skills are stressed, and field trips and library visits are organized as
well. Summer academic programming is designed to mitigate summer learning loss
that can negatively impact academic achievement.
The combination of the eight College Club key components will contribute to short-,
intermediate-, and long-term expected outcomes. In the short term, these activities are meant to
increase Stride Academy scores, lead to the development of action plans to reduce barriers, and
improve attendance and behavior in the schools. Over the course of middle school, the College
Club activities are intended to help improve core and cumulative GPA, increase educational
aspirations, and improve academic performance. Over the long term, program components are
7 designed to improve college readiness, establish a college-going culture at the BGCDC, increase
educational aspirations, and increase AVID/TOPs (Advancement Via Individual Determination/
Teens of Promise) acceptance. AVID/TOPs is a high school level college prep program
coordinated jointly by the Madison Metropolitan School District (MMSD) and the BGCDC.
AVID is a nationally recognized program that is designed to help students excel in high school
and subsequently attend and graduate from college; the TOPs program is a BGCDC initiative
intended to complement the AVID curriculum in MMSD schools (WISCAPE 2014).
There are several assumptions on which we base the logic model:
a) BGCDC has the financial and human resources necessary to implement program
activities.
b) Once activities are provided, members in the target population will take advantage of
them.
c) Once club members participate in programming, activities will lead to improved
academic performance.
d) Improved academic performance will lead to increased likelihood of participation in high
school level college preparatory programs.
In addition, there several external factors that have the potential to impact the successful
implementation of the logic model:
a) Level of parental or familial engagement in the lives of the members;
b) Uncertain socioeconomic situation of participating families;
c) Funding of BGCDC programs; and
d) Access to additional academic support outside of school and College Club.
Research methodology Our proposed evaluation uses the quasi-experimental design of a difference-in-difference
study. Difference-in-difference is a technique to simulate an experimental design by measuring
the difference in average outcome in the treatment group before and after treatment, minus the
8 difference in average outcome in the control group before and after treatment (Alhoudy [2014]).
The design will follow three cohorts of students in distinct treatment and comparison group
through time. Cohorts are determined by grade-level at initiation of the study, (either sixth,
seventh, or eighth grade), as the program is only for middle school students. Short-term
outcomes will be measured quarterly, while the cohort is in middle school; intermediate
outcomes will be measured yearly, while the cohort is in middle school; and long-term outcomes
will be measured at entry and exit of high school. The proposed evaluation will last 10 years in
its ideal implementation. We propose using mixed methods for data collection throughout the
process to capture quantitative changes in academic performance and qualitative differences in
aspirational outcomes that may change with the program.
This design is both feasible to implement for our client and will address the need for
short-term results to give to funders. Additionally, it is a balance of a more rigorous, but more
expensive method, and a feasible, but much weaker design. The difference-in-difference
approach reduces excess burden on students and education coordinators, while maintaining a
level of scientific rigor through the comparison group and multiple pre- and post-tests. We will
utilize a concurrent comparison group of students consisting of members of the BGCDC, but
who do not participate in College Club. Baseline data will be established through collecting
initial GPA and attendance figures, Stride Academy reading and math skills tests scores,
Qualtrics survey results, and by conducting qualitative interviews with students and parents.
The difference-in-difference design removes the bias that may occur due to changes in
time. This is particularly applicable when researching educational programs, as one assumes
maturation effects as students move through school. Additionally, through pre-post analysis
within each group and across study groups, this design eliminates the bias that may be due to
9 stable differences between those in the program and the comparison group. A related design
limitation is that it is necessary to assume that unobserved characteristics of the College Club
members and students in the comparison group are fixed over time, which may or may not hold
(Shager 2014).
Using quasi-experimental evaluation methods presents several threats to validity. Specific
to this evaluation plan, we are concerned about internal validity, statistical conclusion validity,
and external validity. With regard to internal validity, we are concerned that our comparison
group is unmatched to the treatment group. We will address this by selecting individuals for the
comparison group from the same geographical area, who attend the same schools, who
participate in BGCDC programs, and who are roughly matched on demographic characteristics
to the treatment group. We assume that if all the students are involved in some way in BGCDC
programs, we will minimize selection bias effects. Some selection bias may still exist with this
design due to the selection into the program. We also acknowledge that comparing changes in
achievement between these two groups will yield different results than an evaluation studying
the effects of College Club compared to students with similar characteristics who have no
affiliation with BGCDC. It is important to keep this distinction in mind when interpreting effect
sizes, because this is a more rigorous design, but comparing to students who already are seeking
extra-curricular services will help reduce validity issues. This may be a good area for further
research. We are also concerned with attrition of the comparison group, whether due to the
students leaving the BGCDC or because the student is accepted into the College Club program.
This further complicates the comparison group, but it is the best available solution. If funding
increases and more slots become available, it is likely some of our comparison group members
could be entered into the College Club program. This is most likely for the sixth grade cohort.
10 Due to this evaluation focusing on the program in the middle school years, this is only a minor
concern.
The small sample size presents a threat to statistical conclusion validity from low
statistical power. Initially, we planned to only evaluate College Club at the Allied Center;
however, as previously mentioned, to increase sample size we have chosen to include the
program at the Taft Center in our evaluation as well. Further increasing the sample size is not
feasible at this point, because that would include growing the College Club program. Our
intention with this evaluation is to serve the internal needs of BGCDC, providing a plan for data
collection and analysis to justify sustained and increased investment in the program. A power
calculation can determine the minimum program effect size that will be able to be detected given
the fixed sample. Further information can be found in the upcoming power analysis section.
The limited scope of the existing College Club program presents threats to the external
validity of the evaluation. We plan to assess only the club-based, after school portion of the
College Club program. Will these results be valid with respect to the middle-school and high
school based College Club programs? Or programs in other communities? To address this
concern, we will outline differences between the afterschool and club based programs, and
provide guidance for our clients on what conclusions will be appropriate to assume given the
data. It is also important to consider the conditions specific to Madison that have resulted in the
one of the widest educational achievement gaps and largest black/white arrest and incarceration
disparities in the country. The extent to which poverty and “disadvantage” has become racialized
in Dane County makes it distinct from other communities that may implement a similar program
(WCCF 2013).
11 Rationale
We considered several study designs and evaluation methods prior to selecting our
evaluation plan. First, we examined the feasibility of a rigorous, randomized experimental
method, as this design is the most effective in reducing biases and differences between treatment
and control groups, and in establishing causal links. This method proved not to be appropriate for
our client for several reasons. It is expensive, we have a small sample size, and some of the
students have been in the program for a couple of years, which would not only diminish effects
seen between groups, but ethically we do not want to randomize them out of the program.
Because there has not yet been an impact evaluation done on the College Club program, we plan
to start with a smaller trial to demonstrate potential effects, and a next step could be scaling up
the program and conducting a randomized controlled trial (RCT) across sites.
An additional method we considered was a simple one group, pre-test/post-test design.
While this evaluation method would be much more feasible for our client, it would only
demonstrate progress in students receiving College Club services over time and would not show
the impact of the College Club programming against the counterfactual, a comparison group who
did not receive services. It is important to note that the pre-test/post-test analysis of achievement
among College Club members is still possible within the framework of the proposed evaluation
plan. Standardized data collection tools and intervals (discussed in detail below) will give our
client the flexibility to analyze pre-post measures for internal or external needs.
Complementary evaluations
In additional to the impact evaluation, we recommend conducting an implementation
evaluation to answer the question: are College Club members efficiently utilizing services and
reaching program benchmarks? Efficiently utilizing services will mean maximizing the resources
12 available to College Club in relation to having the greatest impact on the participating students.
The designed evaluation plans to track program outputs and collect implementation data after
each semester while the students are in middle school to determine how they are utilizing
services.
Target population and sampling plan
As previously mentioned, our target population consists of sixth through eighth grade
students who participate in the College Club program at either the Taft or Allied BGCDC
Centers and those students of the same age who participate in other BGCDC programs. Many of
these students are low-income students, students of color and potential first-generation college
students. An important shared characteristic to take into consideration between the groups is that
both the College Club members and those in the comparison group are motivated enough or have
the family and social support to participate in BGCDC programming. While this will not allow
an evaluation of the effect of College Club versus no participation in any extracurricular
activities at the BGCDC, it isolates the unique added value of College Club for students.
Our sample size is relatively fixed. There are 25 members of each the Taft and Allied
College Clubs, giving a treatment population of 50. We will recruit 25 students to serve in the
comparison group at each Center, which will be convenience sampling, using students who are
on the waiting list for College Club and those who are participate in other BGCDC programs.
The overall study population will be approximately 100 individuals.
Power analysis
The statistical power in impact evaluations allows researchers to be able to detect
whether a program has had an effect on the target population (McKenzie 2011). Using the
Optimal Design Software, we can calculate the minimum effect size that will be able to be
13 detected for our population given the standard power threshold of 0.8 (Raudenbush et al. 2011).
For a single-level trial conducted at the individual level, given a sample size of 100 and power of
0.8, the minimal detectable effect size that would be statistically significant is 0.57. The literature
evaluating similar programs shows a wide variation in effect sizes, making it difficult to
accurately predict a potential effect size of College Club.
Due to the fixed nature of the sample for this evaluation, we are limited to using
statistical analysis techniques to improve power and lower the minimal detectable effect size.
Our collection and analysis of repeated post-tests and measures will allow for greater power as
well. Please see Appendix B for power calculation.
Data sources and collection procedures
Data to determine any potential impact of the program as well as the data for the
implementation analysis will be collected at varying points over the 10-year time horizon of the
proposed evaluation. As previously mentioned, baseline data will be established at the beginning
of year one through collecting initial grade point average (GPA) and attendance figures for both
College Club members and those in the comparison group. BGCDC has access to both of these
measures (please see Appendix C). Additional baseline data includes a preliminary Stride
Academy reading and math skills test that will be administered to all participants. This program
allows for stratification of data according to the student and the subject matter. The final baseline
measure will be level of family engagement and student’s educational aspirations, gathered
through Qualtrics survey results (please see Appendix D for preliminary survey tool), and by
conducting qualitative interviews with students and parents. All baseline measures will be
collected for both the treatment and control groups.
14 Implementation data will be collected quarterly for all program outputs relating to the
treatment groups while the students are in middle school. Standardized reporting of tutoring, case
management and mentoring using logs will allow for measurement of the utilization of services
among members, including number of meetings and meeting minutes per week. College
exposure, summer academic programming and field trips will be measured through attendance
logs, and time logged on Stride Academy by students will be captured by the software. School,
teacher, and family engagement will be tracked by the Education Coordinators in their case
management logs. Establishing standard procedures to record and document activities beginning
in year one will improve the reliability of the reporting tools and improve internal validity.
Short term expected outcomes will be measured at the end of each academic quarter
while the students are in middle school. Attendance will be measured by the school each student
attends, and access to this information will have been granted for all BGCDC participants. Stride
Academy scores will be measured for math and reading, and generated using the software.
Action plans to reduce academic barriers will be measured as a percentage of members with
action plans, and this will be tracked using the case management logs.
Increased academic performance will be measured using both core and cumulative GPA.
BGCDC obtains permission to access students’ grades through the participation agreement each
signs with their families. These will be collected and documented at the end of each year of
middle school. Grades and GPA are only one measure of success and there are challenges to
reliability of this measure, because students across the evaluation will attend different schools.
This information will be documented and controlled for by using dummy variables, whose use is
described in more detail in the following section. Educational aspirations as an intermediate
outcome will be assessed through a survey tool (please see Appendix E) administered by the
15 Education Coordinator. Because this tool will be used in repeated years, there is the potential for
students to adjust responses based on prior knowledge of the question or due to social
desirability bias to align their responses with the perceived most desirable answer.
The long-term outcome of improved college readiness will be measured using
components that are predictors of college eligibility. Components to be measured include the
American College Testing (ACT) test, a standardized college readiness assessment; overall grade
point average; and completion of the Free Application for Federal Student Aid (FAFSA), an
application to determine eligibility for student financial aid. Increased AVID/TOPS acceptance
will be determined using AVID/TOPS administrative data upon high school entry. Additionally,
increased educational aspirations will be measured through in-person interviews at high school
entry and exit. The final long-term outcome, to establish a college going culture at BGCDC, will
be a qualitative measure assessed by the Education Coordinator. Appendix F is the proposed data
collection table, providing a snapshot view of this information.
Data analysis procedures
The basic notation we will use for our data analysis of this difference-in-difference model
is as follows, where Y is the sample average of the indicated measure, subscript 0 is the pre-test,
subscript 1 is the post test, T is treatment and C is control group:
Pre
Post
Post-Pre Difference
Treatment
Y0T
Y1T
Y1T – Y0T
Control
Y0C
Y1C
Y1C – Y0C
T-C Difference
Y0T – Y0C
Y1T – Y1C
(Y1T – Y0T) – (Y1C – Y0C)
(Albouy [2014])
16 This formula allows for analysis first within the treatment or the comparison group over
the course of the program and then to assess the differences between the groups. Because there
will be different treatment durations between the sixth, seventh, and eighth grade cohorts, we
propose using “dummy” variables to detect potential differences in effect sizes across the grades.
Dummy variables are variables that take the values of only 0 or 1 and serve as numerical standins for qualitative or categorical values that may be expected to shift the outcome of analysis
(Chinn 2014). There are three categories of interest (sixth, seventh, or eighth grade at initiation
of evaluation) which means two dummy variables will need to be created and compared to the
“base case.” In this situation we propose using eighth grade as the base case and measuring
differences for sixth and seventh graders against their average outcome measures. The three
cases can be defined as follows:
Eighth grade (base case)
Seventh grade
Sixth grade
Dummy Variable One (D1)
0
1
0
Dummy Variable One (D2)
0
0
1
(Chinn 2014)
We propose creating two more variables to control for additional differences between
participants. First is to control for differences in the schools the students attend. Second is to
control for students who have previously participated in the College Club program. This method
will be used to measure the intermediate expected outcomes of improved core and cumulative
GPA.
17 III: Quality Control and Human Subjects Protection
This evaluation requires obtaining the test scores and grades of College Club participants
and of the BGCDC members who are not participants in College Club to accurately assess the
program. This requires consent to gather the data from the schools. The BGCDC gathers consent
from parents for accessing school grades upon a child’s entrance into the club. Since Stride
Academy is also implemented club wide, there is no need for additional consent to be gathered.
The form currently used is provided in Appendix C. This form obtains consent for access to
transcripts, attendance records, schedules, and achievement tests (WKCE and school specific).
We recommend adding a line to the consent form informing parents of the program and the need
for evaluation. Proposed language to add to the form is also included in Appendix C. As long as
this remains the standard practice, no further information is required to be gathered.
The results from this study are intended for local use, staying within the program. If the
intentions of the evaluation shift toward a broader academic focus with the goal of being
published, the Institutional Review Board for the governing body of the program evaluators
should be consulted to identify further consent requirements.
IV: Evaluation Implementation
Timeline
Implementing the program evaluation of College Club requires two timelines to follow.
The first is a short-term, one-year timeline, and the second is a long-term, multi-year timeline.
Examples of both can be found in Appendix G. The short term timeline starts in August, prior to
the beginning of the school year and ends in June, after the completion of the school year.
August is the proposed start of preparatory work for the evaluation with the College Club staff,
18 ensuring understanding of how the evaluation will work and the data that needs to be collected
and reported. After Labor Day, when the school year begins, the members of the BGCDC are
placed into their respective groups (College Club or not), and initial baseline testing is
conducted. From this point until the end of the semester in January, the Club runs as normal.
During this time, data is collected in case management logs, tutoring logs, Stride Academy
reports, mentoring notes and attendance logs. At the end of the semester all participants take the
mid-year tests. The data collected so far is reviewed for completeness, not analysis. The second
semester mirrors the first, with final tests occurring near the end of the school year in June.
Surveys are also administered at this point. Next, the school year data is compiled and analyzed
for results and recommendations. Summer programs begin, and the relevant data is collected
throughout the summer.
The long-term timeline provides the framework to follow College Club and BGCDC
members into high school to see who is accepted and completes a college preparatory program.
High school graduation, college acceptance and college completion data is also collected. High
school records will provide high school graduation information, while college acceptance and
completion will be collected through survey responses. This information will be collected twice a
year, at the start and end of the academic calendar. It will take four years to see the first College
Club participants be eligible for acceptance into college. We recommend following the members
for at least 10 years. This will allow sufficient time for three cohorts of members and students to
have completed a two or four year degree. The data will be gathered through follow-ups
conducted via email by the program.
19 Implementation Factors
Implementation of the evaluation will be hindered by a few factors. At this time there is a
spending freeze for the BGCDC. Without available funds it will be difficult to resource the
evaluation. The Education Director position is currently vacant as well. The Director is the
logical person to oversee the evaluation since it involves two locations. Conducting the
evaluation without the Director is not impossible, but will pose additional challenges. The final
implementation challenge is the school year. This evaluation is designed to begin with a new
school year to most accurately capture results. During the waiting period to begin, materials and
data collecting systems should be prepared to ensure a smoother implementation.
Proposed Budget
The Kellogg Foundation recommends designating 5 to 7 percent of a program’s total
budget to conduct an evaluation (Curnan et al. 2004). The evaluation of College Club will cost
time and money. Fiscal amounts will attempt to be minimized because of the current fiscal
situation. Fiscal costs will be discussed first, followed by time costs.
The majority of fiscal costs for this evaluation are already accounted for in the BGCDC
and College Club budgets. For example, this evaluation relies heavily on access to Stride
Academy software, currently purchased by the BGCDC. The budget should reflect this cost to
ensure it, or comparable software, remains available. The surveys completed by participants,
teachers, staff, and family members can be accessed via computers at the clubs. A few can be
printed out and sent home with members if needed. We assume the minimal printing costs can be
covered by the BGCDC printing budget. Obtaining the long-term data from former participants
in the evaluation can also be accomplished by email. This saves on costs and also will likely be
an easier way to maintain contact as people move.
20 Two fiscal costs should be discussed by the program organizers prior to starting the
evaluation. The first is for an outside analyst to compile the data and provide results. It is
reasonable to hire an analyst for 80 hours a year to review the data and analyze for results. This
gives the analyst time to review the data for completeness after each semester, analyze the data
and answer questions. Using an hourly wage between $27 and $36 this total is between $2,160
and $2,880. This is the amount for the first 3 years when there is the largest quantity of data. The
last 7 years of the proposed evaluation, we estimate the need for an analyst to be 20 hours for one
week each year, for a range between $540 and $720 a year.
The other fiscal cost is any incentives for participating. We recommend having pizza
twice a year to help coordinate participation in baseline and follow-up Stride Academy testing
for all students in the evaluation.. This cost accrues for the first 3 years. To feed 100 people
pizza, an estimated 25 pizzas should be sufficient. At $14 a pizza, the total is $350. A yearly
pizza incentive budget is $700.
The majority of the costs to implement this evaluation are in time. These costs are
incurred by the participants taking tests, tutors filing reports, education coordinators filing logs
and conducting interviews, and mentors and teachers providing feedback to the program. Time
estimates are difficult to measure. The activities listed above are already conducted by College
Club; the evaluation will only increase requirements minimally at the beginning and end of the
school year. People who actively assist College Club can expect less than a 10% increase in their
work load to successfully implement this evaluation. Therefore, if one volunteers 10 hours a
week, s/he can expect to volunteer 11 to ensure proper implementation. The largest time effect
falls on whomever is coordinating the evaluation; this increase in time will vary by week, but is
21 sizable and should be taken into account when identifying the evaluation coordinator. A brief
overview of the budget is in Appendix H.
V: Anticipated Results
Successful implementation of this program evaluation will likely allow a few key lessons
to be learned by College Club. Based on relevant literature, we expect the program to have a
positive effect on the achievement of its participants. However, if no effect is shown, there are a
few possible explanations. A likely explanation is that the program services are being underutilized. The implementation portion of the evaluation will help identify areas to focus efforts
and resources to improve program adherence to obtain expected results. Viewing emerging best
practices by similar programs might help generate ways to improve implementation shortcoming
identified in this evaluation.
If a positive effect emerges, it allows for key areas of organizational learning. The first
three are focused on the participants and the effects of the program, while the fourth is focused
on the program itself. First, this will allow clients to say participation in College Club is
associated with an increase in academic performance by X% across various measures. These
measures are GPA, Stride Academy scores, state test scores, and attendance ratios. Secondly,
participation in College Club is associated with being X times more likely to enter into a college
preparatory program in high school. After four years, this can be furthered to show an
association between College Club participants and college attendance compared to nonparticipants. The third result to be learned from the evaluation is the effect on college readiness.
In particular, College Club will be able to say their participants have X% of predictors of college
eligibility compared to non-participants. Finally, this evaluation will allow the administrators of
22 College Club to identify areas for improvement in terms of program adherence. This information
allows the program to continue to develop areas that need greater attention in the program, adjust
resources to maximize positive results, and identify additional areas for improvement.
Remaining Questions
A few questions will remain after this evaluation is complete. The most prominent
question remaining is if participation in the program is associated with increased high school
graduation and college graduation rates compared to non-participants. For example, just because
a student begins a college preparatory program and is eligible for college does not mean s/he will
obtain a degree from a higher education institute. To answer this question, a longer, more
rigorous evaluation will need to take place. The evaluation presented here is the first step.
Other questions will likely arise from the results of the evaluation as well. One potential
question will be which component of the program is most effective? This evaluation will be
unable to answer this question, but descriptive information can help identify which component is
being utilized the most. The design of the proposed evaluation is quasi-experimental and not a
randomized control trial; therefore, the results will be unable to show causal relationships. The
descriptive data can be used to help infer associations, but nothing more. A more rigorous
randomized controlled trial would be required to more accurately determine which components
of the program are having an effect.
Stakeholders
The results of this evaluation will need to be presented to the stakeholders of the
organization. First, we would recommend a formal presentation to the BGCDC Board and the
College Club Program Director. These stakeholders will be able to make decisions for the
program based on the results and should have the opportunity for a detailed presentation with
23 discussion. The second group of stakeholders include all of the families of the BGCDC and key
personnel from the community and schools. The results of the study should be summarized into
one or two pages for this group and disseminated via email to reduce costs. A point of contact
should be provided for additional information. A town-hall style presentation on the program and
the results of the evaluation could be beneficial to improve awareness and participation. The
final group of stakeholders is the funders. And results should be used in future proposals and
updates. Most of the funders receive at least quarterly updates; this is the ideal opportunity to
highlight pertinent results from the study.
Based on the duration of this evaluation, we recommend disseminating information in
three phases for the first two groups of stakeholders. After the first year of the evaluation is
complete, the descriptive data and initial results from the analysis should be disseminated. After
all three cohorts are out of middle school, the second round of disseminating information takes
place. The final results should be disseminated at the end of the evaluation. Due to the nature of
the third group of stakeholders, the funders, results should be disseminated to them based on the
timing of funding requirements.
24 Appendix A: Logic Model
Program: College Club, Boys & Girls Club of Dane County, Logic Model
Inputs
Outputs
Activities
Tutors (certified); materials
for instruction; classroom
environment;
1. Tutoring; Education
Coordinator, Tutors,
members
Case Managers (certified);
meeting space; information
re actions of student)
2. Case Management;
Education Coordinator,
members
Schools supplies; financial
support; food support;
3. Address/Eliminate
Barriers to Academic
Success; Education
Coordinator, members
Transportation; location to
visit; adult supervision;
food support;
4. College Exposure;
Mentors (volunteers);
meeting space; specific
need materials
5. Mentoring; Education
Coordinator Mentors,
members,
BGCDC Visits; Teacher
and Principal time
6. School and Teacher
Engagement; Education
Coordinator, members,
Teachers, Principals,
Counselors
Family time, resources for
conferences and events
Transportation; supplies
Education Coordinator,
members
7. Family Engagement;
Education Coordinator,
families, members
8. Summer Academic
Programming;
Education Coordinator,
members, library staff
Outputs
Short
Meetings, Stride Academy
sessions and tests
Meetings, Individual
Development Plan
Interviews, Action Plans

Long
A. Increased Stride
Academy scores
A. Increased cumulative
GPA
A. Improved college
readiness
B. Improved attendance
B. Increased core GPA
B. Increased AVID/TOPS
Acceptance
C. Development of
action plans to reduce
barriers
C. Increased educational
aspirations
D. Improved behavior at
school
D. Improved academic
performance
C. Established college
going culture at
BGCDC
D. Increased educational
aspirations
Field trips, College visits
Meetings
Individual Development
Plan
Family Conferences,
recognition events
Field trips, library visits,
reading lists
Assumptions



Outcomes -- Impact
Medium
External Factors
Parental involvement; SES situation; BGCDC program funding, Geographical area
for access to resources
BGCDC has the financial and human resources to implement program activities
Once activities are provided, students in the target group will participate
Once accessed, the activities will lead to improvements in GPA
Improved middle school performance will increase likelihood of participation in
high school college prep programs
Rev. 12/15//2014
25 Comment [IH1]: Do we need to add a Part E for Family Engagement (append ix D findings) Appendix B: Power Calculation
26 Appendix C: BGCDC Contract
PARENT AUTHORIZATION AND RELEASE OF INFORMATION FORM
I provide consent to allow The Boys & Girls Club of Dane County staff electronic access
to the following information about my child using the Madison Metropolitan School
District information systems and/or the Verona Area School District’s data:
Academic progress reports, report cards, unofficial school transcripts,
assignments, teacher comments, teachers’ names, dates of courses taken,
student course schedules, daily attendance, absences, tardy data, test scores
(including Wisconsin state achievement tests and MMSD achievement tests) and
fee payments due and paid.
In addition, to access to my child’s records electronically I authorize MMSD and/or The
Verona Area School District to provide The Boys & Girls Club of Dane County with other
demographic and program service eligibility information for my child. This information
includes Limited English Proficiency status, primary language (other than English),
disability status and primary disability and qualification for federal free and reduced
lunch program.
I understand that this information will remain strictly confidential and will only be used
for the improvement of educational services and resources rendered to my child. The
Boys & Girls Club of Dane County will not further disclose the data to any third party,
researcher or others without obtaining a separate written permission from you.
I understand that this release of information agreement and participation agreement will
remain in effect until my child completes high school, until my child resigns from the
program or until I revoke this consent in writing.
27 PARENT SIGNATURE
DATE
(PRINT NAME)
TELEPHONE
Proposed Feedback on Contract: We recommend the BGCDC adds the following to the contract used by all BGCDC members: ‐
The Boys & Girls Club of Dane County is currently running an evaluation of one of its programs, College Club. The purpose of this evaluation is to better understand how the Club can best support the academic learning of our participants, to identify areas where the Club can improve our role, and to provide further justification/evidence of the positive effects of this program. Your child’s test data will be used in this evaluation, but no personal identifying information will connect scores to your child. 28 Appendix D: Parent/Guardian Survey
The purpose of the survey on family engagement and student educational aspirations is to identify how
involved the participants’ family members are in their learning and how family members view the
potential education attainment level of the student. The following survey questions will help obtain
answers to these questions. The National Center for Educational Statistics (www.nces.ed.gov) is an
available resource to help identify lines of questioning used by other surveys to answer similar questions.
One study on the NCES website is the Education Longitudinal Study of 2002. This study used a specific
survey for parents of 10th graders. Questions 52 through 60 and 69 through 83 are appropriate questions
to include on a survey for this College Club evaluation. The questions below are school-specific;
however, they can be tailored to reflect opinions regarding BGCDC Centers. The complete survey can be
found at the following link:
http://nces.ed.gov/surveys/els2002/pdf/ParentQ_baseyear.pdf
Q1 Since your student's school has opened last fall, how many times have you or your spouse/partner been contacted by the school about the following? None (1) Once or twice (2) Three or four times (3) More than four times (4) Your student's poor performance in school (1) 



Your student's school program for this year (2) 



Your student's plans after leaving high school (3) 



Your student's course selection for entry into college, vocational, or technical school after completing high school (4) 



Your student's poor attendance record at school (5) 



Your student's problem behavior in school (6) 



Your student's positive or good behavior in school (7) 



Participating in school fund‐raising activities or doing volunteer work such as supervising lunch or chaperoning a field trip (8) 



Information on how to help your student at home with specific skills or homework (9) 



Obtaining information for school records such as your address or work telephone number (10) 



29 Q3 Since your student's school has opened last fall, how many times have you or your spouse/partner contacted the school about the following? None (1) Once or twice (2) Three or four times (3) More than four times (4) Your student's poor performance in school (1) 



Your student's school program for this year (2) 



Your student's plans after leaving high school (3) 



Your student's course selection for entry into college, vocational, or technical school after completing high school (4) 



Your student's poor attendance record at school (5) 



Your student's problem behavior in school (6) 



Your student's positive or good behavior in school (7) 



Participating in school fund‐raising activities or doing volunteer work such as supervising lunch or chaperoning a field trip (8) 



Information on how to help your student at home with specific skills or homework (9) 



Obtaining information for school records such as your address or work telephone number (10) 



Q2 In this school year, do you or your spouse/partner do any of the following? Yes (1) No (2) Belong to the school's parent‐teacher organization (1) 

Attend meetings of the parent‐teacher organization (2) 

Take part in the activities of the parent‐
teacher organization (3) 

Act as a volunteer at the school (4) 

Belong to any other organization with several parents from your student's school (5) 

30 Q4 How often do you... Never (1) Seldom (2) Usually (3) Always (4) check that your student has completed all of their homework (1) 



discuss your student's report card with her/him (2) 



know where your student is when s/he is not at home or in school (3) 



make and enforce curfews for your student on school nights (4) 



31 Q5 In the first semester or term of this school year, how often have you and/or your spouse/partner provided advice or information about the following to your student? Never (1) Sometimes (2) Often (3) Selecting courses or programs at school (1) 


Plans and preparation for college entrance exams such as ACT, SAT, or ASVAB (2) 


Applying to college or other schools after high school (3) 


Specific jobs your student might apply for after high school (4) 


Community, national and world events (5) 


Things that are troubling your student (6) 


Q6 Looking back over the past year, how frequently did you and your student participate in the following activities together? Never (1) Rarely (2) Sometimes (3) Often (4) Attending school activities (1) 



Working on homework or school projects (2) 



Attending concerts, plays, or movies outside of school (3) 



Attending sporting events outside of school (4) 



Attending religious services (5) 



Attending family social functions (6) 



Taking day trips or vacations (7) 



Working on a hobby or playing sports (8) 



Going shopping (9) 



Going to restaurants/eating out (10) 



Spending time just talking together (11) 



Doing something else fun together (12) 



32 Q7 How much do you agree or disagree with the following statements? Strongly Agree (1) Agree (2) Neither Agree nor Disagree (3) Disagree (4) Strongly Disagree (5) Most people can learn to be good at math (1) 




You have to be born with the ability to be good at math (2) 




Q8 Looking back over the past year, how many times did the following occur? The parent of one of my student's friends... None (1) Once or twice (2) Three or four times (3) More than four times (4) gave me advice about teachers and/or courses at my student's school (1) 



did me a favor (2) 



received a favor from me (3) 



supervised my student on an educational outing or field trip (4) 



Q9 Are there family rules that are enforced for your student about any of the following activities? Yes (1) No (2) Maintaining a certain grade average (1) 

Doing homework (2) 

Doing household chores (3) 

Watching television (4) 
Q11 In a typical week how many days do you eat at least one meal with your student? 







0 days per week (1) 1 day a week (2) 2 days a week (3) 3 days a week (4) 4 days a week (5) 5 days a week (6) 6 days a week (7) 7 days a week (8) Q12 Do you have a computer in your home that your student may use? 

Yes (1) No (2) Q13 Does this computer have access to the internet? 

Yes (1) No (2) 33 
Q14 Do you or your spouse/partner use a computer to communicate with or get information about your student's school? 


Yes (1) No access to a computer (2) Don't use a computer for these purposes (3) Q15 how often do you or your spouse/partner use a computer in the following ways? Never (1) Once or twice a year (2) Several times a year (3) At least once a month (4) Weekly (5) To communicate with your student's teachers and administrative staff via e‐mail about your student (1) 




To find out what events and activities are happening at your student's school (2) 




To express concern to the school principal or teachers over school practices and policies (3) 




To let school staff, counselors, teachers or the principal know which courses you would like your student to take (4) 




To find out about homework and other school assignments and projects (5) 




Q16 Does your student's school have a voice‐messaging/e‐mail system that you can use for information about school events, activities, and programs, or leave messages for your student's teachers, school administrator or other staff? 


Yes (1) No (2) Don't know (3) 34 Q17 During the school year, how often do you use the school's messaging system? 




Never (1) Once or twice a year (2) Several times a year (3) At least once a month (4) At least once a week (5) Q18 How much do you agree or disagree with each of the following statements concerning your student's school? Strongly agree (1) Agree (2) Neither Agree nor Disagree (3) Disagree (4) Strongly Disagree (5) The school assigns too little homework (1) 




My student is challenged at school (2) 




My student is working hard at school (3) 




The school is preparing students well for jobs in the workplace (4) 




The school is preparing students well for college (5) 




The school is a safe place (6) 




Parents have an adequate say in setting school policy (7) 




Parents work together in supporting school policy (8) 




My student's teachers are well trained (9) 




Drinking on school grounds is a problem at my student's school (10) 




Drug use on school grounds is a problem at my student's school (11) 




The sale or use of drugs on the way to or from school is a problem (12) 




Theft on school grounds is a problem at my student's school (13) 




Violence on school grounds is a 




35 problem at my student's school (14) The lack of discipline in classrooms is a problem at my student's school (15) 


Q19 How satisfied are you with the education your student has received up to now? 



Very Dissatisfied (1) Somewhat Dissatisfied (2) Somewhat Satisfied (3) Very Satisfied (4) Q20 How far in school do you want your student to go? Please mark only the highest level that applies. 






Less than high school graduation (1) High school graduation or GED only (2) Attend or complete a 2‐year school course in a community of vocational school (3) Attend college, but not complete a 4‐year degree (4) Graduate from college (5) Obtain a Master's degree or equivalent (6) Obtain a Ph.D., M.D., or other advanced degree (7) 36 

Q21 How important is each of the following to you in your student's choice of a school to attend after high school? Very important (1) Somewhat important (2) Not important (3) Low expenses (tuition, books, room and board) (1) 


Availability of financial aid, such as a school loan, scholarship or grant (2) 


Availability of specific courses or curriculum (3) 


Strong reputation of the school's athletic programs (4) 


Active social life at the school (5) 


Ability to attend school while living at home (6) 


Chance to live away from home (7) 


A religious environment (8) 


A low crime environment (9) 


A good record for placing graduates in jobs (10) 


A good record for placing graduates in graduate school (11) 


Strong reputation of the school's academic programs (12) 


Easy admission standards (13) 


Racial/ethnic composition of the school (14) 


Size of the school (15) 


Q23 How far in school do you think your student WILL go? Please mark only the highest level that applies. 






Less than high school graduation (1) High school graduation or GED only (2) Attend or complete a 2‐year school course in a community of vocational school (3) Attend college, but not complete a 4‐year degree (4) Graduate from college (5) Obtain a Master's degree or equivalent (6) Obtain a Ph.D., M.D., or other advanced degree (7) Q22 Have you or your spouse/partner done anything specific in order to have some money for your student's education after high school? 

Yes (1) No (2) Appendix E: Student Survey
37 The purpose of the survey on educational aspirations is to identify what level of educational attainment
the student believes s/he can achieve. The following survey questions will help obtain answers to these
questions. The National Center for Educational Statistics (www.nces.ed.gov) is an available resource to
help identify lines of questioning used by other surveys to answer similar questions. One study on the
NCES website is the Education Longitudinal Study of 2002. This study surveyed 10th graders on a
variety of topics, to include what level of education they believe they could obtain. The questions in Part
III (questions 54 to 66) are relevant for this College Club Evaluation. The questions below are schoolspecific; however, they can be tailored to reflect opinions regarding BGCDC Centers. The complete
survey can be found at the following link:
http://nces.ed.gov/surveys/els2002/pdf/StudentQ_baseyear.pdf
Student Educational Aspirations Q1 How important is each of the following to you in your life? Not Important (1) Somewhat Important (2) Very Important (3) Being successful in my line of work (1) 


Finding the right person to marry and having a happy family life (2) 


Having strong friendships (3) 


Being able to find steady work (4) 


Helping other people in my community (5) 


Being able to give my children better opportunities than I've had (6) 


Living close to parents and relatives (7) 


Getting away from this area of the country (8) 


Working to correct social and economic inequalities (9) 


Having children (10) 


Having leisure time to enjoy my own interests (11) 


Becoming an expert in my field of work (12) 


Getting a good education (13) 


38 Q2 Have you taken or are you planning to take any of the following in the next two years? I haven't thought about it (1) No, I don't plan to (2) Yes, this school year (3) Yes, next school year (4) Yes, in 12th grade (5) Pre‐SAT test (PSAT) or Preliminary American College Testing Test (PACT) (1) 




College Board Scholastic Assessment Test (SAT) or American College Testing Service (ACT) (2) 




Advanced Placement (AP) test (3) 




Armed Services Vocational Aptitude Battery (ASVAB) (4) 




Q3 As things stand now, how far in school do you think you will get? 







Less than high school graduation (1) High school graduation or GED only (2) Attend or complete a 2‐year school course in a community or vocational school (3) Attend college, but not complete a 4‐year degree (4) Graduate from college (5) Obtain a Master's degree or equivalent (6) Obtain a Ph.D., M.D., or other advanced degree (7) Don't know (8) Q4 Do you plan to continue your education right after high school or at some time in the future? 





Yes, right after high school (1) Yes, after staying out of school for one year (2) Yes, after staying out of school for over a year (3) Yes, but I don't know when (4) No, I don't plan to continue my education after high school (5) I don't know if I will continue my education after high school (6) Q5 Which of the following do you plan to attend? 


Four‐year college or university (1) Two‐year community college (2) Vocational, technical or trade school (3) 39 Q6 Where have you gone for information about the entrance requirements of various colleges? 










Guidance counselor (1) Teacher (2) Coach (3) Parent (4) Friend (5) Sibling (6) Other relative (7) College publications or websites (8) College representatives (9) College search guides, publications, websites (10) None of the above (11) Q7 Would you like to participate in athletics (not intramurals) at the collegiate level? 

Yes (1) No (2) Q8 Do you hope to receive an athletic scholarship to pay for all or part of your college expenses? 

Yes (1) No (2) Q9 Which of the following are reasons why you have decided NOT to continue your education past high school? Yes (1) 

My grades are not high enough (2) 

I will not need more education for the career I want (3) 

I cannot afford to go on to school (4) 
I'd rather work and make money than go to school (5) 


I plan to be a full‐time homemaker (6) 

I do not feel that going to school is important (7) 

I need to help support my family (8) 

Q10 Write in the name of the job or occupation that you expect or plan to have right after high school 


Click to write Choice 1 (1) ____________________ I don't plan to work right after high school (2) I don't know (3) 40 No (2) I do not like school (1) Q11 Write in the name of the job or occupation that you expect or plan to have at age 30. 


Click to write Choice 1 (1) ____________________ I don't plan to work when I'm 30 (2) I don't know (3) Q12 How far in school do you think your Mother or Father or Guardian want you to go? Mother (1) Father (2) Guardian (3) Less than high school graduation (1) 


High school graduation or GED only (2) 


Attend or complete a 2‐year school course in a community or vocational school (3) 


Attend college, but no complete a 4‐year degree (4) 


Graduate from college (5) 


Obtain a Master's degree or equivalent (6) 


Obtain a Ph.D., M.D., or other advanced degree (7) 


Don't know (8) 


Does not apply (9) 


41 Q13 What do the following people think is the most important thing for you to do right after high school? Does not apply (1) Go to college (2) Get a full‐time job (3) Enter a trade school or an apprenticeship (4) Enter military service (5) Get married (6) They think I should do what I want (7) They don't care (8) I don't know (9) Your mother (1) 








Your father (2) 








Your friends (3) 








A close relative (4) 








School counselor (5) 








Your favorite teacher (6) 








Coach (7) 




42 



Appendix F: Data Collection Table
Output/
Outcome Measure/
Indicator
Tutoring sessions Number of minutes spent in one‐on‐one tutoring Stride Academy Number of minutes sessions spent on online customized lessons Addressing barriers to Number of barriers academic identified achievement College exposure Field Trips Mentoring sessions Case management Participation in college site visits Participation in off campus field trips Minutes spent one‐
on‐one with mentor Complete individual development plan Measurement
Data Source
Interval Outputs Target/
Benchmark* Person
Responsible Tutoring logs Weekly Tutors Online reports from Stride Academy Weekly 60 minutes per week Education Coordinator Interviews with students, families and teachers. Case management logs Site visit attendance log Field trip attendance log Mentoring logs
Monthly Education Coordinator End of each semester
Monthly At least one visit each semester 1 – 2 trips per month Weekly
60 minutes per week
Mentors
Yearly Education Coordinator Case management logs Short-Term Outcomes Increased school attendance Addressed barriers to academic achievement Increased study time Days absent from school Percentage of identified barriers with action plan Change in minutes using Stride Academy per week, additional reported study time School attendance records Case management logs Quarterly
Quarterly Stride academy logs, student surveys Quarterly 43 Improved behavior at Number of school suspensions, expulsions, detentions Reduced barriers to academic achievement Increased educational Member survey aspirations Improved math and Increase evaluation reading proficiency scores from Stride Academy, Percentage of students above proficient levels on WKCE test, Core GPA Intermediate Outcomes School records
End of each semester
Case management logs, teacher and family feedback Mentoring and case management logs Stride Academy logs,
WKCE test scores, Core GPA End of each semester Yearly AVID/TOPS records Entry into high school Follow‐up interviews End of high school School records, Follow‐up interview End of high school
Stride academy: End of each semester WKCE scores: end of 8th grade Core GPA: Yearly Long-Term Outcomes Increased AVID/TOPS acceptance Percentage of students accepted into AVID/TOPS Improved college Number of students readiness who take ACT/SAT, fill out FAFSA, and apply to college Increased educational Number of advanced aspirations high school classes (AP or Honors) 44 Appendix G: Timeline
Short Term College Club Evaluation
Timeline
College Club continues. Data Collected on: Prep materials, discuss eval with staff August Case Management, Stride Logs, mentoring logs, attendance logs. September October College club begins. Initial testing conducted for both college club participants (tx) and BG Club members not in the program (comp). November Second semester testing is conducted. December
January
February
End of semester tests conducted. Review first semester data for completeness, not analysis of outcomes. Look at implementation data 45 End of Semester testing is conducted March
April
College Club continues. Data Collected on: Case Management, Stride Logs, mentoring logs, May
June School year data is analyzed for results July
Summer programs begin, data collected
Summer programs ends, data analyzed Long Range College Club Evaluation
Timeline
3 cohorts in program 2015 st
1 cohort in program 2016 2 cohorts in program 2017 rd
1 cohort graduates HS 2018 2019
3 cohort graduates HS 2020
2021
2nd
cohorts graduates HS 46 2nd cohort graduate from a 4 year college 2022
2023
2024
1st cohort graduate from a 4 year college 2025
A decade of effects can be analyzed 2026
3rd cohort graduate from a 4 year college Appendix H: College Club Evaluation Budget
Year
Year 1
Year 2
Year 3
Year 4
Year 5
Year 6
Year 7
Year 8
Year 9
Year 10
10 Year Total
Professional Analyst
@ $27-$36 an hour
(# hours)
$2160-$2880 (80)
$2160-$2880 (80)
$2160-$2880 (80)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$10,260 - $13,680
Incentive Budget @
$14 Pizzas
(# pizzas)
$700 (50)
$700 (50)
$700 (50)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
$2,100
47 Total
$2,860 – $3,580
$2,860 – $3,580
$2,860 – $3,580
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$540 - $720 (20)
$12,360 - $15,780
References:
Afterschool Alliance. 2006. Evaluations backgrounders: A Summary of formal evaluations of
the academic impact of afterschool programs. Washington, DC: Afterschool Alliance.
Albouy, David. [2014]. Program Evaluation and the Difference in Difference Estimator.
Econometrics Laboratory. University of California-Berkeley. Accessed December 11.
http://eml.berkeley.edu/~webfac/saez/e131_s04/diff.pdf.
Boys & Girls Club of Dane County (BGCDC). 2014. “College Club.”
http://www.bgcdc.org/what-we-do/college-club/.
Britsch, Brenda, Nicky Martin, Amy Stucynski, Bethany Tomala and Patti Tucci. 2005. Literacy
in After-School Programs: A Literature Review. Portland: Northwest Regional
Educational Laboratory.
Chinn, Menzie. 2014. “Lecture 7: Dummy Variables.” Advanced Statistical Methods for Public
Policy Analysis. Public Affairs 819. Madison: University of Wisconsin.
Cosden, Merith, Gale Morrison, Ann Leslie Albanese, and Sandra Macias. 2001. When
homework is not home work: After-school programs for home assistance. Educational
Psychologist 36(3): 211 - 221.
Curnan, S., L. LaCava, D. Sharpstee, M. Lelle, and M. Reece. 2004. “WK Kellogg Foundation
Evaluation Handbook.” https://www.wkkf.org/resource-directory/resource/2010/w-kkellogg-foundation-evaluation-handbook.pdf. 54-57.
Granger, R.C. 2008. After-school programs and academics: Implications for policy, practice and
research. Social Policy Report 22: 3-19.
Harvill, Eleanor, Rebecca Maynard, Hoa Nguyen, Claire Robertson-Kraft, and Namrata
Tognatta. 2012. “Effects of college access programs on college readiness and
enrollment: A meta-analysis.” Washington, DC: Institute of Education Sciences (IES),
US Department of Education (US ED).
Hollister, R. 2003. The Growth in After-school Programs and their Impact. Washington, DC:
The Brookings Institute.
McKenzie, David. 2011. “Power Calculations 101: Dealing with Incomplete Take-up.”
Development Impact: News, views, methods, and insights from the world of impact
evaluation. http://blogs.worldbank.org/impactevaluations/power-calculations-101dealing-with-incomplete-take-up.
Power Hour: Making Minutes Count. 2008. After-School Program Outcomes: A Review of the
Literature and Evaluation Data. Branch Associates, Inc.
48 http://www.bgca.net/programs/eduCareer/Documents/NEWPower_Hour_microsite/Power_Hour_Literature_Review_FINAL_12-02-08.pdf.
Raudenbush, S.W., J. Spybrook, R. Congdon, X. Liu, A. Martinez, H. Bloom, and C. Hill. 2011.
Optimal Design Software for Multi-level and Longitudinal Research (Version 3.0)
[Software]. Available from www.wtgrantfoundation.org.
Shager, Hilary. 2014. “Quasi-Experimental Design Comparison Grid,” Handout from PA 871:
Public Program Evaluation. University of Wisconsin–Madison.
Wisconsin Center for the Advancement of Postsecondary Education (WISCAPE). 2014.
AVID/TOPS: Assessing the Outcomes of a Private-Public Pre-College Program.
http://www.wiscape.wisc.edu/wiscape/projects/ongoing-projects/rp043.
Wisconsin Council on Children and Families (WCCF). 2013. Race to Equity: A Baseline Report
on the State of Racial Disparities in Dane County. http://racetoequity.net/dev/wpcontent/uploads/WCCF-R2E-Report.pdf.
Yancsurak, Lonnie S. 2013. "Effectiveness of a computer-based afterschool intervention to
increase reading comprehension." Walden University.
49