18 public school districts in the state of Texas were involved in
Service Learning Projects. The State of Texas Service Learning
Center hired Dr. Carol Albrecht and her students to evaluate this program. These power point slides outline the steps they took to complete this evaluation.
We met with them to identify their objectives. They wanted to know how the program impacted public school children, teachers, community partners and parents of students.

One
• Identify the Objectives, Conceptualize Variables and
Develop Indicators.
Two
• Select the Sample.
Three
• Construct the Indicators.
Four
•Select the Research Design.
•Collect and Analyze the Data.
Five
•Write Report and Present Findings .
Six

Click here to see the surveys and codebooks.

Click here to see the surveys and codebooks.

An Old Chinese Proverb States:
I HEAR, I FORGET
I SEE, I REMEMBER
I DO, I UNDERSTAND
By “doing” this project we learned and really understood some important components of valid and reliable evaluation research.

We learned to
We learned to
We learned to
We learned to
Carefully
Consider the
Impact of
Timing
Select the
“right”
Indicators
Deal with
Social
Desirability
Use Multi-
Methods and
Multiple Units of Analysis

Carefully
Consider the
Impact of
Timing
Selecting the
“right”
Indicators
Dealing with
Social
Desirability
Using Multi-
Methods and
Multiple Units of Analysis


Timing of the Pre-test
 Many programs are ongoing, and this can have a major impact on the pre-test.
 In our study, many of the students had already participated in a Service Learning activity at some point in their school years.
So, we didn’t have a true “pre” test. The
“pre” test scores were contiminated by prior participation.


Timing of the Post-Test
 The “May” Effect
 Outside environmental/social factors need to be considered.
 In our study, we discovered that both teachers and students were more negative about almost EVERYTHING related to school at the end of the year. This may be explained by two factors:
First, they were just TIRED of school, and looking forward to vacation.
Second, they had just taken standardized tests,
TAKS. These tests were stressful for both teachers and students.

Carefully
Considering the Impact of
Timing
Selecting the
“right”
Indicators
Dealing with
Social
Desirability
Using Multi-
Methods and
Multiple Units of Analysis


Head Start Program
 In the 1960’s the head start program was launched. The objective was to increase the IQ scores of underrepresented populations including children living in poverty.
Early research showed that standardized tests of IQ increased for several years, and the decreased, until there was no difference between the experimental and control groups. While some felt this was evidence for discontinuing the program, parents came forward arguing that the researchers weren’t using the right measurements.


Head Start Program

A group of researchers call The
Perry Preschool Consortium, with
The input of teachers and parents, identified
(1) social, (2) educational and (3) socioeconomic indicators that differentiated preschool participants from a control group up to 19 years after participation in the program. The differences were compelling.


Furthermore, this group argued that the decreasing IQ scores actually provided evidence that environmental factors CAN influence IQ – both positively and negatively. Thus being
In an “enriched” environment (i.e., the Head
Start Program) can increase IQ but then being transferred to an impoverished environment
(i.e., public schools in poor neighborhoods) can decrease IQ.

Schooling Success
High School Graduation or Equivalent
College or Vocational Training
Functional Competence
Ever Classified as Mentally Retarded
Time Spent in Special Education
Social Responsibility
Ever Detained or Arrested
Teen Pregnancies
Employed
Receiving Welfare


Using focus groups and intensive interviews, we looked to
(1) Teachers,
(2) Parents
(3) And student participants as well as past research, to help us identify valid and accurate indicators.
Analysis of this qualitative data indicated
(1) Some students did experience the desired impact.
(2) We needed to use “control” variables to accurately assess the impact of Service Learning.

The following control variables were all highly significantly related to EVERY outcome measurement.
Ownership of Project
• Students who were involved in planning the project, and felt they “had a voice” were significantly more likely to experience positive outcomes.
Amount of Participation
• A significant number of students spent four hours or less involved in the project.
This level of involvement did not result in positive outcomes. However students who spent more time did experience positive outcomes.
Teacher’s Attitudes and Experience
• The teacher’s attitude toward the project was an important factor. If teachers were excited about the program then their students tended to have more positive outcomes.

Whether or not students felt they made decisions about the project
Amount of time students indicated they participated in the project.
Teacher’s self reported attitudes about the project.
1. Leadership Skills
2. Problem Solving Skills
3. Academic Aspirations
4. Liking for School
Student
Success
Results from focus groups with students and intensive interviews with teachers indicated that these were valid indicators of the quality and quantity of participations were related to outcomes.

Mean Score on Outcome Measurements by Amount of Time
Student Planned or Participated in Project
Attitudes Toward Attending College
Attitudes Toward School
Problem Solving Skills
Leadership Skills
Attitudes Toward Attending College
Attitudes Toward School
Problem Solving Skills
Leadership Skills
***p<.0001
Students Planned -Four Hours or
More
Yes No
12.81***
19.84***
17.01***
17.15***
12.50***
18.45***
15.58***
15.85***
Students Participated - Four Hours or
More
Yes No
12.89***
19.54***
16.91***
17.04***
12.18***
18.38***
15.09***
15.46***

Mean Scores on Outcome Measurements by Sense of
Ownership – Whether or Not Students Made Decisions
Students Made Decisions
Yes No
Attitudes Toward College
Attitudes Toward School
Problem Solving Skills
Leadership Skills
13.04***
19.59***
17.26***p
17.51***
12.36***
18.84***
15.58***
15.81***
* p< .05 **p< 0.01 ***p< 0.0001
Click here to see the
Power Point
Presentation.

CHART 1. High School Students’ Perception of How Good They are at Speaking in Front of Groups by Whether or Not They Made
Decisions about Service Learning Projects
(p <0.0001)

CHART 2. High School Students’ Perception of How Good They are at Finding Resources by Whether or Not They Made Decisions about
Service Learning Projects
(p <0.0001)
10 0 . 0 0 %
9 0 . 0 0 %
8 0 . 0 0 %
70 . 0 0 %
6 0 . 0 0 %
50 . 0 0 %
4 0 . 0 0 %
3 0 . 0 0 %
2 0 . 0 0 %
10 . 0 0 %
0 . 0 0 %
St r o ng l y
D i sag r ee
D i sag r ee U nd eci d ed A g r ee St r o ng l y A g r ee
D i d
D i d N o t

Carefully
Considering the Impact of
Timing
Selecting the
“right”
Indicators
Dealing with
Social
Desirability
Using Multi-
Methods and
Multiple Units of Analysis

In Evaluation Research,
Participant Often Evaluate a
Program Positively – EVEN when the Program is “poor” and Ineffective. It may not be seen as socially acceptable to do otherwise.
Beware of
Social
Desirability


Why did we become concerned?
 What are the “danger” signs?

How did we attempt to alleviate it?
 How did we modify the construction of our surveys, our research design and analysis of the data to deal with this problem?


Danger Signs
 Type of research
 Past literature indicates that respondents tend to be very positive when asked about their participation in a program even when it is a poor program. They don’ want to believe they wasted their time, and they often feel an obligation to express appreciation for those who implemented the program.


Danger Signs
 Self selection into the program
 Students and teachers were not required to participate in the program. Therefore the program was more likely to attract participants who already had positive attitudes toward these
“types” of activities.


Danger Signs
 Consistently high scores on every aspect of the program – no variation
 Response Set can occur. This is where respondents give you the same response (usually positive) without seriously considering the question.
 The “ceiling” effect is a similar problem. This is when you get consistently highly positive scores on the pre-test. In this case, there is little room for improvement in scores.

Check List
 Make participation voluntary and make answers anonymous or confidential.
 Vary negative/positive statements in
 Your index
 Avoid misleading/biased questions
 Make statements or questions very specific

Check List – continued
 Make participation voluntary and make answers anonymous or confidential.
 Put “sensitive” questions at the end
 Ask how they would change program “under ideal circumstances”.
 Avoid (1) yes or (2) no answers – ask “degrees” of positive or negative.
 Ask for their input in improving the program – rather than simply evaluating the program for instance:
 NOT – Is this a successful program, but rather - what factors increase or decrease the success of this program.

Check List

If possible, don’t evaluate your own program
 An “outsider” would tend to be more objective and participants would be more likely to provide unbiased answers.

Have a variety of participants evaluate the program so you can look for consistencies/inconsistencies in answers.
 Students
 Teachers
 Parents of participants
 Community Partners

Check List - continued

Use multi-methods so you can compare results across to see if you get similar results and look for additional insights. These could include:
 Focus groups
 Participant observation
 Surveys
 Intensive interviews
 Content analysis
Content analysis is especially important for researchers who identify tangible products (e.g., bushels of grains) as their outcomes.

Check List
 Compare your program with other programs
 Compare across different levels of participation within your sample to see if there are variations
 Compare across different types of participation within your sample (i.e., in our study, we compared across types of
Service Learning projects).

Check List
 Compare across different “types” of participants (This would include males vs. females, parents vs. children, rural vs. urban dwellers).
 Compare scores across questions – especially questions that measure the same outcomes.
 Compare answers across time fall vs. summer participants
The most important thing to remember here is to NOT just ask if the program was successful, but rather, HOW and WHEN it is most successful.

Carefully
Considering the Impact of
Timing
Selecting the
“right”
Indicators
Dealing with
Social
Desirability
Using Multi-
Methods and
Diverse Groups

Laboratory
Experiments
Focus groups
Field Trials
Case Studies
Surveys
Different Research
Designs can provide both additional insights and further support for your results.
Content
Analysis
Participant
Observation
Intensive
Interviews

Intensive
Interviews with
Teachers
Surveys with
Students
On-line surveys of Service
Learning
Coordinators
Evaluation of
Service Learning
Project
Face to Face
Surveys with
Parents
Focus Groups with Students
Mail-Out
Surveys with
Community
Partners.


Examples of data
 that the program is producing the desired outcomes
 collected from Teachers
 Using Telephone Surveys
 Using Focus Groups

Descriptive Statistics for Elementary and Middle/High School Service Learning Teachers:
Extent to Which Teachers Agree with the Following Statements about the Impact of Service
Learning in Their Classroom
Elementary Middle/High School
Number Percent Number Percent
Positive Addition to Classroom Learning
Agree/Strongly Agree 72 90.00
73 96.05
Beneficial for the ALL Students
Agree/Strongly Agree
Motivates Students to be Involved
Agree/Strongly Agree
75
66
93.75
87.50
74
66
97.37
88.89
Helps Students Learn Curriculum
Agree/Strongly Agree
Should be Required for All Students
Agree/Strongly Agree
47
40
58.75
50.00
56
50
73.68
65.79

Descriptive Statistics: Identification of GREATEST BENEFITS by
Middle/High School and Elementary Service Learning Teachers
Elementary Middle/High School
Benefits for Students
Service to Others
Understanding of World
Personal Growth
Help Learn Curriculum
Benefits for Teachers
Student Growth
Service to Others
Involvement with Students
Break in TAKS
Percent
50.00
14.71
30.89
4.41
82.26
16.13
1.61
--
Percent
32.43
25.68
32.43
9.46
73.91
15.94
7.25
2.90

As One Teacher Stated, “Service Learning is the most powerful and impactful thing I ever did in the classroom as a teacher. It hooked me, and I am a believer in the power
Another Teacher Claimed, “I think this program has transcended anything that anyone expected when they began the program. It has extended beyond what they thought it could achieve.”

While Another Claimed, “It teaches kids the skills that are not book skills….skills like how to think, how to plan, how to organize, how to manage - stuff you can read about in a book, but until you do it, you don’t know you have the ability to do it.”

One Teacher Stated, “ school is not as…engaging as when they learn through these projects…they are learning all of these things by action – their great public speaking skills, their writing skills, their marketing…”
Another Teacher Explained, “in the writing TAKS, we had to write with a prompt so it kind of helped with the writing and the reading TAKS too.”


Examples of data
 that the program is producing the desired outcomes
 collected from parents and community partners
 using telephone surveys
 using focus groups

Descriptive Statistics For Parents and Community Partners;
An Evaluation of the Service Learning Program
Positive Addition to Classroom
Beneficial for All Students
Motivates Students to be Involved
Helps Agency Achieve Goals
Is Valued by Agency
Is Valued by Community
Mean
Parents
Range
Community Partners
Mean Range
5.00
0 4.94
1
4.83
4.83
--
--
--
1
--
--
1
--
4.83
4.89
4.39
4.67
4.56
1
1
3
3
3
*on 5 pts. Scale (SD to SA)
**sample size is small

One Community Partner Described Their Relationship with the School, “ We Actually came to the schools…and we were looking for assistance. It’s a great marriage. We are still married.”
And when Describing the Benefits for Students,
“..
we’ve watched students mature into more socially aware students much more mature. It’s amazing. It’s just amazing.”


Timing of data collection is important.

Selecting reliable/valid indicators is critically important. Spend some time doing this.

IF you are doing Evaluation Research, plan ways to reduce the impact of social desirability on your results.

Use multi-methods when feasibility to provide additional insights and greater support for your results.

Try to gather information from all the different groups that may be impacted by the program (i.e., parents, students etc.).