PRT 595 Program Evaluation
Final Reflection Paper
Task #1
There will always a need to be able to understand how we get to where we are. Understanding
program evaluation, is understanding what it takes to assess social problems, and how we can improve
existing conditions. More specifically, program evaluation is about using social research, looking at social
problems, and finding a way to improve social conditions. Stakeholders are the groups, individuals, or
agencies who are tasked with identifying the need for evaluation as they are best defined as those who
have decision-making powers and a vested interest on how programs operate, and are usually the
experts in implementing processes for social change. In the simplest terms, evaluation can be how we
do, what we do. Evaluations are identified in five basic types: 1) Needs Assessment; 2) Assessment of
Program Theory; 3) Assessment of Program Process; 4) Impact Assessment, and 5) Efficiency
Assessment. When planning for evaluation, the evaluation position should be molded on the basis of a
particular evaluation type, or a combination of types. Evaluation should have a purpose whether they
are formative (info that helps guide program improvement), summative (intended benefits based on
effective resources), and knowledge generation (info describing the nature and effects of an
intervention.) Evaluation questions are an iatrical key to effective evaluation. Good evaluation
questions should guide the evaluation design, and should be tied into the program goals and objectives.
When examining the types of evaluation, programs can take on positions that need the
approach of any of the five given types of evaluation. When looking at needs assessment, the focus is to
answer questions of the social conditions; to identify targets (people, customers, consumers to serve),
and identify where the program operates. The intent is to address the need for the program or
determine whether a new one is needed. In my profession of recreation/youth sports, specifically our
program goals are to provide quality of life experiences in a social setting/backdrop of youth sports;
promote health and physical activities; build friendships; promote sportsmanship, improve social skills,
and instill a sense of community and involvement. My customers/targets are our youth aged 5-15.
Needs assessment identifies a population in need, lays the foundation for program evaluation, and uses
its stakeholders to accomplish the tasks at hand. Program theory and process helps us to get programs
ready to be evaluated. The program theory is the working theory, as it enables us to explain why the
program does what it does. Once a theory explains what the program does and it is well understood, the
theory is then deemed explicit, henceforth an articulated program theory. The articulated program
theory is identified through the program’s identity, or where the stakeholders and evaluators form the
theory. A program theory is a working theory. A critical area of evaluation lies within program theory
and this is where the logic model is introduced in the evaluation process. The ultimate goals are
outcomes; how the program operates. Under program theory we also address impact theory- what the
program does; service utilization plan- how the program operates, and organizational plan- how the
program needs to operate. Program theory answers why our programs exist, it links program elements
together in terms of process, outcomes, and efficiency. When addressing or accessing program
outcomes, we return to the need for improving a social condition. Essentially we start at the end of the
process of the evaluation and work backwards. Important aspects of program outcomes are validity and
proper interpretation of the gathered and analyzed results. When measuring data, the best kind of data
is entrenched in actual numbers. This is objective data. Subjective data would be considered behaviors
and attitudes. Additionally when measuring data, it must be reliable- when it is measured repeatedly it
should measure the same thing or same results. Going back to validity- we must measure what we say
we are measuring, and sensitivity- where values of measure change when there is a change in the object
being measured. We have essentially now reached the end of the logic model of evaluation. Now the
expectations of change should yield the benefits. In my evaluation proposal, the goal was to measure
ways to increase program participation in youth summer sports clinics as attendance has been
considerably lower in recent summers, and we are evaluating how we got here and how we can improve
on participation. In closing task #1, we cannot assume outcomes will change the way we expect them to
do. In reaching the gold standard of evaluation we have to determine how the change is related to the
program itself. We know we don’t have to be perfect, just good enough. To close the gap in the number
of kids enrolled versus the number of kids not enrolled in the program, is the basis for our evaluation
and determining how this gap can be closed is our purpose.
Task #2
“We’re Selling, Are You Buying”?
A Recreator’s Guide To Evaluation
Session Description:
This session will provide you with basic guidelines on how to perform a successful evaluation of
your recreation programming. In evaluating your recreation programs, you will grasp working
knowledge as it implies to using social research methods to reach your destination of a concise working
evaluation model. This session will bring forth the issues and instruments needed in identifying social
conditions that warrant evaluation that will enable your agency to address those social issues needing to
be measured and the methods needed to impact the changes from an improvement perspective. So, we
are selling, are you buying?
Session Learning Outcomes:
1. What instrument tools will you employ to conduct your evaluation?
a. Identify program staff, your stakeholders, and what capital is needed to conduct your
evaluation.
2. Identify your outputs. What are you doing? Focus on a specific program needing evaluation.
a. What activities are you providing you see those problems in need for social change?
b. Who are your clients, customers, or consumers? Who are your targets?
3. How will your findings lead to positive outcomes? Outcomes are where changes are impacted.
a. What can be identified in the short term (change)?
b. What can be identified at the midpoint?
c. What changes will occur in the long term?
Session Outline:
I.
Tailoring Your Evaluation
a. Look at a specific set of circumstances in order to answer the questions related to the
problem.
b. Ask yourself, what is the purpose of the evaluation; the reason the program needs
evaluation, and what are your available resources?
II.
Types of Evaluation- focus on a specific, your problem however, may take on more than one
discipline.
a. Needs Assessment- Do you need this program? This lays the groundwork for your
evaluation.
b. Program Theory- What is your program’s design?
c. Program Process- How is your program implemented?
d. Outcomes/Impacts- Evaluating the program’s results and effects.
e. Program Efficiency- How are the cost and benefits related?
III.
Identify Your Targets
a. Where does your population lie?
b. What is the description of your target population?
c. What is their socio-economic disposition? What are their ages, can they be identified?
IV.
Developing Your Logic Model
a. Gather your team of evaluators.
b. Identify what you are doing. This is your measure of outputs.
c. Identify your things that change. These are your measurements of outcomes.
d. At the end of your logic model, there are the expectations of change, and benefits.
V.
Putting It All Together
a. Evaluation establishes a bottom line need to improve social conditions.
b. The focus of evaluation is centered on a need to retain a program based on its
effectiveness and service to which it serves.
c. It is of utmost importance to assure your data is valid, reliable, and can be interpreted.
d. ‘Go for the gold”! The gold standard, that is. This is measured by the change that is
related to the program itself.
e. Now, go and impact your program!
Speaker Information:
Paul Nilsen, M.S., Webster University. Director, Marine Corps Community Services Semper Fit Division,
Camp Lejeune, NC. 23 years’ experience in facility management, personnel management, outdoor
recreation, including adult and youth programming.
Chris Alger, B.S., Southwest Missouri State University. Sports Branch Head, Marine Corps Community
Services Semper Fit Division, Camp Lejeune, NC. 12 years’ experience specializing in (military
communities) adult and youth programming, outdoor recreation services, and facilities management.
Chris Williams, U.S. Navy (Retired), B.G.S., Indiana University. Youth Sports Manager, Marine Corps
Community Services Semper Fit Division, Camp Lejeune, NC. 8 years’ experience in youth sports
programming and facility management.
Task #3
My role in program evaluation going forward will be to re-define, and re-tool our current
evaluation processes. With the support of my superiors in the sports branch, I am extremely confident
we can do a much better job in program evaluation. I also feel there is much more emphasis that needs
to be placed on our evaluation processes, other than just a check in the box, which has been sort of the
norm. The knowledge gained from PRT 595 has given me a new approach, a greater insight, a new
perspective and more appreciation for program evaluation. My perceptions of what evaluation should
incorporate are vastly improved. I could probably incorporate a self-imposed logic model of myself in
reference to my self-perceived skills and abilities then and now in relation to evaluation. A neat tool for
future classes, could be a PRT 595 pre & post-test just to see if we are great evaluators initially like we
think we are, only to find out the majority of us were not… Of course our responses on the pre-test
would be valid 