Unit 1 Module 4
Explain: Roles of the Evaluator
Introduction to Educational Evaluation
5100-652
Dr. Kristin Koskey
Multiple Personalities
“Detective”
“Honest
broker”
“Leader”
“Counselor”
“Mediator”
“Negotiator”
“Designer”
“Creative
consultant”
“Devil’s advocate”
“Follower”
“Technical geek”
“Problem solver”
Grubbs, 2009; Mark, 2002; Patton, 2007; Skolits et al., 2009
“Judge”
“Organizational
analyst”
“Change agent”
“Communicator”
“Diplomat”
“Public relations
agent”
2
Evaluation
Stage
• Detective Learn about the context
• Negotiator Agree on the parameters of the
evaluation
• Designer Develop the evaluation plan
Active
Evaluation
Stage
• Diplomat Build relationships and trust
• Researcher Implement the plan
• Judge Judge the merit, value, worth
• Reporter Communicate results to stakeholders
Pre
Post
Evaluation
Stage
• Advocator Promote appropriate uses of evaluation
findings
• Reflector Engage in self-reflection
3
Pre-Evaluation Stage
• Role of the detective
 Learn about the context
 Conduct an evaluability assessment
• Role of the negotiator
 Reach an agreement on the evaluation
• Role of the designer
 Develop the evaluation plan
4
Pre-Evaluation Stage: Role of the
Detective
• Learn about the context
1.
2.
3.
4.
5.
What is to be evaluated?
Who is the client?
Why was this program initiated?
What are the program goals?
What activities are believed to lead to these intended
goals?
6. What is the theory behind these program goals
7. What is the political climate surrounding the evaluation?
8. Who are the most relevant stakeholders?
9. Who are the beneficiaries of the program?
10. Has this program been previously evaluated? If so, what
have different stakeholders observed as a result of the
program?
11. What data already exist on this program?
5
Pre-Evaluation Stage: Role of the
Detective Continued
• Determine the evaluability of the project
 Identify whether the program model is clear.
 Check the alignment between the program activities
and program model
 Determine if the program goals are achievable
 Determine if the stakeholders information needs are
feasible
 Assess whether the stakeholders and you agree on the
evaluation goals and uses
6
Pre-Evaluation Stage: Role of Detective
Continued
• When and when not to get involved
Green Light
Program goals are well defined
Program objectives are feasible to achieve given the program activities
and timeframe
Relevant data can be collected or obtained at a reasonable cost (cost in
terms of money and time)
It is clear that the evaluation results will be used for decision-making
and in what ways
Sufficient funds are available to provide for a quality evaluation
7
Pre-Evaluation Stage: Role of Detective
Continued
• When to and when not to get involved
Red Light
Program goals are ill-defined
The request for the evaluation is too
premature for the stage of the program
Program objectives are not feasible to
achieve
A conflict of interest exists
Data needed to achieve the evaluation
goals will not be available or feasible to
collect in the allotted timeframe or
budget
Key stakeholders cannot reach a
consensus on the direction of the
program and/or evaluation
Evaluation would provide trivial
information and the results would not be
used for decision making
Intended users of the results is not clear
8
Pre-Evaluation Stage: Role of
Negotiator
• Reach an agreement on:
– Evaluation to be conducted
– Goals of the evaluation
– How it will be conducted
– Timelines
– Budget
– Who will do what and when
– Use of evaluation findings
9
Pre-Evaluation Stage: Role of Designer
Evaluation model(s) based on the
evaluation goals
Research design
Methodology
Data sources to be collected
• Formative and summative
Strategies for data analysis and
reporting
Budget
10
Active-Evaluation Stage
• Role of the diplomat
 Build good working relationships and trust with the
stakeholders
• Role of the researcher
 Implement the evaluation plan
• Role of the judge
 Program strengths and weaknesses
 Value, merit, worth of whatever is being evaluated
• Role of the reporter
 Determine audience(s)
 Communicate relevant evaluation results
 Provide recommendations
11
Post-Evaluation Stage
• Role of the advocator
 Educate stakeholders how to use the evaluation results
 Promote stakeholders use of the evaluation results
• Role of the reflector
 Engage in self-reflection as a professional
• What worked well and what didn’t work well?
• Is there anything I could have done differently?
• What skills should I strengthen?
12
References
•
•
•
•
•
•
American Evaluation Association (2009). Guiding principles for
evaluators. American Journal of Evaluation, 30(1), 5-6.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program
evaluation: Alternative approaches and practical guidelines (4th ed).
Boston, MA: Pearson.
Grubbs, S. T. (2009). An evaluation of an alternative teacher
certification program: A matter of trust. American Journal of Evaluation,
30(4), 581-586.
Mark, M. M. (2002). Toward better understanding of alternative
evaluator roles. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring
evaluator role and identity (pp. 17-36). Greenwich, CT: Information Age
Publishing.
Patton, M. Q. (2007). Utlization-focused evaluation (4th ed.). Thousand
Oaks, CA: Sage
Skolitis, G. J., Morrow, J. A., & Burr, E. M. (2009). Reconceptualizing
evaluator roles. American Journal of Evaluation, 30(3), 275-295. doi:
10.1177/1098214009338872
13