1
Handout: The Expanded Program Model/Framework for TPD
Table of Contents
Figure 1: The Overview
……………………………………………………….. page 2
Uses of Expanded Frameworks in TPD Design and Evaluation ………………….page 3
Some Illustrative Evaluation Questions about Specific Components……….…….page 4
Contact Information:
The Center for Evaluation and Assessment
210 Lindquist Center South, University of Iowa
Iowa City, IA 52242
www.education.uiowa.edu/cea/
Updated handouts and slides available on our website soon
EMAIL:
d-yarbrough@uiowa.edu
julie-kearney@uiowa.edu
liz-hollingworth@uiowa.edu
valerie-moody@uiowa.edu
Copyright: D. Yarbrough, 2009, 2013, 2014
2
Figure 1: The Expanded, Extended Program Model (EPM)
Applied to Teacher Professional Development “” indicates chronology, i.e., the flow of time;
I. Phase 1--Prior Situation & Context: the Initial Focus for the Intended TPD
A. Context &
Environment of the
Proposed, Intended TPD
B. Purposes that the TPD Might
Serve, Including Needs,
Problems, Areas for Growth

C. Targeted Users of the TPD, Including
Process and Instrumental Users, and Other
Beneficiaries, Including Staff
D. Diagnostic Theory (part of the inclusive Program Theory) informing how the problem and needs that the TPD
will address are determined. This theory resides in/with the designers, staff, participants & stakeholders and is based on
scholarship, practice-wisdom & beliefs (see also Change Model)
|
\/
II. Phase 2--The Intervention Focus, Including Planning & Implementation
Planning
E. Solutions &
Strategies For
the TPD to
Implement in
this context

Implementation
F. Process &
Output Goals &
Objectives for
TPD Planning
in this context
G. Outcome
and Impact
Goals &
Objectives for
TPD Planning
in this context
H. Resources &
Inputs Actually
Used in this
context
I. Activities,
Methods &
Procedures
Actually Used
in this context
J. Outputs
(Products &
Participation)
in this
context
K. Program Theories informing the TPD impact or change model (Impact Theory & Intervention Hypothesis).
Based on scholarship, practice-wisdom & beliefs
L. Program Theories informing the TPD delivery, administration & management (Process theory, organizational and
service delivery plans). Based on scholarship, practice-wisdom & beliefs
|
\/
III. Phase 3--Post Implementation Situation & Results Focus
M. Post TPD Context &
Environment. How has
it changed as a result of
the TPD and other
factors?
N. TPD Outcomes:
Short-, Intermediate-, &
Long-term; Mediating &
Enabling; Gross & Net;
Other Classifications
O. Impacts:
a. Net Outcomes
b. Changes in the
Systems and
Contexts

P. Costs &
Efficiencies. What
are they and how
could they be best
managed?
Q. Program Theories explaining causal conclusions required in arguing that the TPD did or did not meet
intended goals and objectives or purposes (for teachers as learners and their students) as they are understood at
the end. (Impact theory and intervention hypothesis). Based on scholarship, practice-wisdom & beliefs
Copyright: D. Yarbrough, 2009, 2013, 2014
3
Uses for Expanded TPD Frameworks, Revised (October 2014)
1. To guide TPD design, planning, implementation and management
2. To compare specific TPD implementations to one another in order to understand differences
3.1. To guide the description and documentation of TPD programs, especially their components and
elements, in sufficient detail to achieve accuracy and precision
3.2. To guide the investigation, description, and documentation of TPD contexts and how they support,
create barriers for and interact with and change the TPD which is embedded in them.
4. To help clients generate, review, prioritize, and select evaluation questions for further investigation, at
holistic/global, component, or element levels
4.1. What do we already know?
4.2. What do we need to know and why?
4.3. What purposes and uses would this knowledge serve?
5. To clarify program (TPD) models, logic and theory: (social science theories; practitioner theories; and
“common sense” and traditional, cultural, and personal beliefs)
5.1. Diagnostic theory, including causal hypotheses
5.2. Impact (Intervention, causal) theory, including intervention hypotheses and action hypotheses
(agents, mediators, moderators, enabling variables, etc.), Change Models
5.3. Process theory, including service delivery and organizational plans, Action Models
5.4. Theories supporting other key constructs and relationships
6. To frame the processes used to discover, review and select evaluative criteria related to dimensions,
components and elements
6.1. To contextualize and identify dimensions & criteria of value, merit, worth, significance, importance,
or quality and subsequent standards or comparisons to be used
6.2. To guide investigations of values
7. To help match existing TPD to needs to be met, problems to be solved and enhancements or growth to
be supported, i.e., to inform transportability questions
8. To help plan, choose and design TPD evaluation projects, which themselves can be accurately
described following this framework
9. To help plan, choose and design metaevaluation subprojects for TPD evaluations
Copyright: D. Yarbrough, 2009, 2013, 2014
4
Some illustrative evaluation questions addressing the 17 different Framework components (A
through Q), arranged into three time periods (Phases I, II, & III) Draft revised October 2014
Possible Questions about Phase I, the Time Prior to TPD Implementation (A to D)
A. Context, environment, and ecosystem for the proposed, intended TPD







Does the intended TPD take into account the context in all of its important dynamics?
Does the intended TPD fit the context in its current state?
Will some specific conflicts, controversies or other disruptions in the context be caused by the TPD?
If so, how will the TPD respond and/or change to deal with them?
Are possible advantages of these contexts and environments understood as resources in the TPD?
How will the TPD take advantage of them or respond to them?
Is the context well enough understood by TPD staff so that interactions with it can be managed?
What are key factors in the context that will affect TPD components and functioning?
How similar is this context to pilot test or demonstration contexts or to contexts to which stakeholders
want to export this PPSP with expected similar results?
B. The TPD problem complex, needs, deficiencies, growth opportunities, or talent development
identified (and not identified) for addressing by the TPD. What needs/problems can this TPD
solve/meet?
B.1. The problem construction, understanding, definition, identification of root cause(s), methods
for choice and valuing, as well as problems in the immediate context not selected (cf. Preuss, 2003).
The specific talents, needs, deficiencies to be addressed and competing ones not addressed.







Is the problem well-constructed, given its possible complexity and multi-dimensional qualities?
Are competing constructions presented, based on competing diagnostic theories, assumptions, etc.?
Are the constructs and concepts used to define and understand the problem (construct the problem)
supported by sufficiently broadly-based social science research?
Is the problem stated for solving or is it stated as a solution? If stated as a solution, what preconceived
factors are dominant?
Are opportunity costs addressed in the rationale for choosing this proposed problem construction over
others?
What value systems are reflected in this problem construction, definition, and selection?
Are different and competing problems and needs that might be addressed by other TPD or non-TPD
solutions compared and systematic, rational decisions made about which to select for interventions
C. The choice of beneficiaries or customers (targets, participants, etc.) and salient non-beneficiaries
or competing beneficiaries, especially those who suffer real and/or opportunity costs






Once the problems or needs are selected for addressing, are the beneficiaries well-identified?
Are primary, secondary, and tertiary beneficiaries identified?
Are those who lose out and might be disadvantaged (for example, those with competing needs)
identified?
Whose definitions for beneficiaries are used? Are the best definitions used?
How will recruitment and retention of beneficiaries and participants take place?
Are opportunity costs addressed in choosing these beneficiaries over others?
D. The beliefs and belief systems (diagnostic theories) about what caused or is causing the needs,
problems, strengths to be developed or growth opportunities in the real world (causal model of the
Copyright: D. Yarbrough, 2009, 2013, 2014
5
phenomena to be addressed in the real world) including full specification of any competing causal
systems and their elements.



Are the diagnostic theories plausible (for example, are statements plausible arguing that the intended
beneficiaries need what is specified)?
Are the diagnostic theories supported empirically? …logically? …by other scholarship, including
previous evaluations?
Are the diagnostic theories competing with other theories, and if so who wants and is choosing this
approach? What evidence supports the selected theory or theories?
Possible Questions about Phase II—the Intervention Focus, Including Planning &
Implementation Time Periods
TPD Planning For and In this Context
E. The proposed, planned, and implemented TPD as a solution for needs and problems in this
context)
E.1. The methods and techniques used to search for or create solutions (treatments, services,
educational methods, curricula, content, etc.); how final choices were made and what values
affected final choices for this context; the proposed resources needed to constitute the solutions in
an effective manner in this context; the investigation and documentation of the fit of the selected
solution system to the identified problem complex in this context (see B above).









Compared with other possible activities, interventions, treatments, services, (i.e., alternative solutions
for this need/problem or set of needs/problems) is this proposed TPD solution or set of solutions the
best, the most effective and efficient for this context?
What evidence supports the quality of the proposed TPD solution or set of solutions for this context?
What alternative TPD approaches/solutions are available and how has the chosen one been selected in
this context?
What kinds of systems or typologies are used to specify resources required for this solution approach
in this context? Are the kinds and types of resources needed to implement the TPD fully described?
Are mechanisms identified that will be used to secure resources needed for this approach in this
context?
Are the expected challenges and barriers to implementing this solution set and moving resources into
“inputs” (applied resources) well-specified and strategies for dealing with them clear, especially related
to this context?
Are competing interests specified for this context, i.e., those in this context who also want these same
critical resources?
How plausible is the assumption that there are sufficient resources available to implement this
proposed TPD strategy effectively in this context?
How well are the constructs used to define, describe and specify the proposed solutions aligned with
the diagnostic theories and the causal theories for this context?
F. Process and output goals and objectives during planning, including for targets’ and other
beneficiaries’ identification and participation and for development of materials and other products
[Ideally, these will be explicitly justified in the articulated program theories but sometimes they remain implicit or
assumed in implemented TPD until discovered and described in evaluations or other investigations during
implementation. Unfortunately, sometimes a number of competing, unarticulated theories guide different staff
members and stakeholders toward different goals and objectives.]
Copyright: D. Yarbrough, 2009, 2013, 2014
6







Are the process and output goals and objectives thoroughly presented, plausible, and defensible in this
context?
Are the articulated process and output goals the most important and worthy of attention in this context?
Are the process and output goals aligned with the proposed solutions and strategies from Component E
above?
What other process and output goals/objectives are in competition with these? How are these
competing interests resolved for this context?
Are the constructs used in the process and output goal and objective statements for this context
supported by research or other scholarship and aligned with other program theory components?
Is recruitment and retention of beneficiaries/targets addressed in the process and service delivery
components?
What challenges and barriers in this context are anticipated that will interfere with or reduce the
attainment of planned, intended process and output goals?
G. Outcome and impact goals and objectives as proposed and intended












Do the outcome goals and objectives specify the ways that the beneficiaries are supposed to change
from their condition or state before the processes and services to afterwards?
Are impact and outcome goals defined so as to be clear, transparent, observable, assessable, and
evaluable (supported by quantitative/qualitative or mixed methods) in this context?
What value perspectives govern the merit and/or worth of the goals and objectives for this context?
Whose perspectives are most important for specifying these goals and objectives in this context?
Are these outcome and impact goals the most important and worthy in this context?
Which of these goals and objectives are controversial or open to dispute in this context and/or need
more discussion, either in general or in their specification for the selected TPD approach?
Are the constructs used to define and specify the goals and objectives aligned with the constructs used
for the statement of the problem and those used to specify the processes, activities, service delivery,
and outputs in this context?
How plausible are these goals and objectives in this context given the proposed processes, activities
and service delivery mechanisms?
How are the goals and objectives, as proposed and stated, evaluated to document their value as
intended goals and objectives and their alignment to other components in this context, including the
theoretical perspectives in the plan/proposal?
What challenges and barriers in this context are anticipated that will interfere with or reduce the
attainment of planned, intended process and output goals?
Do the goals/objectives also include performance specifications and criteria so that success in
achieving the intended goals is already defined in this context, or has the specification of intended
performance criteria been left for later?
TPD Implementation for and in this Context
H. Resources actually used: human, systemic, financial, time, etc. [tangible and intangible]




What proposed and planned resources were actually used to implement the TPD in this context? What
other new resources were used? What are the kinds and types of resources used?
What kinds of systems or typologies best describe and classify for this context the resources as actually
used?
Are the mechanisms used to identify and encumber or otherwise secure resources in this context clear?
What are they?
Are the expected and unexpected challenges and barriers to moving resources into “inputs” (applied
resources) well-specified for this context? What are they and how were they dealt with in this context?
Copyright: D. Yarbrough, 2009, 2013, 2014
7



What competitors in this context also want these same resources as inputs? How is this competition
being dealt with?
Are there sufficient resources available to implement the program well in this context? To what extent
are resource deficits in this context affecting services, activities, processes, participation, and other
outputs?
What are the costs of the resources used as inputs, individually and in the aggregate, as determined for
this context? What are the opportunity costs in this context associated with these resources?
I. Activities and processes







What scholarship, research, practitioner wisdom or stakeholder beliefs are supportive of or argue for
these activities and processes for this context (see also K and L below)?
What are the activities, processes, and service delivery mechanisms as actually perceived to be
implemented from the different perspectives of program staff, participants, evaluators, and other
observers in this context?
Are the constructs used to conceptualize the activities and processes as implemented in this context the
same as used during planning? Are they aligned with the theoretical constructs from diagnosis and
planning?
If new conceptualizations are being used, what program theories (K and L below) support the ensuing
or revised activities, processes, and service delivery mechanisms in this context?
Are the actual activities, processes, and service delivery mechanisms as perceived to have been
implemented successful in this context, i.e. plausible, affordable, efficient, and effective? What
dimensions of value are used to judge these in this context?
Can these actual activities, processes, and service delivery mechanisms be sustained in this context,
given the available resources?
Would other activities and processes be more effective or efficient in this context?
J. Outputs, including materials produced, and participation, including targets and other
stakeholders










What are the actual outputs and how are they classified in this context?
Are some of the outputs serving as resources in this or other TPD or programs for this or other
contexts? If so, how are they inter-connected and what are the effects, positive and negative?
Are the outputs valuable and important in this context? What dimensions of value are used to judge
and evaluate outputs in this context?
How are outputs currently being made available and/or documented in this context?
What are the mechanisms by which outputs resulted from activities and processes in this context (see
K and L below)?
Are there other needed or important outputs in this context that have been neglected?
How have key constructs used to guide and construct outputs been formulated in this context? Are
they aligned with other components in this framework?
What outputs were actually achieved in this context compared to those that were intended and
proposed?
What was the value (merit, worth, significance, importance) of the outputs in this and other contexts?
Did the outputs result from intended or other, perhaps improvised or serendipitous, activities and
processes?
K. Program theories, including beliefs and systems of beliefs (program intervention theories) that
lead the TPD designers and implementation staff to conclude that their selected solution system will
lead to specific goals and objectives, i.e., meet needs, further growth, or ameliorate or solve
problems in this context (program impact theories).
Copyright: D. Yarbrough, 2009, 2013, 2014
8




How many different intervention theories ground and explain the TPD inputs, activities and outputs in
this context?
What evidence supports the intervention and impact theories in this context?
What competing theories were not used to inform the planning and implementation of the TPD and
why?
Are there other, better theories that could ground or explain this TPD approach in this context?
L. Program theories, including beliefs and belief systems (process theories) about how to make the
inputs and activities work best and lead to positive, intended outcomes in the intended
targets/beneficiaries (including service utilization plans, organizational theories or plans, and/or a
management theories and plans).



Are the actual (as perceived) organizational, management, and service delivery plans for this context
clear, thorough, plausible and supported by scholarship?
Are they supported by social science findings and/or solid practice wisdom?
How were the achieved plans (as perceived in this context) different from or similar to the intended
ones from the perspective of underlying theories
Questions about Phase III—Post Implementation Time Period
M. Post TPD Context and Environment (at the end of implementation or at a revision/recycling
interval)






How has the context and environment in which the TPD is situated changed from the beginning of planning
and implementation to its current post implementation state?
To what extent have the factors resulting in the problems or needs changed as a result of the TPD
implementation and/or other factors, including historical, political, social, economic factors in this context?
How are identified needs which TPD could address or which compete with TPD for resources now
different or the same or similar as pre-intervention?
What critical factors affecting available resources and targets or beneficiaries have changed in this context?
To what extent, if at all, has the TPD had an impact on organizational learning and performance or other
important organizational characteristics?
To what extent has the TPD, if at all, affected system or organizational infrastructure and capacities?
N. Results as anticipated in the TPD goals and objectives description for this context and as
documented for the TPD as perceived to have been implemented, including unintended and
intended outcomes.
N.1. Outcomes: intended/unintended, positive/negative, mediating/ultimate, positive/negative,
short-intermediate-long-term





Are the constructs used to specify outcomes convergent with the constructs used to describe the other
components, including needs, problems, activities, outputs, or other features and elements over the
time and context phases of this evaluation?
Are the outcome constructs and assessment/documentation methods for this/these contexts clear and
thoroughly stated?
Are important positive and negative, intended and unintended outcomes stated and appropriately valid
assessments of them provided for this/these contexts?
Are mediating effects/enabling outcomes assessed with appropriately valid methods for this/these
contexts and the uses to be made in these contexts?
Are the construct definitions guiding assessments for this/these contexts supported by research and
theory?
Copyright: D. Yarbrough, 2009, 2013, 2014
9



Is the logic used in this/these contexts connecting outputs through outcomes, activities, and resources
back to problems and solutions clear?
Does the information collection design allow for gross and net outcomes to be differentiated in and for
this/these contexts? What are the most important gross and net outcomes?
How are dimensions of value (merit and worth) applied in this/these contexts to evaluate the various
types of outcomes?
O. Effects and results: for which the argument is made that in this context, net impact is the result
of the TPD rather than the gross impact of complementary training and other factors outside of the
specific TPD
Much the same as for results (outcomes) in N above, but especially:






Are the mediators and moderators support impact judgments made explicit for this context?
Does the design enable arguments (experimental, quasi-experimental, case-study, etc. enabled
arguments) that warrant impact (causal) conclusions?
Are the constructs used to specify impact assessment convergent with the constructs used to describe
the other components, including needs, problems, activities, outputs, or other features and elements
over the time and context phases of this evaluation?
Are the impact constructs and assessment/documentation methods for this/these contexts clear and
thoroughly stated?
Are the construct definitions guiding assessments for this/these contexts supported by research and
theory?
Is the logic used in this/these contexts connecting impacts through outcomes, activities, and resources
back to problems and solutions clear?
P. Efficiencies, inefficiencies, cost-efficiency, cost-benefits, return on investment






Is this TPD or aspects of this TPD relatively cost effective compared to other interventions in
this context?
What assumptions are made to justify conclusions about cost effectiveness in this context?
What values guide cost-effectiveness studies in this context?
How are benefits defined for this TPD in the context? What values and assumptions guide
analysis and specification of benefits from this TPD in this context?
If and how is the TPD determined to be sufficiently or insufficiently cost-beneficial?
Are cost-effectiveness and cost-benefit frameworks clear and transparent?
Q. Theory explaining causal conclusions




How similar and different are different stakeholders’ causal theories explaining the results
and effects of TPD?
Is the logic used to argue for or against TPD results and effects sound and warranted by the
program theory?
What is the evidence for and against the conclusion that the theories are valid for this TPD in
this/these contexts?
Are there competing program theories? Which program theories are supported by existing
expertise and scholarship?
Copyright: D. Yarbrough, 2009, 2013, 2014
10
The Expanded Evaluation Model (EEM) for the Our Kids Evaluation
The Expanded, Extended Evaluation Model (EEM)
Applied to the Our Kids (OK) Project “” indicates chronology, i.e., the flow of time;
I. Prior Situation & Context: the Initial Focus for the Intended Evaluation
A. Context &
Environment where the OK
Evaluation will be situated
B. Purposes and uses that the
OK Evaluation Might Serve

C. Potential users and beneficiaries of the
OK evaluation
D. Evaluation diagnostic theories informing how the purposes, problems and/or needs that the OK
Evaluation will address are defined, identified, and selected. These theories arise in the context and reside
in/with the evaluation designers and evaluation collaborators, users and stakeholders. A primary goal is to
create a powerful evaluative perspective and awareness through attention to the Program Evaluation
Standards and other evaluation scholarship and research finding.
|
\/
II. Evaluation as Intervention: Planning & Implementation
Planning
E. Solutions &
Strategies for
the OK
Evaluation in
this context

Implementation
F. Process &
Output Goals &
Objectives for
the OK
Evaluation in
this context
G. Outcome
and Impact
Goals &
Objectives for
the OK
Evaluation
H. Resources &
Inputs Actually
Used for the Ok
Evaluation in
this context
I. Activities,
Methods &
Procedures
Actually Used
in this context
J. Actual
Evaluation
Outputs
(Products &
Participation)
K. Evaluation theory informing the OK evaluation intervention/impact model. Based on evaluation theory,
including the PgES3, and other evaluation theories in current research and scholarship, as well as
stakeholder beliefs
L. Evaluation theory informing the delivery, administration & management of the OK evaluation
(Evaluation process Theory). Based on the PGES3 (e.g., Feasibility Standards) and other evaluation practice
theories, as well as general project management best practices and standards.
|
\/
III. Post Evaluation Situation & Results Focus
M. Post Evaluation
Context & Environment,
including changes in the
OK program.
N. Actual Evaluation
Outcomes (process &
findings uses &
influence) in this context

O. Actual
Evaluation Impacts
as documented in
this context
P. Actual Evaluation
Costs, Efficiencies
& Benefits for this
context, RoEvalI
Q. Program theories explaining how to argue that evaluation did or did not meet its purposes as
they are understood at the end of this cycle. (Evaluation impact theory and intervention hypotheses
as well as design logic). Based on evaluation theory and practice theory, as presented in the PgES3
and elsewhere.
Copyright: D. Yarbrough, 2009, 2013, 2014
11
Illustrative Uses for the EEM Frameworks, Revised (October 2014)
1. To guide evaluation design, planning, implementation and management
2. To compare specific evaluation implementations to one another in order to understand differences
across a broad spectrum of sub-dimensions related to evaluation feasibility, propriety, accuracy and utility
3.1. To guide the description and documentation of implementations of evaluations, especially their
components and elements, in sufficient detail to achieve accuracy and precision, in keeping with Standard
A4 of the Accuracy Standards, for example
3.2. To guide the investigation, description, and documentation of evaluation contexts and how they
support, create barriers for and interact with and change the evaluations which are embedded in them (see
Standards F3 Practical Procedures, F4 Contextual Viability or U5 Relevant Information
4. To help clients generate, review, prioritize, and select meta-evaluative questions for further
investigation, at holistic/global, component, or element levels, for example:
4.1. What do we already know about how the evaluation was conducted and its value to (intended)
stakeholders?
4.2. What else do we need to know about the evaluation and for what purposes (why)?
5. To clarify how evaluations work to result in use and influence and produce outcomes and broader
impacts, what is their logic and theory: (not just in the current standards and “sound practices” but in
novel ways that are as yet undiscovered or not widely acknowledged)
5.1. Evaluation diagnostic theories: why is this evaluation needed, who needs it, and what uses and
purposes might it serve, and what results (positive and negative, intended and unintended, etc.) might it
have
5.2. Evaluation impact theories: what and whose needs did this evaluation address, what factors
contributed to or inhibited or prevented impacts, how were these factors involved (agents, mediators,
moderators, enabling variables, etc.)
5.3. Evaluation process theory, including service delivery and organizational plans
5.4. Evaluation theories supporting other key constructs and relationships, including but not limited to
organizational learning theories, environmental systems theories, management theories, cognition of
evaluation/ judgment and so forth.
6. To frame the processes used to discover, review and select metaevaluative criteria related to
dimensions, components and elements in evaluations
6.1. To contextualize and identify dimensions & criteria of evaluation value, merit, worth, significance,
importance, or quality, as well as subsequent standards or comparisons to be used
6.2. To guide investigations of values guiding and embedded in evaluations and their contexts
7. To help match existing evaluation needs to be met, problems to be solved and enhancements or growth
to be supported, i.e., to inform evaluation transportability questions and inform future related evaluation
design and implementation
8. To help plan, choose and design metaevaluation projects
Copyright: D. Yarbrough, 2009, 2013, 2014
12
Some illustrative metaevaluation questions addressing the 17 different EEM components (A
through Q), arranged into three time periods (Phases I, II, & III)
I. Questions about the Time Prior to Evaluation Implementation (A to D)
A. Context, environment, and ecosystem for the proposed, intended evaluation







Does the intended evaluation take into account the context in all of its important dynamics?
Does the intended evaluation fit the context in its current state?
What if any specific conflicts, controversies or other disruptions in the context might be caused by the
evaluation? If so, how will the evaluation respond and/or change to deal with them?
How will possible advantages of these contexts and environments be used as resources for the
evaluation? How will the evaluation take advantage of them or respond to them?
Is the context well enough understood by evaluation staff so that interactions with it can be managed?
What are the most important key factors in the context that will affect evaluation components and
functioning?
What factors support the conclusion that the evaluation can be implemented with feasibility, propriety,
accuracy and utility? What factors raise doubts about the evaluation being implemented with sufficient
quality on these dimensions?
B. What uses and purposes might this evaluation solve/meet?





What conceptual (findings or process) uses are expected or need to be met?
What instrumental (findings or process) uses are expected or need to be met?
How well is evaluation understood in this context? What difficult misunderstandings or unrealistic,
negative expectations exist?
Are opportunity costs addressed in the initial evaluation overtures compared to other uses of the
resources?
What value systems concerning evaluation are reflected this evaluation context?
C. The identification and choice of those stakeholders who are intended to benefit from the
evaluation findings and processes, as well as of non-beneficiaries or competing beneficiaries,
especially those who might suffer real and/or opportunity costs as a result of the evaluation







Who are all the evaluation stakeholders who might be affected by the information and the processes
taking place in the evaluation
Are primary, secondary, and tertiary beneficiaries of the evaluation identified?
What risks with regard to stakeholders will the evaluation pose?
Are those who lose out and might be disadvantaged (for example, those with competing needs)
identified?
Whose definitions for evaluation stakeholders/ beneficiaries are used? Are the best definitions used?
How will recruitment and retention of involved stakeholders take place?
Are risks, including opportunity costs, addressed in choosing these stakeholders over others?
D. The evaluation diagnostic theories about what caused or is causing the needs and purposes for
the evaluation.


To what extent are those commissioning the evaluation focused on specific evaluation solutions or
approaches that pre-select or otherwise privilege or advantage certain stakeholders or certain types of
evaluation? (e.g., empowerment evaluation, methodology-driven evaluation, theory-driven evaluation,
participatory, collaborative, etc.)
In what ways are the evaluation change theories informed by those who fully understand the context
from “non-silo,” multi-disciplinary and indigenous perspectives?
Copyright: D. Yarbrough, 2009, 2013, 2014
13

Are these particular theories about why and how to conduct this evaluation competing with other
theories about what an effective and valuable evaluation would be, and if so who wants and is
choosing this approach? What evidence supports the selected theory or theories?
II. Possible Questions about the Evaluation Planning & Implementation Time Period
E. The proposed, planned, and designed evaluation as a solution for needs and problems in this
context
E.1. The methods and techniques used to design and plan for the evaluation; how final choices were
made and what values affected final choices for the evaluation in this context; the proposed
resources needed for an effective evaluation in this context; the investigation and documentation of
the fit of the proposed evaluation methodology and purposes to the identified purposes and needs in
this context (see B above).








Compared with other possible evaluation methodologies and approaches, will this proposed evaluation
be the most valuable, effective, and efficient for this context?
What evidence supports the quality of the proposed evaluation for this context?
What alternative evaluation approaches and methodologies are available and how has the chosen one
been selected in this context?
Are the kinds and types of resources needed to implement the evaluation fully described and possibly
available?
Are mechanisms identified that will be used to secure resources needed for this evaluation in this
context?
Are the expected challenges and barriers to implementing this evaluation and moving resources into
“inputs” (applied resources) well-specified and strategies for dealing with them clear, especially related
to this context?
Are competing interests specified for this context, i.e., those in this context who also want these same
critical resources for other uses than evaluation implementation?
How plausible is the assumption that there are sufficient resources available to implement this
proposed evaluation effectively in this context?
F. Evaluation process and output goals and objectives during planning
[These rely heavily on the standards for evaluation quality dimensions: utility, feasibility,
propriety, accuracy, and evaluation accountability, see attached checklist and rubrics. The
planning process will in part consist of identifying which of the all the available dimensions and
sub-dimensions of process and output quality to emphasize and prioritize.]





Are the evaluation process (such as increasing evaluation credibility, U1 Evaluator Credibility) and
output goals and objectives (such as identified in A8 Communicating and Reporting) thoroughly
presented, plausible, and defensible in this context?
Are the articulated evaluation process and output goals the most important and worthy of attention in
this context?
Are the evaluation process and output goals aligned with front end and needs assessments from Part I
above?
What other process and output goals/objectives are in competition with these? How are these
competing interests resolved for this context?
Are the constructs used in the process and output goal and objective statements for this context
supported by evaluation standards and evaluation or other research and scholarship?
Copyright: D. Yarbrough, 2009, 2013, 2014
14


Is recruitment and retention of evaluation key informants, participants, and other needed stakeholders
(including those who are intended to benefit from findings and processes) in the evaluation process and
service delivery components?
What challenges and barriers in this context are anticipated that will interfere with or reduce the
attainment of planned, intended evaluation process and output goals?
G. Evaluation outcome and impact goals and objectives as proposed and intended











Do the goals and objectives related to evaluation results specify the ways that the beneficiaries are
supposed to change from their condition or state before the processes, services and outputs to
afterwards? Are the results only related to knowledge (conceptual uses) or are they related as well to
behavior change and action (instrumental and performance uses)?
Are evaluation impact and outcome goals defined so as to be clear, transparent, observable, assessable,
and evaluable (supported by quantitative/qualitative or mixed methods) in this context?
What value perspectives govern the merit and/or worth of the evaluation goals and objectives for this
context?
Whose perspectives are most important for specifying these evaluation goals and objectives in this
context?
Are these evaluation outcome and impact goals the most important and worthy in this context?
Which of these evaluation goals and objectives are controversial or open to dispute in this context
and/or need more discussion, either in general or in their specification for the selected evaluation
approach and methodology?
Are the constructs used to define and specify the evaluation goals and objectives aligned with the
constructs used at Time I and those used to specify the evaluation processes, activities, service
delivery, and outputs in this context?
How plausible are these evaluation goals and objectives in this context given the proposed evaluation
processes, activities and service delivery mechanisms?
How are the evaluation goals and objectives, as proposed and stated, meta-evaluated to document their
value as intended evaluation goals and objectives and their alignment to other components in this
context, including the theoretical perspectives in the plan/proposal?
What challenges and barriers in this context are anticipated that will interfere with or reduce the
attainment of planned, intended evaluation process and output goals?
Do the evaluation goals/objectives also include performance specifications and criteria so that success
in achieving the intended evaluation goals is already defined in this context, or has the specification of
intended evaluation performance criteria been left for later?
Evaluation Implementation for and in this Context
H. Evaluation resources actually used: human, systemic, financial, time, etc. [tangible and
intangible]
[In general, these metaevaluative questions require extensive notes and/or investigations of evaluation
processes and methods, collected in an “evaluation case data set” during the evaluation implementation,
both for on-going monitoring and improvement of the evaluation in real-time and to make possible end of
cycle evaluation accountability determinations.]



What proposed and planned resources were actually used to implement the evaluation in this context?
What other new resources were used? What are the kinds and types of resources used?
What kinds of systems or typologies best describe and classify for this context the resources as actually
used in the evaluation?
Are the mechanisms used to identify and encumber or otherwise secure evaluation resources in this
context clear? What are they?
Copyright: D. Yarbrough, 2009, 2013, 2014
15






Are the expected and unexpected challenges and barriers to moving evaluation resources into
evaluation “inputs” (applied resources) well-specified for this context? What are they and how were
they dealt with in this context?
What competitors in this context also want these same resources? How is this competition being dealt
with? [This is especially important to investigate when program implementation staff are also engaged
in crucial evaluation processes and outputs, including evaluation data collection.]
Are there sufficient resources available to implement the evaluation well in this context? To what
extent are evaluation resource deficits in this context affecting evaluation services, activities,
processes, participation, and other outputs?
Is the evaluation demonstrating appropriate fiduciary responsibility?
What are the costs of the resources used as evaluation inputs, individually and in the aggregate, as
determined for this context? What are the opportunity costs in this context associated with these
resources?
What are possible risks to the evaluation as well as to other important operations, programs and
projects in this context to the use of these resources as evaluation inputs?
I. Evaluation activities and processes







What standards, scholarship, research, practitioner wisdom or stakeholder beliefs are supportive of or
argue for these evaluation activities and processes for this context (see also K and L below)?
What are the evaluation activities, processes, and service delivery mechanisms as actually perceived to
be implemented from the different perspectives of evaluation staff, participants, evaluators, and other
observers in this context?
Are the constructs used to conceptualize the evaluation activities and processes as implemented in this
context the same as used during planning? Are they aligned with the theoretical constructs from
evaluation diagnosis and planning, including those expressed in the PgES3?
If new conceptualizations are being used, what evaluation theories (K and L below) support the
ensuing or revised activities, processes, and service delivery mechanisms in this context?
Are the actual evaluation activities, processes, and service delivery mechanisms as perceived to have
been implemented successful in this context, i.e. plausible, affordable, efficient, and effective? What
dimensions of value are used to judge these in this context?
Can these actual evaluation activities, processes, and service delivery mechanisms be sustained in this
context, given the available resources?
Would other evaluation activities and processes be more effective or efficient in this context?
J. Evaluation participation, including program staff and participants and other evaluation
stakeholders, and evaluation outputs, including communications, information, assessments,
presentations, reports and the like.








What are the evaluation outputs that are reported and documented in this context?
Are the evaluation outputs valuable and important in this context? What dimensions of value are used
to judge and evaluate evaluation outputs in this context?
How are evaluation outputs currently being made available and/or documented in this context?
What are the mechanisms by which evaluation outputs resulted from activities and processes in this
context (see K and L below)?
Are there other needed or important evaluation outputs in this context that have been neglected (types
of information, reports, findings, explanations, judgments)?
How have key constructs used to guide and construct evaluation outputs been formulated in this
context? Are they aligned with other components in this framework?
What evaluation outputs were actually achieved in this context compared to those that were intended
and proposed?
What was the value (merit, worth, significance, importance) of the evaluation outputs in this and other
contexts?
Copyright: D. Yarbrough, 2009, 2013, 2014
16

Did the evaluation outputs result from intended or other, perhaps improvised or serendipitous,
evaluation approaches, methodologies, activities and processes?
K. Evaluation theories informing the OK evaluation intervention/impact model (action model).
Based on best evaluation practice, including the PgES3






What are the different evaluation theoretical approaches that could explain the ways that the evaluation
will have or is having an impact in this context?
What evidence supports these evaluation intervention and impact theories in this context?
What competing evaluation theories were not used to inform the evaluation planning and
implementation and why?
Are there other, better evaluation theories that could ground or explain this evaluation approach in this
context?
In what ways is the chosen evaluation theoretical approach not working or not being implemented in
this context?
What evidence is there for evaluation theory success and/or failure?
L. Evaluation theory informing the delivery, administration & management of the OK evaluation
(Process Theory). Based on the PGES3 (e.g., Feasibility Standards), as well as general project
management best practices and standards.




Are the documented organizational, management, and service delivery plans used to implement the
evaluation in this context clear, thorough, plausible and supported by scholarship?
Are they supported by social science findings and/or solid practice wisdom?
How were the achieved evaluation plans (as documented in this context) different from or similar to
what the adopted theoretical literature supports
What evidence is there for implementation success and/or failure?
III. Questions about the Post Evaluation Time Period
M. Post Evaluation Context and Environment (at the end of evaluation or at a revision/recycling
interval, usually after a significant time period, for example Year 1, Year 2, etc.)






How has the context and environment in which the evaluation is situated changed from the beginning of
planning to its current post evaluation/evaluation time period state?
To what extent have the purposes and needs which the evaluation was designed to address changed as a
result of the evaluation implementation and/or other factors, including historical, political, social, economic
factors in this context?
How are identified needs which the evaluation could address or which compete with evaluation for
resources now different or the same or similar as pre-planning?
What critical factors affecting available resources and targets or beneficiaries have changed in this context?
To what extent, if at all, has the evaluation had an impact on organizational learning and performance or
other important organizational characteristics?
To what extent has the evaluation, if at all, affected system or organizational infrastructure and capacities?
N. Actual Evaluation Outcomes (process & findings uses & influence) in this context
N.1. Uses and outcomes of the evaluation: intended/unintended, positive/negative,
mediating/ultimate, positive/negative, short-intermediate-long-term

Are the constructs used to specify the documented evaluation uses sound and aligned with the planning
constructs?
Copyright: D. Yarbrough, 2009, 2013, 2014
17







Are the constructs and assessment/documentation methods for assessing evaluation uses and outcomes
in this/these contexts clear and thoroughly stated?
Are important positive and negative, intended and unintended uses and outcomes stated and
appropriately valid assessments of them provided for this/these contexts?
Are mediating effects/enabling uses and outcomes assessed with appropriately valid methods for
this/these contexts and the uses to be made in these contexts?
Are the construct definitions guiding assessments of evaluation uses and outcomes for this/these
contexts supported by research and theory?
Is the logic used in this/these contexts connecting evaluation outputs through outcomes, activities, and
resources back to needs for and purposes of the evaluation clear?
Does the information collection design for evaluating the evaluation allow for gross and net outcomes
to be differentiated in and for this/these contexts? What are the most important gross and net
evaluation outcomes?
How are dimensions of value (merit and worth) applied in this/these contexts to evaluate the various
types of evaluation outcomes?
O. Actual Evaluation Impacts as perceived and/or documented in this context
[Investigating the net impact of an evaluation would require an extensive evaluation case data set created to support
causal attributions or would require multiple replications in a solid quasi- experimental or experimental design. This
logic is more often used to evaluate specific contrasting or variable evaluation components for the effectiveness in
contributing to specific kinds of evaluation quality.]
 Not included as a component in the OK metaevaluation
P. Actual Evaluation Costs, Efficiencies & Benefits as perceived and/or documented in this context






Was this evaluation relatively cost effective compared to other evaluations that achieved the
same uses and purposes?
What assumptions are made to justify conclusions about evaluation cost effectiveness in this
context?
What values guide evaluation cost-effectiveness studies in this context?
How are evaluation benefits defined in the context? What values and assumptions guide
analysis and specification of benefits and benefit monetization in this context?
How is the evaluation determined to be sufficiently or insufficiently cost-beneficial?
Are cost-effectiveness and cost-benefit frameworks used to metaevaluate the evaluation clear
and transparent?
Q. Theories explaining causal conclusions about evaluation quality, especially outcomes, uses,
impact, and influence




How similar and different are different stakeholders’ causal theories explaining the value and
uses made of the evaluation?
Is the logic used to argue for or against the impact of the evaluation sound and warranted by
the evaluation theory?
What is the evidence for and against the conclusion that the evaluation theories are valid for
this evaluation in this/these contexts?
Are there competing evaluation theories in this context/situation? Which evaluation theories
are best supported?
Copyright: D. Yarbrough, 2009, 2013, 2014
18
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. (2011).
The program evaluation standards: A guide for evaluators and evaluation
users (3rd ed.). Thousand Oaks, CA: Sage. [Non-exclusive rights are freely
given to all evaluation stakeholders to use, quote, or apply these
standard names and statements with exact wording and with attribution
to this citation or to the JCSEE, at www.jcsee.org]
Utility Standards
The utility standards are intended to increase the extent to which program stakeholders find evaluation
processes and products valuable in meeting their needs.








U1 Evaluator Credibility Evaluations should be conducted by qualified people who establish
and maintain credibility in the evaluation context.
U2 Attention to Stakeholders Evaluations should devote attention to the full range of
individuals and groups invested in the program and affected by its evaluation.
U3 Negotiated Purposes Evaluation purposes should be identified and continually negotiated
based on the needs of stakeholders.
U4 Explicit Values Evaluations should clarify and specify the individual and cultural
values underpinning purposes, processes, and judgments.
U5 Relevant Information Evaluation information should serve the identified and emergent needs
of stakeholders.
U6 Meaningful Processes and Products Evaluations should const ruct activities, descriptions,
and judgments in ways that encourage participants to rediscover, reinterpret, or revise their
understandings and behaviors.
U7 Timely and Appropriate Communicating and Reporting Evaluations should attend to
the continuing information needs of their multiple audiences.
U8 Concern for Consequences and Influence Evaluations should promote responsible
and adaptive use while guarding against unintended negative consequences and misuse.
Feasibility Standards
The feasibility standards are intended to increase evaluation effectiveness and efficiency.




F1 Project Management Evaluations should use effective project management strategies.
F2 Practical Procedures Evaluation procedures should be practical and responsive to the way
the program operates.
F3 Contextual Viability Evaluations should recognize, monitor, and balance the cultural
and political interests and needs of individuals and groups.
F4 Resource Use Evaluations should use resources effectively and efficiently.
Propriety Standards
The propriety standards support what is proper, fair, legal, right and just in evaluations.



P1 Responsive and Inclusive Orientation Evaluations should be responsive to stakeholders
and their communities.
P2 Formal Agreements Evaluation agreements should be negotiated to make obligations explicit
and take into account the needs, expectations, and cultural contexts of clients and other
stakeholders.
P3 Human Rights and Respect Evaluations should be designed and conducted to protect
human and legal rights and maintain the dignity of participants and other stakeholders.
Copyright: D. Yarbrough, 2009, 2013, 2014
19

P4 Clarity and Fairness Evaluations should be understandable and fair in addressing
stakeholder needs and purposes.

P5 Transparency and Disclosure Evaluations should provide complete descriptions of
findings, limitations, and conclusions to all stakeholders, unless doing so would violate legal and
propriety obligations.
P6 Conflicts of Interests Evaluations should openly and honestly identify and address real
or perceived conflicts of interests that may compromise the evaluation.
P7 Fiscal Responsibility Evaluations should account for all expended resources and comply
with sound fiscal procedures and processes.


Accuracy Standards
The accuracy standards are intended to increase the dependability and truthfulness of evaluation
representations, propositions, and findings, especially those that support interpretations and judgments
about quality.








A1 Justified Conclusions and Decisions Evaluation conclusions and decisions should be
explicitly justified in the cultures and contexts where they have consequences.
A2 Valid Information Evaluation information should serve the intended purposes and support
valid interpretations.
A3 Reliable Information Evaluation procedures should yield sufficiently dependable and
consistent information for the intended uses.
A4 Explicit Program and Context Descriptions Evaluations should document programs and
their contexts with appropriate detail and scope for the evaluation purposes.
A5 Information Management Evaluations should employ systematic information collection,
review, verification, and storage methods.
A6 Sound Designs and Analyses Evaluations should employ technically adequate designs
and analyses that are appropriate for the evaluation purposes.
A7 Explicit Evaluation Reasoning Evaluation reasoning leading from information and analyses
to findings, interpretations, conclusions, and judgments should be clearly and completely
documented.
A8 Communication and Reporting Evaluation communications should have adequate scope
and guard against misconceptions, biases, distortions, and errors.
Evaluation Accountability Standards
The evaluation accountability standards encourage adequate documentation of evaluations and a
metaevaluative perspective focused on improvement and accountability for evaluation processes and
products.



E1 Evaluation Documentation Evaluations should fully document their negotiated purposes
and implemented designs, procedures, data, and outcomes.
E2 Internal Metaevaluation Evaluators should use these and other applicable standards to
examine the accountability of the evaluation design, procedures employed, information collected,
and outcomes.
E3 External Metaevaluation Program evaluation sponsors, clients, evaluators, and other
stakeholders should encourage the conduct of external metaevaluations using these and
other applicable standards.
Copyright: D. Yarbrough, 2009, 2013, 2014