EVALUATION OF US DISPUTE
RESOLUTION PROGRAMS FOR
ENVIRONMENTAL AND COMPLEX
PUBLIC POLICY DISPUTES
Andy Rowe
GHK International
ghkint.us@ghkint.com
www.ghkint.com
1
DISCLAIMER
This presentation has been prepared by the author and
represents only my views. The agencies participating in
or contracting the work on which this presentation is
based do not necessarily subscribe to any or all of what is
contained in the presentation.
Andy Rowe
2
CHEAPER FASTER
BETTER?
 There appears to be more belief than evidence
underlying this claim
 This is really a statement about program
effectiveness:
“Collaborative processes are more effective than a reasonable or likely
alternative process”
 The evaluation system being used by important
U.S. federal and state agencies is starting to
provide evidence for this claim of effectiveness
3
BACKGROUND
 The evaluation system specifically addresses environmental and
complex public policy disputes
 Features of the system:
•
•
•
•
•
•
Takes a results-based accountability / performance management approach
Takes an improvement orientation
Addresses expected internal and external performance reporting requirements (state
and federal)
Self administered
Has a high level of face validity for both ADR and evaluation professions
Is proving to have a high level of validity and reliability
 Focus on utilization. Results will:
•
•
•
Be easy to interpret
Be useful for improving performance
Be easily comprehended by ADR professionals
4
APPLICATIONS
 U.S. Federal agencies
•
•
•
USEPA Conflict Prevention and Resolution Center
US Institute for Environmental Conflict Resolution
Other federal environmental agencies through contact to USIECR
 U.S. State agencies
•
•
Oregon Dispute Resolution Commission
Florida Conflict Resolution Consortium
 Other applications
•
•
Base for Maryland Alternative Dispute Resolution
Applied in pilot project of 9th Circuit
5
A PERFORMANCE
MANAGEMENT APPROACH
 Primary goal is to use evaluation as a mechanism to
focus organizations on results and achieve success
•
•
While satisfying evaluation goals and standards
Address possible Government Performance Results Act (GPRA)
requirements
 John A. Koskinen’s three GPRA questions are what the
evaluation attempts to address (John McLaughlin
version):
•
•
•
What is your program / agency supposed to achieve?
How will you know when you have achieved this?
How are you doing now?
6
IMPLEMENTATION
 Questionnaires to parties, practitioners and project manager at
conclusion of the process and a follow-up to parties at a later date
•
Mail, web and email administration options
 Response rates are high (will exceed 70% in Oregon pilot) and
100% of practitioners (included their contracts)
 Information was systematically tested and has proved valid and
reliable
 Meets performance standards for the design
•
•
•
•
Managers are “can answer” to “fluent” for first two questions
Resource requirements = 10 hours per case for admin, analysis & reporting
Culture of the organizations shifts towards results focus
Internal stakeholders very satisfied, external not yet known
8
OUTCOME CHARTS ARE THE
KEY TO THE DESIGN
 Similar to logic models / logframes but much more
focused on results
 Outcome charts are best developed collaboratively with
the program managers / staff
 Illustrations that follow are from the Public Policy
Program of the Oregon Dispute Resolution
Commission
•
This is an ADR collaborative process – similar to applications in other
agencies
9
OREGON PUBLIC POLICY DISPUTE RESOLUTION PROGRAM OUTCOMES
Decreased cost
of resolving
disputes
Increased efficiency of
agencies through use of
ADR
ADR in State Government
Oregon
state
agencies
have
practices
and
systems
that
ensure
ADR is
available
Practice of
ADR
supports
agency
mission,
and
receives
positive
feedback
Increase agency and public
satisfaction with process and
results of ADR
Case Management
Dispute Systems
Designs
Dispute resolution
systems are
improved
ADR is successful (time
and resources spent in
disagreement and conflict
are now redirected to
more constructive
purposes)
Mediator Roster
Roster provides
expedited access to
qualified and
appropriate providers
Good
agreements
Good
Process
10
EXTRACTED OUTCOMES FOR OREGON CASE MANAGEMENT
ADR successful (resources spent in disagreement are now used more constructively)
Durable and
implementable
agreement
reached using
ADR
Agreements
reached with
ADR are
complete - there
are no hard
issues left or
deferred.
Good outcomes from process
All parties involved in an
ADR process are
satisfied that the process
was fair and open.
ADR more effective (better
benefits for resources
expended) than other
options for this dispute
Right parties continue to
be engaged, new parties
added as required.
Use of ADR helps
adversarial parties
collaborate
Use of ADR narrows disagreements
Government decision making is
improved through use of the process
ADR is used where it is
the best approach for
this case
Design of the ADR process
is appropriate for the
dispute and needs of parties
Appropriate mediator
leads ADR
Right parties engaged in
process
Non-ADR processes used where the best approach
11
ADR used appropriately
Stakeholder capacity to use ADR is
improved through experience with
this case
Party interactions constructive
Good agreements
Simplifying Complex Practices
 Outcome charts seek to display the responses to the first of the
GPRA questions:
• What is my agency/unit trying to achieve?
 The complexity of any one element of ADR practice is difficult to
articulate in a single outcome but this complexity is required to
respond to the second GPRA question:
• How will we recognize this?
• The mechanism I am currently using to simplify this is nested outcomes – mini
outcome charts under individual outcomes
 Even the nested outcomes can be at too high a level of
generalization to be able to respond to the third GPRA questions
• How are we doing now?
• This is either addressed through further nesting or directly by the questions we ask
13
ILLUSTRATION OF CHARTED AND NESTED OUTCOMES
Nested Outcomes
Expressed as questions
At not time did one of the parties
dominate to the detriment of the
process or others
The neutral helped us manage our time
well.
The neutral made sure that the
concerns of all parties were heard.
The parties followed the direction of
the neutral.
The neutral made sure that the
concerns of all parties were addressed.
The neutral ensured that all parties
were fully engaged in the process.
When things got tense the neutral was
always able to find ways to move
ahead constructively
Nested Outcomes
Charted Outcome
Appropriate neutral –
parties satisfied with
neutral
Appropriate neutral –
neutral understands
issues
Appropriate neutral –
manages process well
Appropriate
Neutral Guides
the Process
Appropriate neutral –
available for process
Appropriate neutral –
parties understand
process
14
FCRC Case "A" Nested Outcome
Results for Right Neutral
Appropriate neutral - neutral
available
Appropriate neutral - neutral
manages process well
Appropriate neutral - neutral
understands issues
Appropriate neutral - parties
understand process
Appropriate neutral guides the
process
6.0
7.0
8.0
9.0
10 . 0
S c o r e 0 ( l o w e s t ) t o 10 ( h i g h e s t )
15
FCRC Case "A" Outcome Results
Agency and party capacity is improved
Government decision making is improved
Intergovernmental coordination is improved
Agreements endure
Agreements are implemented
All parties are satisfed with what they have
Parties reach complete agreements
Collaborative process is more effective
Parties satisfied with the process
Parties communicate and collaborate
Best information used by parties
Disagreement narrows
FCRC designs an appropriate process
Appropriate neutral guides the process
Right parties are effectively engaged
6.0
7.0
8.0
9.0
10.0
Score 0 (lowest) to 10 (highest)
16
OTHER ELEMENTS IN THE
EVALUATION SYSTEM
 Collaborative Processes
•
•
•
•
•
Program Intake
ECR/ADR/Collaborative Processes (longer and shorter forms
Conflict Assessment / Collaborative Assessment / Situation Assessment
Meetings: information gathering / public input
Dispute System Design
 Other ADR Services
•
•
•
Training
Roster program
Contract for neutral services
18
THE EVALUATION DESIGN IS
SUCCESSFUL
 Have established a program theory for ADR in environmental and
complex public policy disputes
 Approach is being implemented by a growing number of agencies
 Positive peer external reviews (practice and evaluation)
 Information is helping program improve
 Information is addressing internal and external accountability
requirements
19
MAIN REMAINING CHALLENGES
 Incremental Benefits
•
•
Inherently difficult because of nature of many ADR cases
In some settings this can be addressed without difficulty
 Effectiveness
•
•
Inherently difficult because of nature of most or perhaps all ADR cases
Lack of useful applied research in this area
 Both incremental benefits and effectiveness are highly charged
concepts in the field
•
•
Many strong believers have vigorously argued the merits of ADR based on intuitive
appeal, inappropriate straw dogs, limited conceptual rigor and poor evidence
These are also linked to an even more charged debate between proponents of
process versus proponents of agreements
 There is a compelling logical and some empirical evidence that
ADR will prove to deliver incremental benefit and be an effective
approach when used appropriately
21
CHALLENGE 1: INCREMENTAL
BENEFITS
 Disputes tend to be heterogeneous
•
•
•
•
Arises from some or all of: characteristics of the dispute, parties, practitioner, general
setting in time and place
Very challenging to find a counterfactual
Counterfactuals will always need to be unique to the case
Disputes often engage a variety of resolution options over their life
 Litigation is not the only alternative
•
•
When litigation is the likely alternative we can get some reasonable information
For many disputes the likely alternatives are: do nothing, legislation or administrative
rule making
 Addressing the challenge
•
The strength of the program theory, fact that claims made by the program theory are
not exceptional  “likely to” should suffice as an evaluation judgment of
incremental benefits
22
CHALLENGE 2: EFFECTIVENESS
 Costs and benefits occur in three phases:
•
•
•
Prior to the dispute resolution process. Many costs have already occurred
such as fact finding, establishing the technical, legal and economic claims,
identifying rights
During the dispute resolution process. Main costs are time of parties,
attorneys, practitioner, travel. Ability to pay can vary dramatically (e.g.
dispute involving a major utility, environmental groups and individual small
farmers)
After the dispute resolution process. The hypothesis is that a good
agreement will significantly reduce costs
 Shelf life of this information is relatively short
 The classes and characteristics of costs and benefits
have not been enumerated, nor weights or likely ranges
23
ADDRESSING THE CHALLENGES
 Incremental Benefits
•
•
•
Use litigation as an alternative where feasible
Undertake GOOD case studies to create illustrative comparators that can
be referred to
Use “likely to” approach in the interim
 Effectiveness
•
•
•
Case studies to establish the categories, ranges and coefficients
Include limited number of cost questions in the meantime
Manage the dance of “doing no harm” by not collecting bad cost data in
the interim
24
MY CLAIM
1. Getting the organization to focus on results generates
immediate and significant benefits
a) Outcome chart  program intent is clear = improved evaluability
b) 3 GPRA questions  requirement for monitoring outcomes =
performance information
2. Worst case  Improved evaluation
a) Utilization enhanced
b) Can report on outcomes
c) Credible information for program improvement
3. Better case  Organizations adopt an evaluative and
performance culture
25
CONTACT INFORMATION
Andy Rowe
GHK International
2707 Stratford Rd.
Columbia SC 29204
(803) 782-2182 voice (803) 782-3618 fax
ghkint.us@ghkint.com
www.ghkint.com
26