SOCHA
Combining Outcome Harvesting/QCA for
Evaluating Stochastic Processes of
Bureaucratic Collaboration and Action
30 September 2016
Presentation by
Carroll T. Patterson, PhD
Partner at SoCha,LLC
SOCHA
What is this about?
• Testing the “Collaboration” Hypothesis
• Applying Outcome Harvesting to Capture Stochastic Outcomes
• Feeding those outcomes into a QCA Model
• Lessons Learned about the Collaboration Hypothesis and OH
SOCHA
The District Operational Plan
• What is it?: A quarterly meeting (3hrs) involving District Technical
Reps, Reps and Implementing Partners across 3 substantive areas:
Health, Economic Growth, and D&G
• What Happens?: Members update on implementation progress,
discuss issues, exchange information, create action items
• When and Where?: From March 2012 – August 2015 (16 meetings)
across 19 districts in Uganda (16x19=304 meetings), involving 44 IPs
for 82 projects.
• So What?: The Collaboration Hypothesis, Next Slide
SOCHA
Problems with Evaluating the DOP and Testing the
Collaboration Hypothesis
• Boxes in the causal chain are not discrete, well defined, easily
measured, and…. “indercatable”
• “Logical Leaps” in the Theory: e.g. how do Q meetings at the district
level reduce HIV Prevalence, Teacher Absenteeism and Malnutrition?
• Didn’t have the “Ring”: One impact indicator to rule them all, find
them, bring them all and in the log frame bind them
• Pinned the End Result of Collaboration to “Hard Outcomes”
• Conclusion: Cannot test the collaboration hypothesis with project
data
SOCHA
Solution: Outcome Harvesting
• Useful tool when the outcomes aren’t at first clear, unpredictable,
varied and difficult to quantify.
• Good set of guidelines for when we have to dig deeper to uncover the
effects of our interventions.
• Outcomes are defined as changes in the “behaviour writ large” (such
as actions, relationships, policies, practices) of one or more social
actors influenced by an intervention.
• 6 Steps: Design, Review, Engage, Substantiate, Analyze, Support Use
• Design: Harvest Outcomes from all 19 DOP districts + 5 “Control”
districts; 2.5 weeks of field collection: 4 international/4 national staff
SOCHA
The Rules of Engagement: Fight the FLUFF!
To fight fluff, we imposed the following parameters on the investigation:
Outcome: Change that can be connected to the DOP meeting but occurred
outside of it.
Tangible: Must pass the who, what, where, when, how test to count…if
not, either classed as “follow up” or “FLUFF” (happened a lot)
Define Results Chain: Identify steps from meeting to outcome. *Note: We
tended to work forwards instead of backwards
Counterfactual: Would the outcome likely have happened without the
meeting. No = Concrete Outcome; Yes = Potential or Non-DOP outcome
Knowing what each project was doing here was key!
Significance: Why is the outcome important? (didn’t find so useful)
SOCHA
Triangulation and Quality Control
Substantiate the Outcome: Two Questions, launched both at Field and HQ
Do you agree with the outcome?
Do you agree with the contribution?
Daily Calls to Field Team: Very easy to lose clarity, drift off point, get lost
in the fluff in the beginning
Immediate Capture: Day on/Day off interview schedule to allow for
immediate write up
Triangulation:
• 105 GoU Officials
• 2 FGD with staff; 30 IP interviews, 145 substantiations (18%
contested, revised), online survey of 35 IP
• Reviewed attendance records, action items, and presentations for all
304 DOP meetings
Documentation whenever possible – link to action items
SOCHA
Collation, aka “Bucketing”
• Harvested 211 outcomes (75 concrete, 74 potential, 24 unattributable, 38 in control districts)
• Organized these according to “buckets”
• Rule = must be discrete categories, i.e. one outcome cannot fit into
two buckets
• Several iterations = define categories, bucketed by separate teams
• Led to 4 “Master-Buckets,” or concepts: Coordination, Collaboration,
Formal Action, Hard Outcomes
• Results: 15 Buckets, 4 meta-concepts
The Harvest
SOCHA
Master
Outcome Bucket
Total
Concrete
Potential
Non DOP
Outcomes
Control
Districts
Coord
1=Outside DMC Reporting
26
9
5
6
6
24
11
8
4
11
2
1
0
4
5
10
7
2
0
1
14
4
3
3
4
24
7
13
2
2
18
5
7
2
4
7
8
8
29
11
5
0
4
12
2
2
4
3
12
4
0
4
0
2
1
0
0
1
3
4
7
4
2
0
1
9
5
211
24
3
1
75
19
4
0
74
19
1
2
24
7
1
2
38
5
8.8
3.9
3.9
3.4
7.6
2=Duplication Avoided
3=District led coordination
4=Collaboration between IPs (not across
DOs)
5=Monitoring (Joint Field Visits)
6=IPs Change Implementation Based Upon
Collab
Need
7=District and IPs collaborate to conduct
field activities or trainings
8=Cross sectoral collaboration (not IP to IP)
9=District directs IPs where to implement
10=District arbitrates stakeholder conflict
11=District changes their process
Formal
12=Policy (New or Changed)
Action
13=District offers new/expands existing
series
14=Availability/quality of services
Hard
Outcome 15=More responsive/targeted investments
Total
Number of Districts applicable
Average
SOCHA
Revised ToC: Stochastic Outcomes
Increased Returns on
Investment involve
increased levels of
complexity;
Yet the connection to DOP
is weaker as complexity and
uncertainty increase
Implications:
“Unbox” the outcomes let the messiness unfold
and empirically define the
categories after the
outcomes unfold
SOCHA
Analysis
• Costing: $6,548 per meeting; Calculated Unit Cost per Outcome Type
using frequency distribution of outcomes
• E.g. It took at average of 9.2 meetings to avoid duplication, thus
costing $60,310.53
• Testing Perceived Leadership Thesis: There is a correlation
Correlation of Perception to
Outcome
Pearson's
Significance
• QCA: Used to Explain Success
Total
Concrete
0.762
0.001
0.514
0.024
SOCHA
QCA: Explaining Success
• Success Threshold: At least 7 outcomes (same as non-DOP); 12
successful DOP districts.
• Factors:
• Action Item Performance: at least 33% were “substantive”; 13
districts
• IP Attendance: 50% average attendance; 9 districts
• Rep Attendance: 33% (no one met 50%); 10 districts
• Workplan Sharing: 80% IPs shared every Q; 4 districts
• Perceived Success: 10 districts
• Grant Disbursement Rate: 80% performance; 14 districts
• Questionable Cost Rate: no more than 20%; 13 districts
• Sector Q-Meeting Performance: 80% performance; 6 districts
SOCHA
QCA Results: Substantive Actions Matched by
Spending
• Success= Substantive Action Items AND High Disbursement
Rates (Necessary Conditions)
• Explanation:
• Substantive action items reflect shared commitment to jointly making
change. BUT – meaningful collaboration in and of itself is insufficient for
action, and requires significant capacity and resources to move forward.
• Note: Although Leadership had a high correlation with higher
outcomes; it was not a necessary condition for success
SOCHA
Conclusions About the DOP Collaboration
Hypothesis
• Low Performing DOPs fell into a downward spiral
• Procedure trumped substance 2/3s of the time
• DOP Coordination was expensive but did not outperform non-DOP
districts in terms of outcomes
• Softer outcomes - e.g. sub-meetings, additional reports, more
monitoring - trumped hard outcomes. Hard to Soft Ratio was 1 to 17.5
• Qualifications:
• Control groups are not representative
• Outcomes are Stochastic and can’t use these results as reliable
predictors
• Engagement is a worthy end in and of itself
• Result: End of DOP
SOCHA
Lessons Learned about OH
Pros
• Very Useful to Capture “Stochastic Processes of Change,” such as
bureaucratic outcomes and
• Practical Approach Based upon Sound Principles
• Push Past the Fluff! (Resistance by Respondent to being Trained)
• Allowed us to MODEL!!!
Cons
• Less Rigorous – Lacked Clear Decision Rules, we filled in the blanks
• Hard fit w/typical Eval timelines and scopes; harvesting takes time
• Resource Intense in Rural Environments: High-End Enumerators
Qualifications
• We rushed the process – not enough time/resources for adequate
“results chaining”, time stamping and substantiation. Moderate bias
towards quantity over quality, so we could model
SOCHA
Thank you!
For more information, Contact:
Carroll Patterson, PhD
Partner at SoCha,LLC
cpatterson@socha.net
www.socha.net
Mauritius:
Thailand:
USA:
Uganda:
+230 5733 7713
+66 838 856 311
+1 302 476 24 20
+256 763 606 311