Assessments and the WB
Assessment Manual
Linn Hammergren
November 10, 2010
Purpose
• Discuss why assessments are important
(and what happens when you don’t do
one)
• Discuss assessment inputs
• Provide ideas on methodologies and
alternative strategies for doing them
• Outline a strategy for conducting an
assessment
Why assessments are important
• Because obviously you can’t resolve a problem until you
identify it and its causes
• Because the obvious answers to the two (implicit)
questions are often wrong
• Because an assessment allows you to understand the
broader context for an operation and the likely
impediments to its success
• Because an assessment provides a format for
discussing strategies with the client
• Because an assessment gives you baseline data and
thus a means of monitoring progress
Why assessments are not done or are
incomplete
•
•
•
•
•
•
They cost money
They take time
Bank thinks identification mission is sufficient
The client thinks they are unnecessary
They reveal inconvenient details
However, taking into account how long and how much it
takes to “prepare a PAD suitable for Board approval” the
investments in a good assessment are marginal.
• However inconvenient the truth of the matter, it is better
to find it early on than to have it emerge post-approval
Examples of findings an assessment might
discover
•
•
•
•
Belief (Mexico) that primary problem with money claim cases is delay to
judgment
– Finding: only 20% of cases reach judgment
– Finding: longer delays for enforcement
Belief (Mexico and many LAC countries) that judges are “overloaded with
work”
– Finding: annual filings often quite low and even when not, most cases
go “nowhere” and so require little work.
– Finding: much of caseload is unattended backlog
Belief (Colombia, Honduras) that new CrPC improves case processing:
– Finding: most cases reaching verdicts are in flagranti
– Finding: major crimes less attended.
Belief (Ghana and other African countries) that fast-track and commercial
courts are more efficient, but studies show low caseloads and delays
Inputs to assessments
• All prior diagnostics and studies – or at least the most important
ones. Many assessments ignore prior work (especially if not by the
WB), which is a waste of resources
• Discussions with the principal client (often the high court judges) as
to their perception of the “problem” to be resolved
• Discussions with other stakeholders to get their views, which are
often different
• Review of any existing statistical material, legal framework, history
of prior and on-going programs
• Fieldwork of various kinds (more on this later)
• In short, a good part of the assessment can occur before the
fieldwork and helps focus the latter. This saves time and resources.
A participatory approach?
•
•
•
•
•
This is controversial although Bank seems to endorse
The client is usually the host government or the judiciary, and both will have
a self-interested cut on what is wrong and how it should be fixed.
Participation is important for client buy-in, but in some cases may prove
overly restrictive – e.g. when client attends and thus censors all meetings
If the client is open to new approaches and inconvenient truths, it can be
very helpful, but if this is not the case, it can limit findings.
Still, to the extent possible:
– Clients should be involved in assessment design
– They should be invited to provide any material they consider essential
(and what they consider essential is also revealing)
– They should be involved in at least part of fieldwork
– They should be informed about progress and problems encountered
– They should be included in analysis and discussion of conclusions
Why not do a completely participatory
assessment?
• Even in the best of cases this is not likely to produce the
best results because:
– Local appreciations are local
– Participants lack the big picture (as regards their
“problems” and the likely solutions)
– Participatory planning encourages Christmas Tree
projects – an ornament for everyone.
• External experts are important because they have
broader experience, and if good can suggest a variety of
different interpretations and solutions.
• The wisdom of crowds is rarely technically based
A problem-oriented approach using institutional,
political economic analysis
• Can be as participatory as possible – techniques are not
incompatible
• To maximize utility for project design – if there are no
problems, an operation is unnecessary
• Problems can be defined by combination of CMU input,
literature review, primary client views, broad-based
interviews and team’s observations
• Initial problem definition tested and refined in later
stages
• Seek explanations (and remedies) through institutional
and political economic analysis
What is a problem?
• Defined here as shortcoming in desired outputs and
downstream impacts, but there is often confusion on this
point
• When judges, police are asked about problems, often
mention salaries, equipment, overwork
• Not the problems of interest here – best treated as
potential causes
• Challenge – to get “system operators” to redefine “their”
problems in terms of output/impacts. Can ask about
consequences of “overwork” for clients, for example
Institutional analysis looks at:
• Relevant state and non-state organizations and
actors
• Formal laws, structure, mandates, and
procedures; human and other resources;
territorial distribution and other factors affecting
use and outputs
• Real practices and informal rules
• Incentive structures – will have to be inferred
• How the above interact to shape outputs and
create problems or performance gaps
Political Economy
• Bank has shied away from this (or mislabeled institutional analysis
to this end)
• Political economy analysis (PEA) looks at power and access to
resources, and at intra- and extra-sector interactions to affect both
– Institutional analysis focuses on how organizational
arrangements influence behavior
– Political economy explains why – who benefits?
• PEA may be omitted from public report, but not from assessment.
– If politicians need compliant judges or are in cahoots with
criminals, this will affect the program
– Example – IADB citizen security projects in Caribbean neglect
ties between politicians and “dons” (gang lords).
• Bank need not engage in politics but should recognize its effects
Assessment stages and content – the Manual
•
•
•
•
•
•
Preparation -- scope, resources, timing, and team
S 1: Preliminary background
S 2: Preliminary problem identification
S 3: Field collection of information – institutional analysis
S 4: Summary overview and analysis
S 5: Comprehensive analysis of partial findings,
prioritization of problems/causes, identification of areas
for interventions
• S 6: Recommendations for Program
Preparation: Initial Considerations
• Initial definition of assessment scope by TTL, CMU, client
– Often targeted at institution or problem
– Team may need to negotiate some modification, especially as regards
incorporating more context (Court’s problem is not always the Court)
• Resources and timing --- usually decided (too?) early
– Both often underestimated, but there is never enough
– 2-4 weeks in field (with more time for writing) absolute minimum, but superficiality
can be combated with experienced team (knows country, knows methods) ,
access to documents, or with very limited focus, small country, etc.
– Even with large team, more time, lack of focus/organization can produce
ambiguous results
• Assessment team: skills needed; identification of local/foreign
experts; decisions as to inclusion of Bank and/or client members
• Norms, ground rules, roles: set by team and TTL
• Design and selection of methodologies
Stage 1: Preliminary background
• Get team up to speed on country and overall characteristics, history,
broader development problems
• Less necessary if use local experts or internationals with past
experience
• However, will be needed as part of assessment document, and also
to ensure common team understanding of where they are starting
• May be useful to have team meetings to discuss, especially if prior
exposure to country varies greatly
• Example: team for Honduras labor assessment (1 local, 1
international with country experience, 3 substantive experts with no
country exposure or language skills) – would have been helpful to
start here
Stage 2: Preliminary problem identification
• Start with working hypotheses based on:
– What people say (from interviews, written documents,
international assessments, opinion polls)
– Team’s initial observations – this is where international
experience counts as a point of reference
• Over course of evaluation, problem definition may change – as may
early guesses as to underlying causes
• Even if findings contradict conventional beliefs, latter should be
mentioned in study. What people think is important and will affect
what can be recommended
• Also useful to tie problems to groups mentioning them – not
everyone agrees on what is wrong or why (which is why reviews
should never be limited to judges and government authorities).
Stage 3: Data collection
• Some can be done prior to fieldwork– using
written or internet documents (including laws,
budgets, statistics) and out-of-country interviews
• Rest will inevitably be a mix of interviews,
observation, surveys, statistical reviews, and
whatever else the team can muster
• Aim – to do more than repeat what informants
tell you; test their versions with other data
sources and also explore alternative hypotheses
•
•
•
•
•
•
•
•
•
Methodology (Fieldwork and other Data
collection)
Assessments use multiple methodologies; several described in manual
Combine hard with soft data sources
Soft data (from interviews, observation) provide insights; hard data
(statistics, surveys) used to check and to generate hypotheses directly
No data source is perfect, and selection depends on what is available, time
and financing, what is being sought
Also depends on team skills – do they know statistics/surveys? Are they
experienced with specific topics?
Assessment may suggest further studies
Most studies will only focus on formal systems – this can be a problem
where most citizens uses informal ones.
Even if someone on the team has experience, legal-pluralism does not lend
itself to a fast assessment. (i.e. Countries with 100 or more local traditions?)
But, legal pluralism may not be amenable to international assistance either.
S. 3 Softer data sources
• Document review
– WB often overlooks prior studies, even its own – team should be
provided with this material
– Laws may not reveal reality, but they are important insofar as they
impede change
• Interviews --- will continue to be a major source of information
– Important to triangulate – multiple points of view
– Structured (set questions) can be quantified, but richer information comes from
unstructured. May thus combine two.
– Try to transcend client’s definition of interview list – need to reach further.
• Observation – a widely ignored source of information
– However requires skill and most probably wider experience
– A good court administrator or IT expert doesn’t need to know the language to
see problems
– However, a local and an international specialist may achieve much more
S. 3: Harder sources
• Surveys – might be considered “soft” as they are
quantified opinions, but in any event, seem to carry more
weight
• Aggregate statistics -- often not available and in any
case, usually limited
• Sampling – costly but revealing.
– However, samplers should be carefully monitored to ensure 1)
their sample is good and 2) their data are also good
– Can always treat as “illustrative” if quality is in doubt.
– In large, heterogeneous countries, what holds in one region may
not in another
S 3, Harder data, some further observations
• Common lack of good statistics is a finding in itself – and may lead
to recommendations on improvements
• Most systems only permit estimations of workloads, congestion, and
possibly resolution rates; for delays, case profiles, differential
outcomes need something more.
• Surveys, sampling, and related methods usually take more time and
money than allowed in assessment. Can take “illustrative” short cuts
– Quick review of random sample of filings from courts “of interest”
– DB style survey of users on delays, outcomes – but their
perceptions often don’t coincide with numbers (case of Mexico)
– If courts agree, can use staff for some data collection, but
supervision required and best to keep simple
– See Dilbert (next slide) on iffy data
A little more on hard data
S. 3: Less conventional data sources and
analysis
• Evaluating quality of judicial decisions – many have tried, but no
consensus on how to do it. Also all methods are labor-intensive,
time-consuming and ultimately, highly subjective
• Focus and study groups – also quasi-soft, but unlike surveys allow
more detail and discussion; useful for explaining statistics and
sampling results.
• Incentive analysis – more an art than a science and also the result
of other methods (focus groups, statistics, surveys). Some
interesting, if hypothetical conclusions:
– Mexico: lawyers are less interested in curbing delays than clients
– Generally – delay does not bother judges until someone (their bosses)
tracks it
– Generally – counting judgments encourages “cherry picking” and
manipulation of numbers
S 4: Summary Overview and analysis
• Conducted on the basis of initial information, before
fieldwork is completed
• Allows fieldwork to focus on further testing of hypotheses
-- may call for surveys, focus groups, or sampling
• Except in longer-term studies may not be possible, but
will at least generate areas “for further study.”
• If done, should orient collection of harder data – for
example, extent of and causes of delays; hypotheses as
to who uses courts and with what results
S. 5: Comprehensive analysis
• Initial list:
– Performance and impact problems
– Interrelated causes of each
– Initial ideas on solutions
• Prioritization:
– Of most significant or most urgent problems (can include
perceptions as well)
– Of causes, by susceptibility to change and broader impact
– Potential points of program entry with short and long-term
results, difficulty of addressing, sources of support and
opposition to change.
S. 6: Recommendations
• Assessment TOR may not require, but is the obvious last
step
• This and prior analysis should involve local stakeholders,
as if they are not in agreement, nothing will happen
• May be useful to hold workshops or seminars to discuss
tentative findings, conclusions, and recommendations
• Results and report content may consequently be
negotiated
• Quick fixes (tactical measures) and longer term reforms
(overall strategy)
Assessment Do’s and Don’ts
• There are no recipes, but have a plan, tentative organization, and
role assignments even if they will change along the way
• Go beyond “descriptive studies” –an overview of the parts and how
they fit together is useful, but assessments must do more than this
and when they don’t, counterparts complain. Save the details for the
Annexes?
• Likewise, assessment should not be a legal review – often a
problem of local experts with little research experience
• Do try for a multi-disciplinary team – to spread the perspectives
• Mix internationals and locals – they each have their use
• Beware, however, of expert bias – trainers inevitably recommend
training (even if not a priority) and IT experts will want systems;
generalists with some substantive knowledge may be preferable,
saving the dedicated experts for the identification mission.
Assessment Do’s and Don’ts (cont.)
•
•
•
•
•
•
•
•
Triangulate sources and mix methodologies to test “what everyone knows”
Avoid sending multiple team members to interview one person
(simultaneously or sequentially) – a waste of time and can intimidate
informants. “Round robin interviews” may work better with groups.
Take informant claims seriously, but not as the unvarnished truth
Protect informants and verify/respect their wish to be identified or not.
Gather all relevant information, vetting for inclusion later – don’t censor
along the way. (Consider multiple versions of report?)
Encourage team meetings on findings and intermediate conclusions – this
can reveal gaps, contradictions, and synergies.
Be prepared to leave things out of the report (not only for sensitivity but for
limited relevance). Data collection is a process, but the report does not
need to repeat all the steps.
Recognize areas for more research later, noting why they are important and
how they might be explored.
Assessment is an iterative process
• The first cut is only that
• Assessments should include provisions
and recommendations for further studies
• However, assessments should be used –
and in many cases they remain on the
shelves
• Can also be useful in motivating clients,
other local participants to find out more.