Evaluation: GRANITE Project
Lucknow, December 19, 2006
Presented by:
S.V.Divvaakar and Pankaj Agrawal
Ace Global Private Limited, New Delhi
Presentation Outline
•
•
•
•
•
Evaluation Objectives
Approach and Methodology
Evaluation Frame
Key Findings under evaluation themes
Conclusions and Recommendations
Evaluation Objectives
• To assess the overall results and
impact of the project at national and
sub-national level
• To assess the need for second
phase of the project
• To provide inputs for design of
Phase 2
Approach and Methodology
• Briefing meeting and participation at
inaugural event of KIC initiative at Jaipur
• Document perusal and secondary research
• Field visits in 4 states: Tamil Nadu, Andhra
Pradesh, Maharashtra, Orissa
– Interviews with
• SRG members
• Media personnel
• Civil Society Organisations
– Visits to selected locations of Reachout
meetings, and discussions with participants
Approach and Methodology
• Interview visits also to Kolkata and Jaipur NCU and CART, and telecon with
Karnataka state partners
• E-mail questionnaires to all state partners
Evaluation Scope
• Five classical elements
• Relevance
• Efficiency
• Results:
– Effectiveness
– Outcomes/Impacts
• Sustainability
• External challenges and constraints
Evaluation Frame: Definitions
Relevance
Appropriateness in relation to
policies, needs, priorities
Efficiency
Productivity of implementation, and
cost effectiveness
Effectiveness
Achievement of target objectives/
outcomes
Impact
Positive and negative effects
caused by the intervention
Sustainability
Continuance of positive effects after
the withdrawal of the intervention
Evaluation Frame: Relevance
What to measure
Appropriateness in relation to
policies, needs, priorities
Whose perspective Beneficiaries
Point of reference
Needs and priorities of donors
and partners
Key questions
• Are objectives aligned to
needs and priorities?
• Should direction be
changed?
•Should activities be
continued?
Findings on Relevance
• Project is highly relevant to the setting.
• A large share of the population lives marginalised
from India’s economic growth, and is engaged in
informal employment, in agriculture and artisan
trades.
• Benefits of economic growth have not percolated
to the weaker sections, for various reasons, incl:
– Lack of awareness of trade and economic issues that
directly or indirectly affect their livelihoods
• Mainstreaming these stakeholders is a key
challenge for India’s economic growth and social
development.
• Improving economic literacy at grassroots level
will enable stakeholders to see the links between
their livelihoods and economic developments,
including those related to trade and globalisation.
Findings on Relevance
• Sectors are well-chosen:
– Agriculture and textiles are the two largest employment/
livelihood sustenance sectors in India.
– Agriculture accounts for 23% of GDP, and 10% of exports
– Textiles and Clothing is the second largest employment
creating sector
– More than 2 million households are engaged in handloom
weaving.
– Trade and globalisation especially WTO agreements have
impacts on both sectors.
• Market access constraints for agriculture exports
• Elimination of market access preferences in handlooms
• States:
– Regional balance
– The eight states selected for the project have sizeable
agriculture sectors, and some have sizeable handloom
textiles sectors.
– Three states have very low export intensities in the sectors.
Findings on Relevance
• State Partners are well-chosen:
– Grassroots presence
– Convening power and credibility
– Some have a history of partnership with
CUTS
• However, varying levels of conversancy
on sector-specific livelihood issues
• Most have limited exposure to trade and
globalisation issues, first exposure for
some…
• Limited budgetary resources
Findings on Relevance
• State Consultative Mechanisms:
– Mechanisms for government consultation with
grassroots stakeholders either do not exist or
are dysfunctional.
– WTO Cells have the mandate to represent
issues directly related to trade and
globalisation but are not carrying out the
mandate due to lack of resources and clarity
– There is a need for enhanced advocacy
based on
• documentation of grassroots impacts/experiences,
• increased articulation of concerns
Evaluation Frame: Efficiency
What to measure
The delivery of project inputs
and assistance
Whose perspective Implementing partners
Point of reference
Best practice standards
Key questions
• To what degree have
components been delivered?
• Could it have been done
better? more quickly?with less
costs?
Project Activities
• Project used a wide range of tools for its
actions.
– Capacity building- training of implementation
partners and media persons
– Reach out meetings with final beneficiaries
and stakeholders, and state level workshops
– Information dissemination: website, news
letters, research reports.
– Advocacy with state and centre government
Project Activities
•
•
•
•
•
•
•
•
•
•
•
•
National launch and training seminars
State Project launch meetings
State Reference Groups
Reachout meetings
Media workshops
State level workshops
Advocacy actions through TPCs, WTO Cells
Briefing papers
Research document
Newsletters
Advocacy document
Website
Launch and Preparatory Activities
• National Launch and Training seminars
– 1st Seminar, Jaipur 4 days
• To provide training on globalisation/ WTO subjects, based on
base line questionnaire analysis
– 2nd Seminar, Kolkata 2 days
• Originally to focus on skills to analyse and articulate sector level
issues
• Lack of continuity in participants
• State Project launch meetings: one day panel discussion,
20-100 participants
• State Reference Groups: 10-108 members
– Original role - to serve as sounding board and monitor
qualitative aspects.
– However, no budgets for structured activities
Field Activities - Reach out meetings
• Meetings
–
–
–
–
–
5 outreach meetings planned per state
3 states have held 7 meetings each
3 states have held 4 meetings each
20-350 participants (U.P, A.P, W.B with most)
No textiles coverage in Maharashtra, Orissa and Karnatakaconscious decision with NCU
• Lack of product/cluster/issue specific focus especially in
agriculture sector led to many generalised events, with
little direct connection to livelihood issues for participants.
• Non-homogenous groups attended most meetings.
• Main issues identified in outreach meetings have a
domestic dimension- remunerative pricing, cost of inputs,
lack of extension services, exploitation by middlemen and
money lenders, unfair competition from organised sector,
etc. These were not, always, international trade aspects.
• In some cases, there was good scope for advocacy
actions addressed at local authorities…
Field Activities- Reach out meetings
• In some locations, specific actions have been
undertaken which enhance livelihood prospects
for the beneficiaries:
– Maharashtra: pomegranate crop protection, EurepGAP
certification
– Uttar Pradesh: Market interventions in mango; GI
registration for chikan kari,etc.
• Main issues identified in outreach meetings have
a domestic dimension: remunerative pricing, cost
of inputs, lack of extension services, exploitation
by middlemen and money lenders, unfair
competition from organised sector, etc.
Field Activities- Media Workshops
• Originally two workshops planned, 1 per year
• No media workshop in Orissa, 2 in A.P, U.P
• Objectives: orientation and training of media persons
on WTO issues, to ensure enhanced reporting
• However, these seem to have been diluted, and
several workshops were very generalised and
unstructured, and lacked specific training content as
such
• Media persons’ interest in training is suspect, but
there has been willingness to publish articles
contributed by GRANITE partners. Good media
coverage of GRANITE in all states.
• Media workshop budgets underutilised, due to low
participation by outstation persons
Field Activities - State Level Workshops
• Originally two workshops, 1 per year
• Objectives: better targeting, wider reach out and
networking
• Not held in 3 states: Tamil Nadu, Maharashtra,
and Orissa; while U.P held two workshops.
• Lack of clarity on objectives reported by many
partners; also low credibility before govt officials
• Focus of workshops not in line with the project
document, and too general.
• Sector-level workshops proposed in 2nd year. Not
held in many states.
• Sequencing of state level workshops not
adhered to.
Field Activities- National Seminars
• Two seminars, one at end of each year. First
event in Nov 2005, Jaipur, took stock of lessons,
issues and problems faced by state partners.
• Second seminar: December 2006, Lucknow, this
one…
• Need to document collective learnings and
develop advocacy messages for policy makers
Dissemination Products
• Briefing papers
– Two briefing papers produced, two more planned (December
2006?).
– Not used as resource materials for reachout meetings or
media workshops.
– Poorly distributed in most states.
• Newsletters
– Quarterly issues, 2000 copies in English, 1000 in regional
languages; i.e. 8 issues, 10,000 copies
– 6 published in English, only 3 to 5 in states.
– Poor levels of distribution (only 100 copies in English…)
– Contents appreciated by recipients incl. SRG members
• Research document- released at WTO Hong Kong meet
– Conclusions are debatable.
• Advocacy document- yet to be prepared
• Website- useful information on project; updates needed.
Gender dimensions
• Project management team had
atleast one lady member (NCU and
7 states)
• Special efforts to mobilize
participation by women stakeholders
in reachout meetings
• Representation on SRGs
Advocacy Actions
• State Trade Policy Councils/ WTO Cells
– Idea of STPCs dropped in view of existing WTO cells
in many states.
– State of WTO cells highly unsatisfactory
– Varying levels of interest in the project
• National Trade Policy Council
– Inter State Trade Council formed following active
representation by CUTS, but not attributable to
GRANITE as such.
• Representations on the Foreign Trade Policy
– Pro poor changes in the form of Focus Products and
Focus Markets incorporated in 2006, again on CUTS
representation, but not flowing from GRANITE activities
as such.
Efficiency- Adequacy of Resources
Inadequate
More than Adequate
Outreach meetings
Project state launch
meetings
Honorarium for nodal
person
Media workshops
Intra state travel
State level workshops
Efficiency- Financial
• Total Project budget: Rs 14.33 million
• Budget for each state: Rs. 1 million
• Budget for centralised expenditure by
NCU: Rs. 6.36 million
– 44% of total budget
• Allocation for each activity specified in the
TORs
Project Budget and Distribution
Overhead
costs
5%
Program
activies
69%
Personnel
costs
26%
Project Budget and Distribution
Allocation (Rs. Lacs)
60
54.53
50
44.4
40
30
30
20
10
7.2
1.92 5.28
0
Personnel costs
NCU
Programme
activities
Overheads
State partners
Efficiency- Disbursements
• Initial disbursements on time
• Subsequent disbursement have been slow
• Only 2 instalments had been received by some
partners, with more than 6 month delays
• Delays mainly due to non-completion of all
previous period activities
• Absence of flexibility to disburse according to
state of completion under individual heads
• Procedures can be improved in Phase 2, to
provide for disbursement to continue all onschedule actions
Efficiency- Cost-effectiveness
• Overall, actions have been cost-effective
• The project has influenced close to 5,000
beneficiaries, @ Rs 2860 per head; even a Rs.
100 / month increase in livelihood means/
incomes justifies the expenditure.
• Unit costs for personnel and events are very
reasonable.
• However, budgets for advocacy networking and
representation are concentrated on international
travel – considered sub-optimal, inappropriately
oriented.
Efficiency- Administration & Coordination
• NCU has undertaken role of a Mother Unit, and
taken a lead in some actions
• There has been backstopping for state partners
in several areas.
• Some partners expressed that NCU should have
provided more training content support and
integrated the lessons from outreach meetings
into learnings at project level.
• Project financial and administrative reporting to
donors has been timely
Factors affecting implementation
• Internal Factors
– Staff constraint/turnover
– State partners lacking in-house expertise /
sector conversancy
• External Factors
– Inadequate engagement with Government
bodies and policy makers
– Limited interest of local media
– Elections in Tamil Nadu: delay in State level
Workshops
– Heavy rains affecting participation in State
launch workshop in Maharashtra
Evaluation Frame: Effectiveness
What to measure
Achievement of objectives
Whose perspective Target group of Beneficiaries
Point of reference
Agreed objectives/ outcomes
Key questions
• To what extent have agreed
objectives been achieved?
• Are activities sufficient to
realise agreed objectives?
Effectiveness- outcomes
• Most target results have been achieved.
• Out of 5 target outcomes, evaluators consider
one to be an output and not an outcome.
• Of the remaining four, there has been partial
success in one, and substantial success in
three, of which one is not considered attributable
to the project itself, although achieved by CUTS.
• Project needs more specific indicators for
results, which are measurable at the level of
beneficiaries.
Effectiveness- Target Outcomes
Eight CSOs capable of analysing,
articulating and advocating on issues of
globalisation
Substantial
success
Establishment of National Trade Policy Partial
Council and at least 6 State Trade Policy success
Councils
Two pro-poor changes in Foreign Trade
Policy 2004-2009
A well-established national
network of CSOs on globalisation issues
Not
attributable
Substantial
success
Study document on the impact of WTO
rules on the chosen sectors in 2 states
Output, not
Outcome
Effectiveness - Other outcomes
• Capacity enhancement of partners
• Increased articulation by grassroots
stakeholders
• Development results for beneficiaries:
– Enhancement of livelihood opportunities
Evaluation Frame: Impact
What to measure
Intended and unintended
positive and negative effects
Whose perspective Society, final Beneficiaries
Point of reference
Pre-intervention baseline
status
Key questions
• What are the positive and
negative effects?
• Are the positive effects more
than the negative effects?
Potential Impact
• Realisation by grassroots stakeholders
that their livelihoods have important
linkages with domestic policies and
international covenants.
• Multiplier effect of the experience gained
by partners in creating a stakeholders’
forum for analysis, articulation and
advocacy of grassroots concerns
Evaluation Frame: Sustainability
What to measure
Likelihood of continuation of
benefits after the intervention
Whose perspective Society, final Beneficiaries
Point of reference
Projected future situation
Key questions
• Are the partners willing and
able to continue activities and
maintain facilities on their
own?
•To what extent does the
impact justify the investment?
Sustainability
• Financial sustainability difficult for some
partners, as GRANITE accounts for large
share of their budgets
• Concerns as to continuity of key persons
after project support is withdrawn
• Institutional structures created by the
project- SRGs, news letters etc. can still
continue with support from CUTS
• Mainstreaming of agriculture and textiles
in partner’s core activities
Key gains from the project
• Reachout meetings acknowledged as an
effective forum for grassroots stakeholders
• Tangible and attributable benefits for
stakeholders through increased awareness
and understanding of trade issues
• Increased reporting in local media on trade
and WTO issues
• Initiation of interface between grassroots
stakeholders and the policy makers
Key gains from the project
• Enhanced capacities of state partners in
understanding, analysing, articulating and
advocating on issues related to trade and
globalisation/WTO issues
• Strengthening of network of civil society
organisations that interact and share
research and capacities /skills on issues of
globalisation
Conclusions and Recommendations
• Project Emphasis:
– Grassroots stakeholders are rather remote
from trade and globalisation, and their
livelihoods are linked more to domestic trade
issues than WTO. Project title tends to
mislead.
– Amend the project emphasis to ‘Trade,
Markets and Livelihoods’, which gives it a
broader scope, including globalisation.
Conclusions and Recommendations
• Sector Scope and coverage:
– Agriculture and textiles are large, complex
sectors with several products/segments, all
not having similar pro poor dimensions. There
are diverse issues relating to trade in each
product/ segment.
– Narrow the focus of the interventions to a few
products common to most states, and a few
cross cutting themes such as:
• Market Structures
• Remunerative Pricing and
• Producer Empowerment
Conclusions and Recommendations
• Sector Scope and coverage:
– Project partners do not have adequate level
of expertise or skills in both sectors, which
skews the thrust of actions across various
states.
– Partners should have the flexibility to select
only one sector, based on conversancy
– More than one partner may be inducted in
such cases
– Narrow the skills gap by inducting sector
specialists to support implementation
partners
Conclusions and Recommendations
• Delivery and execution:
– Sequencing of events has not been followed in some
states, resulting in sub-optimal use of project
resources. Contents at many meetings were too
general and not relevant or specific to local
stakeholders
– There should be rigid adherence to the logic of
progression: outreach meetings - media workshops –
state level workshops
– Adequate prior preparation should precede field
actions, enriched by SRG and external experts
– One outreach meeting is inadequate to achieve any
meaningful transformation at the grassroots level
– There should be at least two outreach meetings per
location, including a follow up at the end of the project
Conclusions and Recommendations
• Delivery and execution:
– Outreach meetings tend to have heterogeneous
participants, covering several products and sectors,
with few issues in common
– Locations and themes should be selected to ensure
homogeneity of stakeholders and also issues to be
discussed in outreach meetings
– There should be an integration of learnings from
various meetings, facilitated by experts and the NCU
– Media and government stakeholders workshops did
not focus on capacity development specifically, nor
was it in demand
– Project should not fritter scarce resources on these
groups considering their other sources of support, and
focus only on advocacy
– Advocacy should be bottom- up, focused at local levels
of government as well
Conclusions and Recommendations
• Delivery and execution:
– Involvement of SRG resource persons was
minimal, due to budget constraints
– A small inner list of SRG persons should be
inducted to act as a sounding board and for
monitoring quality of delivery of the project.
Budgets for their involvement should be
included
Conclusions and Recommendations
• Budget allocation and utilisation:
– Overall, budgets should be enhanced for
outreach meetings, intra-state travel and
remuneration to key persons
– Fungibility in budget items should be allowed
within limits, based on specific justification
Conclusions and Recommendations
• Capacity enhancement:
– Focused training on sector-specific trade and
globalisation issues could not be provided.
– Formal training on sector-specific trade
issues should be included in the project.
Specialist resource persons should be
inducted to bridge knowledge gaps among
partners. Some examples already seen in Phase I.
– Second outreach meeting should be used for
identifying advocacy actions in consultation
with stakeholders
– SMART (specific, measurable, attributable,
realistic and time-bound) indicators should be
developed for project actions.
Thank You
www.aceglobalonline.com