Annual Review - Summary Sheet
This Summary Sheet captures the headlines on programme performance, agreed actions and learning
over the course of the review period. It should be attached to all subsequent reviews to build a
complete picture of actions and learning throughout the life of the programme.
Title: Pakistan: Education Voice and Accountability Fund (Ilm Ideas)
Programme Value: £ 5 million
Programme Code: 202233
Review Date: 20-24 October 2014
Start Date: 15
November 2011
End Date: 14 May 2015
Summary of Programme Performance
Year
2012
2013
2014
Programme Score
A
A+
A+
Risk Rating
Medium Medium Medium
Summary of progress and lessons learnt since last review
The Education Voice and Accountability Fund aims to foster public demand for increased
responsiveness, accountability, and transparency to improve educational access, quality, and
governance at the local and district levels. The programme is currently supporting 25 grants covering 56
districts across Pakistan.
This third Annual Review covers progress made in the first three quarters from January-September,
2014. The milestones for outputs and outcomes are cumulative and represent overall results achieved
since the beginning of Ilm Ideas in January 2012. Over the review period, the VA Fund accelerated
momentum on research and advocacy through 18 live grants of which 14 were awarded after the
previous Annual Review.
Progress on outcomes
The Annual Review team assessed the progress on intended outcomes to be good:
Outcome 1: establish models for
articulation of citizen voice to
hold politicians and officials to
account for delivery of quality
education
Progress: so far, grantees have contributed six VA models,
against a target of eight. Examples include:
- accountability of local politicians based on ranking of
electoral constituencies on education;
- election and mobilization of tehsil and district-level school
councils, school expenditure tracking, and advocacy for
home-based schools for girls.
- Gilgit-Baltistan government, influenced by Association of
Global Humanists and Ethics (AGHE) six home-based
schools supported by the VA Fund, has adopted the
model and established 75 home-based schools in which
over 3,000 girls are getting an education.
Outcome 2: administrative and As of 30 June 2014, 203 administrative or policy decisions had
policy decisions made by duty been made to which grantees contributed through advocacy
bearers
and influencing – overachieved by a high margin against the
1
target of 150. The action taken as a result of these decisions
should be further verified by an external party. A
recommendation is made below.
Progress on outputs
Progress on outputs exceeded expectations and sought to enhance community voice and collective
action to demand better education, the capability and responsiveness of duty bearers1 (mainly district
government officials) and availability of evidence. Consistent with previous Annual Reviews, most of the
milestones have been exceeded by a high margin. Key achievements include: establishment of 251 new
forums for citizen voice (e.g. school councils, village education committees, child clubs); provision of
information, capacity and mobilization of over 18,365 people to demand quality education, and written
submission of 1,117 demands to duty bearers. The grantees engaged over 900 duty bearers, including
members of provincial and national assemblies, members of standing committees on education and
education officials.
Three key lessons are:
1)
The uptake and sustainability of voice and accountability activities and efforts should be built
into individual grants from the beginning to the end. While the grantees have made good
contributions in establishing and mobilising new forums for participation and advocacy at local
level, their replication and scale-up for sustainable impact received insufficient attention and is a
key challenge after the closure of grants.
2)
VA grants should strike a balance between research and advocacy, incentivizing the grantees
to invest early on in building relationships with duty bearers (mainly district government
officials). Almost all VA grants under Ilm Ideas included research as well as advocacy. However, a
disproportionate focus on research has left little time for mobilization, collective action and
advocacy. Thus, many grantees were unable to use the evidence or funding properly to articulate
their demands to duty bearers. A lesson for both DFID and DAI is to ensure that future research
and advocacy programmes consider integrating advocacy and its likely impact from on the outset
by establishing advocacy baselines at logframe development stage.
3)
A robust monitoring and evaluation (M&E) system is extremely important to measure results
from VA interventions and demonstrate value for money. This should build on the current
processes in place for measuring the effects of the programme on different beneficiaries,
measurement of qualitative results, and verification and evaluation. This should involve
adequate and proportionate resource allocated to physical verification of reported results and
evaluation from the beginning.
Summary of recommendations for the next year
The VA Fund is scheduled to close in May 2015. However, DFID is in the process of contracting a second
phase of the Innovation Fund. The project, titled Ilm Ideas II, is planned for inception in early 2015 and
will incorporate lessons learned from this programme. The remaining six months are crucial for
maximizing the sustainability of grantees’ work, embedding activities, transitioning any innovative VA
1
Duty bearers are individuals or departments responsible for service delivery. In the context of the VA Fund, the
term is strictly applicable to government officials, which includes all public sector representatives under the
education departments or ministries.
2
work into Ilm Ideas II, ensuring the timely closure of grants and preparing for the Project Completion
Report. Key recommendations are:
1)
Implement an exit action plan (Plan ready by end January 2015, and implementation from
February 2015): The plan should be brief and consist of practical actions for closing down the
existing programme. We recommend that the plan should be prepared internally by the Ilm
Ideas team, and consider actions for: (1) building grantees’ capacity to maintain positive
activities beyond the life of the programme; (2) actively identifying ways to amplify the work of
advocacy forums and coalitions formed under VA grants; (3) establishing an interface between
people mobilized in districts and Network Coordinators working under DFID’s Transforming
Education in Pakistan (TEP) education advocacy programme; (4) recommendations for
transitioning of successful, innovative and high performing projects into Ilm Ideas II; and (5)
strong risk management to ensure timely closure of live grants.
2)
Agree a clear plan with DFID on the utilization of uncommitted funds: (December-January): As
on September 31 2014, funds worth £491,6622 have yet to be committed. We recommend that
part of these funds should be used to increase results from high-performing live grants through
cost extensions. However, this will require strong risk management to ensure the quality of
deliverables and closure of grants by end April 2015. A no-cost extension is therefore proposed
to manage this risk and ensure that any new grants commissioned at this stage can yield
meaningful results.
3)
Carry out a third-party verification (TPV) of key Government policy and administrative
decisions (i.e. outcome 2) for lesson learning (February-April): 203 decisions made by
Government with contributions from VA Fund partner organizations is a key achievement but
remains unverified. Verification of key Government decisions is crucial to understand what works
in demand creation and learn lessons for the future direction of voice and accountability work in
the DFID Pakistan Education programme. A TPV should be commissioned for this purpose, which
should report directly to DFID.
4)
Focus on the dissemination and opportunities for uptake of research produced (DecemberApril): Ilm Ideas need to proactively increase dissemination of research reports using policy
briefs, and awareness materials. Part of the uncommitted funds could be used for this purpose.
5)
Develop a VfM framework for voice and accountability activities. In addition to identifying and
sharing learning about effective voice and accountability activities in Pakistan, Ilm Ideas should
develop a framework for assessing and applying good VfM in voice and accountability projects.
This will be a useful tool, transferable to other programmes and contexts.
6)
Consider extending VA programme end date. Subsequent to the Annual Review, DAI and DFID
have agreed a no-cost extension to extend the VA programme until October 2015, in line with
the closure of the Innovation Fund. The extension will allow to mitigate some of the delivery
risks and will also provide sufficient time to take forward the recommendations of this AR.
2
The total uncommitted amount comprises of £110,157 savings from the closure of grants and the remaining grant budget of
£381,505. ILM Ideas signed a special grant with Free and Fair Elections Network (FAFEN) in mid November 2014 for £279,535
which resulted in bringing the total unobligated funds down to £212,127.
3
A. Introduction and Context (1 page)
DevTracker Link to Business Case:
DevTracker Link to Log frame:
Pre-Business Case
Project
Logframe updated
periodically therefore
not declared as a
record.
Outline of the programme
Ilm Ideas (learning ideas) is the overarching brand name for two funds; an Education Innovation Fund
and Education Voice and Accountability Fund (henceforth the VA Fund), both established in 2011 and
due to close in 2015. Development Alternatives International (DAI) is the service provider for Ilm Ideas
and the original contract value was £8 million for both Funds. Following scale-up funding in 2014, the
programme is now £11 million in total split: (a) £5 million for the VA Fund; and (b) £6 million for the
Education Innovation Fund. Through DAI, DFID provides grants and funds technical assistance for a
range of research, pilot and scale-up activities which: (a) enhance access to primary and secondary
schools; (b) improve the quality of learning in schools; (c) improve learner and parent satisfaction; and
(d) improve governance of the education system. A joint Annual Review of both funds was carried out in
October 2014. This report focuses on the VA Fund.
Expected results
The desired outcome of the VA Fund is citizens’ voice enhanced for improved service delivery and
accountability in the education system. The outcome targets to be achieved by May 2015 are as
follows:


Outcome 1: 10 functional models which enhance citizen voice for demanding improved education
services; and
Outcome 2: 160 administrative and/or policy decisions made by district and provincial authorities
for enhanced education services.
Achievement of these outcomes is envisaged through:
 Bolstering the role of civil society, citizen groups, media and private sector in creating new or
reinforcing existing platforms for parental voice;
 Generating new evidence;
 Mobilizing citizens and parents to engage duty bearers (mainly district government officials) in
an informed dialogue on service delivery failures; and
 Enabling citizens and parents to channel their demands to the right decision-making processes
for improvements in the education system.
According to the Project Memorandum approved in November 2010, Ilm Ideas was originally
conceptualized to work with civil society, media and parliamentary oversight mechanisms. While the
programme has worked mainly with civil society organizations, it did not engage with media and
parliamentary oversight mechanisms at the scale envisaged in the Project Memorandum. This is partly
due to the fact that Transforming Education in Pakistan (TEP) became operational at about the same
time with a strong focus on media campaigning (Zara Sochiye) and engagement with political leadership
on education. Ilm Ideas found its niche in supporting research, advocacy and mobilization at district
level.
4
What is the context in which UK support is provided?
Pakistan’s education sector is characterised by low rates of Government investment (1.9% compared to
4% in the region), poor management and monitoring and some of the worst education outcomes in the
world including 12.3 million children out of school. Over 90% of spending on education is on teacher
salaries with few incentives for teachers to improve learning outcomes despite consistent increases in
teacher salaries . On any given day, up to 25% of public school teachers do not turn up to school. While
national indicators are useful context, provincial data shows large variation between provinces. For
example, net enrolment of 5-9 year ranges between 62% in Punjab, 54% in Sindh, 52% in Khyber
Pakhtunkhwa and 44% in Balochistan. Pakistan has little hope of meeting its Millennium Development
Goal (MDG) targets for Universal Primary Enrolment, or gender parity in education by 2015.
Innovative ideas, urgency, and new social partnerships are required if Pakistan is to make any progress
in achieving the Millennium Development Goals targets. In response to the national ‘education
emergency’, the United Kingdom Department for International Development (DFID) has made
commitments in Pakistan’s education sector of approximately £645 million over five years. Education is
a key plank of the UK’s development assistance to Pakistan and the overall portfolio (see Figure 1) aims
to benefit 4 million children in school by 2015. Already, 6.3 million primary school children have
benefited through UK support.
Figure 1: DFID Education Programme
In order to achieve this, DFID Pakistan has designed what the Independent Commission on Aid Impact
(ICAI) has termed a ‘sophisticated’ programme consisting of sector budget support, innovative technical
assistance referring specifically to the DFID supported Chief Minister’s Reform Roadmaps in Punjab and
Khyber Pakhtunkhwa and a focus on political will to increase supply and parental demand for better
quality education through a systematic political and social campaign in the form of the Transforming
Education in Pakistan programme.
The VA Fund is a relatively small programme in the context of the challenges Pakistan faces and DFID’s
ambitions to tackle them but represents an important part of the solution in addressing the challenges
to both the supply of and demand for education services. The concept is that small investments can help
to develop and incubate ideas that act as a catalyst for change. However, to be effective in the longterm, the VA Fund needs to be connected to other DFID programmes and partners. On its own it is likely
to produce some promising and inspiring models but little real change unless it finds ‘docking points’
with DFID programmes that operate at scale and policy advice/technical assistance with Governments at
both the federal (there is a newly convened Ministry of Education and Trainings) and provincial levels.
5
B: PERFORMANCE AND CONCLUSIONS (1-2 pages)
Annual outcome assessment
This third Annual Review focused on progress over the first three quarters from January-September
2014. The milestones for outputs and outcomes however are cumulative, and represent the overall
results achieved since the commencement of Ilm Ideas in January 2012 (based on 34 months of
implementation by end of October 2014, and a portfolio of 25 grants covering 56 districts across
Pakistan). In the first three quarters of 2014, the VA Fund was able to accelerate momentum on
research and advocacy through 18 live grants. Seven grants are already completed.
Progress on outcomes is assessed as good, with one target on track to be met, and the other target
exceeded, even after an upward revision of the milestone.
Indicator
Outcome Indicator 1: Number of models which enhance citizen
voice for demanding improved education services
Outcome Indicator 2: Number of administrative and/or policy
decisions made by district and provincial authorities for
enhanced education services
Targets by
2015
Results (July
2014)
10
6
1603
203
The VA Fund was set up to bolster the role of civil society in creating sustainable and safe spaces for
citizen voice and collective action on service delivery failures in the education sector. A key outcome
therefore is to establish and nurture models for the powerful articulation of citizen voice to hold
politicians and officials to account for the delivery of quality education at the local level. The expectation
was that the VA Fund would have supported eight models by December 2014. At the time of this Annual
Review, grantees have contributed the following six VA models with a significant degree of innovation:
1. Consumer Rights
Commission of Pakistan
(CRCP)
2. Institute of Social and
Policy Sciences (I-SAPS)
Formation and mobilization of school councils that operate at tehsil
and district level for the first time in Punjab to channel parental voice
from school level up to district level to demand better education
provision and quality
.
Ranking of political constituencies using an education scorecard
indicating the status of education provision (e.g. student-teacher
ratios, enrolment levels, teacher absentee rates) and holding the
politicians to account for poor education delivery.
3. Association of Global
Humanists and Ethics
(AGHE)
Advocacy for girls’ education and replication of a single teacher, homebased schooling model by the government of Gilgit-Baltistan
benefitting 3,018 girls.
4. Farmers Development
Organisation (FDO)
Building capacity of local people to negotiate and lobby politicians and
making their 'power of vote' conditional to the provision of free, equal
and quality education to their children.
3
This milestone was revised upward from 4 in the previous logframe revision in July 2014.
6
5. Centre for Governance and Use of right to information for accountability and disclosure of
Public Accountability
information on education, more specifically on public expenditure.
(CGPA)
6. Semiotics
Establishment of Education Planning Working Groups as a mechanism
for participatory, grass-roots planning involving citizens, community
representatives, district government and school officials.
The trajectory is positive. Ilm Ideas team believes that grantees would be able to demonstrate four
more VA models from the current portfolio of 18 live grants. The Annual Review team concludes that
this outcome will likely be met by the end of the programme. However, the sustainability and
replicability of these models after the closure of DFID grant will continue to be a challenge. The
programme’s exit plan should articulate plans to address this.
On the second outcome, Ilm Ideas reported that 203 administrative or policy decisions were made by
Government to which the advocacy and lobbying efforts of grantees made a contribution (overachieved
by a high margin against the target of 150 by December 2014). This indicator measures the
responsiveness of duty bearers (mainly district government officials), and is a proxy for the success of
grantees’ work to enable citizens to put pressure on the duty bearers for service delivery improvements.
It is clear from a breakdown in the table below that most of these decisions were administrative and
focused on school infrastructure. The second highest number of decisions made was about the transfer
or appointment of teachers, which is a subjective process and highly politicised. The focus on influencing
policy with wider impact has been limited, given that most VA grants worked locally with little footprint
at the provincial level. From the overall portfolio, four policy decisions were reportedly made as a result
of grantees’ influencing. Examples include the adoption of a home-based schooling model by the GilgitBaltistan Government and allocation of an increased budget for school infrastructure in Rajanpur
district.
Decision type
Category
Missing facilities
Whitewash
Administrative
Reconstitution of school councils
Teacher provision or transfer
Government adoption of home-based
Policy
school model, school upgradation, and
budget allocation
Total
Number
of
decisions
116
44
1
38
4
203
The number of decisions reported is based on information provided by grantees. Documentary evidence
was made available to the review team along with details of Ilm Ideas’ monitoring visits and four rapid
assessments for some of these decisions. However, these would benefit from further external
verification. Voice and advocacy is a complex process, and therefore it is difficult to measure the extent
to which these decisions were made as a direct result of grantees’ influencing, or happened anyway
under the on-going strong drive to improve school infrastructure and inputs in provinces, or both. This is
important to capture in both the VfM framework and any further evaluations that are conducted. The
review team noted, however, that monitoring visits were routinely conducted by Ilm Ideas but these
should be strengthened to further verify or triangulate grantees’ logframe results.
7
Overall output score and description
Consistent with previous Annual Reviews, nearly all milestones under the three outputs were exceeded
by a high margin. Key achievements are:

Ilm Ideas supported the establishment of 251 new forums for participation and citizen voice (e.g.
school councils, children clubs, village education committees); provision of information, capacity
and mobilization of over 18,365 people to demand quality education and submission of over
1,117 demands by citizens facilitated by grantees. Gender-disaggregated data is not available for
these results despite this being a recommendation from the previous Annual Review.

Grantees were able to engage more than 900 duty bearers (mainly district government officials),
including members of provincial and national assemblies, members of standing committees on
education, and education officials. This involved direct contact between members of different
voice mechanisms supported by grantees (e.g. school councils, village education committees,
etc.) and duty bearers. Active citizens were supported to submit written petitions and letters to
demand improvements in their schools.
Output 1
Output 2
Output 3
Output
Strengthened and coordinated
community action
Enhanced responsiveness and capacity
of duty
Increased availability of evidence and
information
Score
A+
A+
A+
Ilm Ideas has supported grantees to undertake research and produce information materials and
briefings for duty bearers. In total, 56 publications have been produced, including policy relevant
research. The Review team visited Chakwal to see Nur Foundation’s research on correlation between
nutritional status and access and learning. The study also assessed the impact of a recent policy decision
for health supervisors to work in schools, and conducted anthropometric measurements revealing a
high incidence of stunting in children. This is a good example of research under the VA Fund to bridge an
important evidence deficit, given the unavailability of Pakistan-specific evidence on education and
nutrition. Plans were being made to engage a multilateral donor and get organisations interested in
health in schools engaged. Nur Foundation was also approaching the Chief Minister of Punjab through
the Punjab Roadmap team. In order to share evidence and project citizen voices on education, grantees
have also worked with the media and organised seminars and dialogues.
However, progress on two indicators did not meet the expectations. Indicator 2.14 was added in the
revised log frame in July 2014 to reflect the work under special grants to promote horizontal
accountability, and build capacity and skills of government officials to perform tasks for scrutiny and
oversight. Two special grants were awarded in April 2014 to Institute of Social and Policy Sciences (ISAPS) and Pakistan Coalition for Education (PCE) to encourage mechanisms of inter-departmental
scrutiny. The initial months of these grants focused on research whereas capacity development activities
will commence in 2015. Indicator 3.25 focused on the preparation of training modules and information
4
Indicator 2.1 is “number of duty bearers (mainly district government officials) engaged in capacity development events to
enhance their ability to improve education management”.
5 Indicator 3.2 is “number of training manuals, toolkits & IEC materials developed to enhance capacity and awareness in the
education sector”
8
materials. At the time of this Annual Review, 29 materials had been developed against the target of 40.
The gap is due to the fact that contracts in the third round were signed in December 2013 and for most
of these grantees, capacity building will follow the completion of on-going research. The target is likely
to be met before May 2015.
The Review team noted that most targets had been overachieved by a high margin. In part, this reflects
the ambition and good grant management capability of the Ilm Ideas team as well as grantees.
However, two other factors explain the fact that most targets have overachieved by a significant margin
consecutively since 2012.

Stretch and ambition within targets. The actual contributions of grantees always exceeded the
targets in the overarching logframe. DFID and the Ilm Ideas team could have created further
ambition by using the logframes of grantees as a basis for setting more stretching targets in the
overarching VA Fund logframe as these were seen to be somewhat conservative. Indicators
denoting quality should also have been considered rather than process indicators and attempts
should be made in the PCR to report against quality in a more nuanced way.

Most of the logframe indicators measured the outreach of grantees, rather than the range or
quality of activities reported. Technical and financial capacity constraints, both within the Ilm
Ideas team and partner organisations, made it difficult to monitor hard-to-achieve and hard-tomeasure qualitative results for voice and accountability interventions. Most indicators in the
logframe, agreed with DFID, measured the outreach of grantees’ work, including some indicators
which could be achieved at scale with little effort. For example, “citizen action” in indicator 1.4
counts letters submitted to politicians and education officials, which can possibly be produced
and printed in bulk on a template and sent to hundreds of people simultaneously within a few
hours. The granularity and meaningfulness of what constitutes a citizen action and an
engagement with duty bearers is not explicitly set out. A key reason is that no budget was
earmarked for evaluation when the grants commenced, and hence, ambitious indicators were
not selected because their monitoring would have required evaluation.
Overall, there are indications that the VA Fund is on a positive trajectory to demonstrate impact in at
least three areas: (1) increased public demand and responsiveness for school improvements, especially
for the provision of physical facilities and teachers; (2) creation of new opportunities for communities to
participate in advocacy activities at local level; and (3) generation of context-specific and policy-relevant
evidence on access barriers, expenditure and nutrition. The Annual Review team noted though that
these achievements have remained less visible for two reasons. First, the VA Fund operates at local level
with little provincial footprint. Secondly, scale-up of the Innovation Fund may have drawn greater
attention both from DFID and Ilm Ideas.
Key lessons
1) Grantees have made good contributions in terms of establishing and mobilizing new forums for
participation and advocacy at local level, but their replication and scale-up for sustainable impact
remains a challenge. It is important that the uptake and sustainability of voice mechanisms and
participatory structures receives explicit attention from the beginning of a grant being awarded.
2) Almost all VA grants under Ilm Ideas included research as well as advocacy. However, the team
have focused disproportionately on research in small VA grants (albeit to ensure it is robust and
9
of high quality). However, this often left little time for mobilisation for collective action and
advocacy. Thus, grantees were unable to use the evidence properly to articulate their demands
to duty bearers (mainly district government officials). VA grants should strike a balance between
research and advocacy, and invest early on in building relationships with duty bearers instead of
deferring to the end.
3) A robust monitoring and evaluation system is extremely important to demonstrate results from
VA interventions and should be budgeted for from the beginning. Monitoring and evaluation
systems should ensure the disaggregation of results by gender and marginalized groups, build in
qualitative indicators, and set strong monitoring mechanisms to verify, validate and evaluate key
results.
Has the logframe been updated since the last review?
The logframe was revised in July 2014 to incorporate recommendations of the previous Annual Review,
ensuring clear links between impact and outcome indicators; reflection of school level improvements
elicited by the programme; and merger of some indicators to minimize duplication.
C: DETAILED OUTPUT SCORING
Output Title
Strengthened and coordinated community action to demand improved education
services
Output number per LF
1
Output Score
A+
Risk:
Medium
Impact weighting (%):
35%
Risk revised since last AR?
N
Impact weighting % revised
since last AR?
Y
Indicator(s)
1.1: Number of newly established education
forums/coalitions aimed at strengthening community action
1.2: Number of capacity development workshops and/or
information sessions held with civil society groups to enable
them to raise their voice more effectively
1.3: Number of citizens mobilized to demand improved
transparency and accountability in the education system
1.4: Number of citizen actions such as policy
recommendations, petitions and applications submitted to
duty bearers to demand improved education services
Milestones
Progress
200
251
Exceeded
130
142
Exceeded
17,000
18,365 Exceeded
500
1,117 Exceeded
Key points

Ilm Ideas has contributed to building civil society’s capacity to work on accountability of the
education sector and increase community and parental voice. In addition to funds, the grantees
received mentoring and coaching from Ilm Ideas to set up and nurture 251 new avenues and
structures for participation, dialogue, and voice. Examples includes formation of school councils at
tehsil and district levels (CRCP), village education committees (AGHE), children’s clubs (CGN), and
education planning working groups (Semiotics).
10

Capacity development and mobilisation are critical to organise collective action, and hence are an
integral part of most grants provided under the VA Fund. More than 18,000 people – parents,
community, journalists, students, activists – have received information, skills or support to
understand and raise education issues and channel their demands to education authorities.

Based on monitoring information submitted by the grantees, there is an early indication that Ilm
Ideas has explicitly promoted citizen action and a culture of submitting written demands on
educational issues at local level. Six grants so far have facilitated and supported active citizens and
parents to submit over 1,100 demands for improved education services. Most of these demands
pertain to school inputs - provision of missing facilities, whitewash and provision of teachers. The
lesson learning document produced by Ilm Ideas should explore the relative traction of individual
versus community-led advocacy to gain response.
Summary of responses to issues raised in previous annual reviews (where relevant)
Ilm Ideas responded promptly to most recommendations in the Annual Review 2013 and revised the
output indicators, diversified the thematic and institutional dimensions of the portfolio, introduced
special grants for horizontal accountability, and provided mentoring to grantees. Important
recommendations which have yet to be acted upon are: (1) gender-disaggregation of targets, and (2)
verification and evaluation of well performing projects. Although an evaluation framework was
developed and rapid assessment of four projects undertaken, that alone is insufficient to assess impact
and value for money.
Recommendations

Implement an Exit Action Plan with clear thinking and steps to increase the sustainability of
outcomes, especially in relation to the newly-established education forums and coalitions, and
active mass of parents and activists engaged and mobilised by the grantees. The Action Plan can tap
into opportunities for transitioning of successful innovative projects into Ilm Ideas II, and integrating
the new voice mechanisms and activists into TEP (e.g. bringing them in contact with the Network
Coordinators) and demand-side work under DFID’s sector budget support programmes in Punjab
and Khyber Pakhtunkhwa.

Prioritize verification and validation of key results reported by grantees over impact evaluation. As
the evaluation framework suggests, there is no robust baseline to measure the proper impact
evaluation of voice and accountability grants. It is more important to validate key outcomes and
outputs independently (e.g. administrative and policy decisions taken) with some analysis of
pathways of change to learn lessons and assess value for money ahead of the Project Completion
Report (PCR) in 2015.
11
Output Title
Enhanced responsiveness and capacity of duty bearers to improve education services
Output number per LF
2
Output Score
A+
Risk:
Medium
Impact weighting (%):
35%
Risk revised since last AR?
N
Impact weighting % revised
since last AR?
Y
Indicator(s)
2.1: Number of duty bearers engaged in capacity development
events to enhance their ability to improve education management
2.2: Number of duty bearers effectively engaged in advocacy
platforms to enhance education service delivery
2.3: Number of citizens’ recommendations accepted (in writing) by
elected representatives
Milestones
100
0
Progress
Not met
425
969
Exceeded
150
285
Exceeded
Key points

The milestone for indicator 2.1 was not met. This new indicator was added in the revised logframe
in July 2014 to reflect the work under special grants to promote horizontal accountability, and build
capacity and skills of Government officials to perform tasks for mutual scrutiny and oversight. Two
special grants were awarded to I-SAPS and PCE in April 2014 to encourage mechanisms of interdepartmental scrutiny. Activity in the initial months of these grants focused on research; capacity
development will be delivered in 2015, so rapid progress is required at the point of the PCR.

All grants made under the VA Fund have a focus on advocacy and lobbying with duty bearers –
elected representatives, political leaders, district education officials and teachers. In most cases,
their involvement has been through participation in meetings and seminars. A number of grantees
encouraged parents and citizens to submit written demands and recommendations to elected
representatives. 285 recommendations were acknowledged or endorsed, which is an expression of
willingness and interest of elected representative and education officials to hear the voices of
parents and communities. However, an external evaluation would help to track actual change as a
result of these demands.
Summary of responses to issues raised in previous annual reviews (where relevant)
In the 2013 Annual Review, DFID recommended a greater focus in a number of areas: mutual scrutiny
and accountability role of legislatures, standing committees on education, public accounts committees,
courts, procurement authorities and audit departments. This recommendation was made with the view
that the state’s internal (horizontal) accountability mechanisms need strengthening as they have powers
and authority for oversight and scrutiny of service delivery failures and can foster sustainable
accountability routines. The Ilm Ideas team responded to this proposal positively, and awarded two
special grants to I-SAPS and PCE because of previous experience and delivery in this field. These grants
are six months into implementation and have yet to demonstrate their relevance and impact.
Recommendations
 Under the two special grants on horizontal accountability, prioritize capacity development and
advocacy work streams. These grants provide a good opportunity for the VA Fund to engage with
key provincial institutions and parliamentary committees.
12
Output Title
Increased availability of evidence and information on the education system to inform
duty bearers and rights holders
Output number per LF
3
Output Score
A+
Risk:
Medium
Impact weighting (%):
30
Risk revised since last AR?
N
Impact weighting % revised
since last AR?
Yes
Indicator(s)
3.1: Number of publications to generate evidence in the
education sector
3.2: Number of training manuals, toolkits & IEC materials
developed to enhance capacity and awareness in the
education sector
3.3: Number of print and electronic media content
developed and disseminated through
TV/radio/SMS/newspapers to raise awareness regarding
key education developments
3.4: Number of public events held on education such as
street theatres, melas and seminars
Milestones
50
Progress
56
Exceeded
40
29
Underachieved
30
30
Met
45
83
Exceeded
Key points
 Output 3 indicators largely measure activities and events to generate and disseminate
information and evidence. They are envisaged to serve as a proxy for the scale of awareness
raising and outreach.

Nearly all grants had a focus on research and evidence generation. The Fund has supported over
50 publications on a range of topics. Some research studies have policy relevance and systemwide implications. Nur Foundation is investigating the correlation between nutrition and
educational outcomes. I-SAPS has prepared constituency-based report cards on school
performance and used them to rank political constituencies. CGPA and CPDI have developed
simplified budget analyses.

Progress on training, toolkits and information materials remained slow. The key reason is that
most of the VA grants prioritize research to generate analysis, and training and awareness raising
activities are left for outer months.

Media coverage and outreach through seminars exceeded the milestones, but has largely been
less visible because the grantees worked at district or tehsil level away from the mainstream
media.
Summary of responses to issues raised in previous annual reviews (where relevant)
The previous Annual Review recommended that Ilm Ideas provide more coaching and mentoring to
grantees, especially smaller organizations, to increase the impact their work. The Ilm Ideas team
organized a research and advocacy workshop in June 2014 to improve their understanding and skills.
The Review team held meetings with four partner organizations on 20 October in Islamabad. All of them
appreciated the quality of feedback received from Ilm Ideas on their research products. However, the
13
team found less evidence of networking and mutual sharing of information and lessons among the
partner organizations.
Recommendations

Clear thinking and an action plan is needed for wide dissemination and uptake of evidence
generated under different grant projects. Ilm Ideas should consider making these publications
available on open online platforms for permanent public access.

More opportunities for peer-to-peer learning and networking among partner organizations
should be created to amplify the voice of all civil society organizations, and coalitions and
advocacy forums established with the support of the Fund.
D: VALUE FOR MONEY & FINANCIAL PERFORMANCE (1 page)
Key cost drivers and performance
The programme has funded a range of voice and accountability interventions of differing size, scale and
modality. The operating costs vary significantly in different parts of the country, which also drives up
costs. Therefore comparing the relative value for money of different interventions is challenging. The
Review team has compared this programme in relation to similar DFID programmes to draw conclusions
on VfM. A recommendation has also been made to devise a VfM framework for voice and accountability
given the complexity around measuring actual contribution of activities to results.
The main cost drivers for the VA Fund continue to be overheads associated with sub-grant
administration and programme management costs. DAI made scale-up funding available from April 2014
onwards for new scale-up VA activities and special grants. The administration and programme
management costs of sub-grants costs vary across individual projects but generally range from 20-30%
of the individual grant total.
Programme management costs for Ilm ideas account for 27% of total programme costs to date, in line
with the modified budget after scale-up. Fees, which include programme personnel, make up the
largest share. The original budget, prior to scale-up, provided for spend on management and operations
equal to 24% of total programme costs. The increase is explained by the recruitment of personnel to
provide additional capacity for grant management, M&E and outreach activities. Office expenses have
also increased as the Ilm Ideas innovation management team moved out of a shared premises to
accommodate the innovation fund scale-up.
Whilst these costs seem relatively high, they remain roughly comparable with other challenge funds and
include considerable staff time spent on capacity building with grantees. As a comparison, DFID
Pakistan’s Sub-National Governance (SNG) programme has recently set up an innovation challenge fund.
Fees constitute approximately 23% of the total budgeted amount. As mentioned above, a number of the
projects operate in difficult environments, which include some remote and hard to reach areas of
Pakistan (e.g. AGHE’s work in Gilgit-Baltistan on girls’ education). This contributes significantly to driving
up the associated costs of delivery and monitoring for the Ilm Ideas management team. Adjusting for
the operational costs and overheads that are covered separately as part of total SNG programme
budget, the costs for the VA Fund are easily comparable.
14
VfM performance compared to the original VfM proposition in the business case
Ilm Ideas preceded DFID’s business case process, therefore there are limitations to assessing VfM as no
initial VfM metrics or need for evaluation was identified. Data collected on VA grants has been mainly
activity-based and self-reported with some attempt to gauge impact through beneficiary surveys.
Research also highlights the limitations of unit costs when assessing the VfM of VA initiatives as the
scope for change can vary enormously with context. This further limits the scope for VfM analysis.
However, using data and evidence gathered to date, a total of 25 VA grants have been awarded at an
average value of £150,455. Collectively, these grants have trialled a number of VA models and funded a
range of activities including:
 Establishment of over 250 education fora or coalitions aimed at supporting community action;
 142 workshops or information sessions across the country to raise awareness and build capacity
to articulate and make demands;
 Over 50 publications on a range of educational issues to use as evidence for advocacy and
awareness;
 Mobilisation of more than 18,000 citizens and almost 1,000 government officials; and
 Awareness raising with media actors and development of media content.
However, these figures do not provide any information on the quality, targeting or impact of VA. The
efficiency and effectiveness with which new network structures, research or advocacy activity are
translated into increased citizen action and response needs to be assessed at the grant level. Some
specific examples of projects which have reported on outputs and outcomes are outlined below, with
the caveat that results are largely self-reported by grantees and as yet unverified.
Project cost and output examples
For an outlay of £247,000, I-SAPS supported the dissemination of report cards across three districts in
Punjab ranking performance against a set of education indicators and assisted community groups to
submit demands to their authorities. These activities supported submission of 205 petitions and
whilst attribution cannot be verified, government action has been taken on 62 of them, leading to
provision of boundary walls in 12 schools, toilets in three schools, electricity in 31 schools, and
provision of teachers and furniture in some cases. The project has helped, therefore, to improve
learning environments in some communities. Additional benefits include strengthened community
groups and skills in advocacy.
With a grant of £103,808, the Centre of Peace and Development Initiatives has conducted public
expenditure tracking surveys for two districts in Punjab and disseminated findings through
stakeholder meetings, press briefings and newspaper articles. The impact of the project is harder to
define. The grantee reports contributing to the issue of an ordinance abolishing school user fees.
The Association of Global Humanists and Ethics received a grant of £83,422 to fund research into
constraints to girls’ education and on the back of findings established six home-based schools for girls
in teachers’ homes, enrolling an average of 50 girls per school. The project used a combination of
advocacy and delivery to accommodate cultural norms and enable girls to access school in an
environment that parents felt comfortable with. The project reported parent motivation to send their
girls to school as evidence of the success of its activities. It also reported the Education Department
adopting three of its schools and subsequently replicating its model to provide for 75 schools with
over 3,000 girls enrolled.
15
Assessment of whether the programme continues to represent value for money
The programme has performed well against the logframe indicators and successfully implemented
several VA models in a variety of contexts, and in many cases directly targets marginalised groups. Data
is however insufficient or lacking verification to assess the extent to which these activities have been
effective in enhancing citizens’ voice for improved education services. The review team recognises the
challenges of attributing changes in educational outcomes to VA initiatives which can often be difficult
to isolate from other contributing factors. The recommendation to undertake verification of reported
outcomes and improve qualitative analysis of the contribution of VA projects should provide evidence to
make a more informed assessment of VfM at project completion.
Looking across the portfolio, a number of grants have set up new advocacy groups and structures. As
the programme enters its final year, sustaining (or consolidating) these new groups will be essential to
see the momentum for action survive beyond the lifetime of the grants. To improve value for money,
grantees should ensure the groups have a clear plan for how they will sustain activity and access to
information and resources once the grants end.
Quality of financial management
The table below provides an overview of the Ilm Ideas overall budget and actual spend till September
2014.
Expense lines
Project Budget
Fees
Operational expenses
Innovation Fund grants
VA Fund grants
TOTAL
£2,190,656
£832,392
£3,933,044
£4,000,000
£10,956,093
%age
total
Budget
20%
8%
36%
37%
100%
of Expenses
(as of Sep
2014)
£1,350,649
£350,326
£2,244,233
£2,329,331
£6,274,539
% Spent
62%
42%
57%
58%
57%
The Ilm Ideas overall budget is £10.9 million out of which £5 million is allocated to VA (£4 million grants
+ £1 million towards fee and operational costs) and £5.9 million is allocated to Innovation Fund which
includes £3.9 million grants + £2 million operational costs and fee.
The following pie chart provides a snapshot of VA Fund budget commitments and funds at hand to be
further committed.
Uncommitted,
£101,980
Committed,
£3,898,020
Total Grant
Budget,
£4,000,000
16
Out of the total grant budget 97% has been committed and 3% of the budget has yet to be committed.
Although 97% of the programme budget has been committed, there is no guarantee that the same
would be spent by the end of the programme. The Ilm Ideas team expects 10% savings from each of the
grantees at the closure stage. Overall savings expected by the closure of the programme will be around
£300,000 to £400,000.
Slow spend of committed funds is also a challenge. Over its life time the programme has spent 58% of its
total grant budget with 42% remaining to be spent over the next seven months. Given the programme
spending rate to date, meeting these financial targets over the coming months will be challenging. The
slow spending rate has been attributed to the issues below, which have been addressed by Ilm Ideas in
the third round of applications:



Proposals received were initially lacking in quality.
Most of the grantees are small scale organisations with limited financial absorption capacity.
Late submission of financial reports from grantees resulting in delays in disbursements.
As well as signing grants with the successful applicants from the third round of applications in November
2013, the Ilm Ideas team has issued three special grants (ISAPs, PCE, Century Publications) and are in the
process of finalising a fourth special grant with FAFEN6 at the time of drafting of this report. Special
grants are contributing to an increase in spending rate, but the overall spending challenge remains.
Below are the Ilm Ideas team projections vs spend for the financial year 2014/15.
April to September, 2014
Spend till
30
Forecasted
Variance
September
2014
£359,110
£326,782
£32,329
Fees
Operational expenses (Outreach
£176,628
Activities inc)
VA grants
£1,249,646
Total
£1,785,384
£115,630
£60,998
£820,887
£1,263,299
£428,759
£522,086
Variance
in %
9%
34%
34%
29%
Ilm Ideas performance on financial management has dropped this year. Despite the commitment of
97% of VA grant funds, Ilm Ideas have spent 34% less than their projected spend between April and
September 2014. Until last year (2013/14), monthly projections vs. actual spend on various budget
lines variance ranged between 2% (- or +) to 5% (- or +). However this year variance on projections vs.
actual spend went beyond the standard 5% (- or +) and reached an all-time high of 8% (- or +). A
number of factors have contributed to this issue, including staff changes in the Ilm Ideas finance team
and the finance team changing focus from VA to facilitate the scale-up and mobilisation of Innovation
Fund grantees.
6
Free and Fair Election Network (FAFEN) will run an evidence based advocacy campaign to identify, collate and
analyse effective V&A models that have the potential to be replicated by public and private sectors to impact policy
reform in the education sector.
17
Recommendation
It is a DFID corporate requirement to maintain financial forecast variance at an acceptable level.
Therefore from DFID’s perspective the period between now and the end of the programme is crucial in
terms of spending and delivery. The review team recommends:

Ilm Ideas finance team review spending trends of each grant with a view to identifying high risk
grants and invest more effort in managing them.

Ilm Ideas team should also consider revisiting workplans of each of their grantees with a view to
reducing scope of work where there are major slippages and it is unlikely that the grantee could
deliver by the end of the V&A programme. The focus should remain on the quality of delivery.

Ilm Ideas team should maintain financial variance at standard 5% (+ or -) and have monthly
financial review meetings with DFID to keep track of financial performance and flag potential
slippages in spend to DFID well in advance.

DFID discuss with Ilm ideas regarding how the 10% savings from each grantee at closure stage
can best be spent or alternatively returned to DFID following closure of the overall VA
programme.
Date of last narrative financial report
Date of last audited annual statement
18th August 2014
7th July 2014
DAI’s last audit report is unqualified. The audit report states that “the statement presents fairly, in all
material respects, the receipts and payments of the project for the year ended March 31 st, 2014 on the
basis of accounting as described in Note 2.1 to the statement”.
18
E: RISK
Overall risk rating: Medium
Overview of programme risk
The programme faces a variety risks at the management, grantee and political level. Detailed risks are
outlined in a risk analysis framework developed by Ilm Ideas. Key high level risks and mitigation
measures are explained below:
Ilm Ideas programme management and administration:
The risk of staffing constraints impacting on delivery have been mitigated through the scale-up funds
and recruitment of additional staff within the Ilm Ideas team. Managing staffing constraints has also
been addressed through engagement with technical experts through the short-term technical assistance
(STTA) pool and prioritising key activities with grantees.
Poor security in some areas of the country such as Federally Administered Tribal Areas (FATA) and
Balochistan means it is difficult to visit projects and monitor progress. Ilm Ideas is mitigating this by
working with third party local M&E providers where possible to monitor grantee progress.
Timely completion of live VA grants and spend is of concern particulary with just seven months left until
the closure of the VA Fund. Ilm team should identify high delivery risk grants and should closely monitor
these between now and the end of the programme.
Verification of results
Reliable evidence is required in addition to the monitoring of project activities to verify the outcomes
achieved (i.e. 203 decisions influenced by VA grantees). It is important to use third party monitoring to
mitigate the risks that may arise from inaccurate representation of results by grantees. It is
recommended that a third party verification is carried out before the Project Completion Review.
Grantee level risks
Many applicants for grants do not have sufficiently strong administrative and financial systems in place
to manage the funds and deliver the proposed activities within the timescale. To mitigate this, the Ilm
Ideas team has instituted detailed checks and balances and provided upfront coaching to grantees
through workshops. The review team was satisfied with the rigour of these checks.
Results and sustainability:
The ability to scale up or replicate VA projects is also a key risk. Five out of six of the VA models have not
been replicated or adopted by the government or been able to find additional sources of funding. Two
grantees - Institute of Social and Policy Sciences (I-SAPS) and Centre for Governance and Public
Accountability (CGPA) – will continue similar VA work under grants secured through other DFID
programmes (Transforming Education in Pakistan) and the Sub-National Governance (SNG) Challenge
Fund).
Outstanding actions from risk assessment
The process for receiving annual audited reports from DAI for the Ilm Ideas programme needs to be
more streamlined and regular. The risk rating should be revised or re-phrased to bring clarity to the
term ‘replicability’ and if it pertains to whether the project is adopted or the idea itself lends itself to
19
being replicated at some later stage. This should be followed up by DFID and DAI as part of lesson
learning from the programme, and in the focus of evaluations.
F: COMMERCIAL CONSIDERATIONS (½ page)
Delivery against planned timeframe
The programme has performed well on technical aspects and has exceeded most of its targets. However
there have been slippages on delivery against the planned timeframe, including delays in signing of VA
grants due to the lack of quality proposals from potential grantees. Delays in implementation timescales
have occurred due to limited grantee capacity to absorb funds as well as grantees’ technical expertise.
Most VA grantees are small scale organisations that have needed consistent coaching support from the
Ilm Ideas team. The recent floods in Punjab (August/September 2014) has also impacted on the delivery
of three existing VA projects.
The scale-up of the Innovation Fund provided additional financial space and technical expertise to the
Ilm Ideas team which has also strengthened the monitoring and evaluation function of the VA fund. One
of the recommendations from last year’s Annual Review was to work on results verification. This has
again been recommended in this review and important for the forthcoming Project Completion Report
(PCR).
Recommendations:

Ilm Ideas team should consider reviewing priorities between now and the end of the project to
avoid potential delays in spend, delivery and reporting. The no-cost extension will also help to
mitigate a number of delivery risks.

Ilm Ideas team may consider giving short extensions to grantees that have been affected due to
floods. However, the extensions should not go beyond the programme end date.

Ilm Ideas needs to strengthen its work on results verification and beneficiary feedback ahead of
the PCR.
Performance of partnership (s)
Ilm Ideas has good systems in place (for example, they have adopted the TAMIS management tool for
grant and financial management). The review team was impressed with the upfront due diligence
efforts undertaken with grantees prior to the award of grants. Engagement through this process has
helped develop grantees’ confidence in the partnership. The review team also met and sought feedback
from VA grantees during the AR process (Centre for Governance and Public Accountability (CGPA), Nur
Foundation and the Institute of Social and Policy Sciences (I-SAPS). All reported that they appreciated
support from the Ilm Ideas team. However, one grantee commented that the Ilm Ideas team engaged
more frequently with grantees’ management offices and less frequently with the project staff on
location.
Ilm Ideas shared details of their M&E systems for verification and validation, which is conducted at 4
levels:
Level 1: 100% documentation review of evidence
Level 2: 1-3 field monitoring visits of all live grants
Level 3: Bi-annual data quality assessments with each grant partner
20
Level 4: Rapid Assessments/external evaluations with selected partners based on contribution to results
(underway).
Results are triangulated by one or all four means based on project timelines
The frequency of monitoring visits by the Ilm Ideas team as reported by grantees ranged from one visit
per grantee to up to three visits per year. VA grantees also commented that they would benefit from
greater opportunities to network and learn from other projects within the portfolio. It is recommended
that Ilm Ideas creates a networking and lesson learning opportunity for VA grantees with a view to
looking at how activities could be embedded and grants could be made more sustainable.
Partnership with DFID
The partnership between DFID and DAI has been working effectively. DFID and DAI have open and
regular channels of communication. This provides both DFID and DAI the opportunity to discuss progress
as well as potential challenges. DAI are also provided regular feedback through DFID’s central Key
Supplier Management process.
DFID has been represented on evaluation panels to select projects for scale-up. DFID has also provided
guidance on strategic issues, such as working with Ilm Ideas to strike a balance in the VA portfolio by
awarding grants to address both horizontal and vertical accountability.
One key area where DAI has done well is foreseeing potential risks and flagging them to DFID in a timely
fashion. DAI’s team has kept DFID engaged at each stage of the programme both at process and
outcome level ensuring the success of the partnership. DAI’s team has found DFID supportive,
particularly on agreeing changes in strategy and updating of logframe targets but, on occasion, needs
DFID to take more timely decisions on approving grants.
The role of DAI’s consortium partner Cambridge Education has been limited to provision of a full-time
Education Advisory position. DAI could have used Cambridge Education more strategically in order to
get the maximum value out of this partnership.
Recommendations
DFID should strive to make more timely decisions on approving grants. This comment was made in
reference to Century Publication’s grant where at final stage of discussions the Ilm Ideas team had to
put the process on hold while DFID assessed potential sensitivities around media engagement through
this project and the likely impact on other DFID funded programmes.
DFID should play a more proactive role in facilitating coordination between Ilm Ideas and other DFID
programmes particularly where objectives of the programmes overlap and communicate potential
delays early, and provide an indication of the direction of travel where approval processes may take
longer.
It is recommended that Ilm Ideas creates a networking and lesson learning opportunity for VA grantees
with a view to looking at how activities could be embedded and grants could be made more sustainable.
21
Asset monitoring and control
DAI has a standard asset policy in place. The Finance Officer has overall responsibility for maintaining a
Fixed Assets Register. Grant Managers are responsible for maintaining an Asset Register containing
details of assets purchased by grantees. DFID carries out annual asset verification exercises. To date
assets in use by DAI have been found to be well-maintained and appropriately labelled. In the last asset
verification exercise, cross-referencing of assets with the records in the register was found to be in
order. In line with DAI’s Asset Policy, the ownership of assets purchased by grantees is transferred to
grantees at the end of the project. The transfer of ownership is conditional upon retention of assets at
project locations to ensure beneficiaries continue using the assets beyond project closure.
DAI or its grantees did not report loss or theft of any of the assets since the last asset verification
exercise in 2013.
The next asset verification exercise was completed in November 2014 with a focus on the new assets
purchased over the past year due to the relocation of the Ilm Ideas offices.
G: CONDITIONALITY (½ page)
Update on partnership principles (if relevant)
Not Applicable. The programme sits out of government systems and works through not-for-profit
organisations.
H: MONITORING & EVALUATION (½ page)
Evidence and evaluation
All grantees receive a budget to monitor their activities, although some of the Voice and Accountability
projects are either research-based or have substantial research components. Most of the research
products were aimed at producing evidence for awareness and advocacy on educational issues, and
were not meant to verify the results or evaluate outcomes of the projects. While DAI has a grant
monitoring system which generates reports on quantitative indicators, the evaluation and verification of
results could be further strengthened. The llm Ideas team is able to monitor grants with the help of third
party monitors in difficult to reach areas. However no budget was earmarked for evaluation at the
beginning of the VA Fund in January 2012. The 2013 Annual Review recommended that DAI undertake
evaluation of two to three high-performing projects for lesson learning. An evaluation framework was
developed and four rapid assessments conducted. At this stage of programme implementation, the
available evidence meant it was hard to form a full assessment of whether the Theory of Change and
assumptions used in the programme design were actually working out in practice. For future
programmes, monitoring and evaluation mechanisms should be built in from the beginning and
sufficient funds should be set aside for this purpose.
Given no baseline exists to warrant a proper evaluation of the VA Fund or individual grants, but
retrofitting a baseline and an evaluation would not be useful at this stage. Instead, a third-party
verification (TPV) should be done for key results under Outcome 2 i.e. administrative and policy
decisions. The TPV should collect evidence about and verify a sample of actions reported to have been
22
taken for missing facilities, teacher appointment and transfers, and all policy decisions reported. This
should also include some analysis of how grantees were able to influence these decisions. The TPV
results will provide important information as to what VA work can deliver in education, along with key
lessons for monitoring and evaluation within DFID’s grant windows. The TPV should be directly
supervised by DFID.
Monitoring progress throughout the review period
The Annual Review team held meetings with four grantees in Islamabad and asked questions about
monitoring. The team also visited a Government school in Chakwal where the Nur Foundation is doing a
research study on the correlation between nutritional status and learning outcomes. These meetings
confirmed that monitoring routines had been established, including quarterly reports and field visits by
the Ilm Ideas team. The AR team noted that so far 12 field monitoring visits had been made, including to
remote districts. The frequency of monitoring visits by the Ilm Ideas team ranged from one to three
visits per grantee per year. The Ilm Ideas team had also put in place regular reporting requirements for
the grantees. This has helped smooth the management of grants, but verification of results remained
confined to desk-based documentary review. Therefore, an independent TPV for verification of results
ahead of the PCR is strongly recommended.
23