Synergy of Government and Nongovernmental Bodies
Involvement in Monitoring and Evaluation
The presentation dwells on short falls and positive effects of country led evaluation
systems based on a survey of monitoring and evaluation (M&E) procedures under two
types of projects being implemented by Ministry of Labor and Social Policy. The EU
Pre-Accession and International Projects Directorate with the ministry is in charge of
PHARE projects. For the current financial year the PHARE Program amounts to EUR
40 m round. Currently the Bulgarian Social Investment Fund (BSIF) with the same
ministry runs a development World Bank and Labor Ministry joint project named
Social Investment and Employment Promotion Project of total capitalization of EUR
66.7 m. These specific projects with a single ministry help us sample the M&E
process with all the development projects of the WB and PHARE funded projects on
the national scale.
The ultimate goal of M&E process is to present to beneficiaries and stakeholders a
fair and unbiased perception of project progress achieved. Progress measures with the
extent which the original project goals have been achieved to. On operational level the
goals to be achieved are commonly presented under the form of Goal Matrix based on
benchmark indicators (key performance indicators). Effectiveness of M&E process
measures with consistency of the fundamentals which are principles, regulations and
organizational arrangements set up with managing authorities and beneficiaries.
Effective M&E process attempts to follow principles that are inherent to each
effective organization dealing with public funds: transparency; objectivity;
centralization-vs-decentralization; delegation; accountability, coordination-vscommunication. Strike an effective balance between centralization and
decentralization of data collection, analysis and finally, decision making is topical to
setting up the M&E institutional framework. Duly abiding by principles and
institutional framework is incumbent upon government, non-government bodies and
specific projects’ management units. Strike an effective balance between them is a
challenge. It relates to the strategic issue of institutionalizing M&E. Actually a
complex matrix of responsibilities evolves where a variety of interests commingle.
That could prove counter productive to effective M&E. What generally is needed is a
good political will and general competence of the central authority to allow for
implementing the arrangements that had been reckoned relevant.
Challenges to M&E Effectiveness
The mainstay of issues to be considered is secure M&E effectiveness through
involving outside, ie, non-government bodies. Why is that topical?
1
1)
Including outside bodies stands for an integral and autonomous M&E structure
which does not necessarily follow the administrative (functional) line structures. The
problem points to avoiding administrative weaknesses through overcoming the
inevitable restrictions of communication. There exists a formal, meaning
depersonalized and legally bound, set of relationships. Expertise is critical to
administrations that are vertically integrated. What happens to M&E effectiveness in
the instances of poor coordination and narrow technical expertise? Obviously M&E
process endures serious short falls due to lack of balance between communication and
coordination. Communication loses effectiveness which is concomitant to poor
analytical value of the final M&E produce.
2)
Unlike decision making M&E needs intensive communication on the account
of coordination, ie, on the account of mitigating the harshest effects of a rigid line
administration. The functional organizational structure becomes less effective and
irrelevant when a ministry’s administration diversifies its programs endeavoring to
cover broader beneficiaries’ needs. M&E could be likened to a specific function that
for the sake of its effectiveness needs more autonomy within the classic and, I am
afraid, obsolete for project management, administrative structure. It is well known
from management that in order to overcome these shortfalls in communication and
coordination divisional structure is put forward. It would allow M&E process to
assume an autonomous discretion and institutional framework within a distinct
administrative entity, be it ministry’s directorate, project management unit or like. In a
word, delegation of power to outside M&E bodies would allow for introducing flat
M&E structure as a counterweight to the rigid line administrative structure.
3)
Apart from the above including outside bodies stands for 1) a larger
participation of beneficiaries’ organization; 2) building capacity at local level.
4)
Therefore, increasing effectiveness and efficiency of M&E would call for
overcoming the deficiencies of the centralized line administration without challenge
it. The latter implies as well decentralization of data collection, processing, analysis
and decision making as a direct effect of instituting more or less flat administrative
structure earmarked for M&E.
But is it possible to decentralize M&E within the official structures? Two patterns of
M&E organization that had been adopted for the PHARE funded project and SIEP
Project help answer the question.
2
M&E Patterns: PHARE Program with the Labor Ministry
EU Pre-Accession and International Projects Directorate
Financial Department
- Financial Coordinator
- Technical Coordinator
-financial Controller (Representative of the State Agency for
Internal financial Control)
M&E Unit in charge
of operational
monitoring
Regional Level (28 Regional Coordinators)
Beneficiary Level
M&E Patterns: SIEP Project
Steering Committee Based on Tri-Partite Principle
Executive Director
Departments
Technical
Financial
Advisory Services
Legal Advisor
M&E Coordinator
Local Level Monitoring – By Competitively Selected Private Inspection
Companies
Local
Communities and
Municipalities
LocalLevel
Level–(Communities
& Municipalities)
3
Both M&E patterns evidence a high level of centralization of communication and
decision making regarding M&E, in particular. Primarily, that is because the M&E
arrangements follow the line functional structures of the respective official
institutions. Is it possible under these circumstances to decentralize M&E within the
official structures? I am afraid the answer is “no” because:
1)
Both development projects and EU funded projects have been designed,
planned and implemented at the Government’s discretion. They are Government led
projects and intend to meet challenges the Government reckons relevant. The
respective Government institutions are supposed to have justified their relevance to
the donors.
2)
M&E is a multi-faceted process but financial/accounting audit plays the
leading role within it. By rule financial audit is centralized because the official
institutions which are in charge of project implementation use their own centralized
bodies for this purpose. Save of institutional financial control a heavily centralized
national body which is the State Agency for Internal Financial Control enforces and
institutes a centralized pyramidal pattern of financial control.
3)
Lack of integrated to the Labor Ministry’s (or whatever else ministry being in
charge of PHARE project or development project implementation) Management
Information System (MIS) poses serious problems to M&E. Integrated MIS is a
stepping stone for designing more complex organizational structures allowing for both
flat M&E institutional arrangements and coordinated data analysis and decision
making based on the ministry’s administrative regulations. When imbedded in a
ministry’s communication and decision making regulations individual data collection
and subsequent analysis for the purposes of M&E can not be but centralized. The
issue of M&E information sharing evolves as well. The official bodies run a variety of
projects of specific individual M&E arrangements but for the sake of political
expediency the use of information produced has been always subjected to huge
centralization. It would not be physically possible to uphold a flat structured M&E
system based on decentralized data processing and analysis and centralized data
release arrangements. The other way round, centralization of compound data use
services as a sort of justification for centralizing the entire M&E process.
4
Evidences and Lessons Learned
1)
Mechanical divide between financial and technical monitoring is available.
Financial monitoring is formal; technical monitoring is substantial. Evaluating
projects’ impact requests integrity of both formal and substantial approaches. There is
more to M&E effectiveness than financial accountability. Sometime perfect financial
accountability can prove a shield for technical, ie, substantial improprieties (poor
quality, poor beneficiary need identification, cost inefficient project appraisal, inflated
administrative costs, short falls in attaining critical performance indicators etc.). M&E
procedures are being based on a clear-cut goal matrix and performance indicators.
Gauging headway made against them entails sound analytical work that requests
specialized professional skills. The below presented SIEP Project Key Performance
Indicators sample the requirement of integrated analysis of technical and financial
individual information. The synergy impact on reducing poverty and generating
sustainable jobs is equally due to financial efficiency and accountability, capacity
building with vulnerable communities, and instituting sustainable patterns of project
implementation and follow-through.
Gauging progress against such indicators as Number of Community Based Organizations
Established and Number of Partnerships with Private and Public Institutions Established relates to
the very process of identification, facilitation and training local communities and their
local organizations. Working with a pre-set number of formal indicators implies
introducing many other informal or supporting indicators. For instance, counting the
number of Community Based Organizations implies defining the scope and specifics
of local community which is per se analytical work. We fall in the same situation
when measuring the level of satisfaction of infrastructure rehabilitated with local
communities and municipalities.
Key Performance Indicators (accomplishment as of July 2005)
№
1
2
3
4
5
6
7
7.1
8
8.1
Indicators
Reduced L/T Poverty in the
Targeted Poor Communities
Number of Beneficiaries
Number of Municipality
Infrastructure Microprojects
Number of Community Based
Microprojects
Number of Municipality SubProjects Completed
Number of Community SubProjects Completed
Project Cost (BGN) – Additional
Resources to Poor Communities
Community Project Cost (BGN)
– Additional Resources to Poor
Communities
Contract Cost
Community Contract Cost
Average
Total
Targeted
16.36
Accomplishment
15.67
104.39
99,133,003.00
72.4
432,840.00
321.00
55.00
174.00
224,301.87
71,776,598.11
125,199.94
196,382.01
99,240.75
6,885,996.59
60,682,040.47
5,458,241.12
5
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Percentage of Poor
Communities/Municipalities
Targeted
Unemployment Rate in Targeted
Poor Communities/Municipalities
Number of Beneficiaries
Cost/Beneficiary
Percentage of Effective
Microprojects
Number of Effective
Microprojects
Out of Them: Municipality MP
Out of Them: Community Based
Duration (month)
Mandays
Number of employees
Cost/employee
Long Term Unemployed
Labor Content
Percentage of Microproject
Wage Payments Going to
Unskilled Workers %
Average Gross salary
Average Gross salary to National
Average %
Number of Municipalities
Participating
Number of Community Based
Organizations Established
Number of Partnerships with
Private and Public Institutions
Established
Administrative Costs as % of
Project Expenditures
B.A. (satisfaction%)
Level of satisfaction of local
infrastructure (%)
Satisfaction of local authorities’
responsiveness
Correspondence to citizens'
priorities
Project disbursement (loan)
29.33
23.64
17.21
137.36
700,000.00
92.42
25.00
117.96
99,356,164.00
36.13
432,840.00
179.20
36.01
3.85
2,015.43
17.45
12,617.82
3.49
29.49
376.00
321.00
55.00
1,446.00
646,954.62
5,601.64
1,120.33
19.05
294.54
101.22
376.00
0.24
35,894,041.33
Source: SIEP Project Appraisal Document and MIS
2)
The above stands for a comprehensive M&E process covering all the project
cycle. There is no specialized body to monitor and evaluate the effectiveness of each
of the phases of the Project Cycle under the projects in questions. M&E specialists
either organized in special unit or single-handedly enter the game too late when in the
implementation and hand-over phases possible flaws of the previous phases may
make themselves felt. For instance poor community identification and facilitation
necessarily entails ineffective capacity building and thus, poor project sustainability,
let apart dubious financial accountability and control by the community itself.
3)
Regarding individual data collection and on-site control M&E effectiveness is
additionally smothered by the centralized pyramidal project managing authorities’
administrations. It is not accidental when beneficiaries lack capacity to institute their
own M&E procedures. In order to overcome this deficiency official M&E procedures
6
should be complemented with beneficiaries’ own monitoring system. Had both
monitoring systems been formally integrated a true synergy of institutional and
informal, ie beneficiaries’, monitoring efforts would have been attained.
4)
In my understanding M&E should commence with planning and designing a
relevant goal matrix based on strategic sector priorities (poverty reduction, sustainable
employment generation, social inclusion of vulnerable minorities etc.). In a word
M&E should be initially imbedded in project planning attempting to track down
whether project goal matrix has been tailored to relevant beneficiaries’ needs.
Unfortunately this is not the case so far. Planning is far from being perfect. First, there
is a mechanical divide between financial (commissioned to the Finance Ministry) and
substantial planning (which falls in the ambit of line ministries); second, local
communities in their capacity of direct beneficiaries are not present at all.
5)
Therefore, planning relevant goal matrices proves topical to the overall project
performance. Obviously a line-ministry’s bodies should take the lead as early as the
project fiche/project appraisal document is being prepared. But involvement of nongovernment bodies is strongly recommended, especially when the goal matrix is to be
set up. Instituting a consistent way of planning target strategies and goal matrices
where government and non-government bodies are represented on equal footing is a
task of immediate precedence. Within this context the big issue of who the planning
authority should be arises.
6)
But even though we stay in the scope of formal/official M&E procedure it
becomes obvious that the monitoring process has not been fully institutionalized.
M&E complements project implementation rather than to be an autonomous process
which is incumbent upon a specialized formal structure which is committed to
monitoring the whole project cycle.
7)
M&E system has not been comprised in an integrated Management
Information System (MIS) so far. The SIEP Project’s MIS made a steady progress
with this regard where its M&E Module had been linked to the other modules. Thus,
on-line information on the project’s current status is available. But generally viewed,
integration of individual projects’ MIS to line-ministries’ one is a challenge far from
having been satisfactorily tackled so far.
8)
The above mentioned types of projects evidence both short falls and strengths
in view of centralization and decentralization of M&E. Centralization of information
flows allows for expedient data processing and minimizing data distortion due to
vested interests and lack of expertise. The relevant principle to be followed with this
regard is first, elaboration of as simple as possible measurable indicators (mandays
created, long term unemployed hired, wage earned etc.) to be covered with individual
7
data by local bodies, local communities inclusive; and second, centralization of data
collection and processing. Involving private companies specialized in monitoring and
control is strongly recommended at least at the initial stage of data collection.
Decentralization allows for involving beneficiaries to take part in data processing and
project evaluation on local and regional level. But securing similarity of data
compilation and interpretation stays a challenge under this system.
9)
The main source of restrictions to instituting a more flexible project
management system, M&E arrangements in particular, stems from the formal nature
of EU and WB funded projects. Primarily, they have been demanded by the
government and, thus, they play instrumental role to its overall policy toward meeting
the EU standards. The main contradiction lies in the incapacity of the official
authorities to set strategy apart from operational matters. Having a hold on strategic
issues (defining Target and Allocation Strategies, project capitalization, defining Goal
Matrix etc.), and rightly, the official institutions feel obliged to keep their hold on
operation management too. The only way to do so is commissioning operational
project management, M&E in particular, to their line administrations, which the least
is not cost efficient.
Alternatives
1) Radical Alternative. It takes stock from the fact that M&E has not been fully
institutionalized. Deficiencies of the now existing M&E systems that have
been evidenced by the above projects stand for an autonomous M&E bodies
where Government, Non-Government and private sector are represented on
equal footing. In this way final beneficiaries would be much more effectively
enabled to have a say on project implementation. But the line-ministries
should keep their lead since the projects in question service primarily
Government priorities and budget commitments.
The M&E bodies would be involved in the whole Project Cycle along with the
official institutions.
The M&E bodies would develop flat administrative structure which is
conducive to more effective communication and individual data collection.
M&E should integrate financial and substantial (technical) audit/monitoring.
Possible risk and weakness is the administration of integrated MIS and
coordination of data processing and analysis, in particular.
2) Realistic Alternative. It takes stock from the centralized way EU and
international development projects, mostly WB funded, have been planned and
implemented so far by the state administration. This pattern of project
financing persists.
The only way to avoid the subsequent flaws is outsourcing. Outsourcing M&E
services with independent private bodies would mitigate the negative effects
of centralization of communication and knowledge sharing. Primarily this
8
would provide M&E with own quasi-autonomous administrative structure
which would be free enough to set up formal relationships with beneficiaries’
monitoring bodies.
Blagovest Georgiev
9