Forming evaluation capacity and culture in the Czech Republic: experience
with the first set of ex ante evaluations of programming documents
(with special focus on evaluation of NDP)
Jiří Blažek, Charles University, Prague, blazek@natur.cuni.cz
Jan Vozáb, Berman Group, Prague, jbv@iol.cz
Introduction
This paper aims at deriving lessons from the first set of ex ante evaluations of programming
documents in the Czech Republic (with special regard to ex ante evaluation of Czech National
Development Plan) and outlines the needed steps in forming evaluation capacity in the
country. Though the paper focuses mainly at practical and organisational aspects of evaluation
it also establishes some broader questions in terms of the programming documents as an
expression of public policies and development needs.
1. Context for support policies and programmes in the Czech Republic
One of the major problems faced by both authors of programming documents and teams
responsible for their ex ante evaluation was the question how to fit the future programmes of
EU cohesion policy within the framework of existing national support programmes. In the
sphere of support policies and programmes the Czech Republic suffers from several principal
shortcomings. These shortcomings stem from an inadequacy of the public development
policies in the Czech Republic. In general it can be stated, that Czech Republic does not have
any comprehensive, structured and objective oriented development policy or set of policies
and corresponding development programmes. Instead – as argued later – there are many
fragmented and partial programmes focused on short-term difficulties. These programmes are
mostly supportive instruments with no explicit development goals/objectives. Hence they are
reactive forms of redistributions open to various partial needs of certain interest groups (e.g.
SMEs, territorial units, larger companies, local governments). The Czech national policies are
not based on pro-active or even strategic programmes focused on medium/long-term
deliberate changes, neither in their individual cases nor in their whole complex. Consequently
the managing and implementing structures for national programmes are created on these
assumptions.
Current status of the public support/development policy is closely related to the way how the
package of Czech support programmes has been developing since the beginning of the
transition.
1.1. Design of policies in the sphere of economic and social cohesion
In the first period of transition, approx. until 1996, the role of regional policy was
intentionally marginalized. The reasons for this „low profile“ approach of the Czech
government were historical and geographical (small inter-regional disparities inherited from
the former Communist regime), economic (until 1996 unusually low rate of unemployment of
only about 3 to 4 percent), and political (proclaimed one - sided liberalism, unwillingness to
intervene and make any “exceptions” in market rules). The official regional policy was very
modest and consisted only in the offer of modest support to small and medium size firms (in
the form of soft loans) in assisted regions selected in principle on the basis of the
1
unemployment rate (e.g. more than 5 percent in 1996) and did not require any
strategic/programming document.
Nevertheless, the Czech government gradually developed a whole array of different policies
with (intended or unintended) significant regional impacts. But these policies do not comprise
a comprehensive approach to economic, social and regional development. Almost every
ministry developed some programme with important regional impacts. There are numerous
examples: the Ministry of the Environment distributes the resources from the State
Environment Fund to projects aiming at improvement of environment in most polluted
regions; the Ministry of Transport supports public transport in rural areas; the Ministry of
Agriculture supports farmers in less favourable conditions or in environmentally protected
areas; the Ministry of Labour allocates funds on active employment policy to district job
centres according to unemployment rate; the Ministry of Trade and Industry supports by
regionally differentiated incentives the foreign investors through its agency Czech Invest.
Altogether, in the Czech Republic, there are currently dozens or even hundreds of support
programmes, each with different priorities, incentives, financial resources, project selection
criteria and different time-schedule of assistance. In addition, the whole „system” of support
programmes is due to frequent changes rather unstable which limits its effectiveness and
efficiency.
The main weakness of this approach lies in its institutional fragmentation and lack of
horizontal co-ordination at the governmental level. This problem had been until 2000 even
multiplied by missing regional self-government. Excessive fragmentation of support programs
significantly increases the administrative costs of implementation of these programmes,
especially of projects elaboration, evaluation, selection and contracting, but also adds up the
costs of financial and physical controls. Huge fragmentation of support programmes thus
limits also the amount of money allocated to support measures and thus further decreases
efficiency of these programmes.
Even when the context for development policies has changed significantly (both in political
and economic sense), overcoming of these accumulated problems proved to be quite difficult.
Consequently, preparation for the EU policy of economic and social cohesion (ESC) is still
hampered by the lack of several key elements: missing stable and legally codified division of
competence among different level of public administration (esp. due to recent abolishment of
the level of districts and introduction of new layer of “smaller districts”), unfinished reform of
administration on a central level, not completed system of financial management and control,
missing system of evaluation. Another problem is the lack of qualified people able to prepare
and implement projects according to demanding EU regulations as proved to be the case in
several pre-accession programmes. Therefore, a key question remains how in this context
design a proper strategy in the form of programming document.
1.2. Conceptual problems in drafting of programming documents
Considerable problems concern the poor horizontal and vertical co-ordination of different
bodies, subjects, programmes and policies. However, there are also conceptual problems
related directly to design of programming documents. These problems are ranging from the
very elementary problems like whether to focus the strategy on narrowing the gap between
the Czech Republic and the EU or on fighting with growing internal regional and social
disparities, through missing strategy in some sectors, the lack of multi-annual budgeting of
public budgets, unclear criteria for selection of partners, to lack of experience with evaluation
2
and monitoring (esp. selection of indicators, quantification of targets and missing
consideration of alternative strategies).
Traditional weakness of Czech programming documents is insufficient outline of underlining
strategy based on a logical chain of intervention. Other problems are safeguarding an internal
consistency of all programming documents and an elimination of mutual overlaps. Nearly
chronic problem of majority of programming documents is disproportionate stress put on
analytical part to the detriment of strategic part, namely, alternative development strategies
were considered only exceptionally and development strategies are outlined only very loosely,
rather in the form of list of priorities covering nearly every field of possible intervention than
explanation of a real strategy.
To this “wide” all encompassing approach contributed fears about insufficient absorption
capacity at the beginning of drafting of these documents in the late 1990´s when it was
expected that the Czech Republic would receive form Structural Funds and Cohesion Fund
4% of its GDP. Therefore, motivation for this broad approach was not to exclude any
potential project by a narrow definition of priorities and measures. As the financial framework
has changed (decreased) significantly (to only about 1% of Czech GDP), this tactic proved to
be counterproductive, but than it was too late to change the strategy and to make significant
cuts in list of priorities.
The key problem, and likely the most important one, underlying the whole programming
effort of creation of the NDP and operational programmes (OPs) is missing experience and
tradition of using supportive tools/incentives for development purposes. Moreover, many of
the tools and measures introduced by EU cohesion policy are either new ones or are proposed
in a new context. Given that the NDP and OPs form the very new kind of programmes in the
Czech Republic, and given the tools/measures are sometimes proposed with only limited
previous experience, the resulting complex set of strategies, priorities, measures and
implementing structures is to a certain extent artificial. Moreover, it is only partially linked to
the existing Czech national policies. Though many of the elements of new structural funds
programmes are aimed at using existing support mechanisms, it is very difficult to
assess/estimate their mutual determination, individual results or even common impacts at
various levels of implementation.
2. Procedures and organisation of the ex ante evaluation.
Ex ante evaluation was organised as a new exercise in the Czech Republic. There has been no
previous experience in evaluating national programmes and policies. Only EU pre-accession
instruments (namely various components of Phare programme) have been evaluated so far.
Yet, this evaluation was made mostly at project level or focused on annual „programmes“
which scope was very limited. Therefore, the capacity to carry out the ex ante evaluation was
rather limited among the Czech institutions (consultants, companies, universities). On the
other had, since the programming effort and preparation of documents for Structural Funds
has started about five years ago, there has been a number of organisations and individual
consultants experienced in planning and programming procedures according to the
methodologies and regulations of the EU.
2.1. Selection of ex ante evaluators.
3
The ex ante evaluation had to cope with two different and to some extent contradictory
requirements. First, the need for a previous experience (both with programming as well as
evaluation techniques and methodologies) so that the evaluation could be done properly in
terms of the EU standards. Second, need to command with a knowledge of the Czech
environment, which was necessary to evaluate the proposed activities and programming
documents on the background of national context. Resulting decision of future managing
authorities, which were responsible for contracting ex ante evaluators, was to hire local
consultants in most of the cases (except OP for Tourism, which was at the very end integrated
into Joint Regional Operational Programme).
Such a decision was complicated with the issue of conflict of interests. Most of the evaluators
had been involved in some of the previous programming efforts. Therefore, the attention was
paid by the managing authorities to select such an evaluator which had not participated in
preparation of a programming document which was subject to evaluation and which had not
have previous relationships to respective authorities. Based on the experience of authors and
given the results of the ex ante evaluations, the conclusion may be rather positive. The
independence of ex ante evaluations has been ensured to the extent that they have not been
bound by any previous obligations to the management authorities. It was also ensured that the
evaluators did not participate in the previous preparation of the programming documents they
were to evaluate.
International expertise in ex ante evaluations.
Some of the ex ante evaluation teams were also strengthened by involving international
experts. They were invited in order to facilitate transfer of know-how with ex ante evaluations
to the Czech Republic. Such an expertise has been used in several major areas.
First, foreign experts responded to ad hoc questions of other evaluators, but brought in also
the skills and knowledge of non-written but commonly known practices. At least in the case
of NDP the stress on well developed strategy supported by solid arguments and carefully
linked to other parts of the document was partially result of the international expertise. They
also assisted the national evaluation teams by pointing out to the facts (or missing points)
which the national evaluators might find less important.
Second, the foreign experts provided methodological guidance in cases when national
evaluation teams were in doubts. The general environment in the EU and in accession
countries is different which resulted in different approach to ex ante evaluation, but certain
common points and methodological approaches had to be followed. It was also helpful to
confront methods and techniques used nationally with techniques (and relevant results) used
abroad. At least the view of ex ante evaluators was broadened, which may have contributed to
a better standard of ex ante evaluation. On the other hand, the teams, which did not use the
service of international experts, produced the evaluation of comparable quality. Therefore, the
role of international experts should not be overestimated.
Third, but exceptionally, international experts worked on some specific parts of ex ante
evaluation by themselves and were directly involved in drafting parts of ex ante evaluation.
Rather considerable obstacle was in this case a limited knowledge of national environment.
Therefore, this approach was used particularly in case of the NDP because the Plan at national
level may be less specific and more formal. As such it is more suitable to be dealt with by
4
foreigners. A specific case of NDP was that the expert involved in the team had a long-term
knowledge and experience from the country.
To sum up, international experts were involved only to the limited extent in the ex ante
evaluations. Where they were involved they proved to be helpful members of the team,
particularly when being supported by national consultants who provided national framework.
The international experts focused mostly on methodological issues, on explanation and use of
common practices, on explanation of regulatory background and formal requirements.
2.2. Process of ex ante evaluations.
Ex ante evaluations were organised, in line with the EU working paper requirements, as
parallel exercise along with drafting programming documents. It was expected, from the very
beginning, that the evaluation will be carried out in two phases. First, it will be an ongoing
evaluation, which would deal with individual chapters, and second, the final evaluation which
would provide for the evaluation of the whole programming document in its complexity and
various internal and external relationships. Though the attitude of several ex ante evaluation
teams was a bit different, generally all teams followed this model. The main goal of this „twolevel“ evaluation was to improve the programming document on an ongoing basis and finally
to provide the comprehensive and overall evaluation to be used in the next stages in
programming process.
First phase, ongoing evaluation, provided for informal remarks and then formal
recommendations on individual chapters of programming documents as they were
continuously drafted. It was assumed this approach would allow both to incorporate a
standpoint of evaluators as well as to proceed with preparation of programming documents at
the same time. However, a serious obstacle appeared when using this approach. Chapters of
programming documents were not always drafted continuously and in a logical order. Rather
they were often prepared almost in a random order which resulted in weak relationships and
low internal coherence among them. This situation made it difficult for the evaluators to issue
fully developed recommendations dealing with all the necessary questions as well as for the
authors of programming documents to incorporate all the recommendations of evaluators.
Second phase, though meant as a separate one, in fact overlapped with the first one. The
comments to the first complete draft of programming documents were made while the
documents themselves were re-drafted based on the previous ex ante recommendations. This
rather complicated and to some extent confusing situation was caused mostly by an enormous
amount of work to be done by programming teams at a very short time. Since programming
documents create an interrelated system with links to each other and with the NDP providing
„an umbrella“ to the operational programmes, the change in one parameter or substantial shift
in one document called forth changes in other documents. In such a situation the evaluators´
external inputs may have caused even more complications to the effort of drafting teams.
Partly because of the reasons described above the approach of ex ante evaluators evolved in
time and resulted into rather close cooperation among the programming and evaluation teams.
Though the attitudes of the evaluation teams were different in cases of various programming
documents it can be made some general conclusions. The approach of ex ante evaluators
usually started as a „distant“ commenting on the outputs of authors of programming
documents and developed over time into a very specific kind of technical assistance as the
evaluations went to their end. Despite the fact that the programming teams and the evaluation
5
teams cooperated to a certain extent the evaluation ongoing outputs were always independent
and often very critical and unkind. Finally, the work of evaluators had a lot of features of
technical assistance because of the need to translate the evaluation remarks, comments and
requirements into some positive inputs into continuous drafting of programming documents.
As it may seem apparent from the above described situation, critical issue in the relationships
between the ex ante evaluators and authors of programming documents was the experience,
skills and role of actors at both sides. Even if the generalisation in this particular case is not
easy because the situation in each of the programming documents was different some
conclusions can be drawn. The evaluations were made by professional teams of experts who
had programming/planning experience, broad knowledge of the relevant field and who were
able to devote substantial amount of their time just for the evaluations. On the other hand the
programming documents were in some cases (but not in all) drafted by professionals from the
ministries whose practical experience with planning exercise and with implementing
programmes were sometimes limited and who had a lot of other duties additionally to their
programming responsibility. These imbalances may also be one of the reasons for the
difficulties during the programming and evaluation process as described in this paper.
3. Objectives of ex ante evaluations in the Czech Republic.
Objectives of the ex ante evaluation are given in general by Regulation 1260/1999 and
specified by Working Papers No. XXX on ex ante evaluation and on monitoring indicators.
Overall, though rather simplified, aims of ex ante evaluation are to improve the quality of
programming documents and consequently the interventions planned. It shall lead to better
addressing the development problems and to more effective solutions.
Traditionally, the focus of ex ante evaluation consists of several partial issues:
(i)
(ii)
(iii)
rationale of market failure and needs to address it,
relevance, effectiveness and efficiency of interventions and, finally,
expected impact of development activities and programmes as a whole.
However, there are some other issues to be dealt with in the Czech Republic and perhaps in
other accessing countries as well. This is because of the limited experience with development
programmes in the Czech Republic (see section 1 of the paper) comparing to the current
members of the EU. Particularly the programmes of the specific nature like the EU
programmes of ESC policy are introduced into the Czech Republic as a new, comprehensive
approach.
Additionally, the Czech Republic still has many features of transition countries despite far
reaching changes over the past ten years. Therefore, the quality of programming documents
and the success of planned interventions are determined by additional factors to these
traditionally concerned in the ex ante evaluations. These factors relate mostly to two broadly
defined areas. First, the institutional environment, which means the ability and capacity of
institutions to identify development needs and problems, to suggest relevant tools and
solutions, and to undertake necessary activities in order to achieve results. Second, the
development needs and problems themselves, which are not clear enough. Often, their nature
and causality is not properly described and understood which makes it difficult to address
them.
6
Given the specific situation as described above, the ex ante evaluation of the Czech National
Development Plan stressed four other objectives:
(i)
(ii)
(iii)
(iv)
to specify and narrow the NDP, its objectives and priorities in order to deal with
limited number of problems,
to clarify, simplify and shorten the NDP so that it can address the problems in a
more efficient way,
to discuss the NDP and its interventions within the framework of the accession of
the Czech Republic to the EU,
to review the NDP against existing national development/support policies,
programmes and their administrative/management structures.
These four additional features of the Czech NDP ex ante evaluation appeared in the ex ante
evaluation of operational programmes as well. Nevertheless, their combinations and
importance varied from programme to programme. Particularly, the first two issues, dealing
with the problem focus of programming documents and with their broad scope were often
significant for increasing quality of the operational programmes. These two issues also
affected the NDP itself most significantly although indirectly.
4. Main weaknesses of NDP as identified by ex ante evaluation
Main weaknesses of NDP as identified by ex ante evaluation can be summarized in the
following points:
Analytical part of NDP was quite excessive, without clear focus on the key problem areas.
The general tendency of the drafting team of NDP could be described as an effort to include
all data available with limited interpretation of findings and rather weak conclusions.
SWOT analysis was originally performed rather formally and with weak ties to situation
analysis. It was structured only poorly, without identification of clear priorities.
Strategic part - instead of description of a strategy based on logical/causal links, the strategy
was stipulated only by an extensive list of priorities, therefore the overall strategy was quite
weak and vague (motto “to improve everything”). Under such condition alternative strategies
even could not be considered. One of important factors contributing to problems with
formulation of overall strategy is the fact that the ministry responsible for drafting of NDP
(Ministry for Regional Development) has no experience with strategy formulation as it is
responsible only for regional policy which is one of the smallest Czech policies. Broad and
partially even vague strategy resulted both into difficulties with division of this all-embracing
strategy among particular OPs and into difficulties with connection of this “strategy” to
priorities and measures of individual OPs. Even more difficult for ex ante evaluation of NDP
was assessment how realistic is achievement of such strategy and priorities via envisaged
OPs.
Implementation system - outline of coordination and monitoring on the level of CSF was
appreciated. On the other hand, in the model of implementation structure for OPs proposed in
NDP was not sufficiently specified which elements of the outlined structure are compulsory
and which are optional. Consequently, the structures for implementation of individual sectoral
7
OPs in the regions remained uncoordinated among themselves as well as with Joint Regional
Operational Programme.
Financial framework – In case of financial framework several omissions or problems were
identified. Firstly, the division of resources among operational programmes was not based on
any rigorous analysis but was based on a rather formal decision. The allocation of resources
was complicated firstly by a shift of resources in favour of CF while volume of SFs was cut
accordingly (67% SFs : 33% CF, instead of originally envisaged ratio 90% SFs : 10% CF)
and secondly by merging of several programming documents. (Until January 2003, the Czech
Republic has been preparing 6 sectoral operational programmes + 1 Joint Regional
operational programme, but then the Government decided that OP Transport and OP
Environment would be merged into OP Infrastructure and OP Tourism would be integrated
into Joint Regional Operational Programme). Secondly, also the envisaged rate of cofinancing was set rather arbitrary without detailed analysis of required rate of co-financing on
the level of measures in each OPs. Thirdly, a possibility to differentiate rate of assistance in
favour of selected measures in line with envisaged strategy has not been used.
5. Main benefits and improvements made as a result of ex ante evaluation of NDP.
Analytical part – as a result of ex ante evaluation this part of NDP was significantly
streamlined and focused though the clear and well structured conclusions are still a weak
point.
Strategy – Chapter devoted to strategy was a chapter which was changed most significantly
as a result of ex ante evaluation. Significant effort of ex ante team was exerted to outline a
real strategy based on logical chain of interventions.
Implementation system – ex ante evaluation pointed to the problem stemming from
improperly formulated Czech Regional Development Act which establishes Management and
Monitoring Committee of CSF while one of the principles of EU cohesion policy is clear
separation of management and monitoring functions. On the other hand, ex ante team was not
able to influence design of implementation structure even in the case where proposed
structures were clearly too complicated and mutually uncoordinated.
Financial framework – in case of drafting of this chapter the members of ex ante team were
directly involved in working group consisting of representatives of Ministry of Finance and of
Ministry for Regional Development. Therefore, the ex ante evaluators were involved in
defining methodological approach of drafting the whole financial framework as well as in
solving problems as they were emerging. From this point of view ex ante team moved quite
close to a position of technical assistance. The main results of this way of involvement of ex
ante team were changes in methodology (e.g. within the process of ex ante verification of
additionality new investments were distinguished from repairs) or in pointing to serious
problems which might significantly limit future absorption capacity.
Namely, the danger stemming from significant and growing municipal debt for absorption
from SFs was stressed. The municipal debt represents not only danger for stability of public
finances but after accession (and partially even now) the new member states will be obliged to
proceed towards the Maastricht convergence criteria. This would also require introduction of
some form of regulation of municipal borrowing. Therefore, the towns and municipalities
should already now pay a special attention to healthy financial management as large current
8
debt might prevent them in the near future from access to generally very favourable support
from SFs. This recommendation is based on experience of some of current EU member states
where municipalities are taking loans to provide their own share of co-financing. This is even
more pressing due to the fact that the most indebted are largest cities which are otherwise
having a significant capacity in preparation of eligible projects.
Finally, and perhaps the most importantly, the ex ante evaluation pointed to the fact that the
whole system of Czech national policies and programmes in the sphere of economic and
social cohesion should be reconsidered in the context of implementation of EU cohesion
policy. More specifically, the fundamental question is whether to continue with array of small
fragmented support programmes along with EU cohesion policy or whether to transform
existing policies in such a way that sources currently allocated to these policies or
programmes would be used to provide additional co-financing to projects in most affected
regions or to most desirable measures.
6. Other problems identified during the ex ante evaluation of NDP and OPs
6.1 The problem of a relation (“distance”) between ex ante evaluators and authors of
programming documents.
One of the common problems which occurred in case of ex ante evaluations of several
programming documents was somewhat unclear position between ex ante team and authors of
programming documents as described in section 2. Sometimes the evaluation gradually got
many features of technical assistance.
Related issue was also somewhat unclear status of comments and objections raised by ex ante
evaluation team. This is in contrast with the fact that the Czech authorities responsible for
drafting of the programming documents (i.e. governmental ministries) have relatively clear
procedural rules for dealing with comments and objections raised by other ministries before
any document is submitted to the Government. In that case each objection raised by other
ministries or other official authority should be specified and in tabelar form explained whether
this objection has been respected and incorporated into the document or not. In latter case
should be provided the reasons why the objection could not be respected. However, in case of
comments and objections of ex ante evaluators situation was different and comments of
evaluators were generally given lower significance.
In case of ex ante evaluation of NDP, the following observation concerning the acceptation of
comments and proposals of ex ante team can be specified. Namely, the comments and
objections which were raised during the process of drafting of the document were generally
accepted more readily than those included in formal report. In other words, the comments and
proposals raised during the drafting of individual parts of NDP were much more easily
incorporated than proposals formulated at final stage of drafting of the whole document.
Obviously, this implies even closer cooperation and connection between authors and ex ante
team.
The capability of programming teams to incorporate the evaluators´ recommendations was
also determined by the position of the programming teams in the relevant ministries. The
programming teams could easily accept the requirements which fall into the area of their
responsibility. Usually, (but again not in all cases) programming teams had their
9
responsibility and rights in the field of the Structural Funds programming documents. But the
programming documents and ESC policies in general are not too much related to the national
support programmes. Moreover, the willingness to strengthen relations between the Czech
national policies and EU programmes is not apparent and not likely in a near future.
Consequently, the capability and willingness of programming teams to deal with
recommendations that required changes of the Czech national support programmes and
policies were very limited or negligible even within their own ministries.
Finally, according to our knowledge none of the managing authorities tried to exert influence
upon ex ante evaluation team to obtain a more favourable report. Nevertheless, the ex ante
teams were aware of the fact that the managing authority is in the same time contractor for the
ex ante evaluation which is likely to lead into some form of “self-censorship”. However, the
scale of this problem is difficult to assess, but some indication might be stemming from the
fact that most ex ante evaluations were in principle positive even though admissibility check
performed by EC identified problems in nearly all programming documents. On the other
hand, one has to stress that all Czech programming document passed this admissibility check
successfully and despite some identified problems were considered as a good basis for
negotiations.
6.2 The problem of missing any clear benchmark for quality of programming
documents and of ex ante evaluation
In case of a country missing any tradition of ex ante evaluation of public programmes – like
the Czech Republic - obviously emerged a problem that demands of each ex ante team
differed. More specifically, if the ex ante team applied strict criteria on a relatively good
programme the final report on ex ante evaluation could be more critical than for a programme
of poorer quality but evaluated less strictly. In addition, in case of ex ante evaluation of NDP
a question emerged whether the evaluation should deal with the whole system (i.e. from
strategy to issues related to project level) or to limit evaluation “only” to issues concerning
directly NDP as an “umbrella” programming document. After some hesitation the team of ex
ante evaluation opted for the first option as otherwise there would be a danger that none of ex
ante teams would point to problems related to the whole system. Similarly, in case of ex ante
evaluation of OPs, the ex ante teams had to deal also with other operational programmes to
check their mutual coordination and to eliminate overlaps.
6.3 Problem of quantification of targets
Along with “standard” methodological issues related to problem with quantification of
targets, in case of Czech programming documents the quantification was complicated also by
several other factors. Firstly, the evaluation of existing national programmes (if any) is
generally performed in a formal way only, in cases of some policies or programmes only by
provision of the rough structure of expenditure within the programme even without providing
data about outputs. The results and impacts are evaluated completely exceptionally.
Consequently, the method of unit cost was of limited relevance. This is especially case of
wider operational programmes with many different priorities and measures but predominately
of NDP. In case of NDP the quantification was not based on quantification of targets by
individual operational programmes but on macroeconomic projection.
The structure of indicators was finally developed in an logical and quite a consistent pattern
which is linked to a proposed strategies, priorities and operations and their particular level of
10
specifications. However, since the macroeconomic simulation of impacts of NDP was not part
of the ex ante evaluation the targets, developed by using macroeconomic projection, are still
to a certain extent questionable
6.4 Problems with time-schedule
Despite the fact that the Czech Republic has already been drafting preliminary versions of
NDP and of other programming documents for several years and the most important subjects
were able to learn important lessons from the whole process, the final versions of
programming documents were produced under considerable time pressure. This put additional
burden also on ex ante evaluators which had to elaborate evaluation in parallel with individual
chapters of programming documents being drafted by programming teams. Since drafting
some of them lagged behind the time-schedule, an OP could even be finished (re-drafted)
after the final version of ex ante evaluation was completed.
In other case, the Government acknowledged NDP before final report of ex ante evaluation
was completed. Consequently the document has been adjusted according to at least most
important recommendations of ex ante evaluation and translated into English in the same
time. Moreover, the Government decided at the very last moment to limit number of OPs
from 6 to 4 despite the fact that the NDP was already acknowledged. All this contributed to
increase of workload at the very final stages of drafting these strategic documents and limited
possibilities to incorporate recommendations of ex ante evaluations more thoroughly.
7. Conclusions and policy implications
The main experience gained during the first generation of ex ante evaluation in The Czech
Republic can be summarized in the following points:

Any institutional structure which would facilitate evaluation is clearly missing in the
Czech Republic. One of the solutions would be establishing a network of evaluators. At
least in the beginning the aim of the network shall be sharing information and experience,
development evaluation methods and techniques in the Czech environment and perhaps
promotion of evaluation, too.

The roles of both evaluators and those who draft programming documents are not always
clear which may result in misunderstandings. It consequently slows down the process of
improvements of programming documents. Moreover, balance of independence and
partnership between previously mentioned actors is difficult to achieve because of the lack
of practice between both groups.

No substantial methodological difficulties were identified during the evaluation of the first
set of Czech programming documents. Nevertheless most of evaluators had to tackle their
limited experience and sometimes missing knowledge of “commonly-known
practices/usages” in the field of evaluation. It was successfully solved by formal or
informal inclusion of international experts into the evaluation teams. From the long-term
perspective in could be helpful for the Czech evaluators to strengthen relationships to the
international evaluation community.

In order to guarantee full independency of evaluators the current system when the
contractor of ex ante evaluation is the Managing Authority (or the government department
11
responsible for preparation of the programming document in question) could be changed.
For example, in the Czech Republic, the contractor for all evaluations could be the
Managing Authority for CSF (Ministry for Regional Development) or National Fund (The
Treasury or the Ministry of Finance) which is not responsible for programming. This
would also facilitate mutual cooperation and more coherent approach of all ex ante teams.

The comments and recommendations of ex ante team proved to be more readily accepted
if they were suggested in early stages of drafting the programming documents. There was
limited willingness and even possibility due to considerable time pressure to reflect
recommendations which were formulated for the first time in final report of ex ante
evaluation. For the same reason, and because of the varying quality of programming
documents and their chapters, some important recommendations could not be given in
early stages of programming.

In the case of a majority of ex ante evaluations, ex ante teams sooner or later succeeded in
establishing themselves as partners commanding different spectrum of know-how then the
drafting team and not as additional (or even formal) burden.
Finally, it is necessary to stress again, that irrespective of the quality of the documents
evaluated, the recommendations are difficult to be incorporated if they address the
weaknesses of development policies itself and may require not only re-drafting the document
but change of an approach to the policy/programme and its objectives in general. Solution of
problems stemming from the overall context of national development policies and support
programmes is possible only by a radical reduction of the number of support programmes,
which should be formulated more broadly and should correspond to priorities defined by
corresponding national and regional strategic documents. A possible impetus for such a
radical restructuring of state support programmes, especially of those aimed at support of final
beneficiaries from decentralized bodies of public administration, might stem form newly
introduced regional self-governments. It is likely that the after period of solving fundamental
problems of their very existence, the representatives of new regions would expect that the
state will decentralize responsibility over at least some support programmes to them.
Even bigger impetus might be stemming from preparation and implementation of EU
cohesion policy in the Czech Republic. Hopefully, at the latest, when the support programmes
within the EU cohesion policy would be launched, the relevance of keeping the whole array
of small and fragmented national policies and programmes shall be questioned. However, the
ex ante evaluation can only point to these and other related problems, but it is clear the
evaluation per se does not have the capacity to initiate any changes in this sphere, and is
perhaps even too weak to initiate a serious discussion on these issues.
A typical example would be the Czech regional policy. It can be realistically expected, that by
the time of accession into the EU would seize the very relevance of the existing Czech
regional policy. Thus, the task of national regional policy should be primarily to eliminate the
leverage effect of support from the Structural Funds (SFs) in the form of matching grants as
subjects from poorer regions would not be able to provide sufficient financial resources for
co-financing of eligible projects. Therefore, the Czech regional policy might provide for
example additional 15% co-financing of projects implemented in most needed regions so as
the local subjects would be able to reach on the support from SFs. This example illustrate one
option how existing Czech support programmes might be integrated with the EU cohesion
policy to achieve a synergy between both programmes.
12
REFFERENCES:
BACHTLER, J., DOWNES, R., GORZELAK, G. (eds.)(2000): Transition, Cohesion and
Regional Policy in Central and Eastern Europe, Ashgate, London, 378 p.
BACHTLER, J., TUROK, I. (1997): The Coherence of EU Regional Policy, Jessica Kingsley
Publishers, London. .
BLAŽEK, J., de VET, J.M. (1999): Ex ante evaluation of Czech National Development Plan
(evaluation of the first draft version), Ministry for Regional Development, Prague.
BLAŽEK, J. (2000): (In)consistency and (In)efficiency of the Czech Regional Policy in the
1990s, Informationen zur Raumentwicklung, Bundesant für Bauwesen und Raumordnung, Nr.
7/8, pp. 373-380
BLAŽEK, J., BOECKHOUT, S. (2000): Regional policy in the Czech Republic and the EU
accession, In: Bachtler J. et al (eds.): Transition, Cohesion and Regional Policy in Central and
Eastern Europe, Ashgate, London, pp. 301-317.
6th Periodic Report on the Social and Economic Situation in the Regions in the Community,
European Commission, Brussels, 1999, 242 p.
Ex ante evaluation of Structural Funds programmes, EC, Brussels, Methodological Working
Paper.
Indicators for monitoring and evaluation: an indicative methodology, EC, Brussels,
Methodological Working Paper.
MEANS collection, 6 volumes, EC, Brussels.
Joint Regional Operational Programme, Ministry for Regional Development, Prague,
February 2003
National Development Plan of the Czech Republic, Ministry for Regional Development,
Prague, March 2003
Ex ante evaluation of OP Human Resource Development, National Observatory of
Employment and Education (in Czech), Prague, January 2003
13