Statistics, Knowledge and Policy OECD World Forum on Key Indicators Palermo, 10-13 November 2004 MEASURING FOR DECISION MAKING OFFICIAL STATISTICS FOR DECISION MAKING AND EVALUATION: TERRITORIAL INDICATORS LUIGI BIGGERI1 1. Foreward Nobody has ever doubted that statistical information and statistical methods are indispensable for taking rational decisions. However for a long time even policy makers have rarely and badly used the statistical information available and, at the same time, producers of official statistics have not always supplied the necessary statistical data, often requested in a not well organized and articulated information demand. Recently things have changed and governments at different levels are more and more interested in formalizing their decision processes and in evaluating their programmes, activities and especially intervention policies in economic and social areas. There are several reasons for such a renewed interest. One of them is the continuous process of decisional and institutional decentralization, with a particular attention to territory. Obviously, this implies adequate information and, above all, specific statistical information systems and indicator sets that official statistics should implement, in the interest not only of public decision-makers but also of citizens, so that they can exercize a documented democratic control. Istat, as the major expression of official statistics in Italy, is ever more often called on to play a considerable role in broadening statistical knowledge of social and economic phenomena for decisionmaking and evaluation purposes at national level. Special emphasis should be placed on deepening territorial dimension, which in Italy is characterized by highly articulated elements and significant gaps in development. 1 President, National Statistical Institute, Italy Rev del 1/11/2004 It should be noted that today, as never before, the sinergies in progress have been so intense between those who, for institutional reasons, demand for qualified information to orient, direct, monitor and evaluate policy-making and those who provide statistical information, in particular official statistics. In this overall framework, the active push coming from institutional bodies interested in a greater and more detailed statistical information helps us better investigate the fields they operate in and, above all, acts as a driving force to strengthening the vicious circle “new demand–new supply”. On the other hand, Istat has always positively reacted to such a push and has more and more often started significant experiences of institutional cooperation in the respect of the principles of reliability, transparency and impartiality required by public statistics. In this report, a statistical theoretical approach as well as the current policy-makers demand for statistical information will be illustrated with particular focus on the production of sectoral and territorial statistics. In this context, the experience made in the construction of territorial indicators within the Operative Project will be presented. The title of the project is “Territorial and sectoral statistical information for 2001-2008 structural policies”, and it is carried out by Istat in collaboration with the Department for development and cohesion policies (DPS) of the Ministry of Economy and Finance. The aim is to highlight how information demand arises, how it is defined and specified, which are the answers as regards the construction of a set of indicators, which are the advantages and which problems are met both in the construction of the indicator system and in their interpretation. To give a reference framework and a logic order to the presentation, we will start recalling the steps that must be implemented to define, carry out and evaluate intervention policies; whereas the development and the principal uses of indicators for the development of regional policies are illustrated by Fabrizio Barca, director of DPS. 2. Policy definition and evaluation: a statistical theoretical approach For the reasons mentioned above, the design of policy interventions and their evaluation have become, as it is used to say, a strategic activity at any level since it is an irremissible means: (i) Of Scientific support for decisional processes to make more adequate choices and to rationalize interventions, programmes and actions. (ii) Of Control of the feasibility and effective implementation of the programmed objectives with interventions and activities, in terms of quantity, costs, efficiency, efficacy, process and product quality and impact of implemented programmes; (iii) For ensuring a guarantee function with reference to users, to the investments made and to the society as a whole. To specify how policy intervention design and the ensuing impact evaluation should be organized, we can refer to the simplified framework illustrated in Picture 1 (Biggeri, 2000): Framework analysis highlights some important and specific aspects that must be taken into consideration to organize policy design and evaluation. As a matter of fact, it is necessary: (a) To analyze the context and real situation (a good knowledge is required of how the phenomenon works and how the involved units behave), and the problem that we have to face, to correctly plan the policy, the intervention and the connected information system; Page 2 of 12 (b) (c) (d) (e) To simulate the actions on which intervention is based with macro or micro models, in order to highlight and evaluate their possibile consequences, the possibile need of modifing the intervention and, in any case, to choose among the various alternative actions; To evaluate each phase of implementation of the actions and the obtained results; To use result and evaluation analyses for learning aims (that within policy evaluation is a very important aspect) and, if necessary, to change the plan and the operative characteristics of the intervention and/or to plan or to improve the information system already available; To disseminate evaluation results also as a means of social control by general public and by interested bodies. Pict. 1. Simplified framework of the design, the implementation and the evaluation of an intervention policy This kind of research is different from others both because of its aims, that are precise and definite, and because of its operative conditions (that involve decisions). Both these elements influence the theoretical design of the research and its practical implementation that, as usual, will have to find an appropriate “compromise” among the three criteria of representativeness, randomization and realism (the latter in particular for explanatory variable choice). Fundamentally for these reasons, even considering the experimental approach, the statistical design that better suits the evaluation process is the one called “controlled investigations” design that comprises the Page 3 of 12 almost-experimental designs and the designs effective for observational studies or, in particular, to make comparisons; all these designs often represent ingenious adaptations of experimental designs (obviously releasing some of the criteria and of the hypotheses that are at their basis). As everyone knows, to project these statistical designs it is essential to deeply know the nature and characteristics of the programme or of the process working model, etc that must be evaluated together with its objectives. For example, it is necessary to know which are the elements that influence programme results and how these elements presumably interact among themselves. Obviously the effects of the different characteristics of the programme and/or of the various elements must be measured using response variables strictly connected to the objectives. Undoubtedly, the analysis must be done considering the real operative conditions and consequently the characteristics of the decisional process (as highlighted in pict. 1). It is therefore strategic to define a reference framework of the situation which is the subject of the study and/or a working model, which can be more or less detailed depending on the evaluation aims, the field of interest of the programme and considering the existing economical and sociological theories. If these theories do not exist or are not convincing, it is necessary to use empirical evidence, through a sociological, economic, managerial, engineering study of the organizations and the processes involved in the programme or in the intervention (in this case, the importance of the interdisciplinary cooperation is evident). Two further considerations derive from the framework. Firstly, it is necessary to have adequate quantitative and qualitative information presented, if necessary, in an appropriate and pertinent statistical information system for definition and management (of the intervention, etc) and for evaluation; there is no doubt that to plan and evaluate in an objective way, it is necessary first “to measure”, then to have adequate measures of the variables of interest and plan the opportune quantitative and qualitative indicators. It is therefore necessary to deeply know the characteristics of data and indicators and also their possible interpretative limitations (expecially if we use proxies of the theoretical indicator, without taking for granted that the supposed associative link between the proxy and the related indicator is unchangeable both in time and space); considering that generally many elements act interdependently to cause results, and it is therefore difficult to isolate the effects caused by each element; taking into account that often there is the need to“standardize” the indicators, to eliminate the effect of the structural characteristics of inputs and/or of the reference framework. Secondly, for interpretation purposes, we always have to remember that evaluation aims at understanding if the policy (the intervention) implemented has reached its planned objective, modifing the phenomenon or, rather, the conditions and/or the behaviours of individuals belonging to the reference population observed in the real situation. As various authors underline, the fundamental methodological problem of an impact evaluation is that generally it cannot be reduced to measure only changes of a phenomenon or of behaviours between the previous situation and the situation after the intervention. Unless the coeteribus paribus condition is respected, this difference is not only attributable to the implemented policy or to the realized intervention, but it also depends on a variety of elements, among which the typical development of the phenomenon, apart from the intervention itself. What we have to measure is the net contribution of the intervention, in other words the difference between what happened after the implemented policy (factual event) and what would have happened without that policy (counterfactual event). Page 4 of 12 3. Policy-makers’ demand for statistical information: Istat efforts to satisfy the requests As already mentioned, after a long period characterized by a substantial lack of quantitative and in particular evaluation culture, in Italy a new approach is now being asserted, more oriented towards the measurement of phenomena and its use for correct policy-making. Istat has tried and tries to meet this demand by fostering the overall growth of the National Statistical System (SISTAN) through the National Statistical Programme and the work of sectoral “quality groups”. In addition, Istat2 organises every year the National Statistical Conference, a significant moment where producers and users of official statistical information convene to discuss as well as to share methods, experiences and perspectives for a more effective meeting between demand and supply of public statistics. Moreover for over ten years Istat has drafted an Annual Report that is an occasion for an evidence-based reflection on Italy’s economic and social situation and for a comparison with other countries’ situations and trends. The Report does not only deal with Italy’s macroeconomic evolution but it also contains detailed analyses on business competitiveness, labour market effectiveness and productivity, features of the welfare system and so on. The Report preparation is also an opportunity to check the quality and consistency of different data on the same phenomenon as well as data on different phenomena arising from different statistical sources. In the end, the Report highlights the most important results obtained from statistical surveys (thus increasing the value added in the presentation of the results of each single survey) and provides possible solutions to problems of economic and social policy. Today, however, to meet policy-makers’ demand for statistical information Istat more and more often participates in ad-hoc conventions with Ministries, Regions etc, that envisage the implementation of statistical information systems for decision-making and for evaluating the impact that action policies have on specific fields. However, the context shows the consequences of some important shortcomings. First of all, little importance is given to the planning of information needs that will have to be met subsequently. This is especially true when planning complex evaluation actions, which often suffer from an overlapping of various disciplinary approaches (economic, sociological, political, financial etc.). Secondly, there is a tendency to underestimate the financial resources and the technical-scientific competence necessary to carry out the planned process of measurement/evaluation; furthermore it is on territory that the most important - as well as specific – information needs are expressed even if financial investments and human resources are inadequate in this field. However, there are also cases of conventions and cooperation relationship that are gaining excellent results, both in the production of regular analysis and evaluation reports and in benchmarking exercises. No doubt, one of them is the cooperation relationship with the DPS of the Ministry for Economy and Finance aimed at building up indicators that can be used for the development of Regional Policies. 2 The organization of the National Statistical Conference is among ISTAT institutional duties as established by paragr.15 of the Legislative Decree 6 September 1989, no. 322 Page 5 of 12 This kind of activity was requested in 1989 when regional policies had become a community goal, thus emphasising the need for all the actors involved in the implementation and programming processes to be provided with a harmonised statistical information basis. Later, this activity widely developed in the context of the problems connected to measurement and evaluation in coincidence with the start of 20002006 cycle of structural funds3, both to identify areas eligible for funding and to check the criteria for breaking down financial resources. The main results are contained in the "Programme of Southern Italy Development" (PSM), where the European pushes to the programming of development policies have been implemented in Italy in an original way, through a marked theoretical and quantitative approach in relation to the past. In this document there is a macroeconomic model containing the main instrument-objective relations through which to develop a new policy of public investments based on: renunciation of past welfarism enhancement of the existing strengths a system of indicators for the measurement of the planned objectives the relation between the impact of priority actions and the overall objective Moreover an attempt was made to clarify the link between the choices made in regional and national programmes (POR) and the overall objectives established for the entire economy of Southern Italy. The first experiences have then enabled us to launch a new phase of activities and studies aimed at supporting official statistics for decision-making through a more organic programme for the production of statistics, indicators and adequate technical tools. This has led to the already mentioned project "Territorial and Sectoral Statistical Information for the 2001-2008 structural policies” whose basic characteristics will be presented below in this report. Such a project, that develops in a wide time lag (September 2001-December 2008) long enough to fulfil the objectives set, represents only one of the numerous experiences that Istat has been carrying out in this field. But perhaps the most representative characteristics of this project are its being an in-depth study of statistical information on territory and its implementation according to the logical framework of Pict. 1, as illustrated above. 4. The project “Territorial and sectorial indicators for structural policies” The project mentioned above is co-financed by Istat (30%) and Structural Funds for Objective 1 areas (70%) and aims to contribute to the overall objective of increasing and improving statistical information, especially at territorial level, so that the effects of structural programmes can be better measured. Although this project represents one of Istat’s strategic activities in partnership, it is not the only one. During the last few years many other relevant and qualified activities have been settled such as designing the information system on Handicap and Child labour within the social policy. Furthermore it has to be mentioned how relevant the surveys on Health and Social Services supply at territorial level is, which will allow the development of other two information subsystems. 3 In programming Structural Funds for 1994-99, Istat launched a project entitled “Information systems for local/territorial policies” aiming at producing detailed statistical information at local level. The project has not been developed much both for the reduced funds and the limited time horizon. Page 6 of 12 In the case of the project on Territorial and Sectoral Statistical Information, the overall objectives of action policies are set at the community level and then adjusted to more specific national objectives. Wider actions at territorial level have emphasised the need for enhancing information and cognitive bases. The increased demand for statistical information arises - above all - from the need to complete and implement the set of “key context” regional indicators and the “breach variables” indicated in the Community Framework of Support for Objective 1 Regions (QCS). These are the primary tools to measure the initial socio-economic situation and the existing structures as well as to monitor progress and results of public interventions towards local development. It is therefore obvious that the main focus of this activity was - and is - what statistical information and what set of indicators are adequate for this kind of measurement and analysis with the aim of continuously improving them. It was thus necessary to study in-depth the methodological design underlying regional indicators for decision-making and evaluation. Moreover it should be noted that each action of the project focuses on territorial variables and indicators that have been defined in close cooperation with the Department for development and cohesion policies (DPS): a) Elaboration - for the purpose of programming development policies - of a system of regional economic accounts concerning employment within specific territorial areas (territorial sections, regions, labour local systems), even through the design of estimation models and the connection between preliminary and final series; b) Updating, control, improvement and implementation of “key context regional indicators” and of “breach variables” since they represent the system of indicators that are at the basis of a process of macro evaluation of the effects of the actions carried out through the Structural Funds; c) New production of “key context regional indicators” and “breach variables” through the inception of activities linked to the production of basic statistical information in those specific areas that represent community and national priorities (such as poverty and social exclusion, water system, ICT equipment and use by enterprises, etc.); d) Specific territorial elaborations and production of integrative indicators with a more detailed territorial articulation than that available at the moment (territorial sections, provinces, local labour systems); The results of the activities implemented by the project until now are a lot. Without going into details4, it can be said that the work begun at the end of 2001 resulted in the: 4 Production of regional statistical indicators. This is probably the most relevant activity of the whole project, considering both available resources and the importance the indicators have for the effect evaluation on Objective 1 regions. There are 111 regional indicators (14 “breach variables” and 97 “key context indicators” articulated in six line of intervention provided for by QCS) organized following the Italian regional division and the deriving territorial aggregations; they are available in time series from 1995 until the most recent available datum. This data base is For a more detailed description of the project contents please visit the website http://www.istat.it/Banche-dat/index.htm or refer to the text presented at the VI National Statistical Conference that is available at http://www.istat.it/Eventi/sestaconf/interventi/cuciani.pdf . Moreover on February 2, 2004 a Seminar on the results of the project was held at Istat, Aula Magna. It was also an opportunity to deepen topics which will be dealt with in the next few years. The related documents can be found at http://www.istat.it/Eventi/seminarioterritorio.htm. Page 7 of 12 integrated with meta-information that explains the objectives, even to non institutional users. Considering the system of indicators, Istat has the duty of: - assuring transparent definitions, classifications, elaboration techniques and evaluation that are agreed at national and international level. - harmonizing the indicators considering the Italian context as regards availability and utility in providing information; Estimate of socio-economic aggregates with a greater breakdown than the traditional one. It is worth noting that big efforts were made to increase the information available for the 784 local employment systems in Italy. In this connection, it is possible to have accurate information about the value added produced and employment by macro sectors of activity, resident population employed, unemployment; these quantities are also available in time series. Enhancement of the existing territorial statistical information. The most relevant activities regard the production of two territorial data banks containing variables and indicators on infrastructures in Italian provinces (NUTS3) and on the most important socio-economic aggregates available for municipalities (level LAU2). The potential of the indicators is enormous: they allow us to make temporal and spatial comparisons, more specific sectoral analyses as well as the analysis of regions’ convergence towards common or optimum values. On the other hand, when evaluating the effects of intervention programmes, “key context indicators” do not allow us to distinguish the so-called exogenous factors; during a given time lag, changes in economy can be caused by factors that are not linked to the actions carried out through the programmes: the increase in active population may cause an increase in unemployment notwithstanding the employment policies implemented by the Structural Funds. 5. Other direct and indirect advantages linked to the implementation of the informative system for structural policies The attention towards territory arises from a general need of phenomenon knowledge in the articulated Italian context still subject to north-south dualism and where often economic and social deviance coexists with situations of excellence. Moreover, territory has become even more important for the implementation of the programmes that are co-financed by Structural Indicators since it is just within territory, and in particular through Regions, that objectives (even though in coherence with national and communitarian levels), policies, actions and result evaluation are determined. From the point of view of statistical information, territory represents an active variable since it is often correlated with many other phenomena; but also from the point of view of action evaluation it represents an active variable since it can influence success or failure of an action (for instance, the level of infrastructures, human and cultural resources of an area, etc). Two important elements stem from this situation: Page 8 of 12 In the phase of intervention planning, the information available with an adequate territorial detail must be generally wider than the information used for evaluation (for proper and further information about the contextual situation, see pict.1); Due to the implicit correlations between territory and social and economic phenomena, it is necessary to study the interrelations between base conditions, tools and the actions to be implemented. It is therefore clear that official statistics has to supply more statistical information that must be compatible with action government levels. However policy areas do not often coincide with the usual administrative sections (regions, provinces and municipalities) that represent the territorial dominions usually utilized by statistical production. Negotiated programming instruments (territorial pacts, area contracts, programme contracts etc.) and territorial integrated Projects - instruments planned for by Operative Regional Programmes (POR) - focus on local realities sometimes at a sub-municipal level, often at a sub-provincial level and more often they represent sections of territory that overlap administrative divisions. Therefore information demand that more and more often is required is addressed to "micro" knowledge of territory and on these dominions the possibility to carry out temporal comparisons is also called for, in order to estimate the effects of actions. Since surveys at territorial level are not always available, this means that it is often necessary to improve and increase the availability of statistical information testing the use of administrative archives (data mining, record linkage and statistical matching), the use of statistical models (estimation models for small areas), the enhancement and the widening of current surveys are only some of the tools that official statistics has to widen knowledge on territory. It deals with the activity of learning and improvement highlighted in Pict. 1. An interesting example in this respect derives from Istat participation in a working group aimed at identifying territorial areas suffering from industrial crises. After a preliminary experimental phase based on the analysis of statistical provincial indicators, a more detailed territorial dimension (local labour systems) was examined. The strength that allowed us to apply an original methodology for the identification of critical areas has been the availability of new data and indicators based on local labour systems, a territorial entity more suitable to underline critical situations than the provincial dimension which often, due to its administrative nature, hides phenomena that occur in circumscribed areas. From the point of view of learning and improvement, another result (output) which is not explicitly represented, but not less important, regards the possibility of systematizing the boost coming from the new demand expressed by the Institutional actors. As a matter of fact, it is clear that improvements in methodology and production (also in the improvement of production timeliness of territorial statistics) both obtained and to be obtained - will become a part of the ordinary supply in order to enhance official statistics as a whole. In this context we have to remember that many of the activities provided for by the project are already an integral part of the National Statistical Programme (NSP). It is clear that a new demand for territorial statistical information cannot be met only through its production and/or dissemination but goes together with an intense activity of organization, analysis, elaboration in the kind of “what happens if” and therefore with the need to supply clear decision-making instruments. Therefore Istat has to manage technical assistance activities and accompanying measures, studies and enhancement of detailed territorial data bases (municipalities and local labour systems). Page 9 of 12 Finally, the publication of statistical information, reports and analyses has - and will have - considerable consequences for local actors and all the other actors involved in local development; in the future, through a wide dissemination of the obtained results, they will have a great availability of social and economic statistics at a regional or – in many cases - sub-regional levels and the citizens can have a social control over the interventions. Increased territorial information has both a direct and indirect impact. If it is true that those who have demanded for specific information are the first users of it, it is also true that the indirect effects on the whole population are far from being irrelevant, also for the scientific world or for their use in various contexts. In particular, the demand for territorial statistics has generated at least three types of impacts: (a) a relational impact, meaning an intense cooperation activity and an exchange of territorial data and methodologies among the various competent Sections of Istat; which has surely contributed to increase sensitiveness to the topics of territorial data and indicators. (b) An institutional impact, through the involvement and sensitization of other administrations or bodies to the provision and production of territorial data; which has contributed to disseminate a “good practice” model of institutional relationships within Istat and Sistan for data production and information sharing. (c) An external impact to meet the demand for information on territorial indicators coming from other actors not directly involved in policy-making (local authorities, universities, students, enterprises, etc.). Moreover, a contribution to the achievement of this goal has been the improvement of data accessibility (internet, paper publishing) for a wider range of users. The attention that Istat is giving to territory in the production of official statistics proves how urging the problem is. Much can - and will - be done in the next years to improve the information available for evaluators, policy makers as well as the academic and scientific world. 6. Open issues and final conclusions The activities and products of this project testify the pivotal role that official statistics plays to support evaluation and policy. Besides, it confers an important appreciation to Istat professionalism. This experience represents a change of course in the relationship between official statistics and the subjects involved in policy choices: in the past these two worlds were nearly detached and the dialogue between them took place in the distance; nowadays the need for correct, transparent, impartial and public statistics represents a crucial element for a more conscious implementation of the activities whose aim is the development of our country. Many are, however, the issues to be solved before fully reaching the goal of production and dissemination of territorial statistical indicators. First of all, it should be stressed that the functions of those who design indicators able to measure or to evaluate policy effects need to be better integrated with the functions of those who are (then) called on to implement them. Such an exchange of competence can prevent, on the one hand, the risk of a bad or distorted reading of the indicators; on the other hand, it is functional to identify the best result among the possible ones. It is, therefore, necessary to point at a more systematic demand from institutional bodies not only through coordinating the needs of the subjects that express it (central administrations, regions, local authorities), Page 10 of 12 but also through setting up common rules and schemes to be applied in the various contexts of evaluation and monitoring. All this aimed at optimising the limited resources available. Another possibility is to improve basic surveys in general. Looking carefully at the territorial dimension, while programming, can affect future production costs as well as the enhancement of quality in basic information. Following this view, awareness in Sistan bodies needs to be increasingly raised, especially in the Regions which are often a not-yet explored information source, and it may also be obtained by strengthening the functions of the “quality groups” envisaged for the drawing up of the National Statistical Programme (NSP). Last, but not least, the issue of the resources available needs to be mentioned. In Italy, considerably few resources are devoted to the production of official statistics and to research activity in general, which of course is a very critical point. The prevailing idea is that information is a good and not a cost; it is rather something all the people have or need to have. Unfortunately, this is increasingly less true and it is even less true when information is required at a detailed territorial level. Efforts should be made to increase the resources available. It is therefore indispensable that those responsible for development actions, especially Regions, include in their budget the necessary means to increase the information basis that can be used for more correct and punctual evaluation and monitoring. This is even truer in view of the new programming cycle of the Structural Funds, a large share of which has currently been allocated to the Regions through the POR. The growing complexity of development policies - in terms of variety of areas involved, interventions at detailed territorial levels, cognitive needs for programming, evaluation and monitoring – challenges official statistics and the National Statistical System in terms of efforts, projects, survey methodologies, coordination and resources to be used. In this changing context, we cannot ignore the implications that will arise from the reform process of EU structural policies after the enlargement to 25 Member States and the mounting request for subsidiarity coming from Regions in programming their structural policies. The above mentioned project is considerably improving data and territorial statistical indicator availability while keeping a high-degree reliability, typical of official statistics. But to improve the culture of measurement, in particular about territory - as everyone is expecting - all the actors involved have to give their indispensable contribution. If from one side we ask for systematic, well projected and adequately financed territorial statistical information, from the other side it is the duty of official statistics and of all the Sistan bodies to direct and to manage this demand, using the highest number of available resources. From the confluence of these two urges, we expect a better knowledge of territory and the availability of adequate indicators for implementation and control of development policies. In other words, a better service for society and our country in the next few years. Page 11 of 12 References Biggeri L., Problemi riguardanti la misura statistica per le valutazioni delle politiche, in Rivista Italiana di Economia, Demografia e Statistica, Roma, Ottobre-Dicembre 2000. Cipe, “Orientamenti per la programmazione degli investimenti nel periodo 2000-2006 per lo sviluppo del Mezzogiorno”, Delibera n.71/99 del 14/5/99; Cipe, "Programma Statistico Nazionale", Delibera n.97/2001 del 15/11/2001 ISTAT, I sistemi locali del lavoro 1991, Roma 1997; ISTAT, Programma Statistico Nazionale, triennio 2002/2004, Roma 2001; Ministero del Tesoro, del Bilancio e della Programmazione Economica , Programma di Sviluppo del Mezzogiorno (PSM), Roma, 30 settembre 1999; Ministero dell’Economia e delle Finanze, “Quadro Comunitario di Sostegno per le regioni italiane Obiettivo 1 2000-2006”; Ministero dell’Economia e delle Finanze, Programma Operativo Nazionale “Assistenza tecnica e Azioni di sistema”, Roma, gennaio 2001; Ministero dell’Economia e delle Finanze, Programma Operativo Nazionale "Assistenza tecnica e Azioni di sistema" per le Regioni Obiettivo 1 2000-2006- Complemento di programmazione, Roma, 16 luglio 2002; Pellegrini G., Fondi Strutturali 2000-2006: il contributo del gruppo di contatto, Giornale del Sistan n.13 2000, Roma. Page 12 of 12