Session 4. Evaluation of publicly funded
research development and innovation:
The Integral Monitoring and Evaluation System
(SISE)
Alfonso Beltrán García-Echániz
Director General Adjunto
Fundación Española para la Ciencia y la Tecnología
The Integral Monitoring and Evaluation System (SISE)
WHAT IS SISE?
SISE was launched by the Central Government together with the National Plan
for RD&I 2004-2007
It is a tool for:
 Publicizing their activities and making them more transparent, thus providing the
Spanish society with a better understanding of RD&I activities funded with public
money
 Controlling the management of publicly funded RD&I programs
 Producing all data required for evaluating the RD&I public policy
The rationale behind SISE is the need for decision makers to count in with a
constant feedback about the implementation and the outcomes of their policies
The Integral Monitoring and Evaluation System (SISE)
WHAT IS SISE?
SISE works as the permanent observatory on the Spanish Science and Technology
System (SECYT)
SISE promotes analysis and studies that provides decision makers with a deeper
and more accurate understanding of the Spanish RD&I System
The Integral Monitoring and Evaluation System (SISE)
MAIN GOALS:
1.
Identifying which are the main science and technology indicators
2.
Evaluating the Spanish RD&I public policies and monitoring their impacts on
the evolution of the science and technology system
3.
Developing recommendations and proposals for improving the efficiency of
these public policies
4.
Disseminating all SISE outputs among the actors of the Spanish Science and
Technology System
5.
Contributing to a better data provision, including information related to
regions’ performance, and women’s participation in the RD&I system
The Integral Monitoring and Evaluation System (SISE)
STRUCTURE:
Program Monitoring
Commissions
34 commissions, one per each National Program and
Strategic Action included in the National Plan for RD&I
Shaped by decision makers and by scientists
Each commission produces a monitoring report on its
Program, that includes information on funding schemes, on
the amount of available resources, on the entities involved
in its management, on the applications’ rate of success, etc.
National Plan
Monitoring
Commission (COSEP)
Shaped by scientists, by public policies assessors, and by
decision makers from the Regional Governments
The COSEP Report includes the information contained in
each Program Report, together with further analysis on
public funding and management of the National Plan for
RD&I areas
The Integral Monitoring and Evaluation System (SISE)
SISE in the National Plan for RD&I 2008-2011:
Monitoring Reports
COSEP Report
(National Programmes)
(Monitoring Plan Commission)
SISE Report
CICYT
(Interministerial Commission for Science and
Technology)
National Plan for RD&I
The Integral Monitoring and Evaluation System (SISE)
SISE in the National Plan for RD&I 2008-2011:
SISE has been included in the National Plan for RD&I 2008-2011 as
the key mechanism for monitoring and evaluating research and
innovation policies, in order to review current activities and identify
the need for new initiatives
Three evaluation levels operate:
 Ex-ante evaluations: Focused on evaluating proposals
 On-going evaluations: Through the RD&I National Plan’s Annual
Report, and the Working Program’s Monitoring Report
 Ex-post evaluations: Focused on the outcomes and the outputs of
publicly funded RD&I activities.  PROEVAR
The Integral Monitoring and Evaluation System (SISE)
SISE OUTPUTS:
Three SISE Reports have been produced so far:
• The SISE Report 2005
• The SISE Report 2006
• The SISE Report 2007
Each of these Reports contained specific recommendations for the
Interministerial Commission for Science and Technology (CICYT).
Most of the recommendations were converted into specific political measures.
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2005:
1.
To create a reflection forum for the promotion of a National Strategy on
Science and Technology that encompass a new National Plan for Science,
Technology and Innovation. (December 2006 ENCYT  July 2007 National Plan
2008-2011)
2.
To design an All-Round Popular Science Communication Plan that fostered
scientific and technological culture. ( 2007 Science Year)
3.
To publish before January the annual Working Program of the National Plan,
containing all the forthcoming RD&I calls and official announcements, as well
the amount of money awarded to them, and what are the management units
responsible for each of them. ( Working Program 2006 and Working Program
2007)
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2005 (II):
4.
To take into consideration a division between civil RD&I entries and military
RD&I entries in the National Budget. ( National Budget 2007)
5.
To try to increase the participation of public research centres and private
enterprises in the European Framework Program ( EUROINGENIO 2007)
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2006:
1.
The National Plan for RD&I 2008-2011 should include a set of indicators to
measure its management and its outputs.
2.
The National Plan for RD&I 2008-2011 should be submitted to the Permanent
Commission before the end of July 2007, in order for it to be passed in
accordance with the Advisory Board and the Science and Technology Governing
Body. (Passed through the Permanent Commission, the Advisory Board, and
the Standing Commission in June and July 2007; Passed through the Cabinet in
September 2007)
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2006 (II):
3.
To develop an electronic platform for collecting data regarding all public calls
for RD&I projects, both from the Central Administration and from the Regional
Governments. ( SISE Platform 2007)
4.
To include in the annual Working Program all Region’s RD&I public policies, in
order to promote a fluent dialogue between the Autonomous Regions and the
Central Government. ( Working Program 2008)
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2007:
1. To carry out a pilot study on public sector technology purchase under the legal
figure of public-private collaboration contract. (The Secretary of State for
Telecommunications and the Information Society, together with the Public
Administration Ministry).
2. To develop an All-Round Popular Science Communication Plan (PICODI) for
disseminating scientific and technological culture. (The Spanish Foundation for
Science and Technology - FECYT).
3. To develop an in-depth evaluation analysis on the 2000-2003 National Plan.
(FECYT).
The Integral Monitoring and Evaluation System (SISE)
Main Outputs and Outcomes - SISE Report 2007 (II):
4.
To analyse which are the most suitable mechanisms to integrate the
Autonomous Regions within the National Plan’s actions towards human
resources, infrastructures, and institutional strengthening. (Ministry for Science
and Innovation).
5.
To evaluate the use of the financial expenditures allocated to R&D through the
National Budget (Ministry For Economy and Finances).
All these recommendations have been included in the Working Program for 2008
The Integral Monitoring and Evaluation System (SISE)
The Working Program for 2008
The Working Program is the annual updating tool of the National Plan. It includes:
 A full timetable with all forthcoming public calls, including deadlines and management
units.
 An expenditures’ distribution per knowledge areas and priority programs.
 A review on the strategic actions.
The evolution of the carry out of the Working Program in followed through the
Monitoring Report. It includes:
 A follow-up of the public calls published on the National or Regional Bulletins
 An analysis of each call content.
 Half-yearly reports on the Working Program development.