Models of Evaluation For
Research Proposals in
Turkey
Prof. Dr. Omer CEBECI
Vice President – Funding, TUBITAK
(The Scientific and Technical Research Council of Turkey)
&
S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU,
F. COSKUN, O. OZPEYNIRCI, B. DIKMEN,
G. KOZANOGLU, H. GULER & O. KARA
SUMMARY
• Research funding in Turkey, TUBITAK & recent boost
• Three-dimensional evalution used by TUBITAK for the
evaluation/selection of research project proposals
grouped under three categories:
•
•
•
•
(1) curiosity driven academic research, and
(2) customer driven applied research,
both in universities and research institutions, and
(3) technological and innovation driven research conducted
by the private industry, plus
• (4) research equipment & infrastructure.
• Details of the “Phrase-anchored rating scale”
Actors of Turkish National Science
and Technology System
(policy makers)
BTYK- Supreme Council of
Science and Technology
TUBITAK-The Scientific and Technical
Research Council of Turkey
TAEK- The Turkish Atomic
Energy Commission
MIT-Ministry of Industry and Trade
TOBB-Union of Chambers and
Commodity Exchange of Turkey
ME-Ministry of Education
The Turkish Academy
of Sciences (TÜBA)
HEC-Higher Education Council (YÖK)
KOSGEB- Small and Medium Sized
Industry and
Development Organizations
TTGV- Technology Development
Foundation of Turkey
TÜRKAK- Accreditation Board
DPT- State Planning Organization
DTM-Foreign Trade and Treasury
Actors of Turkish National Science
and Technology System
(science and research performers)
Turkish
Industry
Turkish
State
Universities
TUBITAK National Research
Other
Centers and Institutes
National Research
Marmara Research Center (MAM)
Centers
Institute of Energy
and Institutes
Chemistry and Environment Institute
Food Institute
Materials Institute
Information Technologies Research Institute
Earth and Marine Sciences Research Institute
Information Technologies and Electronics
Research Institute (BİLTEN)
National Electronics and Criptology
Turkish
Research Institute (UEKAE)
Private
Defence Industry Research and Development
Universities
Institute(SAGE)
and Research
Basic Science Research Institute
Centers
National Academic Network Center
Genetic Engineering
and Biotechnology Research Institute
National Metrology Institute
Turkish Industry Management Institute
National Observatory
Turkish National Science,
Technology and Innovation System
President of Republic
Prime Minister
BTYK
MEB
YÖK
STB
Universities
KOSGEB
TSE
DIE
TPE
DTM
DPT
TOBB
TÜBA
TÜBİTAK
TTGV
TÜRKAK
R&D Institutions
BTYK- Supreme Council of Science and Technology
STB- Minister of Industry and Trade
MEB- Ministry of National Education
TÜBİTAK- The Scientific and Technological Research Council of Turkey
YÖK- Higher Education Council (YÖK)
TÜBA- The Turkish Academy of Sciences (TÜBA)
DPT- State Planning Organization
TÜRKAK- Accreditation Board
TPE- Turkish Patent Institute
TSE- Turkish Standards Institution
DIE- State Institute of Statistics
DTM- Undersecretariat of the Prime Minister for Foreign Trade
TOBB-Union of Chambers and Commodity
Exchange of Turkey
KOSGEB- Small and Medium Industry Development Organizations
TTGV- Technology Development Foundation of Turkey
National S&T Indicators
Number of triadic patents per million population
2,5
2
Total R&D personnel per
1000 employment
Japan
1,5
Germany
1
0,5
0
US
Number of papers
per million
population
EU-15
GERD as a percentage
of GDP
GERD per capita
population
EU-15 Values (2000-2001):
GERD per capita population:467. 6 (PPP $)
Number of triadic patents per million population:36 GERD as a percentage of GDP:1,89
Total R&D personnel per 1000 employment: 10.4
Number of papers per million population:822
National S&T Indicators
Number of triadic patents per million population
1
EU-15
Total R&D personnel per
1000 employment
GERD as a percentage
of GDP
0,5
Hungary
Spain
Number of
papers
per million
population
Turkey
GERD per capita
population
Share of Turkey in World Publications (%)
Scientific Publications
1,20
1,11
1,10
1,00
0,94
0,90
0,86
0,80
0,70
0,64
0,60
0,50
0,57
0,52
0,46
0,40
1998
1999
2000
2001
Year
2002
2003
2004
Knowledge as a Tool
Knowledge is a tool that can be used for
a variety of social objectives,
including:
 Meeting Basic Human Needs
 Increasing Safety & Security
 Improving the Quality of Life
Economic Growth and Development
Knowledge - R&D Cost per Kg
Concrete
1 cent
Cement
5 cent
Steel
50 cent
Aluminium
1,5 $
Automobile
10 – 100 $
Passenger Aircraft
100 – 1.000 $
Fiberoptical Cable
3.000 $
Combat aircraft
10.000 $
Communication Satellite 100.000 $
Microprocessor (Si) Chip 4.000.000 $
Share of TUBITAK in
TR Fiscal Budget
46
50
45
40
30
10
2001
10
9
8
8
8
8
9
10
13
13
12
9
8
8
1999
5
7
8
9
10
11
9
10
8
7
8
10
8
10
11
15
10
15
16
20
17
25
8
ONBİNDE
35
5
0
1971
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
2003
2005
Research Proposals
*from universities and research institutions:
1) curiosity driven academic research
2) customer driven applied research
*from private industry:
3) technological & innovation driven research
TUBITAK Research Project
Proposals
Number of Proposal
Science and Technology Project Proposals
3000
2500
Career
2000
International
Sector
1500
Units
Infrastructure
1000
Research
500
0
1999
2000
2001
2002
Years
2003
2004
Running Research Projects
3000
2600
2500
2000
1500
1353
1212 1199
1000
694
885
889
929
899
1995
1996
1997
1998
750
982
833
837
1999
2000
587
430
500
264
0
1990
1991
1992
1993
1994
2001
2002
2003
2004
2005
EVALUATION & MEASUREMENT
TARİHTEN BUGÜNE
SAĞLIK ve ÖLÇME
• Ateş ve nabız – 1625
(Santario ve Galileo)
• Torricelli - 1643
• Tansiyon - 1733 (Hales)
• 1896 - Riva-Rocci
• Korotkov - 1905
• Ateş - nabız - tansiyon ölçüm ve
kayıtları ile hasta izleme - 1920
ÖLÇME ve DEĞERLENDİRME
“ Ölçemediğinizi anlıyamazsınız ! ”
Lord Kelvin (1824 - 1907)
(If you can’t measure it you don’t understand it)
“Her hesaba katılması gerekenin
sayılır olması gerekmez ! “
Albert Einstein (1879 - 1955)
(Not everything that can be counted counts &
not everything that counts can be counted)
Çıktı mı ? Sonuç mu ? Etki mi?
Çıktıların sonucu olarak gerçekleşen ETKİ !
OUTPUT – OUTCOME – IMPACT
IMPACTS achieved as consequences of the
OUTCOMES resulting from the
realisation of the OUTPUTS
OUTPUT – OUTCOME – IMPACT
IMPACTS achieved as consequences of the
OUTCOMES resulting from the
realisation of the OUTPUTS
( My eye jelly project )
Intended / Expected / Desired IMPACT :
. . . in full control of my mandate . . .
Unintended / Unexpected / Unwanted
BUT ACTUAL REAL IMPACT :
. . . under strict control of my wife ! . .
Evaluation Criteria
three-dimensional
evaluation criteria
developed in collaboration with
researchers & reviewers
1) Curiosity Driven
Academic Research
- the three dimensions 1. intellectual/scientific/professional merit
2. expected impact of the anticipated
outcome
3. achievability of the research with the
proposed research team,
equipment/facilities and methods
2) Customer Driven
Applied Research
- the three dimensions 1. research and development merit
2. expected impact of the anticipated
outcome
3. achievability of the research with the
proposed research team,
equipment/facilities and methods
3) Technological & Innovation
Driven Research
- the three dimensions 1. technological level of the research
2. innovative level of the product/outcome
3. feasibility of the process
Evaluation Process
all three-criteria are given equal weights
Phrase-Anchored Rating Scale
versus
Likert Scale
sub-criteria phrases describing
*very competitive
*competitive
*not-competitive
features of the proposal
Very competitive
*scientifically and professionally
outstanding & very well justified project
&
*points to an opportunity for a major
contribution to the advancement of the
knowledge and/or to the resolution of a
problem of practical importance
Competitive
*scientifically and professionally competent
and justified proposal which will make a
contribution to the advancement of
knowledge &/or the resolution of a
problem of practical value
& therefore
*support is suggested if funds are available
Not competitive
*work routine in character
*scientifically and professionally
unsatisfactory and poorly organized
Evaluation Process
5-8 Individual Reviewers
evaluate 10-15 proposals
by referring to the
sub-criteria phrases
Panels
a meeting for a final verdict
Curiosity / Merit = V. Comp.
• Outcomes have high potentials for publication
in journals or books listed in international
indexes
• The originality of the work has been
supported by extensive and critical literature
survey
• Hypothesis for evaluating the research topic
is very well defined
• Explanation and analysis of the expected
outcomes reveal the superiority of the work
in comparison to the existing science and
technology.
Curiosity / Merit = Not comp.
• Scientific consistency and the rationale of
the research are not clearly explained.
• A clear scientific / technological question is
not put forward.
• Project is more like an investigation / data
collection / routine work.
• References provided include similar studies &
literature survey does not point out the basis
/ importance of the project.
• The research is based on unreliable data and
hypothesis.
Curiosity / Impact = V. Comp.
• Very important in terms of sustainable
development of the country /very likely to be
implemented to find solutions for the
problems of society
• Very likely to be employed in different
scientific & technological fields
• Very likely to generate new projects
• Commercialization potential of the outcomes
is very high
• The project is supported by international,
national or industrial sources.
Curiosity / Impact = Not comp.
• Potential for adding value to science and
technology is low
• Subject of the project is not among the
priorities of the country
• Not likely to result in intellectual
property worth-protecting.
Curiosity / Achiev. = V. Comp.
Project Team
• The team is experienced in national /
international projects related with
same/similar subject
• They have experience as advisors /authors /
referees / editors / book authors
• The can allocate enough time for the project
• Competencies / responsibilities / roles of the
team members are well defined and adequate
• End users of the project outcomes are also
members of the team.
Curiosity / Achiev. = Not Comp.
Project Team
• The team is not experienced in conducting
projects of this size
• The knowledge and awareness of the team is
not sufficient
• They do not have important publications in
the subject of the project
• Some of the team members are irrelevant for
the project
• Essential competencies are lacking
Curiosity / Achieve. = V. Comp.
Infrastructure
• Infrastructure of the institution is very
adequate for the project
• Additional equipment requested within the
scope of the project is very compatible with
the existing infrastructure and the project
• Existing sources / equipment are used rather
than purchasing new sources
• Requested equipment can also be used in
other / future projects
Curiosity / Achieve. = Not Comp.
Infrastructure
• Infrastructure of the institution is not
adequate for the project, unless supported
with major equipment
• Equipment requested within the scope of the
project is not compatible with the existing
infrastructure and the project
Curiosity / Achieve. = V. Comp.
Methodology
• Approach / methodology are very well
designed to reach the target
• Methodology is correct and well-defined; and
standard methods and literature are cited
• Preliminary experiments have been conducted
to rationalize the hypothesis
• Alternatives (plan B) have been considered if
difficulties are encountered
Curiosity / Achieve. = Not Comp.
Methodology
• Methodology is not adequate to reach the
target
• Relations between the experiments and
hypothesis are not well defined
• Methodology is not explained with a common
scientific basis
• Possible problems and limitations are not
considered
• Statistical analytical requirements are not
considered
Curiosity / Achieve. = V. Comp.
Timeline
• Proposed period & time schedule are realistic
Budget
• Proposed budget is realistic and well-justified
• Project is also supported by other institutions
Curiosity / Achieve. = Not Comp.
Timeline
• Proposed period and time schedule are not
synchronized
• Time schedule is not adequate
Budget
• Budget is not well-define and requested
amount is too low / high
• There is no possibility of support from other
institutions
Customer / Merit = V. Comp.
• Aims to develop a national / international
novel technology (methodology, system,
product, process / technique)
• Brings comparable superiority to the existing
system
• Outcome will be a technology to be protected
under the intellectual property rights
• Work consists of a scientific and rationale
approach
• An interdisciplinary project with the
collaboration of the Public- Private IndustryUniversity-Research Institutions
Customer / Merit = Not comp.
• Literature survey and market survey are not
satisfactory, work is based on unreliable data
• No scientific / technological rationale and
integrity in the project
• It is a study that had been done before in the
country
• Aim, objective and motivation are not clear
• Outcome is not qualified as “applicable/usable”
• Not related with a real need / problem
Technological Level = V. Comp.
• Technology / product developed aims to fill a
gap in the existing technology or replace the
existing technology within the following 3 years
• Has an interdisciplinary approach to solve more
than one problem
• Rationale of the R&D is well established
(theoretical / analytical / experimental)
• Added value of the anticipated outcome of the
R&D project is considered
• Has contribution to increase R&D staff
• A doctorate / masters study is incorporated
within the project and the outcomes have
potentials to be published in
national/international journals.
Technological Level = Not comp.
• Literature survey and market survey are not
adequate
• No scientific/technological advance and
integrity in the project
• R&D rationale (analytical and /or experimental)
is not adequate
• Aim, objective and motivation are not clear.Procedure that has been used will not provide a
change / improvement in the methodology and
technology
Innovation Level = V. comp.
• Outcomes may lead to spin-off company
• Outcomes include production standards
and technical specifications
• Outcomes will affect other sectors as well
• Outcomes very likely to increase the
export capacity of the country
• Very likely to be a basis for generating
new projects / products
Innovation Level = Not comp.
• Outcome is not qualified as “applicable/usable”
• Period of usefulness is very limited or not
effective
• Does not provide new areas of work and as a
result does not increase employment
• Expenditure for the R&D study is far beyond
the expected economical benefit of the
product
• Support provided by the private organization is
not sufficient.
Comments
Reviewers (hundreds):
• The model reduced the,
– burden of the review process on them, &
– subjectivity and variability of the opinions
of the individuals
• Convenience of selecting from a comprehensive
list of phrases as well as
• the freedom of offering their own judgments
Researchers (thousands):
• Sub-criteria phrases to be very instrumental
in guiding the development of their proposals
Thank you
Hundreds of scientists, researchers and
reviewers participated in workshops for
the development of the criteria and
phrases listed in the tables.
Their contributions are gratefully
acknowledged.