Report about R&D Evaluation Workshop in Japan:
Evaluation of Science and
Technology Policy System in Japan:
Country Case Study
http://www.mri.co.jp/PROJECT/2005/20051011_ird01.html
Osamu NAKAMURA, Ph.D.
Director for Technology Evaluation
Technology Evaluation and Research Division, METI
JAPAN
Evaluation of Science and Technology
Policy System in Japan:
Country Case Study
Organizers:
Japan Society for Science Policy and Research Management (JSSPRM)
Ministry of Economy, Trade and Industry (METI)
Ministry of Education, Culture, Sports, Science, and Technology (MEXT)
National Institute of Advanced Industrial Science and Technology (AIST)
National Institute of Science and Technology (NISTEP)
… In cooperation with WREN!
(Washington?/World? Research Evaluation Network)
Date:
June 2-3, 2005
Place:
Mita Public Hall, Tokyo, JAPAN
Policy Evaluation Exercises in
Seoul / Tokyo
In cooperation with WREN!
Policy Evaluation
Program Evaluation
Project Evaluation
General Discussion in Seoul
(May 30 & 31, 2005)
Country Case Study
Exercise in Tokyo
(June 2 & 3, 2005)

Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Session 1:
Science and Technology Policy System in Japan
moderator: Prof. Kondoh (Yokohama Natl. Univ.)
commentator: Dr. Holland (White House, OSTP)
Prof. Georghiou (Manchester Univ.)

Session 2:
Evaluation of Science and Technology Policy and Reflection
of its Result in Japan
moderator: Prof. Niwa (Natl. Graduate Ins. for Policy Studies)
commentator: Dr. Reeve (EC)
Dr. Shipp (NIST)

Session 3:
Evaluation of Science and Technology Programs and
Reflection of its Result in Japan
moderator: Prof. Miyazaki (Tokyo Inst. of Technology)
commentator: Dr. Oros (USDA)
Mr. Teather (NRC of Canada)

Session 4:
Evaluation of National Research Institutes
moderator: Prof. Hayashi (Natl. Inst. for Academic Degrees and Univ. Evaluation)
commentator: Dr. Jordan (SNL)
Mr. Valdez (DOE)
Kick-Off:
Prof. Vonotas
(GWU)
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Progress of R&D Evaluation in Japan
Policy
Program
Project
Institute
2002
Government Policy
Evaluations Act
Impact As.
2005
National Guideline on the Evaluation
for Governmental Research and
Development (III)
Ex-ante Monitoring/ Ex-post
2001
National Guideline on the
Evaluation for Governmental
Research and Development (II)
1997
National
Guideline on the
Evaluation for
Governmental
2001
Research and
Law on the General Rules
Development (I) of Incorporated
Administrative Agencies
Ryo Hirasawa, 30/08/2005, rh@rhirasawa.info
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Session 1: Science and Technology Policy System in Japan
Japanese Science and Technology Policy
System and Evaluation
Dr. Tomohiro IJICHI
Associate Professor, Institute of Innovation Research,
Hitotsubashi University
Affiliated Fellow, National Institute of Science
and Technology Policy,
MEXT
Ministry of Education,
Culture, Sports, Science,
and Technology
Ministry of
Economy,
Trade and
Industry
Ministry
of
Health,
Labour,
and
Welfare
Ministry
of
Agriculture,
Forestry,
and
Fisheries
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Public R&D Budget in Japan
Investigation
6%
Development
23%
20 Billions US$
Applied
27%
Basic
44%
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Comments from Commentators:
Evaluation results should be reflected to the budget on the uniform
basis across different types of researches.
In active research fields, many researchers at different research units
should compete to stimulate each other. It would produce the new
discipline with high yield. So, the balance between the administrative
efficiency and competition of researchers is very important.
Japanese evaluation system has no definite framework of ex-ante
evaluation, so few projects/programs have clear targets to be
accomplished.
An incentive system to R&D projects based on the results of evaluation
should be carefully set in order to promote reasonable competition.
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Session 2: Evaluation of Science and Technology
Policy and Reflection of its Result in Japan
Current Situation and Challenges
for Policy Level Evaluation in Japan
Dr. Jun SUZUKI
Professor, Technological strategy, S&T policy,
Graduate School of Engineering Management,
Shibaura Institute of Technology
1. Backgrounds and track records for
evaluation of science and technology policy
Law on the Evaluation of Science and Technology Policy
Conducted by the Government
（Government Policy Evaluation Act, April 2004）
METI
METI Basic Plan for Policy
Evaluations
achievement
・ex-ante evaluation for 113 policy
measures, an integration of 840
projects since FY2002
・of which 75 policy measures
(42 are R&D) subject to ex-post
evaluation
Case：solar power
generation
MEXT
Basic Plan for Policy
Evaluations
achievement
・3 evaluation procedures
ex-ante, ex-post, and synthesized
evaluation
・ ex-ante and ex-post evaluation
ex-ante: 78 projects
ex-post: 8 projects
・achievement evaluation:
42 policy measures
Case：HIMAC
He stressed the importance to employ the Balanced Score card and
Behavioral Additionality to assess the effects of project performance.
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Comments from Commentators:
It is very important to evaluate not only intended effects in the mid-term
and long-term but also secondary effects.
Besides objectives and goals, management of the performance is also
very important element of program management.
Project budget must be managed in a consistent manner throughout the
project period.
It is unclear in Japan who is responsible for the evaluation results.
The evaluation system in EU sets importance in the balance of:
accountability
transparency
justification
learning
trust in evaluation
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Session 4: Evaluation of National Research Institutes
Revised Evaluation System to Reflect
the Future
A model＠National Institute of Advanced Industrial
Science and Technology (AIST)
・Independent administrative institution under METI
・To perform innovation-oriented basic researches
Dr. Osamu NAKAMURA
Senior Researcher, Evaluation Department, AIST
Sustainable Development by Full Research
Outcomes
Outcomes
Society
AIST
Public resources
AIST places its highest priority on the pursuit of complete research, "Full Research",
engaging Type-I Basic Research to the development of “products” by conducting
intensive Type-II Basic Research.
People’s
Interest
(applause)
dream
Type-I basic
research
Universities
nightmare
reality
Type-II basic
research
Industrial
development
Industries
AIST
Full Research
Evaluation from the view point of OUTCOMES
Procedure of Evaluation
external reviewers:
internal reviewers:
Specialists or Intellectuals
Principal Reviewers
to make sure
the research
＊Roadmap
Evaluation
strategy
＊Output
to knowEvaluation
the level of performances
Strategy of AIST
＊Roadmap for outcomes
＊Output indexes
achievements
＊Management
and
＊Management
Evaluationto perform Full Research
to encourage researchers
Research Units
Evaluation: Rating with comments
Evaluation Department
Nation
Nation
President of AIST
Evaluation Committee
Evaluation
Committee
Notes:
Evaluation is for Encouraging with Love.
 Evaluation is for Communication.
 Evaluation is for Strategy.
 Evaluation is for Creation.
 Evaluation is to be fed back.
 Evaluation reflects the Future.
 Evaluation is like a Compass for Voyage.

“Revised Evaluation System to Reflect the Future” (O. Nakamura, AIST)
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Session 4: Evaluation of National Research Institutes
Evaluation System of RIKEN
・Independent administrative institution under MEXT
・To perform comprehensive researches for natural science
Dr. Masayuki UCHIDA
Evaluation Section, Policy Planning Division
RIKEN
2-1 Advisory Council System of RIKEN
RIKEN
Advisory
Council
Recommendation
President
AC
RIKEN Discovery Research Institute &
Harima Institute
Directors, DRI & HI
AC
RIKEN Frontier Research System
Director，FRS
AC
RIKEN Brain Science Institute
Director，BSI
AC
RIKEN BioResource Center
Director，BRC
AC
RIKEN Genomic Sciences Center
Director，GSC
AC
RIKEN Plant Science Center
Director，PSC
AC
RIKEN SNP Research Center
Director，SNP
AC
RIKEN Research Center for
Allergy and Immunology
Director，RCAI
AC
RIKEN Center for Developmental Biology
Director，CDB
The Board of
Executive Directors
Review
Report
Advice
Response
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Comments from Commentators:
It is very good to perform evaluation of a research institute based on
its strategic roadmap, outputs, and research management.
If you want to improve the research environment, you should set a
target and monitor indicators across years.
It must be emphasized to motivate researchers by setting
appropriate criteria for evaluation.
Important elements to create a good business model of a research
institute:
1. Core competency
2. Initiative and products (deliverables) from it
3. Leadership based on a rational criteria
4. Risk management (monetary risks, market risks, scientific risks,
non-technical risks, etc.)
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Day2
Session 1: Applicability of logic model to policy evaluation
Dr.Gretchen Jordan (SNL)
Mr.George Teather (NRC of Canada)
Session 2: Application of US PART for Japanese public RTD evaluation
Mr. Bill Valdez (DOE)
Dr. Mike Holland (White House, OSTP)
Session 3: Application of EU FW6 Evaluation system for Japanese public RTD
Prof. Luke Georghiou (UM)
Dr. Neville Reeve (EC)
Session 4: Current program evaluation methods
Dr. Stephanie Shipp (NIST)
Session 5: Training program for policy evaluation/R&D evaluation
Prof. Nicholas Vonortas (GEU)
Dr. Cheryl Oros (USDA)
Session 6: Newest evaluation tools
Dr. Tom Fiddaman (Ventana Systems)
Summary:
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
We conducted extensive discussions on methodologies/
focuses on policy/program/project evaluations on Japanese
science and technology policy system as a country case
study with the guidance of WREN– for the first time ever.
Next Steps for Japan (METI):
We are trying to introduce logic models into planning and
evaluation of METI projects and programs.
We consider that the some criteria in PART (US) or FP (EU),
such as those for management, must be incorporated into
Japanese S&T policy evaluation system.
We should get into the network of the world evaluation
community in order to apply global top-level evaluation
methodologies to Japanese S&T policy evaluation system.
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Thank you !
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
Appendix
Evaluation of Science and Technology Policy System in Japan:
Country Case Study
R&D Budget of Japanese Ministries
(including expenses for labors and facilities, 2005)
MIC
MLIT
2%
2%
MAFF
3%
MHLW
4%
others
3%
JDA
4%
36 Billions US$
METI
17%
MEXT
65%
Chiba (HIMAC)
(3) Heavy Iron Medical Accelerator in Chiba
(HIMAC)
◆Background to development
・ Construction of Heavy Iron Medical
Accelerator in Chiba (HIMAC) began in 1984
as the world’s first heavy particle accelerator
for medical treatment, based on the
Comprehensive 10-Year Strategy for Cancer
Control.
・ Clinical experiments began in 1994.
・ The device has been used in 2,184
people up to February 2005.
・ HIMAC was provided to researchers both
in Japan and overseas as equipment for
utilization in biological experiments that
would provide the basis for medical
treatment, as well as in physical and
engineering experiments. Over 500
outside researchers use the device each
year.
・ Highly-advanced medical treatment
began in November 2003, and 341
patients have undergone treatment.
◆Special features：
・ Causes no pain, placing less burden
on patients
・ Induces little or no side effects
・ Can treat lung and liver cancer
within a short period (less than 1
week)
・ Can treat refractory cancers (e.g.,
that of the bone and soft tissues)
that respond poorly to other
treatment methods