2012
Prompt Evaluation of the IRRS
Mission to Greece
LUX, Ivan
IAEA
6/27/2012
Prompt Evaluation of the IRRS mission to Greece
Page | 1
Table of Contents
1.
Introduction................................................................................................................................. 2
2.
Basic Mission Data ....................................................................................................................... 2
3.
Feedback on the Advance Reference Material of the IRRS Mission ........................................... 2
Characteristic opinions .................................................................................................................... 3
General conclusions ........................................................................................................................ 3
4.
Feedback from the Team Members on the Effectiveness of the Mission .................................. 3
Characteristic opinions .................................................................................................................... 3
General conclusion .......................................................................................................................... 4
5.
Feedback from the Host Country on the Effectiveness of the Mission....................................... 4
6.
Evaluation of the Effectiveness and Efficiency of the Mission .................................................... 5
Discussion and conclusions ............................................................................................................. 7
7.
Summary ..................................................................................................................................... 7
Prompt Evaluation of the IRRS mission to Greece
Page | 2
1. Introduction
The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy
Community (represented by the European Commission) and the IAEA among others foresees as an
expected result a performance monitoring based on the evaluation of some key performance
indicators including:


annual reports of European IRRS mission findings, results actions; and
identification and communication of emerging issues, trends and challenges.
The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to
“Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it
calls the IAEA Secretariat

to review the effectiveness of the IRRS peer reviews.
In reply to these requirements a system of performance indicators have been elaborated in order to
measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators
are based on direct feedbacks from the IRRS team members and from the representatives of the
country hosting the IRRS mission.
Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given
in the present Prompt Evaluation Report.
2. Basic Mission Data
Mission type: Reduced Scope First Mission
Host country: Greece
Mission date: 10 - 20 May 2012
Team Leader: Tom RYAN (Ireland)
Deputy Team Leader: Peter JOHNSTON (Australia)
Team Coordinator: Hazem SUMAN (IAEA NSRW)
Deputy Team Coordinator: Trevor BOAL (IAEA NSRW)
Number of external experts: 9
Number of IAEA staff: 5
Number of observers: 0
Scope of the mission:
 Core modules (No. 1 through 10)
 Facilities and activities: radiation sources
 Additional areas (Module 11): transport of radioactive material, control of medical
exposures, occupational radiation protection, control of radioactive discharges and materials
for clearance, environmental monitoring associated with authorized practices for public
radiation protection purposes, control of chronic exposures and remediation
3. Feedback on the Advance Reference Material of the IRRS Mission
The Advance Reference Material (ARM) includes the results of the self-assessment of the host
country (normally performed with the aid of the Self-Assessment Tool – SAT); the Action Plan for
improvement in issues found in the self-assessment, a module-wise summary of the status and
activity of the regulatory body reviewed and a number of other documents needed for an objective
and well informed peer review.
The team members are requested to offer their opinions on the quality of the ARM by rating from 1
to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their
comments to the questions. The questions and the average values of the team members’ ratings are
given below:
Prompt Evaluation of the IRRS mission to Greece
Page | 3
No.
Question
Qr1
How complete do you consider the ARM?
4.17
Qr2
How realistic picture could you obtain on the area you will be reviewing
from the ARM?
What is your overall evaluation on the quality of the ARM?
4.00
Total average
4.06
Qr3
Average mark
4.00
Characteristic opinions
Qr1:


Qr2:

Qr3:




There were some repetitions in the SAT
Answers were not always related to the questions
Translation errors raised some issues
SAT should be revised to include only those questions that can be assessed by the mission
Repetitions should be avoided
New IAEA Safety Standards should be included
More translated documents would have been useful
General conclusions
1) The team members were generally satisfied with the ARM
2) Translation was an issue
3) Revision of SAT suggested
4. Feedback from the Team Members on the Effectiveness of the Mission
The team members are requested to offer their opinions on the effectiveness of the mission by
rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text
their comments on the questions. The questions and the average values of the team members’
ratings are given below1:
No.
Question
Qt1
4.63
Qt3
How effective do you consider the activity of the expert team during the
mission?
How effective do you consider the activity of the IAEA staff in the team
during the mission
How effective do you consider the activity of the Team Leader?
Qt4
How effective do you consider the activity of the Deputy Team Leader?
4.63
Qt5
How satisfied are you with the preparations of the mission (enough time
for preparing yourself, information provided by the IAEA, etc.)?
4.50
Total average
4.66
Qt2
Average mark
4.81
4.75
Characteristic opinions
Qt1:


The team worked well together and solved issues in a constructive way
A great deal of time was spent correcting issues with written English
Qt2:

1
The IAEA staff put in significant additional effort due to difficult personal circumstances of
one of the staff members.
No answers were obtained from the TL and the DTL
Prompt Evaluation of the IRRS mission to Greece
Page | 4
Qt3:


The team leader was very effective in evolving consensus, maintaining timelines and
encouraging team effort
A commendable effort in herding stray cats
Qt4:

Supported well, pragmatic, focussed
Qt5:


Good info from IAEA but not enough time for home preparations
Generally very good
Further comments:



The mission would have required more time, experts had to work long hours
There were a lot of discussions, which is good
IAEA should provide guidance as how to write the report in order to reach consistency
among the missions
General conclusion
1) Although the team was not very experienced (only three experts have had previous IRRS
experience), it worked effectively.
2) The activity of the IAEA staff was generally appreciated
3) The activity of the Team Leader and of the Deputy Team Leader was appreciated by the team
members
4) IRRS Guidance needs improvements.
5. Feedback from the Host Country on the Effectiveness of the Mission
The representatives of the host country are requested to offer their opinions on the effectiveness of
the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also
expanding in free text their comments on the questions. The questions, the host country’s ratings
and the associated comments are given below:
No.
Question
Qh1
How effective do you consider the mission in assisting the continuous
improvement of nuclear safety in your country?
Comment: The combination of the SAT findings and experts’
recommendations/suggestions can provide an efficient and effective
action plan for the next years, improving thus the radiation and nuclear
safety in the country
How objective was the peer review?
Comment: The final outcome can be considered as quite objective.
However, the team members during interviews presented different
degrees of objectivity
How has the mission helped the exchange of information, experience
and good practice with other countries?
Comment: During individual discussions with team members, the
counterparts had the opportunity to exchange information, experience,
good and bad practices. We had very useful practical advices. The
attitude of some team members to fully understand the framework and
system applied in the host country and to propose the most suitable
solution was very much acknowledged.
Qh2
Qh3
Mark
5
3
5
Prompt Evaluation of the IRRS mission to Greece
Page | 5
Qh4
5
Qh5
Qh6
Qh7
How consistent was the use of the IAEA safety requirements and guides
in the mission?
Comment: The use of IAEA safety requirements was very consistent.
How justified are the findings of the peer review?
Comment: All findings were justified. However, we consider that a
further effort could be done in merging recommendations, in case of
significant overlapping. Additionally, the spirit of acknowledging good
practices was not so obvious.
How relevant are the findings of the peer review for the future
development of your regulatory body?
Comment: The majority of the findings were already identified in the
self assessmenmt process and included in the proposed action plan. This
fact makes the argumentation towards the government, the
administration and the personnel much stronger, facilitating this way
the implementation of the action plan. The development of our
organization also depends on general provisions applied to the whole
public sector.
How competent were the reviewers in their review and findings?
Comment: All team members were competent. Additionally, the
experience of some team members coming from regulatory authorities
was very much appreciated, since they presented a further degree of
flexibility, taking into consideration not only theory but practice as well.
Average
4
4
4
4.29
6. Evaluation of the Effectiveness and Efficiency of the Mission
The Performance Indicators developed for the measurement of the effectiveness and efficiency of a
mission were evaluated in the extent as it was made possible by the available data. Accordingly the
Host feedback PI is not available and the Report completion PI could not be evaluated since the
report is still being processed at the time of the preparation of the present evaluation.
In the next figure the results of the evaluation of the performance indicators as well as the overall
effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the
ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention),
whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green
– optimum, white – effective, yellow – acceptable, red – to analyse).
Prompt Evaluation of the IRRS mission to the Slovak Republic
Efficiency and Effectiveness of an IRRS Mission
INPUT QUANTITIES
A priory mission data
Type of mission (1 - first or 2 - follow-up)
Host country
GREECE
value
LIMITS & PARAMETERS
Page |6
first
2012-06-26
PERFORMANCE INDICATORS
Optimum range Acceptable range
min.(>) max.(≤)
min.(>) max.(≤)
Rel.Dist
from Opt.
For average deviation
No(id)
Unit
Value
1
optimum
0.0
C1(T)
-
14
10.1
12.4
9.0
13.5
0.13
GREECE
EFF. INDICATION
col
PI
Team size
effective
0.1
C2(P)
-
110
77
94
68
102
0.17
Report length
Stating time of the mission (yyyy-mm-dd)
2012-05-20
acceptable
0.2
day
30
45
-
30
45
0.30
ARM review time
Ending time of the mission (yyyy-mm-dd)
2012-05-30
needs attention
C3(t ARM)
C4(N AWC)
-
9.00
6.60
-
4.95
6.60
0.00
Advance comments
C5(fbARM)
-
4.06
4.00
5.00
3.00
4.00
0.00
Feedback on ARM
Team experience
Number of modules (nT)
11
Number of supplementary modules (in follow-up) (nM+)
For team size
T 0-first
-0.96
C6(Exp)
-
0.33
0.50
0.66
0.33
1.00
0.30
Number of facility and activity types (nx)
1
λ T-first
1.48
C7(fbHost)
-
4.29
4.00
5.00
3.00
4.00
0.00
Host feedback
Number of additional technical areas (nT)
5
T 0-fu
3.09
C8(fbTeam)
-
4.66
4.00
5.00
3.00
4.00
0.00
Team feedback
Number of policy discussion issues (nP)
Number of NPP units in the host country (ν)
3
λ T-fu
0.71
C9(β1 )
C10(β2 )
-
1.25
1.00
-
0.80
1.00
0.00
Action Plan coverage
-
1.38
1.00
-
0.80
1.00
0.00
Beyond AP coverage
0
Pre-mission data
Number of Team Members (including IAEA) (T)
P 0-first
17.60
C11(RCont)
-
0.89
0.90
1.00
0.80
0.90
0.01
5.83
C12(RTime )
day
-
0
90
90
120
-
0.63
C13(OpenI)
C14(ρ)
-
-
0.00
0.20
0.20
0.40
-
-
1.19
0.75
1.25
0.50
1.50
0.00
Number of Experts in the Team (without IAEA) (Te )
9
λ M-first
λ F-first
Number of Experts with IRRS experience (Tx)
3
P 0-fu
54.20
2012-04-20
λ M-fu
1.55
16
λ F-fu
0.64
9
μ
0.90
ARM distribution date (yyy-mm-dd)
Number of issues in the Action Plan (N AP)
Number of advanced ARM comments from TM's (N AWC)
14
For report length
Number of findings (R+S) in the first mission (for follow-up)
For findings
Mission data
(R+S) av
39
Number of Recommendations (R)
28
R av
15.1
Number of Suggestions (S)
Number of Good Practices (G)
10
G av
(R*G) av
157.4
Number of findings also in the Action Plan (N =)
10
8
12.3
Number of issues remained open (for follow-up)
Number of Report Pages (P)
110
CALCULATED DATA
Conciseness of the Mission Report (ρ)
0.89
Size of nucl.pr. 0.6
Questionnaire data
Average mark on ARM quality
Average mark by the host country
Average mark by the Team Members
Post-mission data
Date of isuance of mission report (yyy-mm-dd)
4.06
4.29
4.66
Mission size
8.25
Opt.team size
11.3
Opt.rep.length
85
fields to input!
EFF.& EFF. OF THE MISSION TO
Mission type:
Missions duration:
2012-05-20
Average Measure of Deviation (Δ):
Overall Effectiveness:
GREECE
first mission
2012-05-30
0.08
effective
Report conciseness
-
Report completion
Open issues left
Balance of findings
Prompt Evaluation of the IRRS mission to the Slovak Republic
Page |7
Discussion and conclusions
1) Although according to the feedback from the team members show that the team was
dedicated and effective, the respective Performance Indicators warn that the team
composition was not ideal. According to the Team Size indicator the size of the team was
higher than the optimum (14 instead of 12) and the Team Experience was in the read range
(only 3 experienced members out of 9).
2) Very likely the longer than optimum Report Length is also related to the excessive number
of team members.
3) The time available for the pre-mission assessment of the ARM was clearly insufficient.
4) The conduct of the mission seems to be optimum as the PIs related to the mission findings as
well as to the team members’ feedbacks are in the green range.
5) The overall effectiveness of the mission is in the white (effective range) and its value (0.08)
is in harmony with other recent missions.
7. Summary
The data of and feedback from the reduced scope first IRRS mission to Greece have been analysed.
The following conclusions are drawn:












The team members were satisfied with the Advanced Reference Material (ARM)
Translation of materials to English as well as the use of English language were issues
The SAT needs further revision
Although the team was not very experienced (only three experts have had previous IRRS
experience), it worked rather dedicated and effective
The activity of the IAEA staff was appreciated by the team members
The activity of the team leaders was appreciated by the team members
The effectiveness of the mission was highly appreciated by the hosts
The competence of the team members were acknowledged, the objectivity of some of them
was questioned by the hosts
The hosts rated the findings sufficiently relevant and justified
The size of the team and the length of the report were not optimum
The findings of the missions cover the Action Plan, go beyond and are balanced
The mission was rated effective by the evaluated Performance Indicators