2/ AMP competency assessment in the Russian Federation
In accordance with the ICAO/WMO Aeronautical Meteorological Competences,
all aviation personnel in the Russian Federation was assessed in 2013.
The works on the field of competency assessment were launched in Russia in 2012
and finished by December 2013.
After the end of the works, Roshydromet sent to the WMO Secretariat a
notification about the results of competency assessment in Russia.
3/Roshydromet structure in aviation meteorological service
In the Russian Federation, the meteorological authority in the field of
meteorological service for aviation is Roshydromet. It includes 9 federal state
institutions, which carry out the meteorological service for civil aviation and air
navigation in Russia. Each institution incorporates operational units such as
aeronautical meteorological centers, aviation meteorological offices and
operational groups, all together are 250 operational units and 2400 people of the
meteorological personal.
All Roshydromet aviation institutions participated in the procedure of AMP
competence assessment.
4/The preparations made by Roshydromet institutions
The Russian Federation held two training workshops on competency assessment
procedure. The Sixth Workshop «AMP Competency Assessment for the members
of the RAII and RAVI and the CIS countries" was held in June 2012 with 43
participants from 17 countries, and The Seventh Workshop "Implementation of the
AMP competency assessment" was held in March 2013 with 65 participants.
Both workshops were a collaborative effort between the WMO Commission for
Aeronautical Meteorology (CAeM), Roshydromet and WMO Regional Training
Centre - Russian State Hydrometeorological University (RSHU).
5/Workshops participants discussed
The workshops demonstrated how the CAeM Competency Assessment Toolkit
could be applied to align national standards with the ICAO/WMO Competency
Standard, and how to develop a plan, tailored to the national meteorological
service, for the implementation of competency assessment.
The workshops motivated paticipants to discover the significance of their own
practices and expertise to develop a competency assessment toolkit and
demonstrate it compliance with the WMO\ICAO Standards.
6/ Workshops results
At the end of the first workshop Roshydromet at the December 2012 issued the
document № 757 "Start of the implementation of the ICAO/WMO aeronautical
IX международный семинар EUMETCAL. Варшава-2014
meteorological personnelc ompetency assessment in Roshydromet", where the
standards of aviation competence for the Russian Federation were approved.
At the end of the second workshop at the April 2013 the document № 171 "The
WMO standards-based implementation of aeronautical meteorological personnel
competency assessment in Roshydromet" was issued, where the plan of
implementation of AMP competency assessment was approved.
7/ Virtual Round Table on AMP Competences
In addition in 5 June 2013 together with the WMO Virtual Laboratory for Training
and Education in Satellite Meteorology (VLab) Avaimettelecom of Roshydromet
conducted in Russian language the online meeting “Virtual Round Table on WMO
Aeronautical Meteorological Competences”.
This online event was attended by 50 experts from different Roshydromet aviation
units, who were involved in the preparation and implementation of competency
assessment. The big advantage of that online meeting was that it allowed
participating to those experts, who for some reason could not take part in the onsite workshops.
8/ Tools/techniques for competency assessment
In the Russian Federationfor while the AMP competency assessment the following
tools were used:
- Portfolio;
- Method of direct observing the meteorologist in an operational setting;
- Experiential questions, modeling the situation (simulators);
- Tests (written examination);
9/ Portfolio
Expert groups in preparation for the competency assessment developed the form of
portfolio Portfolio is the document that demonstrate competence and the
employee`s experience. Portfolio consists of two parts:
- The first part contains information on employee`s education, training
courses, working experience in meteorological services, knowledge of
foreign languages, the certificates and so on.
- The second part contains employee`s performance indicators (such as
accuracy of weather forecasts, the number of errors in aeronautical
meteorological reports, and so on).
1
IX международный семинар EUMETCAL. Варшава-2014
10/ Direct observing in an operational setting
The main method of the assessment was the direct observing of apersonnel in
operational setting. It is a process of verifying that meteorologist has the required
knowledge and skills to do the job. The purpose of this method is to watch the
actual application of knowledge and skills in usual operational settings, as well as
actions in unusual situations and solving problems. The results of assessment by
this method were recorded in the checklist.
11/ Experiential questions
Experiential questions (what would you do if ....?) and modeling situation were
used in addition to the method of direct observation in the cases where the
assessment is carried out during the period of sustained good weather conditions,
and when it was not possible to observe how meteorologist predict (or watch)
hazardous weather conditions. Using this method, special forms were previously
developed for fixing the actions of assessed person.
12/ The theoretical knowledge testing method
Testing - the theoretical knowledge testing method was used in addition to the
basic methods of competency assessment and was mostly in the forms of written
examination. The tests were developed by the expert groups and were approved by
the heads of the structural units. In the tests, person had the limited period, during
which he had to dial the set percentage of correct answers. Test results were
applied to the checklist.
While competence assessing, the experts was mandatory to use portfolio of
meteorologist and the method of direct observation. Experiential questions,
simulating and testing was used only when it was necessary.
13/ Principles of competency assessment held in the Russian Federation
All categories of aeronautical meteorological personnel was assessed apply
uniform principles
Openness–each meteorologist was informed in advance (1 month or more) about
the purposes of assessment, evaluation criteria, tools and methods, place and
timing of assessment;
2
IX международный семинар EUMETCAL. Варшава-2014
Justice (fairness) - the asked questions did not go far beyond the basic knowledge,
the experts were professional and demonstrated a friendly attitude to the each
person;
Uniqueness - each specialist was evaluated individually, comparison with other
persons was not performed;
Reliability and verifiability - all the documentation on competency assessment
stored properly in sufficient qualities and quantities; the assessment results could
be reviewed, for example, on appeal; the inspectors or experts ICAO/WMO can
check all documentation.
14/ The practice of assessing AMP competence
Competency assessment in all Roshydromet institutions was conducted in
accordance with the Roshydromet document - "Regulation on AMP competency
assessment in Roshydromet".
15/ Documentation
To take in account local conditions and prioreties, in the each institution local
documents "Procedure for the AMP competency assessment" were developed.
There were provided the routine competency assessment and unplaned assessment
procedures.
Roshydromet institutions approved schedules for the routine competency
assessment, all specialists were informed about their schedule in advance (one
month before the assessment).
Unplaned assessment was applied to the following specialists:
- New emploees and the professionals after long work pause (for example,
babycaring);
- Professionals, who was rated as "incompetent" (reassessment).
16/ Assessor`s selection
Roshydromet institutions issued documents, which determined the timing of the
assessment and the assessor`s selection.
Experts were appointed by the head of Roshydromet institutions from the most
experienced professionals with the big operational experience, who have expressed
interest in this project. Preference was given to experts who participated in the
workshops on the competency assessment.All together 291 expert sattended the
assessment.
3
IX международный семинар EUMETCAL. Варшава-2014
17/ Tools/techniques
Expert groups developed and filled the portfolio for each meteorologist with the
education and previuos experience information. Expert groups in each institutions
developed checklists to be filled during the assessment.In some Roshydromet
institutions expert groups developed and approved experiential questions and tests.
Tests most often were used during the distant competency assessment ( in a very
faraway areas). Special attention was paid to the staff skills to make the right
decisions in unusual situation.
18/ Assessment procedure
According to the Roshydromet document "Regulations on the AMP competency
assessment" specialist admitted competent after obtaining at least 80% of the
values of each used tool. If the specialist received the result less then 80%, he (she)
assessed "incompetent."
To the person, who rated "incompetent" the corrective actions took place, such as
additional training or further work under the control of expet. During the period of
corrective actions person temporarily suspended from aviation meteorological
service work until the positive result in the further reassessment.
19/ The results of the competency assessment
After the end of competency assessment, the expert group held the meetings to
approve the results in the form of the Report, which stated:
- Place and date of the assessment;
- The head and members of the expert group;
- List of specialistsis being assessed;
- The results of their assessment.
After summarizing all the institutionals reports, the document, which approved the
results of the AMP competency assessment in the Russian Federation, was issued
by the head of Roshydromet.
The WMO Secretariat was informed about the results of competency assessment in
Russia by 01.12.2013.
20/ The problems identified during the competency assessment
AMP assessment in Russia was done in a very short time In general, the following
challenges during the assessment were faced:
1. Limited funding . Assessment was accompanied by difficulties associated
primarily with limited funding, the lack sufficient financial resources for the
4
IX международный семинар EUMETCAL. Варшава-2014
provision of aeronautical meteorological services cause of the incomplete cost
recovery from aviation users in Russian Federation;
2. Selection and training the experts.There is no training of assessors in
educational institutions; there is no corresponding educational program. It was
recommended to involve specialists in expert groups, who participated in WMO
competency assessment workshops. In practice, only the third part of allassessors
participated in that workshops and have certificates of the WMO Regional
Training Center.
3. Selection of assessment tools.
There were no uniformity in the assessment approaches in the Russian
Federation.
-The main method of competency assessment was a method of direct
observation. The disadvantage of this method was the time-consuming for experts
and high costs for the experts delivery to the venues.
- At the several Roshydromet institutions only the experiential questions and
tests was conducted;
- For the «modeling situation and simulators" method, the participation of
educational institutions is required to provide methodological assistance to the
operational units.
21/ Conclusion
2352 aviation specialists were rated "competent" (962 forecasters-,
1390observers),
• 4 of them performed the procedure of "reassessment", after which these
specialists were rated "competent".
• There is no specialists, who were rated "incompetent.
• 98 aviation meteorologists of Roshydromet institutions were not assessed by the
December 2013 for legitimate reasons, such as maternity and childcare leave,
temporary disability, study and annual leave.
• To the June 2014 extra 45 specialists have been assessed with the results of
"competent".
All specialists in Roshydromet operational units’ demonstrated a high level of
theoretical knowledge and practical skills, responsibility and clear understanding
of the needs and challenges of the competency assessment as part of the Quality
Management System.
5