AP3379
Leaflet 1220
LEAFLET 1220
RAF TRAINING EVALUATION POLICY
Purpose:
To ensure that all personnel are aware of the
RAF policy regarding the Evaluation of Training
Scope:
All RAF personnel
References:
JSP 822 Part 2
DSAT QS
DTSM 4
Leaflet Sponsor:
SO1 Training Policy & Assurance
Point of Contact:
SO2/SO3 Evaluation - COS Trg Pol
Additional Reading:
Leaflet 1225 - Training Improvement Form
Leaflet 1226 - Post-Deployment Questionnaire
RAF Evaluation Website
INTRODUCTION
1.
The Defence Systems Approach to Training Quality Standard (DSAT QS)1 requires all
individual training to be subject to the process of evaluation in order to monitor the impact of
training, to assess what has been achieved, and to assess whether the training was effective. The
information gathered during the evaluation process provides the evidence on which training
management, at all levels, base their decisions. This leaflet explains the rationale behind the
RAF’s system of training evaluation and details the responsibility and processes for Training
Delivery Authority (TDA) to conduct Internal Validation (InVal) and the HQ No 22 (Trg) Gp
Evaluation Team to conduct External Validation (ExVal).
DEFINITIONS
2.
Evaluation is the process of measuring the total worth of training to an organisation and is
defined in JSP 822 Part 2 as:
Evaluation is the process of making a judgement as to the worth or value of training and
education. It examines the impact of training and education policies, assessing what has
been achieved, the effectiveness of the policies and whether this has contributed to the
achievement of goals and targets. A specific subset of evaluation is the process of
validation, which examines whether or not the processes and products of training meet
Defence/single Service requirements. Validation is divided into Internal Validation and
External Validation
The Defence Training Support Manual 4 (DTSM 4) details DSAT QS evaluation methodology and,
specifically, the processes and tools for conducting InVal and ExVal.
EVALUATION – MEASUREMENT
3.
It is mandated that the evaluation of training requires an evaluation strategy2, based on the 4
stages of evaluation3. The stages are comparable with Kirkpatrick’s goal based model for
evaluation, which divides evaluation into 4 levels of measurement: Reaction, Learning, Behaviour
and Results. The RAF approach to evaluation uses four levels mapped to the Kirkpatrick model.
The RAF responsibilities and processes related to each level are described below, and displayed
in table format at Annex A.
1
JSP 822 Part 4, DSAT QS 001:2007.
JSP 822 Part 4, DSAT QS 001:2008 11.3.1
3 JSP 822 Part 4, DSAT QS 001:2007 (Table 11.1).
2
1
AP3379 - Revised Sep 14
4.
Level 1 - Feedback. The measurement of students’ reaction to training is conducted during
the InVal process and is the responsibility of the TDA. Reaction can be measured in several ways
including: reaction sheets, end of phase/end of course debriefs and the feedback received by
instructors during lessons. Overall, reaction to training is best measured during and immediately
after training delivery.
5.
Level 2 - Assessment. The measurement of learning (the change in students’ knowledge,
skills and attitudes) is conducted during InVal, relies heavily on the assessment process, and is the
responsibility of the TDA. Learning transfer can be measured using formative4 and summative5
assessment - DTSM 4 provides guidance on writing and implementing an Assessment Strategy;
questionnaires and instructor observations are typical sources of learning data. In addition, first
time and overall pass rates can provide an insight into the effectiveness of learning transfer, where
effective, proven and reliable assessment systems are in place. Learning is most effectively
measured during and immediately after the training delivery. However, it can be very difficult to
measure a change of attitude; genuine attitude change can often only be measured over a long
period of observation of change of behaviour. Therefore, a combination of Level 2 and 3
evaluations may be required to effectively measure changes in attitude.
6.
Level 3 - Relevance. Training events are only effective if they result in appropriate
behaviours in the workplace. The measurement of behaviours of ex-students is conducted during
ExVal, and is the responsibility of the HQ No 22 (Trg) Gp Evaluation Team. Behavioural change
data will typically be gathered using questionnaires (to ex-students and their supervisors) and by
direct observation. Behaviours are, in the main, best measured 6 months or more after the training
delivery has been completed but this time span can vary according to both the training given and
the workplace behaviours being sought. DTSM 4 details the processes and tools for conducting
ExVal.
7.
Level 4 - Business and Operational Goals. Evaluation of Business & Operational Goals
(Level 4) can only be achieved through drawing upon a range of different sources. Some of the
specific mechanisms that 22 (Trg) Gp are engaged with include:
a.
Training Improvement Feedback (TIF) Form. All personnel involved in the delivery
of individual training to the RAF are able to complete a TIF at any stage of training or career.
The TIF can be used to raise both tactical and strategic issues and the Evaluation Team will
engage with the appropriate stakeholders and staff the process until conclusion.
b.
Post-Deployment Questionnaire (PDQ). All personnel returning from operations are
required to complete a PDQ upon their return to the UK. The Evaluation Team produces a
monthly report to the key stakeholders across all appropriate Defence Lines of Development
(DLOD).
c.
AIR Lessons Identified Forum. This is an AIR wide initiative that the Evaluation
Team are engaged with. SO3 Eval conducts the Senior Responsible Officer role as the
immediate poc within 22 (Trg) Gp for those issues associated with training; liaison then takes
place with the appropriate TRA to identify suitable courses of action to address issues. The
involvement of the Evaluation Team in this high level of activity helps contribute towards the
identification of major training themes.
INTERNAL VALIDATION
8.
InVal is the responsibility of the TDA and TE and is used to determine the efficiency and
effectiveness of training and whether it has enabled a student to achieve the course training
4
Assessment used, during a training programme, to identify any weakness in learning or instruction and to
aid the retention of successful learning.
5 Assessment used to measure the knowledge, skills and attitudes acquired by trainees on completion of a
training event.
2
AP3379 - Revised Sep 14
objectives. To achieve this, InVal is conducted to measure the immediate reaction of a student to
a training event, through an after action review, and the learning transfer achieved by the training
activity; all RAF individual training (Phase 1, 2 and 3) is to be subject to InVal. The process acts as
part of the overall quality assurance system and will provide data on whether the processes and
products of training are delivering an output that meets the Training Performance Statement (TPS).
9.
Conducting InVal. In conducting InVal, the TDA will involve personnel from a range of
backgrounds, including students and instructors. InVal data can be gathered from several sources,
including: course documentation, assessment (formative and summative), instructor performance
monitoring, observations, course feedback questionnaires, post-course discussions, and
interviews. In addition, management and assurance activities (eg training audits, inspections and
management reports) can be used to inform InVal. InVal data is required to be collected prior,
during and at the end of the training process, in order to measure learning transfer and capture
trainee reaction.
10. Analysis of InVal Data. InVal can generate a considerable quantity of both qualitative and
quantitative data, some of which, at times, may be contradictory. Therefore, to ensure changes
made to training are positive, data analysis must be effective; it is particularly important that data
analysts are familiar with the concepts of validity, reliability and triangulation, as defined in DTSM
4. In addition, it is important to realise that there are a range of external factors that may influence
the students’ reactions to training and influence the content of InVal feedback (e.g. relationships
with instructors, peer influence, attitude to training, quality of training delivery, etc). These factors,
together with the weight of supporting evidence, should be carefully considered before any
conclusions are reached with respect to the efficiency and effectiveness of training.
11. InVal Reporting. An InVal report should be completed, iaw the trg evaluation strategy,
indicating trends and common factors across individual course types, groups of courses or
throughout the training unit. For a course to be deemed internally valid, it must be proven, by
triangulation of data, that all training and testing meets the requirements of the Training
Objectives (TOs) as contained in the TPS. The InVal report should be staffed and actioned
within the TDA. If requested, a copy must be made available to the HQ No 22 (Trg) Gp
Evaluation Team. To be able to identify a course as being internally valid, the report must be
able to positively state that:
a.
All standards have been transferred from the Training Objectives in the Training
Performance Statement to the Instructor Specifications.
b.
The instructors delivered training from the Instructor Specifications.
c.
Instruction was delivered to the correct standards and conditions and that appropriate
instructional monitoring occurred.
d.
The Assessment Specification was implemented and the Training Objectives were
tested in the specified conditions to the prescribed standards.
EXTERNAL VALIDATION
12. ExVal is the responsibility of the HQ No 22 (Trg) Gp Evaluation Team and is used to
determine the validity of training in preparing individuals for their role. To achieve this, ExVal
primarily determines the impact on individual performance, including the effectiveness of learning
transfer to the workplace, after a training event. To specifically determine when and how each
ExVal should be conducted, the Evaluation Team ExVal assessment tool6 is completed in
conjunction with the relevant TRA. When conducting ExVal, both qualitative and quantitative
data is used to measure changes in behaviour, and the effectiveness of the enhancement of
knowledge, skills and attitudes; ExVal seeks to determine the impact of training on individual
6
Evaluation Team ExVal Priority Matrix can be viewed via the Evaluation team site.
3
AP3379 - Revised Sep 14
performance, with respect to the Operational Performance Statement (OPS). Essentially, ExVal is
used to determine the continuing validity of training and that the OPS and underpinning Training
Objectives meet the requirements of the job.
13.
Triggers for ExVal. The following 3 triggers initiate ExVal action by the Evaluation Team.
a.
Routine. Phase 1 and Phase 2 courses are subject to routine biennial ExVal iaw the
ExVal Schedule, agreed with the course Sponsor/Training Requirements Authority (TRA).
b.
Sponsor/TRA Request. The majority of the Evaluation Teams output should be
generated by taskings from the Training Sponsor/TRA. It is essential that it is AIR Cmd
SMEs that direct the tasking of the Evaluation Team to ensure that the greatest value can be
achieved from the limited resource. However, TRAs should note that it is the process of
considering whether an ExVal is required that is necessary for DSAT QS compliance. TRAs
may decide that owing to a stable training requirement, they are content for an ExVal not to
take place. However, if this is the case, records should be kept such that the decision
process can be demonstrated during any subsequent 2nd Party Audit. TRAs should ensure
that a course is considered for ExVal at least once every 5 years; utilising the TRA tasking
tool to support decision making The Evaluation Team are able to provide direct advice and
guidance to TRAs on criteria to consider when determining the requirement for ExVal.
c.
Other. If concern over the validity of training is highlighted to the Evaluation Team (eg
via the Post-Deployment Questionnaire or Training Improvement Form) then, in agreement
with the course Sponsor/TRA, appropriate ExVal will be conducted.
14. HQ No 22 (Trg) Gp ExVal Process. The HQ No 22 (Trg) Gp ExVal process is at Annex B.
The default process will comprise questionnaire development and distribution, results analysis and
report generation. In addition a Sponsor/TRA feedback questionnaire is also compiled, used to
assess the validity of recommendations made within the ExVal report, this is then followed up with
a further feedback plus questionnaire to determine whether the changes to a course have been
actioned. Should other methods be required, these would be derived in consultation with the
Training Sponsor/TRA. The Evaluation Team utilise an in-house capability to design bespoke
online questionnaires (distributed via DII) to support each individual ExVal project.
CONTRACTED OUT TRAINING
15. Contracted out training should still be subjected to robust methods of evaluation. The
contract should stipulate that those delivering the training should provide mechanisms to generate
InVal outputs. The RAF should retain responsibility for conducting ExVal to determine whether the
training is continuing to meet the requirements of the Front Line Commands. There maybe
occasions where contractors wish to extend the level of evaluation that they conduct. Where this is
the case discussion should occur with the Evaluation Team to ensure that there is no duplication of
effort. The Evaluation Team should ensure that the proposed approach also meets the needs of
the RAF. Any extension of contractor responsibilities with respect to evaluation should not result in
any additional cost to the service.
POINTS OF CONTACT
16.
The sponsors of this leaflet can be contacted as follows:
SO1 Training Policy & Assurance
HQ No 22 (Trg) Group
Air Command
RAF High Wycombe
Bucks
HP14 4UE
4
AP3379 - Revised Sep 14
GPTN: 95221 Ext 6161
22TrgGp-Pol SO1
17. Questions regarding the implementation of RAF Training Evaluation policy should be
directed to:
SO2 Evaluation
HQ No 22 (Trg) Group
Air Command
RAF High Wycombe
Bucks
HP14 4UE
or
SO3 Evaluation
HQ No 22 (Trg) Group
Air Command
RAF High Wycombe
Bucks
HP14 4UE
GPTN: 95221 Ext 7092
22TrgGp-Eval SO2
GPTN: 95221 Ext 7690
22TrgGp-Eval SO3
Annexes:
A.
B.
Summary of the RAF 4 Levels of Evaluation Measurement.
HQ No 22 (Trg) Gp External Validation Centre - External Validation Process.
5
AP3379 - Revised Sep 14
AP 3379
Leaflet 1220
Annex A
SUMMARY OF RAF 4 LEVELS OF EVALUATION MEASUREMENT
Level
Level 1 – Feedback
Level 2 – Assessment
Level 3 – Relevance
Level 4 – Business and
Operational Goals
KirkPatrick Levels
Reaction
Learning
Behaviour
Results
Process
InVal
Assessment results/InVal
ExVal
OpVal
Responsibility
TE
TE
TRA Desk level POC (AP3379
leaflet 1020 Annex F refers)
HQ 22(Trg) Gp Evaluation
Team on behalf of COS Trg
Stakeholders
TE:
 Training Development
 Instructors
 Students
TE:
 Instructors/Invigilators
 Training Managers
 Students
 22(Trg) Gp Evaluation Team
 TRA Desk Level POC
 COS Trg
 22(Trg) Gp Evaluation Team
 TRA Desk Level POC
Triggers
Each Course Iteration
(plus periodically throughout
the course if required).
Each Course Iteration
(plus periodically throughout
the course if required).
 Routine
 Course Sponsor/TRA
 EVC Recommendation
Main Methods
 Reaction Sheets
 End of Phase Debriefs
 End of Course Debriefs
 Instructor Feedback




 Questionnaires
 InVal Reports
 Questionnaires
 Interviews
Reporting
OC Training Establishment to
TRA via CEB
OC Training Establishment to
TRA via CEB
Direct to Desk Level TRA
Direct to relevant AIR Cmd
Business Area
Formative Assessments
Summative Assessments
Instructor Observations
Pass Rates
 Training Improvement Form
 Post-Deployment
Questionnaire
 Level 3 ExVals
 Course Sponsor/TRA
 DLIMS
AP3379 - Revised Sep 14
AP 3379
Leaflet 1220
Annex B
HQ NO 22 (TRG) GROUP EXTERNAL VALIDATION TEAM - EXTERNAL
VALIDATION PROCESS
Confirm ExVal Requirement
Identify Training
Course(s) & Relevant
Training Sponsor
Joint / Tri Service Course
Confirm Lead
Service/TLB
Not RAF Lead
Lead Service/TLB
ExVal Process
Applied
RAF Course
ExVal Tasked by
Training Sponsor
RAF Lead
In consultation with Training
Sponsor:
- Obtain OPS of relevant
course(s)
- Complete ExVal priority matrix
Determine ExVal
TOR and Identify
Training
Establishment poc
Inform Training
Establishment poc
of ExVal tasking
and process
In consultation
with the Training
Sponsor, develop
the data capture
methodology
Obtain course nominal
roll(s) and InVal reports
from Training
Establishment
Data Capture
Methodology
Endorsed by
Ed Psych
Implement Data
Capture
Hasten Data
Capture
Appropriate
Confidence Level
Reached?
No
Yes
Conduct Data
Analysis
Produce &
Distribute Report
AP3379 - Revised Sep 14