National Fighter
Procurement Secretariat
Secrétariat national
d’approvisionnement en chasseurs
Industry Engagement Request
Questionnaire 1
Assessment Methodology
Meeting with Companies
May 03, 2013
Purpose
• To provide companies with details on the assessment
methodology that will be used in assessing your responses
to Questionnaire 1
• To provide clarifications and provide answers to questions
• To provide companies with information on next steps
• We are committed to an open, fair and transparent process
2
Industry Engagement Request
Questionnaire 1
• Canadian fighter mission and threat information
across two time horizons (2020-2030 and 2030+).
• Responses to include defined mission configurations:
– Air-to-Air;
– Air-to-Ground; and
– Air-to-Surface.
• Responses to include capability developments
(planned upgrades).
• Responses to include descriptions of capability,
production and supportability.
3
PRINCIPLES
The following Principles will guide all participants involved in the
assessment of the responses:
• Consistency
–
–
–
•
Objective
–
•
Team leads will ensure that the assessment is well documented, including all
qualitative and professional military judgement aspects.
Individual aircraft are being assessed against each Canada First Defense
Strategy (CFDS) mission.
No aircraft will be screened out. All options will remain on the table. Final
product will be a high level risk assessment.
The National Fighter Procurement Secretariat will provide oversight through all
steps of the assessment process
Use of Third Parties
–
Responses will be shared with the Independent Review Panel (i.e. Panel)
members and any significant differences of opinion between assessment teams
will be brought to the attention of the Panel.
4
Overview of Q1 Assessment
•
The “Capabilities” questions of the Industry Engagement Request will be
assessed using a 3-step assessment process:
– Step 1 – Assessment of Measures of Performance (MOP);
– Step 2 – Assessment of Measures of Effectiveness (MOE); and
– Step 3 – Assessment of Aerospace Capabilities weighted against Canada
First Defense Strategy (CFDS) Mission and criticality to result in
“Operational Risk” assessment of each aircraft for each of the missions.
•
The “Production and Supportability” questions will be assessed using a 2step assessment process:
– Step 1 – Assessment of responses by technical Subject Matter Experts
teams on Military Strategic Assessment Factors – Impact Statements;
and
– Step 2 – Assessment of responses by the Royal Canadian Air Force
Senior leadership level from a missions perspective to result in “Strategic
Risk” assessment of each aircraft for each of the missions.
•
Both the “Operational Risk” and “Strategic Risk” will then be consolidated
into an overall risk profile for each aircraft against each mission.
5
Step 1 of Capabilities Assessment
•
•
•
•
Assessments being performed using responses to the 17 areas of
« Capabilities » contained in the Industry Engagement Request.
National Defence teams assessing specific areas, with oversight from the
National Fighter Procurement Secretariat.
Qualitative assessment - independent of missions.
Results in Measures of Performance assessment for each aircraft.
Sensors
Radio Frequency Sensors
Avionics
Electro-optical or Infrared Sensors
Weapons
Pilot Workload
Air-to-Air Weapons
Communications
Air-to-Ground Weapons
Sensor Integration
Air-to-Surface Weapons
Non-Kinetic Weapons (Electronic Attack)
Self
Protection
Data Link
Performance
Radio Frequency Signature
Infrared Signature
Engine/Airframe
Radio Frequency Self Protection
Combat Radius/Endurance
Infrared Self Protection
Countermeasures
6
Evaluation Grid for Step 1
Rating
•
•
Performance rating of individual
systems
10
Rating scale from 0-10 to allow
sufficient fidelity
8
9
7
•
Classified systems information
represented by a number (supported
by classified references)
6
5
4
3
2
Definition
Excellent Without
appreciable deficiencies
Very Good Limited only by
minor deficiencies
Good Limited by moderate
deficiencies
Poor Limited by major
deficiencies
1
Very Poor Significantly
limited by major deficiencies
0
Non-Existant
7
Step 2 of Capabilities Assessment
•
•
•
•
•
•
•
•
•
•
•
Three Royal Canadian Air Force teams (four members each)
performing this assessment, with oversight from the National Fighter
Procurement Secretariat:
Including fighter pilots and at least one Air Force Air Weapon
Controller on each team.
Significant differences of opinion between assessment teams will be
brought to the attention of the Panel.
The following inputs will be used in this assessment:
Scores and input from the results of Step 1;
Companies responses to the Industry Engagement Request,
including the mission configuration systems;
Vignettes detailed in Appendix “A” of the Industry Engagement
Request;
Detailed threat capabilities from Canadian Forces Threat
Assessment;
Detailed fighter task lists; and
Other information available to Government.
Fighter Measures of Effectiveness areas of consideration detailed in
the Industry Engagement Request.
8
Information on Missions
as per the Canada First Defence Strategy (CFDS)
1 – Conduct daily domestic and continental operations, including in the Arctic and
through NORAD (Vignette 1)
2 – Support a major international event in Canada, such as the 2010 Olympics (Vignette
2)
3 – Respond to a major terrorist attack (9/11-type scenario under CFDS Mission 1
through NORAD) (Vignette 5)
4 – Support civilian authorities during a crisis in Canada such as a natural disaster (No
Vignette)
5 – Lead and/or conduct a major international operation for an extended period
[including complex peace enforcement operation, and coalition state-on-state warfighting] (Vignettes 3 & 4)
6 – Deploy forces in response to crises elsewhere in the world for shorter periods
(Vignette 6)
Of the 6 Canada First Defense Strategy missions above, only 5 relate to the 6 vignettes.
Mission 4 (Support civilian authorities during a crisis in Canada such as a natural
disaster) will be analyzed together with Mission 2, understanding the fighter’s
role in this mission is generally minor.
9
Information on Aerospace Capabilities
1.
2.
3.
4.
5.
6.
7.
Defensive Counter Air (DCA)
Offensive Counter Air (OCA)
Strategic Attack
Close Air Support (CAS)
Land Strike
Tactical Support to Maritime Operations (TASMO)
Intelligence, Surveillance, and Reconnaissance (ISR)
10
Information on Fighter Measures of
Effectiveness
1.
2.
3.
4.
5.
6.
Awareness
Survivability
Reach and Persistence
Responsiveness
Lethality
Interoperability
11
Step 2 of Capabilities Assessment Overview
•
Step 2 will first generate “raw” scores for Measures of
Effectiveness (MOEs).
•
Raw results will then be weighted against the applicable
mission to come up with an Operational Level
Aerospace Capability Score for each aircraft against
each mission over the 2 timeframes.
•
Mission weightings are sensitive and not included as part
of this presentation.
12
Evaluation Grid for Step 2 Raw Scores
•
•
•
Effectiveness rating of
contributing system
performances.
Multiple systems will
contribute to each measure
of effectiveness.
Will vary across all missions.
Rating
10
9
8
7
6
•
Rating scale from 1-10 to
allow sufficient fidelity.
•
Qualitative, and thus
requires Subject Matter
Expert input.
•
Will result in “Raw” scores
for Step 2.
5
4
3
2
1
Definition
Excellent No appreciable
limitations in delivering the
desired effect
Very Good Only minor limitations
in delivering the desired effect
Good Moderate limitations in
delivering the desired effect
Poor Major limitations in
delivering the desired effect
Very Poor Severe limitations in
delivering the desired effect
13
Step 2 Example
CFDS Mission
1. Defence of Canada – Baseline and Arctic
Aerospace
Capability
Defensive Counter Air
Threat
2020-2030:
Non-state civilian aircraft (including low/slow)
State long-range aviation
Classified Threat List (NORAD)
Vignette 1
The Canadian Fighter is conducting normal daily and contingency North
American Aerospace Defence (NORAD) missions at normal alert levels, and
is prepared to react to elevated alert levels. Operations are conducted from
main operating bases as well as Forward Operating Locations and
missions may be over land and over water. Operations are being conducted
in a combined and/or joint environment with NORAD forces. The threats for
these missions can be air and maritime-based.
Tasks
2030+:
Non-state civilian aircraft (including low/slow)
State long-range aviation and escorts
Non-state use of stand-off missile
technologies
Classified Threat List (NORAD)
- Rapidly establish and maintain air presence/policing/deterrence for a
tactically significant period of time in an area of operations.
- Detect, validate, characterize, assess, and warn of unwanted and
unauthorized air activity approaching or operating within North American
domestic airspace including potential attacks against North America. (NCI
10-175, 1.2.1)
- Detect, Identify, Determine Intent, Monitor, Intercept, Report, Negate, or
Destroy potential air threats.
14
Step 2 Example – Assessment of Raw Scores
Defensive
Counter Air
(MOEs) using
Vignette 1 in
2020 to 2030
timeframe
Aircraft
X
Aircraft
Y
Tech Area
Contributing System
Aircraft X
Aircraft Y
Sensors
Radio Frequency Sensors
3
8
Electro-optical or Infrared Sensors
7
7
Air-to-Air Weapons
2
8
Non-Kinetic Weapons
0
5
Radio Frequency Self Protection
2
9
Infrared Self Protection
5
7
Countermeasures
5
8
Data Link
5
10
Communications
7
7
Sensor Integration
2
8
Pilot Workload
8
9
Radio Frequency Signature
4
9
Infrared Signature
3
6
Engine/Airframe
9
6
Combat Radius/Endurance
9
7
Weapons
Lethality
2
Air-to-Ground Weapons
9
Air-to-Surface Weapons
Survivability
Responsiveness
4
9
9
Self Protection
6
Avionics
Reach and
Persistence
Interoperability
Awareness
9
5
6
6
8
8
Performance
15
Step 2 Example – Capability Raw Scores
Measures of Effectiveness (Aircraft X)
Mission
Aerospace
Capability
Defensive
Counter Air
Lethality
Survivability
Reach and
Persistence
Responsiveness
Interoperability
Awareness
2
4
9
9
5
6
Offensive
Counter Air
1.
Defence
of Canada
(Baseline
and Arctic)
Strategic Attack
Close Air
Support
Land Strike
5.8
TASMO
ISR
16
Step 2 Example – Capability Raw Scores
Measures of Effectiveness (Aircraft Y)
Mission
Aerospace
Capability
Defensive
Counter Air
Lethality
Survivability
Reach and
Persistence
Responsiveness
Interoperability
Awareness
9
9
6
6
8
8
Offensive
Counter Air
1.
Defence
of Canada
(Baseline
and Arctic)
Strategic Attack
Close Air
Support
7.66
Land Strike
TASMO
ISR
17
Step 2 - Capability (Weighted)
Applying weighting function to each measure of effect allows capability
to be properly quantified within that Mission
Mission
Aerospace
Capability
(Weighted) Measures of Effectiveness (Aircraft X) (2020-2030)
Lethality
Survivability
Reach and
Persistence
Responsiveness
Interoperability
Awareness
Defensive
Counter Air
Offensive
Counter Air
1.
Defence
of Canada
(Baseline
and
Arctic)
Strategic Attack
Close Air
Support
Land Strike
6.7
TASMO
ISR
18
Step 2 - Capability (Weighted)
Applying weighting function to each measure of effect allows capability
to be properly quantified within that Mission
Mission
Aerospace
Capability
(Weighted) Measures of Effectiveness (Aircraft Y) (2020-2030)
Lethality
Survivability
Reach and
Persistence
Responsiveness
Interoperability
Awareness
Defensive
Counter Air
Offensive
Counter Air
1.
Defence
of Canada
(Baseline
and
Arctic)
Strategic Attack
Close Air
Support
Land Strike
7.1
TASMO
ISR
19
Step 3 of Capabilities Assessment
•
This last step of the Capabilities Assessment takes into
account: a) the weighting of each Fighter Aerospace Capability
within a particular mission and b) criticality as follows:
•
•
DCA – XX%, OCA – XX%, TASMO – XX%, ISR – XX%;
Mission Critical (MC), Mission Essential (ME), or Mission Routine (MR);
•
The Operational Level Score obtained at the end of Step 2 is
aggregately weighted and assessed by Royal Canadian Air
Force against the mission criticality matrix.
•
Aerospace capability limitations will carry risk forward
dependant on their mission criticality (translation matrix on
slide 22 expands).
20
Step 3a – Operational Mission Weighting
Operational level scoring (before Mission criticality) 20202030
CFDS Mission 1 (example)
DCA
OCA
Strategic
Attack
CAS
Land Strikes
TASMO
ISR
Aircraft
A
CFDS
2/4
CFDS 3
V1
V2
V5
CFDS 5 CFDS 5
V3
V4
CFDS6
V6
Aircraft A
Aircraft
B
Aircraft
C
Aircraft
D
Aircraft E
Aircraft
Y
CFDS 1
Aircraft B
Aircraft C
Aircraft D
Aircraft E
7.1
6.1
7.3
Aircraft Y
7.0
Operational Risk 2020-2030
Weighting of Fighter Roles / CFDS missions
+ Mission Criticality
OCA
Strat
Atk.
CAS
Ld Strk.
CFDS
2/4
CFDS 3
V1
V2
V5
CFDS 5 CFDS 5
V3
V4
CFDS 6
V6
Aircraft A
CFDS Mission 1 (example)
DCA
CFDS 1
TASMO
ISR
Aircraft B
Aircraft C
Weighting
roles VS
CFDS
X%
X%
X%
Aircraft D
Aircraft E
Mission
criticality
(C/E/R)
C
C
E
Aircraft Y
21
Step 3b - Mission Criticality Translation Matrix
Weighted Capability
Assessment of each
Aerospace
Capability
Criticality of
Aerospace Capability
Maximum
Operational
Assessment
Definitions
+
Mission Critical
=
+
Mission Essential
=
+
Mission Routine
=
Mission Critical: A capability that delivers a direct effect
as part of its primary function. This effect Is evaluated as
critical to mission success and will pose risk to mission
success if it is not employed.
Mission Essential: A capability that is an essential
enabler to Mission Critical Capabilities; the lack of one
or more Mission Essential Capabilities will pose risk to
mission success.
Mission Routine: A capability that is required for the
mission but either as a routine supporting function or a
very low likelihood of employment. Only in cases where
multiple Mission Routine Capabilities constitute a systems
failure will any significant risk be posed to the mission.
Insignificant limitations
Minor limitations
Moderate limitations
Major limitations
Severe limitations
Low operational risk
Medium operational risk
significant operational risk
High operational risk
Very high operational risk
22
Step 3 – “Operational Risk” Assessment
Rating
10
Operational level scoring (before Mission criticality) 2020-2030
CFDS 1
CFDS 2/4
CFDS 3
CFDS 5
9
CFDS 5
CFDS 6
8
7
6
Vignette
1
Vignette 2
Vignette 5
Vignette 3
Vignette 4
Vignette 6
5
4
Aircraft A
3
Aircraft B
2
Aircraft C
1
Definition
Insignificant limitations in
accomplishing the mission.
Minor limitations in
accomplishing the mission
Moderate limitations in
accomplishing the mission.
Major limitations in
accomplishing the mission
Severe limitations in
accomplishing the mission.
Aircraft D
Aircraft E
Definition
Example
Low operational risk to the RCAF ability
to successfully fulfill the fighter contribution
to CFDS mission X.
Step 3 - End State “Operational Risk”
Mission
Criticality
Translation
Matrix
Medium operational risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
Operational Risk 2020-2030
CFDS 1
CFDS 2/4
CFDS 3
CFDS 5
CFDS 5
CFDS 6
Vignette
1
Vignette 2
Vignette
5
Vignette
3
Vignette
4
Vignette
6
Significant operational risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
Aircraft A
+
MC
=
Aiircraft B
High operational risk to the RCAF ability
to successfully fulfill the fighter contribution
to CFDS mission X.
Aircraft C
+
ME
=
Aircraft D
Aircraft E
+
MR
=
Very high operational risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
Example
23
Assessment of the “Production and
Supportability” Questions
• Step 1 – Review of responses by technical Subject Matter
Expert teams – no scoring.
• Step 2 – Assessment of responses by the Royal Canadian
Air Force Senior leadership level from a Canada First
Defense Strategy missions perspective to result in
“Military Strategic Risk” assessment of each aircraft
against the missions.
• Both Steps will be overseen by the National Fighter
Procurement Secretariat.
• Significant differences of opinion between assessment
teams will be brought to the attention of the Panel.
24
Factor Analysis through Deductive Reasoning
Where does the Industry response fit inside the framework?
Factors
Considerations
Deductions
(step 1)
(step 2)
Likelihood
Strategic Assessment
Definition:
There is a possible risk that…
So What?
Possible Impacts
So What?
Strategic Risk
Assessment
Required mitigations
The mission need is…
Source Documentation
The threat is assessed as…
CFDS
(used during step 2 only)
Industry responses to Industry Engagement Request
Open source study
Government-to-Government Information
Other products from the Evaluation of Options
25
•
Acquisition
This Strategic Assessment Factor (SAF) is an assessment of
factors, other than cost, that would affect acquisition of aircraft
types. This includes:
• planned production periods of aircraft type;
• the manufacturer’s ability to continue or re-establish
production periods;
• the ability of the manufacturer to offer a complete “cradle to
grave” program, including; training, in-service / life cycle
support and disposal;
• the manufacturer’s successful completion of any required
developmental work before the aircraft are ready for
acquisition; and
• the ability of aircraft type to be certified for airworthiness
under Canadian Forces regulations.
26
Supportability and Force Management
•
An assessment of the overall / long-term supportability and
aspects related to management of the fighter force. This includes
factors such as:
 the required quantity of aircraft;
 managed readiness postures and sustainment rations;
 the training system and production of appropriately trained pilots and
maintenance / support personnel (including use of simulators);
 training considerations for maintenance / aircrew personnel;
 aircraft mission availability rates and aircrew / technician workload;
 long-term availability of components needed for aircraft maintenance;
 implementation and sustainment of supporting infrastructure;
 the ability to operate from required locations;
 weapons compatibility and support; and
 the suitability of a reprogramming capability including its flexibility and
responsiveness for new threats and/or theatres of operation or new
capabilities.
27
Integration
• This Strategic Assessment Factor assesses broad
interoperability within the Canadian Forces and with allied
forces. This includes interoperability with air-to-air
refuelling services (other than Canadian Forces), common
ground/spares support with allies and their supply lines,
training systems, data sharing, communications, STANAGs,
and the ability to feed data into Canadian Forces and
Government of Canada networks taking into consideration
national security requirements.
28
Growth Potential
• This Strategic Assessment Factor assesses the growth
potential and technological flexibility to respond to
unforeseen future advances in threat capabilities, to
implement required enhancements to fighter technology,
and to evolve as needed to meet the Canadian Forces’
needs. Factors include analysis of the architecture of
aircraft types, power and cooling capability for new
systems.
29
Response to
IER
Other reports resulting
from evaluation of
options
Step 1
Aircraft - SAF Assessment
Strategic Assessment Factors
- Technical SME level working groups
Supportability &
Force Management
Future Security Env’t
- Problem space defined and scoped by
the use of a “possible risk” based
questionnaire
Integration
- SME comment on likelihood; then
Acquisition
Open Source/Other Info
and Gov-to-Gov Info
Growth Potential
-Information to be collated and
considered in Step 2.
Definition
End State - Strategic Risk
Low strategic risk to the RCAF ability
to successfully fulfill the fighter
contribution to CFDS mission X.
Strategic Risk 2020-2030
Medium strategic risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
Aircraft A
Significant strategic risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
High strategic risk to the RCAF ability
to successfully fulfill the fighter
contribution to CFDS mission X.
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Example
Very high strategic risk to the RCAF
ability to successfully fulfill the fighter
contribution to CFDS mission X.
-Comment on possible impacts and
required mitigations to the overall
fighter program
CFDS 1
CFDS
2/4
CFDS 3
CFDS 5
CFDS 5
CDFDS 6
V1
V2
V5
V3
V4
V6
Step 2
SAF - CFDS assessment
- Senior leadership level Working group
(fighter heavy)
- Problem space defined and scoped by
the Task 2 report
- “Considering the likelihood of possible
impacts to the overall Fighter Program,
and based on your Professional Military
Judgment, how would you rate the
strategic risk to the RCAF ability to
successfully fulfill the fighter contribution
to each CFDS mission?”
30
Final Stage – Overall Mission Risk
Assessment
End State: Mission Risk
Strategic Risk
Operational Risk
31
Operational Risk 2020-2030
CFDS 1
CFDS
2/4
V1
V2
CFDS 3 CFDS 5 CFDS 5 CFDS 6
V5
V3
V4
End State
Mission Risk
Definition
Low risk to the RCAF ability to
successfully fulfill the fighter
contribution to CFDS mission X.
Mission Risk 2020-2030
V6
CFDS 1 CFDS 2/4 CFDS 3
Aircraft A
V1
Aircraft B
Aircraft C
Aircraft A
Aircraft D
Aircraft B
Aircraft E
Aircraft C
Aircraft Y
Aircraft D
V2
V5
CFDS 5
CFDS 5
CFDS 6
V3
V4
V6
Significant risk to the RCAF
ability to successfully fulfill the
fighter contribution to CFDS
mission X.
High risk to the RCAF ability to
successfully fulfill the fighter
contribution to CFDS mission X.
Aircraft E
Mission Risk
(fusion of
Operational and
Strategic risk)
Strategic Risk 2020-2030
CFDS 1
CFDS
2/4
V1
V2
Very high risk to the RCAF
ability to successfully fulfill the
fighter contribution to CFDS
mission X.
Aircraft Y
RCAF to apply
professional
military judgement
Medium risk to the RCAF ability
to successfully fulfill the fighter
contribution to CFDS mission X.
C Air Force
endorsement of
Final Report
CFDS 3 CFDS 5 CFDS 5 CFDS 6
V5
V3
V4
V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y
32
Mission Risk
•
Mission Risk will be an outcome of the fusion of Strategic and
Operational risk by the same RCAF senior leaders assessment
teams who took part in step 2 of the Strategic Assessments.
•
Decision support tools, such as but not limited to a mission risk
translation matrix and the DND/CF Integrated Risk Management
Guidelines will be used to determine a rating.
•
Final assessment will be informed by, but not limited to the
decision support tools provided. Assessors will be allowed to
deviate from the mission risk translation matrix used by + or –
one risk level.
•
Rationale behind a final assessment that deviates from the
mission risk translation matrix used will be provided to the
Independent Review Panel for review.
33
Summing Up
• We are committed to an open, fair and transparent process
– IER and methodology posted publicly on Government website
– All questions and answers during IER shared simultaneously with all
companies
– Details of assessment methodology shared with companies in advance of
IER response deadline
– NFPS overseeing entire assessment process
– Significant differences of opinion between assessment teams will be
brought to the attention of the Panel
– The process will result in an assessment of benefits and risks associated
with each aircraft against each mission. Canada will release a summary
report that respects commercial sensitivities and classified information
restrictions.
34