Aerospace days
Airborne Vision systems for situation
awareness
15/10/2014
Emmanuel KLING
Optronics & Defence Division
Sight Programs Department
0/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Sagem A wide range of embedded Optronic Systems
Thermal
Imagers
I.R. Seekers
AASM
Hand-held
Optronics
Dismounted
Soldier
Land
Optronics
UAVs
Aerosurveillance
Naval
Optronics
Airborne
Optronics
1/
CONFIDENTIEL / DATE / DIRECTION
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Proprietary key technologies
Optronic Sensors
•
•
•
•
Cooled and uncooled InfraRed
optroniques
AnyComposants
bandwidth
Lasers
ITAR Free
Critical HW & SW
•
•
•
•Stabilisation
Stability
&
• Localisation
Geo location
Ligne de Visée
Airborne application standards
Robust / simple / reliable
Low weight
Real Time Video Processing
•
•
•
•
2/
Fog / low visibility
Line of sight management
DAL level D to B
DO 178 Intégration
B or C
DO254
Optomécanique
Proved design
•
•
•
Robust to bad weather conditions
3D-Obstacle detection
Image Fusion
Enhanced reality
Tracking
CONFIDENTIEL / DATE / DIRECTION
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Sagem Optronics in airborne applications

Observation
EUROFLIRTM : Gyro-stabilized Electro-Optical Airborne gimbals for
-

SPERWER Tactical UAS
PATROLLER UAS
NH90, PANTHER, COUGAR helicopters
Fixed-wings aircrafts
Targeting
 Helicopter sights



STRIX familiy (for TIGER HAP, HAD, ARH)
Osiris (for TIGER UHT)
Situation Awareness
 To reduce the risk of accident during all phases of the flight





Mid Air Collisions between aircrafts
CFIT (Controlled Flight into Terrain)
Obstacle collisions on ground
Additional Operational credit during landing phase
All aircrafts : airliners, business jet, small aircraft, helicopters, UAs
3/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Some “Situation Awareness” devices in Avionics
LOAS
SWORD
HELLAS
LOAM
…
Obstacle Warning
Systems
Collision Avoidance
Systems
Terrain Awareness
& Warning Systems
Synthetic Vision
Systems
Enhanced Vision
Systems
4/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Traffic Separation Rules
Helicopter Flight
« In flight »
Approach
5/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Collision Avoidance products

TCAS – Transponders – PCAS – ADS-B
 TCAS (Traffic collision Avoidance Systems)




involves communication between all aircrafts equipped with an
appropriate transponder
builds a three dimensional map of aircraft in the airspace
Is only able to interact with aircraft that have a correctly operating mode
C or mode S transponder
Elaborates avoidance tactics
 PCAS (Portable Collision Avoidance System)

Passive devices “hearing” transponder communications
 ADS-B (Automatic dependent surveillance-broadcast)



Active systems broadcasting aircraft current track, position and altitude
given by GPS to ATC and to other equipped ADS-B aircraft
Elaborates avoidance tactics
Aircraft sensing by cooperative means




Effective but requires that all aircraft in flight should be equipped with
TCAS / ADS-B
High Certification cost (TCAS & transponders)
PCAS & ADS-B not yet certified worldwide
Fixed and limited channel data bandwidth (ADS-B)
6/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
The need of a non cooperative and independent traffic
surveillance system

Mid Air UAV and helicopters
 Operates mainly in uncontrolled airspaces
 TCAS and transponders are not well designed for helicopters since they are not installed on
every following aircrafts






helicopters
gliders
ULM
UAV
Balloons
Paragliders

We propose to increase the capability in traffic separation with a Traffic Sensing
product using non-cooperative and independent means

Independant
 The aircraft is not dependent on any external element (eg GPS) to make traffic surveillance
in its environment.

Non-cooperative
 Traffic monitoring is done without dedicated radio emission provided by other aircrafts.
Therefore, it applies to all types of aircraft.
7/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Controlled Flight Into Terrain - Existing equipments

TAWS (Terrain Awareness and Warning System)

Electronics coupled to the navigation system and to a Digital cartographic and
terrain database




Developed since the 2000s, initially for military aviation
Also for airliners and business aircraft, and now for helicopters
Returns alerts about CFIT if the extrapolated trajectory of the plane "cuts" the ground
SVS - Synthetic Vision System




synthetic Image generation on the basis of terrain and cartographic database, and
according to the trajectory of the aircraft
Eventually merge with the images provided by an EVS, Becomes a CVS
(Combined Vision System)
CVS and SVS are subject to regulations set by D0315 (aircraft only).
There is no equivalent regulation for such equipment on board helicopters.
On airplanes, SVS shall display obstacles which are higher than 200 feet …

(DO309)Helicopters operate in widely diverse environment, including
under visual and instrument flight rules (VFR and IFR) and in both onand off-airport operations, often with significant low altitude exposure.

These systems seem not well-suited to helicopter operations since the
used database is DTED1 or lower (100 m cell size) and not robust to
ground asperities
8/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
HTAWS Regulation (RTCA DO 309)

Position Source



Terrain and Obstacle database




Some recommendations (no requirements) regarding the type of GPS or WAAS used.
If the position source is corrupted, the HTAWS must be switch off
No requirement on the quality of the database or its accuracy, nor on the recurrence of its update
“The highest resolution database should be used where necessary and available.”(DO309)
“The HTAWS does NOT guarantee successful recovery from a conflict due to factors such as pilot response,
aircraft performance and database limitation” (DO309)
Typical Terrain Database accuracies

For a DTED level 1 :




It appears that:




Individual Cell size : 100m
Horizontal Accuracy Dr = (x2+y2)1/2 < 50 m @90%
Altitude Accuracy D z < 30 m @90%
Only a database with metric accuracy allows to show the ground surface and generate alerts on small
obstacles (buildings, rocks, pylons…) in low or very low altitude
High resolution Databases (metric accuracy) are often classified and localized mainly around strategic sites
(airports, military bases)
The accuracy of the navigation data and spatial accuracy of the database must be compatible and
constistent.
Therefore, the operational interest of HTAWS is still relatively low compared to the
challenge of helicopter flight => the need of a system for detecting obstacles on
real measurements: these systems are the OWS (Obstacle Warning Systems)
9/
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Enhanced Flight Vision Systems

Situational awareness

Approach stability in low visibility conditions
10 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
DO315 - Glossary

RVR :
 Runway Visual Reference ,in aviation meteorology, is the distance over which a pilot of an
aircraft on the centreline of the runway can see the runway surface markings delineating the
runway or identifying its centre line. RVR is normally expressed in feet or meters.

« Enhanced Flight Visibility » :
 “average forward horizontal distance, from the cockpit of an aircraft in flight, at which prominent
topographical objects may be clearly distinguished and identified by day or night by a pilot using
an enhanced flight vision system
 -> An EFVS, then, is the means by which the pilot meets the enhanced flight visibility
requirement

THRE : THReshold Elevation

TDZ : TouchDown Zone
11 /
CONFIDENTIEL / DATE / DIRECTION
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Visual References

Approach light system, if installed;

or

Visual references: Runway threshold,
identified by at least one of the
following:
 Beginning of the runway landing surface,
 Threshold lights, or Runway end identifier
lights

AND Touchdown zone, identified by at
least one of the following:
 Runway touchdown zone landing surface,
 Touchdown zone lights,
 Touchdown zone markings, or Runway
lights.
12 /
CONFIDENTIEL / DATE / DIRECTION
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Active Obstacle Warning Systems for low altitude flights

AOWS – Active Obstacle Warning System
 Principle :


Scanning the space by a RADAR or LASER beam
Range measurement allows to detect possible
obstacles.
13 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Active OWS Example (ASC Inc.)
3D Flash LADAR Helicopter Landing
Sensor for Brownout and Reduced
Visual Cue
14 /
CONFIDENTIEL / DATE / DIRECTION
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Active Obstacle Warning Systems for low altitude flights

Some products







Lord / Sword (ELOP) : FO LASER, 24 kg, 600w
Eagle Owl (BAe Systems) : FO LASER, 20 kg
LOAM (Finmeccanica) : FO LASER, 25 kg
LOAS (Goodrich) : FO LASER, 15 kg
OAsys (Amphitec) : RADAR FMCW (Ka band)
HELLAS (Airbus Defence) : FO LASER, 25 kg
Typical sell prices :
 150 k$ - 300 k$

The OWS provide a real answer to the problem of obstacles in the flight at very low
altitude, especially on cable detection.

Like all active systems, this capability shall not be at the expense of human safety
-> need to switch off active systems during final approach to preserve personal
safety (LASER) or electronic devices (RADAR).

Cost, size, consumption and the lack of any certification rules associated with
these products are some reasons why this market is reserved to military markets
15 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
POSAS Concept : A passive optronic situation awareness system
dedicated to UAS and helicopters
 to
offer standard EFVS operational credit in approach phase (with Runway
Visual References Recognition)
 to
strengthen the traffic separation capability using non-cooperative and
independent means in addition to cooperative means
 to
offer a credible, passive, compact and low cost alternative compared to
current active obstacle warning systems based on LASER or RADAR
 to
provide solutions to extend TAWS and SVS operational credit at low
altitude.
POSAS
Wide FOV EVS
Traffic Surveillance
Passive OWS
3D Mapping
16 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
POSAS Concept : How should it work ?

Integration of distributed low cost optronic sensors
 Uncooled infrared sensors operating in the LWIR spectral band
 Day and night “hot spot” detection
 Low light level sensors operating in the visible and near infrared spectral bands
 Aircraft strobe ligths, helipad lights, buildings warning lights…

Integration of advanced image processing technologies
 Stereovision & Simultaneous Localisation and Mapping
 3D Terrain mapping
 Navigation data accuracies improvement
 Hot spot detection Algorithms
 Reminder : tracks with a constant bearing are on a collision trajectory
17 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Mid Air collision avoidance system

MIDCAS Programme (AED contract, 2008) :

UAV insertion in the traffic

Wide selection of sensors / devices to be tested :





Distributed Uncooled Infrared Sensors (Sagem)
Near Infrared Sensors (Diehl BGT Defence)
RADAR (THALES)
Cooperative devices : TCAS, ADS-b
Sensors installed on a Sky-Y UAV (Alenia Aeronautica) and DGA EV testbed (CASA
Aircraft)
Sensor Head
Uncooled LWIR micro camera
18 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
DGA EV Testbed
POWS Demonstrator

Application : Flight Safety for Rotary Wings in degraded visual environment

Flight cleared demonstrator currently under test


6 uncooled Infrared Cameras
1 visible HD camera

Wide area surveillance : 150°x 80°

3D StereoVision

Automatic obstacle detection

Fly test on a DGA EV helicopter Test Bed (PUMA)
19 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
POWS Demonstrator
20 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
STEREOVISION
Left image



Goal: compute a dense distance map (one distance
for each pixel) from a pair of images
Prerequisite: need to know precisely the geometry
of the stereo rig (focal, pitch, baseline, etc.)
Steps:
 Rectification: adjust the images as if they were coplanar (linear operation)
 Block matching: for each pixel of the left image, find its
correspondent in the right image
 Triangulation: compute the distance map (meters) from
the disparity map (pixel)
21 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Rectification
Right image
Rectification
Rectified left
image
Rectified right
image
Block matching
Disparity map
(pixel)
Triangulation
Distance map
(m)
Simultaneous Localisation and Mapping
Low level extraction
Temporal association
High level
Interest point detection
(Harris, SIFT, Kanade…)
3D non-linear filtering
Tracks
(3D over time)
Images
INS/Vision hybridation
Associated data
22 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
POSAS : Main Operational Capabilities

Enhanced Vision (Wide FOV)

Obstacles detection and ground mapping (3D)
Detection & ranging
Terrain (ground)
Hills
Structures (buildings)
Pylons
Câbles
Oil rigs
Ships
Street lights
Trees
Helipad
TAWS
AOWS
POWS
?
 Obstacle Detection performances

Between 0,6 NM and 10 NM (according to object type and meteorological conditions)
 Localisation and ranging performances


Angular accuracy (azimut, elevation) : < 1mrad @ 3s
Range Accuracy : < 1% @ 3s up to 1Mile
23 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
POSAS : Additional Operational Capabilities
 Providing of a “USER” obstacle and terrain database for HTAWS and SVS
 Metric-class accuracy (obstacle location)
 3D terrain mapping on a corridor along the path of the helicopter with a width of about 1 km
 Terrain Aided Navigation : the software compares the measured 3D mapping of the terrain with
its internal database. The correlation peak gives the position of the aircraft in geographic
coordinates.
 Improvement
of navigation Data accuracies by INS / Vision hybridization
 Compared to the INS alone, the gains expected by hybridization with vision are:




Attitude: error reduction of 35%
Position: error reduction of 90%
Speed: error reduction of 85%
Strong gains are expected in z (better than GPS)
Average standard error in z: 30 cm! (between 5cm and 2m on a typical scenario)
24 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
DVE SYSTEM ARCHITECTURE (NIAG SG 167)
OBSTACLE WARNING SYSTEM
DTED
2+
ARCHITECTURE À
L’ÉTUDE
MSD
ACFT Geo Position
MAGRS/EGI (GPS)
Current Equipment
Augmenting Equipment
Optional Equipment
1553/Ethernet
Display System (CMS)
ACT I VE
C H
S
E
L
CL R
C H
STB Y
BR
T
SEC
ARM
ACT
O FF
NI G HT
DAY
SYM
PO W ER
O FF
NAV
FLI R
STAT
AVI O NI CS PO WER S ELECT
WYP T
HDG
BRT
HO L D
SPEE D
VS L M T
ALT
R
AUTO
NAC
ARM
I NC
ON
ON
ACT
ON
HO L D
ON
HDD
G RO WTH
ACT
HVR CPLD
REM
CPLD
ARM
RF I
CHAN
PUSH
X10
HO L D
ARM
ACT
HO VER
S
E
L
ACT I VE
CH
CL R
C H
STB Y
SEC
BR
T
ON
ACT
ON
DECR
O FF
NI G HT
DAY
O FF
NI G HT
DAY
SYM
SYM
SYST *
9
AVI O NI CS EQ UI PM E NT
ADF
ARC 210 #2
DEU - CO PI L O T
PO W ER
O FF
ON
ON
DM S
FM HO M I NG
I FF
I NS 1
I NS 2
I NS 3
RAD ALT
ON
ON
ON
ON
ON
ON
ON
SATC O M
TACA N
ON
ON
8
EST
XXX
FEED
90
HYD 1
FAI L
HYD 2
FAI L
HYD 3
FAI L
XX XX
EST
XXXX X
USAB LE FUEL
EST
SEL
T 1 2
XXX
NP
XXX
1
5 4
FAI L
NR
NP
N
P
350 60 80
100
KNO TS120
140
I AS 160
180
250 200
ON
O FF
XXX
300
ENG
TRQ
FLAP S
0
9 ALT
9 60 0
9
3
6
FEED
XXX
XXX
7
BRT
SEQ
CO N
BRT
CO N
BRT
O FF
CO N
BRT
C/ A
ACK
C/ A
NI G HT
DAY
ACK
XXX
CO M M 1
01N/ 251. 000
OIL
PSI
OC
XXX TAG B
XXXX
HYD1
XXX
XXXX
HYD2
XXXX
HYD3
RAM P O PEN
APU O N
EM C O N
BARO PRES
29. 9 2
RAD ALT
CO M M 1
13 N / 123. 50
XXX
110. 30
XXX
MW G B
XXX
TCN CHAN
123X
BARO
PSI
OC XXX
PRES
29. 9 2
RAD ALT
LO W SET
200
M O DE 3 CO DE
7600 C
CAC
2
3
4
5
6
7
8
9
O
ACK
EM C O NI NAV
ENAV
WYP T
I FF
EM E R
DEL
H
B
C
I
J
P
V
Q
W
SYS
STAT
CO M M FPLN
M SN
TI M ER PG
0
A
CO N
ACK
CO M M 1
01N/ 251. 000
XXX
HY D SY S 1 PRES S HI
HY D SY S 1 PRES S HI
HY D SY S 1 PRES S HI
L FE ED TANK LO
AFT CRO WN O PEN
RADAR B EACO N FAI L
DC B US 1 FAI L
SET
M O DE 3 CO DE
7600 C
/
PG
EI CA S
C/ A
1
LTS
2
5
6
7
8
9
ENTE R SEQ
/
F
M
N
H
T
U
O
Y
I FF
DEL
E
L
S
CO M M FPLN
0
I DEN T
D
K
X
3
4
SEQ
RTN
CLR
R
CDU
EM C O N I NAV
ECS
W/ W
G
A
Z
B
P
V
N
PO L
Proc
PSI
O
C
XXX
CAC
CDU
I2HMD
Proc
I2HMD
XXX
PSI
XXX
VO R / I LS FRE Q
XXX
TCN CHAN
123X
LO W
200
XXX
XXXX
MG T
VO R / I LS FRE Q
110. 30
1553/Ethernet
BRT
C/ A
PRG B
XXX
NG
XXXX
CO M M 1
13 N / 123. 50
1
DEUs : Display Electronic Units
DTED : Digital Terrain Elevation Data
EV : Enhanced Vision
SV : Synthetic Vision
HMD : Helmet Mounted Display
CMS : Cockpit Management System
DAS : Distributed Aperture System
HRTI : High Resolution Terrain Information
POWS : Passive Obstacle warning System
AOWS : Active Obstacle Warning System
HDD : Head Down Display
CRS
W
RO
G
ARM
ACT
APPR
CO N TRO L PA NEL
AREA PER
PHO TO ETCH
DRAW I NG
AOWS
EV_SV Fusion
Symb Processor
DEUs
DEUs
ARM
ACT
ARM
SYM
PFD
PWR
ALL
ON
POWS
TACA N
G
SRCH PAT
I
N
A
V
O FF
NI G HT
DAY
VO R / I LS
TH
G RO WTH
E
N
A
V
CPLD
RF I
CHAN
TH
HDD
W
Dig
Map Dig
Map
RO
DAS Uncooled-IR /
LLLTV
Synthetic Vision
DTED/HRTI
Data
Update
1553/Ethernet
G
Flight Control
Computers
Terrain
Database Generator
and SV Processor
C
J
M SN
ENAV
WYP T ECS
W/ W
TI M ER PG
RTN
EM E R PG
ENTE R
CLR
LTS
SEQ
SEQ
I DEN T
D
E
F
G
K
L
M
N
Q
R
S
T
U
W
X
Y
Z
HMD Head Tracker
FO FA
V ST W
SLO W
SLEW
PRES S DESI G NATE
DIRECT
HMD Tracked FOR - Display
I2HMD
(COTS/GDE)
DIRECT
HMD Tracked FOR - Display
25 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
HMD Head Tracker
BRT
C/ A
ACK
Conclusion
A Passive Optronic System to increase Situation Awareness
 to
strengthen the traffic separation capability using non-cooperative and
independent means
 OK, by using distributed staring sensors and air surveillance algorithms
 to
offer a credible, compact and low cost alternative compared to current
obstacle warning systems based on LASER or RADAR
 OK, by using distributed staring sensors and SLAM/Stereovision algorithms, the
system can provide alerts when detecting obstacles on ground
 to
provide solutions to extend HTAWS and SVS operational credit at low
altitude.
 OK, by using SLAM / stereovision Algorithms : the system may elaborate a “user”
3D mapping of the neighborhood and may improve the accuracies of the navigation
data (INS/Vision hybridization, Terrain Aided Navigation)
 to
provide day / night enhanced vision for HDD, HuD, Helmet Mounted
Display…
26 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.
Questions ?
27 /
Ce document et les informations qu’il contient sont la propriété de Sagem. Ils ne doivent pas être copiés ni communiqués à un tiers sans l’autorisation préalable et écrite de Sagem.