Ref. Ares(2017)529646 - 31/01/2017 Technical note D4.3 Preliminary description of visionbased navigation and guidance approaches 31st December, 2016 (M10) Author: Yoko Watanabe, Balint Vanek, Akos Zarandy, Antal Hiba, Shinji Suzuki, Ryota Mori Publish date: 31/01/2017 1 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Document Information Sheet General information Deliverable no. 4.3 Document title Preliminary description of vision-based navigation and guidance approaches PU (Public) Dissemination level Project name Validation of Integrated Safety-enhanced Intelligent flight cONtrol EU EU-H2020 GA-690811 Grant no. Japan NEDO GA- 062800 EU Hugues Felix (EC) Project officers Japan Hiroyuki Hirabayashi (NEDO) EU Yoko Watanabe (ONERA) Coordinators Japan Shinji Suzuki (the University of Tokyo) Lead beneficiary ONERA WP no. 4.2 Due date 31/12/2016 (M10) Delivered date 31/01/2017 Revision 8 Revised date 27/01/2017 Approvals Authors EU Yoko Watanabe (ONERA), Balint Vanek (SZTAKI), Akos Zarandy (SZTAKI), Antal Hiba (SZTAKI) Japan Shinji Suzuki (UTOKYO), Ryota Mori (ENRI) Task leader ONERA WP leader UTOKYO 2 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) History Revision 1 2 Date 13/12/2016 21/12/2016 3 22/12/2016 4 09/01/2017 5 16/01/2017 6 24/01/2017 7 26/01/2017 8 27/01/2017 Modifications Creation SZTAKI 1st input 1st update on the vision-based navigation approach (ONERA) 2nd update on the vision-based navigation approach (ONERA) Integration of UTOKYO’s input on the trahectory optimization SZTAKI 2nd input 3rd update on the vision-based navigation approach (ONERA) Update by UTOKYO, RICOH and ENRI 4th update on the vision-based navigation approach (ONERA) Authors Yoko Watanabe Antal Hiba Yoko Watanabe Yoko Watanabe Antal Hiba Yoko Watanabe Ryota Mori Yoko Watanabe 3 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Table of Contents Introduction ............................................................................................................................... 6 1. VISION project overview ............................................................................................... 6 2. WP4.2 overview............................................................................................................. 7 3. Objective of the document............................................................................................ 8 Preliminary design of vision-based navigation ........................................................................ 9 1. Sensor failure scenarios................................................................................................. 9 a. GNSS-based approach .................................................................................................... 9 b. ILS-based approach ...................................................................................................... 10 2. Reference coordinate frames and transformations................................................... 11 a. Inertial frame: FI........................................................................................................... 11 b. ECEF coordinate frame: FECEF ..................................................................................... 11 c. Geodetic coordinate frame: FGEO ............................................................................... 11 d. Local NED and ENU frames: FNED, FENU..................................................................... 12 e. Runway fixed frame: FRWY .......................................................................................... 13 f. Vehicle carried frame: FV ............................................................................................. 13 g. Aircraft body frame: FB ................................................................................................ 13 h. Camera frame and pixel-coordinates on an image: FC, Fi ........................................... 14 3. The aircraft kinematic model ...................................................................................... 15 4. Navigation sensors ...................................................................................................... 16 a. AHRS (IMU + Magnetometer) ...................................................................................... 16 b. GNSS/SBAS ................................................................................................................... 16 c. Baro-altimeter .............................................................................................................. 17 d. Inclinometer ................................................................................................................. 17 e. Vision systems .............................................................................................................. 17 f. ILS ................................................................................................................................. 19 a. Navigation performance criteria .................................................................................. 20 b. Integrity monitoring ..................................................................................................... 21 6. Preliminary design of vision-aided navigation system .............................................. 27 a. Estimator design........................................................................................................... 27 b. Fault detection and estimation (FDE) .......................................................................... 29 7. Validation plan............................................................................................................. 30 a. Validation of the image processing algorithms............................................................ 30 b. Validation of the estimation algorithms ...................................................................... 30 c. Plan for 1st flight test .................................................................................................... 30 d. Preliminary plan for further flight tests ....................................................................... 31 8. Others........................................................................................................................... 31 Preliminary design of vision-based guidance ......................................................................... 32 1. Trajectory optimization for small deviation from the nominal glide path ............... 32 a. Trajectory optimization................................................................................................ 32 b. Decision making ........................................................................................................... 32 c. Validation plan ............................................................................................................. 32 2. Image-based obstacle avoidance approach ............................................................... 33 a. Obstacle detection and possible avoidance maneuvers.............................................. 33 b. Detection and tracking ................................................................................................. 33 4 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) c. Decision based on tracks.............................................................................................. 33 Terminology ............................................................................................................................. 35 References................................................................................................................................ 36 5 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Introduction 1. VISION project overview To contribute towards the global goal of the aircraft accident rate reduction, this EU-Japan collaborative research project called VISION (Validation of Integrated Safety-enhanced Intelligent flight cONtrol) has the objectives of investigating, developing, and above all, validating advanced aircraft Guidance, Navigation and Control (GN&C) solutions that can automatically detect and overcome some critical flight situations. The focus of this project is on the final approach phase where more than half of the fatal accidents have occurred in the last decade. The VISION project tackles the following two different types of fault scenarios, which cover a dominant part of the causal factors of the fatal aircraft accidents in the world. • Flight control performance recovery from o Actuator faults/failures (jamming, authority deterioration, etc.) o Sensor failures (lack of airspeed etc.) • Navigation and guidance performance recovery from o Sensor failures (lack of SBAS, lack of ILS, etc.) o Proximity of unexpected obstacles These two fault types have completely different natures and so require different approaches. While the first recovery scenario set will be achieved by applying fault detection and fault tolerant control techniques, the second one will be achieved by proposing precision navigation and approach guidance systems using new onboard sensing technologies. Flight validations of the developed techniques and methods will be performed on JAXA’s MuPAL-alpha experimental aircraft in Japan for the first scenario and on USOL K-50 UAV platform in Europe for the second scenario. Figure 1 illustrates how the European and Japanese partners from academia, national research institutes and industries will make synergetic and complementary efforts to achieve these ambitious scientific objectives within this VISION project, in order to improve global aviation safety. Figure 1 EU-Japan mutual efforts to the VISION project 6 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 2. WP4.2 overview WP4 of VISION project is dedicated to the second type of failure scenario, “Navigation and Guidance performance recovery.” WP4.2 aims at developing navigation and guidance algorithms which use vision information, provided by the systems developed in WP 4.1, for the purpose of maintaining flight safety and navigability in the fault scenarios defined in WP 2.2. The objectives of this workpackage are the following: • • • • To specify the vision information and its performance criteria to be fed back into the aircraft navigation and guidance system for each fault scenario, To develop vision-based navigation algorithms which estimate the aircraft flight path relative to a runway, from the specified vision information in the navigation data degradation/failure scenario, To develop vision-based guidance algorithms which modify the aircraft flight path in order to ensure flight safety upon the detection of unexpected obstacles on the current planned path, and To adapt and improve the performance of the developed systems through analysing results of simulations and intermediate flight tests. Figure 2 shows the relationship of WP4.2 with other sub-WPs. Figure 2 The VISION work-packages 7 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 3. Objective of the document WP4.2 has been active since M4 (06/2016) and continues until M30 (08/2018). The first six month of this work-package 4.2 was devoted to • Investigation in existing approaches for vision-based navigation and guidance for aircraft final approach and landing, • Study in navigation performance criteria and integrity monitoring algorithms, especially for GNSS-based approach, and • Preliminary design of vision-based navigation and guidance for specific fault scenarios defined in WP2.2. This document provides current results of the above listed tasks, focusing on descriptions of preliminary design of vision-based navigation (in case of sensor failure) and guidance (in case of obstacle detection) systems, which are to be evaluated with data recorded during the first flight test campaign. 8 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Preliminary design of vision-based navigation 1. Sensor failure scenarios This section recalls the navigation sensor failure scenarios, defined in WP2.2 and provided in D2.1 “Fault scenario descriptions,” for which we propose a vision-based navigation system to mitigate a critical situation. Two different approach procedures are considered in this VISION project; the GNSSbased LPV approach (APV-I or SBAS CAT-I) and the classical ILS approach. The interest of using onboard vision in case of failure for both scenarios is to allow the pilot to continue the approach until a decision height corresponding to APV-I or SBAS CAT-I instead of triggering a go-around (or missed approach) procedure. VISION system only addresses the final segment approach, supposing that an accurate relative position with respect to a runway is available on the initial segment for initializing the visual tracker. a. GNSS-based approach In the first set of the sensor failure scenarios, it is supposed that the aircraft applies the GNSS-based LPV approach procedure. In this procedure, horizontal and vertical approach guidance uses the GNSS positioning based on GPS signals and the SBAS augmentation [dgac-2011]. The following four GNSS failures were listed in D2.1. 1) Degradation of GPS signals due to a reduced number of visible satellites with SBAS augmentation 2) Lack of SBAS signals with GPS signals fully available 3) Degradation of GPS signals due to ionosphere interference with SBAS augmentation 4) Lack of SBAS signals and GPS signals due to jamming We consider that the GNSS/SBAS functions correctly at the beginning of the final approach phase and then the failure occurs before the aircraft reaches at DA. GNSS-based system causes positioning errors with various reasons, such as ionosphere interference, lower atmosphere, multi-path effect, and the satellite constellation. SBAS system provides the augmentation information which improves the positioning accuracy. Figure 3 Vision-aided navigation system with degraded GNSS positioning 9 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) During the last WP4 technical meeting (held in Madrid in 10/2016, in conjunction with PM1), the WP4 partners agreed on prioritizing the scenario 2 (lack of SBAS with GPS) then 4 (lack of SBAS and GPS) over the other two. It was also re-confirmed the need of a supervision loop which provides continuously a Horizontal and Vertical Protection Levels (HPL/VPL) corresponding to an estimated error of position with a given probability of missed detection. Figure 3 illustrates a vision-aided navigation system, proposed in this project, with degraded GNSS positioning. b. ILS-based approach The second set of the sensor failure scenario supposes the classical ILS approach procedure. In this procedure, the ILS provides lateral and vertical deviations from a 3-degree final glide path with precision. In VISION project, we consider the following two ILS failures. 5) Lack of ILS information during the ILS approach 6) Misleading of ILS information due to ILS secondary lob As in the GNSS-based navigation scenario, we suppose that the ILS functions at the beginning of the final approach phase and then the failure occurs before the aircraft reaches at DA. During flight validation, as ONERA flight test site is not equipped with ILS, ILS sensor will be emulated onboard by using real-time data and a-priori knowledge on the environment (runway location, elevation and size). The WP4 partners agreed on that scenario 5, which coincide with Scenario 2 in terms of available sensor set, is prioritized over 6. Figure 4 shows a vision-aided ILS-like approach guidance system in case of ILS failure. Figure 4 Vision-aided ILS-like approach guidance system 10 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 2. Reference coordinate frames and transformations Before introducing the navigation systems design, this section overviews different reference coordinate frames. a. Inertial frame: FI An inertial reference frame is a fixed frame, in which Newton’s second law is valid. b. ECEF coordinate frame: FECEF ECEF (Earth-Centered Earth-Fixed) coordinate frame is a Cartesian frame whose origin is at the center of the earth, X-axis pointing to an intersection of the equator and the prime meridian (latitude ϕ = longitude λ = 0°), Z-axis to the North pole, and Y-axis completing the right-hand-rule. The ECEF frame is fixed on the earth, which is rotating about the ECEF Z-axis with a rotational speed ωE with respect to the inertial frame. We denote a position of a point represented in FECEF as c. Geodetic coordinate frame: FGEO The geodetic coordinate frame, widely used in GNSS-based navigation and measurements, is an Earth-fixed frame which represents a point near the earth surface in terms of latitude ϕ, longitude λ, and height h. The latitude is an angle in the meridian plane from the equatorial plane to the ellipsoid normal. The longitude is the angle in the equatorial plane from the prime meridian to the projection of the point onto the equatorial plane. The height is a distance to the point from the earth ellipsoid surface in its normal direction. As the earth ellipsoid surface coincides with the mean-sea-level (MSL), the height h of this geodetic coordinate frame is an altitude above MSL. A relation between the position in FECEF and the coordinates (ϕ, λ, h) in FGEO is given as follows. (1) where Ra=6,378,137.0 m is the length of the Earth semi-major axis and e=0.08181919 is the Earth eccentricity (values according to WGS84 [WGS-1984]). From (1), it is straightforward that the longitude λ can be obtained by (2) Finding (ϕ, h) from the position in FECEF has some difficulty. The Bowring’s method [Bowring-1976], which uses a center of curvature of the earth surface point corresponding to the point of interest (h=0 in the equation (1)), is widely used to obtain a good approximate solution without iteration. This method calculates the latitude ϕ by (3) where is a reduced latitude approximate by 11 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Then, the MSL altitude can be obtained by using the resulting latitude ϕ. (4) Our scenarios of VISION project consider a final approach phase of an aircraft, which is a near-ground operation. Hence, the altitude h remains small and so does the approximation error. d. Local NED and ENU frames: FNED, FENU The NED (North-East-Down) or ENU (East-North-Up) frame is a reference frame locally fixed to a point on the earth surface and its X-Y-Z axes pointing to North-East-Down (NED frame) or East-NorthUp (ENU frame). The Z-axis aligns the earth ellipsoid normal direction. These frames are suited for local navigation and local measurements, which is the case for our final approach scenarios. Let pECEF0 be the origin of the NED (or ENU) frame expressed in FECEF, and (ϕ0, λ0, h0) be its geodetic coordinates. Then, a point given by pECEF in FECEF can be expressed in the NED (or ENU) frame as follows. (5) where Ri represents a rotation matrix about i-axis. Figure 5 shows the three Earth-fixed coordinate frames; FECEF, FGEO and FNED. Those frames are fixed to the rotating earth and hence not an inertial frame (i.e., Newton’s second law does not apply). However, in many airplane dynamics problems, the Earth’s rotation ωE relative to FI can be neglected and those Earth-fixed frames can be used as an inertial frame [Etkin-1972]. In our navigation system design, the NED frame is used as an inertial reference. Figure 5 Earth-fixed coordinate frames (ECEF, GEO, and NED) [Cai-2011] 12 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) e. Runway fixed frame: FRWY In our application of the local relative navigation for the aircraft final approach, we define a frame locally fixed to the runway on which the aircraft is approaching. Let pECEFthd be a runway reference point, which locates at an intersection of the runway threshold and center line, expressed in the ECEF frame. We set this point as an origin of the runway fixed frame. Assume a flat runway with an approach direction ϕRWY from the North and a slight slope of θRWY from the horizontal plane. Then, as shown in Figure 6, we define the runway frame orientation so that a rotation matrix from FNED to FRWY becomes RY(θRWY)Rz(ψRWY). Hereinafter, we define the NED frame at pECEFthd. Then the transformation from FNED to FRWY is simply given by the rotation. Figure 6 Runway Frame (5) f. Vehicle carried frame: FV The vehicle carried frame is defined as the NED frame with its origin at the aircraft center-of-mass. g. Aircraft body frame: FB The aircraft body frame is a frame fixed to an aircraft, with its origin at the aircraft center-of-mass. A plane of symmetry of the aircraft body defines the X-Z plane, with the X- and Z-axes pointing forward and downward respectively. Then, Y-axis completes the right-hand-rule, resulting in pointing rightward (See Figure 7). In general, the aircraft attitude is expressed by the Euler angles (φ, θ, ψ) which rotate the Vehicle carried frame to the aircraft body frame. For local navigation problem, we can neglect the earth curvature and so the vehicle carried frame has the same 3D orientation as the NED frame defined somewhere on the earth in the vicinity of the aircraft. Let pNEDa be the aircraft center-of-mass position expressed in FNED. Then, the transformation from FNED to FB is given by (6) In the navigation filter design, instead of the Euler angles, the quaternion q is often used to represent the attitude in order to avoid the singularity occurring at ±90° pitch angle. A relation between the Euler angles and the quaternion q = [q0 q1 q2 q3] is given by The rotation matrix RB can be written in function of q as follows. (7) 13 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) h. Camera frame and pixel-coordinates on an image: FC, Fi A camera coordinate frame is fixed to a camera with its origin at the center-of-projection (COP) of the camera, its Z-axis aligned with the camera optical axis (which is perpendicular to the image plane), X- and Y-axes aligned with the horizontal (rightward in image width) and vertical (downward in image height) axes of the image plane. Let pNEDc be the camera’s COP and (φc, θc, ψc) be the Euler angles representing the camera attitude with respect to FNED. Then, the position of a point in FC is give by Let pBc and RCB be a position and an orientation (represented by a rotation matrix from FB to FC) of the camera expressed in the aircraft body frame. Then, pC and RC can be written as (8) In VISION project, we consider cameras rigidly fixed to the aircraft body. Then pBc and RCB become known constant, and so pC and RC can be determined by pB and RB, i.e. the aircraft position and attitude, as seen in the equation (8). For Stereo-vision system, the left camera’s frame is usually used as a reference frame. Figure 7 Different coordinate frames (NED, Body, Camera and Pixel-coordinate frames) 2D pixel-coordinates are defined by a frame fixed on an image with its origin at the image left-top, xaxis pointing to the right (in an image width direction) and y-axis pointing down (in an image height direction), as seen in Figure 7. Assuming a pin-hole camera model, a transformation from the 3D camera frame to 2D pixel-coordinates is determined by the camera’s intrinsic parameters; axis-skew s, focal length (fx, fy) and offset of the image center (principle point) (x0, y0). Let pi be a 2D pixel coordinate of the point pC expressed in the camera frame. Then, pi is given by (9) 14 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) where KC is called the camera intrinsic matrix, which is determined by a camera calibration process, for example, by using a Matlab calibration toolbox from Caltech1. ZC is the image depth and αC is its inverse. αC is called inverse depth and is widely used in computer vision in order to exploit the linearity property in the measurement equation (9) [Matthies1989][Civera-2008]. The same concept of inverse range is also introduced in bearing-only target tracking problems [Aidala-1983]. 3. The aircraft kinematic model The state of the aircraft is represented by the followings. • Position expressed in the ECEF frame: pECEF = (XECEF, YECEF, ZECEF) • Translational velocity expressed in the body frame: vB = (u, v, w) • Quaternion rotating from the vehicle carried frame to the body frame: qB = (q0, q1, q2, q3) • Angular velocity expressed in the body frame: ωB = (p, q, r) The VISION project treats the aircraft local navigation problem during the final approach. By neglecting the earth’s rotation and curvature, the aircraft kinematic equations are given by (10) where the rotation matrices RNED and RB are given by (5) and (7) respectively, and the 4x4 matrix Ω is defined as below. aB is the aircraft translational acceleration expressed in the body frame. 1 https://www.vision.caltech.edu/bouguetj/calib_doc/ 15 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 4. Navigation sensors This section summarizes navigation sensors available onboard an aircraft, and their measurement models. a. AHRS (IMU + Magnetometer) AHRS (Attitude and Heading Reference System) combines 3 axis gyroscopes, accelerometers and magnetometers to provide the attitude information. It outputs measurements of the nongravitational acceleration, angular velocity and attitude of the sensor at high frequency. The IMU accelerometer and gyroscope measurements incorporate a bias. AHRS uses the earth magnetic field and the gravity vector to compensate the angular velocity bias and estimate the attitude. We suppose that the AHRS sensor is fixed at the aircraft center-of-mass with its 3 axes aligned with the aircraft body frame axes. Then the AHRS measurements are modelled as follows. (11) (12) (13) where b denote the measurement bias, and ν the measurement noise modelled as a zero-mean Gaussian noise. g =[0 0 g] is the earth gravity vector defined in the NED frame. As we consider the near-ground operation, the normal gravity magnitude g at the latitude ϕ can be approximated by that at the earth ellipsoid surface: where e is the Earth eccentricity, ge = 9.7803253359 m/s2 and gp = 9.8321849378 m/s2 are the normal gravity at the equator and at the pole, respectively [WGS-1984]. For more simplicity, we can even ignore the variation of the gravity with latitude and approximate it by constant g = g(ϕ0) at the origin of the local reference frame. b. GNSS/SBAS GNSS (Global Navigation Satellite System) provides pseudo-range measurements from visible satellites to a receiver. The pseudo-range is obtained by multiplying the signal travel time from the satellite to the receiver by the speed of light c. As the satellite clock and the receiver clock are not synchronized, the pseudo-range measurement includes a bias due to their offset. Then the pseudorange measurement of i-th visible satellite is modelled by (14) where pECEFsati and pECEFrec are the position in FECEF, τI and τ are the clock bias of the i-th satellite and the receiver, respectively. The receiver position in FECEF can be obtained from the aircraft position pECEF and the fixed receiver position in FB. Here we use an estimate of the satellite clock bias and its estimation error is included in the noise (that’s why there is a “hat” on τi). νρI is a zeromean Gaussian measurement noise which includes errors from different sources such as the satellite clock and ephemeris errors, compensation errors in ionosphereic and tropospheric signal delays, multi-path effects, etc [Faurie-2011]. Standard deviation of this measurement noise, denoted by σUERE (UERE stands for User Equivalent Range Error), is calculated and provided by the receiver. The receiver clock-bias can be modelled as a 2nd order random process, such as, 16 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) (15) SBAS (Satellite-Based Augmentation System) is a GNSS augmentation system based on geostationary satellites which broadcast correction information on the satellite clock and ephemeris, and the ionospheric delay. Such correction information is collected by a network of ground stations receiving and monitoring signals from GNSS satellites. The SBAS correction significantly reduces the errors in pseudo-range measurement (i.e. σUERE). c. Baro-altimeter Barometric altimeter measures air pressure to obtain altitude information. A relation between the air pressure pair and the MSL altitude h is given by where g is the gravity acceleration, R = 287.058 J·kg·K−1 is the specific gas constant, L = 0.0065 K/m is the decrease rate of the atmosphere temperature in height, and T0 = 15°C = 288.15 K is the standard temperature [icao-1993]. This gives a = 2.2557x10−4 and b = 5.2557. P0 is the pressure adjusted to the standard atmosphere at the Sea-Level, and this information is given as the QNH pressure. Then by using it, the barometer pressure measurement can be converted to the altitude measurement which is modelled as (16) with a non-zero Gaussian measurement noise. d. Inclinometer Two inclinometers are used to measure roll (φ) and pitch (θ) angles of the aircraft. e. Vision systems In VISION project, two types of vision systems are proposed to use in order to increase an accuracy and integrity of the aircraft relative navigation to the runway in case of sensor failure. Detailed description of preliminary designs of these two vision systems is given in D2.1 “Preliminary description of vision sensor for navigation and guidance recovery scenario”. Vision system 1 provides a relative position and orientation information of cameras with respect to the runway, while Vision system 2 provides also the relative motion (linear and angular velocity) information. Relative position and orientation By using the stereo vision system, Vision System 1 and 2 will provide a relative position and orientation of the camera with respect to the runway. In other words, it finds a transformation between the camera frame FC and the runway frame FRWY, which is represented by a rotation matrix RRWY/C and the threshold position in the camera frame pCthd. The vision system 1 and 2 will be designed to output the 3D relative position Dp of the camera to the threshold point expressed in the runway frame FRWY and the relative orientation angles (Dφ, Dθ, Dψ) of the camera (FC) to the runway (FRWY), giving the rotation matrix RRWY/C. 17 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) The error model of Dp is obtained by an error model of pCthd, which is obtained from the pixelcoordinate measurement pi and the scaling factor measurement αC in equation (9). By assuming unbiased Gaussian error model for pi and αC , the inverse transformation of (9) gives the error model of Dp. The error model of the relative attitude (Dφ, Dθ, Dψ) can be modelled as zero-mean Gaussian with changing sigma’s. The details will be provided in the deliverable D4.2 “Preliminary description of vision-based system for navigation and guidance recovery scenario” to be submitted in M11. Those measurements can be expressed in function of the aircraft pose (pECEF, qB), the known camera pose parameters (pBc, RCB) in the aircraft body, and the runway pose parameters (pECEFthd, θRWY, ψRWY) as follows. Relative motion (linear and angular velocity) In addition to the relative pose (position and orientation) information, the vision system 2 also provides the relative motion information, i.e., linear and angular velocity of the camera with respect to the runway in the camera frame FC. This can be done either by deriving from the camera pose estimation for the two successive images or by calculating the velocities directly from a kind of visual odometry (estimation of camera displacement). Or the following. Based on simulation results, the output error can be modeled with experimentally determined set of Gaussian distributions, where sigma is the function of Dp. Again, the details will be provided in the deliverable D4.2 “Preliminary description of vision-based system for navigation and guidance recovery scenario” to be submitted in M11. 18 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) f. ILS ILS is composed of the lateral localizer (LOC) and the vertical glide slope (GS), which provides final approach guidance information, that is, the deviations of the aircraft flight path from the desired glide course (normally 3° glide path). Each of the LOC and GS antennas emits two lobes of signals modulated at 90Hz and 150Hz centered at the desired glide path in horizontal and vertical plan respectively (Figure 8). The deviations from the desired glide path can be measured by taking a difference in amplitude of the two signals, called DDM (Difference in depth of modulation). DDM = DM90 – DM150 Figure 8 ILS LOC and GS signal emissions where DM is the percentage modulation depth of 90 /150 Hz signals. DDM becomes zero when the aircraft is on the desired glide course. DDM is linear with respect to a deviation angle (denoted ∆θLOC and ∆θGS in Figure 8) from the front course line (DDM=0). where KLOC and KGS are called the angular displacement sensitivity. For the localizer, the nominal displacement sensitivity is defined in distance (but not in angle) and it is 0.00145 DDM/m at the ILS reference datum (= runway threshold) within the half course sector. For the glide slope, KGS = 0.0875 is defined [icao-2006]. 19 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 5. Navigation performance requirement and Integrity monitoring This section provides the current specifications of GNSS and ILS for being used in the aircraft final approach procedure defined by ICAO. a. Navigation performance criteria GNSS According to ICAO’s GNSS manual [icao-2005], the aircraft navigation system should be evaluated against the following four criteria: • Accuracy: The difference between the estimated and actual aircraft position. • Integrity: The ability of the navigation system to alert the user within the prescribed time period (time-to-alert), when it should not be used for the intended flight phase. • Continuity: The capability of the navigation system to perform its function without interruptions during the intended flight operation. • Availability: The portion of time during which the navigation system is delivering the required accuracy, integrity and continuity. GNSS alone cannot meet the stringent aviation requirements of those four criteria, thus the augmentation systems are proposed such as SBAS and ABAS with other information sources. The navigation performance required for the GNSS-based approach procedure is given in [icao-2006] and summarized in the table below (Table 1). Table 1 Navigation performance requirement for GNSS-based approach [icao-2006] GNS S Accuracy Horizontal Vertical (95%) (95%) 220m N/A (720ft) Continuity Availability Continuity risk -4 0.99 to NPA N/A 1x10 to -8 0.99999 1x10 /h -7 -6 APV 16m 20m 10s 40m 50m 0.99 to 8x10 / 2x10 / (130ft) (164ft) 0.99999 I (52ft) (66ft) approach 15s -7 -6 APV 16m 8m 6s 40m 20m 0.99 to 2x10 / 8x10 / II (52ft) (26ft) (130ft) (66ft) 0.99999 approach 15s -7 -6 CAT I 16m 6 to 4m 6s 40m 35 to 10m 0.99 to 2x10 / 8x10 / (130ft) 0.99999 (52ft) (20-13ft) (115-33ft) approach 15s (NPA: Non-Precision Approach, APV: Approach with Vertical Guidance, HAL/VAL: Horizontal/Vertical Alert Limit) Integrity risk -7 1x10 /h Integrity TimeHAL to-alert 10s 556m (0.3NM) VAL ILS The navigation performance required to ILS is also given in [icao-2006] and summarized in Table 2. Table 2 Navigation performance requirement for ILS-based approach [icao-2006] ILSLOC CAT III CAT II CAT I DDM (95%) Accuracy Modula-tion tolerance 0.031 to 0.005 0.031 to 0.005 0.031 to 0.015 ±1.0% ±1.5% ±2.5% Integrity Alert limit of Alert limit of course displacealignment ment error sensitivity 3m ±10% (10ft) 7.5m ±17% (25ft) 10.5m ±17% (35ft) 0.015DDM Continuity Max Continuity duration of risk signal absence -6 2s 2x10 / 30s -6 5s 2x10 / 15s 10s 4x10 / 15s -6 20 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Availability False signal risk -9 0.5x10 / approach -9 0.5x10 / approach -7 1.0x10 / approach ILSGS CAT III CAT II CAT I DDM (95%) 0.035 to 0.023 0.035 to 0.023 0.035 Accuracy Modul Course a-tion alignm toleran ent ce Alert limit of course alignmen t error Integrity Alert limit of displace ment sensitivit y error ±25% Alert limit of DDM= 0.0875 displace ment 0.7475θ ±1.0% +0.04θ -0.075θ +0.10θ ±1.5% +0.075 θ -0.075θ +0.10θ ±25% ±2.5% +0.075 θ -0.075θ +0.10θ ±25% Continuity Max Continu duratio ity risk n of signal absence Availability False signal risk -6 0.5x10 / approach -6 0.5x10 / approach -6 1.0x10 / approach 2s 2x10 / 15s 0.7475θ 2s 2x10 / 15s 0.7475θ 6s 4x10 / 15s -9 -9 -7 b. Integrity monitoring Integrity monitoring function is the must for the aircraft navigation system in order to avoid a pilot (or auto-pilot), by providing a warning, to continue using erroneous information in guidance and control. GNSS Integrity Monitoring ABAS (Aircraft-Based Augmentation System) applies Receiver-Based Integrity Monitoring (RAIM) algorithms on redundant satellite information to perform fault detection, identification and exclusion [Brown-1992] [Lee-1996]. The RAIM algorithms are based on a statistical test on a residual of the measured pseudo-range and a predicted value of that using the least-square error solution. RAIM requires measurements from at least five satellites for fault detection, and six satellites for fault exclusion. If an altitude input (e.g. baro-altimeter) is provided, these numbers of minimum satellites can be reduced by one. • LS-RAIM Let ρ = [ρ1 ρ2 … ρN]T be a GNSS measurement vector consisting of a set of pseudo-range measurements of N visible satellites. Then, it can be written as a nonlinear function of the receiver position pECEFrec and its clock bias τ as given in (14): (17) Let be a predicted value of the receiver position and clock bias before taken the measurements. Usually, it coincides with the estimate at the previous time step. Then the GNSS measurement vector (17) can be linearized about this prediction as follows. (18) Then the linearized GNSS measurement equation is formulated as (19) 21 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) A new estimate for the receiver position and clock bias can be obtained as a least-square solution. (20) The associated estimation error covariance will be PGNSS = GρRρGρ T where Rρ is the covariance matrix of the measurement noise νρ. The LS-RAIM (Least Square-error RAIM) algorithm uses a residual: (21) Let us consider the following two hypotheses. • H0: H0 No failure on any of the N visible satellites. • H1: H1 Failure on one of the N visible satellites, and no failure on the others. Note that the probability of having simultaneous failures on multiple satellites is quite low and hence it is usually neglected in the RAIM algorithm. As a failure introduces a bias in the pseudo-range measurement on the i-th satellite, the residual (21) becomes (22) The sum of the square of the residual (error) is defined as SSE = ∆yρ T Rρ -1∆yρ , and the statistical test is applied on the following value. (23) Under the hypothesis H0, SSE results in a Chi-squared distribution with N-4 degree of freedom, . For the fault detection, the decision test value Td can be derived by a maximum allowable false detection probability pfd such that the following equation satisfies (See also Figure 9). (24) The failure case H1 is detected when T > Td, otherwise no failure case H0 is assumed. 22 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Figure 9 Chi-squared distribution and Decision test value • Protection levels The protection levels are defined by the smallest detectable position error by means of the test statistics T > Td, while satisfying the missed detection requirement. Consider the case of failure on the i-th satellite (but not on any other satellites). Then the measurement residual ∆yρ in (22) incorporates an bias b on its i-th element, and the SSE value now follows the non-central Chisquared distribution with N-4 degree of freedom, denoted as where λ is a noncentrality parameter (Figure 9). The minimum value for λ which satisfies the missed detection probability (pmd) can be identified by (25) for a given pmd and the decision value Td defined in (24). From the relation between the noncentrality parameter λ and the pseudo-measurement bias b, the smallest detectable bias of the i-th satellite measurement ρi can be obtained by (26) where σρi is the standard deviation of the measurement noise νρi, and Dρii is the i-th diagonal element of the matrix Dρ defined in (24). Finally, the position (as well as the receiver clock bias) estimation bias induced by this pseudo-range bias is derived from the equation (20). where Gρ(:,i) is the i-th column of the matrix Gρ defined in (20). From this, the horizontal and vertical protection levels (HPL, VPL) are defined by 23 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) (27) These correspond to the smallest detectable horizontal/vertical position error which satisfies the miss detection probability requirement. In other words, these are the largest horizontal/vertical position error bias possible when not detecting the occurring failure. • Fault exclusion Once the failure is detected by means of T > Td, the next step is to identify the error source (satellite) and exclude it. In order to do so, a bank of N estimators {Ei | i=1,2,…N}, each of which uses (N-1) pseudo-range measurements excluding one from the i-th satellite, is used. The estimated receiver position and clock bias as well as the test value Ti are calculated in the same manner as in (20) and (23) for each estimator Ei with a different set of (N-1) pseudo-range measurements. Under the assumption H1, only one estimator out of N does not use the erroneous measurement and will provide a correct estimation result. If the failure occurs on the i-th satellite measurement, the estimator Ei is such an estimator. We define the decision value for the data exclusion Te in the same manner as in (24) but with the requirement of the maximum false decision of exclusion pfde instead of pfd. Then, if only one estimator, say Ej, out of the N estimators {Ei | i=1,2,…N} gives T ≤ Te, the failure can be identified on the j-th satellite and its pseudo-range measurement will be excluded. • AAIM (Aircraft Autonomous Integrity Monitoring) Unlike RAIM which is self-contained on the GNSS receiver, AAIM algorithms use other sensors (INS, baro-altimeter, etc.) to improve the FDE (Fault Detection and Exclusion) capability. In AAIM, the GNSS pseudo-range measurements ρ = [ρ1 ρ2 … ρN]T are fused with the measurements from other available sensors to estimate the receiver state (position, velocity, etc.) and the sensor bias including the GNSS receiver clock bias: x = [pT … ]T. Then the MSS (Multiple Solution Separation) method [Brenner-1998] is applied directly in the position (but not the pseudo-range) domain in order to perform the FDE. Usually, AAIM algorithm is to detect the failure on one of the N GNSS measurements assuming that the other used sensors are failure-free. Like in RAIM, consider the full-set estimator E0 which uses all the N pseudo-range measurements, and a bank of N subset estimators {Ei | i=1,2,…N}, each of which uses (N-1) pseudo-range measurements excluding one from the i-th satellite. Then the separation ∆xi is defined for each subset estimator Ei as a difference in the estimated states between this estimator Ei and the full-set one E0. The separation ∆xi and its covariance matrix ∆Pi are given by (28) (29) where Pi is the error covariance of the estimation state of Ei , and hρiT is the Jacobian matrix (a row vector) of the measurement ρi with respect to the estimation state x. The horizontal and vertical positions and their associated covariance matrices are given by 24 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) (30) Eigenvalue decomposition can be applied to the horizontal separation covariance matrix ∆Phi and the separation ∆pHi can be projected on the eigenvectors: (31) Then the test value of the horizontal separations is defined and is often approximated as follows. (32) In case of no failure, these test values follow a zero-mean Gaussian distribution with a standard deviation of ∆σΗi1. In case of presence of failure on one of the pseudo-range measurements used in Ei, they follow a non-null mean Gaussian distribution. Based on the maximum allowable false detection probability criterion pfd, the decision value TdHi is calculated as follows. (33) where erfc() is the complementary error function. The failure is detected when THi > TdHi. The same statistical test is performed also on the vertical separation, by using TVi = |∆pVi| and ∆σVi = √∆PVi. These two tests on the horizontal and vertical separations are independently performed for each of the N subset estimators. That is why the minimum false detection probability pfd is divided by 2N, a total number of the statistical tests. Similarly to the RAIM algorithms, the horizontal and vertical protection level (HPL and VPL) are to be defined based on the missed detection probability criterion (pmd). HPL is defined as the maximum position error possible without detecting any failure when satisfying this criterion: (34) By the triangle inequality, the HPL can be approximated by the following in a conservative manner. (35) Consider the case of failure of the j-th satellite, where the j-th subset estimator Ej functions correctly and its estimation error follows a zero-mean Gaussian distribution with a known covariance matrix PHi. Then, the HPL for this case of the j-th satellite failure, denoted as HPLj, is obtained by solving (36) 25 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Through the similar operation as (31) by using Eigenvalue decomposition, (37) where σΗi1 is the largest standard deviation of the covariance PHi. HPLj can be obtained for all j=1,2,…N. Then the HPL is defined as its maximum value. (38) In the same way, the vertical protection level can be obtained as follows. (39) ILS Integrity Monitoring Transmission of ILS signals are continuously monitored, and an installation is automatically switched off (cease of radiation or removal of navigation component from the carrier) when anomaly is detected. The anomaly is detected when at least one of the alert limits listed in Table 2 is exceeded. 26 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 6. Preliminary design of vision-aided navigation system This section provides description of the preliminary design of vision-aided navigation systems for SBAS or ILS failure scenarios, defined in D2.1 and recalled in Section 1. Figure 10 illustrates an overview of the aircraft navigation system, which is composed of the estimator and the integrity monitoring system for fault detection and exclusion (FDE). Figure 10 Overview of Aircraft Navigation System a. Estimator design In this preliminary design, we suppose that the aircraft attitude qB and angular velocity ωB are estimated separately from the position and linear velocity by a fusion of AHRS and inclinometer measurements. This attitude estimator is assumed to give the un-biased estimated attitude and angular velocity as well as their error covariance at the INS frequency. Then, the objective of this estimator is to reconstitute the aircraft position and translational velocity from available onboard sensors. As shown in Figure 10, the estimator is based on INS integration and correction of its drift by the other sensors (GNSS, vision, barometer) through non-linear/linear Kalman filtering techniques. The estimator runs at a frequency of INS (AHRS), and after each INS integration, each of newly arrived sensor measurements will be examined for its validity and used for the estimation correction if it is valid. Estimation state vector The estimation sate vector is defined as follows. 27 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) (40) where pNED and vNED are the the aircraft position and linear velocity expressed in the local NED frame fixed at the known runway threshold position pECEFthd, (τ, vτ) are the GNSS receiver clock bias and drift, and ba is the accelerometer bias. INS integration By taking a time-derivative of the estimation sate vector (40), the process model is given by (41) By discretizing this process model, and the INS integration will be given by (42) The error covariance matrix of this predicted state becomes (43) where the last term is due to the attitude estimation error and Measurement update At each time step, newly arrived measurements will be used to make a correction on the predicted state (42). The sensor set includes GNSS, baro-altimeter, vision systems and ILS. The integrity monitoring algorithm, to be presented in the next section, is applied first so that the erroneous information source will be removed. Then a Kalman filter-like update process (EKF, UKF, etc.) is applied on the sensor models of only the selected (non-excluded by the FDE function) sensor measurements. The estimation error on the attitude and the angular velocity would be also considered when propagating the error covariance. 28 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) The GNSS pseudo-range measurements (instead of the resulting receiver position and clock bias) are directly used in the estimation filter, which results in a tight fusion of GNSS with INS and other sensors. For the purpose of integrity monitoring, the sensor measurement sets should be independent each other, and so it is preferable to use row sensor data (especially for vision system). b. Fault detection and estimation (FDE) Like GNSS, it is very important for the aircraft navigation system to provide the integrity monitoring function. It is proposed here to apply the similar process to the AAIM algorithms (presented in Section 5.b) to perform FDE in case of the data fusion of different sensors. Note that, in this work, the INS (AHRS) is assumed to function always correctly and never fail. GNSS-RAIM As stated before, GNSS sensor normally provides a set of redundant measurements, while the other sensors do not. Therefore, for GNSS, the RAIM algorithm introduced in Section 5.b can be already applied before the sensor fusion. In the RAIM, the initial receiver position can be obtained from the predicted state from the INS integration (42). So the erroneous pseudo-range measurement is supposed to be removed already before the measurement update. FDE for sensor fusion Consider that measurements from m different sensors are newly arrived at a time step k+1. Two approaches can be proposed for the FDE. The first one is to apply the test statistics on the innovation term for each of the m sensor independently. It is the similar test as in the RAIM, the equation (23). Then the decision threshold is used to detect the sensor failure. The second approach is to apply a similar process to the AAIM algorithm. In this approach, a bank of the m subset estimation filters {Ei | i=1,2,…m}, each of which uses measurement from the (m-1) sensors except the i-th sensor. The updated estimation state of each Ei is compared to that of the full-set estimator E0 which uses the available measurements from all the m sensors. Then the MSS (Multiple Solution Separation) method can be applied in the same way as in the AAIM algorithm (Section 5.b) to detect the sensor failure. The horizontal and vertical protection levels can be also derived in the same way as in (38, 39) based on the estimation error covariance matrix. In the preliminary design, the occurrence of failure is assumed to be at the current time step but not before. Extension of the work by assuming the failure occurrence sometime in the past could be considered later in the project, by keeping the bank of estimation filters from the previous time step for the FDE. 29 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 7. Validation plan This section overviews the validation plan of the vision-based navigation systems for the sensor failure scenarios. a. Validation of the image processing algorithms Validation will be made in three phases: • Own simple environment: Synthetic projections of runway from known camera position and orientation. Input error is only caused by numeric representation (no pixelation). Zero error of image processing algorithm (features are known). Goal: Validation of position and orientation extraction from known image features. Status: Goal has already reached. • FlightGear-based environment: Realistic computer graphics provides input for image processing. The image is pixelized, realistic distance-based blurs. Goal: Testing and preliminary validation of Image processing algorithms. The reached accuracy is compared to navigation performance requirements. Status: In progress. • Offline/Online image-based navigation on real data: All necessary data is recorded during flight tests for offline repeatability. Goal 1: Best possible accuracy without real-time constraints. Final Goal: Real-time online operation in closed-loop flight test. Status: Preparation for first capture-only flight experiment. b. Validation of the estimation algorithms Validation will be made in three phases: • Numerical simulation: The estimator designs (including the FDE functions) will firstly be tested in numerical simulation environment with simulated sensors. ENRI provides the GNSS and ILS sensor models (nominal and failure), which will be integrated in Matlab simulations for the estimator validation. The image processor outputs will be also simulated based on the models given in Section 4.e. • Offline test on real flight (ground) test data: All necessary data (image + sensor data) is recorded during flight tests for offline repeatability. The GPS-RTK/AHRS navigation filter on the flight avionics provides the true reference of the trajectory. The developed estimation algorithms will be tested on these real data, with a real image processing results obtained offline/online. • Online test on the platform: After the above simulation validations, the estimation algorithms will be implemented on the onboard software architecture along with the image processor, and will be tested online onboard the K50 platform during the flight. It will be tested in a open-loop manner (i.e., the estimation results will not be fed-back into the flight guidance and control module), then in a closed-loop manner. c. Plan for 1st flight test The primary goal of first flight test is to ensure vision sensor operation on the K-50 platform and record video data synchronized with other sensor data. Operations on Ground before first flight: • Installation of vision sensors to K-50 airframe: Mounting camera modules, mounting image processing payload computers, power supply, cabling between camera modules and image processing computers, cabling Onera payload computer and image processing computers. • Testing operation of modules: Testing operation of vision sensor after mounting while K50’s engine is operating. • Calibration of vision sensor: Measure the position and orientation of cameras after mounting. Tune anti-vibration equipment. • Testing communication of vision sensor: RS232 communication test, UDP test. Flight experiments: 30 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) • • Recording: Record video data with other sensor data during final approach and landing. Different approach paths (and orientations) are needed for better evaluation of image processing algorithms. Reliability test: The system has to operate during the experiments. d. Preliminary plan for further flight tests • • Pseudo closed-loop test: The vision sensor calculates and sends its output to ONERA payload computer, but it is not used by the control loop. Closed-loop test: The vision sensor’s output is integrated to the control. 8. Others Different modules use different frames (local NED/ENU, body, runway fixed, camera, etc.) introduced in Section 2. These differences require special attention during the project. 31 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Preliminary design of vision-based guidance 1. Trajectory optimization for small deviation from the nominal glide path a. Trajectory optimization UTOKYO works on online trajectory optimization. The scenario is defined as follows; when a plane is approaching the runway, a flying obstacle on the glide path is detected, or a moving obstacle is detected on the runway. The optimal flight path which can avoid the obstacle is computed based on the detected obstacle’s position. Research on potential trajectory optimization algorithms is ongoing. The optimal trajectory should avoid the obstacle with sufficient reliability based on the image sensor information and the onboard-information (aircraft position and attitude). The image sensor can detect the relative distance to the obstacle, but it is difficult to detect the exact position of the obstacle due to the presence of noise in the data. Besides, the possible detection range and the camera angle are also limited, and the optimal trajectory should keep the constraints associated with the aircraft flight envelope and ATC. Therefore, the position of the obstacle should be estimated first, and then the optimal trajectory can be generated to avoid the obstacle. The constraints mentioned above are considered in the optimization, and the objective function will consist of the distance from the obstacle, aircraft acceleration, etc. b. Decision making In the optimal computation, the fundamental decision must be determined whether a landing approach continues or go-around is carried out. Decision making algorithms are investigated based on situations. Present operations and aeronautics laws are checked at the same time. c. Validation plan Since it is difficult to conduct a flight experiment of the whole scenario for safety reasons, it will be validated in a simulation with the required data obtained offline. The offline data will be obtained in advance with two drones. A drone (drone A) will act as an approaching aircraft with an image sensor and GPS/INS installed, while another drone (drone B) will act as a flying obstacle with GPS. Drone A will fly straight on a nominal path, and drone B will intersect the nominal path at certain timing. Drone A will detect the drone B by image sensor, and the relative distance to the drone B will be obtained. Since Drone A will have a GPS/INS installed, its own position and attitude can also be obtained. Drone B has a GPS, so the accuracy of the relative distance between drone A and drone B will also be investigated. This data acquisition experiment is expected to be conducted several times to simulate various cases. Finally, using the obtained data (image sensor data and aircraft position/attitude data), a whole scenario including trajectory optimization will be numerically simulated. In numerical simulations, a fixed-wing aircraft model of UAV will be considered. 32 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) 2. Image-based obstacle avoidance approach Our approach is based on recent image-based obstacle avoidance results achieved at SZTAKI [Bauer2016, Hiba-2016, Zsedrovits-2016]. We have reached the proof of concept level real-time on-board avoidance in the case of sky-background detections. There are many interesting contributions on this topic [Fasano-2008, Fasano-2015, Ramasamy-2014], most of them utilize RADAR/LIDAR and IR sensors beyond the visible light spectrum video. IR is very useful in bad weather conditions and in the detection of jet engines. Our results with visible light spectrum images can be applied to small- and micro-UAVs and could enhance the reliability of sensor fusion approaches that lead to obstacle avoidance products for mid- and large-sized UAVs and aircrafts. a. Obstacle detection and possible avoidance maneuvers SZTAKI works on image-based flying obstacle avoidance. The task is to detect and track unknownsized flying objects on camera screen, and alert the piloting system if an object moving towards our aircraft. The system assumes straight flight paths and chooses the most appropriate from predefined avoidance maneuvers. RICOH works also on obstacle detection on runway, with the help of stereo vision sensor. In the presence of an obstacle on runway the aircraft starts a missed approach procedure (go-around). b. Detection and tracking Image-based obstacle or intruder aircraft detection often divided to two scenarios: detection against sky background, detection against non-sky background. We will use different methods for the two scenarios. The basic steps of detection are: sky-ground separation, blob detection, filtering false objects. After filtering the remaining blobs are added to the tracker. We prefer tracking false objects than losing a real object. False objects can be eliminated based on multiple observations, however they can corrupt the tracks of real objects. c. Decision based on tracks The tracks consist centroid (x,y) and size (Sx,Sy) information about the intruder aircraft. Monocular vision-based techniques investigate the trends of centroid and size. A trivial example is to initiate an avoidance maneuver if the size of the object on the image plane increases. Our decision making approach assumes that the intruder and the own aircraft follow a straight flight path with constant speed. We model the intruder as a disk with radius r in the horizontal plane. With these assumptions we could estimate the closest point of approach (CPA) as a multiplicand of 2r (the size of the intruder is not known), and also the time to closest point of approach (TTCPA). It is easy to give an appropriate threshold on CPA and TTCPA. For instance, we initiate an avoidance maneuver if the intruder approaces us closer than 10 times its size, and we want to make this decision 20 seconds before this CPA realizes. For details see [Bauer-2016]. 33 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Figure 11 Basic properties of detected aircraft on the image plane Figure 12 Disk model of intruder object. P is the image plane 34 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) Terminology AAIM ABAS AGL AHRS ATC CAT COP CPA DA(H) DDM DGAC DoF ECEF EKF ENU FAA FDE GCS GN&C GNSS GPS GS HAL HITL HPL ICAO ILS IMU INS LOC LPV NED PL RAIM RNAV RTK SBAS SSE TCH TTCPA UAV UKF VAL VPL WP : Aircraft Autonomous Integrity Monitoring : Aircraft-Based Augmentation System : Above Ground Level : Attitude and Heading Reference System : Air Traffic Control : CATegory : Center of Projection : Closest Point of Approach : Decision Altitude (Height) : Difference in Depth of Modulation : Direction Générale de l’Aviation Civile (French Civil Aviation Authority) : Degrees of Freedom : Earth-Centered Earth-Fixed : Extended Kalman Filter : East-North-Up : Federal Aviation Administration (U.S.A.) : Fault Detection and Exclusion : Ground Control Station : Guidance, Navigation and Control : Global Navigation Satellite System : Global Positioning System : Glide Slope : Horizontal Alert Level : Hardware-In-The-Loop : Horizontal Protection Level : International Civil Aviation Organization : Instrument Landing System : Inertial Measurement Unit : Inertial Navigation System : LOCalizer : Localizer Performance with Vertical guidance : North-East-Up : Payload : Receiver Autonomous Integrity Monitoring : aRea NAVigation : Real-Time Kinematic : Satellite-Based Augmentation System : Sum of Squared Errors : Threshold Crossing Height : Time to Closest Point of Approach : Unmanned Aerial Vehicle : Unscented Kalman Filter : Vertical Alert Level : Vertical Protection Level : Work Package 35 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) References [ec-2012] [dgac-2011] [faa-2015] [casa] [WGS-1984] [Etkin-1972] [Bowring1976] [Cai-2011] [Matthies1989] [Civera-2008] [Aidala-1983] [Faurie-2011] [icao-1993] [icao-2005] [icao-2006] [Brown-1992] [Lee-1996] [Brenner1998] [Bauer-2016] [Hiba-2016] [Zsedrovits2016] [Fasano-2008] [Fasano-2015] “RNAV Approaches”, Eurocontrol, 2012. “Technical Guidelines O1 – PBN Guidelines for RNP APCH operations also known as RNAV(GNSS) Edition No.2”, DGAC, 2011. “Aeronautical Information Manual – Official Guide to Basic Flight Information and ATC Procedures”, US Department of Transportation FAA, 2015. “Go-arounds”, Australian Government Civil Aviation Safety Authority, https://www.casa.gov.au/standard-page/go-arounds “World Geodetic System 1984 – Its Definition and Relationships with Local Geodetic Systems”, National Imagery and Mapping Agency, 1984. B. Etkin, “Dynamics of Atomospheric Flight”, John Wiley & Sons, 1972. B.R. Bowring, “Transformation from Spatial to Geographical Coofdinates”, Survey Review 23, 1976. G. Cai, B.M. Chen and T.H. Lee, “Unmanned Rotorcraft Systems”, Chapter: Coordinate Systems and Transformations, Springer, 2011. L. Matthies and T. Kanade, “Kalman Filter-based Algorithms for Estimating Depth from Image Sequences”, International Journal of Computer Vision, 1989. J. Civera, A.J. Davison and J.M. Martinez Montiel, “Inverse Depth Parametrization for Monocular SLAM”, IEEE Transactions on Robotics, vol.24, no.5, 2008. V.J. Aidala and S.E. Hammel, “Utilization of Modified Polar Coordinates for BearingsOnly Tracking”, IEEE Transactions on Automatic Control, vol.28, no.3, 1983. F. Faurie, “Algorithmes de contrôle d’intégrité pour la navigation hybride GNSS et systèmes de navigation inertielle en présence de multiples mesures satellitaires défaillantes,” Ph.D. thesis, University of Bordeaux, 2011. “Manual of the ICAO Standard Atmosphere”, ICAO Doc 7488/3, 1993. “Global Navigation Satellite System (GNSS) Manual”, ICAO Doc 9849/457, 2005. “Aeronautical Telecommunications: Volume 1 Radio Navigation Aid”, Annex 10 to the Convention on International Civil Aviation, ICAO, 2006. R. Grover Brown, “A baseline of GPS RAIM Scheme and a Note on the Equivalence of Three RAIM methods”, Journal of the Institute of Navigation, vol.39, no.3, 1992. Y. Lee, K. Van Dyke, B. Decleene, J. Studenny and M. Beckmann, “Summary of RTCA SC-159 GPS Integrity Working Group Activities”, Journal of the Institute of Navigation, vol.43, no.3, 1996. M.A. Brenner, “Navigation system with solution separation apparatus for detecting accuracy failures”, US Patent 5760737 A, 1998. P. Bauer, A. Hiba, B. Vanek, A. Zarandy, and J. Bokor. Monocular image-based time to collision and closest point of approach estimation. Mediterranean Conference on Control and Automation, 2016. A. Hiba, T. Zsedrovits, P. Bauer, and A/ Zarandy. Fast horizon detection for airborne visual systems. International Conference on Unmanned Aircraft Systems, 2016. T. Zsedrovits, P. Bauer, B. J. M. Pencz, A. Hiba, I. Gozse, M. Kisantal, M. Nemeth, Z. Nagy, B. Vanek, A. Zarandy, and J. Bokor. Onboard visual sense and avoid system for small aircraft. IEEE Aerospace and Electronic Systems Magazine, 2016. G. Fasano, D. Accardo, A. Moccia, C. Carbone, U. Ciniglio, F. Corraro, and S. Luongo. Multi-sensor-based fully autonomous non-cooperative collision avoidance system for unmanned air vehicles. Journal of aerospace computing, information, and communication, 5(10):338_360, 2008. G. Fasano, D. Accardo, A. Elena Tirri, A. Moccia, and E. De Lellis. Sky region obstacle detection and tracking for vision-based UAS sense and avoid. Journal of Intelligent & Robotic Systems, pages 1_24, 2015. 36 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019) [Ramasamy2014] S. Ramasamy, R. Sabatini, and A. Gardi. Avionics sensor fusion for small size unmanned aircraft sense-and-avoid. In Metrology for Aerospace (MetroAeroSpace), 2014 IEEE, pages 271_276. IEEE, 2014. 37 EU-H2020 GA-690811 NEDO GA-062800 VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)