SOFT COMPUTING IN THE SMART COCKPIT
A Workshop
The 2003 International Symposium on Intelligent Control
Houston, Texas
October 5-8, 2003
ISIC-2003-1
Valasek, Ioerger, Painter
INSTRUCTORS
Dr. John Valasek - Assoc. Professor, Aero. Eng., Texas A&M.
valasek@aero.tamu.edu
979 845-1685.
Dr. Tom Ioerger - Assoc. Professor, Cptr. Sci., Texas A&M.
ioerger@cs.tamu.edu 979 845-0161.
Dr. John Painter - Professor, EE, Aero., Cptr. Sci. (Ret.), TAMU
altair@tca.net
ISIC-2003-2
979 696-0429.
Valasek, Ioerger, Painter
COURSE OUTLINE
SECTION I
• Introduction To The Smart Cockpit And Free Flight
• Free Flight Functionality
• Architecture: Aircraft & Simulator, Software & Hardware
SECTION II
• Artificial Intelligence In The Cockpit
• Intelligent Computing Techniques
• Software Agents: The Key To Smart Cockpit Software
SECTION III
• Real Time Flight Simulation: The Basics
• Real Time Flight Simulation For Free Flight
SECTION IV
• Selected Research Results
• Conclusion: Summing It Up
ISIC-2003-3
Valasek, Ioerger, Painter
SMART COCKPIT COMPUTING - WHAT IS IT?
• COCKPIT FUNCTIONAL INTEGRATION VIA SOFTWARE
 Helping the Pilot Visualize, Understand, and Fly.
 Helping the Airplane Automate Nominal Flying Tasks.
 Helping Air Traffic Control With Transition to “Free Flight.”
• VIEWING AVIATION AS A SOFTWARE PROBLEM.
 Reducing Pilot Task Load Via Automation.
 Increasing General Aviation (GA) Pilot Performance in Weather.
 Increasing GA Weather Access to Un-Instrumented Airports.
 Increasing Flight Safety With
- Automatic Collision Avoidance.
- Automatic Weather Avoidance.
- Automatic Terrain Avoidance.
- All Three at the Same Time.
ISIC-2003-4
Valasek, Ioerger, Painter
SMART COCKPIT COMPUTING - WHY IS IT?
• HAVE YOU FLOWN, LATELY?
 Major City Airports are Traffic Saturated.
 Air Travel is Geared to Using Major Cities.
 Only 715 Airports are Weather Instrumented.
 The Answer is Not More Jumbo Jets.
• WHY NOT FLIGHT-ENABLE THE MEDICALLY QUALIFIED?
 There are 5,400 Public Airports.
 Flying Can Become a Public Utility.
 Computing Technology Can Satisfy the Major Requirements.
- Inexpensive Airport Weather Instrumentation .
- GA Aircraft Relative Affordability.
- GA Pilot Weather Proficiency Increased.
- GA Pilot Training Affordability.
ISIC-2003-5
Valasek, Ioerger, Painter
IS THE SMART COCKPIT JUST FOR GA?
• TECHNIQUES APPLY TO AIR TRANSPORT AS WELL AS GA.
 Improves Situational Awareness and Multi-Tasking.
 Reduces Pilot Work-Load.
 Works for the Pilot, Not Vice-Versa.
 Works for the Pilot, But Doesn’t Replace Him.
• BUY THIS TECHNOLOGY “BY THE YARD.”
 Software Architecture is Modular - by Function.
 Some Functions Require Additional Cockpit Instrumentation.
 Implement Functions as You Can Afford Them.
 Augmentable Functionality is the “Name of the Game.”
ISIC-2003-6
Valasek, Ioerger, Painter
SMART COCKPIT COMPUTING - WHERE IS IT?
• AT THE CROSS-ROADS OF MULTI-DISCIPLINES.
 Flight Control & Air Traffic Management.
 Intelligent Control & Soft Computing.
 AI & Software Agents.
 Human Factors & User Interface Design.
• IN THE HANDS OF MULTIPLE AGENCIES.
 Government - NASA Langley Research Center (SATS).
 Academia - Aeronautical Engineering, Computer Science, Etc.
 Industry - Avionics Companies (FMS, GPS, etc.).
ISIC-2003-7
Valasek, Ioerger, Painter
SATS - THE PROGRAM
Small Aircraft Transportation System
• FIVE-YEAR DEMONSTRATION GOALS (2001-2005)
 Technology and Operational Capability:
1. Higher Volume @ Non-ILS Airports
2. Decreased Landing Minimums (Weather).
3. Increased Single-Pilot/Mission Safety/Reliability.
4. Integration of SATS Traffic With NAS.
• THE OPERATIONAL/TECHNOLOGY KEYS.
 Increasing Cockpit Technology Increases Operational Capability.
 “Self-Controlled Airspace” (SCA) for Smaller Airports.
 Increasing “Usable” Airports by 750% (4,685 + 715 = 5,400).
 ATC-Acceptable Procedural/Spatial Interface and Hand-off.
ISIC-2003-8
Valasek, Ioerger, Painter
SATS - THE APPROACH
Small Aircraft Transportation System
ATC: FAA Air Traffic
Control.
IAF & FAF: Initial- and
Final-Approach Fixes.
ADS-B: Automatic
Dependent Surveillance
Broadcast (Radar Xpndr.)
FAF
RUNWAY
AMM: Airport Management
Module (Digital Data-Link)
• ATC Clears Aircraft to SCA Holding Stack at IAF.
• Self-Separation via ADS-B (Req. Conflict Mgt. Software).
• Approach Sequencing and Airport Info. via AMM.
ISIC-2003-9
Valasek, Ioerger, Painter
SMART COCKPIT RESEARCH - DOING IT
• REQUIRES A FIXED-BASE FLIGHT SIMULATOR.
 General-Purpose Cockpit.
 Realistic Controls.
- Stick, Rudder, Throttles, Flaps, Trim, Gear, Brakes.
- It Must be Realistic to Fly.
 General Purpose Panel Displays.
- Touch-Screen LCDs Are Good (for Research/Development).
 Forward-Projection Screen System.
- At Least 90-degrees Wide.
- 160-degrees is Better (Wrap-Around, 3-Screen).
• THE SIMULATOR IS A SOFTWARE DEVELOPMENT TOOL.
 Software Modules Developed in MATLAB® - Ported to C++.
 Software Modules Integrated in the Flight Simulator Computers.
 Software Functionality Validated/Corrected by Pilots Flying It.
ISIC-2003-10
Valasek, Ioerger, Painter
FREE FLIGHT - WHAT IS IT?
• A MOVING TARGET.
 Its Specification is Not Yet Stable.
 Ideal Agreed, but Details Disagreed.
 Players: FAA, NASA, Aviation Industry.
• THE ORIGINAL IDEAL1
 “Freedom of Choice” … of IFR Routes (RTCA - 1995).
 Pilot Selection of Trajectory … in Real Time.
 “Max/Max” Solution for Safety/Efficiency.
• THE PROBLEM.
 Requires New Air/Ground Operations and Technology.
1Control Applications and Challenges in Air Traffic Management, by Joseph W. Jackson and Steven M. Green,
Proceedings of the American Control Conference, June, 1998, pp. 1772-1788.
ISIC-2003-11
Valasek, Ioerger, Painter
FREE-FLIGHT - COCKPIT FUNCTIONALITY
• VOICE RADIO.
• NAVIGATION
• PANEL INSTRUMENTS
• SITUATION DISPLAY
- Global Positioning System (GPS).
- Flat LCD With Touch-Screen.
- Flat LCD With Touch-Screen.
(Moving Map Reqs. Nav. Data Base).
• AVOIDANCE FUNCTIONS:
 Collision - On Situation Display (Reqs. ADS-B Beacon Add-on).
 Weather On Situation Display (Reqs. A/G Digital Data Link).
 Terrain - On Situation Display (Reqs. Nav. Data Base).
• HEAD-UP DISPLAY (HUD)
- For Eyes Out of the Cockpit.
- Instrument and Approach Displays.
• APPROACH AIDS:
- GPS, ILS, and/or SATS.
- On HUD and/or Situation Display.
• PILOT ADVISOR
- On HUD and/or Situation Display.
(Req. Flight Mode Interpreter)
ISIC-2003-12
Valasek, Ioerger, Painter
BASIC GUIDANCE LOOP(S)
FLIGHT MANAGEMENT SYSTEM
FLIGHT
PLAN
Closed-Loop
Open-Loop
NAVIGATION
+
GUIDANCE
AUTOPILOT
PILOT
Manual
CONTROL
AIRFRAME
NOTE: CDU Interfaces Not Explicit
ISIC-2003-13
Valasek, Ioerger, Painter
THE BASIC NAVIGATION TRIANGLE
The Problem is “Wind”
Air Vector:
Heading/True-Air-Speed = 090°/150
Wind Vector:
WD/WV = 350°/20
Ground Vector: Track/Ground-Speed = 097°/155
• CONVENTIONS.
 Map Convention: North (N) is “Up.”
 Directions Measured Clockwise, From True North (360°)
 Air Vector and Ground Vector are “To” Direction.
 Wind Vector is “From” Direction.
• AIR NAVIGATION REQUIRES COMPUTATION OF “WIND.”
 Manual Wind Computation Uses Mechanical Computer - “E6B.”
 Automated Digital Wind Computation in Avionics.
ISIC-2003-14
Valasek, Ioerger, Painter
GLOBAL POSITIONING SYSTEM (GPS)
 SV1
• THE NEW SOLE MEANS OF AIR NAVIGATION.
SV9


• A SATELLITE MULTI-RANGING SYSTEM
 Global Coverage. All-Weather Capability.
 24 12-Hour Satellites (3 On-Orbit Spares).
 Position/Velocity Output Latitude, Longitude, & Altitude.

 4 Range Measurements - 3 Position Coordinates
SV17
& Precise Time (12 pico-sec.).
 Position Error: ~ A Few Meters.
 Position/Velocity Output - Lat., Long., & Alt.
 Soon to Replace Visual Omni Range (VOR)
& Instrument Ldg. System (ILS).
ISIC-2003-15
Valasek, Ioerger, Painter
NAVIGATION/GUIDANCE FUNCTION
Manual Flying
• BASIC NAVIGATION FUNCTION.
 GPS - Time, Position (LAT, LNG, ALT), Velocity (TRK, GS, ROC).
 Wind Computing - A/C Data Input (MAG HDG/VAR, IAS, TEMP).
 Guidance (Pilot Computed) - Waypoint HDG, ETE, ETA.
•AUGMENTED NAVIGATION FUNCTON.
 Navigation Data Base (Waypoint Location Data).
 Guidance Computing - Adds Altitude (AGL) and Approach NAV.
• HIGH-LEVEL NAVIGATION FUNCTION.
 Flight Plan Driven - Requires Pilot Entry of Flight Plan.
 Guidance Computing - Adds Waypoint Maneuver Alerts (Turn, Climb).
ISIC-2003-16
Valasek, Ioerger, Painter
NAVIGATION/GUIDANCE FUNCTION
Autopilot Flying
• AUTOPILOT (A/P) FUNCTION.
 Autopilot Drives A/C Control Surfaces (Servo).
- Ailerons, Elevator, Rudder, Throttle (Optional).
 Autopilot Inputs - HDG, ALT, IAS (Optional).
 Inputs Manually by Pilot, or Computer-Generated.
•FLIGHT MANAGEMENT SYSTEM (FMS) FUNCTION.
 Automates Guidance for High-Level NAV.
 Computes and Issues Guidance Commands to Autopilot.
 Simultaneous Automatic Navigation and Maneuver.
 Automates All Flight Phases, Including Approach/Landing.
• NAVIGATION/GUIDANCE FUNCTIONAL HIERARCHY.
 Manual Flying
Autopilot
FMS
ISIC-2003-17
Valasek, Ioerger, Painter
COCKPIT DISPLAYS, BY FUNCTION
• THE PILOT’S VIRTUAL WORLD.
 It’s All About Situational Awareness.
 The Pilot “Reckons” His Situation in 4D Space-Time.
 Build a 4D Mental Image … Using 2D Displays.
• DISPLAY DIFFERENTIATION AND TYPING.
 “Internal” Situation - The Airplane
(Trad. Gauges & Switches
 “External” Situation - The Flight.
Long-Term - Relatively Static (Trad. Maps & Books).
Short-Term - Very Dynamic (Traditionally, Gauges).
 Internal and Short-Term External Display Commonalities.
 Two Different Genre of Display.
ISIC-2003-18
Valasek, Ioerger, Painter
COCKPIT DISPLAY - GUIDANCE & NAV
• DISPLAYS TAILORED TO TEMPORAL INFO I/O NEEDS.
• INFORMATION OUT, ONLY.
 Long-Term - Navigation (ex. - Moving Map on Panel Display).
 Short-Term - Guidance/Control (ex. - A/C Dynamics on HUD).
 Continuously Functioning, Disparate Displays.
• INFO INPUT AND OUTPUT - Flight and/or System Management.
 Long-Term - Flight Planning (ex. - One mode of Panel MFD).
 Short-Term - System Management (ex. - A/P, FMS Modes).
 Sequentially Functioning, Multi-Function Display (MFD).
 I/O Displays Called “Control/Display Unit” (CDU)
ISIC-2003-19
Valasek, Ioerger, Painter
THE VIRTUAL RUNWAY
• HUD DISPLAY OF A “VIRTUAL” RUNWAY OUTLINE.
 Generated from GPS and Aeronautical Data-Base.
 “Breaking Out” at 200 Feet, on an ILS Approach.
ISIC-2003-20
Valasek, Ioerger, Painter
HAZARD AVOIDANCE FUNCTION
• HAZARDS TO FLIGHT.
 Severe Weather - (ex. - Thunderstorms).
 Conflicting Air Traffic - (ex. - VFR Collisions).
 Controlled Flight Into Terrain - (ex. - Mountain Flying).
• FREE FLIGHT HAZARD AVOIDANCE REQUIREMENTS.
 Cockpit Acquisition of Hazard Data - (Digital Radio Data).
 Cockpit Computation of Avoidance Trajectories - (Guidance).
 Cockpit Display of Hazard/Avoidance Imagery - (Map).
• FINER POINTS.
 Cockpit Trajectory-Balancing Between Multiple Hazards.
 Automated Negotiation Between Aircraft and Air Traffic Control.
ISIC-2003-21
Valasek, Ioerger, Painter
COCKPIT DISPLAY - HAZARD AVOIDANCE
• THREE HAZARDS OF DIFFERING TEMPORALITY.
 Terrain - Long-Term (No Dynamics).
 Weather - Long- to Medium-Term - (Dynamics Not A/C-Scale).
 Traffic - Medium- to Short-Term - (Dynamics A/C-Scale).
• DISPLAY CHOICES.
 All Three Situations Require Info-Out, Only, Display.
 Imagery of All Three Hazards Can Overlay Moving Map (NAV).
 Long-Term Guidance Vectors Can Also Overlay Moving Map.
 Display “Clutter” is a Human Factors Issue, Here.
 Short-Term Steering Commands on A/C Dynamics Display (HUD).
ISIC-2003-22
Valasek, Ioerger, Painter
SOFTWARE ARCHITECTURE
Form Follows Function
• BUILDING A DEVELOPMENT TOOL AND ENVIRONMENT.
 When You’re Up to Your Armpits in Alligators … ?
 … Remember, the Goal is a Research Tool.
 Don’t be a Slave to Current Cockpit Avionics.
 This Problem is Like GPS … 90% Software.
• HOW TO LAY OUT THE SOFTWARE ARCHITECTURE.
 Modularize, Modularize, Modularize !!
 Structure the Whole, Function by Function.
 Go for Independent, Communicating, Software Modules.
- “Independence” as in Task Independence, not Data Independence.
- Hence, “Communicating” Modules (AI’s “Message-Passing”).
 Let the Computing Hardware Take Care of Itself.
 Think About Project Management and Configuration Control.
ISIC-2003-23
Valasek, Ioerger, Painter
ARCHITECTURE - DATA-FLOW DIAGRAM
Aircraft
Dynamics
HUD
FCS
Pilot
A/P
CDU
FMS
FLT
PLN
NAV
NAV
DB
MFD
TFC
GEN
(MAP)
WX
GEN
TFC
AVD
EXEC
AVD
WX
AVD
ISIC-2003-24
Valasek, Ioerger, Painter
SOFTWARE ARCHITECTURE REALIZATION
IN A FLIGHT SIMULATOR
• COST-EFFECTIVE SIMULATOR IMPLIES …
… DISTRIBUTED COMPUTING.
 Use “Simple” Computers, One Per Display.
 Displays:
- MFD, CDU, HUD, and the Three-Screen, Projected “World.”
 Projecting “The World” May Not Be So Simple - Graphics Engine.
- Surface Geometry Generation is Compute-Intensive.
- Pilot Requires 30 Frame per Second Refresh Rate.
 Use a “Projected HUD” - Simplify Cockpit Hardware.
 Simulator Controller Station - One More Computer & Display.
 Settle on 4 Computers - 3 MS-Windows PCs, 1 SGI Unix Machine.
ISIC-2003-25
Valasek, Ioerger, Painter
SIMULATOR HARDWARE ARCHITECTURE
Projection Screen With HUD
Projector
Center
CPTR
GRAPHICS
Flaps Throttles
Stick
&
Rudder
Gear
&
Trim
MFD
(button)
CDU
(touch)
CTRLR
(kybd)
PC-1
MFD
PC-2
CDU
PC-3
CTRLR
Ethernet
Multiplexed Serial Port Adapter
ISIC-2003-26
Valasek, Ioerger, Painter
ARTIFICIAL INTELLIGENCE IN THE COCKPIT
• A FUZZY “FLIGHT MODE INTERPRETER.”1
 Fuzzy Decision Tool - Bayes Connectives.
 Flight “Modes” as State Partition.
 Taxi; Take-off; Climb-out; Cruise; Hold; Initial, Final, and Missed
Approach; and Land - 9 Modes.
 Membership Functions Modeled for Particular Aircraft.
• A RULE-BASED “PILOT ADVISOR.”
 Keeping the Flight Within the “Envelope.”
 Mode-based - Driven by Flight Mode Interpreter.
 Rules for Instrument Flight.
 Rules for Performance of This Particular Airplane.
1 “Hypertrapezoidal Fuzzy Dynamic State Interpreter,” U.S. Patent 6,272,477,B1,
by Wallace E. Kelly, III and John H. Painter, Aug. 7th, 2001.
ISIC-2003-27
Valasek, Ioerger, Painter
HYPERTRAPEZOIDAL MEMBERSHIP FUNCTIONS
• A 2-DIMENSIONAL PROJECTION OF A 9-DIMENSIONAL M.F.
 9 State-Variables and 9 Modeled Flight Modes.
CRUISE
INITIAL APPROACH
`
1
0.5
FINAL APPROACH
0
150
LANDING
3000
100
airspeed
[knots]
2000
50
1000
0
0
ISIC-2003-28
altitude
[feet]
Valasek, Ioerger, Painter
FUZZY LOGIC - OTHER COCKPIT APPLICATIONS
• AUTOMATING DISPLAY CALL-UP AND FORMATTING.
 Mode-Driven, With Pilot Over-ride.
• BLENDING MULTIPLE GUIDANCE TRAJECTORIES.
(A Fuzzy Executive Guidance Agent.)
 Multiple Hazard-Avoidance Trajectories.
 Example: Nominal Flight Plan versus Weather and/or Traffic.
 Define Spatial Trajectory-Risk Functions.
 Normalize the Set of Evaluated Risks (0-1).
 Prioritize Individual Avoidance Trajectory Generators.
 Minimize Highest Risk, First, Then Lower Risks - Sequence.
ISIC-2003-29
Valasek, Ioerger, Painter
COURSE OUTLINE
SECTION I
• Introduction To The Smart Cockpit And Free Flight
• Free Flight Functionality
• Architecture: Aircraft & Simulator, Software & Hardware
SECTION II
• Artificial Intelligence In The Cockpit
• Intelligent Computing Techniques
• Software Agents: The Key To Smart Cockpit Software
SECTION III
• Real Time Flight Simulation: The Basics
• Real Time Flight Simulation For Free Flight
SECTION IV
• Selected Research Results
• Conclusion: Summing It Up
ISIC-2003-30
Valasek, Ioerger, Painter
Intelligent Computing Techniques
Software Engineering
 Object-oriented programming
Data communications: TCP/IP
 Modular design
 Message Passing
Intelligent Agents
 model complex decision-making/protocols
 simulate autonomous behavior
ISIC-2003-31
Valasek, Ioerger, Painter
Data Communications
TCP/IP enables modularization & scalability (>1 machine)
example: connect independent flight dynamics engines to
common server
typically 10MB/s
pass text strings (e.g. set/get state vars)
buffering, blocking
ISIC-2003-32
Valasek, Ioerger, Painter
128.135.194.2
128.135.194.3
SERVER
Create socket on port 5000
CLIENT
listen for connections
Open Socket connection to
128.135.194.2, port 5000
accept connection
write block of bytes
read block of bytes
write block of bytes
ISIC-2003-33
read block of bytes
Valasek, Ioerger, Painter
Two Flavors: TCP vs. UDP
specify type when create socket
TCP
 guaranteed delivery; messages arrive in order
 checked for errors
UDP
 non-guaranteed; not error-corrected
 much faster!
choice depends on tolerance of msg failures
 example: comm. protocol vs. graphics updates
ISIC-2003-34
Valasek, Ioerger, Painter
Inter-Module Communications
FLT SIM
PA
TCP/IP
TCP/IP
CMD
GEN
HUD
GEN
A/P
DSPLY
HNDLR
FCS
FLT
CTRLS
EQMO
DSPLY
HNDLR
FMI
COCKPIT DATA SIM
SIM
RADAR
SIM
ADS-B
NAV
MOD
WX
AGT
TFC
AGT
FLT
PLN
FMS
EXEC
DATA
OBJ
SIM
CPDL
FMS LOGIC
HDD
Left
PILOT
JPSN
DB
Dashed lines denote
virtual connectivity.
HDD
Right
HUD
DSPLY
HNDLR
SPiFI
ISIC-2003-35
Valasek, Ioerger, Painter
Other Communications Tech.
CORBA - industry standard
 www.omg.org/gettingstarted/corbafaq.html
 network/object-oriented, location-independence
 Interface Definition Language: classes, methods
 remote method invocation (lang/OS independent)
HLA - High Level Architecture
 www.dmso.mil/public/transition/hla
 standard for mil/gov simulations
 federations, broadcasts, time management
 objects, interactions
 SOM/FOM - interface definitions, methods
ISIC-2003-36
Valasek, Ioerger, Painter
XML
Data representation format
 for scenarios, msgs/cmds, config, logs, etc.
Define tags (like HTML)
<events>
<simevent id=“54321” time=“1:21”>
<type>request_takeoff_clearance</type>
<aircraft>US789<aircraft>
<location>KDEN</location>
</simevent>
<simevent>
...
</simevent>
</events>
ISIC-2003-37
Valasek, Ioerger, Painter
Generic parsers available
 Xerces - Java, C++
Apache: xml.apache.org/xerces2-j
IBM: www.alphaworks.ibm.com/tech/xml4j
 SAX => incremental, parse when needed
 DOM => batch, produce “object trees”
root Doc node
(list of child nodes...)
Element: type
text=takeoff
Element: simevent
attribute: time=1:21
attribute: id=54321
Element: location
text=KDEN
ISIC-2003-38
Element: simevent
attribute: time=1:35
attribute: id=54322
Element: aircraft
texst=US789
Valasek, Ioerger, Painter
What are Agents?
Essential Characteristics:
 Situated
can sense and take actions in dynamic environment
 Goal-oriented
 Autonomous
 Social/collaborative
 Adaptive
ISIC-2003-39
Valasek, Ioerger, Painter
Agent Architectures
Production Systems
 Reactive, trigger rules, CLIPS, SOAR
Search Algorithms: A* (WX agent)
Planning Algorithms
Hierarchical Task Networks (Retsina, TRL)
Decision Theoretic
 Markov Decision Processes, maximize payoff
Cognitive (Mentalistic)
 BDI: beliefs, desires, intentions
 JACK, PRS, dMARS
ISIC-2003-40
Valasek, Ioerger, Painter
Roles for Agents in Aviation
Simulate other aircraft, controllers
In cockpit: planning flight path, managing fuel,
maintaining stability of flight, monitoring traffic or
weather conflicts…
On ground (TRACON, ARTCC): planning trajectories,
resolving conflicts, approach metering, handling
emergencies, coordination with ground ops, airlines, etc.
ISIC-2003-41
Valasek, Ioerger, Painter
Collaboration Models
Teamwork




Hierarchical vs. distributed (command vs. consensus)
Key concepts: roles and responsibilities
Shared plans: implicit coordination, synchronization
Theoretical basis: Joint Intentions
Negotiation protocols
 Distributed Constraint Satisfaction
 Share justifications and beliefs to determine compromise
 Monotonic Concession Protocol
Auctions
 Bids based on marginal utility
 Contract networks
ISIC-2003-42
Valasek, Ioerger, Painter
Role of Simulated “Mental Attitudes”
Intent – transmit more than position/vector
 Desire to avoid weather, flight plan, will be turning north, descending due to
turbulence, reason for deviation…
Beliefs
 shared info (weather, congestion, aircraft emergencies)
 common picture of situation
 common knowledge: STAR’s, fixes, active runways, traffic patterns
 manage uncertainty
ISIC-2003-43
Valasek, Ioerger, Painter
Concepts for Development of MultiAgents for Free Flight
Strategic (trajectory planning/management) vs. Tactical
(avoidance maneuvers)
Actionable decisions:
 Alter flight path: heading, altitude, speed
Factors: weather, terrain, traffic
Constraints: fuel, speed/alt range
Preferences: time, fuel cost, comfort
ISIC-2003-44
Valasek, Ioerger, Painter
Negotiation
Utility function: Flight Plans => score
Negotiation by “argumentation”
 State what is wrong with proposed solution and why
 Communicate preferences as well as constraints
make up when behind schedule
minimize fuel consumption
maneuver limitations (safety, comfort)
Monotonic Concession Protocol (Rosenschein and Zlotkin)
 define a finite set of alternative trajectories
 each agent ranks trajectories by utility, proposes best
 take turns proposing next best deal till utilities match
ISIC-2003-45
Valasek, Ioerger, Painter
Negotiation Flowchart
no
weather,
terrain,
traffic info.
identify &
contact other
aircraft (bogey)
conflict
detection
generate
alternative
trajectories
receive counterproposal from bogey
propose next-best
solution for ownship
rank them
by utility
is utility of bogey’s
no
solution>=ownship’s
yes
are there
alternatives left?
conflict
detected?
yes
propose
highest-ranked
solution for
ownship
yes
exchange
candidate
solutions with
bogey
confirm
agreement on last
proposal
no
ask ATC
for help
inform ATC
ISIC-2003-46
Valasek, Ioerger, Painter
TRL Agents
“Task Representation Language”
Developed at Texas A&M Comp. Sci. Dept.
 contact: ioerger@cs.tamu.edu
knowledge bases
 declarative: rule base (“domain knowledge”)
 procedural: plans/methods for achieving goals
connection to simulator
 read state information
 trigger actions
agents can communicate with each other
ISIC-2003-47
Valasek, Ioerger, Painter
TRL Agent Architecture
TRL agent
TRL Task
Decomposition
Hierarchy
TRL KB:
tasks &
methods
APTE
Algorithm
Process
Nets
operators
messages
Simulation
ISIC-2003-48
assert,
query,
retract
results
sensing
JARE KB:
facts &
Horn-clauses
messages
Other
Agents
Valasek, Ioerger, Painter
Example Task Description
(task flight-plan-1 ()
(method
(sequence (takeoff KCLL 16)
(climb-out 3000)
(turn-heading 350)
(fly-direct-to KCNW)
(descend 500)
(land KCNW 17L))))
command to
simulator
invoke
sub-task
Things to add:
• interaction with ATC (set new way-points, altitudes...)
• handling developing weather (while (not cloudy)...)
ISIC-2003-49
Valasek, Ioerger, Painter
The SATS Airport Controller
How to simulate this with agents?
 encode the formal protocol as plans in TRL
AMM agent - simple task, 1st come-1st serve
pseudo-ADS-B=TCP/IP, test robustness of protocol w.r.t.
communications failures
test empirically with various scenarios
 arbitrary number of aircraft
 effects of timing, positions, speeds...
 test handoff from ATC, entry to SCA
 arrival/departure frequencies (Poisson distr.)
ISIC-2003-50
Valasek, Ioerger, Painter
COURSE OUTLINE
SECTION I
• Introduction To The Smart Cockpit And Free Flight
• Free Flight Functionality
• Architecture: Aircraft & Simulator, Software & Hardware
SECTION II
• Artificial Intelligence In The Cockpit
• Intelligent Computing Techniques
• Software Agents: The Key To Smart Cockpit Software
SECTION III
• Real Time Flight Simulation: The Basics
• Real Time Flight Simulation For Free Flight
SECTION IV
• Selected Research Results
• Conclusion: Summing It Up
ISIC-2003-51
Valasek, Ioerger, Painter
FLIGHT SIMULATION SYSTEM
real-time flight simulator
 Fixed-base: Commander 700; AV-8A Harrier, F-5A Freedom Fighter
 SGI Onyx Reality II sim engine
 Networked bank of PC’s
 Center stick; sidestick
 155o projected field of view
 30 Hz refresh rate
 Programmable Head Up Display
ISIC-2003-52
Valasek, Ioerger, Painter
FLIGHT SIMULATION SYSTEM
real-time flight simulator
 Head Down Displays (HDD)
 Reconfigurable
 CRT; touchscreen LCD
Moving Map
NAV Display
 Autopilot
 Glide slope capture
 Heading
 Altitude
 Pitch attitude
FMS & Autopilot Interface
Touch--Sensitive Screen
Gear
Handle
 Flight Management System (FMS)
 Jeppesen data base
 Pre-flight planning and enroute updating
 Moving map display
ISIC-2003-53
Valasek, Ioerger, Painter
HARDWARE ARCHITECTURE
real-time flight simulator
Bledsoe
Video
Signal
Output
Video
Signal
Output
Gannon
TCP/IP
Simulation and External Display
SGI Onyx Reality II
Aikman
UDP
GAPATS & Agent
System PC
Favre
SPiFI PC
Warner
ISIC-2003-54
Flutie
Valasek, Painter, Ioerger
SOFTWARE ARCHITECTURE
Multiple PCs - Windows
HEAD UP
DISPLAY
NAV
FLIGHT
INTERP
SGI Machine - Unix
HEAD DOWN
DISPLAY-L
EFS EXECUTIVE
PILOT
ADVISOR
DATA
OBJECT
TCPIP
GAPATS.CPP
(GAPATS.H)
ENVIRONMENT
AGENT
DATAOBJ
AGENT
EXECUTIVE
SPIFI
HEAD DOWN
DISPLAY-R
FLIGHT PLAN
AGENT
WEATHER
AGENT
TRAFFIC
AGENT
RADAR
DATA (SIM)
ADS-B
DATA (SIM)
ISIC-2003-55
Pilot
Flight
Controls
FCS
PROJECTED
3-SCREEN
DISPLAY
COMGEN (1)
COMGEN (2)
TRAG
ROLCAS
PITCAS
YAWCAS
EFS Dynamics
Valasek, Ioerger, Painter
SIMULATION SYSTEM SOFTWARE
COMPONENTS
Aircraft model
• Six-degree-of-freedom dynamics model
• CD & R model
• FMS model (including Autopilot model)
• Pilot model
• ADS-B model
• Additional research model
ADS-B
Flight Plan
Aircraft Trajectory
Flight
Objectives
Dynamics
Model
Pilot
FMS(Autopilot)/Navigation
CD & R Model
Other Research Models
Communication
Controller
ISIC-2003-56
Surveillance
Valasek, Ioerger, Painter
FLIGHT SIMULATION SYSTEM
 Combined GAPATS/Agents
system functions like a
simplified FMS
 Soft Pilot/FMS Interface
(SPiFI)
Pilot
Commands
SPiFI
Agent
System
Autopilot
Command
Generator
Engineering
Flight
Simulator
GAPATS
Tracker
ISIC-2003-57
Valasek, Ioerger, Painter
HEAD UP DISPLAY SYMBOLOGY
current
waypoint
marker
heading tape
rate of
climb
airspeed
altitude
glide slope
data
text area
runway
outline
pitch
angle
localizer
data
ISIC-2003-58
Valasek, Painter, Ioerger
HEAD UP DISPLAY IN FLIGHT
ISIC-2003-59
Valasek, Ioerger, Painter
HEAD UP DISPLAY IN FLIGHT
ISIC-2003-60
Valasek, Ioerger, Painter
HEAD UP DISPLAY IN FLIGHT
night and weather
ISIC-2003-61
Valasek, Ioerger, Painter
NAV/MAP DISPLAY SYMBOLOGY
ISIC-2003-62
Valasek, Painter, Ioerger
CONTROL EFFECTORS
commercial air transport
primary controls
secondary controls
outboard aileron
flight spoilers
leading edge flaps
ground spoilers
stabilator
upper and
lower rudders
throttle
inboard flap
inboard aileron
outboard flap
Boeing 777-300
ISIC-2003-63
Valasek, Ioerger, Painter
LIFT AND DRAG FORCES
definitions
Lift
XB

U1
Drag
ZB
Grumman F11F-1 Tiger
ISIC-2003-64
Valasek, Ioerger, Painter
AERODYNAMIC ANGLES
definitions
Sideslip Angle
Angle-of-Attack
VP
XB
XB
VP




XI
Note: All Angles Shown Are Positive
Grumman F11F-1 Tiger
ISIC-2003-65
Valasek, Ioerger, Painter
BODY AXIS
component definitions
Aerodynamic and Thrust Forces
Acceleration of Gravity
Aerodynamic and Thrust Moments
Linear and Angular Velocities
Note: Positive Signs Shown
Reference 2-1
ISIC-2003-66
Valasek, Ioerger, Painter
EULER ATTITUDE ANGLES
definition
Reference 2-2
ISIC-2003-67

yaw attitude angle

pitch attitude angle

roll attitude angle
Valasek, Ioerger, Painter
EQUATION SUMMARY
Linear Motion

m(U  VR  WQ )  mg x  FA x  FT x

m(V  UR  WP )  mg y  FA y  FT y

m(W  UQ  VP )  mg z  FA z  FT z
Drag
Equation
" Drag
Equation
"
" Sideforce
Equation
"
Sideforce
Equation
" LiftLift
Equation
"
Equation
Angular Motion
Assuming the x-z plane is plane of symmetry, i.e., I xy  I yz  0


I xx P  I xz R  I xz PQ  ( I zz  I yy ) RQ  L A  LT
Rolling Moment Equation

I yy Q  ( I xx  I zz ) PR  I xz ( P 2  R 2 )  M A  M T

Pitching Moment Equation

I zz R  I xz P  ( I yy  I xx ) PQ  I xz QR  N A  N T
ISIC-2003-68
Yawing Moment Equation
Valasek, Ioerger, Painter
FORCES AND MOMENTS
shorthand notation
 Dividing by mass, each term becomes a longitudinal linear or angular
acceleration.
 Letting
 moment  1 
   moment variable
 variable  m 
, dimensional derivatives are the linear
or angular acceleration per change in the associated motion variable.
 M  is the pitch angular acceleration imparted to the airplane as the result
of a unit change in angle-of-attack.

1
f X  X u u  X   X   X q q  X   E  X   F

m

1
f Z  Z u u  Z   Z   Z q q  Z   E  Z   F

m

1
m  M u u  M   M   M q q  M   E  M   F

I YY

E

F
E

F
E
F
 There will be separate equations for the aerodynamic and thrust forces
and moments.
ISIC-2003-69
Valasek, Ioerger, Painter
S & C DERIVATIVES
wind tunnel testing
 OPTIONS
 Static testing
 Dynamic testing
 DIFFICULTIES
 Tests are not dedicated to estimation of performance
 Reynold’s number effects on drag dependent terms
 Cost of testing
 Wind tunnel availability
ISIC-2003-70
Valasek, Ioerger, Painter
STABILITY DERIVATIVES
relative importance and prediction accuracy
Relative
Importance*
Estimated Prediction
Accuracy**
CL
α
10
± 5%
Cm
α
10
10
CD
α
5
10
4
40
7
40
CD
α
1
50
CL
u
8
20
Cm
8
20
6
CL
q
Cm
Derivative
CL
α
Cm
α
CD
CD
u
u
Cy
β
C
l
β
Cn
β
Cy

C
l

Cn

Relative
Importance*
Estimated Prediction
Accuracy**
7
±20%
10
20
10
15
2
60
2
60
4
60
4
50
10
15
20
Cy
p
C
lp
Cn
p
8
90
3
20
Cy
4
30
9
20
7
40
1
20
Cl
p
Cn
p
9
25
q
q
Derivative
p
Reference 2-1
* 10 = Major, 5 = Minor, 0 = Negligible
** Using theoretical methods. With use of tunnel data, better accuracy can be achieved
ISIC-2003-71
Valasek, Ioerger, Painter
EQU ATION S OF M OTION
observations
 The Equations of Motion are a set of 9 differential equations:




First order
Nonlinear
Coupled
Ordinary
 The Variables are:
 U, V, W, P, Q, R,  ,  and 
 The aerodynamic and thrust Forcing Functions:
 FAx , FT x , FA y , FT y , FAz , FTz , L A , LT , M A , M T , N A and N T
are functions of:
 Velocity
 Angle-of-attack
 Sideslip angle
 Time
 Altitude
 Configuration
ISIC-2003-72
Valasek, Ioerger, Painter
AIRCRAFT DYNAMIC MODES
standard longitudinal modes
 SHORT PERIOD
The primary and most useful standard longitudinal dynamic mode.






Second-order
Stable, or unstable
High frequency, well damped
Exhibited mostly in angle-of-attack and body-axis pitch rate
Specified in military flying qualities regulations
Pitch maneuverability is based upon controlling and shaping this mode
speed remains constant,
angle-of-attack and pitch attitude vary
Reference 3-1
ISIC-2003-73
Valasek, Ioerger, Painter
AIRCRAFT DYNAMIC MODES
standard longitudinal modes
 PHUGOID
The secondary standard longitudinal dynamic mode (nuisance mode).






Second-order
Stable, or unstable
Low frequency, very lightly damped
Exhibited mostly in velocity and pitch attitude angle
Specified in military flying qualities regulations
Name derived from the Greek word for fly: “phugos”
angle-of-attack remains constant,
speed and pitch attitude vary
Reference 3-1
ISIC-2003-74
Valasek, Ioerger, Painter
AIRCRAFT DYNAMIC MODES
example: Dutch roll mode
Reference 3-1
ISIC-2003-75
Valasek, Ioerger, Painter
AIRCRAFT DYNAMIC MODES
example: roll mode
Reference 3-1
ISIC-2003-76
Valasek, Ioerger, Painter
AIRCRAFT DYNAMIC MODES
example: spiral mode
Reference 3-1
ISIC-2003-77
Valasek, Ioerger, Painter
FLIGHT CONTROL PROBLEM
statement
The aircraft flight control design problem is to develop and
implement an algorithm that closes the loop between the
sensors and actuators, such that the aircraft accomplishes its
mission, as dictated in the requirements and mission
specification, with flying qualities deemed acceptable to the
pilot.
ISIC-2003-78
Valasek, Ioerger, Painter
FLIGHT CONTROLLER DESIGN
methodology
 REQUIREMENTS DEFINITION
 Based on intended mission and aircraft class
military: specified entirely by Department of Defense; usually non-negotiable
civilian: specified jointly between customer and manufacturer; negotiable
 Design iterations imply tradeoffs in requirements
how do requirements translate into assumptions?
 MODELING
 “Goodness”
origin
validity
assumptions
accuracy
suitability
– dynamic order
– state-space or frequency domain?
 Refinement
ISIC-2003-79
Valasek, Ioerger, Painter
CLASSES OF CONTROLLERS
Autopilots
Function: Provide pilot relief and special functions
pilot
commanded
motion
variables
command
shaping
+-
controller
compensator
Notes:
controls
vehicle
motion
vehicle
sensors
1. Pilot inputs are outer-loop variables such as airspeed, heading, altitude, etc.
2. Limited-authority system.
ISIC-2003-80
Valasek, Ioerger, Painter
TYPICAL SPECIFICATIONS
G.A. autopilot
 CONFIGURATIONS
 Clean:
 Climb:
 Power Approach:
gear and flaps up
gear up, flaps down
gear down, flaps down
 AIRSPEED COMMAND AND HOLD SYSTEM




Input type:
hold commanded airspeed:
maximum airspeed error:
Maximum values:
  T  70% of maximum
ramp
anywhere in the range 0 h 
10,000 feet
 2 KIAS
Rockwell Commander C700
ISIC-2003-81
Valasek, Ioerger, Painter
TYPICAL SPECIFICATIONS
G.A. autopilot
 PITCH ATTITUDE COMMAND AND HOLD






Input type:
Maximum positive commanded change in :
Maximum negative commanded change in  :
90% rise time on  :
Maximum overshoot on  :
Maximum values:
  e  5 degrees
ramp
10 degrees at SLS altitude
-8 degrees at SLS altitude
5 seconds
15%
 ALTITUDE COMMAND AND HOLD





Input type:
Commanded change in h:
Maximum climb rate:
Maximum error:
Maximum values:
  e  5 degrees
ramp
anywhere in the range 0  h  15,000 feet
Standard Instrument Climb of 500 feet/minute
50 feet of commanded altitude
ISIC-2003-82
Valasek, Ioerger, Painter
AUTOPILOTS
pitch attitude command and hold
Purpose:
Feedback Variable(s):
Typical Sensor(s):
Primary Control Variable:
Typical Compensators:
 ref
Maintain commanded pitch attitude
Pitch rate inner-loop and pitch attitude outer-loop
Rate gyro and vertical gyro
Elevator
Gain; and second-order pole-zero canceling compensator
 ec
e


K


( ) e
e
e
q
q
e
c
1
S

Kq
vertical
gyro
ISIC-2003-83
Valasek, Ioerger, Painter
AUTOPILOTS
pitch attitude command and hold
 Response to commanded 5 degree ramp in pitch attitude angle
Response to 5 deg theta command
 [deg]
6
4
2
0
0
2
4
6
8
10
0
2
4
6
8
10
0
2
4
Time (sec)
6
8
10
q [deg/sec]
4
2
0
-2
-4
e actual [deg]
4
2
0
-2
ISIC-2003-84
Valasek, Ioerger, Painter
AUTOPILOTS
altitude command and hold
Feedback Variable(s):
Typical Sensor(s):
Primary Control Variable:
Typical Compensators:
H
href

Altitude, Pitch Attitude Rate
Barometric Altimeter, Rate Gyro
Elevator
Gain, Lead-Lag, Proportional Derivative

ea
Kh
eh
lead-lag

e e

e
elevator
servo
eRG
aircraft
dynamics

h
K
PD
ISIC-2003-85
Valasek, Ioerger, Painter
AUTOPILOTS
altitude command and hold
PD = 10S+1
hdot (ft/s)
700
300
300
-100
-100
700
12
elevator (deg)
h (ft)
hcom (ft)
700
300
-100
0
25
50
75
time (sec)
100
ISIC-2003-86
0
-12
0
25
50
75
time (sec)
100
Valasek, Ioerger, Painter
IMPLEMENTATION
trim
Case 1: Operation about a Single Trim Condition
The following controller modes operate about a single trim condition:
Yaw Damper
Pitch Damper
Roll Damper
Wing Leveler
Heading Hold
Velocity Hold
Operational Procedure:
The pilot establishes the desired trim condition and engages the mode.
If he wishes to change the trim condition, he disengages the mode,
establishes the new trim condition, and re-engages the mode.
ISIC-2003-87
Valasek, Ioerger, Painter
IMPLEMENTATION
trim
Case 2 Automatic Transition between Two Trim Conditions
Flight controllers that require an automatic transition between two
trim conditions include:
Velocity Command
Rate of Climb Command
Glide Slope Capture and Tracking
Turn Rate Control
Operating Procedures:
The flight control system must be able to automatically handle trim changes
as well as changes in the vehicle’s open-loop dynamics.
ISIC-2003-88
Valasek, Ioerger, Painter
COURSE OUTLINE
SECTION I
• Introduction To The Smart Cockpit And Free Flight
• Free Flight Functionality
• Architecture: Aircraft & Simulator, Software & Hardware
SECTION II
• Artificial Intelligence In The Cockpit
• Intelligent Computing Techniques
• Software Agents: The Key To Smart Cockpit Software
SECTION III
• Real Time Flight Simulation: The Basics
• Real Time Flight Simulation For Free Flight
SECTION IV
• Selected Research Results
• Conclusion: Summing It Up
ISIC-2003-89
Valasek, Ioerger, Painter
AGENT BASED HIERARCHICAL SYSTEM
Executive Agent
Weather
Radar Data
Ground
Weather
Service
Other Traffic
Info...
Traffic
Agent
Weather
Agent
Other
Weather Info.
...
Flight
ADS-B
ATC
Radar
Plan
Info.
Overall Structure of Hierarchical Agent System
ISIC-2003-90
Valasek, Ioerger, Painter
WEATHER AGENT
objective : safest and shortest route
 OTHER CONSTRAINTS
 Minimum segment length
length of any segment in flight
path cannot be less than this
End point

 Maximum turning angle
turns of angles greater than this
are not allowed
 Minimum number of turns
L
Starting point
 METHOD USED
L = segment length
 Modified A* Search
 Regions with intensity greater
than 25 dBZ are set as inaccessible
ISIC-2003-91
 = turning angle
Valasek, Ioerger, Painter
TRAFFIC CD&R AGENT
 Inputs are ADS-B state vectors of aircraft in immediate airspace
Rh  ( x1  x2 ) 2  ( y1  y2 ) 2
Alert Zone
Rv  z1  z2
IF Rh <Rhp .AND. ABS(Rv)<Rvp
THEN avoidance=.TRUE.
ELSE avoidance =.FALSE.
 Detects potential traffic conflicts




Non-Cooperative agent
Protected Zone
Only considers aircraft in alert zone
Overlap of protected zones prohibited
Size of zones determined by several spatial or temporal factors
 Calculates evasion maneuvers to avoid other protected zones
 Knowledge based expert system and optimal control
ISIC-2003-92
Valasek, Ioerger, Painter
EXECUTIVE AGENT
 Arbitrator between lower-level agents
 Intelligent behavior
 Fuzzy synthesis
evaluation method
 Determines ultimate flight
guidance submitted to:
Weather
Radar Image
Weather
Agent
 FMS
 Autopilot
 Pilot
Flight Management System
Rule-Based
Arbitrator
Weather
Conflict
Evaluation
Traffic
Conflict
Evaluation
TC Path
Selection
Agents
Controller
Traffic
Situation
ISIC-2003-93
Traffic
Agent
Valasek, Ioerger, Painter
CONCLUSION
So, You Really Think …
That Software …
Can Fly …
An Airplane ?
Yeah, Right !!
ISIC-2003-94
Valasek, Ioerger, Painter
FLIGHT SIMULATION LAB
points of contact
 Director
John Valasek, Ph.D.
Aerospace Engineering Department
Texas A&M University
3141 TAMU
College Station, TX 77843-3141
Thomas R. Ioerger, Ph.D.
Computer Science Department
Texas A&M University
3112 TAMU
College Station, TX 77843-3112
(979) 845-1685
valasek@aero.tamu.edu
(979) 845-0161
ioerger@cs.tamu.edu
 FSL Web Page
 Web Page
 http://flutie.tamu.edu/~fsl
 http://faculty.cs.tamu.edu/ioerger/
ISIC-2003-95
Valasek, Ioerger, Painter