Providing Integrity for Satellite
Navigation: Lessons Learned (Thus
Far) from the Financial Collapse of
2008 – 2009
Sam Pullen
Stanford University
spullen@stanford.edu
ION GNSS 2009
Savannah, GA.
24 September 2009
Overview and Motivation
• My interest in this subject comes from my background in
Probabilistic Risk Assessment (PRA), which formed the
basis for my Ph.D. dissertation (Stanford, 1996).
– Optimal satellite design
– Optimal design of GPS integrity augmentations
• Studying and understanding failures of the past are the
key to improving risk assessment.
– Motivation for Hurricane Katrina presentation from ION GNSS 2008
• Since SatNav augmentations have been demonstrated to
be safe (with substantial margin), the benefit of this work
for SatNav is improving risk-assessment for future
systems and upgrades.
– Find means to reduce margin against “unknown unknowns”
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
2
Origins: The Changing Debt Market
Yield (%) on 10-Year Treasury Bonds: 1964 - 2009
Low Treasury yields created
demand for higher-yielding
investments.
Yield (%)
Year
Source: http://finance.yahoo.com
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
3
A Simplified Picture of the Financial Market
Collapse
“Wine Glass Pyramid” Overview of Collateralized Debt Market
overflows
when full
mortgage
payments
can further insure via
credit default swaps
1st Tranche (AAA, 3%)
2nd Tranche (AA, 5%)
3rd Tranche (BBB, 7%)
4th Tranche
(Unrated, 10%)
Source: Paddy Hirsch, “Crisis Explainer: Uncorking CDOs,” http://soundlearning.
publicradio.org/subjects/economics_finance/financial_crisis/uncorking_cdos.shtml
Also see: Jonathan Jarvis, “The Crisis of Credit Visualized,” http://www.vimeo.com/3261363
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
4
The Efficient Market Hypothesis (EMH)
• Origins (“random walk”) go back to 1900, but
codified and popularized by the Eugene Fama
(“Chicago school” of economists, 1960s).
• Expresses the concept that today’s prices reflect all
available information, properly (“rationally”) judged.
• “The (market) price is right” – the foundation of
quantitative economics
– Traditional linear analysis with Markov state transitions
– Gaussian (or log-Normal) market-state transition
probabilities are assumed (definition of random walk).
• Despite limited supporting evidence, EMH became
widely accepted (and exploited) because of its
academic and mathematical elegance/convenience.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
5
Non-Gaussian Stock Market Behavior
P DeGrauwe, et al, “How Abnormal was the Stock Market in October 2008?”
Euro Intelligence, 11 Nov. 2008. http://www.eurointelligence.com/article.581+M5f21b8d26a3.0.html
schg = 1.032%
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
6
An Honest Explanation from a Leading
Master of Quantitative Finance
Paul Wilmott, Frequently Asked Questions in Quantitative
Finance (Wiley, 2007), pp. 33 – 35:
In finance we often assume that equity returns are normally distributed.
… We find ourselves using the normal distribution quite naturally for
many financial processes.
As often with mathematical ‘laws’ there is the ‘legal’ small print, in this
case the conditions under which the Central Limit Theorem applies. … Of
course, financial data may not satisfy all of these, or indeed, any. In
particular, it turns out that if you try to fit equity returns data with nonnormal distributions you often find that the best distribution is one that
has infinite variance. Not only does it complicate the nice mathematics of
normal distributions and the Central Limit Theorem, it also results in
infinite volatility. This is appealing to those who want to produce the best
models of financial reality but does rather spoil many decades of financial
theory and practice based on volatility as a measure of risk for example.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
7
One Disastrous Outcome:
The Copula Model for Credit Risk Correlation
See Felix Salmon in Wired Magazine (March 2009)
• Before 2000, debt markets were much more conservative
due to the complexity of modeling default risk and the
small data base of major loan defaults.
• David Li of JP Morgan RiskMetrics group “removed” this
difficulty by assuming a Gaussian copula formulation with
a single correlation parameter g derived from comparative
market prices.
– Justified by EMH since market prices confer “best” knowledge
• This approach led to dramatic growth in the creditderivatives market until its fatal flaws were revealed by
housing market crash of 2007-08.
– Nationwide (correlated) loan defaults were not captured by model.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
8
Wilmott Actually Predicted this Disaster
Years Beforehand
Paul Wilmott, “The Use, Misuse, and Abuse of Mathematics in Finance,” Philosph
Trans: Math, Phys and Eng Sci, Vol. 358, No. 1765 (Jan. 2000), pp. 63-73.
Abstract (conclusion)
Unfortunately, as the mathematics of finance reaches
higher levels so the level of common sense seems to
drop. There have been some well-publicized cases of
large losses sustained by companies because of their
lack of understanding of financial instruments. In this
article we look at the history of financial modelling, the
current state of the subject and possible future
directions. It is clear that a major rethink is
desperately required if the world is to avoid a
mathematician-led market meltdown.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
9
The Financial Modeler’s Manifesto
(in response to the crisis)
Excerpted from Derman and Wilmott, The Financial Modelers’ Manifesto (Jan. 2009):
http://www.wilmott.com/blogs/paul/index.cfm/2009/1/8/Financial-Modelers-Manifesto
Our experience in the financial arena has taught us to be very humble in applying mathematics to
markets, and to be extremely wary of ambitious theories, which are in the end trying to model
human behavior. We like simplicity, but we like to remember that it is our models that are simple,
not the world … The greatest danger is the age-old sin of idolatry. Financial markets are alive but a
model, however beautiful, is an artifice. No matter how hard you try, you will not be able to breathe
life into it. To confuse the model with the world is to embrace a future disaster driven by the belief
that humans obey mathematical rules.
MODELERS OF ALL MARKETS, UNITE! You have nothing to lose but your illusions.
The Modelers' Hippocratic Oath
~ I will remember that I didn't make the world, and it doesn't satisfy my equations.
~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
~ I will never sacrifice reality for elegance without explaining why I have done so.
~ Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make
explicit its assumptions and oversights.
~ I understand that my work may have enormous effects on society and the economy, many of them
beyond my comprehension.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
10
Key Lessons Applicable to Engineering
Risk Modeling
1. Precise modeling of the unknown is not possible.
– Therefore, probabilistic models built upon uncertainty are
preferred to deterministic ones.
2. Simplified risk models may be justified for specific
threats, but model limitations must be given as
much weight as the results.
– Study assumptions carefully and “carry them forward”
with the results so that they are not “lost in time”.
– Avoid “falling in love” with models – keep a critical mind.
3. In particular, avoid extrapolating from a flawed
model into the realm of absurdity.
– Initial errors may be tolerable until exploited too
aggressively by a follow-up model.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
11
Specific Lessons for SatNav Integrity
As with other engineering risk analyses, SatNav
integrity shares features with financial models:
1. Deterministic equations: cause-and-effect behavior is
assumed known; uncertainty introduced by random
variables with known distributions.
»
The SatNav solution – worst-case modeling: where
significant uncertainty cannot be removed, “worst-case”
simplifications are derived to bound unknown reality.
2. Gaussian distribution: most random perturbations
are modeled as Gaussian (or Gaussian variations)
»
24 September 2009
The SatNav solution: theory and data are combined to
determine “inflation” factors such that the unknown “true”
distribution is bounded at sufficiently low probabilities.
Lessons Learned from 2008-09 Financial Crisis
12
Summary
• The financial crisis illustrates the perils of risk assess-ment based upon hubris and over-simplified models.
• This experience provides many useful lessons:
– Probabilistic models are better when uncertainty is large.
– When using deterministic models:
» Emphasize assumptions when presenting results.
» Avoid over-extrapolating from results.
– Remain open to new threats and threat model changes.
• SatNav integrity models are deterministic but apply
multiple levels of caution against uncertainty.
– Care applied to insure proper use of Gaussian distribution
– “Worst-case” error mitigation theoretically bounds all
remaining uncertainty but often impacts user performance.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
13
Backup Slides follow…
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
14
My Background in Risk Assessment
• Research on Probabilistic Risk Analysis (PRA) formed the
core of my Ph.D. dissertation (1996).
• PRA in my thesis was applied to optimal design of
satellites and GPS integrity augmentations (RAIM, WAAS).
• Since then, my work has focused on optimal design and
verification of GPS augmentation systems with predesigned (and highly-constrained) architectures.
– Focus on GBAS (LAAS) ionosphere and ephemeris threats
– Focus on optimal diagnosis and isolation of detected faults
• Risk analysis failures outside of GNSS reinforce basic
PRA principles and provide important lessons:
– Hurricane Katrina (ION GNSS 2008)
– Recent financial crisis (this paper …)
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
15
Opening Thoughts
"The experience of being proved
disastrously wrong is salutary.
No economist should be denied it, and
none are."
- John Kenneth Galbraith (early 1980s)
Also by Galbraith:
"The only function of economic forecasting is
to make astrology look respectable."
Sources: J. A. Smith, The Idea Brokers: Think Tanks and the Rise of the New Policy Elite (1993).
http://www.fool.co.uk/news/comment/2006/c060502g.htm
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
16
Overview
• Serious flaws in mathematical modeling directly
contributed to the financial collapse of 2008 - 2009.
• The caution and consideration applied to integrity
assurance for satellite navigation stands in sharp
contrast to the hubris of the financial community.
• However, elements of the faulty financial models cited
above exist in most traditional forms of risk analysis.
• This briefing examines what lessons, if any, can be
learned that are relevant to risk assessment in
general and SatNav integrity analysis in particular.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
17
A Serious Outcome:
Value-at-Risk (VaR) Modeling
See Joe Nocera in NYT Magazine (4 January 2009)
• Devised by JP Morgan in the 1990’s to provide
standardized trade and company-wide risk modeling
– uses traditional linear-Gaussian statistics
• Key selling point: a single “Value at Risk” output
– Represents a lower confidence bound, for a given percentile
(95th or 99th) and duration, on the amount that could be lost.
– VaR results available to managers in near-real time.
• Ease of use and simplicity of results led to massive
over-dependence and abuse.
– Uncertainty parameters set using limited historical data.
– Used as basis for capital requirements – led to insufficient
reserves when financial crisis hit.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
18
“Misleading Information” Error Criterion
0
10
Gaussian
dist. model
-2
10
Probability Density
Vertical
Protection
Level
Nominal
Error plus 4s
Bias Fault
VPL
VAL
(for this
fault state)
(for this
flight
operation)
-4
10
Vertical
Alert Limit
-6
10
-8
10
PFFMD
-10
10
-6
-4
-2
0
2
Kfault
4
6
8
10
12
14
16
KFFMD
Normalized Vertical Position Error (no. of sigmas)
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
19
Model Definition vs. Reality
In practice, maximum error bounds (VPL, VAL) are very conservative
 actual “hazard level” at or just above VAL is low (if not zero).
 VAL + d is not materially more hazardous than VAL – d.
Typical Model
Level of
Hazard
“Real-World”
Model
VAL
24 September 2009
Error Size (meters)
Lessons Learned from 2008-09 Financial Crisis
20
Resulting Region of “Most Threatening”
Error (Snapshot View)
Range Error = MERR
(range-domain bound)
Error giving
max. hazard
probability is
much lower
than MERR!
= Er / MERR (normalized range error)
Source: T. Zaugg, Proc. ION 58th Annual Meeting, June 2002.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
21
Dealing with the Unknown in GBAS:
Ionosphere Anomaly Threat Modeling
Based upon most severe anomalies observed in CONUS since 1999
450
Worst-case
gradients along
this upper bound.
400
Slope [mm/km]
350
300
250
200
L1-L2 and L1 CMC
150
L1 CMC
L1 CMC (low-elev)
100
50
0
0
10
20
30
40
50
60
70
80
90
Elevation [deg]
Boundaries of resulting “threat model” for LAAS in CONUS
Source: S. Datta-Barua, J. Lee, et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing
Systems,” Submitted to AIAA J. of Aircraft (August 2009, under review).
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
22
Dealing with the Unknown in GBAS:
Ionosphere Anomaly Threat Modeling
Based upon most severe anomalies observed in CONUS since 1999
450
high elevation
400
low elevation
350
Slope [mm/km]
Worst-case
gradients along
this upper bound.
300
250
L1-L2 and L1 CMC
200
L1 CMC
150
L1 CMC (low-elev)
100
50
0
0
100
200
300
400
500
600
700
800
Ground Speed [m/s]
Boundaries of resulting “threat model” for LAAS in CONUS
Source: S. Datta-Barua, J. Lee, et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing
Systems,” Submitted to AIAA J. of Aircraft (August 2009, under review).
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
23
Iono. Anomaly “Wedge” Model Geometry
Simplified Ionosphere Front Model:
a linear ramp defined by constant speed,
slope, and width
Front Speed
200 m/s
Front Slope
425 mm/km
Airplane Speed
~ 70 m/s
(synthetic baseline due
to smoothing ~ 14 km)
24 September 2009
LGF IPP Speed
200 m/s
Front Width
25 km
Max. ~ 6
km at DH
GBAS Ground Station
Lessons Learned from 2008-09 Financial Crisis
24
Impact of Ionosphere Anomaly Model and
Worst-Case Error
RTCA-24 Constellation; All-in-view, all 1-SV-out, and all 2-SV-out subsets
included; 2 satellites impacted simultaneously by ionosphere anomaly
0.14
Most errors (~ 75%) are exactly zero due to
detection/exclusion, but all zero errors have
been removed from the histogram.
Parameter
0.12
0.1
Vast majority of nonzero errors are well
below tolerable limit.
PDF
0.08
0.06
0.04
0.02
0
24 September 2009
Inflation
(“geometry
screening”)
added to remove
28.8-meter
geometries
with
tolerable limit
unsafe
(CAT
I PA)errors,
but many good
geometries are
removed as well
Worst-case
 significant
error, or
“MIEV”,loss.
is 
availability
41 m
0
5
10
15
20
25
30
35
User Vertical Position Error (meters)
Lessons Learned from 2008-09 Financial Crisis
40
45
25
Features of Worst-Case Mitigation
• Theoretically, mitigating worst-case errors covers all
threat scenarios and removes unquantifiable risk due
to “unknown unknowns.”
• Worst-case mitigation almost always protects integrity
with substantial margin (as in iono. anomaly example).
• Key limitation: “worst case” model remains dependent
on un-provable assumptions.
– Ongoing vigilance needed to monitor validity of key assumptions.
• On the “cost” side, the difficulty of mitigating worstcase scenarios stresses the resulting system:
– User benefits may be significantly degraded.
– Loss of availability may have unforeseen safety implications.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
26
How Much do Modeling Issues Matter?
• Unlike absurdities in financial modeling, assumptions
made in GNSS risk modeling are “hubris-free” and are
almost always conservative.
• Furthermore, in theory, focus on and mitigation of
worst-case anomalies also covers all other threats.
• Unfortunately, “worst-case” anomalies are, by
definition, difficult to counter and require extensive
hardware/software/personnel resources.
• As a result, risk mitigation may become mis-focused,
and sources of risk that do not easily fit the above
models may get “assumed away”.
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
27
Similarities to Hurricane Katrina Lessons
• A key lesson from Katrina is the importance of
maintaining flexibility and adaptability in riskmitigation systems.
– New Orleans hurricane “threat model” did not change as
better information became available.
– Lengthy political battles prevented improvements to hurricane
defense system to address worsening threat understanding
and flaws in levees discovered over time.
• The most obvious similarity here is the consistent
refusal of mainstream financial economics to consider
the obvious violations of EMH and their implications.
– Even today, anecdotal evidence suggests that mainstream
financial economists are mostly “sticking with their story …”
24 September 2009
Lessons Learned from 2008-09 Financial Crisis
28