Real-Time Signalextraction
(MDFA) and Algorithmic Trading
marc.wildi@zhaw.ch
http://blog.zhaw.ch/idp/sefblog
http://www.idp.zhaw.ch/usri
http://www.idp.zhaw.ch/MDFA-XT
http://www.idp.zhaw.ch/sef
Background
• Hybrid math/econ.
• IDP-ZHAW → Projects with econ. partners
• Forecasting
– Health-care (cost expenditures)
– Macro (real-time economic indicators: EURI Eurostatproject)
– Finance (MDFA-XT, large hedge-fund)
– Engineering (Telecom, load forecasts)
• Eclectic/disparate range of applications
• Common methodological approach(es)
– In-house developments: (M)DFA
– R-package “signalextraction” on CRAN
A Classical Algorithmic
Trading Approach
Timing System
SP500 Daily Closures
MA(200), Equally Weighted
P.5 (drawdowns), p.7 (timing
system), p.10 (performance)
Problem: (Too) Long Periods with
Systematic Underperformance
Why do Traders Frequently
Adopt/Prefer Filter Crossings?
Filter Characteristics
Why MDFA?
http://blog.zhaw.ch/idp/sefblog/index.php?/archives/54Intermezzo-Why-do-Traders-Often-Consider-Crossings-ofTrading-Filter-Pairs.html
Log-MSCI and MA(45)
Filter Characteristics
1 44
1 44
MA(45):
log( MSCI t  k )  ( )=  exp(ik )

45 k 0
45 k 0
1 44
Id-MA(45)  1-( ) = 1-  exp(ik )
45 k 0
• Amplitude function:
– Which signal is extracted?
• Time-shift:
– How large is the delay?
1 44
|1-()| = 1-  exp(ik)
45 k 0
 1 44

Arg 1-  exp(ik ) 
Arg(1-( ))
 45 k 0

=


Timing System (MSCI-Weekly)
|1-( )| = 1-
1 199
 exp(ik)
200 k 0
|1-( )| = 1-
1 199
 exp(ik)
200 k 0
Arg(1-( ))

|1-( )|
|1-( )| = 1-
1 199
 exp(ik)
200 k 0
More General Crossings:
MA(45,black)-MA(22,red)=crossing
(blue)
Conclusions
• Crossing-rules are (an unnecessarily
cumbersome way of implementing) bandpass
filters
• Crossing-rules (bandpass) have small time
delays
• Why MDFA?
– Flexible efficient real-time (bandpass) design
– Fast and smooth
Fundamental Trading
http://www.idp.zhaw.ch/usri
SP500
http://blog.zhaw.ch/idp/sefblog
USRI (MDFA) and SP500
Performance in Logs
Student Thesis p.19 Long Term
Performances Fundam. Trading
Conclusion
• Damp or avoid all massive recession
draw-downs effectively
– Ideal for risk-averse investors (pension funds)
• Fundamental Trading: truly out of sample
– Focus on Macro-data (finance data ignored)
– NBER
• Disadvantage: `insufficiently active’
– Texto: «Difficult to justify fees»
MDFA-XT
http://www.idp.zhaw.ch/MDFA-XT
MSCI (+BRIC)
http://blog.zhaw.ch/idp/sefblog
Log-MSCI and MA(45)
MDFA vs. MA(45) weekly data
MDFA (blue) Faster
Five Trading Filters
Different Trading Frequencies
Filter « Unfrequent »
Filter « Unfrequent to Mid»
Filter « Mid »
Filter « Frequent »
Conclusion
• Higher trading frequencies are associated
with
– Bandpass shifted to the right
• More flexible than traditional filter-crossings
– Smaller delays/time shifts
Performances
Setting
• Total degenerative trading costs of 0.3%
per order (small fund)
• Long only
• No risk free interest rates
Performance « Unfrequent »
Performance « Unfrequent to Mid»
Performance « Mid»
Performance « Mid to Frequent »
Performance « Frequent »
Conclusions
• Higher trading frequencies are associated
with
– Slight reduction of performance
– Larger draw-downs
• USRI would avoid draw-downs and then the
performance would improve
– Increased market activity (fees!)
• Combination with USRI possible
(recommended)
• Filters will be available on-line in late July
Real-Time Signalextraction
A SEF-Blog Excel-Tutorial
http://blog.zhaw.ch/idp/sefblog
Excel-Tutorial on SEF-Blog
• http://blog.zhaw.ch/idp/sefblog/index.php?/
archives/65-Real-Time-Detection-ofTurning-Points-a-Tutorial-Part-I-MeanSquare-Error-Norm.html
• http://blog.zhaw.ch/idp/sefblog/index.php?/
archives/67-Real-Time-Detection-ofTurning-Points-a-Tutorial-Part-IIEmphasizing-Turning-Points.html
Purposes
• Yoga exercises to detach from main-stream
maximum likelihood world
• First Blog-entry: how traditional econometric
approach `works’
– Intuitively straightforward
– Good (optimal) mean-square performances
– People have become lazy-minded
• Second Blog-Entry: the early detection of turning
points
– Is a (strongly) counterintuitive exercise
– Generates seemingly (strongly) misspecified filter
designs
• Warning → Learning (→ Illumination?)
Excel-Tutorial on SEF-Blog
AR(1)
pole: argument
pole: magnitude
0
0.9
Normalization
AR(1)
0.1
0.9
AR(2)
pole 1: argument
pole 1: magnitude
pole 2: argument
pole 2: magnitude
Normalization
AR(1)
AR(2)
ARMA(2,2)
pole 1: argument
pole 1: magnitude
pole 2: argument
pole 2: magnitude
Normalization
AR(1)
AR(2)
0.26179939
0.8
-0.26179939
0.8
0.1
1.54548132
-0.64
4.5
0.31415927
0.99
-0.31415927
0.99
0.06
1.8830919
-0.9801
zero 1:
zero 1:
zero 2:
zero 2:
argument 0.31415927
magnitude
0.4
argument -0.31415927
magnitude
0.4
MA(1)
MA(2)
0.76084521
-0.16
10
4
8
3.5
Amplitude function
AR(1)
6
2.5
Amplitude function
AR(2)
4
2
Amplitude function
ARMA(2,2)
2
3
1.5
Time-shift AR(1)
Time-shift AR(2)
Time-shift ARMA(2,2)
Frequency-cycle
Frequency-cycle
1
2.99
2.81
2.64
2.46
2.29
2.11
1.94
1.76
1.58
1.41
1.23
0.7
1.06
0.88
0.53
-2
0.35
0
0.5
0.18
0
2.7
2.94
2.45
2.21
1.96
1.72
1.47
1.23
0.98
0.74
0.49
0
0.25
0
-4
4
1.5
3
1
Peak Correlation AR(1)
2
1
Filtered time series ARMA(2,2)
Peak Correlation AR(2)
0.5
0
Peak Correlation
ARMA(2,2))
0
-2 -1 0
-0.5
1
2
3
4
5
6
7
8
9 10
Peak Correlation ModelBased ARMA(2,2) (Rpackage)
1
-1
-2
-3
-4
-1
30 59 88 117 146 175 204 233 262 291 320 349 378
Model-based MMSE Real-time
estimate from R
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Frequencies
0
0.00628319
0.01256637
0.01884956
0.02513274
0.03141593
0.03769911
0.0439823
0.05026548
0.05654867
0.06283185
0.06911504
0.07539822
0.08168141
0.08796459
0.09424778
0.10053096
0.10681415
0.11309734
0.11938052
0.12566371
0.13194689
0.13823008
0.14451326
0.15079645
0.15707963
0.16336282
0.169646
0.17592919
0.18221237
0.18849556
0.19477874
0.20106193
0.20734512
0.2136283
0.21991149
0.22619467
0.23247786
0.23876104
0.24504423
0.25132741
0.2576106
0.26389378
0.27017697
0.27646015
0.28274334
0.28902652
0.29530971
0.30159289
0.30787608
0.31415927
0.32044245
0.32672564
0.33300882
0.33929201
0.34557519
0.35185838
0.35814156
0.36442475
0.37070793
0.37699112
0.3832743
0.38955749
0.39584067
0.40212386
0.40840704
0.41469023
0.42097342
0.4272566
exp(-iomega)
1
0.999980260856137+6.28314396555895E-003i
0.999921044203816+1.25660398833526E-002i
0.999822352380809+1.88484397154082E-002i
0.9996841892833+2.51300954433374E-002i
0.999506560365732+3.14107590781283E-002i
0.999289472640589+3.76901826699345E-002i
0.999032934678125+4.39681183178649E-002i
0.998736956606017+5.02443181797696E-002i
0.998401550108975+5.65185344820245E-002i
0.998026728428272+6.27905195293134E-002i
0.997612506361225+6.90600257144057E-002i
0.997158900260614+7.53268055279327E-002i
0.99666592803403+8.15906115681575E-002i
0.996133609143173+8.78511965507432E-002i
0.99556196460308+9.41083133185143E-002i
0.9949510169813+0.100361714851215i
0.994300790396999+0.10661115427526i
0.993611310520008+0.112856384873482i
0.992882604569814+0.11909716009487i
0.992114701314478+0.125333233564305i
0.991307631069507+0.131564359092282i
0.990461425696651+0.137790290684638i
0.989576118602651+0.144010782552252i
0.988651744737914+0.150225589120757i
0.987688340595138+0.156434465040231i
0.986685944207868+0.162637165194883i
0.985644595148998+0.168833444712734i
0.984564334529205+0.175023058975276i
0.98344520499533+0.181205763627137i
0.982287250728689+0.187381314585725i
0.981090517443334+0.19354946805086i
0.979855052384247+0.199709980514407i
0.978580904325472+0.205862608769881i
0.977268123568193+0.212007109922055i
0.975916761938747+0.218143241396543i
0.974526872786577+0.224270760949381i
0.973098510982127+0.230389426676591i
0.971631732914674+0.236498997023724i
0.970126596490106+0.242599230795408i
0.968583161128631+0.248689887164854i
0.967001487762435+0.254770725683382i
0.965381638833274+0.260841506289897i
0.96372367829001+0.266901989320375i
0.962027671586086+0.272951935517325i
0.960293685676943+0.278991106039229i
0.958521789017376+0.285019262469976i
0.95671205155883+0.291036166828272i
0.954864544746643+0.297041581577035i
0.952979341517219+0.303035269632774i
0.951056516295154+0.309016994374947i
0.949096144990295+0.314986519655305i
0.947098304994744+0.32094360980721i
0.945063075179805+0.326888029654942i
0.942990535892864+0.332819544522987i
0.940880768954226+0.338737920245291i
0.938733857653874+0.344642923174517i
0.936549886748193+0.350534320191259i
0.934328942456612+0.356411878713251i
0.932071112458211+0.362275366704546i
0.929776485888251+0.368124552684678i
0.927445153334661+0.373959205737801i
0.925077206834458+0.379779095521801i
0.922672739870115+0.385583992277397i
0.92023184736587+0.391373666837203i
0.917754625683981+0.397147890634781i
0.915241172620917+0.402906435713663i
0.912691587403503+0.408649074736349i
0.910105970684996+0.414375580993284i
Real-Time Signalextraction
1. Traditional Econometrics
Task: Extract the Cycle
Signal: Ct 

  k X t k
k 

T 1
ˆ  ˆ X
Estimate: C
 k t k
T
k 0
Standard Econometric Approach
• Proceeding:
– Identify a time-series model (ARIMA/state
space)
– Extend the series by optimal forecasts
– Apply the symmetric filter on the extended
time series
• X-12-ARIMA, TRAMO, STAMP, R/S+…
• Claim:
– One-sided filter is optimal (mean-square
sense)
– Assumption: DGP/true model
ARMA(2,2)-Diagnostics
Real-Time Model-Based Filter
Real-Time Signalextraction
2. Excel Example
(Replication of Model-Based
Approach)
Parameters (ARMA(2,2)-FILTER)
• ARMA(2,2)-Filter (not model)
ARMA(2,2)
pole 1: argument 0.26179939 zero 1: argument
0.26179939
pole 1: magnitude
0.99 zero 1: magnitude
0.9
pole 2: argument -0.26179939 zero 2: argument -0.26179939
pole 2: magnitude
0.99 zero 2: magnitude
0.9
Normalization
0.11
AR(1)
1.91253314 MA(1)
1.73866649
AR(2)
-0.9801 MA(2)
-0.81
A Seemingly Virtuous Design
(amplitude)
1.2
1
0.8
Amplitude function
ARMA(2,2)
0.6
Frequency-cycle
0.4
0.2
3.02
2.76
2.51
2.26
2.01
1.76
1.51
1.26
1.01
0.75
0.5
0.25
0
0
A Seemingly Virtuous Design (time
shift)
5
4
3
2
-2
-3
-4
3.1
2.94
2.78
2.61
2.45
2.29
2.12
1.96
1.8
1.63
1.47
1.31
1.14
0.98
0.82
0.65
-1
0.49
Frequency-cycle
0.33
0
0.16
Time-shift ARMA(2,2)
0
1
A Seemingly Virtuous Design
(Peak Correlation)
• Correlation between real-time estimate
and cycle as a function of time-lag k
1.2
1
0.8
0.6
0.4
Peak Correlation
ARMA(2,2))
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
-2
-1
0
1
2
3
4
5
6
7
8
9
10
1
T

 (k )  
1
 T
T
 Cˆ C
j  k 1
T

j |k |1
j
j k
,k  0
Cˆ j |k |C j , k  0
Signal and Estimate
(Estimate: Filter Tweaked by Hand)
4
3
2
1
Filtered time series
ARMA(2,2)
0
1
-1
-2
-3
-4
27 53 79 105 131 157 183 209 235 261 287 313 339 365 391
Cycle
Real-Time Signalextraction
3. Excel Example (Turning Point
Revelation)
Parameters ARMA(2,2)-FILTER
Seemingly Misspecified Design
• ARMA(2,2)-Filter (not model)
ARMA(2,2)
pole 1: argument 0.31415927 zero 1: argument
pole 1: magnitude
0.99 zero 1: magnitude
pole 2: argument -0.31415927 zero 2: argument
pole 2: magnitude
0.99 zero 2: magnitude
Normalization
0.06
AR(1)
1.8830919 MA(1)
AR(2)
-0.9801 MA(2)
0.31415927
0.4
-0.31415927
0.4
0.76084521
-0.16
A Seemingly Misspecified Design
Amplitude
4.5
4
3.5
3
Amplitude function
ARMA(2,2)
2.5
2
Frequency-cycle
1.5
1
0.5
3.02
2.76
2.51
2.26
2.01
1.76
1.51
1.26
1.01
0.75
0.5
0.25
0
0
A Seemingly Misspecified Design
Time Shift
7
6
5
4
3
Time-shift ARMA(2,2)
2
Frequency-cycle
1
-2
-3
3.1
2.94
2.78
2.61
2.45
2.29
2.12
1.96
1.8
1.63
1.47
1.31
1.14
0.98
0.82
0.65
0.49
0.33
-1
0.16
0
0
A Seemingly Misspecified Design
Peak-Correlations
1.5
1
Peak Correlation
ARMA(2,2))
0.5
0
-2 -1
-0.5
-1
0
1
2
3
4
5
6
7
8
9
10
Peak Correlation
Model-Based
ARMA(2,2) (Rpackage)
A Seemingly Misspecified Design
Filtered Series and Signal
4
3
2
1
Filtered time series ARMA(2,2)
0
1
-1
-2
-3
-4
26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 401
Cycle
Comparison: Seemingly Virtuous
vs. Seemingly Misspecified
4
3
2
1
Misspecified ARMA(2,2)
0
1
-1
-2
-3
-4
32 63 94 125 156 187 218 249 280 311 342 373
Model-based ARMA(2,2)
Comparison: Seemingly Virtuous
vs. Seemingly Misspecified
3
2
1
Misspecified ARMA(2,2)
0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52
-1
-2
-3
Model-based ARMA(2,2)
Conclusions
• Seemingly misspecified design is
– Faster
– Smoother (less false TP’s or “alarms”)
– Not mean-square optimal
– Much better in a TP-perspective
From Excel to MDFA
• Tweak filter parameters `by hand’ in Excel
Tutorial
• Shortcomings of example
– Unrealistically simple artificial simulation
exercise
– In practice:
• more complex nuisances and/or signals
– Include information from more than one time
series (multivariate framework)
• Wish: a formal optimization criterion
• Welcome to DFA and MDFA
DFA
Direct Filter Approach
Mean-Square
DFA: Direct Filter Approach
E[(Yt  Y t ) ]  min( filter _ parameters)
2
• Idea: estimate mean-square filter error
efficiently
T
1
E[(Yt  Y t )2 ]   (Yt  Y t ) 2  eT
T t 1
2

T
(T 1)/2

k  (T 1)/2
| (k )  (k ) | ITX (k )  eT '
2
Optimization Criterion (I(0))
2
T
(T 1) / 2

k  (T 1) / 2
| (k )  (k ) | ITX (k )  Min
2
• Minimize a (uniformly) superconsistent
estimate of an (uniformly) efficient
estimate of the filter mean-square error
• (Customized) Efficiency enters explicitly in
the Design of the Optimization Criterion
Did You Say and/or Mean
“Periodogram”?
• Periodogram is a typical example of “statisticbashing”
– Inconsistent estimate of spectral density
– Smoothing (parametric or non-parametric)
• Periodogram has wonderful statistical properties
– Sufficiency (Larry Brethorst)
– One can derive nice formal efficiency results in realtime signalextraction
• Working on a series of new Blog entries about
the topic to rehabilitate – to some extent … - the
periodogram
Performances (Efficiency of
Univariate DFA)
• Business Survey Data (KOF, FED,
2004,2005)
– X-12-ARIMA, Tramo/Seats
– MSE-gain ~30%
• US- and Euro-GDP (2008):
– CF
– turning-points anticipated by 1-2 quarters
• ESI (2006):
– Dainties
– TP‘s discovered 2-3 months earlier
Performances (Efficiency) by
Relying on the Periodogram
• TP-filters won NN3 (2007) and NN5 (2008)
forecasting competitions (~60 participants)
– IIF and University of Lancaster
– Monthly Macro- and Financial Data (111 time
series) and daily financial data (111 time
series)
– Outperformed winner and runner-up of
prestigious M3 competition, X-12-ARIMA,
Tramo, Forecast-Pro, Autobox, Exponential
smoothing: Simple, Holt, Damped,…
– Neural nets, artificial intelligence
– http://blog.zhaw.ch/idp/sefblog
DFA
Direct Filter Approach
Turning Points (TP)
Controlling the Time Delay
(Customization)
[T / 2]
 2
ˆ ( ) |2 I ( )
|

(

)

A
k
k
NX
k

 T  k 0
min ˆ 
 2 [T / 2] 2 A( ) Aˆ ( )(1  cos(
ˆ ( )))I ( )
k
k
k
NX
k
k 0


T
•λ>1: emphasize the time delay in the pass-band
•λ=1: best level filter
Customization: Controlling time
delay and smoothness
 2 [T / 2]
2
ˆ ( ) |2 I ( )
W
(

)
|

(

)

A
k
k
k
NX
k
 T  k 0
min ˆ 
 2  [T / 2] 2W ( ) 2 A( ) Aˆ ( )(1  cos(
ˆ ( )))I ( )
k
k
k
k
NX
k
k 0

T
• Stronger damping of highfrequency noise in stop-band
• Smaller time delays in pass-band
• W(ω) is monotonic (increasing)
and λ>1
W ( )

Amplitude DFA TP-filter (blue) vs.
seem. virtuous level filter (KOFBarometer)
Delay TP-filter (blue) vs. seem.
Virtuous level filter
(KOF-Barometer)
TP-Detection
• Smoother and Faster!
• Poor Mean-Square Performances
MDFA
Real-Time Multivariate Filter
X t ,W1t ,...,Wmt , t  1,..., T
rt 


k 
k
X t k

Target: Yt 


k 
k
X t k


ˆ
ˆ
ˆ
   0 k X t k    1kW1,t k  ...    mkWm ,t k 
k 0
k 0
 k 0

Objective: choose ˆik such that rt 2 is smallest possible
E[rt ]FILTERWEIGHTS  Min
2
•`Direct Filter Approach’
Real-Time Filter Cointegration
Constraints (Rank=1)
Ct  X t  1W1t  ...   mWmt
rt =((0)-ˆ (0))X t  ˆ 1 (0)W1t  ...  ˆ m (0)Wmt  Z t

I(0)

I(0)
Zt

((0)-ˆ (0))X t  ˆ 1 (0)W1t  ...  ˆ m (0)Wmt

ˆ i (0)  i ((0)-ˆ (0))

I(0)
I(0)
Efficiency (Theorem 4.1, Wildi2008,
Wildi/Sturm2008)
2
T
(T 1) / 2

k  (T 1) / 2
| Tr '(k ) |  E[rt ]  eT
2
2
• The error term eT is smallest possible
uniformly
• Uniform efficiency ↔ Customization
Optimal (Efficient) Criterion under
Cointegration (Rank=1)
•Filter Restrictions are satisfied
2
T
(T 1) / 2

k  (T 1) / 2
| Tr '(k ) |2

Min

 (0)  exp(ik )(k )

Tr '(k )  (0)TC (k )  exp(ik ) 
 (0)  T X (k ) 
1  exp(ik )



 (0)  exp(ik )(k )

 (0)  T X (k ) 

1  exp(ik )


 ˆ h (0)  exp(ik )ˆ h (k )

  h (0)  T Wh (k )


1  exp(ik )
h 1 

m
Performances MDFA
• Output-gap US- and Euro-GDP (2008):
– CF and multivariate CF
– turning-points anticipated by 1-2 quarters
• USRI
– Outperformed Markov-switching (Chauvet,
Chauvet/Piger), Dynamic factor models
(CFNAI), state space models (ADS), HodrickPrescott (OECD-CLI), Christiano-Fitzgerald
– SEF-Blog
• MDFA-XT
• EURI
WARNING!!!
• THIS IS NOT A PUSH-THE-BUTTON
APPROACH
• Formula 1 racer: it can be fast (Ferrari) and
reliable (Mercedes) but you have to tweak it
carefully: Ferrades/Mercearri
–
–
–
–
Filter design (ZPC)
Filter constraints (emphasize frequency zero)
Understanding/interpreting: `intelligence’
2008-Book: http://www.idp.zhaw.ch/sef
• Happy to provide support given financial
incentives
Contact/Links
Contact/Links
• marc.wildi@zhaw.ch
• http://blog.zhaw.ch/idp/sefblog
– Illustrate methodological issues by relying on `realworld‘ projects with economic partners
• http://www.idp.zhaw.ch/usri
– Real-Time US Recession Indicator
• http://www.idp.zhaw.ch/MDFA-XT
– Experimental Trader for MSCI Emerging Markets
– Filters on-line late July
• http://www.idp.zhaw.ch/sef
– Signal Extraction & Forecasting Site
– Books, Articles, Software