Shortfall Surprises
KWAKU ABROKWAH AND GEORGE SOFIANOS
KWAKU ABROKWAH
is an analyst, Equity
Execution Strategies, at
Goldman, Sachs & Co. in
New York, NY.
kwaku.abrokwah@gs.com
GEORGE SOFIANOS
is a vice president, Equity
Execution Strategies, at
Goldman, Sachs & Co. in
New York, NY.
george.sofianos@gs.com
n this article, we use a large sample of
order executions to better understand
“shortfall surprises”: the difference
between actual and expected trading
costs. Our trading cost measure is execution
shortfall, including both liquidity impact and
the opportunity cost of slow executions.1
Exhibit 1 summarizes our main findings. In
our sample, on average, only 20% of actual
shortfall is predictable pre-trade. Focusing on
the non-predictable component, we investigate the following possible reasons for the
shortfall surprise:
I
• Price surprise. A higher-than-expected
price increase over the execution horizon
should result in a higher-than-expected
actual shortfall on buy orders and lowerthan-expected shortfall on sell orders.
We find that, on average, the price surprise explains 42% of the shortfall surprise. For orders in large-cap stocks that
take more than an hour to execute, the
price surprise explains 73% of the shortfall surprise. Splitting the price surprise
into its alpha and market components,
the alpha surprise explains 38% of the
shortfall surprise, and the market surprise only 4%.
• Volume surprise. Higher-than-expected
volume over the execution horizon
should result in lower-than-expected
shortfall. Surprisingly, we find that the
SUMMER 2007
volume surprise explains very little of
the shortfall surprise (only 0.3%), indicating that higher-than-expected volume
does not reduce trading costs.
• Volatility surprise. Higher-than-expected
volatility over the execution horizon
should result in higher-than-expected
shortfall. We find that the volatility
surprise explains only 0.1% of the shortfall surprise.
• Spread surprise. Higher-than-expected
quoted spreads over the execution horizon
should result in higher-than-expected
shortfall. We find that the spread surprise
explains 1.8% of the shortfall surprise.
Price surprises, therefore, are by far the
most important factor explaining shortfall
surprises; the volume, volatility, and spread
surprises have little explanatory power. These
findings have important implications for posttrade analysis, pre-trade tools, the development
of algorithms, and the choice of execution
strategies.
In interpreting post-trade execution
quality, for example, we must somehow control for the underlying execution-horizon price
move. For pre-trade analysis, our findings suggest that the only way to improve the pre-trade
t-cost estimates is to better predict the alphamove over the execution horizon. Better pretrade volatility and volume estimates will not
help much. The same applies for algorithm
THE JOURNAL OF TRADING
11
EXHIBIT 1
Summary of Main Findings1
1
Sample period June 1 through December 31, 2006; 241,610 orders.
R-square from univariate regression of actual on expected shortfall.
3
Incremental R-square from multivariate regression analysis of shortfall surprise on the five factor surprises: volatility, spread, volume, EH-alpha and EH-market.
2
development and the choice of execution strategies; only
better predictors of execution-horizon alpha will significantly reduce execution shortfall.
In the next section, we develop our framework for
explaining the shortfall surprise, followed by a section on
our data sample and the construction of our variables.
The following section presents our empirical findings. In
the section after that, we focus on our puzzling finding
that volume surprises do not affect trading costs. The next
section elaborates on the distinction between volatility
and price surprises. We conclude with a discussion of the
implications of our empirical findings, and possible extensions of our analysis.
A FRAMEWORK FOR ANALYZING
THE SHORTFALL SURPRISE
We begin by formally defining execution shortfall, our
t-cost measure. For buy orders, execution shortfall is the
execution price minus the prevailing mid-quote at order
arrival (strike price) as percent of strike price.2 For sell
12
SHORTFALL SURPRISES
orders, execution shortfall is the strike price minus the execution price as percent of the strike price. In Exhibit 2, we
introduce a hypothetical order execution that we will use
throughout this section to illustrate our framework. The
trader receives an order to buy 60,000 shares. The strike
price at order arrival is $25.00. The trader executes the
order over time in three executions and the volumeweighted execution price is $25.06. The execution shortfall in this example is six cents or 24 basis points (bps).
In addition to liquidity impact, execution shortfall
includes the opportunity cost of delayed execution. The
opportunity cost arises because the price may move away
from the trader over the execution horizon. This price
move has both a market and a stock-specific (alpha) component and we estimate both. Exhibit 3 shows the three
components of execution shortfall: liquidity impact, alpha
loss, and market loss. In our example, the trader is buying
in a rising market, so the opportunity cost is positive and
increases the shortfall. If the price were falling, the opportunity cost would be negative, reducing the shortfall, and
possibly resulting in negative shortfall. Unlike liquidity
SUMMER 2007
EXHIBIT 2
Execution Shortfall
impact, which is never negative, execution shortfall can
be negative.3
In this article, we analyze the shortfall surprise (SS)
that we define as actual shortfall (SA) minus expected
shortfall (SE):
SS = SA – SE
(1)
1. Stock capitalization: large-cap, mid-cap, and smallcap stocks.
2. Listing market: NYSE, AMEX, and NASDAQ
stocks.
3. Order size: the actual dollar value of the order
executed.
4. Volatility over the execution horizon.
5. Quoted spreads over the execution horizon.
6. Trading volume over the execution horizon.
To generate the shortfall surprise, we must first estimate expected shortfall. Expected shortfall is the shortfall a trader can predict with only pre-trade information:
when the trader receives the order, but before execution.
To estimate expected shortfall, we need estimates for both
liquidity impact (LE) and the expected underlying price
move over the execution horizon (EH-price or PE):
The first three factors (stock capitalization, listing
market, and order size) we know with certainty pre-trade.
The other factors (volatility, spread, and volume) we do
not know with certainty pre-trade, and we must use their
expected values to estimate liquidity impact (LE):
SE = LE + PE
LE = L(Y, XE) > 0
(2)
We estimate liquidity impact using the Goldman
Sachs t-cost model.4 The model uses the following six
factors to predict liquidity impact:
SUMMER 2007
(3)
where Y are the factors known with certainty pretrade, and XE are the factors not known with certainty
pre-trade.
THE JOURNAL OF TRADING
13
EXHIBIT 3
The Three Components of Execution Shortfall
We next discuss our EH-price concept. In Exhibit 4,
we assume the market component of EH-price is zero
and show how we measure the stock-specific alpha move
over the execution horizon (EH-alpha). Continuing the
Exhibit 2 example, the trader received the order at 12:00,
executed 20,000 shares immediately, another 20,000 shares
at 13:00, and the final 20,000 shares at 14:00. The execution horizon is two hours (order arrival to last execution), and the volume-weighted execution turnaround
time is one hour. The execution turnaround time is the
order’s half-life, taking into account that the order execution is spread over the two-hour horizon.5 We define EHalpha as the price move over the order’s half-life, aside
from the liquidity impact of the trade itself.
In calculating EH-alpha, it is critical to use a posttrade price after the liquidity impact of the trade subsides. In our example, liquidity impact subsides at 15:00,
so any price after that will work. In our empirical analysis,
we use the closing price to measure the actual EH-alpha
for each order in our sample. This procedure assumes
that the closing price is not affected by the impact of the
14
SHORTFALL SURPRISES
trade. Our empirical analysis below confirms this is indeed
the case.
In Exhibit 4, we show how we use the closing
price to calculate EH-alpha. We first calculate the alphato-close. For buy orders, alpha-to-close is the closing
price minus the strike price as percent of the strike
price. For sell orders, alpha-to-close is the strike price
minus the closing price as percent of the strike price.
The alpha-to-close in our example is 40 bps. We then
“allocate” the alpha-to-close to the order in proportion to the order’s half-life. We derive the allocation
factor (Φ) by dividing the order’s half-life (one hour)
by the time from arrival to market close (four hours).6
So, in our example, the allocation factor is 1/4 and EHalpha is 10 bps.
In practice, the underlying price move has both
market and alpha components:
EH-price = EH-alpha + βEH-market
(4)
where β is the intra-day stock beta.
SUMMER 2007
EXHIBIT 4
Execution-Horizon Alpha
1
If the time of last execution is after 16:00, it is possible for the ratio to exceed 1. In these cases (129 orders), we set the ratio to 1.
In our empirical analysis, we use S&P 500 ETF
prices to decompose the underlying price move into EHmarket and EH-alpha. In Exhibit 5, continuing our
example, the market-to-close move is 16 bps and allocating over the execution horizon EH-market is four bps.
Combining Equations (2) to (4) and, for illustration
purposes, using a linear version of the liquidity impact
model L (Y, X), expected shortfall is given by:
SE = a + bY + c XE + β ME + AE
(5)
where ME is the expected EH-market and AE is the
expected EH-alpha.
Suppose the liquidity impact is 10 bps. To get the
total shortfall, we must add the EH-price move. In
Exhibit 5, the EH-price move is 14 bps so the total
shortfall is 24 bps: 10 bps impact and 14 bps price loss.
SUMMER 2007
We next derive the shortfall surprises factor model
we use in our analysis. In Equation 5, replacing the
expected values of the various factors by their post-trade
actual values, the actual shortfall (SA) is given by:
SA = a + bY + cXA + β MA + AA + U
(6)
Actual shortfall is also influenced by other factors,
(U), not included in our empirical analysis. Most of these
other factors are random. Some of them, however, may
be systematic but difficult to quantify even post-trade; for
e.g., trader skill or the presence of natural counterparties.
Subtracting expected shortfall (Equation 5) from
actual shortfall (Equation 6) we get our basic shortfall surprises model:
SS = c (XA – XE) + β (MA – ME) + (AA – AE) + U (7)
THE JOURNAL OF TRADING
15
EXHIBIT 5
The Market and Alpha Components
Or, writing out the factors in full:
S s = c1 (VOLA – VOLE ) + c 2 (SPREADA – SPREADE )
Volatility surprise
Spread surprise
+ c 3 (VLM – VLM ) + β(EH-market A – EH-market E )
A
E
Volume surprise
A
Market surprise
E
+ (EH-alpha – EH-alpha ) + U
(8)
Alpha surprise
In our analysis of the shortfall surprises, we focus on
the five input surprises: 1) the volatility surprise, 2) the
spread surprise, 3) the volume surprise, 4) the market surprise, and 5) the alpha surprise. The expected EH-market
and EH-alpha are hard to estimate and, in our empirical
analysis, we assume they are zero. We describe the surprises in greater detail in the next section. Note that in
the surprises model, the factors known with certainty
pre-trade (Y) drop out. Also, because of the way we constructed EH-price, the coefficient of the alpha surprises
16
SHORTFALL SURPRISES
in the model is one, and the coefficient of the market surprise is β, the intra-day stock beta.
In deriving our shortfall surprise model (8), we assume
that the liquidity impact model we use to generate expected
impact is correctly specified. But the model may be misspecified; we may have omitted systematic variables (U in
our specification), or the model coefficients may be biased.
Model misspecification creates another possible source of
shortfall surprises. In the Appendix, we discuss this issue
more formally and decompose shortfall surprises into a
misspecification component and the input surprises component. In our empirical analysis, we carefully constructed
our sample to minimize the risk of model misspecification.
SAMPLE CONSTRUCTION
AND DESCRIPTION
Our final estimation sample consists of 241,610 client
orders executed by Goldman Sachs over seven months,
SUMMER 2007
June through December 2006. To minimize the risk of
model misspecification, we restricted our sample to closely
match the sample we use in estimating the Goldman Sachs
t-cost model. Our final sample consists of market, notheld orders, executed on an agency basis by the Goldman
Sachs U.S. single-stock trading desk.7 We also aggressively
filtered unusual orders; we dropped odd lots, kept only
orders in common stocks, dropped ADRs and ETFs,
OTCBB and Pink Sheets, dropped crosses, dropped orders
in extreme price stocks (less than $1 or more than $150),
dropped tick sensitive orders (e.g., sell short), etc. Finally,
we dropped orders with data errors.
Exhibit 6 summarizes the order composition of the
final sample. The 241,610 orders in our sample are evenly
divided between buys and sells. One quarter of the orders
are NASDAQ stocks8 and the balance are NYSE stocks.9
Of the total orders, 64% are in large-cap stocks and 9%
in small-cap stocks. Only 296 orders exceed 25% of
average daily volume, and 89% of the orders are less than
10,000 shares. The majority of the orders (92%) have an
execution half-life less than 15 minutes; 13,464 orders
have a half-life more than 30 minutes. The average stock
price in our sample is $34.
Exhibit 7 summarizes order characteristics for the
overall sample. The average order size executed is 9,420
shares, ranging from 18 shares to 29 million shares. The
value-weighted average order size executed is 5% of ADV
and the participation rate is 21%.10 The value-weighted
half-time is 65 minutes, but the median is only 15 seconds and ranges from instantaneous executions to all-day
executions. The value-weighted average actual shortfall
is 23 bps, but the median is only 2 bps. Actual shortfall
ranges widely from –757 bps to +1,069 bps.
Exhibit 8 confirms that, at least on average, the
closing prices we use in calculating the EH-price move are
not affected by liquidity impact. In Exhibit 8, we plot the
average shortfall, same-day closing price and the closing
prices over the next five days for all the orders in our
sample. We measure all prices relative to the order-arrival
price, and include both buy and sell orders, but flip the sign
of the sells. We use value-weighted averages, so large trades
get more weight in the average. The average closing price
is 26 bps higher than the strike price, but there is little
reversal over the next five days. The absence of reversal
reassures us that the closing price is not affected by the
temporary liquidity impact of the trades themselves.
We next describe how we construct the shortfall
surprises and the five factor surprises we use in our analysis.
SUMMER 2007
For each order in our sample, we use the Goldman Sachs
t-cost model to estimate the expected liquidity impact.
The model is re-estimated monthly using the most recent
nine months of data. For each order, we use the most
recent version of the model as of the date of the order to
generate out-of-sample estimates of that order’s expected
impact. For an order received on July 7, for example, we
use the model estimated with data prior to July 7. The
monthly re-estimation of the model ensures the coefficients are not stale and again reduces the risk of model
misspecification.
The Goldman Sachs t-cost model gives different
impact estimates depending on the execution horizon
specified. In our analysis, we estimate the expected impact
using the order’s actual execution horizon: order arrival
EXHIBIT 6
Order Types
Sample period June 1 through December 31, 2006.
THE JOURNAL OF TRADING
17
EXHIBIT 7
Overall Sample Summary Statistics1
1
Sample period June 1 through December 31, 2006; (241,610 orders).
Simple mean.
3
Value-weighted mean.
2
to last execution. To generate our impact estimates, we
must also specify pre-trade estimates for volatility, quoted
spreads and trading volume:
• Volatility is measured as the percent difference
between the intra-day high and low price, adjusted
to the order’s execution horizon, by using the same
allocation factor Φ that we use in calculating EHalpha (Exhibit 4).
• Quoted spread is the time-weighted spread over the
execution horizon, taking into account the spread
smile (the U-shaped intraday spread pattern).11
EXHIBIT 8
Closing Prices are Not Affected by Liquidity Impact
Sample period June 1 through December 31, 2006; 241,610 buy and sell orders (Sign of sell orders flipped). Value-weighted averages.
18
SHORTFALL SURPRISES
SUMMER 2007
Exhibit 9 shows a scatter plot of actual shortfall (vertical axis) against expected shortfall (horizontal axis).
Actual shortfall equals expected shortfall along the
45 degree line. Orders with actual shortfall exceeding
expected shortfall are above the 45 degree line and orders
with actual shortfall less than expected shortfall are below.
Along the vertical, we see the large range in actual shortfall from a minimum of –757 bps to a maximum of +1,069
bps. Along the horizontal, we see the range in expected
shortfall estimates from a minimum of 0.3 bps to a maximum of 334 bps. Because we assume expected EHmarket and EH-alpha are zero, the expected shortfall is
always positive (just the liquidity impact).
The number in each bubble indicates the number
of orders clustering in close proximity to that point. About
• Trading volume is measured over the execution
horizon, taking into account the volume smile (the
U-shaped intraday volume pattern).
We generate pre-trade estimates for these three
factors using their median value over the prior 21
trading days.
To estimate expected shortfall (Equation 5), we also
need pre-trade estimates for EH-alpha and EH-market.
Unlike liquidity impact, where reliable pre-trade estimates are widely available, pre-trade estimates of EHmarket and EH-alpha are difficult to find. As is typically
the case in practice, in our analysis we assume that pretrade, the expected EH-market and expected EH-alpha
are both zero.
EXHIBIT 9
Scatter plot of Actual Vs Expected Shortfall
0
50
100
150
200
250
300
Sample period June 1 through December 31, 2006; (241,610 orders).
SUMMER 2007
THE JOURNAL OF TRADING
19
210,000 orders are clustered between –60 and +35 bps of
actual shortfall and between 0 and 15 bps of expected
shortfall. The remaining orders show a large dispersion,
both in the expected and actual shortfall. Our objective
in this article is to better understand this dispersion: the
shortfall surprise.
Exhibit 10 summarizes the post-trade actual values
of the five factors we use to explain the shortfall surprise.
All five factors range widely, mirroring the large range of
actual shortfall. Volatility, for example, ranges from 0% to
20% and quoted spreads from 0 bps to 882 bps. The actual
EH-market move ranges from –119 bps to +121 bps, but
averages to zero by all measures (median, simple, and
weighted means). The actual EH-alpha, however, has a
much bigger range (from –557 bps to +836 bps) and the
value-weighted mean is 10 bps.
In our shortfall surprises model (Equation 8), the
shortfall surprise depends on the five factor surprises.
Exhibit 11 summarizes our estimates of the shortfall surprise and the five factor surprises. The first column shows
the forecasting accuracy of our pre-trade measures and gives
an indication of the potential explanatory power of each
surprise. If, for example, a factor is perfectly forecasted,
then there is no factor surprise and that factor cannot explain
the shortfall surprise. We measured forecasting accuracy by
estimating univariate regressions of actual values on expected
values and report the unexplained variation (one minus
R-square). The unexplained variation tells us how much
of the actual variation in each variable is not explained by
the expected value of that variable.
Exhibit 11 shows that 80% of the actual shortfall
variation is not explained by expected shortfall (or conversely, that the pre-trade estimates explain only 20% of
the post-trade variation in actual shortfall). Expected
volatility leaves only 23% of the variation in actual
volatility unexplained. Expected spreads and volume
leave about 50% of the post-trade variation unexplained.
Since we assume they are zero, expected EH-market and
EH-alpha leave 100% of the variation in the actual EHalpha and EH-move unexplained. The EH-market and
AH-alpha surprises, therefore, have great power to
potentially explain the shortfall surprise. Exhibit 11 also
shows the large range of all the surprises, another measure of forecasting accuracy. In general, all five factors
have large surprises and, therefore, potentially large
explanatory power in our shortfall surprise regressions,
to which we now turn.
THE EMPIRICAL FINDINGS
Our preferred regression specification for the shortfall surprise (SS) model is:
EXHIBIT 10
The Five Factors: Actual Values1
1
Sample period June 1 through December 31, 2006; (241,610 orders).
Simple mean.
3
Value-weighted mean.
2
20
SHORTFALL SURPRISES
SUMMER 2007
EXHIBIT 11
The Five Factor Surprises: Actual Minus Expected Values1
1
Sample period June 1 through December 31, 2006; 241,610 orders.
One minus the R-square from univariate regression of actual to expected.
2
S s = c0 + c1(VOLA – VOLE )
Volatility surprise
⎛
VLM A ⎞
+ c 2 (SPREADA – SPREADE )+ c 3 ⎜ log
⎟
⎝
VLM E ⎠
Spread surprise
Volume surprise
A
E
+ c4 (EH-market – EH-market )
Market surprise
+ c5 (EH-alpha A – EH-alpha E )
Alpha surprise
We experimented with many alternative specifications (for e.g., with and without intercepts, different functional forms for the five factors, etc.) and, in all cases, the
results are similar.12
Exhibit 12 shows the results from the estimation of
our preferred specification over our whole sample of
241,610 orders. The overall fit of the regression is 44%
(R-square), and the coefficients of all five factor surprises
are significant and have the right sign. Higher-thanexpected volatility and spreads lead to higher-thanexpected shortfall while higher-than-expected volume
leads to lower-than-expected shortfall. Similarly, higherthan-expected market and alpha moves result in higherthan-expected shortfall. Our hypothesis is that the
coefficient of EH-alpha in the regression should be one,
and Exhibit 12 shows that this is indeed the case; a five
basis-point increase in EH-alpha will lead to a five
SUMMER 2007
basis-point increase in the actual shortfall. The coefficient
of EH-market in our regression is an estimate of the shortterm intra-day beta on our sample; the results in Exhibit
12 suggest that this is also close to one.13
To better understand the relative importance of the
five factor surprises in explaining the shortfall surprise,
we also ran a series of univariate regressions of the shortfall surprise on each of the factor surprises in turn. In
Exhibit 13, columns four to eight, we report the R-squares
from these univariate regressions. The first row shows the
results of running the regressions on all the orders in our
sample. In the univariate regressions, the EH-alpha surprise is by far the most important factor explaining almost
40% of the shortfall surprise variation. The other four
factors have little explanatory power; EH-market explains
5.8%, volatility explains 3.0%, spreads explain 1.5% and,
most surprising, volume only explains 0.2% of the shortfall surprise.
Exhibit 14 shows that the covariances across the
five factor surprises, while small, are not always zero.
The correlation between the EH-alpha and volatility
surprises, for example, is 25% and the correlation
between the volume and volatility surprises is 17%.
Because the covariances across the factor surprises are not
zero, the univariate regression results can be misleading.
To double check, we also performed an incremental
analysis, where we evaluate the contribution of each
THE JOURNAL OF TRADING
21
EXHIBIT 12
The Shortfall Surprise Multivariate Regression1
1
Sample period June 1 through December 31, 2006; t-statistics in brackets below the coefficients.
factor surprise by dropping it from the full multivariate
regression. The overview results in Exhibit 1 are from
the incremental approach and are similar to the univariate regression results.
The last three columns in Exhibit 13 show another
example of the incremental approach. Column 10 shows
the full five-factor regression, and the regression in column
nine drops the two EH-price surprises (market and alpha).
Dropping the EH-price surprises reduces the explanatory
power of the regressions from 44% to only 4.5%, confirming that the price surprises account for almost all of
the explanatory power in our shortfall surprise regressions.
In Exhibit 13, rows two to four, we divide our
sample by stock capitalization: orders in large-cap stocks
(>$7.5 billion), mid-cap stocks, and small-cap stocks
(<$1 billion). As we move from large-cap to small-cap
stocks, the accuracy of the expected shortfall forecasts as
measured by the root mean-square error (RMSE) falls.14
The expected shortfall RMSE is 13 bps for orders in largecap stocks, but 49 bps for orders in small-cap stocks, indicating that the shortfall surprise is higher in small-cap
stocks.
Our findings on the five factor surprises are similar
across stock capitalization buckets. In all three buckets,
EXHIBIT 13
Shortfall Surprise Attribution: An Overview1
1
Sample period June 1 through December 31, 2006.
Root mean-square error: the square root of 1/N Σ(SAi – SEi)2, where SAi is the post-trade actual shortfall and SEi is the pre-trade expected shortfall for each order.
3
The average executed order size in this bucket is 775 shares.
2
22
SHORTFALL SURPRISES
SUMMER 2007
EXHIBIT 14
Covariance of the Five Surprises1
1
Sample period June 1 through December 31, 2006; (241,610 orders).
Log of ratio of actual to expected volume.
2
the EH-price surprise explains most of the shortfall surprise. The volatility and spread surprises explain more of
the shortfall surprise in small-cap stocks than in the other
two buckets, but the numbers are still small. In the univariate regressions, for example, the spread surprise
explains 4.3% of the shortfall surprise in small-cap stocks,
but only 0.1% of the surprise in large-cap stocks. The
volume surprise explains very little of the shortfall surprise
across all stock capitalizations.
Because we construct EH-price using closing prices,
one concern is that the closing prices are affected by liquidity impact and therefore are not a pure measure of the
underlying price move. In Exhibit 9, we showed that, at
least on average, the closing prices in our sample are not
affected by liquidity impact. The liquidity impact of the
larger trades in our sample is more likely to affect the
closing price.15 So, in Exhibit 13, rows five to six, we further test the sensitivity of our results to the quality of the
closing price by dropping the larger trades in our sample.
In row five, we drop all orders greater than 25% of ADV
(297 orders) and in row six, we drop all orders greater
than 15% of ADV (1,228 orders). In both cases, dropping
the outlier large orders does not affect our results.
In the last row in Exhibit 13, we focus on the small
orders in our sample: 177,191 orders less than 0.1% ADV
(average order size 775 shares). As expected, for these
small orders, the shortfall forecasting error is low: the
RMSE is only 7 bps compared to 23 bps overall. For these
small orders, the five factor surprises explain only 17% of
SUMMER 2007
the much smaller shortfall surprise. But still, across the
five factor surprises, the EH-price surprise is by far the
most important factor in explaining the shortfall surprise:
volatility, spread, and volume surprises explain 2.3%, while
EH-price explains 14.7%.
In Exhibit 15, we focus on the time dimension of
order executions. A quick execution exposes the order
to little EH-price movement, so we expect the EH-price
surprise to explain less of the shortfall surprise. At the
other end, an order that takes several hours to execute is
exposed to potentially large price moves, so we expect
EH-price to be much more important in explaining the
shortfall surprise. The results in Exhibit 15 confirm this
hypothesis.
Exhibit 15 is divided in four panels: Panel A covers
orders in all stocks, Panel B focuses on orders in large-cap
stocks, Panel C focuses on mid-cap stocks and Panel D
focuses on small-cap stocks. The top row in each panel
replicates the information in Exhibit 13. The next two
rows in each panel divide the orders into orders with halflife more than 30 minutes and orders with half-life less than
30 minutes. The last two rows in each panel repeat this
split, using a 60-minute cut-off.
The first thing to notice in Exhibit 15 is that the
forecasting accuracy of expected shortfall falls sharply as
the execution horizon increases (column 4). In Panel A,
for example, the RMSE for orders with half-life more
than 60 minutes is 89 bps compared to only 15 bps for
orders with half-life less than 30 minutes. For orders in
THE JOURNAL OF TRADING
23
EXHIBIT 15
The Shortfall Surprise Attribution: Details1
1
Sample period June 1 through December 31, 2006.
small-cap stocks (Panel D), the RMSE for orders with
half-life more than 60 minutes is even higher: 118 bps.
The forecasting accuracy results in Exhibits 13 and 15
strikingly quantify what most traders know from experience: most of the shortfall surprise comes from large orders
that take time to execute.
In Exhibit 15, column 11 shows the explanatory
power of the full five-factor regression model. Across the
various buckets, the R-square of the full model ranges
from 75.3% (orders >60 minutes, large-cap stocks) to
8.0% (orders <30 minutes, large-cap stocks). We next
examine the contribution of the five-factor surprises in
explaining the shortfall surprise. The last column in
Exhibit 15 shows that, consistent with our hypothesis,
the EH-price surprise is much more important in
explaining the shortfall surprise for slow executions. In
24
SHORTFALL SURPRISES
Panel A, for example, EH-price explains 63.5% of the
shortfall surprise in orders with half-life more than 60
minutes, but only 6.3% of the surprise for orders with
half-life less than 30 minutes. For orders in large-cap stocks
with half-life more than 60 minutes, EH-price explains
a remarkable 73% of the shortfall surprise.
Looking at the two components of EH-price
(market and alpha), the results in Exhibit 15 across all
buckets are consistent with the results in Exhibit 13: EHalpha is by far the most important factor in explaining
the shortfall surprise. For orders in large-cap stocks with
half-life more than 60 minutes, for example, EH-alpha
explains 65.3% of the shortfall surprise while EH-market
explains only 12.5%. For orders with half-life less than
30 minutes, EH-alpha explains 6.1% of the shortfall surprise while EH-market explains only 0.8%.
SUMMER 2007
The summary statistics on EH-alpha and EH-market
in Exhibit 10 provide a clue why the alpha component
dominates: EH-alpha has a bigger range (–557 bps to
+839 bps) than EH-price (–119 bps to +121 bps). Out
of the 241,610 orders in our sample, 3,173 orders are
associated with an EH-alpha move exceeding 50 bps (in
absolute terms), but only 407 orders are associated with
an EH-market move exceeding 50 bps.
Exhibit 15, column 10, confirms the robustness of
our findings on the volatility, spread, and volume surprises: consistently small explanatory power across all
buckets. Finally, Exhibit 15, column three, confirms the
robustness of the absence of any volume effect. Across all
buckets, the volume surprise explains less than 1% of the
shortfall surprise.
Perhaps the most striking observation about
Exhibits 13 and 15 is the robustness of our main results.
Whichever way we cut the data, EH-alpha is by far the
most important factor in explaining the shortfall surprise;
while volatility, spread, and volume surprises explain very
little. The volume results are particularly surprising and,
in the next section, we take a closer look.
THE VOLUME SURPRISE
Our most puzzling finding is that volume surprises
do not affect trading costs. This seems counterintuitive:
more volume than anticipated should surely reduce trading
costs. This intuition, however, rests on the presumption
that higher volume is caused by an increase in the supply
of liquidity (Case A in Exhibit 16). Our empirical findings suggest that an increase in volume is caused by an
increase in both liquidity supply and demand (Case C in
Exhibit 19): a hundred more traders supply liquidity, but
another hundred traders demand more liquidity. An
increase in uncertainty, for example, will lead to an increase
in both liquidity supply and demand. Similarly, the popular “participate” execution strategies and algorithms
automatically generate liquidity demand in proportion to
increases in liquidity supply.
In our multivariate regressions where we control
for the variation in all five factors, all else being equal,
the volume surprise reduces trading costs, but by a tiny
amount; for e.g., volume twice the expected amount
reduces trading costs by less than 0.3 bps. In the univariate regressions of shortfall surprise on volume surprise, a surprise increase in volume actually increases
trading costs (Case B in Exhibit 16). Equivalently, the
SUMMER 2007
simple correlation between the volume and shortfall surprises is small, but positive, 4%. The striking difference
between the multivariate and univariate results suggests that
other factor surprises correlated the volume surprise (for
e.g., volatility and EH-alpha in Exhibit 14), and increase
costs outweighing the beneficial effects of higher-thanexpected volume.
We performed extensive additional robustness tests
of our finding on the volume surprise. In all cases, volume
surprises explain very little of the shortfall surprise. In
Exhibit 17, for example, we focused on high-volume surprise events: 20,116 orders in our sample associated with
a volume surprise greater than one standard deviation.
Again, there is little change: volume surprises explain only
1.4% of the shortfall surprise and the EH-price move
explains 60.2%.
The important implication of our volume findings
is that traders and algorithms cannot naively exploit
volume surprises to reduce trading costs. To beneficially
exploit a volume surprise, execution strategies must distinguish between liquidity supply and liquidity demand
surprises, which is hard to do. Taking advantage of
higher-than-expected volume has one unambiguous
beneficial effect: traders fill orders faster reducing execution risk.
THE VOLATILITY SURPRISE
Another empirical finding worth highlighting is the
sharp distinction between the small effect on trading costs
of volatility surprises and the large effect of EH-price surprises. The important distinction here is that volatility
surprises are unsigned price moves while EH-price surprises are signed price moves, depending on the direction
of the trade.16 For buy orders, EH-price is closing price
minus strike price, while for sell orders, EH-alpha is strike
price minus closing price (see Exhibit 18).
The unsigned EH-price is just another measure of
execution-horizon volatility: closing price minus strike
price (as percent of strike price) for both buys and sells.
Let’s call this measure of volatility EH-volatility. In
Exhibit 18, we compare the results of regressing the shortfall surprise on the signed and unsigned EH-price. In
the univariate regression of shortfall surprises on signed
EH-price, EH-price explains more than 40% of the variation in the shortfall surprise. However, in the univariate
regression of the shortfall surprise on EH-volatility (the
unsigned EH-price), the unsigned EH-price explains less
THE JOURNAL OF TRADING
25
EXHIBIT 16
The Volume Surprise: Why It Does Not Matter?
EXHIBIT 17
High Volume Surprises1
1
Sample period June 1 through December 31, 2006.
|VLMA – VLME| > one standard deviation.
2
26
SHORTFALL SURPRISES
SUMMER 2007
EXHIBIT 18
The Volatility Surprise
1
Sample period June 1 through December 31, 2006.
than one percent of the variation in the shortfall
surprise. We find similar results in a multivariate regression of the shortfall surprise on both the signed and
unsigned EH-price: all the explanatory power comes
from the signed price move.
What is striking about the analysis in Exhibit 18 is
that, by design, EH-price and EH-volatility are exactly
the same measure, except that one is signed based on the
direction of the trade and the other is not. The important implication of our findings in this section is that
traders and algorithms cannot exploit volatility surprises
to reduce trading costs unless they can accurately predict
the direction of the price move.17
IMPLICATIONS AND CONCLUSION
Our empirical findings have important implications
for post-trade execution quality analysis (EQA) and for
pre-trade tool, for the development of algorithms, and
for the choice of execution strategy. We conclude with a
discussion of these implications.
SUMMER 2007
Post-trade EQA
Our finding that price surprises are the main cause
of shortfall surprises means that, in interpreting post-trade
EQA, it is essential to control for the underlying execution-horizon price move. Our preferred way of doing
this is by reporting the EH-price move alongside the
actual shortfall, or equivalently, by using the EH-price to
split actual shortfall into liquidity impact and price loss.18
Suppose actual shortfall is 20 bps and the EH-price is 18
bps: the liquidity impact is only 2 bps, so this is good execution. If the EH-price is 4 bps, however, 20 bps is poor
execution (16 bps liquidity impact). We avoid using the
usual VWAP benchmarks to control for price surprises,
because VWAP benchmarks may be affected by the impact
of the trade itself and may give misleading results.19
The importance of price surprises also suggests that
post-trade EQA to evaluate execution strategies and algorithms can only be meaningfully done with a large sample
of orders. In small samples, extraneous price surprises will
dominate the statistics and obscure the true performance
being evaluated.
THE JOURNAL OF TRADING
27
Pre-trade Tools and Development
of Algorithms
Trading cost models are widely used to generate
pre-trade cost estimates and choose the right execution
strategy. Cost models are also embedded in various algorithms.20 These trading-cost models have relatively low
forecasting power and model developers are trying hard
to improve them. Our findings suggest that the only way
to make a significant difference in forecasting accuracy is
by coming up with better forecasts of EH-price. Better
forecasts of volume and volatility, in particular, will not
improve the forecasting accuracy of trading cost models.
But how can we come up with better EH-price
forecasts? One way is to continue improving models for
short-term price forecasts. Work is being done on this front
in the context of short-term quantitative trading. The
challenge here is that a better short-term price forecasting
model is more valuable as a proprietary trading tool, rather
than a widely available pre-trade tool. But the synergies
exist and the possibility of combining short-term trading
and improved execution strategies within a buy-side
trading desk is intriguing.
Another way to improve the accuracy of shortfall
estimates is by allowing users to adjust the estimates based
on the user’s view of the expected EH-price. Sell-side
traders may have a good “feel for the market,” so allowing
them to adjust cost estimates accordingly can be useful.
Similarly, buy-side traders may have a better understanding
of portfolio managers’ intentions and alpha signal. The
Goldman Sachs PortX algorithm, for example, allows
traders to input their alpha-to-close expectation and uses
this input to adjust the pre-trade cost estimates, and the
optimum execution strategy, accordingly.
Our findings also have important implications for the
development of dynamic algorithms that react real-time
to market changes. For a dynamic algorithm to usefully
react to a volume surprises, it must be able to distinguish
between supply and demand causes of the surprise, which
is very challenging. Similarly, dynamic algorithms cannot
usefully exploit volatility surprises, unless they can predict the direction of the market.
Choice of Execution Strategy
Our findings in this article strikingly quantify one
of the main themes in our research over the past five
years:21 in choosing an execution strategy that takes more
28
SHORTFALL SURPRISES
than a few minutes to complete, by far the most important factor is the expected underlying price move (EHalpha). But is there any empirical evidence that traders
correctly anticipate EH-alpha and adapt their execution
strategies accordingly? In Rakhlin and Sofianos [2006a],
we found no evidence that buy-side traders allocated
orders across passive VWAP algorithms and more aggressive shortfall algorithms based on expected EH-alpha.
But that article was only the beginning of an ongoing
research effort.
If traders cannot correctly anticipate EH-alpha, then
execution strategies should be modified to reflect this fact.
Segmenting orders between passive and aggressive executions, for example, requires reliable EH-alpha forecasts.
In the absence of reliable EH-alpha forecasts, adopting a
single, relatively passive, execution strategy may work best.
The downside of a passive strategy is high execution risk
(a large variation in actual shortfall). If execution risk is
a concern, then an alternative strategy is to let the trader’s
risk aversion drive the choice of execution strategy, and
accept a higher liquidity impact. Yet another alternative
is to request capital and pay a premium to completely
eliminate execution risk.
The shortfall surprises framework that we developed in this article can be used more generally to quantify and compare the value-added of alternative execution
strategies. By analyzing shortfall surprises, instead of actual
shortfall, we control for the usual factors affecting execution quality and can focus on the incremental value of
additional factors. Consider again the comparison between
passive VWAP and aggressive shortfall algorithms. VWAP
algorithms should outperform shortfall algorithms when
EH-price is low and the reverse when EH-price is high.
Using our shortfall surprises framework, for VWAP executions, the shortfall surprise should be negative (actual
less than expected) when EH-price is low and positive
when EH-price is high and the reverse for the shortfall
executions. In doing this comparison, we do not have to
control for order size, stock capitalization, or the other
“usual suspects,” because expected shortfall already reflects
these factors.
We can also use the shortfall surprises framework
to quantify the value-added of high-touch executions.
The rise of low-touch trading has created a pressing question: why pay more than 12 bps in commissions to use
the high-touch if one can execute for less than 3 bps in
commissions low-touch? The answer, of course, is that for
difficult orders, high-touch executions may add value by
SUMMER 2007
reducing the indirect costs of trading: liquidity impact and
opportunity cost. But to decide between high-touch and
low-touch, we must explicitly quantify high-touch valueadded, and determine the order difficulty cut-off, above
which it is worth executing high-touch. In future work,
we plan to use our shortfall surprises framework to address
these questions.
APPENDIX
MODEL MISSPECIFICATION
AND THE SHORTFALL SURPRISE
In this Appendix, we formally decompose the shortfall surprise into the surprise caused by model misspecification (e.g.,
omitted factors), and the component caused by the surprise in the
included factors. Suppose the true model for actual shortfall is:
SA = α + β Y + γ XA + δ Z + PA + u
where Y = factors known with certainty pre-trade, XA =
actual values of factors not known with certainty pre-trade,
Z = factors not included in the liquidity impact model, PA =
actual execution-horizon price move, and u is a random
variable.
Suppose the estimated liquidity impact model is:
SE = a + b Y + c XE + PE
The other using the actual values of the X and P factors
(perfect-foresight estimates):
SEA = a + b Y + c XA + PA
The shortfall surprise is SS = SA – SE, and by adding and
subtracting SEA, we decompose SS into two components, SS1
and SS2:
SS = ( SA – SEA) – (SEA – SE) = SS1 – SS2
But
SS1 = SA – SEA = (α + β Y + γ XA + δ Z + PA + u )
– (a + b Y + c XA + PA)
Re-arranging:
SS1 = (α – a) + (β - b) Y + (γ – c) XA + δ Z + u
If the impact model is not misspecified a = α, b = β, c = γ, δ =
0, and SS1 = u, a random variable.
The shortfall surprise component SS1, therefore, measures the extent the shortfall surprise SS is caused by other factors (model misspecification). Similarly:
SS2 = SEA – SE = c (XA – XE) + (PA – PE)
L=a+bY+cX
Post-trade, we can generate two estimates from the impact
model. One using the expected values of the X and P factors:
The shortfall surprise component, SS2, therefore, measures the extent the shortfall surprise SS is caused by the factor
surprises (what we are testing in this article).
EXHIBIT A1
Further Disaggregating the Shortfall Surprise1
1
Sample period June 1 through December 31, 2006; 241,610 orders.
SUMMER 2007
THE JOURNAL OF TRADING
29
Using the Goldman Sachs t-cost model and the data in
our sample, we estimated both SE and the perfect foresight
shortfall SEA. We then decomposed the shortfall surprise into
the other-factors component (SS1), and the factor-surprise component (SS2), and estimated the shortfall surprise model only on
the SS2 component.
Exhibit A1 shows our empirical results using this decomposition. Our estimated other-factors component SS1 accounts
for 56% of the shortfall surprise, exactly matching the “unexplained” component in our one-step procedure in Exhibit 1.
In the regression of the factor-surprises component SS2, on the
five factors, we get an R-square of 93%, and the overall
explanatory power of each of the five surprises is similar to
the results from our one-step procedure (Exhibit 1). For
example, the alpha surprise explains 80% of the variation in
SS2, which itself accounts for 44% of the overall shortfall-surprise variation; so in the two-step approach, the alpha surprise explains 35% of the overall shortfall compared to 38%
in the one-step procedure.
ENDNOTES
The authors thank Jeff Bacidore, Tianwu (Michael) Cai,
Barbara Dunn, Oliver Hansch, Kilian Mie, and Ingrid Tierens,
all at Goldman, Sachs & Co., for their comments.
1
See Sofianos [2006] for a discussion of execution shortfall.
2
Orders that arrive before the market opens (9:30 am),
we strike at the opening price.
3
Liquidity impact will never be negative for liquidityseeking orders. Liquidity-providing orders, for e.g., limit orders,
conditional on execution, may have a negative impact (save the
spread).
4
For more details on the model, see Rodella [2005].
5
Only if the trader had executed all 60,000 shares at 14:00
would the execution horizon and the execution turnaround
time be the same (two hours).
6
This allocation procedure assumes the alpha is growing
linearly from order arrival to market close.
7
We exclude limit orders, held orders, and any orders where
Goldman Sachs provided capital at the request of the client.
8
NASDAQ GS, NMS, and Small Cap.
9
The sample also includes a few AMEX stocks.
10
We define participation rate as the executed quantity as
a percent of the actual overall market trading volume in that
stock from strike time (order arrival time) to the time of the
last execution.
11
We use only firm and valid NBBO quotes in our
calculation.
12
We tried, for example, several different ways of specifying the volume surprise: actual minus expected volume, the
log of actual divided by expected (the specification we use in
our analysis), the actual participation minus the expected par-
30
SHORTFALL SURPRISES
ticipation, etc. Moving expected shortfall to the right hand side
of the regressions also does not make much difference; it has a
coefficient of one.
13
These results on the coefficients of EH-market and EHalpha are robust across all specifications and sub-samples that we
estimated.
14
We define the RMSE as the square root of 1/N ∑(SAi –
SEi)2, where SA is the post-trade actual shortfall and SE is the pretrade expected shortfall for each order.
15
At the same time, however, the larger the trade, the
higher the likelihood that the trade was generated by a strong
alpha signal (the true EH-alpha is higher).
16
For further discussion of this important point, see
Rakhlin and Sofianos [2006], pp. 43–45.
17
In this article, we focus on agency executions. Volatility
surprises may be more important for capital commitment
(principal) trades. In a capital commitment trade, the dealer
prices volatility (execution risk) in the price premium/discount. So an increase in volatility will make principal trades
more expensive.
18
For examples of our use of the decomposition of shortfall into liquidity impact and price loss, see Cai and Sofianos
[2006] and Rakhlin and Sofianos [2006].
19
For further discussion of this point, see Sofianos [2006].
20
For example, the Goldman Sachs single-stock 4Cast
and portfolio PortX algorithms. See Rakhlin and Sofianos
[2006a, 2006b] for descriptions of these two algorithms.
21
Bacidore and Sofianos [2002, 2003] through Cai and
Sofianos [2006].
REFERENCES
Bacidore, Jeffrey and George Sofianos. “Evaluating Execution
Quality for Large Orders.” Goldman Sachs Trading and Market
Structure Report (2002).
——. “Choosing the Best Order Execution Strategy.” Goldman
Sachs Trading and Market Structure Report (2003).
Cai, Tianwu (Michael) and George Sofianos. “Multi-day Executions.” Journal of Trading, Vol. 1, No. 3 (2006), pp. 25–33.
Rakhlin, Dmitry and George Sofianos. “The Choice of Execution Algorithm: VWAP or Shortfall.” Journal of Trading,
Vol. 1, No. 1 (2006a), pp. 26–32.
——. “The Impact of an Increase in Volatility on Trading
Costs.” Journal of Trading, Vol. 1, No. 2 (2006b), pp. 43–50.
Rodella, Elena. “Introducing Cost Wizard: Comparing TwoHour and All-Day Executions.” Goldman Sachs Cost Wizard
Reports, Issue 1 (May 12, 2005).
SUMMER 2007
Sofianos, George. “Execution Benchmarks: VWAP or Pretrade Prices.” Journal of Trading, Vol. 1, No. 1 (2006), pp. 22–25.
To order reprints of this article, please contact Dewey Palmieri at
dpalmieri@iijournals.com or 212-224-3675
This material was prepared by the Goldman Sachs Equity Execution
Strategies Group and is not the product of Goldman Sachs Global Investment
Research. It is not a research report and should not be construed as such.
The information in this article has been taken from trade data and
other sources we deem reliable, but we do not represent that such information is accurate or complete, and it should not be relied upon as such. This
information is indicative, based on among other things, market conditions at
the time of writing, and is subject to change without notice. Goldman Sachs’
SUMMER 2007
algorithmic models derive pricing and trading estimates based on historical
volume patterns, real-time market data, and parameters selected by the GSAT
user. The ability of Goldman Sachs’ algorithmic models to achieve the performance described in this article may be impacted by changes in market conditions, systems or communications failures, etc. Finally, factors such as order
quantity, liquidity, and the parameters selected by the GSAT user, may impact
the performance results.
The opinions expressed in this article are those of the authors and do
not necessarily represent the views of Goldman, Sachs & Co. These opinions
represent the authors’ judgment at this date and are subject to change.
Goldman, Sachs & Co. is not soliciting any action based on this article.
It is for general information and does not constitute a personal recommendation or take into account the particular investment objectives, financial situations, or needs, of individual users. Before acting on any advice or
recommendation in this article, users should consider whether it is suitable for
their particular circumstances.
Copyright: Summer 2007, Goldman, Sachs & Co.
THE JOURNAL OF TRADING
31