Taming Uncertainty: All You Need to Know About
Measuring Forecast Accuracy, But Are Afraid to
Ask
“Television won’t be able to hold on to any market it captures after the first six months. People will soon
get tired of staring at a plywood box every night.”
A WELL-KNOWN MOVIE MOGUL IN THE EARLY DAYS OF TV.
This White Paper 08 describes
• Why it is a necessary to define first what a forecast error is
• What bias and precision means for accuracy measurement
• How, when and why to make accuracy measurements
• The systematic steps in a forecast evaluation process.
After reading this paper, you should be able to
• Understand the difference between fitting errors and forecast errors
• Recognize that there is no one best measure of accuracy for products, customers and hierarchies,
industry forecasts and time horizons
• Realize that simple averaging is not a best practice for summarizing accuracy measurements
• Engage with potential users of demand forecasts to clearly define their forecast accuracy
requirements
The Need for Measuring Forecast Accuracy
The Institutional Broker's Estimate System (I/B/E/S), a service that tracks financial analysts’ estimates,
reported that forecasts of corporate yearly earnings that are made early in the year are persistently
optimistic, that is to say, upwardly biased. In all but 2 of the 12 years studied (1979 - 1991), analysts
revised their earnings forecasts downward by the end of the year. In a New York Times article (January 10,
1992), Jonathan Fuerbringer writes that this pattern of revisions was so clear that the I/B/E/S urged stock
market investors to take early-in-the year earnings forecasts with a grain of salt.
It is generally recognized in most organizations that accurate forecasts are essential in achieving
measurable improvements in their operations. In demand forecasting and planning organizations, analysts
use accuracy measures for evaluating their forecast models on an ongoing basis, and hence accuracy is
something that needs to be measured at all levels of a product and customer forecasting database. Some of
the questions you will need to ask include:
•
•
•
Will the model prove reliable for forecasting units or revenues over a planned forecast horizon,
such as item-level product demand for the next 12 weeks or aggregate revenue demand for the next
4 quarters?
Will a forecast have a significant impact on marketing, sales, budgeting, logistics and production
activities?
Will a forecast have an effect on inventory investment or customer service?
In a nutshell, inaccurate forecasts can have a direct effect on setting inadequate safety stocks, ongoing
capacity problems, massive rescheduling of manufacturing plans, chronic late shipments to customers, and
2
adding expensive manufacturing flexibility resources (Exhibit 1). In addition, the Income Statement and
Balance Sheet are also impacted by poor demand forecasts in financial planning (Exhibit 2).
Operational Impact of Poor Forecasts
Poor Forecast
Stock outs
And Late
Orders
Inaccurate
Production
Schedules
Schedule
Changes
Raw Material
Expediting
Loss of volume breaks
Higher freight costs
Higher raw material
inventory
Part deliveries
Inter-warehouse
transfers
Lost Sales
Yield losses
Productivity losses
Higher WIP inventory
Higher WIP inventory
Reduced
Revenues
Higher Shipping
Costs
Increased
Safety Stock
Increased Finished Goods
Inventory
Higher Warehousing Costs
© 2008 - 2015 CPDF_Delphus, Inc.
Source: C. Conradi – SYSPRO 2013
5
Exhibit 1 Operational Impact of Poor Forecasts (Source: S. C. Conradie, Noeticconsulting 2013)
Financial Impact of Poor Demand Forecasts
Sales
Income
Statement
Costs
Assets
Balance
Sheet
Liabilities
© 2008 - 2015 CPDF_Delphus, Inc.
Source: C. Conradi – SYSPRO 2013
Service levels
Out of stocks
Reliability
Logistics costs
Inventory Levels
Obsolete stocks
Direct distribution costs
Inventories
Cash
Fixed Assets
Short term debt
Long term debt
6
Exhibit 2 Financial Impact of Poor Forecasts (Source: S. C. Conradie, Noeticconsulting 2013)
3
Analyzing Forecast Errors
Whether a method that has provided a good fit to historical data will also yield accurate forecasts for future
time periods is an unsettled issue. Intuition suggests that this may not necessarily be the case. There is no
guarantee that past patterns will persist in future periods. For a forecasting technique to be useful, we must
demonstrate that it can forecast reliably and with consistent accuracy. It is not sufficient to simply produce a
model that performs well only in historical (within sample) fit periods. One must also measure forecasting
performance with multiple metrics, while not just with a single, commonly used metric, like the Mean
Absolute Percentage Error (MAPE) or even a weighted MAPE (to be defined precisely later in this paper).
At the same time, the users of a forecast need forecasting results on a timely basis. Using a forecasting
technique and waiting one or more periods for history to unfold in future periods is not practical because
our advice as forecasters will not be timely. Consequently, we advocate the use of holdout test periods for
the analysis and evaluation of forecast errors on an ongoing basis.
Generally, the use of a single measure of accuracy involving simple averages is not a best practice
as you will misinterpret the results when underlying distributions are not normal. Even with approximate
normality a simple average of numbers may lead to misleading uses. With minimal effort, you can
complement a traditional measure with an outlier-resistant alternative to safeguard against the common
presence of an outlier or unusual values in accuracy measurements. It should always be good practice to
report accuracy measures in pairs to insure against misleading results. When the pair are similar (in a
practical sense), then report the traditional result. Otherwise, go back and investigate the data going into the
calculation, using domain information whenever possible.
Two important aspects of forecast accuracy measurement are bias and precision. Bias is a problem
of direction: Forecasts are typically too low (downward bias) or typically too high (upward bias). Precision
is an issue of magnitudes: Forecast errors can be too large (in either direction) using a particular forecasting
technique. Consider first a simple situation - forecasting a single product or item. The attributes that should
satisfy most forecasters include lack of serious bias, acceptable precision, and superiority over naive
models.
Lack of Bias
If forecasts are typically too low, we say that they are downwardly biased; if too high, they are upwardly
biased. If overforecasts and underforecasts tend to cancel one another out (i.e., if an average of the forecast
errors is approximately zero), we say that the forecasts are unbiased.
Bias refers to the tendency of a forecast to be predominantly toward one side of the truth.
Bias is a problem of direction. If we think of forecasting as aiming darts at a target, then a bias
implies that the aim is off-center. That is, the darts land repeatedly toward the same side of the target
(Exhibit 3). In contrast, if forecasts are unbiased, they are evenly distributed around the target.
Bias and Precision
Unbiased
Biased
Precision
© 2008 - 2015 Delphus, Inc.
Exhibit 3 Biased, Unbiased and Precise Forecasts
7
4
What is an Acceptable Precision?
Imprecision is a problem if the forecast errors tend to be too large. Exhibit 4 shows three patterns that differ
in terms of precision. The upper two forecasts are the less precise - as a group, they are farther from the
target. If bias is thought of as bad aim, then imprecision is a lack of constancy or steadiness. The precise
forecast is generally right around the target.
Precision refers to the distance between the forecasts as a result of using a particular forecasting
technique and the corresponding actual values.
Forecasts and Forecast Errors
© 2008 - 2015 Delphus, Inc.
8
Exhibit 4 (a) Bar Charts and (b) Tables Showing Actuals (A), Forecasts and Forecast Errors for Three
Forecasting Techniques
Exhibit 4 illustrates the measurement of bias and precision for three hypothetical forecasting
techniques. In each case, the fit period is periods 1 - 20. Shown in the top row of Exhibit 4b are actual
values for the last four periods (21 – 24). The other three rows contain forecasts using forecasting
techniques X, Y, and Z. These are shown in a bar chart in Exhibit 4. What can we say about how good these
forecast models are? On the graph, the three forecasts do not look all that different.
But, what is a forecast error? It is not unusual to find different definitions and interpretations for
concepts surrounding accuracy measurement among analysts, planners and managers in the same company.
Exhibit 4 records the deviations between the actuals and their forecasts. Each deviation represents a
forecast error (or forecast miss) for the associated period defined by:
Forecast error (E) = Actual (A) - Forecast (F)
If (F – A) is the preferred use in some organizations but not others, then demand forecasters and
planners should name it something else, like forecast variance, a more conventional meaning for revenueoriented planners. The distinction is important because of the interpretation of bias in under- and
overforecasting situations. Consistency is paramount here.
Contrast this with a fitting error (or residual) of a model over a fit period, which is:
Fitting error = Actual (A) - Model fit
5
Forecast error) is a measure of forecast accuracy. By convention, it is defined as A – F. On the other
hand, Fitting error (or residual) is a measure of model adequacy: Actual – Model Fit.
In Exhibit 4b, the forecast error shown for technique X is 1.8 in period 21. This represents the
deviation between the actual value in forecast period 21 (= 79.6) and the forecast using technique X (=
77.8). In forecast period 22, the forecast using technique X was lower than actual value for that period
resulting in a forecast error of 6.9. The period 24 forecast using technique Z was higher than that period's
actual value; hence, the forecast error is negative (- 3.4). When we overforecast, we must make a negative
adjustment to reach the actual value. Note that if the forecast is less than the actual value, the miss is a
positive number; if the forecast is more than the actual value, the miss is a negative number.
In this example, it appears that Model X is underforecasting (positive forecast errors), hence it is
biased. Model Z appears to have a forecast (#22) that should be market for investigation; it is somewhat
large relative to the rest of the numbers.
To identify patterns of upward- and downward-biased forecasts, we start by comparing the number
of positive and negative misses. In Exhibit 4b, Technique X under-forecasts in all four periods, indicative of
a persistent (downward) bias. Technique Y underforecasts and overforecasts with equal frequency;
therefore, it exhibits no evidence of bias in either direction. Technique Z is biased slightly toward
overforecasting. As one measure of forecast accuracy (Exhibit 5), we calculate a percentage error PE =
100% * (A – F)/A.
Forecast Percentage Error
© 2008 - 2015 Delphus, Inc.
9
Exhibit 5 (a) Bar chart and (b) Table of the Forecast Error as Percentage Error (PE) between Actuals and
Forecasts for Three Techniques
To reduce bias in a forecasting technique, we can either (1) reject any technique that projects with
serious bias in favor of a less-biased alternative (after we have first compared the precision and complexity
of the methods under consideration) or (2) investigate the pattern of bias in the hope of devising a bias
adjustment; for example, we might take the forecasts from method X and adjust them upward to try to offset
the tendency of this method to underforecast. Also, forecasts for Techniques Y and Z could be averaged
after placing Forecast X aside for the current forecasting cycle.
6
Ways to Evaluate Accuracy
A number of forecasting competitions were conducted by the International Institute
of Forecasters (IIF; www.forecasters.org) to assess the effectiveness of statistical
forecasting techniques and determine which techniques are among the best. Starting
with the original M competition in 1982, Spyros Makridakis and his co-authors
reported results in the Journal of Forecasting, in which they compared the accuracy
of about 20 forecasting techniques across a sample of 111 time series – a very small
dataset by today’s standards. A subset of the methods was tested on 1001 time series,
still not very large, but substantial enough for the computer processing power at the
time. The last 12 months of each series were held out and the remaining data were
used for model fitting. Using a range of measures on a holdout sample, the
International Journal of Forecasting (IJF) conducted a competition in 1997 comparing a range of
forecasting techniques across a sample of 3003 time series. Known as the M3 competition, these data and
results can be found at the website www.maths.monash.edu.au/∼hyndman/forecasting/. The IJF
published a number of other papers evaluating the results of these competitions. These competitions have
become the basis for how we should measure forecast accuracy in practice.
Forecast accuracy measurements are performed in order to assess the accuracy of a forecasting
technique.
The Fit Period versus the Test Period
In measuring forecast accuracy, a portion of the historical data is withheld, and reserved for evaluating
forecast accuracy. Thus, the historical data are first divided into two parts: an initial segment (the fit
period) and a later segment (the holdout sample). The fit period is used to develop a forecasting model.
Sometimes, the fit period is called the within-sample training, initialization, or calibration period. Next,
using a particular model for the fit period, model forecasts are made for the later segment. Finally, the
accuracy of these forecasts is determined by comparing the projected values with the data in the holdout
sample. The time period over which forecast accuracy is evaluated is called the test period, validation
period, or holdout-sample period.
A forecast accuracy test should be performed with data from a holdout sample.
When implementing software or spreadsheet code, we can minimize confusion and
misinterpretation if a common notation for the concepts in these calculations is used. Let Yt (1) denote the a
one-period-ahead forecast of Yt and Yt (m) the m-period-ahead forecast. For example, a forecast for t = 25
that is the one-period-ahead forecast made at t = 24 is denoted by Y24 (1) and the a two-period-ahead
forecast for the same time, t = 25, made at t = 23 is denoted by Y23(2). Generally, forecasters are interested
in multi period-ahead forecasts because lead times longer than one period are required for business to act
on a forecast. We use the following conventions for a time series:
{Yt , t = 1,2, …, T}
the historical dataset up to and including period t = T
{Ŷ t , t =1,2, . . . , T}
the data set of fitted values that result from fitting a model to historical
data
YT+m
the future value of Yt , m periods after t = T
{YT (1), YT (2), …, YT (m)}, the one- to m-period ahead-forecasts made from t = T
Consequently, we see that forecast errors and fit errors (residuals) refer to:
7
Y1 - Ŷ t , Y2 - Ŷ 2 , . . . , YT - Ŷ T the fit errors or residuals from a fit
YT+1 - YT (1), YT+2 - YT (2), . . . , YT+m - YT(m) the forecast errors (not to be confused with the
residuals from a fit)
In dealing with forecast accuracy, it is important to distinguish between forecast errors and fitting
errors.
Goodness-of-Fit versus Forecast Accuracy
We need to assess forecast accuracy rather than just calculate overall goodness-of-fit statistics for the
following reasons:
• Goodness-of-fit statistics may appear to give better results than forecasting-based calculations, but
goodness-of-fit statistics measure model adequacy over a fitting period that may not be
representative of the forecasting period.
• When a model is fit, it is designed to reproduce the historical patterns as closely as possible. This
may create complexities in the model that capture insignificant patterns in the historical data,
which may lead to over-fitting.
• By adding complexity, we may not realize that insignificant patterns in the past are unlikely to
persist into the future. More important, the more subtle patterns of the future are unlikely to have
revealed themselves in the past.
• Exponential smoothing models are based on updating procedures in which each forecast is made
from smoothed values in the immediate past. For these models, goodness-of-fit is measured from
forecast errors made in estimating the next time period ahead from the current time period. These
are called one-period-ahead forecast errors (also called one-step-ahead forecast errors). Because it
is reasonable to expect that errors in forecasting the more distant future will be larger than those
made in forecasting the next period into the future, we should avoid accuracy assessments based
exclusively on one-period-ahead errors.
When assessing forecast accuracy, we may want to know about likely errors in forecasting more than
one period-ahead.
Item-Level versus Aggregate Performance
Forecast evaluations are also useful in multi-series comparisons. Production and inventory managers
typically need demand or shipment forecasts for hundreds to tens of thousands of items (SKUs) based on
historical data for each item. Financial forecasters need to issue forecasts for dozens of budget categories in
a strategic plan on the basis of past values of each source of revenue. In a multi-series comparison, the
forecaster should appraise the method based not only on its performance for the individual item but also on
the basis of its overall accuracy when tested over various summaries of the data. How to do that most
effectively will depend on the context of the forecasting problem and can, in general, not be determined a
priori. In the next section, we discuss various measures of forecast accuracy.
Absolute Errors versus Squared Errors
The use of absolute errors and that based on squared errors are both useful. There is a good argument for
consistency in the sense that a model's forecasting accuracy should be evaluated on the same basis used to
develop (fit) the model. The standard basis for model fit is the least-squares criterion that is minimizing the
mean-square-error MSE between the actual and fitted values. To be consistent, we should evaluate forecast
accuracy based on squared error measures, such as the root mean squared error (RMSE). It is useful to test
how well different methods do in forecasting using a variety of accuracy measures. Forecasts are put to a
wide variety of uses in any organization, and no single forecaster can dictate on how they must be used and
interpreted.
8
Sometimes costs or losses due to forecast errors are in direct proportion to the size of the error double the error leads to double the cost. For example, when a soft drink distributor realized that the costs
of shipping its product between distribution centers was becoming prohibitive, it made a study of the
relationship between under-forecasting (not enough of the right product at the right place, thus requiring a
transshipment, or backhaul, from another distribution center) and the cost of those backhauls. As shown in
Exhibit 6, over-forecasts of 25% or higher appeared strongly related to an increase in the backhaul of
pallets of product. In this case, the measures based on absolute errors are more appropriate. In other cases,
small forecast errors do not cause much harm and large errors may be devastating; then, we would want to
stress the importance of (avoidance of) large errors, which is what squared-error measures accomplish.
Exhibit 6 Backhauls in Pallets versus Forecast Error
Measures of Forecast Accuracy
Measures of bias
There are two common measures of bias. The mean error (ME) is the sum of the forecast errors divided by
the number of periods in the forecast horizon (h) for which forecasts were made:
ME = [∑ (At - Ft )] / h
= [∑ (Ai - Fi )] / h
(sum from t = T+1 to t = T+h)
sum from i = 1 to i = h
The mean percentage error (MPE) is
MPE = 100 * [∑ (At - Ft ) / At ] / h
sum from t = T+1 to t = T+h
= 100 * [∑ (Ai - Fi ) / Ai ] / h
sum from i = 1 to i = h
The MPE and ME are useful supplements to a count of the frequency of under- and over-forecasts.
The ME gives the average of the forecast errors expressed in the units of measurement of the data; the MPE
gives the average of the forecast errors in terms of percentage and is unit-free.
If we examine the hypothetical example in Exhibits 7 and 8, we find that for technique X ME =
3.8, and MPE = 4.2%. This means that, technique X, we underforecast by an average of 3.8 units per
period. Restated in terms of the MPE, this means that the forecasts from technique X were below the actual
values by an average of 4.2%. A positive value for the ME or MPE signals downward bias in the forecasts
using technique X.
The two most common measures of bias are the mean error (ME) and the mean percentage error
(MPE).
9
Compared to technique X, the ME and MPE for technique Y are much smaller. These results are
not surprising. Because technique Y resulted in an equal number of under-forecasts and overforecasts, we
expect to find that the average error is close to zero.
On the other hand, the low ME and MPE values for technique Z are surprising. We have
previously seen that technique Z over-forecasts in three of the four periods. A closer look at the errors from
technique Z shows that there was a very large under-forecast in one period and relatively small overforecasts in the other three periods. Thus, the one large underforecast offset the three small over-forecasts,
yielding a mean error close to 0. The lesson here is that averages can be distorted by one or just a few
unusually large errors. One should consider calculating non-conventional measures like the Median Error
MdE and the Median Percentage Error MdPE. We should always use multiple error measures as a
supplement to the analysis of the individual errors, as unusual values can readily distort an incorrect
interpretation.
Measures of Precision
Certain indicators of precision are based on the absolute values of the forecast errors. By taking an absolute
value, we eliminate the possibility that under-forecasts and over-forecasts negate one another. Therefore, an
average of the absolute forecast errors reveals simply how far apart the forecasts are from the actual values.
It does not tell us if the forecasts are biased. The absolute errors are shown in Exhibit 10.9, and the absolute
values of the percentage errors are shown in Exhibit 7.
Absolute Error
© 2008 - 2015 Delphus, Inc.
13
Exhibit 7 (a) Bar chart and (b) Table Showing Forecast Error as the Absolute Difference |E| between Actuals
and Forecasts for Three Techniques
10
Absolute Percentage Error
© 2008 - 2015 Delphus, Inc.
14
Exhibit 8 (a) Bar chart and (b) Table Showing Forecast Error as the Absolute Percentage Difference |PE(%)|
between Actuals and Forecasts for Three Techniques.
|(PE) | = 100 • | (A - F)| / A |
The most common averages of absolute values are mean absolute error (MAE), mean absolute
percentage error (MAPE), and median absolute percentage error (MdAPE).
MAE = [∑ | (At - Ft ) | ] / h
= [∑ | (Ai - Fi ) | ] / h
sum t = T + 1 to t = T + h
sum from i = 1 to i = h
MAPE = 100 * [∑ | (At- Ft )| / At ] / h
= 100 * [∑ | (Ai - Fi )| / Ai ] / h
sum t = T + 1 to t = T+ h
sum from i = 1 to i = h
MdAPE = Median value of {| (Ai - Fi )| / Ai | i = 1 , . . . h }
Interpretations of the averages of absolute error measures are straightforward. In Exhibit 7, we
calculate that the MAE is 4.6 for technique Z, from which we can conclude that the forecast errors from this
technique average 4.6 per period. The MAPE, from Exhibit 8, is 5.2%, which tells us that the period
forecast errors average 5.2%.
The MdAPE for technique Z, from Exhibit 9, is approximately 4.1% (the average of the two
middle values: 4.4 and 3.8). Thus, half the time the forecast errors exceeded 4.1%, and half the time they
were smaller than 4.1%. When there is a serious outlier among the forecast errors, as with technique Z, it is
useful to calculate the MdAPE in addition to the MAPE because medians are less sensitive (more resistant)
than mean values to distortion from outliers. This is why the MdAPE is a full percentage point below the
MAPE for technique Z. Sometimes, as with technique Y, the MdAPE and the MAPE are virtually identical.
In this case, we can report the MAPE because it is the far more common measure.
11
Exhibit 9 (a) Bar chart and (b) Table Showing Forecast Error as Squared Difference (E2) between Actuals and
Forecasts for Three Techniques
Exhibit 10 (a) Bar chart and (b) Table Showing Forecast Error as Squared Percentage Difference PE(%))2
between Actuals and Forecasts for Three Techniques
In addition to the indicators of precision calculated from the absolute errors, certain measures are
commonly used that are based on the squared values of the forecast errors (Exhibit 10), such as the MSE
and RMSE. (RMSE is the square root of MSE.) A notable variant on RMSE is the root mean squared
percentage error (RMSPE), which is based on the squared percentage errors (Exhibit 10).
MSE = [∑ (At - Ft )2 ] / h
= [∑ (Ai- Fi )2 ] / h
sum from t = T+1 to t = T+h
sum from i = 1 to i = h
RMSPE = √ 100 * [∑ {(At- Ft ) / At }2 ] / h
= 100 * [∑ {(Ai - Fi ) / Ai }2 ] / h
sum t = T+1 to t = T+h
sum from i = 1 to i = h
To calculate RMSPE, we square each percentage error in Exhibit 10. The squares are then averaged, and
the square root is taken of the average.
The MSE is expressed in the square of the units of the data. This may make it difficult to interpret
when referring to dollar volumes, for example. By taking the square root of the MSE, we return to the units
of measurement of the original data.
We can simplify the interpretation of RMSE and present it as if it were the mean absolute error
(MAE); the forecasts of technique Y are in error by an average of 4. There is little real harm in doing this;
just keep in mind that the RMSE is generally somewhat larger than the MAE. (compare Exhibits 8 and 10).
12
Some forecasters present the RMSE within the context of a prediction interval. Here is how it
might sound: "Assuming the forecast errors are normally distributed, we can be approximately 68% sure
that technique Y's forecasts will be accurate to within 4.0 on the average." However, it is best to not to make
these statements too precise because it requires us to assume a normal distribution of forecast errors, a rarity
in demand forecasting demand forecasting practice.
The RMSE can be interpreted as a standard error of the forecasts.
The RMSPE is the percentage version of the RMSE (just as the MAPE is the percentage version of
the MAE). Because the RMPSE for technique Y equals 4.5%, we may suggest that "the forecasts of
technique Y have a standard or average error of approximately 4.5%."
If we desire a squared error measure in percentage terms, the following shortcut is sometimes
taken. Divide the RMSE by the mean value of the data over the forecast horizon. The result can be
interpreted as the standard error as a percentage of the mean of the data and is called the coefficient of
variation (because it is a ratio of a standard deviation to a mean). Based on technique Y, average actual
values over periods 21 - 24 were 88.2 (Exhibit 1), and the RMSE for technique Y was found to be 4.0
(Exhibit 10). Hence, the coefficient of variation is 4/88.2 = 4.5%, a result that is virtually identical to the
RMSPE, in this case.
Comparison with Naive Techniques
After reviewing the evidence of bias and precision, we may want to select technique Y as the preferred of
the three candidates. Based on the evaluation, technique Y's forecast for periods 21 - 24 shows no indication
of bias - two under-forecasts and two over-forecasts - and proves slightly more precise than the other two
techniques, no matter which measure of precision is used. The MAE for technique Y reveals that the
forecast errors over periods 21 - 24 average approximately 3.5 per period, and the MAPE indicates that
these errors come to just under 4% on average.
To provide additional perspective on technique Y, we might contrast technique Y's forecasting
record with that of a very simplistic procedure, one requiring little or no thought or effort. Such procedures
are called naive techniques. The continued time and effort investment in technique Y might not be worth the
cost if it can be shown to perform no better than a naïve technique. Sometimes the contrast with a naive
technique is sobering. The forecaster thinks an excellent model has been developed and finds that it barely
outperforms a naive technique.
If the forecasting performance of technique Y is no better or is worse than that of a naive technique, it
suggests that in developing technique Y we have not accomplished very much.
A standard naive forecasting technique has emerged for data that are yearly or otherwise lack
seasonality; it is called a NAÏVE_1 or Naive Forecast1 (NF1). We see later that there are also naïve
forecasting techniques for seasonal data. A NAÏVE_1 forecast of the following time period is simply the
value in effect during this time period. Alternatively stated, the forecast for any one time period is the value
observed during the previous time period.
A NAIVE_1 is called a no change forecasting technique, because its forecasted value is unchanged from
the previously observed value.
Exhibit 11 Actuals and NAIVE_1 (one-period-ahead) Forecasts for Technique Y. (Exhibit 3)
13
Exhibit 11 shows the forecast for our example (Exhibit 3). Note that one-period-ahead forecasts
for periods 22 - 24 are the respective actuals for periods 21 - 23. The forecast for period 21 (=73.5) is the
actual value for period 20. When we appraise NAIVE_1's performance, we note it is biased. It
underestimated periods 21 - 23 and overestimated period 24. By always projecting no change, a NAIVE_1
will underforecasts whenever data increase and over-forecast whenever the data are declining. We do not
expect a NAIVE_1 to work very well when the data are steadily trending in one direction.
The error measures for NAIVE_1 in Exhibit 11 are all higher than the corresponding error
measures for technique Y, indicating that technique Y was more accurate than NAIVE_1. By looking at the
relative error measures, we can be more specific about the advantage of technique Y over NAIVE_1.
Because a naive technique has no prescription for a random error term, the NAIVE_1 technique
can be viewed as a no-change, no-chance method (in the sense of our forecasting objectives) to predict
change in the presence of uncertainty (chance). Hence, it is useful as a benchmark-forecasting model to
beat.
Relative Error Measures
A relative error measure is a ratio; the numerator is a measure of the precision of a forecasting method;
the denominator is the analogous measure for a naive technique.
Exhibit 12 Relative Error Measures for Technique Y Relative to NAIVE_1. (Exhibit 3)
Exhibit 12 provides a compendium of relative error measures that compare technique Y with the
NAIVE_1. The first entry (=0.73) is the ratio of the MAE for technique Y to the MAE for the NAIVE_1
(3.55/4.87 = 0.73). It is called the relative absolute error (RAE). The RAE shows that the average forecast
error using technique Y is 73% of the average error using NAIVE_1. This implies a 27% improvement of
technique Y over NAIVE_1.
Expressed as an improvement score, the relative error measure is sometimes called a forecast
coefficient (FC), as listed in the last column of Exhibit 12. When FC = 0, this indicates that the technique in
question was no better than NAIVE_1 technique. When FC < 0, the technique was in fact less precise than
the NAIVE_1, whereas when FC > 0 the forecasting technique was more precise than NAIVE_1. When FC
= 1, the forecasting technique is perfect.
The second row of Exhibit 12 shows the relative error measures based on the MAPE. The results
almost exactly mirror those based on the MAE, and the interpretations are identical. The relative error
measure based on the RMSE (which is the RMSE of technique Y divided by RMSE of the NAIVE_1) was
originally proposed by in 1966 by the economist Henri Theil and it is known as Theil's U or Theil's U2
statistic. As with the RAE, a score less that 1 suggests an improvement over the naive technique.
The relative errors based on the squared-error measures, RMSE and RMSPE, suggest that
technique Y in Exhibit 12 was about 40% more precise than the NAIVE_1. The relative errors based on
MAE and MAPE put the improvement at approximately 25 – 30%. Why the difference? Error measures
based on absolute values, such as the MAE and MAPE, give a weight (emphasis) to each time period in
proportion to the size of the forecast error. The RMSE and RMSPE, in contrast, are measures based on
squared errors and squaring gives more weight to large errors. A closer look at Exhibit 11 shows that the
NAIVE_1 commits a very large error in forecast period 22.
Predictive Visualization Techniques
Predictive Visualization techniques for forecast accuracy measurement provide a perspective on how
serious a forecasting bias may be. If the bias exceeds a designated threshold, a "red flag" is raised
concerning the forecasting approach. The forecaster should then reexamine the data to identify changes in
14
the trend, seasonal, or other business patterns, which in turn suggest some adjustment to the forecasting
approach.
Ladder Charts
By plotting the current year’s monthly forecasts on a ladder chart, a forecaster can determine whether the
seasonal pattern in the forecast looks reasonable. The ladder chart in Exhibit 13 consists of six items of
information for each month of the year: average over the past 5 years, the past year’s performance, the 5year low, the 5-year high, the current year-to-date, and the monthly forecasts for the remainder of the year.
The 5-year average line usually provides the best indication of the seasonal pattern, assuming this pattern is
not changing significantly over time. In fact, this is a good check for reasonableness that can be done before
submitting the forecast for approval.
A ladder chart is a simple yet powerful tool for monitoring forecast results.
The level of the forecast can be checked for reasonableness relative to the prior year (dashed line
in Exhibit 13). The forecaster can determine whether or not the actuals are consistently overrunning or
underrunning the forecast. In this example, the forecast errors are positive for 3 months and negative for 3
months. The greatest differences between the actual and forecast values occur in March and April, but here
the deviations are of opposite signs; additional research should be done to uncover the cause of the unusual
March-April pattern. The forecasts for the remainder of the current year look reasonable, although some
minor adjustments might be made. The ladder chart is one of the best predictive visualization tools for
quickly identifying the need for major changes in forecasts.
Exhibit 13 Ladder Chart
Prediction-Realization Diagram
Another useful visual approach to monitoring forecast accuracy is the predictionrealization diagram introduced by the economist Henri Theil in 1958. If the
predicted values are indicated on the vertical axis and the actual values on the
horizontal axis, a straight line with a 450 slope will represent perfect forecasts. This
is called the line of perfect forecasts (Exhibit 14). In practice, the predictionrealization diagram is sometimes rotated so that the line of perfect forecasts
appears horizontal.
The prediction-realization diagram indicates how well a model or forecaster has predicted turning points
and also how well the magnitude of change has been predicted given that the proper direction of change
has been forecast.
The diagram has six sections. Points falling in sections II and V are a result of turning-point errors.
In Section V, a positive change was predicted, but the actual change was negative. In Section II, a negative
change was predicted, but positive change occurred. The remaining sections involve predictions that were
15
correct in sign but wrong in magnitude. Points above the line of perfect forecasts reflect actual changes that
were less than predicted. Points below the line of perfect forecasts represent actual changes that were
greater than predicted.
Exhibit 14 Prediction-Realization Diagram
The prediction-realization diagram can be used to record forecast results on an ongoing basis.
Persistent overruns or underruns indicate the need to adjust the forecasts or to reestimate the model. In this
case, a simple error pattern is evident and we can raise or lower the forecast based on the pattern and
magnitude of the errors.
More important, the diagram indicates turning-point errors that may be due to misspecification or
missing variables in the model. The forecaster may well be at a loss to decide how to modify the model
forecasts. An analysis of other factors that occurred when the turning-point error was realized may result in
inclusion of a variable in the model that was missing from the initial specification.
Exhibit 15 depicts two prediction-realization diagrams for performance results from two
forecasters in their respective sales regions of a motorcycle manufacturer. By plotting percentage changes in
the monthly forecast versus the percentage change in the monthly actuals, you can see the effect of turning
points on performance. In the first region, six of the forecasts are close to the line of perfect fit (45 degree
line), indicating a good job. The explanation of the other two forecasts suggest that the June point can be
attributed to a push to make up for lost retail sales in the first five months because of late production. In the
second region, for all eight months it showed an inability to detect a continuing decline as well as some
excessive optimism. Interestingly, on the basis of a measure of MAPE, the worse performing region has the
lower MAPE. On balance, it should be noted that the growth patterns are also quite different in the two
regions, which may need to be taken in consideration.
16
Exhibit 15 Prediction-Realization Diagrams for Two Regions of a Motorcycle Manufacturer: Region A (top),
Region B (bottom)
What Are Prediction Intervals?
A major shortcoming in the demand forecaster’s practice of forecasting techniques is the absence of error
bounds on forecasts. By usine the State Space Forecasting Methodology for Exponential smoothing and
ARIMA modeling, the demand forecaster can create prediction limits that reflect a more realistic,
unsymmetrical, as well as the conventional, symmetrical error bounds on forecasts. Why is this important?
A simple example will make the point.
Prediction Intervals for Time Series Models
When no error distribution can be assumed for a method that will make it a model, as with moving
averages, we cannot construct model-based prediction intervals on forecasts to measure uncertainty.
However, empirical prediction intervals can be constructed using an approach by Goodman and Williams in
a 1971 article entitled “A simple method for the construction of empirical confidence limits for economic
forecasts” in the Journal of the American Statistical Association. With the Goodman/Williams procedure,
we go through the ordered data, making forecasts at each time point. Then, the comparisons of these
forecasts with the actuals (that are known) yield an empirical distribution of forecast errors. If the future
errors are assumed to be distributed like the empirical errors, then the empirical distribution of these
17
observed errors can be used to set prediction intervals for subsequent forecasts. In some practical cases, the
theoretical size and the empirical size of the intervals have been found to agree closely.
Prediction intervals are used as a way of expressing the uncertainty in the future values that are derived
from models, and they play a key role in tracking forecasts from these models.
In regression models, it is generally assumed that random errors are additive in the model equation
and have a normal distribution with zero mean and constant variance. The variance of the errors can be
estimated from the time series data. The estimated standard deviation (square root of the variance) is
calculated for the forecast period and is used to develop the desired prediction interval. Although 95%
prediction intervals are frequently used, the range of forecast values for volatile series may be so great that
it might also be useful to show the limits for a lower level of probability, say 80%. This would narrow the
interval. It is common to express prediction intervals about the forecast, but in the tracking process, it is
more useful to deal with prediction intervals for the forecast errors (Error = Actual – Forecast) on a periodby-period or cumulative basis.
A forecaster often deals with aggregated data and would like to know the prediction intervals about
an annual forecast based on a model for monthly or quarterly data. The annual forecast is created by
summing twelve (independent) monthly (or four quarterly) predictions. Developing the appropriate
probability limits requires that the variance for the sum of the prediction errors be calculated. This can be
determined from the variance formula
Var (∑ ei) = ∑ Var (ei) + 2 ∑ Cov (ei , ej )
If the forecast errors have zero covariance (Cov) – in particular, if they are independent of one another, - the
variance of the sum equals the sum of the prediction variances.
The most common form of correlation in the forecast errors for time series models is positive
autocorrelation. In this case, the covariance term is positive and the prediction intervals derived would be
too small. In the unusual case of negative covariances, the prediction intervals would be too wide. Rather
than deal with this complexity, most software programs assume the covariance is small and inconsequential.
In typical forecasting situations, prediction intervals are probably too conservative or narrow.
Prediction Interval as Percentages
Many users of forecasts find the need for prediction intervals obtuse. In such cases, the forecaster may find
more acceptance of this technique if the prediction intervals for forecasts are expressed in terms of percentages.
For example, a 95% probability interval for a forecast might be interpreted that we are 95% sure that the
forecast will be within ± 15% (say) of the actual numerical value. The percentage miss associated with any
prediction interval can be calculated from the formula:
Percent miss = [(Estimated standard error of the forecast) (t factor) x 100% ] / (Forecast)
where t is the tabulated value of the Student t distribution for the appropriate degrees of freedom and
confidence level. This calculation based on the assumption of normality of forecast errors and provides an
estimate of the percent miss for any particular period. Values can be calculated for all periods (e.g., for each
month of the year), resulting in a band of prediction intervals spanning the year.
It could also be useful to determine a prediction interval for a cumulative forecast error. Under the
assumption that forecast errors are independent and random, the cumulative percentage miss will be smaller
than the percentage miss for an individual period because the positive and negative errors will cancel to some
extent. As an approximate rule, the average period (say monthly or quarterly) percentage miss is calculated by
dividing the average monthly percentage miss by √12 ( the square root of the number of periods of the
forecast) or the average quarterly percentage miss by √ 4 (=2).
18
Using Prediction Intervals as Early Warning Signals
One of the simplest tracking signals is based on the ratio of the sum of the forecast errors to the mean
absolute error MAE. It is called the cumulative sum tracking signal (CUSUM). For certain CUSUM
measures, a threshold or upper limit of |0.5| suggests that the forecast errors are no longer randomly
distributed about 0 but rather are congregating too much on one side (i.e., forming a biased pattern).
More sophisticated tracking signals involve the taking of weighted or smoothed averages of
forecast errors. The best-known standard tracking signal, due to Trigg Leach dates back to 1964 and will be
described shortly. Tracking signals are especially useful when forecasting a large number of products at a
time, as is the case in spare parts inventory management systems.
.
A tracking signal is a ratio of a measure of bias to a companion measure of precision.
Warning signals can be seen as predictive visualizations by plotting the forecast errors over time
together with the associated prediction intervals and seeing whether the forecast errors continually lie above
or below the zero line. Even though the individual forecast errors may well lie within the appropriate
prediction interval for the period (say monthly), a plot of the cumulative sum of the errors may indicate that
their sum lies outside its prediction interval.
An early warning signal is a succession of overruns and underruns.
A type of warning signal is evident in Exhibits 16 and 17. It can be seen that the monthly forecast
errors lie well within the 95% prediction interval for 9 of the 12 months, with two of the three exceptions
occurring in November and December. Exhibit 16 suggests that the individual forecast errors lie within their
respective prediction intervals. However, it is apparent that none of the forecast errors are negative certainly, they do not form a random pattern around the zero line. Hence, there appears to be a bias. To
determine whether the bias in the forecast is significant, we review the prediction intervals for cumulative
sums of forecast errors.
Exhibit 16 Monthly Forecast Errors and Associated Prediction Intervals for a Time Series Model
The cumulative prediction interval (Exhibit 17) confirms the problem with the forecast bias. The
cumulative forecast errors fall on the outside of the prediction interval for all twelve periods. This model is
clearly under-forecasting. Either the model has failed to capture a strong cyclical pattern during the year or
the data are growing rapidly and the forecaster has failed to make a proper specification in the model.
Using these two plots, the forecaster would probably be inclined to make upward revisions in the
forecast after several months. It certainly would not be necessary to wait until November to recognize the
problem.
19
Exhibit 17 Cumulative Forecast Errors and Associated Cumulative Prediction Intervals for a Time Series
Model
Another kind of warning signal occurs when too many forecast errors fall outside the prediction
intervals. For example, with a 90% prediction interval, we expect only 10% (approximately 1 month per
year) of the forecast errors to fall outside the prediction interval. Exhibit 18 shows a plot of the monthly
forecast errors for a time series model. In this case, five of the twelve errors lie outside the 95% interval.
Clearly, this model is unacceptable as a predictor of monthly values. However, the monthly error pattern
may appear random and the annual forecast (cumulative sum of twelve months) might be acceptable.
Exhibit 19 shows the cumulative forecast errors and the cumulative prediction intervals. It appears that the
annual forecast lies within the 95% prediction interval and is acceptable.
Exhibit 18 Monthly Forecast Errors and Prediction Intervals for a Time Series Model
Exhibit 19 Cumulative Forecast Errors and Cumulative Prediction Intervals for a Time Series Model
The conclusion that may be reached from monitoring the two sets of forecast errors is that neither
model is wholly acceptable, and that neither can be rejected. In one case, the monthly forecasts were good
20
but the annual forecast was not. In the other case, the monthly forecasts were not good, but the annual
forecast turned out to be acceptable. Whether the forecaster retains either model depends on the purpose for
which the model was constructed. It is important to note that, by monitoring cumulative forecast errors, the
forecaster was able to determine the need for a forecast revision more quickly than if the forecaster were
only monitoring monthly forecasts. This is the kind of early warning that management should expect from
the forecaster.
Trigg Tracking Signal
The Trigg tracking signal proposed by D. W. Trigg in 1964 indicates the presence of nonrandom forecast
errors; it is a ratio of two smoothed errors Et and Mt . The numerator Et is a simple exponential smooth of
the forecast errors et , and the denominator Mt is a simple exponential smooth of the absolute values of the
forecast errors. Thus,
Tt = Et / Mt
Et = α et + (1 - α) Et - 1
Mt = α |et | + (1 - α) Mt - 1
where et = Yt – Ft the difference between the observed value Yt and the forecast Ft.
Trigg shows that when Tt exceeds 0.51 for α = 0.1 or 0.74 for α = 0.2, the forecast errors are
nonrandom at the 95% probability level. Exhibit 20 shows a sample calculation of the Trigg tracking signal
for an adaptive smoothing model of seasonally adjusted airline passenger data. The tracking signal correctly
provides a warning at period 15 after five consecutive periods in which the actual exceeded the forecast.
Period 11 has the largest forecast error, but no warning is provided because the sign of the forecast error
became reversed. It is apparent that the model forecast errors can increase substantially above prior
experience without a warning being signaled as long as the forecast errors change sign. Once a pattern of
over- or underforecasting is evident, a warning is issued.
Exhibit 20 Trigg’s Tracking Signal (α = 0.1) for an Adaptive Smoothing Model of Seasonally Adjusted Airline
Data
Spreadsheet Example: How to Apply a Tracking Signal
In these White Papers we discuss a variety of forecasting models, from simple exponential smoothing to
multiple linear regression. Once we start using these models on an ongoing basis, we cannot expect them to
supply good forecasts indefinitely. The patterns of our data may change, and we will have to adjust our
forecasting model or perhaps select a new model better suited to the data. For example, suppose that we are
using simple exponential smoothing, a model that assumes that the data fluctuate about a constant level.
Now assume that a trend develops. We have a problem because the level exponential smoothing forecasts
21
will not keep up with a trend and we will be consistently under-forecasting. How do we detect this
problem?
One answer is to periodically test the data for trends and other patterns. Another is to examine a
graph of the forecasts each time that we add new data to our model. Both methods quickly become
cumbersome, especially when we forecast a large number of product SKUs by locations. Visually checking
all these data after every forecast cycle is out of the question!
Quick and Dirty Control
The tracking signal, a simple quality control method, is just the tool for the job. When a forecast for a given
time period is too large, the forecast error has a negative sign; when a forecast is too small, the forecast
error is positive. Ideally, the forecasts should vary around the actual data and the sum of the forecast errors
should be near zero. But if the sum of the forecast errors departs from zero in either direction, the
forecasting model may be out of control.
Just how large should the sum of the forecast errors grow before we act? To answer this, we need a
basis for measuring of the relative dispersion, or scatter, of the forecast errors. If forecast errors are widely
scattered, a relatively large sum of errors is not unusual. A standard measure of variability such as the
standard deviation could be used as a measure of dispersion, but more commonly the mean absolute error
(MAE) is used to define the tracking signal
:
Tracking signal = (Sum of forecast errors) / (MAE)
Typically the control limit on the tracking signal is set at ±4.0. If the signal goes outside this range
we should investigate. Otherwise, we can let our forecasting model run unattended. There is a very small
probability, less than 1%, that the signal will go outside the range ±4.0 due to chance.
The Tracking Signal in Action
Exhibit 21 is a graph of actual sales and the forecasts produced by simple exponential smoothing. For the
first 6 months, the forecast errors are relatively small. Then, a strong trend begins in month 7. Fortunately,
as shown in Exhibit 22, the tracking signal sounds an alarm immediately.
Exhibit 21 Monthly Forecasts and Sales for a Time Series Model
22
Exhibit 22 Monthly Forecast Errors and Tracking Signal
Column F in Exhibit 23 contains the mean absolute error. This is updated each month to keep pace
with changes in the errors. Column G contains the running sum of the forecast errors, and column H holds
the value of the tracking signal ratio. A formula in column I displays the label ALARM if the signal is
outside the range ±4.0. We can now take a step-by-step approach to building the spreadsheet. For those
unfamiliar with simple exponential smoothing (Download free White Paper 07 at
http://cpdftraining.org/PDFgallery2.htm), the forecasting formulas are explained as they are entered.
Exhibit 23 Actuals, Forecasts, and Forecast Tracking Signal
Step 1: Set up a spreadsheet. Enter the labels in rows 1 - 5. In cell B2, enter a smoothing weight of 0.1. In
cell F2, enter a control limit of 4.0. Number the months in the column A and enter the actual sales in
column B.
Step 2: Compute the monthly forecasts and forecast errors. To get the smoothing process started, we
must specify the first forecast in cell C7. A good rule of thumb is to set the first forecast equal to the first
actual data value. Therefore, enter +B7 in cell C7. In cell D7, enter the formula +B7-C7, which calculates
the forecast error.
In cell C8, the first updating of the forecast occurs. Enter the formula +C7*$B$2*D7 in cell C7.
This formula states that the forecast for month 2 is equal to the forecast for month 1 plus the smoothing
weight times the error in month 1. Copy the updating formula in cell C8 to the range C9 to C19. Copy the
error-calculating formula in cell D7 to range D8 to D18. Because the actual sales figure is not yet available
for month 13, cells B19 and D19 are blank.
23
Exponential smoothing works by adjusting the forecasts in a direction opposite to that of the error,
working much like a thermostat, an automatic pilot, or a cruise control on an automobile. Examine the way
the forecasts change with the errors. The month 1 error is zero, so the month 2 forecast is unchanged. The
error is positive in month 2, so the month 3 forecast goes up. In month 3 the error is negative, so the month
4 forecast goes down.
The amount of adjustment each month is controlled by the weight in cell B2. A small weight
avoids the mistake of overreacting to purely random changes in the data. But a small weight also means that
the forecasts are slow to react to true changes. A large weight has just the opposite effect – the forecasts
react quickly to both randomness and true changes. Some experimentation is always necessary to find the
best weight for the data.
Step 3: Compute the Mean Absolute Error (MAE). Column E converts the actual errors to absolute
values. In cell E7, enter the absolute value of E7, and then copy this cell to range E8 to E18. The mean
absolute error, in column F is updated via exponential smoothing much like the forecasts. In cell F6, a
starting value for the MAE must be specified. This must be representative of the errors or the tracking
signal can give spurious results. A good rule of thumb is to use the average of all available errors to start up
the MAE. But here we will do an evaluation of how the tracking signal performs without advance
knowledge of the surge in sales in month 7. We use the average of the absolute errors for months 2 - 6 only
as the initial MAE (month 1 should be excluded because the error is zero by design). In cell F6, enter the
formula for the average of E8 to E12.
To update the MAE, in cell F7 enter $B$2*E2+(1-$B$2)*F6. Copy this formula to the range F8 to
F12. This formula updates the MAE using the same weight as the forecast. The result is really a weighted
average of the MAE, which adapts to changes in the scatter of the errors. The new MAE each month is
computed as 0.1 times the current absolute error plus 0.9 times the last MAE.
Step 4: Sum the forecast errors. This simple operation is considered a separate step because a mistake
would be disastrous here. We must sum the actual errors, not the absolute errors. The only logical starting
value for the sum is zero, which should be entered in cell G6. In cell G7, enter +D7+G6 and copy this to
range G8 to G18.
Step 5: Compute the tracking signal. The tracking signal value in column H is the ratio of the error sum
to the MAE. In cell H6, enter +G6/F6 and copy this to range H7 to H18.
Step 6: - Enter the ALARM formulas
In cell I6, enter an IF statement for ABS (H6)>F3,"ALARM","") and copy it to range I7 to I19. This
formula takes the absolute value of the tracking signal ratio. If that value is greater than the control limit, the
label ALARM is displayed; otherwise, the cell is blank.
Adapting the Tracking Signal to Other Spreadsheets
The formulas in columns E through I in Exhibit 23 are not specific to simple exponential smoothing. They
can be used in any other forecasting spreadsheet as long as the actual period-by-period forecast errors are
listed in a column. If we use a forecasting method that does not include weights, such as linear regression,
we will also have to assign a weight for updating the MAE - we simply use a weight of 0.1 in such
applications. Concerning the weights used to update forecasts, experience suggests that there is not much to
be gained by experimenting with the weights in the MAE.
In large forecasting spreadsheets, macros can be used to operate the monitoring control system.
For example, we could write a macro that checks the last cell in the tracking signal column of each data set.
The macro could sound a beep and pause if a signal is out of range. Or we could write a macro that prints
out a report showing the location of each offending signal for later investigation.
In Exhibit 23, the tracking signal gives an immediate warning when our forecasts are out of
control. Do not expect this to happen every time. The tracking signal is by no means a perfect monitoring
control device, and only drastic changes in the data will cause immediate warnings. More subtle changes
may take some time to detect.
24
Statistical theory tells us that the exact probability that the tracking signal will go outside the range
±4.0 depends on many factors, such as the type of data, the forecasting model in use, and the parameters of
the forecasting model such as the smoothing weights. Control limits of ±4.0 are a good starting point for
tracking much business data. If experiments show that these limits miss important changes in the data,
reduce them to 3.5 or 3.0. If the signal gives false alarms, cases in which the signal is outside the limits but
there is no apparent problem, increase the limits to 4.5 or 5.0. We recommend adding a tracking signal to
every forecasting model. They save a great deal of work by letting us manage by exception.
Mini Case: Accuracy Measurements of Transportation Forecasts
The “Hauls” data are proprietary but real. The problem is to investigate the need for data quality
adjustments prior to creating a useful forecasting model. Does it matter if a few data points look unusual?
For trend/seasonal data, how does it impact seasonal factors, monthly bias and overall precision? We would
like to get an assessment of the impact of an outlier (just one) and how we would measure forecast
accuracy.
Preliminary Analysis. Based on the three years (36 monthly observations), the total variation can be
decomposed into trend, seasonal and other using the Excel DataAnalysis Add-in (ANOVA without
replication). Exhibit 24 shows Seasonality (44%), Trend (1%) and Other (55%). Trend/Seasonal models
are expected to show a significant amount of uncertainty in the prediction limits. After an outlier adjustment
(shown below), the total variation breaks down to Seasonality (47%), Trend (1%) and Other (52%). But,
how accurate might the forecasts be?
Exhibit 24 ANOVA Pie Chart for Hauls Data
Exploratory Forecast
1. We proceed by modeling the unadjusted data with a trend/seasonal exponential smoothing model
and test first for within sample outliers. The Model is a State Space Model (N,M,M); Local Level
(No Trend), Multiplicative Seasonality, Multiplicative Error). The latter means that the prediction
limits are asymmetrical (cf. www.peerforecaster.com for free download and license of
PEERForecaster.xla Excel Addin)
2. In Exhibit 25, the model residuals are shown in Column 1. The Descriptive Statistics (Excel >
DataAnalysis Addin) are used to determine the bounds for outlier detection in the residuals.
Outlier Detection and Correction
3. For outlier detection, the traditional method is Mean +/- St.Dev. The traditional method is not
outlier resistant as both the Mean and St.Dev are distorted when the residuals contain one or more
outlier. The outlier-resistant nonconventional method is based on the calculations (a) 75th
percentile + 1.5 (Interquartile Distance), and (b) 25th Percentile – 1.5 (Interquartile Distance).
4. Both outlier detection methods give the same conclusion (in this case): Outlier in residuals = 867.
The outlier can be corrected (conservatively) by replacing the actual (= 5265) with the Upper
Bound or more commonly with the fitted model value for that period (= 4397.62).
5. Before making adjustments, outliers should always be reviewed for reasonableness with a domain
expert familiar with the historical data. It is critical, however, to take data quality steps before
modeling as outlier effects can be confounded with the main effects, like seasonal factors, special
events and promotions.
25
Exhibit 25 Outlier Calculation with Tradtional and Nonconventional Methods
Forecast Testing
6. We choose a hold-out sample of twelve months and create a 12-period ahead forecast for
evaluation. The outlier (October of the second year) could have a significant impact on exponential
smoothing forecasts because it occurs just before the 12-month hold-out period. The first period in
the data is a December.
7. By choosing the fitted value as outlier replacement (Exhibit 26), we rerun the trend/seasonal
exponential smoothing model with the outlier-adjusted data. The Haul data are shown in the third
column.
8. In Exhibit 27, we show a credible forecast (Change) with prediction limits as a measure of
uncertainty (Chance). The trend/seasonal exponential smoothing model is a State Space Model
(N,M,A): Local Level (No trend), Multiplicative seasonality and Additive error. The latter means
that the bounds for the prediction limits are symmetrical around the forecast.
Preliminary Forecast
9. Exhibit 28 shows Predictive Visualization of a preliminary forecast with Outlier-Adjusted Hauls
Data (Ad,M,A); Local Trend (Damped), Multiplicative Seasonality, Additive Error). The latter
means that the bounds for the prediction limits are symmetrical around the forecast.
10. The 12 actuals in the hold-out sample lie within the error bounds (not shown)
26
Exhibit 26 Outlier Replacement for October in 2nd Year: Actual = 5265, Adjusted Value = 4397.62
Exhibit 27 Predictive Visualization of Outlier-Adjusted Hauls Data with 12-Month Hold-out Period
Exhibit 28 Predictive Visualization of a Preliminary Forecast with Outlier-Adjusted Hauls Data
27
Model Evaluation
11. Exhibit 29 shows Bias and Precision measures for forecasts in the Hold-Out Sample With and
Without Outlier-Adjusted Haul Data. The Mean Error (ME) and Mean Absolute Percentage Error
(MAPE) are not outlier resistant, unlike their Median counterparts. The ME is severly distorted by
the October outlier, while the Median Error (MdE) gives a more reliable result for the bias. The
effect of an outlier also appears to be greater on the MAPE than the MdAPE.
12. In practice, both measures should always be calculated and compared. If the MAPE and MdAPE
appear to be quite different (from an “impact” perspective), then investigate the underlying
numbers for stray values. If comparable, then report the MAPE as this measure is typically better
known. Here a MAPE of 6% would be a credible measure of performance based on the
comparison in this one case.
13. Exhibit 30 shows a comparison of seasonal factors from the analysis with and without outlier
adjustment. The impact is greatest for October, but also shows some effect in December/January.
Exhibit 29 Bias and Precision Measures for Forecasts in the Hold-Out Sample With and Without OutlierAdjusted Haul Data
Exhibit 30 Comparison of Multiplicative Seasonal Factors ( With and Without Outlier-Adjusted Haul Data)
28
We come up with standard forecast evaluation procedure as follows:
1.
2.
3.
4.
Calculate the ME, MPE, MAPE, MAE and the RMSE. Contrast these measures of accuracy.
Recommend and calculate a resistant set of measures for each one of the measures
Create a Naïve_12 (12-period-ahead) forecast for a 12-month holdout sample, by starting with
January forecast = January (previous year) actual, February forecast = February (previous
year) actual, and so on. Construct a relative error measure, like RAE, comparing the forecasts
with the Naïve_12 forecasts. Can you do a better job with a model than a Naïve_12 method?
On a month-to-month basis these accuracy measures might not be meaningful to management
(too large, perhaps), so you suggest looking at the results on a quarterly basis. Aggregate the
original data and forecasts into quarterly periods (or buckets). Repeat (1) and (2) using a
Naïve_4 as the benchmark forecast to beat.
Final Takeaways
One goal of an effective demand forecasting strategy is to improve forecast accuracy through the use of
forecasting performance measurement. Forecasters use accuracy measures to evaluate the relative
performance of forecasting models and assess alternative forecast scenarios. Accuracy measurement
provides management with a tool to establish credibility for the way the forecast was derived and to provide
incentives to enhance professionalism in the organization.
•
•
•
•
•
It should always be good practice to report accuracy measures in pairs to insure against
misleading results. When the pair are similar (in a practical sense), then report the
traditional result. Otherwise, go back and investigate the data going into the calculation,
using domain information whenever possible
We recommend the use of relative error measures to see how much of an improvement is
made in choosing a particular forecasting model over some benchmark or "naive"
forecasting technique.
Relative error measures also provide a reliable way of ranking forecasting models on the
basis of their usual accuracy over sets of time series.
Test a forecasting model with hold out samples by splitting the historical data into two
parts, using the first segment as the fit period. Then, the forecasting model is used to
make forecasts for a number of additional periods (the forecast horizon). Because there
are actual values withheld in the holdout sample, one can assess forecast accuracy by
comparing the forecasts against the known figures. Not only do we see how well the
forecasting method has fit the more distant past (fit period) but also how well it would
have forecast the test period.
Tracking signals are useful when large numbers of items must be monitored for accuracy.
This is often the case in inventory systems. When the tracking signal for an item exceeds
the threshold level, the forecaster’s attention should be promptly drawn to the problem.
The tracking of results is essential to ensure the continuing relevance of the forecast. By
properly tracking forecasts and assumptions, the forecaster can inform management when
a forecast revision is required. It should not be necessary for management to inform the
forecaster that something is wrong with the forecast. The techniques for tracking include
ladder charts, prediction-realization diagrams, prediction intervals, and tracking signals
that identify nonrandom error patterns. Through tracking, we can better understand the
models, their capabilities, and the uncertainty associated with the forecasts derived from
them.
29
Contents
Taming Uncertainty: All You Need to Know About Measuring Forecast Accuracy, But Are Afraid to Ask
..........................................................................................................................
The Need for Measuring Forecast Accuracy.......................................................................................... 1
Analyzing Forecast Errors ...................................................................................................................... 3
Lack of Bias............................................................................................................................................ 3
What is an Acceptable Precision?........................................................................................................... 4
Ways to Evaluate Accuracy ..................................................................................................................... 6
The Fit Period versus the Test Period..................................................................................................... 6
Goodness-of-Fit versus Forecast Accuracy ............................................................................................ 7
Item-Level versus Aggregate Performance............................................................................................. 7
Absolute Errors versus Squared Errors................................................................................................... 7
Measures of Forecast Accuracy............................................................................................................... 8
Measures of bias ..................................................................................................................................... 8
Measures of Precision............................................................................................................................. 9
Comparison with Naive Techniques ..................................................................................................... 12
Relative Error Measures ....................................................................................................................... 13
Predictive Visualization Techniques ..................................................................................................... 13
Ladder Charts ....................................................................................................................................... 14
Prediction-Realization Diagram ........................................................................................................... 14
What Are Prediction Intervals? ............................................................................................................ 16
Prediction Intervals for Time Series Models ........................................................................................ 16
Prediction Interval as Percentages ........................................................................................................ 17
Using Prediction Intervals as Early Warning Signals ........................................................................... 18
Trigg Tracking Signal........................................................................................................................... 20
Spreadsheet Example: How to Apply a Tracking Signal .................................................................... 20
Quick and Dirty Control ....................................................................................................................... 21
The Tracking Signal in Action.............................................................................................................. 21
Adapting the Tracking Signal to Other Spreadsheets ........................................................................... 23
Mini Case: Accuracy Measurements of Transportation Forecasts.................................................... 24
Final Takeaways ..................................................................................................................................... 28