Faith Eherts, Erin Kreher, Ryan Dreedon, Eric Donten
Since the city under observation is Louisville, KY, temperatures were expected to range
from the mid-80s (at the commencement of the project, in early September) to about 50 degrees
(beginning of December). It was found that, when the mean absolute errors and mean errors of
each provider (The Weather Channel/TWC, Accuweather, and Weather Underground) were
determined and compared (see Chart 1), The Weather Channel very clearly comes out on top –
with the smallest error values reported virtually every day ahead in the extended forecast.
However, these charts do not display the standard deviations on any given day – say, day five and, based on a statistical one-way ANOVA test, the differences between the errors of the
providers on this day are not significant enough to draw any valid conclusions. On the other
hand, when the one-way ANOVA test is done using the values from day 10, there is sufficient
evidence to determine that one piece of data significantly differs from the others. Therefore,
although one could not legitimately conclude that TWC puts out the most accurate forecasts, it
can be said that Weather Underground puts out the least accurate. Each provider’s biases can
also be unveiled using these error calculations. It was observed that TWC harbors a very slight
but very consistent cold bias, since every forecast date, on average, has negative error values.
Weather Underground shares with TWC a cold bias trend, but one that is far stronger – and gets
stronger as the forecast day goes out - than that of TWC. On the other hand, Accuweather is more
variable, having days where error is slightly negative and others where it is slightly positive,
making it more difficult to discern any patterned bias. Climatology, interestingly, usually underpredicted the temperature, suggesting that this autumn was warmer than average. Based on
Graph 1, it would be expected that climatology becomes more accurate than TWC forecasts after
day 12 (and for Accuweather and Weather Underground after day nine, graphs not shown).
Frequency distributions (graphs two through five) provide proof that mean absolute error is not
always the most efficient way to compare data. In comparing these skewed-left histograms, it is
easier to understand which provider has the most accurate and consistent forecasts. The accuracy
of the forecasts, specifically for seven days ahead, also shifts dramatically depending on the
season. By performing a difference of means test on the data gathered from September 16th
through October 16th and from November 1st through November 30th, there is sufficient statistical
evidence to suggest a fairly dramatic shift in absolute value error as the season changes – and
that forecast accuracy was higher during the beginning of autumn than at the end. Another error
measure that can be utilized to assess the accuracy of forecasts is the squared error. The squared
error is able to convey information about the dispersion of the forecasted temperatures about the
actual recorded temperature. The mean squared errors at the bottom of each column in the
spreadsheet are equal to the average variance of the forecasted temperatures for each particular
day, and indicate that providers with large variances are inconsistent. A good way to measure the
consistency of a provider as a given day approaches is to find the sum of the absolute values of
the differences between forecasted temperatures of adjacent days (See Example 1). The results
are as follows: TWC, 10.9 F; Accuweather, 10.4 F; and Weather Underground, 13.7 F. A oneway ANOVA test indicates that there is, once again, not enough statistical evidence to definitely
conclude that one of these means stands out from the others (even though it appears that
Accuweather is the most consistent). All of these conclusions may have turned out much
differently (and possibly more accurately) if the data was gathered over a larger period of time
(covering more seasons and creating more accurate averages) or in more locations besides just
Louisville. Also, if more than only the temperature was record (such as rainfall or cloud cover),
the prediction accuracy of each provider could be better determined.
Chart 1
Forecast
Provider
Mean Errors Corresponding to Forecast Date
The Weather
Channel
Accuweather
Weather
Underground
Climatology
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-0.5
-0.4
-1.1
-0.5
-0.5
-1.0
-1.4
-1.5
-0.6
-0.7
---
---
---
---
---
-0.3
-0.5
0.08
0.07
0.66
0.98
1.50
0.95
2.32
2.03
0
-0.1
-0.7
0.29
0.93
-0.3
-0.4
-0.6
-0.3
-0.4
0.17
0.67
-2.1
-1.6
-5.5
---
---
---
---
---
-1.1
---
---
---
---
---
---
---
---
---
---
---
---
---
---
Forecast
Provider
Mean Absolute Errors Corresponding to Forecast Date
The Weather
Channel
Accuweather
Weather
Underground
Climatology
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1.76
2.26
2.74
3.11
3.63
4.32
4.91
5.68
6.69
6.43
---
---
---
---
---
1.76
2.31
2.73
3.04
3.76
4.30
4.76
5.59
7.77
7.84
8.54
8.91
9.18
9.84
9.44
1.97
2.52
2.95
3.47
4.47
4.20
5.59
5.22
7.37
10.9
---
---
---
---
---
7.71
---
---
---
---
---
---
---
---
---
---
---
---
---
---
Graph 1
The figure below plots mean absolute forecast error as a function of forecast date for The Weather Channel.
Graph 2
The Weacther Channel:
7-Day Absolute Error Distribution
14
12
10
8
6
Frequency
4
2
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
More
0
Graph 3
Accuweather:
7-Day Absolute Error Distribution
14
12
10
8
6
4
2
0
Frequency
Graph 4
Weather Underground:
7-Day Absolute Error Distribution
14
12
10
8
6
Frequency
4
2
0
Graph 5:
Climatology:
7-Day Absolute Error Distribution
10
8
6
4
Frequency
2
0
Example 1
For example, if a forecast for seven days away in State College was 32F and the forecast for that
same day as we made our way through the week became 33F, 36F, 34F, 38F, 36F, 40F, and 41F
(for each day between the today and seven days from now), then the consistency of the
forecasting provider would be abs(32F – 33F) + abs(33F – 36F) + abs(34F – 36F) + abs(38F –
34F) + abs(36F – 38F) + abs(40F – 36F) + abs(41F – 40F) = 17F.