Summary
This article discusses the two main KPI's you can use in StockIQ for evaluating forecast quality, and the variations on them, and how they are calculated
Applies To
All Items in StockIQ
Process
Forecast Error = (Actual Value of sales) - (Forecasted Value)Regardless of what error measure you use, in StockIQ, we find it useful to show the value for the actual error value itself, along with a percentage, and also the dollar/currency value associated with the error, to help you prioritize the most important measures first.
Mean Absolute Error (MAE)
Since knowing the error of any one period is not often very useful, most of the time you measure error over a number of samples, so you end up with a total error over that time period.
Further, since you want to know in total how wrong you were, typically you will take the absolute value of the error, so you're summing up how wrong you were, on either side, for your observations.
Divide your sum by the number of periods you measured error for, and you have the average of the absolute value of your errors, otherwise known as Mean Absolute Error (MAE), which tells you the average number of units of error you can expect from that forecast series in each forecasting period.
Since MAE is a units-based number, you don't know if a given MAE is good or bad without knowing something about your product. Say you have an MAE of 10. Is that good? If you sell 500 units a month, that is great! If you sell 5 units a month, that's pretty terrible.
To make the MAE number into a percent, divide by your average usage over the period that you have measured, and you have wMAPE (weighted Mean Absolute Percent Error), which is then a number you can compare between different SKUs. This works out to the same thing as adding up how many units you missed by across the whole window and dividing that by how many units actually sold.
wMAPE vs MAPE
StockIQ uses wMAPE (weighted Mean Absolute Percent Error), not plain MAPE (Mean Absolute Percent Error). The names differ by one letter, but they produce entirely different values. wMAPE, which is what StockIQ shows: Add up how many units you missed by across the whole window. Add up how many units actually sold. Divide the first by the second.
wMAPE = (total units missed) ÷ (total units sold)
This answers the question: "Of the demand we actually got, what share did we miss by?"
MAPE (Mean Absolute Percent Error), which is what you may be used to from other tools: Work out how wrong each single period was as a percentage of that period's own sales. Then average those percentages together.
MAPE = average of (units missed in a period ÷ units sold that period)
This answers the question: "How wrong was a typical period?"
The difference is when you divide. wMAPE pools everything and divides once, so a period contributes in proportion to how much demand it actually had. MAPE divides inside each period and then treats every period as equally important, so a month that sold 4 units counts exactly as much as a month that sold 4,000.
Here is a real lag-3 window where the two disagree completely. Forecast is a flat 100 a month; August collapses.
wMAPE = 120 missed ÷ 220 sold = 55%.
MAPE = the average of 17%, 11% and 900% = 309%.
Both are arithmetically correct. But nothing in that quarter was 309% wrong. We shipped a forecast of 300 against demand of 220. The 309% is one 10-unit month shouting over two normal ones. That is the whole reason StockIQ uses wMAPE.
Why wMAPE is the better measure
- Small periods can't hijack the number. MAPE's per-period division means a tiny denominator produces an enormous percentage that then drags the average. wMAPE never divides by a single period, only by the window total, so slow movers and seasonal troughs stay in proportion.
- Zero-demand periods don't break it. A period with no sales makes MAPE mathematically undefined, so tools that use it have to drop the period, substitute a fudge, or report infinity. wMAPE just adds that period's miss to the numerator and moves on. This matters a lot in slow-moving inventory, where zero months are normal.
- It doesn't quietly reward under-forecasting. Under-forecast a period and MAPE can never exceed 100% for it; over-forecast the same period and MAPE is unbounded. Tune a forecast to minimize MAPE and you get pushed downward, which for a replenishment system means stockouts. wMAPE treats a miss of 50 units the same in either direction.
- It rolls up honestly. Because wMAPE is one total divided by another total, the item, category and company numbers are all built the same way and the roll-up is meaningful. An average of per-item MAPEs is an average of averages and doesn't reconcile to anything.
wMAPE is also known as WAPE or the MAD/Mean ratio. Earlier versions of StockIQ labeled it MAE% or MAEP (Mean Absolute Error Percent); the calculation has not changed, only the name. It is a deliberate choice, not a variant of MAPE that got the letters wrong.
wMAPE is also the only one of the two you can put a currency symbol next to. Multiply wMAPE's numerator by unit cost and you have the money at stake, which is precisely what the Error $ column does. There is no equivalent operation on a MAPE.
NOTE: StockIQ no longer offers RMSE% (root mean squared error percent) as an error calculation. Every forecast error percent in StockIQ is wMAPE.
Bias
The other important calculation to monitor in forecasting is your bias. A bias in a forecast means you are consistently over- or under-forecasting a product over time, and is also highly problematic. As with error, bias is most useful when measured over a period of time, versus looking at single values.
As with the error calculations, converting the absolute bias quantity into a percentage is typically helpful as well.
Single Period Bias = (Forecasted Value) - (Actual Value)
Average Bias = (Sum of Single Period Biases) / (Number of Periods Measured)
Bias Percent = (Average Bias) / (Average Actuals Over Periods Measured)Error Calculation Time Periods (Leads & Lags)
The other question when calculating your average error is over what time period to calculate the average error. This is not always straightforward.
For current period forecasts and statistical forecasts, your only option is to calculate backwards looking error values. This gives you an expectation of what your error values might be knowing everything you know today.
In StockIQ, we keep forecast snapshots of history, so that you can see how your forecasts have changed over time. These are called "Lagging Forecasts." It is common to want to see your "Lag 1" "Lag 2" or "Lag 3" forecasts, meaning you want to see forecasts from 1, 2, and 3 periods ago. Forecast Error Lag in the Global Demand Forecast Settings picks the lag the Forecast Error screens use: Auto (one lead time ago) or a fixed 1 to 12 periods.
These are interesting since you can ask the question "What is my forecast accuracy for the lag when I am ordering", e.g. "what is my forecast error when I am one lead time away from receiving my product?"
Consider the following simple time series of a product where we sit on December 1st of a year, and we are looking to calculate error using a 6-month MAE:
Backwards Looking Error
The green, yellow, red, and orange areas show the periods we would use to calculate a backwards looking error value on each of the forecasts, given the period in which they were captured. You can set the length for your error calculations in the Global Demand Forecast Settings. 6 and 12 months are common horizons for this.
Error Vs Current & Back-Testing
The gray area on each of the forecast snapshots represents periods where actuals were UNKNOWN at the time the snapshot was taken.
For forecast snapshots and back-dated statistical models, where demand history is hidden from the algorithm (such as for training the forecast model), you can also calculate an error "Versus Current". This means we are calculating your error from current period back, but from a snapshot in time that did not know some of the results at the time the snapshot was taken.
This incorporates the periods that were not known at the time the snapshot is generated, giving you a sense of what your forecast error will be at that amount of lag.
This approach is used when evaluating statistical models for performance, as it avoids over-fitting the model to your current data, since it tests how well the model can perform against unknown values by hiding known data, then testing the model on the known data after it is generated.