Skip to main content

Forecast Error Screen

Updated

Summary

The Forecast Error Screen has the goal of helping you understand the average forecast error of your items over time, as well as give you some data about per-period error amounts of your statistical and operational forecasts.

You can reach it by clicking to Forecast --> Forecast Error

The screen is divided into two main sections, with a master report at the top, referred to as the Forecast Error Detail, and some tabs of additional information at the bottom.

 

Forecast Error Detail Grid

The Forecast Error Detail grid contains the details about your forecast error information.  You can view saved forecast error at all levels of your forecast hierarchy using the options in the Display Options (see below).  By default, StockIQ starts with data shown at the bottom of your hierarchy, which is often the item-site-forecast-group level.

In addition to the usual item, site, and category columns, the following data is available, including if there is an alert on this record, then you will see an alert priority icon, along with an Alert State column.  See Forecast Error Alert

After the initial category and alert columns, the same sets of columns are available in four different "bands", which differ based on the time periods considered in the error calculations. Only the first band, Lead-Time Ago Errors (Vs Current Actuals), is shown by default. 

NOTE: When Forecast Error Lag in Global Demand Forecast Settings is set to a fixed lag, the bands and columns that say "Lead-Time Ago" or "LT Ago" read "Lag N" instead (for example Lag 3 Errors (Vs Current Actuals) and Lag 3 Snapshot), and every node uses the snapshot from N forecast periods ago.

  • Lead-Time Ago Errors (vs Current Actuals) - These values are all calculated from whatever forecast snapshot is one lead-time ago, aka the Lead-Time Lag.  This then compares the error of that snapshot vs now-known and backdated values to calculate the average error of this lagging forecast snapshot.  So, if today is June 1st, and the lead-time-ago snapshot is from March 1st, that is a 3-period Lag.  The error will be calculated using error values from June of last year to May of this year, which now includes 3 periods not known when the forecast snapshot was taken (March, April, May), so your average error includes 9 known and 3 unknown periods for the error calculation.  This reduces under-reporting error due to "Overfitting" of the forecast. Matches the last group of 'Forecast Errors @ Lead Time Lag -mmddyy - Vs Current Actuals' in Forecast Manager Fcst Error tab.
  • Lag Period Errors - Unlike the Lead-Time Ago vs Current (above), this looks only at the errors in the lag period, e.g. the now known values vs values unknown at the time of snapshot.  In the example above, this would be your average error of March, April, and May time periods.  Shows in Snapshots Tab grid as Lag wMAPE in Forecast Manager.
  • Current Forecast Errors - This is a backwards looking calculations of your current forecasts, with no lag period involved.  Matches the First group column 'Forecast Error -Current' in Forecast Manager Forecast Error tab.
  • Lead-Time Ago Errors (Backwards Looking) - This is backwards looking calculations of the lead-time-ago snapshot.  Using the example above, if today is June 1st, and we have a snapshot from 3 months ago on March 1st, this would be your average error from March of the previous year to February of this year. Matches the Middle column 'Forecast Errors @ Lead Time Lag' in Forecast Manager Forecast Error tab.  These calculations are a snapshot of calculations from one lead time ago, so if data has changed that is not reflected.    

Each of these error calculations can be useful for different reasons.  The important thing is to decide on which set you want to standardize on and use that.

After selecting one or a few of those groups, in each of these groups, the following columns are available. The Current Forecast Errors group does not include the LT Ago Snapshot and LT Ago Lag columns, and the Lead-Time Ago Errors (Backwards Looking) group does not include the FVA columns:

  • LT Ago Snapshot - Shows the date of the snapshot being used as "one lead time ago" (Lag N Snapshot when Forecast Error Lag is a fixed lag: the snapshot from N periods ago).  Since we will not always have a snapshot from exactly one lead time ago, StockIQ will use the most recent snapshot taken on or before that date.
  • LT Ago Lag (Lag N Lag with a fixed Forecast Error Lag) - Shows the number of periods-ago corresponding to this lag.  For example, if today is June 1st, and the snapshot is from April 1st, your lag is 2 periods = 2 months.
  • Benchmark wMAPE - Average error (vs current) of the naive benchmark forecast, as wMAPE (weighted mean absolute percent error): total units missed divided by total units sold.
  • Benchmark MAE - Mean (aka Average) Absolute Error (in units) of the benchmark forecast.
  • Benchmark Error $ - Units of error translated to enterprise value.  This is used to prioritize which errors are most important to the business.
  • Benchmark Bias % - Bias percentage of the benchmark.
  • Stat wMAPE - Average error (vs current) of the current statistical forecast, as wMAPE.
  • Stat MAE - Mean (aka Average) Absolute Error (in units) of the stat forecast.
  • Stat Error $ - Units of error translated to enterprise value.  This is used to prioritize which errors are most important to the business.
  • Stat Bias % - Bias percentage of the stat model.
  • Stat vs Benchmark FVA - Calculates the forecast-value-added of the statistical forecast vs the naive forecast, e.g. are we doing a better job than the simplest algorithm possible.
  • Op Fcst wMAPE - Average error (vs current) of the operational forecast, as wMAPE.
  • Op Fcst MAE - Mean (aka Average) Absolute Error (in units) of the current op forecast series.
  • Op Fcst Error $ - Units of error translated to enterprise value.  This is used to prioritize which errors are most important to the business.
  • Op Fcst Bias % - Bias percentage of the operational forecast.
  • Op Fcst vs Statistical FVA - Calculates the forecast-value-added of the operational forecast vs the current statistical model.  

IMPORTANT: If the Op Fcst vs Statistical FVA is negative on manually forecasted items, this indicates the demand planner may not be adding any value and this item should return to auto forecast.  Similarly, if it is positive, it indicates they are providing good insight versus what the algorithm is able to produce.

An Item Tags column is also available after the bands.

 

Forecast Error Bands Deeper Dive

All three snapshot error bands compare the same frozen forecast — the one captured in the snapshot — against actuals. What differs is only which periods get scored. 

Backwards Looking scores the 12 closed periods before the snapshot was taken. Lag Period scores only the periods that were still in the future on snapshot day and have since closed. Vs Current Actuals scores a rolling 12-period window that ends today — so it contains the lag periods plus enough pre-snapshot periods to fill 12 out. It is a blend, and its window slides forward every month while the forecast in it stays frozen.

Nothing goes "back to the start of forecasting." The window length is one global setting, Error Sample Length, which ships as Yearly — 12 periods on a monthly forecast, 52 on a weekly one.

 

  • Lag Period Error

    • Question- “For the months I could not have known about, how good was that forecast?” This is the only band that is purely out-of-sample.

    • Window- snapshot period … last closed period  (inclusive)

    • Forecast used- The snapshot frozen on snapshot day.

    • Actuals used- Today's actuals, restatements and all.

    • Watch for- Sample counts are tiny. A lag of 3 means the percentage is built from 3 data points, so it is volatile at item level and only really stabilizes when rolled up. It is blank when the snapshot is from the current period (no closed lag periods yet).

  • Backwards Looking Error

    • Question- “What error number was on this screen on the day the snapshot was taken?” It is a stored value copied off the forecast header at snapshot time, not something recomputed for the band.

    • Window- 12 periods ending the period before the snapshot

    • Forecast used- The snapshot, but note these were already closed periods when the snapshot was taken, so the comparison is a fit measure, not a prediction.

    • Actuals used- Actuals as they stood on snapshot day.

    • Watch for- One thing can move it after the fact: changing Error Sample Length sets a flag that makes the next Calculate recompute and overwrite the stored snapshot values. After that, historical numbers no longer match what was on screen back then

  • Average Error Vs Current Actuals

    • Question- “Judged on everything we know now, how good was that frozen forecast over the most recent full year?”

    • Window- 12 periods ending the last closed period anchored on TODAY

    • Forecast used- The snapshot frozen. The forecast does not move; the window does.

    • Actuals used- Today's actuals.

    • Watch for- This is the band that gets mis-explained. It is not the snapshot's own historical error, and it is not anchored to the snapshot date. It is a rolling window ending now. It therefore mixes lag periods (genuine forecast error) with pre-snapshot periods (fit). As a snapshot ages past the sample length, the pre-snapshot part drops out entirely and this band converges on the Lag Period number; while the snapshot is young, it is mostly the backward-looking part.

  • Current Forecast Errors

    • Question- “How well does the forecast as it stands right now line up with the last 12 closed periods?”

    • Window- 12 periods ending the last closed period

    • Forecast used- The live operational forecast. Its historical periods are never rewritten by the nightly run. The statistical forecast is only ever merged in from the current period forward.

    • Actuals used- Today's actuals.

    • Watch for- This is effectively a lag-0 number. Because the current period keeps being rewritten every night until it closes, each historical period holds the forecast as it stood at the end of that period — the last, best-informed version. That is why this band almost always looks better than the lead-time-ago bands, and it is the honest answer to “why does my error jump when I look at the snapshot columns?” It is not a bug and not a different formula; it is a shorter forecast horizon.

 

The lag comes from the Forecast Error Lag setting

Forecast Error Lag in Global Demand Forecast Settings chooses the lag snapshot. Set to a fixed lag of 1 to 12, every node uses the most recent snapshot on or before the start of the period that many forecast periods ago, and the columns read "Lag N". Set to Auto (lead time), the default, the LT Ago columns on the header grid pick the most recent snapshot on or before today − average planning lead time, where that lead time is the average of the node's active planning lead times, in days. So under Auto, “lead-time ago” means a different lag for every node, and LT Ago Lag is the resulting distance in periods. A reported outcome, not an input. Two items in the same category legitimately show different lags. The Compare Forecasts tab lists every retained snapshot with its own Lag, and highlights the row that is the lag snapshot.

TIP: Edison can report fixed-horizon accuracy, where each closed period is scored against the forecast made N periods before it, and how accuracy changes from lag 1 to lag 12. Try "What was our lag-3 wMAPE over the last 6 months for this item?"

How one number is built

Every band runs the same four-step loop over its own window. Only the window and the forecast change.

  1. Line the frozen forecast up against actuals, period by period, over the window. Negative forecast values are floored at zero first.
  2. Accumulate four running totals: sum of actuals, sum of forecast and sum of absolute errors, plus the sample count.
  3. Divide each by the sample count to get MAE, Average Usage and Average Forecast. These are what get stored.
  4. Derive the displayed percentages at read time: wMAPE = MAE ÷ Average Usage, which is the same as total units missed ÷ total units sold, Bias % = (Average Forecast ÷ Average Usage) − 1, and Error $ = MAE × the node's average unit cost.

Three consequences worth knowing: 

  • Blanks are deliberate. If total actuals in the window come to zero, or the absolute errors sum to zero, the calculation is marked invalid and the cells come back empty rather than as 0% or a divide-by-zero.
  • wMAPE is capped. The stored ratio is clamped at 100, i.e. 10,000%. A cell reading exactly 10,000.0% has hit the cap; the true figure is higher.
  • Error $ uses today's cost. Every band — including snapshots years old — values the miss at the node's current-period average unit cost, deliberately, so the dollar columns stay comparable across snapshots. It is not a historical cost.

Forecast Chart Tab


The Forecast Chart tab is the first tab and is shown by default. Its Trend Chart shows the forecast and actuals for the record selected in the Forecast Error Detail grid.

Snapshots Tab

The snapshots tab allows you to see more about each snapshot and how things have changed over time - see the Forecast Snapshots tab for more information.

Use the "Show Snapshot" select box at the top right of the tab to switch the grid between Forecast Quantity, Error Units, Error Percent, and Error Value. Below the grid, the Selected Snapshots and Per-Period Error Over Time charts plot the snapshots you select.

Since there are potentially a lot of snapshots, this table can be a bit easier to view if you use the "maximize" button to expand that chart panel when you are comparing forecasts closely.

Compare Forecasts

The Compare Forecasts tab allows you to compare measures from your different forecast series over different lag periods, to see how they are changing over time.

This tab is broken up into a grid and a chart displaying measures from the grid.

Compare Forecasts Grid

The grid columns are as follows:

  • Series - What is the forecast series for which this row applies?  Note that you can easily filter on this column so you can focus on one series at a time if you wish.  Also, you can group on this column as well.  You will see values like:
    • Benchmark - Values from the naive benchmark forecast.  These are backdated to each snapshot date during the nightly calculation.
    • Stat Forecast - Stat forecasts are backdated to each snapshot date during the nightly calculation (hiding data from the algorithm) so you can see "what would my stat forecast have been at the time?"
    • Operational Forecast - This will be values from your operational forecast snapshots at various time periods.
    • <Other Forecasts> - You may have other forecasts for which StockIQ will keep snapshots and be able to show at different lag periods.
  • Date - Shows the date of the snapshot/series that is being shown.  The lead-time-ago snapshot's date is highlighted.
  • Lag - The lag, in number of periods, that date corresponds to, e.g. 1 month ago, 2 months ago, etc.

After these are bands of different errors, showing the same "Vs Current", "Backwards Looking" and "Lag period" as described above.

 

Interesting things to do with this grid include:

  • Group by lag period to visually see errors of each series over time as they change
  • Filter on one series to see how its error changes over time
  • Filter on one lag period to see how each series compared at a given number of periods ago
  • Combine some of the above with Benchmark vs Stat or Stat vs Operational

Compare Forecasts Chart

The chart will display whatever error measure you select using the "Chart Measure" select box at the top right of the tab.  Use this to provide a nice visual of your key metric of choice over time, on the series(one or many) over time:

If Edison is enabled, use the Insights button at the right of the tabs to have Edison summarize this tab.

Display Options

There are a handful of display options to tweak what you see in the Forecast Error screen:

  • Select Hierarchy- Which forecast hierarchy you would like to view 
  • Hierarchy Level - Which level of the hierarchy at which to view your forecast error history data.  Typically this will be at your bottom level - item-site-forecast group.
  • Forecast Series - Which forecast series should be displayed in the forecast error detail grid, e.g. your operational forecast, or one of your supplemental forecast series.
  • Show Annotations- Shows annotations that were made in Forecast Manager

Alert Settings

You can change the alert settings in the Forecast Error Alert settings screen by clicking the "Alerts" button. The button is only shown to users with the Edit Forecast Alert Settings permission.

Troubleshooting

  • Keep in mind some metrics will be blank if historical data does not exist.  StockIQ stores snapshots of historical data such as forecasts as well as calculations at that time.   Also, StockIQ changes and updates may clear out some of the saved data.
  • Snapshots are kept for the Max Demand Forecast Snapshot Age set in Global Data Settings (180 days by default), so older snapshots will not appear. If the forecast series has no Reference Demand Series set, the error columns will be blank. Changing the Error Sample Length in Global Demand Forecast Settings overwrites the stored error values on the next calculation.

Tags/Metadata

Forecast Error Tab

 

Related articles