Time Series Forecast Metrics Selection Guide for Different Scenarios

Map forecast accuracy metrics to business scenarios: choose MAE, RMSE, MASE, WAPE, pinball loss or CRPS by cost structure, zeros and horizon.

Time Series Forecast Metrics Selection Guide for Different Scenarios

No single accuracy metric is correct for every forecast. The metric you choose encodes what your organization is punished for: a stockout, a missed revenue target, a throttled service, an overstaffed shift. Selecting one is therefore a business-cost decision expressed in statistical notation, and the statistics should follow the cost, not the other way round.

This guide covers point, quantile, interval and distributional metrics, and maps them to five recurring enterprise scenarios: demand and inventory, revenue and finance, capacity and infrastructure load, staffing, and the zero-heavy series that cut across all of them. Each scenario section states the decision the forecast drives, the dominant error cost, a recommended primary metric, a diagnostic companion, and the metric to avoid.

Every section assumes the same evaluation discipline: compare against naive and seasonal-naive baselines, validate with rolling-origin cross-validation rather than a single split, and measure at the horizons the business actually acts on. A metric without that protocol is a number, not evidence.

Last refreshed: 20 November 2026.

What Forecast Accuracy Metrics Can and Cannot Tell You

A metric measures forecast error against a stated loss. It cannot tell you whether the forecast is useful for a decision until you know which errors the decision punishes and how heavily. Four families cover most practice, and each carries a documented boundary.

Scale-dependent measures. Mean absolute error (MAE) and root mean squared error (RMSE) are expressed in the units of the data. The FPP3 accuracy chapter notes that measures based only on the error term are scale-dependent and cannot be used to compare series in different units. Within one series, MAE and RMSE differ in what they reward: squared-error measures optimize for the mean, while absolute-error measures such as MAE and MASE optimize for the median, a distinction that matters on skewed distributions such as intermittent demand (Hewamalage, Ackermann & Bergmeir, 2023).

Percentage-based measures. MAPE, sMAPE, RMSPE and WAPE divide the error by the actual value or by a total. They are scale-free, which makes them popular for comparing many series. They are also the family with the most documented failure modes: percentage errors are infinite or undefined when any actual is zero, extremely skewed when an actual is close to zero, and they assume the unit of measurement has a meaningful zero, so a percentage error is meaningless for temperature on the Celsius or Fahrenheit scale (Hyndman & Koehler, 2006). MAPE and MdAPE also penalize positive errors more heavily than negative ones, which is what prompted the "symmetric" variants; those in turn are not as symmetric as the name suggests and can take negative values.

Scaled measures. MASE divides the mean absolute error by the in-sample MAE of the naive forecast, so a value below one means the forecast beats the average naive forecast on the training data, and above one means it is worse (Hyndman, "Measuring forecast accuracy"). RMSSE is the squared-error analogue. Because numerator and denominator share the data's scale, MASE is comparable across series and remains defined when actuals hit zero.

Relative and counting measures. Relative measures divide a method's error by a benchmark's error; percentage-better scores count how often a method beats the benchmark across series and time steps. Both are useful for ranking many candidate models, and both inherit the benchmark's competence at the horizon you care about.

Two limits apply to all four families. First, a metric is a proxy for a loss function, and a proxy that does not match the cost structure will rank models in the wrong order. Second, a metric computed on fitted values tells you how well a model describes the past, not how well it forecasts; Hyndman's guidance is to always compute accuracy on test data not used when estimating the model. The sections that follow turn these boundaries into scenario-specific choices.

How to Map Metrics to Enterprise Forecasting Scenarios

The mapping below is an illustrative inference from the cost structure of each decision, not observed benchmark data. Use it to choose a starting metric, then validate the choice against your own error costs.

Scenario

Decision the forecast drives

Dominant error cost

Primary metric

Secondary diagnostic

Avoid, and why

Demand and inventory

Reorder points, safety stock, allocation

Stockout cost usually exceeds holding cost; symmetric at the SKU level

MASE against seasonal naive

WAPE at category level; fill-rate tracking

MAPE — zeros and small counts make it undefined or explosive

Revenue and finance

Plan, guidance, budget allocation

Aggregate bias; small-value noise

WAPE on the consolidated figure

Mean error (bias) by period

Item-level MAPE when the decision is made on the total

Capacity and infrastructure load

Provisioning, autoscaling thresholds, headroom

Under-forecast of peaks; over-forecast wastes spend

Pinball loss at the upper quantile

Interval coverage at nominal level

MAE alone — a low mean error can still miss every peak

Staffing and scheduling

Shift coverage, service-level commitments

Two-sided: under-coverage harms service and compliance, over-coverage costs labor

Pinball loss at the service-level quantile

Coverage check per shift

Symmetric point error treated as the whole story

Zero-heavy or intermittent series (any scenario)

Any of the above

Undefined or distorted percentage error

MASE or WAPE

Percentage of series beating naive

MAPE, sMAPE, RMSPE

Three patterns run through the table. Where the cost is roughly symmetric, a scaled absolute-error measure is defensible and easy to explain. Where the cost is asymmetric, no point metric captures it and you need a quantile or interval metric. Where the data contains zeros or small counts, the percentage family drops out regardless of scenario. The subsections explain the reasoning behind each row.

Demand and Inventory Planning

Intermittent and lumpy demand makes percentage errors unusable, so the defensible default is a scaled measure such as MASE evaluated against a seasonal-naive baseline, with WAPE as a category-level companion. Note that fill-rate and stockout costs are asymmetric even when the SKU-level error metric is symmetric; the pinball-loss section shows how to price that asymmetry into a quantile target.

Revenue and Financial Forecasting

Finance acts on the consolidated number, so judge the forecast there: report WAPE and mean error (bias) on the total, since WAPE survives zero and small values that break MAPE. Optimizing item-level MAPE while the decision is made on the aggregate is a mismatch — the hierarchy section explains how level-by-level reporting exposes it.

Capacity and Infrastructure Load Planning

Provisioning is driven by upper quantiles and interval coverage, not mean error. A model with low MAE can still under-provision if it misses peaks, so score the quantile you actually provision against with pinball loss and check empirical coverage; the probabilistic evaluation section covers the full toolkit.

Staffing and Workforce Scheduling

Staffing is a two-sided cost problem: under-coverage carries service and compliance consequences, over-coverage carries direct labor cost. Forecast and score at the quantile implied by your service-level target, then verify coverage per shift rather than trusting a single aggregate error figure.

How to Handle Zeros, Intermittent Demand and Near-Zero Actuals

The zero problem has a decision procedure, not a workaround: identify whether your series contain zeros or small counts, drop the percentage-error family if they do, and move to MASE or WAPE.

Hyndman and Koehler tested the standard measures on three problematic M3 series — one with zeros from intermittent demand, one with negative observations, and one with consecutive repeated values — and found that of the measures examined, only MASE remained usable across all three, because the others produced infinite or undefined values (Hyndman & Koehler, 2006). They also note that the M3 competition avoided large MAPEs by including only positive data, an artificial solution that cannot be applied in practice.

Illustrative arithmetic (not observed data). Suppose a SKU's actual demand over four periods is 0, 0, 2 and 40, and the forecast is 1 in each period. The absolute errors are 1, 1, 1 and 39. MAPE cannot be computed at all, because two periods divide by zero. If you silently drop those periods, MAPE becomes the average of 50% and 97.5%, about 73.75% — a figure driven almost entirely by the period where the actual was 2. WAPE, by contrast, is 42 divided by 42, or 100%, which correctly reflects that the forecast missed the one large period badly. The lesson is not that WAPE is always better; it is that a single near-zero actual can dominate a percentage average and make the number uninterpretable.

WAPE is the practical escape for aggregate reporting because it divides total absolute error by total actual volume, so zeros contribute to the denominator without creating a division by zero. MASE is the escape for cross-series comparison because it scales by the naive forecast's in-sample error. One caution: measures that optimize for the median, including MASE and MdAE, can behave oddly on intermittent series, because constant zeros pull the median toward zero (Hewamalage et al., 2023). Pair a median-based measure with a mean-based diagnostic such as WAPE rather than relying on either alone.

Data characteristic

Use

Avoid

Zeros in actuals

MASE, WAPE

MAPE, sMAPE, RMSPE

Small counts (intermittent demand)

MASE against seasonal naive; WAPE at aggregate

Any per-period percentage error

Negative values

MAE, RMSE, MASE

Percentage errors

Consecutive repeated values

MASE

Measures that divide by the actual

Arbitrary zero point (e.g., temperature)

MAE, RMSE

All percentage errors

Many series on different scales

MASE, RMSSE, relative measures

Raw MAE or RMSE across series

How to Connect Asymmetric Business Costs to Pinball Loss and Quantile Selection

When over-forecasting and under-forecasting cost different amounts, the right metric is pinball (quantile) loss, and the right forecast is a quantile whose level is derived from the cost ratio. The quantile level is a business input, not a statistical default.

Pinball penalty ratio τ/(1−τ) by target quantile Line chart of the under-forecast to over-forecast penalty ratio τ/(1−τ) across target quantile levels from 0.05 to 0.95. The ratio is 1:1 at the median (τ = 0.5), 9:1 at τ = 0.9, and 1:9 at τ = 0.1, showing how the target quantile encodes the asymmetry of business costs. Pinball penalty ratio τ/(1−τ) by target quantile. line data: 0.05 0.053; 0.10 0.111; 0.20 0.25; 0.30 0.429; 0.40 0.667; 0.50 1; 0.60 1.5; 0.70 2.333; 0.80 4; 0.90 9; 0.95 19.Source: Gneiting & Raftery (2007), Strictly proper scoring rules, prediction, and estimation, eq. 41 . Pinball penalty ratioτ/(1−τ) by target quantile 0 5 10 15 20 0.053 0.05 0.111 0.10 0.25 0.20 0.429 0.30 0.667 0.40 1 0.50 1.5 0.60 2.333 0.70 4 0.80 9 0.90 19 0.95 Source: Gneiting & Raftery (2007), Strictly properscoring rules, prediction, and estimation, eq. 41

Definitional ratio of the pinball-loss penalty on an under-forecast to the penalty on an over-forecast, τ/(1−τ), computed from the tick-loss definition in Gneiting & Raftery (2007), eq. 41. This is arithmetic from the loss definition, not an empirical result; the quantile level itself is a business input derived from the under- to over-forecast cost ratio.

Pinball loss scores a quantile forecast by penalizing misses asymmetrically: an observation above the forecast is penalized in proportion to the target quantile level, and an observation below it in proportion to one minus that level. FPP3 defines the quantile score this way and notes that a low value indicates a better estimate of the quantile (FPP3 §5.9). The score is proper: Gneiting and Raftery show that the class of proper scoring rules for quantile forecasts is larger than previously conjectured, and propriety is what makes honest quantile reporting the score-maximizing strategy rather than a hedge (Gneiting & Raftery, 2007).

Illustrative calculation (not observed data). Suppose a stockout costs 9 per unit of unmet demand and holding a surplus unit costs 1. The cost ratio of under- to over-forecasting is 9:1. The cost-minimizing quantile is the under-forecast cost divided by the sum of both costs, or 9 / (9 + 1) = 0.90. Forecast the 90th percentile of demand and score it with pinball loss at p = 0.90. If the ratio were reversed — over-forecasting four times as expensive as under-forecasting — the target quantile would be 1 / (1 + 4) = 0.20. The formula is a transparent consequence of the two costs; the inputs are the part you must get from finance and operations, and they should be documented alongside the metric.

Two practical consequences follow. First, report pinball loss at the quantile you act on, not an average across quantiles, because a model can be strong at the median and weak in the tail that drives the decision. Second, keep a point-forecast metric as a diagnostic for the central tendency, but do not let it override the quantile score when the cost is asymmetric. A model that wins on MAE and loses on pinball loss at p = 0.90 is the wrong model for a stockout-sensitive decision.

How to Evaluate Probabilistic and Interval Forecasts

Coverage alone is not enough. A prediction interval can hit its nominal coverage and still be a poor forecast if it is unnecessarily wide, which is why interval evaluation must score width and misses together.

Coverage, width and interval score for three 95% interval forecasts Grouped bar chart comparing three 95% interval forecasts (I, J, K) on empirical coverage (%), average width, and average interval score. Coverage is nearly identical across all three, while width and interval score differ substantially, illustrating why coverage alone cannot rank interval forecasts. Coverage, width and interval score for three 95% interval forecasts. grouped bar data: Interval I: Empirical coverage (%) 95.01, Average width 4, Average interval score 4.77; Interval J: Empirical coverage (%) 95.08, Average width 5.45, Average interval score 8.04; Interval K: Empirical coverage (%) 94.98, Average width 3.79, Average interval score 5.32.Source: Gneiting & Raftery (2007), Table 2 . Coverage, width and intervalscore for three 95% intervalforecasts Empiricalcoverage (%) Average width Averageinterval score 0 25 50 75 100 95.01 4 4.77 Interval I 95.08 5.45 8.04 Interval J 94.98 3.79 5.32 Interval K Source: Gneiting & Raftery (2007), Table 2

One-step-ahead 95% interval forecasts for a stationary bilinear process, from Gneiting & Raftery (2007), Table 2 (100,000 sequential forecasts on a sample path of length 100,001). All three intervals are calibrated to within a fraction of a percentage point of nominal coverage, yet the interval score separates them; the narrowest interval (K) is not the winner because it pays for its width with misses. Scores are negatively oriented, so lower is better.

Gneiting and Raftery's comparison of three 95% interval forecasts for a stationary bilinear process makes the point concretely. Interval I achieved 95.01% empirical coverage with an average width of 4.00 and an interval score of 4.77; interval J achieved 95.08% coverage but with a width of 5.45 and a score of 8.04; interval K achieved 94.98% coverage with a width of 3.79 and a score of 5.32 (Gneiting & Raftery, 2007). All three are calibrated to within a fraction of a percentage point, yet they are not equally good: the interval score separates them, and the narrowest interval (K) is not the winner either, because it pays for its width with misses.

The governing principle is to maximize sharpness subject to calibration. Calibration is the statistical consistency between the forecast distribution and the observations; sharpness is the concentration of the forecast distribution and is a property of the forecast alone. A wide interval is easy to calibrate and useless for decisions; a narrow interval that misses is worse. The interval score, also called the Winkler score, formalizes the trade-off: it equals the interval length plus a penalty proportional to how far an observation falls outside the interval, so low scores favor narrow intervals that still cover (FPP3 §5.9).

For a full distribution rather than a single interval, use CRPS. It generalizes absolute error and reduces to absolute error when the forecast is a point measure, which lets you compare deterministic and probabilistic forecasts on the same scale (Gneiting & Raftery, 2007). In practice, many teams aggregate quantile scores across the levels they forecast into a weighted interval score; whatever the aggregate, score the levels you act on.

Report the following for any probabilistic forecast:

  • Nominal versus empirical coverage at each interval level you publish, computed on held-out data.

  • Average interval width at each level, so a coverage win cannot hide a width blowout.

  • Interval score or Winkler score per level, which combines width and misses into one comparable number.

  • CRPS if you produce a full predictive distribution, reported in the units of the observation.

  • Pinball loss at each quantile used in a decision, reported separately rather than averaged away.

  • Calibration plot or reliability diagram by horizon, since calibration often degrades as the horizon lengthens.

Why Aggregate and Hierarchy Metrics Can Mask Item-Level Failure

A low total-level error can coexist with severe item-level failure, so a forecast that looks accurate on the consolidated number may still be unusable for the decisions taken below it. Hyndman and Koehler note that scale-dependent measures such as MAE and RMSE cannot compare series in different units, and that percentage-based measures behave badly when actuals are small — the same properties that make an aggregate figure easy to report also make it easy to misread. The practical consequence: report the metric at every level where someone acts on it, and add a distributional view rather than a single average.

Three reporting layers cover most enterprise hierarchies:

  • Total and category level — the number finance and executive reporting uses. Useful for bias and direction, weak for detecting which series failed.

  • SKU, location or service level — the level where replenishment, staffing and provisioning decisions are actually made.

  • Distributional summary — the share of series that beat the naive baseline, and the worst-decile error, so a small number of badly failing series cannot hide inside a good average.

Reconciliation across hierarchy levels is a separate decision from evaluation. Whether bottom-up, top-down or middle-out reconciliation improves or degrades accuracy depends on the method and the data, so measure the reconciled forecast at each level rather than assuming the total-level improvement carries downward. Hewamalage, Ackermann and Bergmeir document how aggregate error measures and weak benchmarks can make methods appear competitive when they are not, and recommend benchmark-relative evaluation across many series rather than a single pooled score. The acceptance checklist below turns this into reportable items.

What Evaluation Protocol Should Accompany Any Metric Choice

A metric is only as trustworthy as the protocol that produced it: evaluate on data not used to fit the model, use rolling-origin cross-validation rather than a single split, and always report skill against naive and seasonal-naive baselines. Hyndman states the first rule directly — accuracy can only be determined by considering how well a model performs on new data that were not used when estimating it, and a model that fits the data well does not necessarily forecast well (Measuring forecast accuracy).

Held-out evaluation. Keep a test period the model never saw, including any preprocessing statistics fitted on the training window. Fitted values and residuals describe in-sample behaviour and are not a substitute for out-of-sample accuracy (FPP3 §5.8).

Rolling-origin validation. Time series cross-validation builds a series of test sets, each using only observations that occurred prior to the observation being forecast, so no future data enters the model (FPP3 §5.10). For multi-step decisions, modify the procedure so the errors evaluated match the horizon you actually commit to — a model tuned on one-step errors can rank differently at four steps.

Baselines and skill. Fit naive and seasonal-naive methods on the same folds and report the candidate's error relative to them. A method that cannot beat the naive forecast on a random-walk-like series has no demonstrated predictive value, and apparent superiority in that setting is likely chance (Hewamalage, Ackermann and Bergmeir, Forecast evaluation for data scientists).

Leakage control. Split before normalising, smoothing or extracting features, and check that no series in the training set contains information about another series' future. Leakage in preprocessing is one of the most common ways a strong-looking result collapses in production.

Significance. When comparing methods across many series, a difference in average error may be noise. Pairwise tests such as Diebold-Mariano or Wilcoxon rank-sum apply to two forecasts at a time; comparing more requires a multiple-testing correction such as Bonferroni.

Own-environment validation. Vendor-published performance figures, including benchmark claims, are claims to reproduce on your data and horizons, not established results to adopt.

Metric Selection Acceptance Checklist

Use this list before a forecast metric goes into a dashboard, a model-selection decision or a planning cycle. Each item maps to a section above; an unchecked item is a known gap, not a formality.

  • Cost structure matched. The primary metric reflects whether over- and under-forecasting carry symmetric or asymmetric costs for this decision.

  • Zero and near-zero handling verified. MAPE, sMAPE and RMSPE are excluded, or the data is confirmed free of zeros and small counts; WAPE or MASE is used where they are not.

  • Baseline comparison included. Naive and seasonal-naive errors are computed on the same folds and reported alongside the candidate.

  • Rolling-origin validation used. Evaluation uses time series cross-validation, not a single train/test split.

  • Horizons matched to decisions. Accuracy is reported at the lead times the business actually plans against, not only at one step.

  • Hierarchy levels reported. Metrics appear at total, category and item/location levels, plus a share-of-series-beating-baseline and worst-decile view.

  • Probabilistic metrics included where costs are asymmetric. Quantile or interval forecasts are scored with pinball loss, the Winkler/interval score or CRPS.

  • Calibration checked. Empirical coverage is compared with nominal coverage, and interval width is reported so a wide interval cannot pass on coverage alone.

  • Failure modes documented. The chosen metric's known weaknesses are written down next to its definition, so future readers do not rediscover them.

  • Review date set. The metric choice is scheduled for review when the cost structure, data characteristics or decision horizon changes.

Frequently Asked Questions

Can I just use MAPE? No, not as a general default. Percentage errors are infinite or undefined when an actual is zero and extremely skewed when actuals are near zero, and they assume a meaningful zero point, which rules them out for scales like temperature (Hyndman & Koehler, Another look at measures of forecast accuracy). Where zeros and small counts are absent and the quantity has a true zero, MAPE remains usable for cross-series comparison.

Is RMSE always better than MAE? No. RMSE squares the errors, so it penalizes large misses more heavily; MAE treats all errors linearly. That is a statement about which cost you care about, not about statistical quality. Choose RMSE when large errors are disproportionately expensive, MAE when the typical miss matters more.

What is a good MASE value? Below 1 means the forecast beats the average naive forecast computed on the training data; above 1 means it is worse (Measuring forecast accuracy). For seasonal series, scale against the seasonal-naive error instead. Because the scaling uses one-step in-sample errors, multi-step MASE values are typically larger than one even for a good method.

How many quantiles should I forecast? As many as the decisions require, and score each one with pinball loss at its own level. A single median forecast serves symmetric costs; provisioning and service-level decisions usually need an upper quantile, sometimes several.

Should I optimize the metric I report? Only when it is a proper scoring rule aligned to the business cost. Pinball loss, CRPS and the interval score are proper, so they reward honest forecasts rather than hedging (Gneiting & Raftery, Strictly proper scoring rules, prediction, and estimation). Optimizing an improper or misaligned metric invites gaming.

How often should the metric choice be revisited? Whenever the cost structure, data characteristics or decision horizon changes — for example when a product line moves from intermittent to steady demand, or when a new service-level target changes the asymmetry of under- and over-coverage.

Further Reading and Sources

These are the primary references behind the framework above, listed for readers who want the full derivations rather than the summary. Inline links in the body point to the specific chapters and papers used; this list is a navigation aid for deeper study. Last reviewed: 20 November 2026.

  • Forecasting: Principles and Practice (3rd ed.) — §5.8 Evaluating point forecast accuracy for scale-dependent, percentage, scaled and relative measures and their failure modes; §5.9 Evaluating distributional forecast accuracy for quantile scores, the Winkler score and CRPS; §5.10 Time series cross-validation for rolling-origin evaluation.

  • Hyndman & Koehler (2006), Another look at measures of forecast accuracy — PDF. The source for MASE, the critique of percentage-error measures, and the M3 evidence on which measures survive zeros and repeated values.

  • Hyndman (2014), Measuring forecast accuracy — PDF. A short, practical treatment of train/test splits and measure selection.

  • Gneiting & Raftery (2007), Strictly proper scoring rules, prediction, and estimation — PDF. Propriety of quantile and interval scores, CRPS, and the interval-score comparison table.

  • Hewamalage, Ackermann & Bergmeir (2023), Forecast evaluation for data scientists: common pitfalls and best practices — PDF. Benchmark selection, data leakage, measure-selection tables and significance testing.

Related Articles