Metrics Reference

Forward Holdout Metrics

  • RMSE Root mean squared error on the observed KPI scale.

  • WMAPE Weighted mean absolute percentage error on the observed KPI scale.

  • Mean error Signed bias on the observed KPI scale.

  • SMAPE Secondary holdout metric on the observed KPI scale.

  • Holdout ELPD / log score Secondary probabilistic diagnostics when compatible posterior outputs are available.

Stability Metrics

  • standardized_posterior_shift Adjacent-refit coefficient shift scaled by posterior uncertainty.

  • allocation_turnover 0.5 * sum(abs(w_t - w_t-1)) across matched channels.

  • marginal_response_rank_corr Spearman correlation of channel marginal-response ranks across adjacent refits on the shared recommendation surface.

Important note:

  • marginal_response_rank_corr is not ROI. It is a rank comparison on the repo-owned recommendation surface. The metric was deliberately renamed from a previous ROI-style label because the current allocator does not compute true ROI.

Interpretation Guidance

  • Holdout metrics address predictive performance.
  • Parameter stability addresses how much posterior media effects move between adjacent refits.
  • Recommendation stability addresses how much the recommended allocation surface moves between adjacent refits under one controlled comparison scenario.

Recommendation stability in the current repo should still be treated as provisional, because the allocator surface is backtest-owned and not yet an owner-approved production policy.