Overview

DSAMbayes-backtest is a dedicated repository for a reproducible walk-forward backtest harness for DSAMbayes marketing mix models.

The repo exists to keep the evaluation contract outside the source modelling repositories. It compares:

  • original DSAMbayes
  • DSAMbayes-Charles-Dev

on the same data, folds, and scoring rules.

What The Repo Owns

  • the manifest and replay contract
  • blocked walk-forward fold planning
  • common KPI-scale holdout scoring
  • adjacent-refit stability analytics
  • repo-owned result artifacts and summary tables
  • a short reporting surface for engineering and peer-review use

Current Scope

The current milestone is intentionally narrow:

  • one single-series time-series pilot at a time
  • one shared comparison contract across both repos
  • one shared external scorer
  • explicit guardrails around lagged and rolling features

Hierarchical / panel backtesting is deferred.

Current Worked Example

The active engineering example is _st, run under an explicit engineering-only scale = FALSE policy. On that example, the repo currently supports:

  • forward holdout scoring
  • parameter stability
  • provisional fixed-budget recommendation stability

The engineering worked example is not the final stakeholder-facing verdict. The intended next substantive use is the UK dataset once it is available.