Chapter 1

Getting Started

Core orientation and the minimum steps to inspect the active backtest harness.

Subsections of Getting Started

Overview

DSAMbayes-backtest is a dedicated repository for a reproducible walk-forward backtest harness for DSAMbayes marketing mix models.

The repo exists to keep the evaluation contract outside the source modelling repositories. It compares:

  • original DSAMbayes
  • DSAMbayes-Charles-Dev

on the same data, folds, and scoring rules.

What The Repo Owns

  • the manifest and replay contract
  • blocked walk-forward fold planning
  • common KPI-scale holdout scoring
  • adjacent-refit stability analytics
  • repo-owned result artifacts and summary tables
  • a short reporting surface for engineering and peer-review use

Current Scope

The current milestone is intentionally narrow:

  • one single-series time-series pilot at a time
  • one shared comparison contract across both repos
  • one shared external scorer
  • explicit guardrails around lagged and rolling features

Hierarchical / panel backtesting is deferred.

Current Worked Example

The active engineering example is _st, run under an explicit engineering-only scale = FALSE policy. On that example, the repo currently supports:

  • forward holdout scoring
  • parameter stability
  • provisional fixed-budget recommendation stability

The engineering worked example is not the final stakeholder-facing verdict. The intended next substantive use is the UK dataset once it is available.

Quickstart

This is the minimum path to inspect the active backtest harness.

1. Validate The Active Manifest

Rscript scripts/dsambayes-backtest.R validate \
  --manifest .planning/research/pilot_manifest.yaml

This checks the active pilot manifest, dataset contract, repo paths, and fold schedule.

2. Inspect The Planned Run Matrix

Rscript scripts/dsambayes-backtest.R plan \
  --manifest .planning/research/pilot_manifest.yaml

This prints the planned repo-by-fold run surface.

3. Do A Dry Run

Rscript scripts/dsambayes-backtest.R run \
  --manifest .planning/research/pilot_manifest.yaml \
  --dry-run

This writes run-scoped result directories and status artifacts without fitting the external repos.

4. Inspect The Current Worked Example

The completed _st engineering example is under:

results_engineering_m1_st_full/_st/engineering_m1_st_scale_false/
run_id=20260407T211743.943118Z__all-repos__all-folds__live/

Key summary files:

  • summary/holdout_summary.csv
  • summary/parameter_stability_summary.csv
  • summary/recommendation_stability_summary.csv

5. Read The Report

See:

That report is the current compact worked example for the library.