run_out_of_sample_exercise()

Run out-of-sample exercise to evaluate model performance.

Usage

run_out_of_sample_exercise(
    df,
    macro_names,
    region_names,
    region_covariate_names,
    n_factors=4,
    aggregate_measure="gva_q_on_q",
    aggregation_region="uk",
    region_measure="gva_q_on_4q",
    region_q_on_q_measure=None,
    region_weights=None,
    region_weights_concentration=None,
    step_size=1,
    init_chunk_size=20,
    lag_qtrs=6,
    lag_qtrs_qoq=1,
    n_its=100000,
    n_posterior_samples=3000
)

Masks the most recent annual regional data by lag_qtrs quarters and fits the model in rolling chunks. Out-of-sample nowcasts and outturns for each step are collected and returned.

Parameters

df: pd.DataFrame

Dataframe containing relevant columns.

macro_names: list[str]

Names of macro series.

region_names: list[str]

Names of regions.

region_covariate_names: list[str]

Names of by-region covariates.

n_factors: int = 4

Number of factors. Defaults to 4.

aggregate_measure: str = "gva_q_on_q"

Nation-wide measure. Defaults to “gva_q_on_q”.

aggregation_region: str = "uk"

Top level geography. Defaults to “uk”.

region_measure: str = "gva_q_on_4q"

Growth measure regional. Defaults to “gva_q_on_4q”.

region_q_on_q_measure: str | None = None

Optional measure name in df supplying published quarterly (q-on-q) regional growth rates as a hard clamp on y_reg. Same format as every other measure (datetime | measure | region | value); values must be decimal growth rates. Inside the out-of-sample window this exercise NaN-masks these rows for every step (as with region_measure), so published quarterly values inside the OOS window do not leak into the nowcast. Older published values remain as clamps. Defaults to None (no q-on-q clamp, backwards compatible).

region_weights: dict[str, float] | None = None

Optional weight of each region in the national aggregate, passed to Ambric at every step. When None the prior centre for the weights is estimated at each step from the data available at that step.

region_weights_concentration: float | None = None

Tightness of the prior on the weights, passed to Ambric.

step_size: int = 1

Quarters to advance per OOS step. Defaults to 1.

init_chunk_size: int = 20

Initial learning window size. Defaults to 20.

lag_qtrs: int = 6

How many quarters before annual regional data are published; drives the OOS mask for region_measure. Tuned for the ONS regional annual GVA release (~6 quarters). Defaults to 6.

lag_qtrs_qoq: int = 1

How many quarters before quarterly regional data are published; drives a separate OOS mask for region_q_on_q_measure. Scot Gov quarterly GDP publishes with ~1 quarter lag, so a smaller value than lag_qtrs is realistic. Only used when region_q_on_q_measure is not None. Defaults to 1.

n_its: int = 100000

Iterations of ADVI for Bayesian inference. Defaults to 100000.

n_posterior_samples: int = 3000

Samples of the posterior. Defaults to 3000.

Returns

pd.DataFrame

pd.DataFrame: Combined out-of-sample nowcasts and outturns across all rolling steps.