graphtoolbox.evaluation.significance

Statistical significance tools for comparing cached forecasts.

Every function operates on prediction arrays, never on model weights, so a full significance analysis runs on the cached forecasts without retraining anything. Given a set of forecasts and the ground truth, the module provides pairwise predictive-accuracy tests, bootstrap standard errors and confidence intervals for error metrics, and the Model Confidence Set.

References

Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy.

Journal of Business & Economic Statistics, 13(3), 253-263.

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of

prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.

Newey, W. K., & West, K. D. (1987). A simple, positive semi-definite,

heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica, 55(3), 703-708.

Hansen, P. R., Lunde, A., & Nason, J. M. (2011). The model confidence set.

Econometrica, 79(2), 453-497.

Kunsch, H. R. (1989). The jackknife and the bootstrap for general stationary

observations. The Annals of Statistics, 17(3), 1217-1241.

class graphtoolbox.evaluation.significance.DMResult(statistic: float, pvalue: float, mean_loss_diff: float, better: str, horizon: int, n: int)

Bases: object

Outcome of a Diebold-Mariano test between two forecasts.

statistic is the HLN-corrected DM statistic with sign convention loss(a) - loss(b): a positive value means forecast a has the larger loss (is worse). mean_loss_diff is that same difference in level, and better names the lower-loss forecast.

statistic: float

pvalue: float

mean_loss_diff: float

better: str

horizon: int

n: int

graphtoolbox.evaluation.significance.diebold_mariano(pred_a, pred_b, target, *, loss: str = 'squared', h: int = 1, names=('a', 'b')) → DMResult

Diebold-Mariano test of equal predictive accuracy between two forecasts.

Uses a Newey-West long-run variance truncated at h - 1 lags (the moving average order of h-step forecast-error differentials) and the Harvey-Leybourne-Newbold small-sample correction, comparing the statistic to a Student-t distribution with n - 1 degrees of freedom.

Parameters:

• pred_a (array-like) – The two forecast series (torch tensors or numpy arrays).

• pred_b (array-like) – The two forecast series (torch tensors or numpy arrays).

• target (array-like) – Ground-truth series.

• loss ({'squared', 'absolute', 'ape'}) – Loss under which accuracy is compared.

• h (int) – Forecast horizon; sets the autocovariance truncation lag to h - 1.

• names (tuple[str, str]) – Labels for the two forecasts, used in the better field.

Return type:

DMResult

graphtoolbox.evaluation.significance.pairwise_dm(preds: Dict[str, ndarray], target, *, loss: str = 'squared', h: int = 1, correction: str = 'holm')

All pairwise Diebold-Mariano tests over a set of forecasts.

Parameters:

• preds (dict[str, array-like]) – Mapping from model name to forecast series.

• target (array-like) – Ground-truth series.

• loss (see diebold_mariano().)

• h (see diebold_mariano().)

• correction ({'holm', 'none'}) – Multiple-testing adjustment applied to the collection of pairwise p-values (one per unordered pair).

Returns:

statistic and pvalue as name-indexed square pandas.DataFrame objects (statistic antisymmetric, p-values symmetric), pvalue_adjusted with the correction applied, and ranking ordering models by mean loss.

Return type:

dict

class graphtoolbox.evaluation.significance.BootstrapResult(metric: str, point: float, se: float, ci_low: float, ci_high: float, level: float, samples: ndarray)

Bases: object

Bootstrap summary for a scalar error metric.

metric: str

point: float

se: float

ci_low: float

ci_high: float

level: float

samples: ndarray

graphtoolbox.evaluation.significance.bootstrap_metric(pred, target, *, metric: str = 'rmse', n_boot: int = 2000, block_len: int = 48, level: float = 0.95, seed: int | None = 0) → BootstrapResult

Moving-block bootstrap standard error and CI for an error metric.

Resampling whole blocks of consecutive time steps preserves the temporal dependence of forecast errors, which an i.i.d. bootstrap would destroy. The default block length of 48 matches one day of half-hourly data.

Parameters:

• pred (array-like) – Forecast and ground-truth series.

• target (array-like) – Forecast and ground-truth series.

• metric ({'rmse', 'mae', 'mape', 'nmae', 'bias'}) – Metric whose sampling uncertainty is estimated.

• n_boot (int) – Number of bootstrap resamples.

• block_len (int) – Block length for the moving-block bootstrap.

• level (float) – Central confidence level for the percentile interval.

• seed (int or None) – Seed for reproducibility.

Return type:

BootstrapResult

graphtoolbox.evaluation.significance.bootstrap_table(preds: Dict[str, ndarray], target, *, metric: str = 'rmse', n_boot: int = 2000, block_len: int = 48, level: float = 0.95, seed: int | None = 0)

Bootstrap point estimate, standard error and CI for every forecast.

Returns a pandas.DataFrame sorted by the metric, one row per model.

class graphtoolbox.evaluation.significance.MCSResult(included: list, pvalues: object, eliminated_order: list, alpha: float)

Bases: object

Outcome of the Model Confidence Set procedure.

included: list

pvalues: object

eliminated_order: list

alpha: float

graphtoolbox.evaluation.significance.model_confidence_set(preds: Dict[str, ndarray], target, *, loss: str = 'squared', alpha: float = 0.1, n_boot: int = 2000, block_len: int = 48, seed: int | None = 0) → MCSResult

Model Confidence Set with the range/T-max statistic and block bootstrap.

Iteratively tests the hypothesis of equal predictive ability across the surviving models and eliminates the worst one until the hypothesis is no longer rejected. The surviving set contains, with probability at least 1 - alpha, the best model. Each model receives an MCS p-value; models with an MCS p-value above alpha form the confidence set.

Parameters:

• preds (dict[str, array-like]) – Mapping from model name to forecast series.

• target (array-like) – Ground-truth series.

• loss ({'squared', 'absolute', 'ape'}) – Loss under which the models are compared.

• alpha (float) – Size of the test; the confidence set has level 1 - alpha.

• n_boot (int) – Moving-block bootstrap settings (shared across elimination steps).

• block_len (int) – Moving-block bootstrap settings (shared across elimination steps).

• seed (int or None) – Seed for reproducibility.

Return type:

MCSResult