Probabilistic forecast reconciliation: Properties, evaluation and score optimisation (original) (raw)

Probabilisitic forecasts in hierarchical time series

2018

Forecast reconciliation involves adjusting forecasts to ensure coherence with aggregation constraints. We extend this concept from point forecasts to probabilistic forecasts by redefining forecast reconciliation in terms of linear functions in general, and projections more specifically. New theorems establish that the true predictive distribution can be recovered in the elliptical case by linear reconciliation, and general conditions are derived for when this is a projection. A geometric interpretation is also used to prove two new theoretical results for point forecasting; that reconciliation via projection both preserves unbiasedness and dominates unreconciled forecasts in a mean squared error sense. Strategies for forecast evaluation based on scoring rules are discussed, and it is shown that the popular log score is an improper scoring rule with respect to the class of unreconciled forecasts when the true predictive distribution coheres with aggregation constraints. Finally, evid...

Probabilistic Forecasts, Calibration and Sharpness JRSSB Submission B6257 Revision 1

2000

Probabilistic forecasts of a continuous variable take the form of predictive densities or pre- dictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and

Combining and Evaluating Probabilistic Forecasts

2009

In presenting this dissertation in partial fulfillment of the requirements for the doctoral degree at the University of Washington, I agree that the Library shall make its copies freely available for inspection. I further agree that extensive copying of the dissertation is allowable only ...

Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Monthly Weather Review, 2015

Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

Forecast reconciliation: A geometric view with new insights on bias correction

International Journal of Forecasting, 2021

A geometric interpretation is developed for so-called reconciliation methodologies used to forecast time series that adhere to known linear constraints. In particular, a general framework is established nesting many existing popular reconciliation methods within the class of projections. This interpretation facilitates the derivation of novel results that explain why and how reconciliation via projection is guaranteed to improve forecast accuracy with respect to a specific class of loss functions. The result is also demonstrated empirically. The geometric interpretation is further used to provide a new proof that forecast reconciliation results in unbiased forecasts provided the initial base forecasts are also unbiased. Approaches for dealing with biased base forecasts are proposed and explored in an extensive empirical study on Australian tourism flows. Overall, the method of bias-correcting before carrying out reconciliation is shown to outperform alternatives that only bias-correct or only reconcile forecasts.

Probabilistic recalibration of forecasts

International Journal of Forecasting, 2019

We present a scheme by which a probabilistic forecasting system whose predictions have poor probabilistic calibration may be recalibrated by incorporating past performance information to produce a new forecasting system that is demonstrably superior to the original, in that one may use it to consistently win wagers against someone using the original system. The scheme utilizes Gaussian process (GP) modeling to estimate a probability distribution over the Probability Integral Transform (PIT) of a scalar predictand. The GP density estimate gives closed-form access to information entropy measures associated with the estimated distribution, which allows prediction of winnings in wagers against the base forecasting system. A separate consequence of the procedure is that the recalibrated forecast has a uniform expected PIT distribution. A distinguishing feature of the procedure is that it is appropriate even if the PIT values are not i.i.d. The recalibration scheme is formulated in a framework that exploits the deep connections between information theory, forecasting, and betting. We demonstrate the effectiveness of the scheme in two case studies: a laboratory experiment with a nonlinear circuit and seasonal forecasts of the intensity of the El Niño-Southern Oscillation phenomenon.

Probabilistic forecasts, calibration and sharpness

Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2007

Probabilistic forecasts of a continuous variable take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple game-theoretic framework allows us to distinguish probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform (PIT) histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy center in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Forecast combination based forecast reconciliation: insights and extensions

2021

In a recent paper, while elucidating the links between forecast combination and cross-sectional forecast reconciliation, Hollyman et al. (2021) have proposed a forecast combination-based approach to the reconciliation of a simple hierarchy. A new Level Conditional Coherent (LCC) point forecast reconciliation procedure was developed, and it was shown that the simple average of a set of LCC, and bottom-up reconciled forecasts (called Combined Conditional Coherent, CCC) results in good performance as compared to those obtained through the state-of-the-art cross-sectional reconciliation procedures. In this paper, we build upon and extend this proposal along some new directions. (1) We shed light on the nature and the mathematical derivation of the LCC reconciliation formula, showing that it is the result of an exogenously linearly constrained minimization of a quadratic loss function in the differences between the target and the base forecasts with a diagonal associated matrix. (2) Endo...