Combining Non-Replicable Forecasts (original) (raw)

Evaluating Individual and Mean Non-Replicable Forecasts

2011

Macroeconomic forecasts are often based on the interaction between econometric models and experts. A forecast that is based only on an econometric model is replicable and may be unbiased, whereas a forecast that is not based only on an econometric model, but also incorporates expert intuition, is non-replicable and is typically biased. In this paper we propose a methodology to analyze the qualities of individual and means of non-replicable forecasts. One part of the methodology seeks to retrieve a replicable component from the non-replicable forecasts, and compares this component against the actual data. A second part modifies the estimation routine due to the assumption that the difference between a replicable and a non-replicable forecast involves measurement error. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the methodological approach using both individuals and mean forecasts.

A new approach for evaluating economic forecasts

2012

Abstract This paper presents a new approach to evaluating multiple economic forecasts. In the past, evaluations have focused on the forecasts of individual variables. However, many macroeconomic variables are forecast at the same time and are used together to describe the state of the economy. It is, therefore, appropriate to examine those forecasts jointly. This specific approach is based on the Sinclair and Stekler (forthcoming) analysis of data revisions.

Combining expert forecasts: Can anything beat the simple average?

International Journal of Forecasting, 2013

ABSTRACT This paper explores the gains from combining expert forecasts from the ECB Survey of Professional Forecasters (SPF). The analysis encompasses combinations based on principal components and trimmed means, performance-based weighting, and least squares estimates of optimal weights, as well as Bayesian shrinkage. For GDP growth and the unemployment rate, only few of the individual forecast combination schemes outperform the simple equally weighted average forecast in a pseudo-out-of-sample analysis, while there is stronger evidence of improvement over this benchmark for the inflation rate. Nonetheless, when we account for the effect of multiple model comparisons through White’s reality check, the results caution against any assumption that the improvements identified would persist in the future.

How Accurate are Government Forecast of Economic Fundamentals?

A government's ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or, expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.

Using forecasts of forecasters to forecast

International Journal of Forecasting, 2007

Quantification techniques are popular methods in empirical research to aggregate the qualitative predictions at the micro-level into a single figure. In this paper, we analyze the forecasting performance of various methods that are based on the qualitative predictions of financial experts for major financial variables and macroeconomic aggregates. Based on the Centre of European Economic Research's Financial Markets Survey, a monthly qualitative survey of around 330 financial experts, we analyze the out-of-sample predictive quality of probability methods and regression methods. Using the modified Diebold-Mariano-Test of Harvey, Leybourne & Newbold (1997), we confront the forecasts based on survey methods with the forecasting performance of standard linear time series approaches and simple random walk forecasts. JEL classification: G10, E30, E31, E37, C10, C42

How accurate are government forecasts of economic fundamentals? The case of Taiwan

International Journal of Forecasting, 2011

A government's ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or, expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate. In this paper, we develop a methodology to evaluate non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that it is the undocumented knowledge of the Taiwanese government that reduces forecast errors substantially.

Evaluating Macroeconomic Forecasts: A Concise Review of Some Recent Developments

Journal of Economic Surveys, 2014

Macroeconomic forecasts are frequently produced, widely published, intensively discussed and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC) and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes non-standard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations.

On the optimality of expert-adjusted forecasts

2007

Official forecasts of international institutions are never purely model-based. Preliminary results of models are adjusted with expert opinions.What is the impact of these adjustments for the forecasts? Are they necessary to get 'optimal' forecasts? When model-based forecasts are adjusted by experts, the loss function of these forecasts is not a mean squared error loss function. In fact, the overall loss function is unknown.To examine the quality of these forecasts, one can rely on the tests for forecast optimality under unknown loss function as developed in Patton and Timmermann (2007). We apply one of these tests to ten variables for which we have model-based forecasts and expert-adjusted forecasts, all generated by the Netherlands Bureau for Economic Policy Analysis (CPB). For almost all variables the added expertise yields better forecasts in terms of fit. In terms of optimality, the effect of adjustments for the forecasts is limited, because for most variables the assu...