How Far Can We Forecast? Statistical Tests of the Predictive Content (original) (raw)

How far can we forecast?: Forecast content horizons for some important macroeconomic time series

2006

For stationary transformations of variables, there exists a maximum horizon beyond which forecasts can provide no more information about the variable than is present in the unconditional mean. Meteorological forecasts, typically excepting only experimental or exploratory situations, are not reported beyond this horizon; by contrast, little generally-accepted information about such maximum horizons is available for economic variables. In this paper we estimate such content horizons for a variety of economic variables, and compare these with the maximum horizons which we observe reported in a large sample of empirical economic forecasting studies. We find that there are many instances of published studies which provide forecasts exceeding, often by substantial margins, our estimates of the content horizon for the particular variable and frequency. We suggest some simple reporting practices for forecasts that could potentially bring greater transparency to the process of making and interpreting economic forecasts.

ESSAYS IN FORECASTING ABSTRACT OF THE DISSERTATION Essays in Forecasting

2009

This dissertation comprises three essays in macroeconomic forecasting. The first essay discusses model selection and predictive accuracy tests in the context of parameter and model uncertainty under recursive and rolling estimation schemes. Particular emphasis is placed on the construction of valid bootstrap procedures for calculating the impact of parameter estimation error on the class of test statistics with limiting distributions that are functionals of Gaussian processes. Results of an empirical investigation of the marginal predictive content of money for income are also presented. The second essay outlines a number of approaches to the selection of factor proxies (ob-served variables that proxy unobserved estimated factors) using statistics based on large sample datasets. This approach to factor proxy selection is examined via a small Monte Carlo experiment and a set of prediction experiments, where evidence supporting our proposed methodology is presented. The third essay co...

“A Bias Recognized Is a Bias Sterilized”: The Effects of a Bias in Forecast Evaluation

Mathematics

Are traditional tests of forecast evaluation well behaved when the competing (nested) model is biased? No, they are not. In this paper, we show analytically and via simulations that, under the null hypothesis of no encompassing, a bias in the nested model may severely distort the size properties of traditional out-of-sample tests in economic forecasting. Not surprisingly, these size distortions depend on the magnitude of the bias and the persistency of the additional predictors. We consider two different cases: (i) There is both in-sample and out-of-sample bias in the nested model. (ii) The bias is present exclusively out-of-sample. To address the former case, we propose a modified encompassing test (MENC-NEW) robust to a bias in the null model. Akin to the ENC-NEW statistic, the asymptotic distribution of our test is a functional of stochastic integrals of quadratic Brownian motions. While this distribution is not pivotal, we can easily estimate the nuisance parameters. To address ...

Essays on Forecast Evaluation in Macroeconomics and International Finance

The George Washington University. ProQuest Dissertations Publishing, 2018

This dissertation shares three common themes: (i) forecasting rare macroeconomic events, i.e. GDP declines and currency crises; (ii) the use of non-parametric methods to evaluate binary indicators, in particular, the advantages of the analysis of the Receiver Operating Characteristic (ROC) curves; and (iii) value of qualitative information from expert surveys and textual analysis in macroeconomic forecasting. Chapter 1 contributes to the literature on evaluation of the qualitative survey directional forecasts using the World Economic Survey (WES) for the U.S. economy in 1989q1-2015q4. I offer a methodology which combines the ROC curves analysis with the traditional analysis of the contingency tables. I propose criteria to assess in-sample and out-of-sample directional predictive value of the binary indicators, including directional forecasts from the qualitative surveys. I find that the WES has high out-of-sample value in forecasting movements in GDP and consumption, and moderate for imports, trade balance, inflation, and short-term interest rate. It has no value in predicting changes in investment and exports. I also motivate and confirm that the WES Economic Climate (EC) indicator is as a more accurate predictor of future movements in the real GDP than future expectations alone. Additionally, I show that the ROC-optimal thresholds yield more accurate predictions than their alternatives proposed by Hutson et al. (2014). Chapter 2 re-examines indicators of currency crises suggested by Kaminsky and Reinhart (1999) and subsequent studies using the ROC curves analysis. I utilize a training set (1975-1995) to confirm a list of indicators with the in-sample predictive value, and test their out-of-sample using data for 1996-2002. Four variables have both in-sample and out-of-sample predictive value: the deviation of the real exchange rate (RER) from a trend, the foreign reserves, the ratio of broad money M2 to reserves, and the decline in exports. I show that the ROC-optimal thresholds issue more accurate signals than the minimum noise-to-signal ratio previously used in the literature. I also employ modified ROC curves to display the relationship between the precision of sent signals and recall of crisis episodes. Finally, I propose forecast combinations using several ad-hoc rules which help to improve forecast accuracy. Chapter 3 contributes to the discussion of asymmetric information about the U.S. economy between the Federal Reserve System (FRS) and the Survey of Professional Forecasters (SPF) via textual analysis of the Federal Open Market Committee (FOMC) minutes. It builds on Stekler and Symington (2016), who scored the texts of the FOMC minutes in 2006-2010 to produce the indexes for the current and future outlooks and their calibrations to the U.S. real GDP. I extend their timeseries adding 26 years of observations to cover 1986Q1-2016Q4. Following Ericsson (2016), I interpret the derived calibrations (FMIs) as elicitcasts of the Greenbook (GB) forecasts. Results indicate that the FMIs are unbiased, efficient, rational, and contain the same informational advantage as the GB forecasts. The forecast encompassing tests suggest that both the FMIs and the SPF forecasts contain their own unique knowledge and can learn from each other. I find that the SPF forecasters already pay close attention to the FOMC minutes available to them at the time of forecast deadline and efficiently use its information in their set. Yet, they could improve their forecasts should the FOMC minutes from the first quarterly meetings become available without a three-week publication lag. During their second quarterly meetings, the FOMC policy-makers accounted only for their own earlier assessment of the U.S. macroeconomy – they did not put weight on the SPF forecasts released a few weeks earlier in the same quarter. The results are robust to the use of alternative scale. Overall, I find that directional forecasts are informative. The qualitative WES survey can produce accurate directional macroeconomic forecasts. The ROC curves analysis helps to set an association between the consensus scores and the growth rates as well as to find accurate indicators of currency crisis. The qualitative statements from monetary policy deliberations can be converted in to the GDP growth forecasts with unique information about the US economy.

Evaluating Macroeconomic Forecasts: A Concise Review of Some Recent Developments

Journal of Economic Surveys, 2014

Macroeconomic forecasts are frequently produced, widely published, intensively discussed and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC) and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes non-standard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations.

How far ahead can we forecast? Evidence from cross-country surveys

International Journal of Forecasting, 2007

Using monthly GDP forecasts from Consensus Economics, Inc. for 18 developed countries, reported over 24 different forecast horizons during the period 1989-2004, we find that the survey forecasts do not have much value when the horizon goes beyond 18 months. Using two alternative approaches to measure the flow of new information in fixed-target survey forecasts, we find that the biggest improvement in forecasting performance comes when the forecast horizon is around 14 months. The dynamics of information accumulation over forecast horizons can provide both the forecasters and their clients with an important clue in their selection of the timing and frequency in the use of forecasting services. The limits to forecasting that these private market forecasters exhibit are indicative of the current state of macroeconomic foresight.

Freedom of Choice in Macroeconomic Forecasting

CESifo Economic Studies, 2010

Different studies provide a surprisingly large variety of controversial conclusions about the forecasting power of an indicator, even when it is supposed to forecast the same time series. In this study we aim to provide a thorough overview of linear forecasting techniques and draw conclusions useful for the identification of the predictive relationship between leading indicators and time series. In a case study for Germany we forecast two possible representations of industrial production. Further on we consider a large variety of time-varying specifications. In a horse race with nine leading indicators plus an AR benchmark model we demonstrate the variance of assessment across target variables and forecasting settings (50 per horizon). We show that it is nearly always possible to find situations in which one indicator proved to have better predicting power compared to another. Nevertheless, the freedom of choice can be useful to identify robust leading indicators. JEL: C52, C53, E37

Macroeconomic forecasting: Debunking a few old wives' tales

Journal of Business Cycle Measurement …, 2007

The forecasting profession, especially when producing forecasts intended to support economic policy, does not currently enjoy a good reputation. Complaints are sometimes voiced about its lack of scientific discipline, which in turn implies that the forecast results may be viewed ...

Using forecasts of forecasters to forecast

International Journal of Forecasting, 2007

Quantification techniques are popular methods in empirical research to aggregate the qualitative predictions at the micro-level into a single figure. In this paper, we analyze the forecasting performance of various methods that are based on the qualitative predictions of financial experts for major financial variables and macroeconomic aggregates. Based on the Centre of European Economic Research's Financial Markets Survey, a monthly qualitative survey of around 330 financial experts, we analyze the out-of-sample predictive quality of probability methods and regression methods. Using the modified Diebold-Mariano-Test of Harvey, Leybourne & Newbold (1997), we confront the forecasts based on survey methods with the forecasting performance of standard linear time series approaches and simple random walk forecasts. JEL classification: G10, E30, E31, E37, C10, C42

The evolution of consensus in macroeconomic forecasting

International Journal of Forecasting, 2004

When professional forecasters repeatedly forecast macroeconomic variables, their forecasts may converge over time towards a consensus. The evolution of consensus is analyzed with Blue Chip data under a parametric polynomial decay function that permits flexibility in the decay path. For the most part, this specification fits the data. We test whether forecast differences decay to zero at the same point in time for a panel of forecasters, and discuss possible explanations for this, along with its implications for studies using panels of forecasters.