Technical Note – RAT: a Robustness Assessment Test for calibrated and uncalibrated hydrological models (original) (raw)
Related papers
Technical note: PMR – a proxy metric to assess hydrological model robustness in a changing climate
2021
The ability of hydrological models to perform in climatic conditions different from those encountered in calibration is crucial to ensure a reliable assessment of the impact of climate change on river regimes and water availability. However, most evaluation studies based on the differential split-sample test (DSST) endorsed the consensus that rainfall-runoff models lack climatic robustness. Models applied under climatologically different conditions typically exhibit substantial errors in streamflow volumes. In this technical note, we propose a new performance metric to evaluate model robustness without applying the DSST, and it can be performed with a single hydrological model calibration. The proxy for model robustness (PMR) is based on the systematic computation of model error on sliding sub-periods of the whole streamflow time series. We demonstrate that the PMR metric shows patterns similar to those obtained with the DSST for a conceptual model on a set of 377 French catchments. An analysis of the sensitivity to the length of the sub-periods shows that this length influences the values of the PMR and its equivalency with DSST biases. We recommend a range of a few years for the choice of sub-period lengths, although this should be context dependent. Our work makes it possible to evaluate the temporal transferability of any hydrological model, including uncalibrated models, at a very low computational cost.
Hydrological Sciences Journal, 2018
Two approaches can be distinguished in studies of climate change impacts on water resources when accounting for issues related to impact model performance: (1) using a multi-model ensemble disregarding model performance, and (2) using models after their evaluation and considering model performance. We discuss the implications of both approaches in terms of credibility of simulated hydrological indicators for climate change adaptation. For that, we discuss and confirm the hypothesis that a good performance of hydrological models in the historical period increases confidence in projected impacts under climate change, and decreases uncertainty of projections related to hydrological models. Based on this, we find the second approach more trustworthy and recommend using it for impact assessment, especially if results are intended to support adaptation strategies. Guidelines for evaluation of global-and basin-scale models in the historical period, as well as criteria for model rejection from an ensemble as an outlier, are also suggested.
2018
Many studies about climate change impacts assessment are published every year. These studies commonly use a hydroclimatic modelling chain, whose principle is to feed impact models with climate models outputs. An important step in this process is to test the validity of impact models in a climate change context. However, this step is not frequently applied. The aim of this study is to test the robustness of two hydrological models with distinct conceptualizations: a global and empirical model (GR4J) and a semi-distributed and physically-based model (SWAT). They both have been calibrated and validated over climate contrasted periods. Despite a higher decrease of performance between calibration and validation for the GR4J model, both of them show relative robustness. Moreover, the stability of parameters between the two calibration periods shows that their value are not much influenced by the climate of the calibration period, and consequently remains valid during the entire projection...
Operational testing of a water balance model for predicting climate change impacts
Agricultural and Forest Meteorology, 1999
The ability of water balance models to incorporate month±month or seasonal variations in climate, snowfall and snowmelt, groundwater¯uctuations, soil moisture characteristics, and natural climatic variability makes them especially attractive for water resources studies of climatic changes. The use of conceptual models to explore the impact of climate changes has increased in recent years. Because of the success claimed for these studies, it is likely that computer simulation of catchments will increasingly be used by and for water resource managers as an aid to decision-making. There is therefore a need for a generally accepted method for demonstrating a model's ®tness for such use. The simple split-sample test method may be reasonable in the simplest case of the`®lling-in missing data' problem but certainly not if the express purpose of the model is to simulate records for conditions different from those corresponding to the calibration record, such as the problem with predicting the effects of climate changes where the data on the changed system are not (and cannot be) available for comparison with the model predictions. Thus, model validation must demonstrate`®tness for the said purpose'.
Assessing the uncertainties of hydrologic model selection in climate change impact studies
The uncertainties associated with atmosphere-ocean General Circulation Models (GCMs) and hydrologic models are assessed by means of multi-modelling and using the statistically downscaled outputs from eight GCM simulations and two emission scenarios. The statistically downscaled atmospheric forcing is used to drive four hydrologic models, three lumped and one distributed, of differing complexity: the Sacramento Soil Moisture Accounting (SAC-SMA) model, Conceptual HYdrologic MODel (HYMOD), Thornthwaite-Mather model (TM) and the Precipitation Runoff Modelling System (PRMS). The models are calibrated based on three objective functions to create more plausible models for the study. The hydrologic model simulations are then combined using the Bayesian Model Averaging (BMA) method according to the performance of each models in the observed period, and the total variance of the models. The study is conducted over the rainfall-dominated Tualatin River Basin (TRB) in Oregon, USA. This study shows that the hydrologic model uncertainty is considerably smaller than GCM uncertainty, except during the dry season, suggesting that the hydrologic model selection-combination is critical when assessing the hydrologic climate change impact. The implementation of the BMA in analysing the ensemble results is found to be useful in integrating the projected runoff estimations from different models, while enabling to assess the model structural uncertainty. Copyright © 2011 John Wiley & Sons, Ltd.
Water Resources Research, 2012
This paper investigates the actual extrapolation capacity of three hydrological models in differing climate conditions. We propose a general testing framework, in which we perform series of split-sample tests, testing all possible combinations of calibration-validation periods using a 10 year sliding window. This methodology, which we have called the generalized split-sample test (GSST), provides insights into the model's transposability over time under various climatic conditions. The three conceptual rainfall-runoff models yielded similar results over a set of 216 catchments in southeast Australia. First, we assessed the model's efficiency in validation using a criterion combining the root-mean-square error and bias. A relation was found between this efficiency and the changes in mean rainfall (P) but not with changes in mean potential evapotranspiration (PE) or air temperature (T). Second, we focused on average runoff volumes and found that simulation biases are greatly affected by changes in P. Calibration over a wetter (drier) climate than the validation climate leads to an overestimation (underestimation) of the mean simulated runoff. We observed different magnitudes of these models deficiencies depending on the catchment considered. Results indicate that the transfer of model parameters in time may introduce a significant level of errors in simulations, meaning increased uncertainty in the various practical applications of these models (flow simulation, forecasting, design, reservoir management, climate change impact assessments, etc.). Testing model robustness with respect to this issue should help better quantify these uncertainties.
The role of hydrological model complexity and uncertainty in climate change impact assessment
Advances in Geosciences, 2009
Little quantitative knowledge is as yet available about the role of hydrological model complexity for climate change impact assessment. This study investigates and compares the varieties of different model response of three hydrological models (PROMET, Hydrotel, HSAMI), each representing a different model complexity in terms of process description, parameter space and spatial and temporal scale. The study is performed in the Ammer watershed, a 709 km 2 catchment in the Bavarian alpine forelands, Germany. All models are driven and validated by a 30-year time-series of observation data. It is expressed by objective functions, that all models, HSAMI and Hydrotel due to calibration, perform almost equally well for runoff simulation over the validation period. Some systematic deviances in the hydrographs and the spatial patterns of hydrologic variables are however quite distinct and thus further discussed.
Climatic Change
Importance of evaluation of global hydrological models (gHMs) before doing climate impact assessment was underlined in several studies. The main objective of this study is to evaluate the performance of six gHMs in simulating observed discharge for a set of 57 large catchments applying common metrics with thresholds for the monthly and seasonal dynamics and summarize them estimating an aggregated index of model performance for each model in each basin. One model showed a good performance, and other five showed a weak or poor performance in most of the basins. In 15 catchments, evaluation results of all models were poor. The model evaluation was supplemented by climate impact assessment for a subset of 12 representative catchments using (1) usual ensemble mean approach and (2) weighted mean approach based on model performance, and the outcomes were compared. The comparison of impacts in terms of mean monthly and mean annual discharges using two approaches has shown that in four basin...