Confidence-Nets: A Step Towards better Prediction Intervals for regression Neural Networks on small datasets (original) (raw)
Related papers
Confident Neural Network Regression with Bootstrapped Deep Ensembles
Cornell University - arXiv, 2022
With the rise of the popularity and usage of neural networks, trustworthy uncertainty estimation is becoming increasingly essential. In this paper we present a computationally cheap extension of Deep Ensembles for a regression setting called Bootstrapped Deep Ensembles that explicitly takes the effect of finite data into account using a modified version of the parametric bootstrap. We demonstrate through a simulation study that our method has comparable or better prediction intervals and superior confidence intervals compared to Deep Ensembles and other state-ofthe-art methods. As an added bonus, our method is better capable of detecting overfitting than standard Deep Ensembles.
Confidence and prediction intervals for neural network ensembles
Neural Networks, 1999. …, 1999
In this paper we propose a new technique that uses the bootstrap to estimate con dence and prediction intervals for neural network (regression) ensembles. Our proposed technique can be applied to any ensemble technique that uses the bootstrap to generate the training sets for the ensemble, such as bagging 1] and balancing 5]. Con dence and prediction intervals are estimated that include a signi cantly improved estimate of underlying model uncertainty (i.e.) the uncertainty of our estimate of the \true" regression. Unlike existing techniques, this estimate of uncertainty will vary according to which ensemble technique is used { if the e ect of using a speci c ensemble technique is to produce less model uncertainty than using another ensemble technique, then this will be re ected in the con dence and prediction intervals. Preliminary results illustrate how our technique can provide more accurate con dence and prediction intervals (intervals that better re ect the desired level of con dence (e.g.) 90%, 95%, etc.) for neural network ensembles than previous attempts.
Tight Prediction Intervals Using Expanded Interval Minimization
ArXiv, 2018
Prediction intervals are a valuable way of quantifying uncertainty in regression problems. Good prediction intervals should be both correct, containing the actual value between the lower and upper bound at least a target percentage of the time; and tight, having a small mean width of the bounds. Many prior techniques for generating prediction intervals make assumptions on the distribution of error, which causes them to work poorly for problems with asymmetric distributions. This paper presents Expanded Interval Minimization (EIM), a novel loss function for generating prediction intervals using neural networks. This loss function uses minibatch statistics to estimate the coverage and optimize the width of the prediction intervals. It does not make the same assumptions on the distributions of data and error as prior work. We compare to three published techniques and show EIM produces on average 1.37x tighter prediction intervals and in the worst case 1.06x tighter intervals across two...
This paper highlights the contribution of utilizing ensemble deep learning with auto-encoders (AEs) for out-of-distribution data detection. The key innovation is treating ensemble UQ as a regression problem, mapping uncertainty distribution to a single model, reducing computational demands. This approach aligns well with the ensemble of AEs' uncertainty distribution, making it valuable for resource-constrained systems and rapid decision-making in computational intelligence.
Credal Deep Ensembles for Uncertainty Quantification
2024 Conference on Neural Information Processing Systems (NeurIPS 2024), 2024
This paper introduces an innovative approach to classification called Credal Deep Ensembles (CreDEs), namely, ensembles of novel Credal-Set Neural Networks (CreNets). CreNets are trained to predict a lower and an upper probability bound for each class, which, in turn, determine a convex set of probabilities (credal set) on the class set. The training employs a loss inspired by distributionally robust optimization which simulates the potential divergence of the test distribution from the training distribution, in such a way that the width of the predicted probability interval reflects the 'epistemic' uncertainty about the future data distribution. Ensembles can be constructed by training multiple CreNets, each associated with a different random seed, and averaging the outputted intervals. Extensive experiments are conducted on various out-of-distributions (OOD) detection benchmarks (CIFAR10/100 vs SVHN/Tiny-ImageNet, CIFAR10 vs CIFAR10-C, ImageNet vs ImageNet-O) and using different network architectures (ResNet50, VGG16, and ViT Base). Compared to Deep Ensemble baselines, CreDEs demonstrate higher test accuracy, lower expected calibration error, and significantly improved epistemic uncertainty estimation.
Prediction intervals for neural networks via nonlinear regression
1998
Standard methods for computing prediction intervals in nonlinear regression can be effectively applied to neural networks when the number of training points is large. Simulations show, however, that these methods can generate unreliable prediction intervals on smaller datasets when the network is trained to convergence. Stopping the training algorithm prior to convergence, to avoid overfitting, reduces the effective number of parameters but can lead to prediction intervals that are too wide.
Towards Consistent Predictive Confidence through Fitted Ensembles
2021 International Joint Conference on Neural Networks (IJCNN), 2021
Deep neural networks are behind many of the recent successes in machine learning applications. However, these models can produce overconfident decisions while encountering outof-distribution (OOD) examples or making a wrong prediction. This inconsistent predictive confidence limits the integration of independently-trained learning models into a larger system. This paper introduces separable concept learning framework to realistically measure the performance of classifiers in presence of OOD examples. In this setup, several instances of a classifier are trained on different parts of a partition of the set of classes. Later, the performance of the combination of these models is evaluated on a separate test set. Unlike current OOD detection techniques, this framework does not require auxiliary OOD datasets and does not separate classification from detection performance. Furthermore, we present a new strong baseline for more consistent predictive confidence in deep models, called fitted ensembles, where overconfident predictions are rectified by transformed versions of the original classification task. Fitted ensembles can naturally detect OOD examples without requiring auxiliary data by observing contradicting predictions among its components. Experiments on MNIST, SVHN, CIFAR-10/100, and ImageNet show fitted ensemble significantly outperform conventional ensembles on OOD examples and are possible to scale.
IEEE access, 2024
Ensembling is a powerful technique to obtain the most accurate results. In some cases, the large number of learners in ensemble learning mostly increases both computational load during the test phase and error rate. To solve this problem, in this paper we propose an Ensemble of Reduced Deep Regression (ERDeR) model, which is a combination of Deep Regressions (DRs), shrinkage methods, and ensemble approaches. The framework of the proposed model contains three phases. The first phase includes base regressions in which parallel DRs are used as learners. The role of these DRs is to extract features of input data and make prediction. In the second phase, to automatically reduce and select the most suitable DRs, shrinkage methods such as Least Absolute Shrinkage and Selection Operator (LASSO) and Elastic Net (EN) are employed. These models are compared with the non-shrinkage model. The last phase is ensemble phase, which consists of three different ensemble methods namely Multi-Layer Perceptron (MLP), Weighted Average (WA), and Simple Average (SA). These ensemble methods are used to aggregate the remaining learners from previous steps. Finally, the proposed model is applied to Monte Carlo simulation data and three real datasets including Boston House Price, Real Estate Valuation and Gold Price per Ounce. The results show that after applying the shrinkage methods the error rate is significantly reduced and the model accuracy is increased. Accordingly, the results of combining shrinkage methods and ensemble approaches not only decreased the computational load during test phase, but also increased the model accuracy.
Prediction intervals for neural network models
Proceedings of the 9th WSEAS International …, 2005
Neural networks are a consistent example of non-parametric estimation, with powerful universal approximation properties. However, the effective development and deployment of neural network applications, has to be based on established procedures for estimating confidence and ...
Construction of Confidence Intervals for Neural Networks
We present the theoretical results about the construction of confidence intervals for a nonlinear regression based on least squares estimation and using the linear Taylor expansion of the nonlinear model output. We stress the assumptions on which these results are based, in order to derive an appropriate methodology for neural black-box modeling; the latter is then analyzed and illustrated on simulated and real processes. We show that the linear Taylor expansion of a nonlinear model output also gives a tool to detect the possible ill-conditioning of neural network candidates, and to estimate their performance. Finally, we show that the least squares and linear Taylor expansion based approach compares favourably with other analytic approaches, and that it is an efficient and economic alternative to the non analytic and computationally intensive bootstrap methods.