Shachi Deshpande | IIT Bombay (original) (raw)

Uploads

Papers by Shachi Deshpande

Research paper thumbnail of Calibrated Uncertainty Estimation Improves Bayesian Optimization

arXiv (Cornell University), Dec 8, 2021

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such... more Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration-i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

Research paper thumbnail of Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

arXiv (Cornell University), Mar 17, 2022

Estimating the effect of an intervention from observational data while accounting for confounding... more Estimating the effect of an intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Research paper thumbnail of Adversarial Calibrated Regression for Online Decision Making

arXiv (Cornell University), Feb 23, 2023

Accurately estimating uncertainty is an essential component of decision-making and forecasting in... more Accurately estimating uncertainty is an essential component of decision-making and forecasting in machine learning. However, existing uncertainty estimation methods may fail when data no longer follows the distribution seen during training. Here, we introduce online uncertainty estimation algorithms that are guaranteed to be reliable on arbitrary streams of datapoints, including data chosen by an adversary. Specifically, our algorithms perform post-hoc recalibration of a blackbox regression model and produce outputs that are provably calibrated-i.e., an 80% confidence interval will contain the true outcome 80% of the time-and that have low regret relative to the learning objective of the base model. We apply our algorithms in the context of Bayesian optimization, an online model-based decision-making task in which the data distribution shifts over time, and observe accelerated convergence to improved optima. Our results suggest that robust uncertainty quantification has the potential to improve online decision-making. Preprint. Under review.

Research paper thumbnail of Calibrated Propensity Scores for Causal Effect Estimation

arXiv (Cornell University), Jun 1, 2023

Propensity scores are commonly used to balance observed covariates while estimating treatment eff... more Propensity scores are commonly used to balance observed covariates while estimating treatment effects. Estimates obtained through propensity score weighing can be biased when the propensity score model cannot learn the true treatment assignment mechanism. We argue that the probabilistic output of a learned propensity score model should be calibrated, i.e. a predictive treatment probability of 90% should correspond to 90% individuals being assigned the treatment group. We propose simple recalibration techniques to ensure this property. We investigate the theoretical properties of a calibrated propensity score model and its role in unbiased treatment effect estimation. We demonstrate improved causal effect estimation with calibrated propensity scores in several tasks including high-dimensional genomewide association studies, where we also show reduced computational requirements when calibration is applied to simpler propensity score models. Preprint. Under review.

Research paper thumbnail of Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation

arXiv (Cornell University), Dec 14, 2021

Accurate probabilistic predictions can be characterized by two properties-calibration and sharpne... more Accurate probabilistic predictions can be characterized by two properties-calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate-a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning.

Research paper thumbnail of Online Calibrated Regression for Adversarially Robust Forecasting

arXiv (Cornell University), Feb 23, 2023

Accurately estimating uncertainty is an essential component of decision-making and forecasting in... more Accurately estimating uncertainty is an essential component of decision-making and forecasting in machine learning. However, existing uncertainty estimation methods may fail when data no longer follows the distribution seen during training. Here, we introduce online uncertainty estimation algorithms that are guaranteed to be reliable on arbitrary streams of datapoints, including data chosen by an adversary. Specifically, our algorithms perform post-hoc recalibration of a blackbox regression model and produce outputs that are provably calibrated-i.e., an 80% confidence interval will contain the true outcome 80% of the time-and that have low regret relative to the learning objective of the base model. We apply our algorithms in the context of Bayesian optimization, an online model-based decision-making task in which the data distribution shifts over time, and observe accelerated convergence to improved optima. Our results suggest that robust uncertainty quantification has the potential to improve online decision-making. Preprint. Under review.

Research paper thumbnail of Multi-Modal Causal Inference with Deep Structural Equation Models

ArXiv, 2022

Accounting for the effects of confounders is one of the central challenges in causal inference. U... more Accounting for the effects of confounders is one of the central challenges in causal inference. Unstructured multi-modal data (images, time series, text) contains valuable information about diverse types of confounders, yet it is typically left unused by most existing methods. This paper seeks to develop techniques that leverage this unstructured data within causal inference to correct for additional confounders that may otherwise not be accounted for. We formalize this task and we propose algorithms based on deep structural equations that treat multi-modal unstructured data as proxy variables. We empirically demonstrate on tasks in genomics and healthcare that unstructured data can be used to correct for diverse sources of confounding, potentially enabling the use of large amounts of data that were previously not used in causal inference.

Research paper thumbnail of Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

Cornell University - arXiv, Mar 17, 2022

Estimating the effect of an intervention from observational data while accounting for confounding... more Estimating the effect of an intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Research paper thumbnail of Calibration Improves Bayesian Optimization

ArXiv, 2021

Bayesian optimization is a procedure that allows obtaining the global optimum of black-box functi... more Bayesian optimization is a procedure that allows obtaining the global optimum of black-box functions and that is useful in applications such as hyper-parameter optimization. Uncertainty estimates over the shape of the objective function are instrumental in guiding the optimization process. However, these estimates can be inaccurate if the objective function violates assumptions made within the underlying model (e.g., Gaussianity). We propose a simple algorithm to calibrate the uncertainty of posterior distributions over the objective function as part of the Bayesian optimization process. We show that by improving the uncertainty estimates of the posterior distribution with calibration, Bayesian optimization makes better decisions and arrives at the global optimum in fewer steps. We show that this technique improves the performance of Bayesian optimization on standard benchmark functions and hyperparameter optimization tasks.

Research paper thumbnail of New Genome Similarity Measures Based on Conserved Gene Adjacencies

Lecture Notes in Computer Science, 2016

Research paper thumbnail of Calibrated and Sharp Uncertainties in Deep Learning via Simple Density Estimation

ArXiv, 2021

Predictive uncertainties can be characterized by two properties—calibration and sharpness. This p... more Predictive uncertainties can be characterized by two properties—calibration and sharpness. This paper argues for reasoning about uncertainty in terms these properties and proposes simple algorithms for enforcing them in deep learning. Our methods focus on the strongest notion of calibration—distribution calibration— and enforce it by fitting a low-dimensional density or quantile function with a neural estimator. The resulting approach is much simpler and more broadly applicable than previous methods across both classification and regression. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead. Our insights suggest simple and improved ways of training deep learning models that lead to accurate uncertainties that should be leveraged to improve performance across downstream applications.

Research paper thumbnail of Calibrated Uncertainty Estimation Improves Bayesian Optimization

arXiv (Cornell University), Dec 8, 2021

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such... more Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration-i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

Research paper thumbnail of Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

arXiv (Cornell University), Mar 17, 2022

Estimating the effect of an intervention from observational data while accounting for confounding... more Estimating the effect of an intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Research paper thumbnail of Adversarial Calibrated Regression for Online Decision Making

arXiv (Cornell University), Feb 23, 2023

Accurately estimating uncertainty is an essential component of decision-making and forecasting in... more Accurately estimating uncertainty is an essential component of decision-making and forecasting in machine learning. However, existing uncertainty estimation methods may fail when data no longer follows the distribution seen during training. Here, we introduce online uncertainty estimation algorithms that are guaranteed to be reliable on arbitrary streams of datapoints, including data chosen by an adversary. Specifically, our algorithms perform post-hoc recalibration of a blackbox regression model and produce outputs that are provably calibrated-i.e., an 80% confidence interval will contain the true outcome 80% of the time-and that have low regret relative to the learning objective of the base model. We apply our algorithms in the context of Bayesian optimization, an online model-based decision-making task in which the data distribution shifts over time, and observe accelerated convergence to improved optima. Our results suggest that robust uncertainty quantification has the potential to improve online decision-making. Preprint. Under review.

Research paper thumbnail of Calibrated Propensity Scores for Causal Effect Estimation

arXiv (Cornell University), Jun 1, 2023

Propensity scores are commonly used to balance observed covariates while estimating treatment eff... more Propensity scores are commonly used to balance observed covariates while estimating treatment effects. Estimates obtained through propensity score weighing can be biased when the propensity score model cannot learn the true treatment assignment mechanism. We argue that the probabilistic output of a learned propensity score model should be calibrated, i.e. a predictive treatment probability of 90% should correspond to 90% individuals being assigned the treatment group. We propose simple recalibration techniques to ensure this property. We investigate the theoretical properties of a calibrated propensity score model and its role in unbiased treatment effect estimation. We demonstrate improved causal effect estimation with calibrated propensity scores in several tasks including high-dimensional genomewide association studies, where we also show reduced computational requirements when calibration is applied to simpler propensity score models. Preprint. Under review.

Research paper thumbnail of Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation

arXiv (Cornell University), Dec 14, 2021

Accurate probabilistic predictions can be characterized by two properties-calibration and sharpne... more Accurate probabilistic predictions can be characterized by two properties-calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate-a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning.

Research paper thumbnail of Online Calibrated Regression for Adversarially Robust Forecasting

arXiv (Cornell University), Feb 23, 2023

Accurately estimating uncertainty is an essential component of decision-making and forecasting in... more Accurately estimating uncertainty is an essential component of decision-making and forecasting in machine learning. However, existing uncertainty estimation methods may fail when data no longer follows the distribution seen during training. Here, we introduce online uncertainty estimation algorithms that are guaranteed to be reliable on arbitrary streams of datapoints, including data chosen by an adversary. Specifically, our algorithms perform post-hoc recalibration of a blackbox regression model and produce outputs that are provably calibrated-i.e., an 80% confidence interval will contain the true outcome 80% of the time-and that have low regret relative to the learning objective of the base model. We apply our algorithms in the context of Bayesian optimization, an online model-based decision-making task in which the data distribution shifts over time, and observe accelerated convergence to improved optima. Our results suggest that robust uncertainty quantification has the potential to improve online decision-making. Preprint. Under review.

Research paper thumbnail of Multi-Modal Causal Inference with Deep Structural Equation Models

ArXiv, 2022

Accounting for the effects of confounders is one of the central challenges in causal inference. U... more Accounting for the effects of confounders is one of the central challenges in causal inference. Unstructured multi-modal data (images, time series, text) contains valuable information about diverse types of confounders, yet it is typically left unused by most existing methods. This paper seeks to develop techniques that leverage this unstructured data within causal inference to correct for additional confounders that may otherwise not be accounted for. We formalize this task and we propose algorithms based on deep structural equations that treat multi-modal unstructured data as proxy variables. We empirically demonstrate on tasks in genomics and healthcare that unstructured data can be used to correct for diverse sources of confounding, potentially enabling the use of large amounts of data that were previously not used in causal inference.

Research paper thumbnail of Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

Cornell University - arXiv, Mar 17, 2022

Estimating the effect of an intervention from observational data while accounting for confounding... more Estimating the effect of an intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Research paper thumbnail of Calibration Improves Bayesian Optimization

ArXiv, 2021

Bayesian optimization is a procedure that allows obtaining the global optimum of black-box functi... more Bayesian optimization is a procedure that allows obtaining the global optimum of black-box functions and that is useful in applications such as hyper-parameter optimization. Uncertainty estimates over the shape of the objective function are instrumental in guiding the optimization process. However, these estimates can be inaccurate if the objective function violates assumptions made within the underlying model (e.g., Gaussianity). We propose a simple algorithm to calibrate the uncertainty of posterior distributions over the objective function as part of the Bayesian optimization process. We show that by improving the uncertainty estimates of the posterior distribution with calibration, Bayesian optimization makes better decisions and arrives at the global optimum in fewer steps. We show that this technique improves the performance of Bayesian optimization on standard benchmark functions and hyperparameter optimization tasks.

Research paper thumbnail of New Genome Similarity Measures Based on Conserved Gene Adjacencies

Lecture Notes in Computer Science, 2016

Research paper thumbnail of Calibrated and Sharp Uncertainties in Deep Learning via Simple Density Estimation

ArXiv, 2021

Predictive uncertainties can be characterized by two properties—calibration and sharpness. This p... more Predictive uncertainties can be characterized by two properties—calibration and sharpness. This paper argues for reasoning about uncertainty in terms these properties and proposes simple algorithms for enforcing them in deep learning. Our methods focus on the strongest notion of calibration—distribution calibration— and enforce it by fitting a low-dimensional density or quantile function with a neural estimator. The resulting approach is much simpler and more broadly applicable than previous methods across both classification and regression. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead. Our insights suggest simple and improved ways of training deep learning models that lead to accurate uncertainties that should be leveraged to improve performance across downstream applications.