State-based confidence bounds for data-driven stochastic reachability using Hilbert space embeddings (original) (raw)
Related papers
Data-Driven Stochastic Reachability Using Hilbert Space Embeddings
ArXiv, 2020
We compute finite sample bounds for approximations of the solution to stochastic reachability problems computed using kernel distribution embeddings, a non-parametric machine learning technique. Our approach enables assurances of safety from observed data, through construction of probabilistic violation bounds on the computed stochastic reachability probability. By embedding the stochastic kernel of a Markov control process in a reproducing kernel Hilbert space, we can compute the safety probabilities for systems with arbitrary disturbances as simple matrix operations and inner products. We present finite sample bounds for the approximation using elements from statistical learning theory. We numerically evaluate the approach, and demonstrate its efficacy on neural net-controlled pendulum system.
2021 60th IEEE Conference on Decision and Control (CDC), 2021
We present algorithms for performing data-driven stochastic reachability as an addition to SReachTools, an open-source stochastic reachability toolbox. Our method leverages a class of machine learning techniques known as kernel embeddings of distributions to approximate the safety probabilities for a wide variety of stochastic reachability problems. By representing the probability distributions of the system state as elements in a reproducing kernel Hilbert space, we can learn the "best fit" distribution via a simple regularized least-squares problem, and then compute the stochastic reachability safety probabilities as simple linear operations. This technique admits finite sample bounds and has known convergence in probability. We implement these methods as part of SReachTools, and demonstrate their use on a double integrator system, on a million-dimensional repeated planar quadrotor system, and a cart-pole system with a black-box neural network controller. CCS CONCEPTS • Computing methodologies → Computational control theory; Kernel methods; • Theory of computation → Stochastic control and optimization.
Stochastic Reachability for Systems up to a Million Dimensions
ArXiv, 2019
We present a solution to the first-hitting time stochastic reachability problem for extremely high-dimensional stochastic dynamical systems. Our approach takes advantage of a nonparametric learning technique known as conditional distribution embeddings to model the stochastic kernel using a data-driven approach. By embedding the dynamics and uncertainty within a reproducing kernel Hilbert space, it becomes possible to compute the safety probabilities for stochastic reachability problems as simple matrix operations and inner products. We employ a convergent approximation technique, random Fourier features, in order to accommodate the large sample sets needed for high-dimensional systems. This technique avoids the curse of dimensionality, and enables the computation of safety probabilities for high-dimensional systems without prior knowledge of the structure of the dynamics or uncertainty. We validate this approach on a double integrator system, and demonstrate its capabilities on a m...
SOCKS: A Stochastic Optimal Control and Reachability Toolbox Using Kernel Methods
25th ACM International Conference on Hybrid Systems: Computation and Control
We present SOCKS, a data-driven stochastic optimal control toolbox based in kernel methods. SOCKS is a collection of data-driven algorithms that compute approximate solutions to stochastic optimal control problems with arbitrary cost and constraint functions, including stochastic reachability, which seeks to determine the likelihood that a system will reach a desired target set while respecting a set of pre-defined safety constraints. Our approach relies upon a class of machine learning algorithms based in kernel methods, a nonparametric technique which can be used to represent probability distributions in a high-dimensional space of functions known as a reproducing kernel Hilbert space. As a nonparametric technique, kernel methods are inherently data-driven, meaning that they do not place prior assumptions on the system dynamics or the structure of the uncertainty. This makes the toolbox amenable to a wide variety of systems, including those with nonlinear dynamics, blackbox elements, and poorly characterized stochastic disturbances. We present the main features of SOCKS and demonstrate its capabilities on several benchmarks. CCS CONCEPTS • Computing methodologies → Computational control theory; Kernel methods; • Theory of computation → Stochastic control and optimization.
A Risk-Sensitive Finite-Time Reachability Approach for Safety of Stochastic Dynamic Systems
2019 American Control Conference (ACC), 2019
A classic reachability problem for safety of dynamic systems is to compute the set of initial states from which the state trajectory is guaranteed to stay inside a given constraint set over a given time horizon. In this paper, we leverage existing theory of reachability analysis and risk measures to devise a risk-sensitive reachability approach for safety of stochastic dynamic systems under non-adversarial disturbances over a finite time horizon. Specifically, we first introduce the notion of a risk-sensitive safe set as a set of initial states from which the risk of large constraint violations can be reduced to a required level via a control policy, where risk is quantified using the Conditional Value-at-Risk (CVaR) measure. Second, we show how the computation of a risk-sensitive safe set can be reduced to the solution to a Markov Decision Process (MDP), where cost is assessed according to CVaR. Third, leveraging this reduction, we devise a tractable algorithm to approximate a risk-sensitive safe set and provide arguments about its correctness. Finally, we present a realistic example inspired from stormwater catchment design to demonstrate the utility of risk-sensitive reachability analysis. In particular, our approach allows a practitioner to tune the level of risk sensitivity from worst-case (which is typical for Hamilton-Jacobi reachability analysis) to risk-neutral (which is the case for stochastic reachability analysis).
Robustness to Incorrect System Models in Stochastic Control
SIAM Journal on Control and Optimization, 2020
In stochastic control applications, typically only an ideal model (controlled transition kernel) is assumed and the control design is based on the given model, raising the problem of performance loss due to the mismatch between the assumed model and the actual model. Toward this end, we study continuity properties of discrete-time stochastic control problems with respect to system models (i.e., controlled transition kernels) and robustness of optimal control policies designed for incorrect models applied to the true system. We study both fully observed and partially observed setups under an infinite horizon discounted expected cost criterion. We show that continuity can be established under total variation convergence of the transition kernels under mild assumptions and with further restrictions on the dynamics and observation model under weak and setwise convergence of the transition kernels. Using these continuity properties, we establish convergence results and error bounds due to mismatch that occurs by the application of a control policy which is designed for an incorrectly estimated system model to a true model, thus establishing positive and negative results on robustness. Compared to the existing literature, we obtain strictly refined robustness results that are applicable even when the incorrect models can be investigated under weak convergence and setwise convergence criteria (with respect to a true model), in addition to the total variation criteria. These entail positive implications on empirical learning in (data-driven) stochastic control since often system models are learned through empirical training data where typically weak convergence criterion applies but stronger convergence criteria do not.
Forward Stochastic Reachability Analysis for Uncontrolled Linear Systems using Fourier Transforms
Proceedings of the 20th International Conference on Hybrid Systems: Computation and Control, 2017
We propose a scalable method for forward stochastic reachability analysis for uncontrolled linear systems with affine disturbance. Our method uses Fourier transforms to efficiently compute the forward stochastic reach probability measure (density) and the forward stochastic reach set. This method is applicable to systems with bounded or unbounded disturbance sets. We also examine the convexity properties of the forward stochastic reach set and its probability density. Motivated by the problem of a robot attempting to capture a stochastically moving, non-adversarial target, we demonstrate our method on two simple examples. Where traditional approaches provide approximations, our method provides exact analytical expressions for the densities and probability of capture. CCS Concepts •Theory of computation → Stochastic control and optimization; Convex optimization; •Computing methodologies → Control methods; Computational control theory;
IEEE Control Systems Letters, 2017
We present a scalable underapproximation of the terminal hitting time stochastic reach-avoid probability at a given initial condition, for verification of high-dimensional stochastic LTI systems. While several approximation techniques have been proposed to alleviate the curse of dimensionality associated with dynamic programming, these techniques cannot handle larger, more realistic systems. We present a scalable method that uses Fourier transforms to compute an underapproximation of the reach-avoid probability for systems with disturbances with arbitrary probability densities. We characterize sufficient conditions for Borel-measurability of the value function. We exploit fixed control sequences parameterized by the initial condition (an open-loop control policy) to generate the underapproximation. For Gaussian disturbances, the underapproximation can be obtained using existing efficient algorithms by solving a convex optimization problem. Our approach produces non-trivial lower bounds and is demonstrated on a 40D chain of integrators.
A Stochastic Approximation Method for Reachability Computations
Lecture Notes in Control and Information Science
We develop a grid-based method for estimating the probability that the trajectories of a given stochastic system will eventually enter a certain target set during a-possibly infinite-look-ahead time horizon. The distinguishing feature of the proposed methodology is that it rests on the approximation of the solution to stochastic differential equations by using Markov chains. From an algorithmic point of view, the probability of entering the target set is computed by appropriately propagating the transition probabilities of the Markov chain backwards in time starting from the target set during the time horizon of interest. We consider air traffic management as an application example. Specifically, we address the problem of estimating the probability that two aircraft flying in the same region of the airspace get closer than a certain safety distance and that an aircraft enters a forbidden airspace area. In this context, the target set is the set of unsafe configurations for the system, and we are estimating the probability that an unsafe situation occurs.
Risk-Bounded Control with Kalman Filtering and Stochastic Barrier Functions
2021 60th IEEE Conference on Decision and Control (CDC), 2021
In this paper, we study Stochastic Control Barrier Functions (SCBFs) to enable the design of probabilistic safe real-time controllers in presence of uncertainties and based on noisy measurements. Our goal is to design controllers that bound the probability of a system failure in finite-time to a given desired value. To that end, we first estimate the system states from the noisy measurements using an Extended Kalman filter, and compute confidence intervals on the filtering errors. Then, we account for filtering errors and derive sufficient conditions on the control input based on the estimated states to bound the probability that the real states of the system enter an unsafe region within a finite time interval. We show that these sufficient conditions are linear constraints on the control input, and, hence, they can be used in tractable optimization problems to achieve safety, in addition to other properties like reachability, and stability. Our approach is evaluated using a simulation of a lane-changing scenario on a highway with dense traffic.