Tyler Westenbroek - Academia.edu (original) (raw)
Papers by Tyler Westenbroek
arXiv (Cornell University), Apr 4, 2017
In this paper, we introduce a preliminary model for interactions in the data market. Recent resea... more In this paper, we introduce a preliminary model for interactions in the data market. Recent research has shown ways in which a data aggregator can design mechanisms for users to ensure the quality of data, even in situations where the users are effort-averse (i.e. prefer to submit lower-quality estimates) and the data aggregator cannot observe the effort exerted by the users (i.e. the contract suffers from the principalagent problem). However, we have shown that these mechanisms often break down in more realistic models, where multiple data aggregators are in competition. Under minor assumptions on the properties of the statistical estimators in use by data aggregators, we show that there is either no Nash equilibrium, or there is an infinite number of Nash equilibrium. In the latter case, there is a fundamental ambiguity in who bears the burden of incentivizing different data sources. We are also able to calculate the price of anarchy, which measures how much social welfare is lost between the Nash equilibrium and the social optimum, i.e. between non-cooperative strategic play and cooperation.
arXiv (Cornell University), Mar 28, 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth vect... more This paper investigates optimal control problems formulated over a class of piecewise-smooth vector fields. Instead of optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system. It is shown that the smooth problems can be used to obtain accurate derivative information about the non-smooth problem, under standard regularity conditions. We then indicate how the regularizations can be used to consistently approximate the non-smooth optimal control problem in the sense of Polak. The utility of these smoothing techniques is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
arXiv (Cornell University), Apr 29, 2019
In recent years, data has played an increasingly important role in the economy as a good in its o... more In recent years, data has played an increasingly important role in the economy as a good in its own right. In many settings, data aggregators cannot directly verify the quality of the data they purchase, nor the effort exerted by data sources when creating the data. Recent work has explored mechanisms to ensure that the data sources share high quality data with a single data aggregator, addressing the issue of moral hazard. Oftentimes, there is a unique, socially efficient solution. In this paper, we consider data markets where there is more than one data aggregator. Since data can be cheaply reproduced and transmitted once created, data sources may share the same data with more than one aggregator, leading to free-riding between data aggregators. This coupling can lead to non-uniqueness of equilibria and social inefficiency. We examine a particular class of mechanisms that have received study recently in the literature, and we characterize all the generalized Nash equilibria of the resulting data market. We show that, in contrast to the single-aggregator case, there is either infinitely many generalized Nash equilibria or none. We also provide necessary and sufficient conditions for all equilibria to be socially inefficient. In our analysis, we identify the components of these mechanisms which give rise to these undesirable outcomes, showing the need for research into mechanisms for competitive settings with multiple data purchasers and sellers.
arXiv (Cornell University), Oct 30, 2015
We present a numerically tractable formulation for computing the optimal control of the class of ... more We present a numerically tractable formulation for computing the optimal control of the class of hybrid dynamical systems whose trajectories are continuous. Our formulation, an extension of existing relaxed-control techniques for switched dynamical systems, incorporates the domain information of each discrete mode as part of the constraints in the optimization problem. Moreover, our numerical results are consistent with phenomena that are particular to hybrid systems, such as the creation of sliding trajectories between discrete modes.
arXiv (Cornell University), Oct 29, 2019
We present a novel approach to control design for nonlinear systems which leverages model-free po... more We present a novel approach to control design for nonlinear systems which leverages model-free policy optimization techniques to learn a linearizing controller for a physical plant with unknown dynamics. Feedback linearization is a technique from nonlinear control which renders the input-output dynamics of a nonlinear plant \emph{linear} under application of an appropriate feedback controller. Once a linearizing controller has been constructed, desired output trajectories for the nonlinear plant can be tracked using a variety of linear control techniques. However, the calculation of a linearizing controller requires a precise dynamics model for the system. As a result, model-based approaches for learning exact linearizing controllers generally require a simple, highly structured model of the system with easily identifiable parameters. In contrast, the model-free approach presented in this paper is able to approximate the linearizing controller for the plant using general function approximation architectures. Specifically, we formulate a continuous-time optimization problem over the parameters of a learned linearizing controller whose optima are the set of parameters which best linearize the plant. We derive conditions under which the learning problem is (strongly) convex and provide guarantees which ensure the true linearizing controller for the plant is recovered. We then discuss how model-free policy optimization algorithms can be used to solve a discrete-time approximation to the problem using data collected from the real-world plant. The utility of the framework is demonstrated in simulation and on a real-world robotic platform.
International Conference on Cyber-Physical Systems, Apr 11, 2016
Wireless sensor-actuator networks (WSANs) are being adopted in process industries because of thei... more Wireless sensor-actuator networks (WSANs) are being adopted in process industries because of their advantages in lowering deployment and maintenance costs. While there has been significant theoretical advancement in networked control design, only limited empirical results that combine control design with realistic WSAN standards exist. This paper presents a cyberphysical case study on a wireless process control system that integrates state-of-the-art network control design and a WSAN based on the WirelessHART standard. The case study systematically explores the interactions between wireless routing and control design in the process control plant. The network supports alternative routing strategies, including single-path source routing and multi-path graph routing. To mitigate the effect of data loss in the WSAN, the control design integrates an observer based on an Extended Kalman Filter with a model predictive controller and an actuator buffer of recent control inputs. We observe that sensing and actuation can have different levels of resilience to packet loss under this network control design. We then propose a flexible routing approach where the routing strategy for sensing and actuation can be configured separately. Finally, we show that an asymmetric routing configuration with different routing strategies for sensing and actuation can effectively improve control performance under significant packet loss. Our results highlight the importance of co-joining the design of wireless network protocols and control in wireless control systems.
arXiv (Cornell University), Apr 21, 2020
This paper introduces a framework for learning a minimum-norm stabilizing controller for a system... more This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we use penalty methods to formulate an unconstrained optimization problem over the parameters of a learned controller, which can be solved using model-free policy optimization algorithms using data collected from the plant. We discuss when the optimization learns a stabilizing controller for the real world system and derive conditions on the structure of the learned controller which ensure that the optimization is strongly convex, meaning the globally optimal solution can be found reliably. We validate the approach in simulation, first for a double pendulum, and then generalize the framework to learn stable walking controllers for underactuated bipedal robots using the Hybrid Zero Dynamics framework. By encoding a large amount of structure into the learning problem, we are able to learn stabilizing controllers for both systems with only minutes or even seconds of training data. * Indicates equal contribution.
2021 60th IEEE Conference on Decision and Control (CDC), Dec 14, 2021
The widespread adoption of nonlinear Receding Horizon Control (RHC) strategies by industry has le... more The widespread adoption of nonlinear Receding Horizon Control (RHC) strategies by industry has led to more than 30 years of intense research efforts to provide stability guarantees for these methods. However, current theoretical guarantees require that each (generally nonconvex) planning problem can be solved to (approximate) global optimality, which is an unrealistic requirement for the derivativebased local optimization methods generally used in practical implementations of RHC. This paper takes the first step towards understanding stability guarantees for nonlinear RHC when the inner planning problem is solved to first-order stationary points, but not necessarily global optima. Special attention is given to feedback linearizable systems, and a mixture of positive and negative results are provided. We establish that, under certain strong conditions, first-order solutions to RHC exponentially stabilize linearizable systems. Crucially, this guarantee requires that state costs applied to the planning problems are in a certain sense 'compatible' with the global geometry of the system, and a simple counterexample demonstrates the necessity of this condition. These results highlight the need to rethink the role of global geometry in the context of optimization-based control.
2022 IEEE 61st Conference on Decision and Control (CDC)
Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite ... more Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite the extensive impacts of methods such as receding horizon control, dynamic programming and reinforcement learning, the design of cost functions for a particular system often remains a heuristicdriven process of trial and error. In this paper we seek to gain insights into how the choice of cost function interacts with the underlying structure of the control system and impacts the amount of computation required to obtain a stabilizing controller. We treat the cost design problem as a two-step process where the designer specifies outputs for the system that are to be penalized and then modulates the relative weighting of the inputs and the outputs in the cost. To characterize the computational burden associated to obtaining a stabilizing controller with a particular cost, we bound the prediction horizon required by receding horizon methods and the number of iterations required by dynamic programming methods to meet this requirement. Our theoretical results highlight a qualitative separation between what is possible, from a design perspective, when the chosen outputs induce either minimum-phase or non-minimum-phase behavior. Simulation studies indicate that this separation also holds for modern reinforcement learning methods.
arXiv (Cornell University), Oct 23, 2017
Hybrid dynamical systems have proven to be a powerful modeling abstraction, yet fundamental quest... more Hybrid dynamical systems have proven to be a powerful modeling abstraction, yet fundamental questions regarding the dynamical properties of these systems remain. In this paper, we develop a novel class of relaxations which we use to recover a number of classic systems theoretic properties for hybrid systems, such as existence and uniqueness of trajectories, even past the point of Zeno. Our relaxations also naturally give rise to a class of provably convergent numerical approximations, capable of simulating through Zeno. Using our methods, we are also able to perform sensitivity analysis about nominal trajectories undergoing a discrete transition-a technique with many practical applications, such as assessing the stability of periodic orbits.
This paper investigates optimal control problems formulated over a class of piecewise-smooth cont... more This paper investigates optimal control problems formulated over a class of piecewise-smooth controlled vector fields. Rather than optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system using tools from singular perturbation theory. Standard, efficient derivative-based algorithms are immediately applicable to solve these smooth approximations to local optimally. Under standard regularity conditions, it is demonstrated that the smooth approximations provide accurate derivative information about the non-smooth problem in the limiting case. The utility of the technique is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton
arXiv (Cornell University), Mar 28, 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth vect... more This paper investigates optimal control problems formulated over a class of piecewise-smooth vector fields. Instead of optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system. It is shown that the smooth problems can be used to obtain accurate derivative information about the non-smooth problem, under standard regularity conditions. We then indicate how the regularizations can be used to consistently approximate the non-smooth optimal control problem in the sense of Polak. The utility of these smoothing techniques is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
arXiv (Cornell University), Aug 13, 2022
Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automat... more Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem. This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller. The method adds a term involving a Control Lyapunov Function (CLF)-an 'energy-like' function from the model-based control literature-to typical cost formulations. Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which is well-known to reduce sample complexity. Moreover, the addition of the CLF term 'robustifies' the search for a stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system. We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1 quadruped with only seconds and a few minutes of finetuning data, respectively. Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders of magnitude less data compared to standard cost designs.
arXiv (Cornell University), Apr 5, 2022
Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite ... more Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite the extensive impacts of methods such as receding horizon control, dynamic programming and reinforcement learning, the design of cost functions for a particular system often remains a heuristicdriven process of trial and error. In this paper we seek to gain insights into how the choice of cost function interacts with the underlying structure of the control system and impacts the amount of computation required to obtain a stabilizing controller. We treat the cost design problem as a two-step process where the designer specifies outputs for the system that are to be penalized and then modulates the relative weighting of the inputs and the outputs in the cost. To characterize the computational burden associated to obtaining a stabilizing controller with a particular cost, we bound the prediction horizon required by receding horizon methods and the number of iterations required by dynamic programming methods to meet this requirement. Our theoretical results highlight a qualitative separation between what is possible, from a design perspective, when the chosen outputs induce either minimum-phase or non-minimum-phase behavior. Simulation studies indicate that this separation also holds for modern reinforcement learning methods.
![Research paper thumbnail of Feedback Linearization for Unknown Systems via Reinforcement Learning. (arXiv:1910.13272v2 [math.OC] UPDATED)](https://attachments.academia-assets.com/110805335/thumbnails/1.jpg)
](arXiv (Cornell University), Apr 23, 2020
![Research paper thumbnail of Learning Min-norm Stabilizing Control Laws for Systems with Unknown Dynamics. (arXiv:2004.10331v1 [math.OC])](https://attachments.academia-assets.com/110805333/thumbnails/1.jpg)
](arXiv (Cornell University), Apr 23, 2020
This paper introduces a framework for learning a minimum-norm stabilizing controller for a system... more This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we formulate an optimization problem over the parameters of a learned controller for the system. The optimization problem can be solved using model-free policy optimization algorithms and data collected from the real-world system. One term in the optimization encourages choices of parameters which minimize control effort, while another term penalizes violations of the safety constraint. If there exists at least one choice of learned parameters which satisfy the CLF constraint then all globally optimal solutions for the optimization also satisfy the constraint if the penalty term is scaled to be large enough. Furthermore, we derive conditions on the structure of the learned controller which ensure that the optimization is strongly convex, meaning the globally optimal solution can be found reliably. We validate the approach in simulation, first for a double pendulum, and then generalize to learn stable walking controllers for underactuated bipedal robots using the Hybrid Zero Dynamics framework. By encoding a large amount of structure into the learning problem, we are able to learn stabilizing controllers for both systems with only minutes or even seconds of training data. * Indicates equal contribution.
2019 IEEE 58th Conference on Decision and Control (CDC), 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth cont... more This paper investigates optimal control problems formulated over a class of piecewise-smooth controlled vector fields. Rather than optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system using tools from singular perturbation theory. Standard, efficient derivativebased algorithms are immediately applicable to solve these smooth approximations to local optimally. Under standard regularity conditions, it is demonstrated that the smooth approximations provide accurate derivative information about the non-smooth problem in the limiting case. The utility of the technique is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
2020 59th IEEE Conference on Decision and Control (CDC), 2020
We analyze stochastic differential equations and their discretizations to derive novel high proba... more We analyze stochastic differential equations and their discretizations to derive novel high probability tracking bounds for exponentially stable time varying systems which are corrupted by process noise. The bounds have an explicit dependence on the rate of convergence for the unperturbed system and the dimension of the state space. The magnitude of the stochastic deviations have a simple intuitive form, and our perturbation bounds also allow us to derive tighter high probability bounds on the tracking of reference trajectories than the state of the art. The resulting bounds can be used in analyzing many tracking control schemes.
2018 IEEE Conference on Decision and Control (CDC), 2018
We introduce a holistic framework for the analysis, approximation and control of the trajectories... more We introduce a holistic framework for the analysis, approximation and control of the trajectories of hybrid dynamical systems which display event-triggered discrete jumps in the continuous state. We begin by demonstrating how to explicitly represent the dynamics of this class of systems using a single piecewise-smooth vector field defined on a manifold, and then employ Filippov's solution concept to describe the trajectories of the system. The resulting hybrid Filippov solutions greatly simplify the mathematical description of hybrid executions, providing a unifying solution concept with which to work. Extending previous efforts to regularize piecewisesmooth vector fields, we then introduce a parameterized family of smooth control systems whose trajectories are used to approximate the hybrid Filippov solution numerically. The two solution concepts are shown to agree in the limit, under mild regularity conditions.
ArXiv, 2020
The main drawbacks of input-output linearizing controllers are the need for precise dynamics mode... more The main drawbacks of input-output linearizing controllers are the need for precise dynamics models and not being able to account for input constraints. Model uncertainty is common in almost every robotic application and input saturation is present in every real world system. In this paper, we address both challenges for the specific case of bipedal robot control by the use of reinforcement learning techniques. Taking the structure of a standard input-output linearizing controller, we use an additive learned term that compensates for model uncertainty. Moreover, by adding constraints to the learning problem we manage to boost the performance of the final controller when input limits are present. We demonstrate the effectiveness of the designed framework for different levels of uncertainty on the five-link planar walking robot RABBIT.
arXiv (Cornell University), Apr 4, 2017
In this paper, we introduce a preliminary model for interactions in the data market. Recent resea... more In this paper, we introduce a preliminary model for interactions in the data market. Recent research has shown ways in which a data aggregator can design mechanisms for users to ensure the quality of data, even in situations where the users are effort-averse (i.e. prefer to submit lower-quality estimates) and the data aggregator cannot observe the effort exerted by the users (i.e. the contract suffers from the principalagent problem). However, we have shown that these mechanisms often break down in more realistic models, where multiple data aggregators are in competition. Under minor assumptions on the properties of the statistical estimators in use by data aggregators, we show that there is either no Nash equilibrium, or there is an infinite number of Nash equilibrium. In the latter case, there is a fundamental ambiguity in who bears the burden of incentivizing different data sources. We are also able to calculate the price of anarchy, which measures how much social welfare is lost between the Nash equilibrium and the social optimum, i.e. between non-cooperative strategic play and cooperation.
arXiv (Cornell University), Mar 28, 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth vect... more This paper investigates optimal control problems formulated over a class of piecewise-smooth vector fields. Instead of optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system. It is shown that the smooth problems can be used to obtain accurate derivative information about the non-smooth problem, under standard regularity conditions. We then indicate how the regularizations can be used to consistently approximate the non-smooth optimal control problem in the sense of Polak. The utility of these smoothing techniques is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
arXiv (Cornell University), Apr 29, 2019
In recent years, data has played an increasingly important role in the economy as a good in its o... more In recent years, data has played an increasingly important role in the economy as a good in its own right. In many settings, data aggregators cannot directly verify the quality of the data they purchase, nor the effort exerted by data sources when creating the data. Recent work has explored mechanisms to ensure that the data sources share high quality data with a single data aggregator, addressing the issue of moral hazard. Oftentimes, there is a unique, socially efficient solution. In this paper, we consider data markets where there is more than one data aggregator. Since data can be cheaply reproduced and transmitted once created, data sources may share the same data with more than one aggregator, leading to free-riding between data aggregators. This coupling can lead to non-uniqueness of equilibria and social inefficiency. We examine a particular class of mechanisms that have received study recently in the literature, and we characterize all the generalized Nash equilibria of the resulting data market. We show that, in contrast to the single-aggregator case, there is either infinitely many generalized Nash equilibria or none. We also provide necessary and sufficient conditions for all equilibria to be socially inefficient. In our analysis, we identify the components of these mechanisms which give rise to these undesirable outcomes, showing the need for research into mechanisms for competitive settings with multiple data purchasers and sellers.
arXiv (Cornell University), Oct 30, 2015
We present a numerically tractable formulation for computing the optimal control of the class of ... more We present a numerically tractable formulation for computing the optimal control of the class of hybrid dynamical systems whose trajectories are continuous. Our formulation, an extension of existing relaxed-control techniques for switched dynamical systems, incorporates the domain information of each discrete mode as part of the constraints in the optimization problem. Moreover, our numerical results are consistent with phenomena that are particular to hybrid systems, such as the creation of sliding trajectories between discrete modes.
arXiv (Cornell University), Oct 29, 2019
We present a novel approach to control design for nonlinear systems which leverages model-free po... more We present a novel approach to control design for nonlinear systems which leverages model-free policy optimization techniques to learn a linearizing controller for a physical plant with unknown dynamics. Feedback linearization is a technique from nonlinear control which renders the input-output dynamics of a nonlinear plant \emph{linear} under application of an appropriate feedback controller. Once a linearizing controller has been constructed, desired output trajectories for the nonlinear plant can be tracked using a variety of linear control techniques. However, the calculation of a linearizing controller requires a precise dynamics model for the system. As a result, model-based approaches for learning exact linearizing controllers generally require a simple, highly structured model of the system with easily identifiable parameters. In contrast, the model-free approach presented in this paper is able to approximate the linearizing controller for the plant using general function approximation architectures. Specifically, we formulate a continuous-time optimization problem over the parameters of a learned linearizing controller whose optima are the set of parameters which best linearize the plant. We derive conditions under which the learning problem is (strongly) convex and provide guarantees which ensure the true linearizing controller for the plant is recovered. We then discuss how model-free policy optimization algorithms can be used to solve a discrete-time approximation to the problem using data collected from the real-world plant. The utility of the framework is demonstrated in simulation and on a real-world robotic platform.
International Conference on Cyber-Physical Systems, Apr 11, 2016
Wireless sensor-actuator networks (WSANs) are being adopted in process industries because of thei... more Wireless sensor-actuator networks (WSANs) are being adopted in process industries because of their advantages in lowering deployment and maintenance costs. While there has been significant theoretical advancement in networked control design, only limited empirical results that combine control design with realistic WSAN standards exist. This paper presents a cyberphysical case study on a wireless process control system that integrates state-of-the-art network control design and a WSAN based on the WirelessHART standard. The case study systematically explores the interactions between wireless routing and control design in the process control plant. The network supports alternative routing strategies, including single-path source routing and multi-path graph routing. To mitigate the effect of data loss in the WSAN, the control design integrates an observer based on an Extended Kalman Filter with a model predictive controller and an actuator buffer of recent control inputs. We observe that sensing and actuation can have different levels of resilience to packet loss under this network control design. We then propose a flexible routing approach where the routing strategy for sensing and actuation can be configured separately. Finally, we show that an asymmetric routing configuration with different routing strategies for sensing and actuation can effectively improve control performance under significant packet loss. Our results highlight the importance of co-joining the design of wireless network protocols and control in wireless control systems.
arXiv (Cornell University), Apr 21, 2020
This paper introduces a framework for learning a minimum-norm stabilizing controller for a system... more This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we use penalty methods to formulate an unconstrained optimization problem over the parameters of a learned controller, which can be solved using model-free policy optimization algorithms using data collected from the plant. We discuss when the optimization learns a stabilizing controller for the real world system and derive conditions on the structure of the learned controller which ensure that the optimization is strongly convex, meaning the globally optimal solution can be found reliably. We validate the approach in simulation, first for a double pendulum, and then generalize the framework to learn stable walking controllers for underactuated bipedal robots using the Hybrid Zero Dynamics framework. By encoding a large amount of structure into the learning problem, we are able to learn stabilizing controllers for both systems with only minutes or even seconds of training data. * Indicates equal contribution.
2021 60th IEEE Conference on Decision and Control (CDC), Dec 14, 2021
The widespread adoption of nonlinear Receding Horizon Control (RHC) strategies by industry has le... more The widespread adoption of nonlinear Receding Horizon Control (RHC) strategies by industry has led to more than 30 years of intense research efforts to provide stability guarantees for these methods. However, current theoretical guarantees require that each (generally nonconvex) planning problem can be solved to (approximate) global optimality, which is an unrealistic requirement for the derivativebased local optimization methods generally used in practical implementations of RHC. This paper takes the first step towards understanding stability guarantees for nonlinear RHC when the inner planning problem is solved to first-order stationary points, but not necessarily global optima. Special attention is given to feedback linearizable systems, and a mixture of positive and negative results are provided. We establish that, under certain strong conditions, first-order solutions to RHC exponentially stabilize linearizable systems. Crucially, this guarantee requires that state costs applied to the planning problems are in a certain sense 'compatible' with the global geometry of the system, and a simple counterexample demonstrates the necessity of this condition. These results highlight the need to rethink the role of global geometry in the context of optimization-based control.
2022 IEEE 61st Conference on Decision and Control (CDC)
Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite ... more Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite the extensive impacts of methods such as receding horizon control, dynamic programming and reinforcement learning, the design of cost functions for a particular system often remains a heuristicdriven process of trial and error. In this paper we seek to gain insights into how the choice of cost function interacts with the underlying structure of the control system and impacts the amount of computation required to obtain a stabilizing controller. We treat the cost design problem as a two-step process where the designer specifies outputs for the system that are to be penalized and then modulates the relative weighting of the inputs and the outputs in the cost. To characterize the computational burden associated to obtaining a stabilizing controller with a particular cost, we bound the prediction horizon required by receding horizon methods and the number of iterations required by dynamic programming methods to meet this requirement. Our theoretical results highlight a qualitative separation between what is possible, from a design perspective, when the chosen outputs induce either minimum-phase or non-minimum-phase behavior. Simulation studies indicate that this separation also holds for modern reinforcement learning methods.
arXiv (Cornell University), Oct 23, 2017
Hybrid dynamical systems have proven to be a powerful modeling abstraction, yet fundamental quest... more Hybrid dynamical systems have proven to be a powerful modeling abstraction, yet fundamental questions regarding the dynamical properties of these systems remain. In this paper, we develop a novel class of relaxations which we use to recover a number of classic systems theoretic properties for hybrid systems, such as existence and uniqueness of trajectories, even past the point of Zeno. Our relaxations also naturally give rise to a class of provably convergent numerical approximations, capable of simulating through Zeno. Using our methods, we are also able to perform sensitivity analysis about nominal trajectories undergoing a discrete transition-a technique with many practical applications, such as assessing the stability of periodic orbits.
This paper investigates optimal control problems formulated over a class of piecewise-smooth cont... more This paper investigates optimal control problems formulated over a class of piecewise-smooth controlled vector fields. Rather than optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system using tools from singular perturbation theory. Standard, efficient derivative-based algorithms are immediately applicable to solve these smooth approximations to local optimally. Under standard regularity conditions, it is demonstrated that the smooth approximations provide accurate derivative information about the non-smooth problem in the limiting case. The utility of the technique is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton
arXiv (Cornell University), Mar 28, 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth vect... more This paper investigates optimal control problems formulated over a class of piecewise-smooth vector fields. Instead of optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system. It is shown that the smooth problems can be used to obtain accurate derivative information about the non-smooth problem, under standard regularity conditions. We then indicate how the regularizations can be used to consistently approximate the non-smooth optimal control problem in the sense of Polak. The utility of these smoothing techniques is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
arXiv (Cornell University), Aug 13, 2022
Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automat... more Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem. This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller. The method adds a term involving a Control Lyapunov Function (CLF)-an 'energy-like' function from the model-based control literature-to typical cost formulations. Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which is well-known to reduce sample complexity. Moreover, the addition of the CLF term 'robustifies' the search for a stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system. We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1 quadruped with only seconds and a few minutes of finetuning data, respectively. Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders of magnitude less data compared to standard cost designs.
arXiv (Cornell University), Apr 5, 2022
Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite ... more Optimal control is an essential tool for stabilizing complex nonlinear systems. However, despite the extensive impacts of methods such as receding horizon control, dynamic programming and reinforcement learning, the design of cost functions for a particular system often remains a heuristicdriven process of trial and error. In this paper we seek to gain insights into how the choice of cost function interacts with the underlying structure of the control system and impacts the amount of computation required to obtain a stabilizing controller. We treat the cost design problem as a two-step process where the designer specifies outputs for the system that are to be penalized and then modulates the relative weighting of the inputs and the outputs in the cost. To characterize the computational burden associated to obtaining a stabilizing controller with a particular cost, we bound the prediction horizon required by receding horizon methods and the number of iterations required by dynamic programming methods to meet this requirement. Our theoretical results highlight a qualitative separation between what is possible, from a design perspective, when the chosen outputs induce either minimum-phase or non-minimum-phase behavior. Simulation studies indicate that this separation also holds for modern reinforcement learning methods.
arXiv (Cornell University), Apr 23, 2020
arXiv (Cornell University), Apr 23, 2020
This paper introduces a framework for learning a minimum-norm stabilizing controller for a system... more This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we formulate an optimization problem over the parameters of a learned controller for the system. The optimization problem can be solved using model-free policy optimization algorithms and data collected from the real-world system. One term in the optimization encourages choices of parameters which minimize control effort, while another term penalizes violations of the safety constraint. If there exists at least one choice of learned parameters which satisfy the CLF constraint then all globally optimal solutions for the optimization also satisfy the constraint if the penalty term is scaled to be large enough. Furthermore, we derive conditions on the structure of the learned controller which ensure that the optimization is strongly convex, meaning the globally optimal solution can be found reliably. We validate the approach in simulation, first for a double pendulum, and then generalize to learn stable walking controllers for underactuated bipedal robots using the Hybrid Zero Dynamics framework. By encoding a large amount of structure into the learning problem, we are able to learn stabilizing controllers for both systems with only minutes or even seconds of training data. * Indicates equal contribution.
2019 IEEE 58th Conference on Decision and Control (CDC), 2019
This paper investigates optimal control problems formulated over a class of piecewise-smooth cont... more This paper investigates optimal control problems formulated over a class of piecewise-smooth controlled vector fields. Rather than optimizing over the discontinuous system directly, we instead formulate optimal control problems over a family of regularizations which are obtained by "smoothing out" the discontinuity in the original system using tools from singular perturbation theory. Standard, efficient derivativebased algorithms are immediately applicable to solve these smooth approximations to local optimally. Under standard regularity conditions, it is demonstrated that the smooth approximations provide accurate derivative information about the non-smooth problem in the limiting case. The utility of the technique is demonstrated in an in-depth example, where we utilize recently developed reduced-order modeling techniques from the dynamic walking community to generate motion plans across contact sequences for a 18-DOF model of a lower-body exoskeleton.
2020 59th IEEE Conference on Decision and Control (CDC), 2020
We analyze stochastic differential equations and their discretizations to derive novel high proba... more We analyze stochastic differential equations and their discretizations to derive novel high probability tracking bounds for exponentially stable time varying systems which are corrupted by process noise. The bounds have an explicit dependence on the rate of convergence for the unperturbed system and the dimension of the state space. The magnitude of the stochastic deviations have a simple intuitive form, and our perturbation bounds also allow us to derive tighter high probability bounds on the tracking of reference trajectories than the state of the art. The resulting bounds can be used in analyzing many tracking control schemes.
2018 IEEE Conference on Decision and Control (CDC), 2018
We introduce a holistic framework for the analysis, approximation and control of the trajectories... more We introduce a holistic framework for the analysis, approximation and control of the trajectories of hybrid dynamical systems which display event-triggered discrete jumps in the continuous state. We begin by demonstrating how to explicitly represent the dynamics of this class of systems using a single piecewise-smooth vector field defined on a manifold, and then employ Filippov's solution concept to describe the trajectories of the system. The resulting hybrid Filippov solutions greatly simplify the mathematical description of hybrid executions, providing a unifying solution concept with which to work. Extending previous efforts to regularize piecewisesmooth vector fields, we then introduce a parameterized family of smooth control systems whose trajectories are used to approximate the hybrid Filippov solution numerically. The two solution concepts are shown to agree in the limit, under mild regularity conditions.
ArXiv, 2020
The main drawbacks of input-output linearizing controllers are the need for precise dynamics mode... more The main drawbacks of input-output linearizing controllers are the need for precise dynamics models and not being able to account for input constraints. Model uncertainty is common in almost every robotic application and input saturation is present in every real world system. In this paper, we address both challenges for the specific case of bipedal robot control by the use of reinforcement learning techniques. Taking the structure of a standard input-output linearizing controller, we use an additive learned term that compensates for model uncertainty. Moreover, by adding constraints to the learning problem we manage to boost the performance of the final controller when input limits are present. We demonstrate the effectiveness of the designed framework for different levels of uncertainty on the five-link planar walking robot RABBIT.