Learning control via probabilistic trajectory optimization

Abstract

A central problem in the field of robotics is to develop real-time planning and control algorithms for autonomous systems to behave intelligently under uncertainty. While classical optimal control provides a general theoretical framework, it relies on strong assumption of full knowledge of the system dynamics and environments. Alternatively, modern reinforcement learning (RL) offers a computational framework for controlling autonomous systems with minimal prior knowledge and user intervention. However, typical RL approaches require many interactions with the physical systems, and suffer from slow convergence. Furthermore, both optimal control and RL have the difficulty of scaling to high-dimensional state and action spaces.

In order to address these challenges, we present probabilistic trajectory optimization methods for solving optimal control problems for systems with unknown or partially known dynamics. Our methods share two key characteristics: (1) we incorporate explicit uncertainty into modeling, prediction and decision making using Gaussian processes; (2) our algorithms bypass the \textit{curse of dimensionality} via local approximation of the value function or linearization of the Hamilton-Jacobi-Bellman (HJB) equation. Compared to related approaches, our methods offer superior combination of data efficiency and scalability. We present experimental results and comparative analyses to demonstrate the strengths of the proposed methods.

In addition, we develop fast Bayesian approximate inference methods which enable probabilistic trajectory optimizer to perform real-time receding horizon control. It can be used to train deep neural network controllers that map raw observations to actions directly. We show that our approach can be used to perform high-speed off-road autonomous driving with low-cost sensors, and without on-the-fly planning and optimization.