Intelligent control through learning and optimization

AMATH / CSE 579

Spring 2015: MW 3:30-4:50, EEB 045



Emo Todorov

Office: CSE 434
Email: todorov@cs.washington.edu

Course Description

Design of near-optimal controllers for complex dynamical systems, using analytical techniques, machine learning, and optimization. Topics from deterministic and stochastic optimal control, reinforcement learning and dynamic programming, numerical optimization in the context of control, and robotics. Prerequisite: vector calculus, linear algebra, and Matlab. Recommended: differential equations, stochastic processes, and optimization.

Homework

Homework 1, due May 3

Homework 2, due May 24

Homework 3, due June 12

Office hours, Thursday June 11, 3:00pm to 4:30pm, CSE 434.

Code

MDP solver: all problem formulations and algorithms

acrobot.m: acrobot dynamics

testacrobot.m: visualize acrobot dynamics

Lecture slides

Introduction

Markov Decision Processes and Bellman Equations

Controlled Diffusions and Hamilton-Jacobi-Bellman Equations

Linearly-Quadratic-Gaussian Controllers and Kalman Filters

Pontryagin's Maximum Principle

Trajectory Optimization

Linearly-Solvable Stochastic Optimal Control Problems

Relevant research papers

Least-Squares Policy Iteration

An Analysis of Linear Models, Linear Value-Function Approximation, and Feature Selection for Reinforcement Learning

Value Function Approximation in Reinforcement Learning Using the Fourier Basis

Least Squares Solutions of the HJB Equation With Neural Network Value-Function Approximators

Reinforcement Learning In Continuous Time and Space

Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes

Dynamical Movement Primitives: Learning Attractor Models forMotor Behaviors

An Iterative Path Integral Stochastic Optimal Control Approach for Learning Robotic Tasks

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Compositionality of Optimal Control Laws

Policy Gradients in Linearly-Solvable MDPs

Robot Trajectory Optimization using Approximate Inference

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Natural Actor-Critic

SIMBICON: Simple Biped Locomotion Control

Optimizing Walking Controllers for Uncertain Inputs and Environments

Trajectory Optimization for Full-Body Movements with Complex Contacts

Dynamic Optimization of Human Walking

Computer Optimization of a Minimal Biped Model Discovers Walking and Running

Optimal Sensorimotor Transformations for Balance

Synthesis of Detailed Hand Manipulations Using Contact Smoothing

Continuous Character Control with Low-Dimensional Embeddings

Motion Fields for Interactive Character Animation

General Readings

A. Barto and R. Sutton (1998) Reinforcement learning: An introduction (online book)

E. Todorov (2006) Optimal control theory (book chapter)

D. Bertsekas (2008) Dynamic programming (lecture slides)

R. Tedrake (2009) Underactuated robotics: Learning, planning and control (lecture notes)

B. Van Roy (2004) Approximate dynamic programming (lecture notes)

P. Abbeel (2009) Advanced robotics (lecture slides)