Intelligent control through learning and optimization

AMATH / CSE 579

Winter 2012: SLN 20029, MW 3:30-4:50, MGH 284



Emo Todorov

Office: Guggenheim 415h
Email: todorov@cs.washington.edu

Course Description

Design of near-optimal controllers for complex dynamical systems, using analytical techniques, machine learning, and optimization. Topics from deterministic and stochastic optimal control, reinforcement learning and dynamic programming, numerical optimization in the context of control, and robotics. Prerequisite: vector calculus, linear algebra, and Matlab. Recommended: differential equations, stochastic processes, and optimization.

Lecture slides

Lecture 1: Introduction

Lecture 2: Markov Decision Processes and Bellman Equations

Lecture 3: Controlled Diffusions and Hamilton-Jacobi-Bellman Equations

Lecture 4: Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters

Lecture 5: Pontryagin's Maximum Principle

Lecture 6: Trajectory-based Optimization

Lecture 7: Linearly-Solvable Stochastic Optimal Control Problems

Homework

Homework 1, due Feb 3

Homework 2, due Feb 29

Default Final Project

Project description, due Mar 16

Code

MDP solver: all problem formulations and algorithms

Acrobot dynamics

Animation of acrobot dynamics

Lecture slides from 2010

Inverse Optimal Control

Applications to biological movement

General Readings

A. Barto and R. Sutton (1998) Reinforcement learning: An introduction (online book)

E. Todorov (2006) Optimal control theory (book chapter)

D. Bertsekas (2008) Dynamic programming (lecture slides)

R. Tedrake (2009) Underactuated robotics: Learning, planning and control (lecture notes)

B. Van Roy (2004) Approximate dynamic programming (lecture notes)

P. Abbeel (2009) Advanced robotics (lecture slides)

Lecture-specific Readings

Lecture 6:
Todorov and Li (2005) A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In proceedings of ACC

Lecture 7:
Todorov (2009) Efficient computation of optimal actions. PNAS 106: 11478-11483

Krishnamurthy and Todorov (2010) Inverse optimal control with linearly-solvable MDPs. In proceedings of ICML

Muico, Lee, Popovic and Popovic (2009) Contact-aware nonlinear control of dynamics characters. In proceedings of SIGGRAPH

Wampler and Popovic (2009) Optimal gait and form for animal locomotion. In proceedings of SIGGRAPH

Bertsekas (2010) Approximate dynamic programming. In Dynamic Programming and Optimal Control, vol 2, 3rd ed

Todorov (2004) Optimality principles in sensorimotor control. Nature Neuroscience