MATH 472/872 Control of Stochastic Systems

Instructor:  Serdar Yuksel

Homework Assignments and Solutions
- [Assignment 1]        Due on Jan. 29th
- [Solutions 1]

- [Assignment 2]        Due on Feb. 16th
- [Solutions 2]

- [Assignment 3]        Due on March 12th by 4pm
- [Solutions 3]

- [Assignment 4]        Due on April 9th by 4pm
- [Solutions 4]


- [Course Syllabus]

- [Supplemental Lecture Notes]

Further Resources:
- [Bruce Hajek: "An Exploration of Random Processes for Engineers"]
- [S. Meyn: Control Techniques for Complex Networks]
- [S. Meyn and R. Tweedie: "Stochastic Stability and Markov Chains"]
- [A Survey Paper on Average Cost Optimal Control]

- Presentations will take place on April 7th from 9am to 12:30pm in Jeffery 225. Attendance is mandatory. Note: The grading for the presentations will be strict and the main criterion will be the group's ability to teach their peers and their own understanding of the material. 
Presentation Schedule:
1 Morgan, Chris, Alison and Callen: Q-Learning
2 Clifford, Lauren, Marwan: Reinforcement Learning
3 Graeme, Griffin, Louis, Steve: Decentralized Q-Learning
4 Adam, Connor, David: Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
5 Anthony, Chester, Xiaofeng: Online Markov Decision Processes under Bandit Feedback
6 Charlie, Mert and Liam: Multi-Armed Bandits
BREAK: 15 minutes
7 Adan, Alfred, Elizabeth, Emma: Learning Priors and Merging with Increasing Information
8 Daniel, Himesh, James: Exponential stability of discrete time filters for bounded observation noise
9 Doug, Jillian, Steven: Controlling IL-7 injections in HIV-infected patients
10 Jeremy, Margo, Stephen, Tyler: Risk-Constrained MDPs
11 Daniel: Optimal Control under Marginal Constraints
12 Chris: Stochastic Stability of Markov Chains
13 Diego: Non-Linear Filtering
14 Yang Chen:  Spiral Down Effects on Airline Ticket Sales

- Midterm will be held on Wednesday March 14th from 7pm to 9pm in Jeffery 126.
- Office Hours for this week: Monday: 11:30am-12:30am / Wednesday 10:30am-11:30am.  Otherwise, please make an appointment.

An Incomplete List of Possible Project Topics and Publications

Undergraduate students should have groups consisting of 3 students for their presentations and reports; in exceptional circumstances 4 students is also acceptable with instructor's approval (e.g., when there is research or extensive algorithmic aspects involved).

The grading for the presentations will be strict and the main criterion will be the group's ability to teach their peers and their own understanding of the material.  

Your reports should be less than 8 pages and contain the following: 1/3 of the report should focus on the problem description, known results and a literature review sufficient enough to reflect your knowledge of the field; 1/3 of the report on the paper's main contributions and results; you do not need to give details of the proofs but must give a convincing sketch; and 1/3 of the report should be on your review, comments, and critique on the paper with regard to its limitations, possible generalizations, applications etc.

Some sample reports from previous semesters: Andrew Brennan, Jeff Calder, Naci Saldi, Ben Wallace, M. Ebeling-Rump, M. Kao, Z. Hervieux-Moore