5 of 5 written

Reinforcement Learning Theory

The math spine of RL: MDPs and value functions, the Bellman operators, policy gradients, and the trust-region and natural-gradient methods that stabilize them.

Part 0

MDPs

MDPs, Returns, and Value Functions

The Markov decision process, the object a simulator becomes once you add a reward.

MDP

Part 1

Bellman & Value Iteration

Bellman Operators and Value Iteration

The Bellman operator as a contraction, and value iteration converging to the optimal value.

Bellman

Part 2

Policy Gradients

Policy Gradients and REINFORCE

The policy gradient theorem, REINFORCE, and why a baseline cuts the variance.

policy gradient

Part 3

Trust Regions

Natural Gradient, TRPO, and PPO

Following the steepest direction in policy space under a KL metric, the idea behind TRPO and PPO.

natural gradient

Part 4

Actor-Critic

Actor-Critic and Advantage Estimation

Learning a critic to cut variance, and the bias-variance dial of generalized advantage estimation.

actor-critic