4 of 4 written

Probability & Information Theory

The probability behind RL and state estimation: expectation and Monte Carlo, entropy and KL divergence, the score-function gradient, and Bayesian filtering.

Part 0

Expectation & Monte Carlo

Expectation, Variance, and Monte Carlo

Estimating an expectation by sampling, the operation every policy-gradient update is really doing.

Monte Carlo

Part 1

Entropy & Divergence

Entropy, KL Divergence, and Cross-Entropy

The information measures behind max-entropy RL, the TRPO trust region, and cross-entropy losses.

entropy

Part 2

Score-Function Gradient

The Score-Function Gradient

The likelihood-ratio trick that differentiates through sampling, and the baseline that tames its variance.

score function

Part 3

Bayesian Estimation

Bayesian Inference and the Kalman Filter

Recursive Bayesian estimation, and the Kalman filter that tracks a physical state through noise.

Bayes