PPO from scratch
Proximal Policy Optimization built from first principles in PyTorch.
Pythonopen source
Problem
Most PPO code people reuse hides the parts that actually matter behind framework abstractions. The goal here was to implement PPO end to end with nothing hidden, so every component - advantage estimation, the policy objective, the value loss and the entropy term - is visible and correct.
Approach
- Generalised Advantage Estimation (GAE) computed with a backward pass over each trajectory, blending bias and variance through the lambda parameter.
- The clipped surrogate objective, taking the minimum of the unclipped and clipped probability-ratio terms so a single update cannot move the policy too far.
- A clipped value-function loss and an entropy bonus to keep exploration alive, combined into one scalar loss.
- The standard PPO training loop: collect rollouts, compute advantages, then run several epochs of minibatch updates over the same batch.
Tech
PyTorchNumPyGymnasiumPython
Status
Shipped and open source. It doubles as the reference implementation behind my blog post on implementing PPO from scratch.