Proximal Policy Optimization

Define the Clipped surrogate objective function to be the following:

$$L^{CLIP}(\theta )={\hat{\mathbb{E}}}_t\left[\min \right(r_t(\theta ){\hat{A}}_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon ){\hat{A}}_t]$$

where the ratio function $r_t$ is define as follows:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{old}}(a_t|s_t)}$$

Explanation of each part:

• The ratio function $r_t$ tells us how much more likely we are to take action $a_t$ at state $s_t$ with our current policy than with our old policy.
  This is the part we want to clip. If $r_t$ is much bigger than $1$ or much smaller than $1$ that means that our policy changed a lot. This is what we don’t want to happen so we clip it to fall into the range $[1-\epsilon ,1+\epsilon ]$.
• The advantage function ${\hat{A}}_t$ tells us how much better the action we have taken at timestep $t$ was compared to the mean.

The following table demonstrates how and when the objective function is clipped.

The final PPO objective function looks like the following:

$$L_t^{CLIP+VF+S}(\theta )={\hat{\mathbb{E}}}_t[L_t^{CLIP}(\theta )-c_1{L}_t^{VF}(\theta )+c_{2}S[{\pi }_{\theta }](s_t)]$$

Where $c_1$ and $c_2$ are hyperparameters to weigh each part.
$L_t^{VF}$ is the Squared-error value loss.
$S[{\pi }_{\theta }]$ is the entropy bonus to ensure sufficient exploration.