diff --git a/docs/10.md b/docs/10.md index d20798ee1228174e6fcd5c8ccacecfb8f107da76..d459e06af49a06ae4e1c840f1b2fa87b558c9229 100644 --- a/docs/10.md +++ b/docs/10.md @@ -270,7 +270,35 @@ $$ **引理 4.1** $$ -J(\pi')-J(\pi) = \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\inf}\gamma^{t} A^{\pi}(s_t,a_t)]。 +J(\pi')-J(\pi) = \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)]。 $$ -证明: \ No newline at end of file +证明: + +$$ +\mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)] = \mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty}\gamma^{t} [r(s_t,a_t)+\gamma V^{\pi}(s_{t+1}) - V^{\pi}(s_t)]] +$$ + +$$ += \mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty} \gamma^{t}r(s_t,a_t)] + \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t}[\gamma V^{\pi}(s_{t+1}) - V^{\pi}(s_t)]] +$$ + +$$ += J(\pi') - \mathbb{E}_ {\tau\sim\pi'}[V^{\pi}(s_0)] +$$ + +$$ +J(\pi')-J(\pi)。 +$$ + +证明完毕。$\diamondsuit$ + +因此,我们有: + +$$ +\mathop{\max}_ {\pi'} J(\pi') = \mathop{\max}_{\pi'} J(\pi')-J(\pi) +$$ + +$$ += \mathop{\max}_ {\pi'} \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)]。 +$$ \ No newline at end of file