test

docs/10.md

@@ -270,7 +270,35 @@ $$
 **引理 4.1**
 
 $$
-J(\pi')-J(\pi) = \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\inf}\gamma^{t} A^{\pi}(s_t,a_t)]。
+J(\pi')-J(\pi) = \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)]。
 $$
 
-证明：
\ No newline at end of file
+证明：
+
+$$
+\mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)] = \mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty}\gamma^{t} [r(s_t,a_t)+\gamma V^{\pi}(s_{t+1}) - V^{\pi}(s_t)]]
+$$
+
+$$
+= \mathbb{E}_ {\tau\sim\pi'} [\sum_{t=0}^{\infty} \gamma^{t}r(s_t,a_t)] + \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t}[\gamma V^{\pi}(s_{t+1}) - V^{\pi}(s_t)]]
+$$
+
+$$
+= J(\pi') - \mathbb{E}_ {\tau\sim\pi'}[V^{\pi}(s_0)]
+$$
+
+$$
+J(\pi')-J(\pi)。
+$$
+
+证明完毕。$\diamondsuit$
+
+因此，我们有：
+
+$$
+\mathop{\max}_ {\pi'} J(\pi') = \mathop{\max}_{\pi'} J(\pi')-J(\pi)
+$$
+
+$$
+= \mathop{\max}_ {\pi'} \mathbb{E}_ {\tau\sim\pi'}[\sum_{t=0}^{\infty}\gamma^{t} A^{\pi}(s_t,a_t)]。
+$$
\ No newline at end of file