test

docs/8&9.md

@@ -251,4 +251,19 @@ $$
 \nabla_{\theta} \mathbb{E}_ {\pi_{\theta}}[r_{t'}] = \mathbb{E}_ {\pi_{\theta}}[r_{t'}\sum_{t=0}^{t'} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t)]。
 $$
 
-由于 $\sum_{t'=t}^{T-1}r_{t'}^{(i)}$ 就是回报 $G_t^{(i)}$，
\ No newline at end of file
+由于 $\sum_{t'=t}^{T-1}r_{t'}^{(i)}$ 就是回报 $G_t^{(i)}$，我们可以将其随时间步累加，从而得到：
+
+$$
+\nabla_{\theta} V(\theta) = \nabla_{\theta} \mathbb{E}_ {\tau \sim \pi_{\theta}}[R(\tau)] = \mathbb{E}_ {\pi_{\theta}}[\sum_{t'=0}^{T-1}r_{t'}\sum_{t=0}^{t'} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t)]
+\tag{12}
+$$
+
+$$
+= \mathbb{E}_ {\pi_{\theta}}[\sum_{t'=0}^{T-1} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) \sum_{t'=t}^{T-1}r_{t'}]
+\tag{13}
+$$
+
+$$
+= \mathbb{E}_ {\pi_{\theta}}[\sum_{t'=0}^{T-1} G_t \nabla_{\theta} \log \pi_{\theta}(a_t|s_t)]。
+\tag{14}
+$$
\ No newline at end of file