diff --git a/docs/10.md b/docs/10.md index d3bd05f7d8367d5449457249b46a0645162b7925..2932d481b7274c9754019834e4b04919a9813102 100644 --- a/docs/10.md +++ b/docs/10.md @@ -148,5 +148,9 @@ $$ 最后的等式中,所有轨迹的概率和为 1。在倒数第二个等式中,由于 $b$ 为常值,我们可以将提出至积分外面(例如,$b$ 可以为平均回报,$b = \frac{1}{N}\sum_{i=1}^{N}r(\tau)$)。即使 $b$ 是状态 $s$ 的函数,这一项也是无偏的: $$ -\mathbb{E}_ {\tau\sim\pi_{\theta}(\tau)} [\nabla_{\theta}\log \pi_{\theta}(\tau) b(s_t)] = \mathbb{E}_ {s_{0:t},a_{0:(t-1)}} [\mathbb{E}_ {s_{(t+1):T},a_{t:(T-1)}} [\nabla_{\theta}\log \pi_{\theta}(a_t|s_t)b(s_t)]] +\mathbb{E}_ {\tau\sim\pi_{\theta}(\tau)} [\nabla_{\theta}\log \pi_{\theta}(\tau) b(s_t)] +$$ + +$$ += \mathbb{E}_ {s_{0:t},a_{0:(t-1)}} [\mathbb{E}_ {s_{(t+1):T},a_{t:(T-1)}} [\nabla_{\theta}\log \pi_{\theta}(a_t|s_t)b(s_t)]] $$ \ No newline at end of file