Deploying to gh-pages from @ 8f47f4cb 🚀

dd50cb7a · PaParaZz1 · 15097372 · dd50cb7a · dd50cb7a · dd50cb7a
44 changed file
--- a/_modules/ding/config/config.html
+++ b/_modules/ding/config/config.html
--- a/_modules/ding/model/common/head.html
+++ b/_modules/ding/model/common/head.html
@@ -699,103 +699,6 @@
        <span class="n">logit</span> <span class="o">=</span> <span class="n">a</span> <span class="o">-</span> <span class="n">a</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="o">+</span> <span class="n">v</span>
        <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;logit&#39;</span><span class="p">:</span> <span class="n">logit</span><span class="p">}</span></div></div>

-<span class="k">class</span> <span class="nc">StochasticDuelingHead</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
-
-    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
-        <span class="bp">self</span><span class="p">,</span>
-        <span class="n">hidden_size</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
-        <span class="n">output_size</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
-        <span class="n">action_shape</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
-        <span class="n">layer_num</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span>
-        <span class="n">a_layer_num</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
-        <span class="n">v_layer_num</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
-        <span class="n">activation</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">]</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">ReLU</span><span class="p">(),</span>
-        <span class="n">norm_type</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
-        <span class="n">noise</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
-
-    <span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
-        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
-<span class="sd">        Overview:</span>
-<span class="sd">            Init the Head according to arguments.</span>
-<span class="sd">        Arguments:</span>
-<span class="sd">            - hidden_size (:obj:`int`): The ``hidden_size`` used before connected to ``DuelingHead``</span>
-<span class="sd">            - output_size (:obj:`int`): The num of output</span>
-<span class="sd">            - a_layer_num (:obj:`int`): The num of layers used in the network to compute action output</span>
-<span class="sd">            - v_layer_num (:obj:`int`): The num of layers used in the network to compute value output</span>
-<span class="sd">            - activation (:obj:`nn.Module`):</span>
-<span class="sd">                The type of activation function to use in ``MLP`` the after ``layer_fn``,</span>
-<span class="sd">                if ``None`` then default set to ``nn.ReLU()``</span>
-<span class="sd">            - norm_type (:obj:`str`):</span>
-<span class="sd">                The type of normalization to use, see ``ding.torch_utils.fc_block`` for more details</span>
-<span class="sd">            - noise (:obj:`bool`): Whether use noisy ``fc_block``</span>
-<span class="sd">        &quot;&quot;&quot;</span>
-        <span class="nb">super</span><span class="p">(</span><span class="n">DuelingHead</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
-        <span class="k">if</span> <span class="n">a_layer_num</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
-            <span class="n">a_layer_num</span> <span class="o">=</span> <span class="n">layer_num</span>
-        <span class="k">if</span> <span class="n">v_layer_num</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
-            <span class="n">v_layer_num</span> <span class="o">=</span> <span class="n">layer_num</span>
-        <span class="n">layer</span> <span class="o">=</span> <span class="n">NoiseLinearLayer</span> <span class="k">if</span> <span class="n">noise</span> <span class="k">else</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span>
-        <span class="n">block</span> <span class="o">=</span> <span class="n">noise_block</span> <span class="k">if</span> <span class="n">noise</span> <span class="k">else</span> <span class="n">fc_block</span>
-        <span class="bp">self</span><span class="o">.</span><span class="n">A</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">(</span>
-            <span class="n">MLP</span><span class="p">(</span>
-                <span class="n">hidden_size</span> <span class="o">+</span> <span class="n">action_shape</span><span class="p">,</span>
-                <span class="n">hidden_size</span> <span class="o">+</span> <span class="n">action_shape</span><span class="p">,</span>
-                <span class="n">hidden_size</span> <span class="o">+</span> <span class="n">action_shape</span><span class="p">,</span>
-                <span class="n">a_layer_num</span><span class="p">,</span>
-                <span class="n">layer_fn</span><span class="o">=</span><span class="n">layer</span><span class="p">,</span>
-                <span class="n">activation</span><span class="o">=</span><span class="n">activation</span><span class="p">,</span>
-                <span class="n">norm_type</span><span class="o">=</span><span class="n">norm_type</span>
-            <span class="p">),</span> <span class="n">block</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">output_size</span><span class="p">)</span>
-        <span class="p">)</span>
-        <span class="bp">self</span><span class="o">.</span><span class="n">V</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">(</span>
-            <span class="n">MLP</span><span class="p">(</span>
-                <span class="n">hidden_size</span><span class="p">,</span>
-                <span class="n">hidden_size</span><span class="p">,</span>
-                <span class="n">hidden_size</span><span class="p">,</span>
-                <span class="n">v_layer_num</span><span class="p">,</span>
-                <span class="n">layer_fn</span><span class="o">=</span><span class="n">layer</span><span class="p">,</span>
-                <span class="n">activation</span><span class="o">=</span><span class="n">activation</span><span class="p">,</span>
-                <span class="n">norm_type</span><span class="o">=</span><span class="n">norm_type</span>
-            <span class="p">),</span> <span class="n">block</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
-        <span class="p">)</span>
-
-    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">a</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">mu_t</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">sigma_t</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">sample_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">:</span>
-        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
-<span class="sd">        Overview:</span>
-<span class="sd">            Use encoded embedding tensor to predict Dueling output.</span>
-<span class="sd">            Parameter updates with DuelingHead&#39;s MLPs forward setup.</span>
-<span class="sd">        Arguments:</span>
-<span class="sd">            - s (:obj:`torch.Tensor`):</span>
-<span class="sd">                The encoded embedding state tensor, determined with given ``hidden_size``, i.e. ``(B, N=hidden_size)``.</span>
-<span class="sd">            - a (:obj:`torch.Tensor`):</span>
-<span class="sd">                The encoded embedding action tensor, determined with ``action_size``, i.e. ``(B, N=action_size)``.</span>
-<span class="sd">            - mu_t (:obj:`torch.Tensor`):</span>
-<span class="sd">                The mu output of actor head at timestep t, size (B, action_size)</span>
-<span class="sd">            - sigma_t (:obj:`torch.Tensor`):</span>
-<span class="sd">                The sigma output of actor head at timestep t, size (B, action_size)</span>
-<span class="sd">            - sample_size (:obj:`int`):</span>
-<span class="sd">                The num to samples for action when computing the Q value</span>
-<span class="sd">        Returns:</span>
-<span class="sd">            - outputs (:obj:`Dict`):</span>
-<span class="sd">                Run ``MLP`` with ``DuelingHead`` setups and return the result prediction dictionary.</span>
-<span class="sd">                Necessary Keys:</span>
-<span class="sd">                    - pred (:obj:`torch.Tensor`): Pred tensor of size ``(B, 1)``.</span>
-<span class="sd">        &quot;&quot;&quot;</span>
-        <span class="n">batch_size</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
-        <span class="n">hidden_size</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
-        <span class="n">action_size</span> <span class="o">=</span> <span class="n">a</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
-        <span class="n">state_cat_action</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">((</span><span class="n">s</span><span class="p">,</span><span class="n">a</span><span class="p">),</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>  <span class="c1"># size (B, action_size + state_size)</span>
-        <span class="n">a_val</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">A</span><span class="p">(</span><span class="n">state_cat_action</span><span class="p">)</span>  <span class="c1"># size (B, 1)</span>
-        <span class="n">s_val</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">V</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>  <span class="c1"># size (B,1)</span>
-        <span class="n">mu_t</span> <span class="o">=</span> <span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="n">mu_t</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span><span class="o">.</span><span class="n">expand</span><span class="p">((</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">sample_size</span><span class="p">,</span> <span class="n">action_size</span><span class="p">,))</span>  <span class="c1"># size (B, sample_size, action_size) </span>
-        <span class="n">sigma_t</span> <span class="o">=</span> <span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="n">sigma_t</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span><span class="o">.</span><span class="n">expand</span><span class="p">((</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">sample_size</span><span class="p">,</span> <span class="n">action_size</span><span class="p">))</span>  <span class="c1"># size (B, sample_size, action_size) </span>
-        <span class="n">expand_s</span> <span class="o">=</span> <span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span><span class="o">.</span><span class="n">expand</span><span class="p">((</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">sample_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">))</span>  <span class="c1"># size (B, sample_size, action_size) </span>
-        <span class="n">action_sample</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">mu_t</span><span class="p">,</span> <span class="n">sigma_t</span><span class="p">)</span>  <span class="c1"># size (B, sample_size, action_size) </span>
-        <span class="n">state_cat_action_sample</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">((</span><span class="n">expand_s</span><span class="p">,</span><span class="n">action_sample</span><span class="p">),</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>   <span class="c1"># size (B, sample_size, action_size + hidden_size) </span>
-        <span class="n">a_val_sample</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">A</span><span class="p">(</span><span class="n">state_cat_action_sample</span><span class="p">)</span>  <span class="c1"># size (B, sample_size, 1)</span>
-        <span class="n">a_val_sample</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">squeeze</span><span class="p">(</span><span class="n">a_val_sample</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>  <span class="c1"># (B, sample_size)</span>
-        <span class="n">logit</span> <span class="o">=</span> <span class="n">a_val</span> <span class="o">-</span> <span class="n">a_val_sample</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="o">+</span> <span class="n">s_val</span>  <span class="c1"># size (B,1)</span>
-        <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;pred&#39;</span><span class="p">:</span> <span class="n">logit</span><span class="p">}</span>

 <div class="viewcode-block" id="RegressionHead"><a class="viewcode-back" href="../../../../api_doc/model/common/head.html#ding.model.common.head.RegressionHead">[docs]</a><span class="k">class</span> <span class="nc">RegressionHead</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>

@@ -921,8 +824,8 @@
 <span class="sd">                Run ``MLP`` with ``ReparameterizationHead`` setups and return the result prediction dictionary.</span>

 <span class="sd">                Necessary Keys:</span>
-<span class="sd">                    - mu (:obj:`torch.Tensor`) Tensor of cells of updated mu values of size ``(B, action_size)``</span>
-<span class="sd">                    - sigma (:obj:`torch.Tensor`) Tensor of cells of updated sigma values of size ``(B, action_size)``</span>
+<span class="sd">                    - mu (:obj:`torch.Tensor`) Tensor of cells of updated mu values, with same size as ``x``.</span>
+<span class="sd">                    - sigma (:obj:`torch.Tensor`) Tensor of cells of updated sigma values, with same size as ``x``.</span>
 <span class="sd">        Examples:</span>
 <span class="sd">            &gt;&gt;&gt; head =  ReparameterizationHead(64, 64, sigma_type=&#39;fixed&#39;)</span>
 <span class="sd">            &gt;&gt;&gt; inputs = torch.randn(4, 64)</span>
@@ -1010,7 +913,6 @@
    <span class="s1">&#39;reparameterization&#39;</span><span class="p">:</span> <span class="n">ReparameterizationHead</span><span class="p">,</span>
    <span class="c1"># multi</span>
    <span class="s1">&#39;multi&#39;</span><span class="p">:</span> <span class="n">MultiHead</span><span class="p">,</span>
-    <span class="s1">&#39;sdn&#39;</span><span class="p">:</span> <span class="n">StochasticDuelingHead</span><span class="p">,</span>
 <span class="p">}</span>
 </pre></div>


--- a/_modules/ding/policy/acer.html
+++ b/_modules/ding/policy/acer.html
--- a/_modules/ding/policy/sql.html
+++ b/_modules/ding/policy/sql.html
--- a/_modules/ding/worker/collector/base_serial_collector.html
+++ b/_modules/ding/worker/collector/base_serial_collector.html
@@ -175,7 +175,7 @@
 <span class="n">INF</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="s2">&quot;inf&quot;</span><span class="p">)</span>


-<span class="k">class</span> <span class="nc">ISerialCollector</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
+<div class="viewcode-block" id="ISerialCollector"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector">[docs]</a><span class="k">class</span> <span class="nc">ISerialCollector</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">    Overview:</span>
 <span class="sd">        Abstract baseclass for serial collector.</span>
@@ -185,7 +185,7 @@
 <span class="sd">        envstep</span>
 <span class="sd">    &quot;&quot;&quot;</span>

-    <span class="nd">@classmethod</span>
+<div class="viewcode-block" id="ISerialCollector.default_config"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector.default_config">[docs]</a>    <span class="nd">@classmethod</span>
    <span class="k">def</span> <span class="nf">default_config</span><span class="p">(</span><span class="bp">cls</span><span class="p">:</span> <span class="nb">type</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">EasyDict</span><span class="p">:</span>
        <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">        Overview:</span>
@@ -196,42 +196,42 @@
 <span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">cfg</span> <span class="o">=</span> <span class="n">EasyDict</span><span class="p">(</span><span class="n">copy</span><span class="o">.</span><span class="n">deepcopy</span><span class="p">(</span><span class="bp">cls</span><span class="o">.</span><span class="n">config</span><span class="p">))</span>
        <span class="n">cfg</span><span class="o">.</span><span class="n">cfg_type</span> <span class="o">=</span> <span class="bp">cls</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">+</span> <span class="s1">&#39;Dict&#39;</span>
-        <span class="k">return</span> <span class="n">cfg</span>
+        <span class="k">return</span> <span class="n">cfg</span></div>

-    <span class="nd">@abstractmethod</span>
+<div class="viewcode-block" id="ISerialCollector.reset_env"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector.reset_env">[docs]</a>    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">reset_env</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_env</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">BaseEnvManager</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
        <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">        Overview:</span>
 <span class="sd">            Reset collector&#39;s environment. In some case, we need collector use the same policy to collect \</span>
 <span class="sd">                data in different environments. We can use reset_env to reset the environment.</span>
 <span class="sd">        &quot;&quot;&quot;</span>
-        <span class="k">raise</span> <span class="ne">NotImplementedError</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>

-    <span class="nd">@abstractmethod</span>
+<div class="viewcode-block" id="ISerialCollector.reset_policy"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector.reset_policy">[docs]</a>    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">reset_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_policy</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">namedtuple</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
        <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">        Overview:</span>
 <span class="sd">            Reset collector&#39;s policy. In some case, we need collector work in this same environment but use\</span>
 <span class="sd">                different policy to collect data. We can use reset_policy to reset the policy.</span>
 <span class="sd">        &quot;&quot;&quot;</span>
-        <span class="k">raise</span> <span class="ne">NotImplementedError</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>

-    <span class="nd">@abstractmethod</span>
+<div class="viewcode-block" id="ISerialCollector.reset"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector.reset">[docs]</a>    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_policy</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">namedtuple</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">_env</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">BaseEnvManager</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
        <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">        Overview:</span>
 <span class="sd">            Reset collector&#39;s policy and environment. Use new policy and environment to collect data.</span>
 <span class="sd">        &quot;&quot;&quot;</span>
-        <span class="k">raise</span> <span class="ne">NotImplementedError</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>

-    <span class="nd">@abstractmethod</span>
+<div class="viewcode-block" id="ISerialCollector.collect"><a class="viewcode-back" href="../../../../api_doc/worker/collector/base_serial_collector.html#ding.worker.collector.base_serial_collector.ISerialCollector.collect">[docs]</a>    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">collect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">per_collect_target</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="n">Any</span><span class="p">]:</span>
        <span class="sd">&quot;&quot;&quot;</span>
 <span class="sd">        Overview:</span>
 <span class="sd">            Collect the corresponding data according to the specified target and return. \</span>
 <span class="sd">                There are different definitions in episode and sample mode.</span>
 <span class="sd">        &quot;&quot;&quot;</span>
-        <span class="k">raise</span> <span class="ne">NotImplementedError</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>

    <span class="nd">@abstractproperty</span>
    <span class="k">def</span> <span class="nf">envstep</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
@@ -239,7 +239,7 @@
 <span class="sd">        Overview:</span>
 <span class="sd">            Get the total envstep num.</span>
 <span class="sd">        &quot;&quot;&quot;</span>
-        <span class="k">raise</span> <span class="ne">NotImplementedError</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>


 <span class="k">def</span> <span class="nf">create_serial_collector</span><span class="p">(</span><span class="n">cfg</span><span class="p">:</span> <span class="n">EasyDict</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ISerialCollector</span><span class="p">:</span>

--- a/_modules/ding/worker/collector/comm/base_comm_collector.html
+++ b/_modules/ding/worker/collector/comm/base_comm_collector.html
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>ding.worker.collector.comm.base_comm_collector &mdash; DI-engine 0.1.0 documentation</title>
+  
+
+  
+  
+  
+  
+
+  
+  <script type="text/javascript" src="../../../../../_static/js/modernizr.min.js"></script>
+  
+    
+      <script type="text/javascript" id="documentation_options" data-url_root="../../../../../" src="../../../../../_static/documentation_options.js"></script>
+        <script data-url_root="../../../../../" id="documentation_options" src="../../../../../_static/documentation_options.js"></script>
+        <script src="../../../../../_static/jquery.js"></script>
+        <script src="../../../../../_static/underscore.js"></script>
+        <script src="../../../../../_static/doctools.js"></script>
+    
+    <script type="text/javascript" src="../../../../../_static/js/theme.js"></script>
+
+    
+
+  
+  <link rel="stylesheet" href="../../../../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/css/theme.css" type="text/css" />
+    <link rel="index" title="Index" href="../../../../../genindex.html" />
+    <link rel="search" title="Search" href="../../../../../search.html" />
+    <link href="../../../../../_static/css/style.css" rel="stylesheet" type="text/css">
+
+</head>
+
+<body class="wy-body-for-nav">
+
+   
+  <div class="wy-grid-for-nav">
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search" >
+          
+
+          
+            <a href="../../../../../index.html" class="icon icon-home"> DI-engine
+          
+
+          
+          </a>
+
+          
+            
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../../../../../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+              
+            
+            
+              <p class="caption" role="heading"><span class="caption-text">User Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../installation/index.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../quick_start/index.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../key_concept/index.html">Key Concept</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../intro_rl/index.html">Introduction to RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../hands_on/index.html">Hands on RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../best_practice/index.html">Best Practice</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../api_doc/index.html">API Doc</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../faq/index_en.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../feature/index.html">Feature</a></li>
+</ul>
+<p class="caption" role="heading"><span class="caption-text">Developer Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../guide/index.html">Developer Guide</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../tutorial_dev/index.html">Tutorial-Developer</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../architecture/index.html">Architecture Design</a></li>
+</ul>
+
+            
+          
+        </div>
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" aria-label="top navigation">
+        
+          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+          <a href="../../../../../index.html">DI-engine</a>
+        
+      </nav>
+
+
+      <div class="wy-nav-content">
+        
+        <div class="rst-content">
+        
+          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    
+      <li><a href="../../../../../index.html">Docs</a> &raquo;</li>
+        
+          <li><a href="../../../../index.html">Module code</a> &raquo;</li>
+        
+      <li>ding.worker.collector.comm.base_comm_collector</li>
+    
+    
+      <li class="wy-breadcrumbs-aside">
+        
+      </li>
+    
+  </ul>
+
+  
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <h1>Source code for ding.worker.collector.comm.base_comm_collector</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
+<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
+<span class="kn">from</span> <span class="nn">easydict</span> <span class="kn">import</span> <span class="n">EasyDict</span>
+
+<span class="kn">from</span> <span class="nn">ding.utils</span> <span class="kn">import</span> <span class="n">get_task_uid</span><span class="p">,</span> <span class="n">import_module</span><span class="p">,</span> <span class="n">COMM_COLLECTOR_REGISTRY</span>
+<span class="kn">from</span> <span class="nn">..base_parallel_collector</span> <span class="kn">import</span> <span class="n">create_parallel_collector</span><span class="p">,</span> <span class="n">BaseParallelCollector</span>
+
+
+<div class="viewcode-block" id="BaseCommCollector"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector">[docs]</a><span class="k">class</span> <span class="nc">BaseCommCollector</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Overview:</span>
+<span class="sd">        Abstract baseclass for common collector.</span>
+<span class="sd">    Interfaces:</span>
+<span class="sd">        __init__, get_policy_update_info, send_metadata, send_stepdata</span>
+<span class="sd">        start, close, _create_collector</span>
+<span class="sd">    Property:</span>
+<span class="sd">        collector_uid</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+
+<div class="viewcode-block" id="BaseCommCollector.__init__"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.__init__">[docs]</a>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Initialization method.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - cfg (:obj:`EasyDict`): Config dict</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_cfg</span> <span class="o">=</span> <span class="n">cfg</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_end_flag</span> <span class="o">=</span> <span class="kc">True</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_collector_uid</span> <span class="o">=</span> <span class="n">get_task_uid</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="BaseCommCollector.get_policy_update_info"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.get_policy_update_info">[docs]</a>    <span class="nd">@abstractmethod</span>
+    <span class="k">def</span> <span class="nf">get_policy_update_info</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Get policy information in corresponding path.</span>
+<span class="sd">            Will be registered in base collector.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - path (:obj:`str`): path to policy update information.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>
+
+<div class="viewcode-block" id="BaseCommCollector.send_metadata"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.send_metadata">[docs]</a>    <span class="nd">@abstractmethod</span>
+    <span class="k">def</span> <span class="nf">send_metadata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">metadata</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Store meta data in queue, which will be retrieved by callback function &quot;deal_with_collector_data&quot;</span>
+<span class="sd">            in collector slave, then will be sent to coordinator.</span>
+<span class="sd">            Will be registered in base collector.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - metadata (:obj:`Any`): meta data.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>
+
+<div class="viewcode-block" id="BaseCommCollector.send_stepdata"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.send_stepdata">[docs]</a>    <span class="nd">@abstractmethod</span>
+    <span class="k">def</span> <span class="nf">send_stepdata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">stepdata</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Save step data in corresponding path.</span>
+<span class="sd">            Will be registered in base collector.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - stepdata (:obj:`Any`): step data.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="k">raise</span> <span class="ne">NotImplementedError</span></div>
+
+<div class="viewcode-block" id="BaseCommCollector.start"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.start">[docs]</a>    <span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Start comm collector.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_end_flag</span> <span class="o">=</span> <span class="kc">False</span></div>
+
+<div class="viewcode-block" id="BaseCommCollector.close"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector.close">[docs]</a>    <span class="k">def</span> <span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Close comm collector.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_end_flag</span> <span class="o">=</span> <span class="kc">True</span></div>
+
+    <span class="nd">@property</span>
+    <span class="k">def</span> <span class="nf">collector_uid</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
+        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_collector_uid</span>
+
+<div class="viewcode-block" id="BaseCommCollector._create_collector"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.base_comm_collector.BaseCommCollector._create_collector">[docs]</a>    <span class="k">def</span> <span class="nf">_create_collector</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_info</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">BaseParallelCollector</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Receive ``task_info`` passed from coordinator and create a collector.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - task_info (:obj:`dict`): Task info dict from coordinator. Should be like \</span>
+<span class="sd">        Returns:</span>
+<span class="sd">            - collector (:obj:`BaseParallelCollector`): Created base collector.</span>
+<span class="sd">        Note:</span>
+<span class="sd">            Four methods(&#39;send_metadata&#39;, &#39;send_stepdata&#39;, &#39;get_policy_update_info&#39;), and policy are set.</span>
+<span class="sd">            The reason why they are set here rather than base collector is, they highly depend on the specific task.</span>
+<span class="sd">            Only after task info is passed from coordinator to comm collector through learner slave, can they be</span>
+<span class="sd">            clarified and initialized.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="n">collector_cfg</span> <span class="o">=</span> <span class="n">EasyDict</span><span class="p">(</span><span class="n">task_info</span><span class="p">[</span><span class="s1">&#39;collector_cfg&#39;</span><span class="p">])</span>
+        <span class="n">collector</span> <span class="o">=</span> <span class="n">create_parallel_collector</span><span class="p">(</span><span class="n">collector_cfg</span><span class="p">)</span>
+        <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">&#39;send_metadata&#39;</span><span class="p">,</span> <span class="s1">&#39;send_stepdata&#39;</span><span class="p">,</span> <span class="s1">&#39;get_policy_update_info&#39;</span><span class="p">]:</span>
+            <span class="nb">setattr</span><span class="p">(</span><span class="n">collector</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">))</span>
+        <span class="k">return</span> <span class="n">collector</span></div></div>
+
+
+<span class="k">def</span> <span class="nf">create_comm_collector</span><span class="p">(</span><span class="n">cfg</span><span class="p">:</span> <span class="n">EasyDict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">BaseCommCollector</span><span class="p">:</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Overview:</span>
+<span class="sd">        Given the key(comm_collector_name), create a new comm collector instance if in comm_map&#39;s values,</span>
+<span class="sd">        or raise an KeyError. In other words, a derived comm collector must first register,</span>
+<span class="sd">        then can call ``create_comm_collector`` to get the instance.</span>
+<span class="sd">    Arguments:</span>
+<span class="sd">        - cfg (:obj:`EasyDict`): Collector config. Necessary keys: [import_names, comm_collector_type].</span>
+<span class="sd">    Returns:</span>
+<span class="sd">        - collector (:obj:`BaseCommCollector`): The created new comm collector, should be an instance of one of \</span>
+<span class="sd">        comm_map&#39;s values.</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+    <span class="n">import_module</span><span class="p">(</span><span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;import_names&#39;</span><span class="p">,</span> <span class="p">[]))</span>
+    <span class="k">return</span> <span class="n">COMM_COLLECTOR_REGISTRY</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">cfg</span><span class="o">.</span><span class="n">type</span><span class="p">,</span> <span class="n">cfg</span><span class="o">=</span><span class="n">cfg</span><span class="p">)</span>
+</pre></div>
+
+           </div>
+           
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2021, OpenDILab Contributors
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.Navigation.enable(true);
+      });
+  </script>
+
+  
+  
+    
+   
+
+</body>
+</html>
\ No newline at end of file
--- a/_modules/ding/worker/collector/comm/flask_fs_collector.html
+++ b/_modules/ding/worker/collector/comm/flask_fs_collector.html
--- a/_modules/ding/worker/collector/comm/utils.html
+++ b/_modules/ding/worker/collector/comm/utils.html
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>ding.worker.collector.comm.utils &mdash; DI-engine 0.1.0 documentation</title>
+  
+
+  
+  
+  
+  
+
+  
+  <script type="text/javascript" src="../../../../../_static/js/modernizr.min.js"></script>
+  
+    
+      <script type="text/javascript" id="documentation_options" data-url_root="../../../../../" src="../../../../../_static/documentation_options.js"></script>
+        <script data-url_root="../../../../../" id="documentation_options" src="../../../../../_static/documentation_options.js"></script>
+        <script src="../../../../../_static/jquery.js"></script>
+        <script src="../../../../../_static/underscore.js"></script>
+        <script src="../../../../../_static/doctools.js"></script>
+    
+    <script type="text/javascript" src="../../../../../_static/js/theme.js"></script>
+
+    
+
+  
+  <link rel="stylesheet" href="../../../../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../../_static/css/theme.css" type="text/css" />
+    <link rel="index" title="Index" href="../../../../../genindex.html" />
+    <link rel="search" title="Search" href="../../../../../search.html" />
+    <link href="../../../../../_static/css/style.css" rel="stylesheet" type="text/css">
+
+</head>
+
+<body class="wy-body-for-nav">
+
+   
+  <div class="wy-grid-for-nav">
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search" >
+          
+
+          
+            <a href="../../../../../index.html" class="icon icon-home"> DI-engine
+          
+
+          
+          </a>
+
+          
+            
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../../../../../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+              
+            
+            
+              <p class="caption" role="heading"><span class="caption-text">User Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../installation/index.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../quick_start/index.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../key_concept/index.html">Key Concept</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../intro_rl/index.html">Introduction to RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../hands_on/index.html">Hands on RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../best_practice/index.html">Best Practice</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../api_doc/index.html">API Doc</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../faq/index_en.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../feature/index.html">Feature</a></li>
+</ul>
+<p class="caption" role="heading"><span class="caption-text">Developer Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../guide/index.html">Developer Guide</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../tutorial_dev/index.html">Tutorial-Developer</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../../architecture/index.html">Architecture Design</a></li>
+</ul>
+
+            
+          
+        </div>
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" aria-label="top navigation">
+        
+          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+          <a href="../../../../../index.html">DI-engine</a>
+        
+      </nav>
+
+
+      <div class="wy-nav-content">
+        
+        <div class="rst-content">
+        
+          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    
+      <li><a href="../../../../../index.html">Docs</a> &raquo;</li>
+        
+          <li><a href="../../../../index.html">Module code</a> &raquo;</li>
+        
+      <li>ding.worker.collector.comm.utils</li>
+    
+    
+      <li class="wy-breadcrumbs-aside">
+        
+      </li>
+    
+  </ul>
+
+  
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <h1>Source code for ding.worker.collector.comm.utils</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">torch</span>
+<span class="kn">from</span> <span class="nn">ding.interaction.slave</span> <span class="kn">import</span> <span class="n">Slave</span><span class="p">,</span> <span class="n">TaskFail</span>
+
+
+<div class="viewcode-block" id="NaiveCollector"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.utils.NaiveCollector">[docs]</a><span class="k">class</span> <span class="nc">NaiveCollector</span><span class="p">(</span><span class="n">Slave</span><span class="p">):</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Overview:</span>
+<span class="sd">        A slave, whose master is coordinator.</span>
+<span class="sd">        Used to pass message between comm collector and coordinator.</span>
+<span class="sd">    Interfaces:</span>
+<span class="sd">        _process_task, _get_timestep</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+
+    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_prefix</span> <span class="o">=</span> <span class="n">prefix</span>
+
+<div class="viewcode-block" id="NaiveCollector._process_task"><a class="viewcode-back" href="../../../../../api_doc/worker/collector/comm.html#ding.worker.collector.comm.utils.NaiveCollector._process_task">[docs]</a>    <span class="k">def</span> <span class="nf">_process_task</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task</span><span class="p">):</span>
+        <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Process a task according to input task info dict, which is passed in by master coordinator.</span>
+<span class="sd">            For each type of task, you can refer to corresponding callback function in comm collector for details.</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - cfg (:obj:`EasyDict`): Task dict. Must contain key &quot;name&quot;.</span>
+<span class="sd">        Returns:</span>
+<span class="sd">            - result (:obj:`Union[dict, TaskFail]`): Task result dict, or task fail exception.</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="n">task_name</span> <span class="o">=</span> <span class="n">task</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span>
+        <span class="k">if</span> <span class="n">task_name</span> <span class="o">==</span> <span class="s1">&#39;resource&#39;</span><span class="p">:</span>
+            <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;cpu&#39;</span><span class="p">:</span> <span class="s1">&#39;20&#39;</span><span class="p">,</span> <span class="s1">&#39;gpu&#39;</span><span class="p">:</span> <span class="s1">&#39;1&#39;</span><span class="p">}</span>
+        <span class="k">elif</span> <span class="n">task_name</span> <span class="o">==</span> <span class="s1">&#39;collector_start_task&#39;</span><span class="p">:</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">task_info</span> <span class="o">=</span> <span class="n">task</span><span class="p">[</span><span class="s1">&#39;task_info&#39;</span><span class="p">]</span>
+            <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;message&#39;</span><span class="p">:</span> <span class="s1">&#39;collector task has started&#39;</span><span class="p">}</span>
+        <span class="k">elif</span> <span class="n">task_name</span> <span class="o">==</span> <span class="s1">&#39;collector_data_task&#39;</span><span class="p">:</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
+            <span class="n">data_id</span> <span class="o">=</span> <span class="s1">&#39;./</span><span class="si">{}</span><span class="s1">_</span><span class="si">{}</span><span class="s1">_</span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_prefix</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_info</span><span class="p">[</span><span class="s1">&#39;task_id&#39;</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">count</span><span class="p">)</span>
+            <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_get_timestep</span><span class="p">(),</span> <span class="n">data_id</span><span class="p">)</span>
+            <span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;data_id&#39;</span><span class="p">:</span> <span class="n">data_id</span><span class="p">,</span> <span class="s1">&#39;buffer_id&#39;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_info</span><span class="p">[</span><span class="s1">&#39;buffer_id&#39;</span><span class="p">],</span> <span class="s1">&#39;unroll_split_begin&#39;</span><span class="p">:</span> <span class="mi">0</span><span class="p">}</span>
+            <span class="n">data</span><span class="p">[</span><span class="s1">&#39;task_id&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_info</span><span class="p">[</span><span class="s1">&#39;task_id&#39;</span><span class="p">]</span>
+            <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">20</span><span class="p">:</span>
+                <span class="k">return</span> <span class="p">{</span>
+                    <span class="s1">&#39;task_id&#39;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_info</span><span class="p">[</span><span class="s1">&#39;task_id&#39;</span><span class="p">],</span>
+                    <span class="s1">&#39;collector_done&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
+                    <span class="s1">&#39;cur_episode&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+                    <span class="s1">&#39;cur_step&#39;</span><span class="p">:</span> <span class="mi">314</span><span class="p">,</span>
+                    <span class="s1">&#39;cur_sample&#39;</span><span class="p">:</span> <span class="mi">314</span><span class="p">,</span>
+                <span class="p">}</span>
+            <span class="k">else</span><span class="p">:</span>
+                <span class="k">return</span> <span class="n">data</span>
+        <span class="k">else</span><span class="p">:</span>
+            <span class="k">raise</span> <span class="n">TaskFail</span><span class="p">(</span>
+                <span class="n">result</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;message&#39;</span><span class="p">:</span> <span class="s1">&#39;task name error&#39;</span><span class="p">},</span> <span class="n">message</span><span class="o">=</span><span class="s1">&#39;illegal collector task &lt;</span><span class="si">{}</span><span class="s1">&gt;&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task_name</span><span class="p">)</span>
+            <span class="p">)</span></div>
+
+    <span class="k">def</span> <span class="nf">_get_timestep</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+        <span class="k">return</span> <span class="p">[</span>
+            <span class="p">{</span>
+                <span class="s1">&#39;obs&#39;</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">4</span><span class="p">),</span>
+                <span class="s1">&#39;next_obs&#39;</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">4</span><span class="p">),</span>
+                <span class="s1">&#39;reward&#39;</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="p">))</span><span class="o">.</span><span class="n">float</span><span class="p">(),</span>
+                <span class="s1">&#39;action&#39;</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">)),</span>
+                <span class="s1">&#39;done&#39;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+            <span class="p">}</span>
+        <span class="p">]</span></div>
+</pre></div>
+
+           </div>
+           
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2021, OpenDILab Contributors
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.Navigation.enable(true);
+      });
+  </script>
+
+  
+  
+    
+   
+
+</body>
+</html>
\ No newline at end of file
--- a/_modules/ding/worker/coordinator/base_parallel_commander.html
+++ b/_modules/ding/worker/coordinator/base_parallel_commander.html
--- a/_modules/ding/worker/coordinator/base_serial_commander.html
+++ b/_modules/ding/worker/coordinator/base_serial_commander.html
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>ding.worker.coordinator.base_serial_commander &mdash; DI-engine 0.1.0 documentation</title>
+  
+
+  
+  
+  
+  
+
+  
+  <script type="text/javascript" src="../../../../_static/js/modernizr.min.js"></script>
+  
+    
+      <script type="text/javascript" id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
+        <script data-url_root="../../../../" id="documentation_options" src="../../../../_static/documentation_options.js"></script>
+        <script src="../../../../_static/jquery.js"></script>
+        <script src="../../../../_static/underscore.js"></script>
+        <script src="../../../../_static/doctools.js"></script>
+    
+    <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+    
+
+  
+  <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+    <link rel="index" title="Index" href="../../../../genindex.html" />
+    <link rel="search" title="Search" href="../../../../search.html" />
+    <link href="../../../../_static/css/style.css" rel="stylesheet" type="text/css">
+
+</head>
+
+<body class="wy-body-for-nav">
+
+   
+  <div class="wy-grid-for-nav">
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search" >
+          
+
+          
+            <a href="../../../../index.html" class="icon icon-home"> DI-engine
+          
+
+          
+          </a>
+
+          
+            
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+              
+            
+            
+              <p class="caption" role="heading"><span class="caption-text">User Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation/index.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../quick_start/index.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../key_concept/index.html">Key Concept</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../intro_rl/index.html">Introduction to RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../hands_on/index.html">Hands on RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../best_practice/index.html">Best Practice</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api_doc/index.html">API Doc</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq/index_en.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../feature/index.html">Feature</a></li>
+</ul>
+<p class="caption" role="heading"><span class="caption-text">Developer Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../guide/index.html">Developer Guide</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial_dev/index.html">Tutorial-Developer</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../architecture/index.html">Architecture Design</a></li>
+</ul>
+
+            
+          
+        </div>
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" aria-label="top navigation">
+        
+          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+          <a href="../../../../index.html">DI-engine</a>
+        
+      </nav>
+
+
+      <div class="wy-nav-content">
+        
+        <div class="rst-content">
+        
+          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    
+      <li><a href="../../../../index.html">Docs</a> &raquo;</li>
+        
+          <li><a href="../../../index.html">Module code</a> &raquo;</li>
+        
+      <li>ding.worker.coordinator.base_serial_commander</li>
+    
+    
+      <li class="wy-breadcrumbs-aside">
+        
+      </li>
+    
+  </ul>
+
+  
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <h1>Source code for ding.worker.coordinator.base_serial_commander</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span>
+<span class="kn">from</span> <span class="nn">easydict</span> <span class="kn">import</span> <span class="n">EasyDict</span>
+<span class="kn">import</span> <span class="nn">copy</span>
+
+
+<div class="viewcode-block" id="BaseSerialCommander"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/commander.html#ding.worker.coordinator.base_serial_commander.BaseSerialCommander">[docs]</a><span class="k">class</span> <span class="nc">BaseSerialCommander</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+    <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Overview:</span>
+<span class="sd">        Base serial commander class.</span>
+<span class="sd">    Interface:</span>
+<span class="sd">        __init__, step</span>
+<span class="sd">    Property:</span>
+<span class="sd">        policy</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+
+    <span class="nd">@classmethod</span>
+    <span class="k">def</span> <span class="nf">default_config</span><span class="p">(</span><span class="bp">cls</span><span class="p">:</span> <span class="nb">type</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">EasyDict</span><span class="p">:</span>
+        <span class="n">cfg</span> <span class="o">=</span> <span class="n">EasyDict</span><span class="p">(</span><span class="n">copy</span><span class="o">.</span><span class="n">deepcopy</span><span class="p">(</span><span class="bp">cls</span><span class="o">.</span><span class="n">config</span><span class="p">))</span>
+        <span class="n">cfg</span><span class="o">.</span><span class="n">cfg_type</span> <span class="o">=</span> <span class="bp">cls</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">+</span> <span class="s1">&#39;Dict&#39;</span>
+        <span class="k">return</span> <span class="n">cfg</span>
+
+    <span class="n">config</span> <span class="o">=</span> <span class="p">{}</span>
+
+<div class="viewcode-block" id="BaseSerialCommander.__init__"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/commander.html#ding.worker.coordinator.base_serial_commander.BaseSerialCommander.__init__">[docs]</a>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
+            <span class="bp">self</span><span class="p">,</span>
+            <span class="n">cfg</span><span class="p">:</span> <span class="nb">dict</span><span class="p">,</span>
+            <span class="n">learner</span><span class="p">:</span> <span class="s1">&#39;BaseLearner&#39;</span><span class="p">,</span>  <span class="c1"># noqa</span>
+            <span class="n">collector</span><span class="p">:</span> <span class="s1">&#39;BaseSerialCollector&#39;</span><span class="p">,</span>  <span class="c1"># noqa</span>
+            <span class="n">evaluator</span><span class="p">:</span> <span class="s1">&#39;InteractionSerialEvaluator&#39;</span><span class="p">,</span>  <span class="c1"># noqa</span>
+            <span class="n">replay_buffer</span><span class="p">:</span> <span class="s1">&#39;IBuffer&#39;</span><span class="p">,</span>  <span class="c1"># noqa</span>
+            <span class="n">policy</span><span class="p">:</span> <span class="n">namedtuple</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
+    <span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Init the BaseSerialCommander</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - cfg (:obj:`dict`): the config of commander</span>
+<span class="sd">            - learner (:obj:`BaseLearner`): the learner</span>
+<span class="sd">            - collector (:obj:`BaseSerialCollector`): the collector</span>
+<span class="sd">            - evaluator (:obj:`InteractionSerialEvaluator`): the evaluator</span>
+<span class="sd">            - replay_buffer (:obj:`IBuffer`): the buffer</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_cfg</span> <span class="o">=</span> <span class="n">cfg</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_learner</span> <span class="o">=</span> <span class="n">learner</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_collector</span> <span class="o">=</span> <span class="n">collector</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_evaluator</span> <span class="o">=</span> <span class="n">evaluator</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_replay_buffer</span> <span class="o">=</span> <span class="n">replay_buffer</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_info</span> <span class="o">=</span> <span class="p">{}</span>
+        <span class="k">if</span> <span class="n">policy</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">policy</span> <span class="o">=</span> <span class="n">policy</span></div>
+
+<div class="viewcode-block" id="BaseSerialCommander.step"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/commander.html#ding.worker.coordinator.base_serial_commander.BaseSerialCommander.step">[docs]</a>    <span class="k">def</span> <span class="nf">step</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            Step the commander</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="c1"># Update info</span>
+        <span class="n">learn_info</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_learner</span><span class="o">.</span><span class="n">learn_info</span>
+        <span class="n">collector_info</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;envstep&#39;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_collector</span><span class="o">.</span><span class="n">envstep</span><span class="p">}</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_info</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">learn_info</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_info</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">collector_info</span><span class="p">)</span>
+        <span class="c1"># update kwargs</span>
+        <span class="n">collect_kwargs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_policy</span><span class="o">.</span><span class="n">get_setting_collect</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_info</span><span class="p">)</span>
+        <span class="k">return</span> <span class="n">collect_kwargs</span></div>
+
+    <span class="nd">@property</span>
+    <span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="s1">&#39;Policy&#39;</span><span class="p">:</span>  <span class="c1"># noqa</span>
+        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_policy</span>
+
+    <span class="nd">@policy</span><span class="o">.</span><span class="n">setter</span>
+    <span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_policy</span><span class="p">:</span> <span class="s1">&#39;Policy&#39;</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>  <span class="c1"># noqa</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_policy</span> <span class="o">=</span> <span class="n">_policy</span></div>
+</pre></div>
+
+           </div>
+           
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2021, OpenDILab Contributors
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.Navigation.enable(true);
+      });
+  </script>
+
+  
+  
+    
+   
+
+</body>
+</html>
\ No newline at end of file
--- a/_modules/ding/worker/coordinator/comm_coordinator.html
+++ b/_modules/ding/worker/coordinator/comm_coordinator.html
--- a/_modules/ding/worker/coordinator/coordinator.html
+++ b/_modules/ding/worker/coordinator/coordinator.html
--- a/_modules/ding/worker/coordinator/resource_manager.html
+++ b/_modules/ding/worker/coordinator/resource_manager.html
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>ding.worker.coordinator.resource_manager &mdash; DI-engine 0.1.0 documentation</title>
+  
+
+  
+  
+  
+  
+
+  
+  <script type="text/javascript" src="../../../../_static/js/modernizr.min.js"></script>
+  
+    
+      <script type="text/javascript" id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
+        <script data-url_root="../../../../" id="documentation_options" src="../../../../_static/documentation_options.js"></script>
+        <script src="../../../../_static/jquery.js"></script>
+        <script src="../../../../_static/underscore.js"></script>
+        <script src="../../../../_static/doctools.js"></script>
+    
+    <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+    
+
+  
+  <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+    <link rel="index" title="Index" href="../../../../genindex.html" />
+    <link rel="search" title="Search" href="../../../../search.html" />
+    <link href="../../../../_static/css/style.css" rel="stylesheet" type="text/css">
+
+</head>
+
+<body class="wy-body-for-nav">
+
+   
+  <div class="wy-grid-for-nav">
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search" >
+          
+
+          
+            <a href="../../../../index.html" class="icon icon-home"> DI-engine
+          
+
+          
+          </a>
+
+          
+            
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+              
+            
+            
+              <p class="caption" role="heading"><span class="caption-text">User Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation/index.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../quick_start/index.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../key_concept/index.html">Key Concept</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../intro_rl/index.html">Introduction to RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../hands_on/index.html">Hands on RL</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../best_practice/index.html">Best Practice</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api_doc/index.html">API Doc</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq/index_en.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../feature/index.html">Feature</a></li>
+</ul>
+<p class="caption" role="heading"><span class="caption-text">Developer Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../guide/index.html">Developer Guide</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial_dev/index.html">Tutorial-Developer</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../architecture/index.html">Architecture Design</a></li>
+</ul>
+
+            
+          
+        </div>
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" aria-label="top navigation">
+        
+          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+          <a href="../../../../index.html">DI-engine</a>
+        
+      </nav>
+
+
+      <div class="wy-nav-content">
+        
+        <div class="rst-content">
+        
+          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    
+      <li><a href="../../../../index.html">Docs</a> &raquo;</li>
+        
+          <li><a href="../../../index.html">Module code</a> &raquo;</li>
+        
+      <li>ding.worker.coordinator.resource_manager</li>
+    
+    
+      <li class="wy-breadcrumbs-aside">
+        
+      </li>
+    
+  </ul>
+
+  
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <h1>Source code for ding.worker.coordinator.resource_manager</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">random</span>
+
+
+<div class="viewcode-block" id="NaiveResourceManager"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/resource_manager.html#ding.worker.coordinator.resource_manager.NaiveResourceManager">[docs]</a><span class="k">class</span> <span class="nc">NaiveResourceManager</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+    <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Overview:</span>
+<span class="sd">        the naive resource manager</span>
+<span class="sd">    Interface:</span>
+<span class="sd">        __init__, assign_collector, assign_learner, update</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+
+<div class="viewcode-block" id="NaiveResourceManager.__init__"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/resource_manager.html#ding.worker.coordinator.resource_manager.NaiveResourceManager.__init__">[docs]</a>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            init the resouce manager</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_worker_type</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;collector&#39;</span><span class="p">,</span> <span class="s1">&#39;learner&#39;</span><span class="p">]</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="p">{}</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_worker_type</span><span class="p">}</span></div>
+
+<div class="viewcode-block" id="NaiveResourceManager.assign_collector"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/resource_manager.html#ding.worker.coordinator.resource_manager.NaiveResourceManager.assign_collector">[docs]</a>    <span class="k">def</span> <span class="nf">assign_collector</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">collector_task</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            assign the collector_task randomly and return the resouce info</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - collector_task (:obj:`dict`): the collector task to assign</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="n">available_collector_list</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;collector&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
+        <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">available_collector_list</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
+            <span class="n">selected_collector</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">sample</span><span class="p">(</span><span class="n">available_collector_list</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+            <span class="n">info</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;collector&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="n">selected_collector</span><span class="p">)</span>
+            <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;collector_id&#39;</span><span class="p">:</span> <span class="n">selected_collector</span><span class="p">,</span> <span class="s1">&#39;resource_info&#39;</span><span class="p">:</span> <span class="n">info</span><span class="p">}</span>
+        <span class="k">else</span><span class="p">:</span>
+            <span class="k">return</span> <span class="kc">None</span></div>
+
+<div class="viewcode-block" id="NaiveResourceManager.assign_learner"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/resource_manager.html#ding.worker.coordinator.resource_manager.NaiveResourceManager.assign_learner">[docs]</a>    <span class="k">def</span> <span class="nf">assign_learner</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learner_task</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            assign the learner_task randomly and return the resouce info</span>
+<span class="sd">        Arguments:</span>
+<span class="sd">            - learner_task (:obj:`dict`): the learner task to assign</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="n">available_learner_list</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;learner&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
+        <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">available_learner_list</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
+            <span class="n">selected_learner</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">sample</span><span class="p">(</span><span class="n">available_learner_list</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+            <span class="n">info</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;learner&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="n">selected_learner</span><span class="p">)</span>
+            <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;learner_id&#39;</span><span class="p">:</span> <span class="n">selected_learner</span><span class="p">,</span> <span class="s1">&#39;resource_info&#39;</span><span class="p">:</span> <span class="n">info</span><span class="p">}</span>
+        <span class="k">else</span><span class="p">:</span>
+            <span class="k">return</span> <span class="kc">None</span></div>
+
+    <span class="k">def</span> <span class="nf">have_assigned</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">id</span><span class="p">,</span> <span class="n">worker_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
+        <span class="k">assert</span> <span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_worker_type</span><span class="p">,</span> <span class="s2">&quot;invalid worker_type: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
+        <span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="s1">&#39;collector&#39;</span><span class="p">:</span>
+            <span class="k">return</span> <span class="n">worker_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;collector&#39;</span><span class="p">]</span>
+        <span class="k">elif</span> <span class="n">name</span> <span class="o">==</span> <span class="s1">&#39;learner&#39;</span><span class="p">:</span>
+            <span class="k">return</span> <span class="n">worker_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="s1">&#39;learner&#39;</span><span class="p">]</span>
+
+    <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">id</span><span class="p">,</span> <span class="n">worker_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
+        <span class="k">assert</span> <span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_worker_type</span><span class="p">,</span> <span class="s2">&quot;invalid worker_type: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
+        <span class="k">if</span> <span class="n">worker_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="n">name</span><span class="p">]:</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="n">worker_id</span><span class="p">)</span>
+            <span class="k">return</span> <span class="kc">True</span>
+        <span class="k">else</span><span class="p">:</span>
+            <span class="k">return</span> <span class="kc">False</span>
+
+<div class="viewcode-block" id="NaiveResourceManager.update"><a class="viewcode-back" href="../../../../api_doc/worker/coordinator/resource_manager.html#ding.worker.coordinator.resource_manager.NaiveResourceManager.update">[docs]</a>    <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">worker_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">resource_info</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
+        <span class="sa">r</span><span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">        Overview:</span>
+<span class="sd">            update the reource info</span>
+<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="k">assert</span> <span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_worker_type</span><span class="p">,</span> <span class="s2">&quot;invalid worker_type: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_resource_info</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="n">worker_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">resource_info</span></div></div>
+</pre></div>
+
+           </div>
+           
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2021, OpenDILab Contributors
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.Navigation.enable(true);
+      });
+  </script>
+
+  
+  
+    
+   
+
+</body>
+</html>
\ No newline at end of file
--- a/_modules/ding/worker/coordinator/solo_parallel_commander.html
+++ b/_modules/ding/worker/coordinator/solo_parallel_commander.html
--- a/_modules/ding/worker/learner/base_learner.html
+++ b/_modules/ding/worker/learner/base_learner.html
--- a/_modules/ding/worker/learner/comm/base_comm_learner.html
+++ b/_modules/ding/worker/learner/comm/base_comm_learner.html
--- a/_modules/ding/worker/learner/comm/flask_fs_learner.html
+++ b/_modules/ding/worker/learner/comm/flask_fs_learner.html
--- a/_modules/ding/worker/learner/learner_hook.html
+++ b/_modules/ding/worker/learner/learner_hook.html
--- a/_modules/ding/worker/replay_buffer/advanced_buffer.html
+++ b/_modules/ding/worker/replay_buffer/advanced_buffer.html
--- a/_modules/ding/worker/replay_buffer/base_buffer.html
+++ b/_modules/ding/worker/replay_buffer/base_buffer.html
--- a/_modules/ding/worker/replay_buffer/episode_buffer.html
+++ b/_modules/ding/worker/replay_buffer/episode_buffer.html
--- a/_modules/ding/worker/replay_buffer/naive_buffer.html
+++ b/_modules/ding/worker/replay_buffer/naive_buffer.html
--- a/_modules/ding/worker/replay_buffer/utils.html
+++ b/_modules/ding/worker/replay_buffer/utils.html
--- a/_modules/index.html
+++ b/_modules/index.html
--- a/api_doc/config/config.html
+++ b/api_doc/config/config.html
--- a/api_doc/model/common/head.html
+++ b/api_doc/model/common/head.html
@@ -845,8 +845,8 @@ Parameter updates with ReparameterizationHead’s MLPs forward setup.</p>
 <dt>outputs (<code class="xref py py-obj docutils literal notranslate"><span class="pre">Dict</span></code>):</dt><dd><p>Run <code class="docutils literal notranslate"><span class="pre">MLP</span></code> with <code class="docutils literal notranslate"><span class="pre">ReparameterizationHead</span></code> setups and return the result prediction dictionary.</p>
 <dl class="simple">
 <dt>Necessary Keys:</dt><dd><ul class="simple">
-<li><p>mu (<code class="xref py py-obj docutils literal notranslate"><span class="pre">torch.Tensor</span></code>) Tensor of cells of updated mu values of size <code class="docutils literal notranslate"><span class="pre">(B,</span> <span class="pre">action_size)</span></code></p></li>
-<li><p>sigma (<code class="xref py py-obj docutils literal notranslate"><span class="pre">torch.Tensor</span></code>) Tensor of cells of updated sigma values of size <code class="docutils literal notranslate"><span class="pre">(B,</span> <span class="pre">action_size)</span></code></p></li>
+<li><p>mu (<code class="xref py py-obj docutils literal notranslate"><span class="pre">torch.Tensor</span></code>) Tensor of cells of updated mu values, with same size as <code class="docutils literal notranslate"><span class="pre">x</span></code>.</p></li>
+<li><p>sigma (<code class="xref py py-obj docutils literal notranslate"><span class="pre">torch.Tensor</span></code>) Tensor of cells of updated sigma values, with same size as <code class="docutils literal notranslate"><span class="pre">x</span></code>.</p></li>
 </ul>
 </dd>
 </dl>

--- a/api_doc/worker/collector/base_serial_collector.html
+++ b/api_doc/worker/collector/base_serial_collector.html
--- a/api_doc/worker/collector/comm.html
+++ b/api_doc/worker/collector/comm.html
--- a/api_doc/worker/collector/index.html
+++ b/api_doc/worker/collector/index.html
@@ -197,7 +197,7 @@
 <li class="toctree-l1"><a class="reference internal" href="comm.html">worker.collector.comm</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="comm.html#base-comm-collector">base_comm_collector</a><ul>
 <li class="toctree-l3"><a class="reference internal" href="comm.html#basecommcollector">BaseCommCollector</a></li>
-<li class="toctree-l3"><a class="reference internal" href="comm.html#create-comm-collector">create_comm_collector</a></li>
+<li class="toctree-l3"><a class="reference internal" href="comm.html#module-ding.worker.collector.comm.base_comm_collector.create_comm_collector">create_comm_collector</a></li>
 </ul>
 </li>
 <li class="toctree-l2"><a class="reference internal" href="comm.html#flask-fs-collector">flask_fs_collector</a><ul>

--- a/api_doc/worker/coordinator/comm_coordinator.html
+++ b/api_doc/worker/coordinator/comm_coordinator.html
--- a/api_doc/worker/coordinator/commander.html
+++ b/api_doc/worker/coordinator/commander.html
--- a/api_doc/worker/coordinator/coordinator.html
+++ b/api_doc/worker/coordinator/coordinator.html
--- a/api_doc/worker/coordinator/resource_manager.html
+++ b/api_doc/worker/coordinator/resource_manager.html
--- a/api_doc/worker/learner/comm_learner.html
+++ b/api_doc/worker/learner/comm_learner.html
--- a/api_doc/worker/learner/index.html
+++ b/api_doc/worker/learner/index.html
--- a/api_doc/worker/learner/learner.html
+++ b/api_doc/worker/learner/learner.html
--- a/api_doc/worker/replay_buffer/buffer.html
+++ b/api_doc/worker/replay_buffer/buffer.html
--- a/genindex.html
+++ b/genindex.html
--- a/hands_on/acer.html
+++ b/hands_on/acer.html
--- a/hands_on/sqil.html
+++ b/hands_on/sqil.html
--- a/objects.inv
+++ b/objects.inv
--- a/py-modindex.html
+++ b/py-modindex.html
--- a/quick_start/index.html
+++ b/quick_start/index.html
@@ -423,7 +423,7 @@ DI-engine saves and loads checkpoints in the same way as PyTorch.</p>
 </pre></div>
 </div>
 <p>To deploy this in a more elegant way, DI-engine is configured to use
-<code class="xref py py-class docutils literal notranslate"><span class="pre">Learner</span> <span class="pre">Hook</span></code> to handle these cases. The saving hook is
+<a class="reference internal" href="../api_doc/worker/learner/learner.html#ding.worker.learner.learner_hook.LearnerHook" title="ding.worker.learner.learner_hook.LearnerHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">Learner</span> <span class="pre">Hook</span></code></a> to handle these cases. The saving hook is
 automatically called after training iterations. And to load &amp; save checkpoints at the beginning and
 in the end, users can simply add one-line code before &amp; after training as follows.</p>
 <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">learner</span><span class="o">.</span><span class="n">call_hook</span><span class="p">(</span><span class="s1">&#39;before_run&#39;</span><span class="p">)</span>

--- a/searchindex.js
+++ b/searchindex.js