提交 e41d6701 编写于 作者: T Travis CI

Deploy to GitHub Pages: e57a40b8

上级 8b924d10
...@@ -106,9 +106,11 @@ See function `_addup_repetitive_outputs_` in `backward.py` for implementation de ...@@ -106,9 +106,11 @@ See function `_addup_repetitive_outputs_` in `backward.py` for implementation de
In our framework, variables can be marked as *no_gradient*, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some `grad_op` are marked as *no_gradient*, the `grad_op` itself can be skipped in backward pass. In our framework, variables can be marked as *no_gradient*, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some `grad_op` are marked as *no_gradient*, the `grad_op` itself can be skipped in backward pass.
But these unnecessary gradients still need to be creating and initialized by something, otherwise following `grad_op`s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ `fill_zeros_like_op` to initialize them as all zeros. Another situation is all the gradient inputs of some `grad_op` are marked as *no_gradient*, which means all of them can be considered as zeros. For `grad_op`s are in essence the propagation of gradients, all the outputs are definitely zeros when all gradient inputs are zeros. Therefore the `grad_op` can also be skipped.
This features are implemented in function `_remove_no_grad_branch_`. It checks new created `grad_op`s one-by-one, removes whose outputs are all in `no_grad_set` or inserts `fill_zeros_like_op` when its necessary. We can get the `no_grad_set` from the `_append_backward_ops_` argument `no_grad_dict` or generate it on the fly by scanning all variables' `no_gradient` attribute(True or False). It should be noted that all these zero gradients still need to be creating and initialized by something, otherwise following `grad_op`s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ `fill_zeros_like_op` to initialize them as all zeros.
This features are implemented in function `_remove_no_grad_branch_`. It checks new created `grad_op`s one-by-one, removes who can be skipped and inserts `fill_zeros_like_op` when its necessary. We can get the `no_grad_set` from the `_append_backward_ops_` argument `no_grad_dict` or generate it on the fly by scanning all variables' `no_gradient` attribute(True or False).
### Creating Backward Variables ### Creating Backward Variables
......
...@@ -301,8 +301,9 @@ for op in reversed(block.ops): ...@@ -301,8 +301,9 @@ for op in reversed(block.ops):
<div class="section" id="no-gradient-variables"> <div class="section" id="no-gradient-variables">
<span id="no-gradient-variables"></span><h4>No Gradient Variables<a class="headerlink" href="#no-gradient-variables" title="Permalink to this headline"></a></h4> <span id="no-gradient-variables"></span><h4>No Gradient Variables<a class="headerlink" href="#no-gradient-variables" title="Permalink to this headline"></a></h4>
<p>In our framework, variables can be marked as <em>no_gradient</em>, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, the <code class="docutils literal"><span class="pre">grad_op</span></code> itself can be skipped in backward pass.</p> <p>In our framework, variables can be marked as <em>no_gradient</em>, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, the <code class="docutils literal"><span class="pre">grad_op</span></code> itself can be skipped in backward pass.</p>
<p>But these unnecessary gradients still need to be creating and initialized by something, otherwise following <code class="docutils literal"><span class="pre">grad_op</span></code>s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> to initialize them as all zeros.</p> <p>Another situation is all the gradient inputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, which means all of them can be considered as zeros. For <code class="docutils literal"><span class="pre">grad_op</span></code>s are in essence the propagation of gradients, all the outputs are definitely zeros when all gradient inputs are zeros. Therefore the <code class="docutils literal"><span class="pre">grad_op</span></code> can also be skipped.</p>
<p>This features are implemented in function <code class="docutils literal"><span class="pre">_remove_no_grad_branch_</span></code>. It checks new created <code class="docutils literal"><span class="pre">grad_op</span></code>s one-by-one, removes whose outputs are all in <code class="docutils literal"><span class="pre">no_grad_set</span></code> or inserts <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> when its necessary. We can get the <code class="docutils literal"><span class="pre">no_grad_set</span></code> from the <code class="docutils literal"><span class="pre">_append_backward_ops_</span></code> argument <code class="docutils literal"><span class="pre">no_grad_dict</span></code> or generate it on the fly by scanning all variables&#8217; <code class="docutils literal"><span class="pre">no_gradient</span></code> attribute(True or False).</p> <p>It should be noted that all these zero gradients still need to be creating and initialized by something, otherwise following <code class="docutils literal"><span class="pre">grad_op</span></code>s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> to initialize them as all zeros.</p>
<p>This features are implemented in function <code class="docutils literal"><span class="pre">_remove_no_grad_branch_</span></code>. It checks new created <code class="docutils literal"><span class="pre">grad_op</span></code>s one-by-one, removes who can be skipped and inserts <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> when its necessary. We can get the <code class="docutils literal"><span class="pre">no_grad_set</span></code> from the <code class="docutils literal"><span class="pre">_append_backward_ops_</span></code> argument <code class="docutils literal"><span class="pre">no_grad_dict</span></code> or generate it on the fly by scanning all variables&#8217; <code class="docutils literal"><span class="pre">no_gradient</span></code> attribute(True or False).</p>
</div> </div>
</div> </div>
<div class="section" id="creating-backward-variables"> <div class="section" id="creating-backward-variables">
......
因为 它太大了无法显示 source diff 。你可以改为 查看blob
...@@ -106,9 +106,11 @@ See function `_addup_repetitive_outputs_` in `backward.py` for implementation de ...@@ -106,9 +106,11 @@ See function `_addup_repetitive_outputs_` in `backward.py` for implementation de
In our framework, variables can be marked as *no_gradient*, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some `grad_op` are marked as *no_gradient*, the `grad_op` itself can be skipped in backward pass. In our framework, variables can be marked as *no_gradient*, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some `grad_op` are marked as *no_gradient*, the `grad_op` itself can be skipped in backward pass.
But these unnecessary gradients still need to be creating and initialized by something, otherwise following `grad_op`s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ `fill_zeros_like_op` to initialize them as all zeros. Another situation is all the gradient inputs of some `grad_op` are marked as *no_gradient*, which means all of them can be considered as zeros. For `grad_op`s are in essence the propagation of gradients, all the outputs are definitely zeros when all gradient inputs are zeros. Therefore the `grad_op` can also be skipped.
This features are implemented in function `_remove_no_grad_branch_`. It checks new created `grad_op`s one-by-one, removes whose outputs are all in `no_grad_set` or inserts `fill_zeros_like_op` when its necessary. We can get the `no_grad_set` from the `_append_backward_ops_` argument `no_grad_dict` or generate it on the fly by scanning all variables' `no_gradient` attribute(True or False). It should be noted that all these zero gradients still need to be creating and initialized by something, otherwise following `grad_op`s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ `fill_zeros_like_op` to initialize them as all zeros.
This features are implemented in function `_remove_no_grad_branch_`. It checks new created `grad_op`s one-by-one, removes who can be skipped and inserts `fill_zeros_like_op` when its necessary. We can get the `no_grad_set` from the `_append_backward_ops_` argument `no_grad_dict` or generate it on the fly by scanning all variables' `no_gradient` attribute(True or False).
### Creating Backward Variables ### Creating Backward Variables
......
...@@ -314,8 +314,9 @@ for op in reversed(block.ops): ...@@ -314,8 +314,9 @@ for op in reversed(block.ops):
<div class="section" id="no-gradient-variables"> <div class="section" id="no-gradient-variables">
<span id="no-gradient-variables"></span><h4>No Gradient Variables<a class="headerlink" href="#no-gradient-variables" title="永久链接至标题"></a></h4> <span id="no-gradient-variables"></span><h4>No Gradient Variables<a class="headerlink" href="#no-gradient-variables" title="永久链接至标题"></a></h4>
<p>In our framework, variables can be marked as <em>no_gradient</em>, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, the <code class="docutils literal"><span class="pre">grad_op</span></code> itself can be skipped in backward pass.</p> <p>In our framework, variables can be marked as <em>no_gradient</em>, it means that the gradient of this variable is unnecessary and can be considered as zero in model training. Apparently, when all the outputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, the <code class="docutils literal"><span class="pre">grad_op</span></code> itself can be skipped in backward pass.</p>
<p>But these unnecessary gradients still need to be creating and initialized by something, otherwise following <code class="docutils literal"><span class="pre">grad_op</span></code>s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> to initialize them as all zeros.</p> <p>Another situation is all the gradient inputs of some <code class="docutils literal"><span class="pre">grad_op</span></code> are marked as <em>no_gradient</em>, which means all of them can be considered as zeros. For <code class="docutils literal"><span class="pre">grad_op</span></code>s are in essence the propagation of gradients, all the outputs are definitely zeros when all gradient inputs are zeros. Therefore the <code class="docutils literal"><span class="pre">grad_op</span></code> can also be skipped.</p>
<p>This features are implemented in function <code class="docutils literal"><span class="pre">_remove_no_grad_branch_</span></code>. It checks new created <code class="docutils literal"><span class="pre">grad_op</span></code>s one-by-one, removes whose outputs are all in <code class="docutils literal"><span class="pre">no_grad_set</span></code> or inserts <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> when its necessary. We can get the <code class="docutils literal"><span class="pre">no_grad_set</span></code> from the <code class="docutils literal"><span class="pre">_append_backward_ops_</span></code> argument <code class="docutils literal"><span class="pre">no_grad_dict</span></code> or generate it on the fly by scanning all variables&#8217; <code class="docutils literal"><span class="pre">no_gradient</span></code> attribute(True or False).</p> <p>It should be noted that all these zero gradients still need to be creating and initialized by something, otherwise following <code class="docutils literal"><span class="pre">grad_op</span></code>s who take these gradients as inputs take the risk of using uninitialized memory. In our code, we employ <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> to initialize them as all zeros.</p>
<p>This features are implemented in function <code class="docutils literal"><span class="pre">_remove_no_grad_branch_</span></code>. It checks new created <code class="docutils literal"><span class="pre">grad_op</span></code>s one-by-one, removes who can be skipped and inserts <code class="docutils literal"><span class="pre">fill_zeros_like_op</span></code> when its necessary. We can get the <code class="docutils literal"><span class="pre">no_grad_set</span></code> from the <code class="docutils literal"><span class="pre">_append_backward_ops_</span></code> argument <code class="docutils literal"><span class="pre">no_grad_dict</span></code> or generate it on the fly by scanning all variables&#8217; <code class="docutils literal"><span class="pre">no_gradient</span></code> attribute(True or False).</p>
</div> </div>
</div> </div>
<div class="section" id="creating-backward-variables"> <div class="section" id="creating-backward-variables">
......
此差异已折叠。
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册