Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
兽拳
Paddle
提交
f8bc4ecb
P
Paddle
项目概览
兽拳
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
1
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
1
Issue
1
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
f8bc4ecb
编写于
11月 06, 2017
作者:
D
dangqingqing
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Fix the doc for momentum and adam optimizer.
上级
c8122e09
变更
2
显示空白变更内容
内联
并排
Showing
2 changed file
with
24 addition
and
24 deletion
+24
-24
python/paddle/trainer_config_helpers/optimizers.py
python/paddle/trainer_config_helpers/optimizers.py
+1
-1
python/paddle/v2/optimizer.py
python/paddle/v2/optimizer.py
+23
-23
未找到文件。
python/paddle/trainer_config_helpers/optimizers.py
浏览文件 @
f8bc4ecb
...
...
@@ -116,7 +116,7 @@ class AdamOptimizer(BaseSGDOptimizer):
m(w, t) & =
\\
beta_1 m(w, t-1) + (1 -
\\
beta_1)
\\
nabla Q_i(w)
\\\\
v(w, t) & =
\\
beta_2 v(w, t-1) + (1 -
\\
beta_2)(
\\
nabla Q_i(w)) ^2
\\\\
w & = w -
\\
frac{
\\
eta}{
\\
sqrt{v(w,t) +
\\
epsilon}}
w & = w -
\\
frac{
\\
eta
m(w, t)
}{
\\
sqrt{v(w,t) +
\\
epsilon}}
:param beta1: the :math:`
\\
beta_1` in equation.
:type beta1: float
...
...
python/paddle/v2/optimizer.py
浏览文件 @
f8bc4ecb
...
...
@@ -11,11 +11,6 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Optimizers(update equation) for SGD method.
TODO(yuyang18): Complete comments.
"""
import
paddle.trainer_config_helpers.config_parser_utils
as
config_parser_utils
import
paddle.trainer_config_helpers.optimizers
as
v1_optimizers
...
...
@@ -101,32 +96,37 @@ class Optimizer(object):
class
Momentum
(
Optimizer
):
"""
SGD Optimizer.
SGD is an optimization method, trying to find a neural network that
minimize the "cost/error" of it by iteration. In paddle's implementation
SGD Optimizer is synchronized, which means all gradients will be wait to
calculate and reduced into one gradient, then do optimize operation.
Momentum Optimizer.
The neural network consider the learning problem of minimizing an objective
function, that has the form of a sum
When sparse=False, the momentum update formula is as follows:
.. math::
Q(w) =
\\
sum_{i}^{n} Q_i(w)
v_{t} &= k * v_{t-1} -
\\
gamma_t / (g_{t} +
\\
lambda w_{t-1})
\\\\
w_{t} &= w_{t-1} + v_{t}
\\\\
The value of function Q sometimes is the cost of neural network (Mean
Square Error between prediction and label for example). The function Q is
parametrised by w, the weight/bias of neural network. And weights is what to
be learned. The i is the i-th observation in (trainning) data
.
where, :math:`k` is momentum, :math:`
\\
lambda` is decay rate,
:math:`
\\
gamma_t` is learning rate at the t'th iteration.
:math:`w_{t}` is the weight as the t'th iteration.
And the :math:`v_{t}` is the history momentum variable
.
So, the SGD method will optimize the weight by
When sparse=True, the update scheme:
.. math::
w = w -
\\
eta
\\
nabla Q(w) = w -
\\
eta
\\
sum_{i}^{n}
\\
nabla Q_i(w)
\\
alpha_t &=
\\
alpha_{t-1} / k
\\\\
\\
beta_t &=
\\
beta_{t-1} / (1 +
\\
lambda
\\
gamma_t)
\\\\
u_t &= u_{t-1} -
\\
alpha_t
\\
gamma_t g_t
\\\\
v_t &= v_{t-1} +
\\
tau_{t-1}
\\
alpha_t
\\
gamma_t g_t
\\\\
\\
tau_t &=
\\
tau_{t-1} +
\\
beta_t /
\\
alpha_t
where :math:`k` is momentum, :math:`
\\
lambda` is decay rate,
:math:`
\\
gamma_t` is learning rate at the t'th iteration.
where :math:`
\\
eta` is learning rate. And :math:`n` is batch size.
:param momentum: the momentum factor.
:type momentum: float
:param sparse: with sparse support or not, False by default.
:type sparse: bool
"""
def
__init__
(
self
,
momentum
=
None
,
sparse
=
False
,
**
kwargs
):
...
...
@@ -146,7 +146,7 @@ class Adam(Optimizer):
m(w, t) & =
\\
beta_1 m(w, t-1) + (1 -
\\
beta_1)
\\
nabla Q_i(w)
\\\\
v(w, t) & =
\\
beta_2 v(w, t-1) + (1 -
\\
beta_2)(
\\
nabla Q_i(w)) ^2
\\\\
w & = w -
\\
frac{
\\
eta}{
\\
sqrt{v(w,t) +
\\
epsilon}}
w & = w -
\\
frac{
\\
eta
m(w, t)
}{
\\
sqrt{v(w,t) +
\\
epsilon}}
:param beta1: the :math:`
\\
beta_1` in equation.
:type beta1: float
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录