Optimizer

Momentum

class paddle.v2.optimizer.Momentum(momentum=None, sparse=False, **kwargs)

Momentum Optimizer.

When sparse=False, the momentum update formula is as follows:

\[\begin{split}v_{t} &= k * v_{t-1} - \gamma_t (g_{t} + \lambda w_{t-1}) \\ w_{t} &= w_{t-1} + v_{t} \\\end{split}\]

where, \(k\) is momentum, \(\lambda\) is decay rate, \(\gamma_t\) is learning rate at the t’th iteration. \(w_{t}\) is the weight as the t’th iteration. And the \(v_{t}\) is the history momentum variable.

When sparse=True, the update scheme:

\[\begin{split}\alpha_t &= \alpha_{t-1} / k \\ \beta_t &= \beta_{t-1} / (1 + \lambda \gamma_t) \\ u_t &= u_{t-1} - \alpha_t \gamma_t g_t \\ v_t &= v_{t-1} + \tau_{t-1} \alpha_t \gamma_t g_t \\ \tau_t &= \tau_{t-1} + \beta_t / \alpha_t\end{split}\]

where \(k\) is momentum, \(\lambda\) is decay rate, \(\gamma_t\) is learning rate at the t’th iteration.

Parameters:
  • momentum (float) – the momentum factor.
  • sparse (bool) – with sparse support or not, False by default.

Adam

class paddle.v2.optimizer.Adam(beta1=0.9, beta2=0.999, epsilon=1e-08, **kwargs)

Adam optimizer. The details of please refer Adam: A Method for Stochastic Optimization

\[\begin{split}m(w, t) & = \beta_1 m(w, t-1) + (1 - \beta_1) \nabla Q_i(w) \\ v(w, t) & = \beta_2 v(w, t-1) + (1 - \beta_2)(\nabla Q_i(w)) ^2 \\ w & = w - \frac{\eta m(w, t)}{\sqrt{v(w,t) + \epsilon}}\end{split}\]
Parameters:
  • beta1 (float) – the \(\beta_1\) in equation.
  • beta2 (float) – the \(\beta_2\) in equation.
  • epsilon (float) – the \(\epsilon\) in equation. It is used to prevent divided by zero.

Adamax

class paddle.v2.optimizer.Adamax(beta1=0.9, beta2=0.999, **kwargs)

Adamax optimizer.

The details of please refer this Adam: A Method for Stochastic Optimization

\[\begin{split}m_t & = \beta_1 * m_{t-1} + (1-\beta_1)* \nabla Q_i(w) \\ u_t & = max(\beta_2*u_{t-1}, abs(\nabla Q_i(w))) \\ w_t & = w_{t-1} - (\eta/(1-\beta_1^t))*m_t/u_t\end{split}\]
Parameters:
  • beta1 (float) – the \(\beta_1\) in the equation.
  • beta2 (float) – the \(\beta_2\) in the equation.

AdaGrad

class paddle.v2.optimizer.AdaGrad(**kwargs)

Adagrad(for ADAptive GRAdient algorithm) optimizer.

For details please refer this Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.

\[\begin{split}G &= \sum_{\tau=1}^{t} g_{\tau} g_{\tau}^T \\ w & = w - \eta diag(G)^{-\frac{1}{2}} \circ g\end{split}\]

DecayedAdaGrad

class paddle.v2.optimizer.DecayedAdaGrad(rho=0.95, epsilon=1e-06, **kwargs)

AdaGrad method with decayed sum gradients. The equations of this method show as follow.

\[\begin{split}E(g_t^2) &= \rho * E(g_{t-1}^2) + (1-\rho) * g^2 \\ learning\_rate &= 1/sqrt( ( E(g_t^2) + \epsilon )\end{split}\]
Parameters:
  • rho (float) – The \(\rho\) parameter in that equation
  • epsilon (float) – The \(\epsilon\) parameter in that equation.

AdaDelta

class paddle.v2.optimizer.AdaDelta(rho=0.95, epsilon=1e-06, **kwargs)

AdaDelta method. The details of adadelta please refer to this ADADELTA: AN ADAPTIVE LEARNING RATE METHOD.

\[\begin{split}E(g_t^2) &= \rho * E(g_{t-1}^2) + (1-\rho) * g^2 \\ learning\_rate &= sqrt( ( E(dx_{t-1}^2) + \epsilon ) / ( \ E(g_t^2) + \epsilon ) ) \\ E(dx_t^2) &= \rho * E(dx_{t-1}^2) + (1-\rho) * (-g*learning\_rate)^2\end{split}\]
Parameters:
  • rho (float) – \(\rho\) in equation
  • epsilon (float) – \(\rho\) in equation

RMSProp

class paddle.v2.optimizer.RMSProp(rho=0.95, epsilon=1e-06, **kwargs)

RMSProp(for Root Mean Square Propagation) optimizer. For details please refer this slide.

The equations of this method as follows:

\[\begin{split}v(w, t) & = \rho v(w, t-1) + (1 - \rho)(\nabla Q_{i}(w))^2 \\ w & = w - \frac{\eta} {\sqrt{v(w,t) + \epsilon}} \nabla Q_{i}(w)\end{split}\]
Parameters:
  • rho (float) – the \(\rho\) in the equation. The forgetting factor.
  • epsilon (float) – the \(\epsilon\) in the equation.