Feature request: RAdam, LookAhead implement and combine them in one
Created by: parap1uie-s
Is there any implementation plan for RAdam and LookAhead?
RAdam and LookAhead are two recently proposed optimization methods, with higher stability of the optimization process and enable faster convergence.
More over, there is a method about combine RAdam and LookAhead into one optimizer, implemented by fast.ai called ranger.
PaddlePaddle是否有实现RAdam及LookAhead的开发计划？
RAdam和LookAhead是两种最近才提出的优化方法，据报告这些方法提升了训练稳定性，可以更快，也更好地让模型收敛。
进一步地，有一种将RAdam和LookAhead融合到一个优化器中的方法，目前fast.ai已有相关实现，被命名为ranger。
相关资料：