Created by: LifeIsStrange

RAdam is consistently better than adam. https://arxiv.org/pdf/1908.03265v1

Lookahead merits to be tried too https://arxiv.org/pdf/1907.08610v1.pdf Maybe it can be used on top of RAdam.