BF16 optimizer for BF16+ZeRO Stage 1 (#2706)
* BF16 optimizer only with ZeRO stage 1.
* Updating to grad accum of fp32 for BF16 ZeRO1 case.
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Showing
想要评论请 注册 或 登录
* BF16 optimizer only with ZeRO stage 1.
* Updating to grad accum of fp32 for BF16 ZeRO1 case.
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>