-
由 Rana Ali Amjad 提交于
* Changes for bfloat16 Zero2 * Cleaned up additional comments and debugging code * Adapted fp16_master_weights_and_grads option to cover BF16 * Reverted fp16_master_weights_and_gradients extension to BFloat16 and minor cleanup * Fixed formatting and variable naming errors recognized in testing * Added relevant unit tests for bfloat16 with ZeRO-2 * Updates conditions for skipping BFloat16 unit tests * Added check for NCCL inconsistent version naming convention * Update skip message for Bfloat16 tests to mention additional checks Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
648f7bfa