• R
    Bfloat16 zero2 (#1398) · 648f7bfa
    Rana Ali Amjad 提交于
    * Changes for bfloat16 Zero2
    
    * Cleaned up additional comments and debugging code
    
    * Adapted fp16_master_weights_and_grads option to cover BF16
    
    * Reverted fp16_master_weights_and_gradients extension to BFloat16 and minor cleanup
    
    * Fixed formatting and variable naming errors recognized in testing
    
    * Added relevant unit tests for bfloat16 with ZeRO-2
    
    * Updates conditions for skipping BFloat16 unit tests
    
    * Added check for NCCL inconsistent version naming convention
    
    * Update skip message for Bfloat16 tests to mention additional checks
    Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
    648f7bfa
config.py 41.5 KB