• Z
    Add pure fp16 training with master weights. (#27712) · be3777a5
    Zhen Wang 提交于
    * add the weight decay func for the momentum op
    
    * Add the multi_precision function in Momentum Optimizer.
    
    * Make sure that the initial value of master weights are same with the fp16 weights.
    
    * add static loss scaling.
    
    * add the rescale_grad function in the pure fp16 training.
    
    * use the original momentum updating method.
    
    * Polish some codes, such as variable names.
    
    * add docstring for apis.
    
    * update the var creation details of _create_master_weight.
    
    * not modify codes about imperative momentum updating.
    
    * Fix the error of test_dist_sparse_tensor_load_momentum UT.
    
    * add unit test for multi precision fp16 training.
    
    * add more unit tests for CI.
    
    * Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT.
    
    * For CI Coverage Checking.
    be3777a5
fp16_utils.py 17.9 KB