• Z
    [Cherry-pick] Add pure fp16 training with master weights. (#29301) · d8ea8a06
    Zhen Wang 提交于
    * Add pure fp16 training with master weights. (#27712)
    
    * add the weight decay func for the momentum op
    
    * Add the multi_precision function in Momentum Optimizer.
    
    * Make sure that the initial value of master weights are same with the fp16 weights.
    
    * add static loss scaling.
    
    * add the rescale_grad function in the pure fp16 training.
    
    * use the original momentum updating method.
    
    * Polish some codes, such as variable names.
    
    * add docstring for apis.
    
    * update the var creation details of _create_master_weight.
    
    * not modify codes about imperative momentum updating.
    
    * Fix the error of test_dist_sparse_tensor_load_momentum UT.
    
    * add unit test for multi precision fp16 training.
    
    * add more unit tests for CI.
    
    * Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT.
    d8ea8a06
fp16_utils.py 17.8 KB