Created by: zhangting2020

PR types

Function optimization

PR changes

OPs

Describe

This Op performs batch norm on input x, and adds the result to input y. Then it performs activation on the sum. We use cuDNN API to implements this function, the following points need to be noted:

• cudnnBatchNormalizationForwardTrainingEx requires inputs x and z must be float16.

• The data format of inputs must be NHWC [batch, in_height, in_width, in_channels].

This Op will be used in automatic mixed precision training of the resnet model. The following image is part of the model. The red parts represent the inputs of this Op. The green parts represent the computation performed by the Op.

Performance of ResNet50 AMP Training

Test on V100, CUDA 10.1, cuDNN 7.6, single card, BS=128

• before：1015.18 imgs/s
• after：1085.98 imgs/s，+6.9%