Implement normalization methods(BatchNorm/LayerNorm/BatchRenorm) as functions in a common header file
Created by: zhouxiao-coder
Current Status
In the new Paddle core, we currently have a batch_norm_op which essentially is an implementation of fused spatial batch normalization method. Like most other operators, most computations are directly included in BatchNorm kernels.
Suggestion
I suggest we abstract away some normalization calculations and implement them as functions/functors in a common header file like normalization.h to better reuse existing code.
Reasons
- Normalization methods are evolving rapidly, new variants of BatchNorm come out from time to time, e.g. layer normalization and batch renormalization. They often share similar structures.
- Normalization layers are often used in deep architectures repeatedly, so it should be efficient. We may need to write a lot of fused kernels to support this.
- RNN units also benefit from using some normalization methods, we should support it in c++ code.