Created by: chengduoZH
Dependent PR: #14208, #14202, #14201
This PR adds switch_to_fp16
to allow the user to specify some operators's computation with fp16.
For example:
import paddle.fluid as fluid
img = fluid.layers.data(name='img', shape=[1, 28, 28], dtype='float32')
label = fluid.layers.data(name='label', shape=[1], dtype='int64')
with fluid.contrib.switch_to_fp16(fluid.default_main_program()):
prediction = fluid.layers.fc(input=img, size=200, act='tanh')
prediction = fluid.layers.cast(prediction, np.float32)
loss = fluid.layers.cross_entropy(input=prediction, label=label)
switch_to_fp16
casts the dtype
of the operator's input to fp16
, and the dtype
of the output is inferred by calling infer_var_type
, in most case, the output's dtype
is fp16
too.
Note: some operator should not apply fp16
computation, because the output of those operators maybe overflows. For example, exp
, log
, softmax
, these operators' output's range is larger then it's input's.
In order to guide users to use fp16
, this PR adds white_list
, black_list
and gray_list
.
-
white_list
is the collection of operators which is safe when using fp16 computation. -
black_list
is the collection of operators which may be overflows when using fp16 computation. -
gray_list
is the collection of operators whose gradient may be overflows when using fp16 computation.
These lists are incomplete now, please feel free to make it completed.