Created by: Jie-Fang
add auto loss scaling.
Usage:
- Cast specific inputs to float16
imgs = fluid.layers.cast(images, "float16")
- Decorate optimizer and minimize scaled loss
optimizer = fluid.optimizer.Adam(learning_rate=0.001)
mp_optimizer = fluid.contrib.mixed_precision.decorate(
optimizer=optimizer, init_loss_scaling=8.0, use_dynamic_loss_scaling=True)
scaled_loss, _, _ = mp_optimizer.minimize(avg_cost)
In the decorate function, you can set incr_every__n_steps
, decr_every_n_nan_or_inf
, incr_ratio
, and decr_ratio
to control the auto loss scaling details.