Created by: guru4elephant

… card to multi gpu card with fleet api

test=develop

With new apis, a user can switch from single gpu card to multi gpu card on a single machine with fleet API. An single gpu card example is as follows:

import paddle.fluid as fluid

# you should define a loss by a neural nets, and here is just an example
total_loss = fluid.layers.data(name="loss", shape=[1])
role = role_maker.MultiProcessRoleMaker()
optimizer = fluid.optimizer.Adam(learning_rate=0.01)
opts, param_and_grads = optimizer.minimize(total_loss)

A multi process gpu training program is as follows:

import paddle.fluid as fluid
from paddle.fluid.incubate.fleet.collective import fleet, DistributedStrategy
import paddle.fluid.incubate.fleet.base.role_maker as role_maker

# you should define a loss by a neural nets, and here is just an example
total_loss = fluid.layers.data(name="loss", shape=[1])
role = role_maker.MultiProcessRoleMaker()
fleet.init(role)
optimizer = fluid.optimizer.Adam(learning_rate=0.01)
optimizer = fleet.distributed_optimizer(optimizer)
opts, param_and_grads = optimizer.minimize(total_loss)

A start command of multi-process gpu training is

python -m paddle.distributed.launch --gpus 8 my_distriburted_trainer.py