Created by: typhoonzero
Reason of this change:
- IR passes need to modify startup program if the pass add/rename/delete variables, and the variable needs an initializer.
- Passes are executed by ParallelExecutor, and we want to run statup after
passes.Apply
, so we need to run startup in ParallelExecutor constructor. - To keep API compatibility, we can allow user to call
executor.run(startup)
before ParallelExecutor constructor - User may need to load a model before running, need to be compatible of loading models
Cases
Consider below cases, this change should work on all of them:
# Normal cases
exe.run(startup)
pe = fluid.ParallelExecutor()
pe.run()
# single node load model and fine tune
exe.run(startup)
fluid.load_vars() # or fluid.load_persistables etc.
pe = fluid.ParallelExecutor()
pe.run()
# single node load model and fine tune
exe.run(startup)
fluid.load_vars() # or fluid.load_persistables etc.
pe = fluid.ParallelExecutor()
pe.run()
# single node simplified
pe = fluid.ParallelExecutor() # run startup inside here
pe.run()
# single node load model
fluid.load_vars()
pe = fluid.ParallelExecutor() # run startup inside here
pe.run()
# distributed mode
main, startup = fluid.DistributedTranspiler().transpile(...)
exe.run(startup)
fluid.load_vars()
pe = fluid.ParallelExecutor() # run startup inside here
pe.run()