Created by: Superjomn

Currently, the refactored Paddle framework does not include any concurrency design, both the distributed cluster computation and multi-device accelerations' design are in process.

I think it's a good time point to add the dependency engine(DE) to simplify our computational model. DE is the same role of CPU. It manages the computation resources and scheduling threads in multiple devices.

All the other computational tasks execute by pushing their job to DE, and DE determines which task is ready to run by analysis the status of resource it depends on (either READ or WRITE), and push the available jobs to a task queue.

Each device might have a different job queue with a thread pool, which makes multi-device parallel execution possible.

What's more, to some extent, nodes in a cluster can be treated as special devices and this view will make the design of multi-device acceleration and distributed cluster framework similar and simpler.

All in all, DE is a unified way to parallelize computation across different devices.

reference

mxnet engine mxnet engine header file simple implementation with less than 500 lines of code