Created by: reyoung

We assume each computational library is a place and has its own DeviceContext. So the inheritance relationship of CUDADevCtx, CUDNNDevCtx, and NCCLDevCtx is:

                 /- CUDNNDeviceContext
CUDADeviceContext
                 `- NCCLDeviceContext

Question 1: How about an operator will need both NCCL context and CUDNN context? Should we provide an NCCLAndCUDNNDeviceContext? Or we just let this operator get two DeviceContext instances?

Question 2: Should we implement a decoration pattern for CUDNNDevCtx and NCCLDevCtx? i.e. The sample code is as below:

class CUDADevCtx;
class CUDNNDevCtx : public CUDADevCtx {
public:
  CUDNNDevCtx(CUDADevCtx* cuda_dev_ctx): cuda_dev_ctx_(cuda_dev_ctx) {}
private:
  CUDADevCtx* cuda_dev_ctx_;
};

Question 3: If we make DeviceContext as global variables, how do we use different streams for copying data and computation in the same GPU?

Question 4: If we make DeviceContext as global variables, why we need DeviceContext class? Why not just some global functions?