Model OverviewΒΆ
Neural network, which is often called model
in DI-engine, is one of the most important modules in the whole pipelines. It is usually a part of RL policy or reward model. Generally, DI-engine recommands to pack up all
the trainable neural network parameters in model
and other parameter-free operations like calculating loss and selecting action are placed in policy
.
Due to the complexity and diversity of neural network
design and obs/action space, it is hard to abstract a series of general models for each algorithms. Therefore, we design some basic modules in ding/model/common
and ding/torch_utils/network
, and offer
serveral template models for different algorithms in ding/model/template
. Users can set the arguments of template models in some simple environments and implement their own networks referring to the interfaces of template models. Here is a structure graph about the mentioned scheme:
Note
For the input/output arguments of forward
and related methods, DI-engine first provides defailed information(such as object type and tensor shape) in code comment, and we agree on the following rules:
For input, if the input is a torch.Tensor variable, DI-engine directly pass it into model directly, and for other compound inputs, DI-engine uses dict-type input and must record the specific keys in comment.
For output, it is always dict-type output variable, and also need to reocrd the related keys.
Tip
There are often multiple neural networks in a RL algorithms. DI-engine recommands the following design rules:
For the same/similar network architecture but different usage, e.g.: target network in Double DQN and teacher network in Knowledge Distillation, we recommands to use the related model_wrapper.
For the different architectures, such as actor-critic, we recommands that different networks should be different member variables in the whole model.
Note
As to code implementation and arrangement, DI-engine use the following rules:
If a neural network is utilized in more than 2 algorithms and environments, it shoule be added into the main repo. Otherwise, the specific use site is a better choice.
If a neural network is a kind of general deep learning module, it should be located in
ding/torch_utils/network
.For some RL specific networks, such as DuelingHead, it shoule be implemented in
ding/model/commom
.