Conditional clearing of MKL-DNN cache in Executor's destructor
Created by: grygielski
In current state, MKL-DNN cache is being cleared on destruction of an Executor
object. This mechanism was introduced by https://github.com/PaddlePaddle/Paddle/pull/20241 to reduce the length of hashing keys.
However, it appeared to be a problem, especially during training models with e.g. conditional_op
or similar ops that create their own Executor
object inside. Upon destruction of this local Executor, whole MKL-DNN cache is being cleared, thus resulting in crashes during training or slow-downs during inference.
The solution would be to somehow distinguish these 2 "types" of Executors where one is created to execute the whole program (therefore clear mkl-dnn cache on destruction) and the other is created locally in some operators (which doesn't clear mkl-dnn cache when destroyed).