Created by: jczaja

Code related to cross entropy is guarded with a functionality that turn -INF , +INF into -1e20 and 1e20 respectively. For example:

https://github.com/PaddlePaddle/Paddle/blob/ddf5783644c95195831101be31dd406d9e84e598/paddle/fluid/operators/math/cross_entropy.cc#L50

https://github.com/PaddlePaddle/Paddle/blob/ddf5783644c95195831101be31dd406d9e84e598/paddle/fluid/operators/cross_entropy_op.h#L81

It helps when logarithm of 0 (or very close to 0) is computed . On the other hand there is a mechanism of catching Arithmetic exception enabled:

https://github.com/PaddlePaddle/Paddle/blob/4c55a6022a0a758295177371fc67c6800658b286/paddle/api/Util.cpp#L30

Hence, when we have math::TolerableValue()(std::log(0)); math::Tolerable will not have a chance to work, as exception will be triggerred (DIV by 0).

My question is if math::TolerableValue()(std::log(0)); is a valid situation, or it should not occur?

Broader context of this question is that I'm working on Softmax operator optimization using MKL-DNN primitives, and would like to know if I need to add EPS to MKL-DNN softmax forward results or I can assume that TolerableValue will correct any -INF, +INF . If I can assume that std::log(0) is valid situation then Floating point exception should not be caught .

Please advice