RuntimeWarning: Invalid value encountered in median
Created by: dale1202
我使用的是Paddle提供的官方镜像 paddlepaddle:1.6.3-gpu-cuda10.0-cudnn7 GPU卡数1个,用的是P4机器。 跑的是PaddleDetection里面的faster_rcnn_r101_fpn_1x.yml配置,数据下载是COCO2017。 我跑1分钟就会出现这个错误。然后就一直是NAN了,辛苦帮忙看下
/usr/local/lib/python2.7/dist-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median r = func(a, **kwargs) Warning: NaN or Inf found in input tensor. Warning: NaN or Inf found in input tensor. Warning: NaN or Inf found in input tensor. Warning: NaN or Inf found in input tensor. Warning: NaN or Inf found in input tensor.