按照教程安装，运行自定义模型，遇到CUDNN_STATUS_BAD_PARAM (#24309) · Issue · PaddlePaddle / Paddle

按照教程安装，运行自定义模型，遇到CUDNN_STATUS_BAD_PARAM

Created by: lartpang

版本、环境信息：

使用的文档https://www.paddlepaddle.org.cn/install/quick里的安装方法：conda install paddlepaddle-gpu cudatoolkit=10.0。

安装后：

GPU：1080Ti，PaddlePaddle，CUDA和CUDNN版本使用的时conda安装时自动安装的版本：
- cudatoolkit 10.0.130 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
- cudnn 7.3.1 cuda10.0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
- paddlepaddle-gpu 1.7.2.post107 pypi_0 pypi
系统环境：Ubuntu 18.04
Python版本：python 3.7.6 h8356626_5_cpython https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge

模型信息

1）模型名称 2）使用数据集名称 3）使用算法名称 4）模型链接

复现信息：如为报错，请给出复现环境、复现步骤
问题描述：请详细描述您的问题，同步贴出报错信息、日志/代码关键片段

本来想搭一个类似于PyTorch代码的模型，代码如下：

import numpy as np
import paddle.fluid.dygraph as pf_d


class MyLayer(pf_d.Layer):
    def __init__(self):
        super(MyLayer, self).__init__()
        self.conv1 = pf_d.Conv2D(3, 32, 3, 1, 1, 1)
        self.bn_relu1 = pf_d.BatchNorm(32)
        self.conv2 = pf_d.Conv2D(32, 1, 3, act='sigmoid')
    
    def forward(self, inputs):
        print(inputs.shape, var_inp.dtype)
        x = self.conv1(inputs)
        print(x.shape, var_inp.dtype)
        x = self.bn_relu1(x)
        x = self.conv2(x)
        return x


if __name__ == '__main__':
    np_inp = np.random.randn(3, 3, 320, 320).astype(np.float32)
    with pf_d.guard():
        var_inp = pf_d.to_variable(np_inp)
        my_layer = MyLayer()
        x = my_layer(var_inp)
        print(x.shape)

结果遇到了问题。

报错信息

/home/lart/.conda/envs/pd17/bin/python /home/lart/coding/MINet_Paddle/main.py
W0505 19:35:42.455104 15774 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.2, Runtime API Version: 10.0
W0505 19:35:42.457126 15774 device_context.cc:245] device: 0, cuDNN Version: 7.3.
W0505 19:35:42.457134 15774 device_context.cc:271] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.6, but CUDNN version in your machine is 7.3, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
[3, 3, 320, 320] VarType.FP32
[3, 32, 320, 320] VarType.FP32
Traceback (most recent call last):
  File "/home/lart/coding/MINet_Paddle/main.py", line 26, in <module>
    x = my_layer(var_inp)
  File "/home/lart/.conda/envs/pd17/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 304, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lart/coding/MINet_Paddle/main.py", line 16, in forward
    x = self.bn_relu1(x)
  File "/home/lart/.conda/envs/pd17/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 304, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lart/.conda/envs/pd17/lib/python3.7/site-packages/paddle/fluid/dygraph/nn.py", line 1177, in forward
    outs = core.ops.batch_norm(inputs, attrs, outputs)
paddle.fluid.core_avx.EnforceNotMet: 

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2   paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
3   std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
4   paddle::imperative::PreparedOp::Run(std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const*, std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const*, std::unordered_map<std::string, boost::variant<boost::blank, int, float, std::string, std::vector<int, std::allocator<int> >, std::vector<float, std::allocator<float> >, std::vector<std::string, std::allocator<std::string> >, bool, std::vector<bool, std::allocator<bool> >, paddle::framework::BlockDesc*, long, std::vector<paddle::framework::BlockDesc*, std::allocator<paddle::framework::BlockDesc*> >, std::vector<long, std::allocator<long> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, boost::variant<boost::blank, int, float, std::string, std::vector<int, std::allocator<int> >, std::vector<float, std::allocator<float> >, std::vector<std::string, std::allocator<std::string> >, bool, std::vector<bool, std::allocator<bool> >, paddle::framework::BlockDesc*, long, std::vector<paddle::framework::BlockDesc*, std::allocator<paddle::framework::BlockDesc*> >, std::vector<long, std::allocator<long> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > > const*)
5   paddle::imperative::OpBase::Run(std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const&, std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const&)
6   paddle::imperative::Tracer::TraceOp(std::string const&, std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const&, std::map<std::string, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::vector<std::shared_ptr<paddle::imperative::VarBase>, std::allocator<std::shared_ptr<paddle::imperative::VarBase> > > > > > const&, std::unordered_map<std::string, boost::variant<boost::blank, int, float, std::string, std::vector<int, std::allocator<int> >, std::vector<float, std::allocator<float> >, std::vector<std::string, std::allocator<std::string> >, bool, std::vector<bool, std::allocator<bool> >, paddle::framework::BlockDesc*, long, std::vector<paddle::framework::BlockDesc*, std::allocator<paddle::framework::BlockDesc*> >, std::vector<long, std::allocator<long> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, boost::variant<boost::blank, int, float, std::string, std::vector<int, std::allocator<int> >, std::vector<float, std::allocator<float> >, std::vector<std::string, std::allocator<std::string> >, bool, std::vector<bool, std::allocator<bool> >, paddle::framework::BlockDesc*, long, std::vector<paddle::framework::BlockDesc*, std::allocator<paddle::framework::BlockDesc*> >, std::vector<long, std::allocator<long> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >)

----------------------
Error Message Summary:
----------------------
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
  - New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
  - Recommended issue content: all error stack information
  [Hint: CUDNN_STATUS_BAD_PARAM] at (/paddle/paddle/fluid/operators/batch_norm_op.cu:287)


Process finished with exit code 1

不得不说，飞桨各种功能挺丰富，看着挺诱人。但是报错提示是在不人性化了。。。我看了半天没看懂是个啥问题。

另外文档问题如何指出？我觉得我发现了一些问题。

PaddlePaddle / Paddle 大约 2 年 前同步成功

按照教程安装，运行自定义模型，遇到CUDNN_STATUS_BAD_PARAM

版本、环境信息：

模型信息

报错信息

PaddlePaddle / Paddle
大约 2 年前同步成功