paddle v2 模型训练异常报错
Created by: duanyuzhuo
模式:cpu本地 描述:模型训练阶段,网络构建成功,后续报错如下
Traceback (most recent call last):
File "./thirdparty/train.py", line 147, in <module>
dssm_trainer.train()
File "./thirdparty/train.py", line 137, in train
num_passes=self.model_conf.get_int("train", "PASS_NUM")
File "/home/disk3/duanyuzhuo/dyz_file/fromTian/cluster_train_cpu_nordma/paddle/python27-gcc482/lib/python2.7/site-packages/paddle/v2/trainer.py", line 153, in train
in_args = feeder(data_batch)
File "paddle/python27-gcc482/lib/python2.7/site-packages/py_paddle/dataprovider_converter.py", line 282, in __call__
return self.convert(dat, argument)
File "paddle/python27-gcc482/lib/python2.7/site-packages/paddle/v2/data_feeder.py", line 134, in convert
return DataProviderConverter.convert(self, reorder_data(dat), argument)
File "paddle/python27-gcc482/lib/python2.7/site-packages/py_paddle/dataprovider_converter.py", line 267, in convert
scanner.pre_scan(each_step)
File "paddle/python27-gcc482/lib/python2.7/site-packages/py_paddle/dataprovider_converter.py", line 224, in pre_scan
for each in dat:
TypeError: 'int' object is not iterable
因为报错定位到paddle脚本,通过参考其它issue(地址:https://github.com/PaddlePaddle/Paddle/issues/2062) 定位到应该是数据读取的问题,看起来是序列数据获取时并非是序列导致,排查过数据本身可以确定要feed进去的数据并非不可迭代。需要的话可以提供脚本和运行环境。 辛苦排查!