Created by: yran1011

环境: cuda10.0 cudnn7.6 RTX2080ti PaddlePaddle 1.8.1 使用配置文件: blazeface.yml 数据集:wider_face 修改学习率 batch_size 等都报同样错误 2020-05-29 11:48:35,923-INFO: iter: 840, lr: 0.001000, 'loss': '6.801611', time: 0.514, eta: 1 day, 21:36:46 2020-05-29 11:48:46,167-INFO: iter: 860, lr: 0.001000, 'loss': '6.761255', time: 0.505, eta: 1 day, 20:45:01 2020-05-29 11:48:57,223-INFO: iter: 880, lr: 0.001000, 'loss': '6.585557', time: 0.561, eta: 2 days, 1:41:11 2020-05-29 11:49:07,257-INFO: iter: 900, lr: 0.001000, 'loss': '6.676511', time: 0.506, eta: 1 day, 20:49:45 2020-05-29 11:49:19,619-INFO: iter: 920, lr: 0.001000, 'loss': '6.596319', time: 0.615, eta: 2 days, 6:29:31 2020-05-29 11:49:31,520-INFO: iter: 940, lr: 0.001000, 'loss': '6.578011', time: 0.583, eta: 2 days, 3:38:20 2020-05-29 11:49:44,018-INFO: iter: 960, lr: 0.001000, 'loss': '6.779737', time: 0.639, eta: 2 days, 8:37:10 2020-05-29 11:49:54,386-INFO: iter: 980, lr: 0.001000, 'loss': '6.576465', time: 0.513, eta: 1 day, 21:29:59 2020-05-29 11:50:05,546-INFO: iter: 1000, lr: 0.001000, 'loss': '6.522973', time: 0.559, eta: 2 days, 1:31:14 2020-05-29 11:50:17,470-INFO: iter: 1020, lr: 0.001000, 'loss': '6.584724', time: 0.601, eta: 2 days, 5:16:22 2020-05-29 11:50:26,784-INFO: iter: 1040, lr: 0.001000, 'loss': '6.423560', time: 0.437, eta: 1 day, 14:43:18 2020-05-29 11:50:36,031-INFO: iter: 1060, lr: 0.001000, 'loss': '6.354516', time: 0.491, eta: 1 day, 19:31:08 2020-05-29 11:50:37,214-INFO: fail to map op [CropImageWithDataAchorSampling_c097e9] with error: Unable to allocate 49.5 GiB for an array with shape (47081, 47081, 3) and data type float64 and stack: Traceback (most recent call last): File "/home/terrywang/RanYu/PaddleDetection/ppdet/data/reader.py", line 45, in call data = f(data, ctx) File "/home/terrywang/RanYu/PaddleDetection/ppdet/data/transform/operators.py", line 924, in call image_height, self.target_size) File "/home/terrywang/RanYu/PaddleDetection/ppdet/data/transform/op_helper.py", line 382, in crop_image_sampling sample_img = np.zeros((height, width, 3)) MemoryError: Unable to allocate 49.5 GiB for an array with shape (47081, 47081, 3) and data type float64

2020-05-29 11:50:37,236-WARNING: recv endsignal from outq with errmsg[consumer[consumer-aef-1] failed to map with error:[Unable to allocate 49.5 GiB for an array with shape (47081, 47081, 3) and data type float64]] 2020-05-29 11:50:38,315-WARNING: recv endsignal from outq with errmsg[consumer[consumer-aef-4] exits for reason[consumer[consumer-aef-1] failed to map with error:[Unable to allocate 49.5 GiB for an array with shape (47081, 47081, 3) and data type float64]]] 2020-05-29 11:50:40,888-WARNING: recv endsignal from outq with errmsg[consumer[consumer-aef-0] exits for reason[consumer[consumer-aef-4] exits for reason[consumer[consumer-aef-1] failed to map with error:[Unable to allocate 49.5 GiB for an array with shape (47081, 47081, 3) and data type float64]]]]