Error running with GPU
Created by: sgrvinod
I'm trying to use the pretrained wmt14
translation model from the demo. It works fine on the CPU, but as soon as I set --use_gpu=1
, it gives the following error:
I0222 16:08:09.291610 30305 Util.cpp:155] commandline: /usr/bin/../opt/paddle/bin/paddle_trainer --job=test --config=translation/gen.conf --save_dir=data/wmt14_model --use_gpu=1 --num_passes=13 --test_pass=12 --trainer_count=1 --config_args=beam_size=3,max_length=100
*** stack smashing detected ***: /usr/bin/../opt/paddle/bin/paddle_trainer terminated
*** Aborted at 1487808490 (unix time) try "date -d @1487808490" if you are using GNU date ***
PC: @ 0x7f81132fcc37 (unknown)
*** SIGABRT (@0x7661) received by PID 30305 (TID 0x7f8115609840) from PID 30305; stack trace: ***
@ 0x7f8114f05330 (unknown)
@ 0x7f81132fcc37 (unknown)
@ 0x7f8113300028 (unknown)
@ 0x7f81133392a4 (unknown)
@ 0x7f81133d0bbc (unknown)
@ 0x7f81133d0b60 (unknown)
@ 0x8954ce hl_create_global_resources()
@ 0x8959f9 hl_specify_devices_start()
@ 0x895d2d hl_start()
@ 0x818602 paddle::initMain()
@ 0x52ac5b main
@ 0x7f81132e7f45 (unknown)
@ 0x540c05 (unknown)
@ 0x0 (unknown)
/usr/bin/paddle: line 109: 30305 Aborted (core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}
My NVIDIA-SMI, if useful:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48 Driver Version: 367.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A |
| 23% 38C P8 15W / 250W | 1MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 0000:03:00.0 Off | N/A |
| 23% 37C P8 16W / 250W | 1MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 TITAN X (Pascal) Off | 0000:04:00.0 Off | N/A |
| 23% 33C P8 8W / 250W | 1MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 TITAN X (Pascal) Off | 0000:05:00.0 On | N/A |
| 28% 47C P8 17W / 250W | 460MiB / 12188MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
I'm using
PaddlePaddle 0.9.0, compiled with
with_avx: ON
with_gpu: ON
with_double: OFF
with_python: ON
with_rdma: OFF
with_glog: ON
with_gflags: ON
with_metric_learning:
with_timer: OFF
with_predict_sdk:
Cuda V8.0.44 Ubuntu 14.04
Any help would be appreciated, thanks!