Created by: jtyoui

`140it [03:24, 1.25s/it]train loss 0.00011 150it [03:39, 1.63s/it]train loss 0.00007 160it [03:55, 1.49s/it]train loss 0.00009 166it [04:03, 1.33s/it]Traceback (most recent call last): File "es.py", line 85, in opt.minimize(loss) File "</usr/local/lib/python3.6/site-packages/decorator.py:decorator-gen-65>", line 2, in minimize File "/usr/local/lib/python3.6/site-packages/paddle/fluid/dygraph/base.py", line 273, in impl return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 837, in minimize loss, startup_program=startup_program, params_grads=params_grads) File "/usr/local/lib/python3.6/site-packages/ernie/optimization.py", line 158, in apply_optimize L.assign(p * (1. - self.wd * self.current_step_lr()), p) File "</usr/local/lib/python3.6/site-packages/decorator.py:decorator-gen-62>", line 2, in current_step_lr File "/usr/local/lib/python3.6/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/paddle/fluid/framework.py", line 216, in impl return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 349, in current_step_lr if current_lr: File "/usr/local/lib/python3.6/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py", line 215, in bool return self.nonzero() File "/usr/local/lib/python3.6/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py", line 212, in nonzero return bool(np.all(tensor.array() > 0)) paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::platform::GpuMemcpySync(void*, void const*, unsigned long, cudaMemcpyKind)

Error Message Summary:

ExternalError: Cuda error(77), an illegal memory access was encountered. [Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] at (/paddle/paddle/fluid/platform/gpu_info.cc:281)

`