Created by: randomtutu

----------- Configuration Arguments ----------- batch_size: 2 checkpoints: ./checkpoints dev_set: /home/wb-wzft515778/LARK/ERNIE/data//lcqmc/test.tsv do_lower_case: True do_test: True do_train: True do_val: True epoch: 3 ernie_config_path: config/ernie_config.json in_tokens: False init_checkpoint: None init_pretraining_params: /home/wb-wzft515778/LARK/ERNIE/ckpt//params label_map_config: None learning_rate: 2e-05 loss_scaling: 1.0 lr_scheduler: linear_warmup_decay max_seq_len: 16 metrics: True num_iteration_per_drop_scope: 1 num_labels: 2 random_seed: 1 save_steps: 1000 skip_steps: 10 test_set: /home/wb-wzft515778/LARK/ERNIE/data//lcqmc/test.tsv train_set: /home/wb-wzft515778/LARK/ERNIE/data//lcqmc/train.tsv use_cuda: True use_fast_executor: False use_fp16: False validation_steps: 100 verbose: True vocab_path: config/vocab.txt warmup_proportion: 0.0 weight_decay: 0.0

attention_probs_dropout_prob: 0.1 hidden_act: relu hidden_dropout_prob: 0.1 hidden_size: 768 initializer_range: 0.02 max_position_embeddings: 513 num_attention_heads: 12 num_hidden_layers: 12 type_vocab_size: 2 vocab_size: 18000

Device count: 1 Num train examples: 102477 Max train steps: 153715 Num warmup steps: 0 Theoretical memory usage in training: 2102.707 - 2202.836 MB W0321 17:00:02.179066 66953 device_context.cc:263] Please NOTE: device: 0, CUDA Capability: 60, Driver API Version: 9.2, Runtime API Version: 9.0 W0321 17:00:02.179129 66953 device_context.cc:271] device: 0, cuDNN Version: 7.0. Traceback (most recent call last): File "run_classifier.py", line 283, in main(args) File "run_classifier.py", line 141, in main exe.run(startup_prog) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/executor.py", line 525, in run use_program_cache=use_program_cache) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/executor.py", line 591, in run exe.run(program.desc, scope, 0, True, True) paddle.fluid.core.EnforceNotMet: Invoke operator fill_constant error. Python Callstacks: File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/framework.py", line 1382, in prepend_op attrs=kwargs.get("attrs", None)) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/initializer.py", line 167, in call stop_gradient=True) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/framework.py", line 1198, in create_var kwargs'initializer' File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 402, in set_variable_initializer initializer=initializer) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/layers/tensor.py", line 137, in create_global_var value=float(value), force_cpu=force_cpu)) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 92, in create_global_learning_rate persistable=True) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 224, in create_optimization_pass self.create_global_learning_rate() File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 350, in apply_gradients optimize_ops = self.create_optimization_pass(params_grads) File "/home/wb-wzft515778/anaconda3/envs/paddle/lib/python3.6/site-packages/paddle/fluid/optimizer.py", line 405, in minimize optimize_ops = self.apply_gradients(params_grads) File "/home/wb-wzft515778/LARK/ERNIE/optimization.py", line 127, in optimization , param_grads = optimizer.minimize(loss) File "run_classifier.py", line 108, in main loss_scaling=args.loss_scaling) File "run_classifier.py", line 283, in main(args) C++ Callstacks: Enforce failed. Expected allocating <= available, but received allocating:14920696472 > available:14265745152. Insufficient GPU memory to allocation. at [/paddle/paddle/fluid/platform/gpu_info.cc:216] PaddlePaddle Call Stacks: 0 0x7f76e04d190dp void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const*, int) + 365 1 0x7f76e04d1c57p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87 2 0x7f76e1f9baf8p paddle::platform::GpuMaxChunkSize() + 744 3 0x7f76e1f922a9p 4 0x7f779ab35bb9p 5 0x7f76e1f91a7dp paddle::memory::legacy::GetGPUBuddyAllocator(int) + 109 6 0x7f76e1f92767p void* paddle::memory::legacy::Allocpaddle::platform::CUDAPlace(paddle::platform::CUDAPlace const&, unsigned long) + 39 7 0x7f76e1f92e35p paddle::memory::allocation::LegacyAllocator::AllocateImpl(unsigned long, paddle::memory::allocation::Allocator::Attr) + 389 8 0x7f76e1f9513bp paddle::memory::allocation::Allocator::Allocate(unsigned long, paddle::memory::allocation::Allocator::Attr) + 27 9 0x7f76e1f868e3p paddle::memory::allocation::AllocatorFacade::Alloc(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long, paddle::memory::allocation::Allocator::Attr) + 435 10 0x7f76e1f86a01p paddle::memory::allocation::AllocatorFacade::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long, paddle::memory::allocation::Allocator::Attr) + 33 11 0x7f76e1c48540p paddle::memory::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long, paddle::memory::allocation::Allocator::Attr) + 48 12 0x7f76e1f43576p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, paddle::framework::proto::VarType_Type, paddle::memory::allocation::Allocator::Attr, unsigned long) + 150 13 0x7f76e095fc01p paddle::operators::FillConstantKernel::Compute(paddle::framework::ExecutionContext const&) const + 497 14 0x7f76e0962c23p std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernelpaddle::platform::float16 >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) + 35 15 0x7f76e1ee5f23p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 659 16 0x7f76e1ee3795p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 341 17 0x7f76e05edf1ap paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 218 18 0x7f76e05eff15p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 261 19 0x7f76e04b5a7bp 20 0x7f76e04fc79ep 21 0x7f779b07e744p _PyCFunction_FastCallDict + 340 22 0x7f779b10557ep 23 0x7f779b12a38ap _PyEval_EvalFrameDefault + 778 24 0x7f779b0fe8e4p 25 0x7f779b0ff771p 26 0x7f779b105505p 27 0x7f779b12b147p _PyEval_EvalFrameDefault + 4295 28 0x7f779b0fe8e4p 29 0x7f779b0ff771p 30 0x7f779b105505p 31 0x7f779b12a38ap _PyEval_EvalFrameDefault + 778 32 0x7f779b0ff53bp 33 0x7f779b105505p 34 0x7f779b12a38ap _PyEval_EvalFrameDefault + 778 35 0x7f779b100289p PyEval_EvalCodeEx + 809 36 0x7f779b10101cp PyEval_EvalCode + 28 37 0x7f779b1833c4p 38 0x7f779b1837c1p PyRun_FileExFlags + 161 39 0x7f779b1839c3p PyRun_SimpleFileExFlags + 451 40 0x7f779b1874b3p Py_Main + 1555 41 0x7f779b05002ep main + 238 42 0x7f779a781401p __libc_start_main + 241 43 0x7f779b130e0ep