Created by: yuye-img

/usr/bin/python3.5 /media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/train.py [INFO: train.py: 295]: Namespace(batch_size=None, config='configs/attention_lstm.txt', enable_ce=False, epoch=10, learning_rate=None, log_interval=10, model_name='AttentionLSTM', no_memory_optimize=False, no_use_pyreader=False, pretrain=None, resume=None, save_dir='checkpoints', use_gpu=False, valid_interval=1) [INFO: config.py: 66]: ---------------- Train Arguments ---------------- [INFO: config.py: 68]: TRAIN: [INFO: config.py: 70]: use_gpu:False [INFO: config.py: 70]: filelist:dataset/youtube8m/train.list [INFO: config.py: 70]: weight_decay:0.0008 [INFO: config.py: 70]: num_samples:5000000 [INFO: config.py: 70]: decay_epochs:[5] [INFO: config.py: 70]: pretrain_base:None [INFO: config.py: 70]: learning_rate:0.001 [INFO: config.py: 70]: decay_gamma:0.1 [INFO: config.py: 70]: num_gpus:8 [INFO: config.py: 70]: epoch:10 [INFO: config.py: 70]: batch_size:1024 [INFO: config.py: 68]: MODEL: [INFO: config.py: 70]: drop_rate:0.5 [INFO: config.py: 70]: embedding_size:512 [INFO: config.py: 70]: bone_nework:None [INFO: config.py: 70]: name:AttentionLSTM [INFO: config.py: 70]: feature_dims:[1024, 128] [INFO: config.py: 70]: dataset:YouTube-8M [INFO: config.py: 70]: num_classes:3862 [INFO: config.py: 70]: lstm_size:1024 [INFO: config.py: 70]: feature_names:['rgb', 'audio'] [INFO: config.py: 70]: feature_num:2 [INFO: config.py: 70]: topk:20 [INFO: config.py: 68]: INFER: [INFO: config.py: 70]: filelist:dataset/youtube8m/infer.list [INFO: config.py: 70]: batch_size:1 [INFO: config.py: 68]: VALID: [INFO: config.py: 70]: filelist:dataset/youtube8m/val.list [INFO: config.py: 70]: batch_size:1024 [INFO: config.py: 68]: TEST: [INFO: config.py: 70]: filelist:dataset/youtube8m/test.list [INFO: config.py: 70]: batch_size:128 [INFO: config.py: 71]: ------------------------------------------------- The CPU_NUM is not specified, you should set CPU_NUM in the environment variable list, i.e export CPU_NUM=1. CPU_NUM indicates that how many CPUPlace are used in the current task. !!! The default number of CPUPlaces is 1.

[WARNING: compiler.py: 239]: You can try our memory optimize feature to save your memory usage: # create a build_strategy variable to set memory optimize option build_strategy = compiler.BuildStrategy() build_strategy.enable_inplace = True build_strategy.memory_optimize = True

     # pass the build_strategy to with_data_parallel API
     compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
         loss_name=loss.name, build_strategy=build_strategy)
  
 !!! Memory optimize is our experimental feature !!!
     some variables may be removed/reused internal to save memory usage, 
     in order to fetch the right value of the fetch_list, please set the 
     persistable property to true for each variable in fetch_list

     # Sample
     conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None) 
     # if you need to fetch conv1, then:
     conv1.persistable = True

I0801 09:07:25.790300 4196 parallel_executor.cc:329] The number of CPUPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies I0801 09:07:25.808480 4196 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1 [WARNING: compiler.py: 239]: You can try our memory optimize feature to save your memory usage: # create a build_strategy variable to set memory optimize option build_strategy = compiler.BuildStrategy() build_strategy.enable_inplace = True build_strategy.memory_optimize = True

     # pass the build_strategy to with_data_parallel API
     compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
         loss_name=loss.name, build_strategy=build_strategy)
  
 !!! Memory optimize is our experimental feature !!!
     some variables may be removed/reused internal to save memory usage, 
     in order to fetch the right value of the fetch_list, please set the 
     persistable property to true for each variable in fetch_list

     # Sample
     conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None) 
     # if you need to fetch conv1, then:
     conv1.persistable = True

share_vars_from is set, scope is ignored. I0801 09:07:25.817261 4196 parallel_executor.cc:329] The number of CPUPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies I0801 09:07:25.818228 4196 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1 [INFO: train_utils.py: 30]: ------- learning rate [0.], learning rate counter [-1] ----- Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layers/io.py", line 596, in provider_thread raise ex File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layers/io.py", line 578, in provider_thread for tensors in func(): File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layers/io.py", line 628, in tensor_provider for slots in paddle_reader(): File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/data_feeder.py", line 416, in reader_creator for item in reader(): File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/datareader/feature_reader.py", line 69, in reader nframes = record[b'nframes'] KeyError: b'nframes'

/usr/local/lib/python3.5/dist-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /usr/local/lib/python3.5/dist-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) [INFO: train_utils.py: 198]: [TRAIN] Epoch 0 training finished, average time: nan Traceback (most recent call last): File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/tools/train_utils.py", line 172, in train_with_pyreader train_outs = train_exe.run(fetch_list=train_fetch_list) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/parallel_executor.py", line 280, in run return_numpy=return_numpy) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 666, in run return_numpy=return_numpy) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 528, in _run_parallel exe.run(fetch_var_names, fetch_var_name) paddle.fluid.core_avx.EOFException: There is no next data. at [/paddle/paddle/fluid/operators/reader/read_op.cc:92]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/train.py", line 300, in train(args) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/train.py", line 288, in train test_metrics=valid_metrics) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/tools/train_utils.py", line 200, in train_with_pyreader "epoch{}".format(epoch)) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/tools/train_utils.py", line 221, in save_model fluid.io.save_persistables(exe, model_path, main_program=program) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 512, in save_persistables filename=filename) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 193, in save_vars filename=filename) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 231, in save_vars executor.run(save_program) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 651, in run use_program_cache=use_program_cache) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 749, in run exe.run(program.desc, scope, 0, True, True, fetch_var_name) paddle.fluid.core_avx.EnforceNotMet: Invoke operator save error. Python Callstacks: File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/framework.py", line 1771, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 215, in save_vars 'file_path': os.path.join(save_dirname, new_var.name) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 193, in save_vars filename=filename) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/io.py", line 512, in save_persistables filename=filename) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/tools/train_utils.py", line 221, in save_model fluid.io.save_persistables(exe, model_path, main_program=program) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/tools/train_utils.py", line 200, in train_with_pyreader "epoch{}".format(epoch)) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/train.py", line 288, in train test_metrics=valid_metrics) File "/media/hxc/新加卷/models-develop/PaddleCV/PaddleVideo/train.py", line 300, in train(args) C++ Callstacks: holder should not be null Tensor not initialized yet when Tensor::type() is called. at [/paddle/paddle/fluid/framework/tensor.h:139] PaddlePaddle Call Stacks: 0 0x7f208c10eb58p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const*, int) + 360 1 0x7f208c10eea7p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87 2 0x7f208c10f84bp paddle::framework::Tensor::type() const + 107 3 0x7f208d450dbdp paddle::framework::GetDataTypeOfVar(paddle::framework::Variable const*) + 157 4 0x7f208c9273d3p paddle::operators::SaveOp::GetExpectedKernelType(paddle::framework::ExecutionContext const&) const + 67 5 0x7f208d45420bp paddle::framework::OperatorWithKernel::ChooseKernel(paddle::framework::RuntimeContext const&, paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 235 6 0x7f208d4556c8p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 728 7 0x7f208d455cd1p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529 8 0x7f208d453b7bp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 267 9 0x7f208c28dc7ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 206 10 0x7f208c290cffp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool) + 143 11 0x7f208c100e7dp 12 0x7f208c13d95ep 13 0x4ea137p PyCFunction_Call + 119 14 0x53c176p PyEval_EvalFrameEx + 23030 15 0x53fc97p 16 0x53bc93p PyEval_EvalFrameEx + 21779 17 0x53fc97p 18 0x53b83fp PyEval_EvalFrameEx + 20671 19 0x53fc97p 20 0x53b83fp PyEval_EvalFrameEx + 20671 21 0x53fc97p 22 0x53b83fp PyEval_EvalFrameEx + 20671 23 0x53fc97p 24 0x53b83fp PyEval_EvalFrameEx + 20671 25 0x53fc97p 26 0x53b83fp PyEval_EvalFrameEx + 20671 27 0x53fc97p 28 0x53b83fp PyEval_EvalFrameEx + 20671 29 0x5401efp 30 0x53bc93p PyEval_EvalFrameEx + 21779 31 0x53fc97p 32 0x5409bfp PyEval_EvalCode + 31 33 0x60cb42p 34 0x60efeap PyRun_FileExFlags + 154 35 0x60f7dcp PyRun_SimpleFileExFlags + 444 36 0x640256p Py_Main + 1110 37 0x4d0001p main + 225 38 0x7f20b41de830p __libc_start_main + 240 39 0x5d6999p _start + 41

Process finished with exit code 1