RuntimeError: Failed to load model file , please make sure model file is saved with the following APIs: save_params, save_persistables, save_vars
Created by: zjykzj
Hi PaddleOCR, i want to finetune pretrained model using rec_chinese_common_train.yml, but error happens
environment
I just downloaded the latest repo in gitee.com and downloaded the pretrained model
and install requirements
$ python -m pip install -r requirements.txt
# 如果您的机器安装的是CUDA10，请运行以下命令安装
$python3 -m pip install paddlepaddle-gpu==1.7.2.post107 -i https://pypi.tuna.tsinghua.edu.cn/simplereproduce
put the pretrained model into PaddleOCR, like this
$ tree pretrain_models/
pretrain_models/
├── ch_rec_r34_vd_crnn
│ ├── best_accuracy.pdmodel
│ ├── best_accuracy.pdopt
│ ├── best_accuracy.pdparams
│ ├── model
│ └── params
├── ch_rec_r34_vd_crnn_infer.tar
└── ch_rec_r34_vd_crnn.tar
1 directory, 7 filesuse tar xf ... to extract it
run the command like this:
$ CUDA_VISIBLE_DEVICES=1 python3 tools/train.py -c configs/rec/rec_chinese_common_train_zhonglian.yml -o Global.pretrain_weights=./pretrain_models/ch_rec_r34_vd_crnn/
2020-08-17 11:09:31,142-INFO: {'Global': {'debug': False, 'algorithm': 'CRNN', 'use_gpu': True, 'epoch_num': 3000, 'log_smooth_window': 20, 'print_batch_step': 10, 'save_model_dir': './output/rec_CRNN_zhonglian', 'save_epoch_step': 3, 'eval_batch_step': 2000, 'train_batch_size_per_card': 128, 'test_batch_size_per_card': 128, 'image_shape': [3, 32, 320], 'max_text_length': 25, 'character_type': 'ch', 'character_dict_path': './ppocr/utils/ppocr_keys_v1.txt', 'loss_type': 'ctc', 'reader_yml': './configs/rec/rec_chinese_reader.yml', 'pretrain_weights': './pretrain_models/ch_rec_r34_vd_crnn/', 'checkpoints': None, 'save_inference_dir': None, 'infer_img': None, 'distort': True}, 'Architecture': {'function': 'ppocr.modeling.architectures.rec_model,RecModel'}, 'Backbone': {'function': 'ppocr.modeling.backbones.rec_resnet_vd,ResNet', 'layers': 34}, 'Head': {'function': 'ppocr.modeling.heads.rec_ctc_head,CTCPredict', 'encoder_type': 'rnn', 'SeqRNN': {'hidden_size': 256}}, 'Loss': {'function': 'ppocr.modeling.losses.rec_ctc_loss,CTCLoss'}, 'Optimizer': {'function': 'ppocr.optimizer,AdamDecay', 'base_lr': 0.0005, 'beta1': 0.9, 'beta2': 0.999}, 'TrainReader': {'reader_function': 'ppocr.data.rec.dataset_traversal,SimpleReader', 'num_workers': 8, 'img_set_dir': './train_data', 'label_file_path': './train_data/rec_gt_train.txt'}, 'EvalReader': {'reader_function': 'ppocr.data.rec.dataset_traversal,SimpleReader', 'img_set_dir': './train_data', 'label_file_path': './train_data/rec_gt_test.txt'}, 'TestReader': {'reader_function': 'ppocr.data.rec.dataset_traversal,SimpleReader'}}
2020-08-17 11:09:33,025-INFO: places would be ommited when DataLoader is not iterable
W0817 11:09:34.215936 25802 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.2, Runtime API Version: 10.0
W0817 11:09:34.220605 25802 device_context.cc:245] device: 0, cuDNN Version: 7.6.
2020-08-17 11:09:35,960-INFO: Loading parameters from ./pretrain_models/ch_rec_r34_vd_crnn/...
2020-08-17 11:09:35,960-WARNING: ./pretrain_models/ch_rec_r34_vd_crnn/.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-08-17 11:09:35,960-WARNING: ./pretrain_models/ch_rec_r34_vd_crnn/.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py:789: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/io.py", line 1865, in load_program_state
filename=file_name)
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/io.py", line 793, in load_vars
executor.run(load_prog)
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 790, in run
six.reraise(*sys.exc_info())
File "/home/zj/anaconda3/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 785, in run
use_program_cache=use_program_cache)
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 838, in _run_impl
use_program_cache=use_program_cache)
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 912, in _run_program
fetch_var_name)
paddle.fluid.core_avx.EnforceNotMet:
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2 paddle::framework::DeserializeFromStream(std::istream&, paddle::framework::LoDTensor*, paddle::platform::DeviceContext const&)
3 paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, float>::LoadLodTensor(std::istream&, paddle::platform::Place const&, paddle::framework::Variable*, paddle::framework::ExecutionContext const&) const
4 paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
5 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, float>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, double>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, int>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, signed char>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, long> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
6 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const
7 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
8 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
9 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool)
10 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool, bool)
------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op
attrs=kwargs.get("attrs", None))
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/io.py", line 773, in load_vars
attrs={'file_path': os.path.join(dirname, new_var.name)})
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/io.py", line 1865, in load_program_state
filename=file_name)
File "tools/../ppocr/utils/save_load.py", line 56, in _load_state
state = fluid.io.load_program_state(path)
File "tools/../ppocr/utils/save_load.py", line 78, in load_params
state = _load_state(path)
File "tools/../ppocr/utils/save_load.py", line 124, in init_model
load_params(exe, program, path)
File "tools/train.py", line 82, in main
init_model(config, train_program, exe)
File "tools/train.py", line 121, in <module>
main()
----------------------
Error Message Summary:
----------------------
InvalidArgumentError: tensor version 118883584 is not supported, Only version 0 is supported
[Hint: Expected version == 0U, but received version:118883584 != 0U:0.] at (/paddle/paddle/fluid/framework/lod_tensor.cc:287)
[operator < load > error]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tools/train.py", line 121, in <module>
main()
File "tools/train.py", line 82, in main
init_model(config, train_program, exe)
File "tools/../ppocr/utils/save_load.py", line 124, in init_model
load_params(exe, program, path)
File "tools/../ppocr/utils/save_load.py", line 78, in load_params
state = _load_state(path)
File "tools/../ppocr/utils/save_load.py", line 56, in _load_state
state = fluid.io.load_program_state(path)
File "/home/zj/anaconda3/lib/python3.7/site-packages/paddle/fluid/io.py", line 1868, in load_program_state
"Failed to load model file , please make sure model file is saved with the "
RuntimeError: Failed to load model file , please make sure model file is saved with the following APIs: save_params, save_persistables, save_varsi don't know what's wrong and how to fix it. Looking forward to your help