Prediction error on test set in training process
Created by: leyiwang
if args.do_eval:
dev_data_generator = processor.data_generator(batch_size=args.batch_size, phase='dev',
with_pos=with_pos)
dev_loader.decorate_sample_list_generator(dev_data_generator, places=fluid.CPUPlace())
dev_program = trainer._main_program.clone(for_test=True)
train_loader.decorate_sample_list_generator(train_data_generator, places=fluid.CPUPlace())
train_fetch_list = ["accuracy_0.tmp_0", "accuracy_0.tmp_1", "create_tensor_0"]
steps = 0
for epoch in range(args.epoch):
for train_data in train_loader:
steps += 1
outputs = trainer.run(feed=train_data, fetch=train_fetch_list)
acc, num_seqs, loss = outputs
if steps % args.disp_freq == 0:
print("[train] epoch: {0} step: {1}, accuray: {2}, num_seqs: {3},"
" loss: {4}".format(epoch, steps, acc, num_seqs, loss))
if steps % args.valid_freq and args.do_eval:
total_cost, total_acc, total_num_seqs = [], [], []
for dev_data in dev_loader:
np_acc, np_num_seqs, np_loss = trainer.exe.run(program=dev_program,
feed=dev_data,
fetch_list=train_fetch_list,
return_numpy=False)
I want to evaluate the model in the training process, and create a dev_program
program for test as above in fl_trainer.py
module. However, an error occurred when the trainer is executed, the error detail is as follows:
------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2459, in append_op
attrs=kwargs.get("attrs", None))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 1835, in dropout
'dropout_implementation': dropout_implementation,
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/classifiers/textcnn.py", line 86, in _conv_layer
convs_drop = fluid.layers.dropout(convs_out, self.dropout_rate)
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/classifiers/textcnn.py", line 43, in build_model
convs_drop = self._conv_layer(emb)
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/federated/fl_master.py", line 67, in _init_model
self.accuracy, self.num_seqs, self.loss = self.model.build_model()
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/federated/fl_master.py", line 35, in __init__
self._init_model(args.model_type)
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/federated/fl_master.py", line 83, in create_fl_job
model = Model(args)
File "/Users/wangleyi/PycharmProjects/sms-ai-screening/audit/src/paddle_classifier/federated/fl_master.py", line 110, in <module>
create_fl_job(endpoints=server_endpoint_list, output=configure_save_dir)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
----------------------
Error Message Summary:
----------------------
PaddleCheckError: Expected ctx->Attrs().Get<bool>("is_test") == false, but received ctx->Attrs().Get<bool>("is_test"):1 != false:0.
GradOp is only callable when is_test is false at [/home/teamcity/work/ef54dc8a5b211854/paddle/fluid/operators/dropout_op.cc:108]
I learned that this error is caused by the program
define later than optimizer. However the optimizer is defined in the 'fl_master. py' module, how can I use the dev_program
if it is defined in the fl_master.py
?