Created by: sasuke96

Hi I was getting assertion error while using ernie on my own data .I was able to run it for MSRA NER data i did the pre processing on my text of removing null values converting it to utf-8 and saving it as tsv file

warnings.warn("Caution! 'set_gradient_clip' is not recommended " [INFO] 2020-07-11 06:44:08,114 [ init.py: 88]: Load pretraining parameters from /axp/rim/emvgml/dev/Vishesh/ernie/ernie_gen_large/params. Traceback (most recent call last): File "/axp/rim/emvgml/dev/Vishesh/ernie/ERNIE-repro/ernie/reader/task_reader.py", line 270, in f for i in wrapper(): File "/axp/rim/emvgml/dev/Vishesh/ernie/ERNIE-repro/ernie/reader/task_reader.py", line 261, in wrapper examples, batch_size, phase=phase): File "/axp/rim/emvgml/dev/Vishesh/ernie/ERNIE-repro/ernie/reader/task_reader.py", line 223, in _prepare_batch_data self.tokenizer) File "/axp/rim/emvgml/dev/Vishesh/ernie/ERNIE-repro/ernie/reader/task_reader.py", line 403, in _convert_example_to_record tokens, labels = self._reseg_token_label(tokens, labels, tokenizer) File "/axp/rim/emvgml/dev/Vishesh/ernie/ERNIE-repro/ernie/reader/task_reader.py", line 397, in _reseg_token_label assert len(ret_tokens) == len(ret_labels) AssertionError /adshome/vgupt137/.local/lib/python3.6/site-packages/paddle/fluid/executor.py:1070: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.")