未验证 提交 3dd14e4e 编写于 作者: L liu zhengxi 提交者: GitHub

Fix delimiter (#5075)

* fix delimiter
......@@ -30,6 +30,8 @@ def get_default_tokenizer():
"""
def _split_tokenizer(x, delimiter=None):
if delimiter == "":
return list(x)
return x.split(delimiter)
return _split_tokenizer
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册
反馈
建议
客服 返回
顶部