- 17 5月, 2022 1 次提交
-
-
由 Chen Weihang 提交于
* adapt faster tokenizer op * add eager test * add unittest
-
- 24 4月, 2022 1 次提交
-
-
由 pangyoki 提交于
* test=py3-eager * test=py3-eager * test=py3-eager
-
- 25 3月, 2022 1 次提交
-
-
由 Jiabin Yang 提交于
* refactor eager flags * fix flags error when we switch from eager to dygraph * fix ci problem * fix ci * fix ci * merge develop and fix code style * merge develop and fix code style * fix op test error * fix op test error * fix op test error * fix op test error * fix op test error * merge develop
-
- 22 12月, 2021 1 次提交
-
-
由 Zhanlue Yang 提交于
-
- 20 10月, 2021 1 次提交
-
-
由 Steffy-zxf 提交于
Add Tokenizer related functionalities for Transformer model in order that the process of training and predicting is consistent. * support the text string as an input Tensor * support the "VOCAB"unordered_map<wstring, int> as an input Tensor to lookup tokens * Tokenizer used for BERT. This tokenizer applies an end-to-end, text string to wordpiece tokenization. * It first applies basic tokenization, followed by wordpiece tokenization.
-