1. 23 2月, 2022 1 次提交
  2. 20 10月, 2021 1 次提交
    • S
      Add FasterTokenizer Operator (#34491) · 3f2d6a3f
      Steffy-zxf 提交于
      Add Tokenizer related functionalities for Transformer model in order that the process of training and predicting is consistent.
      
      * support the text string as an input Tensor
      * support the "VOCAB"unordered_map<wstring, int> as an input Tensor to lookup tokens
      * Tokenizer used for BERT. This tokenizer applies an end-to-end, text string to wordpiece tokenization.
      * It first applies basic tokenization, followed by wordpiece tokenization.
      3f2d6a3f
  3. 31 8月, 2021 1 次提交
  4. 27 8月, 2021 1 次提交
  5. 26 8月, 2021 1 次提交
    • S
      Add copy from tensor (#34406) · ac33c0ca
      Shang Zhizhou 提交于
      * add api
      
      * temp save
      
      * revert
      
      * copytocpu async ok
      
      * fix style
      
      * copy sync ok
      
      * fix compile error
      
      * fix compile error
      
      * api done
      
      * update python async api
      
      * fix compile
      
      * remove async python api; add c++ async unittest
      
      * remove python async api
      
      * update unittest
      
      * update unittest
      
      * add C++ unittest for copytensor
      
      * add unittest
      
      * update namespace utils to class TensorUtils
      
      * add unittest
      
      * update unittest
      
      * update unittest
      
      * update code style
      
      * update code style
      
      * update unittest
      ac33c0ca
  6. 12 8月, 2021 1 次提交
  7. 14 7月, 2021 1 次提交
  8. 05 3月, 2021 1 次提交