1. 17 1月, 2023 1 次提交
    • Y
      [PHI]Change feed_op to phi kernel (#49116) · f7f1dc03
      YuanRisheng 提交于
      * change feed_op to phi kernel
      
      * fix ci bugs
      
      * fix build bugs
      
      * fix ci bugs
      
      * fix compile bugs
      
      * fix ci bugs
      
      * perfect code
      
      * perfect comment code
      
      * fix install bugs
      
      * modify code according comment
      
      * remove visitor in feed_op
      
      * modify according comment
      
      * perfect code according comment
      
      * add infershape
      
      * fix py3 bugs
      
      * fix getexpected kernel type
      
      * fix getexpected kernel type
      
      * fix ci bugs
      
      * add registry for custom device
      
      * fix py3 bugs
      
      * fix floating point error
      
      * fix py3 test bugs
      f7f1dc03
  2. 12 12月, 2022 1 次提交
  3. 20 10月, 2021 1 次提交
    • S
      Add FasterTokenizer Operator (#34491) · 3f2d6a3f
      Steffy-zxf 提交于
      Add Tokenizer related functionalities for Transformer model in order that the process of training and predicting is consistent.
      
      * support the text string as an input Tensor
      * support the "VOCAB"unordered_map<wstring, int> as an input Tensor to lookup tokens
      * Tokenizer used for BERT. This tokenizer applies an end-to-end, text string to wordpiece tokenization.
      * It first applies basic tokenization, followed by wordpiece tokenization.
      3f2d6a3f