1. 23 11月, 2021 1 次提交
    • Z
      [cherry-pick]Refactor Heterogenous Pipeline Parameter Server (#37446) · 4dc426f4
      zmx 提交于
      * bug fix for  DeserializeSelectedRows. test=develop (#36520)
      
      * fix SerializeSelectedRows (#36543)
      
      * bug fix for  DeserializeSelectedRows. test=develop
      
      * fix bug for SerializeSelectedRows. test=develop
      
      * update. test=develop
      
      * [Heterps]Refactor Heter Pipeline Parameter Server (#36845)
      
      * change username
      
      * fix
      
      * fix
      
      * fix
      
      * fix
      
      * fix
      
      * update
      
      * update
      
      * update unittests
      
      * fix
      
      * update
      
      * fix
      
      * update
      
      * fix
      
      * fix
      
      * fix
      
      * update
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update send_and_recv op. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * update. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix ut. test=develop
      
      * fix unit. notest,test=coverage
      
      * fix ut. notest, test=coverage
      
      * update. notest,test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix. notest, test=coverage
      
      * fix. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * add func. notest, test=coverage
      
      * fix ut. notest, test=coverage
      
      * fix. test=develop
      
      * fix. test=develop
      
      * Fix unit test for send_and_recv_cpu & send_and_recv_gpu (#37129)
      
      * [heterps]fix ut for heter_pipeline_trainer.cc  (#37136)
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * [heterps]bug fix for local training with --heter_worker_num (#37166)
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * [heterps]Refactor heterogenous worker (#37244)
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * refactor heter trainer. test=develop
      
      * fix. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * [heterps]add heterps mode judgement (#37298)
      
      * [heterps]change default executor for heter trainer (#37314)
      
      * fix pslib. test=develop
      
      * add device to train_from_dataset. test=develop
      
      * refine fleet.stop_worker. test=develop
      
      * fix ut. test=develop
      
      * fix ut. test=develop
      
      * fix executor & ut. test=develop
      
      * fix executor & ut. test=develop
      
      * fix executor & ut. test=develop
      
      * [heterps]remove api for heter pipeline ps (#37396)
      
      * fix api. test=develop
      
      * fix api. test=develop
      
      * fix code style. test=release/2.2
      
      * fix CMakeLists. test=develop (#37454)
      4dc426f4
  2. 26 10月, 2021 1 次提交
    • S
      [Cherry-pick] Add FasterTokenizer Operator (#36716) · edff5b79
      Steffy-zxf 提交于
      * Add FasterTokenizer Operator (#34491)
      
      Add Tokenizer related functionalities for Transformer model in order that the process of training and predicting is consistent.
      
      * support the text string as an input Tensor
      * support the "VOCAB"unordered_map<wstring, int> as an input Tensor to lookup tokens
      * Tokenizer used for BERT. This tokenizer applies an end-to-end, text string to wordpiece tokenization.
      * It first applies basic tokenization, followed by wordpiece tokenization.
      
      * optimize fast tokenizer
      
      * remove const_cast
      Co-authored-by: Nzhoushunjie <zhoushunjie@baidu.com>
      Co-authored-by: Nwawltor <fangzeyang0904@hotmail.com>
      edff5b79
  3. 17 9月, 2021 1 次提交
    • W
      GeneratePass for Python Pass (#35708) · f6db9806
      wuhuanzhou 提交于
      #### 背景
      
      #35602 提供Python侧开发子图替换类Pass的方式:
      
      - 利用Paddle Python API或者辅助类型定义子图program用来匹配/替换图;
      - Python侧注册Pass时,将注册函数最终转换为protobuf定义的PassDesc数据形式,供C++侧进行解析完成Pass实例注册。
      
      本PR即为根据PassDesc规则描述解析生成Pass实例。
      
      #### 方案设计
      
      ##### Pass规则验证
      
      在以往的Pass开发中,会存在随着算子迭代引发的匹配失效或者错误匹配的问题,该问题可以通过扫描算子支持的参数设置及参数类型等来判断是否应该使用该Pass或者给出提示需要修改Pass代码。
      
      当前Pass开发中提供了算子兼容性OpCompatSensiblePass用于解决上述问题。但同时还存在不足:由于以往Pass开发在运行时才能获取到pattern信息,所以需要在执行Pass时才可以判断。
      
      使用PassDesc表示的Pass可以在执行Pass前验证上述问题,这个过程在VerifyDesc中完成。
      
      ##### 根据匹配子图构造pattern
      
      GeneratePass对于图匹配和替换使用GraphPatternDecetor完成,构造匹配pattern实际上就是将对应对象成员PDPattern中添加PDNode和边关系。该过程在函数`InitGeneratePattern`中完成,该函数没有作为GeneratePass的成员方法,主要出于后续可能开发新的Decetor考虑,GeneratePass与Decetor的操作是没有关联的。
      
      初始化pattern主要通过遍历匹配子图program的全部算子实现:
      
      1. 添加当前算子对应PDNode及限制条件(算子类型、属性限制等);
      2. 遍历当前算子对应输入并从pattern中尝试获取PDNode:
         - 在pattern中获取到PDNode且为输出节点:表示属于匹配子图的中间节点,将该PDNode设置为中间节点;
         - 在pattern中没有获取到PDNode:添加该输入PDNode并设置作为输入节点;
         - 设置输入到算子的边关系;
      3. 遍历当前算子对应输出:
         - 在pattern中获取到PDNode且为输入节点:表示属于匹配子图的中间节点,将该PDNode设置为中间节点;
         - 在pattern中没有获取到PDNode:添加该输入PDNode并设置作为输出节点;
         - 设置算子到输出的边关系;
      
      ##### 根据替换子图操作graph
      
      替换子图操作的过程在`GetGenerateRewrite`函数中完成,与`InitGeneratePattern`类似没有作为GeneratePass的成员方法。
      
      生成替换子图操作过程如下:
      
      1. 判断冗余替换子图;
      2. 遍历替换子图program的全部算子添加替换子图Node:
         1. 添加当前算子的Node及属性设置;
         2. 遍历当前算子对应输入,添加中间variable节点;
         3. 遍历当前算子对应输出,添加中间variable节点;
         4. 添加输入/输出节点与算子节点的边关系;
      3. 删除匹配图中属于中间节点的Node;
      
      ##### 优化子图验证
      
      对于替换子图或者替换后的计算图是否可以正确运行等,可以在执行Pass时验证,从而防止在后续执行计算图时出现异常。
      
      当前Pass执行直接修改计算图,验证失败时无法很好的完成还原操作,目前子图验证暂时默认成功,留到后续改进。
      f6db9806
  4. 16 9月, 2021 1 次提交
    • W
      Python support register pass via PassDesc (#35602) · bab39eb2
      wuhuanzhou 提交于
      PR主要功能:针对fusion等子图替换场景,支持Python侧开发并注册Pass。
      
      背景
      Pass是指输入一个深度学习计算图Graph,依照一定条件进行修改,输出修改后的Graph的过程;
      当前PaddlePadle框架编写Pass代码存在以下问题:
      用户需要手写Graph的条件匹配、在Graph上的修改代码;
      对Graph操作需要深入底层框架代码,了解Graph的结构,并且知道相关Pass写法;
      我们提出了针对fusion等子图替换类Pass的优化方案以支持用户在Python侧开发注册Pass,提升二次开发体验:
      用户只需要输入匹配和替换的子图描述,由深度学习框架编写的代码来生成匹配和替换的逻辑,不需要用户对Graph进行匹配和替换操作;
      API级别的替换,用户可以通过Paddle的Python API构造子图,从而不需要知道Graph的结构,也能写Paddle的Graph Pass代码
      bab39eb2
  5. 07 9月, 2021 1 次提交
  6. 24 8月, 2021 1 次提交
  7. 18 8月, 2021 2 次提交
  8. 11 8月, 2021 1 次提交
  9. 10 8月, 2021 1 次提交
    • C
      copy boost/any.hpp to utils and replace boost::any with self defined any (#34613) · 12892929
      chentianyu03 提交于
      * add any.hpp to utils and replace boost::any with self defined paddle::any
      
      * add copy any.hpp to custom op depends
      
      * modify any.hpp include path
      
      * remove boost from setup.py.in
      
      * add copy any.hpp to custom op depends
      
      * move any.hpp to paddle/utils/ dirs
      
      * move any.h to extension/include direction
      
      * copy utils to right directions
      12892929
  10. 05 8月, 2021 1 次提交
    • H
      New executor dev (#34407) · 012d12b5
      hong 提交于
      * first test version
      
      * add test exec;
      
      * add data transfer; test=develop
      
      * add new exec head;
      
      * add memcpy; test=develop
      
      * add python fetch
      
      * add new test
      
      * add graph node; test=develop
      
      * remove useless new executor test; test=develop
      
      * remove gperf dependency; test=develop
      
      * fix compile bugs; test=develop
      
      * remove useless code; test=develop
      
      * remove useless code; test=develop
      
      * add uni test; test=develop
      
      * polish code; test=develop
      
      * polish code; test=develop
      
      * add interpreter cmakefile; test=develop
      
      * remove useless code; test=develop
      012d12b5
  11. 03 8月, 2021 2 次提交
  12. 20 7月, 2021 1 次提交
  13. 15 7月, 2021 3 次提交
  14. 14 7月, 2021 1 次提交
  15. 13 7月, 2021 1 次提交
  16. 02 7月, 2021 2 次提交
  17. 29 6月, 2021 1 次提交
  18. 07 6月, 2021 1 次提交
  19. 03 6月, 2021 1 次提交
  20. 25 5月, 2021 1 次提交
  21. 18 5月, 2021 1 次提交
  22. 10 5月, 2021 1 次提交
  23. 07 5月, 2021 1 次提交
  24. 28 4月, 2021 1 次提交
  25. 15 4月, 2021 1 次提交
  26. 09 4月, 2021 1 次提交
  27. 30 3月, 2021 1 次提交
  28. 18 3月, 2021 1 次提交
  29. 04 3月, 2021 1 次提交
  30. 27 2月, 2021 1 次提交
  31. 25 2月, 2021 1 次提交
  32. 24 2月, 2021 2 次提交
  33. 22 2月, 2021 1 次提交
  34. 10 2月, 2021 1 次提交
    • C
      New custom operator extension mechanism (#30690) · f649442d
      Chen Weihang 提交于
      * initial commit: simple demo
      
      * polish copyright format
      
      * add grap op simple demo
      
      * adapt uncertain number of argument
      
      * change trait marco name
      
      * add place & dtype support for add kernel
      
      * add dispath and infershape func
      
      * poish code & add notes
      
      * add dynamic_loader dep for paddle_framework
      
      * add new custom op test dir
      
      * polish impl details
      
      * add unittest for new custom op
      
      * fix failed unittest
      
      * Costum op (#1)
      
      * fix compile error
      
      * wrap framework tensor with LoDTensor
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * add CustomTensor default constructor
      
      * add size() for CustomTensor
      
      * make size const for CustomTensor
      
      * refactor place related api to circle the concept
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * make place const
      
      * make Tensor copy
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * remove additional head of framework
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * add gpu test
      
      * merge latest cwh code in
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * Remove ShareData from user && Change CustomTensor to Tensor && Support more data type (#2)
      
      * fix compile error
      
      * wrap framework tensor with LoDTensor
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * add CustomTensor default constructor
      
      * add size() for CustomTensor
      
      * make size const for CustomTensor
      
      * refactor place related api to circle the concept
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * make place const
      
      * make Tensor copy
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * remove additional head of framework
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * add gpu test
      
      * merge latest cwh code in
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * hid share data from and to
      
      * rename CustomTensor to Tensor
      
      * refactor register design & add test
      
      * change op_funtion to op_meta_info
      
      * split op meta info into .h and .cc
      
      * move get methods into friend class
      
      * move OpMetaInfoHelper into framework space
      
      * move CustomTensorUtils into framework space
      
      * change pybind api name
      
      * move PD C API into op meta info
      
      * add register custom op api
      
      * remove inference cmake change
      
      * refactor copy to api && change Reshape to lowercase && support more dtype && add more test (#3)
      
      * fix compile error
      
      * wrap framework tensor with LoDTensor
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * add CustomTensor default constructor
      
      * add size() for CustomTensor
      
      * make size const for CustomTensor
      
      * refactor place related api to circle the concept
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * make place const
      
      * make Tensor copy
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * remove additional head of framework
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * add gpu test
      
      * merge latest cwh code in
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * hid share data from and to
      
      * rename CustomTensor to Tensor
      
      * support multi dtype
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * fix copy to error
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * polish detail & error message
      
      * polish test details
      
      * Add cast api && Change copy related api to copy_to && add more test (#4)
      
      * fix compile error
      
      * wrap framework tensor with LoDTensor
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * add CustomTensor default constructor
      
      * add size() for CustomTensor
      
      * make size const for CustomTensor
      
      * refactor place related api to circle the concept
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * fix compile error
      
      * make place const
      
      * make Tensor copy
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * debug CustomTensor core
      
      * remove additional head of framework
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * use back to shared ptr for custom tensor
      
      * add gpu test
      
      * merge latest cwh code in
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * adjust ut code of custom op
      
      * hid share data from and to
      
      * rename CustomTensor to Tensor
      
      * support multi dtype
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * remove lod, make reshape lowercase, add copy test and refactor copy api
      
      * fix copy to error
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add more test
      
      * add type cast
      
      * add cast and make copy to api
      
      * add cast and make copy to api
      
      * add cast and make copy to api
      
      * add cast and make copy to api
      
      * merge cwh code
      
      * merge cwh code
      
      * merge cwh code
      
      * merge cwh code
      
      * merge cwh code
      
      * add more error log
      
      * add more error log
      
      * polish code
      
      * used for test
      
      * remove test comment
      
      * remove test comment
      
      * fix uint8 type error
      
      * fix lost uint8 type error
      
      * add test for coverage
      
      * polish details by reviewer comments
      
      * add prefix for DISABLE_COPY_AND_ASSIGN
      Co-authored-by: NJiabin Yang <360788950@qq.com>
      f649442d