提交 · 6ab441bb442fd3f8dba1a25abd7fae4bd4d8aa5e · 机器未来 / Paddle

23 4月, 2022 1 次提交

[XPUPS]add hashtable interface (#41987) (#42110) · 6ab441bb

由 zmxdream 提交于 4月 23, 2022

* add hashtable interface. test=develop

* update. test=develop

* update. test=develop

* fix. test=develop

* fix optimizer config for xpups. test=develop

* fix. test=develop

* fix. test=develop

6ab441bb

20 4月, 2022 1 次提交

cherry pick recent updates in graph-engine to release2.3 (#42027) · ef78c9c2

由 seemingwang 提交于 4月 20, 2022

* gpu_graph engine optimization+ (#41455)

* extract sub-graph

* graph-engine merging

* fix

* fix

* fix heter-ps config

* test performance

* test performance

* test performance

* test

* test

* update bfs

* change cmake

* test

* test gpu speed

* gpu_graph_engine optimization

* add ssd layer to graph_engine

* fix allocation

* fix syntax error

* fix syntax error

* fix pscore class

* fix

* recover test

* recover test

* fix spelling

* recover

* fix

* Cpu gpu graph engine (#41942)

* extract sub-graph

* graph-engine merging

* fix

* fix

* fix heter-ps config

* test performance

* test performance

* test performance

* test

* test

* update bfs

* change cmake

* test

* test gpu speed

* gpu_graph_engine optimization

* add ssd layer to graph_engine

* fix allocation

* fix syntax error

* fix syntax error

* fix pscore class

* fix

* recover test

* recover test

* fix spelling

* recover

* fix

* fix linking problem

* remove comment

ef78c9c2

19 4月, 2022 2 次提交

Z
[XPUPS]add rename for heter_ps.cu (#41922) (#41968) · 13202ff7
由 zmxdream 提交于 4月 19, 2022
```
* add rename for heter_ps.cu

* update. test=develop

* update. test=develop

* fix. test=develop
```
13202ff7

[cherry-pick] XPUPS Adaptation (#41917) · a9d8b947

由 Fan Zhang 提交于 4月 19, 2022

* XPUPS Adaptation (#40991)

* Adapt XPUPS - 1st version - 3.24

* Adapt XPUPS - update XPU PushSparse -  2nd version - 3.24

* Adapt XPUPS - add XPU PullSparseOp - 3nd version - 3.25

* refactor heter comm kernel

* update. test=develop

* Adapt XPUPS - modify by compilation - 4th version - 3.27

* update calc_shard_offset. test=develop

* update xpu kernel. test=develop

* update args of calc_shard_offset

* update. test=develop

* remove customGradMerger

* update. test=develop

* heter_comm update

* heter_comm update

* update calc_shard_offset. test=develop

* heter_comm update

* update args of calc_shard_offset

* update. test=develop

* remove customGradMerger

* update. test=develop

* fix. test=develop

* update. test=develop

* update. test=develop

* update optimizer kernel

* Adapt XPUPS - use WITH_XPU_KP and modify wrapper kernel function - 5th version - 3.30

* update. test=develop

* update pslib.cmake

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* Adapt XPUPS - modify by kp compilation  - 6th version - 3.30

* update. test=develop

* update. test=develop

* update. test=develop

* update optimizer kernel

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* fix. test=develop

* fix. test=develop

* used by minxu

* update heter_comm_inl

* fix. test=develop

* Adapt XPUPS - modify by kp compilation  - 7th version - 3.30

* fix. test=develop

* add optimizer kernel. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* 3.31 update

* Adapt XPUPS - update kp compilation path  - 8th version - 3.31

* add optimizer kernel. test=develop

* fix kunlun not support size_t. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix kunlun not support size_t. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update heter_comm_kernel.kps 3.31

* fix. test=develop

* fix. test=develop

* update heter_comm_kernel.kps 3.31

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update heter_comm.h 3.31

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update hashtable. test=develop

* update. test=develop

* Adapt XPUPS - update by kp compilation  - 9th version - 4.1

* update hashtable. test=develop

* fix. test=develop

* update hashtable 4.1

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* Adapt XPUPS - update by kp compilation  - 10th version - 4.1

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update. test=develop

* modify by compilation 4.1

* update. test=develop

* update. test=develop

* fix. test=develop

* modify by compilation 4.1

* update. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* modify by compilation 4.1

* fix. test=develop

* fix. test=develop

* fix. test=develop

* modify by compilation 4.1 19:30

* fix. test=develop

* update ps_gpu_wrapper.kps 4.1

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* Adapt XPUPS - update by kp compilation  - 11th version - 4.1

* fix. test=develop

* Adapt XPUPS - update by kp compilation  - 12nd version - 4.2

* fix. test=develop

* fix. test=develop

* modify by compilation 4.2

* 4.2 update

* fix. test=develop

* template init. test=develop

* update 4.6

* fix. test=develop

* template init. test=develop

* 4.6 modify by compilation

* hashtable template init. test=develop

* hashtable template init. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=devlop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=devlop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* Adapt XPUPS - update by kp compilation  - 13nd version - 4.7

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* 4.11 update

* fix. test=develop

* fix. test=develop

* 4.11 update

* update by pre-commit

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* 4.12 update

* fix. test=develop

* Adapt XPUPS - update by kp compilation  - 14th version - 4.13

* 4.13 update

* 4.14 update

* 4.14 update

* 4.14 update

* 4.14 modify by merged latest compilation

* retry CI 4.14

* 4.15 pass static check

* 4.15 modify by gpups CI

* 3.16 update by gpups CI - modify ps_gpu_wrapper.h

* 4.16 update

* 4.16 pass xpu compile

* 4.16 retry CI

* 4.16 update
Co-authored-by: Nzmxdream <zhangminxu01@baidu.com>

* modify ps_gpu_wrapper.cc

* update
Co-authored-by: Nzmxdream <zhangminxu01@baidu.com>

a9d8b947

18 4月, 2022 1 次提交

[cherry-pick]XPUPS add support for kunlun2 (#41916) · 3a2fb4cf

由 zmxdream 提交于 4月 18, 2022

* [XPUPS]add support for kunlun2 (#40985)


[XPUPS]add support for kunlun2
Co-authored-by: NWorgenZhang <frank08081993@gmail.com>

* [XPUPS]fix hashtable_kernel.kps (#41790)

* refactor heter comm kernel

* update. test=develop

* update calc_shard_offset. test=develop

* update xpu kernel. test=develop

* update args of calc_shard_offset

* update. test=develop

* remove customGradMerger

* update. test=develop

* update. test=develop

* fix. test=develop

* update. test=develop

* update. test=develop

* update optimizer kernel

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* update. test=develop

* fix. test=develop

* fix. test=develop

* add optimizer kernel. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix kunlun not support size_t. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update hashtable. test=develop

* update. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* update. test=develop

* update. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* template init. test=develop

* hashtable template init. test=develop

* fix. test=develop

* fix. test=devlop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix hashtable_kernel. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop
Co-authored-by: NWorgenZhang <frank08081993@gmail.com>

* [XPUPS]modify xpu_kp.cmake with HETERPS&PSLIB (#41760)

* modify xpu_kp.cmake with HETERPS&PSLIB

* fix. test=develop

* fix. test=develop

* fix. test=develop

* fix. test=develop
Co-authored-by: NWorgenZhang <frank08081993@gmail.com>

3a2fb4cf

02 4月, 2022 1 次提交

统一ps refine (#41234) · b3270adf

由 zhaocaibei123 提交于 4月 02, 2022

* update name

* update name

* fix test

* fix fleet bind

* update name

* update name

* fix test

* fix gpups wrapper

* remove Push/Pull/Load/Save with context in client and wrapper base class

* fix

* fix
Co-authored-by: Nesythan <esythan@126.com>

b3270adf

01 4月, 2022 1 次提交

[GPUPS]fix CMakeLists with pslib (#41225) · 4da4265a

由 zmxdream 提交于 4月 01, 2022

* fix cmake. test=develop

* fix. test=develop

* fix dep for graphs_ps_gpu. test=develop

* update. test=develop

* update. test=develop

4da4265a

28 3月, 2022 2 次提交
- W
  delete commonsparsetable and communicator from gpups (#40973) · 630f5b89
  由 wangguanqun 提交于 3月 28, 2022
```
* trainer and worker

* delete commonsparsetable from gpups

* delete vlog

* codestyle

* delete communicator from gpups
```
  630f5b89
- T
  [HeterPS] So Parser (#40750) · cadc4e6a
  由 Thunderbrook 提交于 3月 28, 2022
```
* So Parser

* add macro

* add macro

* slotrecord

* add macro

* code format
```
  cadc4e6a
24 3月, 2022 1 次提交

test gpu graph engine's performance (#40775) · 83ae1619

由 seemingwang 提交于 3月 24, 2022

* extract sub-graph

* graph-engine merging

* fix

* fix

* fix heter-ps config

* test performance

* test performance

* test performance

* test

* test

* update bfs

* change cmake

83ae1619

23 3月, 2022 1 次提交

two-phase training for ps (#40762) · b1a4668c

由 zhaocaibei123 提交于 3月 23, 2022

* fix benchmark and communicator config

* fix bugs of the_one_ps

* multi program and fix bug in optimizer

* multi program in the_one_ps

* public commcontext

* ps optimizer multi programs

* cvm & datanorm backend

* fix dim

* fix unittest

* fix

* the one ps merge

* remove comm

* add DownpourLiteWorker

* all

* fix

* fix

* device worker downpour lite

* fix

* fix bug in global shuffle

* save inference model

* fix & add log

* fix

* remove log

* fix

* fix save summary

* fix

* fix pscore

* fix

* fix

* fix

* fix

* fix

* remove logs

* fix

* fix

* fix

* fix

* fix

* add some comments

* fix
Co-authored-by: Nesythan <esythan@126.com>

b1a4668c

21 3月, 2022 1 次提交

[GpuPs] Update graph sampling method (#40085) · 78fad09b

由 Siming Dai 提交于 3月 21, 2022

* gpu ps graph engine

* remove logs

* Add neighbor sampling method

* Add actual_sample_size and offset for sampling

* Delete Chinese comment

* Fix code style
Co-authored-by: seemingwang <zsasuke@qq.com>

78fad09b

17 3月, 2022 1 次提交
- S
  merge cpu and gpu graph engines (#40597) · 31776199
  由 seemingwang 提交于 3月 17, 2022
```
* extract sub-graph

* graph-engine merging

* fix

* fix

* fix heter-ps config
```
  31776199
15 3月, 2022 1 次提交
- T
  add shard_id (#40261) · 6b7d4845
  由 Thunderbrook 提交于 3月 15, 2022
```
* shard_id

* format
```
  6b7d4845
04 3月, 2022 1 次提交
- T
  Dump cpu xingneng (#40068) · 5496a7ab
  由 Thunderbrook 提交于 3月 04, 2022
```
* dump cpu

* code format
```
  5496a7ab
28 2月, 2022 1 次提交

[Pten->Phi PR4] Rename pten in funcs to phi (#39961) · eb42dd52

由 Chen Weihang 提交于 2月 28, 2022

* rename pten_utils to phi_utils

* rename pten_utils target

* rename Pten to Phi

* replace pten with phi

* resolve conflict

eb42dd52

21 2月, 2022 1 次提交
- S
  gpu ps graph engine (#39699) · 05982c10
  由 seemingwang 提交于 2月 21, 2022
```
* gpu ps graph engine

* remove logs
```
  05982c10
20 2月, 2022 1 次提交

[PTen->Phi PR1] Change pten dirname and namespace to phi (#39748) · dcfe1986

由 Chen Weihang 提交于 2月 20, 2022

* rename pten dir to phi

* rename namespace to phi

* rename infrt pten dir to phi

* resolve conflict

* rename pten to phi in cmake

* revert all infrt change

* change needed files

* fix infrt failed

* fix inference failed

dcfe1986

19 2月, 2022 1 次提交

[Pten]Unify paddle/pten::framework::ddim into pten::ddim (#39614) · 2fe04264

由 Aurelius84 提交于 2月 19, 2022

* Unify paddle/pten::framework::ddim into pten::ddim

* fix paddle namespace

* compile sucessfully

* fix npu src file

* fix conflict

* fix conflict

* fix tensorrt compiler error

* fix conflict

* fix conflict

* fix tesst file conflict

* fix conflict

* fix mlu file conflict

* fix mlu file conflict

* fix cinn header file conflict

* fix conflict

* fix conflict

* fix conflict

* fix conflict

2fe04264

15 2月, 2022 1 次提交

[PTen]Migrate proto::VarType outside of Pten (#39411) · 7e7e9404

由 Aurelius84 提交于 2月 15, 2022

* #1 migrate dist-related type()-> dtype()

* move datatype function from pten -> fluid/framework

* change type() in imperative into convert(dtype())

* modify xx_tensor->type into xx_tensor->dtype

* change the set_type interface and the caller

* modify xx_tensor.type into xx_tensor.dtype

* fix mutable_data(place, dtype())

* change caller of mutable_data in pten and distributed

* change the caller of mutable_data in fluid/framework

* change the caller of mutable_data in imperative directory

* mutable_data: inference

* update the call of mutable_data

* transfer MakePenScalarArray MakePtenScalar ResetHolderWithType

* pass the compile. the next step is remove VarType in Pten

* fix all and remove VarType from pten. success in linux. Next task is other platform

* fix conflict with develop

* fix compiled error

* Fix reset conversion

* fix conflict

* fix compiled problem

* fix typo

* Fix << in tensor_utils.cc

* fix type->dtype

* fix unittest

* fix tensor init constructor

* fix DataTypeSize for BFloat16

* fix code style

* fix npu compiled error

* fix npu

* compile npu sucessfully

* fix conflict

* fix conflict
Co-authored-by: Nxiongkun <xiongkun03@baidu.com>

7e7e9404

28 1月, 2022 1 次提交

[PSLIB] Add Metrics Module, Support User-defined Add Metric (#38789) · 2e6be886

由 Fan Zhang 提交于 1月 28, 2022

* [PSLIB] Add Metrics Module, Support User-defined Add Metric

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* modify role_maker

* update CMakeLists.txt

2e6be886

27 1月, 2022 1 次提交
- T
  compile for afs api (#39113) · 4748486e
  由 Thunderbrook 提交于 1月 27, 2022
```
* compile for afs api

* with pslib
```
  4748486e
25 1月, 2022 2 次提交

Y

reconstruct directory of ps (#39191) · 2bf9b844
由 yaoxuefeng 提交于 1月 25, 2022

2bf9b844

[Move selected_rows PR ] Change the relationship of [include/Cmake]. (#39128) · 2bafd338

由 Weilong Wu 提交于 1月 25, 2022

* Added selected_rows and rw_lock to pten

* Renamed the unit test target to fix CI

* Removed Class SelectedRows in Fluid, changed include/cmake relationship, use pten::SelectedRows in Fluid

* Remove rw_lock.h,rw_lock_test.cc in fluid

* Use pten::RWLock and pten::AutoRDLock, fix CI

* Use pten::SelectedRows

* Use pten::SelectedRows

* Fix to pass NPU CI

* Use pten::SelectedRows, to pass NPU CI

* To fix NPU CI

* To fix NPU CI again

2bafd338

18 1月, 2022 1 次提交
- Z
  [GPUPS]Fix ps_gpu_wrapper (#38993) · 4aa91fd6
  由 zmxdream 提交于 1月 18, 2022
```
* update

* fix ps_gpu_wrapper. test=develop

* fix ps_gpu_wrapper. test=develop
```
  4aa91fd6
17 1月, 2022 1 次提交

[Pten] Replace platform::Place to pten::Place. (#38899) · c48a9ad5

由 Wilber 提交于 1月 17, 2022

* add pten::Place data structure.

* update ci problem

* fix ci problem

* update

* using platform::Place=pten::Place

* remove BOOST_GET_CONST for CPUPlace and GPUPlace

* compile pass 25%.

* compile pass 45%

* compile pass 60%

* remove boost_get for xpu npu mlu and ipu

* compile pass on cpu and gpu.

* fix compile problem

* fix compile error.

* update

* fix ci problem

* update

* ci approve

* fix ci problem

* fix ci eager test problem

* remove BOOST_GET_CONST

* fix npu compile

c48a9ad5

10 1月, 2022 2 次提交

Y

add retry on pull dense sync (#38793) · 0a7cb901
由 yaoxuefeng 提交于 1月 10, 2022

0a7cb901

[Unify Tensors PR ] framework::Tensor inherits from DenseTensor,test=allcases (#38632) · 5c73a6ea

由 Zhanlue Yang 提交于 1月 10, 2022

* Added shared_ptr<Allocation> member & corresponding interfaces to Storage

* Removed original pten::Allocation from Storage and adjusted the interfaces accordingly

* Fixed issues with storage offset

* Used place to malloc allocation for TensorStorage

* [Unify Tensors PR #3]Ported framework::Tensor interfaces to pten::DenseTensor

* Fixed issues with place

* Added comments

* Moved mutable_data with stream argument to DenseTensor

* Added set_offset interface

* Fixed CI issues,test=allcases

* [Unify Tensors PR #4] Port LoDTensor interfaces to DenseTensor

* Removed friend class EigenTensor/EigenMatrix/EigenVector from Tensor

* Modified framework::Tensor to inherit from DenseTensor

* Reverted changes too pten_layout() interface

* Removed friend classes

* Rearranged cfunction calls from tensor.data<void>() to tensor.data()

* Fixed CI issues

* Fixed lite issues

* Fixed data() interface issues,test=allcases

* Resolved IsInitialized() issues

* Fixed ResetHolder() issues

* Fixed MKLDNN & Storage issues

* Resolved ShareBufferWith() issues

* Fixed LoD issues

5c73a6ea

04 1月, 2022 1 次提交
- Y
  heter context support dynamic mf dim (#38487) · 59888bba
  由 yaoxuefeng 提交于 1月 04, 2022
```
heter context support dynamic mf dim
```
  59888bba
31 12月, 2021 1 次提交
- Z
  
  fix compile error for fleetwrapper with -DWITH_TESTING=ON (#38603) · 761055f0
  由 zmxdream 提交于 12月 31, 2021
  
  761055f0
29 12月, 2021 2 次提交
- Y
  add hashtable dynamic mf support (#38493) · d9c174d1
  由 yaoxuefeng 提交于 12月 29, 2021
```
add hashtable dynamic mf support
```
  d9c174d1
- Y
  add dynamic mf size api (#38436) · 7411dab5
  由 yaoxuefeng 提交于 12月 29, 2021
```
add dynamic mf size api
```
  7411dab5
23 12月, 2021 1 次提交
- Y
  add mem pool (#37127) · 745477fe
  由 yaoxuefeng 提交于 12月 23, 2021
```
add mem pool
```
  745477fe
20 12月, 2021 1 次提交
- 石
  
  changes the call AllocShared to Alloc, test=develop (#38258) · bb0713b2
  由石晓伟提交于 12月 20, 2021
  
  bb0713b2
16 12月, 2021 1 次提交
- D
  [psgpu]add checknan print and fix trainer device (#38131) · 092839d6
  由 danleifeng 提交于 12月 16, 2021
```
* trainer_device fix and checknan tool for psgpu;test=develop

* disable show_one_table;test=develop
```
  092839d6
03 12月, 2021 1 次提交
- R
  refine structure for cuda and rocm (#37202) · a6d2fddb
  由 ronnywang 提交于 12月 03, 2021
```
* refine structure for cuda and rocm

* update

* update

* update

* update
```
  a6d2fddb
29 11月, 2021 2 次提交
- T
  [HeterPs] fix allocation (#37476) · 27a5f52b
  由 Thunderbrook 提交于 11月 29, 2021
```
* auc temp

* cuballocator

* code format

* code format
```
  27a5f52b
- X
  
  delete gloo connect retry (#37616) · 41564184
  由 xiaoxiao-luomu 提交于 11月 29, 2021
  
  41564184
17 11月, 2021 1 次提交
- D
  
  fix compile error when pslib use cpu branch;test=develop (#37248) · 0057c12d
  由 danleifeng 提交于 11月 17, 2021
  
  0057c12d
16 11月, 2021 1 次提交
- D
  
  [psgpu]fix pipe bug:save and pull overlap; test=develop (#37233) · 62ec644f
  由 danleifeng 提交于 11月 16, 2021
  
  62ec644f

机器未来 / Paddle 与 Fork 源项目一致

机器未来 / Paddle
与 Fork 源项目一致