提交 · 5e839e4da584d073c065a60c39db4f81b16df110 · 机器未来 / Paddle

12 1月, 2021 1 次提交

add sparse embedding & load vars for 2.0 & gloo bug fix (#30306) · 5e839e4d

由 tangwei12 提交于 1月 12, 2021

* add sparse embedding & load vars for 2.0

Change-Id: I36b59ed5f015189dc9d9d2e34a9357722d369f1b

* fix hdfs gloo

Change-Id: Ia84d579053720ad804183e54c9a04b4f031c79c6

* fix gloo hdfs

Change-Id: I5ab982fd483cddc10adcdef0b8aa83aca976cb9e

* move loadvar/sparse embedding from incubute to static

Change-Id: I57081d3545ad2efab78c72420d2162c0eacaf3a0

5e839e4d

05 1月, 2021 1 次提交
- T
  add topo-aware in heter-ps (#30087) · 0b8e1fad
  由 Thunderbrook 提交于 1月 05, 2021
```
* add topo aware

* resource.h

* topo aware

* format
```
  0b8e1fad
29 12月, 2020 1 次提交
- T
  
  add include (#29952) · 0ca6de17
  由 Thunderbrook 提交于 12月 29, 2020
  
  0ca6de17
25 12月, 2020 1 次提交
- Y
  
  remove duplicate ut names (#29809) · 2a01756b
  由 YUNSHEN XIE 提交于 12月 25, 2020
  
  2a01756b
23 12月, 2020 1 次提交

heter box (#29734) · 09b6e719

由 Thunderbrook 提交于 12月 23, 2020

* 　add heter box

* add trainer, worker, wrapper...

* format

* for ci

* format

* remove boost get

* boost & copyright

* rename

* 　rename

* format

* format

* format
Co-authored-by: Nyaoxuefeng6 <yaoxuefeng@baidu.com>

09b6e719

14 12月, 2020 1 次提交
- L
  
  update, test=develop (#29559) · ff6a1450
  由 lilong12 提交于 12月 14, 2020
  
  ff6a1450
23 11月, 2020 1 次提交

support ps-gpu (#28752) · 0073f9bd

由 Thunderbrook 提交于 11月 23, 2020

* ps gpu transpile

* ps gpu

* remove op

* gps trainer

* local ps

* add macro

* HeterBox

* def cuda

* tab

* code style

* style

Co-authored-by: Thunderbrook <a754913769#163.com>

0073f9bd

14 10月, 2020 1 次提交
- W
  
  revert 4 files, from clear include by iwyu, test=develop (#27895) · 41aad9bf
  由 wanghuancoder 提交于 10月 14, 2020
  
  41aad9bf
25 9月, 2020 1 次提交

add xpu in heter mode (#27000) · 6f69a4cb

由 Thunderbrook 提交于 9月 25, 2020

* add xpu in heter mode
test=develop

* BOOST_CONST_GET; PADDLE_THROW
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* refine
test=develop

* refine
test=develop

* refine
test=develop

* refine code
test=develop

6f69a4cb

24 9月, 2020 1 次提交

use iwyu clean include (#27267) · df43905f

由 wanghuancoder 提交于 9月 24, 2020

* use iwyu clean include, test=develop, test=win

* compilation error, test=develop

* fix compilation error2, test=develop

* fix compilation error3, test=develop

* fix compilation error4, test=develop

* fix compilation error5, test=develop

* fix compilation error6, test=develop

* fix compilation error7, test=develop

* fix compilation error8, test=develop

* fix compilation error8, test=develop

* fix compilation error10, test=develop

* fix compilation error11, test=develop

df43905f

18 9月, 2020 1 次提交

【paddle.fleet】gloo and util (#27213) · 99626502

由 tangwei12 提交于 9月 18, 2020

* fix worker endpoints

* fix gloo wrapper for hdfs

* GPU fleetrun support gloo

* parameterserver fleetrun support gloo

* fix get server endpoint

99626502

14 9月, 2020 1 次提交
- C
  
  polish framework error message part 8 (#27269) · 79149c8e
  由 Chen Weihang 提交于 9月 14, 2020
  
  79149c8e
02 9月, 2020 1 次提交
- T
  fix eigen in push sparse; fix hadoop command (#26872) · 52057484
  由 Thunderbrook 提交于 9月 02, 2020
```
* fix eigen in push sparse; fix hadoop command
test=develop

* add log in load_combine_op
test=develop
```
  52057484
31 8月, 2020 1 次提交
- Y
  
  fleet add save with whitelist test=develop (#23376) · a47d92d8
  由 yaoxuefeng 提交于 8月 31, 2020
  
  a47d92d8
27 8月, 2020 1 次提交
- L
  [api 2.0] add collective op for cpu using gloo and paddle.distributed.* apis (#26552) · 1c681383
  由 lilong12 提交于 8月 27, 2020
```
add collective op for cpu using gloo and paddle.distributed.* apis
```
  1c681383
18 8月, 2020 1 次提交
- T
  fix heter proto (#26093) · a83e0f26
  由 Thunderbrook 提交于 8月 18, 2020
```
test=develop
```
  a83e0f26
07 8月, 2020 2 次提交

T
fix compile error with mkl (#26030) · fd2947ba
由 Thunderbrook 提交于 8月 07, 2020
```
test=develop
```
fd2947ba

【paddle.fleet】fleet_util move to paddle.fleet (#25805) · 2191a083

由 123malin 提交于 8月 07, 2020

* test=develop,test=document_fix, remove the out args

* fleet_util move to paddle.fleet
Co-authored-by: NWuHaobo <wuhaobo1994@gmail.com>
Co-authored-by: Ntangwei12 <tangwei12@baidu.com>

2191a083

06 8月, 2020 1 次提交

add heter ps mode (#25682) · 0cb60c70

由 Thunderbrook 提交于 8月 06, 2020

* add heter ps mode

* code style
test=develop

* add with_pslib
test=develop

* unitest
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* code style
test=develop

* test monitor
test=develop

* prepare trainer
test=develop

* code style
test=develop

0cb60c70

11 7月, 2020 1 次提交

Fix index overflow bug of the CUDA kernel loop increment (#25435) · 0b54d54f

由 Chen Weihang 提交于 7月 11, 2020

* fix softmax_with_cross_entropy cuda kernel overflow bug, test=develop

* replace old macro & for condition, test=develop

* polish details, test=develop

0b54d54f

10 6月, 2020 1 次提交

support CMatchAuc (#24990) · 1c224e26

由 hutuxian 提交于 6月 10, 2020

Support CMatchAucCalculator based on CMatchRankAucCalculator with a new parameter ignore_rank

1c224e26

04 6月, 2020 1 次提交

fix problem in dump and add log (#24891) · b8f17a04

由 hutuxian 提交于 6月 04, 2020

* Fix the field length in LoD scenario
* Fix the missed lod info when copy tensor in dump field
* Add some log to make debug easy

b8f17a04

03 6月, 2020 1 次提交

Replace all errors thrown by LOG(FATAL) with PADDLE_THROW (#24759) · d1062d52

由 Chen Weihang 提交于 6月 03, 2020

* remove REPLACE_ENFORCE_GLOG compile option & add ci rule prohibit LOG(FATAL) using, test=develop

* remove ci test case, test=develop

* replace all LOG(FATAL) & polish message, test=develop

* fix typo, test=develop

* polish error info detail, test=develop

d1062d52

26 5月, 2020 1 次提交
- S
  
  fix conflict, test=develop (#24238) · 95089204
  由 ShenLiang 提交于 5月 26, 2020
  
  95089204
25 5月, 2020 1 次提交
- H
  Support AucRunner in PaddleBox (#22884) · e6b87b31
  由 hutuxian 提交于 5月 25, 2020
```
* Support AucRunner in PaddleBox
* update some code style
```
  e6b87b31
11 5月, 2020 2 次提交

change InitializeGPU to InitializeGPUAndLoadModel (#24377) · 123255cf

由 hutuxian 提交于 5月 11, 2020

* Add InitializeGPUAndLoadModel to solve random hang when downloading sparse parameters.
* Update SaveBase to solve test problem.

123255cf

Add macro BOOST_GET to enrich the error information of boost :: get (#24175) · aa0f254f

由 Chen Weihang 提交于 5月 11, 2020

* add new macro BOOST_GET_SAFELY & unittests, test=develop

* add different macro type, test=develop

* fix get macro type in executor, test=develop

* four macro part change backup

* using one macro for all case, test=develop

* revert attribute change, test=develop

* change to three func to solve gcc4.8 bug, test=develop

* polish some details, test=develop

aa0f254f

30 4月, 2020 1 次提交
- X
  add timeout and http store in communication (#23436) · 1034ca31
  由 xujiaqi01 提交于 4月 30, 2020
```
* add timeout and http store in communication, add revert and confirm in fleet
* test=develop
```
  1034ca31
17 4月, 2020 1 次提交
- H
  
  support set_test_mode and set comlog level(#23905) · df64a966
  由 hutuxian 提交于 4月 17, 2020
  
  df64a966
11 4月, 2020 1 次提交
- X
  add save with prefix (#23449) · d98084e7
  由 xujiaqi01 提交于 4月 11, 2020
```
* add save with prefix
* test=develop
```
  d98084e7
10 4月, 2020 1 次提交
- H
  Add AfsAPI in PaddleBox (#23419) · 94a3789f
  由 hutuxian 提交于 4月 10, 2020
```
* Involves AfsAPI to resolve slow downloading.
* Mainly used in PaddleBox
```
  94a3789f
01 4月, 2020 1 次提交
- X
  add fleet pslib pull and push sparse op and push dense op (#23139) · 3a45767d
  由 xujiaqi01 提交于 4月 01, 2020
```
* add fleet pslib pull and push sparse op and push dense op
* test=develop
```
  3a45767d
26 3月, 2020 2 次提交
- X
  add clear one table (#23089) · 68ea1ad5
  由 xujiaqi01 提交于 3月 26, 2020
```
* add clear_one_table
* test=develop
```
  68ea1ad5
- D
  add MaskAucCalculator in paddlebox (#23157) · ae3bb16d
  由 danleifeng 提交于 3月 26, 2020
```
* add maskauc in paddlebox; test=develop
```
  ae3bb16d
20 3月, 2020 1 次提交
- H
  
  Add need_save_delta parameter to solve OOM (#23097) · 0c30098f
  由 hutuxian 提交于 3月 20, 2020
  
  0c30098f
25 2月, 2020 1 次提交

PaddleBox Framework Part2 (#22466) · 175954d8

由 hutuxian 提交于 2月 25, 2020

* Add two types of Metric Calculator: MultiTaskCalculator & CmatchRankCalculator.
* Add a config for DynamicAdjustChannelNum function to denote whether we will discard the remaining instances when they are not be distributed evenly.
* Remove CPU code in Pull/PushSparse and we will add it back when testing it fully.
* Fix some known issues: such as copying persistable vars after one epoch running.

175954d8

11 2月, 2020 3 次提交

Paddlebox about box_wrapper (#22497) · 1a7962be

由 hutuxian 提交于 2月 11, 2020

Refine PaddleBox Framework, Main functions: 
* Add MetricMsg util class, which can calculate metrics like AUC, bucket_error, COPC.
* Replace FeedPass with new interface: BeginFeedPass & EndFeedPass
* Refactor Pull/Push Sparse Function in box_wrapper.
* Use CUDA Kernel to copy keys and copy feasign between tensor and boxps struct.
* Cache copied keys in pull sparse in order to reuse it in push period.

1a7962be

multi-loss optimization by adding a DownpourOpt worker (#22025) · 2235ee1a

由 yaoxuefeng 提交于 2月 11, 2020

* update

* update test=develop

* update compile set test=develop

* update compile set test=develop

* update test=develop

* update test=develop

* update test=develop

* update compile setting test=develop

* update compile setting test=develop

* update run demo test=develop

* update test=develop

* update test=develop

* fix test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update format test=develop

* update format test=develop

* update style test=develop

* update style test=develop

* change style test=develop

* change style test=develop

* change style test=develop

* add dataset unittest test=develop

* update test=develop

* update for record test=develop

* udpate style for record test=develop

* update for record test=develop

* update for record test=develop

* update for record test=develop

* fix format test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

* update test=develop

2235ee1a

Compile without nccl deps. [1/2] (#22509) · a90fa540

由 Wilber 提交于 2月 11, 2020

支持不依赖nccl进行编译。[1/2]

多卡下，如果没有打开WITH_NCCL开关编译，多卡不能通信，则只能选择一张卡使用。
Co-authored-by: N石晓伟 <39303645+Shixiaowei02@users.noreply.github.com>

a90fa540

05 2月, 2020 1 次提交

add WITH_NCCL option for cmake. (#22384) · 7bc4b095

由 Wilber 提交于 2月 05, 2020

cmake选项中添加了WITH_NCCL，显示指定是否编译NCCL的部分代码，WITH_NCCL默认打开，但如果WITH_GPU为OFF，则关闭WITH_NCCL

添加了PADDLE_WITH_NCCL定义

单机单卡能够关闭NCCL编译，多卡的话需要默认打开NCCL，如果关闭NCCL，则只能使用单卡
Co-authored-by: N石晓伟 <39303645+Shixiaowei02@users.noreply.github.com>

7bc4b095

机器未来 / Paddle 与 Fork 源项目一致

机器未来 / Paddle
与 Fork 源项目一致