1. 30 11月, 2018 1 次提交
    • W
      AsyncExecutor (#14627) · 41e19eb4
      Wang Guibao 提交于
      * AsyncExecutor: C++ side
      
      * Google naming conventions
      
      * Rename MultiExecutor to AsyncExecutor
      
      * pybind with async_executor
      
      * Naming convention
      
      * remove some flags and unused code
      
      * add refactored file of async_executor and data_feed
      
      * clear async executor interface and add data feed factory
      
      * split async executor into executor_thread_worker and async_executor, refactor pybind, add datafeed and corresponding proto
      
      * Fix async_executor interfaces: 1) Remove all protobufs; 2) Stop after each epoch
      
      * refine async_executor_refactor.cc
      
      * add some files about datafeed
      
      * Revert "add some files about datafeed"
      
      This reverts commit 8ee8133a.
      
      * Interface rework
      
      * add MultiSlotDataFeed
      
      * Creating DataFeedDesc from .proto file, then manipulate it (add/del fields etc) from python side
      
      * update data_feed for add MultiSlotDataFeed
      
      * update datafeed and async_executor to run bow_net demo
      
      * fix bug that finish_set_filelist failed in multithread
      
      * delete finish_binding_memory_(flag), because it can not be marked under the current interface
      
      * Fix bug
      
      * update async_executor.py for support set_use_slots
      
      * update async_executor.py for support set_use_slots and set set_dense_slots
      
      * fix bug that when the number of files is less than the number of threads, it will fetch nan
      
      * remove redundant code, and make executor exit when set a illegal queue size
      
      * add batch_size check
      
      * add MultiSlotDesc
      
      * Revert "add MultiSlotDesc"
      
      This reverts commit 2e72ebfa.
      
      * add some checkpoint in DataFeedDesc
      
      * add CheckFile function in MultiSlotDataFeed
      
      * update something error info
      
      * fix deaded lock bug
      
      * Fix fetch variable
      
      * Merge error
      
      * fix code style in async_executor
      
      * using one lock blocking queue replace two lock blocking queue because of some bugs
      
      * update code style
      
      * add utest for data_feed
      
      * Fix fetch var
      
      * update utest for data_feed for multithread
      
      * update SetFileList info
      
      * fix bug in utest of data_feed
      
      * Add comments for python
      
      * Add comments for python code
      
      * Fix pybind.cc with new pybind11 version
      
      * add note for DataFeedDesc's set_use_slots function
      
      * Add save_model
      
      * update data_feed_test for multi-type
      
      * add comment for executor_thread_worker
      
      * Remove unused code
      
      * update data_feed_test for generate test data file
      
      * removed unnecessary interfaces and add comments
      
      * c++ style check
      
      * update data_feed.cc
      
      * AsyncExecutor: C++ side
      
      Google naming conventions
      
      Rename MultiExecutor to AsyncExecutor
      
      pybind with async_executor
      
      Naming convention
      
      remove some flags and unused code
      
      add refactored file of async_executor and data_feed
      
      clear async executor interface and add data feed factory
      
      split async executor into executor_thread_worker and async_executor, refactor pybind, add datafeed and corresponding proto
      
      Fix async_executor interfaces: 1) Remove all protobufs; 2) Stop after each epoch
      
      refine async_executor_refactor.cc
      
      add some files about datafeed
      
      Revert "add some files about datafeed"
      
      This reverts commit 8ee8133a.
      
      add MultiSlotDataFeed
      
      Interface rework
      
      Creating DataFeedDesc from .proto file, then manipulate it (add/del fields etc) from python side
      
      update datafeed and async_executor to run bow_net demo
      
      update async_executor.py for support set_use_slots
      
      Fix bug
      
      update async_executor.py for support set_use_slots and set set_dense_slots
      
      fix bug that when the number of files is less than the number of threads, it will fetch nan
      
      remove redundant code, and make executor exit when set a illegal queue size
      
      add MultiSlotDesc
      
      Revert "add MultiSlotDesc"
      
      This reverts commit 2e72ebfa.
      
      add some checkpoint in DataFeedDesc
      
      Fix fetch variable
      
      fix code style in async_executor
      
      Fix fetch var
      
      add utest for data_feed
      
      Add comments for python
      
      update utest for data_feed for multithread
      
      fix bug in utest of data_feed
      
      Add comments for python code
      
      Fix pybind.cc with new pybind11 version
      
      add note for DataFeedDesc's set_use_slots function
      
      update data_feed_test for multi-type
      
      Add save_model
      
      update data_feed_test for generate test data file
      
      removed unnecessary interfaces and add comments
      
      add comment for executor_thread_worker
      
      Remove unused code
      
      update data_feed.cc
      
      c++ style check
      
      * commit for code style
      
      * commit for code style
      
      * commit for code style
      
      * commit for code style
      
      * Comment away __init__ in async_executor.py
      
      * clang-format fix test=develop
      
      * use PADDLE_THROW instead of exit(-1); use unique_ptr to manage scope var in data_feed_test.cc
      
      * commit for update code style
      
      * commit for update code style
      
      * Add async_executor demo; Remove some methods
      test=develop
      
      * commit for update code style
      
      * commit for update code style
      
      * commit for update code style
      
      * update API.spec
      
      * AsyncExecutor
      test=develop
      
      * AsyncExecutor
      test=develop
      
      * AsyncExecutor
      test=develop
      
      * AsyncExecutor
      test=develop
      
      * Fix API.spec
      test=develop
      
      * Fix API.spec
      test=develop
      
      * Fix windows build error
      test=develop
      
      * FIx windows build error
      test=develop
      
      * FIx windows build error
      test=develop
      
      * FIx windows build error
      test=develop
      
      * Fix Windows Build
      test=develop
      
      * Fix Windows Build
      test=develop
      
      * Fix Windows Build
      test=develop
      
      * Fix code style
      test=develop
      
      * Fix code style
      test=develop
      
      * update datafeed
      
      * Fix code style
      test=develop
      
      * update data_feed_test for test Tensor test=develop
      
      * Fix code style
      test=develop
      
      * Fix windows build failure
      test=develop
      
      * Fix code style and windows build failure
      test=develop
      
      * Fix PYTHON3.5 build failure
      test=develop
      
      * AsyncExecutor API
      test=develop
      41e19eb4