• W
    AsyncExecutor (#14627) · 41e19eb4
    Wang Guibao 提交于
    * AsyncExecutor: C++ side
    
    * Google naming conventions
    
    * Rename MultiExecutor to AsyncExecutor
    
    * pybind with async_executor
    
    * Naming convention
    
    * remove some flags and unused code
    
    * add refactored file of async_executor and data_feed
    
    * clear async executor interface and add data feed factory
    
    * split async executor into executor_thread_worker and async_executor, refactor pybind, add datafeed and corresponding proto
    
    * Fix async_executor interfaces: 1) Remove all protobufs; 2) Stop after each epoch
    
    * refine async_executor_refactor.cc
    
    * add some files about datafeed
    
    * Revert "add some files about datafeed"
    
    This reverts commit 8ee8133a.
    
    * Interface rework
    
    * add MultiSlotDataFeed
    
    * Creating DataFeedDesc from .proto file, then manipulate it (add/del fields etc) from python side
    
    * update data_feed for add MultiSlotDataFeed
    
    * update datafeed and async_executor to run bow_net demo
    
    * fix bug that finish_set_filelist failed in multithread
    
    * delete finish_binding_memory_(flag), because it can not be marked under the current interface
    
    * Fix bug
    
    * update async_executor.py for support set_use_slots
    
    * update async_executor.py for support set_use_slots and set set_dense_slots
    
    * fix bug that when the number of files is less than the number of threads, it will fetch nan
    
    * remove redundant code, and make executor exit when set a illegal queue size
    
    * add batch_size check
    
    * add MultiSlotDesc
    
    * Revert "add MultiSlotDesc"
    
    This reverts commit 2e72ebfa.
    
    * add some checkpoint in DataFeedDesc
    
    * add CheckFile function in MultiSlotDataFeed
    
    * update something error info
    
    * fix deaded lock bug
    
    * Fix fetch variable
    
    * Merge error
    
    * fix code style in async_executor
    
    * using one lock blocking queue replace two lock blocking queue because of some bugs
    
    * update code style
    
    * add utest for data_feed
    
    * Fix fetch var
    
    * update utest for data_feed for multithread
    
    * update SetFileList info
    
    * fix bug in utest of data_feed
    
    * Add comments for python
    
    * Add comments for python code
    
    * Fix pybind.cc with new pybind11 version
    
    * add note for DataFeedDesc's set_use_slots function
    
    * Add save_model
    
    * update data_feed_test for multi-type
    
    * add comment for executor_thread_worker
    
    * Remove unused code
    
    * update data_feed_test for generate test data file
    
    * removed unnecessary interfaces and add comments
    
    * c++ style check
    
    * update data_feed.cc
    
    * AsyncExecutor: C++ side
    
    Google naming conventions
    
    Rename MultiExecutor to AsyncExecutor
    
    pybind with async_executor
    
    Naming convention
    
    remove some flags and unused code
    
    add refactored file of async_executor and data_feed
    
    clear async executor interface and add data feed factory
    
    split async executor into executor_thread_worker and async_executor, refactor pybind, add datafeed and corresponding proto
    
    Fix async_executor interfaces: 1) Remove all protobufs; 2) Stop after each epoch
    
    refine async_executor_refactor.cc
    
    add some files about datafeed
    
    Revert "add some files about datafeed"
    
    This reverts commit 8ee8133a.
    
    add MultiSlotDataFeed
    
    Interface rework
    
    Creating DataFeedDesc from .proto file, then manipulate it (add/del fields etc) from python side
    
    update datafeed and async_executor to run bow_net demo
    
    update async_executor.py for support set_use_slots
    
    Fix bug
    
    update async_executor.py for support set_use_slots and set set_dense_slots
    
    fix bug that when the number of files is less than the number of threads, it will fetch nan
    
    remove redundant code, and make executor exit when set a illegal queue size
    
    add MultiSlotDesc
    
    Revert "add MultiSlotDesc"
    
    This reverts commit 2e72ebfa.
    
    add some checkpoint in DataFeedDesc
    
    Fix fetch variable
    
    fix code style in async_executor
    
    Fix fetch var
    
    add utest for data_feed
    
    Add comments for python
    
    update utest for data_feed for multithread
    
    fix bug in utest of data_feed
    
    Add comments for python code
    
    Fix pybind.cc with new pybind11 version
    
    add note for DataFeedDesc's set_use_slots function
    
    update data_feed_test for multi-type
    
    Add save_model
    
    update data_feed_test for generate test data file
    
    removed unnecessary interfaces and add comments
    
    add comment for executor_thread_worker
    
    Remove unused code
    
    update data_feed.cc
    
    c++ style check
    
    * commit for code style
    
    * commit for code style
    
    * commit for code style
    
    * commit for code style
    
    * Comment away __init__ in async_executor.py
    
    * clang-format fix test=develop
    
    * use PADDLE_THROW instead of exit(-1); use unique_ptr to manage scope var in data_feed_test.cc
    
    * commit for update code style
    
    * commit for update code style
    
    * Add async_executor demo; Remove some methods
    test=develop
    
    * commit for update code style
    
    * commit for update code style
    
    * commit for update code style
    
    * update API.spec
    
    * AsyncExecutor
    test=develop
    
    * AsyncExecutor
    test=develop
    
    * AsyncExecutor
    test=develop
    
    * AsyncExecutor
    test=develop
    
    * Fix API.spec
    test=develop
    
    * Fix API.spec
    test=develop
    
    * Fix windows build error
    test=develop
    
    * FIx windows build error
    test=develop
    
    * FIx windows build error
    test=develop
    
    * FIx windows build error
    test=develop
    
    * Fix Windows Build
    test=develop
    
    * Fix Windows Build
    test=develop
    
    * Fix Windows Build
    test=develop
    
    * Fix code style
    test=develop
    
    * Fix code style
    test=develop
    
    * update datafeed
    
    * Fix code style
    test=develop
    
    * update data_feed_test for test Tensor test=develop
    
    * Fix code style
    test=develop
    
    * Fix windows build failure
    test=develop
    
    * Fix code style and windows build failure
    test=develop
    
    * Fix PYTHON3.5 build failure
    test=develop
    
    * AsyncExecutor API
    test=develop
    41e19eb4
test_async_executor.py 5.1 KB