- 21 1月, 2022 2 次提交
-
-
由 Olatunji Ruwase 提交于
-
由 Justin Chiu 提交于
* Changes for bfloat16 Zero2 * ZeRO stage3 optimizations, with some bug fixes optimizations for stage3: - prefetching improvements - batching allgather calls to amortize fixed overhead and improve bandwidth utilization - batching reduce_scatter calls to amortize fixed overhead and improve bandwidth utilization - using *_base variants of allgather and reduce scatter to reduce memory allocations and data movement - more fine grained synchronization for communication that allows blocking on less work - precomputation of fetching code - using a fetch queue rather than deciding what to (pre)fetch at each iteration - limiting queued coalesced communication ops to reduce memory pressure on pytorch cuda caching allocator (not elegant solution) optimizations for stage3-offload: - made some host-device tensor copies async to improve performance bug fixes and qol improvements: - fix init context method when parent modules modify child weights - speed up model initialization by moving model to GPU before weight initialization - fixed unit test imports so that unit tests can be run from any directory - change performance logging to include memory consumption - add logging w/ model size when done partitioning model new features - bfloat16 support for ZeRO 3 * fix import in ut * ran yapf * improvements to cache flush warn log * backwards compatibility with older versions of pytorch * handle edge case where reduced tensor smaller than world size * moved event synchronization to allgather handle wait() call * removed unnecessary barrier call * formatting fix after resolving merge conflict * skip nvme prefetch when trace not complete * opportunistically avoid memory allocation in allgather coalesced where possible * fix indentation after merge * fixes to account for parameter offload * accounting for torch.cuda.memory_stats not being available * moved partition_all_params to optimizer step * allgathering on params before item gets called * fix param status checks needed after moving partition_all_parameters call to optimizer step * fix grad accumulation with optimizer offload * grad norm computation fix for optimizer offload * change post divide in reduce-scatter to pre divide * fix gradient race condition w/ optimizer offload * improve inf/nan gradient tracking * don't prefetch when not in training mode * format fix after merging * fix prefetching issue when using NVME offload * improved defragmentation for fp16 parameters * relative imports for bf16 tests * changes for bwd compatibility with pytorch 1.2 * remove buffered_reduce_fallback * removed unused parameter offset bookkeeping * fixed tracking for multiple param groups * unbroke bfloat16 config after merge conflict * using base allgather params when only 1 param * cleanup/fixes for fp16 partition defragmentation * switch to CRLF * convert to same new-line style as master * align new line with master * Fix merge issues * switch to CRLF * fix to LF line endings * minor merge fixes * remove extra bfloat16_enabled definition * asserting params inflight for AllGatherHandle * remove get_cuda_mem_allocated_str * Format fixes * fix bfloat16 zero stage check (broken after merge commit) * +self.communication_data_type, -self.allreduce_always_fp32; delete dead code * Add self.reduce_scatter * Format fix * Fix merge issues * iterate over params_to_fetch rather than make another iterator * add some TODOs * remove unnecessary division by micro_step_id * rename config keys "bfloat16" -> "bf16" * rename stage3_gather_fp16_weights_on_model_save -> stage3_gather_16bit_weights_on_model_save * add unit test to check backwards compatibility for gather_16bit_weights * added test to confirm bf16 key bwd compatibility * Format fixes Co-authored-by: NRana Ali Amjad <raamjad@amazon.com> Co-authored-by: NJustin Chiu <justchiu@amazon.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 19 1月, 2022 1 次提交
-
-
由 Jeff Rasley 提交于
Co-authored-by: NReza Yazdani <reyazda@microsoft.com> Co-authored-by: NZhewei Yao <zheweiy@berkeley.edu> Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
-
- 15 1月, 2022 1 次提交
-
-
由 Jeff Rasley 提交于
[ZeRO] Default disable elastic ckpt in stage 1+2 and reduce CPU memory overhead during ckpt load (#1525) Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 29 12月, 2021 2 次提交
-
-
由 Manuel R. Ciosici 提交于
-
由 Stas Bekman 提交于
-
- 14 12月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 01 12月, 2021 1 次提交
-
-
由 Alex Hedges 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 30 11月, 2021 1 次提交
-
-
由 Paige Wang 提交于
wall_clock_breakdown disable failed in moe layer due to incorrectly used a function as property
-
- 27 11月, 2021 1 次提交
-
-
由 Mikhail Druzhinin 提交于
* fp16 allreduce * Undo sparse sum in nan check * communication_data_type instead of fp32_allreduce and fp16_allreduce * sparse_allreduce with fp32 or fp16 data type * FIx communication_data_type checks * Allow only torch data types for communication_data_type * Fix Zero assert messages Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 26 11月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
* allow external gas * init state * add docstring * add missing engine.step Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 19 11月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
-
- 18 11月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
-
- 16 11月, 2021 1 次提交
-
-
由 Mikhail Druzhinin 提交于
Fix partial recovery of sparse_tensor_module_names and dynamically check if gradient data is sparse (#1562) Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 13 11月, 2021 2 次提交
-
-
由 Cheng Li 提交于
* [squash] Staging autotuning v4 Co-authored-by: NCheng Li <pistasable@gmail.com> Co-authored-by: NMinjia Zhang <minjiaz@microsoft.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com> * add new extra, guard xgboost, cleanup dead files (#268) * Fix autotuning docs (#1553) * fix docs * rewording the goal * fix typos * fix typos (#1556) * fix typos * fix format * fix bug (#1557) * fix bug Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NMinjia Zhang <minjiaz@microsoft.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 Olatunji Ruwase 提交于
-
- 12 11月, 2021 1 次提交
-
-
由 Conglong Li 提交于
-
- 02 11月, 2021 1 次提交
-
-
由 Rana Ali Amjad 提交于
* Changes for bfloat16 Zero2 * Cleaned up additional comments and debugging code * Adapted fp16_master_weights_and_grads option to cover BF16 * Reverted fp16_master_weights_and_gradients extension to BFloat16 and minor cleanup * Fixed formatting and variable naming errors recognized in testing * Added relevant unit tests for bfloat16 with ZeRO-2 * Updates conditions for skipping BFloat16 unit tests * Added check for NCCL inconsistent version naming convention * Update skip message for Bfloat16 tests to mention additional checks Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 28 10月, 2021 1 次提交
-
-
由 Olatunji Ruwase 提交于
* Synchronize folder creation; Single latest file writer * Address PR feedback
-
- 27 10月, 2021 2 次提交
-
-
由 Mikhail Druzhinin 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
由 Wenhao Hu 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
-
- 18 10月, 2021 1 次提交
-
-
由 Baizhou Huang 提交于
Co-authored-by: Nskpig <1900012999@pku.edu.cn>
-
- 14 10月, 2021 1 次提交
-
-
由 Ammar Ahmad Awan 提交于
-
- 09 10月, 2021 1 次提交
-
-
由 Conglong Li 提交于
* CL+PP * add TODO
-
- 07 10月, 2021 1 次提交
-
-
由 rpengms 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 05 10月, 2021 1 次提交
-
-
由 eelxpeng 提交于
* Revise param_shapes to be a list of ordered dict * test i can push * add tests; split z2 and z3 into separate funcs Co-authored-by: NXiaopeng Li <xiaopel@amazon.com> Co-authored-by: NStas Bekman <stas@stason.org> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 02 10月, 2021 1 次提交
-
-
由 Alex Hedges 提交于
* Fix typos in docs/ * Fix typos in code comments and output strings * Fix typos in the code itself * Fix typos in tests/ Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 01 10月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
-
- 30 9月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NShaden Smith <shaden.smith@microsoft.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com> Co-authored-by: Neltonzheng <eltonz@microsoft.com> Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>
-
- 23 9月, 2021 1 次提交
-
-
由 Ammar Ahmad Awan 提交于
-
- 21 9月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
-
- 16 9月, 2021 1 次提交
-
-
由 Sean Naren 提交于
-
- 10 9月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
* pass GAS boundary state from PP -> ZeRO * formatting Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 02 9月, 2021 2 次提交
-
-
由 Olatunji Ruwase 提交于
-
由 Hari Prasad 提交于
* Added drop_last to DeepSpeedDataLoader This solves issue #326 * Updated drop_last in engine.py added drop_last as a ds_config as mentioned by @tjruwase * Update engine.py * Update engine.py * updated config.py and constants.py * Update constants.py * added dataloader_ prefix * Update dataloader.py * corrected yapf test errors * Update test_data.py Added dataloader_drop_last unit test * Corrected yapf and formatting issues * updated simple_model.py and test_data.py * Update simple_model.py * pre-commit fix * corrected issues * Update test_data.py * Update test_data.py * Update test_data.py * Update test_data.py * removed batch_size from test_data.py * Update simple_model.py * Update test_data.py * Update test_data.py * Fix unit test issues * Use fp32 to make things work Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 26 8月, 2021 1 次提交
-
-
由 Olatunji Ruwase 提交于
* Callable option for optimizer and scheduler * Add unit test * Formatting * Disable debug prints * Use base optimizer to construct lr scheduler * Formatting * Remove dead import
-
- 25 8月, 2021 1 次提交
-
-
由 Jeff Rasley 提交于
* restore fp16 params if no zero ckpts available * formatting
-
- 17 8月, 2021 2 次提交
-
-
由 Ammar Ahmad Awan 提交于
Co-authored-by: NAlex Muzio <Alex.Muzio@microsoft.com> Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: NConglong Li <conglong.li@gmail.com> Co-authored-by: NFelipe Cruz Salinas <Andres.Cruz@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NReza Yazdani <reyazda@microsoft.com> Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: NShaden Smith <shaden.smith@microsoft.com> Co-authored-by: NYoung Jin Kim <youki@microsoft.com> Co-authored-by: Nbapatra <bapatra@microsoft.com> Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: NShaden Smith <shaden.smith@microsoft.com> Co-authored-by: NYoung Jin Kim <youki@microsoft.com>
-
由 Conglong Li 提交于
Co-authored-by: NConglong Li <conglong.li@gmail.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
- 29 7月, 2021 1 次提交
-
-
由 Olatunji Ruwase 提交于
-