- 24 5月, 2023 1 次提交
-
-
由 Joe Mayer 提交于
* Fixing bf16 test that was missing a config. * Chaning train_batch_size to train_micro_batch_size_per_gpu * Chaning all train_batch_size to train_micro_batch_size_per_gpu
-
- 19 5月, 2023 1 次提交
-
-
由 mzl 提交于
-
- 17 5月, 2023 2 次提交
-
-
由 Ramya Ramineni 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 Olatunji Ruwase 提交于
* Clone tensors to avoid torch.save bloat * Adddocs * Fix clang-formatting * Update docs/code-docs/source/model-checkpointing.rst Co-authored-by: NStas Bekman <stas00@users.noreply.github.com> * Update deepspeed/checkpoint/utils.py Co-authored-by: NStas Bekman <stas00@users.noreply.github.com> * Update deepspeed/checkpoint/utils.py Co-authored-by: NStas Bekman <stas00@users.noreply.github.com> * Fix url * url fix * Tweak docs --------- Co-authored-by: NStas Bekman <stas00@users.noreply.github.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
- 16 5月, 2023 4 次提交
-
-
由 Ma, Guokai 提交于
* add fallback path for kernels used in megatron * temporary numactl WA for SPR 56core * adapt core allocation according to number of ranks * add switch to turn on numactl * detect number of cores on the system * allow select a subset of the cores on the system to bind * remove unneeded changes * add ccl backend * change nccl to ccl * remove unused code * add comm/ccl to ops * initial ccl comm support * first broadcast case passed * add CCL_Backend to DeepSpeed * support comm timer for CPU * support barrier for comm backend * support specify master address from deepspeed command line * support pytorch 2.0 * remove 'block' from api * Tweak for debug Signed-off-by: NCao, Zhong Z <zhong.z.cao@intel.com> * Remove unecessary directory Signed-off-by: NCao, Zhong Z <zhong.z.cao@intel.com> * Add bf16 kernel support for inference * Add temporary torch implement for cpu inference * Add softmax ops cpu fallback for inference * bind cores to numa domain as well * merge latest change in gma/numactl * initial bf16 kernel support with fallback path * initial fallback path for bloom kernel injection * fix softmax attn mask * check KMP_AFFINITY to avoid conflict with numactl * New CCLBackend which utilize TorchBackend for initialization * rollback last change because there is result error * fix bloom injection policy TP could not work issue. injection_policy={BloomBlock: ("self_attention.dense", "mlp.dense_4h_to_h")} * Use TorchBackend to initialize CCLBackend, make behavior consistent * remove comm under deepspeed/ops * add license header * code clean up * fix format issue * remove magic number in main address * add caching support but not turn on by default * change name of inference_cuda_module to inference_module * Check for is_synchronized_device in accelerator before get Event * fix typo * Fix fallback path of softmax kernel on CUDA device for BF16 data type, because CUDA tril does not support BF16 datatype, enforce fp32 data type * add cpu backend files * change CPU_Accelerator op_builder_dir * remove cpu_kernel_path * using CPU_Accelerator on non-cuda device * fix deepspeed.op_builder => deepspeed.ops.op_builder * add alias for num_gpus: num_accelerators * allow loading cpu_builder in build stage * Assume cuda available if torch not installed * add oneccl_binding_pt to requirements * move oneccl-binding-pt to seperate requiremetns-cpu.txt * add missing file * use dependency_links in setuptools.setup() call for additional dependency links * install oneccl_bind_pt in workflows * change oneccl_bind_pt's version from 1.13 to 2.0 * use intel_exention_for_pytorch as indicator that CPU_Accelerator should be used * Add indicator for Accelerator used * change foo.c to foo.cpp * exclude 'cpu' directory in CUDA op builder reflection * add a cpu-inference workflow * run cpu-inference workflow on self-hosted instance * change cpu runs-on node to v100 node * print out python version in workflow * add verbose in pip command to understand oneccl_bind_pt install issue * update cpu-inference workflow * add a stage to detect instance instruction sets * add back bf16 support for CPU inference * enable autoTP for bloom Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * update workflow to detect cpu instruction sets * temporary WA for Intel Extension for PyTorch AVX2 instructioon set detection * change cpu-inference workflow machine to ubuntu-20.04 * add sharded checkpoint loading for AutoTP path to reduce the peak memory in initialization stage Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * enable policy for llama * use a special build ipex to test avx2 detection fix * fix format * fix test fail issue Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * fix gptj sharded checkpoint loading problem Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * return a not implemented build in get_op_builder in cpu_backend * support cpu device in tests * use cpuinfo to extract number of CPUs * use ~/tmp as transfomer cache rather than /blob/ * Add support for mpich launcher with prefer_deepspeed_comm * add missing modification in accelerator * enable IMPI launcher * remove unused file and fix formatting * clean up ccl.cpp * Less confusing error message when certin op builder are not implemented * Fix license header * Add license header * add license headers * add license header * fix cuda specific code in test * update CPU workflow * use numactl to bind to core * allow bind_cores_to_rank in multi-node impi runner * fix format error * Remove InferenceBuilder * fix format error in numa.py * check whether op is in installed ops in ds_report.py * allow override accelerator with DS_ACCELERATOR='cuda','cpu' or 'xpu' * lazy init class_dict in CUDA_Accelerator to avoid cyclic initialization of CUDA_Accelerator * put short path in the beginning in real_accelerator.py * device_count return number of NUMA nodes * fix typo * install numactl in cpu workflow * Follow comments * Better implementation of device_count() and current_device() * remove dependency_link for Intel Extension for DeepSpeed * use check is_synchronized_device in timer only once * remove env mapping WA in cpu_accelerator * fix duplicate definition * fix format error * refine ccl backend selection * move comments to the right place * remove prefer_deepspeed_comm, use CCLBackend by default * refractor fallback path * Fix execution failure in kernel injection path * do not refractory kernel injection fallback path in residual_add because it contains function call with side-effect * guard residual_add fallback path with environ DS_KI_FALLBACK=True * fix format error * add test for allreduce on CPU workflow * fix format error * Fallback to TorchBackend if CCLBackend kernel are not implemented * Update Intel Extension for Pytorch installation link * Don't specify version number of Intel Extension for PyTorch * install oneCCL for CCLBackend * fix link path for CPU comm kernels * fix source oneCCL environment * source oneCCL env before run UT * Give more specific instruction when CCL_ROOT not defined --------- Signed-off-by: NCao, Zhong Z <zhong.z.cao@intel.com> Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> Co-authored-by: Nsdp <sdp@aia-sdp-spr-108864.jf.intel.com> Co-authored-by: NCao, Zhong Z <zhong.z.cao@intel.com> Co-authored-by: NZhenhuan Chen <zhenhuan.chen@intel.com> Co-authored-by: Nbaodii <di.bao@intel.com> Co-authored-by: NWang, Yi A <yi.a.wang@intel.com> Co-authored-by: Njianan-gu <jianan.gu@intel.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Ramya Ramineni 提交于
* Add cg headers hipification * Exclude including cuda_bf16.h on ROCm * Merge * Retricting including cuda_bf16.h with BF16_AVAILABLE var --------- Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 digger yu 提交于
* fix spelling error with deepspeed/runtime/ * fix typo docs/ * fix typo in comments with deepspeed/ --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Joe Mayer 提交于
* Updating fused adam with new Apex bf16 support. * Removing capturable and master weight configs. * resolving pr comments --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 15 5月, 2023 2 次提交
-
-
由 Heyang Qin 提交于
* share inflight registry between PartitionedParameterCoordinator * bound registry to model * make InflightParamRegistry standalone * fix format --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 Yizhou Wang 提交于
* * try to fix broadcast error on multi-node training with ZeroStage3 and TensorParallel=2 * * fix format error * * fix format issue * * add TODO for integrated testing of TP and ZeRO 1/2/3 * fix default pg error --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 13 5月, 2023 4 次提交
-
-
由 mmhab 提交于
DeepSpeedZeRoOffload Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Michael Wyatt 提交于
-
由 Gavin Goodship 提交于
* Update pytorch-profiler.md * Update one-cycle.md --------- Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 digger-yu 提交于
* fix spelling error with deepspeed/runtime/ * fix typo docs/ --------- Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
- 12 5月, 2023 4 次提交
-
-
由 digger-yu 提交于
-
由 Joe Mayer 提交于
* Changing monitor loss to aggregate loss over gas. * Adding self.losses to engine constructor. --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 Molly Smith 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 digger-yu 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
- 11 5月, 2023 3 次提交
-
-
由 Lev Kurilenko 提交于
Co-authored-by: NConnor Holmes <connorholmes@microsoft.com> Co-authored-by: NJeff Rasley <jerasley@microsoft.com> Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>
-
由 Gavin Goodship 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Tian, Feng 提交于
Add snip_momentum structured pruning which can support higher sparse ratio with minor accuracy loss (#3300) Signed-off-by: NTian, Feng <feng.tian@intel.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 10 5月, 2023 5 次提交
-
-
由 Wang, Yi 提交于
fix regression in shard checkpoint loading in AutoTP Path caused by qkv_copy() is deleted and add UT case for shard checkpoint loading in AutoTP (#3457) * add UT case for shard checkpoint loading in AutoTP Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * autoTP path also support shard loading Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> --------- Signed-off-by: NWang, Yi A <yi.a.wang@intel.com>
-
由 Lev Kurilenko 提交于
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>
-
由 MrZhengXin 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>
-
由 digger-yu 提交于
* Update index.md fix spelling error * Update training.md fix spelling error --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
由 digger-yu 提交于
-
- 09 5月, 2023 5 次提交
-
-
由 YiSheng5 提交于
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 LiYu Lu 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 digger-yu 提交于
* fix some spelling error under doc/ * fix spelling error deepspeed/ --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Gavin Goodship 提交于
* Update 2020-09-09-sparse-attention.md * Update MoQ-tutorial.md --------- Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
由 Pablo Emídio S.S 提交于
-
- 08 5月, 2023 1 次提交
-
-
由 Wang, Yi 提交于
remove bloom from unsupported Models Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
- 06 5月, 2023 1 次提交
-
-
由 Yizhou Wang 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 05 5月, 2023 1 次提交
-
-
由 Wang, Yi 提交于
* add sharded checkpoint loading for AutoTP path to reduce the peak memory in initialization stage Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> * fix gptj sharded checkpoint loading problem Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> --------- Signed-off-by: NWang, Yi A <yi.a.wang@intel.com> Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-
- 04 5月, 2023 5 次提交
-
-
由 Connor Holmes 提交于
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
-
由 Michael Wyatt 提交于
-
由 jianan-gu 提交于
* Enable auto TP policy for llama model * Update automatic-tensor-parallelism.md --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com> Co-authored-by: NMolly Smith <112220543+molly-smith@users.noreply.github.com>
-
由 Jeff Rasley 提交于
-
由 Gavin Goodship 提交于
Co-authored-by: NLogan Adams <114770087+loadams@users.noreply.github.com>
-
- 03 5月, 2023 1 次提交
-
-
由 Joe Mayer 提交于
* Adding torch.optim.Adagrad * adding adagrad for zero 1 2 * Adding Adagrad support to zero 3. * Adding documentation and DeepSpeedCPUAdagrad to list. --------- Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
-