Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fix missed subclassed partitioning bug

* fix on exit
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* allow minor cuda version differences

* cleanup

* typo

fixes: s/micro_batch_per_gpu/train_micro_batch_size_per_gpu/
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Change the sparse attention API to be compatible with latest changes on the triton side

* remove compatibility checks for CUDA 11

* Update requirements-sparse_attn.txt
Co-authored-by: NArash Ashari <arashari@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* delay imports for replace policies and fix missing req

* fix issue with _orig_layer_class always being None

* update with inference refs

* updates

* Fix Inference and Quantization tutorial links

* fix inference api

* use correct attention scaling
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NElton Zheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>

* fix Reduce Scatter default value

* Update constants.py
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Align fp16 param wap buffers

* Integrating swap buffer manager for fp16 params

* Support swapping misaligned fp16 parameters

* Support swap into unaligned fp16 buffer

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.11.0 to 1.11.4.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.11.0...v1.11.4)
Signed-off-by: Ndependabot[bot] <support@github.com>
Co-authored-by: Ndependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Align fp16 param wap buffers

* Integrating swap buffer manager for fp16 params

* Support swapping misaligned fp16 parameters

* Round robin partitioning to improve ZeRO-2 Offload CPU copy

* Formatting fixes

* Fix index issues in debug dumps

* Remove debug prints

* Code cleanup

* Remove unintended stage3.py changes

* Add TODO

* is not -> !=

* Use pytest-randomly to seed unit tests.

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* [configure_distributed_model] improve assert

This PR changes the 2 asserts to actually print the names of the params that are wrong. e.g.:
```
fp16 is enabled but the following parameters have dtype that is not fp16: wav2vec2.masked_spec_embed
```

* style
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Fix docstring

* Make screenshots clickable for easier viewing

* Navigation menu in alphabetical order; More clicable screenshots

* Rename 1Cycle doc

* Tweak naming

* Remove no longer used flag

* ZeRO3 Offload release

* Single GPU results

* Rearrange figures

* Single GPU text

* tweak intro

* zero3-offload section

* Add asynchronous i/o docs

* use the original function's name as the key to old_functions dict

* update profile output format

* print at global rank 0

* add flops calculation in bwd pass using time from ds timers

* improve aggregated profiling out to show all depth

* print samples/second

* update readme and examples

* update docs

* fix typo and reorder printing

* fix format

* rename train_step_batch_size to train_micro_batch_size_per_gpu

* clarify batch_size related doc

* Validate env; remove write size from logs

* Performance scripts for auto-tunining/auto-generating aio params of
deepspeed config.

* Formatting fixes

* Address feedback

* Unused parameters assert should be disabled by default

* Fix message

* Invert assert logic in unit test

* Change option for ignoring unused parameters
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Added community tutorials to README

* Update README.md

* NVMe intra-request validation should be on entire file
Optimizer swap buffer sizes should be aligned

* Add fix message for missing aio lib error.

Previously we would bump release version then push release, but then any unreleased commits in master are versioned to the previous release which isn't correct. Instead we will now push to PyPI then bump to next version for un-released commits.

* fix assert

The current assert "Model must initialized in fp16 mode for ZeRO Stage 3." needs TLC - I rewrote it completely to match its cousen assert, so now we have 2 consistent matching asserts:

- f"fp16 is enabled but one or several model parameters have dtype that is not fp16"
- f"fp16 is not enabled but one or several model parameters have dtype of fp16"

* remove f

Fix #1032