Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* undo noise

* another

* Fix bugs about non-contiguous tensor broadcasting

* Fix typo
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* largest_partitioned_params calculation fix

largest partitioned params was getting calculated incorrectly

* Update stage3.py

* Update stage3.py

* formatting fix

* changing sub-group size default to 1e9
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fix links for inference tutorial

* Fix automatic injection. Add the local-attention for GPT-Neo

* fix the inference for generation of large sequences (>1K & <32K)

* fix format
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fix missed subclassed partitioning bug

* fix on exit
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

fixes: s/micro_batch_per_gpu/train_micro_batch_size_per_gpu/
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Change the sparse attention API to be compatible with latest changes on the triton side

* remove compatibility checks for CUDA 11

* Update requirements-sparse_attn.txt
Co-authored-by: NArash Ashari <arashari@microsoft.com>

* delay imports for replace policies and fix missing req

* fix issue with _orig_layer_class always being None

* Fix Inference and Quantization tutorial links

* fix inference api

* use correct attention scaling
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NElton Zheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nniumanar <60243342+niumanar@users.noreply.github.com>

* fix Reduce Scatter default value

* Update constants.py
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Align fp16 param wap buffers

* Integrating swap buffer manager for fp16 params

* Support swapping misaligned fp16 parameters

* Support swap into unaligned fp16 buffer

* Align fp16 param wap buffers

* Integrating swap buffer manager for fp16 params

* Support swapping misaligned fp16 parameters

* Round robin partitioning to improve ZeRO-2 Offload CPU copy

* Formatting fixes

* Fix index issues in debug dumps

* Remove debug prints

* Code cleanup

* Remove unintended stage3.py changes

* Add TODO

* is not -> !=

* Use pytest-randomly to seed unit tests.

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* [configure_distributed_model] improve assert

This PR changes the 2 asserts to actually print the names of the params that are wrong. e.g.:
```
fp16 is enabled but the following parameters have dtype that is not fp16: wav2vec2.masked_spec_embed
```

* style
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* use the original function's name as the key to old_functions dict

* update profile output format

* print at global rank 0

* add flops calculation in bwd pass using time from ds timers

* improve aggregated profiling out to show all depth

* print samples/second

* update readme and examples

* update docs

* fix typo and reorder printing

* fix format

* Unused parameters assert should be disabled by default

* Fix message

* Invert assert logic in unit test

* Change option for ignoring unused parameters
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* NVMe intra-request validation should be on entire file
Optimizer swap buffer sizes should be aligned

* Add fix message for missing aio lib error.

* fix assert

The current assert "Model must initialized in fp16 mode for ZeRO Stage 3." needs TLC - I rewrote it completely to match its cousen assert, so now we have 2 consistent matching asserts:

- f"fp16 is enabled but one or several model parameters have dtype that is not fp16"
- f"fp16 is not enabled but one or several model parameters have dtype of fp16"

* remove f

Fix #1032

* Add additional conditions when checking types of output from the model

* Add test

* Modify test to use torch.tensor as well
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Co-authored-by: NJiangang Zhu <jiangazh@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>