* contiguous gradients should be set to True by default

* Set contiguous gradients to True by default

Features such as reduce_scatter depends on contiguous gradients being True. This is also the preferred default configuration.

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* undo noise

* another

* Fix bugs about non-contiguous tensor broadcasting

* Fix typo
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Fix docstring

* Make screenshots clickable for easier viewing

* Navigation menu in alphabetical order; More clicable screenshots

* Rename 1Cycle doc

* Tweak naming

* Remove no longer used flag

* ZeRO3 Offload release

* Single GPU results

* Rearrange figures

* Single GPU text

* tweak intro

* zero3-offload section

* Add asynchronous i/o docs

* Fix print_per_steps doc

* largest_partitioned_params calculation fix

largest partitioned params was getting calculated incorrectly

* Update stage3.py

* Update stage3.py

* formatting fix

* changing sub-group size default to 1e9
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Add `import os` to inference tutorials

* assign deepspeed-initialized model to hf model

* Add Windows support in README, use c++17 on Windows to support latest vc build tool

* Add detailed cpp build tools version in README
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* unit test for bugfix #1135

* formatter

* fix test in presence of mpi4py
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* fix links for inference tutorial

* Fix automatic injection. Add the local-attention for GPT-Neo

* fix the inference for generation of large sequences (>1K & <32K)

* fix format
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* remove old unit tests, should have been removed in rebase

* formatting

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fix missed subclassed partitioning bug

* fix on exit
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* allow minor cuda version differences

* cleanup

* typo

fixes: s/micro_batch_per_gpu/train_micro_batch_size_per_gpu/
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Change the sparse attention API to be compatible with latest changes on the triton side

* remove compatibility checks for CUDA 11

* Update requirements-sparse_attn.txt
Co-authored-by: NArash Ashari <arashari@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* delay imports for replace policies and fix missing req

* fix issue with _orig_layer_class always being None

* update with inference refs

* updates