Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Add executable permission to `ds_elastic` and `ds_report` in `bin`.

* Automatic `ds_elastic` formatting
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* fix wrong idx bug in invertible LayerNormBackward1

this index bug cause wrong scale grad

* fix unexpected deletion

* fix idx for LayerNormBackward1_fused_add

* move pos defination in LayerNormBackward1 kernels

* fix format error
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Fix ZeRO 2 + Pipelining

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Update README.md

* Update index.md

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* move workspace memory-allocation to PyTorch

* refine the code based on the comments

* remove unnecessary options

* remove bsz from set_seq_len function

Co-authored-by: NCheng Li <pistasable@gmail.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Special thanks to @g-karthik for tracking this issue down.

* Add Linear warmup+decay lr schedule
Update lr schedule unit tests

* LR scheduler unit tests for LR Range Test and 1Cycle

* Disable yapf to preserve parameterizaton

* Disable test_pipe.py for CI debugging

* Disable test_lr_scheduler for CI debugging

* Disable test_lr_scheduler for CI debugging

* Enable all unit tests for CI debugging
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Remove a very verbose print statement.

* Update engine.py

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.10 to 1.11.0.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.10.10...v1.11.0)
Signed-off-by: Ndependabot[bot] <support@github.com>
Co-authored-by: Ndependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Allow DeepSpeed models to be initialized with optimizer=None
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>