Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>

* 1-bit adam (#353)
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NYour Name <you@example.com>
Co-authored-by: Ntanghl1994 <htang14@ur.rochester.edu>
Co-authored-by: NHank <tanghl1994@gmail.com>
Co-authored-by: Nroot <root@node2x12b.cs.rochester.edu>
Co-authored-by: NAmmar Ahmad Awan <awan.ammar@microsoft.com>

* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: NArash Ashari <arashari@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>

Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: NTunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NShaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: NElton Zheng <eltonz@microsoft.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Nyuxionghe <yuxhe@microsoft.com>
Co-authored-by: NArash Ashari <arashari@microsoft.com>

* Add --no_sudo to run without sudo

* Add --pip_mirror to set the pip mirror

* Default to running pip without sudo

* Typo

* Add --pip_sudo to Dockerfile and azure-pipelines.yml
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Porting BingBertSquad test

* Updating default paths.

* Enable model tests.

* Updating DeepSpeedExamples submodule

* Adding BingBertSquad's log uploads.

* Messed up the submodule again :-)

* Enables NCCL backend in @distributed_test

* Adds pytest-forked to avoid CUDA re-initialization issue.

* paste typo

* transcription typo