Merge branch 'olruwase/scheduler_optimizer_bug' of github.com:microsoft/DeepSpeed into olruwase/scheduler_optimizer_bug

Merge branch 'olruwase/scheduler_optimizer_bug' of github.com:microsoft/DeepSpeed into olruwase/scheduler_optimizer_bug

* Add --no_sudo to run without sudo

* Add --pip_mirror to set the pip mirror

* Default to running pip without sudo

* Typo

* Add --pip_sudo to Dockerfile and azure-pipelines.yml
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* add support for deepspeed env file to pass custom env values

* simplify deepspeed config example

* add text about mpirun

* add mpirun support for openmpi 4.0

* add master addr support from args

* switch mpi detection to use mpi4py

* set constant for default distributed port

* Make sure deepspeed_mpi exits in args

* add auto-detect to torch dist init

* update tests to infer distributed init status

* prevent crash if dist_init_required is True but already initiliazed

* only init if safe to do so (forgot to add this file in prev commit)

* Update scripts to handle cases where you have other VMs in your sub

* Support subs with other VMs and fix for PDSH permission error

* Minor fix to support subs with other VMs

* Added shutdown with or without delete VM option

In Azure deallocate is like machine shutdown (and prevents billing). You can restart deallocated VM. To fully drop the VM delete is used. This command with "-d" option will fully delete the VM. Without any argument it justs deallocates / shutd down the VM.

* Update scripts to handle cases where you have other VMs in your sub

* Support subs with other VMs and fix for PDSH permission error

* Minor fix to support subs with other VMs

* Removes DeepSpeedDataSource

* dropping unused imports
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Support legacy optimizer fusion as config option

* Configure for legacy optimizer fusion

* Update configuration jsons for new apex

Also a fix for #94 

Co-authored-by: NShaden Smith <ShadenTSmith@gmail.com>

* add install requirements command line

* add pillow library to fix version

* modify to uppercase
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

bug fixes for adamw/lamb and corresponding tests

* Porting BingBertSquad test

* Updating default paths.

* Enable model tests.

* Updating DeepSpeedExamples submodule

* Adding BingBertSquad's log uploads.

* Messed up the submodule again :-)

* Set up CI with Azure Pipelines for docker build/push

* bump tf version in dockerfile

* Update install.sh

* Fix broken link for the 1Cycle doc.

* Removed the 1Cycle link from README.md.

* Importing 1Cycle tutorial.

* image paths

* Added LR schedule figure

* line wrap

* lowercase name

* Updating README links

* typo