* add mpirun support for openmpi 4.0

* add master addr support from args

* switch mpi detection to use mpi4py

* set constant for default distributed port

* Make sure deepspeed_mpi exits in args

* add auto-detect to torch dist init

* update tests to infer distributed init status

* prevent crash if dist_init_required is True but already initiliazed

* only init if safe to do so (forgot to add this file in prev commit)

* Removes DeepSpeedDataSource

* dropping unused imports
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Support legacy optimizer fusion as config option

* Configure for legacy optimizer fusion

* Update configuration jsons for new apex

Also a fix for #94 

Co-authored-by: NShaden Smith <ShadenTSmith@gmail.com>

bug fixes for adamw/lamb and corresponding tests

* simplifying the batch config, using a single assert to test for validity and allowing for specifying only the micro batch size

* Simplifying Batch Config, Adding ability to specify batch using just micro_batch, and adding a bunch of unit tests

* ran formatting

* Typo fixes and added the config file

* reformatting

* path fixes

* removing print statements

Unit tests for add_XXX_arguments

Co-authored-by: NShaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Fixing file permissions.

Different Optimizers in DeepSpeed.