* Added drop_last to DeepSpeedDataLoader

This solves issue #326

* Updated drop_last in engine.py

added drop_last as a ds_config as mentioned by @tjruwase

* Update engine.py

* Update engine.py

* updated config.py and constants.py

* Update constants.py

* added dataloader_ prefix

* Update dataloader.py

* corrected yapf test errors

* Update test_data.py

Added dataloader_drop_last unit test

* Corrected yapf and formatting issues

* updated simple_model.py and test_data.py

* Update simple_model.py

* pre-commit fix

* corrected issues

* Update test_data.py

* Update test_data.py

* Update test_data.py

* Update test_data.py

* removed batch_size from test_data.py

* Update simple_model.py

* Update test_data.py

* Update test_data.py

* Fix unit test issues

* Use fp32 to make things work
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Remove the wrong function with duplicate name

* fix format.

* add mpu check. fix tests.

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Rename PA_TO_cpu

* Code cleanup

* Revert accidental change

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* add more synchronizations and barriers for resolving gpu-halt issue

* removing unuseful broadcasts

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Callable option for optimizer and scheduler

* Add unit test

* Formatting

* Disable debug prints

* Use base optimizer to construct lr scheduler

* Formatting

* Remove dead import

* restore fp16 params if no zero ckpts available

* formatting

Co-authored-by: NAlex Muzio <Alex.Muzio@microsoft.com>
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: NConglong Li <conglong.li@gmail.com>
Co-authored-by: NFelipe Cruz Salinas <Andres.Cruz@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NShaden Smith <shaden.smith@microsoft.com>
Co-authored-by: NYoung Jin Kim <youki@microsoft.com>
Co-authored-by: Nbapatra <bapatra@microsoft.com>
Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NShaden Smith <shaden.smith@microsoft.com>
Co-authored-by: NYoung Jin Kim <youki@microsoft.com>

Co-authored-by: NConglong Li <conglong.li@gmail.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Use correct input size for splits

* Use smarter partitioning

Make add operation inplace. Without it momentum decays to zero and training has no effect on corresponding parameters

* fix empty grad zero tests

* dont clear grads in stage 1 code path

* prevent none grads from being reduced

* Fix docstring

* Make screenshots clickable for easier viewing

* Navigation menu in alphabetical order; More clicable screenshots

* Rename 1Cycle doc

* Tweak naming

* Remove no longer used flag

* ZeRO3 Offload release

* Single GPU results

* Rearrange figures

* Single GPU text

* tweak intro

* zero3-offload section

* Add asynchronous i/o docs

* Fix print_per_steps doc

* Document round_robin_gradients

* Tweak description

* Trigger CI

* aio: test for libaio with various package managers

* aio: note typical tool used to install libaio package

* setup: abort with error if cannot build requested op

* setup: define op_envvar to return op build environment variable

* setup: call is_compatible once for each op

* setup: only print suggestion to disable op when its envvar not set

* setup: add method to abort from fatal error

* Revert "setup: add method to abort from fatal error"

This reverts commit 0e4cde6b0a650591c3fafface7e27b4efd9aad4f.

* setup: add method to abort from fatal error
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Make round robin gradient partitioning configurable (default False)

* Use the correct default

* Log config setting

Co-authored-by: NIvan Komarov <dfyz@yandex-team.ru>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fixing inference api for FP32 and non-masking GPT-based models

* use a dummy tensor if input_mask is none

* fix input_mask

* minor fix

* send input_mask to compute_attn func for checking

* zero_param_shapes: switch to round_robin_fp16_groups

* add test

* old torch workaround

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* enable async io op on powerpc architectures

* drop any empty strings returned by cxx_args
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>