* Add flexibility of pipeline module and engine

* Separate PRs

* Separate PRs
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Fix spelling errors in inference tutorial

* Remove unused imports in inference tutorial

* Fix inference tutorial code to work with 1 GPU

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NShaden Smith <shaden.smith@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NSamyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Neltonzheng <eltonz@microsoft.com>
Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>

Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.11.4 to 1.12.5.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.11.4...v1.12.5)

---
updated-dependencies:
- dependency-name: nokogiri
  dependency-type: indirect
...
Signed-off-by: Ndependabot[bot] <support@github.com>
Co-authored-by: Ndependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* install HF w. dev extra to get all required packages

* switch ds.init to use param dict instead of json file on disk

* switch back to 'testing' extra

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* [zero_to_fp32] fix padding removal

* style

* fix comments
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NReza Yazdani <reyazda@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* [zero Init] fix regression

* clean up the warning

updated classifiers

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* pass GAS boundary state from PP -> ZeRO

* formatting
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Added 4-byte alignment on NCCL/RCCL

* pre-commit formatting fixes

* Fix for checkpoint loading with optimizer partitioning

* Better assert print

* Added unit tests for nccl/rccl 4-byte alignment

* bug

* Updated alignment to implicit
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Added drop_last to DeepSpeedDataLoader

This solves issue #326

* Updated drop_last in engine.py

added drop_last as a ds_config as mentioned by @tjruwase

* Update engine.py

* Update engine.py

* updated config.py and constants.py

* Update constants.py

* added dataloader_ prefix

* Update dataloader.py

* corrected yapf test errors

* Update test_data.py

Added dataloader_drop_last unit test

* Corrected yapf and formatting issues

* updated simple_model.py and test_data.py

* Update simple_model.py

* pre-commit fix

* corrected issues

* Update test_data.py

* Update test_data.py

* Update test_data.py

* Update test_data.py

* removed batch_size from test_data.py

* Update simple_model.py

* Update test_data.py

* Update test_data.py

* Fix unit test issues

* Use fp32 to make things work
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Remove the wrong function with duplicate name

* fix format.

* add mpu check. fix tests.

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>