Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Refactor distributed tests: checkpointing
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

* Correctly detect offload configuration

* Correctly detect offload configuration

* Handle deprecated cpu offload setting

* Correcly detect zero_offload setting

* Minor tweak
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* Fix the tensor-slicing copy for qkv parameters

* remove the random-generator from context during inference

* formatting
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* add opt replace policy

* simplify inf. api

* fix opt replace policy

* fix use-cash & add relu

* Add support of custom MLP act. function

* Revert "simplify inf. api"

This reverts commit 9e910fcbd5471dec9b3c92008426f5ba590bf0b6.

* fix the inference API (temp. solution)

* fix code formatting

* add unit tests for OPT models.

* refactor pre-attention layer norm configuration

* add support of opt-350m model

* refactor the HF model config initialization

* fix hf model config issue
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

* print warning only once.

* add support for torch param and only warn on gpu 0

* remove type checking. will be done on a new PR with more tests.
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* add cuda 11.7

* formatting

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NConglong Li <conglong.li@gmail.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Refactor Distributed unit tests

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Adding additional instructiosn in the compression tutorial on pre-training distillation and quantization for GPT (#2197)
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Add gradient_average flag support for sparse grads

* formatting fixes

* Add tests
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Fix typos.

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

* Match compute and reduce dtype

* Unit tests
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

re-enable AMD CI with some modifications

Co-authored-by: NCheng Li <pistasable@gmail.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Fix for distributed tests
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* tensor parallelism for mixture of experts
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* Split parameter offload from z3

* Format fixes

* Bug fixes

* Cleanup

* Remove dead code

* Release swap buffers for persisted params

* Format fixes

* Format fixes

* Pass args correctly

* Use pinned memory for nvme offload

* Merge with masster

* Fix missing import

* model pesistence params

* Fix merge issues

* Handle none device

* Usse log_dist