when loading the non-sharded checkpoint update the progress bar (fix by @RezaYazdaniAminabadi) - I've just tested it to work.
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Refactor universal checkpointing and tensor fragments

* Formatting

Co-authored-by: NArash Bakhtiari <arash@bakhtiari.org>
Co-authored-by: NArash Bakhtiari <arashb@users.noreply.github.com>

Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

Co-authored-by: Ncmikeh2 <connorholmes@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nchenguo <chenguo@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Fix RuntimeError: Boolean value of Tensor with more than one value is ambiguous when running test-gptj.py

Add blob storage to CI runners and enable for transformers cache on inference tests

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Correctly detect CPU optimizer usage

* Update nv-transformers-v100.yml (#2259)
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

MoE training with zero stage 1 only works with `contiguous gradients=True`.

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Refactor distributed tests: checkpointing
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

* Correctly detect offload configuration

* Correctly detect offload configuration

* Handle deprecated cpu offload setting

* Correcly detect zero_offload setting

* Minor tweak
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* Fix the tensor-slicing copy for qkv parameters

* remove the random-generator from context during inference

* formatting
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* add opt replace policy

* simplify inf. api

* fix opt replace policy

* fix use-cash & add relu

* Add support of custom MLP act. function

* Revert "simplify inf. api"

This reverts commit 9e910fcbd5471dec9b3c92008426f5ba590bf0b6.

* fix the inference API (temp. solution)

* fix code formatting

* add unit tests for OPT models.

* refactor pre-attention layer norm configuration

* add support of opt-350m model

* refactor the HF model config initialization

* fix hf model config issue
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

* print warning only once.

* add support for torch param and only warn on gpu 0

* remove type checking. will be done on a new PR with more tests.
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* add cuda 11.7

* formatting

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NConglong Li <conglong.li@gmail.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Refactor Distributed unit tests

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>