Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* added unit test for various HF model families and tasks

* formatting

* added missing import

* fixed broken pytest global vars

* modified test to conform to other test structure

* removed gpt-j. it cannot run on V100s (OOM)
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Fix do not updated sparse grads

* Remove call .data for sparse grads
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.13.4 to 1.13.6.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.13.4...v1.13.6)

---
updated-dependencies:
- dependency-name: nokogiri
  dependency-type: indirect
...
Signed-off-by: Ndependabot[bot] <support@github.com>
Co-authored-by: Ndependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* DeepSpeed needs to start cleaning up

* Remove debug prints

* [pipe] prevent deadlock with multiple evals sequence

* style

* style

* style

* align DSE commit w. latest master
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* fix step in adam

* fix backward compatibility and add unittest

* add unittest

* fix unbounded error when there are more than 1 param groups

* fix typo

* remove trailing whitespace

* fix end of file
Co-authored-by: NShuai Zheng <shzheng@amazon.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* GatheredParameters - accept any iterable

* torch tensor is an iterable, so can't use collections.abc.Iterable

* fix

* Fix OOM and type mismatch

* Toggle prefetching

* Disable z3 prefetching for inference (temp workaround)

* Fix zero3 tracing issues

* Remove debug prints

* Enable prefetch for inference

* Code clarity

* Invalidate trace cache

* Trace cache invalidation when needed
Separate nvme prefetch from all-gather prefetch

* Track last used step id

* Use debug name in error message

* Construct param trace from module trace
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NShuai Zheng <shzheng@amazon.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>