* Re-use apex builds for AMD

* Build MI200 tests on PR

* Install on /blob for the first time

* Add the venv version

* Test

* Remove AMD tests from PR - were only needed for tessting.

* add coverage report

* define env vars in shared action

* reduce time for longest running tests

* fix broken shared action

* reduce test time

* reducing Pipeline test times

* further reducing test times

* rework Z3 test

* testing new mp.pool and persistent dist envs

* fix import

* reuse distributed environment for tests with lots of param combos

* fix for dist teardown

* fix pickling issue with pool cache

* actually fix pickling problem

* avoid running pool cache stuff on non-distributed tests

* fix issues with nested mp.pool

* fix for nested pools in Pipeline Engine

* re-add params

* update workflows with pytest opts

* implement feedback

* resolve race condition with port selection

* Update tests/unit/common.py

---------
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* utilize shorter tests for MII

* use cached torch download

* rework zero++ unit tests

* formatting

---------
Co-authored-by: NHeyangQin <heyangqin@microsoft.com>

* Disable AMD workflows in the YML

* Switch from PR to nightly so we can enable the flows here

* update docs to reflect changes in deepspeed-chat training script

* add blogs to ignored changes in unit tests

* Update AMD workflows

* Update MI200 test flow to use torch latest

* Update tolerances to values that pass (will fix before completing PR)

* Revert chyanges to atol

* Rename workflows

* Fix CI badges

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Common location to install libaio-dev

* Update .github/workflows/setup-venv/action.yml
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

---------
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

add reusable workflow that sets up fresh venv for each test and prints relevant environment info

* remove any cupy install when setting up environments

* revert previous changes to run on cu111 runners

* fix for when no cupy is installed

* remove cupy uninstall for workflows not using latest torch version

* update to cu116 for inference tests

* fix pip uninstall line

* move python environment list to after DS install

* remove cupy uninstall

* re-add --forked

* fix how we get cupy version (should be based on nvcc version)

* batch of refactored tests

* more test refactoring

* fp16 test refactor

* more refactors

* added DistributedFixture class

* applied DistributedFixture to first batch of tests as a trial

* added DistributedFixture test and documentation

* last tests

* fixes for refactored tests

* remove subdirs in workflow files

* fix pytest syntax error

* fix another syntax error

* update imports

* use DistFixture with elastic checkpoint test

* missing import

* update to shared class tmpdir for elastic test

* moved test files

* avoid duplicate test file name

* last refactor and moving test files

* formatting

* fix broken import

* testing forked AMD tests

* update abstract method

* use blob storage for accelerate and transformers tests

* upgrade torch for acclerate CI
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NSam Ade Jacobs <samjacobs@microsoft.com>

Refactor Distributed unit tests

re-enable AMD CI with some modifications

* assert no FP16 with AMD CPUs

* add unit test for AMD assert error

* missing import

* downgrade assert to warning

Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* added concurrency to github actions

* fixed problem where one workflow can cancel another workflow

* added tmp file to help test concurrency policy on CI

* removed tmp file to finish testing concurrency policy

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>