* Unify macro definitions and constants in a single file

* Conversion utility implementation.

* Fix reversion from formatting

* Bugfixes after testing with correct DeepSpeed

* Inline markers are available on both HIP + CUDA

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* mem access for quantize kernel

* format

* format fp32

* modify quant kernel

* modify quant kernel2

* modify format

* format

* fix comments in pytest

* fix comments in pytest

* format

* rerun
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: NConnor Holmes <connorholmes@microsoft.com>

Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* Fix formatting

* Remove redundant variable

MOE residual matmul unit tests
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* add quant unit test

* add codeowner

* format fix

* fix undefined symbol: curandSetPseudoRandomGeneratorSeed

* modify ref fn name and add comment

* add comments

* add 4bit quant 16groups

* fix

* modify groups in ref code

* parameterize tensor shape

* single param

* detach tensor

* remove -lcurand flag

* add back -lcurand flag
Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

* ZeRO-Inference blog

* ZeRO-Inference blog

* Format fixes

* Apply feedback

* Feedback

* Update docs/_posts/2022-08-27-zero-inference.md
Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>

* Update docs/_posts/2022-08-27-zero-inference.md
Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>

* Address feedback

* Format fixes

* More tweaks

* long sequence, nvme offload

* Add image
Co-authored-by: NStas Bekman <stas00@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NSam Ade Jacobs <samjacobs@microsoft.com>

* added unit test

* Update pt_binding.cpp

* formatting

* Update test_bias_add.py

Co-authored-by: Nanthony.301 <anthony.301@mri.cluster>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

when loading the non-sharded checkpoint update the progress bar (fix by @RezaYazdaniAminabadi) - I've just tested it to work.
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Refactor universal checkpointing and tensor fragments

* Formatting

Co-authored-by: NArash Bakhtiari <arash@bakhtiari.org>
Co-authored-by: NArash Bakhtiari <arashb@users.noreply.github.com>

Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>

Co-authored-by: Ncmikeh2 <connorholmes@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Nchenguo <chenguo@microsoft.com>

Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Fix RuntimeError: Boolean value of Tensor with more than one value is ambiguous when running test-gptj.py