Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Establish building block of abstract accelerator

* Change .*Tensor variable to @property

* [op builder] add op builder reflection to allow enumerate of builders in all_ops.py and builder_names.py

* change @abstractproperty to @property @abstractmethod
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* rollback ds config changes

* fix format

* Fix error when output_file is a relative path without a prefix (#2397)
Co-authored-by: NBenjamin Steenhoek <benjaminjsteenhoek@gmail.com>

* fix restuls and exprs path to use absolute path

* use base64 encoded ds config as cmd arg

* fix format

* remove assert

* write out optimial config after tuning

* fix format

* no need to update ds config path when encoding ds config

* udpate

* do not use abs path for result and expr dir

* fix conflicts

* fix run mode

* fix format

* fix format
Co-authored-by: NBenjamin Steenhoek <benjaminjsteenhoek@gmail.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* report progress at gradient accumulation boundary

* format

* format

* Adding gradient accumulation dtype config.

* Switching to new DtypeEnum

* Adding standalone check function, and unit tests

* Variable disambiguation

* Adding checks for unsupported states.

* Updating for PR comments.

* Reorganizing unit test.
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* pass down the new DS inference config to replace_transformer_layer.

* remove quantize_settings and rename the ep_mp_group.

* Fix model_config passing. Fixes gptj issue with wrong output.

* fix small bug in gpt-neo.

Co-authored-by: Reza Yazdani and Michael Wyatt

* Change utilization of DS/Triton kernels

* add config at Clip-encoder
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

A mutable default value is dangerous because editing it will change the
value for all future calls to the function. The value is itself edited
later in the function, so this problem will likely be encountered sooner
or later.
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

Adding MII tests to ensure changes to DS-Inference do not break MII

* fixes for new torch.numel return type

* address comment

* Make new InferenceConfig backwards compatible with previous init_inference API
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Initial commit Deepspeed quantization library

* Match function signatures

* Add Quantization Kernel

* adding offset comparision and precommit changes

* format fixes

* FIle name changes

* pt_binding_changes

* test name change

* Integer quantization, minor refactors

* Add directed test_case

* format fixes

* Move param calculation to constructor of params class

* Use local function and add elemsPerBlock

* change function to be specalized

* sub block reduce

* add new schedule

* Add new schedule test case

* fix illegal writes in sch1

* Style fixes in comments
Co-authored-by: NConnor Holmes <connorholmes@microsoft.com>

This PR adds a max_tokens alias to the max_out_tokens argument in the init_inference API to support backwards compatibility after the config refactor PR https://github.com/microsoft/DeepSpeed/pull/2472.

Thanks @molly-smith and @mrwyattii.

* update zero config docs
* add autogenerated docs for pydantic models used in ZeRO and Inference configs

Changes to inference API to use accept a config dict and cleaning up Inference Engine to utilize the newly added inference config.
Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

Co-authored-by: NMichael Wyatt <michaelwyatt@microsoft.com>

* fix for lm-eval nightly tests and add gpt-j to MPtest because OOM on single GPU

* add nv-nightly badge

Co-authored-by: Ncmikeh2 <connorholmes@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>

* Add scale_attn_by_inverse_layer_idx feature

* Fix layer_id bug

* Fix scaling value
Co-authored-by: NConnor Holmes <connorholmes@microsoft.com>
Co-authored-by: NReza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>

* remove any cupy install when setting up environments

* revert previous changes to run on cu111 runners

* fix for when no cupy is installed

* remove cupy uninstall for workflows not using latest torch version

* update to cu116 for inference tests

* fix pip uninstall line

* move python environment list to after DS install

* remove cupy uninstall

* re-add --forked

* fix how we get cupy version (should be based on nvcc version)

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NConnor Holmes <connorholmes@microsoft.com>

* Added MLFLOW environment variables for logging metrics within trainign script

* exporting MLFlow env variables from AML env
Co-authored-by: NCheng Li <pistasable@gmail.com>

* Updating autotune default in docs.

* Running pre-commit.

* don't gather partitioned activations for mp size 1

* add inline comment for the change
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Co-authored-by: NAmmar Ahmad Awan <ammar.awan@microsoft.com>