Fix missing connections from --launcher_args to Slurm srun command.

* Add user defined launcher args for PDSH launcher

* Formatting fixes

* Integrate accelerator abstraction interface into deepspeed/

* Fix error message in fp16/fused_optimizer

* fix error message in fp16/unfused_optimizer.py

* assign get_accelerator().pin_memory() result to input Tensor name

* no need to check cuda and whether nvtx supported

* move try-except into inner most block

* call Event() and Stream() in get_accelerator() for data type

* Make Stream and Event as properties of abstract interface so they can be used as data type in deepspeed

* Apply op_builder backend api change from #2705 from @jeffra

* fix tests where Builder NAME is used

* keep original ...Builder.NAME interface instead of ...Builder().NAME interface

* fix builder closure for installation

* fix randomltd builder

* add comments to clarify create_op_builder and get_op_builder

* fix compatibility with pip install -e
Co-authored-by: NCheng Li <pistasable@gmail.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

Signed-off-by: NDashiell Stander <dstander@protonmail.com>
Co-authored-by: NDashiell Stander <dashiell@ip-172-31-45-20.ec2.internal>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NArpan Jain <t-arpanjain@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Add '-S' argument to pdsh command to return the largest error code from the ssh sessions

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NShuai Zheng <shzheng@amazon.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

force set lf instead of crlf (https://github.com/pre-commit/pre-commit-hooks#mixed-line-ending) (#1598)

* 1-bit adam (#353)
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NYour Name <you@example.com>
Co-authored-by: Ntanghl1994 <htang14@ur.rochester.edu>
Co-authored-by: NHank <tanghl1994@gmail.com>
Co-authored-by: Nroot <root@node2x12b.cs.rochester.edu>
Co-authored-by: NAmmar Ahmad Awan <awan.ammar@microsoft.com>