Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NReza Yazdani <reyazda@microsoft.com>

* Update zero.md

Update to ZeRO tutorial to specify the use of activation checkpointing

* Update zero-offload.md

Use activation checkpointing with ZeRO-Offload
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Progressive layer dropping docs (#499)

* test

* Adding tutorial and news page for pld

* updating the tutorial and posts of PLD

* update the finetune tutorial

* Update PLD tutorial (#512)

* Update installation instructions

* Format fix

* ZeRO tutorial

* Format fixes

* ZeRO-Offload

* ZeRO and ZeRO-Offload tutorials

* Update navigation page

* Format fixes

* Add yuxhe feedback

* Fix blog post link

* Fix OneBit-Adam link
Tweak scheduler example

* Fix date link

* Add DeepSpeed_Adam

* Add PLD tutorial to navigation
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* updating the pld docs

* DeepSpeed implementation of PLD (#508)

* DeepSpeed implementation of PLD

* Format fixes

* Formatting fixes

* Fix broken url

* Address PR feedback

* Bump DSE
Co-authored-by: NMinjia Zhang <33713995+minjiaz@users.noreply.github.com>
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NMinjia Zhang <minjiaz@microsoft.com>

* PLD documentation

* Formatting fixes

* Fix url bug

* PLD documentation

* Formatting fixes

* fixing cpu-adam

* fixing copy with optimizer for data and model parallelism

* fixing cpu-adam

* fix cpu-adam

* fix cpu-adam

* add adamW to CPU-ADAM implementation

* supporting cpu-adam optimizer for zero-offload on deepspeed side

* bump DSE to match cpu-adam updates
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

add compute cap of 6.0, support p100

add compute cap of 6.0 to transformer kernels

* Update installation instructions

* Format fix

* ZeRO tutorial

* Format fixes

* ZeRO-Offload

* ZeRO and ZeRO-Offload tutorials

* Update navigation page

* Format fixes

* Add yuxhe feedback

* Fix blog post link

* Fix OneBit-Adam link
Tweak scheduler example

* Fix date link

* Add DeepSpeed_Adam
Co-authored-by: NShaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Fix printing momentum for non-deepspeed optimizer

Fix printing momentum for non-deepspeed optimizer

* fix momentum access for Adam

fix momentum access for Adam

* gan tutorial

* formatting fix

* adding pointer to repo; adding navigation link
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Do not load optimizer states for fp32 if flag set to False.

* Towards Windows build

* formatting
Co-authored-by: NBruno Cabral <bruno@potelo.com.br>
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>
Co-authored-by: NOlatunji Ruwase <olruwase@microsoft.com>

* Disable default installation of CPU Adam

* Handle cpufeature import/use errors separately

* url fix

* revert absolute path but keep some actual fix

* add real readme

Co-authored-by: NConglong Li <conglong.li@gmail.com>

This reverts commit 01b6e27e.
Co-authored-by: NShaden Smith <ShadenTSmith@gmail.com>

* Activation checkpointing bugfix and unit tests.

* Activation checkpointing bugfix and unit tests.
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Update stage2.py
Co-authored-by: NJeff Rasley <jerasley@microsoft.com>

* Switches fused_optimizer overflow calculation