* support create opt in graph

* add comment
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Tensor): support assign copy interface

* feat(Module): use assign copy

* test(Tensor): add test for tensor assign_copy

* fix(Optim): create buffer data in step()

* rename assign_copy to set_data

* fix(Parameter): fix Parameter set_data bug

* refine log to log_once in data()

* test(ConsistentTensor): add set_data test

* fix(Module): fix Module.to_consistent bug

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(*): support tensor and optimizer serialization

* test(SGD): test sgd state_dict and load_state_dict

* test(Optim): test optim save and load state

* docs(Optim): add documents

* fix(Optim): recover enable_clip_grad for ParamGroup

* refine load_state_dict/state_dict inferface and add pickle file test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add sys conf

* format

* refine warmup api

* refine config

* format

* add WarmupLR

* add warmup lr

* refactor graph folder structure

* export graph

* add amp conf

* deal with lr init step and test cpu

* add WarmUpLR

* add flow config

* GradScaler and config

* rm useless code

* rm useless code

* add api_enable_cudnn_fused_normalization_add_relu

* move config

* move amp

* refactor WarmUpLrScheduler

* add comment

* add arg
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add clip_grad to optimizer

* add clip grad unit test to sgd optimizer

* add sgd graph clip grad test

* refine

* format

* refine

* fix

* merge master

* refine

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix(Optim): fix optimizer list parameters input bug

* refine Optimizer constructor

* format

* test(*): refine graph_rmsprop test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* add valid optimizer case to pass job complete

* refactor and fix optimizer

* support l2 in variable op

* merge master

* format

* fix merge

* rm useless include

* address review

* auto format by CI

* rm internal tensor check after merge master

* restore add loss after merge master
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* refine

* change abs.py for test

* refine

* refine

* refine

* refine

* refine

* refine

* add

* refeom

* refine

* refine

* refien

* refine

* refine

* refine

* refien

* add check

* refine

* refine

* refine

* skip license

* refine

* refien

* refine

* refien

* refien

* refine

* refine

* refine acos

* refien

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* refine

* refien

* refine

* refien

* refine

* refine

* refine

* refein

* refine

* remove signle client twice api

* refine

* refine

* refine

* refien

* refien

* refine

* refien

* refien

* refien

* refine

* refein

* refein

* refein

* refeine

* refein

* refien

* fix

* remove experimental of nn

* refine

* rm oneflow.experimental.F

* refein

* refine

* refien

* refein

* okokok

* remove experimental in test/modules and test/tensor

* refein

* oneflow.python => oneflow

* rm absolute_import

* remove experimental in test/dataloader

* remove experimental call in nn/modules

* remove experimental from nn/ framework/ test_utils

* remove experimental in functionalpy

* refein

* refein

* refein

* refein

* fix python reserved words

* refein

* fix alias

* refein

* refien

* refein

* refein

* refein

* refein

* refine

* refein

* refein

* refein

* add

* refein

* refein

* remove experimental of new merge

* fix

* rm

* refien

* mv setuppy

* refien

* skip fmod grad test

* remove experimental of fmod

* refein

* setup py proj

* refien

* fix error when do multi case test on graph

* remove outdated rst files

* refine and add new rst files

* refein

* check in src

* format

* revert test_fmod.py

* refien

* remove all experimental in python/nn dir (done!)

* remove experimental in test/graph

* refein

* refein

* format

* add ignore

* refein

* add linagl

* refei

* refine rst

* update build whl

* fix

* refein

* refein

* refine

* refine

* auto format by CI

* add license

* add step

* refine

* refien

* refein

* update paths

* updatte paths

* refine

* reifne

* refein

* refein

* refein

* rrefien

* refien

* refein

* refine scatter_nd

* refine oneflow.rst

* refein

* from . import linalg
from . import autograd
from . import optim

* refein

* refine load_minst's docstring

add image.rst module.rst

refine rst files

* reformat

* remove Graph api

* fix file name

* refien

* fix oneflow.nn.image

* import oneflow.nn.image

* refein

* fix from . import lr_scheduler

* fix

* refeine

* refein

* refein

* rm todo

* rm unused box

* refein

* refien

* refein

* refien

* add ignore

* fix distributed api

* refein

* refein

* refine

* refine

* fix doctest

* refein

* update

* refien

* refein

* refein

* refein

* refein

* refein

* fix colition

* refien

* refien

* fix doc

* fix module doc

* fmt

* auto format by CI

* create moduels

* fix

* fix

* auto format by CI

* fix proto problem

* update cmake

* refien

* refein

* refienrefein

* refien

* refein

* add ignore

* refein

* refein

* refien

* auto format by CI

* refein

* refei

* refein

* refien

* refein

* udpate commit

* add ignore

* fix

* refein

* refein

* fix saved_model

* rjefein

* refein
Co-authored-by: NGehangZhang <649975329@qq.com>
Co-authored-by: NYaoChi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nstrint <xiaoyulink@gmail.com>

* change doctest code in abs.py

* del >>> flow.enable_eager_execution() in  modules/abs.py

* del @experimental_api

* del @experimental_api

* slove doctest not pass

* change abs.py for test

* refine acos

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* remove signle client twice api

* refine

* remove experimental of nn

* okokok

* remove experimental in test/modules and test/tensor

* remove experimental in test/dataloader

* remove experimental call in nn/modules

* remove experimental from nn/ framework/ test_utils

* remove experimental in functionalpy

* remove experimental of new merge

* skip fmod grad test

* remove experimental of fmod

* fix error when do multi case test on graph

* remove outdated rst files

* refine and add new rst files

* format

* revert test_fmod.py

* remove all experimental in python/nn dir (done!)

* remove experimental in test/graph

* add linagl

* refine rst

* refine scatter_nd

* refine oneflow.rst

* refine load_minst's docstring

add image.rst module.rst

refine rst files

* reformat

* remove Graph api

* fix file name

* fix index
Co-authored-by: NYaoChi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nstrint <xiaoyulink@gmail.com>
Co-authored-by: Ntsai <jackalcooper@gmail.com>

* fix upsample nearest bug

* Enhance the difficulty of test samples

* fix

* fix

* fix

* fix upsample nearest bug (#5347)
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix upsample bilinear bug

* fix test

* recover code

* align with pytorch

* redesign upsample bilinear

* fix align corner bug

* align bilinear with pytorch

* enchaned 4-dim samples

* fix conflict

* add backward test

* fix bug

* complete grad test

* fix merge conflict

* delete handwritten interpolate

* bypass pytorch interpolate bug in test

* add tensor grad assignment

* add test

* revert upsample changes

* fix TypeError: set_grad(): incompatible function arguments.  bug

* add zero_grad() docstring

* formmat comment

* fix docstring

* fix suggestions

* Update oneflow/python/test/tensor/test_tensor.py
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>

* Update oneflow/api/python/framework/tensor.cpp
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>

* enhance grad setter

* refine zero_grad() docstring and add slice docstring declaration to rst

* adjust optim.rst order

* refine comment

* refine zero_grad() docstring

* fix bug
Co-authored-by: NBBuf <1182563586@qq.com>
Co-authored-by: NXiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>

* added flatten backward

* flatten and softmax backward

* fix bug for not supporting all kinds of iterables in optimizers
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(*): export Optimizer and _LRScheduler

* fix(*): import module

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tensor.zeros_() and soft sync stream instr
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* separate cpu and gpu version of SoftSyncStream

* Remove SyncAutoMemset

* fix compile error
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix wrong parallel_desc()
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* remove unused code
Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* style(Optim): add warning when no param update

* style(Optim): add TODO
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): support custom parameters for optimizer

* feat(Adam): adam support custom parameters

* feat(Adamw): adamw support custom parameters

* feat(RMSprop): rmsprop support custom parameters

* style(Optim): refine adam and adamw
Co-authored-by: NXiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

* refactor(Optim): refine optimizer codes

* docs(SGD): add document for SGD

* docs(SGD): fix code

* test(Adam): fix test_optim_adam bug
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): add sgd

* add adam optimizer

* add adam optimizer

* add adam optimize module

* add adam optimizer module

* fix adam optimizer module bug

* add adam optimizer

* file format

* add adam optimizer docs

* fix comment

* fix comment

* style(*): format codes
Co-authored-by: Nwyg1997 <wyg19970408@gmail.com>

* feat(Optim): add sgd

* fix(*): add no_grad guard in optimizer

* fix(*): fix bug in optimizer test

* style(*): format codes

* style(Optimizer): refine codes

* style(Optimizer): refine codes

* fix(Optimizer): fix bug in ParamGroup

* fix(*): add __init__.py
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>