* support create opt in graph

* add comment
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rewrite pow and logical functional apis.

* bugfix

* Fix logical_or and logical_and unittest

* Rewrite clip

* Fix logical_or and logical_and unittest

* Fix clamp

* Fix logical_xor unittest

* reformat

* Rewrite scatter and scatter_add

* Rewrite fmod and where

* Rename clip functor by clamp

* Rename oneflow.F module to oneflow._C

* Fix merge

* Fix doc

* auto format by CI

* use flow._C

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: NShenghang Tsai <jackalcooper@gmail.com>

* add sys conf

* format

* refine warmup api

* refine config

* format

* add WarmupLR

* add warmup lr

* refactor graph folder structure

* export graph

* add amp conf

* deal with lr init step and test cpu

* add WarmUpLR

* add flow config

* GradScaler and config

* rm useless code

* rm useless code

* add api_enable_cudnn_fused_normalization_add_relu

* move config

* move amp

* refactor WarmUpLrScheduler

* add comment

* add arg
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove single_client api files

* Remove Single-Client API in oneflow default python

* revert oneflow.env

* refine docs

* fix watcher and delete useless python callback

* Remove DestroyGlobalWatcher

* Revert test_deconv for CI

* fix test tol
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Create VariableOp for free EagerTensor in Graph build

* AddFreeEagerTensorToVariableOp

* flow.Tensor and flow.tensor Created ONLY by Eager EmptyOp

* StoreFreeEagerTensorWithNameByGraphName

* Remove NOTE

* add test graph free tensor script

* refine note

* lazy_grad -> lazy_gard

* refine lazy mode guard

* Add True eager tensor test case
Co-authored-by: Nstrint <xiaoyulink@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* pass test on linear with training

* Refactor RuntimeCtx for multi-runtime

* refactor inplace to support nn graph

* block support iterator

* block iter add check

* fix scalar_mul op conf build

* deal with inplace after merge master

* add alexnet graph test

* add cpu test and format

* cout to glog

* deal with Job run finish bug

* refactor lazy deal with inplace

* deal with 0D tensor

* update data path

* address review

* deal with lazy default attr

* mv according to ci

* merge master

* fix for ci

* fix for ci limit

* block proxy func

* support module custom func and refacotr get attr of block

* auto format by CI
Co-authored-by: Nchengtbf <472491134@qq.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* refine

* change abs.py for test

* refine

* refine

* refine

* refine

* refine

* refine

* add

* refeom

* refine

* refine

* refien

* refine

* refine

* refine

* refien

* add check

* refine

* refine

* refine

* skip license

* refine

* refien

* refine

* refien

* refien

* refine

* refine

* refine acos

* refien

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* refine

* refien

* refine

* refien

* refine

* refine

* refine

* refein

* refine

* remove signle client twice api

* refine

* refine

* refine

* refien

* refien

* refine

* refien

* refien

* refien

* refine

* refein

* refein

* refein

* refeine

* refein

* refien

* fix

* remove experimental of nn

* refine

* rm oneflow.experimental.F

* refein

* refine

* refien

* refein

* okokok

* remove experimental in test/modules and test/tensor

* refein

* oneflow.python => oneflow

* rm absolute_import

* remove experimental in test/dataloader

* remove experimental call in nn/modules

* remove experimental from nn/ framework/ test_utils

* remove experimental in functionalpy

* refein

* refein

* refein

* refein

* fix python reserved words

* refein

* fix alias

* refein

* refien

* refein

* refein

* refein

* refein

* refine

* refein

* refein

* refein

* add

* refein

* refein

* remove experimental of new merge

* fix

* rm

* refien

* mv setuppy

* refien

* skip fmod grad test

* remove experimental of fmod

* refein

* setup py proj

* refien

* fix error when do multi case test on graph

* remove outdated rst files

* refine and add new rst files

* refein

* check in src

* format

* revert test_fmod.py

* refien

* remove all experimental in python/nn dir (done!)

* remove experimental in test/graph

* refein

* refein

* format

* add ignore

* refein

* add linagl

* refei

* refine rst

* update build whl

* fix

* refein

* refein

* refine

* refine

* auto format by CI

* add license

* add step

* refine

* refien

* refein

* update paths

* updatte paths

* refine

* reifne

* refein

* refein

* refein

* rrefien

* refien

* refein

* refine scatter_nd

* refine oneflow.rst

* refein

* from . import linalg
from . import autograd
from . import optim

* refein

* refine load_minst's docstring

add image.rst module.rst

refine rst files

* reformat

* remove Graph api

* fix file name

* refien

* fix oneflow.nn.image

* import oneflow.nn.image

* refein

* fix from . import lr_scheduler

* fix

* refeine

* refein

* refein

* rm todo

* rm unused box

* refein

* refien

* refein

* refien

* add ignore

* fix distributed api

* refein

* refein

* refine

* refine

* fix doctest

* refein

* update

* refien

* refein

* refein

* refein

* refein

* refein

* fix colition

* refien

* refien

* fix doc

* fix module doc

* fmt

* auto format by CI

* create moduels

* fix

* fix

* auto format by CI

* fix proto problem

* update cmake

* refien

* refein

* refienrefein

* refien

* refein

* add ignore

* refein

* refein

* refien

* auto format by CI

* refein

* refei

* refein

* refien

* refein

* udpate commit

* add ignore

* fix

* refein

* refein

* fix saved_model

* rjefein

* refein
Co-authored-by: NGehangZhang <649975329@qq.com>
Co-authored-by: NYaoChi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nstrint <xiaoyulink@gmail.com>

* nn.Graph call and launch impl

* Fix bug of 1.pybind11 list->vector; 2.GlobalJobDesc in NNGraph Compile

* Fix bug: AutoTick Multi-Client callback notifier lbn

* Fix bug: callback notifier op check in multi-client

* Fix bug: CheckOpGraph in multi-client

* Fix bug of MultiClientSessionContext new ProfileConf

* Fix bug of create NONE type kernel; add note; refine PlanUtil::PopulateOpAttibute

* Fix bug: CallbackNotifier kernel dtype with uint8; change tick dtype; Feat: add Kernel create fault-tolerant

* inputs tensor list

* rollback of hack

* Using TensorTuple for RunLazyNNGraph

* Fix bug: WaitAndSendIdsKernel handle for Multi-Client

* skip graph runnable test

* refine

* skip test graph optimizer for nn.Graph train job completer error

* Skip SetDefaultVariableConf JobPass in Multi-Client

* Fix bug: register_variable_op_names_and_tensors args with TensorTuple

* refine test graphs

* rm experimental in new scripts
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* change doctest code in abs.py

* del >>> flow.enable_eager_execution() in  modules/abs.py

* del @experimental_api

* del @experimental_api

* slove doctest not pass

* change abs.py for test

* refine acos

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* remove signle client twice api

* refine

* remove experimental of nn

* okokok

* remove experimental in test/modules and test/tensor

* remove experimental in test/dataloader

* remove experimental call in nn/modules

* remove experimental from nn/ framework/ test_utils

* remove experimental in functionalpy

* remove experimental of new merge

* skip fmod grad test

* remove experimental of fmod

* fix error when do multi case test on graph

* remove outdated rst files

* refine and add new rst files

* format

* revert test_fmod.py

* remove all experimental in python/nn dir (done!)

* remove experimental in test/graph

* add linagl

* refine rst

* refine scatter_nd

* refine oneflow.rst

* refine load_minst's docstring

add image.rst module.rst

refine rst files

* reformat

* remove Graph api

* fix file name

* fix index
Co-authored-by: NYaoChi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nstrint <xiaoyulink@gmail.com>
Co-authored-by: Ntsai <jackalcooper@gmail.com>

* add test on add input to graph

* add var into graph

* LazyInterpreter for FetchOutputOpExpr and set op parallel_distribution

* refine input var build

* split file

* rename

* mini refine

* Add note

* LazyInterpret::ApplyImpl for UserOpExpr

* refine test scripts

* add output to graph

* format

* Fix bug of LazyInterpret UserOpExpr for change output lbns

* Add test user op expr test

* fix note mistake

* add userop and test

* address review

* address review

* save i/o/s op_name and tensor for c_nn_graph

* address review

* adjust test

* refine build_graph_state
Co-authored-by: Nchengtbf <472491134@qq.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* core: GlobalMultiClientEnv; refine EagerExecution

* move IsMultiClient to EnvDesc
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* trigger ci test on graph

* add module scope

* Block.paramter or buffer return different value based on context

* add test on scope name prefix

* refine scope build

* refine scope build

* lambda get parameter to create right scope

* check para get

* fix vm:PhysicalRun

* lazy Tensor only create once
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* graph api

* add graph dummy test

* add test

* add recursive module mode

* graph.build test pass

* add detail check on graph inner node

* support config and train

* add repr for debug

* test buffer

* test buffer add

* refine test

* add comment

* refine test

* refactor Node to Block

* add named_state

* refine Graph.named_state()

* add state_tensortuple

* graph._compile()

* add mc session 0

* nn.graph: state tuple to private var; add BlockType; add simple multi client session

* NNGraphIf

* rm old graph.cpp

* nn.graph: add cpp NNGraph; export and call NNGraph

* add comment

* nn.Graph: rm prototype MultiClientSession

* nn.Graph: rm prototype MultiClientSession test

* nn.Graph: add TODO

* nn.Graph: format for review

* nn.Graph: format

* nn.Graph: format

* nn.Graph: pass flake8 check

* job_build_ctx

* support lazy context

* format

* lazy mode

* format

* format

* lazy mode add test

* debug session

* init session and job_build_context

* rm temp code

* build default scope

* add default scope

* add scope proto for debug

* chech scope

* format

* refine MultiClientSession.resource

* address review

* lazy init scope stack in single-client, instantly init scope stack after MultiClientSession created in multi-client

* fix typo

* address review

* fix clear default session

* merger and test
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* graph api

* add graph dummy test

* add test

* add recursive module mode

* graph.build test pass

* add detail check on graph inner node

* support config and train

* add repr for debug

* test buffer

* test buffer add

* refine test

* add comment

* refine test

* refactor Node to Block

* add named_state

* refine Graph.named_state()

* add state_tensortuple

* graph._compile()

* add mc session 0

* nn.graph: state tuple to private var; add BlockType; add simple multi client session

* NNGraphIf

* rm old graph.cpp

* nn.graph: add cpp NNGraph; export and call NNGraph

* add comment

* nn.Graph: rm prototype MultiClientSession

* nn.Graph: rm prototype MultiClientSession test

* nn.Graph: add TODO

* nn.Graph: hack to get Graph object name

* nn.Graph: get obj name

* nn.Graph: get obj name 2

* nn.Graph: format for review

* nn.Graph: format

* nn.Graph: format

* nn.Graph: pass flake8 check

* Update graph.py

* name with init count

* name with init count 2
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* rm nn.Graph.train

* Update graph.py

GraphCofig default predict

* Update test_graph.py

* graph api

* add graph dummy test

* add test

* add recursive module mode

* graph.build test pass

* add detail check on graph inner node

* support config and train

* add repr for debug

* test buffer

* test buffer add

* refine test

* add comment

* refine test

* refactor Node to Block

* add named_state

* refine Graph.named_state()

* add state_tensortuple

* graph._compile()

* add mc session 0

* nn.graph: state tuple to private var; add BlockType; add simple multi client session

* NNGraphIf

* rm old graph.cpp

* nn.graph: add cpp NNGraph; export and call NNGraph

* add comment

* nn.Graph: rm prototype MultiClientSession

* nn.Graph: rm prototype MultiClientSession test

* nn.Graph: add TODO

* nn.Graph: format for review

* nn.Graph: format

* nn.Graph: format

* nn.Graph: pass flake8 check
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>