Merge branch 'dev_fix_broadcast_binary_ops_grad_func' of https://github.com/Oneflow-Inc/oneflow into dev_fix_broadcast_binary_ops_grad_func

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add first version of unary primitive op

* fix

* remove redundant file

* Revert

* fix format

* use has input to check

* restruct

* refine

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* migrate partial_fc

* add test and fix DistributedPariticalFCSample release bug

* fix typos in functional_api.yaml

* initialization

* refine testcase

* skip cpu-only test

* reformat
Co-authored-by: Nbbuf <1182563586@qq.com>
Co-authored-by: NXiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* init

* use cuda elementwise template

* Add half version

* try to use different rand init

* try to use copynd primitives to improve efficiency

* use copynd primitive in narrow backward

* remove redundant templates

* revert

* use elementwise templates

* fix comment

* Remove useless header file

* rename copynd_primitive to copy_nd_primitive

* fix format

* c++ standard: bump to 14

* remove cplusplus_14.h & use cxx14

* optional: add monadic operations `map`, `bind` and `or_else`

* remove useless header

* fix or_else

* fix operator!=

* fix type

* format
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* initial commit, add code for async construct tensor from numpy array

* inital commit to change Maybe to Optional

* delete redundant code

* replace Maybe with Optional

* fix compile errors

* format code

* changes based on review

* format code, fix based on review

* format code

* fix multiclient type

* changes based on review

* changes based on review

* unify calling to IsMultiClirnt

* refector multi_client related code

* restore InMultiClient interface

* double check for unnecessary changes

* remove unnecessary changes

* format code

* Update oneflow/api/python/symbol/job_conf_symbol.cpp

* Update oneflow/api/python/symbol/op_conf_symbol.cpp

* Update oneflow/api/python/symbol/op_node_signature_symbol.cpp

* Update oneflow/core/common/optional.h

* Update oneflow/api/python/symbol/string_symbol.cpp

* Update oneflow/api/python/symbol/scope_symbol.cpp

* Update oneflow/api/python/symbol/placement_symbol.cpp

* Update oneflow/api/python/symbol/op_conf_symbol.cpp
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: NTwice <i@twice.moe>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* Refactor oneflow.Size

* refine

* add pybind11 caster

* Support Shape cast

* refine

* fix size index

* include size header if need export C++ Shape to Python.

* restruct matmul

* restruct narrow

* restruct unsqueeze

* restruct permute

* refine

* refine

* restruct one_hot

* format

* refine

* restruct concat

* refine

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix softmax compile error

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add ReduceLROnPlateau op

* add unit test case

* Mod review comments

* mod review comments

* mod review comments

* mod review comments

* mod review comment

* mod test case

* adjust test case
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Returns py::tuple instead of tensor tuple, and refine split functional api.

* fix and refine code

* fix compile

* modified split_sizes
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: luqiang-guo <702572275@qq.com>

* softmax primitive kernel

* refine

* fix

* refine

* refine

* refine

* refine

* log softmax

* refine

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add todo

* refine

* add attr

* refine

* refine

* add todo

* refine

* add alias c1 for check-oneflow

* fix

* update scripts

* refine

* fix single client env reinit

* add attr

* save and pass mlir module

* fix

* restore module in kernel

* lower in kernel

* refien

* add scf to std

* update lit

* fmt

* add all passes

* add alisas

* refein

* refein

* add check

* fix pass order

* add TODO

* refein

* create jit exe

* refein

* fix arity

* add check and rpint err

* refein

* refein

* refein

* refein

* refein

* refein

* emiit c

* working

* revert

* add err print

* e2e works

* refein

* refein

* refein

* use STATIC_SWITCH_FUNC

* add log

* rename

* use invoke packed

* refein

* add todo

* refein

* rm log

* fix

* refein

* rm

* refein

* add scf to gpu

* add cmake flag for cuda runner

* add CMAKE_CUDA_COMPILER

* refine

* refien

* register gpu kernel

* refein

* add gpu passes

* refein

* add

* refine

* add ptx to cubin pass

* produce cubin

* add gpu to llvm pass

* refein

* add log

* refien

* link mlir cuda runtime lib

* add note

* make gpu runner available in file check

* rm unused

* add to prevent break

* fix with cuda

* edit mlir by hand to have it run on cuda

* rm useless

* add todo

* upgrade llvm

* refein m,irror scripts

* fix for llvm upgrade

* refein cmake

* fix

* fix for llvm upgrade

* remove unused headers

* refeine

* refein

* refactor

* add

* refine

* refine

* cmake first class cuda support

* refine

* refine

* refein

* refine

* refine

* refine

* refein

* add todo

* refine

* pass shared lib path from py

* prevent redef ONEFLOW_CMAKE_BUILD_TYPE

* refine msg

* fix fmt

* fix fmt

* fix fmt

* refine

* refueb

* fix

* refactor jit function outline

* refein

* rm debug log

* rm unnecessary erase

* use 75

* refein

* add allowFoldingUnitDimReshapes

* refine

* Outline JIT func (#6542)

* check in pass impl

* add test

* check in changes

* add todo

* extract func to create attrs

* refine

* refine and mv bert

* refein LLVM_EXTERNAL_LIT

* refine log user_op::AttrValueUtil::ToCppAttrValue

* fix for nd_sbp

* refine log

* fix warnings

* fix

* leverage input_order and output_order

* save lbn_segment_keys as input output order

* refine

* refein

* add CUDATOOLKIT_BIN_ROOT

* finish todo

* finish todo

* finish todo

* add matmul

* rm repetitive code

* add log

* add unary

* add gather

* refine and add gelu

* fix loc

* add mlir conv op (#6559)

* add mlir conv op

* fix conv2d tabelgen bug

* fix merge compile error

* fix comments

* Update mlir-cuda-75.cmake

* add mlir resnet50 test

* add SI32ArrayAttr
Co-authored-by: NShenghang Tsai <jackalcooper@gmail.com>

* backport refactoring of translation

* Add resnet50 mlir dialect part ops (#6607)

* add scalar math ops tablegen

* add pool ops

* add bias_add op

* fix comment

* fix comment

* code format

* add reshape op

* add reduce ops and restruct scalar math ops

* fix bug

* fix typo

* address review

* address review

* rm loggin

* address review

* rm logging

* backport variable rename

* add flag ONEFLOW_MLIR_ENABLE_FUSERS
Co-authored-by: NXiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use git to clean dir

* rm useless to trigger CI

* trigger CI

* refine

* refine

* refine

* refine

* fix typo PopulateOpAttribute

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(autograd.Function): add base class define

* format

* feat(autograd.Function): cache FunctionOpExpr in AutogradFunctionBase
                         and pass autograd.Function name to cpp

* feat(autograd.Function): wrapper PyFunction to FType

* fix(autograd.Function): fix wrapper function capture bug

* feat(autograd.Function): support autograd.Function backward

* feat(autograd.Function): refine apply return value

* fix(autograd.Function): fix autograd.Function name bug

* feat(autograd.Function): refine ctx python api

* feat(*): refine apply interface

* test(autograd.Function): fix ctx interface and add test

* feat(autograd.Function): support mark_non_differentiable

* align ctx.saved_tensors interface

* docs(autograd.Function): export documentation

* refine function names

* refine interface

* use py::args instead of py::object

* refine code

* fix(*): fix `func_name` variable conflict with CHECK_JUST

* feat(autograd.Function): support static call

* docs(autograd.Function): update documentation

* refine code

* add JUST
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Interface primitive::BroadcastElementwiseBinary

* refine
Co-authored-by: Nguo ran <360112263@qq.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Matmul kernels use primitive

* refine

* fix
Co-authored-by: Nguo ran <360112263@qq.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix_naive_eager_boxing_checker

* refine eager_p_to_b_kerne and eager_p_to_s_kernel

* initial commit for adding logical_not operator

* adding logical_not op, debugging Dtype related problems

* finished testing locally, need to add tests

* added tests

* added docs and formatted code

* format file

* format file

* remove python wrapper

* modification based on review

* remove redundant code, format file

* modifications based on reviews

* modifications based on review

* fix duplicate license info

* fix docstring

* fix docstring warning
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix sbp for prelu

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* imporve roll speed

* imporve speed of len(dims) > 1 cases
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dev torch style permute kernel

* Refine

* fix batch permute launch condition

* fix batch permute dispatch logic

* remove redundant header file

* simplified check logic

* use permute primitives in transpose kernels

* fix batch permute logic and avoid mod

* remove redundant templates

* fix grid step

* add grid for loop to avoid the elementnum is too large

* fix bug when hw is not divided by tile size

* refine format

* add a copy kernel as a baseline

* remove annotation

* add copy kernel

* add sync

* use batch permute for profile

* add copy tile baseline

* simplify params for copy kernel

* add slow copy kernel

* use mul to instead mod and remove copy

* use movement size = 4 when h w is modify by 2

* Add temp process for half2

* add half2 specialized kernel

* remove redundant license

* simplified code

* fix format

* fix comment

* fix comment

* use bad for loop condition

* merge half2 in load

* fix bad for loop in batch permute

* refine

* use align storage

* refine

* fix comment

* fix comment

* fix format

* add const and remove redundant header file

* remove register macro

* refine cuda code

* fix guoran comment

* fix format

* fix some details

* remove cuda graph

* fix for 0d tensor
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>