Skip to content

MegEngine
MegEngine

MegEngine 天元 / MegEngine
9 个月 前同步成功

通知 392
Star 4702
Fork 582
MegEngine
MegEngine

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

提交 e296a684 编写于 作者: M Megvii Engine Team
浏览文件
操作

fix(build/doc): fix build md 

GitOrigin-RevId: 7cb159666dbfc740e760030df73aecb71b451eeb
上级 3635af62
隐藏空白更改
内联 并排
Showing with 232 addition and 217 deletion
scripts/cmake-build/BUILD_README.md 100644 → 100755
浏览文件 @ e296a684
# build support status
## host build
* windows build (cpu + gpu)
* linux build (cpu + gpu)
* macos build (cpu only)
## cross build
* windows cross build arm-android (ok)
* windows cross build arm-linux (ok)
* linux cross build arm-android (ok)
* linux cross build arm-linux (ok)
* macos cross build arm-android (ok)
* macos cross build arm-linux (ok but experimental)
* macos cross build ios (ok)
# build env prepare
## package install
### windows host build
```
1: installl Visual Studio (need support LLVM/clang-cl), eg 2019
pls install LLVM-10, VS llvm linker have issue, pls replace lld-link.exe,
download from https://releases.llvm.org/download.html#10.0.0
2: install extension of VS: python/cmake/LLVM
3: CUDA env(if enable CUDA), version detail: project_root_dir/README.md
4: now we support cuda10.1+cudnn7.6+TensorRT6.0 on windows, as windows can
only use dll in fact with cudnn/TensorRT, so please install the same version;
4a: install cuda10.1 to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1
4b: install cudnn7.6 to C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32
4c: install TensorRT6.0 to C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5
4d: add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin to system Path env
4e: add C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32\cuda\bin to system Path env
4f: add C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5\lib Path
if u do not do 4d/4e/4f, CUDA runtime can not find dll
5: install python3 (DFT 3.8.3) to /c/Users/${USER}/mge_whl_python_env/3.8.3 and
put it to PATH env and run python3 -m pip install numpy (if u want to build with training mode or build python whl)
```
### linux host build
```
1: cmake, which version > 3.15.2
2: gcc/g++, which version > 6, (gcc/g++ >= 7, if need build training)
3: install build-essential git git-lfs gfortran libgfortran-6-dev autoconf gnupg flex bison gperf curl
4: zlib1g-dev gcc-multilib g++-multilib lib32ncurses5-dev libxml2-utils xsltproc unzip libtool:
5: librdmacm-dev rdmacm-utils python3-dev swig python3-numpy texinfo
6: CUDA env(if enable CUDA), version detail: project_root_dir/README.md
```
### macos host build
```
1: cmake, which version > 3.15.2
2: install brew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
3: brew install python python3 swig coreutils
4: install at least xcode command line tool: https://developer.apple.com/xcode/
5: about cuda: we do not support CUDA on macos
6: python3 -m pip install numpy (if u want to build with training mode or build python whl)
```
### cross build for arm-android
now we support windows/linux/macos cross build to arm-android
```
1: install unix-like tools, eg MSYS if you are using windows(recommend)
we also support CMD.exe or powershell on windows
1: download NDK from https://developer.android.google.cn/ndk/downloads/
for diff OS platform package, suggested NDK20 or NDK21
2: export NDK_ROOT=NDK_DIR at bash-like env
3: config NDK_ROOT to PATH env at windows control board if use CMD/powershell
```
### cross build for arm-linux
now we support arm-linux on linux and windows fully, also experimental on MACOS
```
1: download toolchains on https://releases.linaro.org/components/toolchain/gcc-linaro/
or https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads
if use windows or linux
2: download https://github.com/thinkski/osx-arm-linux-toolchains if use MACOS
```
### cross build for ios
now we only support cross build to ios on MACOS
```
1: install full xcode: https://developer.apple.com/xcode/
```
## third-party code prepare
### with bash env(linux/macos/unix-like tools on windows, eg: msys etc)
```
./third_party/prepare.sh
./third_party/install-mkl.sh
```
### windows shell env(eg: cmd, powershell etc)
infact if you can use git command on windows, which means you always install
bash.exe at the same dir of git.exe, find it, then you can prepare third-party
code by command:
```
bash.exe ./third_party/prepare.sh
bash.exe ./third_party/install-mkl.sh
```
# how to build
## with bash env(linux/macos/unix-like tools on windows, eg: msys etc)
```
1: host build just use scripts:scripts/cmake-build/host_build.sh
2: cross build to arm-android: scripts/cmake-build/cross_build_android_arm_inference.sh
3: cross build to arm-linux: scripts/cmake-build/cross_build_linux_arm_inference.sh
4: cross build to ios: scripts/cmake-build/cross_build_ios_arm_inference.sh
```
## windows shell env(eg: cmd, powershell etc)
```
1: we do not provide BAT for cmd/powershlel scripts, BUT u can refs for scripts/cmake-build/*.sh
```
## Visual Studio GUI(only for windows host)
```
1: import megengine src to Visual Studio as a project
2: right click CMakeLists.txt, choose config 'cmake config'
choose clang_cl_x86 or clang_cl_x64
3: config other CMAKE config, eg, CUDA ON OR OFF
```
# other arm-linux-like board support
it`s easy to support other customized arm-linux-like board, example:
1: HISI 3516/3519, infact u can just use toolchains from arm developer or linaro
# Build support status
## Host build
* Windows build (cpu and gpu)
* Linux build (cpu and gpu)
* MacOS build (cpu only)
## Cross build
* Windows cross build ARM-Android (ok)
* Windows cross build ARM-Linux (ok)
* Linux cross build ARM-Android (ok)
* Linux cross build ARM-Linux (ok)
* MacOS cross build ARM-Android (ok)
* MacOS cross build ARM-Linux (ok but experimental)
* MacOS cross build IOS (ok)
# Build env prepare
## Package install
### Windows host build
* commands:
```
1: installl Visual Studio (need support LLVM/clang-cl), eg 2019. Please install LLVM-10, VS LLVM linker have issue, please replace lld-link.exe, which can be download from https://releases.llvm.org/download.html#10.0.0
2: install extension of VS: Python/Cmake/LLVM
3: now we support cuda10.1+cudnn7.6+TensorRT6.0 on Windows, as Windows can only use DLL in fact with cudnn/TensorRT, so please install the same version;
3a: install cuda10.1 to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1
3b: install cudnn7.6 to C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32
3c: install TensorRT6.0 to C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5
3d: add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin to system PATH env
3e: add C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32\cuda\bin to system Path env
3f: add C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5\lib Path, if you do not do 4d/4e/4f, CUDA runtime can not find dll
4: install Python3 (default is 3.8.3) to /c/Users/${USER}/mge_whl_python_env/3.8.3 and put it to PATH env and run python3 -m pip install numpy (if you want to build with training mode)
```
### Linux host build
* commands:
```
1: install Cmake, which version >= 3.15.2
2: install gcc/g++, which version >= 6, (gcc/g++ >= 7, if need build training mode)
3: install build-essential git git-lfs gfortran libgfortran-6-dev autoconf gnupg flex bison gperf curl zlib1g-dev gcc-multilib g++-multilib lib32ncurses5-dev libxml2-utils xsltproc unzip libtool librdmacm-dev rdmacm-utils python3-dev swig python3-numpy texinfo
4: CUDA env(if enable CUDA), version detail refer to README.md
```
### MacOS host build
* commands:
```
1: install Cmake, which version >= 3.15.2
2: install brew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
3: brew install python python3 swig coreutils
4: install at least xcode command line tool: https://developer.apple.com/xcode/
5: about cuda: we do not support CUDA on MacOS
6: python3 -m pip install numpy (if you want to build with training mode)
```
### Cross build for ARM-Android
Now we support Windows/Linux/MacOS cross build to ARM-Android
* commands:
```
1: install unix-like tools, eg MSYS if you are using windows(recommend), we also support CMD.exe or powershell on windows
2: download NDK from https://developer.android.google.cn/ndk/downloads/ for diff OS platform package, suggested NDK20 or NDK21
3: export NDK_ROOT=NDK_DIR at bash-like env
4: config NDK_ROOT to PATH env at windows control board if use CMD/powershell
```
### Cross build for ARM-Linux
Now we support ARM-Linux on Linux and Windows fully, also experimental on MacOS
* commands:
```
1: download toolchains from https://releases.linaro.org/components/toolchain/gcc-linaro/ or https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads if use Windows or Linux
2: download toolchains from https://github.com/thinkski/osx-arm-linux-toolchains if use MacOS
```
### Cross build for IOS
Now we only support cross build to IOS from MACOS
* commands:
```
1: install full xcode: https://developer.apple.com/xcode/
```
## Third-party code prepare
With bash env(Linux/MacOS/Unix-Like tools on Windows, eg: msys etc)
* commands:
```
./third_party/prepare.sh
./third_party/install-mkl.sh
```
Windows shell env(eg, CMD, Powershell etc), infact if you can use git command on Windows, which means you always install bash.exe at the same dir of git.exe, find it, then you can prepare third-party code by
* command:
```
bash.exe ./third_party/prepare.sh
bash.exe ./third_party/install-mkl.sh
```
# How to build
## With bash env(Linux/MacOS/Unix-Like tools on Windows, eg: msys etc)
* command:
```
1: host build just use scripts:scripts/cmake-build/host_build.sh
2: cross build to ARM-Android: scripts/cmake-build/cross_build_android_arm_inference.sh
3: cross build to ARM-Linux: scripts/cmake-build/cross_build_linux_arm_inference.sh
4: cross build to IOS: scripts/cmake-build/cross_build_ios_arm_inference.sh
```
## Windows shell env(eg, CMD, Powershell etc)
* command:
```
1: we do not provide BAT for CMD/Powershlel scripts, BUT you can refer for scripts/cmake-build/*.sh
```
## Visual Studio GUI(only for Windows host)
* command:
```
1: import megengine src to Visual Studio as a project
2: right click CMakeLists.txt, choose config 'cmake config' choose clang_cl_x86 or clang_cl_x64
3: config other CMAKE config, eg, CUDA ON OR OFF
```
# Other ARM-Linux-Like board support
It`s easy to support other customized arm-linux-like board, example:
* 1: HISI 3516/3519, infact u can just use toolchains from arm developer or linaro
then call scripts/cmake-build/cross_build_linux_arm_inference.sh to build a ELF
binary, or if you get HISI official toolschain, you just need modify CMAKE_CXX_COMPILER
and CMAKE_C_COMPILER in toolchains/arm-linux-gnueabi* to a real name
2: about Raspberry, just use scripts/cmake-build/cross_build_linux_arm_inference.sh
* 2: about Raspberry, just use scripts/cmake-build/cross_build_linux_arm_inference.sh
scripts/whl/BUILD_PYTHON_WHL_README.md 100644 → 100755
浏览文件 @ e296a684
# python whl package build support status
* windows build (cpu + gpu)
* linux build (cpu + gpu)
* macos build (cpu only)
# build env prepare
## linux
```
1: enable rootless docker env, refs: https://docs.docker.com/engine/security/rootless/
2: cd ./scripts/whl/manylinux2010
3: ./build_image.sh
```
## macos
```
./scripts/whl/macos/macos_whl_env_prepare.sh
```
## windows
```
1: refs scripts/cmake-build/BUILD_README.md windows section build for base windows build
2: install several python or install u care python version, default install dir: /c/Users/${USER}/mge_whl_python_env
a: mkdir /c/Users/${USER}/mge_whl_python_env
b: download python 64-bit install exe
https://www.python.org/ftp/python/3.5.4/python-3.5.4-amd64.exe
https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe
https://www.python.org/ftp/python/3.7.7/python-3.7.7-amd64.exe
https://www.python.org/ftp/python/3.8.3/python-3.8.3-amd64.exe
c: install python-3.5.4-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.5.4 from install gui
d: install python-3.6.8-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.6.8 from install gui
e: install python-3.7.7-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.7.7 from install gui
f: install python-3.8.3-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.8.3 from install gui
3: rename python.exe to python3.exe
a: mv /c/Users/${USER}/mge_whl_python_env/3.5.4/python.exe /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe
b: mv /c/Users/${USER}/mge_whl_python_env/3.6.8/python.exe /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe
c: mv /c/Users/${USER}/mge_whl_python_env/3.7.7/python.exe /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe
d: mv /c/Users/${USER}/mge_whl_python_env/3.8.3/python.exe /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe
4: install needed package for build python whl package
a0: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install --upgrade pip
a1: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install -r imperative/python/requires-test.txt
a2: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install numpy wheel requests tqdm tabulate
b0: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install --upgrade pip
b1: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install -r imperative/python/requires-test.txt
b2: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install numpy wheel requests tqdm tabulate
c0: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install --upgrade pip
c1: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install -r imperative/python/requires-test.txt
c2: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install numpy wheel requests tqdm tabulate
d0: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install --upgrade pip
d1: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install -r imperative/python/requires-test.txt
d2: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install numpy wheel requests tqdm tabulate
```
# how to build
## build for linux
MegBrain delivers `wheel` package with `manylinux2010` tag defined in [PEP-571](https://www.python.org/dev/peps/pep-0571/).
```
export CUDA_ROOT_DIR=/path/to/cuda
export CUDNN_ROOT_DIR=/path/to/cudnn
export TENSORRT_ROOT_DIR=/path/to/tensorrt
./scripts/whl/manylinux2010/build_wheel.sh
```
And you can find all of the outputs in `output` directory.
# Python whl package build support status
* Windows build (cpu and gpu)
* Linux build (cpu and gpu)
* MacOS build (cpu only)
# Build env prepare
## Linux
```bash
1: please refer to: https://docs.docker.com/engine/security/rootless/ to enable rootless docker env
2: cd ./scripts/whl/manylinux2010
3: ./build_image.sh
```
## MacOS
```bash
./scripts/whl/macos/macos_whl_env_prepare.sh
```
## Windows
```
1: refer to scripts/cmake-build/BUILD_README.md Windows section build for base windows build env prepare
2: install several python or install your care about python version, default install dir: /c/Users/${USER}/mge_whl_python_env
a: mkdir /c/Users/${USER}/mge_whl_python_env
b: download python 64-bit install exe
https://www.python.org/ftp/python/3.5.4/python-3.5.4-amd64.exe
https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe
https://www.python.org/ftp/python/3.7.7/python-3.7.7-amd64.exe
https://www.python.org/ftp/python/3.8.3/python-3.8.3-amd64.exe
c: install python-3.5.4-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.5.4 from install gui
d: install python-3.6.8-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.6.8 from install gui
e: install python-3.7.7-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.7.7 from install gui
f: install python-3.8.3-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.8.3 from install gui
3: cp python.exe to python3.exe
a: mv /c/Users/${USER}/mge_whl_python_env/3.5.4/python.exe /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe
b: mv /c/Users/${USER}/mge_whl_python_env/3.6.8/python.exe /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe
c: mv /c/Users/${USER}/mge_whl_python_env/3.7.7/python.exe /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe
d: mv /c/Users/${USER}/mge_whl_python_env/3.8.3/python.exe /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe
4: install needed package for build python whl package
a0: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install --upgrade pip
a1: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install -r imperative/python/requires-test.txt
a2: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install numpy wheel requests tqdm tabulate
b0: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install --upgrade pip
b1: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install -r imperative/python/requires-test.txt
b2: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install numpy wheel requests tqdm tabulate
If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```
ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh
```
If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg:
```
BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh
```
## build for macos
```
./scripts/whl/macos/macos_build_whl.sh
```
If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```
ALL_PYTHON="3.7.7" ./scripts/whl/macos/macos_build_whl.sh
```
## build for windows
```
./scripts/whl/windows/windows_build_whl.sh
```
If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```
ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh
```
If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg:
```
BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh
```
c0: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install --upgrade pip
c1: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install -r imperative/python/requires-test.txt
c2: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install numpy wheel requests tqdm tabulate
d0: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install --upgrade pip
d1: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install -r imperative/python/requires-test.txt
d2: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install numpy wheel requests tqdm tabulate
```
# How to build
## Build for linux
* MegBrain delivers `wheel` package with `manylinux2010` tag defined in [PEP-571](https://www.python.org/dev/peps/pep-0571/).
commands:
```bash
export CUDA_ROOT_DIR=/path/to/cuda
export CUDNN_ROOT_DIR=/path/to/cudnn
export TENSORRT_ROOT_DIR=/path/to/tensorrt
./scripts/whl/manylinux2010/build_wheel.sh
```
* And you can find all of the outputs in `output` directory.If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```bash
ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh
```
* If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg:
```bash
BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh
```
## Build for MacOS
* commands:
```bash
./scripts/whl/macos/macos_build_whl.sh
```
* If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```bash
ALL_PYTHON="3.7.7" ./scripts/whl/macos/macos_build_whl.sh
```
## Build for Windows
* commands:
```bash
./scripts/whl/windows/windows_build_whl.sh
```
* If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg:
```bash
ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh
```
* If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg:
```
BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh
```
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或