diff --git a/scripts/cmake-build/BUILD_README.md b/scripts/cmake-build/BUILD_README.md old mode 100644 new mode 100755 index 7cc2c73f767ab82376496a53d589709caefa50f2..877dce51089cf36f47b3a8fa8aed6c4ba818b049 --- a/scripts/cmake-build/BUILD_README.md +++ b/scripts/cmake-build/BUILD_README.md @@ -1,119 +1,134 @@ -# build support status -## host build -* windows build (cpu + gpu) -* linux build (cpu + gpu) -* macos build (cpu only) -## cross build -* windows cross build arm-android (ok) -* windows cross build arm-linux (ok) -* linux cross build arm-android (ok) -* linux cross build arm-linux (ok) -* macos cross build arm-android (ok) -* macos cross build arm-linux (ok but experimental) -* macos cross build ios (ok) - -# build env prepare -## package install -### windows host build - ``` - 1: installl Visual Studio (need support LLVM/clang-cl), eg 2019 - pls install LLVM-10, VS llvm linker have issue, pls replace lld-link.exe, - download from https://releases.llvm.org/download.html#10.0.0 - 2: install extension of VS: python/cmake/LLVM - 3: CUDA env(if enable CUDA), version detail: project_root_dir/README.md - 4: now we support cuda10.1+cudnn7.6+TensorRT6.0 on windows, as windows can - only use dll in fact with cudnn/TensorRT, so please install the same version; - 4a: install cuda10.1 to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 - 4b: install cudnn7.6 to C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32 - 4c: install TensorRT6.0 to C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5 - 4d: add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin to system Path env - 4e: add C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32\cuda\bin to system Path env - 4f: add C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5\lib Path - if u do not do 4d/4e/4f, CUDA runtime can not find dll - 5: install python3 (DFT 3.8.3) to /c/Users/${USER}/mge_whl_python_env/3.8.3 and - put it to PATH env and run python3 -m pip install numpy (if u want to build with training mode or build python whl) - ``` -### linux host build - ``` - 1: cmake, which version > 3.15.2 - 2: gcc/g++, which version > 6, (gcc/g++ >= 7, if need build training) - 3: install build-essential git git-lfs gfortran libgfortran-6-dev autoconf gnupg flex bison gperf curl - 4: zlib1g-dev gcc-multilib g++-multilib lib32ncurses5-dev libxml2-utils xsltproc unzip libtool: - 5: librdmacm-dev rdmacm-utils python3-dev swig python3-numpy texinfo - 6: CUDA env(if enable CUDA), version detail: project_root_dir/README.md - ``` -### macos host build - ``` - 1: cmake, which version > 3.15.2 - 2: install brew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" - 3: brew install python python3 swig coreutils - 4: install at least xcode command line tool: https://developer.apple.com/xcode/ - 5: about cuda: we do not support CUDA on macos - 6: python3 -m pip install numpy (if u want to build with training mode or build python whl) - ``` -### cross build for arm-android - now we support windows/linux/macos cross build to arm-android - ``` - 1: install unix-like tools, eg MSYS if you are using windows(recommend) - we also support CMD.exe or powershell on windows - 1: download NDK from https://developer.android.google.cn/ndk/downloads/ - for diff OS platform package, suggested NDK20 or NDK21 - 2: export NDK_ROOT=NDK_DIR at bash-like env - 3: config NDK_ROOT to PATH env at windows control board if use CMD/powershell - ``` -### cross build for arm-linux - now we support arm-linux on linux and windows fully, also experimental on MACOS - ``` - 1: download toolchains on https://releases.linaro.org/components/toolchain/gcc-linaro/ - or https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads - if use windows or linux - 2: download https://github.com/thinkski/osx-arm-linux-toolchains if use MACOS - ``` -### cross build for ios - now we only support cross build to ios on MACOS - ``` - 1: install full xcode: https://developer.apple.com/xcode/ - ``` -## third-party code prepare -### with bash env(linux/macos/unix-like tools on windows, eg: msys etc) - ``` - ./third_party/prepare.sh - ./third_party/install-mkl.sh - ``` -### windows shell env(eg: cmd, powershell etc) - infact if you can use git command on windows, which means you always install - bash.exe at the same dir of git.exe, find it, then you can prepare third-party - code by command: - ``` - bash.exe ./third_party/prepare.sh - bash.exe ./third_party/install-mkl.sh - ``` -# how to build -## with bash env(linux/macos/unix-like tools on windows, eg: msys etc) - ``` - 1: host build just use scripts:scripts/cmake-build/host_build.sh - 2: cross build to arm-android: scripts/cmake-build/cross_build_android_arm_inference.sh - 3: cross build to arm-linux: scripts/cmake-build/cross_build_linux_arm_inference.sh - 4: cross build to ios: scripts/cmake-build/cross_build_ios_arm_inference.sh - ``` -## windows shell env(eg: cmd, powershell etc) - ``` - 1: we do not provide BAT for cmd/powershlel scripts, BUT u can refs for scripts/cmake-build/*.sh - ``` -## Visual Studio GUI(only for windows host) - ``` - 1: import megengine src to Visual Studio as a project - 2: right click CMakeLists.txt, choose config 'cmake config' - choose clang_cl_x86 or clang_cl_x64 - 3: config other CMAKE config, eg, CUDA ON OR OFF - ``` - - -# other arm-linux-like board support -it`s easy to support other customized arm-linux-like board, example: -1: HISI 3516/3519, infact u can just use toolchains from arm developer or linaro +# Build support status +## Host build +* Windows build (cpu and gpu) +* Linux build (cpu and gpu) +* MacOS build (cpu only) + +## Cross build +* Windows cross build ARM-Android (ok) +* Windows cross build ARM-Linux (ok) +* Linux cross build ARM-Android (ok) +* Linux cross build ARM-Linux (ok) +* MacOS cross build ARM-Android (ok) +* MacOS cross build ARM-Linux (ok but experimental) +* MacOS cross build IOS (ok) + +# Build env prepare +## Package install +### Windows host build +* commands: +``` +1: installl Visual Studio (need support LLVM/clang-cl), eg 2019. Please install LLVM-10, VS LLVM linker have issue, please replace lld-link.exe, which can be download from https://releases.llvm.org/download.html#10.0.0 +2: install extension of VS: Python/Cmake/LLVM +3: now we support cuda10.1+cudnn7.6+TensorRT6.0 on Windows, as Windows can only use DLL in fact with cudnn/TensorRT, so please install the same version; + 3a: install cuda10.1 to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 + 3b: install cudnn7.6 to C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32 + 3c: install TensorRT6.0 to C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5 + 3d: add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin to system PATH env + 3e: add C:\Program Files\NVIDIA GPU Computing Toolkit\cudnn-10.1-windows10-x64-v7.6.5.32\cuda\bin to system Path env + 3f: add C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT-6.0.1.5\lib Path, if you do not do 4d/4e/4f, CUDA runtime can not find dll +4: install Python3 (default is 3.8.3) to /c/Users/${USER}/mge_whl_python_env/3.8.3 and put it to PATH env and run python3 -m pip install numpy (if you want to build with training mode) +``` + +### Linux host build +* commands: +``` +1: install Cmake, which version >= 3.15.2 +2: install gcc/g++, which version >= 6, (gcc/g++ >= 7, if need build training mode) +3: install build-essential git git-lfs gfortran libgfortran-6-dev autoconf gnupg flex bison gperf curl zlib1g-dev gcc-multilib g++-multilib lib32ncurses5-dev libxml2-utils xsltproc unzip libtool librdmacm-dev rdmacm-utils python3-dev swig python3-numpy texinfo +4: CUDA env(if enable CUDA), version detail refer to README.md +``` + +### MacOS host build +* commands: +``` +1: install Cmake, which version >= 3.15.2 +2: install brew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" +3: brew install python python3 swig coreutils +4: install at least xcode command line tool: https://developer.apple.com/xcode/ +5: about cuda: we do not support CUDA on MacOS +6: python3 -m pip install numpy (if you want to build with training mode) +``` + +### Cross build for ARM-Android +Now we support Windows/Linux/MacOS cross build to ARM-Android + +* commands: +``` +1: install unix-like tools, eg MSYS if you are using windows(recommend), we also support CMD.exe or powershell on windows +2: download NDK from https://developer.android.google.cn/ndk/downloads/ for diff OS platform package, suggested NDK20 or NDK21 +3: export NDK_ROOT=NDK_DIR at bash-like env +4: config NDK_ROOT to PATH env at windows control board if use CMD/powershell +``` + +### Cross build for ARM-Linux +Now we support ARM-Linux on Linux and Windows fully, also experimental on MacOS + +* commands: +``` +1: download toolchains from https://releases.linaro.org/components/toolchain/gcc-linaro/ or https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads if use Windows or Linux +2: download toolchains from https://github.com/thinkski/osx-arm-linux-toolchains if use MacOS +``` + +### Cross build for IOS +Now we only support cross build to IOS from MACOS + +* commands: +``` +1: install full xcode: https://developer.apple.com/xcode/ +``` + +## Third-party code prepare +With bash env(Linux/MacOS/Unix-Like tools on Windows, eg: msys etc) + +* commands: +``` +./third_party/prepare.sh +./third_party/install-mkl.sh +``` + +Windows shell env(eg, CMD, Powershell etc), infact if you can use git command on Windows, which means you always install bash.exe at the same dir of git.exe, find it, then you can prepare third-party code by + +* command: +``` +bash.exe ./third_party/prepare.sh +bash.exe ./third_party/install-mkl.sh +``` + +# How to build +## With bash env(Linux/MacOS/Unix-Like tools on Windows, eg: msys etc) + +* command: +``` +1: host build just use scripts:scripts/cmake-build/host_build.sh +2: cross build to ARM-Android: scripts/cmake-build/cross_build_android_arm_inference.sh +3: cross build to ARM-Linux: scripts/cmake-build/cross_build_linux_arm_inference.sh +4: cross build to IOS: scripts/cmake-build/cross_build_ios_arm_inference.sh +``` + +## Windows shell env(eg, CMD, Powershell etc) + +* command: +``` +1: we do not provide BAT for CMD/Powershlel scripts, BUT you can refer for scripts/cmake-build/*.sh +``` + +## Visual Studio GUI(only for Windows host) + +* command: +``` +1: import megengine src to Visual Studio as a project +2: right click CMakeLists.txt, choose config 'cmake config' choose clang_cl_x86 or clang_cl_x64 +3: config other CMAKE config, eg, CUDA ON OR OFF +``` + + +# Other ARM-Linux-Like board support +It`s easy to support other customized arm-linux-like board, example: + +* 1: HISI 3516/3519, infact u can just use toolchains from arm developer or linaro then call scripts/cmake-build/cross_build_linux_arm_inference.sh to build a ELF binary, or if you get HISI official toolschain, you just need modify CMAKE_CXX_COMPILER and CMAKE_C_COMPILER in toolchains/arm-linux-gnueabi* to a real name -2: about Raspberry, just use scripts/cmake-build/cross_build_linux_arm_inference.sh +* 2: about Raspberry, just use scripts/cmake-build/cross_build_linux_arm_inference.sh diff --git a/scripts/whl/BUILD_PYTHON_WHL_README.md b/scripts/whl/BUILD_PYTHON_WHL_README.md old mode 100644 new mode 100755 index 335d6ab654f07fb51cf6e084bf4621be78a1ca99..3a38c1e8f4bc8a826b4e50185481cb1b01ffca22 --- a/scripts/whl/BUILD_PYTHON_WHL_README.md +++ b/scripts/whl/BUILD_PYTHON_WHL_README.md @@ -1,103 +1,103 @@ -# python whl package build support status -* windows build (cpu + gpu) -* linux build (cpu + gpu) -* macos build (cpu only) -# build env prepare -## linux - - ``` - 1: enable rootless docker env, refs: https://docs.docker.com/engine/security/rootless/ - 2: cd ./scripts/whl/manylinux2010 - 3: ./build_image.sh - - ``` - -## macos - ``` - ./scripts/whl/macos/macos_whl_env_prepare.sh - ``` - -## windows - ``` - 1: refs scripts/cmake-build/BUILD_README.md windows section build for base windows build - 2: install several python or install u care python version, default install dir: /c/Users/${USER}/mge_whl_python_env - a: mkdir /c/Users/${USER}/mge_whl_python_env - b: download python 64-bit install exe - https://www.python.org/ftp/python/3.5.4/python-3.5.4-amd64.exe - https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe - https://www.python.org/ftp/python/3.7.7/python-3.7.7-amd64.exe - https://www.python.org/ftp/python/3.8.3/python-3.8.3-amd64.exe - c: install python-3.5.4-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.5.4 from install gui - d: install python-3.6.8-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.6.8 from install gui - e: install python-3.7.7-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.7.7 from install gui - f: install python-3.8.3-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.8.3 from install gui - 3: rename python.exe to python3.exe - a: mv /c/Users/${USER}/mge_whl_python_env/3.5.4/python.exe /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe - b: mv /c/Users/${USER}/mge_whl_python_env/3.6.8/python.exe /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe - c: mv /c/Users/${USER}/mge_whl_python_env/3.7.7/python.exe /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe - d: mv /c/Users/${USER}/mge_whl_python_env/3.8.3/python.exe /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe - 4: install needed package for build python whl package - a0: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install --upgrade pip - a1: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install -r imperative/python/requires-test.txt - a2: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install numpy wheel requests tqdm tabulate - - b0: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install --upgrade pip - b1: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install -r imperative/python/requires-test.txt - b2: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install numpy wheel requests tqdm tabulate - - c0: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install --upgrade pip - c1: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install -r imperative/python/requires-test.txt - c2: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install numpy wheel requests tqdm tabulate - - d0: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install --upgrade pip - d1: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install -r imperative/python/requires-test.txt - d2: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install numpy wheel requests tqdm tabulate - ``` - -# how to build -## build for linux - MegBrain delivers `wheel` package with `manylinux2010` tag defined in [PEP-571](https://www.python.org/dev/peps/pep-0571/). - - ``` - export CUDA_ROOT_DIR=/path/to/cuda - export CUDNN_ROOT_DIR=/path/to/cudnn - export TENSORRT_ROOT_DIR=/path/to/tensorrt - ./scripts/whl/manylinux2010/build_wheel.sh - ``` - - And you can find all of the outputs in `output` directory. +# Python whl package build support status +* Windows build (cpu and gpu) +* Linux build (cpu and gpu) +* MacOS build (cpu only) + +# Build env prepare +## Linux + +```bash +1: please refer to: https://docs.docker.com/engine/security/rootless/ to enable rootless docker env +2: cd ./scripts/whl/manylinux2010 +3: ./build_image.sh + +``` + +## MacOS +```bash +./scripts/whl/macos/macos_whl_env_prepare.sh +``` + +## Windows +``` +1: refer to scripts/cmake-build/BUILD_README.md Windows section build for base windows build env prepare +2: install several python or install your care about python version, default install dir: /c/Users/${USER}/mge_whl_python_env + a: mkdir /c/Users/${USER}/mge_whl_python_env + b: download python 64-bit install exe + https://www.python.org/ftp/python/3.5.4/python-3.5.4-amd64.exe + https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe + https://www.python.org/ftp/python/3.7.7/python-3.7.7-amd64.exe + https://www.python.org/ftp/python/3.8.3/python-3.8.3-amd64.exe + c: install python-3.5.4-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.5.4 from install gui + d: install python-3.6.8-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.6.8 from install gui + e: install python-3.7.7-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.7.7 from install gui + f: install python-3.8.3-amd64.exe to /c/Users/${USER}/mge_whl_python_env/3.8.3 from install gui +3: cp python.exe to python3.exe + a: mv /c/Users/${USER}/mge_whl_python_env/3.5.4/python.exe /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe + b: mv /c/Users/${USER}/mge_whl_python_env/3.6.8/python.exe /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe + c: mv /c/Users/${USER}/mge_whl_python_env/3.7.7/python.exe /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe + d: mv /c/Users/${USER}/mge_whl_python_env/3.8.3/python.exe /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe +4: install needed package for build python whl package + a0: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install --upgrade pip + a1: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install -r imperative/python/requires-test.txt + a2: /c/Users/${USER}/mge_whl_python_env/3.5.4/python3.exe -m pip install numpy wheel requests tqdm tabulate + + b0: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install --upgrade pip + b1: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install -r imperative/python/requires-test.txt + b2: /c/Users/${USER}/mge_whl_python_env/3.6.8/python3.exe -m pip install numpy wheel requests tqdm tabulate - If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: - - ``` - ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh - ``` - If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg: - - ``` - BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh - ``` - -## build for macos - ``` - ./scripts/whl/macos/macos_build_whl.sh - ``` - If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: - - ``` - ALL_PYTHON="3.7.7" ./scripts/whl/macos/macos_build_whl.sh - ``` -## build for windows - ``` - ./scripts/whl/windows/windows_build_whl.sh - ``` - If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: - - ``` - ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh - ``` - If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg: - - ``` - BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh - ``` + c0: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install --upgrade pip + c1: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install -r imperative/python/requires-test.txt + c2: /c/Users/${USER}/mge_whl_python_env/3.7.7/python3.exe -m pip install numpy wheel requests tqdm tabulate + + d0: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install --upgrade pip + d1: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install -r imperative/python/requires-test.txt + d2: /c/Users/${USER}/mge_whl_python_env/3.8.3/python3.exe -m pip install numpy wheel requests tqdm tabulate +``` + +# How to build +## Build for linux +* MegBrain delivers `wheel` package with `manylinux2010` tag defined in [PEP-571](https://www.python.org/dev/peps/pep-0571/). +commands: +```bash +export CUDA_ROOT_DIR=/path/to/cuda +export CUDNN_ROOT_DIR=/path/to/cudnn +export TENSORRT_ROOT_DIR=/path/to/tensorrt +./scripts/whl/manylinux2010/build_wheel.sh +``` + +* And you can find all of the outputs in `output` directory.If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: +```bash +ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh +``` + +* If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg: +```bash +BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="36m" ./scripts/whl/manylinux2010/build_wheel.sh +``` + +## Build for MacOS +* commands: +```bash +./scripts/whl/macos/macos_build_whl.sh +``` +* If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: +```bash +ALL_PYTHON="3.7.7" ./scripts/whl/macos/macos_build_whl.sh +``` + +## Build for Windows +* commands: +```bash +./scripts/whl/windows/windows_build_whl.sh +``` + +* If you just want to build for a specific Python verison, you can use `ALL_PYTHON` environment variable. eg: +```bash +ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh +``` + +* If you just want to build with cpu only version, you can set `BUILD_WHL_CPU_ONLY` environment 'ON'. eg: +``` +BUILD_WHL_CPU_ONLY="ON" ALL_PYTHON="3.8.3" ./scripts/whl/windows/windows_build_whl.sh +```