[CI][Doc] Add CI Pipelines doc/node tests verification (#3780)

* azp purposes
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* finish all
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* Reorder number point
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* source py$(python.version)/bin/activate
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* improve words
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* restructure
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* nit: change flow
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* fix typo
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>

* When the release CIs will run
Signed-off-by: NChun-Wei Chen <jacky82226@gmail.com>
Co-authored-by: NAshwini Khade <askhade@microsoft.com>

.azure-pipelines/Linux-CI.yml

@@ -144,3 +144,13 @@ jobs:
       python -m pip install -q .
 
     displayName: 'Run ONNX tests'
+
+  - script: |
+      source py$(python.version)/bin/activate
+      # onnx.checker all existing backend data
+      python workflow_scripts/test_generated_backend.py
+      # onnx.checker all generated backend data
+      python onnx/backend/test/cmd_tools.py generate-data
+      python workflow_scripts/test_generated_backend.py
+
+    displayName: Test backend test data

.azure-pipelines/Windows-CI.yml

@@ -48,7 +48,6 @@ jobs:
       git submodule update --init --recursive
       set ONNX_BUILD_TESTS=1
       set ONNX_ML=$(onnx_ml)
-      set ONNX_VERIFY_PROTO_3=$(onnx_verify_proto)
       set CMAKE_ARGS=-DONNX_USE_PROTOBUF_SHARED_LIBS=ON -DONNX_USE_LITE_PROTO=ON -DONNX_WERROR=ON
 
       python setup.py -q install

.github/workflows/win_no_exception_ci.yml

@@ -52,17 +52,13 @@ jobs:
 
     - name: Build and test ONNX binaries
       run: |
-        $arch = 'x64'
-        if ('${{ matrix.architecture }}' -eq 'x86') {
-          $arch = 'Win32'
-        }
         echo "Install protobuf"
         cd ./protobuf_root
         $protobuf_root_dir = Get-Location
         mkdir protobuf_install
         cd ./protobuf/cmake
 
-        cmake -G "Visual Studio 16 2019" -A $arch -DCMAKE_INSTALL_PREFIX="../../protobuf_install" -Dprotobuf_MSVC_STATIC_RUNTIME=ON -Dprotobuf_BUILD_SHARED_LIBS=OFF -Dprotobuf_BUILD_TESTS=OFF -Dprotobuf_BUILD_EXAMPLES=OFF .
+        cmake -G "Visual Studio 16 2019" -A ${{ matrix.architecture }} -DCMAKE_INSTALL_PREFIX="../../protobuf_install" -Dprotobuf_MSVC_STATIC_RUNTIME=ON -Dprotobuf_BUILD_SHARED_LIBS=OFF -Dprotobuf_BUILD_TESTS=OFF -Dprotobuf_BUILD_EXAMPLES=OFF .
         msbuild protobuf.sln /m /p:Configuration=Release
         msbuild INSTALL.vcxproj /p:Configuration=Release
         echo "Protobuf installation complete."
@@ -76,7 +72,7 @@ jobs:
         cd ../../../onnx
 
         echo "Build ONNX"
-        cmake -G "Visual Studio 16 2019" -A $arch -DONNX_USE_PROTOBUF_SHARED_LIBS=OFF -DONNX_USE_LITE_PROTO=ON -DONNX_WERROR=ON -DONNX_DISABLE_EXCEPTIONS=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release -DONNX_USE_MSVC_STATIC_RUNTIME=ON -DONNX_ML=1 -DONNX_BUILD_TESTS=ON -S . -B .setuptools-cmake-build\
+        cmake -G "Visual Studio 16 2019" -A ${{ matrix.architecture }} -DONNX_USE_PROTOBUF_SHARED_LIBS=OFF -DONNX_USE_LITE_PROTO=ON -DONNX_WERROR=ON -DONNX_DISABLE_EXCEPTIONS=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release -DONNX_USE_MSVC_STATIC_RUNTIME=ON -DONNX_ML=1 -DONNX_BUILD_TESTS=ON -S . -B .setuptools-cmake-build\
         cd .setuptools-cmake-build\
         msbuild onnx.sln /m /p:Configuration=Release
 
@@ -89,7 +85,7 @@ jobs:
         cd ..
         git clean -xdf
         echo "Build ONNX with non-static registration for testing selective ONNX schema loading"
-        cmake -G "Visual Studio 16 2019" -A $arch -DONNX_USE_PROTOBUF_SHARED_LIBS=OFF -DONNX_USE_LITE_PROTO=ON -DONNX_WERROR=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release -DONNX_USE_MSVC_STATIC_RUNTIME=ON -DONNX_ML=1 -DONNX_BUILD_TESTS=ON -DONNX_DISABLE_STATIC_REGISTRATION=ON -S . -B .setuptools-cmake-build\
+        cmake -G "Visual Studio 16 2019" -A ${{ matrix.architecture }} -DONNX_USE_PROTOBUF_SHARED_LIBS=OFF -DONNX_USE_LITE_PROTO=ON -DONNX_WERROR=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release -DONNX_USE_MSVC_STATIC_RUNTIME=ON -DONNX_ML=1 -DONNX_BUILD_TESTS=ON -DONNX_DISABLE_STATIC_REGISTRATION=ON -S . -B .setuptools-cmake-build\
         cd .setuptools-cmake-build\
         msbuild onnx.sln /m /p:Configuration=Release
 

docs/CIPipelines.md

+<!--- SPDX-License-Identifier: Apache-2.0 -->
+
+## ONNX CI Pipelines
+
+* CI pipelines matrix:
+
+  |   | When it runs | Config | Test |
+  -- | -- | -- | -- | 
+  [Linux-CI](https://github.com/onnx/onnx/blob/master/.azure-pipelines/Linux-CI.yml) | Every PR | <ul><li>Ubuntu-18.04</li><li>DEBUG=1 or 0</li><li>ONNX_USE_LITE_PROTO=ON or OFF</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_BUILD_TESTS=1</li><li>ONNX_WERROR=ON</li><li>ONNX_ML=1 or 0</li></ul>| <ul><li>ONNX C++ tests</li><li>Test flake8</li><li>Test doc generation</li><li>Test proto generation</li><li>mypy typecheck</li><li>Verify backend node data</li><li>Verify node test generation</li></ul> | 
+  [Windows-CI](https://github.com/onnx/onnx/blob/master/.azure-pipelines/Windows-CI.yml) | Every PR  | <ul><li>vs2017-win2016</li><li>ONNX_USE_LITE_PROTO=ON</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=ON</li><li>ONNX_BUILD_TESTS=1</li><li>ONNX_WERROR=ON</li><li>ONNX_ML=1 or 0</li></ul>| <ul><li>Test building ONNX in conda environment</li><li>Test doc generation</li><li>Test proto generation</li><li>mypy typecheck</li></ul> |
+  [Mac-CI](https://github.com/onnx/onnx/blob/master/.azure-pipelines/MacOS-CI.yml) | Every PR  | <ul><li>macOS-10.14</li><li>DEBUG=1</li><li>ONNX_USE_LITE_PROTO=ON or OFF</li><li>ONNX_ML=1 or 0</li><li>ONNX_BUILD_TESTS=1</li><li>ONNX_WERROR=ON</li></ul>| <ul><li>ONNX C++ tests</li><li>Test flake8</li><li>Test doc generation</li><li>Test proto generation</li></ul>|
+  [Windows_No_Exception CI](https://github.com/onnx/onnx/blob/master/.github/workflows/win_no_exception_ci.yml) | Every PR  | <ul><li>vs2019-winlatest</li><li>ONNX_DISABLE_EXCEPTIONS=ON</li><li>ONNX_USE_LITE_PROTO=ON</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li><li>ONNX_USE_MSVC_STATIC_RUNTIME=ON</li><li>ONNX_DISABLE_STATIC_REGISTRATION=ON or OFF</li></ul>| <ul><li>Only ONNX C++ tests</li><li>Test selective schema loading</li></ul> |
+  [WindowsRelease](https://github.com/onnx/onnx/blob/master/.github/workflows/release_win.yml) | <ul><li>Main branch</li><li>Release branch</li><li>Weekly(1)</li></ul> | <ul><li>Latest Windows</li><li>x86 and x64</li><li>ONNX_USE_LITE_PROTO=ON</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li><li>ONNX_USE_MSVC_STATIC_RUNTIME=OFF</li></ul>| <ul><li> Release Windows wheel</li><li>Release onnx-weekly package</li><li>Verify backend node data</li><li>Verify node test generation</li><li>Verify with different dependency versions - latest numpy version, latest and min supported protobuf version(2)</li><li>Verify ONNX with the latest [ort-nightly](https://test.pypi.org/project/ort-nightly/)(3).</li></ul> |
+  [LinuxRelease_aarch64](https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_aarch64.yml) | <ul><li>Main branch</li><li>Release branch</li><li>Weekly</li></ul>  | <ul><li>Latest manylinux2014_aarch64</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li></ul>| <ul><li> Release Linux aarch64 wheel</li><li>Release onnx-weekly package</li><li>Verify backend node data</li><li>Verify node test generation</li><li>Verify with different dependency versions - latest numpy version, latest and min supported protobuf version</li><li>Verify ONNX with the latest ort-nightly.</li></ul> |
+  [LinuxRelease_i686](https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_i686.yml) | <ul><li>Main branch</li><li>Release branch</li><li>Weekly</li></ul>  | <ul><li>Latest manylinux2010_x86_64</li><li> Release Linux i686 wheel</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li></ul> | <ul><li>Release onnx-weekly package</li><li>Verify backend node data</li><li>Verify node test generation</li><li>Verify with different dependency versions - latest numpy version, latest and min supported protobuf version</li></ul>
+  [LinuxRelease_x86_64](https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_x86_64.yml) | <ul><li>Main branch</li><li>Release branch</li><li>Weekly</li></ul> | <ul><li>Latest manylinux2014_aarch64</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li></ul>| <ul><li> Release Linux x86_64 wheel</li><li>Release onnx-weekly package</li><li>Test TEST_HUB=1(4)</li><li>Verify backend node data</li><li>Verify node test generation</li></li><li>Verify with different dependency versions - latest numpy version, latest and min supported protobuf version</li><li>Verify ONNX with the latest ort-nightly.</li></ul> |
+  [MacRelease](https://github.com/onnx/onnx/blob/master/.github/workflows/release_win.yml) | <ul><li>Main branch</li><li>Release branch</li><li>Weekly</li></ul> | <ul><li>macos-10.15</li><li> MACOSX_DEPLOYMENT_TARGET=10.12(5) </li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li></ul>| <ul><li>Release Mac wheel</li><li>Release onnx-weekly package</li><li>Verify backend node data</li><li>Verify node test generation</li><li>Verify with different dependency versions - latest numpy version, latest and min supported protobuf version</li><li>Verify ONNX with the latest ort-nightly.</li><li>Test source distribution generation</li><li>Test build with source distribution</li><li>Release onnx-weekly source distribution</li></ul>  
+  [Weekly CI with latest onnx.checker](https://github.com/onnx/onnx/blob/master/.github/workflows/weekly_mac_ci.yml) | weekly(6) |<ul><li>macos-latest</li><li>MACOSX_DEPLOYMENT_TARGET=10.15</li><li>ONNX_USE_PROTOBUF_SHARED_LIBS=OFF</li><li>ONNX_ML=1</li></ul>| <ul><li>Test latest ONNX checker</li><li>Test latest ONNX shape inference</li><li>With all models from [onnx/models](https://github.com/onnx/models)(7)</li></ul> |  
+  
+  * (1) When the release CIs will run:
+    * After a PR has been merged into main/rel-* branch
+    * Run weekly (Sunday midnight) and release Python wheel to [onnx-weekly](https://test.pypi.org/project/onnx-weekly/) package on TestPyPI.
+    * Any PR targeting rel-* branch
+    * To manually run them, add a PR label "run release CIs" (only maintainers have permission).
+  * (2) Minimum supported versions are listed [here](https://github.com/onnx/onnx/blob/master/requirements.txt).  
+  * (3) [Test](https://github.com/onnx/onnx/blob/master/onnx/test/test_with_ort.py) ONNX Python wheel with `onnxruntime.InferenceSession` from latest ONNXRuntime. Please note that ort-nightly does not support Linux-i686 and Windows-x86 thus their verification are skipped.  
+  * (4) TEST_HUB=1 will test [onnx.hub](https://github.com/onnx/onnx/blob/master/onnx/test/hub_test.py) by using this API to download an ONNX model from onnx/models. This test is restricted to only 1 pipeline for saving quota usage.
+  * (5) Although the build envioronment is macos-10.15, use MACOSX_DEPLOYMENT_TARGET=10.12 and -p [macosx_10_12_x86_64](https://github.com/onnx/onnx/blob/2e048660ffa8243596aaf3338e60c7c0575458f2/.github/workflows/release_mac.yml#L74) to force the wheel to support 10.12+.  
+  
+  * (6):
+    * The ONNX Model Zoo test will run weekly (Sunday midnight)
+    * To manually trigger it, add a PR label "test ONNX Model Zoo" (only maintainers have permission). Please note that it will need a lot of download bandwidth from [onnx/models](https://github.com/onnx/models) so use it with caution.
+  * (7) Some old deprecated models (opset-1) are [skipped](https://github.com/onnx/onnx/blob/master/workflow_scripts/config.py).

docs/CONTRIBUTING.md

@@ -88,7 +88,9 @@ After having installed mypy, you can run the type checks:
 ```
 python setup.py typecheck
 ```
+# CI Pipelines
 
+Every PR needs to pass CIs before merge. CI pipelines details are [here](CIPipelines.md).
 
 # Other developer documentation
 

docs/OnnxReleases.md

 <!--- SPDX-License-Identifier: Apache-2.0 -->
 
-The ONNX project, going forward, will plan to release roughly on a two month cadence. We follow the [Semver](https://semver.org/) versioning approach and will make decisions as a community on a release by release basis on whether to do a major or minor release.
+The ONNX project, going forward, will plan to release roughly on a four month cadence. We follow the [Semver](https://semver.org/) versioning approach and will make decisions as a community on a release by release basis on whether to do a major or minor release.
 
 ## Preparation
 

workflow_scripts/config.py

@@ -4,7 +4,6 @@
 # Skip them in test_model_zoo.py for now
 # TODO: fix these checker failures
 SKIP_CHECKER_MODELS = {'vision/classification/alexnet/model/bvlcalexnet-3.onnx',  # opset1 typeinference function missing
-                       'vision/classification/caffenet/model/caffenet-3.onnx',  # this should be caffenet-3.onnx; missing too
                        'vision/classification/densenet-121/model/densenet-3.onnx',  # opset1 typeinference function missing
                        'vision/classification/inception_and_googlenet/inception_v1/model/inception-v1-3.onnx',  # opset1 typeinference function missing
                        'vision/classification/inception_and_googlenet/inception_v2/model/inception-v2-3.onnx',  # opset1 typeinference function missing