Set up CI-IT action and Split Plugin-Tests (#3956)

* set up CI-IT action and Split Plugin-Tests * update

Set up CI-IT action and Split Plugin-Tests (#3956)
* set up CI-IT action and Split Plugin-Tests * update
125e96e7 · Daming · wu-sheng · b666dc7e · 125e96e7 · 125e96e7
3 changed file
--- a/.github/workflows/ci-it.yaml
+++ b/.github/workflows/ci-it.yaml
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: CI AND IT
+on: [pull_request, push]
+jobs:
+  CI:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 180
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: 'Install & Test'
+        run: |
+          export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx3g'
+          ./mvnw -P"agent,backend,ui,dist,CI-with-IT" clean verify install
+          ./mvnw javadoc:javadoc -Dmaven.test.skip=true
+      - name: 'Check Dependencies Licenses'
+        run: tools/dependencies/check-LICENSE.sh
--- a/.github/workflows/plugins-test.yaml
+++ b/.github/workflows/plugins-test.yaml
@@ -14,12 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
-name: Agent Plugins Tests
+name: PluginsTest
 on: [pull_request]
 jobs:
-  Group1:
+  Spring_Dubbo:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -43,8 +43,6 @@ jobs:
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
      - name: Run spring-tx 4.x+ (10)
        run: bash test/plugin/run.sh spring-tx-scenario
-      - name: Run spring 4.3.x-5.2.x (54)
-        run: bash test/plugin/run.sh spring-4.3.x-scenario
      - name: Run dubbo 2.5.x-2.6.x (10)
        run: bash test/plugin/run.sh dubbo-2.5.x-scenario
      - name: Run dubbo 2.7.x (4)
@@ -52,7 +50,7 @@ jobs:
      - name: Run okhttp 3.0.x-3.14.x (34)
        run: bash test/plugin/run.sh okhttp-scenario
-  Group2:
+  Spring31x_ES5_Gateway:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -74,8 +72,6 @@ jobs:
        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
      - name: Build the Docker image
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
-      - name: Run redisson 3.x (37)
-        run: bash test/plugin/run.sh redisson-scenario
      - name: Run spring 3.1.x-4.0.x (25)
        run: bash test/plugin/run.sh spring-3.1.x-scenario
      - name: Run spring-cloud-gateway 2.1.x (3)
@@ -83,7 +79,7 @@ jobs:
      - name: Run elasticsearch 5.x (3)
        run: bash test/plugin/run.sh elasticsearch-5.x-scenario
-  Group3:
+  Webflux_SocketIO_RabbitMQ:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -107,14 +103,14 @@ jobs:
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
      - name: Run apm-toolkit-trace (1)
        run: bash test/plugin/run.sh apm-toolkit-trace-scenario
-      - name: Run jetty 9.x (63)
+      - name: Run spring-webflux 2.x (18)
-        run: bash test/plugin/run.sh jetty-scenario
+        run: bash test/plugin/run.sh webflux-scenario
      - name: Run netty-socketio 1.x (4)
        run: bash test/plugin/run.sh netty-socketio-scenario
      - name: Run rabbitmq-scenario (12)
        run: bash test/plugin/run.sh rabbitmq-scenario
-  Group4:
+  Feign_Canal:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -140,12 +136,10 @@ jobs:
        run: bash test/plugin/run.sh feign-scenario
      - name: Run customize (1)
        run: bash test/plugin/run.sh customize-scenario
-      - name: Run postgresql 9.4.1207+ (62)
-        run: bash test/plugin/run.sh postgresql-above9.4.1207-scenario
      - name: Run canal 1.0.24-1.1.2 (5)
        run: bash test/plugin/run.sh canal-scenario
-  Group5:
+  Hystrix_SofaRPC_Spring30x:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -169,14 +163,12 @@ jobs:
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
      - name: Run hystrix 1.4.20-1.5.12 (20)
        run: bash test/plugin/run.sh hystrix-scenario
-      - name: Run postgresql 9.2.x-9.4.x (36)
-        run: bash test/plugin/run.sh postgresql-scenario
      - name: Run sofarpc 5.4.0-5.6.2 (23)
        run: bash test/plugin/run.sh sofarpc-scenario
      - name: Run spring 3.0.x (8)
        run: bash test/plugin/run.sh spring-3.0.x-scenario
-  Group6:
+  SpringAsync_gRPC:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -200,12 +192,10 @@ jobs:
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
      - name: Run spring async 4.3.x-5.1.x (35)
        run: bash test/plugin/run.sh spring-async-scenario
-      - name: Run mongodb 3.4.0-3.11.1 (17)
-        run: bash test/plugin/run.sh mongodb-3.x-scenario
      - name: Run grpc 1.6.0-1.25.0 (22)
        run: bash test/plugin/run.sh grpc-scenario
-  Group7:
+  Ehcache_Undertow_Jedis:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -227,10 +217,6 @@ jobs:
        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
      - name: Build the Docker image
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
-      - name: Run elasticsearch-6.x-scenario 6.7.1-6.8.4 (7)
-        run: bash test/plugin/run.sh elasticsearch-6.x-scenario
-      - name: Run kafka 0.11.0.0-2.3.0 (16)
-        run: bash test/plugin/run.sh kafka-scenario
      - name: Run ehcache 2.8.x-2.10.x (19)
        run: bash test/plugin/run.sh ehcache-2.x-scenario
      - name: Run undertow 1.3.0-2.0.27 (23)
@@ -238,7 +224,7 @@ jobs:
      - name: Run jedis 2.4.0-2.9.0 (18)
        run: bash test/plugin/run.sh jedis-scenario
-  Group8:
+  Spring41x_SolrJ-Http:
    runs-on: ubuntu-18.04
    timeout-minutes: 90
    strategy:
@@ -260,10 +246,6 @@ jobs:
        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
      - name: Build the Docker image
        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
-      - name: Run elasticsearch-7.x-scenario 7.0.0-7.4.2 (11)
-        run: bash test/plugin/run.sh elasticsearch-7.x-scenario
-      - name: Run spring-webflux 2.x (18)
-        run: bash test/plugin/run.sh webflux-scenario
      - name: Run spring 4.1.x-4.2.x (20)
        run: bash test/plugin/run.sh spring-4.1.x-scenario
      - name: Run solrj 7.x (12)
@@ -271,4 +253,206 @@ jobs:
      - name: Run httpclient 4.3.x-4.5.x (14)
        run: bash test/plugin/run.sh httpclient-4.3.x-scenario
      - name: Run httpasyncclient 4.0-4.1.3 (7)
        run: bash test/plugin/run.sh httpasyncclient-scenario
\ No newline at end of file
+  MongoDB:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run mongodb 3.4.0-3.11.1 (17)
+        run: bash test/plugin/run.sh mongodb-3.x-scenario
+  Redisson:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run redisson 3.x (37)
+        run: bash test/plugin/run.sh redisson-scenario
+  Spring43:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run spring 4.3.x-5.2.x (54)
+        run: bash test/plugin/run.sh spring-4.3.x-scenario
+  PostgreSQL94:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run postgresql 9.4.1207+ (62)
+        run: bash test/plugin/run.sh postgresql-above9.4.1207-scenario
+  PostgreSQL92:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run postgresql 9.2.x-9.4.x (36)
+        run: bash test/plugin/run.sh postgresql-scenario
+  Jetty:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run jetty 9.x (63)
+        run: bash test/plugin/run.sh jetty-scenario
+  ElasticSearch6_7:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run elasticsearch-7.x-scenario 7.0.0-7.4.2 (11)
+        run: bash test/plugin/run.sh elasticsearch-7.x-scenario
+      - name: Run elasticsearch-6.x-scenario 6.7.1-6.8.4 (7)
+        run: bash test/plugin/run.sh elasticsearch-6.x-scenario
+  Kafka:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v1
+        with:
+          submodules: true
+      - uses: actions/cache@v1
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build SkyWalking Agent
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
+      - name: Build the Docker image
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
+      - name: Run kafka 0.11.0.0-2.3.0 (16)
+        run: bash test/plugin/run.sh kafka-scenario
\ No newline at end of file
--- a/docs/en/guides/Plugin-test.md
+++ b/docs/en/guides/Plugin-test.md
@@ -557,129 +557,37 @@ rather than recompiling it every time.
 Use `${SKYWALKING_HOME}/test/plugin/run.sh -h` to know more command options.
-If the local test passed, then you could add it to Jenkins file, which will drive the tests running on the official SkyWalking INFRA.
+If the local test passed, then you could add it to `.github/workflows/plugins-test.yaml` file, which will drive the tests running on the Github Action of official SkyWalking repository. Please check the prev agent related PRs, and add your case to the fastest group, in order to make the whole test finished as soon as possible.
-We have 3 JenkinsFile to control the test jobs, `jenkinsfile-agent-test`, `jenkinsfile-agent-test-2` and `jenkinsfile-agent-test-3`(maybe will have 4 later)
-each file declares two parallel groups. Please check the prev agent related PRs, and add your case to the fastest group,
-in order to make the whole test finished as soon as possible.
-Every test case is a Jenkins stage. Please use the scenario name, version range and version number to combine the name,
+Every test case is a Github Action Step. Please use the scenario name, version range and version number to combine the name,
-take the existing one as a reference. And update the total case number in `Test Cases Report` stage.
+take the existing one as a reference. 
-Example
+For example:
+Here is a group named 'Kafka'. The latest step is Kafka scenario.
 ```
-stage('Test Cases Report (15)') { # 15=12+3 The total number of test cases
+  Kafka:
-    steps {
+    runs-on: ubuntu-18.04
-        echo "Test Cases Report"
+    timeout-minutes: 90
-    }
+    strategy:
-}
+      fail-fast: true
+    steps:
-stage('Run Agent Plugin Tests') {
+      - uses: actions/checkout@v1
-    when {
+        with:
-        expression {
+          submodules: true
-            return sh(returnStatus: true, script: 'bash tools/ci/agent-build-condition.sh')
+      - uses: actions/cache@v1
-        }
+        with:
-    }
+          path: ~/.m2/repository
-    parallel {
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
-        stage('Group1') {
+          restore-keys: |
-            stages {
+            ${{ runner.os }}-maven-
-                stage('spring-cloud-gateway 2.1.x (3)') { # For Spring clound gateway 2.1.x, having 3 versions to be tested.
+      - uses: actions/setup-java@v1
-                    steps {
+        with:
-                        sh 'bash test/plugin/run.sh gateway-scenario'
+          java-version: 8
-                    }
+      - name: Build SkyWalking Agent
-                }
+        run: ./mvnw clean package -DskipTests -Pagent >/dev/null
-            }
+      - name: Build the Docker image
-            ...
+        run: ./mvnw -f test/plugin/pom.xml clean package -DskipTests docker:build -DBUILD_NO=local >/dev/null
-        }
+      - name: Run kafka 0.11.0.0-2.3.0 (16)
-        stage('Group2') {
+        run: bash test/plugin/run.sh kafka-scenario
-            stages {
+```
-                stage('solrj 7.x (12)') { # For Solrj 7.x, having 12 versions to be tested.
\ No newline at end of file
-                    steps {
-                        sh 'bash test/plugin/run.sh solrj-7.x-scenario'
-                    }
-                }
-            }
-            ...
-        }
-    }
-}
-```
-## The elapsed time list of plugins
-### How to get the Elapsed time of your task?
-Find the button 'detail' of your Workload in the PR page. Enter to the page and get the elapsed time of your task.
-### Workload 1
-#### Group 1 (2709.00s)
-scenario name | versions | elapsed time (sec)
---|---|---
-apm-toolkit-trace | 1 | 87.00
-jetty 9.x | 63 | 2043.00
-netty-socketio 1.x | 4 | 117.00
-rabbitmq-scenario | 12 | 462
-#### Group 2 (2291.98s)
-scenario name | versions | elapsed time (sec)
---|---|---
-feign 9.0.0-9.5.1 | 8 | 172.00
-customize | 1 | 85.64
-postgresql 9.4.1207+ | 62 | 1820.29
-canal 1.0.24-1.1.2 | 5 | 214.05
-### Workload 2
-#### Group 1 (3936.54s)
-scenario name | versions | elapsed time (sec)
---|---|---
-spring-tx 4.x+ | 10 | 555.00
-spring 4.3.x-5.2.x | 54 | 1769.32
-dubbo 2.5.x-2.6.x | 10 | 367.23
-dubbo 2.7.x | 4 | 214.99
-okhttp 3.0.x-3.14.x | 34 | 1030
-#### Group 2 (2550.66s)
-scenario name | versions | elapsed time (sec)
---|---|---
-redisson 3.x | 37 | 1457.77
-spring 3.1.x-4.0.x | 25 | 760.22
-spring-cloud-gateway 2.1.x | 3 | 190.52
-elasticsearch 5.x | 3 | 142.15
-### Workload 3
-#### Group 1 (3090.912s)
-scenario name | versions | elapsed time (sec)
---|---|---
-hystrix-scenario | 20 | 799.00
-postgresql 9.2.x-9.4.x | 36 | 1243.03
-sofarpc 5.4.0-5.6.2 | 23 | 817.77
-spring 3.0.x | 8 | 231.11
-#### Group 2 (2433.33s)
-scenario name | versions | elapsed time (sec)
---|---|---
-spring async 4.3.x-5.1.x | 35 | 967.70
-mongodb 3.4.0-3.11.1 | 17 | 1465.63
-grcp 1.6.0-1.25.0 | 25 | 627.00
-### Workload 4
-#### Group 1 (2463.00s)
-scenario name | versions | elapsed time (sec)
---|---|---
-elasticsearch-6.x-scenario | 7 | 273.00
-kafka 0.11.0.0-2.3.0 | 16 | 628.00
-ehcache 2.8.x-2.10.x | 19 | 442.00
-undertow 1.3.0-2.0.27 | 23 | 596.00
-jedis 2.4.0-2.9.0 ｜ 18 ｜ 524.00
-#### Group 2 (2398.155s)
-scenario name | versions | elapsed time (sec)
---|---|---
-elasticsearch-7.x-scenario | 11 | 250.00
-spring-webflux 2.x | 18 | 705.60
-spring 4.1.x-4.2.x | 20 | 574.75
-solrj 7.x | 12 | 367.05
-httpclient 4.3.x-4.5.x | 14 | 300.61
-httpasyncclient 4.0-4.1.3 | 7 | 200.11