[wasm] Split `runtime-wasm` pipeline into `runtime-wasm-libtests`, and...

[wasm] Split `runtime-wasm` pipeline into `runtime-wasm-libtests`, and `runtime-wasm-non-libtests` (#74859)

Adds two new pipelines:
- `runtime-wasm-libtests` - runs all the wasm library tests
- `runtime-wasm-non-libtests` - runs all the wasm non-library tests like WBT, debugger etc.

And `runtime-wasm` runs the combination of the two above.

- For all 3 pipelines, jobs are run only if they weren't run on the default (`runtime`) pipeline

## Which pipeline should I run?

* if `runtime` didn't run the tests that would be useful for testing the changes, then:
  - If the changes only relate to libraries, the runtime, then use `runtime-wasm-libtests`
  - If the changes only affect `Wasm.Build.Tests`, or debugger tests, or runtime tests, then use `runtime-wasm-non-libtests`
  - Use `runtime-wasm` when you want to run *all* the wasm tests

# Updated CI matrix for wasm

## PR:

* `only-pc` means `only on relevant path changes`

### Run by default

* `runtime` runs jobs only when relevant paths change. And for `AOT`, only smoke tests are run.

| .                 | runtime                       |
| ----------------- | --------------------          |
| libtests          | linux+windows: all,   only-pc |
| libtests eat      | linux+windows: smoke, only-pc |
| libtests aot      | linux+windows: smoke, only-pc |
| high resource aot | none                          |
| Wasm.Build.Tests  | linux+windows:        only-pc |
| Debugger tests    | linux+windows:        only-pc |
| Runtime tests     | linux+windows:        only-pc |


### Run manually with `/azp run ..`

* `runtime-wasm*` pipelines are triggered manually, and they only run the jobs that would not run on any default pipelines based on path changes.
* The `AOT` jobs run only smoke tests on `runtime`, and on `runtime-wasm*` pipelines all the `AOT` tests are run.

| .                 | runtime-wasm               | runtime-wasm-libtests | runtime-wasm-non-libtests |
| ----------------- | -------------------------- | --------------------  | --------------------      |
| libtests          | linux+windows: all         | linux+windows: all    | none                      |
| libtests eat      | linux:         all         | linux:         all    | none                      |
| libtests aot      | linux+windows: all         | linux+windows: all    | none                      |
| high resource aot | linux+windows: all         | linux+windows: all    | none                      |
| Wasm.Build.Tests  | linux+windows              | none                  | linux+windows             |
| Debugger tests    | linux+windows              | none                  | linux+windows             |
| Runtime tests     | linux                      | none                  | linux                     |
| Perftrace         | linux: all tests           | linux: all tests      |                           |
| Multi-thread      | linux: all tests           | linux: all tests      |                           |

* `runtime-extra-platforms` does not run any wasm jobs on PRs
* `high resource aot` runs a few specific library tests with AOT, that require more memory to AOT.

## Rolling build (twice a day):

* `runtime` runs all the wasm jobs, but `AOT` still only runs smoke tests.
* `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by `runtime`.

| .                 | runtime                    | runtime-extra-platforms (always run) |
| ----------------- | -------------              | ------------------------------------ |
| libtests          | linux+windows: all         | none                                 |
| libtests eat      | linux: all                 | none                                 |
| libtests aot      | linux+windows: smoke       | linux+windows: all                   |
| high resource aot | none                       | linux+windows: all                   |
|                   |                            |                                      |
| Wasm.Build.Tests  | linux+windows              | none                                 |
| Debugger tests    | linux+windows              | none                                 |
| Runtime tests     | linux                      | none                                 |
| Perftrace         | linux: build only          | none                                 |
| Multi-thread      | linux: build only          | none                                 |

* `high resource aot` runs a few specific library tests with AOT, that require more memory to AOT.

eng/pipelines/common/templates/wasm-build-only.yml

@@ -27,16 +27,6 @@ jobs:
         value: ${{ parameters.alwaysRun }}
       - name: allWasmContainsChange
         value: $[ dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'] ]
-      - name: isDefaultPipeline
-        # default pipeline, so not runtime-wasm, and not
-        # runtime-extra-platforms+rolling build
-        value: ${{
-            and(
-              ne(parameters.isWasmOnlyBuild, true),
-              or(
-                ne(parameters.isExtraPlatformsBuild, true),
-                eq(variables['isRollingBuild'], true)))
-          }}
       - name: shouldRunOnDefaultPipelines
         value: $[
           or(

eng/pipelines/common/templates/wasm-build-tests.yml

@@ -26,16 +26,6 @@ jobs:
         value: $[ dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'] ]
       - name: alwaysRunVar
         value: ${{ parameters.alwaysRun }}
-      - name: isDefaultPipeline
-        # default pipeline, so not runtime-wasm, and not
-        # runtime-extra-platforms+rolling build
-        value: ${{
-            and(
-              ne(parameters.isWasmOnlyBuild, true),
-              or(
-                ne(parameters.isExtraPlatformsBuild, true),
-                eq(variables['isRollingBuild'], true)))
-          }}
       - name: shouldRunOnDefaultPipelines
         value: $[
           or(

eng/pipelines/common/templates/wasm-debugger-tests.yml

@@ -25,16 +25,6 @@ jobs:
         value: $[ dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'] ]
       - name: alwaysRunVar
         value: ${{ parameters.alwaysRun }}
-      - name: isDefaultPipeline
-        # default pipeline, so not runtime-wasm, and not
-        # runtime-extra-platforms+rolling build
-        value: ${{
-            and(
-              ne(parameters.isWasmOnlyBuild, true),
-              or(
-                ne(parameters.isExtraPlatformsBuild, true),
-                eq(variables['isRollingBuild'], true)))
-          }}
       - name: shouldRunOnDefaultPipelines
         value: $[
           or(

eng/pipelines/common/templates/wasm-library-tests.yml

@@ -34,16 +34,6 @@ jobs:
         value: ${{ parameters.alwaysRun }}
       - name: allWasmContainsChange
         value: $[ dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'] ]
-      - name: isDefaultPipeline
-        # default pipeline, so not runtime-wasm, and not
-        # runtime-extra-platforms+rolling build
-        value: ${{
-            and(
-              ne(parameters.isWasmOnlyBuild, true),
-              or(
-                ne(parameters.isExtraPlatformsBuild, true),
-                eq(variables['isRollingBuild'], true)))
-          }}
       - name: shouldRunOnDefaultPipelines
         value: $[
           or(

eng/pipelines/common/templates/wasm-runtime-tests.yml

@@ -25,16 +25,6 @@ jobs:
         value: 10
       - name: timeoutPerTestCollectionInMinutes
         value: 200
-      - name: isDefaultPipeline
-        # default pipeline, so not runtime-wasm, and not
-        # runtime-extra-platforms+!rolling build
-        value: ${{
-            and(
-              ne(parameters.isWasmOnlyBuild, true),
-              or(
-                ne(parameters.isExtraPlatformsBuild, true),
-                eq(variables['isRollingBuild'], true)))
-          }}
       - name: shouldRunOnDefaultPipelines
         value: $[
           or(
@@ -49,6 +39,7 @@ jobs:
       nameSuffix: AllSubsets_Mono_RuntimeTests
       runtimeVariant: monointerpreter
       buildArgs: -s mono+libs -c $(_BuildConfig)
+      timeoutInMinutes: 180
       # if !alwaysRun, then:
       #   if this is runtime-wasm (isWasmOnlyBuild):
       #     - then run only if it would not have run on default pipelines (based

eng/pipelines/common/variables.yml

@@ -18,20 +18,27 @@ variables:
 - name: isNotExtraPlatformsBuild
   value: ${{ ne(variables['Build.DefinitionName'], 'runtime-extra-platforms') }}
 - name: isWasmOnlyBuild
-  value: ${{ eq(variables['Build.DefinitionName'], 'runtime-wasm') }}
+  value: ${{ in(variables['Build.DefinitionName'], 'runtime-wasm', 'runtime-wasm-libtests', 'runtime-wasm-non-libtests') }}
 - name: isRunSmokeTestsOnly
-  value: ${{ and(ne(variables['Build.DefinitionName'], 'runtime-extra-platforms'), ne(variables['Build.DefinitionName'], 'runtime-wasm')) }}
+  value: ${{ notin(variables['Build.DefinitionName'], 'runtime-extra-platforms', 'runtime-wasm', 'runtime-wasm-libtests', 'runtime-wasm-non-libtests') }}
 - name: isNotSpecificPlatformOnlyBuild
-  value: ${{ ne(variables['Build.DefinitionName'], 'runtime-wasm') }}
+  value: ${{ notin(variables['Build.DefinitionName'], 'runtime-wasm', 'runtime-wasm-libtests', 'runtime-wasm-non-libtests') }}
 
 # We only run evaluate paths on runtime, runtime-staging and runtime-community pipelines on PRs
 # keep in sync with /eng/pipelines/common/xplat-setup.yml
 - name: dependOnEvaluatePaths
-  value: ${{ and(eq(variables['Build.Reason'], 'PullRequest'), in(variables['Build.DefinitionName'], 'runtime', 'runtime-staging', 'runtime-community', 'runtime-extra-platforms', 'runtime-wasm')) }}
+  value: ${{ and(eq(variables['Build.Reason'], 'PullRequest'), in(variables['Build.DefinitionName'], 'runtime', 'runtime-staging', 'runtime-community', 'runtime-extra-platforms', 'runtime-wasm', 'runtime-wasm-libtests', 'runtime-wasm-non-libtests', 'dotnet-linker-tests', 'runtime-dev-innerloop')) }}
 - name: debugOnPrReleaseOnRolling
   ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
     value: Release
   ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
     value: Debug
 
+- name: isDefaultPipeline
+  value: $[ and(
+              ne(variables['isWasmOnlyBuild'], true),
+              or(
+                ne(variables['isExtraPlatformsBuild'], true),
+                eq(variables['isRollingBuild'], true))) ]
+
 - template: /eng/pipelines/common/perf-variables.yml

eng/pipelines/common/xplat-setup.yml

@@ -18,7 +18,7 @@ jobs:
     shouldContinueOnError: ${{ or(eq(parameters.shouldContinueOnError, true), and(ne(parameters.shouldContinueOnError, 'forceFalse'), endsWith(variables['Build.DefinitionName'], 'staging'), eq(variables['Build.Reason'], 'PullRequest'))) }}
 
     # keep in sync with /eng/pipelines/common/variables.yml
-    dependOnEvaluatePaths: ${{ and(eq(variables['Build.Reason'], 'PullRequest'), in(variables['Build.DefinitionName'], 'runtime', 'runtime-staging', 'runtime-community', 'runtime-extra-platforms', 'runtime-wasm')) }}
+    dependOnEvaluatePaths: ${{ and(eq(variables['Build.Reason'], 'PullRequest'), in(variables['Build.DefinitionName'], 'runtime', 'runtime-staging', 'runtime-community', 'runtime-extra-platforms', 'runtime-wasm', 'runtime-wasm-libtests', 'runtime-wasm-non-libtests', 'dotnet-linker-tests', 'runtime-dev-innerloop')) }}
 
     variables:
       # Disable component governance in our CI builds. These builds are not shipping nor

eng/pipelines/runtime-extra-platforms-wasm.yml

@@ -7,6 +7,8 @@ parameters:
   isExtraPlatformsBuild: false
   isWasmOnlyBuild: false
   isRollingBuild: false
+  excludeLibTests: false
+  excludeNonLibTests: false
 
 jobs:
 
@@ -60,7 +62,7 @@ jobs:
 # - run everything, if relevant paths changed
 # - For runtime-wasm, force run all the jobs
 #
-- ${{ if ne(parameters.isRollingBuild, true) }}:
+- ${{ if and(ne(parameters.isRollingBuild, true), ne(parameters.excludeLibTests, true)) }}:
   # Library tests
   # these run on runtime also
   - template: /eng/pipelines/common/templates/wasm-library-tests.yml
@@ -174,6 +176,7 @@ jobs:
       isWasmOnlyBuild: ${{ parameters.isWasmOnlyBuild }}
       alwaysRun: ${{ parameters.isWasmOnlyBuild }}
 
+- ${{ if and(ne(parameters.isRollingBuild, true), ne(parameters.excludeNonLibTests, true)) }}:
   # Wasm.Build.Tests
   - template: /eng/pipelines/common/templates/wasm-build-tests.yml
     parameters:

eng/pipelines/runtime-wasm-libtests.yml

+trigger: none
+
+variables:
+  - template: /eng/pipelines/common/variables.yml
+
+jobs:
+
+#
+# Evaluate paths
+#
+- template: /eng/pipelines/common/evaluate-default-paths.yml
+
+- template: /eng/pipelines/runtime-extra-platforms-wasm.yml
+  parameters:
+    isExtraPlatformsBuild: ${{ variables.isExtraPlatformsBuild }}
+    isWasmOnlyBuild: ${{ variables.isWasmOnlyBuild }}
+    isRollingBuild: ${{ variables.isRollingBuild }}
+    excludeNonLibTests: true
+    excludeLibTests: false

eng/pipelines/runtime-wasm-non-libtests.yml

+trigger: none
+
+variables:
+  - template: /eng/pipelines/common/variables.yml
+
+jobs:
+
+#
+# Evaluate paths
+#
+- template: /eng/pipelines/common/evaluate-default-paths.yml
+
+- template: /eng/pipelines/runtime-extra-platforms-wasm.yml
+  parameters:
+    isExtraPlatformsBuild: ${{ variables.isExtraPlatformsBuild }}
+    isWasmOnlyBuild: ${{ variables.isWasmOnlyBuild }}
+    isRollingBuild: ${{ variables.isRollingBuild }}
+    excludeNonLibTests: false
+    excludeLibTests: true

src/mono/wasm/README.md

@@ -230,38 +230,60 @@ npm update
 ### How do I know which jobs run on CI, and when?
 
 ## PR:
-* `runtime-extra-platforms`, and `runtime-wasm` run only when manually triggered with a comment - `/azp run <pipeline-name>`
-* `runtime`, and `runtime-staging`, run jobs only when relevant paths change. And for `EAT`, and `AOT`, only smoke tests are run.
-* And when `runtime-wasm` is triggered manually, it runs *all* the wasm jobs completely
-
-| .                 | runtime               | runtime-staging         | runtime-extra-platforms(manual only) | runtime-wasm (manual only) |
-| ----------------- | --------------------  | ---------------         | ------------------------------------ | -------                    |
-| libtests          | linux: all,   only-pc | windows: all,   only-pc | linux+windows: all, only-pc          | linux+windows: all, always |
-| libtests eat      | linux: smoke, only-pc | -                       | linux:         all, only-pc          | linux:         all, always |
-| libtests aot      | linux: smoke, only-pc | windows: smoke, only-pc | linux+windows: all, only-pc          | linux+windows: all, always |
-| high resource aot | none                  | none                    | linux+windows: all, only-pc          | linux+windows: all, always |
-| Wasm.Build.Tests  | linux:        only-pc | windows:        only-pc | linux+windows: only-pc               | linux+windows              |
-| Debugger tests    | -                     | linux+windows:  only-pc | linux+windows: only-pc               | linux+windows              |
-| Runtime tests     | linux:        only-pc | -                       | linux: only-pc                       | linux                      |
 
+* `only-pc` means `only on relevant path changes`
+
+### Run by default
+
+* `runtime` runs jobs only when relevant paths change. And for `AOT`, only smoke tests are run.
+
+| .                 | runtime                       |
+| ----------------- | --------------------          |
+| libtests          | linux+windows: all,   only-pc |
+| libtests eat      | linux+windows: smoke, only-pc |
+| libtests aot      | linux+windows: smoke, only-pc |
+| high resource aot | none                          |
+| Wasm.Build.Tests  | linux+windows:        only-pc |
+| Debugger tests    | linux+windows:        only-pc |
+| Runtime tests     | linux+windows:        only-pc |
+
+
+### Run manually with `/azp run ..`
+
+* `runtime-wasm*` pipelines are triggered manually, and they only run the jobs that would not run on any default pipelines based on path changes.
+* The `AOT` jobs run only smoke tests on `runtime`, and on `runtime-wasm*` pipelines all the `AOT` tests are run.
+
+| .                 | runtime-wasm               | runtime-wasm-libtests | runtime-wasm-non-libtests |
+| ----------------- | -------------------------- | --------------------  | --------------------      |
+| libtests          | linux+windows: all         | linux+windows: all    | none                      |
+| libtests eat      | linux:         all         | linux:         all    | none                      |
+| libtests aot      | linux+windows: all         | linux+windows: all    | none                      |
+| high resource aot | linux+windows: all         | linux+windows: all    | none                      |
+| Wasm.Build.Tests  | linux+windows              | none                  | linux+windows             |
+| Debugger tests    | linux+windows              | none                  | linux+windows             |
+| Runtime tests     | linux                      | none                  | linux                     |
+| Perftrace         | linux: all tests           | linux: all tests      | none                      |
+| Multi-thread      | linux: all tests           | linux: all tests      | none                      |
+
+* `runtime-extra-platforms` does not run any wasm jobs on PRs
 * `high resource aot` runs a few specific library tests with AOT, that require more memory to AOT.
 
 ## Rolling build (twice a day):
 
-* `runtime`, and `runtime-staging`, run all the wasm jobs unconditionally, but `EAT`, and `AOT` still run only smoke tests.
-* `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by the above two pipelines.
-
-* jobs w/o `only-pc` are always run
-
-| .                 | runtime                   | runtime-staging       | runtime-extra-platforms (always run) | runtime-wasm (manual only) |
-| ----------------- | -------------             | ---------------       | ------------------------------------ | ------                     |
-| libtests          | linux: all(v8/chr)        | windows: all          | none                                 | N/A                        |
-| libtests eat      | linux: smoke              | -                     | linux: all                           |                            |
-| libtests aot      | linux: smoke              | windows: smoke        | linux+windows: all                   |                            |
-| high resource aot | none                      | none                  | linux+windows: all                   |                            |
-|                   |                           |                       |                                      |                            |
-| Wasm.Build.Tests  | linux: always             | windows: always       | none                                 |                            |
-| Debugger tests    | -                         | linux+windows: always | none                                 |                            |
-| Runtime tests     | linux: always             | -                     | none                                 |                            |
+* `runtime` runs all the wasm jobs, but `AOT` still only runs smoke tests.
+* `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by `runtime`.
+
+| .                 | runtime                    | runtime-extra-platforms (always run) |
+| ----------------- | -------------              | ------------------------------------ |
+| libtests          | linux+windows: all         | none                                 |
+| libtests eat      | linux: all                 | none                                 |
+| libtests aot      | linux+windows: smoke       | linux+windows: all                   |
+| high resource aot | none                       | linux+windows: all                   |
+|                   |                            |                                      |
+| Wasm.Build.Tests  | linux+windows              | none                                 |
+| Debugger tests    | linux+windows              | none                                 |
+| Runtime tests     | linux                      | none                                 |
+| Perftrace         | linux: build only          | none                                 |
+| Multi-thread      | linux: build only          | none                                 |
 
 * `high resource aot` runs a few specific library tests with AOT, that require more memory to AOT.