[JIT] CodeGen verification testing (#75102)

* Add test

* Add CHECKs

* Build scripts and test settings

* COMPlus -> DOTNET

* Fix filenames

* Missing >

* Missing quotes

* use %scriptPath%

* Rework properties, start two examples

* Arch demo

* Cleanup

* Remove tab

* Easier environment variables.  Undo precommand changes.

* undo blank line

* clean CHECKs

* Draft of bash

* Bash, conditionals

* More variables, start on run.cmd/sh/py

* another output

* Support in cmd/bash for RunningDisasmChecks

* copy, factor, formatting

* Initial work to include FileCheck. Added SuperFileCheck.

* Able to build SuperFileCheck

* Do not DisasmCheck TypeEquality_r for now. Update some FileChecks to follow SuperFileCheck rules.

* Partially wiring up SuperFileCheck to tests

* Piping list of method names from SuperFileCheck to JitDisasm

* Handling bash a little bit

* Moving SuperFileCheck to tests/Common

* Few tweaks

* Building SuperFileCheck as part of the test build

* Tweaking a few things

* Fixed a bug

* Moving SuperFileCheck back to src\coreclr\tools. Removed checks from TypeEquality_r.

* Restore original logic in Runtime_73681

* Trying to add CI leg for disasmchecks

* Use x64 package if x86 platform detected for JIT tools package

* Remove innerloop for disasmchecks

* Trying to fix build. Only run in Windows for now.

* Update Runtime_73681.cs

Trying to fail test

* Trying to fix build

* Update Runtime_73681.cs

* Update Runtime_73681.cs

* Fixing a few issues

* Trying to run disasmchecks as part of CI

* Trying to run disasmchecks

* Trying to run disasmchecks

* Trying to run disasmchecks

* Revert a change

* Trying to run disasmchecks

* Trying to run disasmchecks

* build SuperFileCheck on non-windows

* few tweaks

* Trying to fix CI

* Including SuperFileCheck for tests

* Cleanup

* More cleanup

* Cleanup

* Changed SuperFileCheck to not publish everything. Changed SuperFileCheck's lookup for FileCheck.

* Invoking SuperFileCheck using dotnet

* Making the test pass

* Only run disasm checks for coreclr and not mono

* Using HasBatchDisasmCheck and HasBashDisasmCheck to determine to run the check

* Enabling filecheck on linux and osx

* Added more comments

* Added ARM64 specific test. Do not run SuperFileCheck if no methods were found.

* Added documentation. Changed disasm-output.

* Minor doc tweak

* Minor doc tweak

* Minor doc tweak

* Minor doc tweak

* Minor doc tweak

* Cleanup. Trying to fix linux

* Fixing test

* Add information on additional functionality

* cleanup

* Add FileCheck snippet

* Undo environment variable changes

* Feedback from Mark

* Cleanup

* Trying to fix linux test run

* Trying to fix linux test run

* A few missing changes from the original branch

* Enable OSX for disasm checks

* cleanup / comment

* Force test failure

* Update Runtime_73681.cs

* Set env vars after errorlevel check

* Reverting back on setting environment variables in test. Added new FileCheck test for mod optimization

* Force a failure by changing the register

* Ignore native binaries for superpmi collect

* Update Runtime_34937.cs

* Force the correct failure

* Update Runtime_34937.cs

* Update Runtime_34937.cs

* Adding specific OS check prefixes. Changed dump-input context amount

* Added getting fully qualified method names with wildcards for SuperFileCheck

* More tests. Fixed a few issues with generics.

* Disabling generic support

* Error if it cannot find enclosing type declaration

* Fixing build

* Remove namespac

* Bring generics back, but in a limited form
Co-authored-by: NMark Plesko <markples@microsoft.com>

docs/workflow/testing/coreclr/disasm-checks.md

+# Disassembly output verification checks
+There are tests that the runtime executes that will be able to verify X64/ARM64 assembly output from the JIT.
+The tools used to accomplish this are LLVM FileCheck, SuperFileCheck, and the JIT's ability to output disassembly using `DOTNET_JitDisasm`. LLVM FileCheck is built in https://www.github.com/dotnet/llvm-project and provides several packages for the various platforms. See more about LLVM FileCheck and its syntax here: https://llvm.org/docs/CommandGuide/FileCheck.html. SuperFileCheck is a custom tool located in https://www.github.com/dotnet/runtime. It wraps LLVM FileCheck and provides a simplified workflow for writing these tests in a C# file by leveraging Roslyn's syntax tree APIs.
+# What is FileCheck?
+From https://www.llvm.org/docs/CommandGuide/FileCheck.html:
+
+> **FileCheck** reads two files (one from standard input, and one specified on the command line) and uses one
+to verify the other. This behavior is particularly useful for the testsuite, which wants to verify that the
+output of some tool (e.g. **llc**) contains the expected information (for example, a movsd from esp or
+whatever is interesting). This is similar to using **grep**, but it is optimized for matching multiple
+different inputs in one file in a specific order.
+# Converting an existing test to use disassembly checking
+We will use the existing test `JIT\Regression\JitBlue\Runtime_33972` as an example. The test's intent is to verify that on ARM64, the method `AdvSimd.CompareEqual` behaves correctly when a zero vector is passed as the second argument. Below are snippets of its use:
+```csharp
+    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
+    {
+        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
+    }
+...
+...
+        if (!ValidateResult_Vector64<byte>(AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte>.Zero), Byte.MaxValue))
+            result = -1;
+```
+Currently, the test only verifies that the behavior is correct. It does not verify that the optimal ARM64 instruction was actually used. So now we will add this verification.
+First we need to modify the project file `Runtime_33972.csproj`:
+```xml
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+  </PropertyGroup>
+  <PropertyGroup>
+    <DebugType>None</DebugType>
+    <Optimize>True</Optimize>
+    <AllowUnsafeBlocks>True</AllowUnsafeBlocks>
+  </PropertyGroup>
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs" />
+  </ItemGroup>
+</Project>
+```
+Looking at the `ItemGroup`:
+```xml
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs" />
+  </ItemGroup>
+```
+We want to add `<HasDisasmCheck>true</HasDisasmCheck>` as a child of the `Compile` tag:
+```xml
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs">
+      <HasDisasmCheck>true</HasDisasmCheck>
+    </Compile>
+  </ItemGroup>
+```
+Doing this lets the test builder and runner know that this test has assembly that needs to be verified. Finally, we need to write the assembly check and put the `[MethodImpl(MethodImplOptions.NoInlining)]` attribute on the method `AdvSimd_CompareEqual_Vector64_Byte_Zero`:
+```csharp
+    [MethodImpl(MethodImplOptions.NoInlining)]
+    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
+    {
+        // ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0
+        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
+    }
+```
+And that is it. A few notes about the above example:
+- `ARM64-FULL-LINE` checks to see if there is an exact line that matches the disassembly output of the method `AdvSimd_CompareEqual_Vector64_Byte_Zero` - leading and trailing spaces are ignored.
+- Method bodies that have FileCheck syntax, e.g. `ARM64-FULL-LINE:`/`X64:`/etc, must have the attribute `[MethodImpl(MethodImplOptions.NoInlining)]`. If it does not, then an error is reported.
+- FileCheck syntax outside of a method body will also report an error.
+# Additional functionality
+LLVM has a different setup where each test file is passed to `lit`, and `RUN:` lines inside the test specify
+configuration details such as architectures to run, FileCheck prefixes to use, etc.  In our case, the build
+files handle a lot of this with build conditionals and `.cmd`/`.sh` file generation.  Additionally, LLVM tests
+rely on the order of the compiler output corresponding to the order of the input functions in the test file.
+When running under the JIT, the compilation order is dependent on execution, not the source order.
+
+Functionality that has been added or moved to MSBuild:
+- Conditionals controlling test execution
+- Automatic specificiation of `CHECK` and `<architecture>` as check prefixes
+
+Functionality that has been added or moved to SuperFileCheck:
+- Each function is run under a separate invocation of FileCheck. SuperFileCheck adds additional `CHECK` lines
+  that search for the beginning and end of the output for each function. This ensures that output from
+  different functions don't contaminate each other. The separate invocations remove any dependency on the
+  order of the functions.
+- `<check-prefix>-FULL-LINE:` - same as using FileCheck's `<check-prefix>:`, but checks that the line matches exactly; leading and trailing whitespace is ignored.
+- `<check-prefix>-FULL-LINE-NEXT:` - same as using FileCheck's `<check-prefix>-NEXT:`, but checks that the line matches exactly; leading and trailing whitespace is ignored.

eng/Subsets.props

@@ -313,6 +313,8 @@
 
     <ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and '$(TargetArchitecture)' != '$(BuildArchitecture)'" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />
     <ProjectToBuild Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)'" Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_crossarch.csproj" Category="clr" />
+
+    <ProjectToBuild Condition="'$(TargetOS)' == 'windows' or ('$(TargetOS)' == 'Linux' and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')) or '$(TargetOS)' == 'OSX'" Include="$(CoreClrProjectRoot)tools\SuperFileCheck\SuperFileCheck.csproj" Category="clr" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subset.Contains('+clr.toolstests+'))">

eng/Version.Details.xml

@@ -48,6 +48,38 @@
       <Uri>https://github.com/dotnet/llvm-project</Uri>
       <Sha>8688ea2538ef1287c6619d35a26b5750d355cc18</Sha>
     </Dependency>
+    <Dependency Name="runtime.linux-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-musl-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>754d13817d834b716d339183e21aac7d2489c496</Sha>
+    </Dependency>
+    <Dependency Name="runtime.win-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.win-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.osx.11.0-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
     <Dependency Name="System.CommandLine" Version="2.0.0-beta4.22355.1">
       <Uri>https://github.com/dotnet/command-line-api</Uri>
       <Sha>5618b2d243ccdeb5c7e50a298b33b13036b4351b</Sha>

eng/Versions.props

@@ -204,5 +204,14 @@
     <SwixPackageVersion>1.1.87-gba258badda</SwixPackageVersion>
     <WixPackageVersion>1.0.0-v3.14.0.5722</WixPackageVersion>
     <MonoWorkloadManifestVersion>6.0.0-preview.5.21275.7</MonoWorkloadManifestVersion>
+    <!-- JIT Tools -->
+    <runtimelinuxarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxmuslarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxmuslarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxmuslx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxmuslx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimewinarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimewinarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimewinx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimewinx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimeosx110arm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimeosx110arm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimeosx1012x64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimeosx1012x64MicrosoftNETCoreRuntimeJITToolsVersion>
   </PropertyGroup>
 </Project>

src/coreclr/scripts/superpmi_collect_setup.py

@@ -168,6 +168,9 @@ native_binaries_to_ignore = [
     "vcruntime140_1.dll",
     "R2RDump.exe",
     "R2RTest.exe",
+    "FileCheck.exe",
+    "SuperFileCheck.exe",
+    "llvm-mca.exe",
     "superpmi.exe",
     "superpmi-shim-collector.dll",
     "superpmi-shim-counter.dll",

src/coreclr/tools/SuperFileCheck/SuperFileCheck.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <AssemblyName>SuperFileCheck</AssemblyName>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>
+    <PlatformTarget>AnyCPU</PlatformTarget>
+    <AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
+    <AppendTargetFrameworkToOutputPath Condition="'$(BuildingInsideVisualStudio)' == 'true'">true</AppendTargetFrameworkToOutputPath>
+    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
+    <EnableDefaultEmbeddedResourceItems>false</EnableDefaultEmbeddedResourceItems>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+    <OutputPath>$(RuntimeBinDir)\SuperFileCheck</OutputPath>
+  </PropertyGroup>
+
+  <PropertyGroup>
+    <!-- Massage the OutputRid into an JITTools package RID that we can download -->
+    <_jitToolsRidPlatformIndex>$(OutputRid.LastIndexOf('-'))</_jitToolsRidPlatformIndex>
+    <JITToolsRidWithoutPlatform>$(OutputRid.Substring(0, $(_jitToolsRidPlatformIndex)))</JITToolsRidWithoutPlatform>
+    <JITToolsRidPlatform>$(OutputRid.Substring($(_jitToolsRidPlatformIndex)).TrimStart('-'))</JITToolsRidPlatform>
+
+    <!-- If it's not win/osx/linux-musl, it's a non-portable Linux. Treat as Linux. -->
+    <JITToolsRidWithoutPlatform Condition="'$(JITToolsRidWithoutPlatform)' != 'win' and '$(JITToolsRidWithoutPlatform)' != 'osx' and '$(JITToolsRidWithoutPlatform)' != 'linux-musl'">linux</JITToolsRidWithoutPlatform>
+
+    <!-- OSX builds have a version -->
+    <JITToolsRidWithoutPlatform Condition="'$(JITToolsRidWithoutPlatform)' == 'osx' and '$(JITToolsRidPlatform)' == 'x64'">osx.10.12</JITToolsRidWithoutPlatform>
+    <JITToolsRidWithoutPlatform Condition="'$(JITToolsRidWithoutPlatform)' == 'osx' and '$(JITToolsRidPlatform)' == 'arm64'">osx.11.0</JITToolsRidWithoutPlatform>
+
+    <!-- There are no x86 packages, so use x64 -->
+    <JITToolsRidPlatform Condition="'$(JITToolsRidPlatform)' == 'x86'">x64</JITToolsRidPlatform>
+    <JITToolsRidPlatform Condition="'$(JITToolsRidPlatform)' == 'arm' and '$(JITToolsRidWithoutPlatform)' == 'win'">arm64</JITToolsRidPlatform>
+    <JITToolsRid Condition="'$(JITToolsRid)' == ''">$(JITToolsRidWithoutPlatform)-$(JITToolsRidPlatform)</JITToolsRid>
+
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'linux-arm64'">$(runtimelinuxarm64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'linux-x64'">$(runtimelinuxx64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'linux-musl-arm64'">$(runtimelinuxmuslarm64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'linux-musl-x64'">$(runtimelinuxmuslx64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'win-arm64'">$(runtimewinarm64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'win-x64'">$(runtimewinx64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'osx.11.0-arm64'">$(runtimeosx110arm64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+    <JITToolsVersion Condition="'$(JITToolsVersion)' == '' and '$(JITToolsRid)' == 'osx.10.12-x64'">$(runtimeosx1012x64MicrosoftNETCoreRuntimeJITToolsVersion)</JITToolsVersion>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="$(MicrosoftCodeAnalysisCSharpVersion)" />
+    <PackageReference Include="runtime.$(JITToolsRid).Microsoft.NETCore.Runtime.JIT.Tools">
+      <Version>$(JITToolsVersion)</Version>
+    </PackageReference>
+  </ItemGroup>
+
+</Project>

src/tests/Common/CLRTest.Execute.Bash.targets

@@ -430,8 +430,13 @@ echo /usr/bin/env bash $(InputAssemblyName) %24(printf "'%s' " "${CLRTestExecuti
 CLRTestExitCode=$?
 CLRTestExpectedExitCode=0
       ]]></BashCLRTestLaunchCmds>
-    </PropertyGroup>
 
+      <BashCLRTestPostCommands><![CDATA[
+$(BashDisasmCheckPostCommands)
+$(CLRTestBashPostCommands)
+      ]]></BashCLRTestPostCommands>
+
+    </PropertyGroup>
     <PropertyGroup>
       <BashEnvironmentVariables>
 @(CLRTestBashEnvironmentVariable -> '%(Identity)', '%0a')

src/tests/Common/CLRTest.Execute.Batch.targets

@@ -301,6 +301,7 @@ $(BatchIlrtTestLaunchCmds)
 if defined RunCrossGen2 (
   call :TakeLock
 )
+
 ECHO %LAUNCHER% %ExePath% %CLRTestExecutionArguments%
 %LAUNCHER% %ExePath% %CLRTestExecutionArguments%
 set CLRTestExitCode=!ERRORLEVEL!
@@ -338,6 +339,12 @@ cmd /c $(InputAssemblyName)
 set CLRTestExitCode=!ERRORLEVEL!
 set CLRTestExpectedExitCode=0
       ]]></BatchCLRTestLaunchCmds>
+
+      <BatchCLRTestPostCommands><![CDATA[
+$(BatchDisasmCheckPostCommands)
+$(CLRTestBatchPostCommands)
+      ]]></BatchCLRTestPostCommands>
+
     </PropertyGroup>
     <PropertyGroup>
       <BatchEnvironmentVariables>
@@ -445,7 +452,7 @@ $(CLRTestBatchPreCommands)
 REM Launch
 $(BatchCLRTestLaunchCmds)
 REM PostCommands
-$(CLRTestBatchPostCommands)
+$(BatchCLRTestPostCommands)
 $(BatchCLRTestExitCodeCheck)
       ]]></_CLRTestExecutionScriptText>
     </PropertyGroup>

src/tests/Common/CLRTest.Jit.targets

@@ -20,6 +20,11 @@ WARNING:   When setting properties based on their current state (for example:
 -->
 <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 
+  <PropertyGroup>
+    <BashScriptSnippetGen>$(BashScriptSnippetGen);GetIlasmRoundTripBashScript;GetDisasmCheckBashScript</BashScriptSnippetGen>
+    <BatchScriptSnippetGen>$(BatchScriptSnippetGen);GetIlasmRoundTripBatchScript;GetDisasmCheckBatchScript</BatchScriptSnippetGen>
+  </PropertyGroup>
+
   <!--
   ***********************************************************************************************
   ildasm / ilasm round trip testing
@@ -187,6 +192,115 @@ IF NOT DEFINED DoLink (
     </PropertyGroup>
   </Target>
 
+  <Target Name="GetDisasmCheckData"
+          Returns="$(HasDisasmCheck);@(DisasmCheckFiles)">
+    <ItemGroup>
+      <DisasmCheckFiles Include="%(Compile.Identity)" Condition="'%(Compile.HasDisasmCheck)' == 'true'" />
+    </ItemGroup>
+    <PropertyGroup>
+      <HasDisasmCheck>false</HasDisasmCheck>
+      <HasDisasmCheck Condition="@(DisasmCheckFiles->Count()) &gt; 0 And '$(CLRTestKind)' == 'BuildAndRun'">true</HasDisasmCheck>
+    </PropertyGroup>
+  </Target>
+
+  <!--
+  The source code file(s) (specified by HasDisasmCheck in the test's project file) contain
+  the patterns used by FileCheck.  They need to be copied to the output directory for the
+  test to be self-contained.
+  -->
+  <Target Name="PropagateHasDisasmCheckToCopy"
+          BeforeTargets="AssignTargetPaths">
+    <ItemGroup>
+      <Compile Update="@(Compile)"
+               Condition="'%(Compile.HasDisasmCheck)' == 'true'"
+               CopyToOutputDirectory="PreserveNewest" />
+    </ItemGroup>
+  </Target>
+
+  <Target Name="GetDisasmCheckBashScript"
+          Returns="$(HasBashDisasmCheck);$(BashDisasmOutputFile);$(BashDisasmCheckPostCommands)"
+          DependsOnTargets="GetDisasmCheckData">
+    <PropertyGroup>
+      <HasBashDisasmCheck>false</HasBashDisasmCheck>
+      <HasBashDisasmCheck Condition="'$(HasDisasmCheck)' == 'true' and '$(RuntimeFlavor)' == 'coreclr' and ('$(TargetOS)' == 'Linux' or '$(TargetOS)' == 'OSX') and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')">true</HasBashDisasmCheck>
+
+      <BashDisasmOutputFile>/dev/null</BashDisasmOutputFile>
+      <BashDisasmOutputFile Condition="'$(HasBashDisasmCheck)' == 'true'">$(scriptPath)__jit_disasm.out</BashDisasmOutputFile>
+
+      <BashDisasmListOutputFile>/dev/null</BashDisasmListOutputFile>
+      <BashDisasmListOutputFile Condition="'$(HasBashDisasmCheck)' == 'true'">$(scriptPath)__jit_disasm_list.out</BashDisasmListOutputFile>
+
+      <BashCLRTestPreCommands Condition="'$(HasBashDisasmCheck)' == 'true'"><![CDATA[
+@(DisasmCheckFiles -> '  dotnet $CORE_ROOT/SuperFileCheck/SuperFileCheck.dll --csharp-list-method-names "%(Identity)" --allow-unused-prefixes --check-prefixes=CHECK,$(TargetArchitecture.ToUpperInvariant()),$(TargetArchitecture.ToUpperInvariant())-$(TargetOS.ToUpperInvariant()) > "$(BashDisasmListOutputFile)"
+  ERRORLEVEL=$?
+  export COMPlus_JitDisasm=`cat $(BashDisasmListOutputFile)`
+  export COMPlus_JitDiffableDasm=1
+  export COMPlus_JitStdOutFile=$(BashDisasmOutputFile)
+  if [[ $ERRORLEVEL -ne 0 ]]
+  then
+    echo EXECUTION OF FILECHECK - FAILED $ERRORLEVEL
+    exit 1
+  fi', '%0a')
+]]>
+      </BashCLRTestPreCommands>
+
+      <BashDisasmCheckPostCommands></BashDisasmCheckPostCommands>
+      <BashDisasmCheckPostCommands Condition="'$(HasBashDisasmCheck)' == 'true'"><![CDATA[
+if [[ -n $COMPlus_JitDisasm ]]; then
+  @(DisasmCheckFiles -> '  dotnet $CORE_ROOT/SuperFileCheck/SuperFileCheck.dll --csharp "%(Identity)" --allow-unused-prefixes --check-prefixes=CHECK,$(TargetArchitecture.ToUpperInvariant()),$(TargetArchitecture.ToUpperInvariant())-$(TargetOS.ToUpperInvariant()) --dump-input-context 25 --input-file "$(BashDisasmOutputFile)"
+    ERRORLEVEL=$?
+    if [[ $ERRORLEVEL -ne 0 ]]
+    then
+      echo EXECUTION OF FILECHECK - FAILED $ERRORLEVEL
+      exit 1
+    fi', '%0a')
+fi
+]]>
+      </BashDisasmCheckPostCommands>
+    </PropertyGroup>
+  </Target>
+
+  <Target Name="GetDisasmCheckBatchScript"
+          Returns="$(HasBatchDisasmCheck);$(BatchDisasmOutputFile);$(BatchDisasmCheckPostCommands)"
+          DependsOnTargets="GetDisasmCheckData">
+    <PropertyGroup>
+      <HasBatchDisasmCheck>false</HasBatchDisasmCheck>
+      <HasBatchDisasmCheck Condition="'$(HasDisasmCheck)' == 'true' and '$(RuntimeFlavor)' == 'coreclr'">true</HasBatchDisasmCheck>
+
+      <BatchDisasmOutputFile>NUL</BatchDisasmOutputFile>
+      <BatchDisasmOutputFile Condition="'$(HasBatchDisasmCheck)' == 'true'">$(scriptPath)__jit_disasm.out</BatchDisasmOutputFile>
+
+      <BatchDisasmListOutputFile>NUL</BatchDisasmListOutputFile>
+      <BatchDisasmListOutputFile Condition="'$(HasBatchDisasmCheck)' == 'true'">$(scriptPath)__jit_disasm_list.out</BatchDisasmListOutputFile>
+
+      <CLRTestBatchPreCommands Condition="'$(HasBatchDisasmCheck)' == 'true'">
+<![CDATA[
+      $(CLRTestBatchPreCommands)
+@(DisasmCheckFiles -> '  dotnet %CORE_ROOT%\SuperFileCheck\SuperFileCheck.dll --csharp-list-method-names "%(Identity)" --check-prefixes=CHECK,$(TargetArchitecture.ToUpperInvariant()),$(TargetArchitecture.ToUpperInvariant())-$(TargetOS.ToUpperInvariant()) > "$(BatchDisasmListOutputFile)"
+  IF NOT "!ERRORLEVEL!" == "0" (
+    ECHO EXECUTION OF FILECHECK LISTING METHOD NAMES - FAILED !ERRORLEVEL!
+    Exit /b 1
+  )', '%0d%0a')
+  for /F "delims=" %%g in ($(BatchDisasmListOutputFile)) do set COMPlus_JitDisasm=%%g
+  set COMPlus_JitDiffableDasm=1
+  set COMPlus_JitStdOutFile=$(BatchDisasmOutputFile)
+]]>
+      </CLRTestBatchPreCommands>
+
+      <BatchDisasmCheckPostCommands></BatchDisasmCheckPostCommands>
+      <BatchDisasmCheckPostCommands Condition="'$(HasBatchDisasmCheck)' == 'true'"><![CDATA[
+IF NOT "%COMPlus_JitDisasm%" == "" (
+  @(DisasmCheckFiles -> '  dotnet %CORE_ROOT%\SuperFileCheck\SuperFileCheck.dll --csharp "%(Identity)" --allow-unused-prefixes --check-prefixes=CHECK,$(TargetArchitecture.ToUpperInvariant()),$(TargetArchitecture.ToUpperInvariant())-$(TargetOS.ToUpperInvariant()) --dump-input-context 25 --input-file "$(BatchDisasmOutputFile)"
+    IF NOT "!ERRORLEVEL!" == "0" (
+      ECHO EXECUTION OF FILECHECK - FAILED !ERRORLEVEL!
+      Exit /b 1
+    )', '%0d%0a')
+)
+]]>
+      </BatchDisasmCheckPostCommands>
+    </PropertyGroup>
+  </Target>
+
   <!--
   ***********************************************************************************************
   SuperPMI collection of CoreCLR tests

src/tests/Common/Directory.Build.targets

@@ -87,6 +87,11 @@
         <IncludeSubFolders>True</IncludeSubFolders>
       </RunTimeArtifactsIncludeFolders>
 
+      <!-- Used by disasm output verification tests -->
+      <RunTimeArtifactsIncludeFolders Include="SuperFileCheck/">
+        <IncludeSubFolders>True</IncludeSubFolders>
+      </RunTimeArtifactsIncludeFolders>
+
       <!-- XUnit runner harness assemblies that we don't want to mix in with the framework in Core_Root -->
       <RunTimeArtifactsIncludeFolders Include="xunit/" />
     </ItemGroup>

src/tests/Directory.Build.targets

@@ -383,6 +383,11 @@
         <IncludeSubFolders>True</IncludeSubFolders>
       </RunTimeArtifactsIncludeFolders>
 
+      <!-- Used by disasm output verification tests -->
+      <RunTimeArtifactsIncludeFolders Include="SuperFileCheck/">
+        <IncludeSubFolders>True</IncludeSubFolders>
+      </RunTimeArtifactsIncludeFolders>
+
       <!-- XUnit runner harness assemblies that we don't want to mix in with the framework in Core_Root -->
       <RunTimeArtifactsIncludeFolders Include="xunit/" />
     </ItemGroup>

src/tests/JIT/Regression/JitBlue/Runtime_33972/Runtime_33972.csproj

@@ -6,8 +6,20 @@
     <DebugType>None</DebugType>
     <Optimize>True</Optimize>
     <AllowUnsafeBlocks>True</AllowUnsafeBlocks>
+    <CLRTestBatchPreCommands><![CDATA[
+$(CLRTestBatchPreCommands)
+set COMPlus_TieredCompilation=0
+set COMPlus_JITMinOpts=0
+]]></CLRTestBatchPreCommands>
+    <BashCLRTestPreCommands><![CDATA[
+$(BashCLRTestPreCommands)
+export COMPlus_TieredCompilation=0
+export COMPlus_JITMinOpts=0
+]]></BashCLRTestPreCommands>
   </PropertyGroup>
   <ItemGroup>
-    <Compile Include="$(MSBuildProjectName).cs" />
+    <Compile Include="$(MSBuildProjectName).cs">
+      <HasDisasmCheck>true</HasDisasmCheck>
+    </Compile>
   </ItemGroup>
 </Project>

src/tests/JIT/Regression/JitBlue/Runtime_34937/Runtime_34937.cs

@@ -9,12 +9,35 @@ class Program
     [MethodImpl(MethodImplOptions.NoInlining)]
     static uint PerformMod_1(uint i)
     {
+        // X64-FULL-LINE:      mov [[REG0:[a-z]+]], [[REG1:[a-z0-9]+]]
+        // X64-FULL-LINE-NEXT: and [[REG0]], 7
+
+        // ARM64-FULL-LINE: and w0, w0, #7
+
         return i % 8;
     }
 
     [MethodImpl(MethodImplOptions.NoInlining)]
     static int PerformMod_2(int i)
     {
+        // X64-FULL-LINE:              mov [[REG0:[a-z]+]], [[REG1:[a-z]+]]
+        // X64-FULL-LINE-NEXT:         sar [[REG0]], 31
+        // X64-FULL-LINE-NEXT:         and [[REG0]], 15
+        // X64-FULL-LINE-NEXT:         add [[REG0]], [[REG1]]
+        // X64-FULL-LINE-NEXT:         and [[REG0]], -16
+        // X64-WINDOWS-FULL-LINE-NEXT: mov [[REG2:[a-z]+]], [[REG1]]
+        // X64-WINDOWS-FULL-LINE-NEXT: sub [[REG2]], [[REG0]]
+        // X64-WINDOWS-FULL-LINE-NEXT: mov [[REG0]], [[REG2]]
+        // X64-LINUX-FULL-LINE-NEXT:   sub [[REG1]], [[REG0]]
+        // X64-LINUX-FULL-LINE-NEXT:   mov [[REG0]], [[REG1]]
+        // X64-OSX-FULL-LINE-NEXT:     sub [[REG1]], [[REG0]]
+        // X64-OSX-FULL-LINE-NEXT:     mov [[REG0]], [[REG1]]
+
+        // ARM64-FULL-LINE:      and w1, w0, #15
+        // ARM64-FULL-LINE-NEXT: negs w0, w0
+        // ARM64-FULL-LINE-NEXT: and w0, w0, #15
+        // ARM64-FULL-LINE-NEXT: csneg w0, w1, w0, mi
+
         return i % 16;
     }
 
@@ -27,6 +50,12 @@ static int PerformMod_3(int i, int j)
     [MethodImpl(MethodImplOptions.NoInlining)]
     static int MSUB(int a, int b, int c)
     {
+        // X64-FULL-LINE:      imul [[REG0:[a-z]+]], [[REG1:[a-z0-9]+]]
+        // X64-FULL-LINE-NEXT: mov [[REG2:[a-z]+]], [[REG3:[a-z]+]]
+        // X64-FULL-LINE-NEXT: sub [[REG2]], [[REG0]]
+
+        // ARM64-FULL-LINE: msub w0, w1, w2, w0
+
         return a - b * c;
     }
 

src/tests/JIT/Regression/JitBlue/Runtime_34937/Runtime_34937.csproj

@@ -6,9 +6,21 @@
     <DebugType />
     <Optimize>True</Optimize>
     <AllowUnsafeBlocks>True</AllowUnsafeBlocks>
+    <CLRTestBatchPreCommands><![CDATA[
+$(CLRTestBatchPreCommands)
+set COMPlus_TieredCompilation=0
+set COMPlus_JITMinOpts=0
+]]></CLRTestBatchPreCommands>
+    <BashCLRTestPreCommands><![CDATA[
+$(BashCLRTestPreCommands)
+export COMPlus_TieredCompilation=0
+export COMPlus_JITMinOpts=0
+]]></BashCLRTestPreCommands>
   </PropertyGroup>
   <ItemGroup>
-    <Compile Include="$(MSBuildProjectName).cs" />
+    <Compile Include="$(MSBuildProjectName).cs">
+      <HasDisasmCheck>true</HasDisasmCheck>
+    </Compile>
   </ItemGroup>
   <ItemGroup>
     <ProjectReference Include="$(TestSourceDir)Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj" />

src/tests/JIT/Regression/JitBlue/Runtime_73681/Runtime_73681.cs

+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System;
+using System.Runtime.CompilerServices;
+
+public class Program
+{
+    public static int Main()
+    {
+        Console.WriteLine(CallFoo(new C()));
+        return 100;
+    }
+
+    [MethodImpl(MethodImplOptions.NoInlining)]
+    private static int CallFoo<T>(T val) where T : IFace
+    {
+        // This is testing that a constrained.callvirt through a T variable doesn't use a helper lookup.
+        // CHECK-NOT: CORINFO_HELP
+        return val.Foo();
+    }
+}
+
+public interface IFace
+{
+    int Foo();
+}
+
+public class C : IFace
+{
+    public int Foo() => 0;
+}

src/tests/JIT/Regression/JitBlue/Runtime_73681/Runtime_73681.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <Optimize>True</Optimize>
+  </PropertyGroup>
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs">
+      <HasDisasmCheck>true</HasDisasmCheck>
+    </Compile>
+  </ItemGroup>
+</Project>