This change reduces the VM used and memory allocated in the emit
tests by moving the WinRT tests to their own DLL and removing
the extra mscorlib reference.

The cause of most of the allocations in the Emit unit tests was
serializing the WinRT assemblies across app domains. There were
three reasons the DLLs needed to be serialized:

1) PeVerify currently requires a separate app domain to be used if
   any of the references of the assembly to be verified have already
   been loaded into the current app domain and have a different MVID.

2) The WinRT tests all use a separate version of mscorlib, so they
   will all reference an assembly with a different MVID than the one
   loaded.

3) Most of the WinRT tests were accidentally pulling two references
   to mscorlib, also always leading to a conflict.

All of these issues should be resolved. The CSharp.Emit unit tests
now allocated 400 MB less and have > 100 MB smaller peak VM.

Fixes #386.

In the emit tests we currently check every module to see if we're
emitting an assembly with a name we've already seen. PeVerify
requires that we create a new AppDomain for every conflict we find.
However, mscorlib will always produce a conflict, so this ignores
mscorlib when checking for a conflict and doesn't create a new
AppDomain if it is found during emit.

   The CreateInstanceAndUnwrap call below is failing somewhat randomly in our test runs with an error claiming it can't load Roslyn.Test.Utilities.  So far we've been unable to actually catch this failure under a debugger because as previously stated the failure is random.  The only interesting data we have is that it's much more likely to fail in xUnit2 parallel mode than xUnit1.

     Looking at the code there seems to be an obvious explanation.  The Environment.CurrentDirectory value becomes the resolver path for the newly created AppDomain instance.  If this value doesn't point to the test binaries directory then it would explain the failure.  This is a process wide value that can easily be changed by other code.  Even if the code is careful to reset the value afterwards the intermediate value would be visible across AppDomains and cause this problem.

     A more stable value is AppDomain.CurrentDomain.BaseDirectory.  This value is AppDomain specific, immutable, and was already sufficient to load this particular assembly.
 (changeset 1409768)

This particular exception in Dispose is masking a real failure in our suite runs.   This particular scenario can only happen, based on my code reading, if an exception happens trying to create the RuntimeAssemblyManager instance.  In that case domain != null && assemblyManager == null.   That exception propagates out the calling code which has HostedRuntimeEnvironment in a using block hence it goes to Dispose.  The Dispose method thinks this combination is invalid and throws a new exception thus hiding the real and original problem.

     Removing the check in Dispose so the real exception will make it into our xUnit logs and we can track down the flaky behavior. (changeset 1407359)

     Moves options that were previously on CompilationOptions but were not used until emit phase to EmitOptions: fileAlignment, baseAddress, highEntropyVirtualAddressSpace, subsystemVersion, runtimeMetadataVersion. They hold on values written to various PE headers. We can now easily add other similar PE flags to EmitOptions, which is a common customer request, without affecting code that works with compilation options in other layers (workspaces, project system, etc.).

     Removes EmitMeadataOnly method and instead adds a MetadataOnly flag to EmitOptions. Removes MetadataOnlyEmitOptions - they were not used and can now be easily added as bools to EmitOptions.

     Moves pdbFilePath and outputName from parameters of Emit to EmitOptions.

IDE: remove tracking of options that were moved to EmitOptions, since the IDE doesn't care about options that don't affect compilation. (changeset 1348623)

Rollback change that added a reference to the Compilation on CompilationTestData.  I misunderstood the root cause of the issue that caused a test to AV when visualizing IL. (changeset 1348099)

Add a reference in CompilationTestData to hold on to the Compilation that was used to initialize it.  The IL visualizer requires Symbols of the Compilation to still be alive when visualizing tokens.

Most of the time, we hold a reference to the compilation in the test method, so the lifetime of the Symbols would be long enough for all test verification to occur.  However, in some tests, the Compilation is created in a helper and then thrown away as soon as the PE image is emitted.  Verification steps later in the test method could result in AVs if a GC occurs at the wrong time. (changeset 1347747)

This shelveset implements following changes to metadata reference compiler API in order to remove duplication in the public surface and prevent users from unexpected metadata lifetime issues:

     1) MetadataImageReference and MetadataFileReference overlap
     Currently MetadataImageReference can be constructed from a Stream. MetadataImageReference supports metadata prefetch (reading the blob into memory and close the underlying stream) as well deferred reading (no content is read until the reference is consumed during compilation). MetadataFileReference only supports deferred reading.

     Lifetime of MetadataFileReference is non-deterministic, the underlying file is locked until no references exist and we GC the metadata.  On the other hand, it is possible to construct MetadataImageReference in such a way that allows controlling the underlying resources deterministically.

     Remove MetadataFileReference, use MetadataImageReference instead.

     2) Lifetime management
     AssemblyMetadata and ModuleMetadata objects hold on resources and implement IDisposable. When using MetadataFileReference constructors and some MetadataImageReference constructors to create references the underlying metadata objects are created implicitly and the user doesn’t have a way to explicitly dispose them.

     Make MetadataImageReference constructors internal and instead add factory method GetReference on AssemblyMetadata/ModuleMetadata. The usage pattern is:

     using (var metadata = AssemblyMetadata.CreateFromXxx(…))
     {
         var compilation = CSharpCompilation.Create(syntaxTrees, new[] { metadata.GetReference() });
         …
     }

     In addition the shelveset makes MetadataImageReference internal and adds the following convenience APIs, that are not the most efficient but are very convenient, easy to discover and safe to use for customers that don’t wanna explicitly manage the lifetime of metadata objects. (changeset 1345987)

 AssemblyMetadata and ModuleMetadata factories should not read the content of the PE file/metadata blob.

     Currently the factories read the headers, Assembly and Modules tables, which might result in BadImageFormatException being thrown. This exception is not turned into a compilation diagnostic because it happens before the compilation is created. A user of the Roslyn API thus needs to handle these errors in two places (as an exception and as a diagnostics). The content reading should be deferred until the AssemblyMetadata/ModuleMetadata is queried for content.

     This change defers metadata/PE headers reading and decoding until the AssemblyMetadata and ModuleMetadata properties/methods are called.
     For metadata created from files it aims to match the usage pattern of other APIs working with FileStream. The metadata factory opens the file, which might throw IO exception, but doesn't start reading the stream until the compiler asks for it, therefore it doesn't throw BadImageFormatException. The functionality is equivalent to the user opening a FileStream and creating metadata from that stream (except for a slight complication with multi-module assemblies, which are rare). Thus the API for metadata creation are consistent among in-memory byte array, stream, and file path.  (changeset 1342462)

Replaces CompilationOptions.Optimize and CompilationOptions.DebugInformationKind with a single enum OptimizationLevel that has two values: Release and Debug.
Removes dependency in IL generation on whether PDB stream is passed to Emit or not. IL shall only differ between Release and Debug.

Command line arguments /optimize, /debug and /pdb remain the same, their values map to either Release or Debug. The value of /optimize is mapped to Release or Debug. /debug+, /debug:full and /debug:pdbonly have all the same effect - enable generating PDB file. They don't influence the generated IL.

Debug information/instrumentation (sequence points, dead stores, nops, etc.) doesn't need to be emitted for synthesized methods that don't contain user code (property GenerateDebugInfo on a symbol returns true).

During lowering always create bound sequence nodes. In some scenarios (PDB stream is not passed, we are emitting helper that contains no user code) these nodes are not used. However, in mainstream scenarios we always emit PDBs, so optimizing the bound nodes away is optimizing for uncommon case and just increases test matrix. Lambda, iterator and async lowering have to handle presence of bound sequence point nodes. We get more test coverage if they are always present.

Another reason why to always create bound sequence points is to avoid complexity and ambiguity when checking if a symbol needs debug information. Some symbols might need debug information during some lowering phases but not other. For example, the generated body of a method symbol that represents an async kick-off method doesn't need debug information since it doesn't contain any user code. However, all source code contained in an async method should have debug information emitted. Bound tree for such code is lowered in the context of the async method symbol, so the value of GenerateDebugInfo for the async method symbol would need to differ depending on which phase of compilation are we in.

Finally, IL body deduplication can always be enabled (in Release and Debug), even in EnC scenarios. Removing another compiler knob further simplifies testing.
 (changeset 1316430)

Updates tests to use either Release or Debug. Skips tests that are broken when PDB emit is enabled. (changeset 1314373)

For every conditional branch that consumes a value of an expression that might contain calls to user code and that jumps across user code with sequence points, we need to store the value into a synthesized named local, emit stloc and ldloc and place a hidden sequence point on the ldloc instruction. Furthermore, the synthesized local variable has to be associated with the statement syntax whose lowering produces the conditional branch so that EnC infrastructure can map the local variable defined for the branch to the previous generation.

In essence this is what’s going on:

[|if (F())|]
{ … true … }
else
{ … false … }

IL:
call F()
stloc $value
<-- hidden sequence point -->
ldloc $value
brfalse label_false
<-- sequence point for “{“ of true-block -->
… true …
br label_end
label_false:
<-- sequence point for “{“of false-block -->
… false …
label_end:

When the call to F() returns after an update has been performed on the caller the code execution continues in the old version of the caller body until a special remapping breakpoint is hit. These breakpoints are put on all sequence points of the method. If we didn’t put the hidden sequence point in front of the conditional branch the execution would continue in the old code until another sequence point is hit. Such a sequence point can be arbitrary (suppose e.g. that the else block of the if statement is deleted by an edit). The debugger only knows how to map IP of active statements in the old IL to an IP in the new IL, not arbitrary sequence points. By inserting the hidden sequence point we force the mapping to be performed at well-defined point that we can map to the new method body.

The presence of the hidden sequence point then implies the need of storing the value of the if-statement condition to a long-lived local and reloading it back right after the sequence point. The store will happen before the IP is remapped and the load will happen in the new frame. The synthesized local must be allocated the same slot in the new frame so that its value is transferred from the old frame to the new frame.

Fixes bug 927151: Edit and Continue breaks If statements while inside the conditional method.

This change also includes an update to the test infrastructure to
1) make it easier to update failing test that checks for an expected IL
If a VerifyIL fails it synthesizes a temp file containing the unit test source and prints out a link that launches a diff tool (using ROSLYN_DIFFTOOL environment variable).
2) to be able to directly test correspondence of sequence points to IL.
VerifyIL has now an optional argument “sequencePoints”. When specified the printed IL will contain marks that designate presence of sequence points at IL offsets. “-“ for regular sequence point, “~” for hidden sequence points.
I have updated some tests that are supposed to validate correspondence of sequence points to IL instructions to use this new format.  (changeset 1299447)

Converts non-portable Compilation.Emits that accepts file paths to extension methods and removes SyntaxFactory.ParseFile.

Cleans up Emit methods on CSharpCompilation and VisualBasicCompilation.
Reorders parameters of Emit methods so that cancellation token is the last one to follow a common convention. (changeset 1271389)

We make the compilers (both C# and VB) deterministic (identical inputs cause identical outputs). In theory this may make distributed and incremental builds much easier to implement. It will also allow one to determine easily if any API has changed or not by simply generating a reference assembly and checking if it is identical to the previous one.

Two things needed to change:
(1) The timestamp in the header is replaced with 0 (which is specifically allowed by the spec)
(2) The module version ID Guid (Mvid) is computed by hashing the contents of the generated assembly (with zero
      where the Mvid will go for the purposes of computing the hash)

The name of the "private implementation details" class no longer includes the Mvid.

In order to simplify things, I removed the parameter from a few emit APIs where the caller provides the Guid for the Mvid. It turns out that nobody nowhere used it. So when emitting in the normal way, we don't know the Mvid ahead of time. For edit-and-continue, however, we always reuse the Mvid for the original assembly as before.

I implemented this assuming that everyone wants this new behavior. Some may worry that perhaps some unknown tool depends on the presence of the timestamp. I'd prefer to push it this way and see if anything breaks, because it will be simpler for it to just always work this way rather than having an option to turn it on and off. Also, I look forward to seeing if there is any performance impact due to running SHA1 over the whole in-memory stream. I suspect it will be unmeasurably small.

By the way, strong name signing happens after this is done, so there is no need to worry that a signature will introduce nondeterminism in the mvid. The strong name signature includes mvid in the data it signs. See Compilation.SerializeToPeStream to see the sequence. (changeset 1235486)