This is necessary in order to present strings that originate from external APIs (e.g. the System.XML) with proper localization when they are embedded in diagnostics.
***NO_CI***
 (changeset 1408070)

After some discussion between Neal, Sri and myself we decided to make AsyncQueue<T> and CompilationEvent internal. The justification for this was:

- At the time there is no need for this type to be public. Ideally the IDE and Compiler would be structured in a way that making it public was needed but for now it is not done that way.
- AsyncQueue<T> feels like a type which should be in BCL, not Roslyn.
- No concrete need for making this public at the moment.

Given that we decided to make it internal for now. One of the main hopes here is we can convince the BCL to take ownership of the AsyncQueue<T>. If we make it public now, even if they add such a type we'd never get to consume it.
 (changeset 1404591)

***NO_CI***
 (changeset 1400352)

AnalyzerDriver is currently tightly coupled with the sequence and kind of CompilationEvents being produced by the compilation. It requires that the first event on the queue is a compilation started event and the final event is the compilation end event. If either of these events are not produced OR are out of expected order then the driver can deadlock.

This change cleans this up by:
1) Ignoring the CompilationStartedEvent in the analyzer driver: This event was only triggering initialization of some lazy fields on the driver: compilation, analysis scope, declaration analyzers and event processing task. This code has been refactored into an Initialize method which is triggered in AnalyzerDriver.Create immediately after a new compilation is created with driver's event queue.

2) Clean up ProcessCompilationCompletedAsync: Move the await of syntax tree analyzer tasks and DiagnosticQueue.Complete invocation out of this method into the primary driver task. This method now just invokes all the compilation end actions and doesn't do any post processing.

I have also cleaned up the analyzer test framework a bit. (changeset 1392961)

Add an instance public Kind() method on CSharpSyntaxNode and VisualBasicSyntaxNode as a replacement for obsoleted CSharpKind(), VBKind(). (changeset 1389507)

Marks CSharpKind and VBKind extension methods as Obsolete and adds equivalent extension methods Kind for both languages.
Marks CSharpKind and VBKind instance methods on language specific syntax nodes Obsolete as well.
 (changeset 1389494)

Marks CSharpKind and VBKind extension methods as Obsolete and adds equivalent extension methods Kind for both languages.
Marks CSharpKind and VBKind instance methods on language specific syntax nodes Obsolete as well. (changeset 1388816)

Due to CLR bug 1094313 the added members can't be referenced from another assembly. For now we report a rude edit if we detect a MemberRef or TypeRef to a symbol added during debugging.
 (changeset 1384081)

Explicit interface implementations in C# weren't getting found by the symbol searcher because the members are private.  The fix is to search for symbols via INamespaceOrTypeSymbol.GetMembers() instead of MergedTypeDeclaration.MemberNames. (changeset 1382653)

IsDefinedOrImplementedInSourceTree – when called for an implicitly declared instance constructor, return the answer for the containing type. This ensures that an error about missing parameter-less base constructor is returned for syntax tree containing declaration of a derived type. (changeset 1378066)

Add a diagnostic (not yet part of the toolset) to ensure CancellationToken arguments are last* and fix violations.

* Unless they are followed by params, out, ref, or optional parameters (without themselves being optional), or are interface implementations or overrides.

* I suppressed the warnings in the TaskExtensions and TaskFactoryExtensions types, since we're trying to mirror the TPL's APIs there.

Includes some tests for C# scenarios, and a fixer for C# that just reorders the signature (arguably, it's better to just let the user invoke Change Signature themself). (changeset 1370978)

     ContainsSymbolsWIthName and GetSymbolsWithName which can be used to either check whether certain name exist in the compilation without using symbol or get those symbols that has the name.

     also added corresponding API in workspace layer so that it can be used in IDE as well for things like SymbolFinder

     we are planning to use it further more in features such as "NavigateTo" (changeset 1370769)

In some parts of our codebase, we use "verbatim crefs" where the compiler just passes the cref name out to the XML file and doesn't do any analysis of it. While there are some legitimate uses of these, there were many abuses in our source code. Worst of all, in some cases we had verbatim crefs on public types that referenced types that had since been renamed, or the syntax was wrong, etc. With this change, there's now an analyzer that issues a warning for all uses of them. In the places that are fair uses of them, the warning is suppressed. Going forward, adding suppressions should be considered fine as long as:

1) it's a reference to something in another layer or language
2) it's some framework type that's clearly never going to change ever

Otherwise, a proper, verifiable syntax should be used. Again, this isn't to say that verbatim crefs are universally bad, but just that we've proven we make mistakes with them and some guards should be put in place. (changeset 1360175)

***NO_CI***
 (changeset 1356181)

Implements CR feedback for EmitOptions change: rename EmitOptions.OutputName to OutputNameOverride and adds TolerateErrors and IncludePrivateMembers.  (changeset 1348868)

Diagnostic.Severity property used to denote different things for compiler and user diagnostics. For compiler diagnostics, this property denotes the error code severity, i.e. the default severity of the diagnostic. For warnings as errors, this property still returned DiagnosticSeverity.Warning and the consumer had to check for the IsWarnAsError property to get the effective severity. For user diagnostics, this property denotes the effective severity based on the compilation options.

This change makes the following changes:
1) Change Diagnostic.Severity to always represent the effective severity of the diagnostic for both user and compiler diagnostics. So for warn as error, this will return error severity.
2) Introduce Diagnostic.DefaultSeverity to represent the default severity of the diagnostic. This is the error code severity for compiler diagnostics and diagnostic descriptor severity for user diagnostics.
3) Change IsWarningAsError property to return true iff Diagnostic.DefaultSeverity = Warning and Diagnostic.Severity = Error. (changeset 1348667)

     Moves options that were previously on CompilationOptions but were not used until emit phase to EmitOptions: fileAlignment, baseAddress, highEntropyVirtualAddressSpace, subsystemVersion, runtimeMetadataVersion. They hold on values written to various PE headers. We can now easily add other similar PE flags to EmitOptions, which is a common customer request, without affecting code that works with compilation options in other layers (workspaces, project system, etc.).

     Removes EmitMeadataOnly method and instead adds a MetadataOnly flag to EmitOptions. Removes MetadataOnlyEmitOptions - they were not used and can now be easily added as bools to EmitOptions.

     Moves pdbFilePath and outputName from parameters of Emit to EmitOptions.

IDE: remove tracking of options that were moved to EmitOptions, since the IDE doesn't care about options that don't affect compilation. (changeset 1348623)

Rollback change that added a reference to the Compilation on CompilationTestData.  I misunderstood the root cause of the issue that caused a test to AV when visualizing IL. (changeset 1348099)

Add a reference in CompilationTestData to hold on to the Compilation that was used to initialize it.  The IL visualizer requires Symbols of the Compilation to still be alive when visualizing tokens.

Most of the time, we hold a reference to the compilation in the test method, so the lifetime of the Symbols would be long enough for all test verification to occur.  However, in some tests, the Compilation is created in a helper and then thrown away as soon as the PE image is emitted.  Verification steps later in the test method could result in AVs if a GC occurs at the wrong time. (changeset 1347747)

This change implements a new DiagnosticAnalyzer for equality which adds the following rules (VB and C#)

1. If a struct overrides Object.Equals it should implement IEquatable<T>. This avoids boxing and extra type checks in many scenarios
2. If a struct or class implements IEquatable<T> it should override Object.Equals. Without this the equality implementation is simply incomplete and will unexpectedly fail when Object.Equals

There were a number of violations in the Roslyn source tree of these rules which I cleaned up as a part of this change. A couple of notes on that work

- In several places Equals and GetHashCode were overridden simply to throw an exception. Rather than implement IEquatable<T> (and pay the cost of the interface JIT) I just suppressed the analysis
- Hash utility class cannot be used from RoslynTaoActions because it comes from 2 different referenced DLLs. For now I just manually implemented hashshing, can use extern aliases to solve if there is a strong desire for this

 (changeset 1345744)

Enable fast array literal initialization for enums if the target framework is 4.5 or above. (changeset 1339077)

Consider:

File A.cs:

  partial class MyPartialClass
  {
      partial void MyPartialMethod(MyUndefinedType m);
  }

File B.cs:

  partial class MyPartialClass
  {
      partial void MyPartialMethod(MyUndefinedType m)
      {
          c = new MyUndefinedType(23, true);
      }
  }

Currently, we show squiggles (and corresponding items in the Error List) under "MyUndefinedType" in the signatures of both the partial method declaration (in A.cs) and implementation (in B.cs). However, we show no squiggles for any errors in the body of the implementation. We expect two: one under "c" as it is an undefined variable, and another under "MyUndefinedType" in the object creation.

The issue is in how we decide which symbols to analyze when collecting method body diagnostics. When analyzing B.cs, we pass in a filter that causes us to ignore everything defined outside of that file. The symbol we get by default for MyPartialMethod, however, is only considered to be defined in A.cs. As such we filter it out instead of checking for diagnostics.

When we come across a symbol representing a source method the filter needs to explicitly check if it is partial. If it is, then it must be the symbol for the declaration, and we need to ask it for the symbol for the implementation and check that against the filter as well. (changeset 1338052)

Replace IDiagnosticAnalyzer with DiagnosticAnalyzer, and move to an action-based diagnostic analysis model. (changeset 1337816)

Stop validating warning ids specified in /nowarn and /warnaserror and allow custom diagnostics to be specified via both (in both VB and C#).

Also, start reporting custom info diagnostics on the command line. The compiler generated info diagnostics for unused imports have now been changed to 'hidden' so that we don't start seeing them all over the place during builds. (changeset 1315557)

Switch the command line compilers to use the newly implemented event queue based AnalyzerDriver for executing diagnostic analyzers. This change primarily (a) Fixes AnalyzerDriver.AnalyzeDeclaringReference to correctly compute the syntax nodes to be analyzed by syntax and code block analyzers (b) Port all unit tests to use the new driver (c) Delete the current AnalyzerDriver. (changeset 1308719)

Portable compilers don't need them anymore, except for a few of them that are used in for validation and changing extension of file names (embedded resources, netmodules, output name override).
 (changeset 1282511)

Splits Microsoft.CodeAnalysis, Microsoft.CodeAnalysis.CSharp and Microsoft.CodeAnalysis.VisualBasic into portable and desktop assemblies.

Existing Core\Source, CSharp\Source and VisualBasic\Source directories are renamed to Core\Portable, CSharp\Portable and VisualBasic\Portable.
New sibling Desktop folders are added and non-portable source is moved there.
"System.Runtime" references has to be removed in order for the portable project system magic to automatically add facade references.
 (changeset 1281686)

Converts non-portable Compilation.Emits that accepts file paths to extension methods and removes SyntaxFactory.ParseFile.

Cleans up Emit methods on CSharpCompilation and VisualBasicCompilation.
Reorders parameters of Emit methods so that cancellation token is the last one to follow a common convention. (changeset 1271389)

Stores Encoding on SourceText so that we can always calculate checksum of the underlying binary data, even if the SourceText wasn’t created from a Stream.

Adds WithRootAndOptions and WithFilePath methods to SyntaxTree and implements them for all subclasses. I decided to merge WithRoot and WithOptions to a single method WithRootAndOptions since setting one without the other doesn't make much sense (the options should correspond to the root node).

Propagates the Encoding throughout workspaces and services.
Refactors recoverable syntax trees so that they can implement With* methods.

Allow encoding on SourceText to be unspecified (null) and report an emit diagnostic when attempting to emit debug information for a syntax tree that is backed by encoding-less source text. If user creates a syntax tree from a string they previously read from a file and then try to emit debuggable compilation with such tree the original encoding of the file might not match the default UTF8 (they might different by presence of BOM, which is quite subtle). To avoid confusion we don't set UTF8 encoding by default. Instead we report the diagnostic and the user needs to explicitly provide an encoding to the syntax tree factory method.

Avoid leaking DecoderFallbackException out of EncodedStringText.Create. Throw InvalidDataException instead to reduce the amount of exceptions that the caller needs to deal with.

Also moves HasSameText extension method from Services to SourceText and optimizes the comparison for the case we the SHA1 checksums are available.
 (changeset 1260412)

To allow filtering of diagnostics when you don't have a compilation, this change extracts the filtering logic into separate types. This makes it possible to, say, create a SyntaxTree and filter the resulting syntax diagnostics without needing a full compilation. (changeset 1256179)

Provide option to opt-in to deterministic (non-zero) timestamp using "/features:deterministic". (changeset 1250882)

In C# lowering of async method stored the state machine type on SourceMethodSymbol so it can later be used when emitting synthesized attributes of the method. This may cause issues when emitting the same compilation twice with different options that affect the state machine type.
     Instead we should store the state machine type to the current PEModuleBuilder. Rather then actually storing it there and adding a dependency on PEModuleBuilder to the attribute generation this change introduces ModuleCompilationState (similar to TypeCompilationState) and threads that one.
     This type is designed to be mutable during compilation (lowering) and immutable during emit. The idea is that all data currently stored in PEModuleBuilder that are produced during compilation and only read in emit could be moved there. Only emit specific data would remain on PEModuleBuilder.

     Also refactors Compilation.Compile to remove some duplication and make it cleaner.
 (changeset 1249664)

We need to be more conscious about producing synthesized symbols during lowering in order to enable EnC of method bodies whose lowering produce such symbols (e.g. lambdas, iterators, dynamic sites, etc.).

     This change refactors synthesized symbols in C# so that all symbols produced during lowering phase implement an interface (ISynthesizedMethodBodyImplementationSymbol) that can be used in the emit phase to retrieve information necessary for updating the corresponding metadata entities should they enter EnC.

     I also attempted to somewhat improve and increase consistency of the naming of synthesized symbols involved in this change.

     No new functionality is added by this change. (changeset 1247789)