Revert "Revert "Disable diagnostic service for projects that we know we don't have all information to provide good results.""

Revert "Disable diagnostic service for projects that we know we don't have all information to provide good results."

this should make roslyn to turn off full project diagnostics when intellisense build is failed for the project.

when we detect intellisense build failure, VS should put actionable data in error list detail pane.
also bing search for the issue should work as well if one prefer blogs.

this change also fix some issue on cross language p2p references. before there was case where we fail to connect cross language p2p reference causing a lot of errors.

this change also include bulk diagnostic update events improvement. this should let us to deal better (especially when we clean those up) when there are a lot of diagnostics.

Changes include:

1. Enhance CompilationWithAnalyzers to allow computing analyzer diagnostics for a specific tree/span within a compilation and/or for a subset of analyzers. Implementation ensures no duplicate analysis by tracking partial analysis state and caching the reported analyzer diagnostics.

2. Overview of the new APIs added to CompilationWithAnalyzers:
    1. GetAnalyzerSyntaxDiagnostics(tree, analyzers, ct)
        1. Analogous to SyntaxTree.GetDiagnostics(ct)
        2. Gets analyzer diagnostics reported by executing syntax tree actions on the given tree.
    2. GetAnalyzerSemanticDiagnostics(semanticModel, spanOpt, analyzers, ct)
        1. Analogous to SemanticModel.GetDiagnostics(spanOpt, ct)
        2. Gets analyzer diagnostics reported by executing rest of the non-compilation actions on the given tree span.
    3. GetAnalyzerCompilationDiagnostics(analyzers, ct)
        1. Gets rest of the analyzer diagnostics which are reported by either of the following means:
            1. Compilation actions (and compilation end actions)
            2. Non-compilation actions reporting diagnostics on different tree: Executing a symbol action on a symbol definition in a tree, can report diagnostic on its partial definition in some other tree.

3. Simplify IDE analyzer driver by switching it to using the new CompilationWithAnalyzers APIs for analyzer diagnostic computation. Both the IDE and compiler drivers now use the compilation event queue model for driving analysis.

added explict DocumentCloseAsync to incrementalAnalyzer so that re-analysis and close action have different behavior.

outcome of this change is that, when a user close a document without saving, diagnostics will be cleared as expected.

previously we made system to prefer live diagnostic over build diagnostic.
also when we merge those two, we used very detail info to find out build and live diagnostics are same or not.

but the issue was diagnostics from build didn't have all information live one has, also diagnostics from build is static
when live one is changing as user type in IDE. which lead to race where build diagnostic might be already out-dated on arrival.

new system is that we prefer diagnostic from build over live. and don't distinguish them by detail info but whether live can produce
same diagnostics build produced. as user change file, we will replace build diagnostics in affected files to live diagnostics.

this should remove chance where build diagnostic got stuck in the system until next build. and as user change files, live diagnostic
over take build diagnostics.

1) Statically created LocalizableString instances by analyzers were holding onto instances of AnalyzerExecutor (which holds onto the compilation on which it executes) for exception reporting, causing us to leak compilations in command line builds. Fixed this by making sure that we unregister these exception handlers during analyzer cleanup in CommonCompiler, we already did so for VisualStudioAnalyzer created in IDE.

2) Switch all the state caches in AnalyzerManager to be keyed with analyzer, and clear all state for analyzer when host disposes it. Performance analysis showed that analyzers that capture the CompilationStartAnalysisContext in its RegisterCompilationStartAction via some lambda were rooting the compilation objects.

Above two changes got rid of all the static and dependent handles rooting compilations during command line builds, and I see a perceived reduction in memory used by VBCSCompiler during building Roslyn.

3) Fix the IDE onAnalyzerException delegate to not capture project instance, but instead use the projectId. Otherwise, VSIX analyzers that live for lifetime of VS instance would leak compilations.

While I was at it, I also got rid of functionality added to MetadataCache that was caching and re-using analyzer instances across AnalyzerFileReference instances, we had already decided to instead keep lifetime of analyzer instances bound by lifetime of owning AnalyzerFileReference.

Fixes #249: Fix IDE diagnostics test framework to use DiagnosticAnalyzerService to compute diagnostics rather than DiagnosticAnalyzerDriver.

This change serves multiple purposes:
1) Makes our test framework consistent with the IDE scenario in fetching diagnostics. We have missed out on multiple regressions where the core IDE DiagnosticAnalyzerDriver does the right thing and hence the tests pass, but the DiagnosticService in IDE does additional filtering and certain diagnostics are not reported in VS. This change should avoid such bugs.
2) Enable testing for diagnostics V2 engine work that JohnHamby is doing to merge IDE and compiler analyzer drivers. For this engine, there is no IDE driver and hence existing IDE diagnostics tests can't be run. This change should enable running IDE diagnostics on V2 engine.
3) Move a whole bunch of of test only code in DiagnosticAnalyzerService_Test.cs into the test layer.

all tests passing. now StateManager owns all state related operation.

Add partial doc comments to BaseDiagnosticIncrementalAnalyzer sufficient
for others to validate my understanding.

Add partial doc comments to BaseDiagnosticIncrementalAnalyzer sufficient
for others to validate my understanding.

put middle man in between diagnostic incremental analyzer so that two version can be changed without re-starting VS.

moved v1 engine to its own namespace and updated sources accordingly.

also added BaseDiagnosticIncrementalAnalyzer which contains all methods that engine needs to implement to make all other services which consume IDiagnosticAnalyzerService to just work.

probably later we will change it but for now, this should let us test v2 engine isolated without too much changes in other part of code base