I am just a simple developer trying to send these cancellation tokens downstream where they belong

* Fix nullable annotations for ValueSource

* Refactor WeakValueSource

* Replace OptionalValueSource<T> with ValueSource<Optional<T>>

This reverts commit 6ac8006e.

* Document load failures

* Delete WorkspaceFailureOutputPane

* Remove dead code

* MSBuildWorkspace cleanup

This reverts commit 720f5c5e, reversing
changes made to 6a6043c3.

changes include
make SourceText::GetChecksum and AnalyzerTelemetry contructor public.

added MustRunInProc in IBuiltInAnalyzer and some clean up around serializing Solution/Project/DocumentId and how DocumentState is exposed.

simplified temporary storage service's temporary storage management and added ability to attach to existing temporary storage

solution checksum and serialization service.

added remote host client
- this gives an ability for host (vs) to talk to remote host (service hub)

rename and moving files between feature/workspace layers
- only real change is having ICompilerDiagnosticAnalyzer interface which can either have inproc implementation or out of proc implementation.
- inproc is needed since diagnostics are in feature layer and one who uses feature layer out side of VS host need an implementation.

added RemoteWorkspace
- RemoteWorkspace has host agnostic implementation of roslyn features/services/workspace that will run in remote host

added service hub component and setup project for service hub
- service hub component is basically thin layer that deals with converting data to pass in to RemoteWorkspace

made devdiv insertion tool to ignore servicehub related files

support byte and char array natively in ObjectReader/Writer

Replace AsyncSemaphore with SemaphoreSlim. AsyncSemaphore is no longer
necessary since we moved to 4.5.2

     Background:

     As discussed in the dev team all hands all code under the GitHub dotnet foundation is using a single process for contribution, API review, infrastructure and coding style. The idea is to present a unified view to our customer and give them a single story for contributing to any project under the dotnet foundation.

     https://github.com/dotnet/corefx/wiki/Contributing#c-coding-style

     The coding style transition is automated using a Roslyn based rewrite tool:

     https://github.com/dotnet/codeformatter

     This will be applied in stages across our developer tree. Right now the focus is on the Open directory as this is what is being presented on github. Code owners will be contacted before the transition happens.

     Note: this is a soft style requirement. There are no build errors that come from this change.
 (changeset 1408227)

The current cost-based cache has no domain knowledge of whether items will be needed again or are safe to collect.  It is therefore usually over-sized. This change implements an explicit cache that must be enabled by consumers. The cache will hold trees and compilations as needed and release them when clients are done with them.

Currently, we enable explicit caching of the active document. Rename tracking and the solution crawler also request caching for each project they process. (changeset 1372494)

Move a few types back to the portable layer to allow a CustomWorkspace to be produced without needing the desktop layer.

As part of this, I split apart ITemporaryStorageService's ITemporaryStorage into two separate types, so it's clear whether you're storing text or storing binary streams. The merged implementation allowed some leakage of implementation details. (changeset 1353513)

This change splits the Workspaces layer into two parts, mirroring the Portable/Desktop split that the compilers did. The core parts of the Workspaces layer (managing documents and projects, formatting, some refactoring, code fixes) is kept in the portable subset, with a few non-portable pieces remaining, notably MSBuild support.

This change has a major impact on how MEF now works in Roslyn. Traditional MEF (“MEFv1”) is not portable, and so we must move the Workspaces layer over to using the Microsoft.Composition NuGet package (“MEFv2”) which is. The APIs are distinct in that each has its own namespace, but the concepts are more or less identical. It requires some care though: the workspaces layer is simple in that it only references MEFv2, but higher layers have to reference both versions to use metadata attributes. Exports using metadata attributes from the editor (say, ContentTypeAttribute) must use MEFv1 attributes to export, whereas exports for the workspaces layer must use MEFv2 attributes. The rule is subtle and yet simple, and so a diagnostic is provided which catches any offenses to prevent confusion.

This also has some impact in how we create MEF hosts: if you wish to host just the base workspaces layer, we can use MEFv2 to compose everything. The HostServices implementation (MefV1HostServices) that consumes a MEFv1 ExportProvider. If you’re in Visual Studio, you can use this implementation to get the full set of host services provided in Visual Studio.

Otherwise, most of the changes in here are minor: we react to some APIs that have been moved/renamed in the portable subset we are targeting, and also align our various exception helper utilities with the compiler’s precedent. (changeset 1349276)

our Logger used to get both FeatureId and FunctionId. FeatureId used to be used to control a group of functionId.

now, I am flattening them so that we can group functionId by its name rather than using both FeatureId and FunctionId. (changeset 1311395)

our snapshot model used to go back to the end (text) to re-construct everything from scratch in case needed information is kicked out from memory (or cache)

this change makes that layer to be one up to tree rather than text. one reason is deserializing is 2x faster than parsing and second reason is most of work we do relies on tree not text. so once tree is parsed the text we bring back to memory no longer needed until tree gets kicked out from memory again. this should cut the middle man and let us go straight to tree.

text will be still snapshoted to temporary storage, but once initial parsing is done right after solution load, it will be rarely brought into memory again unless a feature explicitly needs text to do some operation.

here is some perf information between parsing/serialization/deserialization.

...

[11412] *** RUN : 1
[11412] vbsource : 2556 files, size : 56979199 chars (it is unicode so 2 bytes per 1 char)
[11412] cssource : 6465 files, size : 54516545 chars

[11412] *** Iteration : 0
[11412] vb parsed : 2556, time took : 34329ms, size : 56979199 chars (same here)
[11412] cs parsed : 6465, time took : 24430ms, size : 54516545 chars

[11412] vb serialzied : 2556, time took : 20959ms, size : 69724370 bytes (length of stream)
[11412] cs serialzied : 6465, time took : 20917ms, size : 59704755 bytes

[11412] vb deserialzied : 2556, time took : 15616ms, size : 56979199 chars
[11412] cs deserialzied : 6465, time took : 14912ms, size : 54516545 chars

...

as a follow up change, I will re-adjust some of our default cache values.
 (changeset 1219990)