Related to #8817, #8814.

Fell out of our normal cadence for this during the push for RTM. Now that we
had a bit more breathing room getting us back on track here.

Old: Scopes followed the bound tree, so the outermost scope was within the
try-catch block synthesized in async MovedNext methods.

New: If the entire try block would be a hoisted local scope, we remove the
marker node (BoundStateMachineScope) and insert a new one around the
entire method body.

Upshot: hoisted locals are still in scope when stopped on method closing
braces.

     Background:

     As discussed in the dev team all hands all code under the GitHub dotnet foundation is using a single process for contribution, API review, infrastructure and coding style. The idea is to present a unified view to our customer and give them a single story for contributing to any project under the dotnet foundation.

     https://github.com/dotnet/corefx/wiki/Contributing#c-coding-style

     The coding style transition is automated using a Roslyn based rewrite tool:

     https://github.com/dotnet/codeformatter

     This will be applied in stages across our developer tree. Right now the focus is on the Open directory as this is what is being presented on github. Code owners will be contacted before the transition happens.

     Note: this is a soft style requirement. There are no build errors that come from this change.
 (changeset 1408227)

1) Change MethdCompiler.BindMethodBody to associate correct syntax with BoundBlocks it creates when binding constructor with constructor initializer call (two bound blocks are created – outer one defines a closure scope for constructor parameters, the inner one defines a closure scope for variables in the body).
2) Introduce MethodDebugId – a method ordinal and generation ordinal pair
3) Introduce LamdbaDebugInfo and ClosureDebugInfo to represent information (syntax offset) we use to figure out how to map lambdas and closures to the previous generation.
4) Adds a new PDB CDI record (#7) to store lambda and closure debug info.
5) Generalizes CalculateLocalSyntaxOffset to handle positions in field/property initializers and constructor initializers. Use it to calculate syntax offsets of lambdas and closure scopes. (TODO: rename CalculateLocalSyntaxOffset to CalculateSyntaxOffset).
6) Replace lambda and scope ordinal dispenser integers with array builders that collect LambdaDebugInfo and ClosureDebugInfo.
7) Use TryGet- pattern for all VariableSlotAllocator APIs.
8) Implements mapping of lambda method and display class names to previous generation via VariableSlotAllocator.
 (changeset 1407240)

The compiler emits catch-all handler to all async methods. In case of async void methods we also emit the offset of the catch handler to PDB, so that the debugger can pretend that exceptions thrown by the async method are unhandled. Otherwise they are lost. Currently the compiler emits wrong offset, so the debugger is completely confused and displays "no source location" when such exception occurs.

     The fix is to remove a synthesized "nop" instruction that was incorrectly used to track the catch handler IL offset. Instead mark the synthesized BoundCatchHandler node by a flag and create the IL marker in the CodeGen when emitting instructions for such bound node.

     Affects both C# and VB.

     Fixes bug 1114809. (changeset 1405564)

Implements a new approach to generating unique synthesized member names (see also the attached email).

     1) Assigns each member a "member ordinal" - the index in the members array of the containing type. The ordinal is propagated thru lowering rewriters which uses it to make generated names unique.
     2) We no longer calculate "overload ordinal" for names of state machine types, we use the member ordinal of the async/iterator method instead.
     3) Removes GenerateTempNumber from TypeCompiationState. Combination of method ordinal and other indices is used to make generated names unique:
     - Display classes use ordinal of the method containing the lambda and closure scope ordinal (unique within a method). Static display class is shared across all non-generic methods of a containing class and thus doesn't need a unique number in name (it;'s simple named "c<>").
     - Lambda methods include containing method ordinal (unless emitted to a display class whose name already includes it) and lambda ordinal (unique within a method).
     - The same for fields caching lambdas.
     - Dynamic site containers - no longer include method name in the type name, instead method ordinal is used.

     4) Expression compiler - rename generated type to "<>x" to avoid confusion with "<>c" static display class.
     5) Avoid replacing "." with "_" in member names, other than type names. While displaying stack frames the EE extracts method name from synthesized lambdas method names and displays it. Thus we were displaying "I_F" instead of "I.F" for explicitly implemented methods.
      In type names replace "." with "-". Replacing with "_" lead to duplicate type names in emitted assembly (2 iterator methods with names "I_F" and "I.F" - explicitly implemented iface method). (changeset 1390962)

We also add all of the relevant optimizations requested for current and future
formatting performance improvements. We now use full overload resolution
(using a new binder type specialized for synthesizing invocations) to
select an appropriate string.Format method or format factory method, which
enables it to handle params, generics, arbitrary argument conversions, etc, and we
optimize the special case of no fill-ins by processing the {{ and }} escapes,
producing a literal in the generated code. (changeset 1388224)

The await rewriter was attempting to manually replace "dynamic" with "object" in an expression's awaiter type and related methods for await expressions that are typed as "dynamic". Not only is this unnecessary--this sort of rewrite will be done at emit time--it is also incorrect given the pattern-based nature of await binding.

I also simplified the code that finds and validates the various methods and types needed for an await expression while making this fix.
***NO_CI***
 (changeset 1383986)

Moves most of TryCreateVariableSlotAllocator implementation to a base class shared between C# and VB.
 (changeset 1370161)

Use a dedicated dispenser for awaiter slot ids instead of CompilationState.GenerateTempNumber() and avoid possibility of duplicate awaiter fields. (changeset 1368084)

Hoist all long-lived (user-defined and synthesized) variables to state machine fields in debug builds.

In order to provide good debugging experience and enable EnC of async and iterator methods we need to lift locals whose lifetime may span state machine suspension points to fields of the state machine.

For example,

IEnumerable<int> Enumerate() { int x = 1; F(x); yield return 1; }

might be changed to

IEnumerable<int> Enumerate() { int x = 1; F(x); yield return 1; G(x); }

during debugging (or G(x) might just be evaluated in EE while the program is broken on a breakpoint following the yield statement) and thus x has to be hoisted upfront even though it’s lifetime doesn’t cross suspension point in the initial compilation.

In addition to lifting the locals we also need to generate an EnC map for them (syntax offset + debug id + type -> slot index), so that we can reuse the state of existing locals during EnC.

The implementation is similar to unhoisted local variable mapping and exploits an existing notion of “slots” used by the EE to encode local scopes of hoisted variables.
All hoisted variables are assigned slot indices in a syntax order (just to make it deterministic, the order doesn’t matter). The variables are hoisted to fields with names <>5__N (user-defined – this name pattern is already recognized by both Dev12 and Roslyn EEs) and <>s__N (synthesized). N is a slot number. For these locals we emit EnC custom debug information in the same format as we do for unhoisted locals. We attach this CDI to the iterator/async method.
 (changeset 1364072)

Current approach and issues

     When emitting method body of a method updated during EnC the compiler needs to assign local variables the same slots they had before the edit, so that state stored in those locals gets preserved.
     To implement this slot mapping we used to encode information into synthesized variable names that identified the syntax node that produced the variable and ultimately allowed us to calculate the slot mapping.
     For user variables we identified the slot by name. In both cases we relied on an assumption that the compiler assigns slots in syntax order. This assumption is false in some cases. We could make it true but the requirement would be fragile and hard to enforce when new features are implemented in the compiler since the entire lowering pipeline would need to preserve the assumption.

     New approach
     In the new approach we calculate a "syntax offset" for each user-defined and long-lived synthesized variable. Every such variable symbol has to be associated with a syntax node (its declarator). In usual cases this is the textual distance of the declarator from the start of the method body. It gets a bit complicated when the containing method body is not contiguous (constructors). If the variable is in the body of the constructor the definition of syntax offset is the same. If the variable is defined in a constructor  initializer or in a member initializer (this is only possible when declaration expressions or closures in primary constructors are involved) then the distance is a negative sum of the widths of all the initializers that succeed the declarator of the variable in the emitted constructor body plus the relative offset of the declarator from the start of the containing initializer.

     If a single node is a declarator for multiple variables of the same synthesized kind (it can only happen for synthesized variables) we calculate additional number "ordinal" for such variable. We assign the ordinals to the synthesized variables with the same kind and syntax offset in the order as they appear in the lowered bound tree. It is important that a valid EnC edit can't change the ordinal of a synthesized variable. If it could it would need to be assigned a different kind or associated with a different declarator node.

     To support EnC of async method we will assign another number "subordinal" to certain synthesized locals (produced by spilling by-ref variables) to simplify the mapping. Assigning of such number is not yet implemented in this change.

     Together (syntax offset, ordinal, subordinal) form a LocalDebugId of a variable. Since this id needs to be stored in PDB for both user-defined and long-lived synthesized variables we can’t encode it in the name of the variable (the names of user-defined variables have to be their user specified names). Therefore this change introduces a new custom debug info record that is associated with a method. The record encodes the kind and id for each local slot of the method.

     As a consequence we no longer emit names for synthesized long-lived variables (“CS$...”) unless it’s necessary for backward comp with Dev12 EE (only display class local names need to be emitted for this reason). (changeset 1348066)

During lowering Async method, we generate labels for state machine transition and this information is also consumed by Debugger through Pdb. Not realizing that debugger relies on this information, C# generates one additional label "Initial state" at the beginning of rewriting, which adds 1 to state numbering, causing index mismatch between state number and async-info in Pdb (since Pdb does not contain intiail state).

It turns out we don't need to create this "initial state" label since we set the default value to be -1 (representing 'NotStartedStateMachine') in the constructor and dispatch will fail and fall through to the body part where "initial state" label locates. Note that VB async has already been implemented in this manner. This shelvset removes creation/use of initial states in C# and clean up affected test cases. (changeset 1331364)

Preparation for hoisting more locals in Debug builds. I noticed we can use less maps while rewriting a method to a class.

     MethodToClassRewriter:
     - parameterMap could be moved up to LambdaRewriter since it's only needed when rewriting lambdas.
     - variablesCaptured is replaced by a virtual NeedsProxy method which is implemented on LambdaRewriter and StateMachineRewriter using existing maps (so we can save another HashSet).
        We can also distinguish between "captured" variables and variables we create proxy field for. The former are deduced from syntax. A variable may be captured but not lifted into a field (e.g. expression tree lambda parameter). Or a variable can be lifted to field but not captured (in Debug builds we are going to lift user defined variables that are not captured to allow their later capture during EnC).

     LambdaRewriter:
     - variablesCaptured encodes the same information as keys of captured syntax multi-dictionary.
     - declaredInsideExpressionLambda is not needed either. When visiting ET lambdas we used to add their parameters to both variableBlock and declaredInsideExpressionLambda maps. Since ET lambda parameters are never lifted to closure we can avoid adding them to variableBlock map instead of excluding them later via declaredInsideExpressionLambda lookup.

     Adds internal IReadOnlySet similar to IReadOnlyList and IReadOnlyDictionary so that we can specify an intent of not mutating a set. (changeset 1317999)

Replaces CompilationOptions.Optimize and CompilationOptions.DebugInformationKind with a single enum OptimizationLevel that has two values: Release and Debug.
Removes dependency in IL generation on whether PDB stream is passed to Emit or not. IL shall only differ between Release and Debug.

Command line arguments /optimize, /debug and /pdb remain the same, their values map to either Release or Debug. The value of /optimize is mapped to Release or Debug. /debug+, /debug:full and /debug:pdbonly have all the same effect - enable generating PDB file. They don't influence the generated IL.

Debug information/instrumentation (sequence points, dead stores, nops, etc.) doesn't need to be emitted for synthesized methods that don't contain user code (property GenerateDebugInfo on a symbol returns true).

During lowering always create bound sequence nodes. In some scenarios (PDB stream is not passed, we are emitting helper that contains no user code) these nodes are not used. However, in mainstream scenarios we always emit PDBs, so optimizing the bound nodes away is optimizing for uncommon case and just increases test matrix. Lambda, iterator and async lowering have to handle presence of bound sequence point nodes. We get more test coverage if they are always present.

Another reason why to always create bound sequence points is to avoid complexity and ambiguity when checking if a symbol needs debug information. Some symbols might need debug information during some lowering phases but not other. For example, the generated body of a method symbol that represents an async kick-off method doesn't need debug information since it doesn't contain any user code. However, all source code contained in an async method should have debug information emitted. Bound tree for such code is lowered in the context of the async method symbol, so the value of GenerateDebugInfo for the async method symbol would need to differ depending on which phase of compilation are we in.

Finally, IL body deduplication can always be enabled (in Release and Debug), even in EnC scenarios. Removing another compiler knob further simplifies testing.
 (changeset 1316430)

Initial refactoring to support reuse and clearing of state machine field slots associated with all kinds of captured locals (synthesized or user defined). (changeset 1312733)

For every conditional branch that consumes a value of an expression that might contain calls to user code and that jumps across user code with sequence points, we need to store the value into a synthesized named local, emit stloc and ldloc and place a hidden sequence point on the ldloc instruction. Furthermore, the synthesized local variable has to be associated with the statement syntax whose lowering produces the conditional branch so that EnC infrastructure can map the local variable defined for the branch to the previous generation.

In essence this is what’s going on:

[|if (F())|]
{ … true … }
else
{ … false … }

IL:
call F()
stloc $value
<-- hidden sequence point -->
ldloc $value
brfalse label_false
<-- sequence point for “{“ of true-block -->
… true …
br label_end
label_false:
<-- sequence point for “{“of false-block -->
… false …
label_end:

When the call to F() returns after an update has been performed on the caller the code execution continues in the old version of the caller body until a special remapping breakpoint is hit. These breakpoints are put on all sequence points of the method. If we didn’t put the hidden sequence point in front of the conditional branch the execution would continue in the old code until another sequence point is hit. Such a sequence point can be arbitrary (suppose e.g. that the else block of the if statement is deleted by an edit). The debugger only knows how to map IP of active statements in the old IL to an IP in the new IL, not arbitrary sequence points. By inserting the hidden sequence point we force the mapping to be performed at well-defined point that we can map to the new method body.

The presence of the hidden sequence point then implies the need of storing the value of the if-statement condition to a long-lived local and reloading it back right after the sequence point. The store will happen before the IP is remapped and the load will happen in the new frame. The synthesized local must be allocated the same slot in the new frame so that its value is transferred from the old frame to the new frame.

Fixes bug 927151: Edit and Continue breaks If statements while inside the conditional method.

This change also includes an update to the test infrastructure to
1) make it easier to update failing test that checks for an expected IL
If a VerifyIL fails it synthesizes a temp file containing the unit test source and prints out a link that launches a diff tool (using ROSLYN_DIFFTOOL environment variable).
2) to be able to directly test correspondence of sequence points to IL.
VerifyIL has now an optional argument “sequencePoints”. When specified the printed IL will contain marks that designate presence of sequence points at IL offsets. “-“ for regular sequence point, “~” for hidden sequence points.
I have updated some tests that are supposed to validate correspondence of sequence points to IL instructions to use this new format.  (changeset 1299447)

To support EnC well the compiler needs to carefully distinguish between long-lived and short-lived synthesized variables. Long-lived variables may cross sequence points and thus need to be mapped the the corresponding variables in the previous generation when emitting EnC deltas. Short-lived local variables shall never cross sequence points.

This change revisits helpers that create synthesized variables and assigns each synthesized variable a SynthesizedLocalKind. Negative kind values are reserved for short-lived variables, positive values for long-lived variables.

The change removed the option to give a synthesized variable an explicit name. The names of the variables are assigned at code-gen time based on their SynthesizedLocalKind. This is to enforce that the names of long-lived variables follow a naming pattern that allows use to reverse-engineer the SynthesizedLocalKind value from the variable name.

Also renames DebugInformationKind.PDBOnly to DebugInformationKind.PdbOnly to follow .NET naming guidelines.
 (changeset 1293017)

The compiler generates a local variable to store an exception object in synthesized try-catch handler of an async method. It also used to generate a hidden sequence point in the middle of it's lifespan. This sequence point is unnecessary. Once removed the variable name can also be removed. (changeset 1287817)

Splits Microsoft.CodeAnalysis, Microsoft.CodeAnalysis.CSharp and Microsoft.CodeAnalysis.VisualBasic into portable and desktop assemblies.

Existing Core\Source, CSharp\Source and VisualBasic\Source directories are renamed to Core\Portable, CSharp\Portable and VisualBasic\Portable.
New sibling Desktop folders are added and non-portable source is moved there.
"System.Runtime" references has to be removed in order for the portable project system magic to automatically add facade references.
 (changeset 1281686)

Currently the async state machine is a struct. Struct is better for performance (no allocation) but prevents us to modify local variables in the async method in EnC. The CLR doesn't support adding fields to a struct. (changeset 1252494)

Rolling back changeset that switched async state machine to a class. Investigating some issues.  (changeset 1252130)

 Currently the async state machine is a struct. Struct is better for performance (no allocation) but prevents us to modify local variables in the async method in EnC. The CLR doesn't support adding fields to a struct.

     This change switches the state machine to a class when emitting unoptimized debug code (/debug:full /optimize-). It keeps it struct otherwise. (changeset 1251785)

We need to be more conscious about producing synthesized symbols during lowering in order to enable EnC of method bodies whose lowering produce such symbols (e.g. lambdas, iterators, dynamic sites, etc.).

     This change refactors synthesized symbols in C# so that all symbols produced during lowering phase implement an interface (ISynthesizedMethodBodyImplementationSymbol) that can be used in the emit phase to retrieve information necessary for updating the corresponding metadata entities should they enter EnC.

     I also attempted to somewhat improve and increase consistency of the naming of synthesized symbols involved in this change.

     No new functionality is added by this change. (changeset 1247789)