This closes #1394

This closes #1396.

This closes #1391.

This extends StateHandle with a materialize() method. The asynchronous
handle represents state and can materialize it when required.

This closes #1387

This closes #1385

[FLINK-2924] [streaming] Execute compactions in background thread + keep connections alive on empty snapshots

Closes #1305

This closes #1360.

- Add RequiredParameters class to handle required parameters in UDFs which can be checked
  and validated against the parameters extracted in ParameterTool
- A required parameter is represented by an Option which has by default only a name and can
  be extended to include possible values, a type, a default value
- Any validation failure will throw a RequiredParametersException

This closes #1097

This closes #1382

This is not to fulfill what FLINK-3009 is originally asking,
but I'm piggybacking to contribute a docker environment with jekyll
so that users won't have to deal with environment versions.

Author: Jun Aoki <jaoki@apache.org>
Author: jaoki <jaoki@apache.org>

Closes #1363 from jaoki/dockerized-docgen and squashes the following commits:

7bf3661 [Jun Aoki] Added apache label
36869e7 [jaoki] [FLINK-3009] Add dockerized jekyll environment

This closes #1310.

This closes #1389.

This closes #1341

When a candidate for a bulk iteration is instantiated, then the optimizer creates candidates
for the step function. It is then checked that there exists a candidate solution for the step
function whose properties met the properties of the input to the bulk iteration. Sometimes
it is necessary to add a no-op plan node to the end of the step function to generate the
correct properties. These new candidates have to be added to the final set of the accepted
candidates.

This commit adds that these new candidates are properly added to the set of accepted candidates.

Fix test and add new iteration tests

Add predecessor operator and dynamic path information to no op operator in bulk iterations

This closes #1388.

Streaming sources were directly assigned their InputFormat in the StreamingJobGraphGenerator. As a consequence, the input formats were directly serialized/deserialized by Akka when the JobGraph was sent to the JobManager. In cases where the user provided a custom input format or an input format with custom types, this could lead to a ClassDefNotFoundException, because the system class loader instead of the user code class loader is used by Akka for the deserialization.

The problem was fixed by wrapping the InputFormat into a UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's configuration. By instantiating stream sources as InputFormatVertices, the corresponding InputFormat is retrieved from the Configuration in the initializeOnMaster method call.

This closes #1368.

This closes #1345

This closes #1369