This closes #1360.

- Add RequiredParameters class to handle required parameters in UDFs which can be checked
  and validated against the parameters extracted in ParameterTool
- A required parameter is represented by an Option which has by default only a name and can
  be extended to include possible values, a type, a default value
- Any validation failure will throw a RequiredParametersException

This closes #1097

This closes #1382

This is not to fulfill what FLINK-3009 is originally asking,
but I'm piggybacking to contribute a docker environment with jekyll
so that users won't have to deal with environment versions.

Author: Jun Aoki <jaoki@apache.org>
Author: jaoki <jaoki@apache.org>

Closes #1363 from jaoki/dockerized-docgen and squashes the following commits:

7bf3661 [Jun Aoki] Added apache label
36869e7 [jaoki] [FLINK-3009] Add dockerized jekyll environment

This closes #1310.

This closes #1389.

This closes #1341

When a candidate for a bulk iteration is instantiated, then the optimizer creates candidates
for the step function. It is then checked that there exists a candidate solution for the step
function whose properties met the properties of the input to the bulk iteration. Sometimes
it is necessary to add a no-op plan node to the end of the step function to generate the
correct properties. These new candidates have to be added to the final set of the accepted
candidates.

This commit adds that these new candidates are properly added to the set of accepted candidates.

Fix test and add new iteration tests

Add predecessor operator and dynamic path information to no op operator in bulk iterations

This closes #1388.

Streaming sources were directly assigned their InputFormat in the StreamingJobGraphGenerator. As a consequence, the input formats were directly serialized/deserialized by Akka when the JobGraph was sent to the JobManager. In cases where the user provided a custom input format or an input format with custom types, this could lead to a ClassDefNotFoundException, because the system class loader instead of the user code class loader is used by Akka for the deserialization.

The problem was fixed by wrapping the InputFormat into a UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's configuration. By instantiating stream sources as InputFormatVertices, the corresponding InputFormat is retrieved from the Configuration in the initializeOnMaster method call.

This closes #1368.

This closes #1345

This closes #1369

Removes the possibility to go from FAILED state back to RESTARTING. This was only used in a test
case. It was a breaking the terminal state semantics of the FAILED state.

This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.

With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.

This closes #1353

This also adds to the deprecated classes pointers forward to the designated classes.

This closes #1380

[FLINK-3005] [core] Bump commons-collections version to fix object deserialization remote command execution vulnerability

This closes #1381

This closes #1371

FLINK-3041: Twitter Streaming Description section of Streaming Programming guide refers to an incorrect example 'TwitterLocal'

This closes #1379.

This closes #1376

This fixes a problem that would occur if a Trigger registers a new
processing-time trigger in the onProcessingTime method. The problem is
that onProcessingTime() is called while traversing the set of active
triggers. If onProcessingTime() tries to register a new processing-time
trigger this leads to a concurrent modification exception.

- replace deprecated tool which prevents uploading of snapshot binaries
- remove Java 6 jar check

This closes #1266

This closes #1346

In addition to the REST-compliant "DELETE /jobs/<jobid>", allows
cancellation also via a special GET request of the form
"GET /jobs/<jobid>/yarn-cancel".

That enables us to cancel jobs from the web frontend on YARN while
keeping a REST-compliant DELETE alternative.

This closes #1344.

This closes #1373

This closes #2233.