This is needed because the Hadoop IF/OF's are using Hadoop's FileSystem stack, which is using
the security credentials passed in the JobConf / Job class in the getSplits() method.

Note that access to secured Hadoop 1.x using Hadoop IF/OF's is not possible with this change.
This limitation is due to missing methods in the old APIs.

- Add some comments & change dependency scope to test

Closes #851

This closes #837.

This closes #689

Adds pipeline comments

Adds pipeline IT case

[FLINK-2050] [ml] Introduces new pipelining mechanism using implicit classes to wrap the algorithm logic

This closes #704.

This closes #421

[FLINK-1629][FLINK-1630][FLINK-1547] Add option to start Flink on YARN in a detached mode. YARN container reallocation.

This commit is changing:
[FLINK-1629]: users can now "fire and forget" jobs to YARN or YARN sessions to there. (Detached mode)
[FLINK-1630]: YARN is now reallocating failed YARN containers during the lifetime of a YARN session.
[FLINK-1547]: Users can now specify if they want the ApplicationMaster (= the JobManager = the entire YARN session) to restart on failure, and how often. After the first restart, the session will behave like a detached session. There is now backup of state between the old and the new AM.

The whole resource negotiation process between the RM and the AM has been reworked.
Flink is now much more flexible when requesting new containers and also giving back uneeded containers.

A new test case is testing the container restart. It is also verifying that the web frontend is proplery started,
that the logfile access is possible and
that the configuration values the user specifies when starting the YARN session are visible in the web frontend.

This closes #468

This closes #454

- users can now also deploy Flink on YARN for executing a single job.
- The flink-yarn project has been moved out of the flink-addons module
- the MiniYARNCluster is used for testing Flink on YARN
- There is now a (undocumented) Java interface Flink's YARN client, allowing users to manually control the Yarn session.
- ALL ports used by Flink when running on YARN are automatically determined. In the past users reported problems with blocked ports (YARN is telling the client the RPC address of the application master)
- The checks before deployment have been improved to give better error messages if the user is requesting too many resources for a YARN session

Default hadoop version is now hadoop2
Generate specific pom avoids version collisions with dependencies
Updated poms for new release scripts

Conflicts:
	flink-addons/flink-avro/pom.xml
	flink-addons/flink-hadoop-compatibility/pom.xml
	flink-addons/flink-hbase/pom.xml
	flink-addons/flink-jdbc/pom.xml
	flink-addons/flink-spargel/pom.xml
	flink-addons/flink-tachyon/pom.xml
	flink-quickstart/flink-quickstart-java/pom.xml
	flink-quickstart/flink-quickstart-scala/pom.xml

  - Move sling JSON dependency to streaming connectors (prev streaming core)
  - Exclude YARN API from transitive mapreduce-core dependencies we use the dependency
    only for .mapred and .mapreduce interfaces
  - Remove some unnecessary dependencies (junit in java8)
  - Manage Kryo dependency
  - Cleanup minor POM warnings.

to HadoopFileSystem wrapper, which supports all subclasses of org.apache.hadoop.fs.FileSystem.
This allows us to let users use all file systems with support for HDFS.
The change has been tested with Tachyon, Google Cloud Storage Hadoop Adapter and HDFS.

The change also cleans up the Hadoop dependency exclusions.