Before, the external jar loading tests where using the flink-ml jar
without the scala shell package actually declaring this as a dependency.
Now we built our own jar and have no unlisted dependencies anymore.

This closes #4249.

[FLINK-6773] [checkpoint] Introduce compression (snappy) for keyed state in full checkpoints and savepoints

Default values for compiler version are 1.6 and were causing Intellij to
constantly switch language level to 1.6, which in turn was causing
compilation errors. It worked fine for compiling from console using
maven, because those values are separetly set in maven-compiler-plugin
configuration.

This closes #4213.

This closes #4245.

This closes #4247.

[FLINK-5893] [RM] Fix the bug of race condition for removing previous JobManagerRegistration in ResourceManager

This closes #3399.

This closes #4226.

This closes #4227.

This closes #4209.

This closes #4197.

This closes #4156.

This closes #3416.

This closes #4168.

This closes #3833.

This closes #4185.

This closes #4016.

This closes #4123.

This closes #4022.

This closes #4178.

This closes #3951.

This closes #4109.

This closes #4035.

This closes #4211.

This closes #6575.

This closes #3776.

Previously, YARN log aggregation fails because it depoends on the HDFS
delegation token, which we do not upload if Kerberos keytabs are used.
We did not include HDFS delegation tokens when keytabs are used because
the UGI would prioritize the delegation token (which expires) if both
are present.

To address this, changes include:

1. Change Flink YARN client to always upload delegation tokens when
security is enabled. This would then allow log aggregation.

2. Filter out HDFS delegation token from the tokens fetched from HDFS
when populating the UGI. This allows the UGI to always use Kerberos
tickets instead of the HDFS delegation token.

This closes #3855.

This closes #4216.

The root cause for the instability in ES 1.x is that we're testing both
embedded client mode and transport client mode against the same
Elasticsearch environment. This commit removes the embedded mode
testing, as that method is generally discouraged anyways.

This closes #4191.

The testStartFromKafkaCommitOffsets ITCases are covering too much within
one single test. The case verifies that whatever offset was committed to
Kafka, Flink rreads it correctly and can uses that as the correct
starting point for exactly-once.

This over-engineered test was instable that we needed to first read some
records and wait until some records is committed. This wait is hard to
define.

It is in fact sufficient to have 2 separate tests to cover the tested
behaviour:
- test that committed Kafka offsets are correct (there is already a
  ITCase for this, i.e. `runCommitOffsetsToKafka`)
- test that committed offsets are correctly picked up and used correctly
  (there is actually also a test for this, i.e.
`runStartFromGroupOffsets`)

Hence, this test can be removed without harming test coverage.

This closes #4190.

This closes #3746.

This closes #3889.

This closes #4212.

- Move Java example tests into a single ITCase
- Add ITCase for Scala examples

This closes #2761.

This closes #4195.

The current organization of the Gelly examples retains full flexibility
by handling the Graph input to the algorithm Driver and having the
Driver overload interfaces for the various output types. The outputs
must be made independent in order to support Transforms which are
applied between the Driver and Output (and also between the Input and
Driver).

This closes #4179