This closes #3953.

The ClusterClient should not call HighAvailabilityServices#closeAndCleanupAllData when being shut down.
The reason is that this call will delete all HA data needed for a future recovery. Only the JobManager
should be allowed to decide when to discard HA data or not.

This closes #3904.

This closes #3907.

This closes #3952.

This closes #3950.

This closes #3854.

- wrap calls to MasterTriggerRestoreHook (and its factory) such that the user classloader is applied

This closes #3933.

This closes #3913.

The pom.xml for flink-quickstart-java and flink-quickstart-scala must
specify scala.version and scala.binary.version.

This closes #3910

In order to guard against deletions of ZooKeeper nodes which are still being used
by a different ZooKeeperStateHandleStore, we have to introduce a locking mechanism.
Only after all ZooKeeperStateHandleStores have released their lock, the ZNode is
allowed to be deleted.

THe locking mechanism is implemented via ephemeral child nodes of the respective
ZooKeeper node. Whenever a ZooKeeperStateHandleStore wants to lock a ZNode, thus,
protecting it from being deleted, it creates an ephemeral child node. The node's
name is unique to the ZooKeeperStateHandleStore instance. The delete operations
will then only delete the node if it does not have any children associated.

In order to guard against oprhaned lock nodes, they are created as ephemeral nodes.
This means that they will be deleted by ZooKeeper once the connection of the
ZooKeeper client which created the node timed out.

This commit wraps up some general improvements to the new Kafka sink
custom partitioning API, most notably:
1. remove deprecated constructors from base classes, as they are not
user-facing.
2. modify producer IT test to test custom partitioning for dynamic
topics.
3. improve documentation and Javadocs of the new interfaces.

This closes #3901.

This closes #3928.

Introduce more generic parameters

Note that the official Docker images for Flink are community supported
and not an official release of the Apache Flink PMC.

This closes #3932

This closes #3887

This closes #3930.

This closes #3929.

This closes #3905.

This closes #3936.

This closes #3931

This closes #3926.

This commit adds the config snapshot of the key serializer of keyed
backends to its checkpoints. This allows the oppurtunity to upgrade key
serializers, as well as state migration in the future in the case of
incompatible old and new key serializers.

This closes #3925.

This closes #3912.

This closes #3919.

This closes #3922.