This closes #1174

Hadoop 2.7.0 pulls in an older Netty version, which clashes with our version.
This makes sure to only pull in our version.

This closes #1146

Tests would fail because of version conflicts because the tests include
the original curator dependencies even though we shade them away in the
final build result.

This also fixes dependency management entries for curator dependencies.
We shade it, therefore it cannot be in the dependency management
section in the root pom.

This closes #1113

-----
Closing unresponsive PR:
This closes #862

This closes #941

This also bumps surefire/failsafe version to 2.8.1

This closes #1093

This closes #1076

[FLINK-2291] [runtime] Add ZooKeeper support to elect a leader from a set of JobManager. The leader will then be retrieved from ZooKeeper by the TaskManagers.

Refactors FlinkMiniCluster to support multiple JobManager

Adds proper remote address resolution for actors

Clean up of LeaderElection and LeaderRetrievalService. Removes synchronization to avoid deadlock.

Adds ZooKeeper start option to TestBaseUtils.startCluster

Removes registration session IDs, using the leader session IDs instead. Sets the leader session ID
 directly in the grantLeadership method. Let the LeaderElectionService select the leader session I
D. Return leader session ID to LeaderRetrievalListeners.

Removes direct ActorRef interaction

Introduces LeaderRetrievalService for the Client and the CliFrontend.

Make ApplicationClient to use the LeaderRetrievalService for JobManager resolution

Adds LeaderElection/Retrieval tests

Added test for exception forwarding from the CuratorFramework to a Contender

Adds test job submission with changing leaders

Adds new test cases for job cleanup after leader election change

Adds new LeaderChangeStateCleanup test case

Adds LeaderElectionRetrievalTestingCluster

Introduces ListeningBehaviour for job submissions

Relocation of org.apache.curator in flink-shaded-hadoop jar

Adds Apache ZooKeeper and Apache Curator to LICENSE and NOTICE files

Increases zookeeper connection timeout to 20000 ms for the KafkaITCase to fix failing tests on Travis

Increased timeouts of ZooKeeperLeaderElectionTest for Travis

Makes the WebInfoServer and the WebRuntimeMonitor to use the LeaderRetrievalService to retrieve the current leading JobManager

Adds proper synchronization to ZooKeeperLeaderElectionService. Fixes StateCheckpointedITCase and PartitionedStateCheckpointingITCase

Adds configuration description for new ZooKeeper configuration values

Fixed port selection of JobManager at startup

Improves logging output

Extends masters file to also specify the webui ports

Adds proper network interface resolution by retrieving the current leader address

Makes the ZooKeeperLeaderElectionService write the leader information in ephemeral nodes so that the information is deleted once the leader has terminated. Fixes a bug in the TaskManager due to call by name semantics of scheduler.scheduleOnce.

Adds jobManagerURL to TriggerTaskManagerRegistration message

Enables findConnectingAddress to use the ZooKeeperLeaderRetrievalService. This allows to test the connection to a possibly changing master node.

Changes startup scripts to respect the recovery mode instead of the ZK_QUORUM

Adjust travis log file to only log zookeeper errors

Updates high availability setup guide

Adds TestLogger to leader election tests

This closes #1016.

[FLINK-2584] [java api] Downgrade version of javakaffee kryo serializers, for compatibility with kryo 2.4

This is needed because the Hadoop IF/OF's are using Hadoop's FileSystem stack, which is using
the security credentials passed in the JobConf / Job class in the getSplits() method.

Note that access to secured Hadoop 1.x using Hadoop IF/OF's is not possible with this change.
This limitation is due to missing methods in the old APIs.

- Add some comments & change dependency scope to test

[FLINK-1901] [core] enable sample with fixed size on the whole dataset.

[FLINK-1901] [core] add more comments for RandomSamplerTest.

[FLINK-1901] [core] refactor PoissonSampler output Iterator.

[FLINK-1901] [core] move sample/sampleWithSize operator to DataSetUtils.

Adds notes for commons-math3 to LICENSE and NOTICE file

This closes #949.

Python operators are no longer serialized and shipped across the
cluster. Instead the plan file is executed on each node, followed by
usage of the respective operator object.

- removed dill library
- filepaths are always explicitly passed to python
- fix error reporting

This closes #931.

 - Adds a separate Maven project for easier maintenance. Also allows users to refer to runtime without web libraries.
 - Simple HTTP server based on Netty HTTP (slim dependency, since we use netty anyways)
 - REST URL parsing via Netty Router
 - Abstract stubs for handlers that deal with errors and request/response
 - First set of URL request handlers that produce JSON responses

This closes #677
This closes #623
This closes #297

- FLINK-2288: Setup ZooKeeper for distributed coordination
  * Add FlinkZooKeeperQuorumPeer to wrap ZooKeeper's quorum peers with
    utilities to write required config values (default datadir, myid)
  * Add default conf/zoo.cfg config for ZooKeeper
  * Add startup scripts for ZooKeeper quorum
  * Add conf/masters file for HA masters

- FLINK-2302: Allow multiple instances to run on single host
  * Multiple TaskManager and JobManager instances can run on a single
    host.

Closes #851

[storm-compat] Moved Storm-compatibility to flink-contrib and split flink-contrib into small sub-projects

Closes #573

- remove Maven dependencies
- add docs to show how to register your own serializer with Kryo

This closes #794.

Closes #778

This closes #729.

This closes #783

This closes #621
This closes #442

[FLINK-1923] [runtime] Replaces asynchronous logging with synchronous logging using grizzled-slf4j wrapper for Scala.

This closes #628

This closes #35

This closes #604

Fixes several minor issues
Requesting non-available data throws Exception
Python process shutodwn more reliable
Synchronization done via TCP

This closes #586

This closes #189

With this change, users can submit a Flink job to a YARN cluster without having a local client monitoring the Application Master or job. You can basically fire and forget a Flink job to YARN.
For supporting this, the ApplicationMaster can now monitor the status of a job and shutdown itself once it is in a terminal state.

The change also verifies that various ways of setting the parallelism on YARN are passed through the system correctly (per job, session).

There was a bug in YARN container creation which made the configuration values for the heap offset useless. This change fixes this error.

All mentioned features and bugs are covered by the flink-yarn-tests.

This closes #542

This closes #421