Consolidation is done via plugin management.

Closes #1753

Removes curator dependency exclusions from flink-runtime. This resolves
NoClassDefFoundError exceptions when running `mvn test`.

This partialy reverts e31a4d8a.

This closes #1596

This closes #1584

- add suffixes
- adapt change-scala-version script

This closes #1529.

  - Removes Apache Commons HTTP Client
  - Removes Apache Commons IO
  - Removes Jettison JSONparser and consolidates tests to use Jackson instead (which is included anyways)

This closes #1331

This closes #1301.

0.10-SNAPSHOT continues on branch release-0.10

Internal actor states must only be modified within the actor thread.
This avoids all the well-known issues coming with concurrency.

Fix RemoveCachedJob by introducing RemoveJob

Fix JobManagerITCase

Add removeJob which maintains the job in the SubmittedJobGraphStore

Make revokeLeadership not remove the jobs from the state backend

Fix shading problem with curator by hiding CuratorFramework in ChaosMonkeyITCase

[FLINK-2842] [documentation] Remove Flink S3FileSystem, extend documentation to use Hadoop S3FileSystem.

This closes #1245

- make new web one the default
- adapt tests
- make web directory a resource to be included in the fat jar
- serve static files of web interface dynamic through the class loader
- run on YARN
- remove Jetty dependencies from poms

This squashes the following commits:
- Refactor JobManager's start actors method to be reusable
- Yarn refactoring to introduce yarn testing functionality
- Add support for testing yarn cluster. Extracted JobManager's and TaskManager's testing messages into stackable traits.
- Implement YarnHighAvailabilityITCase using Akka messages for synchronization.
- Logging statements
- Fix registration at JobManager when the leader address is null
- Fix curator dependency conflict
- Shades Flink's curator dependency in flink-runtime so that it cannot be overriden by external dependencies in the class path. This solves the problem with Hadoop 2.6.0 which adds Curator 2.6.0 to the class path. The curator version of this Hadoop version is not compatible to Flink's Curator version 2.8.0. Furthermore, Flink's Guava version is forced to be included in flink-shaded-curator jar to avoid to many different Guava version in the resulting dist jar.
- Unify two shade executions of flink-runtime into one
- Exclude log4j and slf4j-log4j12 dependency from flink-shaded-curator
- Set default number of application attempts to 1 in standalone case

This closes #1213

Hadoop 2.7.0 pulls in an older Netty version, which clashes with our version.
This makes sure to only pull in our version.

[FLINK-2705] [yarn] Excludes commons-codec:1.3 from aws-java-jdk dependency to resolve dependency ambiguity

This closes #1162.

Tests would fail because of version conflicts because the tests include
the original curator dependencies even though we shade them away in the
final build result.

This also fixes dependency management entries for curator dependencies.
We shade it, therefore it cannot be in the dependency management
section in the root pom.

This closes #1076

[FLINK-2291] [runtime] Add ZooKeeper support to elect a leader from a set of JobManager. The leader will then be retrieved from ZooKeeper by the TaskManagers.

Refactors FlinkMiniCluster to support multiple JobManager

Adds proper remote address resolution for actors

Clean up of LeaderElection and LeaderRetrievalService. Removes synchronization to avoid deadlock.

Adds ZooKeeper start option to TestBaseUtils.startCluster

Removes registration session IDs, using the leader session IDs instead. Sets the leader session ID
 directly in the grantLeadership method. Let the LeaderElectionService select the leader session I
D. Return leader session ID to LeaderRetrievalListeners.

Removes direct ActorRef interaction

Introduces LeaderRetrievalService for the Client and the CliFrontend.

Make ApplicationClient to use the LeaderRetrievalService for JobManager resolution

Adds LeaderElection/Retrieval tests

Added test for exception forwarding from the CuratorFramework to a Contender

Adds test job submission with changing leaders

Adds new test cases for job cleanup after leader election change

Adds new LeaderChangeStateCleanup test case

Adds LeaderElectionRetrievalTestingCluster

Introduces ListeningBehaviour for job submissions

Relocation of org.apache.curator in flink-shaded-hadoop jar

Adds Apache ZooKeeper and Apache Curator to LICENSE and NOTICE files

Increases zookeeper connection timeout to 20000 ms for the KafkaITCase to fix failing tests on Travis

Increased timeouts of ZooKeeperLeaderElectionTest for Travis

Makes the WebInfoServer and the WebRuntimeMonitor to use the LeaderRetrievalService to retrieve the current leading JobManager

Adds proper synchronization to ZooKeeperLeaderElectionService. Fixes StateCheckpointedITCase and PartitionedStateCheckpointingITCase

Adds configuration description for new ZooKeeper configuration values

Fixed port selection of JobManager at startup

Improves logging output

Extends masters file to also specify the webui ports

Adds proper network interface resolution by retrieving the current leader address

Makes the ZooKeeperLeaderElectionService write the leader information in ephemeral nodes so that the information is deleted once the leader has terminated. Fixes a bug in the TaskManager due to call by name semantics of scheduler.scheduleOnce.

Adds jobManagerURL to TriggerTaskManagerRegistration message

Enables findConnectingAddress to use the ZooKeeperLeaderRetrievalService. This allows to test the connection to a possibly changing master node.

Changes startup scripts to respect the recovery mode instead of the ZK_QUORUM

Adjust travis log file to only log zookeeper errors

Updates high availability setup guide

Adds TestLogger to leader election tests

This closes #1016.

[FLINK-2521] [tests] Adds TestLogger class which automatically logs the currently executed test names and the reasons for a failure.

Makes test bases extend TestLogger to add automatic test name logging

Frames test logging statements to make them more prominent

This closes #1015.

 - Adds a separate Maven project for easier maintenance. Also allows users to refer to runtime without web libraries.
 - Simple HTTP server based on Netty HTTP (slim dependency, since we use netty anyways)
 - REST URL parsing via Netty Router
 - Abstract stubs for handlers that deal with errors and request/response
 - First set of URL request handlers that produce JSON responses

This closes #677
This closes #623
This closes #297

- FLINK-2288: Setup ZooKeeper for distributed coordination
  * Add FlinkZooKeeperQuorumPeer to wrap ZooKeeper's quorum peers with
    utilities to write required config values (default datadir, myid)
  * Add default conf/zoo.cfg config for ZooKeeper
  * Add startup scripts for ZooKeeper quorum
  * Add conf/masters file for HA masters

- FLINK-2302: Allow multiple instances to run on single host
  * Multiple TaskManager and JobManager instances can run on a single
    host.

Closes #851

[FLINK-1923] [runtime] Replaces asynchronous logging with synchronous logging using grizzled-slf4j wrapper for Scala.

This closes #628

This closes #35

This closes #617.

This closes #421

- Renames runtime intermediate result classes:
  a) Removes "Intermediate" prefix
  b) Queue => Subpartition
  c) Iterator => View

- [FLINK-1350] Adds a spillable result subpartition variant for BLOCKING
  results, which writes data to memory first and starts to spill
  (asynchronously) if not enough memory is available to produce the
  result in-memory only.

  Receiving tasks of BLOCKING results are only deployed after *all*
  partitions have been fully produced. PIPELINED and BLOCKING results can not
  be mixed.

- [FLINK-1359] Adds simple state tracking to result partitions with
  notifications after partitions/subpartitions have been consumed. Each
  partition has to be consumed at least once before it can be released.

  Currently there is no notion of historic intermediate results, i.e. results
  are released as soon as they are consumed.

This closes #454

This closes #249.

This closes #254.

to HadoopFileSystem wrapper, which supports all subclasses of org.apache.hadoop.fs.FileSystem.
This allows us to let users use all file systems with support for HDFS.
The change has been tested with Tachyon, Google Cloud Storage Hadoop Adapter and HDFS.

The change also cleans up the Hadoop dependency exclusions.

 - Remove obsolete code from old IPC net utils
 - Smaller Writable/IOReadableWritable serialzation buffer start size (most messages are rather small)
 - For message logging, make system calls (timestamps) only in debug mode
 - Clean up warnings / code simplifications

Removed Disclaimer file
Eliminated unnecessary incubating substrings
Bumped version to 0.9-SNAPSHOT

Removed dead instance cleanup from InstanceManager so that Akka's watch mechanism is the current mean to detect dead instances.