After 38698c0b, there are now two
executions defined for the Surefire plugin: unit-tests and
integration-tests. In addition, there is an implicit default execution
called default-test. This leads to the unit tests to be executed
twice. This renames unit-tests to default-test to prevent duplicate
execution.

This closes #2019

  - Metric groups are generally thread-safe
  - Metric groups are closable. Closed groups do not register metric objects and more.
  - TaskManager's JobMetricsGroup auto disposes when all TaskMetricGroups are closed
  - Maven project with metric reporters renamed to 'flink-metric-reporters'
  - Various code style cleanups

Failures during execution of the integration tests with the Maven
Failsafe plugin were silently ignored on Travis with Maven version
3.2.5. The problem is that failures are not passed on correctly from the
'integration-test' phase (where failures are recorded and tolerated) to
the 'verify' phase.

The cause of the error is most likely SUREFIRE-1127. An exception in the
'integration-test' is sometimes not flushed back to disk where it is
evaluated in the 'verify' phase. Bumping the version of the Failsafe
plugin from 2.18.1 to 2.19.1 fixes the issue but introduces classpath
related problems.

We don't have to rely on Failsafe for running integration
tests. Surefire offers the same feature set we use from the Failsafe
plugin.

- replace Failsafe with Surefire
- update to latest Surefire version
- get rid of duplicate property declaration

This closes #2003

This closes #1952

- link from CONTRIBUTING.md to PR template
- add it to excluded file of rat plugin

This closes #1729.

Consolidation is done via plugin management.

Closes #1753

Deployment will be handled by ASF's Jenkins instances which trigger
the deployment of snapshot versions every night. This way, we have not
only separated deployment from CI testing, but also improved the
reliability of the test and deployment infrastructure.

- remove Travis dependency from Maven deploy script

- set force-dependency version to Flink version

This assures deployment works when only credentials for the snapshot
deployment are available. Previously, we deployed snapshots with release
credentials. These credentials would be over-privileged.

- remove old credentials from Travis config file

This closes #1619.

This ensures that all properties of the root pom are properly
resolved by running the Shade plugin. Thus, our root pom does not have
to depend on a Scala version just because it holds the Scala version
properties.

This closes #1755

This closes #1701

[FLINK-3459] [build] Fix conflicting dependencies commons-collections, commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which pulls in transitively the commons-collection, commons-beanutils and common-beanutils-core depedencies. Commons-beanutils and commons-collection contain classes which live in the same namespace. They are also binary compatible but not binary identical. This is a problem for the sbt assembly plugin which checks for binary identity. In order to solve the problem, we bump the commons-configuration version to 1.7 so that only commons-beanutils is pulled in. This is necessary, because the transitive promotion of dependencies of the shade plugin only excludes the commons-beanutils dependency only from the directly depending dependency. All parent dependencies won't have the exclusion. This is a problem for SBT which will pull the dependency as part of one of the parents, then. Moreover, we replace commons-beanutils by commons-beanutils-bean-collections which contains only the non-conflicting classes wrt commons-collections.

This closes #1682.

This closes #1615

[FLINK-3373] [build] Revert HTTP Components versions because of incompatibility with Hadoop >= 2.6.0

[FLINK-3373] [build] Bump version of transitine HTTP Components dependency to 4.4.4 (core) / 4.5.1 (client)

- add suffixes
- adapt change-scala-version script

This closes #1529.

For adding Kafka 0.9.0.0 support, this commit changes the following:
- Split up of the kafka connector into a flink-connector-kafka-(base|0.9|0.8) with different dependencies
- The base package contains common test cases, classes and implementations (the producer for 0.9 and 0.8 relies on exactly the same code)
- the 0.8 package contains a kafka connector implementation against the SimpleConsumer (low level) API of Kafka 0.8. There are some additional tests for the ZK offset committing
- The 0.9 package relies on the new Consumer API of Kafka 0.9.0.0
- Support for metrics for all producers and the 0.9 consumer through Flink's accumulators.

This closes #1489

This closes #1492
This closes #1482

[FLINK-3195] [examples] Consolidate batch examples into one project, unify batch and streaming examples under on parent project

Netty versions >= 4.0.28.Final contain an improvement by Netty, which slices
a Netty buffer instead of doing a memory copy (https://github.com/netty/netty/issues/3704)
in the LengthFieldBasedFrameDecoder. In some situations, this interacts badly
with our Netty pipeline leading to OutOfMemory error.

To reproduce, run batch WordCount with a range partitioner and reasonably large
data.

This closes #1477

This closes #1427

This closes #1456

This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.

With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.

[FLINK-3005] [core] Bump commons-collections version to fix object deserialization remote command execution vulnerability

This closes #1381

This closes #1257

This closes #1316.

- This ensures Checkstyle works properly, e.g. in IDEs

This closes #1301.

0.10-SNAPSHOT continues on branch release-0.10

A collection of small documentation and grammar updates.

This closes #1277

[FLINK-2354] [runtime] Replace old StateHandleProvider by StateStorageHelper in ZooKeeperStateHandleStore

The old StateHandleProvider used in ZooKeeperStateHandleStore had to be replaced because the state backend implementation has changed. Since the new state backend could not be used anymore, a new StateStorageHelper interface has been created. The default implementation FileSystemStateStorageHelper stores the given state onto the specified file system and returns a FileSerializableStateHandle.

Various fixes due to rebasing.

Squashes:
5a88d5e [tests] Add HttpTestClient for testing HTTP responses
656d6d6 Split WebMonitor and LeaderRetrievalService start up
a7e8da8 Move generated /web files to src/main/resources

Add comment to webMonitorPort attribute and make line breaks more Scalaesque

Don't block on leader retrieval and only resolve associated job manager once

Make JobManagerRetriever independent of redirecting logic

This closes #1202.

- make new web one the default
- adapt tests
- make web directory a resource to be included in the fat jar
- serve static files of web interface dynamic through the class loader
- run on YARN
- remove Jetty dependencies from poms