This closes #2209.

(The groups being tests starting from A-N and N-Z)

This closes #2201

Makes the JUnit test utils (TestLogger, retry rules, ...) properly available to
other projects without the 'flink-core' test-jar, via the 'flink-test-utils-junit' project.

Makes the ForkableMiniCluster, TestEnvironment,  and other test utilities available in the 'main'
scope of the 'flink-test-utils' project.

Creates a 'flink-test-utils-parent' project that holds the 'flink-test-utils-junit' and
'flink-test-utils' project.

Also moves some tests between projects and inlines some very simple utility functions in
order to simplify some test jar dependencies.

Some of the changes include:

- Remove unneeded dependencies (nano, wget)
- Remove apt lists to reduce image size
- Reduce number of layers on the docker image (best docker practice)
- Remove useless variables and base the code in generic ones e.g.
FLINK_HOME
- Change the default JDK from oracle to openjdk-8-jre-headless, based on
two reasons:

1. You cannot legally repackage the oracle jdk in docker images
2. The open-jdk headless is more appropriate for a server image (no GUI stuff)

- Return port assignation to the standard FLINK one:

Variable: docker-flink -> flink

taskmanager.rpc.port: 6121 -> 6122
taskmanager.data.port: 6122 -> 6121
jobmanager.web.port: 8080 -> 8081

This closes #2176

- always ship the lib folder
- properly setup the classpath from the supplied ship files
- cleanup deploy() method of YarnClusterDescriptor
- add test case

This closes #2187

This closes #2112

Jar arguments with a single '-' were not parsed correctly if options were
present. For example, in `./flink run <options> file.jar -arg value` the
jar arguments would be parsed as "arg" and "value". Interestingly, this only
happened when <options> were present.

The issue has been fixed in commons-cli 1.3.1.

A test case was added to test for regressions.

This closes #2139

The ScalaShellITCase sometimes gets stuck before test execution with no output
in the logs. We ran about a hundred builds against Surefire 2.18.1 which
confirmed that the failures don't occur with this version.

Waiting for an upstream fix until this can be reverted.

This closes #2101

This makes running test locally a more pleasant experience. It still
uses all exposed CPU cores (virtual or real). A custom forkCount can be
set using the flink.forkCount property, e.g. -Dflink.forkCount=4.

The Flink documentation build process is currently quite messy. These
changes move us to a new build process with proper dependency
handling. It assures that we all use the same dependency versions for
consistent build output. Also, it eases the automated building process
on other systems (like the ASF Buildbot). The goal was to make the
documentation build process easier and self-contained.

- use Ruby's Bundler Gem to install dependencies
- update README
- adapt Dockerfile
- add additional rules to .gitignore
- change default doc output path from /target to /content
(default path of the flink-web repository)

This closes #2033

This closes #2042

After 38698c0b, there are now two
executions defined for the Surefire plugin: unit-tests and
integration-tests. In addition, there is an implicit default execution
called default-test. This leads to the unit tests to be executed
twice. This renames unit-tests to default-test to prevent duplicate
execution.

This closes #2019

  - Metric groups are generally thread-safe
  - Metric groups are closable. Closed groups do not register metric objects and more.
  - TaskManager's JobMetricsGroup auto disposes when all TaskMetricGroups are closed
  - Maven project with metric reporters renamed to 'flink-metric-reporters'
  - Various code style cleanups

Failures during execution of the integration tests with the Maven
Failsafe plugin were silently ignored on Travis with Maven version
3.2.5. The problem is that failures are not passed on correctly from the
'integration-test' phase (where failures are recorded and tolerated) to
the 'verify' phase.

The cause of the error is most likely SUREFIRE-1127. An exception in the
'integration-test' is sometimes not flushed back to disk where it is
evaluated in the 'verify' phase. Bumping the version of the Failsafe
plugin from 2.18.1 to 2.19.1 fixes the issue but introduces classpath
related problems.

We don't have to rely on Failsafe for running integration
tests. Surefire offers the same feature set we use from the Failsafe
plugin.

- replace Failsafe with Surefire
- update to latest Surefire version
- get rid of duplicate property declaration

This closes #2003

This closes #1952

- link from CONTRIBUTING.md to PR template
- add it to excluded file of rat plugin

This closes #1729.

Consolidation is done via plugin management.

Closes #1753

Deployment will be handled by ASF's Jenkins instances which trigger
the deployment of snapshot versions every night. This way, we have not
only separated deployment from CI testing, but also improved the
reliability of the test and deployment infrastructure.

- remove Travis dependency from Maven deploy script

- set force-dependency version to Flink version

This assures deployment works when only credentials for the snapshot
deployment are available. Previously, we deployed snapshots with release
credentials. These credentials would be over-privileged.

- remove old credentials from Travis config file

This closes #1619.

This ensures that all properties of the root pom are properly
resolved by running the Shade plugin. Thus, our root pom does not have
to depend on a Scala version just because it holds the Scala version
properties.

This closes #1755

This closes #1701

[FLINK-3459] [build] Fix conflicting dependencies commons-collections, commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which pulls in transitively the commons-collection, commons-beanutils and common-beanutils-core depedencies. Commons-beanutils and commons-collection contain classes which live in the same namespace. They are also binary compatible but not binary identical. This is a problem for the sbt assembly plugin which checks for binary identity. In order to solve the problem, we bump the commons-configuration version to 1.7 so that only commons-beanutils is pulled in. This is necessary, because the transitive promotion of dependencies of the shade plugin only excludes the commons-beanutils dependency only from the directly depending dependency. All parent dependencies won't have the exclusion. This is a problem for SBT which will pull the dependency as part of one of the parents, then. Moreover, we replace commons-beanutils by commons-beanutils-bean-collections which contains only the non-conflicting classes wrt commons-collections.

This closes #1682.

This closes #1615

[FLINK-3373] [build] Revert HTTP Components versions because of incompatibility with Hadoop >= 2.6.0

[FLINK-3373] [build] Bump version of transitine HTTP Components dependency to 4.4.4 (core) / 4.5.1 (client)

- add suffixes
- adapt change-scala-version script

This closes #1529.

For adding Kafka 0.9.0.0 support, this commit changes the following:
- Split up of the kafka connector into a flink-connector-kafka-(base|0.9|0.8) with different dependencies
- The base package contains common test cases, classes and implementations (the producer for 0.9 and 0.8 relies on exactly the same code)
- the 0.8 package contains a kafka connector implementation against the SimpleConsumer (low level) API of Kafka 0.8. There are some additional tests for the ZK offset committing
- The 0.9 package relies on the new Consumer API of Kafka 0.9.0.0
- Support for metrics for all producers and the 0.9 consumer through Flink's accumulators.

This closes #1489

This closes #1492
This closes #1482

[FLINK-3195] [examples] Consolidate batch examples into one project, unify batch and streaming examples under on parent project

Netty versions >= 4.0.28.Final contain an improvement by Netty, which slices
a Netty buffer instead of doing a memory copy (https://github.com/netty/netty/issues/3704)
in the LengthFieldBasedFrameDecoder. In some situations, this interacts badly
with our Netty pipeline leading to OutOfMemory error.

To reproduce, run batch WordCount with a range partitioner and reasonably large
data.

This closes #1477

This closes #1427

This closes #1456

This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.

With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.