Upgrades
  JUnit from 4.11 to 4.12
  Mockito from 1.9.5 to 1.10.19
  PowerMock from 1.5.5 to 1.6.5

This closes #2597

This closes #2458

This closes #2549.

[FLINK-3929] Added Keytab based Kerberos support to enable secure Flink cluster deployment(addresses HDHS, Kafka and ZK services)

FLINK-3929 Added MiniKDC support for Kafka, Zookeeper, RollingFS and Yarn integration test modules

The version change didn't cause the Scalastyle errors. Seems like the
only viable solution to prevent random failures of the Scalastyle
plugin is to disable Scalastyle checks for the affected source file.

This closes #2462

- update checkstyle plugin to 2.17
- update scalastyle plugin to 0.8.0

Implemented Mesos AppMaster including:
- runners for AppMaster and TaskManager
- MesosFlinkResourceManager as a Mesos framework
- ZK persistent storage for Mesos tasks
- reusable scheduler actors for:
  - offer handling using Netflix Fenzo (LaunchCoordinator)
  - reconciliation (ReconciliationCoordinator)
  - task monitoring (TaskMonitor)
  - connection monitoring (ConnectionMonitor)
- lightweight HTTP server to serve artifacts to the Mesos fetcher (ArtifactServer)
- scenario-based logging for:
  - connectivity issues
  - offer handling (receive, process, decline, rescind, accept)
- incorporated FLINK-4152, FLINK-3904, FLINK-4141, FLINK-3675, FLINK-4166

- Add redirect layout
- Remove Maven artifact name warning
- Add info box if stable, but not latest
- Add font-awesome 4.6.3
- Add sidenav layout

This closes #2387.

This commit moves all 'Writable' related code to the 'flink-hadoop-compatibility' project
and uses reflection in 'flink-core' to instantiate WritableTypeInfo when needed.

This closes #2338

This closes #2343

This moves the API compatibility checks into the API projects that use stability annotations.
Previously, every project ran the tests, regardless of whether it contained public API classes or not.

This closes #2334

This closes #2324

This closes #2209.

(The groups being tests starting from A-N and N-Z)

This closes #2201

Makes the JUnit test utils (TestLogger, retry rules, ...) properly available to
other projects without the 'flink-core' test-jar, via the 'flink-test-utils-junit' project.

Makes the ForkableMiniCluster, TestEnvironment,  and other test utilities available in the 'main'
scope of the 'flink-test-utils' project.

Creates a 'flink-test-utils-parent' project that holds the 'flink-test-utils-junit' and
'flink-test-utils' project.

Also moves some tests between projects and inlines some very simple utility functions in
order to simplify some test jar dependencies.

Some of the changes include:

- Remove unneeded dependencies (nano, wget)
- Remove apt lists to reduce image size
- Reduce number of layers on the docker image (best docker practice)
- Remove useless variables and base the code in generic ones e.g.
FLINK_HOME
- Change the default JDK from oracle to openjdk-8-jre-headless, based on
two reasons:

1. You cannot legally repackage the oracle jdk in docker images
2. The open-jdk headless is more appropriate for a server image (no GUI stuff)

- Return port assignation to the standard FLINK one:

Variable: docker-flink -> flink

taskmanager.rpc.port: 6121 -> 6122
taskmanager.data.port: 6122 -> 6121
jobmanager.web.port: 8080 -> 8081

This closes #2176

- always ship the lib folder
- properly setup the classpath from the supplied ship files
- cleanup deploy() method of YarnClusterDescriptor
- add test case

This closes #2187

This closes #2112

Jar arguments with a single '-' were not parsed correctly if options were
present. For example, in `./flink run <options> file.jar -arg value` the
jar arguments would be parsed as "arg" and "value". Interestingly, this only
happened when <options> were present.

The issue has been fixed in commons-cli 1.3.1.

A test case was added to test for regressions.

This closes #2139

The ScalaShellITCase sometimes gets stuck before test execution with no output
in the logs. We ran about a hundred builds against Surefire 2.18.1 which
confirmed that the failures don't occur with this version.

Waiting for an upstream fix until this can be reverted.

This closes #2101

This makes running test locally a more pleasant experience. It still
uses all exposed CPU cores (virtual or real). A custom forkCount can be
set using the flink.forkCount property, e.g. -Dflink.forkCount=4.

The Flink documentation build process is currently quite messy. These
changes move us to a new build process with proper dependency
handling. It assures that we all use the same dependency versions for
consistent build output. Also, it eases the automated building process
on other systems (like the ASF Buildbot). The goal was to make the
documentation build process easier and self-contained.

- use Ruby's Bundler Gem to install dependencies
- update README
- adapt Dockerfile
- add additional rules to .gitignore
- change default doc output path from /target to /content
(default path of the flink-web repository)

This closes #2033

This closes #2042

After 38698c0b, there are now two
executions defined for the Surefire plugin: unit-tests and
integration-tests. In addition, there is an implicit default execution
called default-test. This leads to the unit tests to be executed
twice. This renames unit-tests to default-test to prevent duplicate
execution.

This closes #2019

  - Metric groups are generally thread-safe
  - Metric groups are closable. Closed groups do not register metric objects and more.
  - TaskManager's JobMetricsGroup auto disposes when all TaskMetricGroups are closed
  - Maven project with metric reporters renamed to 'flink-metric-reporters'
  - Various code style cleanups

Failures during execution of the integration tests with the Maven
Failsafe plugin were silently ignored on Travis with Maven version
3.2.5. The problem is that failures are not passed on correctly from the
'integration-test' phase (where failures are recorded and tolerated) to
the 'verify' phase.

The cause of the error is most likely SUREFIRE-1127. An exception in the
'integration-test' is sometimes not flushed back to disk where it is
evaluated in the 'verify' phase. Bumping the version of the Failsafe
plugin from 2.18.1 to 2.19.1 fixes the issue but introduces classpath
related problems.

We don't have to rely on Failsafe for running integration
tests. Surefire offers the same feature set we use from the Failsafe
plugin.

- replace Failsafe with Surefire
- update to latest Surefire version
- get rid of duplicate property declaration

This closes #2003

This closes #1952

- link from CONTRIBUTING.md to PR template
- add it to excluded file of rat plugin

This closes #1729.

Consolidation is done via plugin management.

Closes #1753

Deployment will be handled by ASF's Jenkins instances which trigger
the deployment of snapshot versions every night. This way, we have not
only separated deployment from CI testing, but also improved the
reliability of the test and deployment infrastructure.

- remove Travis dependency from Maven deploy script

- set force-dependency version to Flink version

This assures deployment works when only credentials for the snapshot
deployment are available. Previously, we deployed snapshots with release
credentials. These credentials would be over-privileged.

- remove old credentials from Travis config file

This closes #1619.

This ensures that all properties of the root pom are properly
resolved by running the Shade plugin. Thus, our root pom does not have
to depend on a Scala version just because it holds the Scala version
properties.

This closes #1755

This closes #1701

[FLINK-3459] [build] Fix conflicting dependencies commons-collections, commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which pulls in transitively the commons-collection, commons-beanutils and common-beanutils-core depedencies. Commons-beanutils and commons-collection contain classes which live in the same namespace. They are also binary compatible but not binary identical. This is a problem for the sbt assembly plugin which checks for binary identity. In order to solve the problem, we bump the commons-configuration version to 1.7 so that only commons-beanutils is pulled in. This is necessary, because the transitive promotion of dependencies of the shade plugin only excludes the commons-beanutils dependency only from the directly depending dependency. All parent dependencies won't have the exclusion. This is a problem for SBT which will pull the dependency as part of one of the parents, then. Moreover, we replace commons-beanutils by commons-beanutils-bean-collections which contains only the non-conflicting classes wrt commons-collections.

This closes #1682.