The ScalaShellITCase sometimes gets stuck before test execution with no output
in the logs. We ran about a hundred builds against Surefire 2.18.1 which
confirmed that the failures don't occur with this version.

Waiting for an upstream fix until this can be reverted.

This closes #2101

This closes #2099

This commit is a combination of several commits/changes. It combines
changes to the file input formats and the streaming file read operator
and integrates them into the API.

These are the messages of the other two commits:

[FLINK-3717] Make FileInputFormat checkpointable

This adds a new interface called CheckpointableInputFormat
which describes input formats whose state is queryable,
i.e. getCurrentState() returns where the reader is
in the underlying source, and they can resume reading from
a user-specified position.

This functionality is not yet leveraged by current readers.

[FLINK-3889] Refactor File Monitoring Source

This is meant to replace the different file
reading sources in Flink streaming. Now there is
one monitoring source with DOP 1 monitoring a
directory and assigning input split to downstream
readers.

In addition, it makes the new features added by
FLINK-3717 work together with the aforementioned entities
(the monitor and the readers) in order to have
fault tolerant file sources and exactly once guarantees.

This does not replace the old API calls. This
will be done in a future commit.

It adds a method failExternally() to the StreamTask, so that custom Operators
can make their containing task fail when needed.

This closes #2049

[revert] [FLINK-3944] [tableAPI] Reverts "Add rewrite rules to reorder Cartesian products and joins."

This reverts commit 85793a25.
Added rewrite rules blow up search space which cannot be effectively pruned without cardinality estimates.

This closes #2098

These rules are necessary to resolve join orders that initially contain Cartesian products
due to the order in which base relations are added in the FROM clause (SQL) or joined (Table API).

This closes #2044

[hotfix] [core] Make sure Dropwizard reporters do not block operator creation/teardown while reporting

The "report()" method used to hold the same lock that was needed to create and shutdown metrics.
Holding that lock too long could delay operator creation and teardown.

This also removes the no longer needed dependency on dropwizard in flink-core.

This closes #2063

This also does some minor code cleanups, logging fixes, and smoother concurrency handling.

This closes #2080

This closes #2054

Adds retries and timeout scaling to all iteration tests, which rely
on iteration timeouts. The way the tests rely on these timoeuts is
prone to races. If the failures occur again, I vote to ignore the
tests until iteration termination is fixed properly.

Example test failures:
- https://s3.amazonaws.com/archive.travis-ci.org/jobs/134215892/log.txt
- https://s3.amazonaws.com/archive.travis-ci.org/jobs/134215975/log.txt

This closes #2087.

This closes #1953

This closes #2057

This closes #2082

Increase the level of significance from p=0.01 to p=0.001 and add retry annotations
to random sampler tests. This should decrease the probability of failing random
sampler tests.

This closes #2076

This closes #2011

This closes #2047

As says documentation for MessageAcknowledgingSourceBase.restoreState()

This method is invoked when a function is executed as part of a recovery run. Note that restoreState() is called before open().

So current implementation

1. Fails on restoreState with NullPointerException, jobs fail to restart.
2. Does not restore anything because following open erases all checkpoint data immediately.
3. As consequence, violates exactly once rule because processed but not acknowledged list erased.

Proposed change fixes that.

This closes #2062

This closes #2065

This patch adds the possibilty for th user of the RabbitMQ
Streaming Connector to customize the queue which is used. There
are use-cases in which you want to set custom parameters for the
queue (i.e. TTL of the messages if Flink reboots) or the
possibility to bind the queue to an exchange afterwards.

The commit doesn't change the actual behaviour but makes it
possible for users to override the newly create `setupQueue`
method and cutomize their implementation. This was not possible
before.

This closes #2073

This closes #2059

The global clustering coefficient measures the connectedness of a graph.
Scores range from 0.0 (no triangles) to 1.0 (complete graph).

This closes #1997

This closes #2067

This closes #2005

This closes #2064

Before, it could happen that an asynchronous checkpoint was going on
when trying to do cleanup. Now we protect cleanup and asynchronous
checkpointing by a lock.

This reverts commit 98a93955.

This closes #2068

This closes #2021

This closes #2036