This closes #4673

This closes #4632

This closes #4562

This closes #4676

This closes #4575

Program output is written to TaskManager logs.

This closes #4641

The Simplify parameter should accept and set the parallelism when
calling the Simplify algorithms.

The LocalClusteringCoefficient "count triangles" reduce now uses the
assigned ("little") parallelism as this computation is proportional to
the number of vertices (the combine computation is proportional to the
potentially much larger number of triangles).

The ignored CombineHint on the HITS all-reduces have been removed.

This closes #4346

There exist some tests with empty graphs but the `EmptyGraph` in
`AsmTestBase` contained vertices but no edges. Add a new `EmptyGraph`
without vertices and test both empty graphs for each algorithm.

EmptyGraph now generates the proper TypeInformation (for Edge<> not
Tuple3) which had prevented adding edges due to a union incompatibility.

GraphGeneratorUtils#vertexSet now uses a hash-combine for distinct.

`PageRank` optionally includes zero-degree vertices in the results
(at a performance cost).

This closes #4405

[FLINK-7617] [runtime] Remove string format in BitSet to improve the performance of BuildSideOuterjoin

This closes #4668.

This closes #4622.

- Improve deprecation message in Javadocs
- Remove usage of ProducerConfigConstants in code wherever possible
- Remove usage of ProducerConfigConstants in documentation code snippets

This closes #4473.

[FLINK-7440] [kinesis] Eagerly check that provided schema and partitioner is serializable in FlinkKinesisProducer

This commit also adds a test to verify that the FlinkKinesisProducer is
serializable.

This closes #4537.

[FLINK-7440] [kinesis] Eagerly check serializability of deserialization schema in FlinkKinesisConsumer

This commit also adds tests for verifying that the FlinkKinesisConsumer
itself is serializable.

This closes #4526.

Previously, the AbstractPartitionDiscoverer tracked discovered
partitions by keeping only the largest discovered partition id. All
fetched partition metadata with ids smaller than this id would be
considered as discovered. This assumption of contiguous partition ids is
too naive for corner cases where there may be undiscovered partitions
that were temporariliy unavilable before and were shadowed by
discoverered partitions with largerer partition ids.

This commit changes to use a set to track seen partitions. This also
removes the need of pre-sorting fetched partitions.

Closing StreamOperators is still part of the StreamTask's running lifecycle,
because operators which perform asynchronous operations usually finish their
work when the StreamOperator is closed. Since this also entails that errors
can occur and that a checkpointing operation is triggered, we should only set
the StreamTask's isRunning to false after all StreamOperators have been closed.

Furthermore, this commit introduces a while guard for the waiting condition in
ContinuousFileReaderOperator#close.

This closes #4650.

FutureUtils.retryWithDelay executes the given operation of type
Callable<CompletableFuture<T>> n times and waits in between retries the given
delay. This allows to retry an operation with a specified delay.

Make retry and retry with delay future properly cancellable

This closes #4637.

This closes #4608.

This closes #4507.

This closes #4640.

This closes #4633.

This closes #4546.

This commit lets the JobMaster extend from FencedRpcEndpoint. This enables
automatic fencing of all messages. Moreover, this PR introduces the
JobMasterId which is the new leader id/fencing token replacing the UUID. This
improves type safety when passing multiple fencing tokens around.

This closes #4583.

This commit waits for the completion of the shutdown of the ResourceManager
before shutting down the ResourceManagerRuntimeServices. The JobManagerServices are
now exclusively passed in to the JobManagerRunner which means that it is no
longer responsible for shutting the JobManagerServices down. Additionally,
it waits until the JobMaster has been shut down before closing the
LeaderElectionService as well as the JobManagerMetricGroup.

The JobManagerServices are now managed by the caller of the JobManagerRunner. This
allows to reuse them across multiple JobManagerRunners.

The RpcEndpoint#postStop method is now called by the UntypedActor#postStop method,
which ensures that the RpcEndpoint's method is also called if only the underlying
RpcService is shut down (without explicitly shutting down the RpcEndpoint).

This closes #4596.

Properly fences the ResourceManager by letting it extend the FencedRpcEndpoint.
Moreover, this PR introduces a ResourceManagerId which replaces the UUID as
leader id/fencing token. This will give us more type safety when defining rpc
interfaces.

This closes #4582.

Let the Dispatcher extend the FencedRpcEndpoint and introduce DispatcherId which
replaces the UUID as leader id/fencing token.

This closes #4584.

This commit filters out duplicate JobManager gained leadership messges coming from
the JobLeaderService. This avoid opening multiple connections to the JobManager
which consumes resources. Moreover, this commit properly closes all
JobManagerConnections in case of a shut down.

This closes #4595.

This commit changes the JobMaster such that it's leader session id is only set
in the main thread. This is done by passing the leader session id to the
startJobExecution method. Moreover, this commit returns a future from the
start and suspend methods which can be used to wait on the completion of these
operations.

This closes #4579.

Make all external calls from the RpcEndpoint's main thread non blocking by
executing them as a runnable in an Executor.

Make FatalErrorHandler calls non asynchronous

This closes #4540.

This commit changes the behaviour of the WebRuntimeMonitor to not longer block serving
threads by waiting on the result of futures. Instead the RequestHandler now returns a
CompletableFuture<FullHttpResponse> which is written out to the Netty channel upon
completion. This will improve the performance of our WebRuntimeMonitor.

This closes #4527.

Introduce FencedRpcEndpoint which requires all RPC messages to have a
fencing token attached. Based on the received fencing token and the
actual fencing token, the message will either be discarded if they are
not equal or it will be processed. That way we are able to filter out
old messages or messages which originate from a split brain situation

Add support for callAsyncWithoutFencing

Introduce local and remote fenced messages

This closes #4578.

Enrich the RestClientException with the HttpResponseStatus in case of failures. This
makes the exception handling on the client side easier.

This closes #4588.

The termination future is completed when the ClusterEntrypoint shuts down. This
allows for easier testing.

This closes #4589.