This is the Javadoc of DataStream.rescale() that describes the
behaviour:

Sets the partitioning of the {@link DataStream} so that the output elements
are distributed evenly to a subset of instances of the next operation in a round-robin
fashion.

The subset of downstream operations to which the upstream operation sends
elements depends on the degree of parallelism of both the upstream and downstream operation.
For example, if the upstream operation has parallelism 2 and the downstream operation
has parallelism 4, then one upstream operation would distribute elements to two
downstream operations while the other upstream operation would distribute to the other
two downstream operations. If, on the other hand, the downstream operation has parallelism
2 while the upstream operation has parallelism 4 then two upstream operations will
distribute to one downstream operation while the other two upstream operations will
distribute to the other downstream operations.

In cases where the different parallelisms are not multiples of each other one or several
downstream operations will have a differing number of inputs from upstream operations.

This was causing problems with TypeSerializers that choke on null
values, especially in the Scala KeyedStream.*WithState() family of
functions.

This closes #1591

tl;dr Change default Netty configuration to be relative to number of slots,
i.e. configure one memory arena (in PooledByteBufAllocator) per slot and use one
event loop thread per slot. Behaviour can still be manually overwritten. With
this change, we can expect 16 MB of direct memory allocated per task slot by
Netty.

Problem: We were using Netty's default PooledByteBufAllocator instance, which
is subject to changing behaviour between Netty versions (happened between
versions 4.0.27.Final and 4.0.28.Final resulting in increased memory
consumption) and whose default memory consumption depends on the number of
available cores in the system. This can be problematic for example in YARN
setups where users run one slot per task manager on machines with many cores,
resulting in a relatively high number of allocated memory.

Solution: We instantiate a PooledByteBufAllocator instance manually and wrap
it as a NettyBufferPool. Our instance configures one arena per task slot as
default. It's desirable to have the number of arenas match the number of event
loop threads to minimize lock contention (Netty's default tried to ensure this
as well), hence the number of threads is changed as well to match the number
of slots as default. Both number of threads and arenas can still be manually
configured.

This closes #1593.

This closes #1428

This closes #1590

The GradientDescent implementation did not work with sparse input data
because it requires the gradient to be dense. This patch makes sure that
the gradient sum is always dense.

This closes #1587.

This closes #1592.

This closes #1574

This closes #1584

This closes #1568

Parallelizes cluster start-up and tear-down when pdsh is available

This closes #1523

Added a method to set a default application name for the Flink Yarn session CLI.
Switched the order, such that this name can now be overwritten by the command line.

This closes #1558

This changes window operator to use the new partitioned state
abstraction for keeping window contents instead of custom internal
state and the checkpointed interface.

For now, timers are still kept as custom checkpointed state, however.

WindowOperator now expects a StateIdentifier for MergingState, this can
either be for ReducingState or ListState but WindowOperator is agnostic
to the type of State. Also the signature of WindowFunction is changed to
include the type of intermediate input. For example, if a ReducingState
is used the input of the WindowFunction is T (where T is the input
type). If using a ListState the input of the WindowFunction would be of
type Iterable[T].

Add new state types ValueState, ListState and ReducingState, where
ListState and ReducingState derive from interface MergingState.

ValueState behaves exactly the same as OperatorState. MergingState is a
stateful list to which elements can be added and for which the elements
that it contains can be obtained. If using a ListState the list of
elements is actually kept, for a ReducingState a reduce function is used
to combine all added elements into one. To create a ValueState the user
passes a ValueStateIdentifier to
StreamingRuntimeContext.getPartitionedState() while they would pass a
ListStateIdentifier or ReducingStateIdentifier for the other state
types.

This change is necessary to give the system more information about the
nature of the operator state. We want this to be able to do incremental
snapshots. This would not be possible, for example, if the user had a
List as a state. Inside OperatorState this list would be opaque and
Flink could not create good incremental snapshots.

This also refactors the StateBackend. Before, the logic for partitioned
state was spread out over StreamingRuntimeContext,
AbstractStreamOperator and StateBackend. Now it is consolidated in
StateBackend.

This also adds support for partitioned state in two-input operators.

This closes #1582, #1576.

Old guides are redirected to the new pages.

The LegacyFetcher used the given KeyedDeserializationSchema across multiple threads even though
it is not thread-safe. This commit fixes the problem by cloning the KeyedDeserializationSchema
before giving it to the SimpleConsumerThread.

Add clone method for Java serializable objects to InstantiationUtil

This closes #1577.

[FLINK-3314] [streaming] Fix case where early cancel messages do not properly cancel a stream operator.

  - Clean up generics
  - Clean and safe disposal of initialized resources
  - Add names to asynchronous materialization threads
  - Fix concurrent modification of  materialization threads set

This deactivates the validation of results, which is not reliably possible under the current model (timeout on feedback).
This test for now only checks that the job executes properly.

Also adds proper logging property files for the examples projects.