This closes #1618

This closes #1585

This closes #1599

[FLINK-3373] [build] Revert HTTP Components versions because of incompatibility with Hadoop >= 2.6.0

[FLINK-3373] [build] Bump version of transitine HTTP Components dependency to 4.4.4 (core) / 4.5.1 (client)

This also cleans up the generics in the RocksDB state classes.

This closes #1608

This closes #1601

This closes #1602

Removes curator dependency exclusions from flink-runtime. This resolves
NoClassDefFoundError exceptions when running `mvn test`.

This partialy reverts e31a4d8a.

This closes #1596

This closes #1578.

This closes #1598.

This is the Javadoc of DataStream.rescale() that describes the
behaviour:

Sets the partitioning of the {@link DataStream} so that the output elements
are distributed evenly to a subset of instances of the next operation in a round-robin
fashion.

The subset of downstream operations to which the upstream operation sends
elements depends on the degree of parallelism of both the upstream and downstream operation.
For example, if the upstream operation has parallelism 2 and the downstream operation
has parallelism 4, then one upstream operation would distribute elements to two
downstream operations while the other upstream operation would distribute to the other
two downstream operations. If, on the other hand, the downstream operation has parallelism
2 while the upstream operation has parallelism 4 then two upstream operations will
distribute to one downstream operation while the other two upstream operations will
distribute to the other downstream operations.

In cases where the different parallelisms are not multiples of each other one or several
downstream operations will have a differing number of inputs from upstream operations.

This was causing problems with TypeSerializers that choke on null
values, especially in the Scala KeyedStream.*WithState() family of
functions.

This closes #1591

tl;dr Change default Netty configuration to be relative to number of slots,
i.e. configure one memory arena (in PooledByteBufAllocator) per slot and use one
event loop thread per slot. Behaviour can still be manually overwritten. With
this change, we can expect 16 MB of direct memory allocated per task slot by
Netty.

Problem: We were using Netty's default PooledByteBufAllocator instance, which
is subject to changing behaviour between Netty versions (happened between
versions 4.0.27.Final and 4.0.28.Final resulting in increased memory
consumption) and whose default memory consumption depends on the number of
available cores in the system. This can be problematic for example in YARN
setups where users run one slot per task manager on machines with many cores,
resulting in a relatively high number of allocated memory.

Solution: We instantiate a PooledByteBufAllocator instance manually and wrap
it as a NettyBufferPool. Our instance configures one arena per task slot as
default. It's desirable to have the number of arenas match the number of event
loop threads to minimize lock contention (Netty's default tried to ensure this
as well), hence the number of threads is changed as well to match the number
of slots as default. Both number of threads and arenas can still be manually
configured.

This closes #1593.

This closes #1428

This closes #1590

The GradientDescent implementation did not work with sparse input data
because it requires the gradient to be dense. This patch makes sure that
the gradient sum is always dense.

This closes #1587.

This closes #1592.

This closes #1574