- 08 2月, 2016 5 次提交
-
-
由 Aljoscha Krettek 提交于
This is the Javadoc of DataStream.rescale() that describes the behaviour: Sets the partitioning of the {@link DataStream} so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion. The subset of downstream operations to which the upstream operation sends elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 4, then one upstream operation would distribute elements to two downstream operations while the other upstream operation would distribute to the other two downstream operations. If, on the other hand, the downstream operation has parallelism 2 while the upstream operation has parallelism 4 then two upstream operations will distribute to one downstream operation while the other two upstream operations will distribute to the other downstream operations. In cases where the different parallelisms are not multiples of each other one or several downstream operations will have a differing number of inputs from upstream operations.
-
由 Aljoscha Krettek 提交于
This was causing problems with TypeSerializers that choke on null values, especially in the Scala KeyedStream.*WithState() family of functions.
-
由 Ufuk Celebi 提交于
-
由 Stephan Ewen 提交于
-
由 mjsax 提交于
This closes #1591
-
- 07 2月, 2016 1 次提交
-
-
由 Ufuk Celebi 提交于
tl;dr Change default Netty configuration to be relative to number of slots, i.e. configure one memory arena (in PooledByteBufAllocator) per slot and use one event loop thread per slot. Behaviour can still be manually overwritten. With this change, we can expect 16 MB of direct memory allocated per task slot by Netty. Problem: We were using Netty's default PooledByteBufAllocator instance, which is subject to changing behaviour between Netty versions (happened between versions 4.0.27.Final and 4.0.28.Final resulting in increased memory consumption) and whose default memory consumption depends on the number of available cores in the system. This can be problematic for example in YARN setups where users run one slot per task manager on machines with many cores, resulting in a relatively high number of allocated memory. Solution: We instantiate a PooledByteBufAllocator instance manually and wrap it as a NettyBufferPool. Our instance configures one arena per task slot as default. It's desirable to have the number of arenas match the number of event loop threads to minimize lock contention (Netty's default tried to ensure this as well), hence the number of threads is changed as well to match the number of slots as default. Both number of threads and arenas can still be manually configured. This closes #1593.
-
- 05 2月, 2016 9 次提交
-
-
由 Aljoscha Krettek 提交于
-
由 Robert Metzger 提交于
This closes #1428
-
由 Stephan Ewen 提交于
This closes #1590
-
由 Till Rohrmann 提交于
The GradientDescent implementation did not work with sparse input data because it requires the gradient to be dense. This patch makes sure that the gradient sum is always dense. This closes #1587.
-
由 Ufuk Celebi 提交于
-
由 Stefano Baghino 提交于
This closes #1592.
-
由 Aljoscha Krettek 提交于
-
由 Márton Balassi 提交于
This closes #1574
-
由 Márton Balassi 提交于
-
- 04 2月, 2016 16 次提交
-
-
由 Aljoscha Krettek 提交于
-
由 Stephan Ewen 提交于
-
由 Robert Metzger 提交于
This closes #1584
-
由 Kostas Kloudas 提交于
This closes #1568
-
由 Greg Hogan 提交于
Parallelizes cluster start-up and tear-down when pdsh is available This closes #1523
-
由 Johannes 提交于
Added a method to set a default application name for the Flink Yarn session CLI. Switched the order, such that this name can now be overwritten by the command line. This closes #1558
-
由 Stephan Ewen 提交于
-
由 Stephan Ewen 提交于
-
由 Aljoscha Krettek 提交于
-
由 Stephan Ewen 提交于
-
由 Stephan Ewen 提交于
-
由 Aljoscha Krettek 提交于
-
由 Aljoscha Krettek 提交于
This changes window operator to use the new partitioned state abstraction for keeping window contents instead of custom internal state and the checkpointed interface. For now, timers are still kept as custom checkpointed state, however. WindowOperator now expects a StateIdentifier for MergingState, this can either be for ReducingState or ListState but WindowOperator is agnostic to the type of State. Also the signature of WindowFunction is changed to include the type of intermediate input. For example, if a ReducingState is used the input of the WindowFunction is T (where T is the input type). If using a ListState the input of the WindowFunction would be of type Iterable[T].
-
由 Aljoscha Krettek 提交于
Add new state types ValueState, ListState and ReducingState, where ListState and ReducingState derive from interface MergingState. ValueState behaves exactly the same as OperatorState. MergingState is a stateful list to which elements can be added and for which the elements that it contains can be obtained. If using a ListState the list of elements is actually kept, for a ReducingState a reduce function is used to combine all added elements into one. To create a ValueState the user passes a ValueStateIdentifier to StreamingRuntimeContext.getPartitionedState() while they would pass a ListStateIdentifier or ReducingStateIdentifier for the other state types. This change is necessary to give the system more information about the nature of the operator state. We want this to be able to do incremental snapshots. This would not be possible, for example, if the user had a List as a state. Inside OperatorState this list would be opaque and Flink could not create good incremental snapshots. This also refactors the StateBackend. Before, the logic for partitioned state was spread out over StreamingRuntimeContext, AbstractStreamOperator and StateBackend. Now it is consolidated in StateBackend. This also adds support for partitioned state in two-input operators.
-
由 Ufuk Celebi 提交于
This closes #1582, #1576.
-
由 Ufuk Celebi 提交于
Old guides are redirected to the new pages.
-
- 03 2月, 2016 8 次提交
-
-
由 Till Rohrmann 提交于
-
由 Maximilian Michels 提交于
-
由 Till Rohrmann 提交于
The LegacyFetcher used the given KeyedDeserializationSchema across multiple threads even though it is not thread-safe. This commit fixes the problem by cloning the KeyedDeserializationSchema before giving it to the SimpleConsumerThread. Add clone method for Java serializable objects to InstantiationUtil This closes #1577.
-
由 Aljoscha Krettek 提交于
-
由 Stephan Ewen 提交于
[FLINK-3314] [streaming] Fix case where early cancel messages do not properly cancel a stream operator.
-
由 Stephan Ewen 提交于
-
由 Stephan Ewen 提交于
-
由 Stephan Ewen 提交于
- Clean up generics - Clean and safe disposal of initialized resources - Add names to asynchronous materialization threads - Fix concurrent modification of materialization threads set
-
- 02 2月, 2016 1 次提交
-
-
由 Stephan Ewen 提交于
This deactivates the validation of results, which is not reliably possible under the current model (timeout on feedback). This test for now only checks that the job executes properly. Also adds proper logging property files for the examples projects.
-