diff --git a/docs/streaming_guide.md b/docs/streaming_guide.md index 2b7f36097571f6b1fd49de03930fc15218267825..3a9a1c457d35cd555c3aac74c181660069e53390 100644 --- a/docs/streaming_guide.md +++ b/docs/streaming_guide.md @@ -30,7 +30,7 @@ Introduction ------------ -Flink Streaming is an extension of the batch Flink API for high-throughput, low-latency data stream processing. The system can connect to and process data streams from many data sources like Apache Kafka, RabbitMQ, Apache Flume, Twitter and also from any user defined data source. Data streams can be transformed and modified using high-level functions similar to the ones provided by the batch processing API. Flink Streaming provides native support for iterative stream processing. The processed data can be pushed to different output types. +Flink Streaming is a system for high-throughput, low-latency data stream processing. The system can connect to and process data streams from different data sources like file sources, web sockets, message queues(Apache Kafka, RabbitMQ, Apache Flume, Twitter…) and also from any user defined data source using a very simple interface. Data streams can be transformed and modified to create new data streams using high-level functions similar to the ones provided by the batch processing API. Flink Streaming natively supports flexible, data-driven windowing semantics and iterative stream processing. The processed data can be pushed to different output types. Flink Streaming API ----------- @@ -70,13 +70,13 @@ Add the following dependency to your `pom.xml` to use the Flink Streaming. -Create a data stream flow with our Java or Scala API as described below. In order to create your own Flink Streaming program we encourage you to start with the [skeleton](#program-skeleton) and gradually add your own [transformations](#transformations). The remaining sections act as references for additional transformations and advanced features. +In order to create your own Flink Streaming program we encourage you to start with the [skeleton](#program-skeleton) and gradually add your own [transformations](#transformations). The remaining sections act as references for additional transformations and advanced features. Example Program --------------- -The following program is a complete, working example of streaming WordCount. You can copy & paste the code to run it locally. +The following program is a complete, working example of streaming WordCount that incrementally counts the words coming from a web socket. You can copy & paste the code to run it locally.
Evaluates a boolean function for each element and retains those for which the function returns true.
-
-
- IMPORTANT: The system assumes that the function does not modify the elements on which the predicate is applied. Violating this assumption
- can lead to incorrect results.
-
-
+
+
A filter that filters out zero values:
Combines a group of elements into a single element by repeatedly combining two elements +
Combines a stream of elements into another stream by repeatedly combining two elements
into one and emits the current state after every reduction. Reduce may be applied on a full, windowed or grouped data stream.
- IMPORTANT: The streaming and the batch reduce functions have different semantics. A streaming reduce on a full or grouped data stream emits the current reduced value for every new element on a data stream. On a windowed data stream it works as a batch reduce: it produces at most one value.
+ IMPORTANT: The streaming and the batch reduce functions have different semantics. A streaming reduce on a data stream emits the current reduced value for every new element on a data stream. On a windowed data stream it works as a batch reduce: it produces at most one value per window.
-
- A reducer that sums up the incoming stream:
Merges two or more datastreams creating a new stream containing all the elements from all the streams.
+Merges two or more data streams creating a new stream containing all the elements from all the streams.
{% highlight java %} dataStream.merge(otherStream1, otherStream2, …) {% endhighlight %} @@ -424,7 +447,7 @@ dataStream.merge(otherStream1, otherStream2, …) ---------- -The following transformations are available on data sets of Tuples: +The following transformations are available on data streams of Tuples:
Selects a subset of fields from the tuples {% highlight java %} -DataSet |
@@ -489,12 +512,8 @@ data.flatMap { str => str.split(" ") }
Filter |
Evaluates a boolean function for each element and retains those for which the function returns true.
- - IMPORTANT: The streaming and the batch reduce functions have different semantics. A streaming reduce on a full or grouped data stream emits the current reduced value for every new element on a data stream. On a windowed data stream it works as a batch reduce: it produces at most one value. + IMPORTANT: The streaming and the batch reduce functions have different semantics. A streaming reduce on a data stream emits the current reduced value for every new element on a data stream. On a windowed data stream it works as a batch reduce: it produces at most one value per window. - - A reducer that sums up the incoming stream: + + A reducer that sums up the incoming stream, the result is a stream of intermediate sums: {% highlight scala %} dataStream.reduce{ _ + _ } {% endhighlight %} @@ -523,7 +542,7 @@ dataStream.reduce{ _ + _ } |
Merge |
- Merges two or more datastreams creating a new stream containing all the elements from all the streams. +Merges two or more data streams creating a new stream containing all the elements from all the streams. {% highlight scala %} dataStream.merge(otherStream1, otherStream2, …) {% endhighlight %} @@ -544,9 +563,11 @@ dataStream.merge(otherStream1, otherStream2, …) Some transformations require that the elements of a `DataStream` are grouped on some key. The user can create a `GroupedDataStream` by calling the `groupBy(key)` method of a non-grouped `DataStream`. Keys can be of three types: fields positions (applicable for tuple/array types), field expressions (applicable for pojo types), KeySelector instances. +Aggregation or reduce operators called on `GroupedDataStream`s produce elements on a per group basis. + ### Aggregations -The Flink Streaming API supports different types of pre-defined aggregation operators similarly to the batch API. +The Flink Streaming API supports different types of pre-defined aggregations `DataStreams`. The common property of these operators, just like reduce on streams, they produce the stream of intermediate aggregate values. Types of aggregations: `sum(field)`, `min(field)`, `max(field)`, `minBy(field, first)`, `maxBy(field, first)`. @@ -558,22 +579,26 @@ There is also an option to apply user defined aggregations with the usage of the ### Window operators -Flink streaming provides very flexible windowing semantics to create arbitrary windows (also referred to as discretizations or slices) of the data streams and apply reduce, map or aggregation transformations on the windows acquired. Windowing can be used for instance to create rolling aggregations of the most recent N elements, where N could be defined by Time, Count or any arbitrary user defined measure. +Flink streaming provides very flexible data-driven windowing semantics to create arbitrary windows (also referred to as discretizations or slices) of the data streams and apply reduce, map or aggregation transformations on the windows acquired. Windowing can be used for instance to create rolling aggregations of the most recent N elements, where N could be defined by Time, Count or any arbitrary user defined measure. -The user can control the size (eviction) of the windows and the frequency of reduction or aggregation calls (trigger) on them in an intuitive API (some examples): +The user can control the size (eviction) of the windows and the frequency of transformation or aggregation calls (trigger) on them in an intuitive API. We will describe the exact semantics of these operators in the [policy based windowing](#policy-based-windowing) section. + +Some examples: * `dataStream.window(eviction).every(trigger).reduceWindow(…)` * `dataStream.window(…).every(…).mapWindow(…).flatten()` * `dataStream.window(…).every(…).groupBy(…).aggregate(…).getDiscretizedStream()` -The core abstraction of the Windowing semantics is the `WindowedDataStream` and the `StreamWindow`. The `WindowedDataStream` is created when we call the `window(…)` method of the DataStream and represents the windowed discretisation of the underlying stream. The user can think about it simply as a `DataStream
@@ -642,7 +669,7 @@ dataStream.window(triggerPolicy, evictionPolicy)
-By default most triggers can only trigger when a new element arrives. This might not be suitable for all the use-cases, especially when time based windowing is applied. To also provide triggering between elements so called active policies (the two interfaces controlling this special behaviour is ActiveTriggerPolicy and CentralActiveTrigger) can be used. The predefined time-based policies are already implemented in such a way and can hold as an example for user defined active policy implementations.
+By default triggers can only trigger when a new element arrives. This might not be suitable for all the use-cases with low data rates . To also provide triggering between elements so called active policies (the two interfaces controlling this special behaviour is `ActiveTriggerPolicy` and `CentralActiveTrigger`) can be used. The predefined time-based policies are already implemented in such a way and can hold as an example for user defined active policy implementations.
Time-based trigger and eviction policies can work with user defined `TimeStamp` implementations, these policies already cover most use cases.
@@ -686,7 +713,7 @@ windowedDataStream.mapWindow(windowMapFunction)
#### Grouped transformations on windowed data streams
-Calling the `groupBy(…)` method on a windowed stream groups the elements by the given fields inside the stream windows. The window sizes (evictions) and slide sizes (triggers) will be calculated on the whole stream (in a global fashion), but the user defined functions will be applied on a per group basis. This means that for a call `windowedStream.groupBy(…).reduceWindow(…)` will transform each window into another window consisting of as many elements as groups, with the reduced values per key. Similarly the `mapWindow` transformation is applied per group as well.
+Calling the `groupBy(…)` method on a windowed stream groups the elements by the given fields inside the stream windows. The window sizes (evictions) and slide sizes (triggers) will be calculated on the whole stream (in a global fashion), but the user defined functions will be applied on a per group basis inside the window. This means that for a call `windowedStream.groupBy(…).reduceWindow(…)` will transform each window into another window consisting of as many elements as keys in the original window, with the reduced values per key. Similarly the `mapWindow` transformation is applied per group as well.
The user can also create discretisation on a per group basis calling `window(…).every(…)` on an already grouped data stream. This will apply the discretisation logic independently for each key.
@@ -761,11 +788,13 @@ For example `dataStream.window(Count.of(100)).maxBy(field)` would create global
### Temporal database style operators
-While database style operators like joins (on key) and crosses are hard to define properly on data streams, a straightforward implementation is to apply these operators on windows of the data streams.
+While database style operators like joins (on key) and crosses are hard to define properly on data streams, a straightforward interpretation is to apply these operators on windows of the data streams.
+
+Currently join and cross operators are supported only on time windows. We are working on alleviating this limitations in the next release.
-Currently join and cross operators are supported on time windows.
+Temporal operators take the current windows of both streams and apply the join/cross logic on these window pairs.
-The Join transformation produces a new Tuple DataStream with two fields. Each tuple holds a joined element of the first input DataStream in the first tuple field and a matching element of the second input DataStream in the second field for the current window.
+The Join transformation produces a new Tuple DataStream with two fields. Each tuple holds a joined element of the first input DataStream in the first tuple field and a matching element of the second input DataStream in the second field for the current window. The user can also supply a custom join function to control the produced elements.
The following code shows a default Join transformation using field position keys:
@@ -788,7 +817,7 @@ dataStream1.join(dataStream2)
-The Cross transformation combines two DataStreams into one DataStream. It builds all pairwise combinations of the elements of both input DataStreams in the current window, i.e., it builds a temporal Cartesian product.
+The Cross transformation combines two `DataStream`s into one `DataStream`. It builds all pairwise combinations of the elements of both input DataStreams in the current window, i.e., it builds a temporal Cartesian product. The user can also supply a custom cross function to control the produced elements
@@ -806,8 +835,8 @@ dataStream1 cross dataStream2 onWindow (windowing_params)
### Co operators
-Co operators allow the users to jointly transform two DataStreams of different types providing a simple way to jointly manipulate a shared state. It is designed to support joint stream transformations where merging is not appropriate due to different data types or in case the user needs explicit tracking of the joined stream origin.
-Co operators can be applied to ConnectedDataStreams which represent two DataStreams of possibly different types. A ConnectedDataStream can be created by calling the `connect(otherDataStream)` method of a DataStream. Please note that the two connected DataStreams can also be merged data streams.
+Co operators allow the users to jointly transform two `DataStream`s of different types providing a simple way to jointly manipulate streams with a shared state. It is designed to support joint stream transformations where merging is not appropriate due to different data types or in case the user needs explicit tracking of the origin of individual elements.
+Co operators can be applied to `ConnectedDataStream`s which represent two `DataStream`s of possibly different types. A `ConnectedDataStream` can be created by calling the `connect(otherDataStream)` method of a `DataStream`. Please note that the two connected `DataStream`s can also be merged data streams.
#### Map on ConnectedDataStream
Applies a CoMap transformation on two separate DataStreams, mapping them to a common output type. The transformation calls a `CoMapFunction.map1()` for each element of the first input and `CoMapFunction.map2()` for each element of the second input. Each CoMapFunction call returns exactly one element.
@@ -936,6 +965,8 @@ Iterable
@@ -961,6 +992,8 @@ Data streams only receive the elements directed to selected output names. The us
The outputs of an operator are directed by implementing a function that returns the output names for the value. The data is sent to all the outputs returned by the function (referenced by their name). This way the direction of the outputs can be determined by the value of the data sent.
Every output will be emitted to the selected outputs exactly once, even if you add the same output names more than once.
+
+The functionality provided by output splitting can also be achieved efficiently (due to operator chaining) by multiple filter operators.
|