Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
35b4da27
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
35b4da27
编写于
6月 27, 2016
作者:
J
Jonas Traub
提交者:
Aljoscha Krettek
6月 27, 2016
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-4121] Add timeunit (ms) to docs for timestamps and watermarks
This closes #2165
上级
d34bdaf7
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
19 addition
and
18 deletion
+19
-18
docs/apis/streaming/event_timestamps_watermarks.md
docs/apis/streaming/event_timestamps_watermarks.md
+17
-16
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/watermark/Watermark.java
...a/org/apache/flink/streaming/api/watermark/Watermark.java
+2
-2
未找到文件。
docs/apis/streaming/event_timestamps_watermarks.md
浏览文件 @
35b4da27
...
@@ -48,14 +48,13 @@ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
...
@@ -48,14 +48,13 @@ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
</div>
</div>
</div>
</div>
## Assigning Timestamps
## Assigning Timestamps
In order to work with
*Event Time*
, Flink needs to know the events'
*timestamps*
, meaning each element in the
In order to work with
*Event Time*
, Flink needs to know the events'
*timestamps*
, meaning each element in the
stream needs to get its event timestamp
*assigned*
. That happens usually by accessing/extracting the
stream needs to get its event timestamp
*assigned*
. That happens usually by accessing/extracting the
timestamp from some field in the element.
timestamp from some field in the element.
Timestamp assignment goes hand-in-hand with generating watermarks, which tell the system about
Timestamp assignment goes hand-in-hand with generating watermarks, which tell the system about
the progress in event time.
the progress in event time.
There are two ways to assign timestamps and generate Watermarks:
There are two ways to assign timestamps and generate Watermarks:
...
@@ -63,6 +62,8 @@ There are two ways to assign timestamps and generate Watermarks:
...
@@ -63,6 +62,8 @@ There are two ways to assign timestamps and generate Watermarks:
1.
Directly in the data stream source
1.
Directly in the data stream source
2.
Via a timestamp assigner / watermark generator: in Flink timestamp assigners also define the watermarks to be emitted
2.
Via a timestamp assigner / watermark generator: in Flink timestamp assigners also define the watermarks to be emitted
<span
class=
"label label-danger"
>
Attention
</span>
Both timestamps and watermarks are specified as
millliseconds since the Java epoch of 1970-01-01T00:00:00Z.
### Source Functions with Timestamps and Watermarks
### Source Functions with Timestamps and Watermarks
...
@@ -116,18 +117,18 @@ those timestamps will be overwritten by the TimestampAssigner. Similarly, Waterm
...
@@ -116,18 +117,18 @@ those timestamps will be overwritten by the TimestampAssigner. Similarly, Waterm
Timestamp Assigners take a stream and produce a new stream with timestamped elements and watermarks. If the
Timestamp Assigners take a stream and produce a new stream with timestamped elements and watermarks. If the
original stream had timestamps and/or watermarks already, the timestamp assigner overwrites them.
original stream had timestamps and/or watermarks already, the timestamp assigner overwrites them.
The timestamp assigners usually are specified immediately after the data source but it is not strictly required to do so.
The timestamp assigners usually are specified immediately after the data source but it is not strictly required to do so.
A common pattern is, for example, to parse (
*MapFunction*
) and filter (
*FilterFunction*
) before the timestamp assigner.
A common pattern is, for example, to parse (
*MapFunction*
) and filter (
*FilterFunction*
) before the timestamp assigner.
In any case, the timestamp assigner needs to be specified before the first operation on event time
In any case, the timestamp assigner needs to be specified before the first operation on event time
(such as the first window operation). As a special case, when using Kafka as the source of a streaming job,
(such as the first window operation). As a special case, when using Kafka as the source of a streaming job,
Flink allows the specification of a timestamp assigner / watermark emitter inside
Flink allows the specification of a timestamp assigner / watermark emitter inside
the source (or consumer) itself. More information on how to do so can be found in the
the source (or consumer) itself. More information on how to do so can be found in the
[
Kafka Connector documentation
](
{{
site.baseurl }}/apis/streaming/connectors/kafka.html).
[
Kafka Connector documentation
](
{{
site.baseurl }}/apis/streaming/connectors/kafka.html).
**NOTE:**
The remainder of this section presents the main interfaces a programmer has
**NOTE:**
The remainder of this section presents the main interfaces a programmer has
to implement in order to create her own timestamp extractors/watermark emitters.
to implement in order to create her own timestamp extractors/watermark emitters.
To see the pre-implemented extractors that ship with Flink, please refer to the
To see the pre-implemented extractors that ship with Flink, please refer to the
[
Pre-defined Timestamp Extractors / Watermark Emitters
](
{{
site.baseurl }}/apis/streaming/event_timestamp_extractors.html) page.
[
Pre-defined Timestamp Extractors / Watermark Emitters
](
{{
site.baseurl }}/apis/streaming/event_timestamp_extractors.html) page.
<div
class=
"codetabs"
markdown=
"1"
>
<div
class=
"codetabs"
markdown=
"1"
>
...
@@ -137,7 +138,7 @@ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEn
...
@@ -137,7 +138,7 @@ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEn
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
DataStream
<MyEvent>
stream = env.readFile(
DataStream
<MyEvent>
stream = env.readFile(
myFormat, myFilePath, FileProcessingMode.PROCESS_CONTINUOUSLY, 100,
myFormat, myFilePath, FileProcessingMode.PROCESS_CONTINUOUSLY, 100,
FilePathFilter.createDefaultFilter(), typeInfo);
FilePathFilter.createDefaultFilter(), typeInfo);
DataStream
<MyEvent>
withTimestampsAndWatermarks = stream
DataStream
<MyEvent>
withTimestampsAndWatermarks = stream
...
@@ -157,7 +158,7 @@ val env = StreamExecutionEnvironment.getExecutionEnvironment
...
@@ -157,7 +158,7 @@ val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val stream: DataStream[MyEvent] = env.readFile(
val stream: DataStream[MyEvent] = env.readFile(
myFormat, myFilePath, FileProcessingMode.PROCESS_CONTINUOUSLY, 100,
myFormat, myFilePath, FileProcessingMode.PROCESS_CONTINUOUSLY, 100,
FilePathFilter.createDefaultFilter());
FilePathFilter.createDefaultFilter());
val withTimestampsAndWatermarks: DataStream[MyEvent] = stream
val withTimestampsAndWatermarks: DataStream[MyEvent] = stream
...
@@ -176,7 +177,7 @@ withTimestampsAndWatermarks
...
@@ -176,7 +177,7 @@ withTimestampsAndWatermarks
#### **With Periodic Watermarks**
#### **With Periodic Watermarks**
The
`AssignerWithPeriodicWatermarks`
assigns timestamps and generates watermarks periodically (possibly depending
The
`AssignerWithPeriodicWatermarks`
assigns timestamps and generates watermarks periodically (possibly depending
on the stream elements, or purely based on processing time).
on the stream elements, or purely based on processing time).
The interval (every
*n*
milliseconds) in which the watermark will be generated is defined via
The interval (every
*n*
milliseconds) in which the watermark will be generated is defined via
...
@@ -202,7 +203,7 @@ public class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermar
...
@@ -202,7 +203,7 @@ public class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermar
@Override
@Override
public long extractTimestamp(MyEvent element, long previousElementTimestamp) {
public long extractTimestamp(MyEvent element, long previousElementTimestamp) {
long timestamp = element.getCreationTime();
long timestamp = element.getCreationTime();
currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
return timestamp;
return timestamp;
}
}
...
@@ -229,7 +230,7 @@ public class TimeLagWatermarkGenerator extends AssignerWithPeriodicWatermarks<My
...
@@ -229,7 +230,7 @@ public class TimeLagWatermarkGenerator extends AssignerWithPeriodicWatermarks<My
@Override
@Override
public Watermark getCurrentWatermark() {
public Watermark getCurrentWatermark() {
// return the watermark as current time minus the maximum time lag
// return the watermark as current time minus the maximum time lag
return new Watermark(System.currentTimeMillis() - maxTimeLag);
return new Watermark(System.currentTimeMillis() - maxTimeLag);
}
}
}
}
...
@@ -249,7 +250,7 @@ class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks[MyEv
...
@@ -249,7 +250,7 @@ class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks[MyEv
var currentMaxTimestamp: Long;
var currentMaxTimestamp: Long;
override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = {
override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = {
val timestamp = element.getCreationTime()
val timestamp = element.getCreationTime()
currentMaxTimestamp = max(timestamp, currentMaxTimestamp)
currentMaxTimestamp = max(timestamp, currentMaxTimestamp)
timestamp;
timestamp;
}
}
...
@@ -273,7 +274,7 @@ class TimeLagWatermarkGenerator extends AssignerWithPeriodicWatermarks[MyEvent]
...
@@ -273,7 +274,7 @@ class TimeLagWatermarkGenerator extends AssignerWithPeriodicWatermarks[MyEvent]
}
}
override def getCurrentWatermark(): Watermark = {
override def getCurrentWatermark(): Watermark = {
// return the watermark as current time minus the maximum time lag
// return the watermark as current time minus the maximum time lag
new Watermark(System.currentTimeMillis() - maxTimeLag)
new Watermark(System.currentTimeMillis() - maxTimeLag)
}
}
}
}
...
...
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/watermark/Watermark.java
浏览文件 @
35b4da27
...
@@ -48,11 +48,11 @@ public final class Watermark extends StreamElement {
...
@@ -48,11 +48,11 @@ public final class Watermark extends StreamElement {
// ------------------------------------------------------------------------
// ------------------------------------------------------------------------
/** The timestamp of the watermark */
/** The timestamp of the watermark
in milliseconds
*/
private
final
long
timestamp
;
private
final
long
timestamp
;
/**
/**
* Creates a new watermark with the given timestamp.
* Creates a new watermark with the given timestamp
in milliseconds
.
*/
*/
public
Watermark
(
long
timestamp
)
{
public
Watermark
(
long
timestamp
)
{
this
.
timestamp
=
timestamp
;
this
.
timestamp
=
timestamp
;
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录