Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
3d339de8
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
3d339de8
编写于
5月 13, 2020
作者:
A
Aljoscha Krettek
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-17661] Add Scala API for using new WatermarkStrategy/WatermarkGenerator
上级
e0b5d51c
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
35 addition
and
37 deletion
+35
-37
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala
...ala/org/apache/flink/streaming/api/scala/DataStream.scala
+35
-37
未找到文件。
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala
浏览文件 @
3d339de8
...
...
@@ -20,6 +20,7 @@ package org.apache.flink.streaming.api.scala
import
org.apache.flink.annotation.
{
Internal
,
Public
,
PublicEvolving
}
import
org.apache.flink.api.common.ExecutionConfig
import
org.apache.flink.api.common.eventtime.
{
TimestampAssigner
,
WatermarkGenerator
,
WatermarkStrategy
}
import
org.apache.flink.api.common.functions.
{
FilterFunction
,
FlatMapFunction
,
MapFunction
,
Partitioner
}
import
org.apache.flink.api.common.io.OutputFormat
import
org.apache.flink.api.common.operators.ResourceSpec
...
...
@@ -814,32 +815,40 @@ class DataStream[T](stream: JavaStream[T]) {
}
/**
* Assigns timestamps to the elements in the data stream and periodically creates
* watermarks to signal event time progress.
* Assigns timestamps to the elements in the data stream and generates watermarks to signal
* event time progress. The given [[WatermarkStrategy is used to create a [[TimestampAssigner]]
* and [[org.apache.flink.api.common.eventtime.WatermarkGenerator]].
*
* This method creates watermarks periodically (for example every second), based
* on the watermarks indicated by the given watermark generator. Even when no new elements
* in the stream arrive, the given watermark generator will be periodically checked for
* new watermarks. The interval in which watermarks are generated is defined in
* [[org.apache.flink.api.common.ExecutionConfig#setAutoWatermarkInterval(long)]].
* For each event in the data stream, the [[TimestampAssigner#extractTimestamp(Object, long)]]
* method is called to assign an event timestamp.
*
* Use this method for the common cases, where some characteristic over all elements
* should generate the watermarks, or where watermarks are simply trailing behind the
* wall clock time by a certain amount.
* For each event in the data stream, the
* [[WatermarkGenerator#onEvent(Object, long, WatermarkOutput)]] will be called.
*
* For the second case and when the watermarks are required to lag behind the maximum
* timestamp seen so far in the elements of the stream by a fixed amount of time, and this
* amount is known in advance, use the
* [[BoundedOutOfOrdernessTimestampExtractor]].
* Periodically (defined by the [[ExecutionConfig#getAutoWatermarkInterval()]]), the
* [[WatermarkGenerator#onPeriodicEmit(WatermarkOutput)]] method will be called.
*
* Common watermark generation patterns can be found in the
* [[org.apache.flink.api.common.eventtime.WatermarkStrategies]] class.
*/
def
assignTimestampsAndWatermarks
(
watermarkStrategy
:
WatermarkStrategy
[
T
])
:
DataStream
[
T
]
=
{
val
cleanedStrategy
=
clean
(
watermarkStrategy
)
asScalaStream
(
stream
.
assignTimestampsAndWatermarks
(
cleanedStrategy
))
}
/**
* Assigns timestamps to the elements in the data stream and periodically creates
* watermarks to signal event time progress.
*
* For cases where watermarks should be created in an irregular fashion, for example
* based on certain markers that some element carry, use the
* [[AssignerWithPunctuatedWatermarks]].
* This method uses the deprecated watermark generator interfaces. Please switch to
* [[assignTimestampsAndWatermarks(WatermarkStrategy]] to use the
* new interfaces instead. The new interfaces support watermark idleness and no longer need
* to differentiate between "periodic" and "punctuated" watermarks.
*
* @see AssignerWithPeriodicWatermarks
* @see AssignerWithPunctuatedWatermarks
* @see #assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)
* @deprecated please use [[assignTimestampsAndWatermarks()]]
*/
@deprecated
@PublicEvolving
def
assignTimestampsAndWatermarks
(
assigner
:
AssignerWithPeriodicWatermarks
[
T
])
:
DataStream
[
T
]
=
{
asScalaStream
(
stream
.
assignTimestampsAndWatermarks
(
assigner
))
...
...
@@ -849,25 +858,14 @@ class DataStream[T](stream: JavaStream[T]) {
* Assigns timestamps to the elements in the data stream and periodically creates
* watermarks to signal event time progress.
*
* This method creates watermarks based purely on stream elements. For each element
* that is handled via [[AssignerWithPunctuatedWatermarks#extractTimestamp(Object, long)]],
* the [[AssignerWithPunctuatedWatermarks#checkAndGetNextWatermark()]] method is called,
* and a new watermark is emitted, if the returned watermark value is larger than the previous
* watermark.
* This method uses the deprecated watermark generator interfaces. Please switch to
* [[assignTimestampsAndWatermarks(WatermarkStrategy]] to use the
* new interfaces instead. The new interfaces support watermark idleness and no longer need
* to differentiate between "periodic" and "punctuated" watermarks.
*
* This method is useful when the data stream embeds watermark elements, or certain elements
* carry a marker that can be used to determine the current event time watermark.
* This operation gives the programmer full control over the watermark generation. Users
* should be aware that too aggressive watermark generation (i.e., generating hundreds of
* watermarks every second) can cost some performance.
*
* For cases where watermarks should be created in a regular fashion, for example
* every x milliseconds, use the [[AssignerWithPeriodicWatermarks]].
*
* @see AssignerWithPunctuatedWatermarks
* @see AssignerWithPeriodicWatermarks
* @see #assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks)
* @deprecated please use [[assignTimestampsAndWatermarks()]]
*/
@deprecated
@PublicEvolving
def
assignTimestampsAndWatermarks
(
assigner
:
AssignerWithPunctuatedWatermarks
[
T
])
:
DataStream
[
T
]
=
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录