Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
c151a537
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
c151a537
编写于
9月 19, 2017
作者:
B
Bowen Li
提交者:
Aljoscha Krettek
9月 25, 2017
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-7635] Support side output in ProcessWindowFunction
上级
68a99d7a
变更
10
隐藏空白更改
内联
并排
Showing
10 changed file
with
124 addition
and
7 deletion
+124
-7
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/InternalProcessApplyWindowContext.java
...unctions/windowing/InternalProcessApplyWindowContext.java
+6
-0
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/ProcessWindowFunction.java
...eaming/api/functions/windowing/ProcessWindowFunction.java
+9
-0
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/ProcessOperator.java
...apache/flink/streaming/api/operators/ProcessOperator.java
+1
-4
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
...streaming/runtime/operators/windowing/WindowOperator.java
+7
-0
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/functions/InternalProcessWindowContext.java
...ors/windowing/functions/InternalProcessWindowContext.java
+6
-0
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/functions/InternalWindowFunction.java
...operators/windowing/functions/InternalWindowFunction.java
+3
-0
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/ProcessWindowFunction.scala
.../streaming/api/scala/function/ProcessWindowFunction.scala
+6
-3
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/util/ScalaProcessWindowFunctionWrapper.scala
...ala/function/util/ScalaProcessWindowFunctionWrapper.scala
+5
-0
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/SideOutputITCase.scala
...g/apache/flink/streaming/api/scala/SideOutputITCase.scala
+46
-0
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/SideOutputITCase.java
...apache/flink/test/streaming/runtime/SideOutputITCase.java
+35
-0
未找到文件。
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/InternalProcessApplyWindowContext.java
浏览文件 @
c151a537
...
...
@@ -21,6 +21,7 @@ package org.apache.flink.streaming.api.functions.windowing;
import
org.apache.flink.annotation.Internal
;
import
org.apache.flink.api.common.state.KeyedStateStore
;
import
org.apache.flink.streaming.api.windowing.windows.Window
;
import
org.apache.flink.util.OutputTag
;
/**
* Internal reusable context wrapper.
...
...
@@ -64,4 +65,9 @@ public class InternalProcessApplyWindowContext<IN, OUT, KEY, W extends Window>
public
KeyedStateStore
globalState
()
{
return
context
.
globalState
();
}
@Override
public
<
X
>
void
output
(
OutputTag
<
X
>
outputTag
,
X
value
)
{
context
.
output
(
outputTag
,
value
);
}
}
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/ProcessWindowFunction.java
浏览文件 @
c151a537
...
...
@@ -23,6 +23,7 @@ import org.apache.flink.api.common.functions.AbstractRichFunction;
import
org.apache.flink.api.common.state.KeyedStateStore
;
import
org.apache.flink.streaming.api.windowing.windows.Window
;
import
org.apache.flink.util.Collector
;
import
org.apache.flink.util.OutputTag
;
/**
* Base abstract class for functions that are evaluated over keyed (grouped) windows using a context
...
...
@@ -85,5 +86,13 @@ public abstract class ProcessWindowFunction<IN, OUT, KEY, W extends Window> exte
* State accessor for per-key global state.
*/
public
abstract
KeyedStateStore
globalState
();
/**
* Emits a record to the side output identified by the {@link OutputTag}.
*
* @param outputTag the {@code OutputTag} that identifies the side output to emit to.
* @param value The record to emit.
*/
public
abstract
<
X
>
void
output
(
OutputTag
<
X
>
outputTag
,
X
value
);
}
}
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/ProcessOperator.java
浏览文件 @
c151a537
...
...
@@ -73,10 +73,7 @@ public class ProcessOperator<IN, OUT>
this
.
currentWatermark
=
mark
.
getTimestamp
();
}
private
class
ContextImpl
extends
ProcessFunction
<
IN
,
OUT
>.
Context
implements
TimerService
{
private
class
ContextImpl
extends
ProcessFunction
<
IN
,
OUT
>.
Context
implements
TimerService
{
private
StreamRecord
<
IN
>
element
;
private
final
ProcessingTimeService
processingTimeService
;
...
...
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
浏览文件 @
c151a537
...
...
@@ -774,6 +774,13 @@ public class WindowOperator<K, IN, ACC, OUT, W extends Window>
public
KeyedStateStore
globalState
()
{
return
WindowOperator
.
this
.
getKeyedStateStore
();
}
public
<
X
>
void
output
(
OutputTag
<
X
>
outputTag
,
X
value
)
{
if
(
outputTag
==
null
)
{
throw
new
IllegalArgumentException
(
"OutputTag must not be null."
);
}
output
.
collect
(
outputTag
,
new
StreamRecord
<>(
value
,
window
.
maxTimestamp
()));
}
}
/**
...
...
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/functions/InternalProcessWindowContext.java
浏览文件 @
c151a537
...
...
@@ -22,6 +22,7 @@ import org.apache.flink.annotation.Internal;
import
org.apache.flink.api.common.state.KeyedStateStore
;
import
org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction
;
import
org.apache.flink.streaming.api.windowing.windows.Window
;
import
org.apache.flink.util.OutputTag
;
/**
* Internal reusable context wrapper.
...
...
@@ -66,4 +67,9 @@ public class InternalProcessWindowContext<IN, OUT, KEY, W extends Window>
public
KeyedStateStore
globalState
()
{
return
internalContext
.
globalState
();
}
@Override
public
<
X
>
void
output
(
OutputTag
<
X
>
outputTag
,
X
value
)
{
internalContext
.
output
(
outputTag
,
value
);
}
}
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/functions/InternalWindowFunction.java
浏览文件 @
c151a537
...
...
@@ -22,6 +22,7 @@ import org.apache.flink.api.common.functions.Function;
import
org.apache.flink.api.common.state.KeyedStateStore
;
import
org.apache.flink.streaming.api.windowing.windows.Window
;
import
org.apache.flink.util.Collector
;
import
org.apache.flink.util.OutputTag
;
/**
* Internal interface for functions that are evaluated over keyed (grouped) windows.
...
...
@@ -63,5 +64,7 @@ public interface InternalWindowFunction<IN, OUT, KEY, W extends Window> extends
KeyedStateStore
windowState
();
KeyedStateStore
globalState
();
<
X
>
void
output
(
OutputTag
<
X
>
outputTag
,
X
value
);
}
}
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/ProcessWindowFunction.scala
浏览文件 @
c151a537
...
...
@@ -18,11 +18,10 @@
package
org.apache.flink.streaming.api.scala.function
import
java.io.Serializable
import
org.apache.flink.annotation.PublicEvolving
import
org.apache.flink.api.common.functions.AbstractRichFunction
import
org.apache.flink.api.common.state.KeyedStateStore
import
org.apache.flink.streaming.api.scala.OutputTag
import
org.apache.flink.streaming.api.windowing.windows.Window
import
org.apache.flink.util.Collector
...
...
@@ -88,6 +87,10 @@ abstract class ProcessWindowFunction[IN, OUT, KEY, W <: Window]
* State accessor for per-key global state.
*/
def
globalState
:
KeyedStateStore
}
/**
* Emits a record to the side output identified by the [[OutputTag]].
*/
def
output
[
X
](
outputTag
:
OutputTag
[
X
],
value
:
X
);
}
}
flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/util/ScalaProcessWindowFunctionWrapper.scala
浏览文件 @
c151a537
...
...
@@ -22,6 +22,7 @@ import org.apache.flink.api.common.functions.RuntimeContext
import
org.apache.flink.configuration.Configuration
import
org.apache.flink.streaming.api.functions.windowing.
{
ProcessWindowFunction
=>
JProcessWindowFunction
}
import
org.apache.flink.streaming.api.functions.windowing.
{
ProcessAllWindowFunction
=>
JProcessAllWindowFunction
}
import
org.apache.flink.streaming.api.scala.OutputTag
import
org.apache.flink.streaming.api.scala.function.
{
ProcessWindowFunction
=>
ScalaProcessWindowFunction
}
import
org.apache.flink.streaming.api.scala.function.
{
ProcessAllWindowFunction
=>
ScalaProcessAllWindowFunction
}
import
org.apache.flink.streaming.api.windowing.windows.Window
...
...
@@ -56,6 +57,8 @@ final class ScalaProcessWindowFunctionWrapper[IN, OUT, KEY, W <: Window](
override
def
windowState
=
context
.
windowState
()
override
def
globalState
=
context
.
globalState
()
override
def
output
[
X
](
outputTag
:
OutputTag
[
X
],
value
:
X
)
=
context
.
output
(
outputTag
,
value
)
}
func
.
process
(
key
,
ctx
,
elements
.
asScala
,
out
)
}
...
...
@@ -71,6 +74,8 @@ final class ScalaProcessWindowFunctionWrapper[IN, OUT, KEY, W <: Window](
override
def
windowState
=
context
.
windowState
()
override
def
globalState
=
context
.
globalState
()
override
def
output
[
X
](
outputTag
:
OutputTag
[
X
],
value
:
X
)
=
context
.
output
(
outputTag
,
value
)
}
func
.
clear
(
ctx
)
}
...
...
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/SideOutputITCase.scala
浏览文件 @
c151a537
...
...
@@ -234,6 +234,52 @@ class SideOutputITCase extends StreamingMultipleProgramsTestBase {
assertEquals
(
util
.
Arrays
.
asList
((
"3"
,
3
),
(
"4"
,
4
)),
lateResultSink
.
getResult
)
}
/**
* Test ProcessWindowFunction side output.
*/
@Test
def
testProcessWindowFunctionSideOutput
()
{
val
resultSink
=
new
TestListResultSink
[
String
]
val
sideOutputResultSink
=
new
TestListResultSink
[
String
]
val
env
=
StreamExecutionEnvironment
.
getExecutionEnvironment
env
.
setParallelism
(
1
)
env
.
setStreamTimeCharacteristic
(
TimeCharacteristic
.
EventTime
)
val
dataStream
=
env
.
fromElements
((
"1"
,
1
),
(
"2"
,
2
),
(
"5"
,
5
),
(
"3"
,
3
),
(
"4"
,
4
))
val
sideOutputTag
=
OutputTag
[
String
](
"side"
)
val
windowOperator
=
dataStream
.
assignTimestampsAndWatermarks
(
new
TestAssigner
)
.
keyBy
(
i
=>
i
.
_1
)
.
window
(
TumblingEventTimeWindows
.
of
(
Time
.
milliseconds
(
1
)))
.
process
(
new
ProcessWindowFunction
[(
String
,
Int
)
,
String
,
String
,
TimeWindow
]
{
override
def
process
(
key
:
String
,
context
:
Context
,
elements
:
Iterable
[(
String
,
Int
)],
out
:
Collector
[
String
])
:
Unit
=
{
for
(
in
<-
elements
)
{
out
.
collect
(
in
.
_1
)
context
.
output
(
sideOutputTag
,
"sideout-"
+
in
.
_1
)
}
}
})
windowOperator
.
getSideOutput
(
sideOutputTag
)
.
addSink
(
sideOutputResultSink
)
windowOperator
.
addSink
(
resultSink
)
env
.
execute
()
assertEquals
(
util
.
Arrays
.
asList
(
"1"
,
"2"
,
"5"
),
resultSink
.
getResult
)
assertEquals
(
util
.
Arrays
.
asList
(
"sideout-1"
,
"sideout-2"
,
"sideout-5"
),
sideOutputResultSink
.
getResult
)
}
}
class
TestAssigner
extends
AssignerWithPunctuatedWatermarks
[(
String
,
Int
)]
{
...
...
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/SideOutputITCase.java
浏览文件 @
c151a537
...
...
@@ -29,6 +29,7 @@ import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks
import
org.apache.flink.streaming.api.functions.ProcessFunction
;
import
org.apache.flink.streaming.api.functions.source.SourceFunction
;
import
org.apache.flink.streaming.api.functions.windowing.AllWindowFunction
;
import
org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction
;
import
org.apache.flink.streaming.api.functions.windowing.WindowFunction
;
import
org.apache.flink.streaming.api.operators.AbstractStreamOperator
;
import
org.apache.flink.streaming.api.operators.OneInputStreamOperator
;
...
...
@@ -547,4 +548,38 @@ public class SideOutputITCase extends StreamingMultipleProgramsTestBase implemen
assertEquals
(
Collections
.
singletonList
(
3
),
lateResultSink
.
getSortedResult
());
}
@Test
public
void
testProcessdWindowFunctionSideOutput
()
throws
Exception
{
TestListResultSink
<
Integer
>
resultSink
=
new
TestListResultSink
<>();
TestListResultSink
<
String
>
sideOutputResultSink
=
new
TestListResultSink
<>();
StreamExecutionEnvironment
see
=
StreamExecutionEnvironment
.
getExecutionEnvironment
();
see
.
setParallelism
(
3
);
see
.
setStreamTimeCharacteristic
(
TimeCharacteristic
.
EventTime
);
DataStream
<
Integer
>
dataStream
=
see
.
fromCollection
(
elements
);
OutputTag
<
String
>
sideOutputTag
=
new
OutputTag
<
String
>(
"side"
){};
SingleOutputStreamOperator
<
Integer
>
windowOperator
=
dataStream
.
assignTimestampsAndWatermarks
(
new
TestWatermarkAssigner
())
.
keyBy
(
new
TestKeySelector
())
.
timeWindow
(
Time
.
milliseconds
(
1
),
Time
.
milliseconds
(
1
))
.
process
(
new
ProcessWindowFunction
<
Integer
,
Integer
,
Integer
,
TimeWindow
>()
{
private
static
final
long
serialVersionUID
=
1L
;
@Override
public
void
process
(
Integer
integer
,
Context
context
,
Iterable
<
Integer
>
elements
,
Collector
<
Integer
>
out
)
throws
Exception
{
out
.
collect
(
integer
);
context
.
output
(
sideOutputTag
,
"sideout-"
+
String
.
valueOf
(
integer
));
}
});
windowOperator
.
getSideOutput
(
sideOutputTag
).
addSink
(
sideOutputResultSink
);
windowOperator
.
addSink
(
resultSink
);
see
.
execute
();
assertEquals
(
Arrays
.
asList
(
"sideout-1"
,
"sideout-2"
,
"sideout-5"
),
sideOutputResultSink
.
getSortedResult
());
assertEquals
(
Arrays
.
asList
(
1
,
2
,
5
),
resultSink
.
getSortedResult
());
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录