Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
ce7a8996
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
ce7a8996
编写于
7月 14, 2014
作者:
G
gyfora
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] addSink added to newapi
上级
ce56b146
变更
8
显示空白变更内容
内联
并排
Showing
8 changed file
with
111 addition
and
32 deletion
+111
-32
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/DataStream.java
.../main/java/eu/stratosphere/api/datastream/DataStream.java
+5
-1
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SinkFunction.java
...ain/java/eu/stratosphere/api/datastream/SinkFunction.java
+13
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SinkInvokable.java
...in/java/eu/stratosphere/api/datastream/SinkInvokable.java
+41
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamExecutionEnvironment.java
...ratosphere/api/datastream/StreamExecutionEnvironment.java
+28
-22
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/invokable/UserSinkInvokable.java
...ratosphere/streaming/api/invokable/UserSinkInvokable.java
+1
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamComponentHelper.java
.../streaming/api/streamcomponent/StreamComponentHelper.java
+3
-2
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/FlatMapTest.java
.../test/java/eu/stratosphere/streaming/api/FlatMapTest.java
+15
-4
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/MapTest.java
.../src/test/java/eu/stratosphere/streaming/api/MapTest.java
+5
-3
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/DataStream.java
浏览文件 @
ce7a8996
...
@@ -79,11 +79,15 @@ public class DataStream<T extends Tuple> {
...
@@ -79,11 +79,15 @@ public class DataStream<T extends Tuple> {
return
context
.
addFlatMapFunction
(
this
,
flatMapper
);
return
context
.
addFlatMapFunction
(
this
,
flatMapper
);
}
}
public
DataStream
<
T
>
addSink
(
SinkFunction
<
T
>
sinkFunction
)
{
return
context
.
addSink
(
this
,
sinkFunction
);
}
public
<
R
extends
Tuple
>
DataStream
<
R
>
map
(
MapFunction
<
T
,
R
>
mapper
)
{
public
<
R
extends
Tuple
>
DataStream
<
R
>
map
(
MapFunction
<
T
,
R
>
mapper
)
{
return
context
.
addMapFunction
(
this
,
mapper
);
return
context
.
addMapFunction
(
this
,
mapper
);
}
}
public
<
R
extends
Tuple
>
DataStream
<
R
>
addDummySink
()
{
public
DataStream
<
T
>
addDummySink
()
{
return
context
.
addDummySink
(
this
);
return
context
.
addDummySink
(
this
);
}
}
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SinkFunction.java
0 → 100755
浏览文件 @
ce7a8996
package
eu.stratosphere.api.datastream
;
import
java.io.Serializable
;
import
eu.stratosphere.api.java.tuple.Tuple
;
public
abstract
class
SinkFunction
<
IN
extends
Tuple
>
implements
Serializable
{
private
static
final
long
serialVersionUID
=
1L
;
public
abstract
void
invoke
(
IN
tuple
);
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SinkInvokable.java
0 → 100755
浏览文件 @
ce7a8996
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.api.datastream
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.streaming.api.StreamCollector
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
SinkInvokable
<
IN
extends
Tuple
>
extends
UserSinkInvokable
<
IN
>
{
private
static
final
long
serialVersionUID
=
1L
;
private
SinkFunction
<
IN
>
sinkFunction
;
public
SinkInvokable
(
SinkFunction
<
IN
>
sinkFunction
)
{
this
.
sinkFunction
=
sinkFunction
;
}
@Override
public
void
invoke
(
StreamRecord
record
,
StreamCollector
<
Tuple
>
collector
)
throws
Exception
{
int
batchSize
=
record
.
getBatchSize
();
for
(
int
i
=
0
;
i
<
batchSize
;
i
++)
{
@SuppressWarnings
(
"unchecked"
)
IN
tuple
=
(
IN
)
record
.
getTuple
(
i
);
sinkFunction
.
invoke
(
tuple
);
}
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamExecutionEnvironment.java
浏览文件 @
ce7a8996
...
@@ -101,6 +101,28 @@ public class StreamExecutionEnvironment {
...
@@ -101,6 +101,28 @@ public class StreamExecutionEnvironment {
return
returnStream
;
return
returnStream
;
}
}
public
<
T
extends
Tuple
>
DataStream
<
T
>
addSink
(
DataStream
<
T
>
inputStream
,
SinkFunction
<
T
>
sinkFunction
)
{
DataStream
<
T
>
returnStream
=
new
DataStream
<
T
>(
this
);
ByteArrayOutputStream
baos
=
new
ByteArrayOutputStream
();
ObjectOutputStream
oos
;
try
{
oos
=
new
ObjectOutputStream
(
baos
);
oos
.
writeObject
(
sinkFunction
);
}
catch
(
IOException
e
)
{
// TODO Auto-generated catch block
e
.
printStackTrace
();
}
jobGraphBuilder
.
setSink
(
"sink"
,
new
SinkInvokable
<
T
>(
sinkFunction
),
"sink"
,
baos
.
toByteArray
());
connectGraph
(
inputStream
.
connectIDs
,
"sink"
,
inputStream
.
ctype
,
inputStream
.
cparam
);
return
returnStream
;
}
public
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addMapFunction
(
public
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addMapFunction
(
DataStream
<
T
>
inputStream
,
final
MapFunction
<
T
,
R
>
mapper
)
{
DataStream
<
T
>
inputStream
,
final
MapFunction
<
T
,
R
>
mapper
)
{
DataStream
<
R
>
returnStream
=
new
DataStream
<
R
>(
this
);
DataStream
<
R
>
returnStream
=
new
DataStream
<
R
>(
this
);
...
@@ -124,35 +146,19 @@ public class StreamExecutionEnvironment {
...
@@ -124,35 +146,19 @@ public class StreamExecutionEnvironment {
return
returnStream
;
return
returnStream
;
}
}
public
static
final
class
DummySink
extends
UserSinkInvokable
<
Tuple1
<
String
>>
{
public
static
final
class
DummySink
extends
SinkFunction
<
Tuple1
<
String
>>
{
private
static
final
long
serialVersionUID
=
1L
;
private
static
final
long
serialVersionUID
=
1L
;
@Override
@Override
public
void
invoke
(
StreamRecord
record
,
StreamCollector
<
Tuple
>
collector
)
throws
Exception
{
public
void
invoke
(
Tuple1
<
String
>
tuple
)
{
for
(
Tuple
tuple
:
record
.
getBatchIterable
())
{
System
.
out
.
println
(
tuple
);
System
.
out
.
println
(
tuple
);
}
}
}
}
}
public
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addDummySink
(
DataStream
<
T
>
inputStream
)
{
public
<
T
extends
Tuple
>
DataStream
<
T
>
addDummySink
(
DataStream
<
T
>
inputStream
)
{
ByteArrayOutputStream
baos
=
new
ByteArrayOutputStream
();
ObjectOutputStream
oos
;
try
{
oos
=
new
ObjectOutputStream
(
baos
);
oos
.
writeObject
(
new
DummySink
());
}
catch
(
IOException
e
)
{
// TODO Auto-generated catch block
e
.
printStackTrace
();
}
jobGraphBuilder
.
setSink
(
"sink"
,
new
DummySink
(),
"sink"
,
baos
.
toByteArray
());
connectGraph
(
inputStream
.
connectIDs
,
"sink"
,
inputStream
.
ctype
,
return
addSink
(
inputStream
,
(
SinkFunction
<
T
>)
new
DummySink
());
inputStream
.
cparam
);
return
new
DataStream
<
R
>(
this
);
}
}
public
void
execute
()
{
public
void
execute
()
{
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/invokable/UserSinkInvokable.java
浏览文件 @
ce7a8996
...
@@ -22,5 +22,6 @@ import eu.stratosphere.api.java.tuple.Tuple;
...
@@ -22,5 +22,6 @@ import eu.stratosphere.api.java.tuple.Tuple;
public
abstract
class
UserSinkInvokable
<
IN
extends
Tuple
>
extends
StreamRecordInvokable
<
IN
,
Tuple
>
public
abstract
class
UserSinkInvokable
<
IN
extends
Tuple
>
extends
StreamRecordInvokable
<
IN
,
Tuple
>
implements
Serializable
{
implements
Serializable
{
private
static
final
long
serialVersionUID
=
1L
;
private
static
final
long
serialVersionUID
=
1L
;
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamComponentHelper.java
浏览文件 @
ce7a8996
...
@@ -24,6 +24,7 @@ import java.util.List;
...
@@ -24,6 +24,7 @@ import java.util.List;
import
org.apache.commons.logging.Log
;
import
org.apache.commons.logging.Log
;
import
org.apache.commons.logging.LogFactory
;
import
org.apache.commons.logging.LogFactory
;
import
eu.stratosphere.api.datastream.SinkFunction
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.functions.MapFunction
;
import
eu.stratosphere.api.java.functions.MapFunction
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.api.java.tuple.Tuple
;
...
@@ -148,10 +149,10 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
...
@@ -148,10 +149,10 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
}
else
if
(
operatorName
.
equals
(
"sink"
))
{
}
else
if
(
operatorName
.
equals
(
"sink"
))
{
UserSinkInvokable
<
Tuple
>
f
=
(
UserSinkInvokable
<
Tuple
>)
in
.
readObject
();
SinkFunction
<
Tuple
>
f
=
(
SinkFunction
<
Tuple
>)
in
.
readObject
();
inTupleTypeInfo
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
inTupleTypeInfo
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
UserSinkInvokable
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
SinkFunction
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
inTupleSerializer
=
inTupleTypeInfo
.
createSerializer
();
inTupleSerializer
=
inTupleTypeInfo
.
createSerializer
();
inDeserializationDelegate
=
new
DeserializationDelegate
<
Tuple
>(
inTupleSerializer
);
inDeserializationDelegate
=
new
DeserializationDelegate
<
Tuple
>(
inTupleSerializer
);
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/FlatMapTest.java
浏览文件 @
ce7a8996
...
@@ -8,6 +8,7 @@ import java.io.ObjectInputStream;
...
@@ -8,6 +8,7 @@ import java.io.ObjectInputStream;
import
org.junit.Test
;
import
org.junit.Test
;
import
eu.stratosphere.api.datastream.DataStream
;
import
eu.stratosphere.api.datastream.DataStream
;
import
eu.stratosphere.api.datastream.SinkFunction
;
import
eu.stratosphere.api.datastream.StreamExecutionEnvironment
;
import
eu.stratosphere.api.datastream.StreamExecutionEnvironment
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.api.java.tuple.Tuple
;
...
@@ -34,6 +35,16 @@ public class FlatMapTest {
...
@@ -34,6 +35,16 @@ public class FlatMapTest {
}
}
}
}
public
static
final
class
MySink
extends
SinkFunction
<
Tuple1
<
String
>>
{
@Override
public
void
invoke
(
Tuple1
<
String
>
tuple
)
{
// TODO Auto-generated method stub
System
.
out
.
println
(
tuple
);
}
}
@Test
@Test
public
void
test
()
throws
Exception
{
public
void
test
()
throws
Exception
{
Tuple1
<
String
>
tup
=
new
Tuple1
<
String
>(
"asd"
);
Tuple1
<
String
>
tup
=
new
Tuple1
<
String
>(
"asd"
);
...
@@ -42,7 +53,7 @@ public class FlatMapTest {
...
@@ -42,7 +53,7 @@ public class FlatMapTest {
DataStream
<
Tuple1
<
String
>>
dataStream0
=
context
.
setDummySource
();
DataStream
<
Tuple1
<
String
>>
dataStream0
=
context
.
setDummySource
();
DataStream
<
Tuple1
<
String
>>
dataStream1
=
context
.
setDummySource
().
connectWith
(
dataStream0
)
DataStream
<
Tuple1
<
String
>>
dataStream1
=
context
.
setDummySource
().
connectWith
(
dataStream0
)
.
partitionBy
(
0
).
flatMap
(
new
MyFlatMap
()).
broadcast
().
add
DummySink
(
);
.
partitionBy
(
0
).
flatMap
(
new
MyFlatMap
()).
broadcast
().
add
Sink
(
new
MySink
()
);
context
.
execute
();
context
.
execute
();
...
@@ -84,11 +95,11 @@ public class FlatMapTest {
...
@@ -84,11 +95,11 @@ public class FlatMapTest {
ObjectInputStream
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
bytes
));
ObjectInputStream
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
bytes
));
UserSinkInvokable
<
Tuple
>
f
=
(
UserSinkInvokable
<
Tuple
>)
in
.
readObject
();
SinkFunction
<
Tuple
>
f
=
(
SinkFunction
<
Tuple
>)
in
.
readObject
();
System
.
out
.
println
(
f
.
getClass
().
getGenericSuperclass
());
System
.
out
.
println
(
f
.
getClass
().
getGenericSuperclass
());
TupleTypeInfo
<
Tuple
>
ts
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
TupleTypeInfo
<
Tuple
>
ts
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
UserSinkInvokable
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
SinkFunction
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
System
.
out
.
println
(
ts
);
System
.
out
.
println
(
ts
);
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/MapTest.java
浏览文件 @
ce7a8996
...
@@ -8,6 +8,7 @@ import java.io.ObjectInputStream;
...
@@ -8,6 +8,7 @@ import java.io.ObjectInputStream;
import
org.junit.Test
;
import
org.junit.Test
;
import
eu.stratosphere.api.datastream.DataStream
;
import
eu.stratosphere.api.datastream.DataStream
;
import
eu.stratosphere.api.datastream.SinkFunction
;
import
eu.stratosphere.api.datastream.StreamExecutionEnvironment
;
import
eu.stratosphere.api.datastream.StreamExecutionEnvironment
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.functions.FlatMapFunction
;
import
eu.stratosphere.api.java.functions.MapFunction
;
import
eu.stratosphere.api.java.functions.MapFunction
;
...
@@ -22,6 +23,7 @@ import eu.stratosphere.nephele.jobgraph.JobOutputVertex;
...
@@ -22,6 +23,7 @@ import eu.stratosphere.nephele.jobgraph.JobOutputVertex;
import
eu.stratosphere.nephele.jobgraph.JobTaskVertex
;
import
eu.stratosphere.nephele.jobgraph.JobTaskVertex
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamcomponent.StreamInvokableComponent
;
import
eu.stratosphere.streaming.api.streamcomponent.StreamInvokableComponent
;
import
eu.stratosphere.util.Collector
;
import
eu.stratosphere.util.Collector
;
...
@@ -71,7 +73,7 @@ public class MapTest {
...
@@ -71,7 +73,7 @@ public class MapTest {
byte
[]
userFunctionSerialized
=
config
.
getBytes
(
"serializedudf"
,
null
);
byte
[]
userFunctionSerialized
=
config
.
getBytes
(
"serializedudf"
,
null
);
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
userFunctionSerialized
));
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
userFunctionSerialized
));
StreamInvokableComponent
userFunction
=
(
StreamInvokableComponent
)
in
.
readObject
();
UserTaskInvokable
userFunction
=
(
UserTaskInvokable
)
in
.
readObject
();
System
.
out
.
println
(
userFunction
.
getClass
());
System
.
out
.
println
(
userFunction
.
getClass
());
assertTrue
(
true
);
assertTrue
(
true
);
System
.
out
.
println
(
"----------------"
);
System
.
out
.
println
(
"----------------"
);
...
@@ -84,11 +86,11 @@ public class MapTest {
...
@@ -84,11 +86,11 @@ public class MapTest {
ObjectInputStream
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
bytes
));
ObjectInputStream
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
bytes
));
UserSinkInvokable
<
Tuple
>
f
=
(
UserSinkInvokable
<
Tuple
>)
in
.
readObject
();
SinkFunction
<
Tuple
>
f
=
(
SinkFunction
<
Tuple
>)
in
.
readObject
();
System
.
out
.
println
(
f
.
getClass
().
getGenericSuperclass
());
System
.
out
.
println
(
f
.
getClass
().
getGenericSuperclass
());
TupleTypeInfo
<
Tuple
>
ts
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
TupleTypeInfo
<
Tuple
>
ts
=
(
TupleTypeInfo
)
TypeExtractor
.
createTypeInfo
(
UserSinkInvokable
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
SinkFunction
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
System
.
out
.
println
(
ts
);
System
.
out
.
println
(
ts
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录