Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
e34bcd2c
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
e34bcd2c
编写于
7月 14, 2014
作者:
G
gaborhermann
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] Created StreamComponentFactory for the common methods of StreamSource and StreamTask
上级
5d40e871
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
60 addition
and
93 deletion
+60
-93
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamComponentFactory.java
...eu/stratosphere/streaming/api/StreamComponentFactory.java
+53
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamSource.java
...main/java/eu/stratosphere/streaming/api/StreamSource.java
+3
-47
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamTask.java
...c/main/java/eu/stratosphere/streaming/api/StreamTask.java
+4
-46
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamComponentFactory.java
0 → 100644
浏览文件 @
e34bcd2c
package
eu.stratosphere.streaming.api
;
import
java.util.List
;
import
java.util.Map
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.event.task.EventListener
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.streaming.partitioner.DefaultPartitioner
;
import
eu.stratosphere.streaming.partitioner.FieldsPartitioner
;
import
eu.stratosphere.types.Key
;
import
eu.stratosphere.types.Record
;
import
eu.stratosphere.types.StringValue
;
public
final
class
StreamComponentFactory
{
//TODO: put setConfigInputs here
public
static
void
setAckListener
(
Map
<
String
,
StreamRecord
>
recordBuffer
,
String
sourceInstanceID
,
List
<
RecordWriter
<
Record
>>
outputs
)
{
EventListener
eventListener
=
new
AckEventListener
(
sourceInstanceID
,
recordBuffer
);
for
(
RecordWriter
<
Record
>
output
:
outputs
)
{
// TODO: separate outputs
output
.
subscribeToEvent
(
eventListener
,
AckEvent
.
class
);
}
}
public
static
void
setPartitioner
(
Configuration
taskConfiguration
,
int
nrOutput
,
List
<
ChannelSelector
<
Record
>>
partitioners
)
{
Class
<?
extends
ChannelSelector
<
Record
>>
partitioner
=
taskConfiguration
.
getClass
(
"partitionerClass_"
+
nrOutput
,
DefaultPartitioner
.
class
,
ChannelSelector
.
class
);
try
{
if
(
partitioner
.
equals
(
FieldsPartitioner
.
class
))
{
int
keyPosition
=
taskConfiguration
.
getInteger
(
"partitionerIntParam_"
+
nrOutput
,
1
);
Class
<?
extends
Key
>
keyClass
=
taskConfiguration
.
getClass
(
"partitionerClassParam_"
+
nrOutput
,
StringValue
.
class
,
Key
.
class
);
partitioners
.
add
(
partitioner
.
getConstructor
(
int
.
class
,
Class
.
class
)
.
newInstance
(
keyPosition
,
keyClass
));
}
else
{
partitioners
.
add
(
partitioner
.
newInstance
());
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
"partitioner error"
+
" "
+
"partitioner_"
+
nrOutput
);
System
.
out
.
println
(
e
);
}
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamSource.java
浏览文件 @
e34bcd2c
...
@@ -21,18 +21,13 @@ import java.util.Map;
...
@@ -21,18 +21,13 @@ import java.util.Map;
import
java.util.TreeMap
;
import
java.util.TreeMap
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.event.task.EventListener
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.template.AbstractInputTask
;
import
eu.stratosphere.nephele.template.AbstractInputTask
;
import
eu.stratosphere.streaming.api.invokable.DefaultSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.DefaultSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.partitioner.DefaultPartitioner
;
import
eu.stratosphere.streaming.partitioner.FieldsPartitioner
;
import
eu.stratosphere.streaming.test.RandIS
;
import
eu.stratosphere.streaming.test.RandIS
;
import
eu.stratosphere.types.Key
;
import
eu.stratosphere.types.Record
;
import
eu.stratosphere.types.Record
;
import
eu.stratosphere.types.StringValue
;
public
class
StreamSource
extends
AbstractInputTask
<
RandIS
>
{
public
class
StreamSource
extends
AbstractInputTask
<
RandIS
>
{
...
@@ -72,7 +67,7 @@ public class StreamSource extends AbstractInputTask<RandIS> {
...
@@ -72,7 +67,7 @@ public class StreamSource extends AbstractInputTask<RandIS> {
numberOfOutputs
=
taskConfiguration
.
getInteger
(
"numberOfOutputs"
,
0
);
numberOfOutputs
=
taskConfiguration
.
getInteger
(
"numberOfOutputs"
,
0
);
for
(
int
i
=
1
;
i
<=
numberOfOutputs
;
i
++)
{
for
(
int
i
=
1
;
i
<=
numberOfOutputs
;
i
++)
{
setPartitioner
(
taskConfiguration
,
i
);
StreamComponentFactory
.
setPartitioner
(
taskConfiguration
,
i
,
partitioners
);
}
}
for
(
ChannelSelector
<
Record
>
outputPartitioner
:
partitioners
)
{
for
(
ChannelSelector
<
Record
>
outputPartitioner
:
partitioners
)
{
...
@@ -81,63 +76,24 @@ public class StreamSource extends AbstractInputTask<RandIS> {
...
@@ -81,63 +76,24 @@ public class StreamSource extends AbstractInputTask<RandIS> {
}
}
setUserFunction
(
taskConfiguration
);
setUserFunction
(
taskConfiguration
);
setAckListener
();
StreamComponentFactory
.
setAckListener
(
recordBuffer
,
sourceInstanceID
,
outputs
);
}
}
public
void
setUserFunction
(
Configuration
taskConfiguration
)
{
public
void
setUserFunction
(
Configuration
taskConfiguration
)
{
Class
<?
extends
UserSourceInvokable
>
userFunctionClass
=
taskConfiguration
Class
<?
extends
UserSourceInvokable
>
userFunctionClass
=
taskConfiguration
.
getClass
(
"userfunction"
,
DefaultSourceInvokable
.
class
,
.
getClass
(
"userfunction"
,
DefaultSourceInvokable
.
class
,
UserSourceInvokable
.
class
);
UserSourceInvokable
.
class
);
try
{
try
{
this
.
userFunction
=
userFunctionClass
.
newInstance
();
this
.
userFunction
=
userFunctionClass
.
newInstance
();
this
.
userFunction
.
declareOutputs
(
outputs
,
sourceInstanceID
,
recordBuffer
);
this
.
userFunction
.
declareOutputs
(
outputs
,
sourceInstanceID
,
recordBuffer
);
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
}
}
}
}
private
void
setPartitioner
(
Configuration
taskConfiguration
,
int
nrOutput
)
{
Class
<?
extends
ChannelSelector
<
Record
>>
partitioner
=
taskConfiguration
.
getClass
(
"partitionerClass_"
+
nrOutput
,
DefaultPartitioner
.
class
,
ChannelSelector
.
class
);
try
{
if
(
partitioner
.
equals
(
FieldsPartitioner
.
class
))
{
int
keyPosition
=
taskConfiguration
.
getInteger
(
"partitionerIntParam_"
+
nrOutput
,
1
);
Class
<?
extends
Key
>
keyClass
=
taskConfiguration
.
getClass
(
"partitionerClassParam_"
+
nrOutput
,
StringValue
.
class
,
Key
.
class
);
partitioners
.
add
(
partitioner
.
getConstructor
(
int
.
class
,
Class
.
class
)
.
newInstance
(
keyPosition
,
keyClass
));
}
else
{
partitioners
.
add
(
partitioner
.
newInstance
());
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
"partitioner error"
+
" "
+
"partitioner_"
+
nrOutput
);
System
.
out
.
println
(
e
);
}
}
public
void
setAckListener
()
{
EventListener
eventListener
=
new
AckEventListener
(
sourceInstanceID
,
recordBuffer
);
for
(
RecordWriter
<
Record
>
output
:
outputs
)
{
// TODO: separate outputs
output
.
subscribeToEvent
(
eventListener
,
AckEvent
.
class
);
}
}
@Override
@Override
public
void
registerInputOutput
()
{
public
void
registerInputOutput
()
{
setConfigInputs
();
setConfigInputs
();
}
}
@Override
@Override
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamTask.java
浏览文件 @
e34bcd2c
...
@@ -21,18 +21,13 @@ import java.util.Map;
...
@@ -21,18 +21,13 @@ import java.util.Map;
import
java.util.TreeMap
;
import
java.util.TreeMap
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.event.task.EventListener
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.RecordReader
;
import
eu.stratosphere.nephele.io.RecordReader
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.template.AbstractTask
;
import
eu.stratosphere.nephele.template.AbstractTask
;
import
eu.stratosphere.streaming.api.invokable.DefaultTaskInvokable
;
import
eu.stratosphere.streaming.api.invokable.DefaultTaskInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.partitioner.DefaultPartitioner
;
import
eu.stratosphere.streaming.partitioner.FieldsPartitioner
;
import
eu.stratosphere.types.Key
;
import
eu.stratosphere.types.Record
;
import
eu.stratosphere.types.Record
;
import
eu.stratosphere.types.StringValue
;
//TODO: Refactor, create common ancestor with StreamSource
//TODO: Refactor, create common ancestor with StreamSource
public
class
StreamTask
extends
AbstractTask
{
public
class
StreamTask
extends
AbstractTask
{
...
@@ -43,7 +38,7 @@ public class StreamTask extends AbstractTask {
...
@@ -43,7 +38,7 @@ public class StreamTask extends AbstractTask {
private
UserTaskInvokable
userFunction
;
private
UserTaskInvokable
userFunction
;
private
int
numberOfInputs
;
private
int
numberOfInputs
;
private
int
numberOfOutputs
;
private
int
numberOfOutputs
;
private
static
int
numTasks
=
0
;
private
static
int
numTasks
=
0
;
private
String
taskInstanceID
=
""
;
private
String
taskInstanceID
=
""
;
private
Map
<
String
,
StreamRecord
>
recordBuffer
;
private
Map
<
String
,
StreamRecord
>
recordBuffer
;
...
@@ -62,7 +57,6 @@ public class StreamTask extends AbstractTask {
...
@@ -62,7 +57,6 @@ public class StreamTask extends AbstractTask {
}
}
private
void
setConfigInputs
()
{
private
void
setConfigInputs
()
{
Configuration
taskConfiguration
=
getTaskConfiguration
();
Configuration
taskConfiguration
=
getTaskConfiguration
();
numberOfInputs
=
taskConfiguration
.
getInteger
(
"numberOfInputs"
,
0
);
numberOfInputs
=
taskConfiguration
.
getInteger
(
"numberOfInputs"
,
0
);
...
@@ -73,7 +67,7 @@ public class StreamTask extends AbstractTask {
...
@@ -73,7 +67,7 @@ public class StreamTask extends AbstractTask {
numberOfOutputs
=
taskConfiguration
.
getInteger
(
"numberOfOutputs"
,
0
);
numberOfOutputs
=
taskConfiguration
.
getInteger
(
"numberOfOutputs"
,
0
);
for
(
int
i
=
1
;
i
<=
numberOfOutputs
;
i
++)
{
for
(
int
i
=
1
;
i
<=
numberOfOutputs
;
i
++)
{
setPartitioner
(
taskConfiguration
,
i
);
StreamComponentFactory
.
setPartitioner
(
taskConfiguration
,
i
,
partitioners
);
}
}
for
(
ChannelSelector
<
Record
>
outputPartitioner
:
partitioners
)
{
for
(
ChannelSelector
<
Record
>
outputPartitioner
:
partitioners
)
{
...
@@ -82,20 +76,10 @@ public class StreamTask extends AbstractTask {
...
@@ -82,20 +76,10 @@ public class StreamTask extends AbstractTask {
}
}
setUserFunction
(
taskConfiguration
);
setUserFunction
(
taskConfiguration
);
setAckListener
();
StreamComponentFactory
.
setAckListener
(
recordBuffer
,
taskInstanceID
,
outputs
);
}
public
void
setAckListener
()
{
EventListener
eventListener
=
new
AckEventListener
(
taskInstanceID
,
recordBuffer
);
for
(
RecordWriter
<
Record
>
output
:
outputs
)
{
// TODO: separate outputs
output
.
subscribeToEvent
(
eventListener
,
AckEvent
.
class
);
}
}
}
public
void
setUserFunction
(
Configuration
taskConfiguration
)
{
public
void
setUserFunction
(
Configuration
taskConfiguration
)
{
Class
<?
extends
UserTaskInvokable
>
userFunctionClass
=
taskConfiguration
Class
<?
extends
UserTaskInvokable
>
userFunctionClass
=
taskConfiguration
.
getClass
(
"userfunction"
,
DefaultTaskInvokable
.
class
,
.
getClass
(
"userfunction"
,
DefaultTaskInvokable
.
class
,
UserTaskInvokable
.
class
);
UserTaskInvokable
.
class
);
...
@@ -105,32 +89,6 @@ public class StreamTask extends AbstractTask {
...
@@ -105,32 +89,6 @@ public class StreamTask extends AbstractTask {
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
}
}
}
private
void
setPartitioner
(
Configuration
taskConfiguration
,
int
nrOutput
)
{
Class
<?
extends
ChannelSelector
<
Record
>>
partitioner
=
taskConfiguration
.
getClass
(
"partitionerClass_"
+
nrOutput
,
DefaultPartitioner
.
class
,
ChannelSelector
.
class
);
try
{
if
(
partitioner
.
equals
(
FieldsPartitioner
.
class
))
{
int
keyPosition
=
taskConfiguration
.
getInteger
(
"partitionerIntParam_"
+
nrOutput
,
1
);
Class
<?
extends
Key
>
keyClass
=
taskConfiguration
.
getClass
(
"partitionerClassParam_"
+
nrOutput
,
StringValue
.
class
,
Key
.
class
);
partitioners
.
add
(
partitioner
.
getConstructor
(
int
.
class
,
Class
.
class
)
.
newInstance
(
keyPosition
,
keyClass
));
}
else
{
partitioners
.
add
(
partitioner
.
newInstance
());
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
"partitioner error"
+
" "
+
"partitioner_"
+
nrOutput
);
System
.
out
.
println
(
e
);
}
}
}
@Override
@Override
...
@@ -148,7 +106,7 @@ public class StreamTask extends AbstractTask {
...
@@ -148,7 +106,7 @@ public class StreamTask extends AbstractTask {
hasInput
=
true
;
hasInput
=
true
;
StreamRecord
streamRecord
=
new
StreamRecord
(
input
.
next
());
StreamRecord
streamRecord
=
new
StreamRecord
(
input
.
next
());
String
id
=
streamRecord
.
popId
();
String
id
=
streamRecord
.
popId
();
//TODO: Enclose invoke in try-catch to properly fail records
//
TODO: Enclose invoke in try-catch to properly fail records
userFunction
.
invoke
(
streamRecord
.
getRecord
());
userFunction
.
invoke
(
streamRecord
.
getRecord
());
System
.
out
.
println
(
this
.
getClass
().
getName
()
+
"-"
+
taskInstanceID
);
System
.
out
.
println
(
this
.
getClass
().
getName
()
+
"-"
+
taskInstanceID
);
System
.
out
.
println
(
recordBuffer
.
toString
());
System
.
out
.
println
(
recordBuffer
.
toString
());
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录