Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
bb01ea01
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
bb01ea01
编写于
7月 14, 2014
作者:
M
Márton Balassi
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] JobGraphBuilder documentation update
上级
3b5ec860
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
95 addition
and
50 deletion
+95
-50
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/JobGraphBuilder.java
...n/java/eu/stratosphere/streaming/api/JobGraphBuilder.java
+93
-48
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
...stratosphere/streaming/api/streamrecord/StreamRecord.java
+2
-2
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/JobGraphBuilder.java
浏览文件 @
bb01ea01
...
...
@@ -45,7 +45,6 @@ import eu.stratosphere.streaming.partitioner.BroadcastPartitioner;
import
eu.stratosphere.streaming.partitioner.FieldsPartitioner
;
import
eu.stratosphere.streaming.partitioner.GlobalPartitioner
;
import
eu.stratosphere.streaming.partitioner.ShufflePartitioner
;
import
eu.stratosphere.types.Key
;
/**
* Object for building Stratosphere stream processing job graphs
...
...
@@ -80,7 +79,8 @@ public class JobGraphBuilder {
* @param InvokableClass
* User defined class describing the source
*/
public
void
setSource
(
String
sourceName
,
final
Class
<?
extends
UserSourceInvokable
>
InvokableClass
)
{
public
void
setSource
(
String
sourceName
,
final
Class
<?
extends
UserSourceInvokable
>
InvokableClass
)
{
setSource
(
sourceName
,
InvokableClass
,
1
);
}
...
...
@@ -94,7 +94,9 @@ public class JobGraphBuilder {
* @param parallelism
* Number of task instances of this type to run in parallel
*/
public
void
setSource
(
String
sourceName
,
final
Class
<?
extends
UserSourceInvokable
>
InvokableClass
,
int
parallelism
)
{
public
void
setSource
(
String
sourceName
,
final
Class
<?
extends
UserSourceInvokable
>
InvokableClass
,
int
parallelism
)
{
final
JobInputVertex
source
=
new
JobInputVertex
(
sourceName
,
jobGraph
);
source
.
setInputClass
(
StreamSource
.
class
);
setComponent
(
sourceName
,
InvokableClass
,
parallelism
,
source
);
...
...
@@ -109,7 +111,8 @@ public class JobGraphBuilder {
* @param InvokableClass
* User defined class describing the task
*/
public
void
setTask
(
String
taskName
,
final
Class
<?
extends
UserTaskInvokable
>
InvokableClass
)
{
public
void
setTask
(
String
taskName
,
final
Class
<?
extends
UserTaskInvokable
>
InvokableClass
)
{
setTask
(
taskName
,
InvokableClass
,
1
);
}
...
...
@@ -123,7 +126,9 @@ public class JobGraphBuilder {
* @param parallelism
* Number of task instances of this type to run in parallel
*/
public
void
setTask
(
String
taskName
,
final
Class
<?
extends
UserTaskInvokable
>
InvokableClass
,
int
parallelism
)
{
public
void
setTask
(
String
taskName
,
final
Class
<?
extends
UserTaskInvokable
>
InvokableClass
,
int
parallelism
)
{
final
JobTaskVertex
task
=
new
JobTaskVertex
(
taskName
,
jobGraph
);
task
.
setTaskClass
(
StreamTask
.
class
);
setComponent
(
taskName
,
InvokableClass
,
parallelism
,
task
);
...
...
@@ -138,7 +143,8 @@ public class JobGraphBuilder {
* @param InvokableClass
* User defined class describing the sink
*/
public
void
setSink
(
String
sinkName
,
final
Class
<?
extends
UserSinkInvokable
>
InvokableClass
)
{
public
void
setSink
(
String
sinkName
,
final
Class
<?
extends
UserSinkInvokable
>
InvokableClass
)
{
setSink
(
sinkName
,
InvokableClass
,
1
);
}
...
...
@@ -152,22 +158,27 @@ public class JobGraphBuilder {
* @param parallelism
* Number of task instances of this type to run in parallel
*/
public
void
setSink
(
String
sinkName
,
final
Class
<?
extends
UserSinkInvokable
>
InvokableClass
,
int
parallelism
)
{
public
void
setSink
(
String
sinkName
,
final
Class
<?
extends
UserSinkInvokable
>
InvokableClass
,
int
parallelism
)
{
final
JobOutputVertex
sink
=
new
JobOutputVertex
(
sinkName
,
jobGraph
);
sink
.
setOutputClass
(
StreamSink
.
class
);
setComponent
(
sinkName
,
InvokableClass
,
parallelism
,
sink
);
log
.
debug
(
"SINK: "
+
sinkName
);
}
private
void
setComponent
(
String
componentName
,
final
Class
<?
extends
UserInvokable
>
InvokableClass
,
int
parallelism
,
AbstractJobVertex
component
)
{
private
void
setComponent
(
String
componentName
,
final
Class
<?
extends
UserInvokable
>
InvokableClass
,
int
parallelism
,
AbstractJobVertex
component
)
{
component
.
setNumberOfSubtasks
(
parallelism
);
Configuration
config
=
new
TaskConfig
(
component
.
getConfiguration
()).
getConfiguration
();
Configuration
config
=
new
TaskConfig
(
component
.
getConfiguration
())
.
getConfiguration
();
config
.
setClass
(
"userfunction"
,
InvokableClass
);
config
.
setString
(
"componentName"
,
componentName
);
components
.
put
(
componentName
,
component
);
numberOfInstances
.
put
(
componentName
,
parallelism
);
}
/**
* Connects to JobGraph components with the given names, partitioning and
* channel type
...
...
@@ -182,22 +193,33 @@ public class JobGraphBuilder {
* @param channelType
* Channel Type
*/
private
void
connect
(
String
upStreamComponentName
,
String
downStreamComponentName
,
Class
<?
extends
ChannelSelector
<
StreamRecord
>>
PartitionerClass
,
ChannelType
channelType
)
{
private
void
connect
(
String
upStreamComponentName
,
String
downStreamComponentName
,
Class
<?
extends
ChannelSelector
<
StreamRecord
>>
PartitionerClass
,
ChannelType
channelType
)
{
AbstractJobVertex
upStreamComponent
=
components
.
get
(
upStreamComponentName
);
AbstractJobVertex
downStreamComponent
=
components
.
get
(
downStreamComponentName
);
AbstractJobVertex
upStreamComponent
=
components
.
get
(
upStreamComponentName
);
AbstractJobVertex
downStreamComponent
=
components
.
get
(
downStreamComponentName
);
try
{
upStreamComponent
.
connectTo
(
downStreamComponent
,
channelType
);
Configuration
config
=
new
TaskConfig
(
upStreamComponent
.
getConfiguration
()).
getConfiguration
();
config
.
setClass
(
"partitionerClass_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
Configuration
config
=
new
TaskConfig
(
upStreamComponent
.
getConfiguration
()).
getConfiguration
();
config
.
setClass
(
"partitionerClass_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
PartitionerClass
);
log
.
debug
(
"CONNECTED: "
+
PartitionerClass
.
getSimpleName
()
+
" - "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
);
log
.
debug
(
"CONNECTED: "
+
PartitionerClass
.
getSimpleName
()
+
" - "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
);
}
catch
(
JobGraphDefinitionException
e
)
{
log
.
error
(
"Cannot connect components with "
+
PartitionerClass
.
getSimpleName
()
+
" : "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
,
e
);
log
.
error
(
"Cannot connect components with "
+
PartitionerClass
.
getSimpleName
()
+
" : "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
,
e
);
}
}
...
...
@@ -213,9 +235,12 @@ public class JobGraphBuilder {
* Name of the downstream component, that will receive the
* records
*/
public
void
broadcastConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
BroadcastPartitioner
.
class
,
ChannelType
.
INMEMORY
);
addOutputChannels
(
upStreamComponentName
,
numberOfInstances
.
get
(
downStreamComponentName
));
public
void
broadcastConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
BroadcastPartitioner
.
class
,
ChannelType
.
INMEMORY
);
addOutputChannels
(
upStreamComponentName
,
numberOfInstances
.
get
(
downStreamComponentName
));
}
/**
...
...
@@ -231,33 +256,42 @@ public class JobGraphBuilder {
* Name of the downstream component, that will receive the
* records
* @param keyPosition
* Position of key in the record
* @param keyClass
* Class of the key Value stored in the record
* Position of key in the tuple
*/
//TODO: remove unused third parameter
public
void
fieldsConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
,
int
keyPosition
)
{
public
void
fieldsConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
,
int
keyPosition
)
{
AbstractJobVertex
upStreamComponent
=
components
.
get
(
upStreamComponentName
);
AbstractJobVertex
downStreamComponent
=
components
.
get
(
downStreamComponentName
);
AbstractJobVertex
upStreamComponent
=
components
.
get
(
upStreamComponentName
);
AbstractJobVertex
downStreamComponent
=
components
.
get
(
downStreamComponentName
);
try
{
upStreamComponent
.
connectTo
(
downStreamComponent
,
ChannelType
.
INMEMORY
);
upStreamComponent
.
connectTo
(
downStreamComponent
,
ChannelType
.
INMEMORY
);
Configuration
config
=
new
TaskConfig
(
upStreamComponent
.
getConfiguration
()).
getConfiguration
();
Configuration
config
=
new
TaskConfig
(
upStreamComponent
.
getConfiguration
()).
getConfiguration
();
config
.
setClass
(
"partitionerClass_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
config
.
setClass
(
"partitionerClass_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
FieldsPartitioner
.
class
);
config
.
setInteger
(
"partitionerIntParam_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
config
.
setInteger
(
"partitionerIntParam_"
+
(
upStreamComponent
.
getNumberOfForwardConnections
()
-
1
),
keyPosition
);
addOutputChannels
(
upStreamComponentName
,
1
);
log
.
debug
(
"CONNECTED: FIELD PARTITIONING - "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
log
.
debug
(
"CONNECTED: FIELD PARTITIONING - "
+
upStreamComponentName
+
" -> "
+
downStreamComponentName
+
", KEY: "
+
keyPosition
);
}
catch
(
JobGraphDefinitionException
e
)
{
log
.
error
(
"Cannot connect components by field: "
+
upStreamComponentName
+
" to "
+
downStreamComponentName
,
log
.
error
(
"Cannot connect components by field: "
+
upStreamComponentName
+
" to "
+
downStreamComponentName
,
e
);
}
}
...
...
@@ -274,8 +308,10 @@ public class JobGraphBuilder {
* Name of the downstream component, that will receive the
* records
*/
public
void
globalConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
GlobalPartitioner
.
class
,
ChannelType
.
INMEMORY
);
public
void
globalConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
GlobalPartitioner
.
class
,
ChannelType
.
INMEMORY
);
addOutputChannels
(
upStreamComponentName
,
1
);
}
...
...
@@ -291,34 +327,43 @@ public class JobGraphBuilder {
* Name of the downstream component, that will receive the
* records
*/
public
void
shuffleConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
ShufflePartitioner
.
class
,
ChannelType
.
INMEMORY
);
public
void
shuffleConnect
(
String
upStreamComponentName
,
String
downStreamComponentName
)
{
connect
(
upStreamComponentName
,
downStreamComponentName
,
ShufflePartitioner
.
class
,
ChannelType
.
INMEMORY
);
addOutputChannels
(
upStreamComponentName
,
1
);
}
private
void
addOutputChannels
(
String
upStreamComponentName
,
int
numOfInstances
)
{
private
void
addOutputChannels
(
String
upStreamComponentName
,
int
numOfInstances
)
{
if
(
numberOfOutputChannels
.
containsKey
(
upStreamComponentName
))
{
numberOfOutputChannels
.
get
(
upStreamComponentName
).
add
(
numOfInstances
);
numberOfOutputChannels
.
get
(
upStreamComponentName
).
add
(
numOfInstances
);
}
else
{
numberOfOutputChannels
.
put
(
upStreamComponentName
,
new
ArrayList
<
Integer
>());
numberOfOutputChannels
.
get
(
upStreamComponentName
).
add
(
numOfInstances
);
numberOfOutputChannels
.
put
(
upStreamComponentName
,
new
ArrayList
<
Integer
>());
numberOfOutputChannels
.
get
(
upStreamComponentName
).
add
(
numOfInstances
);
}
}
private
void
setNumberOfJobInputs
()
{
for
(
AbstractJobVertex
component
:
components
.
values
())
{
component
.
getConfiguration
().
setInteger
(
"numberOfInputs"
,
component
.
getNumberOfBackwardConnections
());
component
.
getConfiguration
().
setInteger
(
"numberOfInputs"
,
component
.
getNumberOfBackwardConnections
());
}
}
private
void
setNumberOfJobOutputs
()
{
for
(
AbstractJobVertex
component
:
components
.
values
())
{
component
.
getConfiguration
().
setInteger
(
"numberOfOutputs"
,
component
.
getNumberOfForwardConnections
());
component
.
getConfiguration
().
setInteger
(
"numberOfOutputs"
,
component
.
getNumberOfForwardConnections
());
}
for
(
String
component
:
numberOfOutputChannels
.
keySet
())
{
Configuration
config
=
components
.
get
(
component
).
getConfiguration
();
List
<
Integer
>
channelNumList
=
numberOfOutputChannels
.
get
(
component
);
List
<
Integer
>
channelNumList
=
numberOfOutputChannels
.
get
(
component
);
for
(
int
i
=
0
;
i
<
channelNumList
.
size
();
i
++)
{
config
.
setInteger
(
"channels_"
+
i
,
channelNumList
.
get
(
i
));
}
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
浏览文件 @
bb01ea01
...
...
@@ -438,9 +438,9 @@ public class StreamRecord implements IOReadableWritable, Serializable {
* @param d
* New value
*/
public
void
setDouble
(
int
tupleNumber
,
int
fieldNumber
,
Double
tuple
)
{
public
void
setDouble
(
int
tupleNumber
,
int
fieldNumber
,
Double
d
)
{
try
{
tupleBatch
.
get
(
tupleNumber
).
setField
(
tuple
,
fieldNumber
);
tupleBatch
.
get
(
tupleNumber
).
setField
(
d
,
fieldNumber
);
}
catch
(
IndexOutOfBoundsException
e
)
{
throw
(
new
NoSuchTupleException
());
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录