Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
f9d03236
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
f9d03236
编写于
7月 14, 2014
作者:
M
Márton Balassi
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] StreamExecutionEnvironment javadoc update
上级
0dea5d1d
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
59 addition
and
18 deletion
+59
-18
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamExecutionEnvironment.java
...tratosphere/streaming/api/StreamExecutionEnvironment.java
+59
-18
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamExecutionEnvironment.java
浏览文件 @
f9d03236
...
...
@@ -28,11 +28,26 @@ import eu.stratosphere.streaming.faulttolerance.FaultToleranceType;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.util.Collector
;
//TODO: add file, elements, rmq source
//TODO: figure out generic dummysink
//TODO:add link to ExecutionEnvironment
/**
* ExecutionEnvironment for streaming jobs. An instance of it is necessary to
* construct streaming topologies.
*
*/
// TODO: add file, elements, rmq source
// TODO: figure out generic dummysink
public
class
StreamExecutionEnvironment
{
JobGraphBuilder
jobGraphBuilder
;
/**
* General constructor specifying the batch size in which the tuples are
* transmitted and their timeout boundary.
*
* @param defaultBatchSize
* number of tuples in a batch
* @param defaultBatchTimeoutMillis
* timeout boundary in milliseconds
*/
public
StreamExecutionEnvironment
(
int
defaultBatchSize
,
long
defaultBatchTimeoutMillis
)
{
if
(
defaultBatchSize
<
1
)
{
throw
new
IllegalArgumentException
(
"Batch size must be positive."
);
...
...
@@ -44,6 +59,9 @@ public class StreamExecutionEnvironment {
defaultBatchSize
,
defaultBatchTimeoutMillis
);
}
/**
* Constructor for transmitting tuples individually with a 1 second timeout.
*/
public
StreamExecutionEnvironment
()
{
this
(
1
,
1000
);
}
...
...
@@ -59,10 +77,19 @@ public class StreamExecutionEnvironment {
}
}
/**
* Partitioning strategy on the stream.
*/
public
static
enum
ConnectionType
{
SHUFFLE
,
BROADCAST
,
FIELD
}
/**
* Sets the batch size of the datastream in which the tuple are transmitted.
*
* @param inputStream
* input datastream
*/
public
<
T
extends
Tuple
>
void
setBatchSize
(
DataStream
<
T
>
inputStream
)
{
for
(
int
i
=
0
;
i
<
inputStream
.
connectIDs
.
size
();
i
++)
{
...
...
@@ -71,6 +98,15 @@ public class StreamExecutionEnvironment {
}
}
// TODO: Link to JobGraph & JobGraphBuilder
/**
* Internal function for assembling the underlying JobGraph of the job.
*
* @param inputStream
* input datastream
* @param outputID
* ID of the output
*/
private
<
T
extends
Tuple
>
void
connectGraph
(
DataStream
<
T
>
inputStream
,
String
outputID
)
{
for
(
int
i
=
0
;
i
<
inputStream
.
connectIDs
.
size
();
i
++)
{
...
...
@@ -95,9 +131,19 @@ public class StreamExecutionEnvironment {
}
public
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addFunction
(
String
functionName
,
//TODO: link to JobGraph, JobVertex, user-defined spellcheck
/**
* Internal function for passing the user defined functions to the JobGraph of the job.
* @param functionName name of the function
* @param inputStream input data stream
* @param function the user defined function
* @param functionInvokable the wrapping JobVertex instance
* @param parallelism number of parallel instances of the function
* @return the data stream constructed
*/
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addFunction
(
String
functionName
,
DataStream
<
T
>
inputStream
,
final
AbstractFunction
function
,
UserTaskInvokable
<
T
,
R
>
functionInvokable
,
int
paralelism
)
{
UserTaskInvokable
<
T
,
R
>
functionInvokable
,
int
paral
l
elism
)
{
DataStream
<
R
>
returnStream
=
new
DataStream
<
R
>(
this
);
ByteArrayOutputStream
baos
=
new
ByteArrayOutputStream
();
...
...
@@ -110,21 +156,15 @@ public class StreamExecutionEnvironment {
}
jobGraphBuilder
.
setTask
(
returnStream
.
getId
(),
functionInvokable
,
functionName
,
baos
.
toByteArray
(),
paralelism
,
para
lelism
);
baos
.
toByteArray
(),
parallelism
,
paral
lelism
);
connectGraph
(
inputStream
,
returnStream
.
getId
());
return
returnStream
;
}
public
<
T
extends
Tuple
,
R
extends
Tuple
>
DataStream
<
R
>
addFunction
(
String
functionName
,
DataStream
<
T
>
inputStream
,
final
AbstractFunction
function
,
UserTaskInvokable
<
T
,
R
>
functionInvokable
)
{
return
addFunction
(
functionName
,
inputStream
,
function
,
functionInvokable
,
1
);
}
public
<
T
extends
Tuple
>
DataStream
<
T
>
addSink
(
DataStream
<
T
>
inputStream
,
SinkFunction
<
T
>
sinkFunction
,
int
paralelism
)
{
SinkFunction
<
T
>
sinkFunction
,
int
paral
l
elism
)
{
DataStream
<
T
>
returnStream
=
new
DataStream
<
T
>(
this
);
ByteArrayOutputStream
baos
=
new
ByteArrayOutputStream
();
...
...
@@ -138,13 +178,13 @@ public class StreamExecutionEnvironment {
}
jobGraphBuilder
.
setSink
(
"sink"
,
new
SinkInvokable
<
T
>(
sinkFunction
),
"sink"
,
baos
.
toByteArray
(),
paralelism
,
para
lelism
);
baos
.
toByteArray
(),
parallelism
,
paral
lelism
);
connectGraph
(
inputStream
,
"sink"
);
return
returnStream
;
}
public
<
T
extends
Tuple
>
DataStream
<
T
>
addSink
(
DataStream
<
T
>
inputStream
,
SinkFunction
<
T
>
sinkFunction
)
{
return
addSink
(
inputStream
,
sinkFunction
,
1
);
...
...
@@ -169,7 +209,8 @@ public class StreamExecutionEnvironment {
ClusterUtil
.
runOnMiniCluster
(
jobGraphBuilder
.
getJobGraph
());
}
public
<
T
extends
Tuple
>
DataStream
<
T
>
addSource
(
SourceFunction
<
T
>
sourceFunction
,
int
parallelism
)
{
public
<
T
extends
Tuple
>
DataStream
<
T
>
addSource
(
SourceFunction
<
T
>
sourceFunction
,
int
parallelism
)
{
DataStream
<
T
>
returnStream
=
new
DataStream
<
T
>(
this
);
ByteArrayOutputStream
baos
=
new
ByteArrayOutputStream
();
...
...
@@ -183,13 +224,13 @@ public class StreamExecutionEnvironment {
}
jobGraphBuilder
.
setSource
(
returnStream
.
getId
(),
sourceFunction
,
"source"
,
baos
.
toByteArray
(),
parallelism
,
parallelism
);
baos
.
toByteArray
(),
parallelism
,
parallelism
);
return
returnStream
;
}
public
DataStream
<
Tuple1
<
String
>>
readTextFile
(
String
path
)
{
return
addSource
(
new
FileSourceFunction
(
path
),
1
);
return
addSource
(
new
FileSourceFunction
(
path
),
1
);
}
public
DataStream
<
Tuple1
<
String
>>
addDummySource
()
{
...
...
@@ -206,7 +247,7 @@ public class StreamExecutionEnvironment {
}
jobGraphBuilder
.
setSource
(
returnStream
.
getId
(),
new
DummySource
(),
"source"
,
baos
.
toByteArray
(),
1
,
1
);
baos
.
toByteArray
(),
1
,
1
);
return
returnStream
;
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录