Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
55f880bd
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
55f880bd
编写于
7月 14, 2014
作者:
G
Gyula Fora
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] javadoc added to fault tolerance buffer
上级
d1fc3385
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
55 addition
and
2 deletion
+55
-2
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/FaultTolerancyBuffer.java
...a/eu/stratosphere/streaming/api/FaultTolerancyBuffer.java
+55
-2
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/FaultTolerancyBuffer.java
浏览文件 @
55f880bd
...
...
@@ -26,10 +26,13 @@ import java.util.TreeMap;
import
eu.stratosphere.nephele.io.RecordWriter
;
/** An object to provide fault tolerance for Stratosphere stream processing.
* It works as a buffer to hold StreamRecords for a task for re-emitting failed, or timed out records.
*/
public
class
FaultTolerancyBuffer
{
private
long
TIMEOUT
=
1000
;
private
Long
timeOfLastUpdate
;
private
Map
<
String
,
StreamRecord
>
recordBuffer
;
private
Map
<
String
,
Integer
>
ackCounter
;
...
...
@@ -41,6 +44,13 @@ public class FaultTolerancyBuffer {
private
int
numberOfOutputs
;
/**Creates fault tolerance buffer object for the given output channels and channel ID
*
* @param outputs
* List of outputs
* @param channelID
* ID of the task object that uses this buffer
*/
public
FaultTolerancyBuffer
(
List
<
RecordWriter
<
StreamRecord
>>
outputs
,
String
channelID
)
{
this
.
timeOfLastUpdate
=
System
.
currentTimeMillis
();
...
...
@@ -53,6 +63,9 @@ public class FaultTolerancyBuffer {
this
.
recordTimestamps
=
new
HashMap
<
String
,
Long
>();
}
/** Adds the record to the fault tolerance buffer. This record will be monitored for acknowledgements and timeout.
*
*/
public
void
addRecord
(
StreamRecord
streamRecord
)
{
recordBuffer
.
put
(
streamRecord
.
getId
(),
streamRecord
);
...
...
@@ -60,6 +73,13 @@ public class FaultTolerancyBuffer {
addTimestamp
(
streamRecord
.
getId
());
}
/** Checks for records that have timed out since the last check and fails them.
*
* @param currentTime
* Time when the check should be made, usually current system time.
* @return
* Returns the list of the records that have timed out.
*/
List
<
String
>
timeoutRecords
(
Long
currentTime
)
{
if
(
timeOfLastUpdate
+
TIMEOUT
<
currentTime
)
{
List
<
String
>
timedOutRecords
=
new
LinkedList
<
String
>();
...
...
@@ -85,6 +105,14 @@ public class FaultTolerancyBuffer {
return
null
;
}
/**Stores time stamp for a record by recordID
* and also adds the record to a map which maps a time stamp to the IDs of records that were emitted at that time.
* <p>
* Later used for timeouts.
*
* @param recordID
* ID of the record
*/
public
void
addTimestamp
(
String
recordID
)
{
Long
currentTime
=
System
.
currentTimeMillis
();
recordTimestamps
.
put
(
recordID
,
currentTime
);
...
...
@@ -98,6 +126,11 @@ public class FaultTolerancyBuffer {
}
}
/**Returns a StreamRecord after removing it from the buffer
*
* @param recordID
* The ID of the record that will be popped
*/
public
StreamRecord
popRecord
(
String
recordID
)
{
System
.
out
.
println
(
"Pop ID: "
+
recordID
);
StreamRecord
record
=
recordBuffer
.
get
(
recordID
);
...
...
@@ -105,6 +138,11 @@ public class FaultTolerancyBuffer {
return
record
;
}
/** Removes a StreamRecord by ID from the fault tolerance buffer, further acks will have no effects for this record.
* @param recordID
* The ID of the record that will be removed
*
*/
void
removeRecord
(
String
recordID
)
{
recordBuffer
.
remove
(
recordID
);
ackCounter
.
remove
(
recordID
);
...
...
@@ -119,6 +157,11 @@ public class FaultTolerancyBuffer {
}
}
/**Acknowledges the record of the given ID, if all the outputs have sent acknowledgments, removes it from the buffer
*
* @param recordID
* ID of the record that has been acknowledged
*/
//TODO: find a place to call timeoutRecords
public
void
ackRecord
(
String
recordID
)
{
if
(
ackCounter
.
containsKey
(
recordID
))
{
...
...
@@ -133,6 +176,11 @@ public class FaultTolerancyBuffer {
//timeoutRecords(System.currentTimeMillis());
}
/**Re-emits the failed record for the given ID, removes the old record and stores it with a new ID.
*
* @param recordID
* ID of the record that has been failed
*/
public
void
failRecord
(
String
recordID
)
{
// Create new id to avoid double counting acks
System
.
out
.
println
(
"Fail ID: "
+
recordID
);
...
...
@@ -140,7 +188,12 @@ public class FaultTolerancyBuffer {
addRecord
(
newRecord
);
reEmit
(
newRecord
);
}
/**
* Emit give record to all output channels
* @param record
* Record to be re-emitted
*/
public
void
reEmit
(
StreamRecord
record
)
{
for
(
RecordWriter
<
StreamRecord
>
output
:
outputs
)
{
try
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录