Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
a34869c0
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
a34869c0
编写于
2月 20, 2015
作者:
P
Paris Carbone
提交者:
Stephan Ewen
3月 10, 2015
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-1638] [streaming] Fault tolerance logic in JM and TM
上级
5061edb8
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
48 addition
and
6 deletion
+48
-6
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
...cala/org/apache/flink/runtime/jobmanager/JobManager.scala
+32
-5
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
...la/org/apache/flink/runtime/taskmanager/TaskManager.scala
+16
-1
未找到文件。
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
浏览文件 @
a34869c0
...
...
@@ -41,7 +41,7 @@ import org.apache.flink.runtime.ActorLogMessages
import
org.apache.flink.runtime.akka.AkkaUtils
import
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager
import
org.apache.flink.runtime.instance.InstanceManager
import
org.apache.flink.runtime.jobgraph.
{
JobGraph
,
JobStatus
,
JobID
}
import
org.apache.flink.runtime.jobgraph.
{
ScheduleMode
,
JobGraph
,
JobStatus
,
JobID
}
import
org.apache.flink.runtime.jobmanager.accumulators.AccumulatorManager
import
org.apache.flink.runtime.jobmanager.scheduler.
{
Scheduler
=>
FlinkScheduler
}
import
org.apache.flink.runtime.messages.JobManagerMessages._
...
...
@@ -97,12 +97,14 @@ class JobManager(val configuration: Configuration,
val
delayBetweenRetries
:
Long
,
val
timeout
:
FiniteDuration
)
extends
Actor
with
ActorLogMessages
with
ActorLogging
{
/** Reference to the log, for debugging */
protected
val
LOG
=
JobManager
.
LOG
val
LOG
=
JobManager
.
LOG
/** List of current jobs running jobs */
protected
val
currentJobs
=
scala
.
collection
.
mutable
.
HashMap
[
JobID
,
(
ExecutionGraph
,
JobInfo
)]()
val
currentJobs
=
scala
.
collection
.
mutable
.
HashMap
[
JobID
,
(
ExecutionGraph
,
JobInfo
)]()
val
barrierMonitors
=
scala
.
collection
.
mutable
.
HashMap
[
JobID
,
ActorRef
]()
/**
* Run when the job manager is started. Simply logs an informational message.
...
...
@@ -299,13 +301,38 @@ class JobManager(val configuration: Configuration,
jobInfo
.
client
!
Failure
(
exception
)
throw
exception
}
barrierMonitors
.
get
(
jobID
)
match
{
case
Some
(
monitor
)
=>
monitor
!
PoisonPill
barrierMonitors
.
remove
(
jobID
)
case
None
=>
}
removeJob
(
jobID
)
}
else
{
newJobStatus
match
{
case
JobStatus
.
RUNNING
=>
currentJobs
.
get
(
jobID
)
match
{
case
Some
((
executionGraph
,
_
))
=>
//FIXME this is just a fast n dirty check for determining streaming jobs
if
(
executionGraph
.
getScheduleMode
==
ScheduleMode
.
ALL
)
{
barrierMonitors
+=
jobID
->
StreamStateMonitor
.
props
(
context
,
executionGraph
)
}
case
None
=>
log
.
error
(
"Cannot create state monitor for job ID {}."
,
jobID
)
new
IllegalStateException
(
"Cannot find execution graph for job ID "
+
jobID
)
}
}
}
case
None
=>
removeJob
(
jobID
)
}
case
BarrierAck
(
jobID
,
jobVertex
,
instanceID
,
checkpoint
)
=>
barrierMonitors
.
get
(
jobID
)
match
{
case
Some
(
monitor
)
=>
monitor
!
BarrierAck
(
jobID
,
jobVertex
,
instanceID
,
checkpoint
)
case
None
=>
}
case
ScheduleOrUpdateConsumers
(
jobId
,
executionId
,
partitionIndex
)
=>
currentJobs
.
get
(
jobId
)
match
{
case
Some
((
executionGraph
,
_
))
=>
...
...
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
浏览文件 @
a34869c0
...
...
@@ -43,7 +43,8 @@ import org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync
import
org.apache.flink.runtime.io.network.NetworkEnvironment
import
org.apache.flink.runtime.io.network.netty.NettyConfig
import
org.apache.flink.runtime.jobgraph.IntermediateDataSetID
import
org.apache.flink.runtime.jobmanager.JobManager
import
org.apache.flink.runtime.jobgraph.tasks.BarrierTransceiver
import
org.apache.flink.runtime.jobmanager.
{
BarrierReq
,
JobManager
}
import
org.apache.flink.runtime.memorymanager.DefaultMemoryManager
import
org.apache.flink.runtime.messages.JobManagerMessages.UpdateTaskExecutionState
import
org.apache.flink.runtime.messages.Messages.
{
Disconnect
,
Acknowledge
}
...
...
@@ -349,6 +350,20 @@ class TaskManager(val connectionInfo: InstanceConnectionInfo,
networkEnvironment
foreach
{
_
.
getPartitionManager
.
failIntermediateResultPartitions
(
executionID
)
}
case
BarrierReq
(
attemptID
,
checkpointID
)
=>
log
.
debug
(
"[FT-TaskManager] Barrier request received for attempt {}"
,
attemptID
)
runningTasks
.
get
(
attemptID
)
match
{
case
Some
(
i
)
=>
if
(
i
.
getExecutionState
==
ExecutionState
.
RUNNING
)
{
i
.
getEnvironment
.
getInvokable
match
{
case
barrierTransceiver
:
BarrierTransceiver
=>
barrierTransceiver
.
broadcastBarrier
(
checkpointID
)
case
_
=>
log
.
error
(
"[FT-TaskManager] Received a barrier for the wrong vertex"
)
}
}
case
None
=>
log
.
error
(
"[FT-TaskManager] Received a barrier for an unknown vertex"
)
}
}
/**
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录