Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
lhongjum2003
DolphinScheduler
提交
09dec3b7
DolphinScheduler
项目概览
lhongjum2003
/
DolphinScheduler
与 Fork 源项目一致
Fork自
apache / DolphinScheduler
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
DolphinScheduler
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
09dec3b7
编写于
11月 17, 2019
作者:
B
Baoqi Wu
提交者:
qiaozhanwei
11月 16, 2019
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix #1245, make scanCommand transactional (#1246)
上级
a1375922
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
36 addition
and
39 deletion
+36
-39
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/ProcessDao.java
...main/java/org/apache/dolphinscheduler/dao/ProcessDao.java
+21
-35
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterSchedulerThread.java
...scheduler/server/master/runner/MasterSchedulerThread.java
+15
-4
未找到文件。
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/ProcessDao.java
浏览文件 @
09dec3b7
...
...
@@ -109,58 +109,44 @@ public class ProcessDao {
/**
*
find one command from command queue, construct process instance
*
handle Command (construct ProcessInstance from Command) , wrapped in transaction
* @param logger logger
* @param host host
* @param validThreadNum validThreadNum
* @param command found command
* @return process instance
*/
@Transactional
(
rollbackFor
=
Exception
.
class
)
public
ProcessInstance
scanCommand
(
Logger
logger
,
String
host
,
int
validThreadNum
){
ProcessInstance
processInstance
=
null
;
Command
command
=
findOneCommand
();
if
(
command
==
null
)
{
public
ProcessInstance
handleCommand
(
Logger
logger
,
String
host
,
int
validThreadNum
,
Command
command
)
{
ProcessInstance
processInstance
=
constructProcessInstance
(
command
,
host
);
//cannot construct process instance, return null;
if
(
processInstance
==
null
){
logger
.
error
(
"scan command, command parameter is error: %s"
,
command
.
toString
());
moveToErrorCommand
(
command
,
"process instance is null"
);
return
null
;
}
logger
.
info
(
String
.
format
(
"find one command: id: %d, type: %s"
,
command
.
getId
(),
command
.
getCommandType
().
toString
()));
try
{
processInstance
=
constructProcessInstance
(
command
,
host
);
//cannot construct process instance, return null;
if
(
processInstance
==
null
){
logger
.
error
(
"scan command, command parameter is error: %s"
,
command
.
toString
());
delCommandByid
(
command
.
getId
());
saveErrorCommand
(
command
,
"process instance is null"
);
return
null
;
}
if
(!
checkThreadNum
(
command
,
validThreadNum
)){
logger
.
info
(
"there is not enough thread for this command: {}"
,
command
.
toString
()
);
return
setWaitingThreadProcess
(
command
,
processInstance
);
}
processInstance
.
setCommandType
(
command
.
getCommandType
());
processInstance
.
addHistoryCmd
(
command
.
getCommandType
());
saveProcessInstance
(
processInstance
);
this
.
setSubProcessParam
(
processInstance
);
delCommandByid
(
command
.
getId
());
return
processInstance
;
}
catch
(
Exception
e
){
logger
.
error
(
"scan command error "
,
e
);
saveErrorCommand
(
command
,
e
.
toString
());
delCommandByid
(
command
.
getId
());
if
(!
checkThreadNum
(
command
,
validThreadNum
)){
logger
.
info
(
"there is not enough thread for this command: {}"
,
command
.
toString
()
);
return
setWaitingThreadProcess
(
command
,
processInstance
);
}
return
null
;
processInstance
.
setCommandType
(
command
.
getCommandType
());
processInstance
.
addHistoryCmd
(
command
.
getCommandType
());
saveProcessInstance
(
processInstance
);
this
.
setSubProcessParam
(
processInstance
);
delCommandByid
(
command
.
getId
());
return
processInstance
;
}
/**
* save error command
* save error command
, and delete original command
* @param command command
* @param message message
*/
private
void
saveErrorCommand
(
Command
command
,
String
message
)
{
@Transactional
(
rollbackFor
=
Exception
.
class
)
public
void
moveToErrorCommand
(
Command
command
,
String
message
)
{
ErrorCommand
errorCommand
=
new
ErrorCommand
(
command
,
message
);
this
.
errorCommandMapper
.
insert
(
errorCommand
);
delCommandByid
(
command
.
getId
());
}
/**
...
...
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterSchedulerThread.java
浏览文件 @
09dec3b7
...
...
@@ -22,6 +22,7 @@ import org.apache.dolphinscheduler.common.thread.ThreadUtils;
import
org.apache.dolphinscheduler.common.utils.OSUtils
;
import
org.apache.dolphinscheduler.common.zk.AbstractZKClient
;
import
org.apache.dolphinscheduler.dao.ProcessDao
;
import
org.apache.dolphinscheduler.dao.entity.Command
;
import
org.apache.dolphinscheduler.dao.entity.ProcessInstance
;
import
org.apache.dolphinscheduler.server.zk.ZKMasterClient
;
import
org.apache.commons.configuration.Configuration
;
...
...
@@ -108,10 +109,20 @@ public class MasterSchedulerThread implements Runnable {
ThreadPoolExecutor
poolExecutor
=
(
ThreadPoolExecutor
)
masterExecService
;
int
activeCount
=
poolExecutor
.
getActiveCount
();
// make sure to scan and delete command table in one transaction
processInstance
=
processDao
.
scanCommand
(
logger
,
OSUtils
.
getHost
(),
this
.
masterExecThreadNum
-
activeCount
);
if
(
processInstance
!=
null
)
{
logger
.
info
(
"start master exex thread , split DAG ..."
);
masterExecService
.
execute
(
new
MasterExecThread
(
processInstance
,
processDao
));
Command
command
=
processDao
.
findOneCommand
();
if
(
command
!=
null
)
{
logger
.
info
(
String
.
format
(
"find one command: id: %d, type: %s"
,
command
.
getId
(),
command
.
getCommandType
().
toString
()));
try
{
processInstance
=
processDao
.
handleCommand
(
logger
,
OSUtils
.
getHost
(),
this
.
masterExecThreadNum
-
activeCount
,
command
);
if
(
processInstance
!=
null
)
{
logger
.
info
(
"start master exec thread , split DAG ..."
);
masterExecService
.
execute
(
new
MasterExecThread
(
processInstance
,
processDao
));
}
}
catch
(
Exception
e
){
logger
.
error
(
"scan command error "
,
e
);
processDao
.
moveToErrorCommand
(
command
,
e
.
toString
());
}
}
}
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录