Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
wk1234123
DolphinScheduler
提交
433b41dd
DolphinScheduler
项目概览
wk1234123
/
DolphinScheduler
与 Fork 源项目一致
Fork自
apache / DolphinScheduler
通知
5
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
DolphinScheduler
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
433b41dd
编写于
4月 07, 2020
作者:
Q
qiaozhanwei
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
refactor-worker merge to dev bug fix
上级
94713cef
变更
8
隐藏空白更改
内联
并排
Showing
8 changed file
with
110 addition
and
38 deletion
+110
-38
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java
...org/apache/dolphinscheduler/common/utils/HadoopUtils.java
+1
-2
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java
.../org/apache/dolphinscheduler/dao/entity/TaskInstance.java
+17
-1
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/builder/TaskExecutionContextBuilder.java
...scheduler/server/builder/TaskExecutionContextBuilder.java
+1
-0
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/entity/TaskExecutionContext.java
.../dolphinscheduler/server/entity/TaskExecutionContext.java
+14
-0
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumer.java
...ler/server/master/consumer/TaskPriorityQueueConsumer.java
+46
-7
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
...phinscheduler/server/worker/runner/TaskExecuteThread.java
+1
-3
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
...ache/dolphinscheduler/service/process/ProcessService.java
+11
-1
pom.xml
pom.xml
+19
-24
未找到文件。
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java
浏览文件 @
433b41dd
...
...
@@ -91,8 +91,7 @@ public class HadoopUtils implements Closeable {
*/
private
void
initHdfsPath
()
{
String
hdfsPath
=
PropertyUtils
.
getString
(
resourceUploadPath
);
Path
path
=
new
Path
(
hdfsPath
);
Path
path
=
new
Path
(
resourceUploadPath
);
try
{
if
(!
fs
.
exists
(
path
))
{
...
...
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java
浏览文件 @
433b41dd
...
...
@@ -29,6 +29,7 @@ import com.baomidou.mybatisplus.annotation.TableName;
import
java.io.Serializable
;
import
java.util.Date
;
import
java.util.List
;
/**
* task instance
...
...
@@ -47,6 +48,8 @@ public class TaskInstance implements Serializable {
*/
private
String
name
;
/**
* task type
*/
...
...
@@ -205,8 +208,12 @@ public class TaskInstance implements Serializable {
private
String
executorName
;
@TableField
(
exist
=
false
)
private
List
<
String
>
resources
;
public
void
init
(
String
host
,
Date
startTime
,
String
executePath
){
public
void
init
(
String
host
,
Date
startTime
,
String
executePath
){
this
.
host
=
host
;
this
.
startTime
=
startTime
;
this
.
executePath
=
executePath
;
...
...
@@ -446,6 +453,15 @@ public class TaskInstance implements Serializable {
||
this
.
getState
().
typeIsCancel
()
||
(
this
.
getState
().
typeIsFailure
()
&&
!
taskCanRetry
());
}
public
List
<
String
>
getResources
()
{
return
resources
;
}
public
void
setResources
(
List
<
String
>
resources
)
{
this
.
resources
=
resources
;
}
/**
* determine if you can try again
* @return can try result
...
...
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/builder/TaskExecutionContextBuilder.java
浏览文件 @
433b41dd
...
...
@@ -50,6 +50,7 @@ public class TaskExecutionContextBuilder {
taskExecutionContext
.
setTaskJson
(
taskInstance
.
getTaskJson
());
taskExecutionContext
.
setWorkerGroup
(
taskInstance
.
getWorkerGroup
());
taskExecutionContext
.
setHost
(
taskInstance
.
getHost
());
taskExecutionContext
.
setResources
(
taskInstance
.
getResources
());
return
this
;
}
...
...
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/entity/TaskExecutionContext.java
浏览文件 @
433b41dd
...
...
@@ -23,6 +23,7 @@ import org.apache.dolphinscheduler.remote.utils.FastJsonSerializer;
import
java.io.Serializable
;
import
java.util.Date
;
import
java.util.List
;
import
java.util.Map
;
/**
...
...
@@ -166,6 +167,10 @@ public class TaskExecutionContext implements Serializable{
*/
private
String
workerGroup
;
/**
* resources full name
*/
private
List
<
String
>
resources
;
/**
* sql TaskExecutionContext
...
...
@@ -433,6 +438,14 @@ public class TaskExecutionContext implements Serializable{
this
.
dependenceTaskExecutionContext
=
dependenceTaskExecutionContext
;
}
public
List
<
String
>
getResources
()
{
return
resources
;
}
public
void
setResources
(
List
<
String
>
resources
)
{
this
.
resources
=
resources
;
}
@Override
public
String
toString
()
{
return
"TaskExecutionContext{"
+
...
...
@@ -462,6 +475,7 @@ public class TaskExecutionContext implements Serializable{
", taskTimeoutStrategy="
+
taskTimeoutStrategy
+
", taskTimeout="
+
taskTimeout
+
", workerGroup='"
+
workerGroup
+
'\''
+
", resources="
+
resources
+
", sqlTaskExecutionContext="
+
sqlTaskExecutionContext
+
", dataxTaskExecutionContext="
+
dataxTaskExecutionContext
+
", dependenceTaskExecutionContext="
+
dependenceTaskExecutionContext
+
...
...
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumer.java
浏览文件 @
433b41dd
...
...
@@ -22,6 +22,8 @@ import org.apache.dolphinscheduler.common.enums.ExecutionStatus;
import
org.apache.dolphinscheduler.common.enums.TaskType
;
import
org.apache.dolphinscheduler.common.enums.UdfType
;
import
org.apache.dolphinscheduler.common.model.TaskNode
;
import
org.apache.dolphinscheduler.common.process.ResourceInfo
;
import
org.apache.dolphinscheduler.common.task.AbstractParameters
;
import
org.apache.dolphinscheduler.common.task.datax.DataxParameters
;
import
org.apache.dolphinscheduler.common.task.procedure.ProcedureParameters
;
import
org.apache.dolphinscheduler.common.task.sql.SqlParameters
;
...
...
@@ -29,10 +31,8 @@ import org.apache.dolphinscheduler.common.thread.Stopper;
import
org.apache.dolphinscheduler.common.utils.EnumUtils
;
import
org.apache.dolphinscheduler.common.utils.FileUtils
;
import
org.apache.dolphinscheduler.common.utils.StringUtils
;
import
org.apache.dolphinscheduler.dao.entity.DataSource
;
import
org.apache.dolphinscheduler.dao.entity.TaskInstance
;
import
org.apache.dolphinscheduler.dao.entity.Tenant
;
import
org.apache.dolphinscheduler.dao.entity.UdfFunc
;
import
org.apache.dolphinscheduler.common.utils.TaskParametersUtils
;
import
org.apache.dolphinscheduler.dao.entity.*
;
import
org.apache.dolphinscheduler.server.builder.TaskExecutionContextBuilder
;
import
org.apache.dolphinscheduler.server.entity.*
;
import
org.apache.dolphinscheduler.server.master.dispatch.ExecutorDispatcher
;
...
...
@@ -47,7 +47,12 @@ import org.springframework.beans.factory.annotation.Autowired;
import
org.springframework.stereotype.Component
;
import
javax.annotation.PostConstruct
;
import
java.util.ArrayList
;
import
java.util.HashSet
;
import
java.util.List
;
import
java.util.Set
;
import
java.util.stream.Collectors
;
import
java.util.stream.Stream
;
/**
* TaskUpdateQueue consumer
...
...
@@ -127,6 +132,12 @@ public class TaskPriorityQueueConsumer extends Thread{
protected
TaskExecutionContext
getTaskExecutionContext
(
int
taskInstanceId
){
TaskInstance
taskInstance
=
processService
.
getTaskInstanceDetailByTaskId
(
taskInstanceId
);
// task type
TaskType
taskType
=
TaskType
.
valueOf
(
taskInstance
.
getTaskType
());
// task node
TaskNode
taskNode
=
JSONObject
.
parseObject
(
taskInstance
.
getTaskJson
(),
TaskNode
.
class
);
Integer
userId
=
taskInstance
.
getProcessDefine
()
==
null
?
0
:
taskInstance
.
getProcessDefine
().
getUserId
();
Tenant
tenant
=
processService
.
getTenantForProcess
(
taskInstance
.
getProcessInstance
().
getTenantId
(),
userId
);
...
...
@@ -145,14 +156,14 @@ public class TaskPriorityQueueConsumer extends Thread{
taskInstance
.
getProcessInstance
().
setQueue
(
StringUtils
.
isEmpty
(
userQueue
)
?
tenant
.
getQueue
()
:
userQueue
);
taskInstance
.
getProcessInstance
().
setTenantCode
(
tenant
.
getTenantCode
());
taskInstance
.
setExecutePath
(
getExecLocalPath
(
taskInstance
));
taskInstance
.
setResources
(
getResourceFullNames
(
taskNode
));
SQLTaskExecutionContext
sqlTaskExecutionContext
=
new
SQLTaskExecutionContext
();
DataxTaskExecutionContext
dataxTaskExecutionContext
=
new
DataxTaskExecutionContext
();
ProcedureTaskExecutionContext
procedureTaskExecutionContext
=
new
ProcedureTaskExecutionContext
();
TaskType
taskType
=
TaskType
.
valueOf
(
taskInstance
.
getTaskType
());
TaskNode
taskNode
=
JSONObject
.
parseObject
(
taskInstance
.
getTaskJson
(),
TaskNode
.
class
);
// SQL task
if
(
taskType
==
TaskType
.
SQL
){
setSQLTaskRelation
(
sqlTaskExecutionContext
,
taskNode
);
...
...
@@ -171,7 +182,6 @@ public class TaskPriorityQueueConsumer extends Thread{
}
return
TaskExecutionContextBuilder
.
get
()
.
buildTaskInstanceRelatedInfo
(
taskInstance
)
.
buildProcessInstanceRelatedInfo
(
taskInstance
.
getProcessInstance
())
...
...
@@ -270,4 +280,33 @@ public class TaskPriorityQueueConsumer extends Thread{
}
return
false
;
}
/**
* create project resource files
*/
private
List
<
String
>
getResourceFullNames
(
TaskNode
taskNode
){
Set
<
Integer
>
resourceIdsSet
=
new
HashSet
<>();
AbstractParameters
baseParam
=
TaskParametersUtils
.
getParameters
(
taskNode
.
getType
(),
taskNode
.
getParams
());
if
(
baseParam
!=
null
)
{
List
<
ResourceInfo
>
projectResourceFiles
=
baseParam
.
getResourceFilesList
();
if
(
projectResourceFiles
!=
null
)
{
Stream
<
Integer
>
resourceInfotream
=
projectResourceFiles
.
stream
().
map
(
resourceInfo
->
resourceInfo
.
getId
());
resourceIdsSet
.
addAll
(
resourceInfotream
.
collect
(
Collectors
.
toSet
()));
}
}
Integer
[]
resourceIds
=
resourceIdsSet
.
toArray
(
new
Integer
[
resourceIdsSet
.
size
()]);
List
<
Resource
>
resources
=
processService
.
listResourceByIds
(
resourceIds
);
List
<
String
>
resourceFullNames
=
resources
.
stream
()
.
map
(
resourceInfo
->
resourceInfo
.
getFullName
())
.
collect
(
Collectors
.
toList
());
return
resourceFullNames
;
}
}
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
浏览文件 @
433b41dd
...
...
@@ -83,11 +83,9 @@ public class TaskExecuteThread implements Runnable {
// task node
TaskNode
taskNode
=
JSONObject
.
parseObject
(
taskExecutionContext
.
getTaskJson
(),
TaskNode
.
class
);
// get resource files
List
<
String
>
resourceFiles
=
createProjectResFiles
(
taskNode
);
// copy hdfs/minio file to local
downloadResource
(
taskExecutionContext
.
getExecutePath
(),
resourceFiles
,
taskExecutionContext
.
getResources
()
,
taskExecutionContext
.
getTenantCode
(),
logger
);
...
...
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
浏览文件 @
433b41dd
...
...
@@ -1487,7 +1487,7 @@ public class ProcessService {
* @return tenant code
*/
public
String
queryTenantCodeByResName
(
String
resName
,
ResourceType
resourceType
){
return
resourceMapper
.
queryTenantCodeByResourceName
(
resName
,
resourceType
.
ordinal
());
return
resourceMapper
.
queryTenantCodeByResourceName
(
resName
,
resourceType
.
ordinal
());
}
/**
...
...
@@ -1766,4 +1766,14 @@ public class ProcessService {
}
/**
* list resources by ids
* @param resIds resIds
* @return resource list
*/
public
List
<
Resource
>
listResourceByIds
(
Integer
[]
resIds
){
return
resourceMapper
.
listResourceByIds
(
resIds
);
}
}
pom.xml
浏览文件 @
433b41dd
...
...
@@ -691,25 +691,25 @@
<include>
**/alert/utils/JSONUtilsTest.java
</include>
<include>
**/alert/utils/MailUtilsTest.java
</include>
<include>
**/alert/utils/PropertyUtilsTest.java
</include>
<
include>
**/api/controller/AccessTokenControllerTest.java
</include
>
<
include>
**/api/controller/AlertGroupControllerTest.java
</include
>
<
include>
**/api/controller/DataAnalysisControllerTest.java
</include
>
<
include>
**/api/controller/DataSourceControllerTest.java
</include
>
<
include>
**/api/controller/ExecutorControllerTest.java
</include
>
<
include>
**/api/controller/LoggerControllerTest.java
</include
>
<
include>
**/api/controller/LoginControllerTest.java
</include
>
<
include>
**/api/controller/MonitorControllerTest.java
</include
>
<
include>
**/api/controller/ProcessDefinitionControllerTest.java
</include
>
<
include>
**/api/controller/ProcessInstanceControllerTest.java
</include
>
<
include>
**/api/controller/ProjectControllerTest.java
</include
>
<
include>
**/api/controller/QueueControllerTest.java
</include
>
<
include>
**/api/controller/ResourcesControllerTest.java
</include
>
<
include>
**/api/controller/SchedulerControllerTest.java
</include
>
<
include>
**/api/controller/TaskInstanceControllerTest.java
</include
>
<
include>
**/api/controller/TaskRecordControllerTest.java
</include
>
<
include>
**/api/controller/TenantControllerTest.java
</include
>
<
include>
**/api/controller/UsersControllerTest.java
</include
>
<
include>
**/api/controller/WorkerGroupControllerTest.java
</include
>
<
!--<include>**/api/controller/AccessTokenControllerTest.java</include>--
>
<
!--<include>**/api/controller/AlertGroupControllerTest.java</include>--
>
<
!--<include>**/api/controller/DataAnalysisControllerTest.java</include>--
>
<
!--<include>**/api/controller/DataSourceControllerTest.java</include>--
>
<
!--<include>**/api/controller/ExecutorControllerTest.java</include>--
>
<
!--<include>**/api/controller/LoggerControllerTest.java</include>--
>
<
!--<include>**/api/controller/LoginControllerTest.java</include>--
>
<
!--<include>**/api/controller/MonitorControllerTest.java</include>--
>
<
!--<include>**/api/controller/ProcessDefinitionControllerTest.java</include>--
>
<
!--<include>**/api/controller/ProcessInstanceControllerTest.java</include>--
>
<
!--<include>**/api/controller/ProjectControllerTest.java</include>--
>
<
!--<include>**/api/controller/QueueControllerTest.java</include>--
>
<
!--<include>**/api/controller/ResourcesControllerTest.java</include>--
>
<
!--<include>**/api/controller/SchedulerControllerTest.java</include>--
>
<
!--<include>**/api/controller/TaskInstanceControllerTest.java</include>--
>
<
!--<include>**/api/controller/TaskRecordControllerTest.java</include>--
>
<
!--<include>**/api/controller/TenantControllerTest.java</include>--
>
<
!--<include>**/api/controller/UsersControllerTest.java</include>--
>
<
!--<include>**/api/controller/WorkerGroupControllerTest.java</include>--
>
<include>
**/api/dto/resources/filter/ResourceFilterTest.java
</include>
<include>
**/api/dto/resources/visitor/ResourceTreeVisitorTest.java
</include>
<include>
**/api/enums/testGetEnum.java
</include>
...
...
@@ -783,11 +783,6 @@
<include>
**/common/utils/StringUtilsTest.java
</include>
<include>
**/common/utils/TaskParametersUtilsTest.java
</include>
<include>
**/common/ConstantsTest.java
</include>
<include>
**/dao/mapper/AccessTokenMapperTest.java
</include>
<include>
**/dao/mapper/AlertGroupMapperTest.java
</include>
<include>
**/dao/mapper/AlertMapperTest.java
</include>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录