Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
6ffe40dc
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
提交
6ffe40dc
编写于
8月 28, 2014
作者:
X
xvrl
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #712 from metamx/indexing-extra-classpaths
Allow indexing tasks to specify extra classpaths.
上级
578ca64e
8bcb73a0
变更
5
隐藏空白更改
内联
并排
Showing
5 changed file
with
39 addition
and
3 deletion
+39
-3
indexing-service/src/main/java/io/druid/indexing/common/task/AbstractTask.java
...main/java/io/druid/indexing/common/task/AbstractTask.java
+6
-0
indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
...n/java/io/druid/indexing/common/task/HadoopIndexTask.java
+13
-1
indexing-service/src/main/java/io/druid/indexing/common/task/Task.java
...ice/src/main/java/io/druid/indexing/common/task/Task.java
+6
-0
indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
...in/java/io/druid/indexing/overlord/ForkingTaskRunner.java
+10
-1
indexing-service/src/test/java/io/druid/indexing/common/task/TaskSerdeTest.java
...est/java/io/druid/indexing/common/task/TaskSerdeTest.java
+4
-1
未找到文件。
indexing-service/src/main/java/io/druid/indexing/common/task/AbstractTask.java
浏览文件 @
6ffe40dc
...
...
@@ -107,6 +107,12 @@ public abstract class AbstractTask implements Task
return
null
;
}
@Override
public
String
getClasspathPrefix
()
{
return
null
;
}
@Override
public
String
toString
()
{
...
...
indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
浏览文件 @
6ffe40dc
...
...
@@ -79,6 +79,8 @@ public class HadoopIndexTask extends AbstractTask
private
final
HadoopIngestionSpec
spec
;
@JsonIgnore
private
final
List
<
String
>
hadoopDependencyCoordinates
;
@JsonIgnore
private
final
String
classpathPrefix
;
/**
* @param spec is used by the HadoopDruidIndexerJob to set up the appropriate parameters
...
...
@@ -96,7 +98,8 @@ public class HadoopIndexTask extends AbstractTask
@JsonProperty
(
"spec"
)
HadoopIngestionSpec
spec
,
@JsonProperty
(
"config"
)
HadoopIngestionSpec
config
,
// backwards compat
@JsonProperty
(
"hadoopCoordinates"
)
String
hadoopCoordinates
,
@JsonProperty
(
"hadoopDependencyCoordinates"
)
List
<
String
>
hadoopDependencyCoordinates
@JsonProperty
(
"hadoopDependencyCoordinates"
)
List
<
String
>
hadoopDependencyCoordinates
,
@JsonProperty
(
"classpathPrefix"
)
String
classpathPrefix
)
{
super
(
...
...
@@ -123,6 +126,8 @@ public class HadoopIndexTask extends AbstractTask
// Will be defaulted to something at runtime, based on taskConfig.
this
.
hadoopDependencyCoordinates
=
null
;
}
this
.
classpathPrefix
=
classpathPrefix
;
}
@Override
...
...
@@ -159,6 +164,13 @@ public class HadoopIndexTask extends AbstractTask
return
hadoopDependencyCoordinates
;
}
@JsonProperty
@Override
public
String
getClasspathPrefix
()
{
return
classpathPrefix
;
}
@SuppressWarnings
(
"unchecked"
)
@Override
public
TaskStatus
run
(
TaskToolbox
toolbox
)
throws
Exception
...
...
indexing-service/src/main/java/io/druid/indexing/common/task/Task.java
浏览文件 @
6ffe40dc
...
...
@@ -98,6 +98,12 @@ public interface Task
*/
public
<
T
>
QueryRunner
<
T
>
getQueryRunner
(
Query
<
T
>
query
);
/**
* Returns an extra classpath that should be prepended to the default classpath when running this task. If no
* extra classpath should be prepended, this should return null or the empty string.
*/
public
String
getClasspathPrefix
();
/**
* Execute preflight actions for a task. This can be used to acquire locks, check preconditions, and so on. The
* actions must be idempotent, since this method may be executed multiple times. This typically runs on the
...
...
indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
浏览文件 @
6ffe40dc
...
...
@@ -161,10 +161,19 @@ public class ForkingTaskRunner implements TaskRunner, TaskLogStreamer
final
List
<
String
>
command
=
Lists
.
newArrayList
();
final
String
childHost
=
String
.
format
(
"%s:%d"
,
node
.
getHostNoPort
(),
childPort
);
final
String
taskClasspath
;
if
(
task
.
getClasspathPrefix
()
!=
null
&&
!
task
.
getClasspathPrefix
().
isEmpty
())
{
taskClasspath
=
Joiner
.
on
(
File
.
pathSeparator
).
join
(
task
.
getClasspathPrefix
(),
config
.
getClasspath
()
);
}
else
{
taskClasspath
=
config
.
getClasspath
();
}
command
.
add
(
config
.
getJavaCommand
());
command
.
add
(
"-cp"
);
command
.
add
(
config
.
getClasspath
()
);
command
.
add
(
taskClasspath
);
Iterables
.
addAll
(
command
,
whiteSpaceSplitter
.
split
(
config
.
getJavaOpts
()));
...
...
indexing-service/src/test/java/io/druid/indexing/common/task/TaskSerdeTest.java
浏览文件 @
6ffe40dc
...
...
@@ -427,7 +427,8 @@ public class TaskSerdeTest
null
),
null
,
null
null
,
"blah"
);
final
String
json
=
jsonMapper
.
writeValueAsString
(
task
);
...
...
@@ -442,5 +443,7 @@ public class TaskSerdeTest
task
.
getSpec
().
getTuningConfig
().
getJobProperties
(),
task2
.
getSpec
().
getTuningConfig
().
getJobProperties
()
);
Assert
.
assertEquals
(
"blah"
,
task
.
getClasspathPrefix
());
Assert
.
assertEquals
(
"blah"
,
task2
.
getClasspathPrefix
());
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录