Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
DolphinScheduler
提交
6609b912
DolphinScheduler
项目概览
apache
/
DolphinScheduler
上一次同步 接近 2 年
通知
707
Star
9572
Fork
3514
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
DolphinScheduler
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
6609b912
编写于
12月 11, 2020
作者:
E
eights
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
add target-dir param in sqoop task hive TargetHiveParameter
上级
0420fe54
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
20 addition
and
2 deletion
+20
-2
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/task/sqoop/targets/TargetHiveParameter.java
...eduler/common/task/sqoop/targets/TargetHiveParameter.java
+13
-0
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sqoop/generator/targets/HiveTargetGenerator.java
...ker/task/sqoop/generator/targets/HiveTargetGenerator.java
+5
-0
dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/sqoop/SqoopTaskTest.java
...phinscheduler/server/worker/task/sqoop/SqoopTaskTest.java
+2
-2
未找到文件。
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/task/sqoop/targets/TargetHiveParameter.java
浏览文件 @
6609b912
...
...
@@ -41,6 +41,11 @@ public class TargetHiveParameter {
* hive overwrite
*/
private
boolean
hiveOverWrite
;
/**
* hive target dir
*/
private
String
hiveTargetDir
;
/**
* replace delimiter
*/
...
...
@@ -117,4 +122,12 @@ public class TargetHiveParameter {
public
void
setHivePartitionValue
(
String
hivePartitionValue
)
{
this
.
hivePartitionValue
=
hivePartitionValue
;
}
public
String
getHiveTargetDir
()
{
return
hiveTargetDir
;
}
public
void
setHiveTargetDir
(
String
hiveTargetDir
)
{
this
.
hiveTargetDir
=
hiveTargetDir
;
}
}
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sqoop/generator/targets/HiveTargetGenerator.java
浏览文件 @
6609b912
...
...
@@ -68,6 +68,11 @@ public class HiveTargetGenerator implements ITargetGenerator {
.
append
(
Constants
.
SPACE
).
append
(
SqoopConstants
.
DELETE_TARGET_DIR
);
}
if
(
StringUtils
.
isNotEmpty
(
targetHiveParameter
.
getHiveTargetDir
()))
{
hiveTargetSb
.
append
(
Constants
.
SPACE
).
append
(
SqoopConstants
.
TARGET_DIR
)
.
append
(
Constants
.
SPACE
).
append
(
targetHiveParameter
.
getHiveTargetDir
());
}
if
(
StringUtils
.
isNotEmpty
(
targetHiveParameter
.
getReplaceDelimiter
()))
{
hiveTargetSb
.
append
(
Constants
.
SPACE
).
append
(
SqoopConstants
.
HIVE_DELIMS_REPLACEMENT
)
.
append
(
Constants
.
SPACE
).
append
(
targetHiveParameter
.
getReplaceDelimiter
());
...
...
dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/sqoop/SqoopTaskTest.java
浏览文件 @
6609b912
...
...
@@ -138,13 +138,13 @@ public class SqoopTaskTest {
+
"\\\"srcQuerySql\\\":\\\"SELECT * FROM person_2\\\",\\\"srcColumnType\\\":\\\"0\\\",\\\"srcColumns\\\":\\\"\\\",\\\"srcConditionList\\\":[],"
+
"\\\"mapColumnHive\\\":[],\\\"mapColumnJava\\\":[{\\\"prop\\\":\\\"id\\\",\\\"direct\\\":\\\"IN\\\",\\\"type\\\":\\\"VARCHAR\\\",\\\"value\\\":\\\"Integer\\\"}]}\","
+
"\"targetParams\":\"{\\\"hiveDatabase\\\":\\\"stg\\\",\\\"hiveTable\\\":\\\"person_internal_2\\\",\\\"createHiveTable\\\":true,\\\"dropDelimiter\\\":false,"
+
"\\\"hiveOverWrite\\\":true,\\\"replaceDelimiter\\\":\\\"\\\",\\\"hivePartitionKey\\\":\\\"date\\\",\\\"hivePartitionValue\\\":\\\"2020-02-16\\\"}\",\"localParams\":[]}"
;
+
"\\\"hiveOverWrite\\\":true,\\\"
hiveTargetDir\\\":\\\"/tmp/sqoop_import_hive\\\",\\\"
replaceDelimiter\\\":\\\"\\\",\\\"hivePartitionKey\\\":\\\"date\\\",\\\"hivePartitionValue\\\":\\\"2020-02-16\\\"}\",\"localParams\":[]}"
;
SqoopParameters
mysqlToHiveParams
=
JSONUtils
.
parseObject
(
mysqlToHive
,
SqoopParameters
.
class
);
String
mysqlToHiveScript
=
generator
.
generateSqoopJob
(
mysqlToHiveParams
,
mysqlTaskExecutionContext
);
String
mysqlToHiveExpected
=
"sqoop import -D mapred.job.name=sqoop_import -m 1 --connect \"jdbc:mysql://192.168.0.111:3306/test\" --username kylo --password \"123456\" "
+
"--query \"SELECT * FROM person_2 WHERE \\$CONDITIONS\" --map-column-java id=Integer --hive-import --hive-database stg --hive-table person_internal_2 "
+
"--create-hive-table --hive-overwrite --delete-target-dir --hive-partition-key date --hive-partition-value 2020-02-16"
;
+
"--create-hive-table --hive-overwrite --delete-target-dir --
target-dir /tmp/sqoop_import_hive --
hive-partition-key date --hive-partition-value 2020-02-16"
;
Assert
.
assertEquals
(
mysqlToHiveExpected
,
mysqlToHiveScript
);
//sqoop CUSTOM job
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录