Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
DolphinScheduler
提交
7b66b66e
DolphinScheduler
项目概览
apache
/
DolphinScheduler
上一次同步 1 年多
通知
705
Star
9572
Fork
3514
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
DolphinScheduler
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
未验证
提交
7b66b66e
编写于
3月 01, 2022
作者:
R
ronyang1985
提交者:
GitHub
3月 01, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[Feature-8527][Datasource] Add dataSource for Amazon Redshift (#8564)
Co-authored-by:
N
Wenjun Ruan
<
wenjun@apache.org
>
上级
10282840
变更
21
隐藏空白更改
内联
并排
Showing
21 changed file
with
522 addition
and
5 deletion
+522
-5
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-all/pom.xml
...datasource-plugin/dolphinscheduler-datasource-all/pom.xml
+4
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/BaseDataSourceParamDTO.java
...gin/datasource/api/datasource/BaseDataSourceParamDTO.java
+3
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftConnectionParam.java
...urce/api/datasource/redshift/RedshiftConnectionParam.java
+37
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceParamDTO.java
...e/api/datasource/redshift/RedshiftDataSourceParamDTO.java
+43
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java
.../api/datasource/redshift/RedshiftDataSourceProcessor.java
+140
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java
...cheduler/plugin/datasource/api/utils/DataSourceUtils.java
+4
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java
.../datasource/redshift/RedshiftDataSourceProcessorTest.java
+98
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/pom.xml
...ource-plugin/dolphinscheduler-datasource-redshift/pom.xml
+44
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceChannel.java
...plugin/datasource/redshift/RedshiftDataSourceChannel.java
+30
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceChannelFactory.java
...datasource/redshift/RedshiftDataSourceChannelFactory.java
+36
-0
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceClient.java
.../plugin/datasource/redshift/RedshiftDataSourceClient.java
+28
-0
dolphinscheduler-datasource-plugin/pom.xml
dolphinscheduler-datasource-plugin/pom.xml
+1
-0
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/enums/DbType.java
...in/java/org/apache/dolphinscheduler/spi/enums/DbType.java
+3
-1
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/Constants.java
...java/org/apache/dolphinscheduler/spi/utils/Constants.java
+21
-2
dolphinscheduler-ui-next/src/service/modules/data-source/types.ts
...cheduler-ui-next/src/service/modules/data-source/types.ts
+1
-0
dolphinscheduler-ui-next/src/views/datasource/list/use-form.ts
...inscheduler-ui-next/src/views/datasource/list/use-form.ts
+5
-0
dolphinscheduler-ui-next/src/views/projects/task/components/node/fields/use-datasource-type.ts
...ojects/task/components/node/fields/use-datasource-type.ts
+5
-0
dolphinscheduler-ui/src/js/conf/home/pages/datasource/pages/list/_source/createDataSource.vue
.../pages/datasource/pages/list/_source/createDataSource.vue
+7
-0
dolphinscheduler-ui/src/js/conf/home/store/dag/state.js
dolphinscheduler-ui/src/js/conf/home/store/dag/state.js
+5
-0
dolphinscheduler-ui/src/js/conf/home/store/datasource/actions.js
...scheduler-ui/src/js/conf/home/store/datasource/actions.js
+2
-2
pom.xml
pom.xml
+5
-0
未找到文件。
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-all/pom.xml
浏览文件 @
7b66b66e
...
...
@@ -56,5 +56,9 @@
<groupId>
org.apache.dolphinscheduler
</groupId>
<artifactId>
dolphinscheduler-datasource-sqlserver
</artifactId>
</dependency>
<dependency>
<groupId>
org.apache.dolphinscheduler
</groupId>
<artifactId>
dolphinscheduler-datasource-redshift
</artifactId>
</dependency>
</dependencies>
</project>
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/BaseDataSourceParamDTO.java
浏览文件 @
7b66b66e
...
...
@@ -24,6 +24,7 @@ import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLD
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceParamDTO
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
...
...
@@ -46,6 +47,7 @@ import com.fasterxml.jackson.annotation.JsonTypeInfo;
* see {@link SQLServerDataSourceParamDTO}
* see {@link Db2DataSourceParamDTO}
* see {@link PrestoDataSourceParamDTO}
* see {@link RedshiftDataSourceParamDTO}
*/
@JsonTypeInfo
(
use
=
JsonTypeInfo
.
Id
.
NAME
,
property
=
"type"
)
@JsonSubTypes
(
value
=
{
...
...
@@ -58,6 +60,7 @@ import com.fasterxml.jackson.annotation.JsonTypeInfo;
@JsonSubTypes
.
Type
(
value
=
SQLServerDataSourceParamDTO
.
class
,
name
=
"SQLSERVER"
),
@JsonSubTypes
.
Type
(
value
=
Db2DataSourceParamDTO
.
class
,
name
=
"DB2"
),
@JsonSubTypes
.
Type
(
value
=
PrestoDataSourceParamDTO
.
class
,
name
=
"PRESTO"
),
@JsonSubTypes
.
Type
(
value
=
RedshiftDataSourceParamDTO
.
class
,
name
=
"REDSHIFT"
),
})
public
abstract
class
BaseDataSourceParamDTO
implements
Serializable
{
...
...
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftConnectionParam.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift
;
import
org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam
;
public
class
RedshiftConnectionParam
extends
BaseConnectionParam
{
@Override
public
String
toString
()
{
return
"RedshiftConnectionParam{"
+
"user='"
+
user
+
'\''
+
", password='"
+
password
+
'\''
+
", address='"
+
address
+
'\''
+
", database='"
+
database
+
'\''
+
", jdbcUrl='"
+
jdbcUrl
+
'\''
+
", driverLocation='"
+
driverLocation
+
'\''
+
", driverClassName='"
+
driverClassName
+
'\''
+
", validationQuery='"
+
validationQuery
+
'\''
+
", other='"
+
other
+
'\''
+
'}'
;
}
}
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceParamDTO.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
public
class
RedshiftDataSourceParamDTO
extends
BaseDataSourceParamDTO
{
@Override
public
String
toString
()
{
return
"RedshiftDataSourceParamDTO{"
+
"name='"
+
name
+
'\''
+
", note='"
+
note
+
'\''
+
", host='"
+
host
+
'\''
+
", port="
+
port
+
", database='"
+
database
+
'\''
+
", userName='"
+
userName
+
'\''
+
", password='"
+
password
+
'\''
+
", other='"
+
other
+
'\''
+
'}'
;
}
@Override
public
DbType
getType
()
{
return
DbType
.
REDSHIFT
;
}
}
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.AbstractDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO
;
import
org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils
;
import
org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam
;
import
org.apache.dolphinscheduler.spi.datasource.ConnectionParam
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
import
org.apache.dolphinscheduler.spi.utils.Constants
;
import
org.apache.dolphinscheduler.spi.utils.JSONUtils
;
import
org.apache.commons.collections4.MapUtils
;
import
org.apache.commons.lang.StringUtils
;
import
java.sql.Connection
;
import
java.sql.DriverManager
;
import
java.sql.SQLException
;
import
java.util.ArrayList
;
import
java.util.LinkedHashMap
;
import
java.util.List
;
import
java.util.Map
;
public
class
RedshiftDataSourceProcessor
extends
AbstractDataSourceProcessor
{
@Override
public
BaseDataSourceParamDTO
createDatasourceParamDTO
(
String
connectionJson
)
{
RedshiftConnectionParam
connectionParams
=
(
RedshiftConnectionParam
)
createConnectionParams
(
connectionJson
);
String
[]
hostSeperator
=
connectionParams
.
getAddress
().
split
(
Constants
.
DOUBLE_SLASH
);
String
[]
hostPortArray
=
hostSeperator
[
hostSeperator
.
length
-
1
].
split
(
Constants
.
COMMA
);
RedshiftDataSourceParamDTO
redshiftDatasourceParamDTO
=
new
RedshiftDataSourceParamDTO
();
redshiftDatasourceParamDTO
.
setPort
(
Integer
.
parseInt
(
hostPortArray
[
0
].
split
(
Constants
.
COLON
)[
1
]));
redshiftDatasourceParamDTO
.
setHost
(
hostPortArray
[
0
].
split
(
Constants
.
COLON
)[
0
]);
redshiftDatasourceParamDTO
.
setDatabase
(
connectionParams
.
getDatabase
());
redshiftDatasourceParamDTO
.
setUserName
(
connectionParams
.
getUser
());
redshiftDatasourceParamDTO
.
setOther
(
parseOther
(
connectionParams
.
getOther
()));
return
redshiftDatasourceParamDTO
;
}
@Override
public
BaseConnectionParam
createConnectionParams
(
BaseDataSourceParamDTO
datasourceParam
)
{
RedshiftDataSourceParamDTO
redshiftParam
=
(
RedshiftDataSourceParamDTO
)
datasourceParam
;
String
address
=
String
.
format
(
"%s%s:%s"
,
Constants
.
JDBC_REDSHIFT
,
redshiftParam
.
getHost
(),
redshiftParam
.
getPort
());
String
jdbcUrl
=
address
+
Constants
.
SLASH
+
redshiftParam
.
getDatabase
();
RedshiftConnectionParam
redshiftConnectionParam
=
new
RedshiftConnectionParam
();
redshiftConnectionParam
.
setUser
(
redshiftParam
.
getUserName
());
redshiftConnectionParam
.
setPassword
(
PasswordUtils
.
encodePassword
(
redshiftParam
.
getPassword
()));
redshiftConnectionParam
.
setOther
(
transformOther
(
redshiftParam
.
getOther
()));
redshiftConnectionParam
.
setAddress
(
address
);
redshiftConnectionParam
.
setJdbcUrl
(
jdbcUrl
);
redshiftConnectionParam
.
setDatabase
(
redshiftParam
.
getDatabase
());
redshiftConnectionParam
.
setDriverClassName
(
getDatasourceDriver
());
redshiftConnectionParam
.
setValidationQuery
(
getValidationQuery
());
redshiftConnectionParam
.
setProps
(
redshiftParam
.
getOther
());
return
redshiftConnectionParam
;
}
@Override
public
ConnectionParam
createConnectionParams
(
String
connectionJson
)
{
return
JSONUtils
.
parseObject
(
connectionJson
,
RedshiftConnectionParam
.
class
);
}
@Override
public
String
getDatasourceDriver
()
{
return
Constants
.
COM_REDSHIFT_JDBC_DRIVER
;
}
@Override
public
String
getValidationQuery
()
{
return
Constants
.
REDHIFT_VALIDATION_QUERY
;
}
@Override
public
String
getJdbcUrl
(
ConnectionParam
connectionParam
)
{
RedshiftConnectionParam
redshiftConnectionParam
=
(
RedshiftConnectionParam
)
connectionParam
;
if
(!
StringUtils
.
isEmpty
(
redshiftConnectionParam
.
getOther
()))
{
return
String
.
format
(
"%s?%s"
,
redshiftConnectionParam
.
getJdbcUrl
(),
redshiftConnectionParam
.
getOther
());
}
return
redshiftConnectionParam
.
getJdbcUrl
();
}
@Override
public
Connection
getConnection
(
ConnectionParam
connectionParam
)
throws
ClassNotFoundException
,
SQLException
{
RedshiftConnectionParam
redshiftConnectionParam
=
(
RedshiftConnectionParam
)
connectionParam
;
Class
.
forName
(
getDatasourceDriver
());
return
DriverManager
.
getConnection
(
getJdbcUrl
(
connectionParam
),
redshiftConnectionParam
.
getUser
(),
PasswordUtils
.
decodePassword
(
redshiftConnectionParam
.
getPassword
()));
}
@Override
public
DbType
getDbType
()
{
return
DbType
.
REDSHIFT
;
}
private
String
transformOther
(
Map
<
String
,
String
>
otherMap
)
{
if
(
MapUtils
.
isNotEmpty
(
otherMap
))
{
List
<
String
>
list
=
new
ArrayList
<>(
otherMap
.
size
());
otherMap
.
forEach
((
key
,
value
)
->
list
.
add
(
String
.
format
(
"%s=%s"
,
key
,
value
)));
return
String
.
join
(
Constants
.
SEMICOLON
,
list
);
}
return
null
;
}
private
Map
<
String
,
String
>
parseOther
(
String
other
)
{
Map
<
String
,
String
>
otherMap
=
new
LinkedHashMap
<>();
if
(
StringUtils
.
isEmpty
(
other
))
{
return
otherMap
;
}
String
[]
configs
=
other
.
split
(
Constants
.
SEMICOLON
);
for
(
String
config
:
configs
)
{
otherMap
.
put
(
config
.
split
(
Constants
.
EQUAL_SIGN
)[
0
],
config
.
split
(
Constants
.
EQUAL_SIGN
)[
1
]);
}
return
otherMap
;
}
}
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java
浏览文件 @
7b66b66e
...
...
@@ -26,6 +26,7 @@ import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLD
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor
;
import
org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor
;
import
org.apache.dolphinscheduler.spi.datasource.ConnectionParam
;
...
...
@@ -52,6 +53,7 @@ public class DataSourceUtils {
private
static
final
DataSourceProcessor
sqlServerProcessor
=
new
SQLServerDataSourceProcessor
();
private
static
final
DataSourceProcessor
db2PROCESSOR
=
new
Db2DataSourceProcessor
();
private
static
final
DataSourceProcessor
prestoPROCESSOR
=
new
PrestoDataSourceProcessor
();
private
static
final
DataSourceProcessor
redshiftProcessor
=
new
RedshiftDataSourceProcessor
();
/**
* check datasource param
...
...
@@ -120,6 +122,8 @@ public class DataSourceUtils {
return
db2PROCESSOR
;
case
PRESTO:
return
prestoPROCESSOR
;
case
REDSHIFT:
return
redshiftProcessor
;
default
:
throw
new
IllegalArgumentException
(
"datasource type illegal:"
+
dbType
);
}
...
...
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift
;
import
org.apache.dolphinscheduler.plugin.datasource.api.plugin.DataSourceClientProvider
;
import
org.apache.dolphinscheduler.plugin.datasource.api.utils.CommonUtils
;
import
org.apache.dolphinscheduler.plugin.datasource.api.utils.DataSourceUtils
;
import
org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
import
org.apache.dolphinscheduler.spi.utils.Constants
;
import
java.sql.DriverManager
;
import
java.util.HashMap
;
import
java.util.Map
;
import
org.junit.Assert
;
import
org.junit.Test
;
import
org.junit.runner.RunWith
;
import
org.mockito.Mockito
;
import
org.powermock.api.mockito.PowerMockito
;
import
org.powermock.core.classloader.annotations.PrepareForTest
;
import
org.powermock.modules.junit4.PowerMockRunner
;
@RunWith
(
PowerMockRunner
.
class
)
@PrepareForTest
({
Class
.
class
,
DriverManager
.
class
,
DataSourceUtils
.
class
,
CommonUtils
.
class
,
DataSourceClientProvider
.
class
,
PasswordUtils
.
class
})
public
class
RedshiftDataSourceProcessorTest
{
private
RedshiftDataSourceProcessor
redshiftDatasourceProcessor
=
new
RedshiftDataSourceProcessor
();
@Test
public
void
testCreateConnectionParams
()
{
Map
<
String
,
String
>
props
=
new
HashMap
<>();
props
.
put
(
"serverTimezone"
,
"utc"
);
RedshiftDataSourceParamDTO
redshiftDatasourceParamDTO
=
new
RedshiftDataSourceParamDTO
();
redshiftDatasourceParamDTO
.
setHost
(
"localhost"
);
redshiftDatasourceParamDTO
.
setPort
(
5439
);
redshiftDatasourceParamDTO
.
setDatabase
(
"dev"
);
redshiftDatasourceParamDTO
.
setUserName
(
"awsuser"
);
redshiftDatasourceParamDTO
.
setPassword
(
"123456"
);
redshiftDatasourceParamDTO
.
setOther
(
props
);
PowerMockito
.
mockStatic
(
PasswordUtils
.
class
);
PowerMockito
.
when
(
PasswordUtils
.
encodePassword
(
Mockito
.
anyString
())).
thenReturn
(
"test"
);
RedshiftConnectionParam
connectionParams
=
(
RedshiftConnectionParam
)
redshiftDatasourceProcessor
.
createConnectionParams
(
redshiftDatasourceParamDTO
);
Assert
.
assertEquals
(
"jdbc:redshift://localhost:5439"
,
connectionParams
.
getAddress
());
Assert
.
assertEquals
(
"jdbc:redshift://localhost:5439/dev"
,
connectionParams
.
getJdbcUrl
());
}
@Test
public
void
testCreateConnectionParams2
()
{
String
connectionJson
=
"{\"user\":\"awsuser\",\"password\":\"123456\",\"address\":\"jdbc:redshift://localhost:5439\""
+
",\"database\":\"dev\",\"jdbcUrl\":\"jdbc:redshift://localhost:5439/dev\"}"
;
RedshiftConnectionParam
connectionParams
=
(
RedshiftConnectionParam
)
redshiftDatasourceProcessor
.
createConnectionParams
(
connectionJson
);
Assert
.
assertNotNull
(
connectionParams
);
Assert
.
assertEquals
(
"awsuser"
,
connectionParams
.
getUser
());
}
@Test
public
void
testGetDatasourceDriver
()
{
Assert
.
assertEquals
(
Constants
.
COM_REDSHIFT_JDBC_DRIVER
,
redshiftDatasourceProcessor
.
getDatasourceDriver
());
}
@Test
public
void
testGetJdbcUrl
()
{
RedshiftConnectionParam
redshiftConnectionParam
=
new
RedshiftConnectionParam
();
redshiftConnectionParam
.
setJdbcUrl
(
"jdbc:redshift://localhost:5439/default"
);
redshiftConnectionParam
.
setOther
(
"DSILogLevel=6;defaultRowFetchSize=100"
);
Assert
.
assertEquals
(
"jdbc:redshift://localhost:5439/default?DSILogLevel=6;defaultRowFetchSize=100"
,
redshiftDatasourceProcessor
.
getJdbcUrl
(
redshiftConnectionParam
));
}
@Test
public
void
testGetDbType
()
{
Assert
.
assertEquals
(
DbType
.
REDSHIFT
,
redshiftDatasourceProcessor
.
getDbType
());
}
@Test
public
void
testGetValidationQuery
()
{
Assert
.
assertEquals
(
Constants
.
REDHIFT_VALIDATION_QUERY
,
redshiftDatasourceProcessor
.
getValidationQuery
());
}
}
\ No newline at end of file
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/pom.xml
0 → 100644
浏览文件 @
7b66b66e
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project
xmlns=
"http://maven.apache.org/POM/4.0.0"
xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
>
<parent>
<artifactId>
dolphinscheduler-datasource-plugin
</artifactId>
<groupId>
org.apache.dolphinscheduler
</groupId>
<version>
2.0.4-SNAPSHOT
</version>
</parent>
<modelVersion>
4.0.0
</modelVersion>
<artifactId>
dolphinscheduler-datasource-redshift
</artifactId>
<packaging>
jar
</packaging>
<dependencies>
<dependency>
<groupId>
org.apache.dolphinscheduler
</groupId>
<artifactId>
dolphinscheduler-spi
</artifactId>
<scope>
provided
</scope>
</dependency>
<dependency>
<groupId>
org.apache.dolphinscheduler
</groupId>
<artifactId>
dolphinscheduler-datasource-api
</artifactId>
</dependency>
</dependencies>
</project>
\ No newline at end of file
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceChannel.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.redshift
;
import
org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam
;
import
org.apache.dolphinscheduler.spi.datasource.DataSourceChannel
;
import
org.apache.dolphinscheduler.spi.datasource.DataSourceClient
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
public
class
RedshiftDataSourceChannel
implements
DataSourceChannel
{
@Override
public
DataSourceClient
createDataSourceClient
(
BaseConnectionParam
baseConnectionParam
,
DbType
dbType
)
{
return
new
RedshiftDataSourceClient
(
baseConnectionParam
,
dbType
);
}
}
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceChannelFactory.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.redshift
;
import
org.apache.dolphinscheduler.spi.datasource.DataSourceChannel
;
import
org.apache.dolphinscheduler.spi.datasource.DataSourceChannelFactory
;
import
com.google.auto.service.AutoService
;
@AutoService
(
DataSourceChannelFactory
.
class
)
public
class
RedshiftDataSourceChannelFactory
implements
DataSourceChannelFactory
{
@Override
public
DataSourceChannel
create
()
{
return
new
RedshiftDataSourceChannel
();
}
@Override
public
String
getName
()
{
return
"redshift"
;
}
}
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-redshift/src/main/java/org/apache/dolphinscheduler/plugin/datasource/redshift/RedshiftDataSourceClient.java
0 → 100644
浏览文件 @
7b66b66e
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.dolphinscheduler.plugin.datasource.redshift
;
import
org.apache.dolphinscheduler.plugin.datasource.api.client.CommonDataSourceClient
;
import
org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam
;
import
org.apache.dolphinscheduler.spi.enums.DbType
;
public
class
RedshiftDataSourceClient
extends
CommonDataSourceClient
{
public
RedshiftDataSourceClient
(
BaseConnectionParam
baseConnectionParam
,
DbType
dbType
)
{
super
(
baseConnectionParam
,
dbType
);
}
}
dolphinscheduler-datasource-plugin/pom.xml
浏览文件 @
7b66b66e
...
...
@@ -38,5 +38,6 @@
<module>
dolphinscheduler-datasource-postgresql
</module>
<module>
dolphinscheduler-datasource-api
</module>
<module>
dolphinscheduler-datasource-all
</module>
<module>
dolphinscheduler-datasource-redshift
</module>
</modules>
</project>
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/enums/DbType.java
浏览文件 @
7b66b66e
...
...
@@ -35,7 +35,9 @@ public enum DbType {
SQLSERVER
(
6
,
"sqlserver"
),
DB2
(
7
,
"db2"
),
PRESTO
(
8
,
"presto"
),
H2
(
9
,
"h2"
);
H2
(
9
,
"h2"
),
REDSHIFT
(
10
,
"redshift"
),
;
@EnumValue
private
final
int
code
;
...
...
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/Constants.java
浏览文件 @
7b66b66e
...
...
@@ -161,6 +161,7 @@ public class Constants {
public
static
final
String
COM_SQLSERVER_JDBC_DRIVER
=
"com.microsoft.sqlserver.jdbc.SQLServerDriver"
;
public
static
final
String
COM_DB2_JDBC_DRIVER
=
"com.ibm.db2.jcc.DB2Driver"
;
public
static
final
String
COM_PRESTO_JDBC_DRIVER
=
"com.facebook.presto.jdbc.PrestoDriver"
;
public
static
final
String
COM_REDSHIFT_JDBC_DRIVER
=
"com.amazon.redshift.jdbc42.Driver"
;
/**
...
...
@@ -174,6 +175,7 @@ public class Constants {
public
static
final
String
SQLSERVER_VALIDATION_QUERY
=
"select 1"
;
public
static
final
String
DB2_VALIDATION_QUERY
=
"select 1 from sysibm.sysdummy1"
;
public
static
final
String
PRESTO_VALIDATION_QUERY
=
"select 1"
;
public
static
final
String
REDHIFT_VALIDATION_QUERY
=
"select 1"
;
/**
* jdbc url
...
...
@@ -187,7 +189,7 @@ public class Constants {
public
static
final
String
JDBC_SQLSERVER
=
"jdbc:sqlserver://"
;
public
static
final
String
JDBC_DB2
=
"jdbc:db2://"
;
public
static
final
String
JDBC_PRESTO
=
"jdbc:presto://"
;
public
static
final
String
JDBC_REDSHIFT
=
"jdbc:redshift://"
;
public
static
final
String
ADDRESS
=
"address"
;
public
static
final
String
DATABASE
=
"database"
;
...
...
@@ -204,6 +206,11 @@ public class Constants {
*/
public
static
final
String
DOUBLE_SLASH
=
"//"
;
/**
* SLASH /
*/
public
static
final
String
SLASH
=
"/"
;
/**
* comma ,
*/
...
...
@@ -214,11 +221,23 @@ public class Constants {
*/
public
static
final
String
COLON
=
":"
;
/**
* AT SIGN
* AT SIGN
@
*/
public
static
final
String
AT_SIGN
=
"@"
;
/**
* SEMICOLON ;
*/
public
static
final
String
SEMICOLON
=
";"
;
/**
* EQUAL_SIGN =
*/
public
static
final
String
EQUAL_SIGN
=
"="
;
/**
* datasource encryption salt
*/
...
...
dolphinscheduler-ui-next/src/service/modules/data-source/types.ts
浏览文件 @
7b66b66e
...
...
@@ -25,6 +25,7 @@ type IDataBase =
|
'
SQLSERVER
'
|
'
DB2
'
|
'
PRESTO
'
|
'
REDSHIFT
'
interface
IDataSource
{
id
?:
number
...
...
dolphinscheduler-ui-next/src/views/datasource/list/use-form.ts
浏览文件 @
7b66b66e
...
...
@@ -202,6 +202,11 @@ const datasourceType: IDataBaseOptionKeys = {
value
:
'
PRESTO
'
,
label
:
'
PRESTO
'
,
defaultPort
:
8080
},
REDSHIFT
:
{
value
:
'
REDSHIFT
'
,
label
:
'
REDSHIFT
'
,
defaultPort
:
5439
}
}
...
...
dolphinscheduler-ui-next/src/views/projects/task/components/node/fields/use-datasource-type.ts
浏览文件 @
7b66b66e
...
...
@@ -75,6 +75,11 @@ export function useDatasourceType(
id
:
8
,
code
:
'
PRESTO
'
,
disabled
:
false
},
{
id
:
9
,
code
:
'
REDSHIFT
'
,
disabled
:
false
}
]
...
...
dolphinscheduler-ui/src/js/conf/home/pages/datasource/pages/list/_source/createDataSource.vue
浏览文件 @
7b66b66e
...
...
@@ -279,6 +279,10 @@
{
value
:
'
PRESTO
'
,
label
:
'
PRESTO
'
},
{
value
:
'
REDSHIFT
'
,
label
:
'
REDSHIFT
'
}
]
}
...
...
@@ -490,6 +494,9 @@
case
'
PRESTO
'
:
defaultPort
=
'
8080
'
break
case
'
REDSHIFT
'
:
defaultPort
=
'
5439
'
break
default
:
break
}
...
...
dolphinscheduler-ui/src/js/conf/home/store/dag/state.js
浏览文件 @
7b66b66e
...
...
@@ -113,6 +113,11 @@ export default {
id
:
8
,
code
:
'
PRESTO
'
,
disabled
:
false
},
{
id
:
9
,
code
:
'
REDSHIFT
'
,
disabled
:
false
}
],
// Alarm interface
...
...
dolphinscheduler-ui/src/js/conf/home/store/datasource/actions.js
浏览文件 @
7b66b66e
...
...
@@ -20,7 +20,7 @@ import io from '@/module/io'
export
default
{
/**
* Data source creation
* @param "type": string,//MYSQL, POSTGRESQL, HIVE, SPARK, CLICKHOUSE, ORACLE, SQLSERVER, PRESTO
* @param "type": string,//MYSQL, POSTGRESQL, HIVE, SPARK, CLICKHOUSE, ORACLE, SQLSERVER, PRESTO
, REDSHIFT
* @param "name": string,
* @param "desc": string,
* @param "parameter":string //{"address":"jdbc:hive2://192.168.220.189:10000","autoReconnect":"true","characterEncoding":"utf8","database":"default","initialTimeout":3000,"jdbcUrl":"jdbc:hive2://192.168.220.189:10000/default","maxReconnect":10,"password":"","useUnicode":true,"user":"hive"}
...
...
@@ -53,7 +53,7 @@ export default {
},
/**
* Query data source list - no paging
* @param "type": string//MYSQL, POSTGRESQL, HIVE, SPARK, CLICKHOUSE, ORACLE, SQLSERVER, PRESTO
* @param "type": string//MYSQL, POSTGRESQL, HIVE, SPARK, CLICKHOUSE, ORACLE, SQLSERVER, PRESTO
, REDSHIFT
*/
getDatasourcesList
({
state
},
payload
)
{
return
new
Promise
((
resolve
,
reject
)
=>
{
...
...
pom.xml
浏览文件 @
7b66b66e
...
...
@@ -435,6 +435,11 @@
<artifactId>
dolphinscheduler-datasource-sqlserver
</artifactId>
<version>
${project.version}
</version>
</dependency>
<dependency>
<groupId>
org.apache.dolphinscheduler
</groupId>
<artifactId>
dolphinscheduler-datasource-redshift
</artifactId>
<version>
${project.version}
</version>
</dependency>
<dependency>
<groupId>
org.apache.dolphinscheduler
</groupId>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录