Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
lijianghuflute
canal
提交
bc056288
canal
项目概览
lijianghuflute
/
canal
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
canal
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
bc056288
编写于
11月 16, 2018
作者:
M
mcy
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
rdb 批量提交优化
上级
fc6c8713
变更
5
展开全部
隐藏空白更改
内联
并排
Showing
5 changed file
with
278 addition
and
309 deletion
+278
-309
client-adapter/launcher/src/main/resources/application.yml
client-adapter/launcher/src/main/resources/application.yml
+10
-10
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/RdbAdapter.java
...om/alibaba/otter/canal/client/adapter/rdb/RdbAdapter.java
+44
-35
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/BatchExecutor.java
...otter/canal/client/adapter/rdb/service/BatchExecutor.java
+1
-28
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/RdbSyncService.java
...tter/canal/client/adapter/rdb/service/RdbSyncService.java
+121
-236
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/support/SimpleDml.java
...aba/otter/canal/client/adapter/rdb/support/SimpleDml.java
+102
-0
未找到文件。
client-adapter/launcher/src/main/resources/application.yml
浏览文件 @
bc056288
...
...
@@ -28,16 +28,16 @@ canal.conf:
-
instance
:
example
groups
:
-
outAdapters
:
#
- name: logger
-
name
:
rdb
key
:
oracle1
properties
:
jdbc.driverClassName
:
oracle.jdbc.OracleDriver
jdbc.url
:
jdbc:oracle:thin:@localhost:49161:XE
jdbc.username
:
mytest
jdbc.password
:
m121212
threads
:
5
commitSize
:
5000
-
name
:
logger
#
- name: rdb
#
key: oracle1
#
properties:
#
jdbc.driverClassName: oracle.jdbc.OracleDriver
#
jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
#
jdbc.username: mytest
#
jdbc.password: m121212
#
threads: 5
#
commitSize: 5000
# - name: rdb
# key: postgres1
# properties:
...
...
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/RdbAdapter.java
浏览文件 @
bc056288
...
...
@@ -5,9 +5,6 @@ import java.sql.SQLException;
import
java.util.*
;
import
java.util.concurrent.ExecutorService
;
import
java.util.concurrent.Executors
;
import
java.util.concurrent.TimeUnit
;
import
java.util.concurrent.atomic.AtomicInteger
;
import
java.util.concurrent.locks.Condition
;
import
java.util.concurrent.locks.Lock
;
import
java.util.concurrent.locks.ReentrantLock
;
...
...
@@ -18,11 +15,14 @@ import org.slf4j.Logger;
import
org.slf4j.LoggerFactory
;
import
com.alibaba.druid.pool.DruidDataSource
;
import
com.alibaba.fastjson.JSON
;
import
com.alibaba.fastjson.serializer.SerializerFeature
;
import
com.alibaba.otter.canal.client.adapter.OuterAdapter
;
import
com.alibaba.otter.canal.client.adapter.rdb.config.ConfigLoader
;
import
com.alibaba.otter.canal.client.adapter.rdb.config.MappingConfig
;
import
com.alibaba.otter.canal.client.adapter.rdb.service.RdbEtlService
;
import
com.alibaba.otter.canal.client.adapter.rdb.service.RdbSyncService
;
import
com.alibaba.otter.canal.client.adapter.rdb.support.SimpleDml
;
import
com.alibaba.otter.canal.client.adapter.support.*
;
@SPI
(
"rdb"
)
...
...
@@ -30,8 +30,8 @@ public class RdbAdapter implements OuterAdapter {
private
static
Logger
logger
=
LoggerFactory
.
getLogger
(
RdbAdapter
.
class
);
private
Map
<
String
,
MappingConfig
>
rdbMapping
=
new
HashMap
<>();
// 文件名对应配置
private
Map
<
String
,
MappingConfig
>
mappingConfigCache
=
new
HashMap
<>();
// 库名-表名对应配置
private
Map
<
String
,
MappingConfig
>
rdbMapping
=
new
HashMap
<>();
// 文件名对应配置
private
Map
<
String
,
MappingConfig
>
mappingConfigCache
=
new
HashMap
<>();
// 库名-表名对应配置
private
DruidDataSource
dataSource
;
...
...
@@ -39,6 +39,12 @@ public class RdbAdapter implements OuterAdapter {
private
int
commitSize
=
3000
;
private
volatile
boolean
running
=
false
;
private
List
<
SimpleDml
>
dmlList
=
Collections
.
synchronizedList
(
new
ArrayList
<>());
private
Lock
syncLock
=
new
ReentrantLock
();
private
ExecutorService
executor
=
Executors
.
newFixedThreadPool
(
1
);
@Override
public
void
init
(
OuterAdapterConfig
configuration
)
{
Map
<
String
,
MappingConfig
>
rdbMappingTmp
=
ConfigLoader
.
load
();
...
...
@@ -81,52 +87,54 @@ public class RdbAdapter implements OuterAdapter {
if
(
commitSize
!=
null
)
{
this
.
commitSize
=
Integer
.
valueOf
(
commitSize
);
}
rdbSyncService
=
new
RdbSyncService
(
this
.
commitSize
,
threads
!=
null
?
Integer
.
valueOf
(
threads
)
:
null
,
dataSource
);
}
rdbSyncService
=
new
RdbSyncService
(
threads
!=
null
?
Integer
.
valueOf
(
threads
)
:
null
,
dataSource
);
private
AtomicInteger
batchRowNum
=
new
AtomicInteger
(
0
);
private
List
<
Dml
>
dmlList
=
Collections
.
synchronizedList
(
new
ArrayList
<>());
private
Lock
syncLock
=
new
ReentrantLock
();
private
Condition
condition
=
syncLock
.
newCondition
();
private
ExecutorService
executor
=
Executors
.
newFixedThreadPool
(
1
);
@Override
public
void
sync
(
Dml
dml
)
{
boolean
first
=
batchRowNum
.
get
()
==
0
;
int
currentSize
=
batchRowNum
.
addAndGet
(
dml
.
getData
().
size
());
dmlList
.
add
(
dml
);
running
=
true
;
if
(
first
)
{
// 开启超时判断
executor
.
submit
(()
->
{
executor
.
submit
(()
->
{
while
(
running
)
{
try
{
syncLock
.
lock
();
if
(!
condition
.
await
(
5
,
TimeUnit
.
SECONDS
))
{
// 批量超时
int
size1
=
dmlList
.
size
();
Thread
.
sleep
(
3000
);
int
size2
=
dmlList
.
size
();
if
(
size1
==
size2
)
{
// 超时提交
sync
();
}
}
catch
(
Exception
e
)
{
logger
.
error
(
e
.
getMessage
(),
e
);
}
finally
{
syncLock
.
unlock
();
}
});
}
}
});
}
@Override
public
void
sync
(
Dml
dml
)
{
String
destination
=
StringUtils
.
trimToEmpty
(
dml
.
getDestination
());
String
database
=
dml
.
getDatabase
();
String
table
=
dml
.
getTable
();
MappingConfig
config
=
mappingConfigCache
.
get
(
destination
+
"."
+
database
+
"."
+
table
);
List
<
SimpleDml
>
simpleDmlList
=
SimpleDml
.
dml2SimpleDml
(
dml
,
config
);
if
(
currentSize
>
commitSize
)
{
dmlList
.
addAll
(
simpleDmlList
);
if
(
dmlList
.
size
()
>
commitSize
)
{
sync
();
}
if
(
logger
.
isDebugEnabled
())
{
logger
.
debug
(
"DML: {}"
,
JSON
.
toJSONString
(
dml
,
SerializerFeature
.
WriteMapNullValue
));
}
}
private
void
sync
()
{
try
{
syncLock
.
lock
();
rdbSyncService
.
sync
(
mappingConfigCache
,
dmlList
);
batchRowNum
.
set
(
0
);
dmlList
.
clear
();
condition
.
signal
();
if
(!
dmlList
.
isEmpty
())
{
rdbSyncService
.
sync
(
dmlList
);
dmlList
.
clear
();
}
}
finally
{
syncLock
.
unlock
();
}
...
...
@@ -226,6 +234,7 @@ public class RdbAdapter implements OuterAdapter {
@Override
public
void
destroy
()
{
running
=
false
;
executor
.
shutdown
();
if
(
rdbSyncService
!=
null
)
{
...
...
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/BatchExecutor.java
浏览文件 @
bc056288
...
...
@@ -23,18 +23,14 @@ public class BatchExecutor {
private
Integer
key
;
private
Connection
conn
;
private
int
commitSize
=
3000
;
private
AtomicInteger
idx
=
new
AtomicInteger
(
0
);
private
ExecutorService
executor
=
Executors
.
newFixedThreadPool
(
1
);
private
Lock
commitLock
=
new
ReentrantLock
();
private
Condition
condition
=
commitLock
.
newCondition
();
public
BatchExecutor
(
Integer
key
,
Connection
conn
,
Integer
commitSize
){
public
BatchExecutor
(
Integer
key
,
Connection
conn
){
this
.
key
=
key
;
this
.
conn
=
conn
;
if
(
commitSize
!=
null
)
{
this
.
commitSize
=
commitSize
;
}
try
{
this
.
conn
.
setAutoCommit
(
false
);
...
...
@@ -71,29 +67,6 @@ public class BatchExecutor {
}
catch
(
SQLException
e
)
{
logger
.
error
(
e
.
getMessage
(),
e
);
}
// int i = idx.incrementAndGet();
//
// // 批次的第一次执行设置延时
// if (i == 1) {
// executor.submit(() -> {
// try {
// commitLock.lock();
// conn.commit(); //直接提交一次
// if (!condition.await(5, TimeUnit.SECONDS)) {
// // 超时提交
// commit();
// }
// } catch (Exception e) {
// logger.error(e.getMessage(), e);
// } finally {
// commitLock.unlock();
// }
// });
// }
//
// if (i == commitSize) {
// commit();
// }
}
public
void
commit
()
{
...
...
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/RdbSyncService.java
浏览文件 @
bc056288
此差异已折叠。
点击以展开。
client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/support/SimpleDml.java
0 → 100644
浏览文件 @
bc056288
package
com.alibaba.otter.canal.client.adapter.rdb.support
;
import
java.util.ArrayList
;
import
java.util.List
;
import
java.util.Map
;
import
com.alibaba.otter.canal.client.adapter.rdb.config.MappingConfig
;
import
com.alibaba.otter.canal.client.adapter.support.Dml
;
import
org.apache.commons.lang.StringUtils
;
public
class
SimpleDml
{
private
String
destination
;
private
String
database
;
private
String
table
;
private
String
type
;
private
Map
<
String
,
Object
>
data
;
private
Map
<
String
,
Object
>
old
;
private
MappingConfig
config
;
public
String
getDestination
()
{
return
destination
;
}
public
void
setDestination
(
String
destination
)
{
this
.
destination
=
destination
;
}
public
String
getDatabase
()
{
return
database
;
}
public
void
setDatabase
(
String
database
)
{
this
.
database
=
database
;
}
public
String
getTable
()
{
return
table
;
}
public
void
setTable
(
String
table
)
{
this
.
table
=
table
;
}
public
String
getType
()
{
return
type
;
}
public
void
setType
(
String
type
)
{
this
.
type
=
type
;
}
public
Map
<
String
,
Object
>
getData
()
{
return
data
;
}
public
void
setData
(
Map
<
String
,
Object
>
data
)
{
this
.
data
=
data
;
}
public
Map
<
String
,
Object
>
getOld
()
{
return
old
;
}
public
void
setOld
(
Map
<
String
,
Object
>
old
)
{
this
.
old
=
old
;
}
public
MappingConfig
getConfig
()
{
return
config
;
}
public
void
setConfig
(
MappingConfig
config
)
{
this
.
config
=
config
;
}
public
static
List
<
SimpleDml
>
dml2SimpleDml
(
Dml
dml
,
MappingConfig
config
)
{
List
<
SimpleDml
>
simpleDmlList
=
new
ArrayList
<>();
int
len
=
dml
.
getData
().
size
();
String
destination
=
StringUtils
.
trimToEmpty
(
dml
.
getDestination
());
String
database
=
dml
.
getDatabase
();
String
table
=
dml
.
getTable
();
for
(
int
i
=
0
;
i
<
len
;
i
++)
{
SimpleDml
simpleDml
=
new
SimpleDml
();
simpleDml
.
setDestination
(
dml
.
getDestination
());
simpleDml
.
setDatabase
(
dml
.
getDatabase
());
simpleDml
.
setTable
(
dml
.
getTable
());
simpleDml
.
setType
(
dml
.
getType
());
simpleDml
.
setData
(
dml
.
getData
().
get
(
i
));
if
(
dml
.
getOld
()
!=
null
)
{
simpleDml
.
setOld
(
dml
.
getOld
().
get
(
i
));
}
simpleDml
.
setConfig
(
config
);
simpleDmlList
.
add
(
simpleDml
);
}
return
simpleDmlList
;
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录