Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
2dot5
ClickHouse
提交
28a3e78a
C
ClickHouse
项目概览
2dot5
/
ClickHouse
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
C
ClickHouse
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
28a3e78a
编写于
3月 14, 2019
作者:
A
alesapin
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Remove 'Adding of unexpected local part to zookeeper' logic because it's redundant.
上级
9121822e
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
13 addition
and
61 deletion
+13
-61
dbms/src/Storages/StorageReplicatedMergeTree.cpp
dbms/src/Storages/StorageReplicatedMergeTree.cpp
+13
-61
未找到文件。
dbms/src/Storages/StorageReplicatedMergeTree.cpp
浏览文件 @
28a3e78a
...
...
@@ -547,45 +547,25 @@ void StorageReplicatedMergeTree::checkParts(bool skip_sanity_checks)
/// There are no PreCommitted parts at startup.
auto
parts
=
data
.
getDataParts
({
MergeTreeDataPartState
::
Committed
,
MergeTreeDataPartState
::
Outdated
});
/// Local parts that are not in ZK.
/** Local parts that are not in ZK.
* In very rare cases they may cover missing parts
* and someone may think that pushing them to zookeeper is good idea.
* But actually we can't precisely determine that ALL missing parts
* covered by this unexpected part. So missing parts will be downloaded.
*/
MergeTreeData
::
DataParts
unexpected_parts
;
for
(
const
auto
&
part
:
parts
)
{
if
(
expected_parts
.
count
(
part
->
name
))
expected_parts
.
erase
(
part
->
name
);
else
unexpected_parts
.
insert
(
part
);
}
/// Which local parts to added into ZK.
MergeTreeData
::
DataPartsVector
parts_to_add
;
UInt64
parts_to_add_rows
=
0
;
/// Which parts should be taken from other replicas.
Strings
parts_to_fetch
;
for
(
const
String
&
missing_name
:
expected_
parts
)
for
(
const
auto
&
part
:
parts
)
{
/// If locally some part is missing, but there is a part covering it, you can replace it in ZK with the covering one.
auto
containing
=
data
.
getActiveContainingPart
(
missing_name
);
if
(
containing
)
{
LOG_ERROR
(
log
,
"Ignoring missing local part "
<<
missing_name
<<
" because part "
<<
containing
->
name
<<
" exists"
);
if
(
unexpected_parts
.
count
(
containing
))
{
parts_to_add
.
push_back
(
containing
);
unexpected_parts
.
erase
(
containing
);
parts_to_add_rows
+=
containing
->
rows_count
;
}
}
if
(
expected_parts
.
count
(
part
->
name
))
parts_to_fetch
.
push_back
(
part
->
name
);
/// these parts we will fetch from other replicas
else
parts_to_fetch
.
push_back
(
missing_name
);
unexpected_parts
.
insert
(
part
);
/// this parts we will place to detached with ignored_ prefix
}
for
(
const
String
&
name
:
parts_to_fetch
)
expected_parts
.
erase
(
name
);
/** To check the adequacy, for the parts that are in the FS, but not in ZK, we will only consider not the most recent parts.
* Because unexpected new parts usually arise only because they did not have time to enroll in ZK with a rough restart of the server.
...
...
@@ -620,16 +600,10 @@ void StorageReplicatedMergeTree::checkParts(bool skip_sanity_checks)
for
(
const
String
&
name
:
parts_to_fetch
)
parts_to_fetch_blocks
+=
get_blocks_count_in_data_part
(
name
);
UInt64
expected_parts_blocks
=
0
;
for
(
const
String
&
name
:
expected_parts
)
expected_parts_blocks
+=
get_blocks_count_in_data_part
(
name
);
std
::
stringstream
sanity_report
;
sanity_report
<<
"There are "
<<
unexpected_parts
.
size
()
<<
" unexpected parts with "
<<
unexpected_parts_rows
<<
" rows ("
<<
unexpected_parts_nonnew
<<
" of them is not just-written with "
<<
unexpected_parts_rows
<<
" rows), "
<<
parts_to_add
.
size
()
<<
" unexpectedly merged parts with "
<<
parts_to_add_rows
<<
" rows, "
<<
expected_parts
.
size
()
<<
" missing obsolete parts (with "
<<
expected_parts_blocks
<<
" blocks), "
<<
parts_to_fetch
.
size
()
<<
" missing parts (with "
<<
parts_to_fetch_blocks
<<
" blocks)."
;
/** We can automatically synchronize data,
...
...
@@ -645,45 +619,23 @@ void StorageReplicatedMergeTree::checkParts(bool skip_sanity_checks)
for
(
const
auto
&
part
:
parts
)
total_rows_on_filesystem
+=
part
->
rows_count
;
UInt64
total_suspicious_rows
=
parts_to_add_rows
+
unexpected_parts_rows
;
UInt64
total_suspicious_rows_no_new
=
parts_to_add_rows
+
unexpected_parts_nonnew_rows
;
bool
insane
=
total_suspicious_rows
>
total_rows_on_filesystem
*
data
.
settings
.
replicated_max_ratio_of_wrong_parts
;
bool
insane
=
unexpected_parts_rows
>
total_rows_on_filesystem
*
data
.
settings
.
replicated_max_ratio_of_wrong_parts
;
if
(
insane
&&
!
skip_sanity_checks
)
{
std
::
stringstream
why
;
why
<<
"The local set of parts of table "
<<
database_name
<<
"."
<<
table_name
<<
" doesn't look like the set of parts "
<<
"in ZooKeeper: "
<<
formatReadableQuantity
(
total_suspiciou
s_rows
)
<<
" rows of "
<<
formatReadableQuantity
(
total_rows_on_filesystem
)
<<
formatReadableQuantity
(
unexpected_part
s_rows
)
<<
" rows of "
<<
formatReadableQuantity
(
total_rows_on_filesystem
)
<<
" total rows in filesystem are suspicious."
;
throw
Exception
(
why
.
str
()
+
" "
+
sanity_report
.
str
(),
ErrorCodes
::
TOO_MANY_UNEXPECTED_DATA_PARTS
);
}
if
(
total_suspicious_rows_no_new
>
0
)
if
(
unexpected_parts_nonnew_rows
>
0
)
LOG_WARNING
(
log
,
sanity_report
.
str
());
/// Add information to the ZK about the parts that cover the missing parts.
for
(
const
MergeTreeData
::
DataPartPtr
&
part
:
parts_to_add
)
{
LOG_ERROR
(
log
,
"Adding unexpected local part to ZooKeeper: "
<<
part
->
name
);
Coordination
::
Requests
ops
;
checkPartChecksumsAndAddCommitOps
(
zookeeper
,
part
,
ops
);
zookeeper
->
multi
(
ops
);
}
/// Remove from ZK information about the parts covered by the newly added ones.
{
for
(
const
String
&
name
:
expected_parts
)
LOG_ERROR
(
log
,
"Removing unexpectedly merged local part from ZooKeeper: "
<<
name
);
removePartsFromZooKeeper
(
zookeeper
,
Strings
(
expected_parts
.
begin
(),
expected_parts
.
end
()));
}
/// Add to the queue jobs to pick up the missing parts from other replicas and remove from ZK the information that we have them.
std
::
vector
<
std
::
future
<
Coordination
::
ExistsResponse
>>
exists_futures
;
exists_futures
.
reserve
(
parts_to_fetch
.
size
());
for
(
const
String
&
part_name
:
parts_to_fetch
)
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录