Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
2dot5
ClickHouse
提交
10bad32f
C
ClickHouse
项目概览
2dot5
/
ClickHouse
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
C
ClickHouse
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
10bad32f
编写于
10月 29, 2020
作者:
N
Nikolai Kochetov
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Refactor code a little bit. Add comment.
上级
f995ef97
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
39 addition
and
36 deletion
+39
-36
src/Storages/MergeTree/MergeTreeReadPool.cpp
src/Storages/MergeTree/MergeTreeReadPool.cpp
+39
-36
未找到文件。
src/Storages/MergeTree/MergeTreeReadPool.cpp
浏览文件 @
10bad32f
...
...
@@ -267,51 +267,41 @@ void MergeTreeReadPool::fillPerThreadInfo(
size_t
sum_marks
;
size_t
part_idx
;
};
std
::
map
<
String
,
std
::
vector
<
PartInfo
>>
parts_per_disk
;
for
(
size_t
i
=
0
;
i
<
parts
.
size
();
++
i
)
{
PartInfo
part_info
{
parts
[
i
],
per_part_sum_marks
[
i
],
i
};
if
(
parts
[
i
].
data_part
->
isStoredOnDisk
())
parts_per_disk
[
parts
[
i
].
data_part
->
volume
->
getDisk
()
->
getName
()].
push_back
(
std
::
move
(
part_info
));
else
parts_per_disk
[
""
].
push_back
(
std
::
move
(
part_info
));
}
const
size_t
min_marks_per_thread
=
(
sum_marks
-
1
)
/
threads
+
1
;
using
PartsInfo
=
std
::
vector
<
PartInfo
>
;
std
::
queue
<
PartsInfo
>
parts_queue
;
auto
it
=
std
::
prev
(
parts_per_disk
.
end
());
auto
*
current_parts
=
&
it
->
second
;
auto
get_next_parts
=
[
&
]()
{
size_t
n
=
parts_per_disk
.
size
();
do
/// Group parts by volume name.
/// We try minimize the number of threads concurrently read from the same volume.
/// It improves the performance for JBOD architecture.
std
::
map
<
String
,
std
::
vector
<
PartInfo
>>
parts_per_disk
;
for
(
size_t
i
=
0
;
i
<
parts
.
size
();
++
i
)
{
++
it
;
if
(
it
==
parts_per_disk
.
end
())
it
=
parts_per_disk
.
begin
();
current_parts
=
&
it
->
second
;
}
while
(
current_parts
->
empty
()
&&
--
n
);
return
!
current_parts
->
empty
();
};
PartInfo
part_info
{
parts
[
i
],
per_part_sum_marks
[
i
],
i
};
if
(
parts
[
i
].
data_part
->
isStoredOnDisk
())
parts_per_disk
[
parts
[
i
].
data_part
->
volume
->
getDisk
()
->
getName
()].
push_back
(
std
::
move
(
part_info
));
else
parts_per_disk
[
""
].
push_back
(
std
::
move
(
part_info
));
}
auto
get_current_parts
=
[
&
]()
{
if
(
!
current_parts
->
empty
())
return
true
;
return
get_next_parts
();
};
for
(
auto
&
info
:
parts_per_disk
)
parts_queue
.
push
(
std
::
move
(
info
.
second
));
}
for
(
size_t
i
=
0
;
i
<
threads
&&
get_next_parts
();
++
i
)
const
size_t
min_marks_per_thread
=
(
sum_marks
-
1
)
/
threads
+
1
;
for
(
size_t
i
=
0
;
i
<
threads
&&
!
parts_queue
.
empty
();
++
i
)
{
auto
need_marks
=
min_marks_per_thread
;
while
(
need_marks
>
0
&&
get_current_parts
())
while
(
need_marks
>
0
&&
!
parts_queue
.
empty
())
{
RangesInDataPart
&
part
=
current_parts
->
back
().
part
;
size_t
&
marks_in_part
=
current_parts
->
back
().
sum_marks
;
const
auto
part_idx
=
current_parts
->
back
().
part_idx
;
auto
&
current_parts
=
parts_queue
.
front
();
RangesInDataPart
&
part
=
current_parts
.
back
().
part
;
size_t
&
marks_in_part
=
current_parts
.
back
().
sum_marks
;
const
auto
part_idx
=
current_parts
.
back
().
part_idx
;
/// Do not get too few rows from part.
if
(
marks_in_part
>=
min_marks_for_concurrent_read
&&
...
...
@@ -333,7 +323,9 @@ void MergeTreeReadPool::fillPerThreadInfo(
marks_in_ranges
=
marks_in_part
;
need_marks
-=
marks_in_part
;
current_parts
->
pop_back
();
current_parts
.
pop_back
();
if
(
current_parts
.
empty
())
parts_queue
.
pop
();
}
else
{
...
...
@@ -362,6 +354,17 @@ void MergeTreeReadPool::fillPerThreadInfo(
if
(
marks_in_ranges
!=
0
)
remaining_thread_tasks
.
insert
(
i
);
}
/// Before processing next thread, change volume if possible.
/// Different threads will likely start reading from different volumes,
/// which may improve read parallelism for JBOD.
/// It also may be helpful in case we have backoff threads.
/// Backoff threads will likely to reduce load for different disks, not the same one.
if
(
parts_queue
.
size
()
>
1
)
{
parts_queue
.
push
(
std
::
move
(
parts_queue
.
front
()));
parts_queue
.
pop
();
}
}
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录