Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
milvus
milvus
提交
6f18587f
M
milvus
项目概览
milvus
/
milvus
9 个月 前同步成功
通知
260
Star
22476
Fork
2472
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
未验证
提交
6f18587f
编写于
7月 26, 2023
作者:
X
xige-16
提交者:
GitHub
7月 26, 2023
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Fix small segment compaction (#21327)
Signed-off-by:
N
xige-16
<
xi.ge@zilliz.com
>
上级
b5e79e7f
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
17 addition
and
6 deletion
+17
-6
configs/milvus.yaml
configs/milvus.yaml
+1
-1
internal/datacoord/compaction_trigger.go
internal/datacoord/compaction_trigger.go
+13
-2
internal/datacoord/compaction_trigger_test.go
internal/datacoord/compaction_trigger_test.go
+2
-2
pkg/util/paramtable/component_param.go
pkg/util/paramtable/component_param.go
+1
-1
未找到文件。
configs/milvus.yaml
浏览文件 @
6f18587f
...
...
@@ -338,7 +338,7 @@ dataCoord:
smallProportion
:
0.5
# The segment is considered as "small segment" when its # of rows is smaller than
# (smallProportion * segment max # of rows).
# A compaction will happen on small segments if the segment after compaction will have
compactableProportion
:
0.5
compactableProportion
:
0.
8
5
# over (compactableProportion * segment max # of rows) rows.
# MUST BE GREATER THAN OR EQUAL TO <smallProportion>!!!
# During compaction, the size of segment # of rows is able to exceed segment max # of rows by (expansionRate-1) * 100%.
...
...
internal/datacoord/compaction_trigger.go
浏览文件 @
6f18587f
...
...
@@ -654,8 +654,7 @@ func (t *compactionTrigger) generatePlans(segments []*SegmentInfo, force bool, i
}
// only merge if candidate number is large than MinSegmentToMerge or if target row is large enough
if
len
(
bucket
)
>=
Params
.
DataCoordCfg
.
MinSegmentToMerge
.
GetAsInt
()
||
len
(
bucket
)
>
1
&&
targetRow
>
int64
(
float64
(
segment
.
GetMaxRowNum
())
*
Params
.
DataCoordCfg
.
SegmentCompactableProportion
.
GetAsFloat
())
{
len
(
bucket
)
>
1
&&
t
.
isCompactableSegment
(
targetRow
,
segment
)
{
plan
:=
segmentsToPlan
(
bucket
,
compactTime
)
log
.
Info
(
"generate a plan for small candidates"
,
zap
.
Int64s
(
"plan segmentIDs"
,
lo
.
Map
(
bucket
,
getSegmentIDs
)),
...
...
@@ -798,6 +797,18 @@ func (t *compactionTrigger) isSmallSegment(segment *SegmentInfo) bool {
return
segment
.
GetNumOfRows
()
<
int64
(
float64
(
segment
.
GetMaxRowNum
())
*
Params
.
DataCoordCfg
.
SegmentSmallProportion
.
GetAsFloat
())
}
func
(
t
*
compactionTrigger
)
isCompactableSegment
(
targetRow
int64
,
segment
*
SegmentInfo
)
bool
{
smallProportion
:=
Params
.
DataCoordCfg
.
SegmentSmallProportion
.
GetAsFloat
()
compactableProportion
:=
Params
.
DataCoordCfg
.
SegmentCompactableProportion
.
GetAsFloat
()
// avoid invalid single segment compaction
if
compactableProportion
<
smallProportion
{
compactableProportion
=
smallProportion
}
return
targetRow
>
int64
(
float64
(
segment
.
GetMaxRowNum
())
*
compactableProportion
)
}
func
isExpandableSmallSegment
(
segment
*
SegmentInfo
)
bool
{
return
segment
.
GetNumOfRows
()
<
int64
(
float64
(
segment
.
GetMaxRowNum
())
*
(
Params
.
DataCoordCfg
.
SegmentExpansionRate
.
GetAsFloat
()
-
1
))
}
...
...
internal/datacoord/compaction_trigger_test.go
浏览文件 @
6f18587f
...
...
@@ -1156,8 +1156,8 @@ func Test_compactionTrigger_PrioritizedCandi(t *testing.T) {
spy
:=
(
tt
.
fields
.
compactionHandler
)
.
(
*
spyCompactionHandler
)
select
{
case
val
:=
<-
spy
.
spyChan
:
//
5
segments in the final pick list
assert
.
Equal
(
t
,
len
(
val
.
SegmentBinlogs
),
5
)
//
6
segments in the final pick list
assert
.
Equal
(
t
,
len
(
val
.
SegmentBinlogs
),
6
)
return
case
<-
time
.
After
(
3
*
time
.
Second
)
:
assert
.
Fail
(
t
,
"failed to get plan"
)
...
...
pkg/util/paramtable/component_param.go
浏览文件 @
6f18587f
...
...
@@ -2177,7 +2177,7 @@ the number of binlog file reaches to max value.`,
p
.
SegmentCompactableProportion
=
ParamItem
{
Key
:
"dataCoord.segment.compactableProportion"
,
Version
:
"2.2.1"
,
DefaultValue
:
"0.5"
,
DefaultValue
:
"0.
8
5"
,
Doc
:
`(smallProportion * segment max # of rows).
A compaction will happen on small segments if the segment after compaction will have`
,
Export
:
true
,
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录