Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
milvus
milvus
提交
5bcb9429
M
milvus
项目概览
milvus
/
milvus
11 个月 前同步成功
通知
261
Star
22476
Fork
2472
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
5bcb9429
编写于
6月 11, 2021
作者:
X
XuanYang-cn
提交者:
zhenshan.cao
6月 15, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Set start positions in datanode (#5757)
Signed-off-by:
N
yangxuan
<
xuan.yang@zilliz.com
>
上级
7beab6ed
变更
5
展开全部
显示空白变更内容
内联
并排
Showing
5 changed file
with
212 addition
and
169 deletion
+212
-169
internal/datanode/collection_replica.go
internal/datanode/collection_replica.go
+28
-0
internal/datanode/data_sync_service.go
internal/datanode/data_sync_service.go
+2
-0
internal/datanode/flow_graph_insert_buffer_node.go
internal/datanode/flow_graph_insert_buffer_node.go
+9
-6
internal/proto/data_service.proto
internal/proto/data_service.proto
+4
-3
internal/proto/datapb/data_service.pb.go
internal/proto/datapb/data_service.pb.go
+169
-160
未找到文件。
internal/datanode/collection_replica.go
浏览文件 @
5bcb9429
...
...
@@ -20,6 +20,7 @@ import (
"go.uber.org/zap"
"github.com/milvus-io/milvus/internal/log"
"github.com/milvus-io/milvus/internal/proto/datapb"
"github.com/milvus-io/milvus/internal/proto/internalpb"
"github.com/milvus-io/milvus/internal/types"
)
...
...
@@ -42,6 +43,7 @@ type Replica interface {
getChannelName
(
segID
UniqueID
)
(
string
,
error
)
setStartPositions
(
segmentID
UniqueID
,
startPos
[]
*
internalpb
.
MsgPosition
)
error
setEndPositions
(
segmentID
UniqueID
,
endPos
[]
*
internalpb
.
MsgPosition
)
error
getAllStartPositions
()
[]
*
datapb
.
SegmentStartPosition
getSegmentPositions
(
segID
UniqueID
)
([]
*
internalpb
.
MsgPosition
,
[]
*
internalpb
.
MsgPosition
)
listOpenSegmentCheckPointAndNumRows
(
segs
[]
UniqueID
)
(
map
[
UniqueID
]
internalpb
.
MsgPosition
,
map
[
UniqueID
]
int64
)
}
...
...
@@ -185,6 +187,32 @@ func (replica *CollectionSegmentReplica) addSegment(
return
nil
}
func
(
replica
*
CollectionSegmentReplica
)
getAllStartPositions
()
[]
*
datapb
.
SegmentStartPosition
{
replica
.
mu
.
RLock
()
defer
replica
.
mu
.
RUnlock
()
result
:=
make
([]
*
datapb
.
SegmentStartPosition
,
0
,
len
(
replica
.
segments
))
for
id
,
seg
:=
range
replica
.
segments
{
if
seg
.
isNew
.
Load
()
.
(
bool
)
{
pos
,
ok
:=
replica
.
startPositions
[
id
]
if
!
ok
{
log
.
Warn
(
"Segment has no start positions"
)
continue
}
result
=
append
(
result
,
&
datapb
.
SegmentStartPosition
{
SegmentID
:
id
,
StartPosition
:
pos
[
0
],
})
seg
.
isNew
.
Store
(
false
)
}
}
return
result
}
func
(
replica
*
CollectionSegmentReplica
)
removeSegment
(
segmentID
UniqueID
)
error
{
replica
.
mu
.
Lock
()
delete
(
replica
.
segments
,
segmentID
)
...
...
internal/datanode/data_sync_service.go
浏览文件 @
5bcb9429
...
...
@@ -119,6 +119,7 @@ func (dsService *dataSyncService) initNodes(vchanInfo *datapb.VchannelInfo) {
zap
.
Int64
(
"SegmentID"
,
fu
.
segID
),
zap
.
Int64
(
"CollectionID"
,
fu
.
collID
),
zap
.
Int
(
"Length of Field2BinlogPaths"
,
len
(
id2path
)),
zap
.
Any
(
"Start Positions"
,
fu
.
startPositions
),
)
req
:=
&
datapb
.
SaveBinlogPathsRequest
{
...
...
@@ -132,6 +133,7 @@ func (dsService *dataSyncService) initNodes(vchanInfo *datapb.VchannelInfo) {
CollectionID
:
fu
.
collID
,
Field2BinlogPaths
:
id2path
,
CheckPoints
:
checkPoints
,
StartPositions
:
fu
.
startPositions
,
Flushed
:
fu
.
flushed
,
}
rsp
,
err
:=
dsService
.
dataService
.
SaveBinlogPaths
(
dsService
.
ctx
,
req
)
...
...
internal/datanode/flow_graph_insert_buffer_node.go
浏览文件 @
5bcb9429
...
...
@@ -76,6 +76,7 @@ type segmentFlushUnit struct {
segID
UniqueID
field2Path
map
[
UniqueID
]
string
checkPoint
map
[
UniqueID
]
segmentCheckPoint
startPositions
[]
*
datapb
.
SegmentStartPosition
flushed
bool
}
...
...
@@ -714,7 +715,9 @@ func flushSegment(
_
,
ep
:=
ibNode
.
replica
.
getSegmentPositions
(
segID
)
sta
,
_
:=
ibNode
.
replica
.
getSegmentStatisticsUpdates
(
segID
)
ibNode
.
setSegmentCheckPoint
(
segID
,
segmentCheckPoint
{
sta
.
NumRows
,
*
ep
[
0
]})
flushUnit
<-
segmentFlushUnit
{
collID
:
collID
,
segID
:
segID
,
field2Path
:
field2Path
}
startPos
:=
ibNode
.
replica
.
getAllStartPositions
()
flushUnit
<-
segmentFlushUnit
{
collID
:
collID
,
segID
:
segID
,
field2Path
:
field2Path
,
startPositions
:
startPos
}
clearFn
(
true
)
}
...
...
internal/proto/data_service.proto
浏览文件 @
5bcb9429
...
...
@@ -245,9 +245,9 @@ message ID2PathList {
repeated
string
Paths
=
2
;
}
message
PositionPair
{
message
SegmentStartPosition
{
internal.MsgPosition
start_position
=
1
;
int
ernal.MsgPosition
end_position
=
2
;
int
64
segmentID
=
2
;
}
message
SaveBinlogPathsRequest
{
...
...
@@ -256,7 +256,8 @@ message SaveBinlogPathsRequest {
int64
collectionID
=
3
;
repeated
ID2PathList
field2BinlogPaths
=
4
;
repeated
CheckPoint
checkPoints
=
5
;
bool
flushed
=
6
;
repeated
SegmentStartPosition
start_positions
=
6
;
bool
flushed
=
7
;
}
message
CheckPoint
{
...
...
internal/proto/datapb/data_service.pb.go
浏览文件 @
5bcb9429
此差异已折叠。
点击以展开。
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录