Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
milvus
milvus
提交
61897166
M
milvus
项目概览
milvus
/
milvus
12 个月 前同步成功
通知
261
Star
22476
Fork
2472
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
61897166
编写于
10月 12, 2021
作者:
T
ThyeeZz
提交者:
GitHub
10月 12, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[skip ci] update docs (#9726)
Signed-off-by:
N
ThyeeZz
<
jialian.ji@zilliz.com
>
上级
062902b0
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
22 addition
and
13 deletion
+22
-13
docs/design_docs/datanode_flowgraph_recovery_design_0604_2021.md
...sign_docs/datanode_flowgraph_recovery_design_0604_2021.md
+22
-13
未找到文件。
docs/design_docs/datanode_flowgraph_recovery_design_0604_2021.md
浏览文件 @
61897166
...
...
@@ -4,36 +4,44 @@ update: 6.4.2021, by [Goose](https://github.com/XuanYang-cn)
update: 6.21.2021, by
[
Goose
](
https://github.com/XuanYang-cn
)
## 1. Common Sense
A. 1 message stream to 1 vchannel, so there are 1 start position and 1 end position in 1 message pack
A. One message stream to one vchannel, so there are one start position and one end position in one message pack
B. Only when datanode flushes, datanode will update every segment's position
An optimization: update position of
-
a. Current flushing segment
-
b. StartPosition of segments never been flushed.
C. DataNode auto-flush is a valid flush.
D. DDL messages are now in DML Vchannels.
-
a. Current flushing segment
-
b. StartPosition of segments never been flushed.
C. DataNode auto-flush is a valid flush.
D. DDL messages are now in DML Vchannels.
## 2. Segments in Flowgraph
![
segments
](
graphs/segments.png
)
## 3. Flowgraph Recovery
### A. Save checkpoints
When a flowgraph flushes a segment, we need to save these things:
-
current segment's binlog paths,
-
current segment positions,
-
all other segments' current positions from replica (If a segment hasn't been flushed, save the position when datanode first meets it.)
Whether save successfully:
-
If succeeded, flowgraph updates all segments' positions to replica
-
If not
-
For a grpc failure( this failure will appear after many times retry internally), crush itself.
-
For a normal failure, retry save 10 times, if fail still, crush itself.
-
For a grpc failure( this failure will appear after many times retry internally), crush itself.
-
For a normal failure, retry save 10 times, if fail still, crush itself.
### B. Recovery from a set of checkpoints
1.
We need all positions of all segments in this vchannel
`p1, p2, ... pn`
A design of WatchDmChannelReq
```
proto
```
proto
message
VchannelInfo
{
int64
collectionID
=
1
;
string
channelName
=
2
;
...
...
@@ -53,16 +61,17 @@ message WatchDmChannelsRequest {
![
recovery
](
graphs/flowgraph_recovery_design.png
)
Supposing we have segment
`s1, s2, s3`
, corresponding position
`p1, p2, p3`
-
Sort positions in reverse order
`p3, p2, p1`
-
Get segments dup range time:
`s3 ( p3 > mp_px > p1)`
,
`s2 (p2 > mp_px > p1)`
,
`s1(zero)`
-
Seek from the earliest, in this example
`p1`
-
Then for every msgPack after seeking
`p1`
, the pseudocode:
-
Sort positions in reverse order
`p3, p2, p1`
-
Get segments dup range time:
`s3 ( p3 > mp_px > p1)`
,
`s2 (p2 > mp_px > p1)`
,
`s1(zero)`
-
Seek from the earliest, in this example
`p1`
-
Then for every msgPack after seeking
`p1`
, the pseudocode:
```
go
const
filter_threshold
=
recovery_time
// mp means msgPack
for
mp
:=
seeking
(
p1
)
{
if
mp
.
position
.
endtime
<
filter_threshod
{
if
mp
.
position
.
endtime
<
filter_threshod
{
if
mp
.
position
<
p3
{
filter
s3
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录