Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
milvus
milvus
提交
4b712284
M
milvus
项目概览
milvus
/
milvus
大约 1 年 前同步成功
通知
261
Star
22476
Fork
2472
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
4b712284
编写于
5月 19, 2021
作者:
X
XuanYang-cn
提交者:
GitHub
5月 19, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add ddl flush design (#5289)
Signed-off-by:
N
yangxuan
<
xuan.yang@zilliz.com
>
上级
9b37cab9
变更
2
显示空白变更内容
内联
并排
Showing
2 changed file
with
80 addition
and
18 deletion
+80
-18
docs/design_docs/datanode_ddl_flush_design_0519_2021.md
docs/design_docs/datanode_ddl_flush_design_0519_2021.md
+68
-0
docs/design_docs/datanode_recovery_design_0513_2021.md
docs/design_docs/datanode_recovery_design_0513_2021.md
+12
-18
未找到文件。
docs/design_docs/datanode_ddl_flush_design_0519_2021.md
0 → 100644
浏览文件 @
4b712284
# DataNode DDL Flush Design
update: 5.19.2021, by
[
Goose
](
https://github.com/XuanYang-cn
)
## Background
Data Definition Language (DDL) is a language used to define data structures and modify data
<sup>
[
1
](
#techterms1
)
</sup>
.
In Milvus terminology, for instance,
`CreateCollection`
and
`DropPartition`
etc. are DDL. In order to recover
or redo DD operations, DataNode flushes DDLs into persistent storages.
Before this design, DataNode buffers DDL chunks by collection, flushes all buffered data in manul/auto flush.
Now in
[
DataNode Recovery Design
](
datanode_recover_design_0513_2021.md
)
, flowgraph : vchannel = 1 : 1, and insert
data of one segment is always in one vchannel. So each flowgraph concerns only about ONE specific collection. For
DDL channels, one flowgraph only cares about DDL operations of one collection.
## Goals
-
Flowgraph knows about which segment/collection to concern.
-
DDNode update masPositions once it buffers ddl about the collection
-
DDNode buffers binlog Paths generated by auto-flush
-
In manul-flush, a background flush-complete goroutinue waits for DDNode and InsertBufferNode both done flushing,
waiting for both binlog paths.
## Detailed design
1.
Redisign of DDL binlog paths and etcd paths for these binlog paths
DDL flushes based on a manul flush of a segment.
**Former design**
```
# minIO/S3 ddl binlog paths
${tenant}/data_definition_log/${collection_id}/ts/${log_idx}
${tenant}/data_definition_log/${collection_id}/ddl/${log_idx}
# etcd paths for ddl binlog paths
${prefix}/${collectionID}/${idx}
```
The minIO/S3 ddl binlog paths seems ok, but etcd paths aren't clear, especially when we want to relate a ddl flush
to a certain segment flush.
**Redesign**
```
# etcd paths for ddl binlog paths
${prefix}/${collectionID}/${segmentID}/${idx}
```
```
message SaveBinlogPathsRequest {
common.MsgBase base = 1;
int64 segmentID = 2;
int64 collectionID = 3;
ID2PathList field2BinlogPaths = 4;
repeated DDLBinlogMeta = 5;
repeated internal.MsgPosition start_positions = 7;
repeated internal.MsgPosition end_positions = 8;
}
```
## TODOs
1.
Refactor auto-flush of ddNode
3.
Refactor etcd paths
<a
name=
"techterms1"
>
[
1]</a>: *[techterms.com
](
https://techterms.com/definition/ddl#:~:text=Stands%20for%20%22Data%20Definition%20Language,SQL%2C%20the%20Structured%20Query%20Language
)
*
docs/design_docs/datanode_recovery_design_0513_2021.md
浏览文件 @
4b712284
...
...
@@ -63,8 +63,6 @@ manul-flush and upload to DataServce together.
```
proto
rpc
SaveBinlogPaths
(
SaveBinlogPathsRequest
)
returns
(
common.Status
){}
message
ID2PathList
{
int64
ID
=
1
;
repeated
string
Paths
=
2
;
...
...
@@ -87,20 +85,16 @@ message SaveBinlogPathsRequest {
The same as DataNode
```
proto
message
FieldFlushMeta
{
// key: ${prefix}/${segmentID}/${fieldID}/${idx}
message
SegmentFieldBinlogMeta
{
int64
fieldID
=
1
;
repeated
string
binlog_paths
=
2
;
}
message
SegmentFlushMeta
{
int64
segmentID
=
1
;
bool
is_flushed
=
2
;
repeated
FieldFlushMeta
fields
=
5
;
string
binlog_path
=
2
;
}
message
DDLFlushMeta
{
int64
collectionID
=
1
;
repeated
string
binlog_paths
=
2
;
// key: ${prefix}/${collectionID}/${idx}
message
DDLBinlogMeta
{
string
ddl_binlog_path
=
1
;
string
ts_binlog_path
=
2
;
}
```
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录