Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
BaiXuePrincess
milvus
提交
42806de4
milvus
项目概览
BaiXuePrincess
/
milvus
与 Fork 源项目一致
从无法访问的项目Fork
通知
7
Star
4
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
42806de4
编写于
5月 22, 2020
作者:
S
shengjun.li
提交者:
JinHai-CN
6月 10, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix MatchNlist (#2401)
Signed-off-by:
N
shengjun.li
<
shengjun.li@zilliz.com
>
上级
2be4005d
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
15 addition
and
12 deletion
+15
-12
CHANGELOG.md
CHANGELOG.md
+3
-0
core/src/index/knowhere/knowhere/index/vector_index/ConfAdapter.cpp
...ndex/knowhere/knowhere/index/vector_index/ConfAdapter.cpp
+11
-12
core/src/index/knowhere/knowhere/index/vector_index/ConfAdapter.h
.../index/knowhere/knowhere/index/vector_index/ConfAdapter.h
+1
-0
未找到文件。
CHANGELOG.md
浏览文件 @
42806de4
...
...
@@ -17,8 +17,11 @@ Please mark all change in change log and use the issue from GitHub
## Bug
-
\#
2378 Duplicate data after server restart
-
\#
2399 The nlist set by the user may not take effect
## Feature
-
\#
2363 Update branch version
-
\#
2381 Upgrade to faiss_1.6.3
## Improvement
-
\#
2307 Disable index SPTAG by default
...
...
core/src/index/knowhere/knowhere/index/vector_index/ConfAdapter.cpp
浏览文件 @
42806de4
...
...
@@ -75,14 +75,13 @@ ConfAdapter::CheckSearch(Config& oricfg, const IndexType type, const IndexMode m
}
int64_t
MatchNlist
(
const
int64_t
&
size
,
const
int64_t
&
nlist
,
const
int64_t
&
per_nlist
)
{
static
float
TYPICAL_COUNT
=
1000000.0
;
if
(
size
<=
TYPICAL_COUNT
/
per_nlist
+
1
)
{
// handle less row count, avoid nlist set to 0
return
1
;
}
else
if
(
int
(
size
/
TYPICAL_COUNT
)
*
nlist
<=
0
)
{
// calculate a proper nlist if nlist not specified or size less than TYPICAL_COUNT
return
int
(
size
/
TYPICAL_COUNT
*
per_nlist
);
MatchNlist
(
int64_t
size
,
int64_t
nlist
)
{
const
int64_t
TYPICAL_COUNT
=
1000000
;
const
int64_t
PER_NLIST
=
16384
;
if
(
nlist
*
TYPICAL_COUNT
>
size
*
PER_NLIST
)
{
// nlist is too large, adjust to a proper value
nlist
=
std
::
max
(
1L
,
size
*
PER_NLIST
/
TYPICAL_COUNT
);
}
return
nlist
;
}
...
...
@@ -101,7 +100,7 @@ IVFConfAdapter::CheckTrain(Config& oricfg, const IndexMode mode) {
// auto tune params
int64_t
nq
=
oricfg
[
knowhere
::
meta
::
ROWS
].
get
<
int64_t
>
();
int64_t
nlist
=
oricfg
[
knowhere
::
IndexParams
::
nlist
].
get
<
int64_t
>
();
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
nq
,
nlist
,
16384
);
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
nq
,
nlist
);
// Best Practice
// static int64_t MIN_POINTS_PER_CENTROID = 40;
...
...
@@ -157,8 +156,8 @@ IVFPQConfAdapter::CheckTrain(Config& oricfg, const IndexMode mode) {
// CheckIntByRange(knowhere::meta::ROWS, nlist, DEFAULT_MAX_ROWS);
// auto tune params
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
oricfg
[
knowhere
::
meta
::
ROWS
].
get
<
int64_t
>
(),
oricfg
[
knowhere
::
IndexParams
::
nlist
].
get
<
int64_t
>
(),
16384
);
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
oricfg
[
knowhere
::
meta
::
ROWS
].
get
<
int64_t
>
(),
oricfg
[
knowhere
::
IndexParams
::
nlist
].
get
<
int64_t
>
()
);
// Best Practice
// static int64_t MIN_POINTS_PER_CENTROID = 40;
...
...
@@ -216,7 +215,7 @@ NSGConfAdapter::CheckTrain(Config& oricfg, const IndexMode mode) {
CheckIntByRange
(
knowhere
::
IndexParams
::
candidate
,
MIN_CANDIDATE_POOL_SIZE
,
MAX_CANDIDATE_POOL_SIZE
);
// auto tune params
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
oricfg
[
knowhere
::
meta
::
ROWS
].
get
<
int64_t
>
(),
8192
,
8192
);
oricfg
[
knowhere
::
IndexParams
::
nlist
]
=
MatchNlist
(
oricfg
[
knowhere
::
meta
::
ROWS
].
get
<
int64_t
>
(),
8192
);
int64_t
nprobe
=
int
(
oricfg
[
knowhere
::
IndexParams
::
nlist
].
get
<
int64_t
>
()
*
0.1
);
oricfg
[
knowhere
::
IndexParams
::
nprobe
]
=
nprobe
<
1
?
1
:
nprobe
;
...
...
core/src/index/knowhere/knowhere/index/vector_index/ConfAdapter.h
浏览文件 @
42806de4
...
...
@@ -11,6 +11,7 @@
#pragma once
#include <algorithm>
#include <memory>
#include <vector>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录