Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
BaiXuePrincess
milvus
提交
a8bd2cf4
milvus
项目概览
BaiXuePrincess
/
milvus
与 Fork 源项目一致
从无法访问的项目Fork
通知
7
Star
4
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
a8bd2cf4
编写于
7月 09, 2020
作者:
S
shengjun.li
提交者:
JinHai-CN
7月 09, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix too many data copies (#2784)
Signed-off-by:
N
shengjun.li
<
shengjun.li@zilliz.com
>
上级
e587c286
变更
5
隐藏空白更改
内联
并排
Showing
5 changed file
with
16 addition
and
20 deletion
+16
-20
CHANGELOG.md
CHANGELOG.md
+2
-0
core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.cpp
...c/index/knowhere/knowhere/index/vector_index/IndexIVF.cpp
+4
-6
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFPQ.cpp
...index/knowhere/knowhere/index/vector_index/IndexIVFPQ.cpp
+6
-6
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQ.cpp
...index/knowhere/knowhere/index/vector_index/IndexIVFSQ.cpp
+3
-5
core/src/index/knowhere/knowhere/index/vector_index/IndexNSG.cpp
...c/index/knowhere/knowhere/index/vector_index/IndexNSG.cpp
+1
-3
未找到文件。
CHANGELOG.md
浏览文件 @
a8bd2cf4
...
...
@@ -19,6 +19,8 @@ Please mark all changes in change log and use the issue from GitHub
-
\#
2649 Search parameter of annoy has conflict with document
-
\#
2690 Remove body parser in show-partitions endpoints
-
\#
2692 Milvus hangs during multi-thread concurrent search
-
\#
2739 Fix mishards start failed
-
\#
2776 Fix too many data copies during creating IVF index
## Feature
-
\#
2319 Redo metadata to support MVCC
...
...
core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.cpp
浏览文件 @
a8bd2cf4
...
...
@@ -66,13 +66,11 @@ void
IVF
::
Train
(
const
DatasetPtr
&
dataset_ptr
,
const
Config
&
config
)
{
GETTENSOR
(
dataset_ptr
)
faiss
::
Index
*
coarse_quantizer
=
new
faiss
::
IndexFlatL2
(
dim
);
int64_t
nlist
=
config
[
IndexParams
::
nlist
].
get
<
int64_t
>
();
faiss
::
MetricType
metric_type
=
GetMetricType
(
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
());
auto
index
=
std
::
make_shared
<
faiss
::
IndexIVFFlat
>
(
coarse_quantizer
,
dim
,
nlist
,
metric_type
);
in
dex
->
train
(
rows
,
(
float
*
)
p_data
);
index_
.
reset
(
faiss
::
clone_index
(
index
.
get
())
);
faiss
::
Index
*
coarse_quantizer
=
new
faiss
::
IndexFlat
(
dim
,
metric_type
);
in
t64_t
nlist
=
config
[
IndexParams
::
nlist
].
get
<
int64_t
>
(
);
index_
=
std
::
shared_ptr
<
faiss
::
Index
>
(
new
faiss
::
IndexIVFFlat
(
coarse_quantizer
,
dim
,
nlist
,
metric_type
));
index_
->
train
(
rows
,
(
float
*
)
p_data
);
}
void
...
...
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFPQ.cpp
浏览文件 @
a8bd2cf4
...
...
@@ -35,13 +35,13 @@ void
IVFPQ
::
Train
(
const
DatasetPtr
&
dataset_ptr
,
const
Config
&
config
)
{
GETTENSOR
(
dataset_ptr
)
faiss
::
Index
*
coarse_quantizer
=
new
faiss
::
IndexFlat
(
dim
,
GetMetricType
(
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
()
));
auto
index
=
std
::
make_shared
<
faiss
::
IndexIVFPQ
>
(
coarse_quantizer
,
dim
,
config
[
IndexParams
::
nlist
].
get
<
int64_t
>
(),
config
[
IndexParams
::
m
].
get
<
int64_t
>
(),
config
[
IndexParams
::
nbits
].
get
<
int64_t
>
());
index
->
train
(
rows
,
(
float
*
)
p_data
);
faiss
::
MetricType
metric_type
=
GetMetricType
(
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
(
));
faiss
::
Index
*
coarse_quantizer
=
new
faiss
::
IndexFlat
(
dim
,
metric_type
);
index_
=
std
::
shared_ptr
<
faiss
::
Index
>
(
new
faiss
::
IndexIVFPQ
(
coarse_quantizer
,
dim
,
config
[
IndexParams
::
nlist
].
get
<
int64_t
>
(),
config
[
IndexParams
::
m
].
get
<
int64_t
>
(),
config
[
IndexParams
::
nbits
].
get
<
int64_t
>
(),
metric_type
)
);
index_
.
reset
(
faiss
::
clone_index
(
index
.
get
())
);
index_
->
train
(
rows
,
(
float
*
)
p_data
);
}
VecIndexPtr
...
...
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQ.cpp
浏览文件 @
a8bd2cf4
...
...
@@ -38,11 +38,9 @@ IVFSQ::Train(const DatasetPtr& dataset_ptr, const Config& config) {
std
::
stringstream
index_type
;
index_type
<<
"IVF"
<<
config
[
IndexParams
::
nlist
]
<<
","
<<
"SQ"
<<
config
[
IndexParams
::
nbits
];
auto
build_index
=
faiss
::
index_factory
(
dim
,
index_type
.
str
().
c_str
(),
GetMetricType
(
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
()));
build_index
->
train
(
rows
,
(
float
*
)
p_data
);
index_
.
reset
(
faiss
::
clone_index
(
build_index
));
index_
=
std
::
shared_ptr
<
faiss
::
Index
>
(
faiss
::
index_factory
(
dim
,
index_type
.
str
().
c_str
(),
GetMetricType
(
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
())));
index_
->
train
(
rows
,
(
float
*
)
p_data
);
}
VecIndexPtr
...
...
core/src/index/knowhere/knowhere/index/vector_index/IndexNSG.cpp
浏览文件 @
a8bd2cf4
...
...
@@ -139,9 +139,7 @@ NSG::Train(const DatasetPtr& dataset_ptr, const Config& config) {
b_params
.
out_degree
=
config
[
IndexParams
::
out_degree
];
b_params
.
search_length
=
config
[
IndexParams
::
search_length
];
auto
p_ids
=
dataset_ptr
->
Get
<
const
int64_t
*>
(
meta
::
IDS
);
GETTENSOR
(
dataset_ptr
)
GETTENSORWITHIDS
(
dataset_ptr
)
index_
=
std
::
make_shared
<
impl
::
NsgIndex
>
(
dim
,
rows
,
config
[
Metric
::
TYPE
].
get
<
std
::
string
>
());
index_
->
SetKnnGraph
(
knng
);
index_
->
Build_with_ids
(
rows
,
(
float
*
)
p_data
,
(
int64_t
*
)
p_ids
,
b_params
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录