Merge remote-tracking branch 'main/master' into caiyd_1548_move_store

CHANGELOG.md

@@ -2,9 +2,10 @@
 
 Please mark all change in change log and use the issue from GitHub
 
-# Milvus 0.8.0 (TBD)
+# Milvus 0.7.1 (TBD)
 
 ## Bug
+-   \#1635 Vectors can be returned by searching after vectors deleted if `cache_insert_data` set true
 
 ## Feature
 

README.md

@@ -31,7 +31,7 @@ Milvus is an open source vector similarity search engine. Built with heterogeneo
 
 For more detailed introduction of Milvus and its architecture, see [Milvus overview](https://www.milvus.io/docs/about_milvus/overview.md).
 
-Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://www.milvus.io/docs/master/releases/release_notes.md).
+Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://www.milvus.io/docs/releases/release_notes.md).
 
 ## Roadmap
 

README_CN.md

@@ -31,7 +31,7 @@ Milvus 是一款开源的特征向量相似性搜索引擎。基于异构众核
 
 若要了解 Milvus 详细介绍和整体架构，请访问 [Milvus 简介](https://www.milvus.io/cn/docs/about_milvus/overview.md)。
 
-通过 [版本发布说明](https://www.milvus.io/cn/docs/master/releases/release_notes.md) 获取最新版本的功能和更新。
+通过 [版本发布说明](https://www.milvus.io/cn/docs/releases/release_notes.md) 获取最新版本的功能和更新。
 
 ## 路线图
 

README_JP.md

@@ -31,7 +31,7 @@ Milvusは特徴ベクトルにむかうオーペンソース高性能類似性
 
 [基本紹介](https://www.milvus.io/docs/about_milvus/overview.md)を参考して詳しい情報を手に入れます。
 
-Milvus [リリースノート](https://www.milvus.io/docs/master/releases/release_notes.md)を読んで最新バージョンや更新情報を手に入れます。
+Milvus [リリースノート](https://www.milvus.io/docs/releases/release_notes.md)を読んで最新バージョンや更新情報を手に入れます。
 
 # ロードマップ
 

core/src/db/engine/ExecutionEngineImpl.cpp

@@ -738,15 +738,22 @@ ExecutionEngineImpl::BuildIndex(const std::string& location, EngineType engine_t
 
     auto status = Status::OK();
     std::vector<segment::doc_id_t> uids;
+    faiss::ConcurrentBitsetPtr blacklist;
     if (from_index) {
         status = to_index->BuildAll(Count(), from_index->GetRawVectors(), from_index->GetRawIds(), conf);
         uids = from_index->GetUids();
+        from_index->GetBlacklist(blacklist);
     } else if (bin_from_index) {
         status = to_index->BuildAll(Count(), bin_from_index->GetRawVectors(), bin_from_index->GetRawIds(), conf);
         uids = bin_from_index->GetUids();
+        bin_from_index->GetBlacklist(blacklist);
     }
     to_index->SetUids(uids);
-    ENGINE_LOG_DEBUG << "set uids " << to_index->GetUids().size() << " for " << location;
+    ENGINE_LOG_DEBUG << "Set " << to_index->GetUids().size() << "uids for " << location;
+    if (blacklist != nullptr) {
+        to_index->SetBlacklist(blacklist);
+        ENGINE_LOG_DEBUG << "Set blacklist for index " << location;
+    }
 
     if (!status.ok()) {
         throw Exception(DB_ERROR, status.message());

core/src/server/web_impl/README.md

@@ -680,7 +680,7 @@ Updates the index type and nlist of a collection.
 
 | Parameter    | Description                                                                                                                                                                                              | Required? |
 | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
-| `index_type` | The type of indexing method to query the collection. Please refer to [Index Types](https://www.milvus.io/docs/reference/index.md) for detailed introduction of supported indexes. The default is "FLAT". | No        |
+| `index_type` | The type of indexing method to query the collection. Please refer to [Milvus Indexes](https://www.milvus.io/docs/guides/index.md) for detailed introduction of supported indexes. The default is "FLAT". | No        |
 | `params`     | The extra params of indexing method to query the collection. Please refer to [Index and search parameters](#Index-and-search-parameters) for detailed introduction of supported indexes.                                              | No        |
 
 ##### Query Parameters
@@ -1543,7 +1543,7 @@ For each index type, the RESTful API has specific index parameters and search pa
 </tr>
 </table>
 
-For detailed information about the parameters above, refer to [Index Types](https://milvus.io/docs/v0.7.0/reference/index.md)
+For detailed information about the parameters above, refer to [Milvus Indexes](https://milvus.io/docs/guides/index.md)
 
 ## Error Codes
 

core/unittest/db/utils.cpp

@@ -65,7 +65,7 @@ static const char *CONFIG_STR =
     "  cpu_cache_capacity: 4             # GB, CPU memory used for cache\n"
     "  cpu_cache_threshold: 0.85         # percentage of data kept when cache cleanup triggered\n"
     "  insert_buffer_size: 4             # GB, maximum insert buffer size allowed\n"
-    "  cache_insert_data: false          # whether load inserted data into cache\n"
+    "  cache_insert_data: true          # whether load inserted data into cache\n"
     "\n"
     "engine_config:\n"
     "  use_blas_threshold: 20\n"
@@ -179,6 +179,7 @@ DBTest::SetUp() {
     milvus::scheduler::JobMgrInst::GetInstance()->Start();
 
     auto options = GetOptions();
+    options.insert_cache_immediately_ = true;
     db_ = milvus::engine::DBFactory::Build(options);
 
     std::string config_path(options.meta_.path_ + CONFIG_FILE);