Merge remote-tracking branch 'main/0.5.1' into dev

Former-commit-id: 1301dfebe6a4adffd65bf50a4a52f6b33765de7f

CHANGELOG.md

@@ -5,6 +5,7 @@ Please mark all change in change log and use the ticket from JIRA.
 # Milvus 0.5.1 (TODO)
 
 ## Bug
+- \#90 - The server start error messages could be improved to enhance user experience
 - \#104 - test_scheduler core dump
 
 ## Improvement

README.md

@@ -20,7 +20,7 @@
 
 ## What is Milvus
 
-Milvus is an open source similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
+Milvus is an open source similarity search engine for massive-scale feature vectors. Built with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
 
 Milvus provides stable Python, Java and C++ APIs.
 
@@ -28,7 +28,7 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
 
 - Heterogeneous computing
 
-  Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency. 
+  Milvus is built with heterogeneous computing architecture for the best performance and cost efficiency. 
 
 - Multiple indexes
 
@@ -64,14 +64,14 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
 
 ## Get started
 
-### Hardware Requirements
+### Hardware requirements
 
 | Component | Recommended configuration           |
 | --------- | ----------------------------------- |
 | CPU       | Intel CPU Haswell or higher         |
 | GPU       | NVIDIA Pascal series or higher      |
-| Memory    | 8 GB or more (depends on data size) |
-| Storage   | SATA 3.0 SSD or higher              |
+| RAM       | 8 GB or more (depends on data size) |
+| Hard drive| SATA 3.0 SSD or higher              |
 
 ### Install using docker
 
@@ -168,6 +168,10 @@ Make sure Java 8 or higher is already installed.
 
 Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code.
 
+## Milvus roadmap
+
+Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
+
 ## Contribution guidelines
 
 Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
@@ -178,9 +182,11 @@ We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to
 
 To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk). 
 
-## Milvus Roadmap
+## Thanks
 
-Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
+We greatly appreciate the help of the following people.
+
+- [akihoni](https://github.com/akihoni) found a broken link and a small typo in the README file.
 
 ## Resources
 
@@ -196,7 +202,6 @@ Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upc
 
 [Milvus roadmap](https://milvus.io/docs/en/roadmap/)
 
-
 ## License
 
 [Apache License 2.0](LICENSE)

ci/jenkinsfile/milvus_build.groovy

@@ -14,7 +14,7 @@ container('milvus-build-env') {
                             sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
                             && export JFROG_USER_NAME='${USERNAME}' \
                             && export JFROG_PASSWORD='${PASSWORD}' \
-                            && export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
+                            && export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
                             && ./build.sh -t ${params.BUILD_TYPE} -d /opt/milvus -j -u -c"
 
                             sh "./coverage.sh -u root -p 123456 -t \$POD_IP"

ci/jenkinsfile/milvus_build_no_ut.groovy

@@ -14,7 +14,7 @@ container('milvus-build-env') {
                             sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
                             && export JFROG_USER_NAME='${USERNAME}' \
                             && export JFROG_PASSWORD='${PASSWORD}' \
-                            && export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
+                            && export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
                             && ./build.sh -t ${params.BUILD_TYPE} -j -d /opt/milvus"
                         }
                     }

core/conf/server_config.template

@@ -2,9 +2,9 @@
 
 server_config:
   address: 0.0.0.0                  # milvus server ip address (IPv4)
-  port: 19530                       # port range: 1025 ~ 65534
+  port: 19530                       # milvus server port, must in range [1025, 65534]
   deploy_mode: single               # deployment type: single, cluster_readonly, cluster_writable
-  time_zone: UTC+8
+  time_zone: UTC+8                  # time zone, must be in format: UTC+X
 
 db_config:
   primary_path: @MILVUS_DB_PATH@    # path used to store data and meta
@@ -14,30 +14,30 @@ db_config:
                                     # Keep 'dialect://:@:/', and replace other texts with real values
                                     # Replace 'dialect' with 'mysql' or 'sqlite'
 
-  insert_buffer_size: 4             # GB, maximum insert buffer size allowed
+  insert_buffer_size: 4             # GB, maximum insert buffer size allowed, must be a positive integer
                                     # sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory
 
   preload_table:                    # preload data at startup, '*' means load all tables, empty value means no preload
                                     # you can specify preload tables like this: table1,table2,table3
 
 metric_config:
-  enable_monitor: false             # enable monitoring or not
+  enable_monitor: false             # enable monitoring or not, must be a boolean
   collector: prometheus             # prometheus
   prometheus_config:
-    port: 8080                      # port prometheus uses to fetch metrics
+    port: 8080                      # port prometheus uses to fetch metrics, must in range [1025, 65534]
 
 cache_config:
-  cpu_cache_capacity: 16            # GB, CPU memory used for cache
-  cpu_cache_threshold: 0.85         # percentage of data that will be kept when cache cleanup is triggered
-  gpu_cache_capacity: 4             # GB, GPU memory used for cache
-  gpu_cache_threshold: 0.85         # percentage of data that will be kept when cache cleanup is triggered
-  cache_insert_data: false          # whether to load inserted data into cache
+  cpu_cache_capacity: 16            # GB, CPU memory used for cache, must be a positive integer
+  cpu_cache_threshold: 0.85         # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
+  gpu_cache_capacity: 4             # GB, GPU memory used for cache, must be a positive integer
+  gpu_cache_threshold: 0.85         # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
+  cache_insert_data: false          # whether to load inserted data into cache, must be a boolean
 
 engine_config:
   use_blas_threshold: 20            # if nq <  use_blas_threshold, use SSE, faster with fluctuated response times
                                     # if nq >= use_blas_threshold, use OpenBlas, slower with stable response times
 
 resource_config:
-  search_resources:                 # define the GPUs used for search computation, valid value: gpux
+  search_resources:                 # define the GPUs used for search computation, must be in format: gpux
     - gpu0
-  index_build_device: gpu0          # GPU used for building index
+  index_build_device: gpu0          # GPU used for building index, must be in format: gpux
\ No newline at end of file

core/src/db/engine/ExecutionEngineImpl.cpp

@@ -18,16 +18,13 @@
 #include "db/engine/ExecutionEngineImpl.h"
 #include "cache/CpuCacheMgr.h"
 #include "cache/GpuCacheMgr.h"
+#include "knowhere/common/Config.h"
 #include "metrics/Metrics.h"
+#include "scheduler/Utils.h"
+#include "server/Config.h"
 #include "utils/CommonUtil.h"
 #include "utils/Exception.h"
 #include "utils/Log.h"
-
-#include "knowhere/common/Config.h"
-#include "knowhere/common/Exception.h"
-#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
-#include "scheduler/Utils.h"
-#include "server/Config.h"
 #include "wrapper/ConfAdapter.h"
 #include "wrapper/ConfAdapterMgr.h"
 #include "wrapper/VecImpl.h"
@@ -260,6 +257,54 @@ ExecutionEngineImpl::Load(bool to_cache) {
 Status
 ExecutionEngineImpl::CopyToGpu(uint64_t device_id, bool hybrid) {
     if (hybrid) {
+        const std::string key = location_ + ".quantizer";
+        std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
+
+        const int64_t NOT_FOUND = -1;
+        int64_t device_id = NOT_FOUND;
+
+        // cache hit
+        {
+            knowhere::QuantizerPtr quantizer = nullptr;
+
+            for (auto& gpu : gpus) {
+                auto cache = cache::GpuCacheMgr::GetInstance(gpu);
+                if (auto cached_quantizer = cache->GetIndex(key)) {
+                    device_id = gpu;
+                    quantizer = std::static_pointer_cast<CachedQuantizer>(cached_quantizer)->Data();
+                }
+            }
+
+            if (device_id != NOT_FOUND) {
+                // cache hit
+                auto config = std::make_shared<knowhere::QuantizerCfg>();
+                config->gpu_id = device_id;
+                config->mode = 2;
+                auto new_index = index_->LoadData(quantizer, config);
+                index_ = new_index;
+            }
+        }
+
+        if (device_id == NOT_FOUND) {
+            // cache miss
+            std::vector<int64_t> all_free_mem;
+            for (auto& gpu : gpus) {
+                auto cache = cache::GpuCacheMgr::GetInstance(gpu);
+                auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
+                all_free_mem.push_back(free_mem);
+            }
+
+            auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
+            auto best_index = std::distance(all_free_mem.begin(), max_e);
+            device_id = gpus[best_index];
+
+            auto pair = index_->CopyToGpuWithQuantizer(device_id);
+            index_ = pair.first;
+
+            // cache
+            auto cached_quantizer = std::make_shared<CachedQuantizer>(pair.second);
+            cache::GpuCacheMgr::GetInstance(device_id)->InsertItem(key, cached_quantizer);
+        }
         return Status::OK();
     }
 

core/src/index/cmake/ThirdPartyPackagesCore.cmake

@@ -244,11 +244,12 @@ if(CUSTOMIZATION)
         # set(FAISS_MD5 "21deb1c708490ca40ecb899122c01403") # commit-id 643e48f479637fd947e7b93fa4ca72b38ecc9a39 branch-0.2.0
         # set(FAISS_MD5 "072db398351cca6e88f52d743bbb9fa0") # commit-id 3a2344d04744166af41ef1a74449d68a315bfe17 branch-0.2.1
         # set(FAISS_MD5 "c89ea8e655f5cdf58f42486f13614714") # commit-id 9c28a1cbb88f41fa03b03d7204106201ad33276b branch-0.2.1
-        set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
+        # set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
+         set(FAISS_MD5 "f3b2ce3364c3fa7febd3aa7fdd0fe380") # commit-id 694e03458e6b69ce8a62502f71f69a614af5af8f branch-0.3.0
     endif()
 else()
-    set(FAISS_SOURCE_URL "https://github.com/facebookresearch/faiss/archive/v1.5.3.tar.gz")
-    set(FAISS_MD5 "0bc12737b23def156f6a1eb782050135")
+    set(FAISS_SOURCE_URL "https://github.com/milvus-io/faiss/archive/1.6.0.tar.gz")
+    set(FAISS_MD5 "eb96d84f98b078a9eec04a796f5c792e")
 endif()
 message(STATUS "FAISS URL = ${FAISS_SOURCE_URL}")
 

core/src/index/knowhere/knowhere/index/vector_index/FaissBaseIndex.h

@@ -38,7 +38,7 @@ class FaissBaseIndex {
     virtual void
     SealImpl();
 
- protected:
+ public:
     std::shared_ptr<faiss::Index> index_ = nullptr;
 };
 

core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVF.cpp

@@ -15,12 +15,12 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/gpu/GpuAutoTune.h>
-#include <faiss/gpu/GpuIndexFlat.h>
+#include <memory>
+
+#include <faiss/gpu/GpuCloner.h>
 #include <faiss/gpu/GpuIndexIVF.h>
 #include <faiss/gpu/GpuIndexIVFFlat.h>
 #include <faiss/index_io.h>
-#include <memory>
 
 #include "knowhere/adapter/VectorAdapter.h"
 #include "knowhere/common/Exception.h"
@@ -86,7 +86,8 @@ GPUIVF::SerializeImpl() {
             faiss::Index* index = index_.get();
             faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(index);
 
-            SealImpl();
+            // TODO(linxj): support seal
+            // SealImpl();
 
             faiss::write_index(host_index, &writer);
             delete host_index;
@@ -130,13 +131,12 @@ void
 GPUIVF::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
     std::lock_guard<std::mutex> lk(mutex_);
 
-    // TODO(linxj): gpu index support GenParams
     if (auto device_index = std::dynamic_pointer_cast<faiss::gpu::GpuIndexIVF>(index_)) {
         auto search_cfg = std::dynamic_pointer_cast<IVFCfg>(cfg);
-        device_index->setNumProbes(search_cfg->nprobe);
+        device_index->nprobe = search_cfg->nprobe;
+        //        assert(device_index->getNumProbes() == search_cfg->nprobe);
 
         {
-            // TODO(linxj): allocate gpu mem
             ResScope rs(res_, gpu_id_);
             device_index->search(n, (float*)data, k, distances, labels);
         }

core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFPQ.cpp

@@ -16,8 +16,10 @@
 // under the License.
 
 #include <faiss/IndexIVFPQ.h>
-#include <faiss/gpu/GpuAutoTune.h>
+#include <faiss/gpu/GpuCloner.h>
 #include <faiss/gpu/GpuIndexIVFPQ.h>
+#include <faiss/index_factory.h>
+
 #include <memory>
 
 #include "knowhere/adapter/VectorAdapter.h"

core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFSQ.cpp

@@ -15,9 +15,10 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/gpu/GpuAutoTune.h>
+#include <faiss/gpu/GpuCloner.h>
+#include <faiss/index_factory.h>
+
 #include <memory>
-#include <utility>
 
 #include "knowhere/adapter/VectorAdapter.h"
 #include "knowhere/common/Exception.h"
@@ -71,13 +72,4 @@ GPUIVFSQ::CopyGpuToCpu(const Config& config) {
     return std::make_shared<IVFSQ>(new_index);
 }
 
-void
-GPUIVFSQ::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
-#ifdef CUSTOMIZATION
-    GPUIVF::search_impl(n, data, k, distances, labels, cfg);
-#else
-    IVF::search_impl(n, data, k, distances, labels, cfg);
-#endif
-}
-
 }  // namespace knowhere

core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFSQ.h

@@ -38,10 +38,6 @@ class GPUIVFSQ : public GPUIVF {
 
     VectorIndexPtr
     CopyGpuToCpu(const Config& config) override;
-
- protected:
-    void
-    search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) override;
 };
 
 }  // namespace knowhere

core/src/index/knowhere/knowhere/index/vector_index/IndexIDMAP.cpp

@@ -15,11 +15,12 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/AutoTune.h>
 #include <faiss/IndexFlat.h>
 #include <faiss/MetaIndexes.h>
-#include <faiss/gpu/GpuAutoTune.h>
+#include <faiss/gpu/GpuCloner.h>
+#include <faiss/index_factory.h>
 #include <faiss/index_io.h>
+
 #include <vector>
 
 #include "knowhere/adapter/VectorAdapter.h"

core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.cpp

@@ -15,15 +15,12 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/AutoTune.h>
-#include <faiss/AuxIndexStructures.h>
 #include <faiss/IVFlib.h>
 #include <faiss/IndexFlat.h>
 #include <faiss/IndexIVF.h>
 #include <faiss/IndexIVFFlat.h>
-#include <faiss/IndexIVFPQ.h>
-#include <faiss/gpu/GpuAutoTune.h>
-#include <faiss/index_io.h>
+#include <faiss/gpu/GpuCloner.h>
+
 #include <chrono>
 #include <memory>
 #include <utility>

core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.h

@@ -30,7 +30,7 @@ namespace knowhere {
 
 using Graph = std::vector<std::vector<int64_t>>;
 
-class IVF : public VectorIndex, protected FaissBaseIndex {
+class IVF : public VectorIndex, public FaissBaseIndex {
  public:
     IVF() : FaissBaseIndex(nullptr) {
     }

core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQ.cpp

@@ -15,7 +15,8 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/gpu/GpuAutoTune.h>
+#include <faiss/gpu/GpuCloner.h>
+#include <faiss/index_factory.h>
 #include <memory>
 
 #include "knowhere/adapter/VectorAdapter.h"
@@ -56,14 +57,7 @@ IVFSQ::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
     if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
         ResScope rs(res, device_id, false);
 
-#ifdef CUSTOMIZATION
-        faiss::gpu::GpuClonerOptions option;
-        option.allInGpu = true;
-
-        auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get(), &option);
-#else
         auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get());
-#endif
 
         std::shared_ptr<faiss::Index> device_index;
         device_index.reset(gpu_index);

core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQHybrid.cpp

@@ -17,19 +17,25 @@
 // under the License.
 
 #include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
-#include <utility>
-#include "faiss/AutoTune.h"
-#include "faiss/gpu/GpuAutoTune.h"
-#include "faiss/gpu/GpuIndexIVF.h"
 #include "knowhere/adapter/VectorAdapter.h"
 #include "knowhere/common/Exception.h"
 
+#include <utility>
+
+#include <faiss/gpu/GpuCloner.h>
+#include <faiss/gpu/GpuIndexIVF.h>
+#include <faiss/index_factory.h>
+
 namespace knowhere {
 
 #ifdef CUSTOMIZATION
 
+// std::mutex g_mutex;
+
 IndexModelPtr
 IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     auto build_cfg = std::dynamic_pointer_cast<IVFSQCfg>(config);
     if (build_cfg != nullptr) {
         build_cfg->CheckValid();  // throw exception
@@ -63,23 +69,25 @@ IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
 
 VectorIndexPtr
 IVFSQHybrid::CopyGpuToCpu(const Config& config) {
+    if (gpu_mode == 0) {
+        return std::make_shared<IVFSQHybrid>(index_);
+    }
     std::lock_guard<std::mutex> lk(mutex_);
 
-    if (auto device_idx = std::dynamic_pointer_cast<faiss::IndexIVF>(index_)) {
-        faiss::Index* device_index = index_.get();
-        faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
+    faiss::Index* device_index = index_.get();
+    faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
 
-        std::shared_ptr<faiss::Index> new_index;
-        new_index.reset(host_index);
-        return std::make_shared<IVFSQHybrid>(new_index);
-    } else {
-        // TODO(linxj): why? jinhai
-        return std::make_shared<IVFSQHybrid>(index_);
-    }
+    std::shared_ptr<faiss::Index> new_index;
+    new_index.reset(host_index);
+    return std::make_shared<IVFSQHybrid>(new_index);
 }
 
 VectorIndexPtr
 IVFSQHybrid::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
+    if (gpu_mode != 0) {
+        KNOWHERE_THROW_MSG("Not a GpuIndex Type");
+    }
+
     if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
         ResScope rs(res, device_id, false);
         faiss::gpu::GpuClonerOptions option;
@@ -105,16 +113,26 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
     FaissBaseIndex::LoadImpl(index_binary);  // load on cpu
     auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
     ivf_index->backup_quantizer();
+    gpu_mode = 0;
 }
 
 void
 IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels,
                          const Config& cfg) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+    //    static int64_t search_count;
+    //    ++search_count;
+
     if (gpu_mode == 2) {
         GPUIVF::search_impl(n, data, k, distances, labels, cfg);
-    } else if (gpu_mode == 1) {
-        ResScope rs(res_, gpu_id_);
-        IVF::search_impl(n, data, k, distances, labels, cfg);
+        //        index_->search(n, (float*)data, k, distances, labels);
+    } else if (gpu_mode == 1) {  // hybrid
+        if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(quantizer_gpu_id_)) {
+            ResScope rs(res, quantizer_gpu_id_, true);
+            IVF::search_impl(n, data, k, distances, labels, cfg);
+        } else {
+            KNOWHERE_THROW_MSG("Hybrid Search Error, can't get gpu: " + std::to_string(quantizer_gpu_id_) + "resource");
+        }
     } else if (gpu_mode == 0) {
         IVF::search_impl(n, data, k, distances, labels, cfg);
     }
@@ -122,16 +140,18 @@ IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distanc
 
 QuantizerPtr
 IVFSQHybrid::LoadQuantizer(const Config& conf) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
     if (quantizer_conf != nullptr) {
         if (quantizer_conf->mode != 1) {
             KNOWHERE_THROW_MSG("mode only support 1 in this func");
         }
     }
-    gpu_id_ = quantizer_conf->gpu_id;
+    auto gpu_id = quantizer_conf->gpu_id;
 
-    if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
-        ResScope rs(res, gpu_id_, false);
+    if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
+        ResScope rs(res, gpu_id, false);
         faiss::gpu::GpuClonerOptions option;
         option.allInGpu = true;
 
@@ -140,7 +160,7 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
         index_composition->quantizer = nullptr;
         index_composition->mode = quantizer_conf->mode;  // only 1
 
-        auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
+        auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
         delete gpu_index;
 
         auto q = std::make_shared<FaissIVFQuantizer>();
@@ -148,16 +168,19 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
         auto& q_ptr = index_composition->quantizer;
         q->size = q_ptr->d * q_ptr->getNumVecs() * sizeof(float);
         q->quantizer = q_ptr;
+        q->gpu_id = gpu_id;
         res_ = res;
         gpu_mode = 1;
         return q;
     } else {
-        KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
+        KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
     }
 }
 
 void
 IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     auto ivf_quantizer = std::dynamic_pointer_cast<FaissIVFQuantizer>(q);
     if (ivf_quantizer == nullptr) {
         KNOWHERE_THROW_MSG("Quantizer type error");
@@ -170,20 +193,27 @@ IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
         //        delete ivf_index->quantizer;
         ivf_index->quantizer = ivf_quantizer->quantizer;
     }
+    quantizer_gpu_id_ = ivf_quantizer->gpu_id;
+    gpu_mode = 1;
 }
 
 void
 IVFSQHybrid::UnsetQuantizer() {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
     if (ivf_index == nullptr) {
         KNOWHERE_THROW_MSG("Index type error");
     }
 
     ivf_index->quantizer = nullptr;
+    quantizer_gpu_id_ = -1;
 }
 
 VectorIndexPtr
 IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
     if (quantizer_conf != nullptr) {
         if (quantizer_conf->mode != 2) {
@@ -192,13 +222,11 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
     } else {
         KNOWHERE_THROW_MSG("conf error");
     }
-    //    if (quantizer_conf->gpu_id != gpu_id_) {
-    //        KNOWHERE_THROW_MSG("quantizer and data must on the same gpu card");
-    //    }
-    gpu_id_ = quantizer_conf->gpu_id;
 
-    if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
-        ResScope rs(res, gpu_id_, false);
+    auto gpu_id = quantizer_conf->gpu_id;
+
+    if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
+        ResScope rs(res, gpu_id, false);
         faiss::gpu::GpuClonerOptions option;
         option.allInGpu = true;
 
@@ -211,18 +239,20 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
         index_composition->quantizer = ivf_quantizer->quantizer;
         index_composition->mode = quantizer_conf->mode;  // only 2
 
-        auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
+        auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
         std::shared_ptr<faiss::Index> new_idx;
         new_idx.reset(gpu_index);
-        auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id_, res);
+        auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id, res);
         return sq_idx;
     } else {
-        KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
+        KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
     }
 }
 
 std::pair<VectorIndexPtr, QuantizerPtr>
 IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& config) {
+    //    std::lock_guard<std::mutex> lk(g_mutex);
+
     if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
         ResScope rs(res, device_id, false);
         faiss::gpu::GpuClonerOptions option;
@@ -242,12 +272,29 @@ IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& c
         auto q = std::make_shared<FaissIVFQuantizer>();
         q->quantizer = index_composition.quantizer;
         q->size = index_composition.quantizer->d * index_composition.quantizer->getNumVecs() * sizeof(float);
+        q->gpu_id = device_id;
         return std::make_pair(new_idx, q);
     } else {
         KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
     }
 }
 
+void
+IVFSQHybrid::set_index_model(IndexModelPtr model) {
+    std::lock_guard<std::mutex> lk(mutex_);
+
+    auto host_index = std::static_pointer_cast<IVFIndexModel>(model);
+    if (auto gpures = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
+        ResScope rs(gpures, gpu_id_, false);
+        auto device_index = faiss::gpu::index_cpu_to_gpu(gpures->faiss_res.get(), gpu_id_, host_index->index_.get());
+        index_.reset(device_index);
+        res_ = gpures;
+        gpu_mode = 2;
+    } else {
+        KNOWHERE_THROW_MSG("load index model error, can't get gpu_resource");
+    }
+}
+
 FaissIVFQuantizer::~FaissIVFQuantizer() {
     if (quantizer != nullptr) {
         delete quantizer;
@@ -307,5 +354,10 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
     GPUIVF::LoadImpl(index_binary);
 }
 
+void
+IVFSQHybrid::set_index_model(IndexModelPtr model) {
+    GPUIVF::set_index_model(model);
+}
+
 #endif
 }  // namespace knowhere

core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQHybrid.h

@@ -17,7 +17,9 @@
 
 #pragma once
 
+#include <faiss/gpu/GpuIndexFlat.h>
 #include <faiss/index_io.h>
+
 #include <memory>
 #include <utility>
 
@@ -29,6 +31,7 @@ namespace knowhere {
 #ifdef CUSTOMIZATION
 struct FaissIVFQuantizer : public Quantizer {
     faiss::gpu::GpuIndexFlat* quantizer = nullptr;
+    int64_t gpu_id;
 
     ~FaissIVFQuantizer() override;
 };
@@ -52,6 +55,9 @@ class IVFSQHybrid : public GPUIVFSQ {
     }
 
  public:
+    void
+    set_index_model(IndexModelPtr model) override;
+
     QuantizerPtr
     LoadQuantizer(const Config& conf);
 
@@ -85,6 +91,7 @@ class IVFSQHybrid : public GPUIVFSQ {
 
  protected:
     int64_t gpu_mode = 0;  // 0,1,2
+    int64_t quantizer_gpu_id_ = -1;
 };
 
 }  // namespace knowhere

core/src/index/knowhere/knowhere/index/vector_index/VectorIndex.h

@@ -48,6 +48,7 @@ class VectorIndex : public Index {
     virtual void
     Seal() = 0;
 
+    // TODO(linxj): Deprecated
     virtual VectorIndexPtr
     Clone() = 0;
 

core/src/index/knowhere/knowhere/index/vector_index/helpers/FaissIO.h

@@ -17,7 +17,7 @@
 
 #pragma once
 
-#include <faiss/AuxIndexStructures.h>
+#include <faiss/impl/io.h>
 
 namespace knowhere {
 

core/src/index/thirdparty/versions.txt

@@ -3,4 +3,4 @@ BOOST_VERSION=1.70.0
 GTEST_VERSION=1.8.1
 LAPACK_VERSION=v3.8.0
 OPENBLAS_VERSION=v0.3.6
-FAISS_VERSION=branch-0.2.1
+FAISS_VERSION=branch-0.3.0
\ No newline at end of file

core/src/index/unittest/CMakeLists.txt

@@ -86,5 +86,6 @@ install(TARGETS test_gpuresource DESTINATION unittest)
 install(TARGETS test_customized_index DESTINATION unittest)
 
 #add_subdirectory(faiss_ori)
+#add_subdirectory(faiss_benchmark)
 add_subdirectory(test_nsg)
 

core/src/index/unittest/Helper.h

@@ -26,7 +26,7 @@
 #include "knowhere/index/vector_index/IndexIVFSQ.h"
 #include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
 
-constexpr int DEVICEID = 0;
+int DEVICEID = 0;
 constexpr int64_t DIM = 128;
 constexpr int64_t NB = 10000;
 constexpr int64_t NQ = 10;

core/src/index/unittest/faiss_benchmark/CMakeLists.txt

+include_directories(${INDEX_SOURCE_DIR}/thirdparty)
+include_directories(${INDEX_SOURCE_DIR}/include)
+include_directories(/usr/local/cuda/include)
+include_directories(/usr/local/hdf5/include)
+
+link_directories(/usr/local/cuda/lib64)
+link_directories(/usr/local/hdf5/lib)
+
+set(unittest_libs
+        gtest gmock gtest_main gmock_main)
+
+set(depend_libs
+        faiss openblas lapack hdf5
+        arrow ${ARROW_PREFIX}/lib/libjemalloc_pic.a
+        )
+
+set(basic_libs
+        cudart cublas
+        gomp gfortran pthread
+        )
+
+add_executable(test_faiss_benchmark faiss_benchmark_test.cpp)
+target_link_libraries(test_faiss_benchmark ${depend_libs} ${unittest_libs} ${basic_libs})
+install(TARGETS test_faiss_benchmark DESTINATION unittest)

core/src/index/unittest/faiss_benchmark/README.md

+### To run this FAISS benchmark, please follow these steps:
+ 
+#### Step 1:
+Download the HDF5 source from:
+  https://support.hdfgroup.org/ftp/HDF5/releases/
+and build/install to "/usr/local/hdf5".
+
+#### Step 2:
+Download HDF5 data files from:
+  https://github.com/erikbern/ann-benchmarks
+
+#### Step 3:
+Update 'milvus/core/src/index/unittest/CMakeLists.txt',
+uncomment "#add_subdirectory(faiss_benchmark)".
+
+#### Step 4:
+Build Milvus with unittest enabled: "./build.sh -t Release -u",
+binary 'test_faiss_benchmark' will be generated.
+
+#### Step 5:
+Put HDF5 data files into the same directory with binary 'test_faiss_benchmark'.
+
+#### Step 6:
+Run test binary 'test_faiss_benchmark'.
+

core/src/index/unittest/test_customized_index.cpp

@@ -16,17 +16,23 @@
 // under the License.
 
 #include <gtest/gtest.h>
+#include <thread>
 
 #include "unittest/Helper.h"
 #include "unittest/utils.h"
 
+#include "knowhere/common/Timer.h"
+
 class SingleIndexTest : public DataGen, public TestGpuIndexBase {
  protected:
     void
     SetUp() override {
         TestGpuIndexBase::SetUp();
-        Generate(DIM, NB, NQ);
-        k = K;
+        nb = 1000000;
+        nq = 1000;
+        dim = DIM;
+        Generate(dim, nb, nq);
+        k = 1000;
     }
 
     void
@@ -119,4 +125,113 @@ TEST_F(SingleIndexTest, IVFSQHybrid) {
     }
 }
 
+// TEST_F(SingleIndexTest, thread_safe) {
+//    assert(!xb.empty());
+//
+//    index_type = "IVFSQHybrid";
+//    index_ = IndexFactory(index_type);
+//    auto base = ParamGenerator::GetInstance().Gen(ParameterType::ivfsq);
+//    auto conf = std::dynamic_pointer_cast<knowhere::IVFSQCfg>(base);
+//    conf->nlist = 16384;
+//    conf->k = k;
+//    conf->nprobe = 10;
+//    conf->d = dim;
+//    auto preprocessor = index_->BuildPreprocessor(base_dataset, conf);
+//    index_->set_preprocessor(preprocessor);
+//
+//    auto model = index_->Train(base_dataset, conf);
+//    index_->set_index_model(model);
+//    index_->Add(base_dataset, conf);
+//    EXPECT_EQ(index_->Count(), nb);
+//    EXPECT_EQ(index_->Dimension(), dim);
+//
+//    auto binaryset = index_->Serialize();
+//
+//
+//
+//    auto cpu_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
+//    cpu_idx->Load(binaryset);
+//    auto pair = cpu_idx->CopyCpuToGpuWithQuantizer(DEVICEID, conf);
+//    auto quantizer = pair.second;
+//
+//    auto quantizer_conf = std::make_shared<knowhere::QuantizerCfg>();
+//    quantizer_conf->mode = 2;  // only copy data
+//    quantizer_conf->gpu_id = DEVICEID;
+//
+//    auto CopyAllToGpu = [&](int64_t search_count, bool do_search = false) {
+//        for (int i = 0; i < search_count; ++i) {
+//            auto gpu_idx = cpu_idx->CopyCpuToGpu(DEVICEID, conf);
+//            if (do_search) {
+//                auto result = gpu_idx->Search(query_dataset, conf);
+//                AssertAnns(result, nq, conf->k);
+//            }
+//        }
+//    };
+//
+//    auto hybrid_qt_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
+//    hybrid_qt_idx->Load(binaryset);
+//    auto SetQuantizerDoSearch = [&](int64_t search_count) {
+//        for (int i = 0; i < search_count; ++i) {
+//            hybrid_qt_idx->SetQuantizer(quantizer);
+//            auto result = hybrid_qt_idx->Search(query_dataset, conf);
+//            AssertAnns(result, nq, conf->k);
+//            //            PrintResult(result, nq, k);
+//            hybrid_qt_idx->UnsetQuantizer();
+//        }
+//    };
+//
+//    auto hybrid_data_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
+//    hybrid_data_idx->Load(binaryset);
+//    auto LoadDataDoSearch = [&](int64_t search_count, bool do_search = false) {
+//        for (int i = 0; i < search_count; ++i) {
+//            auto hybrid_idx = hybrid_data_idx->LoadData(quantizer, quantizer_conf);
+//            if (do_search) {
+//                auto result = hybrid_idx->Search(query_dataset, conf);
+////                AssertAnns(result, nq, conf->k);
+//            }
+//        }
+//    };
+//
+//    knowhere::TimeRecorder tc("");
+//    CopyAllToGpu(2000/2, false);
+//    tc.RecordSection("CopyAllToGpu witout search");
+//    CopyAllToGpu(400/2, true);
+//    tc.RecordSection("CopyAllToGpu with search");
+//    SetQuantizerDoSearch(6);
+//    tc.RecordSection("SetQuantizer with search");
+//    LoadDataDoSearch(2000/2, false);
+//    tc.RecordSection("LoadData without search");
+//    LoadDataDoSearch(400/2, true);
+//    tc.RecordSection("LoadData with search");
+//
+//    {
+//        std::thread t1(CopyAllToGpu, 2000, false);
+//        std::thread t2(CopyAllToGpu, 400, true);
+//        t1.join();
+//        t2.join();
+//    }
+//
+//    {
+//        std::thread t1(SetQuantizerDoSearch, 12);
+//        std::thread t2(CopyAllToGpu, 400, true);
+//        t1.join();
+//        t2.join();
+//    }
+//
+//    {
+//        std::thread t1(SetQuantizerDoSearch, 12);
+//        std::thread t2(LoadDataDoSearch, 400, true);
+//        t1.join();
+//        t2.join();
+//    }
+//
+//    {
+//        std::thread t1(LoadDataDoSearch, 2000, false);
+//        std::thread t2(LoadDataDoSearch, 400, true);
+//        t1.join();
+//        t2.join();
+//    }
+//
+//}
+
 #endif

core/src/index/unittest/test_ivf.cpp

@@ -20,19 +20,12 @@
 #include <iostream>
 #include <thread>
 
-#include <faiss/AutoTune.h>
-#include <faiss/gpu/GpuAutoTune.h>
 #include <faiss/gpu/GpuIndexIVFFlat.h>
 
 #include "knowhere/common/Exception.h"
 #include "knowhere/common/Timer.h"
 #include "knowhere/index/vector_index/IndexGPUIVF.h"
-#include "knowhere/index/vector_index/IndexGPUIVFPQ.h"
-#include "knowhere/index/vector_index/IndexGPUIVFSQ.h"
 #include "knowhere/index/vector_index/IndexIVF.h"
-#include "knowhere/index/vector_index/IndexIVFPQ.h"
-#include "knowhere/index/vector_index/IndexIVFSQ.h"
-#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
 #include "knowhere/index/vector_index/helpers/Cloner.h"
 
 #include "unittest/Helper.h"
@@ -51,6 +44,9 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
         ParameterType parameter_type;
         std::tie(index_type, parameter_type) = GetParam();
         // Init_with_default();
+        //        nb = 1000000;
+        //        nq = 1000;
+        //        k = 1000;
         Generate(DIM, NB, NQ);
         index_ = IndexFactory(index_type);
         conf = ParamGenerator::GetInstance().Gen(parameter_type);
@@ -61,16 +57,6 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
         knowhere::FaissGpuResourceMgr::GetInstance().Free();
     }
 
-    knowhere::VectorIndexPtr
-    ChooseTodo() {
-        std::vector<std::string> gpu_idx{"GPUIVFSQ"};
-        auto finder = std::find(gpu_idx.cbegin(), gpu_idx.cend(), index_type);
-        if (finder != gpu_idx.cend()) {
-            return knowhere::cloner::CopyCpuToGpu(index_, DEVICEID, knowhere::Config());
-        }
-        return index_;
-    }
-
  protected:
     std::string index_type;
     knowhere::Config conf;
@@ -100,8 +86,7 @@ TEST_P(IVFTest, ivf_basic) {
     EXPECT_EQ(index_->Count(), nb);
     EXPECT_EQ(index_->Dimension(), dim);
 
-    auto new_idx = ChooseTodo();
-    auto result = new_idx->Search(query_dataset, conf);
+    auto result = index_->Search(query_dataset, conf);
     AssertAnns(result, nq, conf->k);
     // PrintResult(result, nq, k);
 }
@@ -134,8 +119,7 @@ TEST_P(IVFTest, ivf_serialize) {
 
         index_->set_index_model(model);
         index_->Add(base_dataset, conf);
-        auto new_idx = ChooseTodo();
-        auto result = new_idx->Search(query_dataset, conf);
+        auto result = index_->Search(query_dataset, conf);
         AssertAnns(result, nq, conf->k);
     }
 
@@ -159,8 +143,7 @@ TEST_P(IVFTest, ivf_serialize) {
         index_->Load(binaryset);
         EXPECT_EQ(index_->Count(), nb);
         EXPECT_EQ(index_->Dimension(), dim);
-        auto new_idx = ChooseTodo();
-        auto result = new_idx->Search(query_dataset, conf);
+        auto result = index_->Search(query_dataset, conf);
         AssertAnns(result, nq, conf->k);
     }
 }
@@ -176,8 +159,7 @@ TEST_P(IVFTest, clone_test) {
     index_->Add(base_dataset, conf);
     EXPECT_EQ(index_->Count(), nb);
     EXPECT_EQ(index_->Dimension(), dim);
-    auto new_idx = ChooseTodo();
-    auto result = new_idx->Search(query_dataset, conf);
+    auto result = index_->Search(query_dataset, conf);
     AssertAnns(result, nq, conf->k);
     // PrintResult(result, nq, k);
 
@@ -210,12 +192,6 @@ TEST_P(IVFTest, clone_test) {
     //        }
     //    }
 
-    {
-        if (index_type == "IVFSQHybrid") {
-            return;
-        }
-    }
-
     {
         // copy from gpu to cpu
         std::vector<std::string> support_idx_vec{"GPUIVF", "GPUIVFSQ", "IVFSQHybrid"};
@@ -277,8 +253,7 @@ TEST_P(IVFTest, gpu_seal_test) {
     index_->Add(base_dataset, conf);
     EXPECT_EQ(index_->Count(), nb);
     EXPECT_EQ(index_->Dimension(), dim);
-    auto new_idx = ChooseTodo();
-    auto result = new_idx->Search(query_dataset, conf);
+    auto result = index_->Search(query_dataset, conf);
     AssertAnns(result, nq, conf->k);
 
     auto cpu_idx = knowhere::cloner::CopyGpuToCpu(index_, knowhere::Config());

core/src/scheduler/SchedInst.h

@@ -94,6 +94,7 @@ class OptimizerInst {
             std::lock_guard<std::mutex> lock(mutex_);
             if (instance == nullptr) {
                 std::vector<PassPtr> pass_list;
+                pass_list.push_back(std::make_shared<LargeSQ8HPass>());
                 pass_list.push_back(std::make_shared<HybridPass>());
                 instance = std::make_shared<Optimizer>(pass_list);
             }

core/src/scheduler/optimizer/LargeSQ8HPass.cpp

@@ -26,48 +26,48 @@
 namespace milvus {
 namespace scheduler {
 
-// bool
-// LargeSQ8HPass::Run(const TaskPtr& task) {
-//    if (task->Type() != TaskType::SearchTask) {
-//        return false;
-//    }
-//
-//    auto search_task = std::static_pointer_cast<XSearchTask>(task);
-//    if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
-//        return false;
-//    }
-//
-//    auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
-//
-//    // TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
-//    if (search_job->nq() < 100) {
-//        return false;
-//    }
-//
-//    std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
-//    std::vector<int64_t> all_free_mem;
-//    for (auto& gpu : gpus) {
-//        auto cache = cache::GpuCacheMgr::GetInstance(gpu);
-//        auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
-//        all_free_mem.push_back(free_mem);
-//    }
-//
-//    auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
-//    auto best_index = std::distance(all_free_mem.begin(), max_e);
-//    auto best_device_id = gpus[best_index];
-//
-//    ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
-//    if (not res_ptr) {
-//        SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
-//        // TODO: throw critical error and exit
-//        return false;
-//    }
-//
-//    auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
-//    task->label() = label;
-//
-//    return true;
-// }
+bool
+LargeSQ8HPass::Run(const TaskPtr& task) {
+    if (task->Type() != TaskType::SearchTask) {
+        return false;
+    }
+
+    auto search_task = std::static_pointer_cast<XSearchTask>(task);
+    if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
+        return false;
+    }
+
+    auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
+
+    // TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
+    if (search_job->nq() < 100) {
+        return false;
+    }
+
+    std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
+    std::vector<int64_t> all_free_mem;
+    for (auto& gpu : gpus) {
+        auto cache = cache::GpuCacheMgr::GetInstance(gpu);
+        auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
+        all_free_mem.push_back(free_mem);
+    }
+
+    auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
+    auto best_index = std::distance(all_free_mem.begin(), max_e);
+    auto best_device_id = gpus[best_index];
+
+    ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
+    if (not res_ptr) {
+        SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
+        // TODO: throw critical error and exit
+        return false;
+    }
+
+    auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
+    task->label() = label;
+
+    return true;
+}
 
 }  // namespace scheduler
 }  // namespace milvus

core/src/scheduler/optimizer/LargeSQ8HPass.h

@@ -37,8 +37,8 @@ class LargeSQ8HPass : public Pass {
     LargeSQ8HPass() = default;
 
  public:
-    //    bool
-    //    Run(const TaskPtr& task) override;
+    bool
+    Run(const TaskPtr& task) override;
 };
 
 using LargeSQ8HPassPtr = std::shared_ptr<LargeSQ8HPass>;

core/src/server/Config.cpp

@@ -363,7 +363,9 @@ Config::PrintAll() {
 Status
 Config::CheckServerConfigAddress(const std::string& value) {
     if (!ValidationUtil::ValidateIpAddress(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid server config address: " + value);
+        std::string msg =
+            "Invalid server IP address: " + value + ". Possible reason: server_config.address is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -371,11 +373,14 @@ Config::CheckServerConfigAddress(const std::string& value) {
 Status
 Config::CheckServerConfigPort(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid server config port: " + value);
+        std::string msg = "Invalid server port: " + value + ". Possible reason: server_config.port is not a number.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         int32_t port = std::stoi(value);
         if (!(port > 1024 && port < 65535)) {
-            return Status(SERVER_INVALID_ARGUMENT, "Server config port out of range (1024, 65535): " + value);
+            std::string msg = "Invalid server port: " + value +
+                              ". Possible reason: server_config.port is not in range [1025, 65534].";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -385,7 +390,8 @@ Status
 Config::CheckServerConfigDeployMode(const std::string& value) {
     if (value != "single" && value != "cluster_readonly" && value != "cluster_writable") {
         return Status(SERVER_INVALID_ARGUMENT,
-                      "Invalid server config mode [single, cluster_readonly, cluster_writable]: " + value);
+                      "server_config.deploy_mode is not one of "
+                      "single, cluster_readonly, and cluster_writable.");
     }
     return Status::OK();
 }
@@ -393,15 +399,15 @@ Config::CheckServerConfigDeployMode(const std::string& value) {
 Status
 Config::CheckServerConfigTimeZone(const std::string& value) {
     if (value.length() <= 3) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
+        return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
     } else {
         if (value.substr(0, 3) != "UTC") {
-            return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
+            return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
         } else {
             try {
                 stoi(value.substr(3));
             } catch (...) {
-                return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
+                return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
             }
         }
     }
@@ -411,7 +417,7 @@ Config::CheckServerConfigTimeZone(const std::string& value) {
 Status
 Config::CheckDBConfigPrimaryPath(const std::string& value) {
     if (value.empty()) {
-        return Status(SERVER_INVALID_ARGUMENT, "DB config primary_path empty");
+        return Status(SERVER_INVALID_ARGUMENT, "db_config.db_path is empty.");
     }
     return Status::OK();
 }
@@ -424,7 +430,10 @@ Config::CheckDBConfigSecondaryPath(const std::string& value) {
 Status
 Config::CheckDBConfigBackendUrl(const std::string& value) {
     if (!ValidationUtil::ValidateDbURI(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config backend_url: " + value);
+        std::string msg =
+            "Invalid backend url: " + value + ". Possible reason: db_config.db_backend_url is invalid. " +
+            "The correct format should be like sqlite://:@:/ or mysql://root:123456@127.0.0.1:3306/milvus.";
+        return Status(SERVER_INVALID_ARGUMENT, "invalid db_backend_url: " + value);
     }
     return Status::OK();
 }
@@ -432,7 +441,9 @@ Config::CheckDBConfigBackendUrl(const std::string& value) {
 Status
 Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_disk_threshold: " + value);
+        std::string msg = "Invalid archive disk threshold: " + value +
+                          ". Possible reason: db_config.archive_disk_threshold is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -440,7 +451,9 @@ Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
 Status
 Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_days_threshold: " + value);
+        std::string msg = "Invalid archive days threshold: " + value +
+                          ". Possible reason: db_config.archive_disk_threshold is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -448,13 +461,23 @@ Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
 Status
 Config::CheckDBConfigInsertBufferSize(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config insert_buffer_size: " + value);
+        std::string msg = "Invalid insert buffer size: " + value +
+                          ". Possible reason: db_config.insert_buffer_size is not a positive integer.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         int64_t buffer_size = std::stoi(value) * GB;
+        if (buffer_size <= 0) {
+            std::string msg = "Invalid insert buffer size: " + value +
+                              ". Possible reason: db_config.insert_buffer_size is not a positive integer.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
+        }
+
         uint64_t total_mem = 0, free_mem = 0;
         CommonUtil::GetSystemMemInfo(total_mem, free_mem);
         if (buffer_size >= total_mem) {
-            return Status(SERVER_INVALID_ARGUMENT, "DB config insert_buffer_size exceed system memory: " + value);
+            std::string msg = "Invalid insert buffer size: " + value +
+                              ". Possible reason: db_config.insert_buffer_size exceeds system memory.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -463,7 +486,9 @@ Config::CheckDBConfigInsertBufferSize(const std::string& value) {
 Status
 Config::CheckMetricConfigEnableMonitor(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config auto_bootup: " + value);
+        std::string msg =
+            "Invalid metric config: " + value + ". Possible reason: metric_config.enable_monitor is not a boolean.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -471,7 +496,9 @@ Config::CheckMetricConfigEnableMonitor(const std::string& value) {
 Status
 Config::CheckMetricConfigCollector(const std::string& value) {
     if (value != "prometheus") {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config collector: " + value);
+        std::string msg =
+            "Invalid metric collector: " + value + ". Possible reason: metric_config.collector is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -479,6 +506,8 @@ Config::CheckMetricConfigCollector(const std::string& value) {
 Status
 Config::CheckMetricConfigPrometheusPort(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
+        std::string msg = "Invalid metric port: " + value +
+                          ". Possible reason: metric_config.prometheus_config.port is not in range [1025, 65534].";
         return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config prometheus_port: " + value);
     }
     return Status::OK();
@@ -487,15 +516,25 @@ Config::CheckMetricConfigPrometheusPort(const std::string& value) {
 Status
 Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_capacity: " + value);
+        std::string msg = "Invalid cpu cache capacity: " + value +
+                          ". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
-        uint64_t cpu_cache_capacity = std::stoi(value) * GB;
+        int64_t cpu_cache_capacity = std::stoi(value) * GB;
+        if (cpu_cache_capacity <= 0) {
+            std::string msg = "Invalid cpu cache capacity: " + value +
+                              ". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
+        }
+
         uint64_t total_mem = 0, free_mem = 0;
         CommonUtil::GetSystemMemInfo(total_mem, free_mem);
-        if (cpu_cache_capacity >= total_mem) {
-            return Status(SERVER_INVALID_ARGUMENT, "Cache config cpu_cache_capacity exceed system memory: " + value);
-        } else if (cpu_cache_capacity > static_cast<double>(total_mem * 0.9)) {
-            std::cerr << "Warning: cpu_cache_capacity value is too big" << std::endl;
+        if (static_cast<uint64_t>(cpu_cache_capacity) >= total_mem) {
+            std::string msg = "Invalid cpu cache capacity: " + value +
+                              ". Possible reason: cache_config.cpu_cache_capacity exceeds system memory.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
+        } else if (static_cast<double>(cpu_cache_capacity) > static_cast<double>(total_mem * 0.9)) {
+            std::cerr << "WARNING: cpu cache capacity value is too big" << std::endl;
         }
 
         int32_t buffer_value;
@@ -506,7 +545,10 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
 
         int64_t insert_buffer_size = buffer_value * GB;
         if (insert_buffer_size + cpu_cache_capacity >= total_mem) {
-            return Status(SERVER_INVALID_ARGUMENT, "Sum of cpu_cache_capacity and buffer_size exceed system memory");
+            std::string msg = "Invalid cpu cache capacity: " + value +
+                              ". Possible reason: sum of cache_config.cpu_cache_capacity and "
+                              "db_config.insert_buffer_size exceeds system memory.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -515,11 +557,15 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
 Status
 Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value);
+        std::string msg = "Invalid cpu cache threshold: " + value +
+                          ". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         float cpu_cache_threshold = std::stof(value);
         if (cpu_cache_threshold <= 0.0 || cpu_cache_threshold >= 1.0) {
-            return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value);
+            std::string msg = "Invalid cpu cache threshold: " + value +
+                              ". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -528,7 +574,9 @@ Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
 Status
 Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_capacity: " + value);
+        std::string msg = "Invalid gpu cache capacity: " + value +
+                          ". Possible reason: cache_config.gpu_cache_capacity is not a positive integer.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         uint64_t gpu_cache_capacity = std::stoi(value) * GB;
         int gpu_index;
@@ -539,13 +587,14 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
 
         size_t gpu_memory;
         if (!ValidationUtil::GetGpuMemory(gpu_index, gpu_memory).ok()) {
-            return Status(SERVER_UNEXPECTED_ERROR,
-                          "Fail to get GPU memory for GPU device: " + std::to_string(gpu_index));
+            std::string msg = "Fail to get GPU memory for GPU device: " + std::to_string(gpu_index);
+            return Status(SERVER_UNEXPECTED_ERROR, msg);
         } else if (gpu_cache_capacity >= gpu_memory) {
-            return Status(SERVER_INVALID_ARGUMENT,
-                          "Cache config gpu_cache_capacity exceed GPU memory: " + std::to_string(gpu_memory));
+            std::string msg = "Invalid gpu cache capacity: " + value +
+                              ". Possible reason: cache_config.gpu_cache_capacity exceeds GPU memory.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         } else if (gpu_cache_capacity > (double)gpu_memory * 0.9) {
-            std::cerr << "Warning: gpu_cache_capacity value is too big" << std::endl;
+            std::cerr << "Warning: gpu cache capacity value is too big" << std::endl;
         }
     }
     return Status::OK();
@@ -554,11 +603,15 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
 Status
 Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value);
+        std::string msg = "Invalid gpu cache threshold: " + value +
+                          ". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         float gpu_cache_threshold = std::stof(value);
         if (gpu_cache_threshold <= 0.0 || gpu_cache_threshold >= 1.0) {
-            return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value);
+            std::string msg = "Invalid gpu cache threshold: " + value +
+                              ". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -567,7 +620,9 @@ Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
 Status
 Config::CheckCacheConfigCacheInsertData(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cache_insert_data: " + value);
+        std::string msg = "Invalid cache insert option: " + value +
+                          ". Possible reason: cache_config.cache_insert_data is not a boolean.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -575,7 +630,9 @@ Config::CheckCacheConfigCacheInsertData(const std::string& value) {
 Status
 Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config use_blas_threshold: " + value);
+        std::string msg = "Invalid blas threshold: " + value +
+                          ". Possible reason: engine_config.use_blas_threshold is not a positive integer.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -583,14 +640,18 @@ Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
 Status
 Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value);
+        std::string msg = "Invalid omp thread number: " + value +
+                          ". Possible reason: engine_config.omp_thread_num is not a positive integer.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
     int32_t omp_thread = std::stoi(value);
     uint32_t sys_thread_cnt = 8;
     CommonUtil::GetSystemAvailableThreads(sys_thread_cnt);
     if (omp_thread > static_cast<int32_t>(sys_thread_cnt)) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value);
+        std::string msg = "Invalid omp thread number: " + value +
+                          ". Possible reason: engine_config.omp_thread_num exceeds system cpu cores.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -598,7 +659,8 @@ Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
 Status
 Config::CheckResourceConfigMode(const std::string& value) {
     if (value != "simple") {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config mode: " + value);
+        std::string msg = "Invalid resource mode: " + value + ". Possible reason: resource_config.mode is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -608,12 +670,16 @@ CheckGpuDevice(const std::string& value) {
     const std::regex pat("gpu(\\d+)");
     std::cmatch m;
     if (!std::regex_match(value.c_str(), m, pat)) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value);
+        std::string msg = "Invalid gpu device: " + value +
+                          ". Possible reason: resource_config.search_resources does not match your hardware.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
     int32_t gpu_index = std::stoi(value.substr(3));
     if (!ValidationUtil::ValidateGpuIndex(gpu_index).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value);
+        std::string msg = "Invalid gpu device: " + value +
+                          ". Possible reason: resource_config.search_resources does not match your hardware.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }
@@ -621,12 +687,17 @@ CheckGpuDevice(const std::string& value) {
 Status
 Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value) {
     if (value.empty()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Empty resource config search_resources");
+        std::string msg =
+            "Invalid search resource. "
+            "Possible reason: resource_config.search_resources is empty.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
     for (auto& gpu_device : value) {
         if (!CheckGpuDevice(gpu_device).ok()) {
-            return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config search_resources: " + gpu_device);
+            std::string msg = "Invalid search resource: " + gpu_device +
+                              ". Possible reason: resource_config.search_resources does not match your hardware.";
+            return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
     return Status::OK();
@@ -635,7 +706,9 @@ Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value
 Status
 Config::CheckResourceConfigIndexBuildDevice(const std::string& value) {
     if (!CheckGpuDevice(value).ok()) {
-        return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config index_build_device: " + value);
+        std::string msg = "Invalid index build device: " + value +
+                          ". Possible reason: resource_config.index_build_device does not match your hardware.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
     return Status::OK();
 }

core/src/server/DBWrapper.cpp

@@ -15,7 +15,7 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include <faiss/utils.h>
+#include <faiss/utils/distances.h>
 #include <omp.h>
 #include <cmath>
 #include <string>
@@ -40,12 +40,14 @@ DBWrapper::StartService() {
     engine::DBOptions opt;
     s = config.GetDBConfigBackendUrl(opt.meta_.backend_uri_);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
     std::string path;
     s = config.GetDBConfigPrimaryPath(path);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -54,6 +56,7 @@ DBWrapper::StartService() {
     std::string db_slave_path;
     s = config.GetDBConfigSecondaryPath(db_slave_path);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -62,12 +65,14 @@ DBWrapper::StartService() {
     // cache config
     s = config.GetCacheConfigCacheInsertData(opt.insert_cache_immediately_);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
     std::string mode;
     s = config.GetServerConfigDeployMode(mode);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -78,8 +83,8 @@ DBWrapper::StartService() {
     } else if (mode == "cluster_writable") {
         opt.mode_ = engine::DBOptions::MODE::CLUSTER_WRITABLE;
     } else {
-        std::cerr << "ERROR: mode specified in server_config must be ['single', 'cluster_readonly', 'cluster_writable']"
-                  << std::endl;
+        std::cerr << "Error: server_config.deploy_mode in server_config.yaml is not one of "
+                  << "single, cluster_readonly, and cluster_writable." << std::endl;
         kill(0, SIGUSR1);
     }
 
@@ -87,6 +92,7 @@ DBWrapper::StartService() {
     int32_t omp_thread;
     s = config.GetEngineConfigOmpThreadNum(omp_thread);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -105,6 +111,7 @@ DBWrapper::StartService() {
     int32_t use_blas_threshold;
     s = config.GetEngineConfigUseBlasThreshold(use_blas_threshold);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -115,6 +122,7 @@ DBWrapper::StartService() {
     int32_t disk, days;
     s = config.GetDBConfigArchiveDiskThreshold(disk);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -124,6 +132,7 @@ DBWrapper::StartService() {
 
     s = config.GetDBConfigArchiveDaysThreshold(days);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 
@@ -133,16 +142,20 @@ DBWrapper::StartService() {
     opt.meta_.archive_conf_.SetCriterias(criterial);
 
     // create db root folder
-    Status status = CommonUtil::CreateDirectory(opt.meta_.path_);
-    if (!status.ok()) {
-        std::cerr << "ERROR! Failed to create database root path: " << opt.meta_.path_ << std::endl;
+    s = CommonUtil::CreateDirectory(opt.meta_.path_);
+    if (!s.ok()) {
+        std::cerr << "Error: Failed to create database primary path: " << path
+                  << ". Possible reason: db_config.primary_path is wrong in server_config.yaml or not available."
+                  << std::endl;
         kill(0, SIGUSR1);
     }
 
     for (auto& path : opt.meta_.slave_paths_) {
-        status = CommonUtil::CreateDirectory(path);
-        if (!status.ok()) {
-            std::cerr << "ERROR! Failed to create database slave path: " << path << std::endl;
+        s = CommonUtil::CreateDirectory(path);
+        if (!s.ok()) {
+            std::cerr << "Error: Failed to create database secondary path: " << path
+                      << ". Possible reason: db_config.secondary_path is wrong in server_config.yaml or not available."
+                      << std::endl;
             kill(0, SIGUSR1);
         }
     }
@@ -151,7 +164,8 @@ DBWrapper::StartService() {
     try {
         db_ = engine::DBFactory::Build(opt);
     } catch (std::exception& ex) {
-        std::cerr << "ERROR! Failed to open database: " << ex.what() << std::endl;
+        std::cerr << "Error: failed to open database: " << ex.what()
+                  << ". Possible reason: the meta system does not work." << std::endl;
         kill(0, SIGUSR1);
     }
 
@@ -161,6 +175,7 @@ DBWrapper::StartService() {
     std::string preload_tables;
     s = config.GetDBConfigPreloadTable(preload_tables);
     if (!s.ok()) {
+        std::cerr << s.ToString() << std::endl;
         return s;
     }
 

core/src/wrapper/VecIndex.h

@@ -71,6 +71,7 @@ class VecIndex : public cache::DataObj {
     virtual VecIndexPtr
     CopyToCpu(const Config& cfg = Config()) = 0;
 
+    // TODO(linxj): Deprecated
     virtual VecIndexPtr
     Clone() = 0;
 

core/unittest/db/test_engine.cpp

@@ -108,7 +108,6 @@ TEST_F(EngineTest, ENGINE_IMPL_TEST) {
     ASSERT_EQ(engine_ptr->Dimension(), dimension);
     ASSERT_EQ(engine_ptr->Count(), ids.size());
 
-    status = engine_ptr->CopyToGpu(0, true);
     status = engine_ptr->CopyToGpu(0, false);
     //ASSERT_TRUE(status.ok());
 

core/unittest/db/utils.cpp

@@ -65,10 +65,10 @@ static const char
                    "  cache_insert_data: false          # whether load inserted data into cache\n"
                    "\n"
                    "engine_config:\n"
-                   "  blas_threshold: 20\n"
+                   "  use_blas_threshold: 20\n"
                    "\n"
                    "resource_config:\n"
-                   "  resource_pool:\n"
+                   "  search_resources:\n"
                    "    - gpu0\n"
                    "  index_build_device: gpu0          # GPU used for building index";
 

core/unittest/wrapper/test_wrapper.cpp

@@ -74,7 +74,7 @@ INSTANTIATE_TEST_CASE_P(WrapperParam, KnowhereWrapperTest,
                                             10,
                                             10),
                             std::make_tuple(milvus::engine::IndexType::FAISS_IVFSQ8_CPU, "Default", DIM, NB, 10, 10),
-//                            std::make_tuple(milvus::engine::IndexType::FAISS_IVFSQ8_GPU, "Default", DIM, NB, 10, 10),
+                            std::make_tuple(milvus::engine::IndexType::FAISS_IVFSQ8_GPU, "Default", DIM, NB, 10, 10),
                             std::make_tuple(milvus::engine::IndexType::FAISS_IVFSQ8_MIX, "Default", DIM, NB, 10, 10),
 //                            std::make_tuple(IndexType::NSG_MIX, "Default", 128, 250000, 10, 10),
 //                            std::make_tuple(IndexType::SPTAG_KDT_RNT_CPU, "Default", 128, 250000, 10, 10),

core/unittest/wrapper/utils.cpp

@@ -58,7 +58,7 @@ static const char
                    "  blas_threshold: 20\n"
                    "\n"
                    "resource_config:\n"
-                   "  resource_pool:\n"
+                   "  search_resources:\n"
                    "    - gpu0\n"
                    "  index_build_device: gpu0          # GPU used for building index";