Fix http bug & add binary vectors support (#1073)

* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* change gpu resources invoke way to cmd()

* set advanced config name add DEFAULT

* change config setting to cmd

* modify ..

* optimize code

* assign TableDto' count default value 0 (fix #995)

* check if table exists when show partitions (fix #1028)

* check table exists when drop partition (fix #1029)

* check if partition name is legal (fix #1022)

* modify status code when partition tag is illegal

* update changlog

* add info to /system url

* add binary index and add bin uri & handler method(not completed)

* optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067)

* fix test partition bug

* fix test bug when check insert records

* add binary vectors test

* add default for offset and page_size

* fix uinttest bug

* [skip ci] remove comments

* optimize web code for PR comments

* add new folder named utils
Co-authored-by: Njielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: NJackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: NCai Yudong <yudong.cai@zilliz.com>

CHANGELOG.md

@@ -11,6 +11,13 @@ Please mark all change in change log and use the issue from GitHub
 - \#805 - IVFTest.gpu_seal_test unittest failed
 - \#831 - Judge branch error in CommonUtil.cpp
 - \#977 - Server crash when create tables concurrently
+- \#995 - table count set to 0 if no tables found
+- \#1010 - improve error message when offset or page_size is equal 0
+- \#1022 - check if partition name is legal
+- \#1028 - check if table exists when show partitions
+- \#1029 - check if table exists when try to delete partition
+- \#1066 - optimize http insert and search speed
+- \#1067 - Add binary vectors support in http server
 
 ## Feature
 - \#216 - Add CLI to get server info

core/src/CMakeLists.txt

@@ -96,12 +96,14 @@ aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/handler web_handler_fi
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/component web_conponent_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/controller web_controller_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/dto web_dto_files)
+aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/utils web_utils_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl web_impl_files)
 set(web_server_files
         ${web_handler_files}
         ${web_conponent_files}
         ${web_controller_files}
         ${web_dto_files}
+        ${web_utils_files}
         ${web_impl_files}
         )
 

core/src/server/delivery/request/DropPartitionRequest.cpp

@@ -47,8 +47,19 @@ DropPartitionRequest::OnExecute() {
     std::string table_name = table_name_;
     std::string partition_name = partition_name_;
     std::string partition_tag = tag_;
+
+    bool exists;
+    auto status = DBWrapper::DB()->HasTable(table_name, exists);
+    if (!status.ok()) {
+        return status;
+    }
+
+    if (!exists) {
+        return Status(SERVER_TABLE_NOT_EXIST, "Table " + table_name_ + " not exists");
+    }
+
     if (!partition_name.empty()) {
-        auto status = ValidationUtil::ValidateTableName(partition_name);
+        status = ValidationUtil::ValidateTableName(partition_name);
         if (!status.ok()) {
             return status;
         }
@@ -68,7 +79,7 @@ DropPartitionRequest::OnExecute() {
 
         return DBWrapper::DB()->DropPartition(partition_name);
     } else {
-        auto status = ValidationUtil::ValidateTableName(table_name);
+        status = ValidationUtil::ValidateTableName(table_name);
         if (!status.ok()) {
             return status;
         }

core/src/server/delivery/request/ShowPartitionsRequest.cpp

@@ -48,6 +48,16 @@ ShowPartitionsRequest::OnExecute() {
         return status;
     }
 
+    bool exists = false;
+    status = DBWrapper::DB()->HasTable(table_name_, exists);
+    if (!status.ok()) {
+        return status;
+    }
+
+    if (!exists) {
+        return Status(SERVER_TABLE_NOT_EXIST, "Table " + table_name_ + " not exists");
+    }
+
     std::vector<engine::meta::TableSchema> schema_array;
     status = DBWrapper::DB()->ShowPartitions(table_name_, schema_array);
     if (!status.ok()) {

core/src/server/web_impl/Constants.h

-
 // Licensed to the Apache Software Foundation (ASF) under one
 // or more contributor license agreements.  See the NOTICE file
 // distributed with this work for additional information
@@ -46,6 +45,9 @@ static const char* NAME_ENGINE_TYPE_IVFPQ = "IVFPQ";
 
 static const char* NAME_METRIC_TYPE_L2 = "L2";
 static const char* NAME_METRIC_TYPE_IP = "IP";
+static const char* NAME_METRIC_TYPE_HAMMING = "HAMMING";
+static const char* NAME_METRIC_TYPE_JACCARD = "JACCARD";
+static const char* NAME_METRIC_TYPE_TANIMOTO = "TANIMOTO";
 
 ////////////////////////////////////////////////////
 

core/src/server/web_impl/Types.h

@@ -21,9 +21,9 @@
 #include <unordered_map>
 
 #include <oatpp/core/data/mapping/type/Object.hpp>
+#include <oatpp/web/protocol/http/Http.hpp>
 
 #include "db/engine/ExecutionEngine.h"
-
 #include "server/web_impl/Constants.h"
 
 namespace milvus {
@@ -31,6 +31,8 @@ namespace server {
 namespace web {
 
 using OString = oatpp::data::mapping::type::String;
+using OInt8 = oatpp::data::mapping::type::Int8;
+using OInt16 = oatpp::data::mapping::type::Int16;
 using OInt64 = oatpp::data::mapping::type::Int64;
 using OFloat32 = oatpp::data::mapping::type::Float32;
 template <class T>
@@ -65,10 +67,11 @@ enum StatusCode : int {
     ILLEGAL_METRIC_TYPE = 23,
     OUT_OF_MEMORY = 24,
 
-    // HTTP status code
+    // HTTP error code
     PATH_PARAM_LOSS = 31,
     QUERY_PARAM_LOSS = 32,
     BODY_FIELD_LOSS = 33,
+    ILLEGAL_QUERY_PARAM = 36,
 };
 
 static const std::unordered_map<engine::EngineType, std::string> IndexMap = {
@@ -92,11 +95,17 @@ static const std::unordered_map<std::string, engine::EngineType> IndexNameMap =
 static const std::unordered_map<engine::MetricType, std::string> MetricMap = {
     {engine::MetricType::L2, NAME_METRIC_TYPE_L2},
     {engine::MetricType::IP, NAME_METRIC_TYPE_IP},
+    {engine::MetricType::HAMMING, NAME_METRIC_TYPE_HAMMING},
+    {engine::MetricType::JACCARD, NAME_METRIC_TYPE_JACCARD},
+    {engine::MetricType::TANIMOTO, NAME_METRIC_TYPE_TANIMOTO},
 };
 
 static const std::unordered_map<std::string, engine::MetricType> MetricNameMap = {
     {NAME_METRIC_TYPE_L2, engine::MetricType::L2},
     {NAME_METRIC_TYPE_IP, engine::MetricType::IP},
+    {NAME_METRIC_TYPE_HAMMING, engine::MetricType::HAMMING},
+    {NAME_METRIC_TYPE_JACCARD, engine::MetricType::JACCARD},
+    {NAME_METRIC_TYPE_TANIMOTO, engine::MetricType::TANIMOTO},
 };
 
 }  // namespace web

core/src/server/web_impl/WebServer.h

@@ -22,8 +22,6 @@
 #include <string>
 #include <thread>
 
-#include <oatpp/network/server/Server.hpp>
-
 #include "server/web_impl/component/AppComponent.hpp"
 
 #include "utils/Status.h"

core/src/server/web_impl/dto/TableDto.hpp

@@ -59,14 +59,7 @@ class TableListFieldsDto : public OObject {
     DTO_INIT(TableListFieldsDto, Object)
 
     DTO_FIELD(List<TableFieldsDto::ObjectWrapper>::ObjectWrapper, tables);
-    DTO_FIELD(Int64, count);
-};
-
-class TablesResponseDto : public OObject {
-    DTO_INIT(TablesResponseDto, Object)
-
-    DTO_FIELD(TableListFieldsDto::ObjectWrapper, tables_fields);
-    DTO_FIELD(Int64, page_num);
+    DTO_FIELD(Int64, count) = 0;
 };
 
 #include OATPP_CODEGEN_END(DTO)

core/src/server/web_impl/dto/VectorDto.hpp

@@ -46,14 +46,15 @@ class SearchRequestDto : public OObject {
     DTO_FIELD(List<String>::ObjectWrapper, tags);
     DTO_FIELD(List<String>::ObjectWrapper, file_ids);
     DTO_FIELD(List<List<Float32>::ObjectWrapper>::ObjectWrapper, records);
+    DTO_FIELD(List<List<Int64>::ObjectWrapper>::ObjectWrapper, records_bin);
 };
 
-
 class InsertRequestDto : public oatpp::data::mapping::type::Object {
     DTO_INIT(InsertRequestDto, Object)
 
     DTO_FIELD(String, tag) = VALUE_PARTITION_TAG_DEFAULT;
     DTO_FIELD(List<List<Float32>::ObjectWrapper>::ObjectWrapper, records);
+    DTO_FIELD(List<List<Int64>::ObjectWrapper>::ObjectWrapper, records_bin);
     DTO_FIELD(List<Int64>::ObjectWrapper, ids);
 };
 
@@ -66,17 +67,10 @@ class VectorIdsDto : public oatpp::data::mapping::type::Object {
 class ResultDto : public oatpp::data::mapping::type::Object {
     DTO_INIT(ResultDto, Object)
 
-//    DTO_FIELD(Int64, num);
     DTO_FIELD(String, id);
     DTO_FIELD(String, dit, "distance");
 };
 
-class RowResultsDto : public OObject {
-    DTO_INIT(RowResultsDto, Object)
-
-//    DTO_FIELD(List<ResultDto::ObjectWrapper>::ObjectWrapper, );
-};
-
 class TopkResultsDto : public OObject {
     DTO_INIT(TopkResultsDto, Object);
 

core/src/server/web_impl/handler/WebRequestHandler.h

@@ -23,10 +23,9 @@
 #include <utility>
 
 #include <opentracing/mocktracer/tracer.h>
-#include <oatpp/web/server/api/ApiController.hpp>
-
 #include <oatpp/core/data/mapping/type/Object.hpp>
 #include <oatpp/core/macro/codegen.hpp>
+#include <oatpp/web/server/api/ApiController.hpp>
 
 #include "server/web_impl/Types.h"
 #include "server/web_impl/dto/CmdDto.hpp"
@@ -37,6 +36,7 @@
 #include "server/web_impl/dto/TableDto.hpp"
 #include "server/web_impl/dto/VectorDto.hpp"
 
+#include "db/Types.h"
 #include "server/context/Context.h"
 #include "server/delivery/RequestHandler.h"
 #include "utils/Status.h"
@@ -82,15 +82,18 @@ class WebRequestHandler {
         return context_ptr;
     }
 
+ protected:
+    Status
+    GetTableInfo(const std::string& table_name, TableFieldsDto::ObjectWrapper& table_fields);
+
+    Status
+    CommandLine(const std::string& cmd, std::string& reply);
+
  public:
     WebRequestHandler() {
         context_ptr_ = GenContextPtr("Web Handler");
     }
 
-    Status
-    GetTaleInfo(const std::shared_ptr<Context>& context, const std::string& table_name,
-                std::map<std::string, std::string>& table_info);
-
     StatusDto::ObjectWrapper
     GetDevices(DevicesDto::ObjectWrapper& devices);
 
@@ -115,7 +118,7 @@ class WebRequestHandler {
     GetTable(const OString& table_name, const OQueryParams& query_params, TableFieldsDto::ObjectWrapper& schema_dto);
 
     StatusDto::ObjectWrapper
-    ShowTables(const OInt64& offset, const OInt64& page_size, TableListFieldsDto::ObjectWrapper& table_list_dto);
+    ShowTables(const OString& offset, const OString& page_size, TableListFieldsDto::ObjectWrapper& table_list_dto);
 
     StatusDto::ObjectWrapper
     DropTable(const OString& table_name);
@@ -133,7 +136,7 @@ class WebRequestHandler {
     CreatePartition(const OString& table_name, const PartitionRequestDto::ObjectWrapper& param);
 
     StatusDto::ObjectWrapper
-    ShowPartitions(const OInt64& offset, const OInt64& page_size, const OString& table_name,
+    ShowPartitions(const OString& offset, const OString& page_size, const OString& table_name,
                    PartitionListDto::ObjectWrapper& partition_list_dto);
 
     StatusDto::ObjectWrapper
@@ -150,6 +153,7 @@ class WebRequestHandler {
     StatusDto::ObjectWrapper
     Cmd(const OString& cmd, CommandDto::ObjectWrapper& cmd_dto);
 
+ public:
     WebRequestHandler&
     RegisterRequestHandler(const RequestHandler& handler) {
         request_handler_ = handler;

core/src/server/web_impl/utils/Util.cpp

+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "server/web_impl/utils/Util.h"
+
+namespace milvus {
+namespace server {
+namespace web {
+
+Status
+CopyRowRecords(const OList<OList<OFloat32>::ObjectWrapper>::ObjectWrapper& records, std::vector<float>& vectors) {
+    size_t tal_size = 0;
+    records->forEach([&tal_size](const OList<OFloat32>::ObjectWrapper& row_item) { tal_size += row_item->count(); });
+
+    vectors.resize(tal_size);
+    size_t index_offset = 0;
+    records->forEach([&vectors, &index_offset](const OList<OFloat32>::ObjectWrapper& row_item) {
+        row_item->forEach(
+            [&vectors, &index_offset](const OFloat32& item) { vectors[index_offset++] = item->getValue(); });
+    });
+
+    return Status::OK();
+}
+
+Status
+CopyBinRowRecords(const OList<OList<OInt64>::ObjectWrapper>::ObjectWrapper& records, std::vector<uint8_t>& vectors) {
+    size_t tal_size = 0;
+    records->forEach([&tal_size](const OList<OInt64>::ObjectWrapper& item) { tal_size += item->count(); });
+
+    vectors.resize(tal_size);
+    size_t index_offset = 0;
+    bool oor = false;
+    records->forEach([&vectors, &index_offset, &oor](const OList<OInt64>::ObjectWrapper& row_item) {
+        row_item->forEach([&vectors, &index_offset, &oor](const OInt64& item) {
+            if (!oor) {
+                int64_t value = item->getValue();
+                if (0 > value || value > 255) {
+                    oor = true;
+                } else {
+                    vectors[index_offset++] = static_cast<uint8_t>(value);
+                }
+            }
+        });
+    });
+
+    return Status::OK();
+}
+
+}  // namespace web
+}  // namespace server
+}  // namespace milvus

core/src/server/web_impl/utils/Util.h

+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <vector>
+
+#include "db/Types.h"
+#include "server/web_impl/Types.h"
+#include "utils/Status.h"
+
+namespace milvus {
+namespace server {
+namespace web {
+
+Status
+CopyRowRecords(const OList<OList<OFloat32>::ObjectWrapper>::ObjectWrapper& records, std::vector<float>& vectors);
+
+Status
+CopyBinRowRecords(const OList<OList<OInt64>::ObjectWrapper>::ObjectWrapper& records, std::vector<uint8_t>& vectors);
+
+}  // namespace web
+}  // namespace server
+}  // namespace milvus

core/src/utils/ValidationUtil.cpp

@@ -194,7 +194,33 @@ ValidationUtil::ValidatePartitionName(const std::string& partition_name) {
         return Status(SERVER_INVALID_TABLE_NAME, msg);
     }
 
-    return ValidateTableName(partition_name);
+    std::string invalid_msg = "Invalid partition name: " + partition_name + ". ";
+    // Table name size shouldn't exceed 16384.
+    if (partition_name.size() > TABLE_NAME_SIZE_LIMIT) {
+        std::string msg = invalid_msg + "The length of a partition name must be less than 255 characters.";
+        SERVER_LOG_ERROR << msg;
+        return Status(SERVER_INVALID_TABLE_NAME, msg);
+    }
+
+    // Table name first character should be underscore or character.
+    char first_char = partition_name[0];
+    if (first_char != '_' && std::isalpha(first_char) == 0) {
+        std::string msg = invalid_msg + "The first character of a partition name must be an underscore or letter.";
+        SERVER_LOG_ERROR << msg;
+        return Status(SERVER_INVALID_TABLE_NAME, msg);
+    }
+
+    int64_t table_name_size = partition_name.size();
+    for (int64_t i = 1; i < table_name_size; ++i) {
+        char name_char = partition_name[i];
+        if (name_char != '_' && std::isalnum(name_char) == 0) {
+            std::string msg = invalid_msg + "Partition name can only contain numbers, letters, and underscores.";
+            SERVER_LOG_ERROR << msg;
+            return Status(SERVER_INVALID_TABLE_NAME, msg);
+        }
+    }
+
+    return Status::OK();
 }
 
 Status
@@ -207,7 +233,7 @@ ValidationUtil::ValidatePartitionTags(const std::vector<std::string>& partition_
         if (valid_tag.empty()) {
             std::string msg = "Invalid partition tag: " + valid_tag + ". " + "Partition tag should not be empty.";
             SERVER_LOG_ERROR << msg;
-            return Status(SERVER_INVALID_NPROBE, msg);
+            return Status(SERVER_INVALID_TABLE_NAME, msg);
         }
     }
 

core/unittest/CMakeLists.txt

@@ -78,12 +78,14 @@ aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/handler web_handler_fi
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/component web_conponent_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/controller web_controller_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/dto web_dto_files)
+aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl/utils web_utils_files)
 aux_source_directory(${MILVUS_ENGINE_SRC}/server/web_impl web_impl_files)
 set(web_server_files
         ${web_handler_files}
         ${web_conponent_files}
         ${web_controller_files}
         ${web_dto_files}
+        ${web_utils_files}
         ${web_impl_files}
         )
 

tests/milvus_python_test/test_partition.py

@@ -258,8 +258,8 @@ class TestShowBase:
         '''
         partition_name = gen_unique_str()
         status, res = connect.show_partitions(partition_name)
-        assert status.OK()
-        assert len(res) == 0
+        assert not status.OK()
+        # assert len(res) == 0
 
     def test_show_multi_partitions(self, connect, table):
         '''
@@ -428,4 +428,4 @@ class TestNameInvalid(object):
         partition_name = gen_unique_str()
         status = connect.create_partition(table, partition_name, tag)
         status, res = connect.show_partitions(table_name)
-        assert not status.OK()
\ No newline at end of file
+        assert not status.OK()