提交 95b231a5 编写于 作者: Y yudong.cai

Merge remote-tracking branch 'main/0.5.1' into 0.5.1


Former-commit-id: aecebe10c5104ebc1d77d7e21c0d79844309f371
...@@ -5,6 +5,7 @@ Please mark all change in change log and use the ticket from JIRA. ...@@ -5,6 +5,7 @@ Please mark all change in change log and use the ticket from JIRA.
# Milvus 0.5.1 (TODO) # Milvus 0.5.1 (TODO)
## Bug ## Bug
- \#90 - The server start error messages could be improved to enhance user experience
- \#104 - test_scheduler core dump - \#104 - test_scheduler core dump
## Improvement ## Improvement
......
...@@ -20,7 +20,7 @@ ...@@ -20,7 +20,7 @@
## What is Milvus ## What is Milvus
Milvus is an open source similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources. Milvus is an open source similarity search engine for massive-scale feature vectors. Built with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
Milvus provides stable Python, Java and C++ APIs. Milvus provides stable Python, Java and C++ APIs.
...@@ -28,7 +28,7 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea ...@@ -28,7 +28,7 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
- Heterogeneous computing - Heterogeneous computing
Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency. Milvus is built with heterogeneous computing architecture for the best performance and cost efficiency.
- Multiple indexes - Multiple indexes
...@@ -64,14 +64,14 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea ...@@ -64,14 +64,14 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
## Get started ## Get started
### Hardware Requirements ### Hardware requirements
| Component | Recommended configuration | | Component | Recommended configuration |
| --------- | ----------------------------------- | | --------- | ----------------------------------- |
| CPU | Intel CPU Haswell or higher | | CPU | Intel CPU Haswell or higher |
| GPU | NVIDIA Pascal series or higher | | GPU | NVIDIA Pascal series or higher |
| Memory | 8 GB or more (depends on data size) | | RAM | 8 GB or more (depends on data size) |
| Storage | SATA 3.0 SSD or higher | | Hard drive| SATA 3.0 SSD or higher |
### Install using docker ### Install using docker
...@@ -168,6 +168,10 @@ Make sure Java 8 or higher is already installed. ...@@ -168,6 +168,10 @@ Make sure Java 8 or higher is already installed.
Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code. Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code.
## Milvus roadmap
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
## Contribution guidelines ## Contribution guidelines
Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code. Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
...@@ -178,9 +182,11 @@ We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to ...@@ -178,9 +182,11 @@ We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to
To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk). To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk).
## Milvus Roadmap ## Thanks
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features. We greatly appreciate the help of the following people.
- [akihoni](https://github.com/akihoni) found a broken link and a small typo in the README file.
## Resources ## Resources
...@@ -196,7 +202,6 @@ Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upc ...@@ -196,7 +202,6 @@ Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upc
[Milvus roadmap](https://milvus.io/docs/en/roadmap/) [Milvus roadmap](https://milvus.io/docs/en/roadmap/)
## License ## License
[Apache License 2.0](LICENSE) [Apache License 2.0](LICENSE)
......
...@@ -2,9 +2,9 @@ ...@@ -2,9 +2,9 @@
server_config: server_config:
address: 0.0.0.0 # milvus server ip address (IPv4) address: 0.0.0.0 # milvus server ip address (IPv4)
port: 19530 # port range: 1025 ~ 65534 port: 19530 # milvus server port, must in range [1025, 65534]
deploy_mode: single # deployment type: single, cluster_readonly, cluster_writable deploy_mode: single # deployment type: single, cluster_readonly, cluster_writable
time_zone: UTC+8 time_zone: UTC+8 # time zone, must be in format: UTC+X
db_config: db_config:
primary_path: @MILVUS_DB_PATH@ # path used to store data and meta primary_path: @MILVUS_DB_PATH@ # path used to store data and meta
...@@ -14,30 +14,30 @@ db_config: ...@@ -14,30 +14,30 @@ db_config:
# Keep 'dialect://:@:/', and replace other texts with real values # Keep 'dialect://:@:/', and replace other texts with real values
# Replace 'dialect' with 'mysql' or 'sqlite' # Replace 'dialect' with 'mysql' or 'sqlite'
insert_buffer_size: 4 # GB, maximum insert buffer size allowed insert_buffer_size: 4 # GB, maximum insert buffer size allowed, must be a positive integer
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory # sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload preload_table: # preload data at startup, '*' means load all tables, empty value means no preload
# you can specify preload tables like this: table1,table2,table3 # you can specify preload tables like this: table1,table2,table3
metric_config: metric_config:
enable_monitor: false # enable monitoring or not enable_monitor: false # enable monitoring or not, must be a boolean
collector: prometheus # prometheus collector: prometheus # prometheus
prometheus_config: prometheus_config:
port: 8080 # port prometheus uses to fetch metrics port: 8080 # port prometheus uses to fetch metrics, must in range [1025, 65534]
cache_config: cache_config:
cpu_cache_capacity: 16 # GB, CPU memory used for cache cpu_cache_capacity: 16 # GB, CPU memory used for cache, must be a positive integer
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
gpu_cache_capacity: 4 # GB, GPU memory used for cache gpu_cache_capacity: 4 # GB, GPU memory used for cache, must be a positive integer
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
cache_insert_data: false # whether to load inserted data into cache cache_insert_data: false # whether to load inserted data into cache, must be a boolean
engine_config: engine_config:
use_blas_threshold: 20 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times use_blas_threshold: 20 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times # if nq >= use_blas_threshold, use OpenBlas, slower with stable response times
resource_config: resource_config:
search_resources: # define the GPUs used for search computation, valid value: gpux search_resources: # define the GPUs used for search computation, must be in format: gpux
- gpu0 - gpu0
index_build_device: gpu0 # GPU used for building index index_build_device: gpu0 # GPU used for building index, must be in format: gpux
\ No newline at end of file \ No newline at end of file
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
// specific language governing permissions and limitations // specific language governing permissions and limitations
// under the License. // under the License.
#include "Job.h" #include "scheduler/job/Job.h"
namespace milvus { namespace milvus {
namespace scheduler { namespace scheduler {
......
...@@ -363,7 +363,9 @@ Config::PrintAll() { ...@@ -363,7 +363,9 @@ Config::PrintAll() {
Status Status
Config::CheckServerConfigAddress(const std::string& value) { Config::CheckServerConfigAddress(const std::string& value) {
if (!ValidationUtil::ValidateIpAddress(value).ok()) { if (!ValidationUtil::ValidateIpAddress(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config address: " + value); std::string msg =
"Invalid server IP address: " + value + ". Possible reason: server_config.address is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -371,11 +373,14 @@ Config::CheckServerConfigAddress(const std::string& value) { ...@@ -371,11 +373,14 @@ Config::CheckServerConfigAddress(const std::string& value) {
Status Status
Config::CheckServerConfigPort(const std::string& value) { Config::CheckServerConfigPort(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config port: " + value); std::string msg = "Invalid server port: " + value + ". Possible reason: server_config.port is not a number.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
int32_t port = std::stoi(value); int32_t port = std::stoi(value);
if (!(port > 1024 && port < 65535)) { if (!(port > 1024 && port < 65535)) {
return Status(SERVER_INVALID_ARGUMENT, "Server config port out of range (1024, 65535): " + value); std::string msg = "Invalid server port: " + value +
". Possible reason: server_config.port is not in range [1025, 65534].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -385,7 +390,8 @@ Status ...@@ -385,7 +390,8 @@ Status
Config::CheckServerConfigDeployMode(const std::string& value) { Config::CheckServerConfigDeployMode(const std::string& value) {
if (value != "single" && value != "cluster_readonly" && value != "cluster_writable") { if (value != "single" && value != "cluster_readonly" && value != "cluster_writable") {
return Status(SERVER_INVALID_ARGUMENT, return Status(SERVER_INVALID_ARGUMENT,
"Invalid server config mode [single, cluster_readonly, cluster_writable]: " + value); "server_config.deploy_mode is not one of "
"single, cluster_readonly, and cluster_writable.");
} }
return Status::OK(); return Status::OK();
} }
...@@ -393,15 +399,15 @@ Config::CheckServerConfigDeployMode(const std::string& value) { ...@@ -393,15 +399,15 @@ Config::CheckServerConfigDeployMode(const std::string& value) {
Status Status
Config::CheckServerConfigTimeZone(const std::string& value) { Config::CheckServerConfigTimeZone(const std::string& value) {
if (value.length() <= 3) { if (value.length() <= 3) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value); return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
} else { } else {
if (value.substr(0, 3) != "UTC") { if (value.substr(0, 3) != "UTC") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value); return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
} else { } else {
try { try {
stoi(value.substr(3)); stoi(value.substr(3));
} catch (...) { } catch (...) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value); return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
} }
} }
} }
...@@ -411,7 +417,7 @@ Config::CheckServerConfigTimeZone(const std::string& value) { ...@@ -411,7 +417,7 @@ Config::CheckServerConfigTimeZone(const std::string& value) {
Status Status
Config::CheckDBConfigPrimaryPath(const std::string& value) { Config::CheckDBConfigPrimaryPath(const std::string& value) {
if (value.empty()) { if (value.empty()) {
return Status(SERVER_INVALID_ARGUMENT, "DB config primary_path empty"); return Status(SERVER_INVALID_ARGUMENT, "db_config.db_path is empty.");
} }
return Status::OK(); return Status::OK();
} }
...@@ -424,7 +430,10 @@ Config::CheckDBConfigSecondaryPath(const std::string& value) { ...@@ -424,7 +430,10 @@ Config::CheckDBConfigSecondaryPath(const std::string& value) {
Status Status
Config::CheckDBConfigBackendUrl(const std::string& value) { Config::CheckDBConfigBackendUrl(const std::string& value) {
if (!ValidationUtil::ValidateDbURI(value).ok()) { if (!ValidationUtil::ValidateDbURI(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config backend_url: " + value); std::string msg =
"Invalid backend url: " + value + ". Possible reason: db_config.db_backend_url is invalid. " +
"The correct format should be like sqlite://:@:/ or mysql://root:123456@127.0.0.1:3306/milvus.";
return Status(SERVER_INVALID_ARGUMENT, "invalid db_backend_url: " + value);
} }
return Status::OK(); return Status::OK();
} }
...@@ -432,7 +441,9 @@ Config::CheckDBConfigBackendUrl(const std::string& value) { ...@@ -432,7 +441,9 @@ Config::CheckDBConfigBackendUrl(const std::string& value) {
Status Status
Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) { Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_disk_threshold: " + value); std::string msg = "Invalid archive disk threshold: " + value +
". Possible reason: db_config.archive_disk_threshold is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -440,7 +451,9 @@ Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) { ...@@ -440,7 +451,9 @@ Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
Status Status
Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) { Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_days_threshold: " + value); std::string msg = "Invalid archive days threshold: " + value +
". Possible reason: db_config.archive_disk_threshold is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -448,13 +461,23 @@ Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) { ...@@ -448,13 +461,23 @@ Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
Status Status
Config::CheckDBConfigInsertBufferSize(const std::string& value) { Config::CheckDBConfigInsertBufferSize(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config insert_buffer_size: " + value); std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
int64_t buffer_size = std::stoi(value) * GB; int64_t buffer_size = std::stoi(value) * GB;
if (buffer_size <= 0) {
std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
uint64_t total_mem = 0, free_mem = 0; uint64_t total_mem = 0, free_mem = 0;
CommonUtil::GetSystemMemInfo(total_mem, free_mem); CommonUtil::GetSystemMemInfo(total_mem, free_mem);
if (buffer_size >= total_mem) { if (buffer_size >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "DB config insert_buffer_size exceed system memory: " + value); std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size exceeds system memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -463,7 +486,9 @@ Config::CheckDBConfigInsertBufferSize(const std::string& value) { ...@@ -463,7 +486,9 @@ Config::CheckDBConfigInsertBufferSize(const std::string& value) {
Status Status
Config::CheckMetricConfigEnableMonitor(const std::string& value) { Config::CheckMetricConfigEnableMonitor(const std::string& value) {
if (!ValidationUtil::ValidateStringIsBool(value).ok()) { if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config auto_bootup: " + value); std::string msg =
"Invalid metric config: " + value + ". Possible reason: metric_config.enable_monitor is not a boolean.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -471,7 +496,9 @@ Config::CheckMetricConfigEnableMonitor(const std::string& value) { ...@@ -471,7 +496,9 @@ Config::CheckMetricConfigEnableMonitor(const std::string& value) {
Status Status
Config::CheckMetricConfigCollector(const std::string& value) { Config::CheckMetricConfigCollector(const std::string& value) {
if (value != "prometheus") { if (value != "prometheus") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config collector: " + value); std::string msg =
"Invalid metric collector: " + value + ". Possible reason: metric_config.collector is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -479,6 +506,8 @@ Config::CheckMetricConfigCollector(const std::string& value) { ...@@ -479,6 +506,8 @@ Config::CheckMetricConfigCollector(const std::string& value) {
Status Status
Config::CheckMetricConfigPrometheusPort(const std::string& value) { Config::CheckMetricConfigPrometheusPort(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
std::string msg = "Invalid metric port: " + value +
". Possible reason: metric_config.prometheus_config.port is not in range [1025, 65534].";
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config prometheus_port: " + value); return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config prometheus_port: " + value);
} }
return Status::OK(); return Status::OK();
...@@ -487,15 +516,25 @@ Config::CheckMetricConfigPrometheusPort(const std::string& value) { ...@@ -487,15 +516,25 @@ Config::CheckMetricConfigPrometheusPort(const std::string& value) {
Status Status
Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) { Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_capacity: " + value); std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
uint64_t cpu_cache_capacity = std::stoi(value) * GB; int64_t cpu_cache_capacity = std::stoi(value) * GB;
if (cpu_cache_capacity <= 0) {
std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
uint64_t total_mem = 0, free_mem = 0; uint64_t total_mem = 0, free_mem = 0;
CommonUtil::GetSystemMemInfo(total_mem, free_mem); CommonUtil::GetSystemMemInfo(total_mem, free_mem);
if (cpu_cache_capacity >= total_mem) { if (static_cast<uint64_t>(cpu_cache_capacity) >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "Cache config cpu_cache_capacity exceed system memory: " + value); std::string msg = "Invalid cpu cache capacity: " + value +
} else if (cpu_cache_capacity > static_cast<double>(total_mem * 0.9)) { ". Possible reason: cache_config.cpu_cache_capacity exceeds system memory.";
std::cerr << "Warning: cpu_cache_capacity value is too big" << std::endl; return Status(SERVER_INVALID_ARGUMENT, msg);
} else if (static_cast<double>(cpu_cache_capacity) > static_cast<double>(total_mem * 0.9)) {
std::cerr << "WARNING: cpu cache capacity value is too big" << std::endl;
} }
int32_t buffer_value; int32_t buffer_value;
...@@ -506,7 +545,10 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) { ...@@ -506,7 +545,10 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
int64_t insert_buffer_size = buffer_value * GB; int64_t insert_buffer_size = buffer_value * GB;
if (insert_buffer_size + cpu_cache_capacity >= total_mem) { if (insert_buffer_size + cpu_cache_capacity >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "Sum of cpu_cache_capacity and buffer_size exceed system memory"); std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: sum of cache_config.cpu_cache_capacity and "
"db_config.insert_buffer_size exceeds system memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -515,11 +557,15 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) { ...@@ -515,11 +557,15 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
Status Status
Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) { Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsFloat(value).ok()) { if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value); std::string msg = "Invalid cpu cache threshold: " + value +
". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
float cpu_cache_threshold = std::stof(value); float cpu_cache_threshold = std::stof(value);
if (cpu_cache_threshold <= 0.0 || cpu_cache_threshold >= 1.0) { if (cpu_cache_threshold <= 0.0 || cpu_cache_threshold >= 1.0) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value); std::string msg = "Invalid cpu cache threshold: " + value +
". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -528,7 +574,9 @@ Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) { ...@@ -528,7 +574,9 @@ Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
Status Status
Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) { Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_capacity: " + value); std::string msg = "Invalid gpu cache capacity: " + value +
". Possible reason: cache_config.gpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
uint64_t gpu_cache_capacity = std::stoi(value) * GB; uint64_t gpu_cache_capacity = std::stoi(value) * GB;
int gpu_index; int gpu_index;
...@@ -539,13 +587,14 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) { ...@@ -539,13 +587,14 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
size_t gpu_memory; size_t gpu_memory;
if (!ValidationUtil::GetGpuMemory(gpu_index, gpu_memory).ok()) { if (!ValidationUtil::GetGpuMemory(gpu_index, gpu_memory).ok()) {
return Status(SERVER_UNEXPECTED_ERROR, std::string msg = "Fail to get GPU memory for GPU device: " + std::to_string(gpu_index);
"Fail to get GPU memory for GPU device: " + std::to_string(gpu_index)); return Status(SERVER_UNEXPECTED_ERROR, msg);
} else if (gpu_cache_capacity >= gpu_memory) { } else if (gpu_cache_capacity >= gpu_memory) {
return Status(SERVER_INVALID_ARGUMENT, std::string msg = "Invalid gpu cache capacity: " + value +
"Cache config gpu_cache_capacity exceed GPU memory: " + std::to_string(gpu_memory)); ". Possible reason: cache_config.gpu_cache_capacity exceeds GPU memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else if (gpu_cache_capacity > (double)gpu_memory * 0.9) { } else if (gpu_cache_capacity > (double)gpu_memory * 0.9) {
std::cerr << "Warning: gpu_cache_capacity value is too big" << std::endl; std::cerr << "Warning: gpu cache capacity value is too big" << std::endl;
} }
} }
return Status::OK(); return Status::OK();
...@@ -554,11 +603,15 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) { ...@@ -554,11 +603,15 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
Status Status
Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) { Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsFloat(value).ok()) { if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value); std::string msg = "Invalid gpu cache threshold: " + value +
". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else { } else {
float gpu_cache_threshold = std::stof(value); float gpu_cache_threshold = std::stof(value);
if (gpu_cache_threshold <= 0.0 || gpu_cache_threshold >= 1.0) { if (gpu_cache_threshold <= 0.0 || gpu_cache_threshold >= 1.0) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value); std::string msg = "Invalid gpu cache threshold: " + value +
". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -567,7 +620,9 @@ Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) { ...@@ -567,7 +620,9 @@ Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
Status Status
Config::CheckCacheConfigCacheInsertData(const std::string& value) { Config::CheckCacheConfigCacheInsertData(const std::string& value) {
if (!ValidationUtil::ValidateStringIsBool(value).ok()) { if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cache_insert_data: " + value); std::string msg = "Invalid cache insert option: " + value +
". Possible reason: cache_config.cache_insert_data is not a boolean.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -575,7 +630,9 @@ Config::CheckCacheConfigCacheInsertData(const std::string& value) { ...@@ -575,7 +630,9 @@ Config::CheckCacheConfigCacheInsertData(const std::string& value) {
Status Status
Config::CheckEngineConfigUseBlasThreshold(const std::string& value) { Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config use_blas_threshold: " + value); std::string msg = "Invalid blas threshold: " + value +
". Possible reason: engine_config.use_blas_threshold is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -583,14 +640,18 @@ Config::CheckEngineConfigUseBlasThreshold(const std::string& value) { ...@@ -583,14 +640,18 @@ Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
Status Status
Config::CheckEngineConfigOmpThreadNum(const std::string& value) { Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) { if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value); std::string msg = "Invalid omp thread number: " + value +
". Possible reason: engine_config.omp_thread_num is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
int32_t omp_thread = std::stoi(value); int32_t omp_thread = std::stoi(value);
uint32_t sys_thread_cnt = 8; uint32_t sys_thread_cnt = 8;
CommonUtil::GetSystemAvailableThreads(sys_thread_cnt); CommonUtil::GetSystemAvailableThreads(sys_thread_cnt);
if (omp_thread > static_cast<int32_t>(sys_thread_cnt)) { if (omp_thread > static_cast<int32_t>(sys_thread_cnt)) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value); std::string msg = "Invalid omp thread number: " + value +
". Possible reason: engine_config.omp_thread_num exceeds system cpu cores.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -598,7 +659,8 @@ Config::CheckEngineConfigOmpThreadNum(const std::string& value) { ...@@ -598,7 +659,8 @@ Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
Status Status
Config::CheckResourceConfigMode(const std::string& value) { Config::CheckResourceConfigMode(const std::string& value) {
if (value != "simple") { if (value != "simple") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config mode: " + value); std::string msg = "Invalid resource mode: " + value + ". Possible reason: resource_config.mode is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -608,12 +670,16 @@ CheckGpuDevice(const std::string& value) { ...@@ -608,12 +670,16 @@ CheckGpuDevice(const std::string& value) {
const std::regex pat("gpu(\\d+)"); const std::regex pat("gpu(\\d+)");
std::cmatch m; std::cmatch m;
if (!std::regex_match(value.c_str(), m, pat)) { if (!std::regex_match(value.c_str(), m, pat)) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value); std::string msg = "Invalid gpu device: " + value +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
int32_t gpu_index = std::stoi(value.substr(3)); int32_t gpu_index = std::stoi(value.substr(3));
if (!ValidationUtil::ValidateGpuIndex(gpu_index).ok()) { if (!ValidationUtil::ValidateGpuIndex(gpu_index).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value); std::string msg = "Invalid gpu device: " + value +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
...@@ -621,12 +687,17 @@ CheckGpuDevice(const std::string& value) { ...@@ -621,12 +687,17 @@ CheckGpuDevice(const std::string& value) {
Status Status
Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value) { Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value) {
if (value.empty()) { if (value.empty()) {
return Status(SERVER_INVALID_ARGUMENT, "Empty resource config search_resources"); std::string msg =
"Invalid search resource. "
"Possible reason: resource_config.search_resources is empty.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
for (auto& gpu_device : value) { for (auto& gpu_device : value) {
if (!CheckGpuDevice(gpu_device).ok()) { if (!CheckGpuDevice(gpu_device).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config search_resources: " + gpu_device); std::string msg = "Invalid search resource: " + gpu_device +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
} }
return Status::OK(); return Status::OK();
...@@ -635,7 +706,9 @@ Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value ...@@ -635,7 +706,9 @@ Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value
Status Status
Config::CheckResourceConfigIndexBuildDevice(const std::string& value) { Config::CheckResourceConfigIndexBuildDevice(const std::string& value) {
if (!CheckGpuDevice(value).ok()) { if (!CheckGpuDevice(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config index_build_device: " + value); std::string msg = "Invalid index build device: " + value +
". Possible reason: resource_config.index_build_device does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} }
return Status::OK(); return Status::OK();
} }
......
...@@ -40,12 +40,14 @@ DBWrapper::StartService() { ...@@ -40,12 +40,14 @@ DBWrapper::StartService() {
engine::DBOptions opt; engine::DBOptions opt;
s = config.GetDBConfigBackendUrl(opt.meta_.backend_uri_); s = config.GetDBConfigBackendUrl(opt.meta_.backend_uri_);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
std::string path; std::string path;
s = config.GetDBConfigPrimaryPath(path); s = config.GetDBConfigPrimaryPath(path);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -54,6 +56,7 @@ DBWrapper::StartService() { ...@@ -54,6 +56,7 @@ DBWrapper::StartService() {
std::string db_slave_path; std::string db_slave_path;
s = config.GetDBConfigSecondaryPath(db_slave_path); s = config.GetDBConfigSecondaryPath(db_slave_path);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -62,12 +65,14 @@ DBWrapper::StartService() { ...@@ -62,12 +65,14 @@ DBWrapper::StartService() {
// cache config // cache config
s = config.GetCacheConfigCacheInsertData(opt.insert_cache_immediately_); s = config.GetCacheConfigCacheInsertData(opt.insert_cache_immediately_);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
std::string mode; std::string mode;
s = config.GetServerConfigDeployMode(mode); s = config.GetServerConfigDeployMode(mode);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -78,8 +83,8 @@ DBWrapper::StartService() { ...@@ -78,8 +83,8 @@ DBWrapper::StartService() {
} else if (mode == "cluster_writable") { } else if (mode == "cluster_writable") {
opt.mode_ = engine::DBOptions::MODE::CLUSTER_WRITABLE; opt.mode_ = engine::DBOptions::MODE::CLUSTER_WRITABLE;
} else { } else {
std::cerr << "ERROR: mode specified in server_config must be ['single', 'cluster_readonly', 'cluster_writable']" std::cerr << "Error: server_config.deploy_mode in server_config.yaml is not one of "
<< std::endl; << "single, cluster_readonly, and cluster_writable." << std::endl;
kill(0, SIGUSR1); kill(0, SIGUSR1);
} }
...@@ -87,6 +92,7 @@ DBWrapper::StartService() { ...@@ -87,6 +92,7 @@ DBWrapper::StartService() {
int32_t omp_thread; int32_t omp_thread;
s = config.GetEngineConfigOmpThreadNum(omp_thread); s = config.GetEngineConfigOmpThreadNum(omp_thread);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -105,6 +111,7 @@ DBWrapper::StartService() { ...@@ -105,6 +111,7 @@ DBWrapper::StartService() {
int32_t use_blas_threshold; int32_t use_blas_threshold;
s = config.GetEngineConfigUseBlasThreshold(use_blas_threshold); s = config.GetEngineConfigUseBlasThreshold(use_blas_threshold);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -115,6 +122,7 @@ DBWrapper::StartService() { ...@@ -115,6 +122,7 @@ DBWrapper::StartService() {
int32_t disk, days; int32_t disk, days;
s = config.GetDBConfigArchiveDiskThreshold(disk); s = config.GetDBConfigArchiveDiskThreshold(disk);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -124,6 +132,7 @@ DBWrapper::StartService() { ...@@ -124,6 +132,7 @@ DBWrapper::StartService() {
s = config.GetDBConfigArchiveDaysThreshold(days); s = config.GetDBConfigArchiveDaysThreshold(days);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
...@@ -133,16 +142,20 @@ DBWrapper::StartService() { ...@@ -133,16 +142,20 @@ DBWrapper::StartService() {
opt.meta_.archive_conf_.SetCriterias(criterial); opt.meta_.archive_conf_.SetCriterias(criterial);
// create db root folder // create db root folder
Status status = CommonUtil::CreateDirectory(opt.meta_.path_); s = CommonUtil::CreateDirectory(opt.meta_.path_);
if (!status.ok()) { if (!s.ok()) {
std::cerr << "ERROR! Failed to create database root path: " << opt.meta_.path_ << std::endl; std::cerr << "Error: Failed to create database primary path: " << path
<< ". Possible reason: db_config.primary_path is wrong in server_config.yaml or not available."
<< std::endl;
kill(0, SIGUSR1); kill(0, SIGUSR1);
} }
for (auto& path : opt.meta_.slave_paths_) { for (auto& path : opt.meta_.slave_paths_) {
status = CommonUtil::CreateDirectory(path); s = CommonUtil::CreateDirectory(path);
if (!status.ok()) { if (!s.ok()) {
std::cerr << "ERROR! Failed to create database slave path: " << path << std::endl; std::cerr << "Error: Failed to create database secondary path: " << path
<< ". Possible reason: db_config.secondary_path is wrong in server_config.yaml or not available."
<< std::endl;
kill(0, SIGUSR1); kill(0, SIGUSR1);
} }
} }
...@@ -151,7 +164,8 @@ DBWrapper::StartService() { ...@@ -151,7 +164,8 @@ DBWrapper::StartService() {
try { try {
db_ = engine::DBFactory::Build(opt); db_ = engine::DBFactory::Build(opt);
} catch (std::exception& ex) { } catch (std::exception& ex) {
std::cerr << "ERROR! Failed to open database: " << ex.what() << std::endl; std::cerr << "Error: failed to open database: " << ex.what()
<< ". Possible reason: the meta system does not work." << std::endl;
kill(0, SIGUSR1); kill(0, SIGUSR1);
} }
...@@ -161,6 +175,7 @@ DBWrapper::StartService() { ...@@ -161,6 +175,7 @@ DBWrapper::StartService() {
std::string preload_tables; std::string preload_tables;
s = config.GetDBConfigPreloadTable(preload_tables); s = config.GetDBConfigPreloadTable(preload_tables);
if (!s.ok()) { if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s; return s;
} }
......
...@@ -68,7 +68,7 @@ static const char ...@@ -68,7 +68,7 @@ static const char
" blas_threshold: 20\n" " blas_threshold: 20\n"
"\n" "\n"
"resource_config:\n" "resource_config:\n"
" resource_pool:\n" " search_resources:\n"
" - gpu0\n" " - gpu0\n"
" index_build_device: gpu0 # GPU used for building index"; " index_build_device: gpu0 # GPU used for building index";
......
...@@ -58,7 +58,7 @@ static const char ...@@ -58,7 +58,7 @@ static const char
" blas_threshold: 20\n" " blas_threshold: 20\n"
"\n" "\n"
"resource_config:\n" "resource_config:\n"
" resource_pool:\n" " search_resources:\n"
" - gpu0\n" " - gpu0\n"
" index_build_device: gpu0 # GPU used for building index"; " index_build_device: gpu0 # GPU used for building index";
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册