Skip to content

milvus
milvus

qq_35024513 / milvus
与 Fork 源项目一致

Fork自 milvus / milvus

通知 14
Star 0
Fork 0
milvus
milvus

体验新版 GitCode，发现更多精彩内容 >>

提交 d1c4ab91 编写于 作者: B BossZou 提交者： JinHai-CN
浏览文件
操作

Mishards 010 upgrade & Add config unit in demo config (#2550) 

* Update mishards config
Signed-off-by: NYhz <yinghao.zou@zilliz.com>

* Update mishards config
Signed-off-by: NYhz <yinghao.zou@zilliz.com>

* Upgrade mishards for milvus v0.10.0 (fix #2548)
Signed-off-by: NYhz <yinghao.zou@zilliz.com>

* Launch server fail using demo config (#2549)
Signed-off-by: NYhz <yinghao.zou@zilliz.com>

* [skip ci] Remove surplus config yaml
Signed-off-by: NYhz <yinghao.zou@zilliz.com>
上级 18539c50
隐藏空白更改
内联 并排
Showing with 408 addition and 561 deletion
CHANGELOG.md
浏览文件 @ d1c4ab91
......@@ -21,6 +21,7 @@ Please mark all change in change log and use the issue from GitHub
- \#2368 Make read node detect delete behavior
- \#2373 Build index for small segment waste time on waiting background index thread finish
- \#2394 Drop collection timeout if too many partitions created on collection
- \#2549 Launch server fail using demo config
## Feature
- \#2363 Update branch version
......@@ -34,6 +35,7 @@ Please mark all change in change log and use the issue from GitHub
- \#2489 Add exception throw on mysql meta error
- \#2495 Add creating lock file failure reason.
- \#2516 Improve unit test coverage
- \#2548 Upgrade mishards for milvus v0.10.0
## Task
......
core/conf/demo/server_config.yaml
浏览文件 @ d1c4ab91
......@@ -95,7 +95,7 @@ storage:
wal:
enable: true
recovery_error_ignore: false
buffer_size: 256
buffer_size: 256MB
path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -115,8 +115,8 @@ wal:
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache:
cache_size: 4
insert_buffer_size: 1
cache_size: 4GB
insert_buffer_size: 1GB
preload_collection:
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -142,7 +142,7 @@ cache:
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu:
enable: false
cache_size: 1
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
......@@ -168,7 +168,7 @@ logs:
level: debug
trace.enable: true
path: /var/lib/milvus/logs
max_log_file_size: 1024
max_log_file_size: 1024MB
log_rotate_num: 0
#----------------------+------------------------------------------------------------+------------+-----------------+
......
shards/all_in_one/all_in_one.yml
浏览文件 @ d1c4ab91
......@@ -3,7 +3,7 @@ services:
milvus_wr:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.9.1-gpu-d052920-e04ed5
image: milvusdb/milvus:0.10.0-gpu-d061320-83cc81
ports:
- "0.0.0.0:19540:19530"
volumes:
......@@ -13,7 +13,7 @@ services:
milvus_ro:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.9.1-gpu-d052920-e04ed5
image: milvusdb/milvus:0.10.0-gpu-d061320-83cc81
ports:
- "0.0.0.0:19541:19530"
volumes:
......
shards/all_in_one/ro_server.yml
浏览文件 @ d1c4ab91
......@@ -9,91 +9,99 @@
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.4
version: 0.5
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
# enable | If runinng with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
# role | Milvus deployment role: rw / ro | role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config:
address: 0.0.0.0
port: 19530
deploy_mode: cluster_readonly
time_zone: UTC+8
web_enable: true
web_port: 19121
cluster:
enable: true
role: ro
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
general:
timezone: UTC+8
meta_uri: sqlite://:@:/
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_collection:
auto_flush_interval: 1
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
# | index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
# | to save vector data and index data. | | |
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# file_cleanup_timeout | The time gap between marking a file as 'deleted' and | Integer | 10 (s) |
# | physically deleting this file from disk, range [0, 3600] | | |
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
file_cleanup_timeout: 10
storage:
path: /var/lib/milvus
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | false |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config:
enable_monitor: false
address: 127.0.0.1
port: 9091
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
recovery_error_ignore: false
buffer_size: 256MB
path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# cache_size | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -101,23 +109,22 @@ metric_config:
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config:
cpu_cache_capacity: 4
insert_buffer_size: 1
cache_insert_data: false
cache:
cache_size: 4GB
insert_buffer_size: 1GB
preload_collection:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
# cache_size | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
......@@ -127,85 +134,29 @@ cache_config:
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config:
use_blas_threshold: 1100
gpu_search_threshold: 1000
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
gpu:
enable: true
cache_capacity: 1
search_resources:
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_resources:
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs | Description | Type | Default |
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# debug.enable | Whether to enable debug level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# info.enable | Whether to enable info level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# warning.enable | Whether to enable warning level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# error.enable | Whether to enable error level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# fatal.enable | Whether to enable fatal level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range [512, 4096] | Integer | 1024 (MB) |
......@@ -214,13 +165,23 @@ wal_config:
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: debug
trace.enable: true
debug.enable: true
info.enable: true
warning.enable: true
error.enable: true
fatal.enable: true
path: /var/lib/milvus/logs
max_log_file_size: 1024
max_log_file_size: 1024MB
log_rotate_num: 0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091
shards/all_in_one/wr_server.yml
浏览文件 @ d1c4ab91
......@@ -9,91 +9,99 @@
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.4
version: 0.5
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
# enable | If runinng with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
# role | Milvus deployment role: rw / ro | role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config:
address: 0.0.0.0
port: 19530
deploy_mode: cluster_writable
time_zone: UTC+8
web_enable: true
web_port: 19121
cluster:
enable: true
role: rw
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
general:
timezone: UTC+8
meta_uri: sqlite://:@:/
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_collection:
auto_flush_interval: 1
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
# | index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
# | to save vector data and index data. | | |
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# file_cleanup_timeout | The time gap between marking a file as 'deleted' and | Integer | 10 (s) |
# | physically deleting this file from disk, range [0, 3600] | | |
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
file_cleanup_timeout: 10
storage:
path: /var/lib/milvus
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | false |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config:
enable_monitor: false
address: 127.0.0.1
port: 9091
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
recovery_error_ignore: false
buffer_size: 256MB
path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# cache_size | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -101,23 +109,22 @@ metric_config:
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config:
cpu_cache_capacity: 4
insert_buffer_size: 1
cache_insert_data: false
cache:
cache_size: 4GB
insert_buffer_size: 1GB
preload_collection:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
# cache_size | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
......@@ -127,85 +134,29 @@ cache_config:
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config:
use_blas_threshold: 1100
gpu_search_threshold: 1000
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
gpu:
enable: true
cache_capacity: 1
search_resources:
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_resources:
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs | Description | Type | Default |
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# debug.enable | Whether to enable debug level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# info.enable | Whether to enable info level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# warning.enable | Whether to enable warning level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# error.enable | Whether to enable error level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# fatal.enable | Whether to enable fatal level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range [512, 4096] | Integer | 1024 (MB) |
......@@ -214,14 +165,23 @@ wal_config:
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: debug
trace.enable: true
debug.enable: true
info.enable: true
warning.enable: true
error.enable: true
fatal.enable: true
path: /var/lib/milvus/logs
max_log_file_size: 1024
max_log_file_size: 1024MB
log_rotate_num: 0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091
shards/all_in_one_with_mysql/all_in_one.yml
浏览文件 @ d1c4ab91
......@@ -18,7 +18,7 @@ services:
milvus_wr:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.9.1-gpu-d052920-e04ed5
image: milvusdb/milvus:0.10.0-gpu-d061320-83cc81
volumes:
- /tmp/milvus/db:/var/lib/milvus/db
- ./wr_server.yml:/var/lib/milvus/conf/server_config.yaml
......@@ -29,7 +29,7 @@ services:
milvus_ro:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.9.1-gpu-d052920-e04ed5
image: milvusdb/milvus:0.10.0-gpu-d061320-83cc81
volumes:
- /tmp/milvus/db:/var/lib/milvus/db
- ./ro_server.yml:/var/lib/milvus/conf/server_config.yaml
......
shards/all_in_one_with_mysql/ro_server.yml
浏览文件 @ d1c4ab91
......@@ -9,91 +9,99 @@
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.4
version: 0.5
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
# enable | If runinng with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
# role | Milvus deployment role: rw / ro | role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config:
address: 0.0.0.0
port: 19530
deploy_mode: cluster_readonly
time_zone: UTC+8
web_enable: true
web_port: 19121
cluster:
enable: true
role: ro
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
general:
timezone: UTC+8
meta_uri: mysql://root:milvusroot@milvus-mysql:3306/milvus
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
preload_collection:
auto_flush_interval: 1
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
# | index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
# | to save vector data and index data. | | |
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# file_cleanup_timeout | The time gap between marking a file as 'deleted' and | Integer | 10 (s) |
# | physically deleting this file from disk, range [0, 3600] | | |
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
file_cleanup_timeout: 10
storage:
path: /var/lib/milvus
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | false |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config:
enable_monitor: false
address: 127.0.0.1
port: 9091
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
recovery_error_ignore: false
buffer_size: 256MB
path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# cache_size | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -101,23 +109,22 @@ metric_config:
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config:
cpu_cache_capacity: 4
insert_buffer_size: 1
cache_insert_data: false
cache:
cache_size: 4GB
insert_buffer_size: 1GB
preload_collection:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
# cache_size | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
......@@ -127,85 +134,29 @@ cache_config:
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config:
use_blas_threshold: 1100
gpu_search_threshold: 1000
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
gpu:
enable: true
cache_capacity: 1
search_resources:
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_resources:
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs | Description | Type | Default |
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# debug.enable | Whether to enable debug level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# info.enable | Whether to enable info level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# warning.enable | Whether to enable warning level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# error.enable | Whether to enable error level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# fatal.enable | Whether to enable fatal level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range [512, 4096] | Integer | 1024 (MB) |
......@@ -214,13 +165,23 @@ wal_config:
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: debug
trace.enable: true
debug.enable: true
info.enable: true
warning.enable: true
error.enable: true
fatal.enable: true
path: /var/lib/milvus/logs
max_log_file_size: 1024
max_log_file_size: 1024MB
log_rotate_num: 0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091
shards/all_in_one_with_mysql/wr_server.yml
浏览文件 @ d1c4ab91
......@@ -9,91 +9,99 @@
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.4
version: 0.5
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
# enable | If runinng with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
# role | Milvus deployment role: rw / ro | role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config:
address: 0.0.0.0
port: 19530
deploy_mode: cluster_writable
time_zone: UTC+8
web_enable: true
web_port: 19121
cluster:
enable: true
role: rw
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
general:
timezone: UTC+8
meta_uri: mysql://root:milvusroot@milvus-mysql:3306/milvus
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
preload_collection:
auto_flush_interval: 1
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable web server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
# | index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
# | to save vector data and index data. | | |
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# file_cleanup_timeout | The time gap between marking a file as 'deleted' and | Integer | 10 (s) |
# | physically deleting this file from disk, range [0, 3600] | | |
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
file_cleanup_timeout: 10
storage:
path: /var/lib/milvus
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | false |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config:
enable_monitor: false
address: 127.0.0.1
port: 9091
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
recovery_error_ignore: false
buffer_size: 256MB
path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# cache_size | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
......@@ -101,23 +109,22 @@ metric_config:
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config:
cpu_cache_capacity: 4
insert_buffer_size: 1
cache_insert_data: false
cache:
cache_size: 4GB
insert_buffer_size: 1GB
preload_collection:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
# cache_size | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
......@@ -127,85 +134,29 @@ cache_config:
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config:
use_blas_threshold: 1100
gpu_search_threshold: 1000
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
gpu:
enable: true
cache_capacity: 1
search_resources:
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_resources:
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs | Description | Type | Default |
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# debug.enable | Whether to enable debug level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# info.enable | Whether to enable info level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# warning.enable | Whether to enable warning level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# error.enable | Whether to enable error level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# fatal.enable | Whether to enable fatal level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range [512, 4096] | Integer | 1024 (MB) |
......@@ -214,13 +165,23 @@ wal_config:
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: debug
trace.enable: true
debug.enable: true
info.enable: true
warning.enable: true
error.enable: true
fatal.enable: true
path: /var/lib/milvus/logs
max_log_file_size: 1024
max_log_file_size: 1024MB
log_rotate_num: 0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091
shards/mishards/connections.py
浏览文件 @ d1c4ab91
import copy
import time
import json
import logging
......@@ -5,7 +6,6 @@ import threading
from functools import wraps
from collections import defaultdict
from milvus import Milvus
# from milvus.client.hooks import BaseSearchHook
from mishards import (settings, exceptions, topology)
from utils import singleton
......@@ -254,7 +254,9 @@ class ConnectionGroup(topology.TopoGroup):
uri = kwargs.get('uri', None)
if not uri:
raise RuntimeError('\"uri\" is required to create connection pool')
pool = Milvus(name=name, **kwargs)
milvus_args = copy.deepcopy(kwargs)
milvus_args["max_retry"] = settings.MAX_RETRY
pool = Milvus(name=name, **milvus_args)
status = self.add(pool)
if status != topology.StatusType.OK:
pool = None
......
shards/requirements.txt
浏览文件 @ d1c4ab91
......@@ -14,8 +14,8 @@ py==1.8.0
pyasn1==0.4.7
pyasn1-modules==0.2.6
pylint==2.5.0
#pymilvus==0.2.12
pymilvus-test==0.3.25
pymilvus==0.2.13
#pymilvus-test==0.3.25
pyparsing==2.4.0
pytest==4.6.3
pytest-level==0.1.1
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或