diff --git a/docs/configuration-large.md b/docs/configuration-large.md new file mode 100644 index 0000000000000000000000000000000000000000..5dc3c03e799c34ad700da2106f55baa165e64bc8 --- /dev/null +++ b/docs/configuration-large.md @@ -0,0 +1,27 @@ +# Orchestrator configuration in larger environments + +If monitoring a large number of servers the backend database can become a bottleneck. The following comments refer to using MySQL as the orchestrator backend. + +Some configuration options allow you to control the throughput. These settings are: +- `BufferInstanceWrites` +- `InstanceWriteBufferSize` +- `InstanceFlushIntervalMilliseconds` +- `DiscoveryMaxConcurrency` + +Limit the number of concurrent discoveries made by orchestrator using `DiscoveryMaxConcurrency` and ensure that the backend server's `max_connections` setting is high enough to allow orchestrator to make as many connections as it needs. + +By setting `BufferInstanceWrites: True` in orchestrator when a poll completes the results will be buffered until `InstanceFlushIntervalMilliseconds` has elapsed or `InstanceWriteBufferSize` buffered writes have been made. + +The buffered writes are ordered by the time of the write using a single `insert ... on duplicate key update ...` call. If the same host appears twice only the last write will be written to the database for that host. + +`InstanceFlushIntervalMilliseconds` *should be* well below `InstancePollSeconds` as making this value too high will mean the data is not being written to the orchestrator db backend. This can lead to `not recently checked` problems. Also the different health checks are run against the backend database state so not updating it frequently enough could lead to Orchestrator not detecting the different failure scenarios correctly. + +Suggested values to start with for larger Orchestrator environments might be: +``` + ... + "BufferInstanceWrites": true, + "InstanceWriteBufferSize": 1000, + "InstanceFlushIntervalMilliseconds": 50, + "DiscoveryMaxConcurrency": 1000, + ... +``` diff --git a/docs/configuration.md b/docs/configuration.md index 083119ace3f22a89f0cd7a88876b0b564c67c423..2ee971706bec401c33bdce0b5fde049172f81de0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -19,6 +19,7 @@ Use the following small steps to configure `orchestrator`: - [Raft](configuration-raft.md): configure a [orchestrator/raft](raft.md) cluster for high availability - Security: See [security](security.md) section. - [Key-Value stores](configuration-kv.md): configure and use key-value stores for master discovery. +- [Hints on some settings suitable for larger orchestrator environments](configuration-large.md) ### Configuration sample file