Skip to content

TDengine
TDengine

taosdata / TDengine
大约 2 年 前同步成功

通知 1193
Star 22018
Fork 4786
TDengine
TDengine
“895c936938112cc7a395316cca377a9036996afb”上不存在“docs/examples/csharp/wsConnect/wsConnect.csproj”
未验证 提交 a76bbfe0 编写于 作者: W wade zhang 提交者： GitHub
浏览文件
操作

Merge pull request #16429 from taosdata/docs/daniel-3.0 

doc: reuploading doc updates to 3.0 branch
上级 32ce4b31 3633281a
隐藏空白更改
内联 并排
Showing with 742 addition and 1028 deletion
docs/en/13-operation/02-planning.mdx
浏览文件 @ a76bbfe0
--- ---
sidebar_label: Resource Planning
title: Resource Planning title: Resource Planning
--- ---
It is important to plan computing and storage resources if using TDengine to build an IoT, time-series or Big Data platform. How to plan the CPU, memory and disk resources required, will be described in this chapter. It is important to plan computing and storage resources if using TDengine to build an IoT, time-series or Big Data platform. How to plan the CPU, memory and disk resources required, will be described in this chapter.
## Memory Requirement of Server Side ## Server Memory Requirements
By default, the number of vgroups created for each database is the same as the number of CPU cores. This can be configured by the parameter `maxVgroupsPerDb`. Each vnode in a vgroup stores one replica. Each vnode consumes a fixed amount of memory, i.e. `blocks` \* `cache`. In addition, some memory is required for tag values associated with each table. A fixed amount of memory is required for each cluster. So, the memory required for each DB can be calculated using the formula below: Each database creates a fixed number of vgroups. This number is 2 by default and can be configured with the `vgroups` parameter. The number of replicas can be controlled with the `replica` parameter. Each replica requires one vnode per vgroup. Altogether, the memory required by each database depends on the following configuration options:
``` - vgroups
Database Memory Size = maxVgroupsPerDb * replica * (blocks * cache + 10MB) + numOfTables * (tagSizePerTable + 0.5KB) - replica
``` - buffer
- pages
- pagesize
- cachesize
For example, assuming the default value of `maxVgroupPerDB` is 64, the default value of `cache` is 16M, the default value of `blocks` is 6, there are 100,000 tables in a DB, the replica number is 1, total length of tag values is 256 bytes, the total memory required for this DB is: 64 \* 1 \* (16 \* 6 + 10) + 100000 \* (0.25 + 0.5) / 1000 = 6792M. For more information, see [Database](../../taos-sql/database).
In the real operation of TDengine, we are more concerned about the memory used by each TDengine server process `taosd`. The memory required by a database is therefore greater than or equal to:
``` ```
taosd_memory = vnode_memory + mnode_memory + query_memory vgroups * replica * (buffer + pages * pagesize + cachesize)
``` ```
In the above formula: However, note that this requirement is spread over all dnodes in the cluster, not on a single physical machine. The physical servers that run dnodes meet the requirement together. If a cluster has multiple databases, the memory required increases accordingly. In complex environments where dnodes were added after initial deployment in response to increasing resource requirements, load may not be balanced among the original dnodes and newer dnodes. In this situation, the actual status of your dnodes is more important than theoretical calculations.
1. "vnode_memory" of a `taosd` process is the memory used by all vnodes hosted by this `taosd` process. It can be roughly calculated by firstly adding up the total memory of all DBs whose memory usage can be derived according to the formula for Database Memory Size, mentioned above, then dividing by number of dnodes and multiplying the number of replicas.
```
vnode_memory = (sum(Database Memory Size) / number_of_dnodes) * replica
```
2. "mnode_memory" of a `taosd` process is the memory consumed by a mnode. If there is one (and only one) mnode hosted in a `taosd` process, the memory consumed by "mnode" is "0.2KB \* the total number of tables in the cluster".
3. "query_memory" is the memory used when processing query requests. Each ongoing query consumes at least "0.2 KB \* total number of involved tables".
Please note that the above formulas can only be used to estimate the minimum memory requirement, instead of maximum memory usage. In a real production environment, it's better to reserve some redundance beyond the estimated minimum memory requirement. If memory is abundant, it's suggested to increase the value of parameter `blocks` to speed up data insertion and data query.
## Memory Requirement of Client Side ## Client Memory Requirements
For the client programs using TDengine client driver `taosc` to connect to the server side there is a memory requirement as well. For the client programs using TDengine client driver `taosc` to connect to the server side there is a memory requirement as well.
...@@ -56,10 +48,10 @@ So, at least 3GB needs to be reserved for such a client. ...@@ -56,10 +48,10 @@ So, at least 3GB needs to be reserved for such a client.
The CPU resources required depend on two aspects: The CPU resources required depend on two aspects:
- **Data Insertion** Each dnode of TDengine can process at least 10,000 insertion requests in one second, while each insertion request can have multiple rows. The difference in computing resource consumed, between inserting 1 row at a time, and inserting 10 rows at a time is very small. So, the more the number of rows that can be inserted one time, the higher the efficiency. Inserting in batch also imposes requirements on the client side which needs to cache rows to insert in batch once the number of cached rows reaches a threshold. - **Data Insertion** Each dnode of TDengine can process at least 10,000 insertion requests in one second, while each insertion request can have multiple rows. The difference in computing resource consumed, between inserting 1 row at a time, and inserting 10 rows at a time is very small. So, the more the number of rows that can be inserted one time, the higher the efficiency. If each insert request contains more than 200 records, a single core can process more than 1 million records per second. Inserting in batch also imposes requirements on the client side which needs to cache rows to insert in batch once the number of cached rows reaches a threshold.
- **Data Query** High efficiency query is provided in TDengine, but it's hard to estimate the CPU resource required because the queries used in different use cases and the frequency of queries vary significantly. It can only be verified with the query statements, query frequency, data size to be queried, and other requirements provided by users. - **Data Query** High efficiency query is provided in TDengine, but it's hard to estimate the CPU resource required because the queries used in different use cases and the frequency of queries vary significantly. It can only be verified with the query statements, query frequency, data size to be queried, and other requirements provided by users.
In short, the CPU resource required for data insertion can be estimated but it's hard to do so for query use cases. In real operation, it's suggested to control CPU usage below 50%. If this threshold is exceeded, it's a reminder for system operator to add more nodes in the cluster to expand resources. In short, the CPU resource required for data insertion can be estimated but it's hard to do so for query use cases. If possible, ensure that CPU usage remains below 50%. If this threshold is exceeded, it's a reminder for system operator to add more nodes in the cluster to expand resources.
## Disk Requirement ## Disk Requirement
...@@ -77,6 +69,6 @@ To increase performance, multiple disks can be setup for parallel data reading o ...@@ -77,6 +69,6 @@ To increase performance, multiple disks can be setup for parallel data reading o
## Number of Hosts ## Number of Hosts
A host can be either physical or virtual. The total memory, total CPU, total disk required can be estimated according to the formulae mentioned previously. Then, according to the system resources that a single host can provide, assuming all hosts have the same resources, the number of hosts can be derived easily. A host can be either physical or virtual. The total memory, total CPU, total disk required can be estimated according to the formulae mentioned previously. If the number of data replicas is not 1, the required resources are multiplied by the number of replicas.
**Quick Estimation for CPU, Memory and Disk** Please refer to [Resource Estimate](https://www.taosdata.com/config/config.html). Then, according to the system resources that a single host can provide, assuming all hosts have the same resources, the number of hosts can be derived easily.
docs/en/13-operation/03-tolerance.md
浏览文件 @ a76bbfe0
--- ---
sidebar_label: Fault Tolerance title: Fault Tolerance and Disaster Recovery
title: Fault Tolerance & Disaster Recovery
--- ---
## Fault Tolerance ## Fault Tolerance
...@@ -11,22 +10,21 @@ When a data block is received by TDengine, the original data block is first writ ...@@ -11,22 +10,21 @@ When a data block is received by TDengine, the original data block is first writ
There are 2 configuration parameters related to WAL: There are 2 configuration parameters related to WAL:
- walLevel： - wal_level: Specifies the WAL level. 1 indicates that WAL is enabled but fsync is disabled. 2 indicates that WAL and fsync are both enabled. The default value is 1.
- 0：wal is disabled - wal_fsync_period: This parameter is only valid when wal_level is set to 2. It specifies the interval, in milliseconds, of invoking fsync. If set to 0, it means fsync is invoked immediately once WAL is written.
- 1：wal is enabled without fsync
- 2：wal is enabled with fsync
- fsync：This parameter is only valid when walLevel is set to 2. It specifies the interval, in milliseconds, of invoking fsync. If set to 0, it means fsync is invoked immediately once WAL is written.
To achieve absolutely no data loss, walLevel should be set to 2 and fsync should be set to 1. There is a performance penalty to the data ingestion rate. However, if the concurrent data insertion threads on the client side can reach a big enough number, for example 50, the data ingestion performance will be still good enough. Our verification shows that the drop is only 30% when fsync is set to 3,000 milliseconds. To achieve absolutely no data loss, set wal_level to 2 and wal_fsync_period to 0. There is a performance penalty to the data ingestion rate. However, if the concurrent data insertion threads on the client side can reach a big enough number, for example 50, the data ingestion performance will be still good enough. Our verification shows that the drop is only 30% when wal_fsync_period is set to 3000 milliseconds.
## Disaster Recovery ## Disaster Recovery
TDengine uses replication to provide high availability and disaster recovery capability. TDengine uses replication to provide high availability.
A TDengine cluster is managed by mnode. To ensure the high availability of mnode, multiple replicas can be configured by the system parameter `numOfMnodes`. The data replication between mnode replicas is performed in a synchronous way to guarantee metadata consistency. A TDengine cluster is managed by mnodes. You can configure up to three mnodes to ensure high availability. The data replication between mnode replicas is performed in a synchronous way to guarantee metadata consistency.
The number of replicas for time series data in TDengine is associated with each database. There can be many databases in a cluster and each database can be configured with a different number of replicas. When creating a database, parameter `replica` is used to configure the number of replications. To achieve high availability, `replica` needs to be higher than 1. The number of replicas for time series data in TDengine is associated with each database. There can be many databases in a cluster and each database can be configured with a different number of replicas. When creating a database, the parameter `replica` is used to specify the number of replicas. To achieve high availability, set `replica` to 3.
The number of dnodes in a TDengine cluster must NOT be lower than the number of replicas for any database, otherwise it would fail when trying to create a table. The number of dnodes in a TDengine cluster must NOT be lower than the number of replicas for any database, otherwise it would fail when trying to create a table.
As long as the dnodes of a TDengine cluster are deployed on different physical machines and the replica number is higher than 1, high availability can be achieved without any other assistance. For disaster recovery, dnodes of a TDengine cluster should be deployed in geographically different data centers. As long as the dnodes of a TDengine cluster are deployed on different physical machines and the replica number is higher than 1, high availability can be achieved without any other assistance. For disaster recovery, dnodes of a TDengine cluster should be deployed in geographically different data centers.
Alternatively, you can use taosX to synchronize the data from one TDengine cluster to another cluster in a remote location. For more information, see [taosX](../../reference/taosX).
docs/en/13-operation/17-diagnose.md
浏览文件 @ a76bbfe0
...@@ -13,110 +13,59 @@ Diagnostic steps： ...@@ -13,110 +13,59 @@ Diagnostic steps：
1. If the port range to be diagnosed is being occupied by a `taosd` server process, please first stop `taosd. 1. If the port range to be diagnosed is being occupied by a `taosd` server process, please first stop `taosd.
2. On the server side, execute command `taos -n server -P <port> -l <pktlen>` to monitor the port range starting from the port specified by `-P` parameter with the role of "server". 2. On the server side, execute command `taos -n server -P <port> -l <pktlen>` to monitor the port range starting from the port specified by `-P` parameter with the role of "server".
3. On the client side, execute command `taos -n client -h <fqdn of server> -P <port> -l <pktlen>` to send a testing package to the specified server and port. 3. On the client side, execute command `taos -n client -h <fqdn of server> -P <port> -l <pktlen>` to send a testing package to the specified server and port.
-l <pktlen\>： The size of the testing package, in bytes. The value range is [11, 64,000] and default value is 1,000. Please note that the package length must be same in the above 2 commands executed on server side and client side respectively. -l <pktlen\>： The size of the testing package, in bytes. The value range is [11, 64,000] and default value is 1,000.
Please note that the package length must be same in the above 2 commands executed on server side and client side respectively.
Output of the server side for the example is below: Output of the server side for the example is below:
```bash ```bash
# taos -n server -P 6000 # taos -n server -P 6030 -l 1000
12/21 14:50:13.522509 0x7f536f455200 UTL work as server, host:172.27.0.7 startPort:6000 endPort:6011 pkgLen:1000 network test server is initialized, port:6030
request is received, size:1000
12/21 14:50:13.522659 0x7f5352242700 UTL TCP server at port:6000 is listening request is received, size:1000
12/21 14:50:13.522727 0x7f5351240700 UTL TCP server at port:6001 is listening
...
...
... ...
12/21 14:50:13.523954 0x7f5342fed700 UTL TCP server at port:6011 is listening
12/21 14:50:13.523989 0x7f53437ee700 UTL UDP server at port:6010 is listening
12/21 14:50:13.524019 0x7f53427ec700 UTL UDP server at port:6011 is listening
12/21 14:50:22.192849 0x7f5352242700 UTL TCP: read:1000 bytes from 172.27.0.8 at 6000
12/21 14:50:22.192993 0x7f5352242700 UTL TCP: write:1000 bytes to 172.27.0.8 at 6000
12/21 14:50:22.237082 0x7f5351a41700 UTL UDP: recv:1000 bytes from 172.27.0.8 at 6000
12/21 14:50:22.237203 0x7f5351a41700 UTL UDP: send:1000 bytes to 172.27.0.8 at 6000
12/21 14:50:22.237450 0x7f5351240700 UTL TCP: read:1000 bytes from 172.27.0.8 at 6001
12/21 14:50:22.237576 0x7f5351240700 UTL TCP: write:1000 bytes to 172.27.0.8 at 6001
12/21 14:50:22.281038 0x7f5350a3f700 UTL UDP: recv:1000 bytes from 172.27.0.8 at 6001
12/21 14:50:22.281141 0x7f5350a3f700 UTL UDP: send:1000 bytes to 172.27.0.8 at 6001
... ...
... ...
... request is received, size:1000
12/21 14:50:22.677443 0x7f5342fed700 UTL TCP: read:1000 bytes from 172.27.0.8 at 6011 request is received, size:1000
12/21 14:50:22.677576 0x7f5342fed700 UTL TCP: write:1000 bytes to 172.27.0.8 at 6011
12/21 14:50:22.721144 0x7f53427ec700 UTL UDP: recv:1000 bytes from 172.27.0.8 at 6011
12/21 14:50:22.721261 0x7f53427ec700 UTL UDP: send:1000 bytes to 172.27.0.8 at 6011
``` ```
Output of the client side for the example is below: Output of the client side for the example is below:
```bash ```bash
# taos -n client -h 172.27.0.7 -P 6000 # taos -n client -h 172.27.0.7 -P 6000
12/21 14:50:22.192434 0x7fc95d859200 UTL work as client, host:172.27.0.7 startPort:6000 endPort:6011 pkgLen:1000 taos -n client -h v3s2 -P 6030 -l 1000
network test client is initialized, the server is v3s2:6030
12/21 14:50:22.192472 0x7fc95d859200 UTL server ip:172.27.0.7 is resolved from host:172.27.0.7 request is sent, size:1000
12/21 14:50:22.236869 0x7fc95d859200 UTL successed to test TCP port:6000 response is received, size:1000
12/21 14:50:22.237215 0x7fc95d859200 UTL successed to test UDP port:6000 request is sent, size:1000
response is received, size:1000
... ...
... ...
... ...
12/21 14:50:22.676891 0x7fc95d859200 UTL successed to test TCP port:6010 request is sent, size:1000
12/21 14:50:22.677240 0x7fc95d859200 UTL successed to test UDP port:6010 response is received, size:1000
12/21 14:50:22.720893 0x7fc95d859200 UTL successed to test TCP port:6011 request is sent, size:1000
12/21 14:50:22.721274 0x7fc95d859200 UTL successed to test UDP port:6011 response is received, size:1000
```
The output needs to be checked carefully for the system operator to find the root cause and resolve the problem.
## Startup Status and RPC Diagnostic
`taos -n startup -h <fqdn of server>` can be used to check the startup status of a `taosd` process. This is a common task which should be performed by a system operator, especially in the case of a cluster, to determine whether `taosd` has been started successfully.
`taos -n rpc -h <fqdn of server>` can be used to check whether the port of a started `taosd` can be accessed or not. If `taosd` process doesn't respond or is working abnormally, this command can be used to initiate a rpc communication with the specified fqdn to determine whether it's a network problem or whether `taosd` is abnormal.
## Sync and Arbitrator Diagnostic
```bash total succ: 100/100 cost: 16.23 ms speed: 5.87 MB/s
taos -n sync -P 6040 -h <fqdn of server>
taos -n sync -P 6042 -h <fqdn of server>
``` ```
The above commands can be executed in a Linux shell to check whether the port for sync is working well and whether the sync module on the server side is working well. Additionally, `-P 6042` is used to check whether the arbitrator is configured properly and is working well. The output needs to be checked carefully for the system operator to find the root cause and resolve the problem.
## Network Speed Diagnostic
`taos -n speed -h <fqdn of server> -P 6030 -N 10 -l 10000000 -S TCP`
From version 2.2.0.0 onwards, the above command can be executed in a Linux shell to test network speed. The command sends uncompressed packages to a running `taosd` server process or a simulated server process started by `taos -n server` to test the network speed. Parameters can be used when testing network speed are as below:
-n：When set to "speed", it means testing network speed.
-h：The FQDN or IP of the server process to be connected to; if not set, the FQDN configured in `taos.cfg` is used.
-P：The port of the server process to connect to, the default value is 6030.
-N：The number of packages that will be sent in the test, range is [1,10000], default value is 100.
-l：The size of each package in bytes, range is [1024, 1024 \* 1024 \* 1024], default value is 1024.
-S：The type of network packages to send, can be either TCP or UDP, default value is TCP.
## FQDN Resolution Diagnostic
`taos -n fqdn -h <fqdn of server>`
From version 2.2.0.0 onward, the above command can be executed in a Linux shell to test the resolution speed of FQDN. It can be used to try to resolve a FQDN to an IP address and record the time spent in this process. The parameters that can be used for this purpose are as below:
-n：When set to "fqdn", it means testing the speed of resolving FQDN.
-h：The FQDN to be resolved. If not set, the `FQDN` parameter in `taos.cfg` is used by default.
## Server Log ## Server Log
The parameter `debugFlag` is used to control the log level of the `taosd` server process. The default value is 131. For debugging and tracing, it needs to be set to either 135 or 143 respectively. The parameter `debugFlag` is used to control the log level of the `taosd` server process. The default value is 131. For debugging and tracing, it needs to be set to either 135 or 143 respectively.
Once this parameter is set to 135 or 143, the log file grows very quickly especially when there is a huge volume of data insertion and data query requests. If all the logs are stored together, some important information may be missed very easily and so on the server side, important information is stored in a different place from other logs.
- The log at level of INFO, WARNING and ERROR is stored in `taosinfo` so that it is easy to find important information Once this parameter is set to 135 or 143, the log file grows very quickly especially when there is a huge volume of data insertion and data query requests. Ensure that the disk drive on which logs are stored has sufficient space.
- The log at level of DEBUG (135) and TRACE (143) and other information not handled by `taosinfo` are stored in `taosdlog`
## Client Log ## Client Log
An independent log file, named as "taoslog+<seq num\>" is generated for each client program, i.e. a client process. The default value of `debugFlag` is also 131 and only logs at level of INFO/ERROR/WARNING are recorded. As stated above, for debugging and tracing, it needs to be changed to 135 or 143 respectively, so that logs at DEBUG or TRACE level can be recorded. An independent log file, named as "taoslog+<seq num\>" is generated for each client program, i.e. a client process. The parameter `debugFlag` is used to control the log level. The default value is 131. For debugging and tracing, it needs to be set to either 135 or 143 respectively.
The default value of `debugFlag` is also 131 and only logs at level of INFO/ERROR/WARNING are recorded. As stated above, for debugging and tracing, it needs to be changed to 135 or 143 respectively, so that logs at DEBUG or TRACE level can be recorded.
The maximum length of a single log file is controlled by parameter `numOfLogLines` and only 2 log files are kept for each `taosd` server process. The maximum length of a single log file is controlled by parameter `numOfLogLines` and only 2 log files are kept for each `taosd` server process.
Log files are written in an async way to minimize the workload on disk, but the trade off for performance is that a few log lines may be lost in some extreme conditions. Log files are written in an async way to minimize the workload on disk, but the trade off for performance is that a few log lines may be lost in some extreme conditions. You can configure asynclog to 0 when needed for troubleshooting purposes to ensure that no log information is lost.
docs/en/14-reference/12-config/index.md
浏览文件 @ a76bbfe0
--- ---
sidebar_label: Configuration
title: Configuration Parameters title: Configuration Parameters
description: "Configuration parameters for client and server in TDengine" description: "Configuration parameters for client and server in TDengine"
--- ---
In this chapter, all the configuration parameters on both server and client side are described thoroughly.
## Configuration File on Server Side ## Configuration File on Server Side
On the server side, the actual service of TDengine is provided by an executable `taosd` whose parameters can be configured in file `taos.cfg` to meet the requirements of different use cases. The default location of `taos.cfg` is `/etc/taos`, but can be changed by using `-c` parameter on the CLI of `taosd`. For example, the configuration file can be put under `/home/user` and used like below On the server side, the actual service of TDengine is provided by an executable `taosd` whose parameters can be configured in file `taos.cfg` to meet the requirements of different use cases. The default location of `taos.cfg` is `/etc/taos`, but can be changed by using `-c` parameter on the CLI of `taosd`. For example, the configuration file can be put under `/home/user` and used like below
```bash ```
taosd -c /home/user taosd -c /home/user
``` ```
...@@ -24,8 +21,6 @@ taosd -C ...@@ -24,8 +21,6 @@ taosd -C
TDengine CLI `taos` is the tool for users to interact with TDengine. It can share same configuration file as `taosd` or use a separate configuration file. When launching `taos`, parameter `-c` can be used to specify the location where its configuration file is. For example `taos -c /home/cfg` means `/home/cfg/taos.cfg` will be used. If `-c` is not used, the default location of the configuration file is `/etc/taos`. For more details please use `taos --help` to get. TDengine CLI `taos` is the tool for users to interact with TDengine. It can share same configuration file as `taosd` or use a separate configuration file. When launching `taos`, parameter `-c` can be used to specify the location where its configuration file is. For example `taos -c /home/cfg` means `/home/cfg/taos.cfg` will be used. If `-c` is not used, the default location of the configuration file is `/etc/taos`. For more details please use `taos --help` to get.
From version 2.0.10.0 below commands can be used to show the configuration parameters of the client side.
```bash ```bash
taos -C taos -C
``` ```
...@@ -36,6 +31,11 @@ taos --dump-config ...@@ -36,6 +31,11 @@ taos --dump-config
# Configuration Parameters # Configuration Parameters
:::note
The parameters described in this document by the effect that they have on the system.
:::
:::note :::note
`taosd` needs to be restarted for the parameters changed in the configuration file to take effect. `taosd` needs to be restarted for the parameters changed in the configuration file to take effect.
...@@ -45,19 +45,19 @@ taos --dump-config ...@@ -45,19 +45,19 @@ taos --dump-config
### firstEp ### firstEp
| Attribute | Description | | Attribute | Description |
| ------------- | ---------------------------------------------------------------------------------------------------- | | -------- | -------------------------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The end point of the first dnode in the cluster to be connected to when `taosd` or `taos` is started | | Meaning | The end point of the first dnode in the cluster to be connected to when `taosd` or `taos` is started |
| Default Value | localhost:6030 | | Default | localhost:6030 |
### secondEp ### secondEp
| Attribute | Description | | Attribute | Description |
| ------------- | ---------------------------------------------------------------------------------------------------------------------- | | -------- | ------------------------------------------------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The end point of the second dnode to be connected to if the firstEp is not available when `taosd` or `taos` is started | | Meaning | The end point of the second dnode to be connected to if the firstEp is not available when `taosd` or `taos` is started |
| Default Value | None | | Default | None |
### fqdn ### fqdn
...@@ -65,35 +65,28 @@ taos --dump-config ...@@ -65,35 +65,28 @@ taos --dump-config
| ------------- | ------------------------------------------------------------------------ | | ------------- | ------------------------------------------------------------------------ |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The FQDN of the host where `taosd` will be started. It can be IP address | | Meaning | The FQDN of the host where `taosd` will be started. It can be IP address |
| Default Value | The first hostname configured for the host | | Default Value | The first hostname configured for the host |
| Note | It should be within 96 bytes | | Note | It should be within 96 bytes | |
### serverPort ### serverPort
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------- | | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The port for external access after `taosd` is started | | Meaning | The port for external access after `taosd` is started |
| Default Value | 6030 | | Default Value | 6030 |
:::note :::note
TDengine uses 13 continuous ports, both TCP and UDP, starting with the port specified by `serverPort`. You should ensure, in your firewall rules, that these ports are kept open. Below table describes the ports used by TDengine in details. - Ensure that your firewall rules do not block TCP port 6042 on any host in the cluster. Below table describes the ports used by TDengine in details.
::: :::
| Protocol | Default Port | Description | How to configure | | Protocol | Default Port | Description | How to configure |
| :------- | :----------- | :----------------------------------------------- | :--------------------------------------------------------------------------------------------- | | :------- | :----------- | :----------------------------------------------- | :--------------------------------------------------------------------------------------------- |
| TCP | 6030 | Communication between client and server | serverPort | | TCP | 6030 | Communication between client and server. In a multi-node cluster, communication between nodes. serverPort |
| TCP | 6035 | Communication among server nodes in cluster | serverPort+5 | | TCP | 6041 | REST connection between client and server | Prior to 2.4.0.0: serverPort+11; After 2.4.0.0 refer to [taosAdapter](/reference/taosadapter/) |
| TCP | 6040 | Data syncup among server nodes in cluster | serverPort+10 | | TCP | 6043 | Service Port of TaosKeeper | The parameter of TaosKeeper |
| TCP | 6041 | REST connection between client and server | Please refer to [taosAdapter](../taosadapter/) | | TCP | 6044 | Data access port for StatsD | Configurable through taosAdapter parameters.
| TCP | 6042 | Service Port of Arbitrator | The parameter of Arbitrator | | UDP | 6045 | Data access for statsd | Configurable through taosAdapter parameters.
| TCP | 6043 | Service Port of TaosKeeper | The parameter of TaosKeeper | | TCP | 6060 | Port of Monitoring Service in Enterprise version | |
| TCP | 6044 | Data access port for StatsD | refer to [taosAdapter](../taosadapter/) |
| UDP | 6045 | Data access for statsd | refer to [taosAdapter](../taosadapter/) |
| TCP | 6060 | Port of Monitoring Service in Enterprise version | |
| UDP | 6030-6034 | Communication between client and server | serverPort |
| UDP | 6035-6039 | Communication among server nodes in cluster | serverPort |
### maxShellConns ### maxShellConns
...@@ -104,104 +97,109 @@ TDengine uses 13 continuous ports, both TCP and UDP, starting with the port spec ...@@ -104,104 +97,109 @@ TDengine uses 13 continuous ports, both TCP and UDP, starting with the port spec
| Value Range | 10-50000000 | | Value Range | 10-50000000 |
| Default Value | 5000 | | Default Value | 5000 |
### maxConnections ## Monitoring Parameters
| Attribute | Description | ### monitor
| ------------- | ----------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The maximum number of connections allowed by a database |
| Value Range | 1-100000 |
| Default Value | 5000 |
| Note | The maximum number of worker threads on the client side is maxConnections/100 |
### rpcForceTcp | Attribute | Description |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server only |
| Meaning | The switch for monitoring inside server. The main object of monitoring is to collect information about load on physical nodes, including CPU usage, memory usage, disk usage, and network bandwidth. Monitoring information is sent over HTTP to the taosKeeper service specified by `monitorFqdn` and `monitorProt`.
| Value Range | 0: monitoring disabled, 1: monitoring enabled |
| Default | 1 |
| Attribute | Description | ### monitorFqdn
| ------------- | ------------------------------------------------------------------- |
| Applicable | Server and Client |
| Meaning | TCP is used by force |
| Value Range | 0: disabled 1: enabled |
| Default Value | 0 |
| Note | It's suggested to configure to enable if network is not good enough |
## Monitoring Parameters | Attribute | Description |
| -------- | -------------------------- |
| Applicable | Server Only |
| Meaning | FQDN of taosKeeper monitoring service |
| Default | None |
### monitor ### monitorPort
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------- | --------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The switch for monitoring inside server. The workload of the hosts, including CPU, memory, disk, network, TTP requests, are collected and stored in a system builtin database `LOG` | | Meaning | Port of taosKeeper monitoring service |
| Value Range | 0: monitoring disabled, 1: monitoring enabled | | Default Value | 6043 |
| Default Value | 1 |
### monitorInterval ### monitorInterval
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------ | | -------- | -------------------------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The interval of collecting system workload | | Meaning | The interval of collecting system workload |
| Unit | second | | Unit | second |
| Value Range | 1-600 | | Value Range | 1-200000 |
| Default Value | 30 | | Default Value | 30 |
### telemetryReporting ### telemetryReporting
| Attribute | Description | | Attribute | Description |
| ------------- | ---------------------------------------------------------------------------- | | -------- | ---------------------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | Switch for allowing TDengine to collect and report service usage information | | Meaning | Switch for allowing TDengine to collect and report service usage information |
| Value Range | 0: Not allowed; 1: Allowed | | Value Range | 0: Not allowed; 1: Allowed |
| Default Value | 1 | | Default Value | 1 |
## Query Parameters ## Query Parameters
### queryBufferSize ### queryPolicy
| Attribute | Description | | Attribute | Description |
| ------------- | ---------------------------------------------------------------------------------------- | | -------- | ----------------------------- |
| Applicable | Server Only | | Applicable | Client only |
| Meaning | The total memory size reserved for all queries | | Meaning | Execution policy for query statements |
| Unit | MB | | Unit | None |
| Default Value | None | | Default | 1 |
| Note | It can be estimated by "maximum number of concurrent queries" _ "number of tables" _ 170 | | Notes | 1: Run queries on vnodes and not on qnodes |
2: Run subtasks without scan operators on qnodes and subtasks with scan operators on vnodes.
### ratioOfQueryCores 3: Only run scan operators on vnodes; run all other operators on qnodes.
### querySmaOptimize
| Attribute | Description |
| -------- | -------------------- |
| Applicable | Client only |
| 含义 | SMA index optimization policy |
| Unit | None |
| Default Value | 0 |
| Notes |
0: Disable SMA indexing and perform all queries on non-indexed data.
1: Enable SMA indexing and perform queries from suitable statements on precomputation results.|
| Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Maximum number of query threads |
| Default Value | 1 |
| Note | value range: float number between [0, 2] 0: only 1 query thread; >0: the times of the number of cores |
### maxNumOfDistinctRes ### maxNumOfDistinctRes
| Attribute | Description | | Attribute | Description |
| ------------- | -------------------------------------------- | | -------- | -------------------------------- | --- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The maximum number of distinct rows returned | | Meaning | The maximum number of distinct rows returned |
| Value Range | [100,000 - 100,000,000] | | Value Range | [100,000 - 100,000,000] |
| Default Value | 100,000 | | Default Value | 100,000 |
| Note | After version 2.3.0.0 |
## Locale Parameters ## Locale Parameters
### timezone ### timezone
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------- | | -------- | ------------------------------ |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | TimeZone | | Meaning | TimeZone |
| Default Value | TimeZone configured in the host | | Default Value | TimeZone configured in the host |
:::info :::info
To handle the data insertion and data query from multiple timezones, Unix Timestamp is used and stored in TDengine. The timestamp generated from any timezones at same time is same in Unix timestamp. To make sure the time on client side can be converted to Unix timestamp correctly, the timezone must be set properly. To handle the data insertion and data query from multiple timezones, Unix Timestamp is used and stored in TDengine. The timestamp generated from any timezones at same time is same in Unix timestamp. Note that Unix timestamps are converted and recorded on the client side. To make sure the time on client side can be converted to Unix timestamp correctly, the timezone must be set properly.
On Linux system, TDengine clients automatically obtain timezone from the host. Alternatively, the timezone can be configured explicitly in configuration file `taos.cfg` like below. On Linux system, TDengine clients automatically obtain timezone from the host. Alternatively, the timezone can be configured explicitly in configuration file `taos.cfg` like below. For example:
``` ```
timezone UTC-7 timezone UTC-8
timezone GMT-8 timezone GMT-8
timezone Asia/Shanghai timezone Asia/Shanghai
``` ```
...@@ -239,11 +237,11 @@ To avoid the problems of using time strings, Unix timestamp can be used directly ...@@ -239,11 +237,11 @@ To avoid the problems of using time strings, Unix timestamp can be used directly
| Default Value | Locale configured in host | | Default Value | Locale configured in host |
:::info :::info
A specific type "nchar" is provided in TDengine to store non-ASCII characters such as Chinese, Japanese, and Korean. The characters to be stored in nchar type are firstly encoded in UCS4-LE before sending to server side. To store non-ASCII characters correctly, the encoding format of the client side needs to be set properly. A specific type "nchar" is provided in TDengine to store non-ASCII characters such as Chinese, Japanese, and Korean. The characters to be stored in nchar type are firstly encoded in UCS4-LE before sending to server side. Note that the correct encoding is determined by the user. To store non-ASCII characters correctly, the encoding format of the client side needs to be set properly.
The characters input on the client side are encoded using the default system encoding, which is UTF-8 on Linux, or GB18030 or GBK on some systems in Chinese, POSIX in docker, CP936 on Windows in Chinese. The encoding of the operating system in use must be set correctly so that the characters in nchar type can be converted to UCS4-LE. The characters input on the client side are encoded using the default system encoding, which is UTF-8 on Linux, or GB18030 or GBK on some systems in Chinese, POSIX in docker, CP936 on Windows in Chinese. The encoding of the operating system in use must be set correctly so that the characters in nchar type can be converted to UCS4-LE.
The locale definition standard on Linux is: <Language\>\_<Region\>.<charset\>, for example, in "zh_CN.UTF-8", "zh" means Chinese, "CN" means China mainland, "UTF-8" means charset. On Linux and Mac OSX, the charset can be set by locale in the system. On Windows system another configuration parameter `charset` must be used to configure charset because the locale used on Windows is not POSIX standard. Of course, `charset` can also be used on Linux to specify the charset. The locale definition standard on Linux is: <Language\>\_<Region\>.<charset\>, for example, in "zh_CN.UTF-8", "zh" means Chinese, "CN" means China mainland, "UTF-8" means charset. The charset indicates how to display the characters. On Linux and Mac OSX, the charset can be set by locale in the system. On Windows system another configuration parameter `charset` must be used to configure charset because the locale used on Windows is not POSIX standard. Of course, `charset` can also be used on Linux to specify the charset.
::: :::
...@@ -256,29 +254,37 @@ The locale definition standard on Linux is: <Language\>\_<Region\>.<charset\>, f ...@@ -256,29 +254,37 @@ The locale definition standard on Linux is: <Language\>\_<Region\>.<charset\>, f
| Default Value | charset set in the system | | Default Value | charset set in the system |
:::info :::info
On Linux, if `charset` is not set in `taos.cfg`, when `taos` is started, the charset is obtained from system locale. If obtaining charset from system locale fails, `taos` would fail to start. So on Linux system, if system locale is set properly, it's not necessary to set `charset` in `taos.cfg`. For example: On Linux, if `charset` is not set in `taos.cfg`, when `taos` is started, the charset is obtained from system locale. If obtaining charset from system locale fails, `taos` would fail to start.
So on Linux system, if system locale is set properly, it's not necessary to set `charset` in `taos.cfg`. For example:
``` ```
locale zh_CN.UTF-8 locale zh_CN.UTF-8
``` ```
On Windows system, it's not possible to obtain charset from system locale. If it's not set in configuration file `taos.cfg`, it would be default to CP936, same as set as below in `taos.cfg`. For example
```
charset CP936
```
Refer to the documentation for your operating system before changing the charset.
On a Linux system, if the charset contained in `locale` is not consistent with that set by `charset`, the later setting in the configuration file takes precedence. On a Linux system, if the charset contained in `locale` is not consistent with that set by `charset`, the later setting in the configuration file takes precedence.
```title="Effective charset is GBK" ```
locale zh_CN.UTF-8 locale zh_CN.UTF-8
charset GBK charset GBK
``` ```
```title="Effective charset is UTF-8" The charset that takes effect is GBK.
```
charset GBK charset GBK
locale zh_CN.UTF-8 locale zh_CN.UTF-8
``` ```
On Windows system, it's not possible to obtain charset from system locale. If it's not set in configuration file `taos.cfg`, it would be default to CP936, same as set as below in `taos.cfg`. For example The charset that takes effect is UTF-8.
```
charset CP936
```
::: :::
...@@ -286,819 +292,528 @@ charset CP936 ...@@ -286,819 +292,528 @@ charset CP936
### dataDir ### dataDir
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------- | | -------- | ------------------------------------------ |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | All data files are stored in this directory | | Meaning | All data files are stored in this directory |
| Default Value | /var/lib/taos | | Default Value | /var/lib/taos |
### cache
| Attribute | Description |
| ------------- | ----------------------------- |
| Applicable | Server Only |
| Meaning | The size of each memory block |
| Unit | MB |
| Default Value | 16 |
### blocks
| Attribute | Description |
| ------------- | -------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The number of memory blocks of size `cache` used by each vnode |
| Default Value | 6 |
### days
| Attribute | Description |
| ------------- | ----------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The time range of the data stored in single data file |
| Unit | day |
| Default Value | 10 |
### keep
| Attribute | Description |
| ------------- | -------------------------------------- |
| Applicable | Server Only |
| Meaning | The number of days for data to be kept |
| Unit | day |
| Default Value | 3650 |
### minRows
| Attribute | Description |
| ------------- | ------------------------------------------ |
| Applicable | Server Only |
| Meaning | minimum number of rows in single data file |
| Default Value | 100 |
### maxRows
| Attribute | Description |
| ------------- | ------------------------------------------ |
| Applicable | Server Only |
| Meaning | maximum number of rows in single data file |
| Default Value | 4096 |
### walLevel
| Attribute | Description |
| ------------- | ---------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | WAL level |
| Value Range | 0: wal disabled <br/> 1: wal enabled without fsync <br/> 2: wal enabled with fsync |
| Default Value | 1 |
### fsync
| Attribute | Description |
| ------------- | --------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The waiting time for invoking fsync when walLevel is 2 |
| Unit | millisecond |
| Value Range | 0: no waiting time, fsync is performed immediately once WAL is written; <br/> maximum value is 180000, i.e. 3 minutes |
| Default Value | 3000 |
### update
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------ |
| Applicable | Server Only |
| Meaning | If it's allowed to update existing data |
| Value Range | 0: not allowed <br/> 1: a row can only be updated as a whole <br/> 2: a part of columns can be updated |
| Default Value | 0 |
| Note | Not available from version 2.0.8.0 |
### cacheLast
| Attribute | Description |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Whether to cache the latest rows of each sub table in memory |
| Value Range | 0: not cached <br/> 1: the last row of each sub table is cached <br/> 2: the last non-null value of each column is cached <br/> 3: identical to both 1 and 2 are set |
| Default Value | 0 |
### minimalTmpDirGB ### minimalTmpDirGB
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------- | | -------- | ------------------------------------------------ |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | When the available disk space in tmpDir is below this threshold, writing to tmpDir is suspended | | Meaning | When the available disk space in tmpDir is below this threshold, writing to tmpDir is suspended |
| Unit | GB | | Unit | GB |
| Default Value | 1.0 | | Default Value | 1.0 |
### minimalDataDirGB ### minimalDataDirGB
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------ | | -------- | ------------------------------------------------ |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | hen the available disk space in dataDir is below this threshold, writing to dataDir is suspended | | Meaning | When the available disk space in dataDir is below this threshold, writing to dataDir is suspended |
| Unit | GB | | Unit | GB |
| Default Value | 2.0 | | Default Value | 2.0 |
### vnodeBak
| Attribute | Description |
| ------------- | --------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Whether to backup the corresponding vnode directory when a vnode is deleted |
| Value Range | 0: not backed up, 1: backup |
| Default Value | 1 |
## Cluster Parameters ## Cluster Parameters
### numOfMnodes ### supportVnodes
| Attribute | Description |
| ------------- | ------------------------------ |
| Applicable | Server Only |
| Meaning | The number of management nodes |
| Default Value | 3 |
### replica
| Attribute | Description |
| ------------- | -------------------------- |
| Applicable | Server Only |
| Meaning | The number of replications |
| Value Range | 1-3 |
| Default Value | 1 |
### quorum
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------ |
| Applicable | Server Only |
| Meaning | The number of required confirmations for data replication in case of multiple replications |
| Value Range | 1,2 |
| Default Value | 1 |
### role
| Attribute | Description |
| ------------- | --------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The role of the dnode |
| Value Range | 0: both mnode and vnode <br/> 1: mnode only <br/> 2: dnode only |
| Default Value | 0 |
### balance | Attribute | Description |
| -------- | --------------------------- |
| Attribute | Description | | Applicable | Server Only |
| ------------- | ------------------------ | | Meaning | Maximum number of vnodes per dnode |
| Applicable | Server Only | | Value Range | 0-4096 |
| Meaning | Automatic load balancing | | Default Value | 256 |
| Value Range | 0: disabled, 1: enabled |
| Default Value | 1 |
### balanceInterval
| Attribute | Description |
| ------------- | ----------------------------------------------- |
| Applicable | Server Only |
| Meaning | The interval for checking load balance by mnode |
| Unit | second |
| Value Range | 1-30000 |
| Default Value | 300 |
### arbitrator
| Attribute | Description |
| ------------- | -------------------------------------------------- |
| Applicable | Server Only |
| Meaning | End point of arbitrator, format is same as firstEp |
| Default Value | None |
## Time Parameters ## Time Parameters
### precision
| Attribute | Description |
| ------------- | ------------------------------------------------- |
| Applicable | Server only |
| Meaning | Time precision used for each database |
| Value Range | ms: millisecond; us: microsecond ; ns: nanosecond |
| Default Value | ms |
### rpcTimer
| Attribute | Description |
| ------------- | ------------------ |
| Applicable | Server and Client |
| Meaning | rpc retry interval |
| Unit | milliseconds |
| Value Range | 100-3000 |
| Default Value | 300 |
### rpcMaxTime
| Attribute | Description |
| ------------- | ---------------------------------- |
| Applicable | Server and Client |
| Meaning | maximum wait time for rpc response |
| Unit | second |
| Value Range | 100-7200 |
| Default Value | 600 |
### statusInterval ### statusInterval
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------------------- | | -------- | --------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | the interval of dnode reporting status to mnode | | Meaning | the interval of dnode reporting status to mnode |
| Unit | second | | Unit | second |
| Value Range | 1-10 | | Value Range | 1-10 |
| Default Value | 1 | | Default Value | 1 |
### shellActivityTimer ### shellActivityTimer
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------ | | -------- | --------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The interval for taos shell to send heartbeat to mnode | | Meaning | The interval for taos shell to send heartbeat to mnode |
| Unit | second | | Unit | second |
| Value Range | 1-120 | | Value Range | 1-120 |
| Default Value | 3 | | Default Value | 3 |
### tableMetaKeepTimer
| Attribute | Description |
| ------------- | -------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The expiration time for metadata in cache, once it's reached the client would refresh the metadata |
| Unit | second |
| Value Range | 1-8640000 |
| Default Value | 7200 |
### maxTmrCtrl
| Attribute | Description |
| ------------- | ------------------------ |
| Applicable | Server and Client |
| Meaning | Maximum number of timers |
| Unit | None |
| Value Range | 8-2048 |
| Default Value | 512 |
### offlineThreshold
| Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The expiration time for dnode online status, once it's reached before receiving status from a node, the dnode becomes offline |
| Unit | second |
| Value Range | 5-7200000 |
| Default Value | 86400\*10 (i.e. 10 days) |
## Performance Optimization Parameters ## Performance Optimization Parameters
### numOfThreadsPerCore
| Attribute | Description |
| ------------- | ------------------------------------------- |
| Applicable | Server and Client |
| Meaning | The number of consumer threads per CPU core |
| Default Value | 1.0 |
### ratioOfQueryThreads
| Attribute | Description |
| ------------- | --------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Maximum number of query threads |
| Value Range | 0: Only one query thread <br/> 1: Same as number of CPU cores <br/> 2: two times of CPU cores |
| Default Value | 1 |
| Note | This value can be a float number, 0.5 means half of the CPU cores |
### maxVgroupsPerDb
| Attribute | Description |
| ------------- | ------------------------------------ |
| Applicable | Server Only |
| Meaning | Maximum number of vnodes for each DB |
| Value Range | 0-8192 |
| Default Value | |
### maxTablesPerVnode
| Attribute | Description |
| ------------- | -------------------------------------- |
| Applicable | Server Only |
| Meaning | Maximum number of tables in each vnode |
| Default Value | 1000000 |
### minTablesPerVnode
| Attribute | Description |
| ------------- | -------------------------------------- |
| Applicable | Server Only |
| Meaning | Minimum number of tables in each vnode |
| Default Value | 1000 |
### tableIncStepPerVnode
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | When minTablesPerVnode is reached, the number of tables are allocated for a vnode each time |
| Default Value | 1000 |
### maxNumOfOrderedRes
| Attribute | Description |
| ------------- | ------------------------------------------- |
| Applicable | Server and Client |
| Meaning | Maximum number of rows ordered for a STable |
| Default Value | 100,000 |
### mnodeEqualVnodeNum
| Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The number of vnodes whose system resources consumption are considered as equal to single mnode |
| Default Value | 4 |
### numOfCommitThreads ### numOfCommitThreads
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------------- | | -------- | ---------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | Maximum of threads for committing to disk | | Meaning | Maximum of threads for committing to disk |
| Default Value | | | Default Value | |
## Compression Parameters ## Compression Parameters
### comp
| Attribute | Description |
| ------------- | ------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Whether data is compressed |
| Value Range | 0: uncompressed, 1: One phase compression, 2: Two phase compression |
| Default Value | 2 |
### tsdbMetaCompactRatio
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------- |
| Meaning | The threshold for percentage of redundant in meta file to trigger compression for meta file |
| Value Range | 0: no compression forever, [1-100]: The threshold percentage |
| Default Value | 0 |
### compressMsgSize ### compressMsgSize
| Attribute | Description | | Attribute | Description |
| ------------- | -------------------------------------------------------------------------------- | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The threshold for message size to compress the message.. | | Meaning | The threshold for message size to compress the message. | Set the value to 64330 bytes for good message compression. |
| Unit | bytes | | Unit | bytes |
| Value Range | 0: already compress; >0: compress when message exceeds it; -1: always uncompress | | Value Range | 0: already compress; >0: compress when message exceeds it; -1: always uncompress |
| Default Value | -1 | | Default Value | -1 |
### compressColData ### compressColData
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------- | | -------- | --------------------------------------------------------------------------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | The threshold for size of column data to trigger compression for the query result | | Meaning | The threshold for size of column data to trigger compression for the query result |
| Unit | bytes | | Unit | bytes |
| Value Range | 0: always compress; >0: only compress when the size of any column data exceeds the threshold; -1: always uncompress | | Value Range | 0: always compress; >0: only compress when the size of any column data exceeds the threshold; -1: always uncompress |
| Default Value | -1 | | Default Value | -1 |
| Note | available from version 2.3.0.0 |
### lossyColumns
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | The floating number types for lossy compression |
| Value Range | "": lossy compression is disabled <br/> float: only for float <br/>double: only for double <br/> float \| double: for both float and double |
| Default Value | "" , i.e. disabled |
### fPrecision
| Attribute | Description |
| ------------- | ----------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Compression precision for float type |
| Value Range | 0.1 ~ 0.00000001 |
| Default Value | 0.00000001 |
| Note | The fractional part lower than this value will be discarded |
### dPrecision
| Attribute | Description |
| ------------- | ----------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Compression precision for double type |
| Value Range | 0.1 ~ 0.0000000000000001 |
| Default Value | 0.0000000000000001 |
| Note | The fractional part lower than this value will be discarded |
## Continuous Query Parameters
### stream
| Attribute | Description |
| ------------- | ---------------------------------- |
| Applicable | Server Only |
| Meaning | Whether to enable continuous query |
| Value Range | 0: disabled <br/> 1: enabled |
| Default Value | 1 |
### minSlidingTime
| Attribute | Description |
| ------------- | -------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Minimum sliding time of time window |
| Unit | millisecond or microsecond , depending on time precision |
| Value Range | 10-1000000 |
| Default Value | 10 |
### minIntervalTime
| Attribute | Description |
| ------------- | --------------------------- |
| Applicable | Server Only |
| Meaning | Minimum size of time window |
| Unit | millisecond |
| Value Range | 1-1000000 |
| Default Value | 10 |
### maxStreamCompDelay
| Attribute | Description |
| ------------- | ------------------------------------------------ |
| Applicable | Server Only |
| Meaning | Maximum delay before starting a continuous query |
| Unit | millisecond |
| Value Range | 10-1000000000 |
| Default Value | 20000 |
### maxFirstStreamCompDelay
| Attribute | Description |
| ------------- | -------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Maximum delay time before starting a continuous query the first time |
| Unit | millisecond |
| Value Range | 10-1000000000 |
| Default Value | 10000 |
### retryStreamCompDelay
| Attribute | Description |
| ------------- | --------------------------------------------- |
| Applicable | Server Only |
| Meaning | Delay time before retrying a continuous query |
| Unit | millisecond |
| Value Range | 10-1000000000 |
| Default Value | 10 |
### streamCompDelayRatio
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------ |
| Applicable | Server Only |
| Meaning | The delay ratio, with time window size as the base, for continuous query |
| Value Range | 0.1-0.9 |
| Default Value | 0.1 |
:::info
To prevent system resource from being exhausted by multiple concurrent streams, a random delay is applied on each stream automatically. `maxFirstStreamCompDelay` is the maximum delay time before a continuous query is started the first time. `streamCompDelayRatio` is the ratio for calculating delay time, with the size of the time window as base. `maxStreamCompDelay` is the maximum delay time. The actual delay time is a random time not bigger than `maxStreamCompDelay`. If a continuous query fails, `retryStreamComDelay` is the delay time before retrying it, also not bigger than `maxStreamCompDelay`.
:::
## HTTP Parameters
### http
| Attribute | Description |
| ------------- | ------------------------------ |
| Applicable | Server Only |
| Meaning | Whether to enable http service |
| Value Range | 0: disabled, 1: enabled |
| Default Value | 1 |
### httpEnableRecordSql
| Attribute | Description |
| ------------- | ------------------------------------------------------------------------- |
| Applicable | Server Only |
| Meaning | Whether to record the SQL invocation through REST interface |
| Default Value | 0: false; 1: true |
| Note | The resulting files, i.e. httpnote.0/httpnote.1, are located under logDir |
### httpMaxThreads
| Attribute | Description |
| ------------- | -------------------------------------------- |
| Applicable | Server Only |
| Meaning | The number of threads for RESTFul interface. |
| Default Value | 2 |
### restfulRowLimit
| Attribute | Description |
| ------------- | ------------------------------------------------------------ |
| Applicable | Server Only |
| Meaning | Maximum number of rows returned each time by REST interface. |
| Default Value | 10240 |
| Note | Maximum value is 10,000,000 |
### httpDBNameMandatory
| Attribute | Description |
| ------------- | ---------------------------------------- |
| Applicable | Server Only |
| Meaning | Whether database name is required in URL |
| Value Range | 0:not required, 1: required |
| Default Value | 0 |
| Note | From version 2.3.0.0 |
## Log Parameters ## Log Parameters
### logDir ### logDir
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------- | | -------- | -------------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The directory for writing log files | | Meaning | The directory for writing log files |
| Default Value | /var/log/taos | | Default Value | /var/log/taos |
### minimalLogDirGB ### minimalLogDirGB
| Attribute | Description | | Attribute | Description |
| ------------- | -------------------------------------------------------------------------------------------------- | | -------- | -------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | When the available disk space in logDir is below this threshold, writing to log files is suspended | | Meaning | When the available disk space in logDir is below this threshold, writing to log files is suspended |
| Unit | GB | | Unit | GB |
| Default Value | 1.0 | | Default Value | 1.0 |
### numOfLogLines ### numOfLogLines
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------ | | -------- | ---------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Maximum number of lines in single log file | | Meaning | Maximum number of lines in single log file |
| Default Value | 10,000,000 | | Default Value | 10000000 |
### asyncLog ### asyncLog
| Attribute | Description | | Attribute | Description |
| ------------- | ---------------------------- | | -------- | -------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The mode of writing log file | | Meaning | The mode of writing log file |
| Value Range | 0: sync way; 1: async way | | Value Range | 0: sync way; 1: async way |
| Default Value | 1 | | Default Value | 1 |
### logKeepDays ### logKeepDays
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | -------- | ----------------------------------------------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | The number of days for log files to be kept | | Meaning | The number of days for log files to be kept |
| Unit | day | | Unit | day |
| Default Value | 0 | | Default Value | 0 |
| Note | When it's bigger than 0, the log file would be renamed to "taosdlog.xxx" in which "xxx" is the timestamp when the file is changed last time | | Note | When it's bigger than 0, the log file would be renamed to "taosdlog.xxx" in which "xxx" is the timestamp when the file is changed last time |
### debugFlag ### debugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | --------------------------------------------------------- | | -------- | ------------------------------------------------------------------------------------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Log level | | Meaning | Log level |
| Value Range | 131: INFO/WARNING/ERROR; 135: plus DEBUG; 143: plus TRACE | | Value Range | 131: INFO/WARNING/ERROR; 135: plus DEBUG; 143: plus TRACE |
| Default Value | 131 or 135, depending on the module | | Default Value | 131 or 135, depending on the module |
### mDebugFlag ### tmrDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------ | | -------- | -------------------- |
| Applicable | Server Only | | Applicable | Server and Client |
| Meaning | Log level of mnode | | Meaning | Log level of timer module |
| Value Range | same as debugFlag | | Value Range | same as debugFlag |
| Default Value | 135 | | Default Value | |
### dDebugFlag ### uDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------ | | -------- | ---------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Log level of dnode | | Meaning | Log level of common module |
| Value Range | same as debugFlag | | Value Range | same as debugFlag |
| Default Value | 135 | | Default Value | |
### sDebugFlag
| Attribute | Description |
| ------------- | ------------------------ |
| Applicable | Server and Client |
| Meaning | Log level of sync module |
| Value Range | same as debugFlag |
| Default Value | 135 |
### wDebugFlag
| Attribute | Description |
| ------------- | ----------------------- |
| Applicable | Server and Client |
| Meaning | Log level of WAL module |
| Value Range | same as debugFlag |
| Default Value | 135 |
### sdbDebugFlag
| Attribute | Description |
| ------------- | ---------------------- |
| Applicable | Server and Client |
| Meaning | logLevel of sdb module |
| Value Range | same as debugFlag |
| Default Value | 135 |
### rpcDebugFlag ### rpcDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------- | | -------- | -------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Log level of rpc module | | Meaning | Log level of rpc module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | |
### tmrDebugFlag ### jniDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------- | | -------- | ------------------ |
| Applicable | Client Only |
| Meaning | Log level of jni module |
| Value Range | same as debugFlag |
| Default Value | |
### qDebugFlag
| Attribute | Description |
| -------- | -------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Log level of timer module | | Meaning | Log level of query module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | |
### cDebugFlag ### cDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------- | | -------- | --------------------- |
| Applicable | Client Only | | Applicable | Client Only |
| Meaning | Log level of Client | | Meaning | Log level of Client |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | |
### jniDebugFlag
| Attribute | Description |
| ------------- | ----------------------- |
| Applicable | Client Only |
| Meaning | Log level of jni module |
| Value Range | Same as debugFlag |
| Default Value | |
### odbcDebugFlag ### dDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------ | | -------- | -------------------- |
| Applicable | Client Only | | Applicable | Server Only |
| Meaning | Log level of odbc module | | Meaning | Log level of dnode |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | 135 |
### uDebugFlag ### vDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | -------------------------- | | -------- | -------------------- |
| Applicable | Server and Client | | Applicable | Server Only |
| Meaning | Log level of common module | | Meaning | Log level of vnode |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | | Default Value | |
### mqttDebugFlag ### mDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------ | | -------- | -------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | Log level of mqtt module | | Meaning | Log level of mnode module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | 135 |
### monitorDebugFlag ### wDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------ | | -------- | ------------------ |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | Log level of monitoring module | | Meaning | Log level of WAL module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | 135 |
### qDebugFlag ### sDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------- | | -------- | -------------------- |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Log level of query module | | Meaning | Log level of sync module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | 135 |
### vDebugFlag
| Attribute | Description |
| ------------- | ------------------ |
| Applicable | Server and Client |
| Meaning | Log level of vnode |
| Value Range | Same as debugFlag |
| Default Value | |
### tsdbDebugFlag ### tsdbDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------ | | -------- | ------------------- |
| Applicable | Server Only | | Applicable | Server Only |
| Meaning | Log level of TSDB module | | Meaning | Log level of TSDB module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | |
### cqDebugFlag ### tqDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------ | | -------- | ----------------- |
| Applicable | Server and Client | | Applicable | Server only |
| Meaning | Log level of continuous query module | | Meaning | Log level of TQ module |
| Value Range | Same as debugFlag | | Value Range | same as debugFlag |
| Default Value | | | Default Value | |
## Client Only ### fsDebugFlag
### maxSQLLength | Attribute | Description |
| -------- | ----------------- |
| Applicable | Server only |
| Meaning | Log level of FS module |
| Value Range | same as debugFlag |
| Default Value | |
### udfDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | -------------------------------------- | | -------- | ------------------ |
| Applicable | Client Only | | Applicable | Server Only |
| Meaning | Maximum length of single SQL statement | | Meaning | Log level of UDF module |
| Unit | bytes | | Value Range | same as debugFlag |
| Value Range | 65480-1048576 | | Default Value | |
| Default Value | 1048576 |
### tscEnableRecordSql ### smaDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | -------- | ------------------ |
| Meaning | Whether to record SQL statements in file | | Applicable | Server Only |
| Value Range | 0: false, 1: true | | Meaning | Log level of SMA module |
| Default Value | 0 | | Value Range | same as debugFlag |
| Note | The generated files are named as "tscnote-xxxx.0/tscnote-xxx.1" in which "xxxx" is the pid of the client, and located at same place as client log | | Default Value | |
### maxBinaryDisplayWidth ### idxDebugFlag
| Attribute | Description | | Attribute | Description |
| ------------- | --------------------------------------------------------------------------------------------------- | | -------- | -------------------- |
| Meaning | Maximum display width of binary and nchar in taos shell. Anything beyond this limit would be hidden | | Applicable | Server Only |
| Value Range | 5 - | | Meaning | Log level of index module |
| Default Value | 30 | | Value Range | same as debugFlag |
| Default Value | |
:::info ### tdbDebugFlag
If the length of value exceeds `maxBinaryDisplayWidth`, then the actual display width is max(column name, maxBinaryDisplayLength); otherwise the actual display width is max(length of column name, length of column value). This parameter can also be changed dynamically using `set max_binary_display_width <nn\>` in TDengine CLI `taos`.
::: | Attribute | Description |
| -------- | ------------------ |
| Applicable | Server Only |
| Meaning | Log level of TDB module |
| Value Range | same as debugFlag |
| Default Value | |
### maxWildCardsLength ## Schemaless Parameters
| Attribute | Description | ### smlChildTableName
| ------------- | ----------------------------------------------------- |
| Meaning | The maximum length for wildcard string used with LIKE |
| Unit | bytes |
| Value Range | 0-16384 |
| Default Value | 100 |
| Note | From version 2.1.6.1 |
### clientMerge | Attribute | Description |
| -------- | ------------------------- |
| Applicable | Client only |
| Meaning | Custom subtable name for schemaless writes |
| Type | String |
| Default Value | None |
| Attribute | Description | ### smlTagName
| ------------- | --------------------------------------------------- |
| Meaning | Whether to filter out duplicate data on client side |
| Value Range | 0: false; 1: true |
| Default Value | 0 |
| Note | From version 2.3.0.0 |
### maxRegexStringLen | Attribute | Description |
| -------- | ------------------------------------ |
| Applicable | Client only |
| Meaning | Default tag for schemaless writes without tag value specified |
| Type | String |
| Default Value | _tag_null |
### smlDataFormat
| Attribute | Description | | Attribute | Description |
| ------------- | ------------------------------------ | | -------- | ----------------------------- |
| Meaning | Maximum length of regular expression | | Applicable | Client only |
| Value Range | [128, 16384] | | Meaning | Whether schemaless columns are consistently ordered |
| Default Value | 128 | | Value Range | 0: not consistent; 1: consistent. |
| Note | From version 2.3.0.0 | | Default | 1 |
## Other Parameters ## Other Parameters
### enableCoreFile ### enableCoreFile
| Attribute | Description | | Attribute | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Applicable | Server and Client | | Applicable | Server and Client |
| Meaning | Whether to generate core file when server crashes | | Meaning | Whether to generate core file when server crashes |
| Value Range | 0: false, 1: true | | Value Range | 0: false, 1: true |
| Default Value | 1 | | Default Value | 1 |
| Note | The core file is generated under root directory `systemctl start taosd` is used to start, or under the working directory if `taosd` is started directly on Linux Shell. | | Note | The core file is generated under root directory `systemctl start taosd` is used to start, or under the working directory if `taosd` is started directly on Linux Shell. |
### udf
| Attribute | Description |
| -------- | ------------------ |
| Applicable | Server Only |
| Meaning | Whether the UDF service is enabled |
| Value Range | 0: disable UDF; 1: enabled UDF |
| Default Value | 1 |
## Parameter Comparison of TDengine 2.x and 3.0
| # | **Parameter** | **In 2.x** | **In 3.0** |
| --- | :-----------------: | --------------- | --------------- |
| 1 | firstEp | Yes | Yes |
| 2 | secondEp | Yes | Yes |
| 3 | fqdn | Yes | Yes |
| 4 | serverPort | Yes | Yes |
| 5 | maxShellConns | Yes | Yes |
| 6 | monitor | Yes | Yes |
| 7 | monitorFqdn | No | Yes |
| 8 | monitorPort | No | Yes |
| 9 | monitorInterval | Yes | Yes |
| 10 | monitorMaxLogs | No | Yes |
| 11 | monitorComp | No | Yes |
| 12 | telemetryReporting | Yes | Yes |
| 13 | telemetryInterval | No | Yes |
| 14 | telemetryServer | No | Yes |
| 15 | telemetryPort | No | Yes |
| 16 | queryPolicy | No | Yes |
| 17 | querySmaOptimize | No | Yes |
| 18 | queryBufferSize | Yes | Yes |
| 19 | maxNumOfDistinctRes | Yes | Yes |
| 20 | minSlidingTime | Yes | Yes |
| 21 | minIntervalTime | Yes | Yes |
| 22 | countAlwaysReturnValue | Yes | Yes |
| 23 | dataDir | Yes | Yes |
| 24 | minimalDataDirGB | Yes | Yes |
| 25 | supportVnodes | No | Yes |
| 26 | tempDir | Yes | Yes |
| 27 | minimalTmpDirGB | Yes | Yes |
| 28 | compressMsgSize | Yes | Yes |
| 29 | compressColData | Yes | Yes |
| 30 | smlChildTableName | Yes | Yes |
| 31 | smlTagName | Yes | Yes |
| 32 | smlDataFormat | No | Yes |
| 33 | statusInterval | Yes | Yes |
| 34 | shellActivityTimer | Yes | Yes |
| 35 | transPullupInterval | No | Yes |
| 36 | mqRebalanceInterval | No | Yes |
| 37 | ttlUnit | No | Yes |
| 38 | ttlPushInterval | No | Yes |
| 39 | numOfTaskQueueThreads | No | Yes |
| 40 | numOfRpcThreads | No | Yes |
| 41 | numOfCommitThreads | Yes | Yes |
| 42 | numOfMnodeReadThreads | No | Yes |
| 43 | numOfVnodeQueryThreads | No | Yes |
| 44 | numOfVnodeStreamThreads | No | Yes |
| 45 | numOfVnodeFetchThreads | No | Yes |
| 46 | numOfVnodeWriteThreads | No | Yes |
| 47 | numOfVnodeSyncThreads | No | Yes |
| 48 | numOfQnodeQueryThreads | No | Yes |
| 49 | numOfQnodeFetchThreads | No | Yes |
| 50 | numOfSnodeSharedThreads | No | Yes |
| 51 | numOfSnodeUniqueThreads | No | Yes |
| 52 | rpcQueueMemoryAllowed | No | Yes |
| 53 | logDir | Yes | Yes |
| 54 | minimalLogDirGB | Yes | Yes |
| 55 | numOfLogLines | Yes | Yes |
| 56 | asyncLog | Yes | Yes |
| 57 | logKeepDays | Yes | Yes |
| 58 | debugFlag | Yes | Yes |
| 59 | tmrDebugFlag | Yes | Yes |
| 60 | uDebugFlag | Yes | Yes |
| 61 | rpcDebugFlag | Yes | Yes |
| 62 | jniDebugFlag | Yes | Yes |
| 63 | qDebugFlag | Yes | Yes |
| 64 | cDebugFlag | Yes | Yes |
| 65 | dDebugFlag | Yes | Yes |
| 66 | vDebugFlag | Yes | Yes |
| 67 | mDebugFlag | Yes | Yes |
| 68 | wDebugFlag | Yes | Yes |
| 69 | sDebugFlag | Yes | Yes |
| 70 | tsdbDebugFlag | Yes | Yes |
| 71 | tqDebugFlag | No | Yes |
| 72 | fsDebugFlag | Yes | Yes |
| 73 | udfDebugFlag | No | Yes |
| 74 | smaDebugFlag | No | Yes |
| 75 | idxDebugFlag | No | Yes |
| 76 | tdbDebugFlag | No | Yes |
| 77 | metaDebugFlag | No | Yes |
| 78 | timezone | Yes | Yes |
| 79 | locale | Yes | Yes |
| 80 | charset | Yes | Yes |
| 81 | udf | Yes | Yes |
| 82 | enableCoreFile | Yes | Yes |
| 83 | arbitrator | Yes | No |
| 84 | numOfThreadsPerCore | Yes | No |
| 85 | numOfMnodes | Yes | No |
| 86 | vnodeBak | Yes | No |
| 87 | balance | Yes | No |
| 88 | balanceInterval | Yes | No |
| 89 | offlineThreshold | Yes | No |
| 90 | role | Yes | No |
| 91 | dnodeNopLoop | Yes | No |
| 92 | keepTimeOffset | Yes | No |
| 93 | rpcTimer | Yes | No |
| 94 | rpcMaxTime | Yes | No |
| 95 | rpcForceTcp | Yes | No |
| 96 | tcpConnTimeout | Yes | No |
| 97 | syncCheckInterval | Yes | No |
| 98 | maxTmrCtrl | Yes | No |
| 99 | monitorReplica | Yes | No |
| 100 | smlTagNullName | Yes | No |
| 101 | keepColumnName | Yes | No |
| 102 | ratioOfQueryCores | Yes | No |
| 103 | maxStreamCompDelay | Yes | No |
| 104 | maxFirstStreamCompDelay | Yes | No |
| 105 | retryStreamCompDelay | Yes | No |
| 106 | streamCompDelayRatio | Yes | No |
| 107 | maxVgroupsPerDb | Yes | No |
| 108 | maxTablesPerVnode | Yes | No |
| 109 | minTablesPerVnode | Yes | No |
| 110 | tableIncStepPerVnode | Yes | No |
| 111 | cache | Yes | No |
| 112 | blocks | Yes | No |
| 113 | days | Yes | No |
| 114 | keep | Yes | No |
| 115 | minRows | Yes | No |
| 116 | maxRows | Yes | No |
| 117 | quorum | Yes | No |
| 118 | comp | Yes | No |
| 119 | walLevel | Yes | No |
| 120 | fsync | Yes | No |
| 121 | replica | Yes | No |
| 122 | partitions | Yes | No |
| 123 | quorum | Yes | No |
| 124 | update | Yes | No |
| 125 | cachelast | Yes | No |
| 126 | maxSQLLength | Yes | No |
| 127 | maxWildCardsLength | Yes | No |
| 128 | maxRegexStringLen | Yes | No |
| 129 | maxNumOfOrderedRes | Yes | No |
| 130 | maxConnections | Yes | No |
| 131 | mnodeEqualVnodeNum | Yes | No |
| 132 | http | Yes | No |
| 133 | httpEnableRecordSql | Yes | No |
| 134 | httpMaxThreads | Yes | No |
| 135 | restfulRowLimit | Yes | No |
| 136 | httpDbNameMandatory | Yes | No |
| 137 | httpKeepAlive | Yes | No |
| 138 | enableRecordSql | Yes | No |
| 139 | maxBinaryDisplayWidth | Yes | No |
| 140 | stream | Yes | No |
| 141 | retrieveBlockingModel | Yes | No |
| 142 | tsdbMetaCompactRatio | Yes | No |
| 143 | defaultJSONStrType | Yes | No |
| 144 | walFlushSize | Yes | No |
| 145 | keepTimeOffset | Yes | No |
| 146 | flowctrl | Yes | No |
| 147 | slaveQuery | Yes | No |
| 148 | adjustMaster | Yes | No |
| 149 | topicBinaryLen | Yes | No |
| 150 | telegrafUseFieldNum | Yes | No |
| 151 | deadLockKillQuery | Yes | No |
| 152 | clientMerge | Yes | No |
| 153 | sdbDebugFlag | Yes | No |
| 154 | odbcDebugFlag | Yes | No |
| 155 | httpDebugFlag | Yes | No |
| 156 | monDebugFlag | Yes | No |
| 157 | cqDebugFlag | Yes | No |
| 158 | shortcutFlag | Yes | No |
| 159 | probeSeconds | Yes | No |
| 160 | probeKillSeconds | Yes | No |
| 161 | probeInterval | Yes | No |
| 162 | lossyColumns | Yes | No |
| 163 | fPrecision | Yes | No |
| 164 | dPrecision | Yes | No |
| 165 | maxRange | Yes | No |
| 166 | range | Yes | No |
docs/en/14-reference/13-schemaless/13-schemaless.md
浏览文件 @ a76bbfe0
...@@ -3,7 +3,8 @@ title: Schemaless Writing ...@@ -3,7 +3,8 @@ title: Schemaless Writing
description: "The Schemaless write method eliminates the need to create super tables/sub tables in advance and automatically creates the storage structure corresponding to the data, as it is written to the interface." description: "The Schemaless write method eliminates the need to create super tables/sub tables in advance and automatically creates the storage structure corresponding to the data, as it is written to the interface."
--- ---
In IoT applications, data is collected for many purposes such as intelligent control, business analysis, device monitoring and so on. Due to changes in business or functional requirements or changes in device hardware, the application logic and even the data collected may change. To provide the flexibility needed in such cases and in a rapidly changing IoT landscape, TDengine provides a series of interfaces for the schemaless writing method. These interfaces eliminate the need to create super tables and subtables in advance by automatically creating the storage structure corresponding to the data as the data is written to the interface. When necessary, schemaless writing will automatically add the required columns to ensure that the data written by the user is stored correctly. In IoT applications, data is collected for many purposes such as intelligent control, business analysis, device monitoring and so on. Due to changes in business or functional requirements or changes in device hardware, the application logic and even the data collected may change. Schemaless writing automatically creates storage structures for your data as it is being written to TDengine, so that you do not need to create supertables in advance. When necessary, schemaless writing
will automatically add the required columns to ensure that the data written by the user is stored correctly.
The schemaless writing method creates super tables and their corresponding subtables. These are completely indistinguishable from the super tables and subtables created directly via SQL. You can write data directly to them via SQL statements. Note that the names of tables created by schemaless writing are based on fixed mapping rules for tag values, so they are not explicitly ideographic and they lack readability. The schemaless writing method creates super tables and their corresponding subtables. These are completely indistinguishable from the super tables and subtables created directly via SQL. You can write data directly to them via SQL statements. Note that the names of tables created by schemaless writing are based on fixed mapping rules for tag values, so they are not explicitly ideographic and they lack readability.
...@@ -19,12 +20,12 @@ With the following formatting conventions, schemaless writing uses a single stri ...@@ -19,12 +20,12 @@ With the following formatting conventions, schemaless writing uses a single stri
measurement,tag_set field_set timestamp measurement,tag_set field_set timestamp
``` ```
where : where:
- measurement will be used as the data table name. It will be separated from tag_set by a comma. - measurement will be used as the data table name. It will be separated from tag_set by a comma.
- tag_set will be used as tag data in the format `<tag_key>=<tag_value>,<tag_key>=<tag_value>`, i.e. multiple tags' data can be separated by a comma. It is separated from field_set by space. - `tag_set` will be used as tags, with format like `<tag_key>=<tag_value>,<tag_key>=<tag_value>` Enter a space between `tag_set` and `field_set`.
- field_set will be used as normal column data in the format of `<field_key>=<field_value>,<field_key>=<field_value>`, again using a comma to separate multiple normal columns of data. It is separated from the timestamp by a space. - `field_set`will be used as data columns, with format like `<field_key>=<field_value>,<field_key>=<field_value>` Enter a space between `field_set` and `timestamp`.
- The timestamp is the primary key corresponding to the data in this row. - `timestamp` is the primary key timestamp corresponding to this row of data
All data in tag_set is automatically converted to the NCHAR data type and does not require double quotes ("). All data in tag_set is automatically converted to the NCHAR data type and does not require double quotes (").
...@@ -37,16 +38,18 @@ In the schemaless writing data line protocol, each data item in the field_set ne ...@@ -37,16 +38,18 @@ In the schemaless writing data line protocol, each data item in the field_set ne
| **Serial number** | **Postfix** | **Mapping type** | **Size (bytes)** | | **Serial number** | **Postfix** | **Mapping type** | **Size (bytes)** |
| -------- | -------- | ------------ | -------------- | | -------- | -------- | ------------ | -------------- |
| 1 | none or f64 | double | 8 | | 1 | None or f64 | double | 8 |
| 2 | f32 | float | 4 | | 2 | f32 | float | 4 |
| 3 | i8/u8 | TinyInt/UTinyInt | 1 | | 3 | i8/u8 | TinyInt/UTinyInt | 1 |
| 4 | i16/u16 | SmallInt/USmallInt | 2 | | 4 | i16/u16 | SmallInt/USmallInt | 2 |
| 5 | i32/u32 | Int/UInt | 4 | | 5 | i32/u32 | Int/UInt | 4 |
| 6 | i64/i/u64/u | Bigint/Bigint/UBigint/UBigint | 8 | | 6 | i64/i/u64/u | BigInt/BigInt/UBigInt/UBigInt | 8 |
- `t`, `T`, `true`, `True`, `TRUE`, `f`, `F`, `false`, and `False` will be handled directly as BOOL types. - `t`, `T`, `true`, `True`, `TRUE`, `f`, `F`, `false`, and `False` will be handled directly as BOOL types.
For example, the following data rows indicate that the t1 label is "3" (NCHAR), the t2 label is "4" (NCHAR), and the t3 label is "t3" to the super table named `st` labeled "t3" (NCHAR), write c1 column as 3 (BIGINT), c2 column as false (BOOL), c3 column is "passit" (BINARY), c4 column is 4 (DOUBLE), and the primary key timestamp is 1626006833639000000 in one row. For example, the following data rows indicate that the t1 label is "3" (NCHAR), the t2 label is "4" (NCHAR), and the t3 label
is "t3" to the super table named `st` labeled "t3" (NCHAR), write c1 column as 3 (BIGINT), c2 column as false (BOOL), c3 column
is "passit" (BINARY), c4 column is 4 (DOUBLE), and the primary key timestamp is 1626006833639000000 in one row.
```json ```json
st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4f64 1626006833639000000 st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4f64 1626006833639000000
...@@ -65,18 +68,22 @@ Schemaless writes process row data according to the following principles. ...@@ -65,18 +68,22 @@ Schemaless writes process row data according to the following principles.
``` ```
Note that tag_key1, tag_key2 are not the original order of the tags entered by the user but the result of using the tag names in ascending order of the strings. Therefore, tag_key1 is not the first tag entered in the line protocol. Note that tag_key1, tag_key2 are not the original order of the tags entered by the user but the result of using the tag names in ascending order of the strings. Therefore, tag_key1 is not the first tag entered in the line protocol.
The string's MD5 hash value "md5_val" is calculated after the ranking is completed. The calculation result is then combined with the string to generate the table name: "t_md5_val". "t*" is a fixed prefix that every table generated by this mapping relationship has. The string's MD5 hash value "md5_val" is calculated after the ranking is completed. The calculation result is then combined with the string to generate the table name: "t_md5_val". "t_" is a fixed prefix that every table generated by this mapping relationship has.
You can configure smlChildTableName to specify table names, for example, `smlChildTableName=tname`. You can insert `st,tname=cpul,t1=4 c1=3 1626006833639000000` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
2. If the super table obtained by parsing the line protocol does not exist, this super table is created. 2. If the super table obtained by parsing the line protocol does not exist, this super table is created.
If the subtable obtained by the parse line protocol does not exist, Schemaless creates the sub-table according to the subtable name determined in steps 1 or 2. 3. If the subtable obtained by the parse line protocol does not exist, Schemaless creates the sub-table according to the subtable name determined in steps 1 or 2.
4. If the specified tag or regular column in the data row does not exist, the corresponding tag or regular column is added to the super table (only incremental). 4. If the specified tag or regular column in the data row does not exist, the corresponding tag or regular column is added to the super table (only incremental).
5. If there are some tag columns or regular columns in the super table that are not specified to take values in a data row, then the values of these columns are set to NULL. 5. If there are some tag columns or regular columns in the super table that are not specified to take values in a data row, then the values of these columns are set to
NULL.
6. For BINARY or NCHAR columns, if the length of the value provided in a data row exceeds the column type limit, the maximum length of characters allowed to be stored in the column is automatically increased (only incremented and not decremented) to ensure complete preservation of the data. 6. For BINARY or NCHAR columns, if the length of the value provided in a data row exceeds the column type limit, the maximum length of characters allowed to be stored in the column is automatically increased (only incremented and not decremented) to ensure complete preservation of the data.
7. Errors encountered throughout the processing will interrupt the writing process and return an error code. 7. Errors encountered throughout the processing will interrupt the writing process and return an error code.
8. In order to improve the efficiency of writing, it is assumed by default that the order of the fields in the same Super is the same (the first data contains all fields, and the following data is in this order). If the order is different, the parameter smlDataFormat needs to be configured to be false. Otherwise, the data is written in the same order, and the data in the library will be abnormal. 8. It is assumed that the order of field_set in a supertable is consistent, meaning that the first record contains all fields and subsequent records store fields in the same order. If the order is not consistent, set smlDataFormat to false. Otherwise, data will be written out of order and a database error will occur.
:::tip :::tip
All processing logic of schemaless will still follow TDengine's underlying restrictions on data structures, such as the total length of each row of data cannot exceed 16k bytes. See [TAOS SQL Boundary Limits](/taos-sql/limit) for specific constraints in this area. All processing logic of schemaless will still follow TDengine's underlying restrictions on data structures, such as the total length of each row of data cannot exceed
16KB. See [TAOS SQL Boundary Limits](/taos-sql/limit) for specific constraints in this area.
::: :::
## Time resolution recognition ## Time resolution recognition
...@@ -85,75 +92,74 @@ Three specified modes are supported in the schemaless writing process, as follow ...@@ -85,75 +92,74 @@ Three specified modes are supported in the schemaless writing process, as follow
| **Serial** | **Value** | **Description** | | **Serial** | **Value** | **Description** |
| -------- | ------------------- | ------------------------------- | | -------- | ------------------- | ------------------------------- |
| 1 | SML_LINE_PROTOCOL | InfluxDB Line Protocol | | 1 | SML_LINE_PROTOCOL | InfluxDB Line Protocol |
| 2 | SML_TELNET_PROTOCOL | OpenTSDB Text Line Protocol | | 2 | SML_TELNET_PROTOCOL | OpenTSDB file protocol |
| 3 | SML_JSON_PROTOCOL | JSON protocol format | | 3 | SML_JSON_PROTOCOL | OpenTSDB JSON protocol |
In the SML_LINE_PROTOCOL parsing mode, the user is required to specify the time resolution of the input timestamp. The available time resolutions are shown in the following table. In InfluxDB line protocol mode, you must specify the precision of the input timestamp. Valid precisions are described in the following table.
| **Serial Number** | **Time Resolution Definition** | **Meaning** | | **No.** | **Precision** | **Description** |
| -------- | --------------------------------- | -------------- | | -------- | --------------------------------- | -------------- |
| 1 | TSDB_SML_TIMESTAMP_NOT_CONFIGURED | Not defined (invalid) | | 1 | TSDB_SML_TIMESTAMP_NOT_CONFIGURED | Not defined (invalid) |
| 2 | TSDB_SML_TIMESTAMP_HOURS | hour | | 2 | TSDB_SML_TIMESTAMP_HOURS | Hours |
| 3 | TSDB_SML_TIMESTAMP_MINUTES | MINUTES | 3 | TSDB_SML_TIMESTAMP_MINUTES | Minutes |
| 4 | TSDB_SML_TIMESTAMP_SECONDS | SECONDS | 4 | TSDB_SML_TIMESTAMP_SECONDS | Seconds |
| 5 | TSDB_SML_TIMESTAMP_MILLI_SECONDS | milliseconds | 5 | TSDB_SML_TIMESTAMP_MILLI_SECONDS | Milliseconds |
| 6 | TSDB_SML_TIMESTAMP_MICRO_SECONDS | microseconds | 6 | TSDB_SML_TIMESTAMP_MICRO_SECONDS | Microseconds |
| 7 | TSDB_SML_TIMESTAMP_NANO_SECONDS | nanoseconds | | 7 | TSDB_SML_TIMESTAMP_NANO_SECONDS | Nanoseconds |
In SML_TELNET_PROTOCOL and SML_JSON_PROTOCOL modes, the time precision is determined based on the length of the timestamp (in the same way as the OpenTSDB standard operation), and the user-specified time resolution is ignored at this point.
## Data schema mapping rules In OpenTSDB file and JSON protocol modes, the precision of the timestamp is determined from its length in the standard OpenTSDB manner. User input is ignored.
This section describes how data for line protocols are mapped to data with a schema. The data measurement in each line protocol is mapped as follows: ## Data Model Mapping
- The tag name in tag_set is the name of the tag in the data schema
- The name in field_set is the column's name.
The following data is used as an example to illustrate the mapping rules. This section describes how data in line protocol is mapped to a schema. The data measurement in each line is mapped to a
supertable name. The tag name in tag_set is the tag name in the schema, and the name in field_set is the column name in the schema. The following example shows how data is mapped:
```json ```json
st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4f64 1626006833639000000 st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4f64 1626006833639000000
``` ```
The row data mapping generates a super table: `st`, which contains three labels of type NCHAR: t1, t2, t3. Five data columns are ts (timestamp), c1 (bigint), c3 (binary), c2 (bool), c4 (bigint). The mapping becomes the following SQL statement. This row is mapped to a supertable: `st` contains three NCHAR tags: t1, t2, and t3. Five columns are created: ts (timestamp), c1 (bigint), c3 (binary), c2 (bool), and c4 (bigint). The following SQL statement is generated:
```json ```json
create stable st (_ts timestamp, c1 bigint, c2 bool, c3 binary(6), c4 bigint) tags(t1 nchar(1), t2 nchar(1), t3 nchar(2)) create stable st (_ts timestamp, c1 bigint, c2 bool, c3 binary(6), c4 bigint) tags(t1 nchar(1), t2 nchar(1), t3 nchar(2))
``` ```
## Data schema change handling ## Processing Schema Changes
This section describes the impact on the data schema for different line protocol data writing cases. This section describes the impact on the schema caused by different data being written.
When writing to an explicitly identified field type using the line protocol, subsequent changes to the field's type definition will result in an explicit data schema error, i.e., will trigger a write API report error. As shown below, the If you use line protocol to write to a specific tag field and then later change the field type, a schema error will ocur. This triggers an error on the write API. This is shown as follows:
```json ```json
st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4 1626006833639000000 st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4 1626006833639000000
st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4i 1626006833640000000 st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4i 1626006833640000000
``` ```
The data type mapping in the first row defines column c4 as DOUBLE, but the data in the second row is declared as BIGINT by the numeric suffix, which triggers a parsing error with schemaless writing. The first row defines c4 as a double. However, in the second row, the suffix indicates that the value of c4 is a bigint. This causes schemaless writing to throw an error.
If the line protocol before the column declares the data column as BINARY, the subsequent one requires a longer binary length, which triggers a super table schema change. An error also occurs if data input into a binary column exceeds the defined length of the column.
```json ```json
st,t1=3,t2=4,t3=t3 c1=3i64,c5="pass" 1626006833639000000 st,t1=3,t2=4,t3=t3 c1=3i64,c5="pass" 1626006833639000000
st,t1=3,t2=4,t3=t3 c1=3i64,c5="passit" 1626006833640000000 st,t1=3,t2=4,t3=t3 c1=3i64,c5="passit" 1626006833640000000
``` ```
The first line of the line protocol parsing will declare column c5 is a BINARY(4) field. The second line data write will parse column c5 as a BINARY column. But in the second line, c5's width is 6 so you need to increase the width of the BINARY field to be able to accommodate the new string. The first row defines c5 as a binary(4). but the second row writes 6 bytes to it. This means that the length of the binary column must be expanded to contain the data.
```json ```json
st,t1=3,t2=4,t3=t3 c1=3i64 1626006833639000000 st,t1=3,t2=4,t3=t3 c1=3i64 1626006833639000000
st,t1=3,t2=4,t3=t3 c1=3i64,c6="passit" 1626006833640000000 st,t1=3,t2=4,t3=t3 c1=3i64,c6="passit" 1626006833640000000
``` ```
The second line of data has an additional column c6 of type BINARY(6) compared to the first row. Then a column c6 of type BINARY(6) is automatically added at this point. The preceding data includes a new entry, c6, with type binary(6). When this occurs, a new column c6 with type binary(6) is added automatically.
## Write integrity ## Write Integrity
TDengine provides idempotency guarantees for data writing, i.e., you can repeatedly call the API to write data with errors. However, it does not give atomicity guarantees for writing multiple rows of data. During the process of writing numerous rows of data in one batch, some data will be written successfully, and some data will fail. TDengine guarantees the idempotency of data writes. This means that you can repeatedly call the API to perform write operations with bad data. However, TDengine does not guarantee the atomicity of multi-row writes. In a multi-row write, some data may be written successfully and other data unsuccessfully.
## Error code ##: Error Codes
If it is an error in the data itself during the schemaless writing process, the application will get `TSDB_CODE_TSC_LINE_SYNTAX_ERROR` error message, which indicates that the error occurred in writing. The other error codes are consistent with the TDengine and can be obtained via the `taos_errstr()` to get the specific cause of the error. The TSDB_CODE_TSC_LINE_SYNTAX_ERROR indicates an error in the schemaless writing component.
This error occurs when writing text. For other errors, schemaless writing uses the standard TDengine error codes
found in taos_errstr.
docs/en/20-third-party/01-grafana.mdx
浏览文件 @ a76bbfe0
...@@ -6,9 +6,7 @@ title: Grafana ...@@ -6,9 +6,7 @@ title: Grafana
import Tabs from "@theme/Tabs"; import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem"; import TabItem from "@theme/TabItem";
TDengine can be quickly integrated with the open-source data visualization system [Grafana](https://www.grafana.com/) to build a data monitoring and alerting system. The whole process does not require any code development. And you can visualize the contents of the data tables in TDengine on a dashboard. TDengine can be quickly integrated with the open-source data visualization system [Grafana](https://www.grafana.com/) to build a data monitoring and alerting system. The whole process does not require any code development. And you can visualize the contents of the data tables in TDengine on a dashboard. You can learn more about using the TDengine plugin on [GitHub](https://github.com/taosdata/grafanaplugin/blob/master/README.md).
You can learn more about using the TDengine plugin on [GitHub](https://github.com/taosdata/grafanaplugin/blob/master/README.md).
## Prerequisites ## Prerequisites
...@@ -65,7 +63,6 @@ Restart Grafana service and open Grafana in web-browser, usually <http://localho ...@@ -65,7 +63,6 @@ Restart Grafana service and open Grafana in web-browser, usually <http://localho
Save the script and type `./install.sh --help` for the full usage of the script. Save the script and type `./install.sh --help` for the full usage of the script.
</TabItem> </TabItem>
<TabItem value="manual" label="Install & Configure Manually"> <TabItem value="manual" label="Install & Configure Manually">
Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation. Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation.
...@@ -76,7 +73,7 @@ grafana-cli plugins install tdengine-datasource ...@@ -76,7 +73,7 @@ grafana-cli plugins install tdengine-datasource
sudo -u grafana grafana-cli plugins install tdengine-datasource sudo -u grafana grafana-cli plugins install tdengine-datasource
``` ```
Alternatively, you can manually download the .zip file from [GitHub](https://github.com/taosdata/grafanaplugin/releases/tag/latest) or [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) and unpack it into your grafana plugins directory. You can also download zip files from [GitHub](https://github.com/taosdata/grafanaplugin/releases/tag/latest) or [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) and install manually. The commands are as follows:
```bash ```bash
GF_VERSION=3.2.2 GF_VERSION=3.2.2
...@@ -131,7 +128,7 @@ docker run -d \ ...@@ -131,7 +128,7 @@ docker run -d \
grafana/grafana grafana/grafana
``` ```
You can setup a zero-configuration stack for TDengine + Grafana by [docker-compose](https://docs.docker.com/compose/) and [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) file： You can setup a zero-configuration stack for TDengine + Grafana by [docker-compose](https://docs.docker.com/compose/) and [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) file:
1. Save the provisioning configuration file to `tdengine.yml`. 1. Save the provisioning configuration file to `tdengine.yml`.
...@@ -196,7 +193,7 @@ Go back to the main interface to create a dashboard and click Add Query to enter ...@@ -196,7 +193,7 @@ Go back to the main interface to create a dashboard and click Add Query to enter
As shown above, select the `TDengine` data source in the `Query` and enter the corresponding SQL in the query box below for query. As shown above, select the `TDengine` data source in the `Query` and enter the corresponding SQL in the query box below for query.
- INPUT SQL: enter the statement to be queried (the result set of the SQL statement should be two columns and multiple rows), for example: `select avg(mem_system) from log.dn where ts >= $from and ts < $to interval($interval)`, where, from, to and interval are built-in variables of the TDengine plugin, indicating the range and time interval of queries fetched from the Grafana plugin panel. In addition to the built-in variables, custom template variables are also supported. - INPUT SQL: Enter the desired query (the results being two columns and multiple rows), such as `select _wstart, avg(mem_system) from log.dnodes_info where ts >= $from and ts < $to interval($interval)`. In this statement, $from, $to, and $interval are variables that Grafana replaces with the query time range and interval. In addition to the built-in variables, custom template variables are also supported.
- ALIAS BY: This allows you to set the current query alias. - ALIAS BY: This allows you to set the current query alias.
- GENERATE SQL: Clicking this button will automatically replace the corresponding variables and generate the final executed statement. - GENERATE SQL: Clicking this button will automatically replace the corresponding variables and generate the final executed statement.
...@@ -208,7 +205,11 @@ Follow the default prompt to query the average system memory usage for the speci ...@@ -208,7 +205,11 @@ Follow the default prompt to query the average system memory usage for the speci
### Importing the Dashboard ### Importing the Dashboard
You can install TDinsight dashboard in data source configuration page (like `http://localhost:3000/datasources/edit/1/dashboards`) as a monitoring visualization tool for TDengine cluster. The dashboard is published in Grafana as [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167). Check the [TDinsight User Manual](/reference/tdinsight/) for the details. You can install TDinsight dashboard in data source configuration page (like `http://localhost:3000/datasources/edit/1/dashboards`) as a monitoring visualization tool for TDengine cluster. Ensure that you use TDinsight for 3.x.
![TDengine Database Grafana plugine import dashboard](./import_dashboard.webp)
A dashboard for TDengine 2.x has been published on Grafana: [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167)) 。 Check the [TDinsight User Manual](/reference/tdinsight/) for the details.
For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list: For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list:
......
docs/en/20-third-party/06-statsd.md
浏览文件 @ a76bbfe0
--- ---
sidebar_label: StatsD sidebar_label: StatsD
title: StatsD writing title: StatsD Writing
--- ---
import StatsD from "../14-reference/_statsd.mdx" import StatsD from "../14-reference/_statsd.mdx"
...@@ -12,8 +12,8 @@ You can write StatsD data to TDengine by simply modifying the configuration file ...@@ -12,8 +12,8 @@ You can write StatsD data to TDengine by simply modifying the configuration file
## Prerequisites ## Prerequisites
To write StatsD data to TDengine requires the following preparations. To write StatsD data to TDengine requires the following preparations.
- The TDengine cluster has been deployed and is working properly 1. The TDengine cluster is deployed and functioning properly
- taosAdapter is installed and running properly. Please refer to the [taosAdapter manual](/reference/taosadapter) for details. 2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
- StatsD has been installed. To install StatsD, please refer to [official documentation](https://github.com/statsd/statsd) - StatsD has been installed. To install StatsD, please refer to [official documentation](https://github.com/statsd/statsd)
## Configuration steps ## Configuration steps
...@@ -39,8 +39,12 @@ $ echo "foo:1|c" | nc -u -w0 127.0.0.1 8125 ...@@ -39,8 +39,12 @@ $ echo "foo:1|c" | nc -u -w0 127.0.0.1 8125
Use the TDengine CLI to verify that StatsD data is written to TDengine and can read out correctly. Use the TDengine CLI to verify that StatsD data is written to TDengine and can read out correctly.
``` ```
Welcome to the TDengine shell from Linux, Client Version:3.0.0.0 taos> show databases;
Copyright (c) 2022 by TAOS Data, Inc. All rights reserved. name | created_time | ntables | vgroups | replica | quorum | days | keep | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | cachelast | precision | update | status |
====================================================================================================================================================================================================================================================================================
log | 2022-04-20 07:19:50.260 | 11 | 1 | 1 | 1 | 10 | 3650 | 16 | 6 | 100 | 4096 | 1 | 3000 | 2 | 0 | ms | 0 | ready |
statsd | 2022-04-20 09:54:51.220 | 1 | 1 | 1 | 1 | 10 | 3650 | 16 | 6 | 100 | 4096 | 1 | 3000 | 2 | 0 | ns | 2 | ready |
Query OK, 2 row(s) in set (0.003142s)
taos> use statsd; taos> use statsd;
Database changed. Database changed.
......
docs/en/20-third-party/10-hive-mq-broker.md
浏览文件 @ a76bbfe0
--- ---
sidebar_label: HiveMQ Broker sidebar_label: HiveMQ Broker
title: HiveMQ Broker writing title: HiveMQ Broker Writing
--- ---
[HiveMQ](https://www.hivemq.com/) is an MQTT broker that provides community and enterprise editions. HiveMQ is mainly for enterprise emerging machine-to-machine M2M communication and internal transport, meeting scalability, ease of management, and security features. HiveMQ provides an open-source plug-in development kit. MQTT data can be saved to TDengine via TDengine extension for HiveMQ. Please refer to the [HiveMQ extension - TDengine documentation](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README_EN.md) for details on how to use it. [HiveMQ](https://www.hivemq.com/) is an MQTT broker that provides community and enterprise editions. HiveMQ is mainly for enterprise emerging machine-to-machine M2M communication and internal transport, meeting scalability, ease of management, and security features. HiveMQ provides an open-source plug-in development kit. MQTT data can be saved to TDengine via TDengine extension for HiveMQ. For more information, see [HiveMQ TDengine Extension](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README_EN.md).
\ No newline at end of file
docs/en/27-train-faq/01-faq.md
浏览文件 @ a76bbfe0
--- ---
sidebar_label: FAQ
title: Frequently Asked Questions title: Frequently Asked Questions
--- ---
## Submit an Issue ## Submit an Issue
If the tips in FAQ don't help much, please submit an issue on [GitHub](https://github.com/taosdata/TDengine) to describe your problem. In your description please include the TDengine version, hardware and OS information, the steps to reproduce the problem and any other relevant information. It would be very helpful if you can package the contents in `/var/log/taos` and `/etc/taos` and upload. These two are the default directories used by TDengine. If you have changed the default directories in your configuration, please package the files in your configured directories. We recommended setting `debugFlag` to 135 in `taos.cfg`, restarting `taosd`, then reproducing the problem and collecting the logs. If you don't want to restart, an alternative way of setting `debugFlag` is executing `alter dnode <dnode_id> debugFlag 135` command in TDengine CLI `taos`. During normal running, however, please make sure `debugFlag` is set to 131. If your issue could not be resolved by reviewing this documentation, you can submit your issue on GitHub and receive support from the TDengine Team. When you submit an issue, attach the following directories from your TDengine deployment:
## Frequently Asked Questions 1. The directory containing TDengine logs (`/var/log/taos` by default)
2. The directory containing TDengine configuration files (`/etc/taos` by default)
### 1. How to upgrade to TDengine 2.0 from older version? In your GitHub issue, provide the version of TDengine and the operating system and environment for your deployment, the operations that you performed when the issue occurred, and the time of occurrence and affected tables.
version 2.x is not compatible with version 1.x. With regard to the configuration and data files, please perform the following steps before upgrading. Please follow data integrity, security, backup and other relevant SOPs, best practices before removing/deleting any data. To obtain more debugging information, open `taos.cfg` and set the `debugFlag` parameter to `135`. Then restart TDengine Server and reproduce the issue. The debug-level logs generated help the TDengine Team to resolve your issue. If it is not possible to restart TDengine Server, you can run the following command in the TDengine CLI to set the debug flag:
1. Delete configuration files: `sudo rm -rf /etc/taos/taos.cfg` ```
2. Delete log files: `sudo rm -rf /var/log/taos/` alter dnode <dnode_id> 'debugFlag' '135';
3. Delete data files if the data doesn't need to be kept: `sudo rm -rf /var/lib/taos/` ```
4. Install latest 2.x version
5. If the data needs to be kept and migrated to newer version, please contact professional service at TDengine for assistance.
### 2. How to handle "Unable to establish connection"？ You can run the `SHOW DNODES` command to determine the dnode ID.
When the client is unable to connect to the server, you can try the following ways to troubleshoot and resolve the problem. When debugging information is no longer needed, set `debugFlag` to 131.
1. Check the network ## Frequently Asked Questions
- Check if the hosts where the client and server are running are accessible to each other, for example by `ping` command. ### 1. What are the best practices for upgrading a previous version of TDengine to version 3.0?
- Check if the TCP/UDP on port 6030-6042 are open for access if firewall is enabled. If possible, disable the firewall for diagnostics, but please ensure that you are following security and other relevant protocols.
- Check if the FQDN and serverPort are configured correctly in `taos.cfg` used by the server side.
- Check if the `firstEp` is set properly in the `taos.cfg` used by the client side.
2. Make sure the client version and server version are same. TDengine 3.0 is not compatible with the configuration and data files from previous versions. Before upgrading, perform the following steps:
3. On server side, check the running status of `taosd` by executing `systemctl status taosd` . If your server is started using another way instead of `systemctl`, use the proper method to check whether the server process is running normally. 1. Run `sudo rm -rf /etc/taos/taos.cfg` to delete your configuration file.
2. Run `sudo rm -rf /var/log/taos/` to delete your log files.
3. Run `sudo rm -rf /var/lib/taos/` to delete your data files.
4. Install TDengine 3.0.
5. For assistance in migrating data to TDengine 3.0, contact [TDengine Support](https://tdengine.com/support).
4. If using connector of Python, Java, Go, Rust, C#, node.JS on Linux to connect to the server, please make sure `libtaos.so` is in directory `/usr/local/taos/driver` and `/usr/local/taos/driver` is in system lib search environment variable `LD_LIBRARY_PATH`. ### 4. How can I resolve the "Unable to establish connection" error?
5. If using connector on Windows, please make sure `C:\TDengine\driver\taos.dll` is in your system lib search path. We recommend putting `taos.dll` under `C:\Windows\System32`. This error indicates that the client could not connect to the server. Perform the following troubleshooting steps:
6. Some advanced network diagnostics tools 1. Check the network.
- On Linux system tool `nc` can be used to check whether the TCP/UDP can be accessible on a specified port - For machines deployed in the cloud, verify that your security group can access ports 6030 and 6031 (TCP and UDP).
Check whether a UDP port is open: `nc -vuz {hostIP} {port} ` - For virtual machines deployed locally, verify that the hosts where the client and server are running are accessible to each other. Do not use localhost as the hostname.
Check whether a TCP port on server side is open: `nc -l {port}` - For machines deployed on a corporate network, verify that your NAT configuration allows the server to respond to the client.
Check whether a TCP port on client side is open: `nc {hostIP} {port}`
- On Windows system `Test-NetConnection -ComputerName {fqdn} -Port {port}` on PowerShell can be used to check whether the port on server side is open for access. 2. Verify that the client and server are running the same version of TDengine.
7. TDengine CLI `taos` can also be used to check network, please refer to [TDengine CLI](/reference/taos-shell). 3. On the server, run `systemctl status taosd` to verify that taosd is running normally. If taosd is stopped, run `systemctl start taosd`.
### 3. How to handle "Unexpected generic error in RPC" or "Unable to resolve FQDN" ? 4. Verify that the client is configured with the correct FQDN for the server.
This error is caused because the FQDN can't be resolved. Please try following ways: 5. If the server cannot be reached with the `ping` command, verify that network and DNS or hosts file settings are correct. For a TDengine cluster, the client must be able to ping the FQDN of every node in the cluster.
1. Check whether the FQDN is configured properly on the server side 6. Verify that your firewall settings allow all hosts in the cluster to communicate on ports 6030 and 6041 (TCP and UDP). You can run `ufw status` (Ubuntu) or `firewall-cmd --list-port` (CentOS) to check the configuration.
2. If DSN server is configured in the network, please check whether it works; otherwise, check `/etc/hosts` to see whether the FQDN is configured with correct IP
3. If the network configuration on the server side is OK, try to ping the server from the client side.
4. If TDengine has been used before with an old hostname then the hostname has been changed, please check `/var/lib/taos/taos/dnode/dnodeEps.json`. Before setting up a new TDengine cluster, it's better to cleanup the directories configured.
### 4. "Invalid SQL" is returned even though the Syntax is correct 7. If you are using the Python, Java, Go, Rust, C#, or Node.js connector on Linux to connect to the server, verify that `libtaos.so` is in the `/usr/local/taos/driver` directory and `/usr/local/taos/driver` is in the `LD_LIBRARY_PATH` environment variable.
"Invalid SQL" is returned when the length of SQL statement exceeds maximum allowed length or the syntax is not correct. 8. If you are using Windows, verify that `C:\TDengine\driver\taos.dll` is in the `PATH` environment variable. If possible, move `taos.dll` to the `C:\Windows\System32` directory.
### 5. Whether validation queries are supported? 9. On Linux systems, you can use the `nc` tool to check whether a port is accessible:
- To check whether a UDP port is open, run `nc -vuz {hostIP} {port}`.
- To check whether a TCP port on the server side is open, run `nc -l {port}`.
- To check whether a TCP port on client side is open, run `nc {hostIP} {port}`.
It's suggested to use a builtin database named as `log` to monitor. 10. On Windows systems, you can run `Test-NetConnection -ComputerName {fqdn} -Port {port}` in PowerShell to check whether a port on the server side is accessible.
<a class="anchor" id="update"></a> 11. You can also use the TDengine CLI to diagnose network issues. For more information, see [Problem Diagnostics](https://docs.tdengine.com/operation/diagnose/).
### 6. Can I delete a record? ### 5. How can I resolve the "Unable to resolve FQDN" error?
From version 2.6.0.0 Enterprise version, deleting data can be supported. Clients and dnodes must be able to resolve the FQDN of each required node. You can confirm your configuration as follows:
### 7. How to create a table of over 1024 columns? 1. Verify that the FQDN is configured properly on the server.
2. If your network has a DNS server, verify that it is operational.
3. If your network does not have a DNS server, verify that the FQDNs in the `hosts` file are correct.
4. On the client, use the `ping` command to test your connection to the server. If you cannot ping an FQDN, TDengine cannot reach it.
5. If TDengine has been previously installed and the `hostname` was modified, open `dnode.json` in the `data` folder and verify that the endpoint configuration is correct. The default location of the dnode file is `/var/lib/taos/dnode`. Ensure that you clean up previous installations before reinstalling TDengine.
6. Confirm whether FQDNs are preconfigured in `/etc/hosts` and `/etc/hostname`.
From version 2.1.7.0, at most 4096 columns can be defined for a table. ### 6. What is the most effective way to write data to TDengine?
### 8. How to improve the efficiency of inserting data? Writing data in batches provides higher efficiency in most situations. You can insert one or more data records into one or more tables in a single SQL statement.
Inserting data in batch is a good practice. Single SQL statement can insert data for one or multiple tables in batch. ### 9. Why are table names not fully displayed?
### 9. JDBC Error： the executed SQL is not a DML or a DDL？ The number of columns in the TDengine CLI terminal display is limited. This can cause table names to be cut off, and if you use an incomplete name in a statement, the "Table does not exist" error will occur. You can increase the display size with the `maxBinaryDisplayWidth` parameter or the SQL statement `set max_binary_display_width`. You can also append `\G` to your SQL statement to bypass this limitation.
Please upgrade to latest JDBC driver, for details please refer to [Java Connector](/reference/connector/java) ### 10. How can I migrate data?
### 10. Failed to connect with error "invalid timestamp" In TDengine, the `hostname` uniquely identifies a machine. When you move data files to a new machine, you must configure the new machine to have the same `host name` as the original machine.
The most common reason is that the time setting is not aligned on the client side and the server side. On Linux system, please use `ntpdate` command. On Windows system, please enable automatic sync in system time setting. :::note
### 11. Table name is not shown in full The data structure of previous versions of TDengine is not compatible with version 3.0. To migrate from TDengine 1.x or 2.x to 3.0, you must export data from your older deployment and import it back into TDengine 3.0.
There is a display width setting in TDengine CLI `taos`. It can be controlled by configuration parameter `maxBinaryDisplayWidth`, or can be set using SQL command `set max_binary_display_width`. A more convenient way is to append `\G` in a SQL command to bypass this limitation. :::
### 12. How to change log level temporarily? ### 11. How can I temporary change the log level from the TDengine Client?
Below SQL command can be used to adjust log level temporarily To change the log level for debugging purposes, you can use the following command:
```sql ```sql
ALTER LOCAL flag_name flag_value; ALTER LOCAL local_option
local_option: {
'resetLog'
| 'rpcDebugFlag' value
| 'tmrDebugFlag' value
| 'cDebugFlag' value
| 'uDebugFlag' value
| 'debugFlag' value
}
``` ```
- flag_name can be: debugFlag，cDebugFlag，tmrDebugFlag，uDebugFlag，rpcDebugFlag
- flag_value can be: 131 (INFO/WARNING/ERROR), 135 (plus DEBUG), 143 (plus TRACE)
<a class="anchor" id="timezone"></a> Use `resetlog` to remove all logs generated on the local client. Use the other parameters to specify a log level for a specific component.
### 13. What to do if go compilation fails? For each parameter, you can set the value to `131` (error and warning), `135` (error, warning, and debug), or `143` (error, warning, debug, and trace).
From version 2.3.0.0, a new component named `taosAdapter` is introduced. Its' developed in Go. If you want to compile from source code and meet go compilation problems, try to do below steps to resolve Go environment problems. ### Why do TDengine components written in Go fail to compile?
```sh TDengine includes taosAdapter, an independent component written in Go. This component provides the REST API as well as data access for other products such as Prometheus and Telegraf.
go env -w GO111MODULE=on When using the develop branch, you must run `git submodule update --init --recursive` to download the taosAdapter repository and then compile it.
go env -w GOPROXY=https://goproxy.cn,direct
``` TDengine Go components require Go version 1.14 or later.
### 13. How can I query the storage space being used by my data?
The TDengine data files are stored in `/var/lib/taos` by default. Log files are stored in `/var/log/taos`.
To see how much space your data files occupy, run `du -sh /var/lib/taos/vnode --exclude='wal'`. This excludes the write-ahead log (WAL) because its size is relatively fixed while writes are occurring, and it is written to disk and cleared when you shut down TDengine.
If you want to see how much space is occupied by a single database, first determine which vgroup is storing the database by running `show vgroups`. Then check `/var/lib/taos/vnode` for the files associated with the vgroup ID.
### 15. How is timezone information processed for timestamps?
TDengine uses the timezone of the client for timestamps. The server timezone does not affect timestamps. The client converts Unix timestamps in SQL statements to UTC before sending them to the server. When you query data on the server, it provides timestamps in UTC to the client, which converts them to its local time.
Timestamps are processed as follows:
1. The client uses its system timezone unless it has been configured otherwise.
2. A timezone configured in `taos.cfg` takes precedence over the system timezone.
3. A timezone explicitly specified when establishing a connection to TDengine through a connector takes precedence over `taos.cfg` and the system timezone. For example, the Java connector allows you to specify a timezone in the JDBC URL.
4. If you use an RFC 3339 timestamp (2013-04-12T15:52:01.123+08:00), or an ISO 8601 timestamp (2013-04-12T15:52:01.123+0800), the timezone specified in the timestamp is used instead of the timestamps configured using any other method.
### 16. Which network ports are required by TDengine?
See [serverPort](https://docs.tdengine.com/reference/config/#serverport) in Configuration Parameters.
Note that ports are specified using 6030 as the default first port. If you change this port, all other ports change as well.
### 17. Why do applications such as Grafana fail to connect to TDengine over the REST API?
In TDengine, the REST API is provided by taosAdapter. Ensure that taosAdapter is running before you connect an application to TDengine over the REST API. You can run `systemctl start taosadapter` to start the service.
Note that the log path for taosAdapter must be configured separately. The default path is `/var/log/taos`. You can choose one of eight log levels. The default is `info`. You can set the log level to `panic` to disable log output. You can modify the taosAdapter configuration file to change these settings. The default location is `/etc/taos/taosadapter.toml`.
For more information, see [taosAdapter](https://docs.tdengine.com/reference/taosadapter/).
### 18. How can I resolve out-of-memory (OOM) errors?
OOM errors are thrown by the operating system when its memory, including swap, becomes insufficient and it needs to terminate processes to remain operational. Most OOM errors in TDengine occur for one of the following reasons: free memory is less than the value of `vm.min_free_kbytes` or free memory is less than the size of the request. If TDengine occupies reserved memory, an OOM error can occur even when free memory is sufficient.
TDengine preallocates memory to each vnode. The number of vnodes per database is determined by the `vgroups` parameter, and the amount of memory per vnode is determined by the `buffer` parameter. To prevent OOM errors from occurring, ensure that you prepare sufficient memory on your hosts to support the number of vnodes that your deployment requires. Configure an appropriately sized swap space. If you continue to receive OOM errors, your SQL statements may be querying too much data for your system. TDengine Enterprise Edition includes optimized memory management that increases stability for enterprise customers.
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或