Skip to content

TDengine
TDengine

taosdata / TDengine
大约 2 年 前同步成功

通知 1192
Star 22018
Fork 4786
TDengine
TDengine
未验证 提交 b0e07ef8 编写于 作者: E Elias Soong 提交者： GitHub
浏览文件
操作

Merge pull request #6455 from taosdata/docs/Update-Latest-Feature 

[TD-4669] <docs>: upload English documents & pictures for version 2.0.
上级 20e9a1fb 2e0a381b
隐藏空白更改
内联 并排
Showing with 4065 addition and 0 deletion
documentation20/en/05.insert/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Efficient Data Writing
TDengine supports multiple interfaces to write data, including SQL, Prometheus, Telegraf, EMQ MQTT Broker, HiveMQ Broker, CSV file, etc. Kafka, OPC and other interfaces will be provided in the future. Data can be inserted in a single piece or in batches, data from one or multiple data collection points can be inserted at the same time. TDengine supports multi-thread insertion, nonsequential data insertion, and also historical data insertion.
## <a class="anchor" id="sql"></a> SQL Writing
Applications insert data by executing SQL insert statements through C/C + +, JDBC, GO, or Python Connector, and users can manually enter SQL insert statements to insert data through TAOS Shell. For example, the following insert writes a record to table d1001:
```mysql
```
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31);
```
```
TDengine supports writing multiple records at a time. For example, the following command writes two records to table d1001:
```mysql
```
INSERT INTO d1001 VALUES (1538548684000, 10.2, 220, 0.23) (1538548696650, 10.3, 218, 0.25);
```
```
TDengine also supports writing data to multiple tables at a time. For example, the following command writes two records to d1001 and one record to d1002:
```mysql
```
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31) (1538548695000, 12.6, 218, 0.33) d1002 VALUES (1538548696800, 12.3, 221, 0.31);
```
```
For the SQL INSERT Grammar, please refer to [Taos SQL insert](https://www.taosdata.com/en/documentation/taos-sql#insert)。
**Tips:**
- To improve writing efficiency, batch writing is required. The more records written in a batch, the higher the insertion efficiency. However, a record cannot exceed 16K, and the total length of an SQL statement cannot exceed 64K (it can be configured by parameter maxSQLLength, and the maximum can be configured to 1M).
- TDengine supports multi-thread parallel writing. To further improve writing speed, a client needs to open more than 20 threads to write parallelly. However, after the number of threads reaches a certain threshold, it cannot be increased or even become decreased, because too much frequent thread switching brings extra overhead.
- For a same table, if the timestamp of a newly inserted record already exists, (no database was created using UPDATE 1) the new record will be discarded as default, that is, the timestamp must be unique in a table. If an application automatically generates records, it is very likely that the generated timestamps will be the same, so the number of records successfully inserted will be smaller than the number of records the application try to insert. If you use UPDATE 1 option when creating a database, inserting a new record with the same timestamp will overwrite the original record.
- The timestamp of written data must be greater than the current time minus the time of configuration parameter keep. If keep is configured for 3650 days, data older than 3650 days cannot be written. The timestamp for writing data cannot be greater than the current time plus configuration parameter days. If days is configured to 2, data 2 days later than the current time cannot be written.
## <a class="anchor" id="prometheus"></a> Direct Writing of Prometheus
As a graduate project of Cloud Native Computing Foundation, [Prometheus](https://www.prometheus.io/) is widely used in the field of performance monitoring and K8S performance monitoring. TDengine provides a simple tool [Bailongma](https://github.com/taosdata/Bailongma), which only needs to be simply configured in Prometheus without any code, and can directly write the data collected by Prometheus into TDengine, then automatically create databases and related table entries in TDengine according to rules. Blog post [Use Docker Container to Quickly Build a Devops Monitoring Demo](https://www.taosdata.com/blog/2020/02/03/1189.html), which is an example of using bailongma to write Prometheus and Telegraf data into TDengine.
### Compile blm_prometheus From Source
Users need to download the source code of [Bailongma](https://github.com/taosdata/Bailongma) from github, then compile and generate an executable file using Golang language compiler. Before you start compiling, you need to complete following prepares:
- A server running Linux OS
- Golang version 1.10 and higher installed
- An appropriated TDengine version. Because the client dynamic link library of TDengine is used, it is necessary to install the same version of TDengine as the server-side; for example, if the server version is TDengine 2.0. 0, ensure install the same version on the linux server where bailongma is located (can be on the same server as TDengine, or on a different server)
Bailongma project has a folder, blm_prometheus, which holds the prometheus writing API. The compiling process is as follows:
```bash
```
cd blm_prometheus
go build
```
```
If everything goes well, an executable of blm_prometheus will be generated in the corresponding directory.
### Install Prometheus
Download and install as the instruction of Prometheus official website. [Download Address](https://prometheus.io/download/)
### Configure Prometheus
Read the Prometheus [configuration document](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) and add following configurations in the section of Prometheus configuration file
- url: The URL provided by bailongma API service, refer to the blm_prometheus startup example section below
After Prometheus launched, you can check whether data is written successfully through query taos client.
### Launch blm_prometheus
blm_prometheus has following options that you can configure when you launch blm_prometheus.
```sh
--tdengine-name
If TDengine is installed on a server with a domain name, you can also access the TDengine by configuring the domain name of it. In K8S environment, it can be configured as the service name that TDengine runs
--batch-size
blm_prometheus assembles the received prometheus data into a TDengine writing request. This parameter controls the number of data pieces carried in a writing request sent to TDengine at a time.
--dbname
Set a name for the database created in TDengine, blm_prometheus will automatically create a database named dbname in TDengine, and the default value is prometheus.
--dbuser
Set the user name to access TDengine, the default value is'root '
--dbpassword
Set the password to access TDengine, the default value is'taosdata '
--port
The port number blm_prometheus used to serve prometheus.
```
### Example
Launch an API service for blm_prometheus with the following command:
```bash
./blm_prometheus -port 8088
```
Assuming that the IP address of the server where blm_prometheus located is "10.1.2. 3", the URL shall be added to the configuration file of Prometheus as:
remote_write:
\- url: "http://10.1.2.3:8088/receive"
### Query written data of prometheus
The format of generated data by Prometheus is as follows:
```json
{
Timestamp: 1576466279341,
Value: 37.000000,
apiserver_request_latencies_bucket {
component="apiserver",
instance="192.168.99.116:8443",
job="kubernetes-apiservers",
le="125000",
resource="persistentvolumes", s
cope="cluster",
verb="LIST",
version=“v1"
}
}
```
Where apiserver_request_latencies_bucket is the name of the time-series data collected by prometheus, and the tag of the time-series data is in the following {}. blm_prometheus automatically creates a STable in TDengine with the name of the time series data, and converts the tag in {} into the tag value of TDengine, with Timestamp as the timestamp and value as the value of the time-series data. Therefore, in the client of TDEngine, you can check whether this data was successfully written through the following instruction.
```mysql
use prometheus;
select * from apiserver_request_latencies_bucket;
```
## <a class="anchor" id="telegraf"></a> Direct Writing of Telegraf
[Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a popular open source tool for IT operation data collection. TDengine provides a simple tool [Bailongma](https://github.com/taosdata/Bailongma), which only needs to be simply configured in Telegraf without any code, and can directly write the data collected by Telegraf into TDengine, then automatically create databases and related table entries in TDengine according to rules. Blog post [Use Docker Container to Quickly Build a Devops Monitoring Demo](https://www.taosdata.com/blog/2020/02/03/1189.html), which is an example of using bailongma to write Prometheus and Telegraf data into TDengine.
### Compile blm_telegraf From Source Code
Users need to download the source code of [Bailongma](https://github.com/taosdata/Bailongma) from github, then compile and generate an executable file using Golang language compiler. Before you start compiling, you need to complete following prepares:
- A server running Linux OS
- Golang version 1.10 and higher installed
- An appropriated TDengine version. Because the client dynamic link library of TDengine is used, it is necessary to install the same version of TDengine as the server-side; for example, if the server version is TDengine 2.0. 0, ensure install the same version on the linux server where bailongma is located (can be on the same server as TDengine, or on a different server)
Bailongma project has a folder, blm_telegraf, which holds the Telegraf writing API. The compiling process is as follows:
```bash
cd blm_telegraf
go build
```
If everything goes well, an executable of blm_telegraf will be generated in the corresponding directory.
### Install Telegraf
At the moment, TDengine supports Telegraf version 1.7. 4 and above. Users can download the installation package on Telegraf's website according to your current operating system. The download address is as follows: https://portal.influxdata.com/downloads
### Configure Telegraf
Modify the TDengine-related configurations in the Telegraf configuration file /etc/telegraf/telegraf.conf.
In the output plugins section, add the [[outputs.http]] configuration:
- url: The URL provided by bailongma API service, please refer to the example section below
- data_format: "json"
- json_timestamp_units: "1ms"
In agent section:
- hostname: The machine name that distinguishes different collection devices, and it is necessary to ensure its uniqueness
- metric_batch_size: 100, which is the max number of records per batch wriiten by Telegraf allowed. Increasing the number can reduce the request sending frequency of Telegraf.
For information on how to use Telegraf to collect data and more about using Telegraf, please refer to the official [document](https://docs.influxdata.com/telegraf/v1.11/) of Telegraf.
### Launch blm_telegraf
blm_telegraf has following options, which can be set to tune configurations of blm_telegraf when launching.
```sh
--host
The ip address of TDengine server, default is null
--batch-size
blm_prometheus assembles the received telegraf data into a TDengine writing request. This parameter controls the number of data pieces carried in a writing request sent to TDengine at a time.
--dbname
Set a name for the database created in TDengine, blm_telegraf will automatically create a database named dbname in TDengine, and the default value is prometheus.
--dbuser
Set the user name to access TDengine, the default value is 'root '
--dbpassword
Set the password to access TDengine, the default value is'taosdata '
--port
The port number blm_telegraf used to serve Telegraf.
```
### Example
Launch an API service for blm_telegraf with the following command
```bash
./blm_telegraf -host 127.0.0.1 -port 8089
```
Assuming that the IP address of the server where blm_telegraf located is "10.1.2. 3", the URL shall be added to the configuration file of telegraf as:
```yaml
url = "http://10.1.2.3:8089/telegraf"
```
### Query written data of telegraf
The format of generated data by telegraf is as follows:
```json
{
"fields": {
"usage_guest": 0,
"usage_guest_nice": 0,
"usage_idle": 89.7897897897898,
"usage_iowait": 0,
"usage_irq": 0,
"usage_nice": 0,
"usage_softirq": 0,
"usage_steal": 0,
"usage_system": 5.405405405405405,
"usage_user": 4.804804804804805
},
"name": "cpu",
"tags": {
"cpu": "cpu2",
"host": "bogon"
},
"timestamp": 1576464360
}
```
Where the name field is the name of the time-series data collected by telegraf, and the tag field is the tag of the time-series data. blm_telegraf automatically creates a STable in TDengine with the name of the time series data, and converts the tag field into the tag value of TDengine, with Timestamp as the timestamp and fields values as the value of the time-series data. Therefore, in the client of TDEngine, you can check whether this data was successfully written through the following instruction.
```mysql
use telegraf;
select * from cpu;
```
MQTT is a popular data transmission protocol in the IoT. TDengine can easily access the data received by MQTT Broker and write it to TDengine.
## <a class="anchor" id="emq"></a> Direct Writing of EMQ Broker
[EMQ](https://github.com/emqx/emqx) is an open source MQTT Broker software, with no need of coding, only to use "rules" in EMQ Dashboard for simple configuration, and MQTT data can be directly written into TDengine. EMQ X supports storing data to the TDengine by sending it to a Web service, and also provides a native TDengine driver on Enterprise Edition for direct data store. Please refer to [EMQ official documents](https://docs.emqx.io/broker/latest/cn/rule/rule-example.html#%E4%BF%9D%E5%AD%98%E6%95%B0%E6%8D%AE%E5%88%B0-tdengine) for more details.
## <a class="anchor" id="hivemq"></a> Direct Writing of HiveMQ Broker
[HiveMQ](https://www.hivemq.com/) is an MQTT agent that provides Free Personal and Enterprise Edition versions. It is mainly used for enterprises, emerging machine-to-machine(M2M) communication and internal transmission to meet scalability, easy management and security features. HiveMQ provides an open source plug-in development kit. You can store data to TDengine via HiveMQ extension-TDengine. Refer to the [HiveMQ extension-TDengine documentation](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README.md) for more details.
\ No newline at end of file
documentation20/en/06.queries/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Efficient Data Querying
## <a class="anchor" id="queries"></a> Main Query Features
TDengine uses SQL as the query language. Applications can send SQL statements through C/C + +, Java, Go, Python connectors, and users can manually execute SQL Ad-Hoc Query through the Command Line Interface (CLI) tool TAOS Shell provided by TDengine. TDengine supports the following query functions:
- Single-column and multi-column data query
- Multiple filters for tags and numeric values: >, <, =, < >, like, etc
- Group by, Order by, Limit/Offset of aggregation results
- Four operations for numeric columns and aggregation results
- Time stamp aligned join query (implicit join) operations
- Multiple aggregation/calculation functions: count, max, min, avg, sum, twa, stddev, leastsquares, top, bottom, first, last, percentile, apercentile, last_row, spread, diff, etc
For example, in TAOS shell, the records with vlotage > 215 are queried from table d1001, sorted in descending order by timestamps, and only two records are outputted.
```mysql
taos> select * from d1001 where voltage > 215 order by ts desc limit 2;
ts | current | voltage | phase |
======================================================================================
2018-10-03 14:38:16.800 | 12.30000 | 221 | 0.31000 |
2018-10-03 14:38:15.000 | 12.60000 | 218 | 0.33000 |
Query OK, 2 row(s) in set (0.001100s)
```
In order to meet the needs of an IoT scenario, TDengine supports several special functions, such as twa (time weighted average), spread (difference between maximum and minimum), last_row (last record), etc. More functions related to IoT scenarios will be added. TDengine also supports continuous queries.
For specific query syntax, please see the [Data Query section of TAOS SQL](https://www.taosdata.com/cn/documentation/taos-sql#select).
## <a class="anchor" id="aggregation"></a> Multi-table Aggregation Query
In an IoT scenario, there are often multiple data collection points in a same type. TDengine uses the concept of STable to describe a certain type of data collection point, and an ordinary table to describe a specific data collection point. At the same time, TDengine uses tags to describe the statical attributes of data collection points. A given data collection point has a specific tag value. By specifying the filters of tags, TDengine provides an efficient method to aggregate and query the sub-tables of STables (data collection points of a certain type). Aggregation functions and most operations on ordinary tables are applicable to STables, and the syntax is exactly the same.
**Example 1**: In TAOS Shell, look up the average voltages collected by all smart meters in Beijing and group them by location
```mysql
taos> SELECT AVG(voltage) FROM meters GROUP BY location;
avg(voltage) | location |
=============================================================
222.000000000 | Beijing.Haidian |
219.200000000 | Beijing.Chaoyang |
Query OK, 2 row(s) in set (0.002136s)
```
**Example 2**: In TAOS Shell, look up the number of records with groupId 2 in the past 24 hours, check the maximum current of all smart meters
```mysql
taos> SELECT count(*), max(current) FROM meters where groupId = 2 and ts > now - 24h;
cunt(*) | max(current) |
==================================
5 | 13.4 |
Query OK, 1 row(s) in set (0.002136s)
```
TDengine only allows aggregation queries between tables belonging to a same STable, means aggregation queries between different STables are not supported. In the Data Query section of TAOS SQL, query class operations will all be indicated that whether STables are supported.
## <a class="anchor" id="sampling"></a> Down Sampling Query, Interpolation
In a scenario of IoT, it is often necessary to aggregate the collected data by intervals through down sampling. TDengine provides a simple keyword interval, which makes query operations according to time windows extremely simple. For example, the current values collected by smart meter d1001 are summed every 10 seconds.
```mysql
taos> SELECT sum(current) FROM d1001 INTERVAL(10s);
ts | sum(current) |
======================================================
2018-10-03 14:38:00.000 | 10.300000191 |
2018-10-03 14:38:10.000 | 24.900000572 |
Query OK, 2 row(s) in set (0.000883s)
```
The down sampling operation is also applicable to STables, such as summing the current values collected by all smart meters in Beijing every second.
```mysql
taos> SELECT SUM(current) FROM meters where location like "Beijing%" INTERVAL(1s);
ts | sum(current) |
======================================================
2018-10-03 14:38:04.000 | 10.199999809 |
2018-10-03 14:38:05.000 | 32.900000572 |
2018-10-03 14:38:06.000 | 11.500000000 |
2018-10-03 14:38:15.000 | 12.600000381 |
2018-10-03 14:38:16.000 | 36.000000000 |
Query OK, 5 row(s) in set (0.001538s)
```
The down sampling operation also supports time offset, such as summing the current values collected by all smart meters every second, but requires each time window to start from 500 milliseconds.
```mysql
taos> SELECT SUM(current) FROM meters INTERVAL(1s, 500a);
ts | sum(current) |
======================================================
2018-10-03 14:38:04.500 | 11.189999809 |
2018-10-03 14:38:05.500 | 31.900000572 |
2018-10-03 14:38:06.500 | 11.600000000 |
2018-10-03 14:38:15.500 | 12.300000381 |
2018-10-03 14:38:16.500 | 35.000000000 |
Query OK, 5 row(s) in set (0.001521s)
```
In a scenario of IoT, it is difficult to synchronize the time stamp of collected data at each point, but many analysis algorithms (such as FFT) need to align the collected data strictly at equal intervals of time. In many systems, it’s required to write their own programs to process, but the down sampling operation of TDengine can be easily solved. If there is no collected data in an interval, TDengine also provides interpolation calculation function.
For details of syntax rules, please refer to the [Time-dimension Aggregation section of TAOS SQL](https://www.taosdata.com/en/documentation/taos-sql#aggregation).
\ No newline at end of file
documentation20/en/07.advanced-features/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Advanced Features
## <a class="anchor" id="continuous-query"></a> Continuous Query
Continuous Query is a query executed by TDengine periodically with a sliding window, it is a simplified stream computing driven by timers. Continuous query can be applied to a table or a STable automatically and periodically, and the result set can be passed to the application directly via call back function, or written into a new table in TDengine. The query is always executed on a specified time window (window size is specified by parameter interval), and this window slides forward while time flows (the sliding period is specified by parameter sliding).
Continuous query of TDengine adopts time-driven mode, which can be defined directly by TAOS SQL without additional operation. Using continuous query, results can be generated conveniently and quickly according to the time window, thus down sampling the original collected data. After the user defines a continuous query through TAOS SQL, TDengine automatically pulls up the query at the end of the last complete time period and pushes the calculated results to the user or writes them back to TDengine.
The continuous query provided by TDengine differs from the time window calculation in ordinary stream computing in the following ways:
- Unlike the real-time feedback calculated results of stream computing, continuous query only starts calculation after the time window is closed. For example, if the time period is 1 day, the results of that day will only be generated after 23:59:59.
- If a history record is written to the time interval that has been calculated, the continuous query will not recalculate and will not push the results to the user again. For the mode of writing back to TDengine, the existing calculated results will not be updated.
- Using the mode of continuous query pushing results, the server does not cache the client's calculation status, nor does it provide Exactly-Once semantic guarantee. If the user's application side crashed, the continuous query pulled up again would only recalculate the latest complete time window from the time pulled up again. If writeback mode is used, TDengine can ensure the validity and continuity of data writeback.
### How to use continuous query
The following is an example of the smart meter scenario to introduce the specific use of continuous query. Suppose we create a STables and sub-tables through the following SQL statement:
```sql
create table meters (ts timestamp, current float, voltage int, phase float) tags (location binary(64), groupdId int);
create table D1001 using meters tags ("Beijing.Chaoyang", 2);
create table D1002 using meters tags ("Beijing.Haidian", 2);
...
```
We already know that the average voltage of these meters can be counted with one minute as the time window and 30 seconds as the forward increment through the following SQL statement.
```sql
select avg(voltage) from meters interval(1m) sliding(30s);
```
Every time this statement is executed, all data will be recalculated. If you need to execute every 30 seconds to incrementally calculate the data of the latest minute, you can improve the above statement as following, using a different `startTime` each time and executing it regularly:
```sql
select avg(voltage) from meters where ts > {startTime} interval(1m) sliding(30s);
```
There is no problem with this, but TDengine provides a simpler method, just add `create table {tableName} as` before the initial query statement, for example:
```sql
create table avg_vol as select avg(voltage) from meters interval(1m) sliding(30s);
```
A new table named `avg_vol` will be automatically created, and then every 30 seconds, TDengine will incrementally execute the SQL statement after `as` and write the query result into this table. The user program only needs to query the data from `avg_vol`. For example:
```mysql
taos> select * from avg_vol;
ts | avg_voltage_ |
===================================================
2020-07-29 13:37:30.000 | 222.0000000 |
2020-07-29 13:38:00.000 | 221.3500000 |
2020-07-29 13:38:30.000 | 220.1700000 |
2020-07-29 13:39:00.000 | 223.0800000 |
```
It should be noted that the minimum value of the query time window is 10 milliseconds, and there is no upper limit of the time window range.
In addition, TDengine also supports users to specify the starting and ending times of a continuous query. If the start time is not entered, the continuous query will start from the time window where the first original data is located; If no end time is entered, the continuous query will run permanently; If the user specifies an end time, the continuous query stops running after the system time reaches the specified time. For example, a continuous query created with the following SQL will run for one hour and then automatically stop.
```mysql
create table avg_vol as select avg(voltage) from meters where ts > now and ts <= now + 1h interval(1m) sliding(30s);
```
It should be noted that now in the above example refers to the time when continuous queries are created, not the time when queries are executed, otherwise, queries cannot be stopped automatically. In addition, in order to avoid the problems caused by delayed writing of original data as much as possible, there is a certain delay in the calculation of continuous queries in TDengine. In other words, after a time window has passed, TDengine will not immediately calculate the data of this window, so it will take a while (usually not more than 1 minute) to find the calculation result.
### Manage the Continuous Query
Users can view all continuous queries running in the system through the show streams command in the console, and can kill the corresponding continuous queries through the kill stream command. Subsequent versions will provide more finer-grained and convenient continuous query management commands.
## <a class="anchor" id="subscribe"></a> Publisher/Subscriber
Based on the natural time-series characteristics of data, the data insert of TDengine is logically consistent with the data publish (pub) of messaging system, which can be regarded as a new record inserted with timestamp in the system. At the same time, TDengine stores data in strict accordance with the monotonous increment of time-series. Essentially, every table in TDengine can be regarded as a standard messaging queue.
TDengine supports embedded lightweight message subscription and publishment services. Using the API provided by the system, users can subscribe to one or more tables in the database using common query statements. The maintenance of subscription logic and operation status is completed by the client. The client regularly polls the server for whether new records arrive, and the results will be fed back to the client when new records arrive.
The status of the subscription and publishment services of TDengine is maintained by the client, but not by the TDengine server. Therefore, if the application restarts, it is up to the application to decide from which point of time to obtain the latest data.
In TDengine, there are three main APIs relevant to subscription:
```c
taos_subscribe
taos_consume
taos_unsubscribe
```
Please refer to the [C/C++ Connector](https://www.taosdata.com/cn/documentation/connector/) for the documentation of these APIs. The following is still a smart meter scenario as an example to introduce their specific usage (please refer to the previous section "Continuous Query" for the structure of STables and sub-tables). The complete sample code can be found [here](https://github.com/taosdata/TDengine/blob/master/tests/examples/c/subscribe.c).
If we want to be notified and do some process when the current of a smart meter exceeds a certain limit (e.g. 10A), there are two methods: one is to query each sub-table separately, record the timestamp of the last piece of data after each query, and then only query all data after this timestamp:
```sql
select * from D1001 where ts > {last_timestamp1} and current > 10;
select * from D1002 where ts > {last_timestamp2} and current > 10;
...
```
This is indeed feasible, but as the number of meters increases, the number of queries will also increase, and the performance of both the client and the server will be affected, until the system cannot afford it.
Another method is to query the STable. In this way, no matter how many meters there are, only one query is required:
```sql
select * from meters where ts > {last_timestamp} and current > 10;
```
However, how to choose `last_timestamp` has become a new problem. Because, on the one hand, the time of data generation (the data timestamp) and the time of data storage are generally not the same, and sometimes the deviation is still very large; On the other hand, the time when the data of different meters arrive at TDengine will also vary. Therefore, if we use the timestamp of the data from the slowest meter as `last_timestamp` in the query, we may repeatedly read the data of other meters; If the timestamp of the fastest meter is used, the data of other meters may be missed.
The subscription function of TDengine provides a thorough solution to the above problem.
First, use `taos_subscribe` to create a subscription:
```c
TAOS_SUB* tsub = NULL;
if (async) {
　　// create an asynchronized subscription, the callback function will be called every 1s
　　tsub = taos_subscribe(taos, restart, topic, sql, subscribe_callback, &blockFetch, 1000);
} else {
　　// create an synchronized subscription, need to call 'taos_consume' manually
　　tsub = taos_subscribe(taos, restart, topic, sql, NULL, NULL, 0);
}
```
Subscriptions in TDengine can be either synchronous or asynchronous, and the above code will decide which method to use based on the value of parameter `async` obtained from the command line. Here, synchronous means that the user program calls `taos_consume` directly to pull data, while asynchronous means that the API calls `taos_consume` in another internal thread, and then gives the pulled data to the callback function `subscribe_callback` for processing.
Parameter `taos` is an established database connection and has no special requirements in synchronous mode. However, in asynchronous mode, it should be noted that it will not be used by other threads, otherwise it may lead to unpredictable errors, because the callback function is called in the internal thread of the API, while some APIs of TDengine are not thread-safe.
Parameter `sql` is a query statement in which you can specify filters using where clause. In our example, if you only want to subscribe to data when the current exceeds 10A, you can write as follows:
```sql
select * from meters where current > 10;
```
Note that the starting time is not specified here, so the data of all timers will be read. If you only want to start subscribing from the data one day ago and do not need earlier historical data, you can add a time condition:
```sql
select * from meters where ts > now - 1d and current > 10;
```
The `topic` of the subscription is actually its name, because the subscription function is implemented in the client API, so it is not necessary to ensure that it is globally unique, but it needs to be unique on a client machine.
If the subscription of name `topic` does not exist, the parameter restart is meaningless; However, if the user program exits after creating this subscription, when it starts again and reuses this `topic`, `restart` will be used to decide whether to read data from scratch or from the previous location. In this example, if `restart` is **true** (non-zero value), the user program will definitely read all the data. However, if this subscription exists before, and some data has been read, and `restart` is **false** (zero), the user program will not read the previously read data.
The last parameter of `taos_subscribe` is the polling period in milliseconds. In synchronous mode, if the interval between the two calls to `taos_consume` is less than this time, `taos_consume` will block until the interval exceeds this time. In asynchronous mode, this time is the minimum time interval between two calls to the callback function.
The penultimate parameter of `taos_subscribe` is used by the user program to pass additional parameters to the callback function, which is passed to the callback function as it is without any processing by the subscription API. This parameter is meaningless in sync mode.
After created, the subscription can consume data. In synchronous mode, the sample code is the following as the `else` section:
```c
if (async) {
　　getchar();
} else while(1) {
　　TAOS_RES* res = taos_consume(tsub);
　　if (res == NULL) {
　　　　printf("failed to consume data.");
　　　　break;
　　} else {
　　　　print_result(res, blockFetch);
　　　　getchar();
　　}
}
```
Here is a **while** loop. Every time the user presses the Enter key, `taos_consume` is called, and the return value of `taos_consume` is the query result set, which is exactly the same as `taos_use_result`. In the example, the code using this result set is the function `print_result`:
```c
void print_result(TAOS_RES* res, int blockFetch) {
　　TAOS_ROW row = NULL;
　　int num_fields = taos_num_fields(res);
　　TAOS_FIELD* fields = taos_fetch_fields(res);
　　int nRows = 0;
　　if (blockFetch) {
　　　　nRows = taos_fetch_block(res, &row);
　　　　for (int i = 0; i < nRows; i++) {
　　　　　　char temp[256];
　　　　　　taos_print_row(temp, row + i, fields, num_fields);
　　　　　　puts(temp);
　　　　}
　　} else {
　　　　while ((row = taos_fetch_row(res))) {
　　　　　　char temp[256];
　　　　　　taos_print_row(temp, row, fields, num_fields);puts(temp);
　　　　　　nRows++;
　　　　}
　　}
　　printf("%d rows consumed.\n", nRows);
}
```
Among them, `taos_print_row` is used to process subscription to data. In our example, it will print out all eligible records. In asynchronous mode, it is simpler to consume subscribed data:
```c
void subscribe_callback(TAOS_SUB* tsub, TAOS_RES *res, void* param, int code) {
　　print_result(res, *(int*)param);
}
```
To end a data subscription, you need to call `taos_unsubscribe`:
```c
taos_unsubscribe(tsub, keep);
```
Its second parameter is used to decide whether to keep the progress information of subscription on the client. If this parameter is **false** (zero), the subscription can only be restarted no matter what the `restart` parameter is when `taos_subscribe` is called next time. In addition, progress information is saved in the directory {DataDir}/subscribe/. Each subscription has a file with the same name as its `topic`. Deleting a file will also lead to a new start when the corresponding subscription is created next time.
After introducing the code, let's take a look at the actual running effect. For exmaple:
- Sample code has been downloaded locally
- TDengine has been installed on the same machine
- All the databases, STables and sub-tables required by the example have been created
You can compile and start the sample program by executing the following command in the directory where the sample code is located:
```shell
$ make
$ ./subscribe -sql='select * from meters where current > 10;'
```
After the sample program starts, open another terminal window, and the shell that starts TDengine inserts a data with a current of 12A into **D1001**:
```shell
$ taos
> use test;
> insert into D1001 values(now, 12, 220, 1);
```
At this time, because the current exceeds 10A, you should see that the sample program outputs it to the screen. You can continue to insert some data to observe the output of the sample program.
### Use data subscription in Java
The subscription function also provides a Java development interface, as described in [Java Connector](https://www.taosdata.com/cn/documentation/connector/). It should be noted that the Java interface does not provide asynchronous subscription mode at present, but user programs can achieve the same feature by creating TimerTask.
The following is an example to introduce its specific use. The function it completes is basically the same as the C language example described earlier, and it is also to subscribe to all records with current exceeding 10A in the database.
#### Prepare data
```sql
# Create power Database
taos> create database power;
# Switch to the database
taos> use power;
# Create a STable
taos> create table meters(ts timestamp, current float, voltage int, phase int) tags(location binary(64), groupId int);
# Create tables
taos> create table d1001 using meters tags ("Beijing.Chaoyang", 2);
taos> create table d1002 using meters tags ("Beijing.Haidian", 2);
# Insert test data
taos> insert into d1001 values("2020-08-15 12:00:00.000", 12, 220, 1),("2020-08-15 12:10:00.000", 12.3, 220, 2),("2020-08-15 12:20:00.000", 12.2, 220, 1);
taos> insert into d1002 values("2020-08-15 12:00:00.000", 9.9, 220, 1),("2020-08-15 12:10:00.000", 10.3, 220, 1),("2020-08-15 12:20:00.000", 11.2, 220, 1);
# Query all records with current over 10A from STable meters
taos> select * from meters where current > 10;
ts | current | voltage | phase | location | groupid |
===========================================================================================================
2020-08-15 12:10:00.000 | 10.30000 | 220 | 1 | Beijing.Haidian | 2 |
2020-08-15 12:20:00.000 | 11.20000 | 220 | 1 | Beijing.Haidian | 2 |
2020-08-15 12:00:00.000 | 12.00000 | 220 | 1 | Beijing.Chaoyang | 2 |
2020-08-15 12:10:00.000 | 12.30000 | 220 | 2 | Beijing.Chaoyang | 2 |
2020-08-15 12:20:00.000 | 12.20000 | 220 | 1 | Beijing.Chaoyang | 2 |
Query OK, 5 row(s) in set (0.004896s)
```
#### Example
```java
public class SubscribeDemo {
private static final String topic = "topic-meter-current-bg-10";
private static final String sql = "select * from meters where current > 10";
public static void main(String[] args) {
Connection connection = null;
TSDBSubscribe subscribe = null;
try {
Class.forName("com.taosdata.jdbc.TSDBDriver");
Properties properties = new Properties();
properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8");
String jdbcUrl = "jdbc:TAOS://127.0.0.1:6030/power?user=root&password=taosdata";
connection = DriverManager.getConnection(jdbcUrl, properties);
subscribe = ((TSDBConnection) connection).subscribe(topic, sql, true); // Create a subscription
int count = 0;
while (count < 10) {
TimeUnit.SECONDS.sleep(1); / Wait 1 second to avoid calling consume too frequently and causing pressure on server
TSDBResultSet resultSet = subscribe.consume(); // 消费数据
if (resultSet == null) {
continue;
}
ResultSetMetaData metaData = resultSet.getMetaData();
while (resultSet.next()) {
int columnCount = metaData.getColumnCount();
for (int i = 1; i <= columnCount; i++) {
System.out.print(metaData.getColumnLabel(i) + ": " + resultSet.getString(i) + "\t");
}
System.out.println();
count++;
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (null != subscribe)
subscribe.close(true); // Close the subscription
if (connection != null)
connection.close();
} catch (SQLException throwables) {
throwables.printStackTrace();
}
}
}
}
```
Run the sample program. First, it consumes all the historical data that meets the query conditions:
```shell
# java -jar subscribe.jar
ts: 1597464000000 current: 12.0 voltage: 220 phase: 1 location: Beijing.Chaoyang groupid : 2
ts: 1597464600000 current: 12.3 voltage: 220 phase: 2 location: Beijing.Chaoyang groupid : 2
ts: 1597465200000 current: 12.2 voltage: 220 phase: 1 location: Beijing.Chaoyang groupid : 2
ts: 1597464600000 current: 10.3 voltage: 220 phase: 1 location: Beijing.Haidian groupid : 2
ts: 1597465200000 current: 11.2 voltage: 220 phase: 1 location: Beijing.Haidian groupid : 2
```
Then, add a piece of data to the table via taos client:
```sql
# taos
taos> use power;
taos> insert into d1001 values("2020-08-15 12:40:00.000", 12.4, 220, 1);
```
Because the current of this data is greater than 10A, the sample program will consume it:
```shell
ts: 1597466400000 current: 12.4 voltage: 220 phase: 1 location: Beijing.Chaoyang groupid: 2
```
## <a class="anchor" id="cache"></a> Cache
TDengine adopts a time-driven cache management strategy (First-In-First-Out, FIFO), also known as a write-driven cache management mechanism. This strategy is different from the read-driven data cache mode (Least-Recent-Use, LRU), which directly saves the most recently written data in the system buffer. When the buffer reaches a threshold, the oldest data is written to disk in batches. Generally speaking, for the use of IoT data, users are most concerned about the recently generated data, that is, the current status. TDengine takes full advantage of this feature by storing the most recently arrived (current status) data in the buffer.
TDengine provides data collection in milliseconds to users through query functions. Saving the recently arrived data directly in buffer can respond to the user's query analysis for the latest piece or batch of data more quickly, and provide faster database query response as a whole. In this way, TDengine can be used as a data buffer by setting appropriate configuration parameters without deploying additional caching systems, which can effectively simplify the system architecture and reduce the operation costs. It should be noted that after the TDengine is restarted, the buffer of the system will be emptied, the previously cached data will be written to disk in batches, and the cached data will not reload the previously cached data into the buffer like some proprietary Key-value cache system.
TDengine allocates a fixed size of memory space as a buffer, which can be configured according to application requirements and hardware resources. By properly setting the buffer space, TDengine can provide extremely high-performance write and query support. Each virtual node in TDengine is allocated a separate cache pool when it is created. Each virtual node manages its own cache pool, and different virtual nodes do not share the pool. All tables belonging to each virtual node share the cache pool owned by itself.
TDengine manages the memory pool by blocks, and the data is stored in the form of rows within. The memory pool of a vnode is allocated by blocks when the vnode is created, and each memory block is managed according to the First-In-First-Out strategy. When creating a memory pool, the size of the blocks is determined by the system configuration parameter cache; The number of memory blocks in each vnode is determined by the configuration parameter blocks. So for a vnode, the total memory size is: cache * blocks. A cache block needs to ensure that each table can store at least dozens of records in order to be efficient.
You can quickly obtain the last record of a table or a STable through the function last_row, which is very convenient to show the real-time status or collected values of each device on a large screen. For example:
```mysql
select last_row(voltage) from meters where location='Beijing.Chaoyang';
```
This SQL statement will obtain the last recorded voltage value of all smart meters located in Chaoyang District, Beijing.
## <a class="anchor" id="alert"></a> Alert
In scenarios of TDengine, alarm monitoring is a common requirement. Conceptually, it requires the program to filter out data that meet certain conditions from the data of the latest period of time, and calculate a result according to a defined formula based on these data. When the result meets certain conditions and lasts for a certain period of time, it will notify the user in some form.
In order to meet the needs of users for alarm monitoring, TDengine provides this function in the form of an independent module. For its installation and use, please refer to the blog [How to Use TDengine for Alarm Monitoring](https://www.taosdata.com/blog/2020/04/14/1438.html).
\ No newline at end of file
documentation20/en/08.connector/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Connectors
TDengine provides many connectors for development, including C/C++, JAVA, Python, RESTful, Go, Node.JS, etc.
At present, TDengine connectors support a wide range of platforms, including hardware platforms such as X64/X86/ARM64/ARM32/MIPS/Alpha, and development environments such as Linux/Win64/Win32. The comparison matrix is as follows:
| **CPU** | **X64 64bit** | **X64 64bit** | **X64 64bit** | **X86 32bit** | **ARM64** | **ARM32** | **MIPS Godson** | **Alpha Whenwei** | **X64 TimecomTech** |
| ----------- | ------------- | ------------- | ------------- | ------------- | --------- | --------- | --------------- | ----------------- | ------------------- |
| **OS** | **Linux** | **Win64** | **Win32** | **Win32** | **Linux** | **Linux** | **Linux** | **Linux** | **Linux** |
| **C/C++** | ● | ● | ● | ○ | ● | ● | ○ | ○ | ○ |
| **JDBC** | ● | ● | ● | ○ | ● | ● | ○ | ○ | ○ |
| **Python** | ● | ● | ● | ○ | ● | ● | ○ | -- | ○ |
| **Go** | ● | ● | ● | ○ | ● | ● | ○ | -- | -- |
| **NodeJs** | ● | ● | ○ | ○ | ● | ● | ○ | -- | -- |
| **C#** | ○ | ● | ● | ○ | ○ | ○ | ○ | -- | -- |
| **RESTful** | ● | ● | ● | ● | ● | ● | ○ | ○ | ○ |
Note: ● stands for that has been verified by official tests; ○ stands for that has been verified by unofficial tests.
Note:
- To access the TDengine database through connectors (except RESTful) in the system without TDengine server software, it is necessary to install the corresponding version of the client installation package to make the application driver (the file name is [libtaos.so](http://libtaos.so/) in Linux system and taos.dll in Windows system) installed in the system, otherwise, the error that the corresponding library file cannot be found will occur.
- All APIs that execute SQL statements, such as `tao_query`, `taos_query_a`, `taos_subscribe` in C/C + + Connector, and APIs corresponding to them in other languages, can only execute one SQL statement at a time. If the actual parameters contain multiple statements, their behavior is undefined.
- Users upgrading to TDengine 2.0. 8.0 must update the JDBC connection. TDengine must upgrade taos-jdbcdriver to 2.0.12 and above.
- No matter which programming language connector is selected, TDengine version 2.0 and above recommends that each thread of database application establish an independent connection or establish a connection pool based on threads to avoid mutual interference between threads of "USE statement" state variables in the connection (but query and write operations of the connection are thread-safe).
## <a class="anchor" id="driver"></a> Steps of Connector Driver Installation
The server should already have the TDengine server package installed. The connector driver installation steps are as follows:
**Linux**
**1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)**
- X64 hardware environment: TDengine-client-2.x.x.x-Linux-x64.tar.gz
- ARM64 hardware environment: TDengine-client-2.x.x.x-Linux-aarch64.tar.gz
- ARM32 hardware environment: TDengine-client-2.x.x.x-Linux-aarch32.tar.gz
**2. Unzip the package**
Place the package in any directory that current user can read/write, and then execute following command:
`tar -xzvf TDengine-client-xxxxxxxxx.tar.gz`
Where xxxxxx needs to be replaced with you actual version as a string.
**3. Execute installation script**
After extracting the package, you will see the following files (directories) in the extracting directory:
*install_client. sh*: Installation script for application driver
*taos.tar.gz*: Application driver installation package
*driver*: TDengine application driver
*connector*: Connectors for various programming languages (go/grafanaplugin/nodejs/python/JDBC)
*Examples*: Sample programs for various programming languages (C/C #/go/JDBC/matlab/python/R)
Run install_client.sh to install.
**4. Configure taos.cfg**
Edit the taos.cfg file (default path/etc/taos/taos.cfg) and change firstEP to End Point of the TDengine server, for example: [h1.taos.com](http://h1.taos.com/):6030.
**Tip: If no TDengine service deployed in this machine, but only the application driver is installed, only firstEP needs to be configured in taos.cfg, and FQDN does not.**
**Windows x64/x86**
- **1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)**
- X64 hardware environment: TDengine-client-2.X.X.X-Windows-x64.exe
- X86 hardware environment: TDengine-client-2.X.X.X-Windows-x86.exe
**2. Execute installation, select default vales as prompted to complete**
**3. Installatino path**
Default installation path is: C:\TDengine, with following files(directories):
*taos.exe*: taos shell command line program
*cfg*: configuration file directory
*driver*: application driver dynamic link library
*examples*: sample program bash/C/C #/go/JDBC/Python/Node.js
*include*: header file
*log*: log file
*unins000. exe*: uninstall program
**4. Configure taos.cfg**
Edit the taos.cfg file (default path/etc/taos/taos.cfg) and change firstEP to End Point of the TDengine server, for example: [h1.taos.com](http://h1.taos.com/):6030.
**Note:**
**1. If you use FQDN to connect to the server, you must confirm that the DNS of the local network environment has been configured, or add FQDN addressing records in the hosts file. For example, edit C:\ Windows\ system32\ drivers\ etc\ hosts, and add the following record: 192.168. 1.99 [h1.taos.com](http://h1.taos.com/)**
**2. Uninstall: Run unins000. exe to uninstall the TDengine application driver.**
**Installation verification**
After the above installation and configuration completed, and confirm that the TDengine service has started running normally, the taos client can be logged in at this time.
**Linux environment:**
If you execute taos directly under Linux shell, you should be able to connect to tdengine service normally and jump to taos shell interface. For Example:
```mysql
$ taos
Welcome to the TDengine shell from Linux, Client Version:2.0.5.0
Copyright (c) 2017 by TAOS Data, Inc. All rights reserved.
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep1,keep2,keep(D) | cache(MB)| blocks | minrows | maxrows | wallevel | fsync | comp | precision | status |
=========================================================================================================================================================================================================================
test | 2020-10-14 10:35:48.617 | 10 | 1 | 1 | 1 | 2 | 3650,3650,3650 | 16| 6 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
log | 2020-10-12 09:08:21.651 | 4 | 1 | 1 | 1 | 10 | 30,30,30 | 1| 3 | 100 | 4096 | 1 | 3000 | 2 | us | ready |
Query OK, 2 row(s) in set (0.001198s)
taos>
```
**Windows (x64/x86) environment:**
Under cmd, enter the c:\ tdengine directory and directly execute taos.exe, and you should be able to connect to tdengine service normally and jump to taos shell interface. For example:
```mysql
C:\TDengine>taos
Welcome to the TDengine shell from Linux, Client Version:2.0.5.0
Copyright (c) 2017 by TAOS Data, Inc. All rights reserved.
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep1,keep2,keep(D) | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | precision | status |
===================================================================================================================================================================================================================================================================
test | 2020-10-14 10:35:48.617 | 10 | 1 | 1 | 1 | 2 | 3650,3650,3650 | 16 | 6 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
log | 2020-10-12 09:08:21.651 | 4 | 1 | 1 | 1 | 10 | 30,30,30 | 1 | 3 | 100 | 4096 | 1 | 3000 | 2 | us | ready |
Query OK, 2 row(s) in set (0.045000s)
taos>
```
## <a class="anchor" id="c-cpp"></a> C/C++ Connector
**Systems supported by C/C++ connectors as follows:**
| **CPU Type** | **x64****（****64bit****）** | | | **ARM64** | **ARM32** |
| -------------------- | ---------------------------- | ------- | ------- | --------- | ------------------ |
| **OS Type** | Linux | Win64 | Win32 | Linux | Linux |
| **Supported or Not** | Yes | **Yes** | **Yes** | **Yes** | **In development** |
The C/C + + API is similar to MySQL's C API. When application use it, it needs to include the TDengine header file taos.h (after installed, it is located in/usr/local/taos/include):
```C
#include <taos.h>
```
Note:
- The TDengine dynamic library needs to be linked at compiling. The library in Linux is [libtaos.so](http://libtaos.so/), which installed at/usr/local/taos/driver. By Windows, it is taos.dll and installed at C:\ TDengine.
- Unless otherwise specified, when the return value of API is an integer, 0 represents success, others are error codes representing the cause of failure, and when the return value is a pointer, NULL represents failure.
More sample codes for using C/C++ connectors, please visit https://github.com/taosdata/TDengine/tree/develop/tests/examples/c.
### Basic API
The basic API is used to create database connections and provide a runtime environment for the execution of other APIs.
- `void taos_init()`
Initialize the running environment. If the application does not actively call the API, the API will be automatically called when the application call taos_connect, so the application generally does not need to call the API manually.
- `void taos_cleanup()`
Clean up the running environment and call this API before the application exits.
- `int taos_options(TSDB_OPTION option, const void * arg, ...)`
Set client options, currently only time zone setting (_TSDB_OPTIONTIMEZONE) and encoding setting (_TSDB_OPTIONLOCALE) are supported. The time zone and encoding default to the current operating system settings.
- `char *taos_get_client_info()`
Get version information of the client.
- `TAOS *taos_connect(const char *host, const char *user, const char *pass, const char *db, int port)`
Create a database connection and initialize the connection context. The parameters that need to be provided by user include:
- - host: FQDN used by TDengine to manage the master node
- user: User name
- pass: Password
- db: Database name. If user does not provide it, it can be connected normally, means user can create a new database through this connection. If user provides a database name, means the user has created the database and the database is used by default
- port: Port number
A null return value indicates a failure. The application needs to save the returned parameters for subsequent API calls.
- `char *taos_get_server_info(TAOS *taos)`
Get version information of the server-side.
- `int taos_select_db(TAOS *taos, const char *db)`
Set the current default database to db.
- `void taos_close(TAOS *taos)`
Close the connection, where `taos` is the pointer returned by `taos_connect` function.
### Synchronous query API
Traditional database operation APIs all make synchronous operations. After the application calls an API, it remains blocked until the server returns the result. TDengine supports the following APIs:
- `TAOS_RES* taos_query(TAOS *taos, const char *sql)`
This API is used to execute SQL statements, which can be DQL, DML or DDL statements. Where `taos` parameter is a pointer obtained through `taos_connect`. You can't judge whether the execution result fails by whether the return value is NULL, but to use `taos_errno` function to parse the error code in the result set.
- `int taos_result_precision(TAOS_RES *res)`
The precision of the timestamp field in the returned result set, `0` for milliseconds, `1` for microseconds, and `2` for nanoseconds.
- `TAOS_ROW taos_fetch_row(TAOS_RES *res)`
Get the data in the query result set by rows.
- `int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows)`
The data in the query result set is obtained in batch, and the return value is the number of rows of the obtained data.
- `int taos_num_fields(TAOS_RES *res)` 和 `int taos_field_count(TAOS_RES *res)`
The two APIs are equivalent, and are used to get the number of columns in the query result set.
- `int* taos_fetch_lengths(TAOS_RES *res)`
Get the length of each field in the result set. The return value is an array whose length is the number of columns in the result set.
- `int taos_affected_rows(TAOS_RES *res)`
Get the number of rows affected by the executed SQL statement.
- `TAOS_FIELD *taos_fetch_fields(TAOS_RES *res)`
Get the attributes (data type, name, number of bytes) of each column of data in the query result set, which can be used in conjunction with `taos_num_files` to parse the data of a tuple (one row) returned by `taos_fetch_row`. The structure of `TAOS_FIELD` is as follows:
```c
typedef struct taosField {
char name[65]; // Column name
uint8_t type; // Data type
int16_t bytes; // Number of bytes
} TAOS_FIELD;
```
- `void taos_stop_query(TAOS_RES *res)`
Stop the execution of a query.
- `void taos_free_result(TAOS_RES *res)`
Release the query result set and related resources. After the query is completed, be sure to call the API to release resources, otherwise it may lead to application memory leakage. However, it should also be noted that after releasing resources, if you call functions such as `taos_consume` to obtain query results, it will lead the application to Crash.
- `char *taos_errstr(TAOS_RES *res)`
Get the reason why the last API call failed, and the return value is a string.
- `char *taos_errno(TAOS_RES *res)`
Get the reason why the last API call failed, and the return value is the error code.
**Note:** TDengine 2.0 and above recommends that each thread of a database application establish an independent connection or establish a connection pool based on threads. It is not recommended to pass the connection (TAOS\*) structure to different threads for sharing in applications. Query and write operations based on TAOS structure have multithread safety, but state variables such as "USE statement" may interfere with each other among threads. In addition, C connector can dynamically establish new database-oriented connections according to requirements (this process is not visible to users), and it is recommended to call `taos_close` to close the connection only when the program finally exits.
### Asynchronous query API
In addition to synchronous API, TDengine also provides higher performance asynchronous call API to handle data insertion and query operations. Under the same software and hardware environment, asynchronous API processes data insertion 2 ~ 4 times faster than synchronous API. Asynchronous API adopts a non-blocking call mode and returns immediately before the system really completes a given database operation. The calling thread can handle other work, thus improving the performance of the whole application. Asynchronous API has outstanding advantages in the case of poor network delay.
Asynchronous APIs all need applications to provide corresponding callback function. The callback function parameters are set as follows: the first two parameters are consistent, and the third parameter depends on different APIs. The first parameter param is provided to the system when the application calls the asynchronous API. When used for callback, the application can retrieve the context of the specific operation, depending on the specific implementation. The second parameter is the result set of SQL operation. If it is empty, such as insert operation, it means that there is no record returned. If it is not empty, such as select operation, it means that there is record returned.
Asynchronous APIs have relatively high requirements for users, who can selectively use them according to specific application scenarios. Here are three important asynchronous APIs:
- `void taos_query_a(TAOS *taos, const char *sql, void (*fp)(void *param, TAOS_RES *, int code), void *param);`
Execute SQL statement asynchronously.
- - taos: The database connection returned by calling `taos_connect`
- sql: The SQL statement needed to execute
- fp: User-defined callback function, whose third parameter `code` is used to indicate whether the operation is successful, `0` for success, and negative number for failure (call `taos_errstr` to get the reason for failure). When defining the callback function, it mainly handles the second parameter `TAOS_RES *`, which is the result set returned by the query
- param：the parameter for the callback
- `void taos_fetch_rows_a(TAOS_RES *res, void (*fp)(void *param, TAOS_RES *, int numOfRows), void *param);`
Get the result set of asynchronous queries in batch, which can only be used with `taos_query_a`. Within:
- - res: The result set returned when backcall `taos_query_a`
- fp: Callback function. Its parameter `param` is a user-definable parameter construct passed to the callback function; `numOfRows` is the number of rows of data obtained (not a function of the entire query result set). In the callback function, applications can get each row of the batch records by calling `taos_fetch_rows` forward iteration. After reading all the records in a block, the application needs to continue calling `taos_fetch_rows_a` in the callback function to obtain the next batch of records for processing until the number of records returned (`numOfRows`) is zero (the result is returned) or the number of records is negative (the query fails).
The asynchronous APIs of TDengine all use non-blocking calling mode. Applications can use multithreading to open multiple tables at the same time, and can query or insert to each open table at the same time. It should be pointed out that the **application client must ensure that the operation on the same table is completely serialized**, that is, when the insertion or query operation on the same table is not completed (when no result returned), the second insertion or query operation cannot be performed.
<a class="anchor" id="stmt"></a>
### Parameter binding API
In addition to calling `taos_query` directly for queries, TDengine also provides a Prepare API that supports parameter binding. Like MySQL, these APIs currently only support using question mark `?` to represent the parameters to be bound, as follows:
- `TAOS_STMT* taos_stmt_init(TAOS *taos)`
Create a `TAOS_STMT` object for calling later.
- `int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length)`
Parse a SQL statement and bind the parsing result and parameter information to STMT. If the parameter length is greater than 0, this parameter will be used as the length of the SQL statement. If it is equal to 0, the length of the SQL statement will be automatically judged.
- `int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_BIND *bind)`
For parameter binding, bind points to an array, and it is necessary to ensure that the number and order of elements in this array are exactly the same as the parameters in sql statement. TAOS_BIND is used in the same way as MYSQL_BIND in MySQL and is defined as follows:
```c
typedef struct TAOS_BIND {
int buffer_type;
void * buffer;
unsigned long buffer_length; // Not in use
unsigned long *length;
int * is_null;
int is_unsigned; // Not in use
int * error; // Not in use
} TAOS_BIND;
```
Add the curren bound parameters to the batch. After calling this function, you can call `taos_stmt_bind_param` again to bind the new parameters. It should be noted that this function only supports insert/import statements, and if it is other SQL statements such as select, it will return errors.
- `int taos_stmt_execute(TAOS_STMT *stmt)`
Execute the prepared statement. At the moment, a statement can only be executed once.
- `TAOS_RES* taos_stmt_use_result(TAOS_STMT *stmt)`
Gets the result set of the statement. The result set is used in the same way as when calling nonparameterized. After using it, `taos_free_result` should be called to release resources.
- `int taos_stmt_close(TAOS_STMT *stmt)`
Execution completed, release all resources.
### Continuous query interface
TDengine provides time-driven real-time stream computing APIs. You can perform various real-time aggregation calculation operations on tables (data streams) of one or more databases at regular intervals. The operation is simple, only APIs for opening and closing streams. The details are as follows:
- `TAOS_STREAM *taos_open_stream(TAOS *taos, const char *sql, void (*fp)(void *param, TAOS_RES *, TAOS_ROW row), int64_t stime, void *param, void (*callback)(void *))`
This API is used to create data streams where:
- - taos: Database connection established
- sql: SQL query statement (query statement only)
- fp: user-defined callback function pointer. After each stream computing is completed, TDengine passes the query result (TAOS_ROW), query status (TAOS_RES), and user-defined parameters (PARAM) to the callback function. In the callback function, the user can use taos_num_fields to obtain the number of columns in the result set, and taos_fetch_fields to obtain the type of data in each column of the result set.
- stime: The time when stream computing starts. If it is 0, it means starting from now. If it is not zero, it means starting from the specified time (the number of milliseconds from 1970/1/1 UTC time).
- param: It is a parameter provided by the application for callback. During callback, the parameter is provided to the application
- callback: The second callback function is called when the continuous query stop automatically.
The return value is NULL, indicating creation failed; the return value is not NULL, indicating creation successful.
- `void taos_close_stream (TAOS_STREAM *tstr)`
Close the data flow, where the parameter provided is the return value of `taos_open_stream`. When the user stops stream computing, be sure to close the data flow.
### Data subscription interface
The subscription API currently supports subscribing to one or more tables and continuously obtaining the latest data written to the tables through regular polling.
- `TAOS_SUB *taos_subscribe(TAOS* taos, int restart, const char* topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp, void *param, int interval)`
This function is for starting the subscription service, returning the subscription object in case of success, and NULL in case of failure. Its parameters are:
- - taos: Database connection established
- Restart: If the subscription already exists, do you want to start over or continue with the previous subscription
- Topic: Subject (that is, name) of the subscription. This parameter is the unique identification of the subscription
- sql: The query statement subscribed. This statement can only be a select statement. It should only query the original data, and can only query the data in positive time sequence
- fp: The callback function when the query result is received (the function prototype will be introduced later). It is only used when calling asynchronously, and this parameter should be passed to NULL when calling synchronously
- param: The additional parameter when calling the callback function, which is passed to the callback function as it is by the system API without any processing
- interval: Polling period in milliseconds. During asynchronous call, the callback function will be called periodically according to this parameter; In order to avoid affecting system performance, it is not recommended to set this parameter too small; When calling synchronously, if the interval between two calls to taos_consume is less than this period, the API will block until the interval exceeds this period.
- `typedef void (*TAOS_SUBSCRIBE_CALLBACK)(TAOS_SUB* tsub, TAOS_RES *res, void* param, int code)`
In asynchronous mode, the prototype of the callback function has the following parameters:
- - tsub: Subscription object
- res: Query the result set. Note that there may be no records in the result set
- param: Additional parameters supplied by the client when `taos_subscribe` is called
- code: Error code
- `TAOS_RES *taos_consume(TAOS_SUB *tsub)`
In synchronous mode, this function is used to get the results of subscription. The user application places it in a loop. If the interval between two calls to `taos_consume` is less than the polling cycle of the subscription, the API will block until the interval exceeds this cycle. If a new record arrives in the database, the API will return the latest record, otherwise it will return an empty result set with no records. If the return value is NULL, it indicates a system error. In asynchronous mode, user program should not call this API.
- `void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress)`
Unsubscribe. If the parameter `keepProgress` is not 0, the API will keep the progress information of subscription, and the subsequent call to `taos_subscribe` can continue based on this progress; otherwise, the progress information will be deleted and the data can only be read again.
## <a class="anchor" id="python"></a>Python Connector
See [video tutorials](https://www.taosdata.com/blog/2020/11/11/1963.html) for the use of Python connectors.
### Installation preparation
- For application driver installation, please refer to [steps of connector driver installation](https://www.taosdata.com/en/documentation/connector#driver)
- python 2.7 or >= 3.4 installed
- pip or pip3 installed
### Python client installation
#### Linux
Users can find the connector package for python2 and python3 in the source code src/connector/python (or tar.gz/connector/python) folder. Users can install it through `pip` command:
`pip install src/connector/python/linux/python2/`
or
`pip3 install src/connector/python/linux/python3/`
#### Windows
With Windows TDengine client installed, copy the file "C:\TDengine\driver\taos.dll" to the "C:\ windows\ system32" directory and enter the Windows <em>cmd</em> command line interface:
```cmd
cd C:\TDengine\connector\python
python -m pip install .
```
- If there is no `pip` command on the machine, the user can copy the taos folder under src/connector/python to the application directory for use. For Windows client, after installing the TDengine Windows client, copy C:\ TDengine\driver\taos.dll to the C:\ windows\ system32 directory.
### How to use
#### Code sample
- Import the TDengine client module
```python
import taos
```
- Get the connection and cursor object
```python
conn = taos.connect(host="127.0.0.1", user="root", password="taosdata", config="/etc/taos")
c1 = conn.cursor()
```
- *host* covers all IPs of TDengine server-side, and *config* is the directory where the client configuration files is located
- Write data
```python
import datetime
# Create a database
c1.execute('create database db')
c1.execute('use db')
# Create a table
c1.execute('create table tb (ts timestamp, temperature int, humidity float)')
# Insert data
start_time = datetime.datetime(2019, 11, 1)
affected_rows = c1.execute('insert into tb values (\'%s\', 0, 0.0)' %start_time)
# Insert data in batch
time_interval = datetime.timedelta(seconds=60)
sqlcmd = ['insert into tb values']
for irow in range(1,11):
start_time += time_interval
sqlcmd.append('(\'%s\', %d, %f)' %(start_time, irow, irow*1.2))
affected_rows = c1.execute(' '.join(sqlcmd))
```
- Query data
```python
c1.execute('select * from tb')
# pull query result
data = c1.fetchall()
# The result is a list, with each row as an element
numOfRows = c1.rowcount
numOfCols = len(c1.description)
for irow in range(numOfRows):
print("Row%d: ts=%s, temperature=%d, humidity=%f" %(irow, data[irow][0], data[irow][1],data[irow][2]))
# Use cursor loop directly to pull query result
c1.execute('select * from tb')
for data in c1:
print("ts=%s, temperature=%d, humidity=%f" %(data[0], data[1],data[2]))
```
- Create subscription
```python
# Create a subscription with the topic ‘test’ and a consumption cycle of 1000 milliseconds
# If the first parameter is True, it means restarting the subscription. If it is False and a subscription with the topic 'test 'has been created before, it means continuing to consume the data of this subscription instead of restarting to consume all the data
sub = conn.subscribe(True, "test", "select * from tb;", 1000)
```
- Consume subscription data
```python
data = sub.consume()
for d in data:
print(d)
```
- Unsubscription
```python
sub.close()
```
- Close connection
```python
c1.close()
conn.close()
```
#### Helper
Users can directly view the usage information of the module through Python's helper, or refer to the sample program in tests/examples/Python. The following are some common classes and methods:
- *TDengineConnection* class
Refer to help (taos.TDEngineConnection) in python. This class corresponds to a connection between the client and TDengine. In the scenario of client multithreading, it is recommended that each thread apply for an independent connection instance, but not recommended that multiple threads share a connection.
- *TDegnineCursor* class
Refer to help (taos.TDengineCursor) in python. This class corresponds to the write and query operations performed by the client. In the scenario of client multithreading, this cursor instance must be kept exclusive to threads and cannot be used by threads, otherwise errors will occur in the returned results.
- *connect* method
Used to generate an instance of taos.TDengineConnection.
### Python client code sample
In tests/examples/python, we provide a sample Python program read_example. py to guide you to design your own write and query program. After installing the corresponding client, introduce the taos class through `import taos`. The steps are as follows:
- Get the `TDengineConnection` object through `taos.connect`, which can be applied for only one by a program and shared among multiple threads.
- Get a new cursor object through the `.cursor ()` method of the `TDengineConnection` object, which must be guaranteed to be exclusive to each thread.
- Execute SQL statements for writing or querying through the `execute()` method of the cursor object.
- If a write statement is executed, `execute` returns the number of rows successfully written affected rows.
- If the query statement is executed, the result set needs to be pulled through the fetchall method after the execution is successful.
You can refer to the sample code for specific methods.
## <a class="anchor" id="restful"></a> RESTful Connector
To support the development of various types of platforms, TDengine provides an API that conforms to REST design standards, that is, RESTful API. In order to minimize the learning cost, different from other designs of database RESTful APIs, TDengine directly requests SQL statements contained in BODY through HTTP POST to operate the database, and only needs a URL. See the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1965.html) for the use of RESTful connectors.
### HTTP request format
```
http://<ip>:<PORT>/rest/sql
```
Parameter description:
- IP: Any host in the cluster
- PORT: httpPort configuration item in the configuration file, defaulting to 6041
For example: [http://192.168.0.1](http://192.168.0.1/): 6041/rest/sql is a URL that points to an IP address of 192.168. 0.1.
The header of HTTP request needs to carry identity authentication information. TDengine supports Basic authentication and custom authentication. Subsequent versions will provide standard and secure digital signature mechanism for identity authentication.
- Custom identity authentication information is as follows (We will introduce <token> later)
```
Authorization: Taosd <TOKEN>
```
- Basic identity authentication information is as follows
```
Authorization: Basic <TOKEN>
```
The BODY of HTTP request is a complete SQL statement. The data table in the SQL statement should provide a database prefix, such as \<db-name>.\<tb-name>. If the table name does not have a database prefix, the system returns an error. Because the HTTP module is just a simple forwarding, there is no current DB concept.
Use curl to initiate an HTTP Request through custom authentication. The syntax is as follows:
```bash
curl -H 'Authorization: Basic <TOKEN>' -d '<SQL>' <ip>:<PORT>/rest/sql
```
or
```bash
curl -u username:password -d '<SQL>' <ip>:<PORT>/rest/sql
```
Where `TOKEN` is the string of `{username}:{password}` encoded by Base64, for example, `root:taosdata` will be encoded as `cm9vdDp0YW9zZGF0YQ==`.
### HTTP return format
The return value is in JSON format, as follows:
```json
{
"status": "succ",
"head": ["ts","current", …],
"column_meta": [["ts",9,8],["current",6,4], …],
"data": [
["2018-10-03 14:38:05.000", 10.3, …],
["2018-10-03 14:38:15.000", 12.6, …]
],
"rows": 2
}
```
Description:
- status: Informs whether the operation results are successful or failed.
- head: The definition of the table, with only one column "affected_rows" if no result set is returned. (Starting from version 2.0. 17, it is recommended not to rely on the head return value to judge the data column type, but to use column_meta. In future versions, head may be removed from the return value.)
- column_meta: Starting with version 2.0. 17, this item is added to the return value to indicate the data type of each column in the data. Each column will be described by three values: column name, column type and type length. For example, ["current", 6, 4] means that the column name is "current"; the column type is 6, that is, float type; the type length is 4, which corresponds to a float represented by 4 bytes. If the column type is binary or nchar, the type length indicates the maximum content length that the column can save, rather than the specific data length in this return value. When the column type is nchar, its type length indicates the number of Unicode characters that can be saved, not bytes.
- data: The specific returned data, rendered line by line, if no result set is returned, then only [[affected_rows]]. The order of the data columns for each row in data is exactly the same as the order of the data columns described in column_meta.
- rows: Indicates the total number of rows of data.
Column types in column_meta:
- 1：BOOL
- 2：TINYINT
- 3：SMALLINT
- 4：INT
- 5：BIGINT
- 6：FLOAT
- 7：DOUBLE
- 8：BINARY
- 9：TIMESTAMP
- 10：NCHAR
### Custom authorization code
The HTTP request requires the authorization code `<TOKEN>` for identification. Authorization codes are usually provided by administrators. Authorization codes can be obtained simply by sending `HTTP GET` requests as follows:
```bash
curl http://<ip>:6041/rest/login/<username>/<password>
```
Where `ip` is the IP address of the TDengine database, `username` is the database user name, `password` is the database password, and the return value is in `JSON` format. The meanings of each field are as follows:
- status: flag bit for request result
- code: code of return value
- desc: Authorization code
Sample to get authorization code:
```bash
curl http://192.168.0.1:6041/rest/login/root/taosdata
```
Return value:
```json
{
"status": "succ",
"code": 0,
"desc": "/KfeAzX/f9na8qdtNZmtONryp201ma04bEl8LcvLUd7a8qdtNZmtONryp201ma04"
}
```
### Use case
- Lookup all records of table d1001 in demo database:
```bash
curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'select * from demo.d1001' 192.168.0.1:6041/rest/sql
```
Return value:
```json
{
"status": "succ",
"head": ["ts","current","voltage","phase"],
"column_meta": [["ts",9,8],["current",6,4],["voltage",4,4],["phase",6,4]],
"data": [
["2018-10-03 14:38:05.000",10.3,219,0.31],
["2018-10-03 14:38:15.000",12.6,218,0.33]
],
"rows": 2
}
```
- Craete a database demo:
```bash
curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'create database demo' 192.168.0.1:6041/rest/sql
```
Return value:
```json
{
"status": "succ",
"head": ["affected_rows"],
"column_meta": [["affected_rows",4,4]],
"data": [[1]],
"rows": 1
}
```
### Other cases
### Result set in Unix timestamp
When the HTTP request URL is sqlt, the timestamp of the returned result set will be expressed in Unix timestamp format, for example:
```bash
curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'select * from demo.d1001' 192.168.0.1:6041/rest/sqlt
```
Return value:
```json
{
"status": "succ",
"head": ["ts","current","voltage","phase"],
"column_meta": [["ts",9,8],["current",6,4],["voltage",4,4],["phase",6,4]],
"data": [
[1538548685000,10.3,219,0.31],
[1538548695000,12.6,218,0.33]
],
"rows": 2
}
```
#### Result set in UTC time string
When the HTTP request URL is `sqlutc`, the timestamp of the returned result set will be represented by a UTC time string, for example:
```bash
curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'select * from demo.t1' 192.168.0.1:6041/rest/sqlutc
```
Return value:
```json
{
"status": "succ",
"head": ["ts","current","voltage","phase"],
"column_meta": [["ts",9,8],["current",6,4],["voltage",4,4],["phase",6,4]],
"data": [
["2018-10-03T14:38:05.000+0800",10.3,219,0.31],
["2018-10-03T14:38:15.000+0800",12.6,218,0.33]
],
"rows": 2
}
```
### Important configuration options
Only some configuration parameters related to RESTful interface are listed below. Please refer to the instructions in the configuration file for other system parameters. Note: After the configuration is modified, the taosd service needs to be restarted before it can take effect.
- httpPort: The port number that provides RESTful services externally, which is bound to 6041 by default
- httpMaxThreads: The number of threads started, the default is 2 (starting with version 2.0. 17, the default value is changed to half of the CPU cores and rounded down)
- restfulRowLimit: The maximum number of result sets returned (in JSON format), default 10240
- httpEnableCompress: Compression is not supported by default. Currently, TDengine only supports gzip compression format
- httpdebugflag: Logging switch, 131: error and alarm information only, 135: debugging information, 143: very detailed debugging information, default 131
## <a class="anchor" id="csharp"></a> CSharp Connector
The C # connector supports: Linux 64/Windows x64/Windows x86.
### Installation preparation
- For application driver installation, please refer to the[ steps of installing connector driver](https://www.taosdata.com/en/documentation/connector#driver).
- . NET interface file TDengineDrivercs.cs and reference sample TDengineTest.cs are both located in the Windows client install_directory/examples/C# directory.
- On Windows, C # applications can use the native C interface of TDengine to perform all database operations, and future versions will provide the ORM (dapper) framework driver.
### Installation verification
Run install_directory/examples/C#/C#Checker/C#Checker.exe
```cmd
cd {install_directory}/examples/C#/C#Checker
csc /optimize *.cs
C#Checker.exe -h <fqdn>
```
### How to use C# connector
On Windows system, .NET applications can use the .NET interface of TDengine to perform all database operations. The steps to use it are as follows:
1. Add the. NET interface file TDengineDrivercs.cs to the .NET project where the application is located.
2. Users can refer to TDengineTest.cs to define database connection parameters and how to perform data insert, query and other operations;
This. NET interface requires the taos.dll file, so before executing the application, copy the taos.dll file in the Windows client install_directory/driver directory to the folder where the. NET project finally generated the .exe executable file. After running the exe file, you can access the TDengine database and do operations such as insert and query.
**Note:**
1. TDengine V2.0. 3.0 supports both 32-bit and 64-bit Windows systems, so when. NET project generates a .exe file, please select the corresponding "X86" or "x64" for the "Platform" under "Solution"/"Project".
2. This. NET interface has been verified in Visual Studio 2015/2017, and other VS versions have yet to be verified.
### Third-party Driver
Maikebing.Data.Taos is an ADO.Net provider for TDengine that supports Linux, Windows. This development package is provided by enthusiastic contributor 麦壳饼@@maikebing. For more details:
```
// Download
https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos
// How to use
https://www.taosdata.com/blog/2020/11/02/1901.html
```
## <a class="anchor" id="go"></a> Go Connector
### Installation preparation
- For application driver installation, please refer to the [steps of installing connector driver](https://www.taosdata.com/en/documentation/connector#driver).
The TDengine provides the GO driver taosSql. taosSql implements the GO language's built-in interface database/sql/driver. Users can access TDengine in the application by simply importing the package as follows, see https://github.com/taosdata/driver-go/blob/develop/taosSql/driver_test.go for details.
Sample code for using the Go connector can be found in https://github.com/taosdata/TDengine/tree/develop/tests/examples/go and the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1951.html).
```Go
import (
"database/sql"
_ "github.com/taosdata/driver-go/taosSql"
)
```
**It is recommended to use Go version 1.13 or above and turn on module support:**
```bash
go env -w GO111MODULE=on
go env -w GOPROXY=https://goproxy.io,direct
```
### Common APIs
- `sql.Open(DRIVER_NAME string, dataSourceName string) *DB`
This API is used to open DB and return an object of type \* DB. Generally, DRIVER_NAME is set to the string `taosSql`, and dataSourceName is set to the string `user:password@/tcp(host:port)/dbname`. If the customer wants to access TDengine with multiple goroutines concurrently, it is necessary to create a `sql.Open` object in each goroutine and use it to access TDengine.
**Note**: When the API is successfully created, there is no permission check. Only when Query or Exec is actually executed can the connection be truly created and whether the user/password/host/port is legal can be checked at the same time. In addition, because most of the implementation of the whole driver sinks into libtaos, which taosSql depends on. Therefore, sql.Open itself is particularly lightweight.
- `func (db *DB) Exec(query string, args ...interface{}) (Result, error)`
`sql.Open` built-in method to execute non-query related SQL
- `func (db *DB) Query(query string, args ...interface{}) (*Rows, error)`
`sql.Open` built-in method used to execute query statements
- `func (db *DB) Prepare(query string) (*Stmt, error)`
`sql.Open` built-in method used to create a prepared statement for later queries or executions.
- `func (s *Stmt) Exec(args ...interface{}) (Result, error)`
`sql.Open` built-in method to execute a prepared statement with the given arguments and returns a Result summarizing the effect of the statement.
- `func (s *Stmt) Query(args ...interface{}) (*Rows, error)`
`sql.Open` built-in method to query executes a prepared query statement with the given arguments and returns the query results as a \*Rows.
- `func (s *Stmt) Close() error`
`sql.Open` built-in method to closes the statement.
## <a class="anchor" id="nodejs"></a> Node.js Connector
The Node.js connector supports the following systems:
| **CPU Type** | **x64****（****64bit****）** | | | **aarch64** | **aarch32** |
| -------------------- | ---------------------------- | ------- | ------- | ----------- | ----------- |
| **OS Type** | Linux | Win64 | Win32 | Linux | Linux |
| **Supported or Not** | **Yes** | **Yes** | **Yes** | **Yes** | **Yes** |
See the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1957.html) for use of the Node.js connector.
### Installation preparation
- For application driver installation, please refer to the [steps of installing connector driver](https://www.taosdata.com/en/documentation/connector#driver).
### Install Node.js connector
Users can install it through [npm](https://www.npmjs.com/) or through the source code src/connector/nodejs/. The specific installation steps are as follows:
First, install the node.js connector through [npm](https://www.npmjs.com/).
```bash
npm install td2.0-connector
```
We recommend that use npm to install the node.js connector. If you do not have npm installed, you can copy src/connector/nodejs/ to your nodejs project directory.
We use [node-gyp](https://github.com/nodejs/node-gyp) to interact with the TDengine server. Before installing the node.js connector, you also need to install the following software:
### Linux
- python (recommended v2.7, not currently supported in v3.x.x)
- node 2.0. 6 supports v12. x and v10. x, 2.0. 5 and earlier supports v10. x, and other versions may have package compatibility issues.
- make
- [GCC](https://gcc.gnu.org/) and other C compilers
### Windows
#### Solution 1
Use Microsoft [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) to install all necessary tools by executing npm install --global --production windows-build-tools in cmd command line interface.
#### Solution 2
Mannually install the following tools:
- Install Visual Studio related tools: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community)
- Install [Python](https://www.python.org/downloads/) 2.7 (not supported in v3.x.x) and execute npm config set python python2.7
- Open `cmd`, `npm config set msvs_version 2017`
If the steps above cannot be performed successfully, you can refer to Microsoft's Node.js User Manual [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules).
If you use ARM64 Node.js on Windows 10 ARM, you also need to add "Visual C + + compilers and libraries for ARM64" and "Visual C + + ATL for ARM64".
#### Sample
The sample program source code is located in install_directory/examples/nodejs, and there are:
Node-example.js node.js sample source code Node-example-raw. js
### Installation verification
After installing the TDengine client, the nodejsChecker.js program can verify whether the current environment supports access to TDengine via nodejs.
Steps:
1. Create a new installation verification directory, for example: ~/tdengine-test, copy the nodejsChecker.js source program on github. Download address: （https://github.com/taosdata/TDengine/tree/develop/tests/examples/nodejs/nodejsChecker.js）.
2. Execute the following command:
```bash
npm init -y
npm install td2.0-connector
node nodejsChecker.js host=localhost
```
3. After performing the above steps, the nodejs connection Tdengine instance will be outputted on the command line, and the short-answer of insertion and query will be executed.
### How to use Node.js
The following are some basic uses of node.js connector. Please refer to [TDengine Node.js connector](http://docs.taosdata.com/node) for details.
### Create connection
When using the node.js connector, you must execute <em>require</em> `td2.0-connector`, and then use the `taos.connect` function. The parameter that `taos.connect` function must provide is `host`, and other parameters will use the following default values if they are not provided. Finally, the `cursor` needs to be initialized to communicate with the TDengine server-side.
```javascript
const taos = require('td2.0-connector');
var conn = taos.connect({host:"127.0.0.1", user:"root", password:"taosdata", config:"/etc/taos",port:0})
var cursor = conn.cursor(); // Initializing a new cursor
```
To close the connect:
```javascript
conn.close();
```
#### To execute SQL and insert data
For DDL statements (such as `create database`, `create table`, `use`, and so on), you can use the `execute` method of `cursor`. The code is as follows:
```js
cursor.execute('create database if not exists test;')
```
The above code creates a database named test. For DDL statements, there is generally no return value, and the execute return value of `cursor` is 0.
For Insert statements, the code is as follows:
```js
var affectRows = cursor.execute('insert into test.weather values(now, 22.3, 34);')
```
The return value of the execute method is the number of rows affected by the statement. If the sql above inserts a piece of data into the weather table of the test database, the return value affectRows is 1.
TDengine does not currently support update and delete statements.
#### Query
You can query the database through `cursor.query` function.
```javascript
var query = cursor.query('show databases;')
```
The results of the query can be obtained and printed through `query.execute()` function:
```javascript
var promise = query.execute();
promise.then(function(result) {
result.pretty();
});
```
You can also use the `bind` method of `query` to format query statements. For example: `query` automatically fills the `?` with the value provided in the query statement .
```javascript
var query = cursor.query('select * from meterinfo.meters where ts <= ? and areaid = ?;').bind(new Date(), 5);
query.execute().then(function(result) {
result.pretty();
})
```
If you provide the second parameter in the `query` statement and set it to `true`, you can also get the query results immediately. As follows:
```javascript
var promise = cursor.query('select * from meterinfo.meters where v1 = 30;', true)
promise.then(function(result) {
result.pretty();
})
```
#### Asynchronous function
The operation of asynchronous query database is similar to the above, only by adding `_a` after `cursor.execute`, `TaosQuery.execute` and other functions.
```javascript
var promise1 = cursor.query('select count(*), avg(v1), avg(v2) from meter1;').execute_a()
var promise2 = cursor.query('select count(*), avg(v1), avg(v2) from meter2;').execute_a();
promise1.then(function(result) {
result.pretty();
})
promise2.then(function(result) {
result.pretty();
})
```
### Example
[node-example.js](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example.js) provides a code example that uses the NodeJS connector to create a table, insert weather data, and query the inserted data.
[node-example-raw.js](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example-raw.js) is also a code example that uses the NodeJS connector to create a table, insert weather data, and query the inserted data, but unlike the above, this example only uses cursor.
\ No newline at end of file
documentation20/en/09.connections/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Connections with Other Tools
## <a class="anchor" id="grafana"></a> Grafana
TDengine can quickly integrate with [Grafana](https://www.grafana.com/), an open source data visualization system, to build a data monitoring and alarming system. The whole process does not require any code to write. The contents of the data table in TDengine can be visually showed on DashBoard.
### Install Grafana
TDengine currently supports Grafana 5.2.4 and above. You can download and install the package from Grafana website according to the current operating system. The download address is as follows:
https://grafana.com/grafana/download.
### Configure Grafana
TDengine Grafana plugin is in the /usr/local/taos/connector/grafanaplugin directory.
Taking Centos 7.2 as an example, just copy grafanaplugin directory to /var/lib/grafana/plugins directory and restart Grafana.
```bash
sudo cp -rf /usr/local/taos/connector/grafanaplugin /var/lib/grafana/plugins/tdengine
```
### Use Grafana
#### Configure data source
You can log in the Grafana server (username/password:admin/admin) through localhost:3000, and add data sources through `Configuration -> Data Sources` on the left panel, as shown in the following figure:
![img](page://images/connections/add_datasource1.jpg)
Click `Add data source` to enter the Add Data Source page, and enter TDengine in the query box to select Add, as shown in the following figure:
![img](page://images/connections/add_datasource2.jpg)
Enter the data source configuration page and modify the corresponding configuration according to the default prompt:
![img](page://images/connections/add_datasource3.jpg)
- Host: IP address of any server in TDengine cluster and port number of TDengine RESTful interface (6041), default [http://localhost:6041](http://localhost:6041/)
- User: TDengine username.
- Password: TDengine user password.
Click `Save & Test` to test. Success will be prompted as follows:
![img](page://images/connections/add_datasource4.jpg)
#### Create Dashboard
Go back to the home to create Dashboard, and click `Add Query` to enter the panel query page:
![img](page://images/connections/create_dashboard1.jpg)
As shown in the figure above, select the TDengine data source in Query, and enter the corresponding sql in the query box below to query. Details are as follows:
- INPUT SQL: Enter the statement to query (the result set of the SQL statement should be two columns and multiple rows), for example: `select avg(mem_system) from log.dn where ts >= $from and ts < $to interval($interval)` , where `from`, `to` and `interval` are built-in variables of the TDengine plug-in, representing the query range and time interval obtained from the Grafana plug-in panel. In addition to built-in variables, it is also supported to use custom template variables.
- ALIAS BY: You can set alias for the current queries.
- GENERATE SQL: Clicking this button will automatically replace the corresponding variable and generate the final statement to execute.
According to the default prompt, query the average system memory usage at the specified interval of the server where the current TDengine deployed in as follows:
![img](page://images/connections/create_dashboard2.jpg)
> Please refer to Grafana [documents](https://grafana.com/docs/) for how to use Grafana to create the corresponding monitoring interface and for more about Grafana usage.
#### Import Dashboard
A `tdengine-grafana.json` importable dashboard is provided under the Grafana plug-in directory/usr/local/taos/connector/grafana/tdengine/dashboard/.
Click the `Import` button on the left panel and upload the `tdengine-grafana.json` file:
![img](page://images/connections/import_dashboard1.jpg)
You can see as follows after Dashboard imported.
![img](page://images/connections/import_dashboard2.jpg)
## <a class="anchor" id="matlab"></a> Matlab
MatLab can access data to the local workspace by connecting directly to the TDengine via the JDBC Driver provided in the installation package.
### JDBC Interface Adaptation of MatLab
Several steps are required to adapt Matlab to TDengine. Taking adapting Matlab2017a on Windows10 as an example:
- Copy the file JDBCDriver-1.0.0-dist.ja*r* in TDengine package to the directory ${matlab_root}\MATLAB\R2017a\java\jar\toolbox
- Copy the file taos.lib in TDengine package to ${matlab root dir}\MATLAB\R2017a\lib\win64
- Add the .jar package just copied to the Matlab classpath. Append the line below as the end of the file of ${matlab root dir}\MATLAB\R2017a\toolbox\local\classpath.txt
- ```
$matlabroot/java/jar/toolbox/JDBCDriver-1.0.0-dist.jar
```
- Create a file called javalibrarypath.txt in directory ${user_home}\AppData\Roaming\MathWorks\MATLAB\R2017a_, and add the _taos.dll path in the file. For example, if the file taos.dll is in the directory of C:\Windows\System32，then add the following line in file javalibrarypath.txt:
- ```
C:\Windows\System32
```
- ### Connect to TDengine in MatLab to get data
After the above configured successfully, open MatLab.
- Create a connection:
```matlab
conn = database(‘db’, ‘root’, ‘taosdata’, ‘com.taosdata.jdbc.TSDBDriver’, ‘jdbc:TSDB://127.0.0.1:0/’)
```
* Make a query:
```matlab
sql0 = [‘select * from tb’]
data = select(conn, sql0);
```
* Insert a record:
```matlab
sql1 = [‘insert into tb values (now, 1)’]
exec(conn, sql1)
```
For more detailed examples, please refer to the examples\Matlab\TDEngineDemo.m file in the package.
## <a class="anchor" id="r"></a> R
R language supports connection to the TDengine database through the JDBC interface. First, you need to install the JDBC package of R language. Launch the R language environment, and then execute the following command to install the JDBC support library for R language:
```R
install.packages('RJDBC', repos='http://cran.us.r-project.org')
```
After installed, load the RJDBC package by executing `library('RJDBC')` command.
Then load the TDengine JDBC driver:
```R
drv<-JDBC("com.taosdata.jdbc.TSDBDriver","JDBCDriver-2.0.0-dist.jar", identifier.quote="\"")
```
If succeed, no error message will display. Then use the following command to try a database connection:
```R
conn<-dbConnect(drv,"jdbc:TSDB://192.168.0.1:0/?user=root&password=taosdata","root","taosdata")
```
Please replace the IP address in the command above to the correct one. If no error message is shown, then the connection is established successfully, otherwise the connection command needs to be adjusted according to the error prompt. TDengine supports below functions in *RJDBC* package:
- `dbWriteTable(conn, "test", iris, overwrite=FALSE, append=TRUE)`: Write the data in a data frame iris to the table test in the TDengine server. Parameter overwrite must be false. append must be TRUE and the schema of the data frame iris should be the same as the table test.
- `dbGetQuery(conn, "select count(*) from test")`: run a query command
- `dbSendUpdate (conn, "use db")`: Execute any non-query sql statement. For example, `dbSendUpdate (conn, "use db")`, write data `dbSendUpdate (conn, "insert into t1 values (now, 99)")`, and the like.
- `dbReadTable(conn, "test")`: read all the data in table test
- `dbDisconnect(conn)`: close a connection
- `dbRemoveTable(conn, "test")`: remove table test
The functions below are not supported currently:
- `dbExistsTable(conn, "test")`: if table test exists
- `dbListTables(conn)`: list all tables in the connection
\ No newline at end of file
documentation20/en/10.cluster/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# Installation and Management of TDengine Cluster
Multiple TDengine servers, that is, multiple running instances of taosd, can form a cluster to ensure the highly reliable operation of TDengine and provide scale-out features. To understand cluster management in TDengine 2.0, it is necessary to understand the basic concepts of clustering. Please refer to the chapter "Overall Architecture of TDengine 2.0". And before installing the cluster, please follow the chapter ["Getting started"](https://www.taosdata.com/en/documentation/getting-started/) to install and experience the single node function.
Each data node of the cluster is uniquely identified by End Point, which is composed of FQDN (Fully Qualified Domain Name) plus Port, such as [h1.taosdata.com](http://h1.taosdata.com/):6030. The general FQDN is the hostname of the server, which can be obtained through the Linux command `hostname -f` (how to configure FQDN, please refer to: [All about FQDN of TDengine](https://www.taosdata.com/blog/2020/09/11/1824.html)). Port is the external service port number of this data node. The default is 6030, but it can be modified by configuring the parameter serverPort in taos.cfg. A physical node may be configured with multiple hostnames, and TDengine will automatically get the first one, but it can also be specified through the configuration parameter fqdn in taos.cfg. If you are accustomed to direct IP address access, you can set the parameter fqdn to the IP address of this node.
The cluster management of TDengine is extremely simple. Except for manual intervention in adding and deleting nodes, all other tasks are completed automatically, thus minimizing the workload of operation. This chapter describes the operations of cluster management in detail.
Please refer to the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1961.html) for cluster building.
## <a class="anchor" id="prepare"></a> Preparation
**Step 0:** Plan FQDN of all physical nodes in the cluster, and add the planned FQDN to /etc/hostname of each physical node respectively; modify the /etc/hosts of each physical node, and add the corresponding IP and FQDN of all cluster physical nodes. [If DNS is deployed, contact your network administrator to configure it on DNS]
**Step 1:** If the physical nodes have previous test data, installed with version 1. x, or installed with other versions of TDengine, please delete it first and drop all data. For specific steps, please refer to the blog "[Installation and Uninstallation of Various Packages of TDengine](https://www.taosdata.com/blog/2019/08/09/566.html)"
**Note 1:** Because the information of FQDN will be written into a file, if FQDN has not been configured or changed before, and TDengine has been started, be sure to clean up the previous data （`rm -rf /var/lib/taos/*`）on the premise of ensuring that the data is useless or backed up;
**Note 2:** The client also needs to be configured to ensure that it can correctly parse the FQDN configuration of each node, whether through DNS service or Host file.
**Step 2:** It is recommended to close the firewall of all physical nodes, and at least ensure that the TCP and UDP ports of ports 6030-6042 are open. It is **strongly recommended** to close the firewall first and configure the ports after the cluster is built;
**Step 3:** Install TDengine on all physical nodes, and the version must be consistent, **but do not start taosd**. During installation, when prompted to enter whether to join an existing TDengine cluster, press enter for the first physical node directly to create a new cluster, and enter the FQDN: port number (default 6030) of any online physical node in the cluster for the subsequent physical nodes;
**Step 4:** Check the network settings of all data nodes and the physical nodes where the application is located:
1. Execute command `hostname -f` on each physical node, and check and confirm that the hostnames of all nodes are different (the node where the application driver is located does not need to do this check).
2. Execute `ping host` on each physical node, wherein host is that hostname of other physical node, and see if other physical nodes can be communicated to; if not, you need to check the network settings, or the /etc/hosts file (the default path for Windows systems is C:\ Windows\ system32\ drivers\ etc\ hosts), or the configuration of DNS. If it fails to ping, then we cann't build the cluster.
3. From the physical node where the application runs, ping the data node where taosd runs. If the ping fails, the application cannot connect to taosd. Please check the DNS settings or hosts file of the physical node where the application is located;
4. The End Point of each data node is the output hostname plus the port number, for example, [h1.taosdata.com](http://h1.taosdata.com/): 6030
**Step 5:** Modify the TDengine configuration file (the file/etc/taos/taos.cfg for all nodes needs to be modified). Assume that the first data node End Point to be started is [h1.taosdata.com](http://h1.taosdata.com/): 6030, and its parameters related to cluster configuration are as follows:
```
// firstEp is the first data node connected after each data node’s first launch
firstEp h1.taosdata.com:6030
// Must configure it as the FQDN of this data node. If this machine has only one hostname, you can comment out this configuration
fqdn h1.taosdata.com
// Configure the port number of this data node, the default is 6030
serverPort 6030
// For application scenarios, please refer to the section “Use of Arbitrator”
arbitrator ha.taosdata.com:6042
```
The parameters that must be modified are firstEp and fqdn. At each data node, every firstEp needs to be configured to be the same, **but fqdn must be configured to the value of the data node where it is located**. Other parameters may not be modified unless you have clear reasons.
**The data node dnode added to the cluster must be exactly the same as the 11 parameters in the following table related to the cluster, otherwise it cannot be successfully added to the cluster.**
| **#** | **Configuration Parameter Name** | **Description** |
| ----- | -------------------------------- | ------------------------------------------------------------ |
| 1 | numOfMnodes | Number of management nodes in system |
| 2 | mnodeEqualVnodeNum | A mnode equals to the number of vnodes consumed |
| 3 | offlineThreshold | Offline threshold of dnode to judge if the dnode is offline |
| 4 | statusInterval | The interval for dnode to report its status to mnode |
| 5 | arbitrator | The end point of the arbitrator in system |
| 6 | timezone | Time zone |
| 7 | locale | Location information and coding format of system |
| 8 | charset | Character set encoding |
| 9 | balance | Whether to start load balancing |
| 10 | maxTablesPerVnode | The maximum number of tables that can be created in each vnode |
| 11 | maxVgroupsPerDb | The maximum number of vgroups that can be used per DB |
## <a class="anchor" id="node-one"></a> Launch the First Data Node
Follow the instructions in "[Getting started](https://www.taosdata.com/en/documentation/getting-started/)", launch the first data node, such as [h1.taosdata.com](http://h1.taosdata.com/), then execute taos, start the taos shell, and execute command "show dnodes" from the shell; ", as follows:
```
Welcome to the TDengine shell from Linux, Client Version:2.0.0.0
Copyright (c) 2017 by TAOS Data, Inc. All rights reserved.
taos> show dnodes;
id | end_point | vnodes | cores | status | role | create_time |
=====================================================================================
1 | h1.taos.com:6030 | 0 | 2 | ready | any | 2020-07-31 03:49:29.202 |
Query OK, 1 row(s) in set (0.006385s)
taos>
```
In the above command, you can see that the End Point of the newly launched data node is: [h1.taos.com](http://h1.taos.com/): 6030, which is the firstEP of the new cluster.
## <a class="anchor" id="node-other"></a>Launch Subsequent Data Nodes
To add subsequent data nodes to the existing cluster, there are the following steps:
1. Start taosd at each physical node according to the chapter "[Getting started](https://www.taosdata.com/en/documentation/getting-started/)";
2. On the first data node, use CLI program taos to log in to TDengine system and execute the command:
```
CREATE DNODE "h2.taos.com:6030";
```
Add the End Point of the new data node (learned in Step 4 of the preparation) to the cluster's EP list. **"fqdn: port" needs to be enclosed in double quotation marks**, otherwise an error will occur. Notice that the example "[h2.taos.com](http://h2.taos.com/): 6030" is replaced with the End Point for this new data node.
3. And then execute the command
1. ```
SHOW DNODES;
```
2. Check to see if the new node was successfully joined. If the added data node is offline, then check:
1. - Check whether the taosd of this data node is working properly. If it is not working properly, you need to check the reason first
- Check the first few lines of the data node taosd log file taosdlog.0 (usually in the /var/log/taos directory) to see if the data node fqdn and port number output in the log are the just added End Point. If not, you need to add the correct End Point.
According to the above steps, new data nodes can be continuously added to the cluster.
**Tips**:
- Any data node that has joined the cluster online can be used as the firstEP of the subsequent node to be joined.
- firstEp is only effective when the data node joins the cluster for the first time. After joining the cluster, the data node will save the latest End Point list of mnode and no longer rely on this parameter.
- The two dnode data nodes dnode that are not configured with the firstEp parameter will run independently after startup. At this time, one data node cannot be added to another data node to form a cluster. **You cannot merge two independent clusters into a new cluster**.
## <a class="anchor" id="management"></a> Data Node Management
The above has already introduced how to build clusters from scratch. After the cluster is formed, new data nodes can be added at any time for expansion, or data nodes can be deleted, and the current status of the cluster can be checked.
### Add data nodes
Execute CLI program taos, log in to the system using root account, and execute:
```
CREATE DNODE "fqdn:port";
```
Add the End Point for the new data node to the cluster's EP list. **"fqdn: port" needs to be enclosed in double quotation marks**, otherwise an error will occur. The fqdn and port of a data node's external service can be configured through the configuration file taos.cfg, which is automatically obtained by default. [It is strongly not recommended to configure FQDN with automatic acquisition, which may cause the End Point of the generated data node to be not expected]
### Delete data nodes
Execute the CLI program taos, log in to the TDengine system using the root account, and execute:
```
DROP DNODE "fqdn:port";
```
Where fqdn is the FQDN of the deleted node, and port is the port number of its external server.
<font color=green>**【Note】**</font>
- Once a data node is dropped, it cannot rejoin the cluster. This node needs to be redeployed (emptying the data folder). The cluster migrates the data from the dnode before it completes the drop dnode operation.
- Note that dropping a dnode and stopping the taosd process are two different concepts. Don't be confused: the data migration operation must be performed before deleting a dnode, thus the deleted dnode must remain online. The taosd process cannot be stopped until the delete operation is completed.
- After a data node is dropped, other nodes will perceive the deletion of this dnodeID, and no node in any cluster will receive the request of the dnodeID.
- dnodeID is automatically assigned by the cluster and cannot be specified manually. It is incremented at the time of generation and does not repeat.
### View data nodes
Execute the CLI program taos, log in to the TDengine system using the root account, and execute:
```
SHOW DNODES;
```
All dnodes, fqdn: port for each dnode, status (ready, offline, etc.), number of vnodes, number of unused vnodes in the cluster will be listed. You can use this command to view after adding or deleting a data node.
### View virtual node group
In order to make full use of multi-core technology and provide scalability, data needs to be processed in partitions. Therefore, TDengine will split the data of a DB into multiple parts and store them in multiple vnodes. These vnodes may be distributed in multiple data node dnodes, thus realizing scale-out. A vnode belongs to only one DB, but a DB can have multiple vnodes. vnode is allocated automatically by mnode according to the current system resources without any manual intervention.
Execute the CLI program taos, log in to the TDengine system using the root account, and execute:
```
SHOW VGROUPS;
```
## <a class="anchor" id="high-availability"></a> High-availability of vnode
TDengine provides high-availability of system through a multi-replica mechanism, including high-availability of vnode and mnode.
The number of replicas of vnode is associated with DB. There can be multiple DBs in a cluster. Each DB can be configured with different replicas according to operational requirements. When creating a database, specify the number of replicas with parameter replica (the default is 1). If the number of replicas is 1, the reliability of the system cannot be guaranteed. As long as the node where the data is located goes down, the service cannot be provided. The number of nodes in the cluster must be greater than or equal to the number of replicas, otherwise the error "more dnodes are needed" will be returned when creating a table. For example, the following command will create a database demo with 3 replicas:
```
CREATE DATABASE demo replica 3;
```
The data in a DB will be partitioned and splitted into multiple vnode groups. The number of vnodes in a vnode group is the number of replicas of the DB, and the data of each vnode in the same vnode group is completely consistent. In order to ensure high-availability, the vnodes in a vnode group must be distributed in different dnode data nodes (in actual deployment, they need to be on different physical machines). As long as more than half of the vnodes in a vgroup are working, the vgroup can be normally serving.
There may be data from multiple DBs of data in a data node dnode, so when a dnode is offline, it may affect multiple DBs. If half or more of the vnodes in a vnode group do not work, then the vnode group cannot serve externally and cannot insert or read data, which will affect the reading and writing operations of some tables in the DB to which it belongs.
Because of the introduction of vnode, it is impossible to simply draw a conclusion: "If more than half of the data nodes in the cluster work in dnode, the cluster should work." But for simple cases, it is easier to judge. For example, if the number of replicas is 3 and there are only 3 dnodes, the whole cluster can still work normally if only one node does not work, but if two data nodes do not work, the whole cluster cannot work normally.
## <a class="anchor" id="mnode"></a> High-availability of mnode
TDengine cluster is managed by mnode (a module of taosd, management node). In order to ensure the high-availability of mnode, multiple mnode replicas can be configured. The number of replicas is determined by system configuration parameter numOfMnodes, and the effective range is 1-3. In order to ensure the strong consistency of metadata, mnode replicas are duplicated synchronously.
A cluster has multiple data node dnodes, but a dnode runs at most one mnode instance. In the case of multiple dnodes, which dnode can be used as an mnode? This is automatically specified by the system according to the resource situation on the whole. User can execute the following command in the console of TDengine through the CLI program taos:
```
SHOW MNODES;
```
To view the mnode list, which lists the End Point and roles (master, slave, unsynced, or offline) of the dnode where the mnode is located. When the first data node in the cluster starts, the data node must run an mnode instance, otherwise the dnode of the data node cannot work properly because a system must have at least one mnode. If numOfMnodes is configured to 2, when the second dnode is started, the latter will also run an mnode instance.
To ensure the high-availability of mnode service, numOfMnodes must be set to 2 or greater. Because the metadata saved by mnode must be strongly consistent, if numOfMnodes is greater than 2, the duplication parameter quorum is automatically set to 2, that is to say, at least two replicas must be guaranteed to write the data successfully before notifying the client application of successful writing.
**Note:** A TDengine highly-available system, whether vnode or mnode, must be configured with multiple replicas.
## <a class="anchor" id="load-balancing"></a> Load Balancing
There are three situations in which load balancing will be triggered, and no manual intervention is required.
- When a new data node is added to the cluster, the system will automatically trigger load balancing, and the data on some nodes will be automatically migrated to the new data node without any manual intervention.
- When a data node is removed from the cluster, the system will automatically migrate the data on the data node to other data nodes without any manual intervention.
- If a data node is overheated (too large amount of data), the system will automatically load balance and migrate some vnodes of the data node to other nodes.
When the above three situations occur, the system will start a load computing of each data node to decide how to migrate.
**[Tip] Load balancing is controlled by parameter balance, which determines whether to start automatic load balancing.**
## <a class="anchor" id="offline"></a> Offline Processing of Data Nodes
If a data node is offline, the TDengine cluster will automatically detect it. There are two detailed situations:
- If the data node is offline for more than a certain period of time (configuration parameter offlineThreshold in taos.cfg controls the duration), the system will automatically delete the data node, generate system alarm information and trigger the load balancing process. If the deleted data node is online again, it will not be able to join the cluster, and the system administrator will need to add it to the cluster again.
- After offline, the system will automatically start the data recovery process if it goes online again within the duration of offlineThreshold. After the data is fully recovered, the node will start to work normally.
**Note:** If each data node belonging to a virtual node group (including mnode group) is in offline or unsynced state, Master can only be elected after all data nodes in the virtual node group are online and can exchange status information, and the virtual node group can serve externally. For example, the whole cluster has 3 data nodes with 3 replicas. If all 3 data nodes go down and then 2 data nodes restart, it will not work. Only when all 3 data nodes restart successfully can serve externally again.
## <a class="anchor" id="arbitrator"></a> How to Use Arbitrator
If the number of replicas is even, it is impossible to elect a master from a vnode group when half of the vnodes are not working. Similarly, when half of the mnodes are not working, the master of the mnode cannot be elected because of the "split brain" problem. To solve this problem, TDengine introduced the concept of Arbitrator. Arbitrator simulates a vnode or mnode working, but is simply responsible for networking, and does not handle any data insertion or access. As long as more than half of the vnodes or mnodes, including the Arbitrator, work, the vnode group or mnode group can normally provide data insertion or query services. For example, in the case of 2 replicas, if one node A is offline, but the other node B is normal on and can connect to the Arbitrator, then node B can work normally.
In a word, under the current version, TDengine recommends configuring Arbitrator in double-replica environment to improve the availability.
The name of the executable for Arbitrator is tarbitrator. The executable has almost no requirements for system resources, just need to ensure a network connection, with any Linux server to run it. The following briefly describes the steps to install the configuration:
1. Click [Package Download](https://www.taosdata.com/cn/all-downloads/), and in the TDengine Arbitrator Linux section, select the appropriate version to download and install.
2. The command line parameter -p of this application can specify the port number of its external service, and the default is 6042.
3. Modify the configuration file of each taosd instance, and set parameter arbitrator to the End Point corresponding to the tarbitrator in taos.cfg. (If this parameter is configured, when the number of replicas is even, the system will automatically connect the configured Arbitrator. If the number of replicas is odd, even if the Arbitrator is configured, the system will not establish a connection.)
4. The Arbitrator configured in the configuration file will appear in the return result of instruction `SHOW DNODES`; the value of the corresponding role column will be "arb".
documentation20/en/11.administrator/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# TDengine Operation and Maintenance
## <a class="anchor" id="planning"></a> Capacity Planing
Using TDengine to build an IoT big data platform, computing resource and storage resource need to be planned according to business scenarios. The following is a discussion of the memory, CPU and hard disk space required for the system to run.
### Memory requirements
Each DB can create a fixed number of vgroups, which is the same as the CPU cores by default and can be configured by maxVgroupsPerDb; each replica in the vgroup would be a vnode; each vnode takes up a fixed amount of memory (the size is related to the database's configuration parameters blocks and cache); each table takes up memory related to the total length of the tag; in addition, the system will have some fixed memory overhead. Therefore, the system memory required for each DB can be calculated by the following formula:
```
Database Memory Size = maxVgroupsPerDb * (blocks * cache + 10MB) + numOfTables * (tagSizePerTable + 0.5KB)
```
Example: Assuming a 4-core machine, cache is the default size of 16M, blocks is the default value of 6, assuming there are 100,000 tables, and the total tag length is 256 bytes, the total memory requirement is: 4 * (16 * 6 + 10) + 100,000 * (0.25 + 0.5)/1000 = 499M.
The actual running system often stores the data in different DBs according to different characteristics of the data. All these shall be considered when planning.
If there is plenty of memory, the configuration of Blocks can be increased so that more data will be stored in memory and the query speed will be improved.
### CPU requirements
CPU requirements depend on the following two aspects:
- **Data insertion** TDengine single core can handle at least 10,000 insertion requests per second. Each insertion request can take multiple records, and inserting one record at a time is almost the same as inserting 10 records in computing resources consuming. Therefore, the larger the number of inserts, the higher the insertion efficiency. If an insert request has more than 200 records, a single core can insert 1 million records per second. However, the faster the insertion speed, the higher the requirement for front-end data collection, because records need to be cached and then inserted in batches.
- **Query requirements** TDengine to provide efficient queries, but the queries in each scenario vary greatly and the query frequency too, making it difficult to give objective figures. Users need to write some query statements for their own scenes to determine.
Therefore, only for data insertion, CPU can be estimated, but the computing resources consumed by query cannot be that clear. In the actual operation, it is not recommended to make CPU utilization rate over 50%. After that, new nodes need to be added to bring more computing resources.
### Storage requirements
Compared with general databases, TDengine has an ultra-high compression ratio. In most scenarios, the compression ratio of TDengine will not be less than 5:1, and in some scenarios, maybe over 10:1, depending on the actual data characteristics. The raw data size before compressed can be calculated as follows:
```
Raw DataSize = numOfTables * rowSizePerTable * rowsPerTable
```
Example: 10 million smart meters, each meter collects data every 15 minutes, and the data collected each time is 128 bytes, so the original data amount in one year is: 10000000 * 128 * 24 * 60/15 * 365 = 44.8512 T. The TDengine consumes approximately 44.851/5 = 8.97024 T.
User can set the maximum retention time of data on disk through parameter `keep`. In order to further reduce the storage cost, TDengine also provides tiered storage. The coldest data can be stored on the cheapest storage media. Application access does not need to be adjusted, but lower reading speed.
To improve speed, multiple hard disks can be configured so that data can be written or read concurrently. It should be reminded that TDengine provides high reliability of data in the form of multiple replicas, so it is no longer necessary to use expensive disk arrays.
### Number of physical or virtual machines
According to the above estimation of memory, CPU and storage, we can know how many cores, how much memory and storage space the whole system needs. If the number of data replicas is not 1, the total demand needs to be multiplied by the number of replicas.
Because TDengine provides great scale-out feature, it is easy to decide how many physical or virtual machines need to be purchased according to the total amount and the resources of a single physical/ virtual machine.
**Calculate CPU, memory and storage immediately, see:** [**Resource Estimation**](https://www.taosdata.com/config/config.html)
### Fault Tolerance and Disaster Recovery
### Fault tolerance
TDengine supports WAL (Write Ahead Log) mechanism to realize fault tolerance of data and ensure high-availability of data.
When TDengine receives the application's request packet, it first writes the requested original packet into the database log file, and then deletes the corresponding WAL after the data is successfully written. This ensures that TDengine can recover data from the database log file when the service is restarted due to power failure or other reasons, thus avoiding data loss.
There are two system configuration parameters involved:
- walLevel: WAL level, 0: do not write wal; 1: write wal, but do not execute fsync; 2: write wal and execute fsync.
- fsync: the cycle in which fsync is executed when walLevel is set to 2. Setting to 0 means that fsync is executed immediately whenever there is a write.
To guarantee 100% data safe, you need to set walLevel to 2 and fsync to 0. In that way, the write speed will decrease. However, if the number of threads starting to write data on the application side reaches a certain number (more than 50), the performance of writing data will also be good, only about 30% lower than that of fsync set to 3000 milliseconds.
### Disaster recovery
The cluster of TDengine provides high-availability of the system and implements disaster recovery through the multipl-replica mechanism.
TDengine cluster is managed by mnode. In order to ensure the high reliability of the mnode, multiple mnode replicas can be configured. The number of replicas is determined by system configuration parameter numOfMnodes. In order to support high reliability, it needs to be set to be greater than 1. In order to ensure the strong consistency of metadata, mnode replicas duplicate data synchronously to ensure the strong consistency of metadata.
The number of replicas of time-series data in TDengine cluster is associated with databases. There can be multiple databases in a cluster, and each database can be configured with different replicas. When creating a database, specify the number of replicas through parameter replica. In order to support high reliability, it is necessary to set the number of replicas greater than 1.
The number of nodes in TDengine cluster must be greater than or equal to the number of replicas, otherwise an error will be reported in table creation.
When the nodes in TDengine cluster are deployed on different physical machines and multiple replicas are set, the high reliability of the system is implemented without using other software or tools. TDengine Enterprise Edition can also deploy replicas in different server rooms, thus realizing remote disaster recovery.
## <a class="anchor" id="config"></a> Server-side Configuration
The background service of TDengine system is provided by taosd, and the configuration parameters can be modified in the configuration file taos.cfg to meet the requirements of different scenarios. The default location of the configuration file is the /etc/taos directory, which can be specified by executing the parameter -c from the taosd command line. Such as taosd-c/home/user, to specify that the configuration file is located in the /home/user directory.
You can also use “-C” to show the current server configuration parameters:
```
taosd -C
```
Only some important configuration parameters are listed below. For more parameters, please refer to the instructions in the configuration file. Please refer to the previous chapters for detailed introduction and function of each parameter, and the default of these parameters is working and generally does not need to be set. **Note: After the configuration is modified, \*taosd service\* needs to be restarted to take effect.**
- firstEp: end point of the first dnode in the actively connected cluster when taosd starts, the default value is localhost: 6030.
- fqdn: FQDN of the data node, which defaults to the first hostname configured by the operating system. If you are accustomed to IP address access, you can set it to the IP address of the node.
- serverPort: the port number of the external service after taosd started, the default value is 6030.
- httpPort: the port number used by the RESTful service to which all HTTP requests (TCP) require a query/write request. The default value is 6041.
- dataDir: the data file directory to which all data files will be written. [Default:/var/lib/taos](http://default/var/lib/taos).
- logDir: the log file directory to which the running log files of the client and server will be written. [Default:/var/log/taos](http://default/var/log/taos).
- arbitrator: the end point of the arbiter in the system; the default value is null.
- role: optional role for dnode. 0-any; it can be used as an mnode and to allocate vnodes; 1-mgmt; It can only be an mnode, but not to allocate vnodes; 2-dnode; caannot be an mnode, only vnode can be allocated
- debugFlage: run the log switch. 131 (output error and warning logs), 135 (output error, warning, and debug logs), 143 (output error, warning, debug, and trace logs). Default value: 131 or 135 (different modules have different default values).
- numOfLogLines: the maximum number of lines allowed for a single log file. Default: 10,000,000 lines.
- logKeepDays: the maximum retention time of the log file. When it is greater than 0, the log file will be renamed to taosdlog.xxx, where xxx is the timestamp of the last modification of the log file in seconds. Default: 0 days.
- maxSQLLength: the maximum length allowed for a single SQL statement. Default: 65380 bytes.
- telemetryReporting: whether TDengine is allowed to collect and report basic usage information. 0 means not allowed, and 1 means allowed. Default: 1.
- stream: whether continuous query (a stream computing function) is enabled, 0 means not allowed, 1 means allowed. Default: 1.
- queryBufferSize: the amount of memory reserved for all concurrent queries. The calculation rule can be multiplied by the number of the table according to the maximum possible concurrent number in practical application, and then multiplied by 170. The unit is MB (in versions before 2.0. 15, the unit of this parameter is byte).
- ratioOfQueryCores: set the maximum number of query threads. The minimum value of 0 means that there is only one query thread; the maximum value of 2 indicates that the maximum number of query threads established is 2 times the number of CPU cores. The default is 1, which indicates the maximum number of query threads equals to the number of CPU cores. This value can be a decimal, that is, 0.5 indicates that the query thread with half of the maximum CPU cores is established.
**Note:** for ports, TDengine will use 13 continuous TCP and UDP port numbers from serverPort, so be sure to open them in the firewall. Therefore, if it is the default configuration, a total of 13 ports from 6030 to 6042 need to be opened, and the same for both TCP and UDP.
Data in different application scenarios often have different data characteristics, such as retention days, number of replicas, collection frequency, record size, number of collection points, compression, etc. In order to obtain the best efficiency in storage, TDengine provides the following storage-related system configuration parameters:
- days: the time span for a data file to store data, in days, the default value is 10.
- keep: the number of days to keep data in the database, in days, default value: 3650.
- minRows: the minimum number of records in a file block, in pieces, default: 100.
- maxRows: the maximum number of records in a file block, in pieces, default: 4096.
- comp: file compression flag bit, 0: off; 1: one-stage compression; 2: two-stage compression. Default: 2.
- walLevel: WAL level. 1: write wal, but do not execute fsync; 2: write wal and execute fsync. Default: 1.
- fsync: the period during which fsync is executed when wal is set to 2. Setting to 0 means that fsync is executed immediately whenever a write happens, in milliseconds, and the default value is 3000.
- cache: the size of the memory block in megabytes (MB), default: 16.
- blocks: how many cache-sized memory blocks are in each VNODE (TSDB). Therefore, the memory size used by a VNODE is roughly (cache * blocks), in blocks, and the default value is 4.
- replica: number of replicas; value range: 1-3, in items, default value: 1
- precision: timestamp precision identification, ms for milliseconds and us for microseconds. Default: ms
- cacheLast: whether the sub-table last_row is cached in memory, 0: off; 1: on. Default: 0. (This parameter is supported as of version 2.0. 11)
For an application scenario, there may be data with multiple characteristics coexisting. The best design is to put tables with the same data characteristics in one database. Such an application has multiple databases, and each one can be configured with different storage parameters, thus ensuring the optimal performance of the system. TDengine allows the application to specify the above storage parameter in database creation. If specified, the parameters will override the corresponding system configuration parameters. For example, there is the following SQL:
```
create database demo days 10 cache 32 blocks 8 replica 3 update 1;
```
The SQL creates a database demo, each data file stores 10 days of data, the memory block is 32 megabytes, each VNODE occupies 8 memory blocks, the number of replicas is 3, updates are allowed, and other parameters are completely consistent with the system configuration.
When adding a new dnode to the TDengine cluster, some parameters related to the cluster must be the same as the configuration of the existing cluster, otherwise it cannot be successfully added to the cluster. The parameters that will be verified are as follows:
- numOfMnodes: the number of management nodes in the system. Default: 3.
- balance: whether to enable load balancing. 0: No, 1: Yes. Default: 1.
- mnodeEqualVnodeNum: an mnode is equal to the number of vnodes consumed. Default: 4.
- offlineThreshold: the threshold for a dnode to be offline, exceed which the dnode will be removed from the cluster. The unit is seconds, and the default value is 86400*10 (that is, 10 days).
- statusInterval: the length of time dnode reports status to mnode. The unit is seconds, and the default value is 1.
- maxTablesPerVnode: the maximum number of tables that can be created in each vnode. Default: 1000000.
- maxVgroupsPerDb: the maximum number of vgroups that can be used in each database.
- arbitrator: the end point of the arbiter in system, which is empty by default.
- See Client Configuration for the configuration of timezone, locale and charset.
For the convenience of debugging, the log configuration of each dnode can be temporarily adjusted through SQL statements, and all will be invalid after system restarting:
```mysql
ALTER DNODE <dnode_id> <config>
```
- dnode_id: available from the SQL statement "SHOW DNODES" command
- config: the log parameter to be adjusted, and the value is taken in the following list
resetlog truncates the old log file and creates a new log file debugFlag < 131 135 143 > Set debugFlag to 131, 135 or 143.
For example:
```
alter dnode 1 debugFlag 135;
```
## <a class="anchor" id="client"></a> Client Configuration
The foreground interactive client application of TDengine system is taos and application driver, which shares the same configuration file taos.cfg with taosd. When running taos, use the parameter -c to specify the configuration file directory, such as taos-c/home/cfg, which means using the parameters in the taos.cfg configuration file under the /home/cfg/ directory. The default directory is /etc/taos. For more information on how to use taos, see the help information taos --help. This section mainly describes the parameters used by the taos client application in the configuration file taos.cfg.
**Versions after 2.0. 10.0 support the following parameters on command line to display the current client configuration parameters**
```bash
taos -C 或 taos --dump-config
```
Client configuration parameters:
- firstEp: end point of the first taosd instance in the actively connected cluster when taos is started, the default value is localhost: 6030.
- secondEp: when taos starts, if not impossible to connect to firstEp, it will try to connect to secondEp.
- locale
Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API
TDengine provides a special field type nchar for storing non-ASCII encoded wide characters such as Chinese, Japanese and Korean. The data written to the nchar field will be uniformly encoded in UCS4-LE format and sent to the server. It should be noted that the correctness of coding is guaranteed by the client. Therefore, if users want to normally use nchar fields to store non-ASCII characters such as Chinese, Japanese, Korean, etc., it’s needed to set the encoding format of the client correctly.
The characters inputted by the client are all in the current default coding format of the operating system, mostly UTF-8 on Linux systems, and some Chinese system codes may be GB18030 or GBK, etc. The default encoding in the docker environment is POSIX. In the Chinese versions of Windows system, the code is CP936. The client needs to ensure that the character set it uses is correctly set, that is, the current encoded character set of the operating system running by the client, in order to ensure that the data in nchar is correctly converted into UCS4-LE encoding format.
The naming rules of locale in Linux are: < language > _ < region >. < character set coding >, such as: zh_CN.UTF-8, zh stands for Chinese, CN stands for mainland region, and UTF-8 stands for character set. Character set encoding provides a description of encoding transformations for clients to correctly parse local strings. Linux system and Mac OSX system can determine the character encoding of the system by setting locale. Because the locale used by Windows is not the POSIX standard locale format, another configuration parameter charset is needed to specify the character encoding under Windows. You can also use charset to specify character encoding in Linux systems.
- charset
Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API
If charset is not set in the configuration file, in Linux system, when taos starts up, it automatically reads the current locale information of the system, and parses and extracts the charset encoding format from the locale information. If the automatic reading of locale information fails, an attempt is made to read the charset configuration, and if the reading of the charset configuration also fails, the startup process is interrupted.
In Linux system, locale information contains character encoding information, so it is unnecessary to set charset separately after setting locale of Linux system correctly. For example:
```
locale zh_CN.UTF-8
```
- On Windows systems, the current system encoding cannot be obtained from locale. If string encoding information cannot be read from the configuration file, taos defaults to CP936. It is equivalent to adding the following to the configuration file:
```
charset CP936
```
- If you need to adjust the character encoding, check the encoding used by the current operating system and set it correctly in the configuration file.
In Linux systems, if user sets both locale and charset encoding charset, and the locale and charset are inconsistent, the value set later will override the value set earlier.
```
locale zh_CN.UTF-8
charset GBK
```
- The valid value for charset is GBK.
And the valid value for charset is UTF-8.
The configuration parameters of log are exactly the same as those of server.
- timezone
Default value: get the current time zone option dynamically from the system
The time zone in which the client runs the system. In order to deal with the problem of data writing and query in multiple time zones, TDengine uses Unix Timestamp to record and store timestamps. The characteristics of UNIX timestamps determine that the generated timestamps are consistent at any time regardless of any time zone. It should be noted that UNIX timestamps are converted and recorded on the client side. In order to ensure that other forms of time on the client are converted into the correct Unix timestamp, the correct time zone needs to be set.
In Linux system, the client will automatically read the time zone information set by the system. Users can also set time zones in profiles in a number of ways. For example:
```
timezone UTC-8
timezone GMT-8
timezone Asia/Shanghai
```
- All above are legal to set the format of the East Eight Zone.
The setting of time zone affects the content of non-Unix timestamp (timestamp string, parsing of keyword now) in query and writing SQL statements. For example:
```sql
SELECT count(*) FROM table_name WHERE TS<'2019-04-11 12:01:08';
```
- In East Eight Zone, the SQL statement is equivalent to
```sql
SELECT count(*) FROM table_name WHERE TS<1554955268000;
```
-
In the UTC time zone, the SQL statement is equivalent to
```sql
SELECT count(*) FROM table_name WHERE TS<1554984068000;
```
-
In order to avoid the uncertainty caused by using string time format, Unix timestamp can also be used directly. In addition, timestamp strings with time zones can also be used in SQL statements, such as: timestamp strings in RFC3339 format, 2013-04-12T15: 52: 01.123 +08:00, or ISO-8601 format timestamp strings 2013-04-12T15: 52: 01.123 +0800. The conversion of the above two strings into Unix timestamps is not affected by the time zone in which the system is located.
When starting taos, you can also specify an end point for an instance of taosd from the command line, otherwise read from taos.cfg.
- maxBinaryDisplayWidth
The upper limit of the display width of binary and nchar fields in a shell, beyond which parts will be hidden. Default: 30. You can modify this option dynamically in the shell with the command set max_binary_display_width nn.
## <a class="anchor" id="user"></a>User Management
System administrators can add and delete users in CLI, and also modify passwords. The SQL syntax in the CLI is as follows:
```sql
CREATE USER <user_name> PASS <'password'>;
```
Create a user, and specify the user name and password. The password needs to be enclosed in single quotation marks. The single quotation marks are in English half-width.
```sql
DROP USER <user_name>;
```
Delete a user, root only.
```sql
ALTER USER <user_name> PASS <'password'>;
```
Modify the user password. In order to avoid being converted to lowercase, the password needs to be quoted in single quotation marks. The single quotation marks are in English half-width
```sql
ALTER USER <user_name> PRIVILEGE <write|read>;
```
Modify the user privilege to: write or read, without adding single quotation marks.
Note: There are three privilege levels: super/write/read in the system, but it is not allowed to give super privilege to users through alter instruction at present.
```mysql
SHOW USERS;
```
Show all users
**Note:** In SQL syntax, < > indicates the part that requires user to input, but do not enter < > itself
## <a class="anchor" id="import"></a> Import Data
TDengine provides a variety of convenient data import functions, including imported by script file, by data file, and by taosdump tool.
**Import by script file**
TDengine shell supports source filename command, which is used to run SQL statements from a file in batch. Users can write SQL commands such as database building, table building and data writing in the same file. Each command has a separate line. By running source command in the shell, SQL statements in the file can be run in batches in sequence. SQL statements beginning with '#' are considered comments and are automatically ignored by the shell.
**Import by data file**
TDengine also supports data import from CSV files on existing tables in the shell. The CSV file belongs to only one table, and the data format in the CSV file should be the same as the structure of the table to be imported. When importing, its syntax is as follows:
```mysql
insert into tb1 file 'path/data.csv';
```
Note: if there is descriptive information in the first line of the CSV file, please delete it manually before importing
For example, there is now a sub-table d1001 whose table structure is as follows:
```mysql
taos> DESCRIBE d1001
Field | Type | Length | Note |
=================================================================================
ts | TIMESTAMP | 8 | |
current | FLOAT | 4 | |
voltage | INT | 4 | |
phase | FLOAT | 4 | |
location | BINARY | 64 | TAG |
groupid | INT | 4 | TAG |
```
And the format of the data.csv to import is as follows:
```csv
'2018-10-04 06:38:05.000',10.30000,219,0.31000
'2018-10-05 06:38:15.000',12.60000,218,0.33000
'2018-10-06 06:38:16.800',13.30000,221,0.32000
'2018-10-07 06:38:05.000',13.30000,219,0.33000
'2018-10-08 06:38:05.000',14.30000,219,0.34000
'2018-10-09 06:38:05.000',15.30000,219,0.35000
'2018-10-10 06:38:05.000',16.30000,219,0.31000
'2018-10-11 06:38:05.000',17.30000,219,0.32000
'2018-10-12 06:38:05.000',18.30000,219,0.31000
```
Then we can use the following command to import:
```mysql
taos> insert into d1001 file '~/data.csv';
Query OK, 9 row(s) affected (0.004763s)
```
**Import via taosdump tool**
TDengine provides a convenient database import and export tool, taosdump. Users can import data exported by taosdump from one system into other systems. Please refer to the blog: [User Guide of TDengine DUMP Tool](https://www.taosdata.com/blog/2020/03/09/1334.html).
## <a class="anchor" id="export"></a> Export Data
To facilitate data export, TDengine provides two export methods, namely, export by table and export by taosdump.
**Export CSV file by table**
If user needs to export data from a table or a STable, it can run in a shell
```mysql
select * from <tb_name> >> data.csv;
```
In this way, the data in table tb_name will be exported to the file data.csv in CSV format.
**Export data by taosdump**
TDengine provides a convenient database export tool, taosdump. Users can choose to export all databases, a database or a table in a database, all data or data for a time period, or even just the definition of a table as needed. Please refer to the blog: [User Guide of TDengine DUMP Tool](https://www.taosdata.com/blog/2020/03/09/1334.html)
## <a class="anchor" id="status"></a> System Connection and Task Query Management
The system administrator can query the connection, ongoing query and stream computing of the system from CLI, and can close the connection and stop the ongoing query and stream computing. The SQL syntax in the CLI is as follows:
```mysql
SHOW CONNECTIONS;
```
Show the connection of the database, and one column shows ip: port, which is the IP address and port number of the connection.
```mysql
KILL CONNECTION <connection-id>;
```
Force the database connection to close, where connection-id is the number in the first column displayed in SHOW CONNECTIONS.
```mysql
SHOW QUERIES;
```
Show the data query, where the two numbers separated by colons displayed in the first column are query-id and the connection-id that initiated the query application connection and the number of queries.
```mysql
KILL QUERY <query-id>;
```
Force to close the data query, where query-id is the connection-id: query-no string displayed in SHOW QUERIES, such as "105: 2", copy and paste it.
```mysql
SHOW STREAMS;
```
Show the stream computing, where the first column shows the two numbers separated by colons as stream-id and the connection-id to start the stream application connection and the number of times the stream was initiated.
```mysql
KILL STREAM <stream-id>;
```
Force to turn off the stream computing, in which stream-id is the connection-id: stream-no string displayed in SHOW STREAMS, such as 103: 2, copy and paste it.
## System Monitoring
After TDengine is started, it will automatically create a monitoring database log and write the server's CPU, memory, hard disk space, bandwidth, number of requests, disk read-write speed, slow query and other information into the database regularly. TDengine also records important system operations (such as logging in, creating, deleting databases, etc.) logs and various error alarm information and stores them in the log database. The system administrator can view the database directly from CLI or view the monitoring information through GUI on WEB.
The collection of these monitoring metrics is turned on by default, but you can modify option enableMonitor in the configuration file to turn it off or on.
## <a class="anchor" id="directories"></a> File Directory Structure
After installing TDengine, the following directories or files are generated in the operating system by default:
| **Directory/File** | **Description** |
| ------------------------- | ------------------------------------------------------------ |
| /usr/local/taos/bin | TEngine’s executable directory. The executables are connected to the/usr/bin directory via softly links. |
| /usr/local/taos/connector | TDengine’s various connector directories. |
| /usr/local/taos/driver | TDengine’s dynamic link library directory. Connect to /usr/lib directory via soft links. |
| /usr/local/taos/examples | TDengine’s application example directory for various languages. |
| /usr/local/taos/include | TDengine’s header files of C interface for externally serving. |
| /etc/taos/taos.cfg | TDengine’s default [configuration files]. |
| /var/lib/taos | TDengine’s default data file directory, where the local can be modified via [configuration files]. |
| /var/log/taos | TDengine’s default log file directory, where the local can be modified via [configuration files]. |
**Executables**
All executables of TDengine are stored in the directory /usr/local/taos/bin by default. Including:
- *taosd*: TDengine server-side executable
- *taos*: TDengine Shell executable
- *taosdump*: A data import/export tool
- remove.sh: uninstall the TDengine script, please execute carefully, and link to rmtaos command in the/usr/bin directory. The TDengine installation directory /usr/local/taos will be removed, but/etc/taos,/var/lib/taos,/var/log/taos will remain.
You can configure different data directories and log directories by modifying system configuration file taos.cfg.
## <a class="anchor" id="keywords"></a> TDengine Parameter Limits and Reserved Keywords
- Database name: cannot contain "." and other special characters, and cannot exceed 32 characters
- Table name: cannot contain "." and other special characters, and cannot exceed 192 characters together with the database name to which it belongs
- Table column name: cannot contain special characters, and cannot exceed 64 characters
- Database name, table name, column name cannot begin with a number
- Number of columns in table: cannot exceed 1024 columns
- Maximum length of record: including 8 bytes as timestamp, no more than 16KB (each column of BINARY/NCHAR type will occupy an additional 2 bytes of storage location)
- Default maximum string length for a single SQL statement: 65480 bytes
- Number of database replicas: no more than 3
- User name: no more than 23 bytes
- User password: no more than 15 bytes
- Number of Tags: no more than 128
- Total length of label: cannot exceed 16K bytes
- Number of records: limited by storage space only
- Number of tables: limited only by the number of nodes
- Number of databases: limited only by the number of nodes
- Number of virtual nodes on a single database: cannot exceed 64
At the moment, TDengine has nearly 200 internal reserved keywords, which cannot be used as database name, table name, STable name, data column name or tag column name regardless of case. The list of these keywords is as follows:
| **List of Keywords** | | | | |
| -------------------- | ----------- | ------------ | ---------- | --------- |
| ABLOCKS | CONNECTIONS | GT | MNODES | SLIDING |
| ABORT | COPY | ID | MODULES | SLIMIT |
| ACCOUNT | COUNT | IF | NCHAR | SMALLINT |
| ACCOUNTS | CREATE | IGNORE | NE | SPREAD |
| ADD | CTIME | IMMEDIATE | NONE | STABLE |
| AFTER | DATABASE | IMPORT | NOT | STABLES |
| ALL | DATABASES | IN | NOTNULL | STAR |
| ALTER | DAYS | INITIALLY | NOW | STATEMENT |
| AND | DEFERRED | INSERT | OF | STDDEV |
| AS | DELIMITERS | INSTEAD | OFFSET | STREAM |
| ASC | DESC | INTEGER | OR | STREAMS |
| ATTACH | DESCRIBE | INTERVAL | ORDER | STRING |
| AVG | DETACH | INTO | PASS | SUM |
| BEFORE | DIFF | IP | PERCENTILE | TABLE |
| BEGIN | DISTINCT | IS | PLUS | TABLES |
| BETWEEN | DIVIDE | ISNULL | PRAGMA | TAG |
| BIGINT | DNODE | JOIN | PREV | TAGS |
| BINARY | DNODES | KEEP | PRIVILEGE | TBLOCKS |
| BITAND | DOT | KEY | QUERIES | TBNAME |
| BITNOT | DOUBLE | KILL | QUERY | TIMES |
| BITOR | DROP | LAST | RAISE | TIMESTAMP |
| BOOL | EACH | LE | REM | TINYINT |
| BOTTOM | END | LEASTSQUARES | REPLACE | TOP |
| BY | EQ | LIKE | REPLICA | TRIGGER |
| CACHE | EXISTS | LIMIT | RESET | UMINUS |
| CASCADE | EXPLAIN | LINEAR | RESTRICT | UPLUS |
| CHANGE | FAIL | LOCAL | ROW | USE |
| CLOG | FILL | LP | ROWS | USER |
| CLUSTER | FIRST | LSHIFT | RP | USERS |
| COLON | FLOAT | LT | RSHIFT | USING |
| COLUMN | FOR | MATCH | SCORES | VALUES |
| COMMA | FROM | MAX | SELECT | VARIABLE |
| COMP | GE | METRIC | SEMI | VGROUPS |
| CONCAT | GLOB | METRICS | SET | VIEW |
| CONFIGS | GRANTS | MIN | SHOW | WAVG |
| CONFLICT | GROUP | MINUS | SLASH | WHERE |
| CONNECTION | | | | |
\ No newline at end of file
documentation20/en/12.taos-sql/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# TAOS SQL
TDengine provides a SQL-style language, TAOS SQL, to insert or query data, and support other common tips. To finish this document, you should have some understanding about SQL.
TAOS SQL is the main tool for users to write and query data to TDengine. TAOS SQL provides a style and mode similar to standard SQL to facilitate users to get started quickly. Strictly speaking, TAOS SQL is not and does not attempt to provide SQL standard syntax. In addition, since TDengine does not provide deletion function for temporal structured data, the relevant function of data deletion is non-existent in TAO SQL.
Let’s take a look at the conventions used for syntax descriptions.
- The content in < > is what the user needs to enter, but do not enter < > itself
- [] indicates that the content is optional, but do not enter [] itself
- "|" means you can select one of multiple choices, but you cannot enter | yourself
- "…" means repeating for as many times
In order to better explain the rules and characteristics of SQL syntax, this document assumes that there is a data set. Take smart meters as an example, each smart meter collects three metrics: current, voltage and phase. It is modeled as follows:
```mysql
taos> DESCRIBE meters;
Field | Type | Length | Note |
=================================================================================
ts | TIMESTAMP | 8 | |
current | FLOAT | 4 | |
voltage | INT | 4 | |
phase | FLOAT | 4 | |
location | BINARY | 64 | TAG |
groupid | INT | 4 | TAG |
```
The data set contains data from four smart meters, which correspond to four sub-tables according to the modeling rules of TDengine, and their names are D1001, D1002, D1003 and D1004 respectively.
## <a class="anchor" id="data-type"></a> Data Types
With TDengine, the most important thing is timestamp. When creating and inserting records and querying history records, you need to specify a timestamp. The timestamp has the following rules:
- Time Format: 'YYYY-MM-DD HH:mm:ss.MS', default in milliseconds. For example,'2017-08-12 18:52:58.128'
- Internal Function **now** : this is the current time of the server
- When inserting a record, if timestamp is NOW, then use the server current time.
- Epch Time: a timestamp value can also be a long integer representing milliseconds since 1970-01-01 08:00:00.000.
- Arithmetic operations can be applied to timestamp. For example: now-2h represents a timestamp which is 2 hours ago from the current server time. Units include u( microsecond), a (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks). In `select * from t1 where ts > now-2w and ts <= now-1w`, which queries data of the whole week before two weeks. To specify the interval of down sampling, you can also use n(calendar month) and y(calendar year) as time units.
Default time precision of TDengine is millisecond, you can change it to microseocnd by setting parameter enableMicrosecond.
In TDengine, the following 10 data types can be used in data model of an ordinary table.
| | **Data Type** | **Bytes** | **Note** |
| ---- | ------------- | --------- | ------------------------------------------------------------ |
| 1 | TIMESTAMP | 8 | Time stamp. Default in milliseconds, and support microseconds. Starting from 1970-01-01 00:00:00. 000 (UTC/GMT), the timing cannot be earlier than this time. |
| 2 | INT | 4 | A nullable integer type with a range of [-2^31+1, 2^31-1 ] |
| 3 | BIGINT | 8 | A nullable integer type with a range of [-2^59, 2^59 ] |
| 4 | FLOAT | 4 | A standard nullable float type with 6 -7 significant digits and a range of [-3.4E38, 3.4E38] |
| 5 | DOUBLE | 8 | A standard nullable double float type with 15-16 significant digits and a range of [-1.7E308, 1.7E308] |
| 6 | BINARY | Custom | Used to record ASCII strings. Theoretically, the maximum length can be 16,374 bytes, but since each row of data can be up to 16K bytes, the actual limit is generally smaller than the theoretical value. Binary only supports string input, and single quotation marks are used at both ends of the string, otherwise all English will be automatically converted to lowercase. When using, the size must be specified. For example, binary (20) defines a string with a maximum length of 20 characters, and each character occupies 1 byte of storage space. In this case, if the user string exceeds 20 bytes, an error will be reported. For single quotation marks in strings, they can be represented by escape character backslash plus single quotation marks, that is\ '. |
| 7 | SMALLINT | 2 | A nullable integer type with a range of [-32767, 32767] |
| 8 | TINYINT | 1 | A nullable integer type with a range of [-127, 127] |
| 9 | BOOL | 1 | Boolean type，{true, false} |
| 10 | NCHAR | Custom | Used to record non-ASCII strings, such as Chinese characters. Each nchar character takes up 4 bytes of storage space. Single quotation marks are used at both ends of the string, and escape characters are required for single quotation marks in the string, that is \’. When nchar is used, the string size must be specified. A column of type nchar (10) indicates that the string of this column stores up to 10 nchar characters, which will take up 40 bytes of space. If the length of the user string exceeds the declared length, an error will be reported. |
**Tips**:
1. TDengine is case-insensitive to English characters in SQL statements and automatically converts them to lowercase for execution. Therefore, the user's case-sensitive strings and passwords need to be enclosed in single quotation marks.
2. Avoid using BINARY type to save non-ASCII type strings, which will easily lead to errors such as garbled data. The correct way is to use NCHAR type to save Chinese characters.
## <a class="anchor" id="management"></a>Database Management
- Create a Database
```mysql
CREATE DATABASE [IF NOT EXISTS] db_name [KEEP keep] [DAYS days] [UPDATE 1];
```
Note:
1. KEEP is how long the data of the database is kept, the default is 3650 days (10 years), and the database will automatically delete the data expired;
2. UPDATE marks the database support updating the same timestamp data;
3. Maximum length of the database name is 33;
4. Maximum length of a SQL statement is 65480 characters;
5. Database has more storage-related configuration parameters, see System Management.
- Show current system parameters
- ```mysql
SHOW VARIABLES;
```
- Use a database
Use/switch database
- Drop a database
Delete a database, all data tables included will be deleted. Please use with caution.
- Modify database parameters
- ```mysql
ALTER DATABASE db_name COMP 2;
```
COMP parameter modifies the database file compression flag bit, with the default value of 2 and the value range is [0, 2]. 0 means no compression, 1 means one-stage compression, and 2 means two-stage compression.
```mysql
ALTER DATABASE db_name REPLICA 2;
```
- REPLICA parameter refers to the number of replicas of the modified database, and the value range is [1, 3]. For use in a cluster, the number of replicas must be less than or equal to the number of DNODE.
```mysql
ALTER DATABASE db_name KEEP 365;
```
- The KEEP parameter refers to the number of days to save a modified data file. The default value is 3650, and the value range is [days, 365000]. It must be greater than or equal to the days parameter value.
```mysql
ALTER DATABASE db_name QUORUM 2;
```
-
QUORUM parameter refers to the number of confirmations required for successful data writing, and the value range is [1, 3]. For asynchronous replication, quorum is set to 1, and the virtual node with master role can confirm it by itself. For synchronous replication, it needs to be at least 2 or greater. In principle, Quorum > = 1 and Quorum < = replica number, which is required when starting a synchronization module instance.
```mysql
ALTER DATABASE db_name BLOCKS 100;
```
- BLOCKS parameter is the number of cache-sized memory blocks in each VNODE (TSDB), so the memory size used for a VNODE equals roughly (cache * blocks). Value range is [3,1000].
```mysql
ALTER DATABASE db_name CACHELAST 0;
```
- CACHELAST parameter controls whether last_row of the data subtable is cached in memory. The default value is 0, and the value range is [0, 1]. Where 0 means not enabled and 1 means enabled. (supported from version 2.0. 11)
Tips: After all the above parameters are modified, show databases can be used to confirm whether the modification is successful.
- Show all databases in system
## <a class="anchor" id="table"></a> Table Management
- Create a table
Note:
1. The first field must be a timestamp, and system will set it as the primary key;
2. The max length of table name is 192;
3. The length of each row of the table cannot exceed 16k characters;
4. Sub-table names can only consist of letters, numbers, and underscores, and cannot begin with numbers
5. ​ 10)If the data type binary or nchar is used, the maximum number of bytes should be specified, such as binary (20), which means 20 bytes;
- Create a table via STable
- ```mysql
CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name TAGS (tag_value1, ...);
```
Use a STable as template and assign tag values to create a data table.
- **Create a data table using STable as a template and specify a specific tags column**
- ```mysql
CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name (tag_name1, ...) TAGS (tag_value1, ...);
```
Using the specified STable as a template, specify the values of some tags columns to create a data table. (Unspecified tags columns are set to null values.)
Note: This method has been supported since version 2.0. 17. In previous versions, tags columns were not allowed to be specified, but the values of all tags columns must be explicitly given.
- Create tables in batches
- ```mysql
CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) tb_name2 USING stb_name TAGS (tag_value2, ...) ...;
```
Create a large number of data tables in batches faster. (Server side 2.0. 14 and above)
Note:
1. The method of batch creating tables requires that the data table must use STable as a template.
2. On the premise of not exceeding the length limit of SQL statements, it is suggested that the number of tables in a single statement should be controlled between 1000 and 3000, which will obtain an ideal speed of table building.
- Drop a table
- Show all data table information under the current database
Show all data table information under the current database.
Note: Wildcard characters can be used to match names in like. The maximum length of this wildcard character string cannot exceed 24 bytes.
Wildcard matching: 1) '%' (percent sign) matches 0 to any number of characters; 2) '_' underscore matches one character.
- Modify display character width online
- Get schema information of a table
- Add a column to table
- ```mysql
ALTER TABLE tb_name ADD COLUMN field_name data_type;
```
Note:
1. The maximum number of columns is 1024 and the minimum number is 2;
2. The maximum length of a column name is 64;
- Drop a column in table
- ```mysql
ALTER TABLE tb_name DROP COLUMN field_name;
```
If the table is created through a STable, the operation of table schema changing can only be carried out on the STable. Moreover, the schema changes for the STable take effect for all tables created through the schema. For tables that are not created through STables, you can modify the table schema directly.
## <a class="anchor" id="super-table"></a> STable Management
Note: In 2.0. 15.0 and later versions, STABLE reserved words are supported. That is, in the instruction description later in this section, the three instructions of CREATE, DROP and ALTER need to write TABLE instead of STABLE in the old version as the reserved word.
- Create a STable
Similiar to a standard table creation SQL, but you need to specify name and type of TAGS field.
Note:
1. Data types of TAGS column cannot be timestamp;
2. No duplicated TAGS column names;
3. Reversed word cannot be used as a TAGS column name;
4. The maximum number of TAGS is 128, and at least 1 TAG allowed, with a total length of no more than 16k characters.
- Drop a STable
- ```mysql
DROP STABLE [IF EXISTS] stb_name;
```
Drop a STable automatically deletes all sub-tables created through the STable.
- Show all STable information under the current database
- ```mysql
SHOW STABLES [LIKE tb_name_wildcard];
```
View all STables under the current database and relevant information, including name, creation time, column number, tag number, number of tables created through the STable, etc.
- Obtain schema information of a STable
- ```mysql
DESCRIBE stb_name;
```
- Add column to STable
- Drop column in STable
- ```mysql
ALTER STABLE stb_name DROP COLUMN field_name;
```
## <a class="anchor" id="tags"></a> TAG Management in STable
- Add a tag
Add a new tag to the STable and specify a type of the new tag. The total number of tags cannot exceed 128 and the total length does not exceed 16K characters.
- Drop a tag
- ```mysql
ALTER STABLE stb_name DROP TAG tag_name;
```
Delete a tag of STable. After deleting the tag, all sub-tables under the STable will also automatically delete the same tag.
- Modify a tag name
- ```mysql
ALTER STABLE stb_name CHANGE TAG old_tag_name new_tag_name;
```
Modify a tag name of STable. After modifying, all sub-tables under the STable will automatically update the new tag name.
- Modify a tag value of sub-table
- ```mysql
ALTER TABLE tb_name SET TAG tag_name=new_tag_value;
```
Note: Except that the operation of tag value updating is carried out for sub-tables, all other tag operations (adding tags, deleting tags, etc.) can only be applied to STable, and cannot be operated on a single sub-table. After adding a tag to a STable, all tables established based on that will automatically add a new tag, and the default value is NULL.
## <a class="anchor" id="insert"></a> Data Writing
- Insert a record
```mysql
INSERT INTO tb_name VALUES (field_value, ...);
```
Insert a record into table tb_name
- Insert a record with data corresponding to a given column
Insert a record into table tb_name, and the data corresponds to a given column. For columns that do not appear in the SQL statement, database will automatically populate them with NULL. Primary key (timestamp) cannot be NULL.
- Insert multiple records
- ```mysql
INSERT INTO tb_name VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
- Insert multiple records into table tb_name
- Insert multiple records into a given column
- ```mysql
INSERT INTO tb_name (field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
Insert multiple records into a given column of table tb_name
- Insert multiple records into multiple tables
- ```mysql
INSERT INTO tb1_name VALUES (field1_value1, ...) (field1_value2, ...) ...
tb2_name VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
Insert multiple records into tables tb1_name and tb2_name at the same time
- Insert multiple records per column into multiple tables
- ```mysql
INSERT INTO tb1_name (tb1_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...
tb2_name (tb2_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
Insert multiple records per column into tables tb1_name and tb2_name at the same time
Note: The timestamp of the oldest record allowed to be inserted is relative to the current server time, minus the configured keep value (days of data retention), and the timestamp of the latest record allowed to be inserted is relative to the current server time, plus the configured days value (interval of data storage in the data file, in days). Both keep and days can be specified when the database is created, and the default values are 3650 days and 10 days, respectively.
- <a class="anchor" id="auto_create_table"></a> Automatically create a table when inserting
- ```mysql
INSERT INTO tb_name USING stb_name TAGS (tag_value1, ...) VALUES (field_value1, ...);
```
If user is not sure whether a table exists when writing data, the automatic table building syntax can be used to create a non-existent table when writing. If the table already exists, no new table will be created. When automatically creating a table, it is required to use the STable as a template and specify tags value for the data table.
- Automatically create a table when inserting, and specify a given tags column
During automatic table creation, only the values of some tags columns can be specified, and the unspecified tags columns will be null.
**History writing**: The IMPORT or INSERT command can be used. The syntax and function of IMPORT are exactly the same as those of INSERT.
Note: For SQL statements in insert type, the stream parsing strategy we adopt will still execute the correct part of SQL before the following errors are found. In the following sql, insert statement is invalid, but d1001 will still be created.
```mysql
taos> CREATE TABLE meters(ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS(location BINARY(30), groupId INT);
Query OK, 0 row(s) affected (0.008245s)
taos> SHOW STABLES;
name | created_time | columns | tags | tables |
============================================================================================
meters | 2020-08-06 17:50:27.831 | 4 | 2 | 0 |
Query OK, 1 row(s) in set (0.001029s)
taos> SHOW TABLES;
Query OK, 0 row(s) in set (0.000946s)
taos> INSERT INTO d1001 USING meters TAGS('Beijing.Chaoyang', 2) VALUES('a');
DB error: invalid SQL: 'a' (invalid timestamp) (0.039494s)
taos> SHOW TABLES;
table_name | created_time | columns | stable_name |
======================================================================================================
d1001 | 2020-08-06 17:52:02.097 | 4 | meters |
Query OK, 1 row(s) in set (0.001091s)
```
## <a class="anchor" id="select"></a> Data Query
### Query Syntax:
```mysql
SELECT select_expr [, select_expr ...]
FROM {tb_name_list}
[WHERE where_condition]
[INTERVAL (interval_val [, interval_offset])]
[SLIDING sliding_val]
[FILL fill_val]
[GROUP BY col_list]
[ORDER BY col_list { DESC | ASC }]
[SLIMIT limit_val [SOFFSET offset_val]]
[LIMIT limit_val [OFFSET offset_val]]
[>> export_file];
```
#### SELECT Clause
A select clause can be a subquery of UNION and another query.
#### Wildcard character
The wildcard \* can be used to refer to all columns. For ordinary tables, there’re only ordinary columns in results.
```mysql
taos> SELECT * FROM d1001;
ts | current | voltage | phase |
======================================================================================
2018-10-03 14:38:05.000 | 10.30000 | 219 | 0.31000 |
2018-10-03 14:38:15.000 | 12.60000 | 218 | 0.33000 |
2018-10-03 14:38:16.800 | 12.30000 | 221 | 0.31000 |
Query OK, 3 row(s) in set (0.001165s)
```
For Stables, wildcards contain *tag columns*.
```mysql
taos> SELECT * FROM meters;
ts | current | voltage | phase | location | groupid |
=====================================================================================================================================
2018-10-03 14:38:05.500 | 11.80000 | 221 | 0.28000 | Beijing.Haidian | 2 |
2018-10-03 14:38:16.600 | 13.40000 | 223 | 0.29000 | Beijing.Haidian | 2 |
2018-10-03 14:38:05.000 | 10.80000 | 223 | 0.29000 | Beijing.Haidian | 3 |
2018-10-03 14:38:06.500 | 11.50000 | 221 | 0.35000 | Beijing.Haidian | 3 |
2018-10-03 14:38:04.000 | 10.20000 | 220 | 0.23000 | Beijing.Chaoyang | 3 |
2018-10-03 14:38:16.650 | 10.30000 | 218 | 0.25000 | Beijing.Chaoyang | 3 |
2018-10-03 14:38:05.000 | 10.30000 | 219 | 0.31000 | Beijing.Chaoyang | 2 |
2018-10-03 14:38:15.000 | 12.60000 | 218 | 0.33000 | Beijing.Chaoyang | 2 |
2018-10-03 14:38:16.800 | 12.30000 | 221 | 0.31000 | Beijing.Chaoyang | 2 |
Query OK, 9 row(s) in set (0.002022s)
​```
```
Wildcards support table name prefixes, the two following SQL statements will return all columns:
```mysql
SELECT * FROM d1001;
SELECT d1001.* FROM d1001;
```
In Join query, the results returned by \* with prefix and \* without prefix are different. \* returns all column data of all tables (excluding tags), while wildcards with prefix only return column data of the corresponding table.
```mysql
taos> SELECT * FROM d1001, d1003 WHERE d1001.ts=d1003.ts;
ts | current | voltage | phase | ts | current | voltage | phase |
==================================================================================================================================
2018-10-03 14:38:05.000 | 10.30000| 219 | 0.31000 | 2018-10-03 14:38:05.000 | 10.80000| 223 | 0.29000 |
Query OK, 1 row(s) in set (0.017385s)
```
```mysql
taos> SELECT d1001.* FROM d1001,d1003 WHERE d1001.ts = d1003.ts;
ts | current | voltage | phase |
======================================================================================
2018-10-03 14:38:05.000 | 10.30000 | 219 | 0.31000 |
Query OK, 1 row(s) in set (0.020443s)
```
In the process of using SQL functions for query, some SQL functions support wildcard operation. The difference is that the `count(\*)` function returns only one column, but the `first`,`last`,`last_row` functions return all columns.
```mysql
taos> SELECT COUNT(*) FROM d1001;
count(*) |
========================
3 |
Query OK, 1 row(s) in set (0.001035s)
```
```mysql
taos> SELECT FIRST(*) FROM d1001;
first(ts) | first(current) | first(voltage) | first(phase) |
=========================================================================================
2018-10-03 14:38:05.000 | 10.30000 | 219 | 0.31000 |
Query OK, 1 row(s) in set (0.000849s)
```
#### Tag Column
Since version 2.0. 14, it is supported to specify *tag column* in queries of ordinary tables, and the values of tag columns will be returned together with the data of other ordinary columns.
```mysql
taos> SELECT location, groupid, current FROM d1001 LIMIT 2;
location | groupid | current |
======================================================================
Beijing.Chaoyang | 2 | 10.30000 |
Beijing.Chaoyang | 2 | 12.60000 |
Query OK, 2 row(s) in set (0.003112s)
```
Note: The wildcard \* of ordinary tables does not contain *tag columns*.
#### Obtain the de-duplicated value of a tag column
Since version 2.0. 15, it is supported to specify `DISTINCT` keyword when querying tag columns in STables, which will return all non-duplicate values of given tag columns.
```mysql
SELECT DISTINCT tag_name FROM stb_name;
```
Note: At present, `DISTINCT` keyword only supports deduplication of tag columns of STables, and cannot be used for ordinary columns.
#### Column name in result set
In `SELECT` clause, if there’s no returning of column name in result set, the result set column name defaults to the expression name in `SELECT` clause as the column name. In addition, user can use `AS` to rename the columns in the returned result set. For example:
```mysql
taos> SELECT ts, ts AS primary_key_ts FROM d1001;
ts | primary_key_ts |
====================================================
2018-10-03 14:38:05.000 | 2018-10-03 14:38:05.000 |
2018-10-03 14:38:15.000 | 2018-10-03 14:38:15.000 |
2018-10-03 14:38:16.800 | 2018-10-03 14:38:16.800 |
Query OK, 3 row(s) in set (0.001191s)
```
However, renaming for one single column is not supported for `first(*)`,`last(*)`,`last_row(*)`.
#### Implicit result column
`Select_exprs` can be the name of a column belongs to a table, or it can be a column-based functional expression or calculation formula, with an upper limit of 256. When user uses `interval` or `group by tags` clause, the timestamp column (the first column) and the tag column in `group by` clause are forced to be returned in the final returned result. Later versions can support turning off the output of implicit columns in `group by` clause, and the column output is completely controlled by select clause.
#### List of STable
The `FROM` keyword can be followed by a list of several tables (STables) or result of a subquery.
If you do not specify user's current database, you can use the database name before the table name to specify the database to which the table belongs. For example: `power.d1001` to use tables across databases.
```mysql
SELECT * FROM power.d1001;
------------------------------
USE power;
SELECT * FROM d1001;
```
#### Special Functions
Some special query functions can be performed without using FROM clause. Obtain the current database database ():
```mysql
taos> SELECT DATABASE();
database() |
=================================
power |
Query OK, 1 row(s) in set (0.000079s)
```
If no default database is specified when logging in, and `USE` command is not used to switch data, then `NULL` is returned.
```mysql
taos> SELECT DATABASE();
database() |
=================================
NULL |
Query OK, 1 row(s) in set (0.000184s)
```
Get server and client version numbers:
```mysql
taos> SELECT CLIENT_VERSION();
client_version() |
===================
2.0.0.0 |
Query OK, 1 row(s) in set (0.000070s)
taos> SELECT SERVER_VERSION();
server_version() |
===================
2.0.0.0 |
Query OK, 1 row(s) in set (0.000077s)
```
A server state detection statement. If server is normal, return a number (for example, 1). If server is exceptional, return error code. The SQL syntax can be compatible with the check of TDengine status by connection pool and the check of database server status by third-party tools. And can avoid connection loss of connection pool caused by using a wrong heartbeat detection SQL statement.
```mysql
taos> SELECT SERVER_STATUS();
server_status() |
==================
1 |
Query OK, 1 row(s) in set (0.000074s)
taos> SELECT SERVER_STATUS() AS status;
status |
==============
1 |
Query OK, 1 row(s) in set (0.000081s)
```
#### Special keywords in TAOS SQL
> `TBNAME`: It can be regarded as a special tag in a STable query, representing the name of sub-table involved in the query
>
> _c0: Represents the first column of a table (STable)
#### Tips
Get all sub-table names and related tags information of a STable:
```mysql
SELECT TBNAME, location FROM meters;
```
Statistics of sub-tables number under a STable:
```mysql
SELECT COUNT(TBNAME) FROM meters;
```
The two queries above only support adding filters for TAGS in Where conditional clause. For example:
```mysql
taos> SELECT TBNAME, location FROM meters;
tbname | location |
==================================================================
d1004 | Beijing.Haidian |
d1003 | Beijing.Haidian |
d1002 | Beijing.Chaoyang |
d1001 | Beijing.Chaoyang |
Query OK, 4 row(s) in set (0.000881s)
taos> SELECT COUNT(tbname) FROM meters WHERE groupId > 2;
count(tbname) |
========================
2 |
Query OK, 1 row(s) in set (0.001091s)
```
- You can use \* to return all columns, or given column names. Four operations can be performed on numeric columns, and column names can be given to output columns.
- `WHERE` statement can use various logical decisions to filter numeric values, or wildcards to filter strings
- The output is sorted by default in ascending order by timestamps in the first column, but you can specify descending order (\_c0 refers to the first column timestamp). It is illegal to use ORDER BY to sort other fields.
- Parameter LIMIT controls the number of outputs, and OFFSET specifies which output starts from. LIMIT/OFFSET executes the result set after ORDER BY.
- "> >" output can be exported to a specified file
#### Supported Filtering Operations
| **Operation** | **Note** | **Applicable Data Types** |
| ------------- | ----------------------------- | ----------------------------------- |
| > | larger than | **timestamp** and all numeric types |
| < | smaller than | **timestamp** and all numeric types |
| >= | larger than or equal to | **timestamp** and all numeric types |
| <= | smaller than or equal to | **timestamp** and all numeric types |
| = | equal to | all types |
| <> | not equal to | all types |
| between and | within a certain range | **timestamp** and all numeric types |
| % | match with any char sequences | **binary** **nchar** |
| _ | match with a single char | **binary** **nchar** |
1. To filter the range of multiple fields at the same time, you need to use keyword AND to connect different query conditions. The query filtering between different columns connected by OR are not supported at the moment.
2. For filtering a single field, if it is a time filtering condition, only one condition in a statement can be set; however, for other (ordinary) columns or tag columns, OR keyword can be used for query filtering of combined conditions. For example: ((value > 20 AND value < 30) OR (value < 12)).
3. Since version 2.0. 17, condition filtering supports BETWEEN AND syntax. For example, WHERE col2 BETWEEN 1.5 AND 3.25 means that the query condition is "1.5 ≤ col2 ≤ 3.25".
### SQL Example
- For example, table tb1 is created with the following statement
- ```mysql
CREATE TABLE tb1 (ts TIMESTAMP, col1 INT, col2 FLOAT, col3 BINARY(50));
```
- Query all records of the last hour of tb1
- Look up table tb1 from 2018-06-01 08:00:00. 000 to 2018-06-02 08:00:00. 000, and col3 string is a record ending in'nny ', and the result is in descending order of timestamp:
- ```mysql
SELECT * FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND ts <= '2018-06-02 08:00:00.000' AND col3 LIKE '%nny' ORDER BY ts DESC;
```
- Query the sum of col1 and col2, and name it complex. The time is greater than 2018-06-01 08:00:00. 000, and col2 is greater than 1.2. As a result, only 10 records are outputted, starting from item 5
- ```mysql
SELECT (col1 + col2) AS 'complex' FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND col2 > 1.2 LIMIT 10 OFFSET 5;
```
- Query the records of past 10 minutes, the value of col2 is greater than 3.14, and output the result to the file /home/testoutpu.csv.
- ```mysql
SELECT COUNT(*) FROM tb1 WHERE ts >= NOW - 10m AND col2 > 3.14 >> /home/testoutpu.csv;
```
<a class="anchor" id="functions"></a>
## SQL Functions
TDengine supports aggregations over data, they are listed below:
- **COUNT**
Function: record the number of rows or non-null values in a column of statistics/STable.
Returned result data type: long integer INT64.
Applicable Fields: Applied to all fields.
Applied to: **table, STable**.
Note:
1. 1. You can use \* instead of specific fields, and use *() to return the total number of records.
2. The query results for fields of the same table (excluding NULL values) are the same.
3. If the statistic object is a specific column, return the number of records with non-NULL values in that column.
Example:
- ```mysql
taos> SELECT COUNT(*), COUNT(voltage) FROM meters;
count(*) | count(voltage) |
================================================
9 | 9 |
Query OK, 1 row(s) in set (0.004475s)
taos> SELECT COUNT(*), COUNT(voltage) FROM d1001;
count(*) | count(voltage) |
================================================
3 | 3 |
Query OK, 1 row(s) in set (0.001075s)
```
- **AVG**
- ```mysql
SELECT AVG(field_name) FROM tb_name [WHERE clause];
```
Function: return the average value of a column in statistics/STable.
Return Data Type: double.
Applicable Fields: all types except timestamp, binary, nchar, bool.
Applied to: **table,STable**.
Example:
- ```mysql
taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM meters;
avg(current) | avg(voltage) | avg(phase) |
====================================================================================
11.466666751 | 220.444444444 | 0.293333333 |
Query OK, 1 row(s) in set (0.004135s)
taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM d1001;
avg(current) | avg(voltage) | avg(phase) |
====================================================================================
11.733333588 | 219.333333333 | 0.316666673 |
Query OK, 1 row(s) in set (0.000943s)
```
- **TWA**
- ```mysql
SELECT TWA(field_name) FROM tb_name WHERE clause;
```
Function: Time weighted average function. The time-weighted average of a column in a statistical table over a period of time.
Return Data Type: double.
Applicable Fields: all types except timestamp, binary, nchar, bool.
Applied to: **table**.
- **SUM**
- ```mysql
SELECT SUM(field_name) FROM tb_name [WHERE clause];
```
Function: return the sum of a statistics/STable.
Return Data Type: long integer INMT64 and Double.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Applied to: **table,STable**.
Example:
- ```mysql
taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM meters;
sum(current) | sum(voltage) | sum(phase) |
================================================================================
103.200000763 | 1984 | 2.640000001 |
Query OK, 1 row(s) in set (0.001702s)
taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM d1001;
sum(current) | sum(voltage) | sum(phase) |
================================================================================
35.200000763 | 658 | 0.950000018 |
Query OK, 1 row(s) in set (0.000980s)
```
- **STDDEV**
- ```mysql
SELECT STDDEV(field_name) FROM tb_name [WHERE clause];
```
Function: Mean square deviation of a column in statistics table.
Return Data Type: Double.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Applied to: **table**. (also support **STable** since version 2.0.15.1)
Example:
- **LEASTSQUARES**
Function: Value of a column in statistical table is a fitting straight equation of primary key (timestamp). Start_val is the initial value of independent variable, and step_val is the step size value of independent variable.
Return Data Type: String expression (slope, intercept).
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note: Independent variable is the timestamp, and dependent variable is the value of the column.
Applied to: **table**.
Example:
### Selector Functions
- **MIN**
Function: return the minimum value of a specific column in statistics/STable.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Example:
```mysql
taos> SELECT MIN(current), MIN(voltage) FROM meters;
min(current) | min(voltage) |
======================================
10.20000 | 218 |
Query OK, 1 row(s) in set (0.001765s)
taos> SELECT MIN(current), MIN(voltage) FROM d1001;
min(current) | min(voltage) |
======================================
10.30000 | 218 |
Query OK, 1 row(s) in set (0.000950s)
```
- **MAX**
- ```mysql
SELECT MAX(field_name) FROM { tb_name | stb_name } [WHERE clause];
```
Function: return the maximum value of a specific column in statistics/STable.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Example:
- ```mysql
taos> SELECT MAX(current), MAX(voltage) FROM meters;
max(current) | max(voltage) |
======================================
13.40000 | 223 |
Query OK, 1 row(s) in set (0.001123s)
taos> SELECT MAX(current), MAX(voltage) FROM d1001;
max(current) | max(voltage) |
======================================
12.60000 | 221 |
Query OK, 1 row(s) in set (0.000987s)
```
- **FIRST**
- ```mysql
SELECT FIRST(field_name) FROM { tb_name | stb_name } [WHERE clause];
```
Function: The first non-NULL value written into a column in statistics/STable.
Return Data Type: Same as applicable fields.
Applicable Fields: All types.
Note:
1. 1. To return the first (minimum timestamp) non-NULL value of each column, use FIRST (\*);
2. if all columns in the result set are NULL values, the return result of the column is also NULL;
3. If all columns in the result set are NULL values, no result is returned.
Example:
```mysql
taos> SELECT FIRST(*) FROM meters;
first(ts) | first(current) | first(voltage) | first(phase) |
=========================================================================================
2018-10-03 14:38:04.000 | 10.20000 | 220 | 0.23000 |
Query OK, 1 row(s) in set (0.004767s)
taos> SELECT FIRST(current) FROM d1002;
first(current) |
=======================
10.20000 |
Query OK, 1 row(s) in set (0.001023s)
```
-
- **LAST**
- ```mysql
SELECT LAST(field_name) FROM { tb_name | stb_name } [WHERE clause];
```
Function: The last non-NULL value written by the value of a column in statistics/STable.
Return Data Type: Same as applicable fields.
Applicable Fields: All types.
Note:
1. 1. To return the last (maximum timestamp) non-NULL value of each column, use LAST (\*);
2. If a column in the result set has a NULL value, the returned result of the column is also NULL; if all columns in the result set have NULL values, no result is returned.
Example:
```mysql
taos> SELECT LAST(*) FROM meters;
last(ts) | last(current) | last(voltage) | last(phase) |
========================================================================================
2018-10-03 14:38:16.800 | 12.30000 | 221 | 0.31000 |
Query OK, 1 row(s) in set (0.001452s)
taos> SELECT LAST(current) FROM d1002;
last(current) |
=======================
10.30000 |
Query OK, 1 row(s) in set (0.000843s)
```
-
- **TOP**
Function: The top k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the largest, the one with smaller timestamp is returned.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note:
1. *The range of k value is* 1≤*k*≤100;
2. System also returns the timestamp column associated with the record.
Example:
- ```mysql
taos> SELECT TOP(current, 3) FROM meters;
ts | top(current, 3) |
=================================================
2018-10-03 14:38:15.000 | 12.60000 |
2018-10-03 14:38:16.600 | 13.40000 |
2018-10-03 14:38:16.800 | 12.30000 |
Query OK, 3 row(s) in set (0.001548s)
taos> SELECT TOP(current, 2) FROM d1001;
ts | top(current, 2) |
=================================================
2018-10-03 14:38:15.000 | 12.60000 |
2018-10-03 14:38:16.800 | 12.30000 |
Query OK, 2 row(s) in set (0.000810s)
```
- **BOTTOM**
Function: The last k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the smallest, the one with smaller timestamp is returned.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note:
1. *The range of k value is* 1≤*k*≤100;
2. System also returns the timestamp column associated with the record.
Example:
- ```mysql
taos> SELECT BOTTOM(voltage, 2) FROM meters;
ts | bottom(voltage, 2) |
===============================================
2018-10-03 14:38:15.000 | 218 |
2018-10-03 14:38:16.650 | 218 |
Query OK, 2 row(s) in set (0.001332s)
taos> SELECT BOTTOM(current, 2) FROM d1001;
ts | bottom(current, 2) |
=================================================
2018-10-03 14:38:05.000 | 10.30000 |
2018-10-03 14:38:16.800 | 12.30000 |
Query OK, 2 row(s) in set (0.000793s)
```
- **PERCENTILE**
Function: Percentile of the value of a column in statistical table.
Return Data Type: Double.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note: The range of P value is 0 ≤ P ≤ 100. P equals to MIN when, and equals MAX when it’s 100.
Example:
```mysql
taos> SELECT PERCENTILE(current, 20) FROM d1001;
percentile(current, 20) |
============================
11.100000191 |
Query OK, 1 row(s) in set (0.000787s)
```
-
- **APERCENTILE**
Function: The value percentile of a column in statistical table is similar to the PERCENTILE function, but returns approximate results.
Return Data Type: Double.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note: The range of P value is 0 ≤ P ≤ 100. P equals to MIN when, and equals MAX when it’s 100. APERCENTILE function is recommended, which performs far better than PERCENTILE function.
- **LAST_ROW**
Function: Return the last record of a table (STtable).
Return Data Type: Double.
Applicable Fields: All types.
Note: Unlike last function, last_row does not support time range restriction and forces the last record to be returned.
Example:
- ```mysql
taos> SELECT LAST_ROW(current) FROM meters;
last_row(current) |
=======================
12.30000 |
Query OK, 1 row(s) in set (0.001238s)
taos> SELECT LAST_ROW(current) FROM d1002;
last_row(current) |
=======================
10.30000 |
Query OK, 1 row(s) in set (0.001042s)
```
### Computing Functions
- **DIFF**
Function: Return the value difference between a column and the previous column.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note: The number of output result lines is the total number of lines in the range minus one, and there is no result output in the first line.
Example:
- ```mysql
taos> SELECT DIFF(current) FROM d1001;
ts | diff(current) |
=================================================
2018-10-03 14:38:15.000 | 2.30000 |
2018-10-03 14:38:16.800 | -0.30000 |
Query OK, 2 row(s) in set (0.001162s)
```
- **SPREAD**
- ```mysql
SELECT SPREAD(field_name) FROM { tb_name | stb_name } [WHERE clause];
```
Function: Return the difference between the max value and the min value of a column in statistics /STable.
Return Data Type: Same as applicable fields.
Applicable Fields: All types except binary, nchar, bool.
Note: Applicable for TIMESTAMP field, which indicates the time range of a record.
Example:
- ```mysql
taos> SELECT SPREAD(voltage) FROM meters;
spread(voltage) |
============================
5.000000000 |
Query OK, 1 row(s) in set (0.001792s)
taos> SELECT SPREAD(voltage) FROM d1001;
spread(voltage) |
============================
3.000000000 |
Query OK, 1 row(s) in set (0.000836s)
```
- Four Operations
Function: Calculation results of addition, subtraction, multiplication, division and remainder of values in a column or among multiple columns in statistics/STable.
Returned Data Type: Double.
Applicable Fields: All types except timestamp, binary, nchar, bool.
Note:
1. Calculation between two or more columns is supported, and the calculation priorities can be controlled by parentheses();
2. The NULL field does not participate in the calculation. If a row involved in calculation contains NULL, the calculation result of the row is NULL.
## <a class="anchor" id="aggregation"></a> Time-dimension Aggregation
TDengine supports aggregating by intervals. Data in a table can partitioned by intervals and aggregated to generate results. For example, a temperature sensor collects data once per second, but the average temperature needs to be queried every 10 minutes. This aggregation is suitable for down sample operation, and the syntax is as follows:
```mysql
SELECT function_list FROM tb_name
[WHERE where_condition]
INTERVAL (interval [, offset])
[SLIDING sliding]
[FILL ({NONE | VALUE | PREV | NULL | LINEAR | NEXT})]
SELECT function_list FROM stb_name
[WHERE where_condition]
INTERVAL (interval [, offset])
[SLIDING sliding]
[FILL ({ VALUE | PREV | NULL | LINEAR | NEXT})]
[GROUP BY tags]
```
- The length of aggregation interval is specified by keyword INTERVAL, the min time interval is 10 milliseconds (10a), and offset is supported (the offset must be less than interval). In aggregation queries, the aggregator and selector functions that can be executed simultaneously are limited to functions with one single output: count, avg, sum, stddev, leastsquares, percentile, min, max, first, last. Functions with multiple rows of output results (such as top, bottom, diff, and four operations) cannot be used.
- WHERE statement specifies the start and end time of a query and other filters
- FILL statement specifies a filling mode when data missed in a certain interval. Applicable filling modes include the following:
- - Do not fill: NONE (default filingl mode).
- VALUE filling: Fixed value filling, where the filled value needs to be specified. For example: fill (VALUE, 1.23).
- NULL filling: Fill the data with NULL. For example: fill (NULL).
- PREV filling: Filling data with the previous non-NULL value. For example: fill (PREV).
- NEXT filling: Filling data with the next non-NULL value. For example: fill (NEXT).
Note:
1. When using a FILL statement, a large number of filling outputs may be generated. Be sure to specify the time interval for the query. For each query, system can return no more than 10 million results with interpolation.
2. In a time-dimension aggregation, the time-series in returned results increases strictly monotonously.
3. If the query object is a STable, the aggregator function will act on the data of all tables under the STable that meet the value filters. If group by statement is not used in the query, the returned result increases strictly monotonously according to time-series; If group by statement is used to group in the query, each group in the returned result does not increase strictly monotonously according to time-series.
Example: The statement for building a database for smart meter is as follows:
```mysql
CREATE TABLE meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT);
```
According to the data collected by the smart meter, the average value, maximum value, median current of current data in the past 24 hours are calculated in a phase of 10 minutes, and the current trend with time changes is fitted to a straight line. If there is no calculated value, fill it with the previous non-NULL value. The query statement used is as follows:
```mysql
SELECT AVG(current), MAX(current), LEASTSQUARES(current, start_val, step_val), PERCENTILE(current, 50) FROM meters
WHERE ts>=NOW-1d
INTERVAL(10m)
FILL(PREV);
```
## <a class="anchor" id="limitation"></a> TAOS SQL Boundary Restrictions
- Max database name length is 32
- Max length of table name is 192, and max length of each data row is 16k characters
- Max length of column name is 64, max number of columns allowed is 1024, and min number of columns allowed is 2. The first column must be a timestamp
- Max number of tags allowed is 128, down to 1, and total length of tags does not exceed 16k characters
- Max length of SQL statement is 65480 characters, but it can be modified by system configuration parameter maxSQLLength, and max length can be configured to 1M
- Number of databases, STables and tables are not limited by system, but only limited by system resources
## Other TAOS SQL Conventions
**Restrictions on group by**
TAOS SQL supports group by operation on tags, tbnames and ordinary columns, required that only one column and whichhas less than 100,000 unique values.
**Restrictions on join operation**
TAOS SQL supports join columns of two tables by Primary Key timestamp between them, and does not support four operations after tables aggregated for the time being.
**Availability of is no null**
Is not null supports all types of columns. Non-null expression is < > "" and only applies to columns of non-numeric types.
\ No newline at end of file
documentation20/en/13.faq/docs.md 0 → 100644
浏览文件 @ b0e07ef8
# FAQ
Tutorials & FAQ
## 0.How to report an issue?
If the contents in FAQ cannot help you and you need the technical support and assistance of TDengine team, please package the contents in the following two directories:
1./var/log/taos (if default path has not been modified)
2./etc/taos
Provide the necessary description of the problem, including the version information of TDengine used, the platform environment information, the execution operation of the problem, the characterization of the problem and the approximate time, and submit the Issue on [GitHub](https://github.com/taosdata/TDengine).
To ensure that there is enough debug information, if the problem can be repeated, please modify the/etc/taos/taos.cfg file, add a line of "debugFlag 135" at the end (without quotation marks themselves), then restart taosd, repeat the problem, and then submit. You can also temporarily set the log level of taosd through the following SQL statement.
```
alter dnode <dnode_id> debugFlag 135;
```
However, when the system is running normally, please set debugFlag to 131, otherwise a large amount of log information will be generated and the system efficiency will be reduced.
## 1.What should I pay attention to when upgrading TDengine from older versions to 2.0 and above? ☆☆☆
Version 2.0 is a complete refactoring of the previous version, and the configuration and data files are incompatible. Be sure to do the following before upgrading:
1. Delete the configuration file, execute sudo rm `-rf /etc/taos/taos.cfg`
2. Delete the log file, execute `sudo rm -rf /var/log/taos/`
3. By ensuring that the data is no longer needed, delete the data file and execute `sudo rm -rf /var/lib/taos/`
4. Install the latest stable version of TDengine
5. If you need to migrate data or the data file is corrupted, please contact the official technical support team of TAOS Data to assist
## 2. When encoutered with the error " Unable to establish connection " in Windows, what can I do?
See the [technical blog](https://www.taosdata.com/blog/2019/12/03/jdbcdriver%E6%89%BE%E4%B8%8D%E5%88%B0%E5%8A%A8%E6%80%81%E9%93%BE%E6%8E%A5%E5%BA%93/) for this issue.
## 3. Why I get “more dnodes are needed” when create a table?
See the [technical blog](https://www.taosdata.com/blog/2019/12/03/%E5%88%9B%E5%BB%BA%E6%95%B0%E6%8D%AE%E8%A1%A8%E6%97%B6%E6%8F%90%E7%A4%BAmore-dnodes-are-needed/) for this issue.
## 4. How do I generate a core file when TDengine crashes?
See the [technical blog](https://www.taosdata.com/blog/2019/12/06/tdengine-crash%E6%97%B6%E7%94%9F%E6%88%90core%E6%96%87%E4%BB%B6%E7%9A%84%E6%96%B9%E6%B3%95/) for this issue.
## 5. What should I do if I encounter an error "Unable to establish connection"?
When the client encountered a connection failure, please follow the following steps to check:
1. Check your network environment
2. - Cloud server: Check whether the security group of the cloud server opens access to TCP/UDP ports 6030-6042
- Local virtual machine: Check whether the network can be pinged, and try to avoid using localhost as hostname
- Corporate server: If you are in a NAT network environment, be sure to check whether the server can return messages to the client
2. Make sure that the client and server version numbers are exactly the same, and the open source Community Edition and Enterprise Edition cannot be mixed.
3. On the server, execute systemctl status taosd to check the running status of *taosd*. If not running, start *taosd*.
4. Verify that the correct server FQDN (Fully Qualified Domain Name, which is available by executing the Linux command hostname-f on the server) is specified when the client connects. FQDN configuration reference: "[All about FQDN of TDengine](https://www.taosdata.com/blog/2020/09/11/1824.html)".
5. Ping the server FQDN. If there is no response, please check your network, DNS settings, or the system hosts file of the computer where the client is located.
6. Check the firewall settings (Ubuntu uses ufw status, CentOS uses firewall-cmd-list-port) to confirm that TCP/UDP ports 6030-6042 are open.
7. For JDBC (ODBC, Python, Go and other interfaces are similar) connections on Linux, make sure that libtaos.so is in the directory /usr/local/taos/driver, and /usr/local/taos/driver is in the system library function search path LD_LIBRARY_PATH.
8. For JDBC, ODBC, Python, Go, etc. connections on Windows, make sure that C:\ TDengine\ driver\ taos.dll is in your system library function search directory (it is recommended that taos.dll be placed in the directory C:\ Windows\ System32)
9. If the connection issue still exist
1. - On Linux system, please use the command line tool nc to determine whether the TCP and UDP connections on the specified ports are unobstructed. Check whether the UDP port connection works: nc -vuz {hostIP} {port} Check whether the server-side TCP port connection works: nc -l {port}Check whether the client-side TCP port connection works: nc {hostIP} {port}
- Windows systems use the PowerShell command Net-TestConnection-ComputerName {fqdn} Port {port} to detect whether the service-segment port is accessed
10. You can also use the built-in network connectivity detection function of taos program to verify whether the specified port connection between the server and the client is unobstructed (including TCP and UDP): [TDengine's Built-in Network Detection Tool Use Guide](https://www.taosdata.com/blog/2020/09/08/1816.html).
## 6.What to do if I encounter an error "Unexpected generic error in RPC" or "TDengine error: Unable to resolve FQDN"?
This error occurs because the client or data node cannot parse the FQDN (Fully Qualified Domain Name). For TAOS shell or client applications, check the following:
1. Please verify whether the FQDN of the connected server is correct. FQDN configuration reference: "[All about FQDN of TDengine](https://www.taosdata.com/blog/2020/09/11/1824.html)".
2. If the network is configured with a DNS server, check that it is working properly.
3. If the network does not have a DNS server configured, check the hosts file of the machine where the client is located to see if the FQDN is configured and has the correct IP address.
4. If the network configuration is OK, from the machine where the client is located, you need to be able to ping the connected FQDN, otherwise the client cannot connect to the server
## 7.Although the syntax is corrected, why do I still get the “Invalid SQL" error?
If you confirm that the syntax is correct, for versions older than 2.0, please check whether the SQL statement length exceeds 64K. If it does, this error will also be returned.
## 8. Are “validation queries” supported?
The TDengine does not yet have a dedicated set of validation queries. However, it is recommended to use the database "log" monitored by the system.
## 9. Can I delete or update a record?
TDengine does not support the deletion function at present, and may support it in the future according to user requirements.
Starting from 2.0. 8.0, TDengine supports the function of updating written data. Using the update function requires using UPDATE 1 parameter when creating the database, and then you can use INSERT INTO command to update the same timestamp data that has been written. UPDATE parameter does not support ALTER DATABASE command modification. Without a database created using UPDATE 1 parameter, writing data with the same timestamp will not modify the previous data with no error reported.
It should also be noted that when UPDATE is set to 0, the data with the same timestamp sent later will be discarded directly, but no error will be reported, and will still be included in affected rows (so the return information of INSERT instruction cannot be used for timestamp duplicate checking). The main reason for this design is that TDengine regards the written data as a stream. Regardless of whether the timestamp conflicts or not, TDengine believes that the original device that generates the data actually generates such data. The UPDATE parameter only controls how such stream data should be processed when persistence-when UPDATE is 0, it means that the data written first overwrites the data written later; When UPDATE is 1, it means that the data written later overwrites the data written first. How to choose this coverage relationship depends on whether the data generated first or later is expected in the subsequent use and statistics compile.
## 10. How to create a table with more than 1024 columns?
Using version 2.0 and above, 1024 columns are supported by default; for older versions, TDengine allowed the creation of a table with a maximum of 250 columns. However, if the limit is exceeded, it is recommended to logically split this wide table into several small ones according to the data characteristics.
## 11. What is the most effective way to write data?
Insert in batches. Each write statement can insert multiple records into one or multiple tables at the same time.
## 12. What is the most effective way to write data? How to solve the problem that Chinese characters in nchar inserted under Windows systems are parsed into messy code?
If there are Chinese characters in nchar data under Windows, please first confirm that the region of the system is set to China (which can be set in the Control Panel), then the taos client in cmd should already support it normally; If you are developing Java applications in an IDE, such as Eclipse and Intellij, please confirm that the file code in the IDE is GBK (this is the default coding type of Java), and then initialize the configuration of the client when generating the Connection. The specific statement is as follows:
```JAVA
Class.forName("com.taosdata.jdbc.TSDBDriver");
Properties properties = new Properties();
properties.setProperty(TSDBDriver.LOCALE_KEY, "UTF-8");
Connection = DriverManager.getConnection(url, properties);
```
## 13. JDBC error: the excluded SQL is not a DML or a DDL?
Please update to the latest JDBC driver.
```xml
<dependency>
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
<version>2.0.27</version>
</dependency>
```
## 14. taos connect failed, reason: invalid timestamp.
The common reason is that the server time and client time are not calibrated, which can be calibrated by synchronizing with the time server (use ntpdate command under Linux, and select automatic synchronization in the Windows time setting).
## 15. Incomplete display of table name
Due to the limited display width of taos shell in the terminal, it is possible that a relatively long table name is not displayed completely. If relevant operations are carried out according to the displayed incomplete table name, a Table does not exist error will occur. The workaround can be by modifying the setting option maxBinaryDisplayWidth in the taos.cfg file, or directly entering the command `set max_binary_display_width 100`. Or, use the \\G parameter at the end of the command to adjust how the results are displayed.
## 16. How to migrate data?
TDengine uniquely identifies a machine according to hostname. When moving data files from machine A to machine B, pay attention to the following three points:
- For versions 2.0. 0.0 to 2.0. 6. x, reconfigure machine B's hostname to machine A's.
- For 2.0. 7.0 and later versions, go to/var/lib/taos/dnode, repair the FQDN corresponding to dnodeId of dnodeEps.json, and restart. Make sure this file is identical for all machines.
- The storage structures of versions 1. x and 2. x are incompatible, and it is necessary to use migration tools or your own application to export and import data.
## 17. How to temporarily adjust the log level in command line program taos?
For the convenience of debugging, since version 2.0. 16, command line program taos gets two new instructions related to logging:
```mysql
ALTER LOCAL flag_name flag_value;
```
This means that under the current command line program, modify the loglevel of a specific module (only valid for the current command line program, if taos is restarted, it needs to be reset):
- The values of flag_name can be: debugFlag, cDebugFlag, tmrDebugFlag, uDebugFlag, rpcDebugFlag
- Flag_value values can be: 131 (output error and alarm logs), 135 (output error, alarm, and debug logs), 143 (output error, alarm, debug, and trace logs)
```mysql
ALTER LOCAL RESETLOG;
```
This means wiping up all client-generated log files on the machine.
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或