[](https://bestpractices.coreinfrastructure.org/projects/4201)
[](https://bestpractices.coreinfrastructure.org/projects/4201)
English | [简体中文](README-CN.md) | We are hiring, check [here](https://tdengine.com/careers)
English | [简体中文](README-CN.md) | [Lean more about TSDB](https://tdengine.com/tsdb)
# What is TDengine？
# What is TDengine？
TDengine is an open source, high-performance, cloud native [time-series database](https://tdengine.com/tsdb/what-is-a-time-series-database/) optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. TDengine differentiates itself from other time-seires databases with the following advantages:
TDengine is an open source, high-performance, cloud native [time-series database](https://tdengine.com/tsdb/) optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. TDengine differentiates itself from other time-seires databases with the following advantages:
-**[High-Performance](https://tdengine.com/tdengine/high-performance-time-series-database/)**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
-**[High-Performance](https://tdengine.com/tdengine/high-performance-time-series-database/)**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
...
@@ -33,6 +33,8 @@ TDengine is an open source, high-performance, cloud native [time-series database
...
@@ -33,6 +33,8 @@ TDengine is an open source, high-performance, cloud native [time-series database
-**[Open Source](https://tdengine.com/tdengine/open-source-time-series-database/)**: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered 18.8k stars on GitHub. There is an active developer community, and over 139k running instances worldwide.
-**[Open Source](https://tdengine.com/tdengine/open-source-time-series-database/)**: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered 18.8k stars on GitHub. There is an active developer community, and over 139k running instances worldwide.
For a full list of TDengine competitive advantages, please [check here](https://tdengine.com/tdengine/)
# Documentation
# Documentation
For user manual, system design and architecture, please refer to [TDengine Documentation](https://docs.tdengine.com)([TDengine 文档](https://docs.taosdata.com))
For user manual, system design and architecture, please refer to [TDengine Documentation](https://docs.tdengine.com)([TDengine 文档](https://docs.taosdata.com))
...
@@ -319,6 +321,7 @@ TDengine provides abundant developing tools for users to develop on TDengine. Fo
...
@@ -319,6 +321,7 @@ TDengine provides abundant developing tools for users to develop on TDengine. Fo
Please follow the [contribution guidelines](CONTRIBUTING.md) to contribute to the project.
Please follow the [contribution guidelines](CONTRIBUTING.md) to contribute to the project.
# Join TDengine WeChat Group
# Join TDengine User Community
Add WeChat “tdengine” to join the group，you can communicate with other users.
TDengine is an [open-source](https://tdengine.com/tdengine/open-source-time-series-database/), [cloud-native](https://tdengine.com/tdengine/cloud-native-time-series-database/)time-series database optimized for the Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. This document is the TDengine user manual. It introduces the basic, as well as novel concepts, in TDengine, and also talks in detail about installation, features, SQL, APIs, operation, maintenance, kernel design, and other topics. It’s written mainly for architects, developers, and system administrators.
TDengine is an [open-source](https://tdengine.com/tdengine/open-source-time-series-database/), [cloud-native](https://tdengine.com/tdengine/cloud-native-time-series-database/)[time-series database](https://tdengine.com/tsdb/) optimized for the Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. This document is the TDengine user manual. It introduces the basic, as well as novel concepts, in TDengine, and also talks in detail about installation, features, SQL, APIs, operation, maintenance, kernel design, and other topics. It’s written mainly for architects, developers, and system administrators.
To get an overview of TDengine, such as a feature list, benchmarks, and competitive advantages, please browse through the [Introduction](./intro) section.
To get an overview of TDengine, such as a feature list, benchmarks, and competitive advantages, please browse through the [Introduction](./intro) section.
...
@@ -22,6 +22,8 @@ If you want to know more about TDengine tools, the REST API, and connectors for
...
@@ -22,6 +22,8 @@ If you want to know more about TDengine tools, the REST API, and connectors for
If you are very interested in the internal design of TDengine, please read the chapter [Inside TDengine](./tdinternal), which introduces the cluster design, data partitioning, sharding, writing, and reading processes in detail. If you want to study TDengine code or even contribute code, please read this chapter carefully.
If you are very interested in the internal design of TDengine, please read the chapter [Inside TDengine](./tdinternal), which introduces the cluster design, data partitioning, sharding, writing, and reading processes in detail. If you want to study TDengine code or even contribute code, please read this chapter carefully.
To get more general introduction about time series database, please read through [a series of articles](https://tdengine.com/tsdb/). To lean more competitive advantages about TDengine, please read through [a series of blogs](https://tdengine.com/tdengine/).
TDengine is an open-source database, and we would love for you to be a part of TDengine. If you find any errors in the documentation or see parts where more clarity or elaboration is needed, please click "Edit this page" at the bottom of each page to edit it directly.
TDengine is an open-source database, and we would love for you to be a part of TDengine. If you find any errors in the documentation or see parts where more clarity or elaboration is needed, please click "Edit this page" at the bottom of each page to edit it directly.
TDengine is an open source, high-performance, cloud native[time-series database](https://tdengine.com/tsdb/) optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](../develop/cache), [stream processing](../develop/stream), [data subscription](../develop/tmq) and other functionalities to reduce the system complexity and cost of development and operation.
TDengine is an [open source](https://tdengine.com/tdengine/open-source-time-series-database/), [high-performance](https://tdengine.com/tdengine/high-performance-time-series-database/), [cloud native](https://tdengine.com/tdengine/cloud-native-time-series-database/)[time-series database](https://tdengine.com/tsdb/) optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](../develop/cache), [stream processing](../develop/stream), [data subscription](../develop/tmq) and other functionalities to reduce the system complexity and cost of development and operation.
This section introduces the major features, competitive advantages, typical use-cases and benchmarks to help you get a high level overview of TDengine.
This section introduces the major features, competitive advantages, typical use-cases and benchmarks to help you get a high level overview of TDengine.
...
@@ -43,7 +43,7 @@ For more details on features, please read through the entire documentation.
...
@@ -43,7 +43,7 @@ For more details on features, please read through the entire documentation.
## Competitive Advantages
## Competitive Advantages
By making full use of [characteristics of time series data](https://tdengine.com/tsdb/characteristics-of-time-series-data/), TDengine differentiates itself from other time series databases, with the following advantages.
By making full use of [characteristics of time series data](https://tdengine.com/tsdb/characteristics-of-time-series-data/), TDengine differentiates itself from other [time series databases](https://tdengine.com/tsdb), with the following advantages.
-**[High-Performance](https://tdengine.com/tdengine/high-performance-time-series-database/)**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
-**[High-Performance](https://tdengine.com/tdengine/high-performance-time-series-database/)**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
...
@@ -127,3 +127,8 @@ As a high-performance, scalable and SQL supported time-series database, TDengine
...
@@ -127,3 +127,8 @@ As a high-performance, scalable and SQL supported time-series database, TDengine
-[TDengine vs OpenTSDB](https://tdengine.com/2019/09/12/710.html)
-[TDengine vs OpenTSDB](https://tdengine.com/2019/09/12/710.html)
-[TDengine vs Cassandra](https://tdengine.com/2019/09/12/708.html)
-[TDengine vs Cassandra](https://tdengine.com/2019/09/12/708.html)
-[TDengine vs InfluxDB](https://tdengine.com/2019/09/12/706.html)
-[TDengine vs InfluxDB](https://tdengine.com/2019/09/12/706.html)
## More readings
-[Introduction to Time-Series Database](https://tdengine.com/tsdb/)
-[Introduction to TDengine competitive advantages](https://tdengine.com/tdengine/)
@@ -348,19 +348,15 @@ SELECT ... FROM (SELECT ... FROM ...) ...;
...
@@ -348,19 +348,15 @@ SELECT ... FROM (SELECT ... FROM ...) ...;
:::info
:::info
- Only one layer of nesting is allowed, that means no sub query is allowed within a sub query
- The result of a nested query is returned as a virtual table used by the outer query. It's recommended to give an alias to this table for the convenience of using it in the outer query.
- The result set returned by the inner query will be used as a "virtual table" by the outer query. The "virtual table" can be renamed using `AS` keyword for easy reference in the outer query.
- Sub query is not allowed in continuous query.
- JOIN operation is allowed between tables/STables inside both inner and outer queries. Join operation can be performed on the result set of the inner query.
- JOIN operation is allowed between tables/STables inside both inner and outer queries. Join operation can be performed on the result set of the inner query.
- UNION operation is not allowed in either inner query or outer query.
- The features that can be used in the inner query are the same as those that can be used in a non-nested query.
- The functions that can be used in the inner query are the same as those that can be used in a non-nested query.
-`ORDER BY` inside the inner query is unnecessary and will slow down the query performance significantly. It is best to avoid the use of `ORDER BY` inside the inner query.
-`ORDER BY` inside the inner query is unnecessary and will slow down the query performance significantly. It is best to avoid the use of `ORDER BY` inside the inner query.
- Compared to the non-nested query, the functionality that can be used in the outer query has the following restrictions:
- Compared to the non-nested query, the functionality that can be used in the outer query has the following restrictions:
- Functions
- Functions
- If the result set returned by the inner query doesn't contain timestamp column, then functions relying on timestamp can't be used in the outer query, like `TOP`, `BOTTOM`, `FIRST`, `LAST`, `DIFF`.
- If the result set returned by the inner query doesn't contain timestamp column, then functions relying on timestamp can't be used in the outer query, like INTERP,DERIVATIVE, IRATE, LAST_ROW, FIRST, LAST, TWA, STATEDURATION, TAIL, UNIQUE.
- Functions that need to scan the data twice can't be used in the outer query, like `STDDEV`, `PERCENTILE`.
- If the result set returned by the inner query are not sorted in order by timestamp, then functions relying on data ordered by timestamp can't be used in the outer query, like LEASTSQUARES, ELAPSED, INTERP, DERIVATIVE, IRATE, TWA, DIFF, STATECOUNT, STATEDURATION, CSUM, MAVG, TAIL, UNIQUE.
-`IN` operator is not allowed in the outer query but can be used in the inner query.
- Functions that need to scan the data twice can't be used in the outer query, like PERCENTILE.
As a purpose-built database for storing and processing time-series data, TDengine provides time-series-specific extensions to standard SQL.
As a purpose-built database for storing and processing time-series data, TDengine provides time-series-specific extensions to standard SQL.
These extensions include tag-partitioned queries and windowed queries.
These extensions include partitioned queries and windowed queries.
## Tag-Partitioned Queries
## Partitioned Queries
When you query a supertable, you may need to partition the supertable by tag and perform additional operations on a specific partition. In this case, you can use the following SQL clause:
When you query a supertable, you may need to partition the supertable by some dimensions and perform additional operations on a specific partition. In this case, you can use the following SQL clause:
```sql
```sql
PARTITIONBYpart_list
PARTITIONBYpart_list
...
@@ -17,22 +17,24 @@ PARTITION BY part_list
...
@@ -17,22 +17,24 @@ PARTITION BY part_list
part_list can be any scalar expression, such as a column, constant, scalar function, or a combination of the preceding items.
part_list can be any scalar expression, such as a column, constant, scalar function, or a combination of the preceding items.
A PARTITION BY clause with a tag is processed as follows:
A PARTITION BY clause is processed as follows:
- The PARTITION BY clause must occur after the WHERE clause and cannot be used with a JOIN clause.
- The PARTITION BY clause must occur after the WHERE clause
- The PARTITION BY clause partitions the super table by the specified tag group, and the specified calculation is performed on each partition. The calculation performed is determined by the rest of the statement - a window clause, GROUP BY clause, or SELECT clause.
- The PARTITION BY caluse partitions the data according to the specified dimentions, then perform computation on each partition. The performed computation is determined by the rest of the statement - a window clause, GROUP BY clause, or SELECT clause.
-You can use PARTITION BY together with a window clause or GROUP BY clause. In this case, the window or GROUP BY clause takes effect on every partition. For example, the following statement partitions the table by the location tag, performs downsampling over a 10 minute window, and returns the maximum value:
-The PARTITION BY clause can be used together with a window clause or GROUP BY clause. In this case, the window or GROUP BY clause takes effect on every partition. For example, the following statement partitions the table by the location tag, performs downsampling over a 10 minute window, and returns the maximum value:
The most common usage of PARTITION BY is partitioning the data in subtables by tags then perform computation when querying data in a supertable. More specifically, `PARTITION BY TBNAME` partitions the data of each subtable into a single timeline, and this method facilitates the statistical analysis in many use cases of processing timeseries data.
## Windowed Queries
## Windowed Queries
Aggregation by time window is supported in TDengine. For example, in the case where temperature sensors report the temperature every seconds, the average temperature for every 10 minutes can be retrieved by performing a query with a time window. Window related clauses are used to divide the data set to be queried into subsets and then aggregation is performed across the subsets. There are three kinds of windows: time window, status window, and session window. There are two kinds of time windows: sliding window and flip time/tumbling window. The query syntax is as follows:
Aggregation by time window is supported in TDengine. For example, in the case where temperature sensors report the temperature every seconds, the average temperature for every 10 minutes can be retrieved by performing a query with a time window. Window related clauses are used to divide the data set to be queried into subsets and then aggregation is performed across the subsets. There are three kinds of windows: time window, status window, and session window. There are two kinds of time windows: sliding window and flip time/tumbling window. The query syntax is as follows:
```sql
```sql
SELECTfunction_listFROMtb_name
SELECTselect_listFROMtb_name
[WHEREwhere_condition]
[WHEREwhere_condition]
[SESSION(ts_col,tol_val)]
[SESSION(ts_col,tol_val)]
[STATE_WINDOW(col)]
[STATE_WINDOW(col)]
...
@@ -42,12 +44,6 @@ SELECT function_list FROM tb_name
...
@@ -42,12 +44,6 @@ SELECT function_list FROM tb_name
The following restrictions apply:
The following restrictions apply:
### Restricted Functions
- Aggregate functions and select functions can be used in `function_list`, with each function having only one output. For example COUNT, AVG, SUM, STDDEV, LEASTSQUARES, PERCENTILE, MIN, MAX, FIRST, LAST. Functions having multiple outputs, such as DIFF or arithmetic operations can't be used.
-`LAST_ROW` can't be used together with window aggregate.
- Scalar functions, like CEIL/FLOOR, can't be used with window aggregate.
### Other Rules
### Other Rules
- The window clause must occur after the PARTITION BY clause and before the GROUP BY clause. It cannot be used with a GROUP BY clause.
- The window clause must occur after the PARTITION BY clause and before the GROUP BY clause. It cannot be used with a GROUP BY clause.
...
@@ -82,7 +78,7 @@ These pseudocolumns occur after the aggregation clause.
...
@@ -82,7 +78,7 @@ These pseudocolumns occur after the aggregation clause.
1. A huge volume of interpolation output may be returned using `FILL`, so it's recommended to specify the time range when using `FILL`. The maximum number of interpolation values that can be returned in a single query is 10,000,000.
1. A huge volume of interpolation output may be returned using `FILL`, so it's recommended to specify the time range when using `FILL`. The maximum number of interpolation values that can be returned in a single query is 10,000,000.
2. The result set is in ascending order of timestamp when you aggregate by time window.
2. The result set is in ascending order of timestamp when you aggregate by time window.
3. If aggregate by window is used on STable, the aggregate function is performed on all the rows matching the filter conditions. If `GROUP BY` is not used in the query, the result set will be returned in ascending order of timestamp; otherwise the result set is not exactly in the order of ascending timestamp in each group.
3. If aggregate by window is used on STable, the aggregate function is performed on all the rows matching the filter conditions. If `PARTITION BY` is not used in the query, the result set will be returned in strict ascending order of timestamp; otherwise the result set is not exactly in the order of ascending timestamp in each group.
:::
:::
...
@@ -112,9 +108,9 @@ When using time windows, note the following:
...
@@ -112,9 +108,9 @@ When using time windows, note the following:
Please note that the `timezone` parameter should be configured to be the same value in the `taos.cfg` configuration file on client side and server side.
Please note that the `timezone` parameter should be configured to be the same value in the `taos.cfg` configuration file on client side and server side.
- The result set is in ascending order of timestamp when you aggregate by time window.
- The result set is in ascending order of timestamp when you aggregate by time window.
### Status Window
### State Window
In case of using integer, bool, or string to represent the status of a device at any given moment, continuous rows with the same status belong to a status window. Once the status changes, the status window closes. As shown in the following figure, there are two status windows according to status, [2019-04-28 14:22:07，2019-04-28 14:22:10] and [2019-04-28 14:22:11，2019-04-28 14:22:12]. Status window is not applicable to STable for now.
In case of using integer, bool, or string to represent the status of a device at any given moment, continuous rows with the same status belong to a status window. Once the status changes, the status window closes. As shown in the following figure, there are two state windows according to status, [2019-04-28 14:22:07，2019-04-28 14:22:10] and [2019-04-28 14:22:11，2019-04-28 14:22:12].
...
@@ -130,7 +126,7 @@ The primary key, i.e. timestamp, is used to determine which session window a row
...
@@ -130,7 +126,7 @@ The primary key, i.e. timestamp, is used to determine which session window a row
If the time interval between two continuous rows are within the time interval specified by `tol_value` they belong to the same session window; otherwise a new session window is started automatically. Session window is not supported on STable for now.
If the time interval between two continuous rows are within the time interval specified by `tol_value` they belong to the same session window; otherwise a new session window is started automatically.
@@ -9,15 +9,54 @@ This document describes how to manage permissions in TDengine.
...
@@ -9,15 +9,54 @@ This document describes how to manage permissions in TDengine.
## Create a User
## Create a User
```sql
```sql
CREATEUSERuse_namePASS'password';
CREATEUSERuser_namePASS'password'[SYSINFO{1|0}];
```
```
This statement creates a user account.
This statement creates a user account.
The maximum length of use_name is 23 bytes.
The maximum length of user_name is 23 bytes.
The maximum length of password is 128 bytes. The password can include leters, digits, and special characters excluding single quotation marks, double quotation marks, backticks, backslashes, and spaces. The password cannot be empty.
The maximum length of password is 128 bytes. The password can include leters, digits, and special characters excluding single quotation marks, double quotation marks, backticks, backslashes, and spaces. The password cannot be empty.
`SYSINFO` indicates whether the user is allowed to view system information. `1` means allowed, `0` means not allowed. System information includes server configuration, dnode, vnode, storage. The default value is `1`.
For example, we can create a user whose password is `123456` and is able to view system information.
- ENABLE: Specify whether the user is enabled or disabled. 1 indicates enabled and 0 indicates disabled.
- ENABLE: Specify whether the user is enabled or disabled. 1 indicates enabled and 0 indicates disabled.
- SYSINFO: Specify whether the user can query system information. 1 indicates that the user can query system information and 0 indicates that the user cannot query system information.
- SYSINFO: Specify whether the user can query system information. 1 indicates that the user can query system information and 0 indicates that the user cannot query system information.
For example, you can use below command to disable user `test`:
```sql
taos>alterusertestenable0;
QueryOK,0of0rowsaffected(0.001160s)
```
## Grant Permissions
## Grant Permissions
...
@@ -62,7 +108,7 @@ priv_level : {
...
@@ -62,7 +108,7 @@ priv_level : {
}
}
```
```
Grant permissions to a user.
Grant permissions to a user, this feature is only available in enterprise edition.
Permissions are granted on the database level. You can grant read or write permissions.
Permissions are granted on the database level. You can grant read or write permissions.
...
@@ -92,4 +138,4 @@ priv_level : {
...
@@ -92,4 +138,4 @@ priv_level : {
```
```
Revoke permissions from a user.
Revoke permissions from a user, this feature is only available in enterprise edition.
After TDengine is started, a database named `log` is created automatically to help with monitoring. Information that includes CPU, memory and disk usage, bandwidth, number of requests, disk I/O speed, slow queries, is written into the `log` database at a predefined interval. Additionally, some important system operations, like logon, create user, drop database, and alerts and warnings generated in TDengine are written into the `log` database too. A system operator can view the data in `log` database from TDengine CLI or from a web console.
After TDengine is started, it automatically writes monitoring data including CPU, memory and disk usage, bandwidth, number of requests, disk I/O speed, slow queries, into a designated database at a predefined interval through taosKeeper. Additionally, some important system operations, like logon, create user, drop database, and alerts and warnings generated in TDengine are written into the `log` database too. A system operator can view the data in `log` database from TDengine CLI or from a web console.
The collection of the monitoring information is enabled by default, but can be disabled by parameter `monitor` in the configuration file.
The collection of the monitoring information is enabled by default, but can be disabled by parameter `monitor` in the configuration file.
...
@@ -10,7 +10,7 @@ The collection of the monitoring information is enabled by default, but can be d
...
@@ -10,7 +10,7 @@ The collection of the monitoring information is enabled by default, but can be d
TDinsight is a complete solution which uses the monitoring database `log` mentioned previously, and Grafana, to monitor a TDengine cluster.
TDinsight is a complete solution which uses the monitoring database `log` mentioned previously, and Grafana, to monitor a TDengine cluster.
From version 2.3.3.0, more monitoring data has been added in the `log` database. Please refer to [TDinsight Grafana Dashboard](https://grafana.com/grafana/dashboards/15167) to learn more details about using TDinsight to monitor TDengine.
Please refer to [TDinsight Grafana Dashboard](../../reference/tdinsight) to learn more details about using TDinsight to monitor TDengine.
A script `TDinsight.sh` is provided to deploy TDinsight automatically.
A script `TDinsight.sh` is provided to deploy TDinsight automatically.
...
@@ -30,31 +30,14 @@ Prepare：
...
@@ -30,31 +30,14 @@ Prepare：
2. Grafana Alert Notification
2. Grafana Alert Notification
There are two ways to setup Grafana alert notification.
You can use below command to setup Grafana alert notification.
-An existing Grafana Notification Channel can be specified with parameter `-E`, the notifier uid of the channel can be obtained by `curl -u admin:admin localhost:3000/api/alert-notifications |jq`
An existing Grafana Notification Channel can be specified with parameter `-E`, the notifier uid of the channel can be obtained by `curl -u admin:admin localhost:3000/api/alert-notifications |jq`
- The AliCloud SMS alert built in TDengine data source plugin can be enabled with parameter `-s`, the parameters of enabling this plugin are listed below:
-`-I`: AliCloud SMS Key ID
-`-K`: AliCloud SMS Key Secret
-`-S`: AliCloud SMS Signature
-`-C`: SMS notification template
-`-T`: Input parameters in JSON format for the SMS notification template, for example`{"alarm_level":"%s","time":"%s","name":"%s","content":"%s"}`
-`-B`: List of mobile numbers to be notified
Below is an example of the full command using the AliCloud SMS alert.
```bash
sudo ./TDinsight.sh -a http://localhost:6041 -u root -p taosdata -s\
TDinsight is a solution for monitoring TDengine using the builtin native monitoring database and [Grafana].
TDinsight is a solution for monitoring TDengine using the builtin native monitoring database and [Grafana].
After TDengine starts, it will automatically create a monitoring database `log`. TDengine will automatically write many metrics in specific intervals into the `log` database. The metrics may include the server's CPU, memory, hard disk space, network bandwidth, number of requests, disk read/write speed, slow queries, other information like important system operations (user login, database creation, database deletion, etc.), and error alarms. With [Grafana] and [TDengine Data Source Plugin](https://github.com/taosdata/grafanaplugin/releases), TDinsight can visualize cluster status, node information, insertion and query requests, resource usage, vnode, dnode, and mnode status, exception alerts and many other metrics. This is very convenient for developers who want to monitor TDengine cluster status in real-time. This article will guide users to install the Grafana server, automatically install the TDengine data source plug-in, and deploy the TDinsight visualization panel using the `TDinsight.sh` installation script.
After TDengine starts, it automatically writes many metrics in specific intervals into a designated database. The metrics may include the server's CPU, memory, hard disk space, network bandwidth, number of requests, disk read/write speed, slow queries, other information like important system operations (user login, database creation, database deletion, etc.), and error alarms. With [Grafana] and [TDengine Data Source Plugin](https://github.com/taosdata/grafanaplugin/releases), TDinsight can visualize cluster status, node information, insertion and query requests, resource usage, vnode, dnode, and mnode status, exception alerts and many other metrics. This is very convenient for developers who want to monitor TDengine cluster status in real-time. This article will guide users to install the Grafana server, automatically install the TDengine data source plug-in, and deploy the TDinsight visualization panel using the `TDinsight.sh` installation script.
## System Requirements
## System Requirements
To deploy TDinsight, a single-node TDengine server or a multi-node TDengine cluster and a [Grafana] server are required. This dashboard requires TDengine 2.3.3.0 and above, with the `log` database enabled (`monitor = 1`).
To deploy TDinsight, we need
- a single-node TDengine server or a multi-node TDengine cluster and a [Grafana] server are required. This dashboard requires TDengine 3.0.1.0 and above, with the monitoring feature enabled. For detailed configuration, please refer to [TDengine monitoring configuration](../config/#monitoring-parameters).
- taosAdapter has been instaleld and running, please refer to [taosAdapter](../taosadapter).
- taosKeeper has been installed and running, please refer to [taosKeeper](../taoskeeper).
Please record
- The endpoint of taosAdapter REST service, for example `http://tdengine.local:6041`
- Authentication of taosAdapter, e.g. user name and password
- The database name used by taosKeeper to store monitoring data
## Installing Grafana
## Installing Grafana
We recommend using the latest [Grafana] version 7 or 8 here. You can install Grafana on any [supported operating system](https://grafana.com/docs/grafana/latest/installation/requirements/#supported-operating-systems) by following the [official Grafana documentation Instructions](https://grafana.com/docs/grafana/latest/installation/) to install [Grafana].
We recommend using the latest [Grafana] version 8 or 9 here. You can install Grafana on any [supported operating system](https://grafana.com/docs/grafana/latest/installation/requirements/#supported-operating-systems) by following the [official Grafana documentation Instructions](https://grafana.com/docs/grafana/latest/installation/) to install [Grafana].
### Installing Grafana on Debian or Ubuntu
### Installing Grafana on Debian or Ubuntu
...
@@ -71,7 +79,7 @@ chmod +x TDinsight.sh
...
@@ -71,7 +79,7 @@ chmod +x TDinsight.sh
./TDinsight.sh
./TDinsight.sh
```
```
This script will automatically download the latest [Grafana TDengine data source plugin](https://github.com/taosdata/grafanaplugin/releases/latest) and [TDinsight dashboard](https://grafana.com/grafana/dashboards/15167) with configurable parameters for command-line options to the [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) configuration file to automate deployment and updates, etc. With the alert setting options provided by this script, you can also get built-in support for AliCloud SMS alert notifications.
This script will automatically download the latest [Grafana TDengine data source plugin](https://github.com/taosdata/grafanaplugin/releases/latest) and [TDinsight dashboard](https://github.com/taosdata/grafanaplugin/blob/master/dashboards/TDinsightV3.json) with configurable parameters for command-line options to the [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) configuration file to automate deployment and updates, etc. With the alert setting options provided by this script, you can also get built-in support for AliCloud SMS alert notifications.
Assume you use TDengine and Grafana's default services on the same host. Run `. /TDinsight.sh` and open the Grafana browser window to see the TDinsight dashboard.
Assume you use TDengine and Grafana's default services on the same host. Run `. /TDinsight.sh` and open the Grafana browser window to see the TDinsight dashboard.
...
@@ -106,18 +114,6 @@ Install and configure TDinsight dashboard in Grafana on Ubuntu 18.04/20.04 syste
...
@@ -106,18 +114,6 @@ Install and configure TDinsight dashboard in Grafana on Ubuntu 18.04/20.04 syste
-E, --external-notifier <string> Apply external notifier uid to TDinsight dashboard.
-E, --external-notifier <string> Apply external notifier uid to TDinsight dashboard.
| -e | --tdinsight-editable | TDINSIGHT_DASHBOARD_EDITABLE | If the dashboard is configured to be editable. [Default: false] | -e | --external
| -e | --tdinsight-editable | TDINSIGHT_DASHBOARD_EDITABLE | If the dashboard is configured to be editable. [Default: false] | -e | --external
| -E | --external-notifier | EXTERNAL_NOTIFIER | Apply the external notifier uid to the TDinsight dashboard. | -s
| -E | --external-notifier | EXTERNAL_NOTIFIER | Apply the external notifier uid to the TDinsight dashboard. | -s
| -s | --sms-enabled | SMS_ENABLED | Enable the tdengine-datasource plugin built into Alibaba Cloud SMS webhook. | -s
| -N | --sms-notifier-name | SMS_NOTIFIER_NAME | The name of the provisioning notifier. [Default: `TDinsight Builtin SMS`] | -U
| -U | --sms-notifier-uid | SMS_NOTIFIER_UID | "Notification Channel" `uid`, lowercase of the program name is used by default, other characters are replaced by "-". |-sms
| -D | --sms-notifier-is-default | SMS_NOTIFIER_IS_DEFAULT | Set built-in SMS notification to default value. |-sms-notifier-is-default
Suppose you start a TDengine database on host `tdengine` with HTTP API port `6041`, user `root1`, and password `pass5ord`. Execute the script.
Suppose you start a TDengine database on host `tdengine` with HTTP API port `6041`, user `root1`, and password `pass5ord`. Execute the script.
...
@@ -166,24 +151,10 @@ Use the `uid` value obtained above as `-E` input.
...
@@ -166,24 +151,10 @@ Use the `uid` value obtained above as `-E` input.
sudo ./TDinsight.sh -a http://tdengine:6041 -u root1 -p pass5ord -E existing-notifier
sudo ./TDinsight.sh -a http://tdengine:6041 -u root1 -p pass5ord -E existing-notifier
```
```
If you want to use the [Alibaba Cloud SMS](https://www.aliyun.com/product/sms) service as a notification channel, you should enable it with the `-s` flag add the following parameters.
-`-N`: Notification Channel name, default is `TDinsight Builtin SMS`.
-`-U`: Channel uid, default is lowercase of `name`, any other character is replaced with -, for the default `-N`, its uid is `tdinsight-builtin-sms`.
-`-I`: Alibaba Cloud SMS access key id.
-`-K`: Alibaba Cloud SMS access secret key.
-`-S`: Alibaba Cloud SMS signature.
-`-C`: Alibaba Cloud SMS template id.
-`-T`: Alibaba Cloud SMS template parameters, for JSON format template, example is as follows `'{"alarm_level":"%s", "time":"%s", "name":"%s", "content":"%s"}'`. There are four parameters: alarm level, time, name and alarm content.
-`-B`: a list of phone numbers, separated by a comma `,`.
If you want to monitor multiple TDengine clusters, you need to set up numerous TDinsight dashboards. Setting up non-default TDinsight requires some changes: the `-n``-i``-t` options need to be changed to non-default names, and `-N` and `-L` should also be changed if using the built-in SMS alerting feature.
If you want to monitor multiple TDengine clusters, you need to set up numerous TDinsight dashboards. Setting up non-default TDinsight requires some changes: the `-n``-i``-t` options need to be changed to non-default names, and `-N` and `-L` should also be changed if using the built-in SMS alerting feature.
-s-N'Env1 SMS'-I xx -K xx -S xx -C SMS_XX -T''-B 00000000000 -L 127.0.0.01:10611
```
```
Please note that the configuration data source, notification channel, and dashboard are not changeable on the front end. You should update the configuration again via this script or manually change the configuration file in the `/etc/grafana/provisioning` directory (this is the default directory for Grafana, use the `-P` option to change it as needed).
Please note that the configuration data source, notification channel, and dashboard are not changeable on the front end. You should update the configuration again via this script or manually change the configuration file in the `/etc/grafana/provisioning` directory (this is the default directory for Grafana, use the `-P` option to change it as needed).
...
@@ -249,21 +220,23 @@ Save and test. It will report 'TDengine Data source is working' under normal cir
...
@@ -249,21 +220,23 @@ Save and test. It will report 'TDengine Data source is working' under normal cir
### Importing dashboards
### Importing dashboards
Point to **+** / **Create** - **import** (or `/dashboard/import` url).
In the page of configuring data source, click **Dashboards** tab.
Type the dashboard ID `15167` in the **Import via grafana.com** location and **Load**.
Choose `TDengine for 3.x` and click `import`.
After the importing is done, `TDinsight for 3.x` dashboard is available on the page of `search dashboards by name`.
Once the import is complete, the full page view of TDinsight is shown below.
In the `TDinsight for 3.x` dashboard, choose the database used by taosKeeper to store monitoring data, you can see the monitoring result.
The TDinsight dashboard is designed to provide the usage and status of TDengine-related resources[dnodes, mnodes, vnodes](../../taos-sql/node/) or databases.
The TDinsight dashboard is designed to provide the usage and status of TDengine-related resources, e.g. dnodes, mnodes, vnodes and databases.
Details of the metrics are as follows.
Details of the metrics are as follows.
...
@@ -285,7 +258,6 @@ This section contains the current information and status of the cluster, the ale
...
@@ -285,7 +258,6 @@ This section contains the current information and status of the cluster, the ale
-**Measuring Points Used**: The number of measuring points used to enable the alert rule (no data available in the community version, healthy by default).
-**Measuring Points Used**: The number of measuring points used to enable the alert rule (no data available in the community version, healthy by default).
-**Grants Expire Time**: the expiration time of the enterprise version of the enabled alert rule (no data available for the community version, healthy by default).
-**Grants Expire Time**: the expiration time of the enterprise version of the enabled alert rule (no data available for the community version, healthy by default).
-**Error Rate**: Aggregate error rate (average number of errors per second) for alert-enabled clusters.
-**Error Rate**: Aggregate error rate (average number of errors per second) for alert-enabled clusters.
-**Variables**: `show variables` table display.
### DNodes Status
### DNodes Status
...
@@ -294,7 +266,6 @@ This section contains the current information and status of the cluster, the ale
...
@@ -294,7 +266,6 @@ This section contains the current information and status of the cluster, the ale
-**DNodes Status**: simple table view of `show dnodes`.
-**DNodes Status**: simple table view of `show dnodes`.
-**DNodes Lifetime**: the time elapsed since the dnode was created.
-**DNodes Lifetime**: the time elapsed since the dnode was created.
-**DNodes Number**: the number of DNodes changes.
-**DNodes Number**: the number of DNodes changes.
-**Offline Reason**: if any dnode status is offline, the reason for offline is shown as a pie chart.
### MNode Overview
### MNode Overview
...
@@ -309,7 +280,6 @@ This section contains the current information and status of the cluster, the ale
...
@@ -309,7 +280,6 @@ This section contains the current information and status of the cluster, the ale
1.**Requests Rate(Inserts per Second)**: average number of inserts per second.
1.**Requests Rate(Inserts per Second)**: average number of inserts per second.
2.**Requests (Selects)**: number of query requests and change rate (count of second).
2.**Requests (Selects)**: number of query requests and change rate (count of second).
3.**Requests (HTTP)**: number of HTTP requests and request rate (count of second).
### Database
### Database
...
@@ -319,9 +289,8 @@ Database usage, repeated for each value of the variable `$database` i.e. multipl
...
@@ -319,9 +289,8 @@ Database usage, repeated for each value of the variable `$database` i.e. multipl
1.**STables**: number of super tables.
1.**STables**: number of super tables.
2.**Total Tables**: number of all tables.
2.**Total Tables**: number of all tables.
3.**Sub Tables**: the number of all super table subtables.
3.**Tables**: number of normal tables.
4.**Tables**: graph of all normal table numbers over time.
4.**Table number for each vgroup**: number of tables per vgroup.
5.**Tables Number Foreach VGroups**: The number of tables contained in each VGroups.
### DNode Resource Usage
### DNode Resource Usage
...
@@ -356,12 +325,11 @@ Currently, only the number of logins per minute is reported.
...
@@ -356,12 +325,11 @@ Currently, only the number of logins per minute is reported.
Support monitoring taosAdapter request statistics and status details. Includes.
Support monitoring taosAdapter request statistics and status details. Includes.
1.**http_request**: contains the total number of requests, the number of failed requests, and the number of requests being processed
1.**http_request_inflight**: number of real-time requests.
2.**top 3 request endpoint**: data of the top 3 requests by endpoint group
2.**http_request_total**: number of total requests.
3.**Memory Used**: taosAdapter memory usage
3.**http_request_fail**: number of failed requets.
4.**latency_quantile(ms)**: quantile of (1, 2, 5, 9, 99) stages
4.**CPU Used**: CPU usage of taosAdapter.
5.**top 3 failed request endpoint**: data of the top 3 failed requests by endpoint grouping
JupyterLab is the next generation of the ubiquitous Jupyter Notebook. In this note we show you how to install the TDengine Python connector to connect to TDengine in JupyterLab. You can then insert data and perform queries against the TDengine instance within JupyterLab.
## Install JupyterLab
Installing JupyterLab is very easy. Installation instructions can be found at:
If you don't feel like clicking on the link here are the instructions.
Jupyter's preferred Python package manager is pip, so we show the instructions for pip.
You can also use **conda** or **pipenv** if you are managing Python environments.
````
pip install jupyterlab
````
For **conda** you can run:
````
conda install -c conda-forge jupyterlab
````
For **pipenv** you can run:
````
pipenv install jupyterlab
pipenv shell
````
## Run JupyterLab
You can start JupyterLab from the command line by running:
````
jupyter lab
````
This will automatically launch your default browser and connect to your JupyterLab instance, usually on port 8888.
## Install the TDengine Python connector
You can now install the TDengine Python connector as follows.
Start a new Python kernel in JupyterLab.
If using **conda** run the following:
````
# Install a conda package in the current Jupyter kernel
import sys
!conda install --yes --prefix {sys.prefix} taospy
````
If using **pip** run the following:
````
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install taospy
````
## Connect to TDengine
You can find detailed examples to use the Python connector, in the TDengine documentation here.
Once you have installed the TDengine Python connector in your JupyterLab kernel, the process of connecting to TDengine is the same as that you would use if you weren't using JupyterLab.
Each TDengine instance, has a database called "log" which has monitoring information about the TDengine instance.
In the "log" database there is a [supertable](https://docs.tdengine.com/taos-sql/stable/) called "disks_info".
The code below is used to fetch data from this table into a pandas DataFrame.
````
import sys
import taos
import pandas
def sqlQuery(conn):
df: pandas.DataFrame = pandas.read_sql("select * from log.disks_info limit 500", conn)
print(df)
return df
conn = taos.connect()
result = sqlQuery(conn)
print(result)
````
TDengine has connectors for various languages including Node.js, Go, PHP and there are kernels for these languages which can be found [here](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels).
