@@ -38,7 +38,7 @@ for details. The documentation from our website can also be downloaded locally f
# Building
At the moment, TDengine only supports building and running on Linux systems. You can choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) or from the source code. This quick guide is for installation from the source only.
At the moment, TDengine server only supports running on Linux systems. You can choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) or build it from the source code. This quick guide is for installation from the source only.
To build TDengine, use [CMake](https://cmake.org/) 3.0.2 or higher versions in the project directory.
...
...
@@ -169,9 +169,20 @@ You can modify the file ~/.gitconfig to use ssh protocol instead of https for be
### On Linux platform
You can run the bash script `build.sh` to build both TDengine and taosTools including taosBenchmark and taosdump as below:
```bash
mkdir debug &&cd debug
cmake .. && cmake --build .
./build.sh
```
It equals to execute following commands:
```bash
git submodule update --init--recursive
mkdir debug
cd debug
cmake .. -DBUILD_TOOLS=true
make
```
Note TDengine 2.3.x.0 and later use a component named 'taosAdapter' to play http daemon role by default instead of the http daemon embedded in the early version of TDengine. The taosAdapter is programmed by go language. If you pull TDengine source code to the latest from an existing codebase, please execute 'git submodule update --init --recursive' to pull taosAdapter source code. Please install go language version 1.14 or above for compiling taosAdapter. If you meet difficulties regarding 'go mod', especially you are from China, you can use a proxy to solve the problem.
message(WARNING"The submodules were not downloaded! GIT_SUBMODULE was turned off or failed. Please update submodules manually if you need build them.")
TDengine is a high-performance, scalable time-series database with SQL support. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](/develop/cache), [stream processing](/develop/continuous-query), [data subscription](/develop/subscribe) and other functionalities to reduce the complexity and cost of development and operation.
TDengine is a high-performance, scalable time-series database with SQL support. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](/develop/cache), [stream processing](/develop/continuous-query), [data subscription](/develop/subscribe) and other functionalities to reduce the complexity and cost of development and operation. The major features are listed below:
This section introduces the major features, competitive advantages, suited scenarios and benchmarks to help you get a high level picture for TDengine.
1. Besides [using SQL to insert](/develop/insert-data/sql-writing)，supports [Schemaless writing](/reference/schemaless/)，and supports [InfluxDB LINE](/develop/insert-data/influxdb-line)，[OpenTSDB Telnet](/develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](/develop/insert-data/opentsdb-json) and other protocols.
2. Support seamless integration with third-party data collection agent like [Telegraf](/third-party/telegraf)，[Prometheus](/third-party/prometheus)，[StatsD](/third-party/statsd)，[collectd](/third-party/collectd)，[icinga2](/third-party/icinga2), [Tcollector](/third-party/tcollector), [EMQ](/third-party/emq-broker), [HiveMQ](/third-party/hive-mq-broker). Without a line of code, those agents can write data points into TDengine just by configuration.
3. Support [all kinds of queries](/query-data), including aggregation, nested query, downsampling, interpolation, etc.
4. Support [user defined functions](/develop/udf)
5. Support [caching](/develop/cache). TDengine always save the last data point in cache, so Redis is not needed in some scenarios.
6. Support [continuous query](/develop/continuous-query).
7. Support [data subscription](/develop/subscribe)，and the filter condition can be specified.
8. Support [cluster](/cluster/), so it can gain more processing power by adding more nodes. The high availability is supported by replication.
9. Provide interactive [command line intrerface](/reference/taos-shell) for management, maintainence and ad-hoc query.
10. Provide many ways to [import](/operation/import), [export](/operation/export) data.
11. Provide [monitoring](/operation/monitor) on TDengine running instances.
12. Provide [connectors](/reference/connector/) for [C/C++](/reference/connector/cpp), [Java](/reference/connector/java), [Python](/reference/connector/python), [Go](/reference/connector/go), [Rust](/reference/connector/rust), [Node.js](/reference/connector/node) and other programming languages.
13. Provide [REST API](/reference/rest-api/).
14. Support the seamless integration with [Grafana](/third-party/grafana) for visualization.
15. Support seamless integration with Google Data Studio.
## Major Features
For more detailed features, please read through the whole document.
The major features are listed below:
## TDenginge Highlights
1. Besides [using SQL to insert](/develop/insert-data/sql-writing)，it supports [Schemaless writing](/reference/schemaless/)，and it supports [InfluxDB LINE](/develop/insert-data/influxdb-line)，[OpenTSDB Telnet](/develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](/develop/insert-data/opentsdb-json) and other protocols.
2. Support for seamless integration with third-party data collection agents like [Telegraf](/third-party/telegraf)，[Prometheus](/third-party/prometheus)，[StatsD](/third-party/statsd)，[collectd](/third-party/collectd)，[icinga2](/third-party/icinga2), [TCollector](/third-party/tcollector), [EMQX](/third-party/emq-broker), [HiveMQ](/third-party/hive-mq-broker). Without a line of code, those agents can write data points into TDengine just by configuration.
3. Support for [all kinds of queries](/develop/query-data), including aggregation, nested query, downsampling, interpolation, etc.
4. Support for [user defined functions](/develop/udf)
5. Support for [caching](/develop/cache). TDengine always saves the last data point in cache, so Redis is not needed in some scenarios.
6. Support for [continuous query](/develop/continuous-query).
7. Support for [data subscription](/develop/subscribe) with the capability to specify filter conditions.
8. Support for [cluster](/cluster/), with the capability of increasing processing power by adding more nodes. High availability is supported by replication.
9. Provides interactive [command-line intrerface](/reference/taos-shell) for management, maintainence and ad-hoc query.
10. Provides many ways to [import](/operation/import) and [export](/operation/export) data.
11. Provides [monitoring](/operation/monitor) on TDengine running instances.
12. Provides [connectors](/reference/connector/) for [C/C++](/reference/connector/cpp), [Java](/reference/connector/java), [Python](/reference/connector/python), [Go](/reference/connector/go), [Rust](/reference/connector/rust), [Node.js](/reference/connector/node) and other programming languages.
13. Provides a [REST API](/reference/rest-api/).
14. Supports the seamless integration with [Grafana](/third-party/grafana) for visualization.
15. Supports seamless integration with Google Data Studio.
TDengine makes full use of [the characteristics of time series data](https://tdengine.com/2019/07/09/86.html), such as structured, no transaction, rarely delete or update, etc., and builds its own innovative storage engine and computing engine to differentiate itself from other TSDBs with the following advantages.
For more detail on features, please read through the whole documentation.
## Competitive Advantages
TDengine makes full use of [the characteristics of time series data](https://tdengine.com/2019/07/09/86.html), such as structured, no transaction, rarely delete or update, etc., and builds its own innovative storage engine and computing engine to differentiate itself from other time series databases with the following advantages.
-**High Performance**: TDengine outperforms other time series databases in data ingestion and querying while significantly reducing storage cost and compute costs, with an innovatively designed and purpose-built storage engine.
...
...
@@ -41,29 +45,27 @@ TDengine makes full use of [the characteristics of time series data](https://tde
-**Zero Management**: Installation and cluster setup can be done in seconds. Data partitioning and sharding are executed automatically. TDengine’s running status can be monitored via Grafana or other DevOps tools.
-**Zero Learning Cost**: With SQL as the query language, support for ubiquitous tools like Python, Java, C/C++, Go, Rust, Node.js connectors, there is zero learning cost.
-**Zero Learning Costs**: With SQL as the query language and support for ubiquitous tools like Python, Java, C/C++, Go, Rust, and Node.js connectors, there are zero learning costs.
-**Interactive Console**: TDengine provides convenient console access to the database to run ad hoc queries, maintain the database, or manage the cluster without any programming.
With TDengine, the total cost of ownership of time-seriess data platform can be greatly reduced. Because 1: with its superior performance, the computing and storage resources are reduced significantly; 2：with SQL support, it can be seamlessly integrated with many third party tools, and learning cost/migration cost is reduced significantly; 3: with its simple architecture and zero management, the operation and maintainence cost is reduced.
With TDengine, the total cost of ownership of time-seriess data platform can be greatly reduced. Because 1: with its superior performance, the computing and storage resources are reduced significantly; 2：with SQL support, it can be seamlessly integrated with many third party tools, and learning costs/migration costs are reduced significantly; 3: with its simple architecture and zero management, the operation and maintainence costs are reduced.
## TDengine Technical Ecosystem
## Technical Ecosystem
In the time-series data processing platform, TDengine stands in a role like this diagram below:
On the left side, there are data collection agents like OPC-UA, MQTT, Telegraf and Kafka. On the right side, visualization/BI tools, HMI, Python/R, IoT App can be connected. TDengine itself provides interactive command line interface and web interface for management and maintainence.
On the left side, there are data collection agents like OPC-UA, MQTT, Telegraf and Kafka. On the right side, visualization/BI tools, HMI, Python/R, and IoT Apps can be connected. TDengine itself provides interactive command-line interface and web interface for management and maintainence.
## Suited Scenarios for TDengine
## Suited Scenarios
As a high-performance, scalable and SQL supported time-series database, TDengine's typical application scenarios include but are not limited to IoT, Industrial Internet, Connected Vehicles, IT operation and maintenance, energy, financial market and other fields. But you shall note that TDengine is a purpose-built database and does tons of optimization based on the characteristics of time series data, it cannot be used to process data from web crawlers, social media, e-commerce, ERP, CRM, etc. This section makes a more detailed analysis of the applicable scenarios.
As a high-performance, scalable and SQL supported time-series database, TDengine's typical application scenarios include but are not limited to IoT, Industrial Internet, Connected Vehicles, IT operation and maintenance, energy, financial markets and other fields. TDengine is a purpose-built database optimized for the characteristics of time series data, it cannot be used to process data from web crawlers, social media, e-commerce, ERP, CRM, etc. This section makes a more detailed analysis of the applicable scenarios.
### Characteristics and Requirements of Data Sources
From the perspective of data sources, designers can analyze the applicability of TDengine in target application systems as follows.
| **Data Source Characteristics and Requirements** | **Not Applicable** | **Might Be Applicable** | **Very Applicable** | **Description** |
| A massive amount of total data | | | √ | TDengine provides excellent scale-out functions in terms of capacity, and has a storage structure with matching high compression ratio to achieve the best storage efficiency in the industry.|
...
...
@@ -76,7 +78,7 @@ From the perspective of data sources, designers can analyze the applicability of
| A simple and reliable system architecture | | | √ | TDengine's system architecture is very simple and reliable, with its own message queue, cache, stream computing, monitoring and other functions. There is no need to integrate any additional third-party products. |
| Fault-tolerance and high-reliability | | | √ | TDengine has cluster functions to automatically provide high-reliability and high-availability functions such as fault tolerance and disaster recovery. |
| Standardization support | | | √ | TDengine supports standard SQL and also provides extensions specifically to analyze time-series data. |
| Standardization support | | | √ | TDengine supports standard SQL and provides SQL extensions for time-series data analysis. |
### System Function Requirements
...
...
@@ -91,7 +93,7 @@ From the perspective of data sources, designers can analyze the applicability of
| Very large total processing capacity | | | √ | TDengine’s cluster functions can easily improve processing capacity via multi-server coordination. |
| Extremely high-speed data processing | | | √ | TDengine’s storage and data processing are optimized for IoT, and can process data many times faster than similar products.|
| Extremely fast processing of fine-grained data | | | √ | TDengine has achieved the same or better performance than other relational and NoSQL data processing systems. |
| Extremely fast processing of high resolution data | | | √ | TDengine has achieved the same or better performance than other relational and NoSQL data processing systems. |
### System Maintenance Requirements
...
...
@@ -101,7 +103,7 @@ From the perspective of data sources, designers can analyze the applicability of
| Minimize learning and maintenance costs | | | √ | In addition to being easily configurable, standard SQL support and the Taos shell for ad hoc queries makes maintenance simpler, allows reuse and reduces learning costs.|
| Abundant talent supply | √ | | | Given the above, and given the extensive training and professional services provided by TDengine, it is easy to migrate from existing solutions or create a new and lasting solution based on TDengine.|
## Benchmark comparision between TDengine and other databases
## Comparision with other databases
-[Writing Performance Comparison of TDengine and InfluxDB ](https://tdengine.com/2022/02/23/4975.html)
-[Query Performance Comparison of TDengine and InfluxDB](https://tdengine.com/2022/02/24/5120.html)
In order to explain the basic concepts and show sample code easily, the entire TDengine document takes smart meters as a typical time series data scenario. Assuming that each smart meter collects three metrics of current, voltage, and phase, there are multiple smart meters, and each meter has static attributes like location and group ID, the collected data shall be similar to the following table:
In order to explain the basic concepts and provide some sample code, the TDengine documentation takes smart meters as a typical time series data scenario. Assuming that each smart meter collects three metrics of current, voltage, and phase, there are multiple smart meters, and each meter has static attributes like location and group ID, the collected data will be similar to the following table:
<divclassName="center-table">
<table>
...
...
@@ -100,32 +100,32 @@ In order to explain the basic concepts and show sample code easily, the entire T
<ahref="#model_table1">Table 1: Smart meter example data</a>
</div>
Each row contains the device ID, timestamp, collected metrics (current, voltage, phase as above), and static tags (Location and groupId in Table 1) associated with the devices. Each smart meter generates a row (data point) in a pre-defined timer or triggered by an external event. It is a sequence of data points like a stream.
Each row contains the device ID, time stamp, collected metrics (current, voltage, phase as above), and static tags (location and groupId in Table 1) associated with the devices. Each smart meter generates a row (measurement) in a pre-defined time interval or triggered by an external event. The device produces a sequence of measurements with associated time stamps.
## Metric
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which changes with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases.
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which change with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases.
## Label/Tag
Label/Tag refers to the static properties of sensors, devices or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values. Unlike the collected metric data, the amount of tag data stored does not change over time.
Label/Tag refers to the static properties of sensors, equipment or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values at any time. Unlike the collected metric data, the amount of tag data stored does not change over time.
## Data Collection Point
Data Collection Point(DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipments, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there is a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car, so a car has three data collection points.
Data Collection Point (DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipments, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there could be a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car, so in this example the car would have three data collection points.
## Table
Since time-series data is most likely to be structured data, TDengine adopts the traditional relational database model to process them with a short learning curve. You need to create a database, create tables, then insert data points and execute queries to explore the data.
To make full use of time-series data characteristics, TDengine adopts a strategy "**One Table for One Data Collection Point**". TDengine requires the user to create a table for each data collection point (DCP) to store collected time-series data. For example, if there are over 10 million smart meters, it means 10 million tables shall be created. For the table above, 4 tables shall be created for devices D1001, D1002, D1003, and D1004 to store the data collected. This design has several benefits:
To make full use of time-series data characteristics, TDengine adopts a strategy of "**One Table for One Data Collection Point**". TDengine requires the user to create a table for each data collection point (DCP) to store collected time-series data. For example, if there are over 10 million smart meters, it means 10 million tables should be created. For the table above, 4 tables should be created for devices D1001, D1002, D1003, and D1004 to store the data collected. This design has several benefits:
1. Since the metric data from different DCP is fully independent, the data source of each DCP is unique, and a table has only one writer. In this way, data points can be written in a lock-free manner, and the writing speed can be greatly improved.
1. Since the metric data from different DCP are fully independent, the data source of each DCP is unique, and a table has only one writer. In this way, data points can be written in a lock-free manner, and the writing speed can be greatly improved.
2. For a DCP, the metric data generated by DCP is ordered by timestamp, so the write operation can be implemented by simple appending, which further greatly improves the data writing speed.
3. The metric data from a DCP is continuously stored in block by block. If you read data for a period of time, it can greatly reduce random read operations and improve read and query performance by orders of magnitude.
4. Inside a data block for a DCP, columnar storage is used, and different compression algorithms are used for different data types. Because the change of the metrics from a DCP is not big in a time range, the compression rate is higher.
4. Inside a data block for a DCP, columnar storage is used, and different compression algorithms are used for different data types. Metrics generally don't vary as significantly between themselves over a time range as compared to other metrics, this allows for a higher compression rate.
If the metric data of multiple DPCs are traditionally written into a single table, due to the uncontrollable network delay, the timing of the data from different DCPs arriving at the server cannot be guaranteed, the writing operation must be protected by locks, and the metric data from one DCP cannot be guaranteed to be continuously stored together. **One table for one data collection point can ensure the best performance of insert and query of a single data collection point to the greatest extent.**
If the metric data of multiple DCPs are traditionally written into a single table, due to the uncontrollable network delay, the timing of the data from different DCPs arriving at the server cannot be guaranteed, the writing operation must be protected by locks, and the metric data from one DCP cannot be guaranteed to be continuously stored together. **One table for one data collection point can ensure the best performance of insert and query of a single data collection point to the greatest extent.**
TDengine suggests using DCP ID as the table name (like D1001 in the above table). Each DCP may collect one or multiple metrics (like the current, voltage, phase as above). Each metric has a corresponding column in the table. The data type for a column can be int, float, string and others. In addition, the first column in the table must be a timestamp. TDengine uses the time stamp as the index, and won’t build the index on any metrics stored. Column wise storage is used.
...
...
@@ -133,27 +133,27 @@ TDengine suggests using DCP ID as the table name (like D1001 in the above table)
The design of one table for one data collection point will require a huge number of tables, which is difficult to manage. Furthermore, applications often need to take aggregation operations among DCPs, thus aggregation operations will become complicated. To support aggregation over multiple tables efficiently, the STable(Super Table) concept is introduced by TDengine.
STable is an set for a type of data collection point. A STable contains a set of data collection points (tables) that have the same schema or data structure, but with different static attributes(tags). To describe a STable, in addition to defining the table structure of the metrics, it is also necessary to define the schema of its tags. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established.
STable is a template for a type of data collection point. A STable contains a set of data collection points (tables) that have the same schema or data structure, but with different static attributes (tags). To describe a STable, in addition to defining the table structure of the metrics, it is also necessary to define the schema of its tags. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established.
In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**.
## Subtable
When creating a table for a specific data collection point, the user can use a STable as a template and specifies the tag value of this specific DCP to create it. **The table created by using a STable as the template is called subtable** in TDengine system. The difference between regular table and subtable is:
When creating a table for a specific data collection point, the user can use a STable as a template and specifies the tag values of this specific DCP to create it. **The table created by using a STable as the template is called subtable** in TDengine. The difference between regular table and subtable is:
1. Subtable is a table, all SQL commands applied on a regular table can be applied on subtable.
2. Subtable is a table with extension, it has static tags(labels), and these tags can be added, deleted, and updated afterward. But regular table does not have tags.
3. A subtable belongs to only one STable, but a STable may have many subtables. Regular table does not belong to any STable.
4. A regular table could not converted into a subtable, and vice versa.
2. Subtable is a table with extensions, it has static tags (labels), and these tags can be added, deleted, and updated after it is created. But a regular table does not have tags.
3. A subtable belongs to only one STable, but a STable may have many subtables. Regular tables do not belong to a STable.
4. A regular table can not be converted into a subtable, and vice versa.
The relationship between a STable and the subtables created based on this STable is as follows:
1. A STable contains multiple subtables with the same metric schema but with different tag values.
2. The schema of metrics or labels cannot be adjusted through subtables, and it can only be changed via STable. Changes to the schema of a STable takes effect immediately for all belonged subtables.
2. The schema of metrics or labels cannot be adjusted through subtables, and it can only be changed via STable. Changes to the schema of a STable takes effect immediately for all associated subtables.
3. STable defines only one template and does not store any data or label information by itself. Therefore, data cannot be written to a STable, only to subtables.
Query can be executed on both table(subtable) and STable. For a query on a STable, TDengine will treat the data in all its subtables as a whole data set for processing. TDengine will first find out the subtables that meet the tag filter conditions, then scan the time-series data of these subtables to perform aggregation operation, which can greatly reduce the data sets to be scanned, thus greatly improving the performance of data aggregation across multiple DCPs.
Queries can be executed on both a table (subtable) and a STable. For a query on a STable, TDengine will treat the data in all its subtables as a whole data set for processing. TDengine will first find the subtables that meet the tag filter conditions, then scan the time-series data of these subtables to perform aggregation operation, which can greatly reduce the data sets to be scanned, thus greatly improving the performance of data aggregation across multiple DCPs.
In TDengine system, it is recommend to use a substable instead of a regular table for a DCP.
In TDengine, it is recommended to use a substable instead of a regular table for a DCP.
## Database
...
...
@@ -167,4 +167,4 @@ FQDN (Fully Qualified Domain Name) is the full domain name of a specific compute
Each node of a TDengine cluster is uniquely identified by an End Point, which consists of an FQDN and a Port, such as h1.tdengine.com:6030. In this way, when the IP changes, we can still use the FQDN to dynamically find the node without changing any configuration of the cluster. In addition, FQDN is used to facilitate unified access to the same cluster from the Intranet and the Internet.
TDengine does not recommend using IP address to access the cluster, which is not good for cluster management.
TDengine does not recommend using an IP address to access the cluster, FQDN is recommended for cluster management.
@@ -10,13 +10,13 @@ import AptGetInstall from "./\_apt_get_install.mdx";
## Quick Install
The full package of TDengine includes server(taosd), taosAdapter for connecting with third-party systems and providing RESTful interface, application driver(taosc), command line program(CLI, taos) and some tools. For current version, the server taosd and taosAdapter can only be installed and run on Linux systems, and will support Windows, macOS and other systems in the future. The application driver taosc and TDengine CLI can be installed and run on Windows or Linux. In addition to the RESTful interface, TDengine also provides connectors for a number of programming languages. In versions before 2.4, there is no taosAdapter, and the RESTfule interface is provided by the built-in http service of taosd.
The full package of TDengine includes the server(taosd), taosAdapter for connecting with third-party systems and providing a RESTful interface, client driver(taosc), command-line program(CLI, taos) and some tools. For the current version, the server taosd and taosAdapter can only be installed and run on Linux systems. In the future taosd and taosAdapter will also be supported on Windows, macOS and other systems. The client driver taosc and TDengine CLI can be installed and run on Windows or Linux. In addition to the [RESTful interface](/reference/taosadapter), TDengine also provides connectors for a number of programming languages. In versions before 2.4, there is no taosAdapter, and the RESTful interface is provided by the built-in HTTP service of taosd.
TDengine supports X64/ARM64/MIPS64/Alpha64 hardware platforms, and will support ARM32, RISC-V and other CPU architectures in the future.
<TabsdefaultValue="apt-get">
<TabItemvalue="docker"label="Docker">
If docker is already installed on your computer, execute the following command:
If docker is already installed on your computer, execute the following command:
```shell
docker run -d-p 6030-6049:6030-6049 -p 6030-6049:6030-6049/udp tdengine/tdengine
...
...
@@ -31,7 +31,7 @@ docker ps
Enter into container and execute bash
```shell
docker exec-it <containrid> bash
docker exec-it <container name> bash
```
Then you can execute the Linux commands and access TDengine.
If you like to check the source code, build the package by youself or contribute to the project, please check [TDengine GitHub Repository](https://github.com/taosdata/TDengine)
If you like to check the source code, build the package by yourself or contribute to the project, please check [TDengine GitHub Repository](https://github.com/taosdata/TDengine)
</TabItem>
</Tabs>
...
...
@@ -71,13 +71,13 @@ Check if taosd is running：
systemctl status taosd
```
If everything is fine，you can run TDengine command line interface `taos` to access TDengine and play around it.
If everything is fine, you can run TDengine command-line interface `taos` to access TDengine and test it out yourself.
:::info
- systemctl requires _root_ privileges，if you are not _root_ ，please add sudo before the command.
- To get feedback and keep polishing the prodcut, TDengine is colleting some basic usage information, but you can turn it off by setting telemetryReporting to 0 in configuration file taos.cfg.
- TDengine uses FQDN (usually hostname）as the ID for a node. To make system work, you need to configure the FQDN for the server running taosd, and configure the DNS service or hosts file on the the machine where the application or TDengine CLI runs to ensure that the FQDN can be resolved.
- To get feedback and keep improving the product, TDengine is collecting some basic usage information, but you can turn it off by setting telemetryReporting to 0 in configuration file taos.cfg.
- TDengine uses FQDN (usually hostname）as the ID for a node. To make the system work, you need to configure the FQDN for the server running taosd, and configure the DNS service or hosts file on the the machine where the application or TDengine CLI runs to ensure that the FQDN can be resolved.
-`systemctl stop taosd` won't stop the server right away, it will wait until all the data in memory are flushed to disk. It may takes time depending on the cache size.
TDengine supports the installation on system which runs [`systemd`](https://en.wikipedia.org/wiki/Systemd) for process management，use `which systemctl` to check if the system has `systemd` installed:
...
...
@@ -91,8 +91,8 @@ If the system does not have `systemd`，you can start TDengine manually by execu
:::note
## Command Line Interface
To manage the TDengine running instance，or execute ad-hoc queries, TDengine provides a Command Line Interface(hereinafter referred to as TDengine CLI) taos. To enter into the interactive CLI，execute `taos` on a Linux terminal where TDengine is installed.
To manage the TDengine running instance，or execute ad-hoc queries, TDengine provides a Command Line Interface(hereinafter referred to as TDengine CLI) taos. To enter into the interactive CLI，execute `taos` on a Linux terminal where TDengine is installed.
```bash
taos
...
...
@@ -104,7 +104,7 @@ If it connects to the TDengine server successfully, it will print out the versio
taos>
```
Inside TDengine CLI，you can execute SQL commands to create/drop database/table, and run queries. The SQL command must be eneded with a semicolon. For example:
Inside TDengine CLI，you can execute SQL commands to create/drop database/table, and run queries. The SQL command must be ended with a semicolon. For example:
```sql
createdatabasedemo;
...
...
@@ -119,24 +119,26 @@ select * from t;
2019-07-1501:00:00.000|20|
QueryOK,2row(s)inset(0.003128s)
```
Besides executing SQL commands, system administrator can check running status, add/drop user accounts and manage the running instances. TAOS CLI with application driver can be installed and run on either Linux or windows machine. For more details on CLI, please [check here](../reference/taos-shell/).
Besides executing SQL commands, system administrators can check running status, add/drop user accounts and manage the running instances. TAOS CLI with client driver can be installed and run on either Linux or Windows machines. For more details on CLI, please [check here](../reference/taos-shell/).
## Experience the blazing fast speed
After TDengine server is running，execute `taosBenchmark`(named as taosdemo before) from a Linux terminal：
After TDengine server is running，execute `taosBenchmark` (previously named taosdemo) from a Linux terminal：
```bash
taosBenchmark
```
This command will create a super table "meters" under database "test". Unde "meters", 10000 tables are created with name from "d0" to "d9999". Each table has 10000 rows and each row has four columns (ts, current, voltage, phase). Time stamp is starting from "2017-07-14 10:40:00 000" to "2017-07-14 10:40:09 999". Each table has tags "location" and "groupId". groupId is set 1 to 10 randomly, and location is set to "beijing" or "shanghai".
This command will insert 100 million rows into database quickly. Depends on the hardware configuration, it only takes a dozen seconds for a regular PC server.
This command will create a super table "meters" under database "test". Under "meters", 10000 tables are created with names from "d0" to "d9999". Each table has 10000 rows and each row has four columns (ts, current, voltage, phase). Time stamp is starting from "2017-07-14 10:40:00 000" to "2017-07-14 10:40:09 999". Each table has tags "location" and "groupId". groupId is set 1 to 10 randomly, and location is set to "beijing" or "shanghai".
This command will insert 100 million rows into the database quickly. Time to insert depends on the hardware configuration, it only takes a dozen seconds for a regular PC server.
taosBenchmark provides you command line options and confuguration file to customize the scenarios, like number of tables, number of rows per table, number of columns and more. Please execute `taosBenchmark --help` to list them. For details on running taosBenchmark, please check [reference for taosBenchmark](/reference/taosbenchmark)
taosBenchmark provides command-line options and a configuration file to customize the scenarios, like number of tables, number of rows per table, number of columns and more. Please execute `taosBenchmark --help` to list them. For details on running taosBenchmark, please check [reference for taosBenchmark](/reference/taosbenchmark)
## Experience query speed
After using taosBenchmark to insert a number of rows data, you can execute queries from TDengine CLI to experience the lightning query speed.
After using taosBenchmark to insert a number of rows data, you can execute queries from TDengine CLI to experience the lightning fast query speed.
query the total number of rows under super table "meters"：
...
...
@@ -162,7 +164,7 @@ query the average, maximum, minimum of all rows with groupId=10:
@@ -26,18 +26,18 @@ Any application programs running on any kind of platforms can access TDengine th
There are two ways for a connector to establish connections to TDengine:
1. Connection through the REST API provided by taosAdapter component, this way is called "REST connection" hereinafter.
2. Connection through the TDengine application driver taosc, this way is called "Native connection" hereinafter.
2. Connection through the TDengine client driver (taosc), this way is called "Native connection" hereinafter.
Either way, same or similar APIs are provided by connectors to access database or execute SQL statements, no obvious difference can be observed.
Key differences：
1. With REST connection, it's not necessary to install TDengine application driver taosc, it's more friendly for cross-platform with the cost of 30% performance downgrade. When taosc has an upgrade, application does not need to make changes.
1. With REST connection, it's not necessary to install TDengine client driver (taosc), it's more friendly for cross-platform with the cost of 30% performance downgrade. When taosc has an upgrade, application does not need to make changes.
2. With native connection, full compatibility of TDengine can be utilized, like [Parameter Binding](/reference/connector/cpp#Parameter Binding-api), [Subscription](reference/connector/cpp#Subscription), etc. But taosc has to be installed, some platforms may not be supported.
## Install Client Driver taosc
If choosing to use native connection and the application is not on the same host as TDengine server, TDengine application driver taosc needs to be installed on the host where the application is. If choosing to use REST connection or the application is on the same host as server side, this step can be skipped. It's better to use same version of taosc as the server.
If choosing to use native connection and the application is not on the same host as TDengine server, TDengine client driver taosc needs to be installed on the host where the application is. If choosing to use REST connection or the application is on the same host as server side, this step can be skipped. It's better to use same version of taosc as the server.
### Install
...
...
@@ -52,7 +52,7 @@ If choosing to use native connection and the application is not on the same host
### Verify
After the above installation and configuration are done and making sure TDengine service is already started and in service, the TDengine command line interface `taos` can be launched to access TDengine.以
After the above installation and configuration are done and making sure TDengine service is already started and in service, the TDengine command-line interface `taos` can be launched to access TDengine.
<TabsdefaultValue="linux"groupId="os">
<TabItemvalue="linux"label="Linux">
...
...
@@ -198,7 +198,7 @@ install.packages("RJDBC")
</TabItem>
<TabItemlabel="C"value="c">
If the client driver taosc is already installed, then the C connector is already available.
If the client driver (taosc) is already installed, then the C connector is already available.
The guide is for developers to quickly learn about the functionalities of TDengine, including fundamentals like data model, inserting data, query and advanced features like data subscription, continuous query. For each functionality, sample code of multiple programming languages are provided for developers to get started quickly.
To develop an application, if you are going to use TDengine as the tool to process time-series data, you shall take the following steps:
1. Choose the way for connection to TDengine. No matter what programming language you use, you can always use the REST interface to access TDengine, but you can also use connectors unique to each programming language.
2. Design the data model based on your own application scenarios. According to the data characteristics, you can decide to create one or more databases; learns about static labels, collected metrics, create the STable with right schema, and create the subtables.
3. Decide how to insert data. TDengine supports writing using standard SQL, but also supports schemaless writing, so that data can be written directly without creating tables manually.
4. Based on business requirements, find out what SQL query statements need to be written.
5. If you want to run real-time analysis based on time series data, including various dashboards, it is recommended that you use the TDengine continuous query feature instead of depolying complex streaming processing systems such as Spark or Flink.
6. If your application has modules that need to consume inserted data, and they need to be notified when new data is inserted, it is recommended that you use the data subscription function provided by TDengine without the need to deploy Kafka.
7. In many scenarios (such as fleet management), the application needs to obtain the latest status of each data collection point. It is recommended that you use the cache function of TDengine instead of deploying Redis separately.
8. If you find that the SQL functions of TDengine cannot meet your requirements, then you can use user-defined functions to solve the problem.
This section is organized in the order described above. For ease of understanding, TDengine provides sample code for each supported programming language for each function. If you want to learn more about the use of SQL, you need to check the [SQL manual](/taos-sql/). For a more in-depth understanding of the use of each connector, please read the [Connector Reference Guide](/reference/connector/). If you also want to integrate TDengine with third-party systems, such as Grafana, please refer to the [third-party tools](/third-party/).
If you encounter any problems during the development process, please click ["Submit an issue"](https://github.com/taosdata/TDengine/issues/new/choose) at the bottom of each page and submit it on GitHub right away.
```mdx-code-block
import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
6. Please be noted that all of the parameters mentioned in this section can be configured in configuration file `taosd.cfg` at server side and used by default, can be override if they are specifically in `create database` statement.
6. Please be noted that all of the parameters mentioned in this section can be configured in configuration file `taosd.cfg` at server side and used by default, can be override if they are specified in `create database` statement.
TDengine provides an API that conforms to REST design standards to support development on various platforms, i.e., the REST API. To minimize learning costs, and unlike other databases' REST API design approaches, TDengine operates on databases directly through HTTP POST requests for SQL statements contained in the BODY, requiring only a URL. See the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1965.html) to use the REST connector.
To support the development of various types of platforms, TDengine provides an API that conforms to the REST principle, namely REST API. To minimize the learning cost, different from the other database REST APIs, TDengine directly requests the SQL command contained in the request BODY through HTTP POST to operate the database and only requires a URL.
Note: One difference from the native connector is that the RESTful interface is stateless, so the `USE db_name` directive has no effect. All references to table names and super table names need to be specified with a database name prefix. (Starting with version 2.2.0.0, RESTful supports specifying db_name in the RESTful URL, so if the database name prefix is not specified in the SQL statement, the db_name specified in the URL will be used. (db_name.)
:::note
One difference from the native connector is that the REST interface is stateless, so the `USE db_name` command has no effect. All references to table names and super table names need to specify the database name prefix. (Since version 2.2.0.0, it is supported to specify db_name in RESTful URL. If the database name prefix is not specified in the SQL command, the `db_name` specified in the URL will be used. Since version 2.4.0.0, REST service is provided by taosAdapter by default. And it requires that the `db_name` must be specified in the URL.)
:::
## Installation
The RESTful interface does not rely on any TDengine libraries, so the client does not need to install any TDengine libraries. The client's development language supports the HTTP protocol.
The REST interface does not rely on any TDengine native library, so the client application does not need to install any TDengine libraries. The client application's development language supports the HTTP protocol is enough.
## Verification
If the TDengine server side is already installed, it can be verified as follows:
If the TDengine server is already installed, it can be verified as follows:
The following is an Ubuntu environment using the curl tool (to confirm that it is installed) to verify that the RESTful interface is working.
The following is an Ubuntu environment using the `curl` tool (to confirm that it is installed) to verify that the REST interface is working.
The following example lists all databases, replacing h1.taosdata.com and 6041 (the default) with the actual running TDengine service FQDN and port number.
The following example lists all databases, replacing `h1.taosdata.com` and `6041` (the default port) with the actual running TDengine service FQDN and port number.
The following return value results indicate that the validation passed.
The following return value results indicate that the verification passed.
```json
{
...
...
@@ -78,13 +79,13 @@ The following return value results indicate that the validation passed.
http://<fqdn>:<port>/rest/sql/[db_name]
```
Parameter Description.
Parameter Description:
- fqnd: FQDN or IP address of any host in the cluster
- port: httpPort configuration item in the configuration file, default is 6041
- db_name: Optional parameter that specifies the default database name for the executed SQL statement. (supported since version 2.2.0.0)
- db_name: Optional parameter that specifies the default database name for the executed SQL command. (supported since version 2.2.0.0)
For example, http://h1.taos.com:6041/rest/sql/test is a URL to h1.taos.com:6041 and sets the default database name to test.
For example, `http://h1.taos.com:6041/rest/sql/test` is a URL to `h1.taos.com:6041` and sets the default database name to `test`.
TDengine supports both Basic authentication and custom authentication mechanisms, and subsequent versions will provide a standard secure digital signature mechanism for authentication.
...
...
@@ -100,9 +101,9 @@ TDengine supports both Basic authentication and custom authentication mechanisms
Authorization: Basic <TOKEN>
```
The HTTP request's BODY is a complete SQL statement, and the data table in the SQL statement should be provided with a database prefix, e.g., \<db_name>. \<tb_name>. If the table name does not have a database prefix and the database name is not specified in the URL, the system will return an error because the HTTP module is a simple forwarder and has no concept of the current DB.
The HTTP request's BODY is a complete SQL command, and the data table in the SQL statement should be provided with a database prefix, e.g., `db_name.tb_name`. If the table name does not have a database prefix and the database name is not specified in the URL, the system will response an error because the HTTP module is a simple forwarder and has no awareness of the current DB.
Use curl to initiate an HTTP Request with a custom authentication method, with the following syntax.
Use `curl` to initiate an HTTP request with a custom authentication method, with the following syntax.
where `TOKEN` is the string after Base64 encoding of `{username}:{password}`, e.g. `root:taosdata` is encoded as `cm9vdDp0YW9zZGF0YQ==`
where `TOKEN` is the string after Base64 encoding of `{username}:{password}`, e.g. `root:taosdata` is encoded as `cm9vdDp0YW9zZGF0YQ==`.
## HTTP Return Format
The return value is in JSON format, as follows:
The return result is in JSON format, as follows:
```json
{
...
...
@@ -133,10 +134,10 @@ The return value is in JSON format, as follows:
}
```
Description.
Description:
- status: tell if the operation result is success or failure.
- head: the definition of the table, or just one column "affected_rows" if no result set is returned. (As of version 2.0.17.0, it is recommended not to rely on the head return value to determine the data column type but rather use column_meta. In future versions, the head item may be removed from the return value.)
- head: the definition of the table, or just one column "affected_rows" if no result set is returned. (As of version 2.0.17.0, it is recommended not to rely on the head return value to determine the data column type but rather use column_meta. In later versions, the head item may be removed from the return value.)
- column_meta: this item is added to the return value to indicate the data type of each column in the data with version 2.0.17.0 and later versions. Each column is described by three values: column name, column type, and type length. For example, `["current",6,4]` means that the column name is "current", the column type is 6, which is the float type, and the type length is 4, which is the float type with 4 bytes. If the column type is binary or nchar, the type length indicates the maximum length of content stored in the column, not the length of the specific data in this return value. When the column type is nchar, the type length indicates the number of Unicode characters that can be saved, not bytes.
- data: The exact data returned, presented row by row, or just [[affected_rows]] if no result set is returned. The order of the data columns in each row of data is the same as that of the data columns described in column_meta.
- rows: Indicates how many rows of data there are.
...
...
@@ -236,7 +237,7 @@ Response body:
### Unix timestamps for result sets
When the HTTP request URL uses `sqlt`, the timestamp of the returned result set will be in Unix timestamp format, for example:
When the HTTP request URL uses `/rest/sqlt`, the returned result set's timestamp value will be in Unix timestamp format, for example:
@@ -298,10 +299,9 @@ Only some configuration parameters related to the RESTful interface are listed b
- httpMaxThreads: the number of threads to start, default is 2 (the default value is rounded down to half of the CPU cores with version 2.0.17.0 and later versions).
- restfulRowLimit: the maximum number of result sets (in JSON format) to return. The default value is 10240.
- httpEnableCompress: whether to support compression, the default is not supported. Currently, TDengine only supports the gzip compression format.
- httpDebugFlag: logging switch, default is 131. 131: error and alarm messages only, 135: debug messages, 143: very detailed debug messages.
- httpDbNameMandatory: users must specify the default database name in the RESTful URL. The default is 0, which turns off this check. If set to 1, users must put a default database name in every RESTful URL. Otherwise, it will return an execution error and reject this SQL statement, regardless of whether the SQL statement executed at this time requires a specified database.
:::note
If you are using the REST API provided by taosd, you should write the above configuration in taosd's configuration file taos.cfg. If you use the REST API of taosAdaper, you need to refer to taosAdaper [corresponding configuration method](/reference/taosadapter/).
If you are using the REST API provided by taosd, you should write the above configuration in taosd's configuration file taos.cfg. If you use the REST API of taosAdapter, you need to refer to taosAdapter [corresponding configuration method](/reference/taosadapter/).
TDengine provides a rich application development interface. To facilitate users to develop their own applications quickly, TDengine supports connectors for multiple programming languages, including official connectors for C/C++, Java, Python, Go, Node.js, C#, and Rust. These connectors support connecting to TDengine clusters using both native interfaces (taosc) and REST interfaces (not supported in some languages yet). Community developers have also contributed several unofficial connectors, such as the ADO.NET connector, the Lua connector, and the PHP connector.
TDengine provides a rich set of APIs (application development interface). To facilitate users to develop their applications quickly, TDengine supports connectors for multiple programming languages, including official connectors for C/C++, Java, Python, Go, Node.js, C#, and Rust. These connectors support connecting to TDengine clusters using both native interfaces (taosc) and REST interfaces (not supported in a few languages yet). Community developers have also contributed several unofficial connectors, such as the ADO.NET connector, the Lua connector, and the PHP connector.
...
...
@@ -11,16 +11,14 @@ TDengine provides a rich application development interface. To facilitate users
Currently, TDengine's native interface connectors can support platforms such as X64/X86/ARM64/ARM32/MIPS/Alpha hardware platforms and Linux/Win64/Win32 development environments. The comparison matrix is as follows.
| **Connection Management** | Support | Support | Support | N/A | Support | Support | Support
| Support | Support | N/A | Support | Support | Support
| Support | Support | N/A | Support | Support | Support
| Support | N/A | Support | N/A | Support | N/A
| | N/A | Support | N/A | Support | N/A
| **Schemaless** | Not supported | N/A | Not supported | Not supported | N/A
| N/A | Not Supported | Not Supported | N/A
| **DataFrame** | Not supported | Support | Not supported | N/A | Not supported | Not supported
| **Connection Management** | Support | Support | Support | N/A | Support | Support |
| **Regular Query** | Support | Support | Support | N/A | Support | Support |
| **Continous Query ** | Support | Support | Support | N/A | Support | Support |
| **Parameter Binding** | Not Supported | Not Supported | Not Supported | N/A | Not Supported | Not Supported |
| **Subscription** | Not Supported | Not Supported | Not Supported | N/A | Not Supported | Not Supported |
| **Schemaless** | Not supported | Not Supported | Not Supported | N/A | Not Supported | Not supported |
| **Bulk Pulling (based on WebSocket) **| Support | Not Supported | Not Supported | N/A | Not Supported | Not Supported |
| **DataFrame** | Not supported | Support | Not supported | N/A | Not supported | Not supported |
:::warning
...
...
@@ -84,7 +82,7 @@ import InstallOnLinux from "./_windows_install.mdx";
import VerifyWindows from "./_verify_windows.mdx";
import VerifyLinux from "./_verify_linux.mdx";
## Install client driver
## Install Client Driver
:::info
The client driver needs to be installed if you use the native interface connector on a system that does not have the TDengine server software installed.
@@ -15,8 +15,8 @@ import PkgList from "/components/PkgList";
Once the package is unzipped, you will see the following files in the directory:
- _ install_client.sh_: install script
- _ taos.tar.gz_: application driver package
- _ driver_: TDengine application driver
- _ taos.tar.gz_: client driver package
- _ driver_: TDengine client driver
- _examples_: some example programs of different programming languages (C/C#/go/JDBC/MATLAB/python/R)
You can run `install_client.sh` to install it.
...
...
@@ -27,7 +27,7 @@ import PkgList from "/components/PkgList";
:::tip
1. If the computer does not run the TDengine service but installs the TDengine application driver, then you need to config `firstEP` in `taos.cfg` only, and there is no need to configure `FQDN`;
1. If the computer does not run the TDengine service but installs the TDengine client driver, then you need to config `firstEP` in `taos.cfg` only, and there is no need to configure `FQDN`;
2. If you encounter the "Unable to resolve FQDN" error, please make sure the FQDN in the `/etc/hosts` file of the current computer is correctly configured, or the DNS service is correctly configured.
Since the TDengine client driver is written in C, using the native connection requires loading the client driver shared library file, which is usually included in the TDengine installer. You can install either standard TDengine server installation package or [TDengine client installtion package](/get-started/). For Windows development, you need to install the corresponding [Windows client](https://www.taosdata.com/cn/all-downloads/#TDengine-Windows-Client) for TDengine.
- libtaos.so: After successful installation of TDengine on a Linux system, the dependent Linux version of the client driver libtaos.so file will be automatically copied to /usr/lib/libtaos.so, which is included in the Linux scanable path and does not need to be specified separately.
- libtaos.so: After successful installation of TDengine on a Linux system, the dependent Linux version of the client driver `libtaos.so` file will be automatically linked to `/usr/lib/libtaos.so`, which is included in the Linux scannable path and does not need to be specified separately.
- taos.dll: After installing the client on Windows, the dependent Windows version of the client driver taos.dll file will be automatically copied to the system default search path C:/Windows/System32, again without the need to specify it separately.
Execute `taos` directly from the Linux shell to connect to the TDengine service and enter the TDengine CLI interface, as shown in the following example.
Execute TDengine CLI program `taos` directly from the Linux shell to connect to the TDengine service and enter the TDengine CLI interface, as shown in the following example.
Go to the C:\TDengine directory from `cmd` and execute `taos.exe` directly to connect to the TDengine service and enter the TDengine CLI interface, for example, as follows:
Go to the `C:\TDengine` directory from `cmd` and execute TDengine CLI program `taos.exe` directly to connect to the TDengine service and enter the TDengine CLI interface, for example, as follows:
@@ -26,6 +26,6 @@ import PkgList from "/components/PkgList";
:::tip
1. If you use FQDN to connect to the server, you must ensure the local network environment DNS is configured, or add FQDN addressing records in the `hosts` file, e.g., edit C:\Windows\system32\drivers\etc\hosts and add a record like the following: `192.168.1.99 h1.tados.com`..
2. Uninstall: Run unins000.exe to uninstall the TDengine application driver.
2. Uninstall: Run unins000.exe to uninstall the TDengine client driver.
@@ -17,9 +17,9 @@ import CSQuery from "../../04-develop/04-query-data/_cs.mdx"
import CSAsyncQuery from "../../04-develop/04-query-data/_cs_async.mdx"
TDengine.Connector` is a C# language connector provided by TDengine that allows C# developers to develop C# applications that access TDengine cluster data.
`TDengine.Connector` is a C# language connector provided by TDengine that allows C# developers to develop C# applications that access TDengine cluster data.
The `TDengine.Connector` connector supports connection to TDengine runtime instances via the TDengine client driver (taosc), providing data writing, querying, subscription, schemaless data writing, bind interface data writing, etc. The `TDengine.Connector` currently does not provide a REST connection. REST connection is not yet available. Users can write their RESTful APIs by referring to the [RESTful APIs](https://docs.taosdata.com//reference/restful-api/) documentation.
The `TDengine.Connector` connector supports connect to TDengine instances via the TDengine client driver (taosc), providing data writing, querying, subscription, schemaless writing, bind interface, etc. The `TDengine.Connector` currently does not provide a REST connection interface. Developers can write their RESTful application by referring to the [RESTful APIs](https://docs.taosdata.com//reference/restful-api/) documentation.
This article describes how to install `TDengine.Connector` in a Linux or Windows environment and connect to TDengine clusters via `TDengine.Connector` to perform basic operations such as data writing and querying.
...
...
@@ -31,15 +31,15 @@ The supported platforms are the same as those supported by the TDengine client d
## Version support
Please refer to [version support list](/reference/connector#versionsupport)
Please refer to [version support list](/reference/connector#version-support)
## Supported features
1. connection management
2. general query
3. continuous query
4. parameter binding
5. subscription function
1. Connection Mmanagement
2. General Query
3. Continuous Query
4. Parameter Binding
5. Subscription
6. Schemaless
## Installation Steps
...
...
@@ -50,7 +50,7 @@ Please refer to [version support list](/reference/connector#version support)
* Install TDengine client driver, please refer to [Install client driver](/reference/connector#Install client driver) for details
### Install using dotnet CLI
### Install via dotnet CLI
<Tabs defaultValue="CLI">
<TabItem value="CLI" label="Get C# driver using dotnet CLI">
...
...
@@ -179,7 +179,7 @@ namespace TDengineExample
1. "Unable to establish connection", "Unable to resolve FQDN"
Usually, because the FQDN configuration is incorrect, you can refer to [How to understand TDengine's FQDN thoroughly](https://www.taosdata.com/blog/2021/07/29/2741.html) to solve it. 2.
Usually, it cause by the FQDN configuration is incorrect, you can refer to [How to understand TDengine's FQDN (Chinese)](https://www.taosdata.com/blog/2021/07/29/2741.html) to solve it. 2.
Unhandled exception. System.DllNotFoundException: Unable to load DLL 'taos' or one of its dependencies: The specified module cannot be found.
@@ -16,9 +16,9 @@ import NodeOpenTSDBJson from "../../04-develop/03-insert-data/_js_opts_json.mdx"
import NodeQuery from "../../04-develop/04-query-data/_js.mdx";
import NodeAsyncQuery from "../../04-develop/04-query-data/_js_async.mdx";
`td2.0-connector` and `td2.0-rest-connector` are the official Node.js language connectors for TDengine, through which Node.js developers can develop applications that can access TDengine cluster data.
`td2.0-connector` and `td2.0-rest-connector` are the official Node.js language connectors for TDengine. Node.js developers can develop applications to access TDengine instance data.
`td2.0-connector` is a **native connector** that connects to TDengine runtime instances via the TDengine client driver (taosc) and supports data writing, querying, subscriptions, schemaless interface, and parameter binding interface. The `td2.0-rest-connector` is a **REST connector** that connects to TDengine runtime instances via the REST interface provided by taosAdapter. The REST connector can run on any platform, but performance is slightly degraded, and the interface implements a somewhat different set of functional features than the native interface.
`td2.0-connector` is a **native connector** that connects to TDengine instances via the TDengine client driver (taosc) and supports data writing, querying, subscriptions, schemaless writing, and bind interface. The `td2.0-rest-connector` is a **REST connector** that connects to TDengine instances via the REST interface provided by taosAdapter. The REST connector can run on any platform, but performance is slightly degraded, and the interface implements a somewhat different set of functional features than the native interface.
The Node.js connector source code is hosted on [GitHub](https://github.com/taosdata/taos-connector-node).
...
...
@@ -29,7 +29,7 @@ The REST connector supports all platforms that can run Node.js.
## Version support
Please refer to [version support list](/reference/connector#versionsupport)
Please refer to [version support list](/reference/connector#version-support)
## Supported features
...
...
@@ -53,7 +53,7 @@ Please refer to [version support list](/reference/connector#version support)
### Pre-installation
- Install the Node.js development environment
- If you are using the REST connector, skip this step. However, if you use the native connector, please install the TDengine client driver. Please refer to [install client driver](/reference/connector#install-client-driver) for more details. We use [node-gyp](https://github.com/nodejs/node-gyp) to interact with TDengine instances and also need to install some dependencies mentioned below depending on the specific OS.
- If you are using the REST connector, skip this step. However, if you use the native connector, please install the TDengine client driver. Please refer to [Install Client Driver](/reference/connector#Install-Client-Driver) for more details. We use [node-gyp](https://github.com/nodejs/node-gyp) to interact with TDengine instances and also need to install some dependencies mentioned below depending on the specific OS.
<Tabs defaultValue="Linux">
<TabItem value="Linux" label="Linux system installation dependencies">
...
...
@@ -68,17 +68,17 @@ Please refer to [version support list](/reference/connector#version support)
- Installation method 1
Use Microsoft's [ windows-build-tools ](https://github.com/felixrieseberg/windows-build-tools) to execute `npm install --global --production` from the `cmd` command-line interface to install all the necessary tools.
Use Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) to execute `npm install --global --production` from the `cmd` command-line interface to install all the necessary tools.
- Installation method 2
Manually install the following tools.
- Install Visual Studio related: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community)
- Install Visual Studio related: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community)
- Install [Python](https://www.python.org/downloads/) 2.7 (`v3.x.x` is not supported) and execute `npm config set python python2.7`.
- Go to the `cmd` commandline interface, `npm config set msvs_version 2017`
- Go to the `cmd` command-line interface, `npm config set msvs_version 2017`
Refer to Microsoft's Node.js User Manual [ Microsoft's Node.js Guidelines for Windows ](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows- environment. md#compiling-native-addon-modules).
Refer to Microsoft's Node.js User Manual [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows- environment. md#compiling-native-addon-modules).
If using ARM64 Node.js on Windows 10 ARM, you must add "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64".
...
...
@@ -106,13 +106,13 @@ npm i td2.0-rest-connector
### Installation verification
After installing the TDengine client, use the nodejsChecker.js program to verify that the current environment supports Node.js access to TDengine.
After installing the TDengine client, use the `nodejsChecker.js` program to verify that the current environment supports Node.js access to TDengine.
Validation method.
Verification in details:
- Create a new installation verification directory, e.g. `~/tdengine-test`, and download the [nodejsChecker.js source code](https://github.com/taosdata/TDengine/tree/develop/examples/nodejs/) from GitHub. nodejsChecker.js) to local.
- Create a new installation verification directory, e.g. `~/tdengine-test`, and download the [nodejsChecker.js source code](https://github.com/taosdata/TDengine/tree/develop/examples/nodejs/) from GitHub. to the work directory.
- Execute the following command from the commandline.
- Execute the following command from the command-line.
```bash
npm init -y
...
...
@@ -120,7 +120,7 @@ npm install td2.0-connector
node nodejsChecker.js host=localhost
```
- After executing the above steps, the command line will output the result of nodejsChecker.js connecting to the TDengine instance and performing a simple insert and query.
- After executing the above steps, the command-line will output the result of `nodejsChecker.js` connecting to the TDengine instance and performing a simple insert and query.
## Establishing a connection
...
...
@@ -148,7 +148,7 @@ conn.close();
```
</TabItem>
<TabItem value="rest" label="REST 连接">
<TabItem value="rest" label="REST connection">
Install and refer to `td2.0-rest-connector`package:
3. "Unable to establish connection", "Unable to resolve FQDN"
Usually, it is because the FQDN is not configured correctly. You can refer to [How to understand TDengine's FQDN thoroughly](https://www.taosdata.com/blog/2021/07/29/2741.html).
Usually, root cause is the FQDN is not configured correctly. You can refer to [How to understand TDengine's FQDN (In Chinese)](https://www.taosdata.com/blog/2021/07/29/2741.html).
@@ -8,10 +8,10 @@ description: "taospy is the official Python connector for TDengine. taospy provi
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
`taospy` is the official Python connector for TDengine. `taospy` provides a rich API that makes it easy for Python applications to use TDengine. `taospy` wraps both the [native interface](/reference/connector/cpp) and [REST interface](/reference/rest-api) of TDengine, which correspond to the `taos` and `taosrest` modules of the `taospy` package, respectively.
In addition to wrapping the native and REST interfaces, `taospy` also provides a programming interface that conforms to the [Python Data Access Specification (PEP 249)](https://peps.python.org/pep-0249/). It is easy to integrate `taospy` with many third-party tools, such as [SQLAlchemy](https://www.sqlalchemy.org/) and [pandas](https://pandas.pydata.org/).
`taospy` is the official Python connector for TDengine. `taospy` provides a rich set of APIs that makes it easy for Python applications to access TDengine. `taospy` wraps both the [native interface](/reference/connector/cpp) and [REST interface](/reference/rest-api) of TDengine, which correspond to the `taos` and `taosrest` modules of the `taospy` package, respectively.
In addition to wrapping the native and REST interfaces, `taospy` also provides a set of programming interfaces that conforms to the [Python Data Access Specification (PEP 249)](https://peps.python.org/pep-0249/). It is easy to integrate `taospy` with many third-party tools, such as [SQLAlchemy](https://www.sqlalchemy.org/) and [pandas](https://pandas.pydata.org/).
The connection to the server directly using the native interface provided by the client driver is referred to hereinafter as a "native connection"; the connection to the server using the REST interface provided by taosAdapter is referred to hereinafter as a "REST connection". ".
The connection to the server directly using the native interface provided by the client driver is referred to hereinafter as a "native connection"; the connection to the server using the REST interface provided by taosAdapter is referred to hereinafter as a "REST connection".
The source code for the Python connector is hosted on [GitHub](https://github.com/taosdata/taos-connector-python).
...
...
@@ -22,33 +22,34 @@ The source code for the Python connector is hosted on [GitHub](https://github.co
## Version selection
We recommend using the latest version of `taospy`, regardless of the version of TDengine used.
We recommend using the latest version of `taospy`, regardless what the version of TDengine is.
## Supported features
- Native connections support all the core features of TDeingine, including connection management, SQL execution, parameter binding, subscriptions, and schemaless writing.
- Native connections support all the core features of TDeingine, including connection management, SQL execution, bind interface, subscriptions, and schemaless writing.
- REST connections support features such as connection management and SQL execution. (SQL execution allows you to: manage databases, tables, and supertables, write data, query data, create continuous queries, etc.).
## Installation
### Preparation
1. Install Python. Python >= 3.6 is recommended. If Python is not available on your system, refer to the [Python BeginnersGuide](https://wiki.python.org/moin/BeginnersGuide/Download) to install it. 2.
Install [pip](https://pypi.org/project/pip/). In most cases, the Python installer comes with the pip utility. If not, please refer to [pip docuemntation](https://pip.pypa.io/en/stable/installation/) to install it.
If you use a native connection, you will also need to [install the client driver](../#install client driver). The client software contains the TDengine client dynamic link library (libtaos.so or taos.dll) and the TDengine CLI.
1. Install Python. Python >= 3.6 is recommended. If Python is not available on your system, refer to the [Python BeginnersGuide](https://wiki.python.org/moin/BeginnersGuide/Download) to install it.
2. Install [pip](https://pypi.org/project/pip/). In most cases, the Python installer comes with the pip utility. If not, please refer to [pip docuemntation](https://pip.pypa.io/en/stable/installation/) to install it.
### Install using pip
If you use a native connection, you will also need to [Install Client Driver](/reference/connector#Install-Client-Driver). The client install package includes the TDengine client dynamic link library (`libtaos.so` or `taos.dll`) and the TDengine CLI.
### Install via pip
#### Uninstalling an older version
If you have previously installed an older version of the Python Connector, please uninstall it beforehand.
If you have installed an older version of the Python Connector, please uninstall it beforehand.
```
pip3 uninstall taos taospy
```
:::note
Earlier TDengine client software includes the Python connector. If the Python connector is installed from the client software's installation directory, the corresponding Python package name is `taos`. So the above uninstall command includes `taos`, and it doesn't matter if it doesn't exist.
Earlier TDengine client software includes the Python connector. If the Python connector is installed from the client package's installation directory, the corresponding Python package name is `taos`. So the above uninstall command includes `taos`, and it doesn't matter if it doesn't exist.
:::
...
...
@@ -57,13 +58,13 @@ Earlier TDengine client software includes the Python connector. If the Python co
<Tabs>
<TabItem label="Install from PyPI" value="pypi">
Install the latest version of
Install the latest version of:
```
pip3 install taospy
```
You can also specify a specific version to install.
You can also specify a specific version to install:
For native connections, you need to verify that both the client driver and the Python connector itself are installed correctly. The client driver and Python connector have been installed properly if you can successfully import the `taos` module. In the Python Interactive Shell, you can type.
For native connection, you need to verify that both the client driver and the Python connector itself are installed correctly. The client driver and Python connector have been installed properly if you can successfully import the `taos` module. In the Python Interactive Shell, you can type.
```python
import taos
...
...
@@ -93,7 +94,7 @@ import taos
</TabItem>
<TabItem value="rest" label="REST connection">
For REST connections, verifying that the ``taosrest`'' module can be imported successfully can be done in the Python Interactive Shell by typing.
For REST connections, verifying that the `taosrest` module can be imported successfully can be done in the Python Interactive Shell by typing.
```python
import taosrest
...
...
@@ -109,7 +110,6 @@ If you have multiple versions of Python on your system, you may have various `pi
C:\> pip3 install taospy
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: taospy in c:\users\username\appdata\local\programs\python\python310\lib\site-packages (2.3.0)
ðŸ™'ðŸ™'
:::
...
...
@@ -122,13 +122,13 @@ Before establishing a connection with the connector, we recommend testing the co
Ensure that the TDengine cluster is up and that the FQDN of the machines in the cluster (the FQDN defaults to hostname if you are starting a standalone version) can be resolved locally, by testing with the ping command.
Ensure that the TDengine instance is up and that the FQDN of the machines in the cluster (the FQDN defaults to hostname if you are starting a standalone version) can be resolved locally, by testing with the `ping` command.
```
ping <FQDN>
```
Then test if the cluster can be appropriately connected with TDengine CLI: ``` ping <FQDN>```
Then test if the cluster can be appropriately connected with TDengine CLI:
```
taos -h <FQDN> -p <PORT>
...
...
@@ -145,7 +145,7 @@ For REST connections and making sure the cluster is up, make sure the taosAdapte
All arguments of the ``connect`` function are optional keyword arguments. The following are the connection parameters specified.
All arguments of the `connect()` function are optional keyword arguments. The following are the connection parameters specified.
- `host` : The FQDN of the node to connect to. There is no default value. If this parameter is not provided, the firstEP in the client configuration file will be connected.
- `user` : The TDengine user name. The default value is `root`.
...
...
@@ -182,11 +182,11 @@ All arguments of the ``connect`` function are optional keyword arguments. The fo
- `timezone` : The timezone used to convert the TIMESTAMP data in the query results to python `datetime` objects. The default is the local timezone.
:::warning
`config` and `timezone` are both process-level configurations. we recommend that all connections made by a process use the same parameter values. Otherwise, unpredictable errors may occur.
`config` and `timezone` are both process-level configurations. We recommend that all connections made by a process use the same parameter values. Otherwise, unpredictable errors may occur.
:::
:::tip
The `connect` function returns a `taos.TaosConnection` instance. In client-side multi-threaded scenarios, we recommend that each thread request a separate connection instance rather than sharing a connection between multiple threads.
The `connect()` function returns a `taos.TaosConnection` instance. In client-side multi-threaded scenarios, we recommend that each thread request a separate connection instance rather than sharing a connection between multiple threads.
:::
...
...
@@ -197,7 +197,7 @@ The `connect` function returns a `taos.TaosConnection` instance. In client-side
The `TaosConnection` class contains both an implementation of the PEP249 Connection interface (e.g., the `cursor` method and the `close` method) and many extensions (e.g., the `execute`, `query`, `schemaless_insert`, and `subscribe` methods). .
The `TaosConnection` class contains both an implementation of the PEP249 Connection interface (e.g., the `cursor()` method and the `close()` method) and many extensions (e.g., the `execute()`, `query()`, `schemaless_insert()`, and `subscribe()` methods).
@@ -232,12 +228,12 @@ The `TaosConnection` class contains both an implementation of the PEP249 Connect
```
:::tip
The queried results can only be fetched once. For example, only one of `featch_all` and `fetch_all_into_dict` can be used in the example above. Repeated fetches will result in an empty list.
The queried results can only be fetched once. For example, only one of `fetch_all()` and `fetch_all_into_dict()` can be used in the example above. Repeated fetches will result in an empty list.
:::
##### Use of TaosResult class
In the above example of using the `TaosConnection` class, we have shown two ways to get the result of a query: `featch_all` and `fetch_all_into_dict`. In addition, `TaosResult` also provides methods to iterate through the result set by rows (`rows_iter`) or by data blocks (`blocks_iter`). Using these two methods will be more efficient in scenarios where the query has a large amount of data.
In the above example of using the `TaosConnection` class, we have shown two ways to get the result of a query: `fetch_all()` and `fetch_all_into_dict()`. In addition, `TaosResult` also provides methods to iterate through the result set by rows (`rows_iter`) or by data blocks (`blocks_iter`). Using these two methods will be more efficient in scenarios where the query has a large amount of data.
@@ -271,7 +267,7 @@ The ``TaosRestCursor`` class is an implementation of the PEP249 Cursor interface
##### Use of the RestClient class
The `RestClient` class is a direct wrapper for the [REST API](/reference/rest-api). It contains only a ``sql()` method for executing arbitrary SQL statements and returning the result.
The `RestClient` class is a direct wrapper for the [REST API](/reference/rest-api). It contains only a `sql()` method for executing arbitrary SQL statements and returning the result.
@@ -15,9 +15,9 @@ import RustOpenTSDBTelnet from "../../04-develop/03-insert-data/_rust_opts_telne
import RustOpenTSDBJson from "../../04-develop/03-insert-data/_rust_opts_json.mdx"
import RustQuery from "../../04-develop/04-query-data/_rust.mdx"
`libtaos` is the official Rust language connector for TDengine, through which Rust developers can develop applications that access the TDengine database.
`libtaos` is the official Rust language connector for TDengine. Rust developers can develop applications to access the TDengine instance data.
`libtaos` provides two ways to establish connections. One is the **Native Connection**, which connects to TDengine runtime instances via the TDengine client driver (taosc). The other is **REST connection**, which connects to TDengine runtime instances via taosAdapter's REST interface. The REST connection supports any platform, but the native connection supports all platforms on which the TDengine client can run.
`libtaos` provides two ways to establish connections. One is the **Native Connection**, which connects to TDengine instances via the TDengine client driver (taosc). The other is **REST connection**, which connects to TDengine instances via taosAdapter's REST interface.
The source code for `libtaos` is hosted on [GitHub](https://github.com/taosdata/libtaos-rs).
...
...
@@ -28,7 +28,7 @@ REST connections are supported on all platforms that can run Rust.
## Version support
Please refer to [version support list](/reference/connector#version support)
Please refer to [version support list](/reference/connector#version-support).
The Rust Connector is still under rapid development and is not guaranteed to be backward compatible before 1.0. Recommend to use TDengine version 2.4 or higher to avoid known issues.
...
...
@@ -36,7 +36,7 @@ The Rust Connector is still under rapid development and is not guaranteed to be
### Pre-installation
* Install the Rust development toolchain
* If using the native connection, please install the TDengine client driver. Please refer to [install client driver](/reference/connector#install client driver)
* If using the native connection, please install the TDengine client driver. Please refer to [install client driver](/reference/connector#install-client-driver)
### Adding libtaos dependencies
...
...
@@ -56,7 +56,7 @@ libtaos = "*"
</TabItem
<TabItem value="rest" label="REST connection">
Add [libtaos][libtaos] to the ``Cargo.toml`'' file and enable the ``rest`'' feature.
Add [libtaos][libtaos] to the `Cargo.toml` file and enable the `rest` feature.
The [Builder Pattern](https://doc.rust-lang.org/1.0.0/style/ownership/builders.html) constructor pattern is Rust's solution for handling complex data types or optional configuration types. The [libtaos] implementation uses the connection constructor [TaosCfgBuilder] as the entry point for the TDengine Rust connector. The [TaosCfgBuilder] provides optional configuration of servers, ports, databases, usernames, passwords, etc.
Using the ``default()` method, you can construct a [TaosCfg] with default parameters for subsequent connections to the database or establishing connection pools.
Using the `default()` method, you can construct a [TaosCfg] with default parameters for subsequent connections to the database or establishing connection pools.
```rust
let cfg = TaosCfgBuilder::default().build()? ;
...
...
@@ -206,7 +206,7 @@ let conn: Taos = cfg.connect();
### Connection pooling
In complex applications, recommand to enable connection pooling. Connection pooling for [libtaos] is implemented using [r2d2].
In complex applications, recommand to enable connection pool. Connection pool for [libtaos] is implemented using [r2d2].
As follows, a connection pool with default parameters can be generated.
...
...
@@ -226,7 +226,7 @@ You can set the same connection pool parameters using the connection pool's cons
.build(cfg);
```
In the application code, use ``pool.get()? ` to get a connection object [Taos].
In the application code, use `pool.get()? ` to get a connection object [Taos].
```rust
let taos = pool.get()? ;
...
...
@@ -234,13 +234,13 @@ let taos = pool.get()? ;
The [Taos] structure is the connection manager in [libtaos] and provides two main APIs.
1. ``exec``: Execute some non-query SQL statements, such as ``CREATE`, ``ALTER`, ``INSERT`, etc.
1. `exec`: Execute some non-query SQL statements, such as `CREATE`, `ALTER`, `INSERT`, etc.
```rust
taos.exec().await?
```
2. ``query``: Execute the query statement and return the [TaosQueryData] object.
2. `query`: Execute the query statement and return the [TaosQueryData] object.
```rust
let q = taos.query("select * from log.logs").await?
...
...
@@ -275,17 +275,17 @@ Note that Rust asynchronous functions and an asynchronous runtime are required.
- `.create_database(database: &str)`: Executes the `CREATE DATABASE` statement.
- `.use_database(database: &str)`: Executes the `USE` statement.
In addition, this structure is also the entry point for [Parameter Binding](#Parameter Binding Interface) and [Row Protocol Interface](#Row Protocol Interface). Please refer to the specific API descriptions for usage.
In addition, this structure is also the entry point for [Parameter Binding](#Parameter Binding Interface) and [Line Protocol Interface](#Line Protocol Interface). Please refer to the specific API descriptions for usage.
### Parameter Binding Interface
### Bind Interface
Similar to the C interface, Rust provides a parameter binding interface. First, create a parameter binding object [Stmt] for a SQL statement from the [Taos] object.
Similar to the C interface, Rust provides the bind interface's wraping. First, create a bind object [Stmt] for a SQL command from the [Taos] object.
```rust
let mut stmt: Stmt = taos.stmt("insert into ? values(? ,?)") ? ;
```
The parameter binding object provides a set of interfaces for implementing parameter binding.
The bind object provides a set of interfaces for implementing parameter binding.
##### `.set_tbname(tbname: impl ToCString)`
...
...
@@ -325,7 +325,7 @@ stmt.execute()? ;
//stmt.execute()? ;
```
### Row protocol interface
### Line protocol interface
The line protocol interface supports multiple modes and different precision and requires the introduction of constants in the schemaless module to set.
@@ -8,7 +8,7 @@ import Prometheus from "./_prometheus.mdx"
import CollectD from "./_collectd.mdx"
import StatsD from "./_statsd.mdx"
import Icinga2 from "./_icinga2.mdx"
import Tcollector from "./_tcollector.mdx"
import TCollector from "./_tcollector.mdx"
taosAdapter is a TDengine companion tool that acts as a bridge and adapter between TDengine clusters and applications. It provides an easy-to-use and efficient way to ingest data directly from data collection agent software such as Telegraf, StatsD, collectd, etc. It also provides an InfluxDB/OpenTSDB compatible data ingestion interface that allows InfluxDB/OpenTSDB applications to be seamlessly ported to TDengine.
...
...
@@ -30,26 +30,26 @@ taosAdapter provides the following features.
### Install taosAdapter
taosAdapter has been part of TDengine server software since TDengine v2.4.0.0. If you use the TDengine server, you don't need additional steps to install taosAdapter. You can download taosAdapter from [Taos Data official website](https://taosdata. com/cn/all-downloads/) to download the TDengine server (taosAdapter is included in v2.4.0.0 and above) installation package. If you need to deploy taosAdapter separately on a server other than the TDengine server, you should install the full TDengine on that server to install taosAdapter. If you need to generate taosAdapter using source code compilation, you can refer to the [Building taosAdapter](https://github.com/taosdata/taosadapter/blob/develop/BUILD-CN.md) documentation.
taosAdapter has been part of TDengine server software since TDengine v2.4.0.0. If you use the TDengine server, you don't need additional steps to install taosAdapter. You can download taosAdapter from [TAOSData official website](https://taosdata.com/en/all-downloads/) to download the TDengine server installation package (taosAdapter is included in v2.4.0.0 and later version). If you need to deploy taosAdapter separately on another server other than the TDengine server, you should install the full TDengine on that server to install taosAdapter. If you need to build taosAdapter from source code, you can refer to the [Building taosAdapter](https://github.com/taosdata/taosadapter/blob/develop/BUILD.md) documentation.
### start/stop taosAdapter
On Linux systems, the taosAdapter service is managed by systemd by default. You can use the command `systemctl start taosadapter` to start the taosAdapter service and use the command `systemctl stop taosadapter` to stop the taosAdapter service.
On Linux systems, the taosAdapter service is managed by `systemd` by default. You can use the command `systemctl start taosadapter` to start the taosAdapter service and use the command `systemctl stop taosadapter` to stop the taosAdapter service.
### Remove taosAdapter
Use the command `rmtaos` to remove the TDengine server software, including taosAdapter.
Use the command `rmtaos` to remove the TDengine server software if you use tar.gz package or use package management command like rpm or apt to remove the TDengine server, including taosAdapter.
### Upgrade taosAdapter
taosAdapter and TDengine server need to use the same version. Please upgrade the taosAdapter by upgrading the TDengine server.
You need to upgrade the taosAdapter deployed separately from taosd by upgrading the TDengine server of the deployed server.
You need to upgrade the taosAdapter deployed separately from TDengine server by upgrading the TDengine server on the deployed server.
## taosAdapter parameter list
taosAdapter supports configuration via command-line arguments, environment variables and configuration files. The default configuration file is /etc/taos/taosadapter.toml.
taosAdapter is configurable via command-line arguments, environment variables and configuration files. The default configuration file is /etc/taos/taosadapter.toml on Linux.
Command-line arguments take precedence over environment variables over configuration files. The commandline usage is arg=val, e.g., taosadapter -p=30000 --debug=true. The detailed list is as follows:
Command-line arguments take precedence over environment variables over configuration files. The command-line usage is arg=val, e.g., taosadapter -p=30000 --debug=true. The detailed list is as follows:
```shell
Usage of taosAdapter:
...
...
@@ -156,7 +156,7 @@ See [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/bl
@@ -165,19 +165,19 @@ See [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/bl
- Seamless connection with StatsD
StatsD is a simple yet powerful daemon for aggregating statistical information. Please visit [https://github.com/statsd/statsd](https://github.com/statsd/statsd) for more information.
- Seamless connection with icinga2
icinga2 is a software that collects inspection result metrics and performance data. Please visit [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) for more information.
- Seamless connection to tcollector
icinga2 is a software that collects inspection result metrics and performance data. Please visit [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) for more information.
- Seamless connection to TCollector
TCollector is a client process that collects data from a local collector and pushes the data to OpenTSDB. Please visit [http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html) for more information.
- Seamless connection to node_exporter
node_export is an exporter for machine metrics. Please visit [https://github.com/prometheus/node_exporter](https://github.com/prometheus/node_exporter) for more information.
- Support for Prometheus remote_read and remote_write
remote_read and remote_write are clustering solutions for Prometheus data read and write separation. Please visit [https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) for more information.
remote_read and remote_write are interfaces for Prometheus data read and write from/to other data storage solution. Please visit [https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) for more information.
## Interfaces
### TDengine RESTful interface
You can use any client that supports the http protocol to write data to or query data from TDengine by accessing the RESTful interface address `http://<fqdn>:6041/<APIEndPoint>`. See the [official documentation](/reference/connector#restful) for details. The following EndPoint is supported.
You can use any client that supports the http protocol to write data to or query data from TDengine by accessing the REST interface address `http://<fqdn>:6041/<APIEndPoint>`. See the [official documentation](/reference/connector#restful) for details. The following EndPoint is supported.
```text
/rest/sql
...
...
@@ -200,7 +200,7 @@ Support InfluxDB query parameters as follows.
-`u` TDengine user name
-`p` TDengine password
Note: InfluxDB token verification is not supported at present. Only Basic verification and query parameter validation are supported.
Note: InfluxDB token authorization is not supported at present. Only Basic authorization and query parameter validation are supported.
### OpenTSDB
...
...
@@ -225,17 +225,17 @@ You can use any client that supports the http protocol to access the Restful int
### TCollector
<Tcollector/>
<TCollector/>
### node_exporter
Exporter of hardware and OS metrics exposed by the \*NIX kernel used by Prometheus
node_export is an exporter of hardware and OS metrics exposed by the \*NIX kernel used by Prometheus
- Enable the taosAdapter configuration node_exporter.enable
- Enable the taosAdapter configuration `node_exporter.enable`
- Set the configuration of the node_exporter
- Restart taosAdapter
### prometheus
### Prometheus
<Prometheus/>
...
...
@@ -246,16 +246,16 @@ taosAdapter will monitor its memory usage during operation and adjust it with tw
- pauseQueryMemoryThreshold
- pauseAllMemoryThreshold
Stops processing query requests when the pauseQueryMemoryThreshold threshold is exceeded.
Stops processing query requests when the `pauseQueryMemoryThreshold` threshold is exceeded.
http response content.
HTTP response content.
- code 503
- body "query memory exceeds threshold"
Stops processing all write and query requests when the pauseAllMemoryThreshold threshold is exceeded.
Stops processing all write and query requests when the `pauseAllMemoryThreshold` threshold is exceeded.
http response: code 503
HTTP response: code 503
- code 503
- body "memory exceeds threshold"
...
...
@@ -266,24 +266,24 @@ Status check interface `http://<fqdn>:6041/-/ping`
- Normal returns `code 200`
- No parameter If memory exceeds pauseAllMemoryThreshold returns `code 503`
- Request parameter `action=query` returns `code 503` if memory exceeds pauseQueryMemoryThreshold or pauseAllMemoryThreshold
- Request parameter `action=query` returns `code 503` if memory exceeds `pauseQueryMemoryThreshold` or `pauseAllMemoryThreshold`
Corresponding configuration parameter
``text
monitor.collectDuration monitoring interval environment variable "TAOS_MONITOR_COLLECT_DURATION" (default value 3s)
monitor.incgroup whether to run in cgroup (set to true for running in container) environment variable "TAOS_MONITOR_INCGROUP"
monitor.pauseAllMemoryThreshold memory threshold for no more inserts and queries environment variable "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default 80)
monitor.pauseQueryMemoryThreshold memory threshold for no more queries Environment variable "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default 70)
monitor.collectDuration monitoring interval environment variable `TAOS_MONITOR_COLLECT_DURATION` (default value 3s)
monitor.incgroup whether to run in cgroup (set to true for running in container) environment variable `TAOS_MONITOR_INCGROUP`
monitor.pauseAllMemoryThreshold memory threshold for no more inserts and queries environment variable `TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD` (default 80)
monitor.pauseQueryMemoryThreshold memory threshold for no more queries Environment variable `TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD` (default 70)
```
You can adjust it according to the specific project application scenario and operation strategy, and it is recommended to use operation monitoring software for timely system memory status monitoring. The load balancer can also check the taosAdapter running status through this interface.
You can adjust it according to the specific application scenario and operation strategy, and it is recommended to use operation monitoring software to monitor system memory status timely. The load balancer can also check the taosAdapter running status through this interface.
## taosAdapter Monitoring Metrics
taosAdapter collects http-related metrics, CPU percentage, and memory percentage.
taosAdapter collects HTTP-related metrics, CPU percentage, and memory percentage.
### http interface
### HTTP interface
Provides an interface conforming to [OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md).
...
...
@@ -293,14 +293,14 @@ http://<fqdn>:6041/metrics
### Write to TDengine
taosAdapter supports writing http monitoring, CPU percentage, and memory percentage to TDengine.
taosAdapter supports writing the metrics of HTTP monitoring, CPU percentage, and memory percentage to TDengine.
@@ -320,13 +320,13 @@ This parameter controls the number of results returned by the following interfac
## Troubleshooting
You can check the taosAdapter running status with the`systemctl status taosadapter` command.
You can check the taosAdapter running status with the `systemctl status taosadapter` command.
You can also adjust the level of the taosAdapter log output by setting the --logLevel parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values are: panic, fatal, error, warn, warning, info, debug and trace.
You can also adjust the level of the taosAdapter log output by setting the `--logLevel` parameter or the environment variable `TAOS_ADAPTER_LOG_LEVEL`. Valid values are: panic, fatal, error, warn, warning, info, debug and trace.
## How to migrate from older TDengine versions to taosAdapter
In TDengine server 2.2.x.x or earlier, the taosd process contains an embedded http service. As mentioned earlier, taosAdapter is a standalone software managed using systemd and has its process. And there are some configuration parameters and behaviors that are different between the two. See the following table.
In TDengine server 2.2.x.x or earlier, the TDengine server process (taosd) contains an embedded HTTP service. As mentioned earlier, taosAdapter is a standalone software managed using `systemd` and has its process ID. And there are some configuration parameters and behaviors that are different between the two. See the following table for details.
@@ -21,9 +21,9 @@ There are two ways to install taosBenchmark:
### Configuration and running methods
taosBenchmark supports two configuration methods: [command line arguments](# command line arguments detailed) and [JSON configuration file](# configuration file arguments detailed). These two methods are mutually exclusive, and with only one command line parameter, users can use `-f <json file>` to specify a configuration file when using a configuration file. When running taosBenchmark with command-line arguments and controlling its behavior, users should use other parameters for configuration rather than `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
taosBenchmark supports two configuration methods: [Command-line arguments](#Command-line arguments in detailed) and [JSON configuration file](#Configuration file arguments in detailed). These two methods are mutually exclusive, and with only one command-line parameter, users can use `-f <json file>` to specify a configuration file when using a configuration file. When running taosBenchmark with command-line arguments and controlling its behavior, users should use other parameters for configuration rather than `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
taosBenchmark supports complete performance testing of TDengine. taosBenchmark supports the TDengine functions in three categories: write, query, and subscribe. These three functions are mutually exclusive, and users can select only one of them each time taosBenchmark runs. It is important to note that the type of functionality to be tested is not configurable when using the commandline configuration method, which can only test writing performance. To test the query and subscription performance of the TDengine, you must use the configuration file method and specify the function type to test via the parameter `filetype` in the configuration file.
taosBenchmark supports complete performance testing of TDengine. taosBenchmark supports the TDengine functions in three categories: write, query, and subscribe. These three functions are mutually exclusive, and users can select only one of them each time taosBenchmark runs. It is important to note that the type of functionality to be tested is not configurable when using the command-line configuration method, which can only test writing performance. To test the query and subscription performance of the TDengine, you must use the configuration file method and specify the function type to test via the parameter `filetype` in the configuration file.
**Make sure that the TDengine cluster is running correctly before running taosBenchmark. **
...
...
@@ -37,9 +37,9 @@ taosBenchmark
When run without parameters, taosBenchmark connects to the TDengine cluster specified in `/etc/taos` by default and creates a database named test in TDengine, a super table named `meters` under the test database, and 10,000 tables under the super table with 10,000 records written to each table. Note that if there is already a test database, this table is not used. Note that if there is already a test database, this command will delete it first and create a new test database.
### Run with commandline configuration parameters
### Run with command-line configuration parameters
The `-f <json file>` argument cannot be used when running taosBenchmark with command-line parameters and controlling its behavior. Users must specify all configuration parameters from the command line. The following is an example of testing taosBenchmark writing performance using the command line approach.
The `-f <json file>` argument cannot be used when running taosBenchmark with command-line parameters and controlling its behavior. Users must specify all configuration parameters from the command-line. The following is an example of testing taosBenchmark writing performance using the command-line approach.
```bash
taosBenchmark -I stmt -n 200 -t 100
...
...
@@ -51,7 +51,7 @@ The above command, `taosBenchmark` will create a database named `test`, create a
A sample configuration file is provided in the taosBenchmark installation package under `<install_directory>/examples/taosbenchmark-json`.
Use the following commandline to run taosBenchmark and control its behavior via a configuration file.
Use the following command-line to run taosBenchmark and control its behavior via a configuration file.
```bash
taosBenchmark -f <json file>
...
...
@@ -81,7 +81,7 @@ taosBenchmark -f <json file>
</details>
#### 订阅场景 JSON 配置文件示例
#### Subscription JSON configuration example
<details>
<summary>subscribe.json</summary>
...
...
@@ -92,10 +92,10 @@ taosBenchmark -f <json file>
</details>
## Command Line Parameters Explained
## Command-line argument in detailed
-**-f/--file <json file\>** :
specify the configuration file to use. This file includes All parameters. And users should not use this parameter with other parameters on the commandline. There is no default value.
specify the configuration file to use. This file includes All parameters. And users should not use this parameter with other parameters on the command-line. There is no default value.
-**-c/--config-dir <dir\>** :
specify the directory where the TDengine cluster configuration file. the default path is `/etc/taos`.
...
...
@@ -204,7 +204,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
-**-? /--help** :
Show help information and exit. Users should not use it with other parameters.
3. back up some super or normal tables in the specified database: use `-dbname stbname1 stbname2 tbname1 tbname2 ... ` parameters. Note that the first parameter of this input sequence is the database name, and only one database is supported. The second and subsequent parameters are the names of super or normal tables in that database, separated by spaces.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` commandline parameter.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` command-line parameter.
5. Loose mode backup: taosdump version 1.4.1 onwards provides `-n` and `-L` parameters for backing up data without using escape characters and "loose" mode, which can reduce the number of backups if table names, column names, tag names do not use This can reduce the backup data time and backup data footprint if table names, column names, and tag names do not use `escape character`. If you are unsure about using `-n` and `-L` conditions, please use the default parameters for "strict" mode backup. See the [official documentation](/taos-sql/escape) for a description of escaped characters.
:::tip
...
...
@@ -48,9 +48,9 @@ taosdump internally uses TDengine stmt binding API for writing recovery data and
:::
## Detailed commandline parameter list
## Detailed command-line parameter list
The following is a detailed list of taosdump commandline arguments.
The following is a detailed list of taosdump command-line arguments.
description:Instructions and tips for using the TDengine CLI
---
The TDengine command-line application (hereafter referred to as TDengine CLI) is the cleanest and most common way for users to manipulate and interact with TDengine instances.
The TDengine command-line application (hereafter referred to as `TDengine CLI`) is the most feasility way for users to manipulate and interact with TDengine instances.
## Installation
If executed on the TDengine server-side, there is no need for additional installation as it is already installed automatically. To run on the non-TDengine server-side, the TDengine client driver needs to be installed. For details, please refer to [connector](/reference/connector/).
If executed on the TDengine server-side, there is no need for additional installation steps to install TDengine CLI as it is already included and installed automatically. To run TDengine CLI on the environemtn which no TDengine server running, the TDengine client installation package needs to be installed first. For details, please refer to [connector](/reference/connector/).
## Execution
To access the TDengine CLI, you can execute `taos` from a Linux terminal or Windows terminal.
To access the TDengine CLI, you can execute `taos`command-line utility from a Linux terminal or Windows terminal.
```bash
taos
```
TDengine will display a welcome message and version information if the connection to the service is successful. If it fails, TDengine will print an error message (see [FAQ](/train-faq/faq) to solve the problem of terminal connection failure to the server.) The TDengine CLI prompt symbols are as follows:
TDengine CLI will display a welcome message and version information if it successfully connected to the TDengine service. If it fails, TDengine CLI will print an error message. See [FAQ](/train-faq/faq) to solve the problem of terminal connection failure to the server. The TDengine CLI prompts as follows:
```cmd
taos>
```
After entering the CLI, you can execute various SQL statements, including inserts, queries, and administrative commands.
After entering the TDengine CLI, you can execute various SQL commands, including inserts, queries, or administrative commands.
## Execute SQL scripts
## Execute SQL script file
Run SQL command scripts in the TDengine CLI via the `source` command.
Run SQL command script file in the TDengine CLI via the `source` command.
```sql
taos>source<filename>;
```
## Modify display character width online
## Adjust display width to show more characters
Users can adjust the character display width in TDengine CLI with the following command:
Users can adjust the display width in TDengine CLI to show more characters with the following command:
```sql
taos>SETMAX_BINARY_DISPLAY_WIDTH<nn>;
...
...
@@ -44,31 +45,31 @@ If the displayed content is followed by `...` you can use this command to change
## Command Line Parameters
You can change the behavior of TDengine CLI by configuring command-line parameters. The following command-line arguments are commonly used.
You can change the behavior of TDengine CLI by specifying command-line parameters. The following parameters are commonly used.
-h, --host=HOST: FQDN of the server where the TDengine server is to be connected. Default is to connect to the local service
-P, --port=PORT: Specify the port number to be used by the server
-u, --user=USER: the user name to use when connecting
-p, --password=PASSWORD: the password to use when connecting to the server
--?, --help: print out all command-line arguments
- -h, --host=HOST: FQDN of the server where the TDengine server is to be connected. Default is to connect to the local service
- -P, --port=PORT: Specify the port number to be used by the server. Default is `6030`
- -u, --user=USER: the user name to use when connecting. Default is `root`
- -p, --password=PASSWORD: the password to use when connecting to the server. Default is `taosdata`
--?, --help: print out all command-line arguments
And many more parameters.
-c, --config-dir: Specify the configuration file directory. The default is `/etc/taos`, and the default name of the configuration file in this directory is taos.cfg
-C, --dump-config: Print the configuration parameters of taos.cfg in the directory specified by -c
-d, --database=DATABASE: Specify the database to use when connecting to the server
-D, --directory=DIRECTORY: Import the SQL script file in the specified path
-f, --file=FILE: Execute the SQL script file in non-interactive mode
-k, --check=CHECK: Specify the table to be checked
-l, --pktlen=PKTLEN: Test package size to be used for network testing
-n, --netrole=NETROLE: test scope for network connection test, default is `startup`, The value can be `client`, `server`, `rpc`, `startup`, `sync`, `speed`, or `fqdn`.
-r, --raw-time: output the time to uint64_t
-s, --commands=COMMAND: execute SQL commands in non-interactive mode
-S, --pkttype=PKTTYPE: Specify the packet type used for network testing. The default is TCP. only `netrole` can be specified as either TCP or UDP when speed is specified
-T, --thread=THREADNUM: The number of threads to import data in multi-threaded mode
-s, --commands: Run TDengine commands without entering the terminal
-z, --timezone=TIMEZONE: Specify time zone. Default is local
-V, --version: Print out the current version number
- -c, --config-dir: Specify the directory where configuration file exists. The default is `/etc/taos`, and the default name of the configuration file in this directory is `taos.cfg`
- -C, --dump-config: Print the configuration parameters of `taos.cfg` in the default directory or specified by -c
- -d, --database=DATABASE: Specify the database to use when connecting to the server
- -D, --directory=DIRECTORY: Import the SQL script file in the specified path
- -f, --file=FILE: Execute the SQL script file in non-interactive mode
- -k, --check=CHECK: Specify the table to be checked
- -l, --pktlen=PKTLEN: Test package size to be used for network testing
- -n, --netrole=NETROLE: test scope for network connection test, default is `startup`, The value can be `client`, `server`, `rpc`, `startup`, `sync`, `speed`, or `fqdn`.
- -r, --raw-time: output the timestamp format as unsigned 64-bits integer (uint64_t in C language)
- -s, --commands=COMMAND: execute SQL commands in non-interactive mode
- -S, --pkttype=PKTTYPE: Specify the packet type used for network testing. The default is TCP. can be specified as either TCP or UDP when `speed` is specified to netrole parameter
- -T, --thread=THREADNUM: The number of threads to import data in multi-threaded mode
- -s, --commands: Run TDengine CLI commands without entering the terminal
- -z, --timezone=TIMEZONE: Specify time zone. Default is the value of current configruation file
- -V, --version: Print out the current version number
- You can use the up and down cursor keys to see the history of commands entered
- Change user password: use `alter user` command in TDengine CLI. The default password is `taosdata`.
-ctrl+c to stop a query in progress
- You can use the up and down keys to iterate the history of commands entered
- Change user password: use `alter user` command in TDengine CLI to change user's password. The default password is `taosdata`.
-use Ctrl+C to stop a query in progress
- Execute `RESET QUERY CACHE` to clear the local cache of the table schema
- Execute SQL statements in batches. You can store a series of shell commands (ending with ;, one line for each SQL statement) in a file and execute the command `source <file-name>` in the shell to execute all SQL statements in that file automatically
- Enter `q` to exit taos shell
- Execute SQL statements in batches. You can store a series of shell commands (ending with ;, one line for each SQL command) in a script file and execute the command `source <file-name>` in the TDengine CLI to execute all SQL commands in that file automatically
Note: ● means officially tested and verified, ○ means unofficially tested and verified.
## List of supported platforms for TDengine clients and connectors
TDengine's connector can support a wide range of platforms, including X64/X86/ARM64/ARM32/MIPS/Alpha hardware platforms and Linux/Win64/Win32 development environments.
本章主要介绍如何在容器中启动 TDengine 服务并访问它。可以在 docker run 命令行中或者 docker-compose 文件中使用环境变量来控制容器中服务的行为。
This chapter describes how to start the TDengine service in a container and access it. Users can control the behavior of the service in the container by using environment variables on the docker run command-line or in the docker-compose file.
## 启动 TDengine
## Starting TDengine
TDengine 镜像启动时默认激活 HTTP 服务，使用下列命令
The TDengine image starts with the HTTP service activated by default, using the following command:
```shell
docker run -d--name tdengine -p 6041:6041 tdengine/tdengine
The above command starts a container named "tdengine" and maps the HTTP service end 6041 to the host port 6041. You can verify that the HTTP service provided in this container is available using the following command.
The TDengine server running in the container uses the container's hostname to establish a connection. Using TDengine CLI or various connectors (such as JDBC-JNI) to access the TDengine inside the container from outside the container is more complicated. So the above is the simplest way to access the TDengine service in the container and is suitable for some simple scenarios. Please refer to the next section if you want to access the TDengine service in the container from containerized using TDengine CLI or various connectors in some complex scenarios.
## 在 host 网络上启动 TDengine
## Start TDengine on the host network
```shell
docker run -d--name tdengine --network host tdengine/tdengine
The above command starts TDengine on the host network and uses the host's FQDN to establish a connection instead of the container's hostname. It works too, like using `systemctl` to start TDengine on the host. If the TDengine client is already installed on the host, you can access it directly with the following command.
```shell
$ taos
Welcome to the TDengine shell from Linux, Client Version:2.4.0.0
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
Copyright (c) 2020 by TAOS Data, Inc.
taos> show dnodes;
id | end_point | vnodes | cores | status | role | create_time | offline reason |
The `TAOS_FQDN` environment variable or the `fqdn` configuration item in `taos.cfg` allows TDengine to establish a connection at the specified hostname. This approach provides greater flexibility for deployment.
The above command starts a TDengine service in the container, which listens to the hostname tdengine, and maps the container's port segment 6030 to 6049 to the host's port segment 6030 to 6049 (both TCP and UDP ports need to be mapped). If the port segment is already occupied on the host, you can modify the above command to specify a free port segment on the host. If `rpcForceTcp` is set to `1`, you can map only the TCP protocol.
- The `VERSION` environment variable is used to set the tdengine image tag
-`TAOS_FIRST_EP` must be set on the newly created instance so that it can join the TDengine cluster; if there is a high availability requirement, `TAOS_SECOND_EP` needs to be used at the same time
-`TAOS_REPLICA` is used to set the default number of database replicas. Its value range is [1,3]
We recommend setting with `TAOS_ARBITRATOR` to use arbitrator in a two-nodes environment.
1. taosAdapter is enabled by default in the TDengine container. If you want to disable it, specify the environment variable `TAOS_DISABLE_ADAPTER=true` at startup
2.同时为了部署灵活起见，可以在独立的容器中启动 taosAdapter
2.At the same time, for flexible deployment, taosAdapter can be started in a separate container
Suppose you want to deploy multiple taosAdapters to improve throughput and provide high availability. In that case, the recommended configuration method uses a reverse proxy such as Nginx to offer a unified access entry. For specific configuration methods, please refer to the official documentation of Nginx. Here is an example:
If you want to deploy a container-based TDengine cluster on multiple hosts, you can use docker swarm. First, to establish a docker swarm cluster on these hosts, please refer to the official docker documentation.
@@ -17,14 +17,14 @@ When TDengine is installed, the following directories or files will be created i
## Executable files
All executable files of TDengine are stored in the _/usr/local/taos/bin_ directory by default. These include.
All executable files of TDengine are in the _/usr/local/taos/bin_ directory by default. These include.
- _taosd_: TDengine server-side executable files
- _taos_: TDengine shell executable
- _taos_: TDengine CLI executable
- _taosdump_: data import and export tool
- _taosBenchmark_: TDengine testing tool
- _remove.sh_: script to uninstall TDengine, please execute it carefully, link to the **rmtaos** command in the /usr/bin directory. Will remove the TDengine installation directory /usr/local/taos, but will keep /etc/taos, /var/lib/taos, /var/log/taos
- _taosadapter_: server-side executable that provides RESTful services and accepts writing requests from a variety of other software
- _remove.sh_: script to uninstall TDengine, please execute it carefully, link to the **rmtaos** command in the /usr/bin directory. Will remove the TDengine installation directory `/usr/local/taos`, but will keep `/etc/taos`, `/var/lib/taos`, `/var/log/taos`
- _taosadapter_: server-side executable that provides RESTful services and accepts writing requests from a variety of other softwares
- _tarbitrator_: provides arbitration for two-node cluster deployments
- _run_taosd_and_taosadapter.sh_: script to start both taosd and taosAdapter
- _TDinsight.sh_: script to download TDinsight and install it
...
...
@@ -32,11 +32,9 @@ All executable files of TDengine are stored in the _/usr/local/taos/bin_ directo
- _taosd-dump-cfg.gdb_: script to facilitate debugging of taosd's gdb execution.
:::note
taosBenchmark and taosdump after version 2.4.0.0 require taosTools as a standalone installation.
taosdump after version 2.4.0.0 require taosTools as a standalone installation. A few version taosBenchmark is include in taosTools too.
:::
:::tip
You can configure different data directories and log directories by modifying the system configuration file taos.cfg.
You can configure different data directories and log directories by modifying the system configuration file `taos.cfg`.
The default database name written by taosAdapter is `collectd`. You can also modify the taosAdapter configuration file dbs entry to specify a different name. user and password are the values configured by the actual TDengine. After changing the configuration file, you need to restart the taosAdapter.
- You can also enable the taosAdapter to receive collectd data by using the taosAdapter commandline parameters or by setting environment variables.
- You can also enable the taosAdapter to receive collectd data by using the taosAdapter command-line parameters or by setting environment variables.
The default database name written by the taosAdapter is `icinga2`. You can also modify the taosAdapter configuration file dbs entry to specify a different name. user and password are the values configured by the actual TDengine. You need to restart the taosAdapter after modification.
- You can also enable taosAdapter to receive icinga2 data by using the taosAdapter commandline parameters or setting environment variables.
- You can also enable taosAdapter to receive icinga2 data by using the taosAdapter command-line parameters or setting environment variables.
The default database name written by taosAdapter is `statsd`. To specify a different name, you can also modify the taosAdapter configuration file db entry. user and password fill in the actual TDengine configuration values. After changing the configuration file, you need to restart the taosAdapter.
- You can also enable taosAdapter to receive StatsD data by using the taosAdapter commandline parameters or setting environment variables.
- You can also enable taosAdapter to receive StatsD data by using the taosAdapter command-line parameters or setting environment variables.
The taosAdapter writes to the database with the default name `tcollector`. You can also modify the taosAdapter configuration file dbs entry to specify a different name. user and password fill in the actual TDengine configuration values. After changing the configuration file, you need to restart the taosAdapter.
- You can also enable taosAdapter to receive tcollector data by using the taosAdapter commandline parameters or setting environment variables.
- You can also enable taosAdapter to receive tcollector data by using the taosAdapter command-line parameters or setting environment variables.
import Tcollector from "../14-reference/_tcollector.mdx"
import TCollector from "../14-reference/_tcollector.mdx"
TCollector is part of openTSDB and collects client computer's logs to send to the database.
...
...
@@ -17,7 +17,7 @@ To write data to the TDengine via TCollector requires the following preparations
- TCollector has been installed. Please refer to [official documentation](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html#installation-of-tcollector) for TCollector installation
MQTT is a popular IoT data transfer protocol, [EMQ](https://github.com/emqx/emqx) is an open-source MQTT Broker software, without any code, only need to use "rules" in EMQ Dashboard to do simple configuration. You can write MQTT data directly to TDengine. EMQ X supports saving data to TDengine by sending it to web services and provides a native TDengine driver for direct saving in the Enterprise Edition. Please refer to the [EMQ official documentation](https://www.emqx.io/docs/en/v4.4/rule/rule-engine.html) for details on how to use it. tdengine).
MQTT is a popular IoT data transfer protocol, [EMQX](https://github.com/emqx/emqx) is an open-source MQTT Broker software, without any code, only need to use "rules" in EMQX Dashboard to do simple configuration. You can write MQTT data directly to TDengine. EMQX supports saving data to TDengine by sending it to web services and provides a native TDengine driver for direct saving in the Enterprise Edition. Please refer to the [EMQX official documentation](https://www.emqx.io/docs/en/v4.4/rule/rule-engine.html) for details on how to use it. tdengine).
## Prerequisites
...
...
@@ -34,7 +34,7 @@ Depending on the current operating system, users can download the installation p
Note: The table schema is based on the blog [(In Chinese) Data Transfer, Storage, Presentation, EMQX + TDengine Build MQTT IoT Data Visualization Platform](https://www.taosdata.com/blog/2020/08/04/1722.html) as an example. Subsequent operations are carried out with this blog scenario too. Please modify it according to your actual application scenario.
Note: The table schema is based on the blog [(In Chinese) Data Transfer, Storage, Presentation, EMQX + TDengine Build MQTT IoT Data Visualization Platform](https://www.taosdata.com/blog/2020/08/04/1722.html) as an example. Subsequent operations are carried out with this blog scenario too. Please modify it according to your actual application scenario.
## Configuring EMQX Rules
...
...
@@ -187,4 +187,4 @@ Use the TDengine CLI program to log in and query the appropriate databases and t
Please refer to the [TDengine official documentation](https://docs.taosdata.com/) for more details on how to use TDengine.
EMQX Please refer to the [EMQ official documentation](https://www.emqx.io/docs/en/v4.4/rule/rule-engine.html) for details on how to use EMQX.
EMQX Please refer to the [EMQX official documentation](https://www.emqx.io/docs/en/v4.4/rule/rule-engine.html) for details on how to use EMQX.
A complete TDengine system runs on one or more physical nodes. Logically, it includes data node (dnode), TDengine application driver (TAOSC) and application (app). There are one or more data nodes in the system, which form a cluster. The application interacts with the TDengine cluster through TAOSC's API. The following is a brief introduction to each logical unit.
A complete TDengine system runs on one or more physical nodes. Logically, it includes data node (dnode), TDengine client driver (TAOSC) and application (app). There are one or more data nodes in the system, which form a cluster. The application interacts with the TDengine cluster through TAOSC's API. The following is a brief introduction to each logical unit.
**Physical node (pnode)**: A pnode is a computer that runs independently and has its own computing, storage and network capabilities. It can be a physical machine, virtual machine, or Docker container installed with OS. The physical node is identified by its configured FQDN (Fully Qualified Domain Name). TDengine relies entirely on FQDN for network communication. If you don't know about FQDN, please check [wikipedia](https://en.wikipedia.org/wiki/Fully_qualified_domain_name).
...
...
@@ -30,7 +30,7 @@ A complete TDengine system runs on one or more physical nodes. Logically, it inc
### Node Communication
**Communication mode**: The communication among each data node of TDengine system, and among the application driver and each data node is carried out through TCP/UDP. Considering an IoT scenario, the data writing packets are generally not large, so TDengine uses UDP in addition to TCP for transmission, because UDP is more efficient and is not limited by the number of connections. TDengine implements its own timeout, retransmission, confirmation and other mechanisms to ensure reliable transmission of UDP. For packets with a data volume of less than 15K, UDP is adopted for transmission, and TCP is automatically adopted for transmission of packets with a data volume of more than 15K or query operations. At the same time, TDengine will automatically compress/decompress the data, digital sign/authenticate the data according to the configuration and data packet. For data replication among data nodes, only TCP is used for data transportation.
**Communication mode**: The communication among each data node of TDengine system, and among the client driver and each data node is carried out through TCP/UDP. Considering an IoT scenario, the data writing packets are generally not large, so TDengine uses UDP in addition to TCP for transmission, because UDP is more efficient and is not limited by the number of connections. TDengine implements its own timeout, retransmission, confirmation and other mechanisms to ensure reliable transmission of UDP. For packets with a data volume of less than 15K, UDP is adopted for transmission, and TCP is automatically adopted for transmission of packets with a data volume of more than 15K or query operations. At the same time, TDengine will automatically compress/decompress the data, digital sign/authenticate the data according to the configuration and data packet. For data replication among data nodes, only TCP is used for data transportation.
**FQDN configuration:** A data node has one or more FQDNs, which can be specified in the system configuration file taos.cfg with the parameter “fqdn”. If it is not specified, the system will automatically use the hostname of the computer as its FQDN. If the node is not configured with FQDN, you can directly set the configuration parameter “fqdn” of the node to its IP address. However, IP is not recommended because IP address may be changed, and once it changes, the cluster will not work properly. The EP (End Point) of a data node consists of FQDN + Port. With FQDN, it is necessary to ensure the DNS service is running, or hosts files on nodes are configured properly.
@@ -34,7 +34,7 @@ Please refer to the [official documentation](https://grafana.com/grafana/downloa
### TDengine
Download the latest TDengine-server 2.4.0.x or above from the [Downloads](http://taosdata.com/cn/all-downloads/) page on the Taos Data website and install it.
Download the latest TDengine-server 2.4.0.x or above from the [Downloads](http://taosdata.com/cn/all-downloads/) page on the TAOSData website and install it.
@@ -99,7 +99,7 @@ This chapter describes the differences between OpenTSDB and TDengine at the syst
TDengine currently only supports Grafana for visual kanban rendering, so if your application uses front-end kanban boards other than Grafana (e.g., [TSDash](https://github.com/facebook/tsdash), [Status Wolf](https://github) .com/box/StatusWolf), etc.). You cannot directly migrate those front-end kanbans to TDengine, and the front-end kanban will need to be ported to Grafana to work correctly.
TDengine version 2.3.0.x only supports collectd and StatsD as data collection aggregation software but will provide more data collection aggregation software in the future. If you use other data aggregators on the collection side, your application needs to be ported to these two data aggregation systems to write data correctly.
In addition to the two data aggregator software protocols mentioned above, TDengine also supports writing data directly via InfluxDB's row protocol and OpenTSDB's data writing protocol, JSON format. You can rewrite the logic on the data push side to write data using the row protocols supported by TDengine.
In addition to the two data aggregator software protocols mentioned above, TDengine also supports writing data directly via InfluxDB's line protocol and OpenTSDB's data writing protocol, JSON format. You can rewrite the logic on the data push side to write data using the line protocols supported by TDengine.
In addition, if your application uses the following features of OpenTSDB, you need to understand the following considerations before migrating your application to TDengine.
Even though it's not recommended to deploy TDengine using docker in production system, docker is still very useful in development environment, especially when your host is not Linux. From version 2.0.14.0, the official image of TDengine can support X86-64, X86, arm64, and rm32 .
下文通过 Step by Step 风格的介绍，讲解如何通过 Docker 快速建立 TDengine 的单节点运行环境，以支持开发和测试。
In this chapter a simple step by step guide of using TDengine in docker is introduced.
这条命令，启动一个运行了 TDengine server 的 docker 容器，并且将容器的 6030 到 6049 端口映射到宿主机的 6030 到 6049 端口上。如果宿主机已经运行了 TDengine server 并占用了相同端口，需要映射容器的端口到不同的未使用端口段。（详情参见 [TDengine 2.0 端口说明](/train-faq/faq#port）。为了支持 TDengine 客户端操作 TDengine server 服务， TCP 和 UDP 端口都需要打开。
In the above command, a docker container is started to run TDengine server, the port range 6030-6049 of the container is mapped to host port range 6030-6049. If port range 6030-6049 has been occupied on the host, please change to an available host port range. Regarding the requirements about ports on the host, please refer to [Port Configuration](/reference/config/#serverport).
-**526aa188da767ae94b244226a2b2eec2b5f17dd8eff592893d9ec0cd0f3a1ccd**：这个返回的长字符是容器 ID，我们也可以通过容器 ID 来查看对应的容器
-**docker run**: Launch a docker container
-**-d**: the container will run in background mode
-**-p**: port mapping
-**tdengine/tdengine**: The image from which to launch the container
-**526aa188da767ae94b244226a2b2eec2b5f17dd8eff592893d9ec0cd0f3a1ccd**: the container ID if successfully launched.
进一步，还可以使用 docker run 命令启动运行 TDengine server 的 docker 容器，并使用 `--name` 命令行参数将容器命名为 `tdengine`，使用 `--hostname` 指定 hostname 为 `tdengine-server`，通过 `-v` 挂载本地目录到容器，实现宿主机与容器内部的数据同步，防止容器删除后，数据丢失。
Furthermore, `--name` can be used with `docker run` to specify name for the container, `--hostname` can be used to specify hostname for the container, `-v` can be used to mount local volumes to the container so that the data generated inside the container can be persisted to disk on the host.
-**--hostname=tdengine-server**：设置容器内 Linux 系统的 hostname，我们可以通过映射 hostname 和 IP 来解决容器 IP 可能变化的问题。
-**-v**：设置宿主机文件目录映射到容器内目录，避免容器删除后数据丢失。
-**--name tdengine**: specify the name of the container, the name can be used to specify the container later
-**--hostname=tdengine-server**: specify the hostname inside the container, the hostname can be used inside the container without worrying the container IP may vary
-**-v**: volume mapping between host and container
### 使用 docker ps 命令确认容器是否已经正确运行
### Check the container
```bash
docker ps
```
输出示例如下：
The output is like below:
```
CONTAINER ID IMAGE COMMAND CREATED STATUS ···
c452519b0f9b tdengine/tdengine "taosd" 14 minutes ago Up 14 minutes ···
```
-**docker ps**：列出所有正在运行状态的容器信息。
-**CONTAINER ID**：容器 ID。
-**IMAGE**：使用的镜像。
-**COMMAND**：启动容器时运行的命令。
-**CREATED**：容器创建时间。
-**STATUS**：容器状态。UP 表示运行中。
-**docker ps**: List all the containers
-**CONTAINER ID**: Container ID
-**IMAGE**: The image used for the container
-**COMMAND**: The command used when launching the container
In TDengine CLI, SQL commands can be executed to create/drop databases, tables, STables, and insert or query data. For details please refer to [TAOS SQL](/taos-sql/).
If `-p` used to map ports properly between host and container, it's also able to access TDengine in container from the host as long as `firstEp` is configured correctly for the client on host.
```
$ taos
...
...
@@ -105,61 +106,59 @@ Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
这条命令，通过 REST API 访问 TDengine server，这时连接的是本机的 6041 端口，可见连接成功。
For details of REST API please refer to [REST API]](/reference/rest-api/).
TDengine REST API 详情请参考[官方文档](/reference/rest-api/)。
### Run TDengine server and taosAdapter inside container
### 使用 Docker 容器运行 TDengine server 和 taosAdapter
From version 2.4.0.0, in the TDengine Docker image, `taosAdapter` is enabled by default, but can be disabled using environment variable `TAOS_DISABLE_ADAPTER=true` . `taosAdapter` can also be run alone without `taosd` when launching a container.
- Run `taosAdapter` only in docker container, `TAOS_FIRST_EP` environment variable needs to be used to specify the container name in which `taosd` is running:
Once the execution is finished, a database `test` is created, a STable `meters` is created in database `test`, 10,000 sub tables are created using `meters` as template, named as "d0" to "d9999", while 10,000 rows are inserted into each table, so totally 100,000,000 rows are inserted.
A lot of 3rd party tools can be used to write data into TDengine through `taosAdapter` , for details please refer to [3rd party tools](/third-party/).
可以看到模拟数据已经被写入到 TDengine 中。
There is nothing different from the 3rd party side to access TDengine server inside a container, as long as the end point is specified correctly, the end point should be the FQDN and the mapped port of the host.
TDengine supports multiple ways to write data, including SQL, Prometheus, Telegraf, collectd, StatsD, EMQ MQTT Broker, HiveMQ Broker, CSV file, etc. Kafka, OPC and other interfaces will be provided in the future. Data can be inserted in one single record or in batches, data from one or multiple data collection points can be inserted at the same time. TDengine supports multi-thread insertion, out-of-order data insertion, and also historical data insertion.
TDengine supports multiple ways to write data, including SQL, Prometheus, Telegraf, collectd, StatsD, EMQX MQTT Broker, HiveMQ Broker, CSV file, etc. Kafka, OPC and other interfaces will be provided in the future. Data can be inserted in one single record or in batches, data from one or multiple data collection points can be inserted at the same time. TDengine supports multi-thread insertion, out-of-order data insertion, and also historical data insertion.
## <a class="anchor" id="sql"></a> Data Writing via SQL
...
...
@@ -303,9 +303,9 @@ TCollector is a client-side process that gathers data from local collectors and
Please find taosAdapter configuration and usage from `taosadapter --help` output.
## <a class="anchor" id="emq"></a> Data Writing via EMQ Broker
## <a class="anchor" id="emq"></a> Data Writing via EMQX Broker
[EMQ](https://github.com/emqx/emqx) is an open source MQTT Broker software, with no need of coding, only to use "rules" in EMQ Dashboard for simple configuration, and MQTT data can be directly written into TDengine. EMQX supports storing data to the TDengine by sending it to a Web service, and also provides a native TDengine driver on Enterprise Edition for direct data store. Please refer to [EMQ official documents](https://docs.emqx.io/broker/latest/cn/rule/rule-example.html#%E4%BF%9D%E5%AD%98%E6%95%B0%E6%8D%AE%E5%88%B0-tdengine) for more details.
[EMQX](https://github.com/emqx/emqx) is an open source MQTT Broker software, with no need of coding, only to use "rules" in EMQX Dashboard for simple configuration, and MQTT data can be directly written into TDengine. EMQX supports storing data to the TDengine by sending it to a Web service, and also provides a native TDengine driver on Enterprise Edition for direct data store. Please refer to [EMQX official documents](https://docs.emqx.io/broker/latest/cn/rule/rule-example.html#%E4%BF%9D%E5%AD%98%E6%95%B0%E6%8D%AE%E5%88%B0-tdengine) for more details.
## <a class="anchor" id="hivemq"></a> Data Writing via HiveMQ Broker
query_sql=f'select count(*), avg(c6), sum(c3) from (select * from {tb_name} where c1 >1 or c2 = 2 and c7 like "binar_" and c4 in (3, 5)) where c1 != 2 or c3 = 1 or t1=2 or t1=3 or c8 like "ncha_" and c9 in (true) interval(8d)'
res=tdSql.query(query_sql,True)
tdSql.checkRows(3)
tdSql.checkEqual(int(res[0][1]),17)
tdSql.checkEqual(int(res[0][1]),15)
tdSql.checkEqual(int(res[0][2]),1)
tdSql.checkEqual(int(res[0][3]),38)
tdSql.checkEqual(int(res[1][1]),10)
tdSql.checkEqual(int(res[1][2]),2)
tdSql.checkEqual(int(res[1][3]),17)
tdSql.checkEqual(int(res[2][1]),8)
tdSql.checkEqual(int(res[0][3]),50)
tdSql.checkEqual(int(res[1][1]),15)
tdSql.checkEqual(int(res[1][2]),3)
tdSql.checkEqual(int(res[1][3]),15)
tdSql.checkEqual(int(res[2][1]),5)
tdSql.checkEqual(int(res[2][2]),1)
tdSql.checkEqual(int(res[2][3]),15)
tdSql.checkEqual(int(res[2][3]),5)
## select count avg sum from (condition_A and condition_B and and line and in and ts and condition_tag_A and condition_tag_B and between) where condition_C orr condition_D or condition_tag_C or condition_tag_D or like and in interval
query_sql=f'select count(*), avg(c6), sum(c3) from (select * from {tb_name} where c1 >= 1 and c2 = 2 and c7 like "binar_" and c4 in (3, 5) and ts > "2021-01-11 12:00:00" and t1 < 2 and t1 > 0 and c6 between 0 and 7) where c1 != 2 or c3 = 1 or t1=2 or t1=3 or c8 like "ncha_" and c9 in (true) interval(8d)'
