diff --git a/documentation20/cn/00.index/docs.md b/documentation20/cn/00.index/docs.md index 04de20fd6238e5a5840b4791bf8b9ac0fa1470fa..49cfa12119a9e7bab1926a9ccb959f96d43dba9c 100644 --- a/documentation20/cn/00.index/docs.md +++ b/documentation20/cn/00.index/docs.md @@ -15,7 +15,7 @@ TDengine是一个高效的存储、查询、分析时序大数据的平台，专 * [命令行程序TAOS](/getting-started#console)：访问TDengine的简便方式 * [极速体验](/getting-started#demo)：运行示例程序，快速体验高效的数据插入、查询 * [支持平台列表](/getting-started#platforms)：TDengine服务器和客户端支持的平台列表 -* [Kubenetes部署](https://taosdata.github.io/TDengine-Operator/zh/index.html)：TDengine在Kubenetes环境进行部署的详细说明 +* [Kubernetes部署](https://taosdata.github.io/TDengine-Operator/zh/index.html)：TDengine在Kubernetes环境进行部署的详细说明 ## [整体架构](/architecture) diff --git a/documentation20/cn/08.connector/docs.md b/documentation20/cn/08.connector/docs.md index 8c22d50185d1c30459487682800241b48e782d95..991c3ce6cea887bb7ab1bb7525afa5a549276e76 100644 --- a/documentation20/cn/08.connector/docs.md +++ b/documentation20/cn/08.connector/docs.md @@ -899,7 +899,7 @@ go env -w GOPROXY=https://goproxy.io,direct Node.js连接器支持的系统有： -| **CPU类型** | x64（64bit） | | | aarch64 | aarch32 | +|**CPU类型** | x64（64bit） | | | aarch64 | aarch32 | | ------------ | ------------ | -------- | -------- | -------- | -------- | | **OS类型** | Linux | Win64 | Win32 | Linux | Linux | | **支持与否** | **支持** | **支持** | **支持** | **支持** | **支持** | diff --git a/documentation20/cn/11.administrator/docs.md b/documentation20/cn/11.administrator/docs.md index d0e82a16c8a1faf5568b2b4dd930af5af6646bf4..9e1f627709c3ad2e45e618b6f2b91ab2e80c1f10 100644 --- a/documentation20/cn/11.administrator/docs.md +++ b/documentation20/cn/11.administrator/docs.md @@ -444,7 +444,7 @@ TDengine的所有可执行文件默认存放在 _/usr/local/taos/bin_ 目录下 - 数据库名：不能包含“.”以及特殊字符，不能超过 32 个字符 - 表名：不能包含“.”以及特殊字符，与所属数据库名一起，不能超过 192 个字符 - 表的列名：不能包含特殊字符，不能超过 64 个字符 -- 数据库名、表名、列名，都不能以数字开头 +- 数据库名、表名、列名，都不能以数字开头，合法的可用字符集是“英文字符、数字和下划线” - 表的列数：不能超过 1024 列 - 记录的最大长度：包括时间戳 8 byte，不能超过 16KB（每个 BINARY/NCHAR 类型的列还会额外占用 2 个 byte 的存储位置） - 单条 SQL 语句默认最大字符串长度：65480 byte diff --git a/documentation20/en/00.index/docs.md b/documentation20/en/00.index/docs.md index 381f914e4a2e56e85908712a01d57ad2821de9f6..13f8a8565efea48ebb69702c1447ffd1c2b56b2f 100644 --- a/documentation20/en/00.index/docs.md +++ b/documentation20/en/00.index/docs.md @@ -16,6 +16,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series - [Command-line](/getting-started#console) : an easy way to access TDengine server - [Experience Lightning Speed](/getting-started#demo): running a demo, inserting/querying data to experience faster speed - [List of Supported Platforms](/getting-started#platforms): a list of platforms supported by TDengine server and client +- [Deploy to Kubernetes](https://taosdata.github.io/TDengine-Operator/en/index.html)：a detailed guide for TDengine deployment in Kubernetes environment ## [Overall Architecture](/architecture) @@ -28,7 +29,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series ## [Data Modeling](/model) -- [Create a Library](/model#create-db): create a library for all data collection points with similar features +- [Create a Database](/model#create-db): create a database for all data collection points with similar features - [Create a Super Table(STable)](/model#create-stable): create a STable for all data collection points with the same type - [Create a Table](/model#create-table): use STable as the template, to create a table for each data collecting point @@ -70,7 +71,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series ## [Connector](/connector) - [C/C++ Connector](/connector#c-cpp): primary method to connect to TDengine server through libtaos client library -- [Java Connector(JDBC)](/connector/java): driver for connecting to the server from Java applications using the JDBC API +- [Java Connector(JDBC)]: driver for connecting to the server from Java applications using the JDBC API - [Python Connector](/connector#python): driver for connecting to TDengine server from Python applications - [RESTful Connector](/connector#restful): a simple way to interact with TDengine via HTTP - [Go Connector](/connector#go): driver for connecting to TDengine server from Go applications @@ -111,8 +112,8 @@ TDengine is a highly efficient platform to store, query, and analyze time-series ## TDengine Technical Design -- [System Module](/architecture/taosd): taosd functions and modules partitioning -- [Data Replication](/architecture/replica): support real-time synchronous/asynchronous replication, to ensure high-availability of the system +- [System Module]: taosd functions and modules partitioning +- [Data Replication]: support real-time synchronous/asynchronous replication, to ensure high-availability of the system - [Technical Blog](https://www.taosdata.com/cn/blog/?categories=3): More technical analysis and architecture design articles ## Common Tools @@ -121,7 +122,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series - [TDengine performance comparison test tools](https://www.taosdata.com/blog/2020/01/18/1166.html) - [Use TDengine visually through IDEA Database Management Tool](https://www.taosdata.com/blog/2020/08/27/1767.html) -## [Performance: TDengine vs Others +## Performance: TDengine vs Others - [Performance: TDengine vs InfluxDB with InfluxDB’s open-source performance testing tool](https://www.taosdata.com/blog/2020/01/13/1105.html) - [Performance: TDengine vs OpenTSDB](https://www.taosdata.com/blog/2019/08/21/621.html) @@ -138,4 +139,4 @@ TDengine is a highly efficient platform to store, query, and analyze time-series ## FAQ -- [FAQ: Common questions and answers](/faq) \ No newline at end of file +- [FAQ: Common questions and answers](/faq) diff --git a/documentation20/en/01.evaluation/docs.md b/documentation20/en/01.evaluation/docs.md index 2b9ed68015b209846a3e5ccaf5c1781c83e8540c..89fb6443cb45bac411b0fa5238b4f5b224931c39 100644 --- a/documentation20/en/01.evaluation/docs.md +++ b/documentation20/en/01.evaluation/docs.md @@ -43,26 +43,23 @@ From the perspective of data sources, designers can analyze the applicability of ### System Function Requirements -| | | | | | -| ----------------------------------------------------- | ------------------ | -------------------- | ------------------- | ------------------------------------------------------------ | -| **System Function Requirements** | **Not Applicable** | **Might Applicable** | **Very Applicable** | **Description** | +| **System Architecture Requirements** | **Not Applicable** | **Might Be Applicable** | **Very Applicable** | **Description** | +| ------------------------------------------------- | ------------------ | ----------------------- | ------------------- | ------------------------------------------------------------ | | Require completed data processing algorithms built-in | | √ | | TDengine implements various general data processing algorithms, but has not properly handled all requirements of different industries, so special types of processing shall be processed at the application level. | | Require a huge amount of crosstab queries | | √ | | This type of processing should be handled more by relational database systems, or TDengine and relational database systems should fit together to implement system functions. | ### System Performance Requirements -| | | | | | -| -------------------------------------------- | ------------------ | -------------------- | ------------------- | ------------------------------------------------------------ | -| **System Performance Requirements** | **Not Applicable** | **Might Applicable** | **Very Applicable** | **Description** | +| **System Architecture Requirements** | **Not Applicable** | **Might Be Applicable** | **Very Applicable** | **Description** | +| ------------------------------------------------- | ------------------ | ----------------------- | ------------------- | ------------------------------------------------------------ | | Require larger total processing capacity | | | √ | TDengine’s cluster functions can easily improve processing capacity via multi-server-cooperating. | | Require high-speed data processing | | | √ | TDengine’s storage and data processing are designed to be optimized for IoT, can generally improve the processing speed by multiple times than other similar products. | | Require fast processing of fine-grained data | | | √ | TDengine has achieved the same level of performance with relational and NoSQL data processing systems. | ### System Maintenance Requirements -| | | | | | -| -------------------------------------------- | ------------------ | -------------------- | ------------------- | ------------------------------------------------------------ | -| **System Maintenance Requirements** | **Not Applicable** | **Might Applicable** | **Very Applicable** | **Description** | +| **System Architecture Requirements** | **Not Applicable** | **Might Be Applicable** | **Very Applicable** | **Description** | +| ------------------------------------------------- | ------------------ | ----------------------- | ------------------- | ------------------------------------------------------------ | | Require system with high-reliability | | | √ | TDengine has a very robust and reliable system architecture to implement simple and convenient daily operation with streamlined experiences for operators, thus human errors and accidents are eliminated to the greatest extent. | | Require controllable operation learning cost | | | √ | As above. | -| Require abundant talent supply | √ | | | As a new-generation product, it’s still difficult to find talents with TDengine experiences from market. However, the learning cost is low. As the vendor, we also provide extensive operation training and counselling services. | \ No newline at end of file +| Require abundant talent supply | √ | | | As a new-generation product, it’s still difficult to find talents with TDengine experiences from market. However, the learning cost is low. As the vendor, we also provide extensive operation training and counselling services. | diff --git a/documentation20/en/02.getting-started/docs.md b/documentation20/en/02.getting-started/docs.md index 6d6bda114eecd303dd46ce21ca72768fe0bc5916..fcfb88a6fe2fd48e39241a2ba7e7d2b8e347a3ff 100644 --- a/documentation20/en/02.getting-started/docs.md +++ b/documentation20/en/02.getting-started/docs.md @@ -25,21 +25,13 @@ For more about installation process, please refer [TDengine Installation Package After installation, you can start the TDengine service by the `systemctl` command. ```bash -``` - $ systemctl start taosd - -``` ``` Then check if the service is working now. ```bash -``` - $ systemctl status taosd - -``` ``` If the service is running successfully, you can play around through TDengine shell `taos`. @@ -50,13 +42,11 @@ If the service is running successfully, you can play around through TDengine she - To get better product feedback and improve our solution, TDegnine will collect basic usage information, but you can modify the configuration parameter **telemetryReporting** in the system configuration file taos.cfg, and set it to 0 to turn it off. - TDegnine uses FQDN (usually hostname) as the node ID. In order to ensure normal operation, you need to set hostname for the server running taosd, and configure DNS service or hosts file for the machine running client application, to ensure the FQDN can be resolved. - TDengine supports installation on Linux systems with[ systemd ](https://en.wikipedia.org/wiki/Systemd)as the process service management, and uses `which systemctl` command to detect whether `systemd` packages exist in the system: -- ```bash + + ```bash + $ which systemctl ``` -- $ which systemctl - -- ``` - If `systemd` is not supported in the system, TDengine service can also be launched via `/usr/local/taos/bin/taosd` manually. ## TDengine Shell Command Line @@ -64,28 +54,18 @@ If `systemd` is not supported in the system, TDengine service can also be launch To launch TDengine shell, the command line interface, in a Linux terminal, type: ```bash -``` - $ taos - -``` ``` The welcome message is printed if the shell connects to TDengine server successfully, otherwise, an error message will be printed (refer to our [FAQ](https://www.taosdata.com/en/faq) page for troubleshooting the connection error). The TDengine shell prompt is: ```cmd -``` - taos> - -``` ``` In the TDengine shell, you can create databases, create tables and insert/query data with SQL. Each query command ends with a semicolon. It works like MySQL, for example: ```mysql -``` - create database demo; use demo; @@ -107,8 +87,6 @@ ts | speed | 19-07-15 01:00:00.000| 20| Query OK, 2 row(s) in set (0.001700s) - -``` ``` Besides the SQL commands, the system administrator can check system status, add or delete accounts, and manage the servers. @@ -127,11 +105,7 @@ You can configure command parameters to change how TDengine shell executes. Some Examples: ```bash -``` - $ taos -h 192.168.0.1 -s "use db; show tables;" - -``` ``` ### Run SQL Command Scripts @@ -139,11 +113,7 @@ $ taos -h 192.168.0.1 -s "use db; show tables;" Inside TDengine shell, you can run SQL scripts in a file with source command. ```mysql -``` - taos> source ; - -``` ``` ### Shell Tips @@ -158,11 +128,7 @@ taos> source ; After starting the TDengine server, you can execute the command `taosdemo` in the Linux terminal. ```bash -``` - $ taosdemo - -``` ``` Using this command, a STable named `meters` will be created in the database `test` There are 10k tables under this stable, named from `t0` to `t9999`. In each table there are 100k rows of records, each row with columns （`f1`, `f2` and `f3`. The timestamp is from "2017-07-14 10:40:00 000" to "2017-07-14 10:41:39 999". Each table also has tags `areaid` and `loc`: `areaid` is set from 1 to 10, `loc` is set to "beijing" or "shanghai". @@ -174,51 +140,31 @@ In the TDengine client, enter sql query commands and then experience our lightni - query total rows of records： ```mysql -``` - taos> select count(*) from test.meters; - -``` ``` - query average, max and min of the total 1 billion records： ```mysql -``` - taos> select avg(f1), max(f2), min(f3) from test.meters; - -``` ``` - query the number of records where loc="beijing": ```mysql -``` - taos> select count(*) from test.meters where loc="beijing"; - -``` ``` - query the average, max and min of total records where areaid=10： ```mysql -``` - taos> select avg(f1), max(f2), min(f3) from test.meters where areaid=10; - -``` ``` - query the average, max, min from table t10 when aggregating over every 10s: ```mysql -``` - taos> select avg(f1), max(f2), min(f3) from test.t10 interval(10s); - -``` ``` **Note**: you can run command `taosdemo` with many options, like number of tables, rows of records and so on. To know more about these options, you can execute `taosdemo --help` and then take a try using different options. @@ -274,4 +220,4 @@ Comparison matrix as following: Note: ● has been verified by official tests; ○ has been verified by unofficial tests. -Please visit [Connectors](https://www.taosdata.com/cn/documentation/connector) section for more detailed information. \ No newline at end of file +Please visit [Connectors](https://www.taosdata.com/en/documentation/connector) section for more detailed information. diff --git a/documentation20/en/03.architecture/docs.md b/documentation20/en/03.architecture/docs.md new file mode 100644 index 0000000000000000000000000000000000000000..2e5fc7bd18e7a61d30b658678dab02d7a9ca2cae --- /dev/null +++ b/documentation20/en/03.architecture/docs.md @@ -0,0 +1,436 @@ +# Data Model and Architecture + +## Data Model + +### A Typical IoT Scenario + +In typical IoT, Internet of Vehicles and Operation Monitoring scenarios, there are often many different types of data collecting devices that collect one or more different physical metrics. However, for the collection devices of the same type, there are often many specific collection devices distributed in places. BigData processing system aims to collect all kinds of data, and then calculate and analyze them. For the same kind of devices, the data collected are very regular. Taking smart meters as an example, assuming that each smart meter collects three metrics of current, voltage and phase, the collected data are similar to the following table: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Device IDTime StampCollected MetricsTags
Device IDTime StampcurrentvoltagephaselocationgroupId
d1001153854868500010.32190.31Beijing.Chaoyang2
d1002153854868400010.22200.23Beijing.Chaoyang3
d1003153854868650011.52210.35Beijing.Haidian3
d1004153854868550013.42230.29Beijing.Haidian2
d1001153854869500012.62180.33Beijing.Chaoyang2
d1004153854869660011.82210.28Beijing.Haidian2
d1002153854869665010.32180.25Beijing.Chaoyang3
d1001153854869680012.32210.31Beijing.Chaoyang2
+ +
Table 1: Smart meter example data
+ +Each data record contains the device ID, timestamp, collected metrics (current, voltage, phase as above), and static tags (Location and groupId in Table 1) associated with the devices. Each device generates a data record in a pre-defined timer or triggered by an external event. It is a sequence of data points like a stream. + +### Data Characteristics + +As the data points are a series of data points over time, the data points generated by IoT, Internet of Vehicles, and Operation Monitoring have some strong common characteristics: + +1. Metrics are always structured data; +2. There are rarely delete/update operations on collected data; +3. No need for transactions of traditional databases +4. The ratio of reading is lower but write is higher than typical Internet applications; +5. data flow is uniform and can be predicted according to the number of devices and collection frequency; +6. the user pays attention to the trend of data, not a specific value at a specific time; +7. there is always a data retention policy; +8. the data query is always executed in a given time range and a subset of space; +9. in addition to storage and query operations, various statistical and real-time calculation operations are also required; +10. data volume is huge, a system may generate over 10 billion data points in a day. + +By utilizing the above characteristics, TDengine designs the storage and computing engine in a special and optimized way for time-series data, resulting in massive improvements in system efficiency. + +### Relational Database Model + +Since time-series data is most likely to be structured data, TDengine adopts the traditional relational database model to process them with a shallow learning curve. You need to create a database, create tables with schema definitions, then insert data points and execute queries to explore the data. Standard SQL is used, instead of NoSQL’s key-value storage. + +### One Table for One Collection Point + +To utilize this time-series and other data features, TDengine requires the user to create a table for each collection point to store collected time-series data. For example, if there are over 10 millions smart meters, means 10 millions tables shall be created. For the table above, 4 tables shall be created for devices D1001, D1002, D1003, and D1004 to store the data collected. This design has several advantages: + +1. Guarantee that all data from a collection point can be saved in a continuous memory/hard disk space block by block. If queries are applied only on one point in a time range, this design will reduce the random read latency significantly, thus increase read and query speed by orders of magnitude. +2. Since the data generation process of each collection device is completely independent, means each device has its unique data source, thus writes can be carried out in a lock-free manner to greatly improve the speed. +3. Write latency can be significantly reduced too as the data points generated by the same device will arrive in time order, the new data point will be simply appended to a block. + +If the data of multiple devices are written into a table in the traditional way, due to the uncontrollable network delay, the timing of the data from different devices arriving at the server cannot be guaranteed, the writing operation must be protected by locks, and the data of one device cannot be guaranteed to continuously stored together. **The method of one table for each data collection point can ensure the optimal performance of insertion and query of a single data collection point to the greatest extent.** + +TDengine suggests using collection point ID as the table name (like D1001 in the above table). Each point may collect one or more metrics (like the current, voltage, phase as above). Each metric has a column in the table. The data type for a column can be int, float, string and others. In addition, the first column in the table must be a timestamp. TDengine uses the time stamp as the index, and won’t build the index on any metrics stored. All data will be stored in columns. + +### STable: A Collection of Data Points in the Same Type + +The method of one table for each point will bring a greatly increasing number of tables, which is difficult to manage. Moreover, applications often need to take aggregation operations between collection points, thus aggregation operations will become complicated. To support aggregation over multiple tables efficiently, the [STable(Super Table)](https://www.taosdata.com/en/documentation/super-table) concept is introduced by TDengine. + +STable is an abstract collection for a type of data point. A STable contains a set of points (tables) that have the same schema or data structure, but with different static attributes (tags). To describe a STable (a combination of data collection points of a specific type), in addition to defining the table structure of the collected metrics, it is also necessary to define the schema of its tag. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established. + +In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**. When creating a table for a specific data collection point, the user uses the definition of STable as a template and specifies the tag value of the specific collection point (table). Compared with the traditional relational database, the table (a data collection point) has static tags, and these tags can be added, deleted, and modified afterward. **A STable contains multiple tables with the same time-series data schema but different tag values.** + +When aggregating multiple data collection points with the same data type, TDEngine will first find out the tables that meet the tag filters from the STables, and then scan the time-series data of these tables to perform aggregation operation, which can greatly reduce the data sets to be scanned, thus greatly improving the performance of aggregation calculation. + +## Cluster and Primary Logic Unit + +The design of TDengine is based on the assumption that one single hardware or software system is unreliable and that no single computer can provide sufficient computing and storage resources to process massive data. Therefore, TDengine has been designed according to a distributed and high-reliability architecture since Day One of R&D, which supports scale-out, so that hardware failure or software failure of any single or multiple servers will not affect the availability and reliability of the system. At the same time, through node virtualization and automatic load-balancing technology, TDengine can make the most efficient use of computing and storage resources in heterogeneous clusters to reduce hardware investment. + +### Primary Logic Unit + +Logical structure diagram of TDengine distributed architecture as following: + +![TDengine architecture diagram](page://images/architecture/structure.png) +
Picture 1: TDengine architecture diagram
+ + + +A complete TDengine system runs on one or more physical nodes. Logically, it includes data node (dnode), TDEngine application driver (taosc) and application (app). There are one or more data nodes in the system, which form a cluster. The application interacts with the TDengine cluster through taosc's API. The following is a brief introduction to each logical unit. + +**Physical node (pnode)**: A pnode is a computer that runs independently and has its own computing, storage and network capabilities. It can be a physical machine, virtual machine or Docker container installed with OS. The physical node is identified by its configured FQDN (Fully Qualified Domain Name). TDengine relies entirely on FQDN for network communication. If you don't know about FQDN, please read the blog post "[All about FQDN of TDengine](https://www.taosdata.com/blog/2020/09/11/1824.html)". + +**Data node (dnode):** A dnode is a running instance of the TDengine server-side execution code taosd on a physical node. A working system must have at least one data node. A dnode contains zero to multiple logical virtual nodes (VNODE), zero or at most one logical management node (mnode). The unique identification of a dnode in the system is determined by the instance's End Point (EP). EP is a combination of FQDN (Fully Qualified Domain Name) of the physical node where the dnode is located and the network port number (Port) configured by the system. By configuring different ports, a physical node (a physical machine, virtual machine or container) can run multiple instances or have multiple data nodes. + +**Virtual node (vnode)**: In order to better support data sharding, load balancing and prevent data from overheating or skewing, data nodes are virtualized into multiple virtual nodes (vnode, V2, V3, V4, etc. in the figure). Each vnode is a relatively independent work unit, which is the basic unit of time-series data storage, and has independent running threads, memory space and persistent storage path. A vnode contains a certain number of tables (data collection points). When a new table is created, the system checks whether a new vnode needs to be created. The number of vnodes that can be created on a data node depends on the hardware capacities of the physical node where the data node is located. A vnode belongs to only one DB, but a DB can have multiple vnodes. In addition to the stored time-series data, a vnode also stores the schema and tag values of the included tables. A virtual node is uniquely identified in the system by the EP of the data node and the VGroup ID to which it belongs, and is created and managed by the management node. + +**Management node (mnode)**: A virtual logical unit responsible for monitoring and maintaining the running status of all data nodes and load balancing among nodes (M in figure). At the same time, the management node is also responsible for the storage and management of metadata (including users, databases, tables, static tags, etc.), so it is also called Meta Node. Multiple (up to 5) mnodes can be configured in a TDengine cluster, and they are automatically constructed into a virtual management node group (M0, M1, M2 in the figure). The master/slave mechanism is used to manage between mnodes, and the data synchronization is carried out in a strong consistent way. Any data update operation can only be done on the master. The creation of mnode cluster is completed automatically by the system without manual intervention. There is at most one mnode on each dnode, which is uniquely identified by the EP of the data node to which it belongs. Each dnode automatically obtains the EP of the dnode where all mnodes in the whole cluster are located through internal messaging interaction. + +**Virtual node group (VGroup)**: Vnodes on different data nodes can form a virtual node group to ensure the high reliability of the system. The virtual node group is managed in a master/slave structure. Write operations can only be performed on the master vnode, and the system synchronizes data to the slave vnode via replication, thus ensuring that one single replica of data is copied on multiple physical nodes. The number of virtual nodes in a vgroup equals the number of data replicas. If the number of replicas of a DB is N, the system must have at least N data nodes. The number of replicas can be specified by the parameter replica when creating DB, and the default is 1. Using the multi-replica feature of TDengine, the same high data reliability can be done without the need for expensive storage devices such as disk arrays. Virtual node group is created and managed by management node, and the management node assigns a system unique ID, aka VGroup ID. If two virtual nodes has the same vnode group ID, means that they belong to the same group and the data is backed up to each other. The number of virtual nodes in a virtual node group can be dynamically changed, allowing only one, that is, no data replication. VGroup ID is never changed. Even if a virtual node group is deleted, its ID will not be reused. + +**TAOSC**: TAOSC is the driver provided by TDengine to applications, which is responsible for dealing with the interface interaction between application and cluster, and provides the native interface of C/C + + language, which is embedded in JDBC, C #, Python, Go, Node.js language connection libraries. Applications interact with the whole cluster through taosc instead of directly connecting to data nodes in the cluster. This module is responsible for obtaining and caching metadata; forwarding requests for insertion, query, etc. to the correct data node; when returning the results to the application, taosc also need to be responsible for the final level of aggregation, sorting, filtering and other operations. For JDBC, C/C + +/C #/Python/Go/Node.js interfaces, this module runs on the physical node where the application is located. At the same time, in order to support the fully distributed RESTful interface, taosc has a running instance on each dnode of TDengine cluster. + +### Node Communication + +**Communication mode**: The communication among each data node of TDengine system, and among application driver and each data node is carried out through TCP/UDP. Considering an IoT scenario, the data writing packets are generally not large, so TDengine uses UDP in addition to TCP for transmission, because UDP is more efficient and is not limited by the number of connections. TDengine implements its own timeout, retransmission, confirmation and other mechanisms to ensure reliable transmission of UDP. For packets with a data volume of less than 15K, UDP is adopted for transmission, and TCP is automatically adopted for transmission of packets with a data volume of more than 15K or query operations. At the same time, TDengine will automatically compress/decompress the data, digital sign/authenticate the data according to the configuration and data packet. For data replication among data nodes, only TCP is used for data transmission. + +**FQDN configuration:** A data node has one or more FQDNs, which can be specified in the system configuration file taos.cfg with the parameter "fqdn". If it is not specified, the system will automatically use the hostname of the computer as its FQDN. If the node is not configured with FQDN, you can directly set the configuration parameter fqdn of the node to its IP address. However, IP is not recommended because IP address is variable, and once it changes, the cluster will not work properly. The EP (End Point) of a data node consists of FQDN + Port. With FQDN, it is necessary to ensure the normal operation of DNS service, or configure hosts files on nodes and the nodes where applications are located. + +**Port configuration**: The external port of a data node is determined by the system configuration parameter serverPort in TDengine, and the port for internal communication of cluster is serverPort+5. The data replication operation among data nodes in the cluster also occupies a TCP port, which is serverPort+10. In order to support multithreading and efficient processing of UDP data, each internal and external UDP connection needs to occupy 5 consecutive ports. Therefore, the total port range of a data node will be serverPort to serverPort + 10, for a total of 11 TCP/UDP ports. When using, make sure that the firewall keeps these ports open. Each data node can be configured with a different serverPort. + +**Cluster external connection**: TDengine cluster can accommodate one single, multiple or even thousands of data nodes. The application only needs to initiate a connection to any data node in the cluster. The network parameter required for connection is the End Point (FQDN plus configured port number) of a data node. When starting the application taos through CLI, the FQDN of the data node can be specified through the option-h, and the configured port number can be specified through -p. If the port is not configured, the system configuration parameter serverPort of TDengine will be adopted. + +**Inter-cluster communication**: Data nodes connect with each other through TCP/UDP. When a data node starts, it will obtain the EP information of the dnode where the mnode is located, and then establish a connection with the mnode in the system to exchange information. There are three steps to obtain EP information of the mnode: 1. Check whether the mnodeEpList file exists, if it does not exist or cannot be opened normally to obtain EP information of the mnode, skip to the second step; 2: Check the system configuration file taos.cfg to obtain node configuration parameters firstEp and secondEp (the node specified by these two parameters can be a normal node without mnode, in this case, the node will try to redirect to the mnode node when connected). If these two configuration parameters do not exist or do not exist in taos.cfg, or are invalid, skip to the third step; 3: Set your own EP as a mnode EP and run it independently. After obtaining the mnode EP list, the data node initiates the connection. It will successfully join the working cluster after connected. If not successful, it will try the next item in the mnode EP list. If all attempts are made, but the connection still fails, sleep for a few seconds before trying again. + +**The choice of MNODE**: TDengine logically has a management node, but there is no separated execution code. The server side only has a set of execution code taosd. So which data node will be the management node? This is determined automatically by the system without any manual intervention. The principle is as follows: when a data node starts, it will check its End Point and compare it with the obtained mnode EP List. If its EP exists in it, the data node shall start the mnode module and become a mnode. If your own EP is not in the mnode EP List, the mnode module will not start. During the system operation, due to load balancing, downtime and other reasons, mnode may migrate to the new dnode, while totally transparent without manual intervention. The modification of configuration parameters is the decision made by mnode itself according to resources usage. + +**Add new data nodes:** After the system has a data node, it has become a working system. There are two steps to add a new node into the cluster. Step1: Connect to the existing working data node using TDengine CLI, and then add the End Point of the new data node with the command "create dnode"; Step 2: In the system configuration parameter file taos.cfg of the new data node, set the firstEp and secondEp parameters to the EP of any two data nodes in the existing cluster. Please refer to the detailed user tutorial for detailed steps. In this way, the cluster will be established step by step. + +**Redirection**: No matter about dnode or taosc, the connection to the mnode shall be initiated first, but the mnode is automatically created and maintained by the system, so user does not know which dnode is running the mnode. TDengine only requires a connection to any working dnode in the system. Because any running dnode maintains the currently running mnode EP List, when receiving a connecting request from the newly started dnode or taosc, if it’s not an mnode by self, it will reply the mnode EP List back. After receiving this list, taosc or the newly started dnode will try to establish the connection again. When the mnode EP List changes, each data node quickly obtains the latest list and notifies taosc through messaging interaction among nodes. + +### A Typical Messaging Process + +To explain the relationship between vnode, mnode, taosc and application and their respective roles, the following is an analysis of a typical data writing process. + +![ typical process of TDengine](page://images/architecture/message.png) +
Picture 2 typical process of TDengine
+ +1. Application initiates a request to insert data through JDBC, ODBC, or other APIs. +2. Cache be checked by taosc that if meta data existing for the table. If so, go straight to Step 4. If not, taosc sends a get meta-data request to mnode. +3. Mnode returns the meta-data of the table to taosc. Meta-data contains the schema of the table, and also the vgroup information to which the table belongs (the vnode ID and the End Point of the dnode where the table belongs. If the number of replicas is N, there will be N groups of End Points). If taosc does not receive a response from the mnode for a long time, and there are multiple mnodes, taosc will send a request to the next mnode. +4. Taosc initiates an insert request to master vnode. +5. After vnode inserts the data, it gives a reply to taosc, indicating that the insertion is successful. If taosc doesn't get a response from vnode for a long time, taosc will judge the node as offline. In this case, if there are multiple replicas of the inserted database, taosc will issue an insert request to the next vnode in vgroup. +6. Taosc notifies APP that writing is successful. + +For Step 2 and 3, when taosc starts, it does not know the End Point of mnode, so it will directly initiate a request to the externally serving End Point of the configured cluster. If the dnode that received the request does not have an mnode configured, it will inform the mnode EP list in a reply message, so that taosc will re-issue a request to obtain meta-data to the EP of another new mnode. + +For Step 4 and 5, without caching, taosc can't recognize the master in the virtual node group, so assumes that the first vnodeID is the master and send a request to it. If the requested vnode is not the master, it will reply the actual master as a new target taosc makes a request to. Once the reply of successful insertion is obtained, taosc will cache the information of master node. + +The above is the process of inserting data, and the processes of querying and calculating are completely consistent. Taosc encapsulates and shields all these complicated processes, and has no perception and no special treatment for applications. + +Through taosc caching mechanism, mnode needs to be accessed only when a table is operated for the first time, so mnode will not become a system bottleneck. However, because schema and vgroup may change (such as load balancing), taosc will interact with mnode regularly to automatically update the cache. + +## Storage Model and Data Partitioning/Sharding + +### Storage Model + +The data stored by TDengine include collected time-series data, metadata related to libraries and tables, tag data, etc. These data are specifically divided into three parts: + +- Time-series data: stored in vnode and composed of data, head and last files. The amount of data is large and query amount depends on the application scenario. Out-of-order writing is allowed, but delete operation is not supported for the time being, and update operation is only allowed when update parameter is set to 1. By adopting the model with one table for each collection point, the data of a given time period is continuously stored, and the writing against one single table is a simple add operation. Multiple records can be read at one time, thus ensuring the insert and query operation of a single collection point with best performance. +- Tag data: meta files stored in vnode support four standard operations of add, delete, modify and check. The amount of data is not large. If there are N tables, there are N records, so all can be stored in memory. If there are many tag filtering operations, queries will be very frequent and TDengine supports multi-core and multi-threaded concurrent queries. As long as the computing resources are sufficient, even in face of millions of tables, the filtering results will return in milliseconds. +- Metadata: stored in mnode, including system node, user, DB, Table Schema and other information. Four standard operations of add, delete, modify and query are supported. The amount of these data are not large and can be stored in memory, moreover the query amount is not large because of the client cache. Therefore, TDengine uses centralized storage management, however, there will be no performance bottleneck. + +Compared with the typical NoSQL storage model, TDengine stores tag data and time-series data completely separately, which has two major advantages: + +- Greatly reduce the redundancy of tag data storage: general NoSQL database or time-series database adopts K-V storage, in which Key includes timestamp, device ID and various tags. Each record carries these duplicates, so wasting storage space. Moreover, if the application needs to add, modify or delete tags on historical data, it has to traverse the data and rewrite again, which is extremely expensive to operate. +- Realize extremely efficient aggregation query between multiple tables: when doing aggregation query between multiple tables, it firstly finds out the tag filtered tables, and then find out the corresponding data blocks of these tables to greatly reduce the data sets to be scanned, thus greatly improving the query efficiency. Moreover, tag data is managed and maintained in a full-memory structure, and tag data queries in tens of millions can return in milliseconds. + +### Data Sharding + +For large-scale data management, to achieve scale-out, it is generally necessary to adopt the a Partitioning strategy as Sharding. TDengine implements data sharding via vnode, and time-series data partitioning via one data file for each time range. + +VNode (Virtual Data Node) is responsible for providing writing, query and calculation functions for collected time-series data. To facilitate load balancing, data recovery and support heterogeneous environments, TDengine splits a data node into multiple vnodes according to its computing and storage resources. The management of these vnodes is done automatically by TDengine and completely transparent to the application. + +For a single data collection point, regardless of the amount of data, a vnode (or vnode group, if the number of replicas is greater than 1) has enough computing resource and storage resource to process (if a 16-byte record is generated per second, the original data generated in one year will be less than 0.5 G), so TDengine stores all the data of a table (a data collection point) in one vnode instead of distributing the data to two or more dnodes. Moreover, a vnode can store data from multiple data collection points (tables), and the upper limit of the tables’ quantity for a vnode is one million. By design, all tables in a vnode belong to the same DB. On a data node, unless specially configured, the number of vnodes owned by a DB will not exceed the number of system cores. + +When creating a DB, the system does not allocate resources immediately. However, when creating a table, the system will check if there is an allocated vnode with free tablespace. If so, the table will be created in the vacant vnode immediately. If not, the system will create a new vnode on a dnode from the cluster according to the current workload, and then a table. If there are multiple replicas of a DB, the system does not create only one vnode, but a vgroup (virtual data node group). The system has no limit on the number of vnodes, which is just limited by the computing and storage resources of physical nodes. + +The meda data of each table (including schema, tags, etc.) is also stored in vnode instead of centralized storage in mnode. In fact, this means sharding of meta data, which is convenient for efficient and parallel tag filtering operations. + +### Data Partitioning + +In addition to vnode sharding, TDengine partitions the time-series data by time range. Each data file contains only one time range of time-series data, and the length of the time range is determined by DB's configuration parameter “days”. This method of partitioning by time rang is also convenient to efficiently implement the data retention strategy. As long as the data file exceeds the specified number of days (system configuration parameter ‘keep’), it will be automatically deleted. Moreover, different time ranges can be stored in different paths and storage media, so as to facilitate the cold/hot management of big data and realize tiered-storage. + +In general, **TDengine splits big data by vnode and time as two dimensions**, which is convenient for parallel and efficient management with scale-out. + +### Load Balancing + +Each dnode regularly reports its status (including hard disk space, memory size, CPU, network, number of virtual nodes, etc.) to the mnode (virtual management node) for declaring the status of the entire cluster. Based on the overall state, when an mnode finds an overloaded dnode, it will migrate one or more vnodes to other dnodes. In the process, external services keep running and the data insertion, query and calculation operations are not affected. + +If the mnode has not received the dnode status for a period of time, the dnode will be judged as offline. When offline lasts a certain period of time (the duration is determined by the configuration parameter ‘offlineThreshold’), the dnode will be forcibly removed from the cluster by mnode. If the number of replicas of vnodes on this dnode is greater than one, the system will automatically create new replicas on other dnodes to ensure the replica number. If there are other mnodes on this dnode and the number of mnodes replicas is greater than one, the system will automatically create new mnodes on other dnodes to ensure t the replica number. + +When new data nodes are added to the cluster, with new computing and storage are added, the system will automatically start the load balancing process. + +The load balancing process does not require any manual intervention without application restarted. It will automatically connect new nodes with completely transparence. **Note: load balancing is controlled by parameter “balance”, which determines to turn on/off automatic load balancing.** + +## Data Writing and Replication Process + +If a database has N replicas, thus a virtual node group has N virtual nodes, but only one as Master and all others are slaves. When the application writes a new record to system, only the Master vnode can accept the writing request. If a slave vnode receives a writing request, the system will notifies taosc to redirect. + +### Master vnode Writing Process + +Master Vnode uses a writing process as follows: + +Figure 3: TDengine Master writing process + +1. Master vnode receives the application data insertion request, verifies, and to next step; +2. If the system configuration parameter “walLevel” is greater than 0, vnode will write the original request packet into database log file WAL. If walLevel is set to 2 and fsync is set to 0, TDengine will make WAL data written immediately to ensure that even system goes down, all data can be recovered from database log file; +3. If there are multiple replicas, vnode will forward data packet to slave vnodes in the same virtual node group, and the forwarded packet has a version number with data; +4. Write into memory and add the record to “skip list”; +5. Master vnode returns a confirmation message to the application, indicating a successful writing. +6. If any of Step 2, 3 or 4 fails, the error will directly return to the application. + +### Slave vnode Writing Process + +For a slave vnode, the write process as follows: + +![TDengine Slave Writing Process](page://images/architecture/write_master.png) +
Picture 3 TDengine Slave Writing Process
+ +1. Slave vnode receives a data insertion request forwarded by Master vnode. +2. If the system configuration parameter “walLevel” is greater than 0, vnode will write the original request packet into database log file WAL. If walLevel is set to 2 and fsync is set to 0, TDengine will make WAL data written immediately to ensure that even system goes down, all data can be recovered from database log file; +3. Write into memory and add the record to “skip list”; + +Compared with Master vnode, slave vnode has no forwarding or reply confirmation step, means two steps less. But writing into memory is exactly the same as WAL. + +### Remote Disaster Recovery and IDC Migration + +As above Master and Slave processes discussed, TDengine adopts asynchronous replication for data synchronization. This method can greatly improve the writing performance, with not obvious impact from network delay. By configuring IDC and rack number for each physical node, it can be ensured that for a virtual node group, virtual nodes are composed of physical nodes from different IDC and different racks, thus implementing remote disaster recovery without other tools. + +On the other hand, TDengine supports dynamic modification of the replicas number. Once the number of replicas increases, the newly added virtual nodes will immediately enter the data synchronization process. After synchronization completed, added virtual nodes can provide services. In the synchronization process, master and other synchronized virtual nodes keep serving. With this feature, TDengine can realize IDC room migration without service interruption. It is only necessary to add new physical nodes to the existing IDC cluster, and then remove old physical nodes after the data synchronization is completed. + +However, this asynchronous replication method has a tiny time window of written data lost. The specific scenario is as follows: + +1. Master vnode has completed its 5-step operations, confirmed the success of writing to APP, and then went down; +2. Slave vnode receives the write request, then processing fails before writing to the log in Step 2; +3. Slave vnode will become the new master, thus losing one record + +In theory, as long as in asynchronous replication, there is no guarantee for no losing. However, this window is extremely small, only if mater and slave fail at the same time, and just confirm the successful write to the application before. + +Note: Remote disaster recovery and no-downtime IDC migration are only supported by Enterprise Edition. **Hint: This function is not available yet** + +### Master/slave Selection + +Vnode maintains a Version number. When memory data is persisted, the version number will also be persisted. For each data update operation, whether it is collecting time-series data or metadata, this version number will be increased by one. + +When a vnode starts, the roles (master, slave) are uncertain, and the data is in an unsynchronized state. It’s necessary to establish TCP connections with other nodes in the virtual node group and exchange status, including version and its own roles. Through the exchange, the system implements a master-selection process. The rules are as follows: + +1. If there’s only one replica, it’s always master +2. When all replicas are online, the one with latest version is master +3. Over half of online nodes are virtual nodes, and some virtual node is slave, it will automatically become master +4. For 2 and 3, if multiple virtual nodes meet the requirement, the first vnode in virtual node group list will be selected as master + +See [TDengine 2.0 Data Replication Module Design](https://www.taosdata.com/cn/documentation/architecture/replica/) for more information on the data replication process. + +### Synchronous Replication + +For scenarios with higher data consistency requirements, asynchronous data replication is not applicable, because there is some small probability of data loss. So, TDengine provides a synchronous replication mechanism for users. When creating a database, in addition to specifying the number of replicas, user also needs to specify a new parameter “quorum”. If quorum is greater than one, it means that every time the Master forwards a message to the replica, it needs to wait for “quorum-1” reply confirms before informing the application that data has been successfully written in slave. If “quorum-1” reply confirms are not received within a certain period of time, the master vnode will return an error to the application. + +With synchronous replication, performance of system will decrease and latency will increase. Because metadata needs strong consistent, the default for data synchronization between mnodes is synchronous replication. + +Note: synchronous replication between vnodes is only supported in Enterprise Edition + +## Caching and Persistence + +### Caching + +TDengine adopts a time-driven cache management strategy (First-In-First-Out, FIFO), also known as a Write-driven Cache Management Mechanism. This strategy is different from the read-driven data caching mode (Least-Recent-Used, LRU), which directly put the most recently written data in the system buffer. When the buffer reaches a threshold, the earliest data are written to disk in batches. Generally speaking, for the use of IoT data, users are most concerned about the newly generated data, that is, the current status. TDengine takes full advantage of this feature to put the most recently arrived (current state) data in the buffer. + +TDengine provides millisecond-level data collecting capability to users through query functions. Putting the recently arrived data directly in the buffer can respond to users' analysis query for the latest piece or batch of data more quickly, and provide faster database query response capability as a whole. In this sense, **TDengine can be used as a data buffer by setting appropriate configuration parameters without deploying Redis or other additional cache systems**, which can effectively simplify the system architecture and reduce the operation costs. It should be noted that after the TDengine is restarted, the buffer of the system will be emptied, the previously cached data will be written to disk in batches, and the previously cached data will not be reloaded into the buffer as so in a proprietary key-value cache system. + +Each vnode has its own independent memory, and it is composed of multiple memory blocks of fixed size, and different vnodes are completely isolated. When writing data, similar to the writing of logs, data is sequentially added to memory, but each vnode maintains its own skip list for quick search. When more than one third of the memory block are used, the disk writing operation will start, and the subsequent writing operation is carried out in a new memory block. By this design, one third of the memory blocks in a vnode keep the latest data, so as to achieve the purpose of caching and quick search. The number of memory blocks of a vnode is determined by the configuration parameter “blocks”, and the size of memory blocks is determined by the configuration parameter “cache”. + +### Persistent Storage + +TDengine uses a data-driven method to write the data from buffer into hard disk for persistent storage. When the cached data in vnode reaches a certain volume, TDengine will also pull up the disk-writing thread to write the cached data into persistent storage in order not to block subsequent data writing. TDengine will open a new database log file when the data is written, and delete the old database log file after written successfully to avoid unlimited log growth. + +To make full use of the characteristics of time-series data, TDengine splits the data stored in persistent storage by a vnode into multiple files, each file only saves data for a fixed number of days, which is determined by the system configuration parameter “days”. By so, for the given start and end date of a query, you can locate the data files to open immediately without any index, thus greatly speeding up reading operations. + +For collected data, there is generally a retention period, which is determined by the system configuration parameter “keep”. Data files exceeding this set number of days will be automatically deleted by the system to free up storage space. + +Given “days” and “keep” parameters, the total number of data files in a vnode is: keep/days. The total number of data files should not be too large or too small. 10 to 100 is appropriate. Based on this principle, reasonable days can be set. In the current version, parameter “keep” can be modified, but parameter “days” cannot be modified once it is set. + +In each data file, the data of a table is stored by blocks. A table can have one or more data file blocks. In a file block, data is stored in columns, occupying a continuous storage space, thus greatly improving the reading speed. The size of file block is determined by the system parameter “maxRows” (the maximum number of records per block), and the default value is 4096. This value should not be too large or too small. If it is too large, the data locating in search will cost longer; if too small, the index of data block is too large, and the compression efficiency will be low with slower reading speed. + +Each data file (with a .data postfix) has a corresponding index file (with a .head postfix). The index file has summary information of a data block for each table, recording the offset of each data block in the data file, start and end time of data and other information, so as to lead system quickly locate the data to be found. Each data file also has a corresponding last file (with a .last postfix), which is designed to prevent data block fragmentation when written in disk. If the number of written records from a table does not reach the system configuration parameter “minRows” (minimum number of records per block), it will be stored in the last file first. When write to disk next time, the newly written records will be merged with the records in last file and then written into data file. + +When data is written to disk, it is decided whether to compress the data according to system configuration parameter “comp”. TDengine provides three compression options: no compression, one-stage compression and two-stage compression, corresponding to comp values of 0, 1 and 2 respectively. One-stage compression is carried out according to the type of data. Compression algorithms include delta-delta coding, simple 8B method, zig-zag coding, LZ4 and other algorithms. Two-stage compression is based on one-stage compression and compressed by general compression algorithm, which has higher compression ratio. + +### Tiered Storage + +By default, TDengine saves all data in /var/lib/taos directory, and the data files of each vnode are saved in a different directory under this directory. In order to expand the storage space, minimize the bottleneck of file reading and improve the data throughput rate, TDengine can configure the system parameter “dataDir” to allow multiple mounted hard disks to be used by system at the same time. In addition, TDengine also provides the function of tiered data storage, i.e. storage on different storage media according to the time stamps of data files. For example, the latest data is stored on SSD, the data for more than one week is stored on local hard disk, and the data for more than four weeks is stored on network storage device, thus reducing the storage cost and ensuring efficient data access. The movement of data on different storage media is automatically done by the system and completely transparent to applications. Tiered storage of data is also configured through the system parameter “dataDir”. + + + +dataDir format is as follows: +``` +dataDir data_path [tier_level] +``` + +Where data_path is the folder path of mount point and tier_level is the media storage-tier. The higher the media storage-tier, means the older the data file. Multiple hard disks can be mounted at the same storage-tier, and data files on the same storage-tier are distributed on all hard disks within the tier. TDengine supports up to 3 tiers of storage, so tier_level values are 0, 1, and 2. When configuring dataDir, there must be only one mount path without specifying tier_level, which is called special mount disk (path). The mount path defaults to level 0 storage media and contains special file links, which cannot be removed, otherwise it will have a devastating impact on the written data. + + + +Suppose a physical node with six mountable hard disks/mnt/disk1,/mnt/disk2, …,/mnt/disk6, where disk1 and disk2 need to be designated as level 0 storage media, disk3 and disk4 are level 1 storage media, and disk5 and disk6 are level 2 storage media. Disk1 is a special mount disk, you can configure it in/etc/taos/taos.cfg as follows: + +``` +dataDir /mnt/disk1/taos +dataDir /mnt/disk2/taos 0 +dataDir /mnt/disk3/taos 1 +dataDir /mnt/disk4/taos 1 +dataDir /mnt/disk5/taos 2 +dataDir /mnt/disk6/taos 2 +``` + +Mounted disks can also be a non-local network disk, as long as the system can access it. + + +Note: Tiered Storage is only supported in Enterprise Edition + +## Data Query + +TDengine provides a variety of query processing functions for tables and STables. In addition to common aggregation queries, TDengine also provides window queries and statistical aggregation functions for time-series data. The query processing of TDengine needs the collaboration of client, vnode and mnode. + +### Single Table Query + +The parsing and verification of SQL statements are completed on the client side. SQL statements are parsed and generate an Abstract Syntax Tree (AST), which is then checksummed. Then request metadata information (table metadata) for the table specified in the query from management node (mnode). + +According to the End Point information in metadata information, the query request is serialized and sent to the data node (dnode) where the table is located. After receiving the query, the dnode identifies the virtual node (vnode) pointed to and forwards the message to the query execution queue of the vnode. The query execution thread of vnode establishes the basic query execution environment, immediately returns the query request and starts executing the query at the same time. + +When client obtains query result, the worker thread in query execution queue of dnode will wait for the execution of vnode execution thread to complete before returning the query result to the requesting client. + +### Aggregation by Time Axis, Downsampling, Interpolation + +The remarkable feature that time-series data is different from ordinary data is that each record has a timestamp, so aggregating data with timestamps on the time axis is an important and unique function from common databases. From this point of view, it is similar to the window query of stream computing engine. + +The keyword “interval” is introduced into TDengine to split fixed length time windows on time axis, and the data are aggregated according to time windows, and the data within window range are aggregated as needed. For example: + +```mysql +select count(*) from d1001 interval(1h); +``` + +According to the data collected by device D1001, the number of records stored per hour is returned by a 1-hour time window. + + + +In application scenarios where query results need to be obtained continuously, if there is data missing in a given time interval, the data results in this interval will also be lost. TDengine provides a strategy to interpolate the results of timeline aggregation calculation. The results of time axis aggregation can be interpolated by using keyword Fill. For example: + +```mysql +select count(*) from d1001 interval(1h) fill(prev); +``` + +According to the data collected by device D1001, the number of records per hour is counted. If there is no data in a certain hour, statistical data of the previous hour is returned. TDengine provides forward interpolation (prev), linear interpolation (linear), NULL value populating (NULL), and specific value populating (value). + +### Multi-table Aggregation Query + +TDengine creates a separate table for each data collection point, but in practical applications, it is often necessary to aggregate data from different collection points. In order to perform aggregation operations efficiently, TDengine introduces the concept of STable. STable is used to represent a specific type of data collection point. It is a table set containing multiple tables. The schema of each table in the set is completely consistent, but each table has its own static tag. The tags can be multiple and be added, deleted and modified at any time. Applications can aggregate or statistically operate all or a subset of tables under a STABLE by specifying tag filters, thus greatly simplifying the development of applications. The process is shown in the following figure: + +![Diagram of multi-table aggregation query](page://images/architecture/multi_tables.png) +
Picture 4 Diagram of multi-table aggregation query
+ +1. Application sends a query condition to system; +2. taosc sends the STable name to Meta Node(management node); +3. Management node sends the vnode list owned by the STable back to taosc; +4. taosc sends the computing request together with tag filters to multiple data nodes corresponding to these vnodes; +5. Each vnode first finds out the set of tables within its own node that meet the tag filters from memory, then scans the stored time-series data, completes corresponding aggregation calculations, and returns result to taosc; +6. taosc finally aggregates the results returned by multiple data nodes and send them back to application. + +Since TDengine stores tag data and time-series data separately in vnode, by filtering tag data in memory, the set of tables that need to participate in aggregation operation is first found, which greatly reduces the volume of data scanned and improves aggregation calculation speed. At the same time, because the data is distributed in multiple vnodes/dnodes, the aggregation calculation operation is carried out concurrently in multiple vnodes, which further improves the aggregation speed. Aggregation functions for ordinary tables and most operations are applicable to STables. The syntax is exactly the same. Please see TAOS SQL for details. + +### Precomputation + +In order to effectively improve the performance of query processing, based-on the unchangeable feature of IoT data, statistical information of data stored in data block is recorded in the head of data block, including max value, min value, and sum. We call it a precomputing unit. If the query processing involves all the data of a whole data block, the pre-calculated results are directly used, and no need to read the data block contents at all. Since the amount of pre-calculated data is much smaller than the actual size of data block stored on disk, for query processing with disk IO as bottleneck, the use of pre-calculated results can greatly reduce the pressure of reading IO and accelerate the query process. The precomputation mechanism is similar to the index BRIN (Block Range Index) of PostgreSQL. diff --git a/documentation20/en/04.model/docs.md b/documentation20/en/04.model/docs.md index 5875c66a213619bf028dd9c41894cfdeeda35c74..5ab5e0c6a56d0b5d534386752988cb1adae3b2fa 100644 --- a/documentation20/en/04.model/docs.md +++ b/documentation20/en/04.model/docs.md @@ -1,370 +1,74 @@ -# Data Model and Architecture +# Data Modeling -## ## Data Model +TDengine adopts a relational data model, so we need to build the "database" and "table". Therefore, for a specific application scenario, it is necessary to consider the design of the database, STable and ordinary table. This section does not discuss detailed syntax rules, but only concepts. -### ### A Typical IoT Scenario +Please watch the [video tutorial](https://www.taosdata.com/blog/2020/11/11/1945.html) for data modeling. -In typical IoT, Internet of Vehicles and Operation Monitoring scenarios, there are often many different types of data collecting devices that collect one or more different physical metrics. However, for the collection devices of the same type, there are often many specific collection devices distributed in places. BigData processing system aims to collect all kinds of data, and then calculate and analyze them. For the same kind of devices, the data collected are very regular. Taking smart meters as an example, assuming that each smart meter collects three metrics of current, voltage and phase, the collected data are similar to the following table: +## Create a Database -| | | | | | | | -| ------------- | -------------- | ----------- | ----------- | --------- | ---------------- | ----------- | -| **Device ID** | **Time Stamp** | **current** | **voltage** | **phase** | **location** | **groupId** | -| d1001 | 1538548685000 | 10.3 | 219 | 0.31 | Beijing.Chaoyang | 2 | -| d1002 | 1538548684000 | 10.2 | 220 | 0.23 | Beijing.Chaoyang | 3 | -| d1003 | 1538548686500 | 11.5 | 221 | 0.35 | Beijing.Haidian | 3 | -| d1004 | 1538548685500 | 13.4 | 223 | 0.29 | Beijing.Haidian | 2 | -| d1001 | 1538548695000 | 12.6 | 218 | 0.33 | Beijing.Chaoyang | 2 | -| d1004 | 1538548696600 | 11.8 | 221 | 0.28 | Beijing.Haidian | 2 | -| d1002 | 1538548696650 | 10.3 | 218 | 0.25 | Beijing.Chaoyang | 3 | -| d1001 | 1538548696800 | 12.3 | 221 | 0.31 | Beijing.Chaoyang | 2 | +Different types of data collection points often have different data characteristics, including frequency of data collecting, length of data retention time, number of replicas, size of data blocks, whether to update data or not, and so on. To ensure TDengine working with great efficiency in various scenarios, TDengine suggests creating tables with different data characteristics in different databases, because each database can be configured with different storage strategies. When creating a database, in addition to SQL standard options, the application can also specify a variety of parameters such as retention duration, number of replicas, number of memory blocks, time accuracy, max and min number of records in a file block, whether it is compressed or not, and number of days a data file will be overwritten. For example: +```mysql +CREATE DATABASE power KEEP 365 DAYS 10 BLOCKS 4 UPDATE 1; +``` +The above statement will create a database named “power”. The data of this database will be kept for 365 days (it will be automatically deleted 365 days later), one data file created per 10 days, and the number of memory blocks is 4 for data updating. For detailed syntax and parameters, please refer to [Data Management section of TAOS SQL](https://www.taosdata.com/en/documentation/taos-sql#management). -| **Device ID** | **Time Stamp** | **Collected Metrics** | **Tags** | | | | -| ------------- | -------------- | --------------------- | ----------- | --------- | ---------------- | ----------- | -| **Device ID** | **Time Stamp** | **current** | **voltage** | **phase** | **location** | **groupId** | -| d1001 | 1538548685000 | 10.3 | 219 | 0.31 | Beijing.Chaoyang | 2 | -| d1002 | 1538548684000 | 10.2 | 220 | 0.23 | Beijing.Chaoyang | 3 | -| d1003 | 1538548686500 | 11.5 | 221 | 0.35 | Beijing.Haidian | 3 | -| d1004 | 1538548685500 | 13.4 | 223 | 0.29 | Beijing.Haidian | 2 | -| d1001 | 1538548695000 | 12.6 | 218 | 0.33 | Beijing.Chaoyang | 2 | -| d1004 | 1538548696600 | 11.8 | 221 | 0.28 | Beijing.Haidian | 2 | -| d1002 | 1538548696650 | 10.3 | 218 | 0.25 | Beijing.Chaoyang | 3 | -| d1001 | 1538548696800 | 12.3 | 221 | 0.31 | Beijing.Chaoyang | 2 | +After the database created, please use SQL command USE to switch to the new database, for example: -Table 1: Smart meter example data +```mysql +USE power; +``` -Each data record contains the device ID, timestamp, collected metrics (current, voltage, phase as above), and static tags (Location and groupId in Table 1) associated with the devices. Each device generates a data record in a pre-defined timer or triggered by an external event. It is a sequence of data points like a stream. +Replace the database operating in the current connection with “power”, otherwise, before operating on a specific table, you need to use "database name. table name" to specify the name of database to use. -### Data Characteristics +**Note:** -As the data points are a series of data points over time, the data points generated by IoT, Internet of Vehicles, and Operation Monitoring have some strong common characteristics: +- Any table or STable belongs to a database. Before creating a table, a database must be created first. +- Tables in two different databases cannot be JOIN. -1. Metrics are always structured data; -2. There are rarely delete/update operations on collected data; -3. No need for transactions of traditional databases -4. The ratio of reading is lower but write is higher than typical Internet applications; -5. data flow is uniform and can be predicted according to the number of devices and collection frequency; -6. the user pays attention to the trend of data, not a specific value at a specific time; -7. there is always a data retention policy; -8. the data query is always executed in a given time range and a subset of space; -9. in addition to storage and query operations, various statistical and real-time calculation operations are also required; -10. data volume is huge, a system may generate over 10 billion data points in a day. +## Create a STable -By utilizing the above characteristics, TDengine designs the storage and computing engine in a special and optimized way for time-series data, resulting in massive improvements in system efficiency. +An IoT system often has many types of devices, such as smart meters, transformers, buses, switches, etc. for power grids. In order to facilitate aggregation among multiple tables, using TDengine, it is necessary to create a STable for each type of data collection point. Taking the smart meter in Table 1 as an example, you can use the following SQL command to create a STable: -### Relational Database Model +```mysql +CREATE STABLE meters (ts timestamp, current float, voltage int, phase float) TAGS (location binary(64), groupdId int); +``` -Since time-series data is most likely to be structured data, TDengine adopts the traditional relational database model to process them with a shallow learning curve. You need to create a database, create tables with schema definitions, then insert data points and execute queries to explore the data. Standard SQL is used, instead of NoSQL’s key-value storage. +**Note:** The STABLE keyword in this instruction needs to be written as TABLE in versions before 2.0.15. -### One Table for One Collection Point +Just like creating an ordinary table, you need to provide the table name (‘meters’ in the example) and the table structure Schema, that is, the definition of data columns. The first column must be a timestamp (‘ts’ in the example), the other columns are the physical metrics collected (current, volume, phase in the example), and the data types can be int, float, string, etc. In addition, you need to provide the schema of the tag (location, groupId in the example), and the data types of the tag can be int, float, string and so on. Static attributes of collection points can often be used as tags, such as geographic location of collection points, device model, device group ID, administrator ID, etc. The schema of the tag can be added, deleted and modified afterwards. Please refer to the [STable Management section of TAOS SQL](https://www.taosdata.com/cn/documentation/taos-sql#super-table) for specific definitions and details. -To utilize this time-series and other data features, TDengine requires the user to create a table for each collection point to store collected time-series data. For example, if there are over 10 millions smart meters, means 10 millions tables shall be created. For the table above, 4 tables shall be created for devices D1001, D1002, D1003, and D1004 to store the data collected. This design has several advantages: +Each type of data collection point needs an established STable, so an IoT system often has multiple STables. For the power grid, we need to build a STable for smart meters, transformers, buses, switches, etc. For IoT, a device may have multiple data collection points (for example, a fan for wind-driven generator, some collection points capture parameters such as current and voltage, and some capture environmental parameters such as temperature, humidity and wind direction). In this case, multiple STables need to be established for corresponding types of devices. All collected physical metrics contained in one and the same STable must be collected at the same time (with a consistent timestamp). -1. Guarantee that all data from a collection point can be saved in a continuous memory/hard disk space block by block. If queries are applied only on one point in a time range, this design will reduce the random read latency significantly, thus increase read and query speed by orders of magnitude. -2. Since the data generation process of each collection device is completely independent, means each device has its unique data source, thus writes can be carried out in a lock-free manner to greatly improve the speed. -3. Write latency can be significantly reduced too as the data points generated by the same device will arrive in time order, the new data point will be simply appended to a block. +A STable allows up to 1024 columns. If the number of physical metrics collected at a collection point exceeds 1024, multiple STables need to be built to process them. A system can have multiple DBs, and a DB can have one or more STables. -If the data of multiple devices are written into a table in the traditional way, due to the uncontrollable network delay, the timing of the data from different devices arriving at the server cannot be guaranteed, the writing operation must be protected by locks, and the data of one device cannot be guaranteed to continuously stored together. **The method of one table for each data collection point can ensure the optimal performance of insertion and query of a single data collection point to the greatest extent.** +## Create a Table -TDengine suggests using collection point ID as the table name (like D1001 in the above table). Each point may collect one or more metrics (like the current, voltage, phase as above). Each metric has a column in the table. The data type for a column can be int, float, string and others. In addition, the first column in the table must be a timestamp. TDengine uses the time stamp as the index, and won’t build the index on any metrics stored. All data will be stored in columns. +TDengine builds a table independently for each data collection point. Similar to standard relational data, one table has a table name, Schema, but in addition, it can also carry one or more tags. When creating, you need to use the STable as a template and specify the specific value of the tag. Taking the smart meter in Table 1 as an example, the following SQL command can be used to build the table: -### STable: A Collection of Data Points in the Same Type +```mysql +CREATE TABLE d1001 USING meters TAGS ("Beijing.Chaoyang", 2); +``` -The method of one table for each point will bring a greatly increasing number of tables, which is difficult to manage. Moreover, applications often need to take aggregation operations between collection points, thus aggregation operations will become complicated. To support aggregation over multiple tables efficiently, the [STable(Super Table)](https://www.taosdata.com/en/documentation/super-table) concept is introduced by TDengine. +Where d1001 is the table name, meters is the name of the STable, followed by the specific tag value of tag Location as "Beijing.Chaoyang", and the specific tag value of tag groupId 2. Although the tag value needs to be specified when creating the table, it can be modified afterwards. Please refer to the [Table Management section of TAOS SQL](https://www.taosdata.com/en/documentation/taos-sql#table) for details. -STable is an abstract collection for a type of data point. A STable contains a set of points (tables) that have the same schema or data structure, but with different static attributes (tags). To describe a STable (a combination of data collection points of a specific type), in addition to defining the table structure of the collected metrics, it is also necessary to define the schema of its tag. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established. +**Note: ** At present, TDengine does not technically restrict the use of a STable of a database (dbA) as a template to create a sub-table of another database (dbB). This usage will be prohibited later, and it is not recommended to use this method to create a table. -In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**. When creating a table for a specific data collection point, the user uses the definition of STable as a template and specifies the tag value of the specific collection point (table). Compared with the traditional relational database, the table (a data collection point) has static tags, and these tags can be added, deleted, and modified afterward. **A STable contains multiple tables with the same time-series data schema but different tag values.** +TDengine suggests to use the globally unique ID of data collection point as a table name (such as device serial number). However, in some scenarios, there is no unique ID, and multiple IDs can be combined into a unique ID. It is not recommended to use a unique ID as tag value. -When aggregating multiple data collection points with the same data type, TDEngine will first find out the tables that meet the tag filters from the STables, and then scan the time-series data of these tables to perform aggregation operation, which can greatly reduce the data sets to be scanned, thus greatly improving the performance of aggregation calculation. +**Automatic table creating** : In some special scenarios, user is not sure whether the table of a certain data collection point exists when writing data. In this case, the non-existent table can be created by using automatic table building syntax when writing data. If the table already exists, no new table will be created. For example: -## Cluster and Primary Logic Unit +```mysql +INSERT INTO d1001 USING METERS TAGS ("Beijng.Chaoyang", 2) VALUES (now, 10.2, 219, 0.32); +``` -The design of TDengine is based on the assumption that one single hardware or software system is unreliable and that no single computer can provide sufficient computing and storage resources to process massive data. Therefore, TDengine has been designed according to a distributed and high-reliability architecture since Day One of R&D, which supports scale-out, so that hardware failure or software failure of any single or multiple servers will not affect the availability and reliability of the system. At the same time, through node virtualization and automatic load-balancing technology, TDengine can make the most efficient use of computing and storage resources in heterogeneous clusters to reduce hardware investment. +The SQL statement above inserts records (now, 10.2, 219, 0.32) into table d1001. If table d1001 has not been created yet, the STable meters is used as the template to automatically create it, and the tag value "Beijing.Chaoyang", 2 is marked at the same time. -### Primary Logic Unit +For detailed syntax of automatic table building, please refer to the "[Automatic Table Creation When Inserting Records](https://www.taosdata.com/en/documentation/taos-sql#auto_create_table)" section. -Logical structure diagram of TDengine distributed architecture as following: +## Multi-column Model vs Single-column Model -Figure 1: TDengine architecture diagram +TDengine supports multi-column model. As long as physical metrics are collected simultaneously by a data collection point (with a consistent timestamp), these metrics can be placed in a STable as different columns. However, there is also an extreme design, a single-column model, in which each collected physical metric is set up separately, so each type of physical metrics is set up separately with a STable. For example, create 3 Stables, one each for current, voltage and phase. -A complete TDengine system runs on one or more physical nodes. Logically, it includes data node (dnode), TDEngine application driver (taosc) and application (app). There are one or more data nodes in the system, which form a cluster. The application interacts with the TDengine cluster through taosc's API. The following is a brief introduction to each logical unit. - -**Physical node (pnode)**: A pnode is a computer that runs independently and has its own computing, storage and network capabilities. It can be a physical machine, virtual machine or Docker container installed with OS. The physical node is identified by its configured FQDN (Fully Qualified Domain Name). TDengine relies entirely on FQDN for network communication. If you don't know about FQDN, please read the blog post "[All about FQDN of TDengine](https://www.taosdata.com/blog/2020/09/11/1824.html)". - -**Data node (dnode):** A dnode is a running instance of the TDengine server-side execution code taosd on a physical node. A working system must have at least one data node. A dnode contains zero to multiple logical virtual nodes (VNODE), zero or at most one logical management node (mnode). The unique identification of a dnode in the system is determined by the instance's End Point (EP). EP is a combination of FQDN (Fully Qualified Domain Name) of the physical node where the dnode is located and the network port number (Port) configured by the system. By configuring different ports, a physical node (a physical machine, virtual machine or container) can run multiple instances or have multiple data nodes. - -**Virtual node (vnode)**: In order to better support data sharding, load balancing and prevent data from overheating or skewing, data nodes are virtualized into multiple virtual nodes (vnode, V2, V3, V4, etc. in the figure). Each vnode is a relatively independent work unit, which is the basic unit of time-series data storage, and has independent running threads, memory space and persistent storage path. A vnode contains a certain number of tables (data collection points). When a new table is created, the system checks whether a new vnode needs to be created. The number of vnodes that can be created on a data node depends on the hardware capacities of the physical node where the data node is located. A vnode belongs to only one DB, but a DB can have multiple vnodes. In addition to the stored time-series data, a vnode also stores the schema and tag values of the included tables. A virtual node is uniquely identified in the system by the EP of the data node and the VGroup ID to which it belongs, and is created and managed by the management node. - -**Management node (mnode)**: A virtual logical unit responsible for monitoring and maintaining the running status of all data nodes and load balancing among nodes (M in figure). At the same time, the management node is also responsible for the storage and management of metadata (including users, databases, tables, static tags, etc.), so it is also called Meta Node. Multiple (up to 5) mnodes can be configured in a TDengine cluster, and they are automatically constructed into a virtual management node group (M0, M1, M2 in the figure). The master/slave mechanism is used to manage between mnodes, and the data synchronization is carried out in a strong consistent way. Any data update operation can only be done on the master. The creation of mnode cluster is completed automatically by the system without manual intervention. There is at most one mnode on each dnode, which is uniquely identified by the EP of the data node to which it belongs. Each dnode automatically obtains the EP of the dnode where all mnodes in the whole cluster are located through internal messaging interaction. - -**Virtual node group (VGroup)**: Vnodes on different data nodes can form a virtual node group to ensure the high reliability of the system. The virtual node group is managed in a master/slave structure. Write operations can only be performed on the master vnode, and the system synchronizes data to the slave vnode via replication, thus ensuring that one single replica of data is copied on multiple physical nodes. The number of virtual nodes in a vgroup equals the number of data replicas. If the number of replicas of a DB is N, the system must have at least N data nodes. The number of replicas can be specified by the parameter replica when creating DB, and the default is 1. Using the multi-replica feature of TDengine, the same high data reliability can be done without the need for expensive storage devices such as disk arrays. Virtual node group is created and managed by management node, and the management node assigns a system unique ID, aka VGroup ID. If two virtual nodes has the same vnode group ID, means that they belong to the same group and the data is backed up to each other. The number of virtual nodes in a virtual node group can be dynamically changed, allowing only one, that is, no data replication. VGroup ID is never changed. Even if a virtual node group is deleted, its ID will not be reused. - -**TAOSC**: TAOSC is the driver provided by TDengine to applications, which is responsible for dealing with the interface interaction between application and cluster, and provides the native interface of C/C + + language, which is embedded in JDBC, C #, Python, Go, Node.js language connection libraries. Applications interact with the whole cluster through taosc instead of directly connecting to data nodes in the cluster. This module is responsible for obtaining and caching metadata; forwarding requests for insertion, query, etc. to the correct data node; when returning the results to the application, taosc also need to be responsible for the final level of aggregation, sorting, filtering and other operations. For JDBC, C/C + +/C #/Python/Go/Node.js interfaces, this module runs on the physical node where the application is located. At the same time, in order to support the fully distributed RESTful interface, taosc has a running instance on each dnode of TDengine cluster. - -### Node Communication - -**Communication mode**: The communication among each data node of TDengine system, and among application driver and each data node is carried out through TCP/UDP. Considering an IoT scenario, the data writing packets are generally not large, so TDengine uses UDP in addition to TCP for transmission, because UDP is more efficient and is not limited by the number of connections. TDengine implements its own timeout, retransmission, confirmation and other mechanisms to ensure reliable transmission of UDP. For packets with a data volume of less than 15K, UDP is adopted for transmission, and TCP is automatically adopted for transmission of packets with a data volume of more than 15K or query operations. At the same time, TDengine will automatically compress/decompress the data, digital sign/authenticate the data according to the configuration and data packet. For data replication among data nodes, only TCP is used for data transmission. - -**FQDN configuration:** A data node has one or more FQDNs, which can be specified in the system configuration file taos.cfg with the parameter "fqdn". If it is not specified, the system will automatically use the hostname of the computer as its FQDN. If the node is not configured with FQDN, you can directly set the configuration parameter fqdn of the node to its IP address. However, IP is not recommended because IP address is variable, and once it changes, the cluster will not work properly. The EP (End Point) of a data node consists of FQDN + Port. With FQDN, it is necessary to ensure the normal operation of DNS service, or configure hosts files on nodes and the nodes where applications are located. - -**Port configuration**: The external port of a data node is determined by the system configuration parameter serverPort in TDengine, and the port for internal communication of cluster is serverPort+5. The data replication operation among data nodes in the cluster also occupies a TCP port, which is serverPort+10. In order to support multithreading and efficient processing of UDP data, each internal and external UDP connection needs to occupy 5 consecutive ports. Therefore, the total port range of a data node will be serverPort to serverPort + 10, for a total of 11 TCP/UDP ports. When using, make sure that the firewall keeps these ports open. Each data node can be configured with a different serverPort. - -**Cluster external connection**: TDengine cluster can accommodate one single, multiple or even thousands of data nodes. The application only needs to initiate a connection to any data node in the cluster. The network parameter required for connection is the End Point (FQDN plus configured port number) of a data node. When starting the application taos through CLI, the FQDN of the data node can be specified through the option-h, and the configured port number can be specified through -p. If the port is not configured, the system configuration parameter serverPort of TDengine will be adopted. - -**Inter-cluster communication**: Data nodes connect with each other through TCP/UDP. When a data node starts, it will obtain the EP information of the dnode where the mnode is located, and then establish a connection with the mnode in the system to exchange information. There are three steps to obtain EP information of the mnode: 1. Check whether the mnodeEpList file exists, if it does not exist or cannot be opened normally to obtain EP information of the mnode, skip to the second step; 2: Check the system configuration file taos.cfg to obtain node configuration parameters firstEp and secondEp (the node specified by these two parameters can be a normal node without mnode, in this case, the node will try to redirect to the mnode node when connected). If these two configuration parameters do not exist or do not exist in taos.cfg, or are invalid, skip to the third step; 3: Set your own EP as a mnode EP and run it independently. After obtaining the mnode EP list, the data node initiates the connection. It will successfully join the working cluster after connected. If not successful, it will try the next item in the mnode EP list. If all attempts are made, but the connection still fails, sleep for a few seconds before trying again. - -**The choice of MNODE**: TDengine logically has a management node, but there is no separated execution code. The server side only has a set of execution code taosd. So which data node will be the management node? This is determined automatically by the system without any manual intervention. The principle is as follows: when a data node starts, it will check its End Point and compare it with the obtained mnode EP List. If its EP exists in it, the data node shall start the mnode module and become a mnode. If your own EP is not in the mnode EP List, the mnode module will not start. During the system operation, due to load balancing, downtime and other reasons, mnode may migrate to the new dnode, while totally transparent without manual intervention. The modification of configuration parameters is the decision made by mnode itself according to resources usage. - -**Add new data nodes:** After the system has a data node, it has become a working system. There are two steps to add a new node into the cluster. Step1: Connect to the existing working data node using TDengine CLI, and then add the End Point of the new data node with the command "create dnode"; Step 2: In the system configuration parameter file taos.cfg of the new data node, set the firstEp and secondEp parameters to the EP of any two data nodes in the existing cluster. Please refer to the detailed user tutorial for detailed steps. In this way, the cluster will be established step by step. - -**Redirection**: No matter about dnode or taosc, the connection to the mnode shall be initiated first, but the mnode is automatically created and maintained by the system, so user does not know which dnode is running the mnode. TDengine only requires a connection to any working dnode in the system. Because any running dnode maintains the currently running mnode EP List, when receiving a connecting request from the newly started dnode or taosc, if it’s not an mnode by self, it will reply the mnode EP List back. After receiving this list, taosc or the newly started dnode will try to establish the connection again. When the mnode EP List changes, each data node quickly obtains the latest list and notifies taosc through messaging interaction among nodes. - -### A Typical Messaging Process - -To explain the relationship between vnode, mnode, taosc and application and their respective roles, the following is an analysis of a typical data writing process. - -Figure 2: A typical process of TDengine - -1. Application initiates a request to insert data through JDBC, ODBC, or other APIs. -2. Cache be checked by taosc that if meta data existing for the table. If so, go straight to Step 4. If not, taosc sends a get meta-data request to mnode. -3. Mnode returns the meta-data of the table to taosc. Meta-data contains the schema of the table, and also the vgroup information to which the table belongs (the vnode ID and the End Point of the dnode where the table belongs. If the number of replicas is N, there will be N groups of End Points). If taosc does not receive a response from the mnode for a long time, and there are multiple mnodes, taosc will send a request to the next mnode. -4. Taosc initiates an insert request to master vnode. -5. After vnode inserts the data, it gives a reply to taosc, indicating that the insertion is successful. If taosc doesn't get a response from vnode for a long time, taosc will judge the node as offline. In this case, if there are multiple replicas of the inserted database, taosc will issue an insert request to the next vnode in vgroup. -6. Taosc notifies APP that writing is successful. - -For Step 2 and 3, when taosc starts, it does not know the End Point of mnode, so it will directly initiate a request to the externally serving End Point of the configured cluster. If the dnode that received the request does not have an mnode configured, it will inform the mnode EP list in a reply message, so that taosc will re-issue a request to obtain meta-data to the EP of another new mnode. - -For Step 4 and 5, without caching, taosc can't recognize the master in the virtual node group, so assumes that the first vnodeID is the master and send a request to it. If the requested vnode is not the master, it will reply the actual master as a new target taosc makes a request to. Once the reply of successful insertion is obtained, taosc will cache the information of master node. - -The above is the process of inserting data, and the processes of querying and calculating are completely consistent. Taosc encapsulates and shields all these complicated processes, and has no perception and no special treatment for applications. - -Through taosc caching mechanism, mnode needs to be accessed only when a table is operated for the first time, so mnode will not become a system bottleneck. However, because schema and vgroup may change (such as load balancing), taosc will interact with mnode regularly to automatically update the cache. - -## Storage Model and Data Partitioning/Sharding - -### Storage Model - -The data stored by TDengine include collected time-series data, metadata related to libraries and tables, tag data, etc. These data are specifically divided into three parts: - -- Time-series data: stored in vnode and composed of data, head and last files. The amount of data is large and query amount depends on the application scenario. Out-of-order writing is allowed, but delete operation is not supported for the time being, and update operation is only allowed when update parameter is set to 1. By adopting the model with one table for each collection point, the data of a given time period is continuously stored, and the writing against one single table is a simple add operation. Multiple records can be read at one time, thus ensuring the insert and query operation of a single collection point with best performance. -- Tag data: meta files stored in vnode support four standard operations of add, delete, modify and check. The amount of data is not large. If there are N tables, there are N records, so all can be stored in memory. If there are many tag filtering operations, queries will be very frequent and TDengine supports multi-core and multi-threaded concurrent queries. As long as the computing resources are sufficient, even in face of millions of tables, the filtering results will return in milliseconds. -- Metadata: stored in mnode, including system node, user, DB, Table Schema and other information. Four standard operations of add, delete, modify and query are supported. The amount of these data are not large and can be stored in memory, moreover the query amount is not large because of the client cache. Therefore, TDengine uses centralized storage management, however, there will be no performance bottleneck. - -Compared with the typical NoSQL storage model, TDengine stores tag data and time-series data completely separately, which has two major advantages: - -- Greatly reduce the redundancy of tag data storage: general NoSQL database or time-series database adopts K-V storage, in which Key includes timestamp, device ID and various tags. Each record carries these duplicates, so wasting storage space. Moreover, if the application needs to add, modify or delete tags on historical data, it has to traverse the data and rewrite again, which is extremely expensive to operate. -- Realize extremely efficient aggregation query between multiple tables: when doing aggregation query between multiple tables, it firstly finds out the tag filtered tables, and then find out the corresponding data blocks of these tables to greatly reduce the data sets to be scanned, thus greatly improving the query efficiency. Moreover, tag data is managed and maintained in a full-memory structure, and tag data queries in tens of millions can return in milliseconds. - -### Data Sharding - -For large-scale data management, to achieve scale-out, it is generally necessary to adopt the a Partitioning strategy as Sharding. TDengine implements data sharding via vnode, and time-series data partitioning via one data file for each time range. - -VNode (Virtual Data Node) is responsible for providing writing, query and calculation functions for collected time-series data. To facilitate load balancing, data recovery and support heterogeneous environments, TDengine splits a data node into multiple vnodes according to its computing and storage resources. The management of these vnodes is done automatically by TDengine and completely transparent to the application. - -For a single data collection point, regardless of the amount of data, a vnode (or vnode group, if the number of replicas is greater than 1) has enough computing resource and storage resource to process (if a 16-byte record is generated per second, the original data generated in one year will be less than 0.5 G), so TDengine stores all the data of a table (a data collection point) in one vnode instead of distributing the data to two or more dnodes. Moreover, a vnode can store data from multiple data collection points (tables), and the upper limit of the tables’ quantity for a vnode is one million. By design, all tables in a vnode belong to the same DB. On a data node, unless specially configured, the number of vnodes owned by a DB will not exceed the number of system cores. - -When creating a DB, the system does not allocate resources immediately. However, when creating a table, the system will check if there is an allocated vnode with free tablespace. If so, the table will be created in the vacant vnode immediately. If not, the system will create a new vnode on a dnode from the cluster according to the current workload, and then a table. If there are multiple replicas of a DB, the system does not create only one vnode, but a vgroup (virtual data node group). The system has no limit on the number of vnodes, which is just limited by the computing and storage resources of physical nodes. - -The meda data of each table (including schema, tags, etc.) is also stored in vnode instead of centralized storage in mnode. In fact, this means sharding of meta data, which is convenient for efficient and parallel tag filtering operations. - -### Data Partitioning - -In addition to vnode sharding, TDengine partitions the time-series data by time range. Each data file contains only one time range of time-series data, and the length of the time range is determined by DB's configuration parameter “days”. This method of partitioning by time rang is also convenient to efficiently implement the data retention strategy. As long as the data file exceeds the specified number of days (system configuration parameter ‘keep’), it will be automatically deleted. Moreover, different time ranges can be stored in different paths and storage media, so as to facilitate the cold/hot management of big data and realize tiered-storage. - -In general, **TDengine splits big data by vnode and time as two dimensions**, which is convenient for parallel and efficient management with scale-out. - -### Load Balancing - -Each dnode regularly reports its status (including hard disk space, memory size, CPU, network, number of virtual nodes, etc.) to the mnode (virtual management node) for declaring the status of the entire cluster. Based on the overall state, when an mnode finds an overloaded dnode, it will migrate one or more vnodes to other dnodes. In the process, external services keep running and the data insertion, query and calculation operations are not affected. - -If the mnode has not received the dnode status for a period of time, the dnode will be judged as offline. When offline lasts a certain period of time (the duration is determined by the configuration parameter ‘offlineThreshold’), the dnode will be forcibly removed from the cluster by mnode. If the number of replicas of vnodes on this dnode is greater than one, the system will automatically create new replicas on other dnodes to ensure the replica number. If there are other mnodes on this dnode and the number of mnodes replicas is greater than one, the system will automatically create new mnodes on other dnodes to ensure t the replica number. - -When new data nodes are added to the cluster, with new computing and storage are added, the system will automatically start the load balancing process. - -The load balancing process does not require any manual intervention without application restarted. It will automatically connect new nodes with completely transparence. **Note: load balancing is controlled by parameter “balance”, which determines to turn on/off automatic load balancing.** - -## Data Writing and Replication Process - -If a database has N replicas, thus a virtual node group has N virtual nodes, but only one as Master and all others are slaves. When the application writes a new record to system, only the Master vnode can accept the writing request. If a slave vnode receives a writing request, the system will notifies taosc to redirect. - -### Master vnode Writing Process - -Master Vnode uses a writing process as follows: - -Figure 3: TDengine Master writing process - -1. Master vnode receives the application data insertion request, verifies, and to next step; -2. If the system configuration parameter “walLevel” is greater than 0, vnode will write the original request packet into database log file WAL. If walLevel is set to 2 and fsync is set to 0, TDengine will make WAL data written immediately to ensure that even system goes down, all data can be recovered from database log file; -3. If there are multiple replicas, vnode will forward data packet to slave vnodes in the same virtual node group, and the forwarded packet has a version number with data; -4. Write into memory and add the record to “skip list”; -5. Master vnode returns a confirmation message to the application, indicating a successful writing. -6. If any of Step 2, 3 or 4 fails, the error will directly return to the application. - -### Slave vnode Writing Process - -For a slave vnode, the write process as follows: - -Fiture 4: TDengine Slave Writing Process - -1. Slave vnode receives a data insertion request forwarded by Master vnode. -2. If the system configuration parameter “walLevel” is greater than 0, vnode will write the original request packet into database log file WAL. If walLevel is set to 2 and fsync is set to 0, TDengine will make WAL data written immediately to ensure that even system goes down, all data can be recovered from database log file; -3. Write into memory and add the record to “skip list”; - -Compared with Master vnode, slave vnode has no forwarding or reply confirmation step, means two steps less. But writing into memory is exactly the same as WAL. - -### Remote Disaster Recovery and IDC Migration - -As above Master and Slave processes discussed, TDengine adopts asynchronous replication for data synchronization. This method can greatly improve the writing performance, with not obvious impact from network delay. By configuring IDC and rack number for each physical node, it can be ensured that for a virtual node group, virtual nodes are composed of physical nodes from different IDC and different racks, thus implementing remote disaster recovery without other tools. - -On the other hand, TDengine supports dynamic modification of the replicas number. Once the number of replicas increases, the newly added virtual nodes will immediately enter the data synchronization process. After synchronization completed, added virtual nodes can provide services. In the synchronization process, master and other synchronized virtual nodes keep serving. With this feature, TDengine can realize IDC room migration without service interruption. It is only necessary to add new physical nodes to the existing IDC cluster, and then remove old physical nodes after the data synchronization is completed. - -However, this asynchronous replication method has a tiny time window of written data lost. The specific scenario is as follows: - -1. Master vnode has completed its 5-step operations, confirmed the success of writing to APP, and then went down; -2. Slave vnode receives the write request, then processing fails before writing to the log in Step 2; -3. Slave vnode will become the new master, thus losing one record - -In theory, as long as in asynchronous replication, there is no guarantee for no losing. However, this window is extremely small, only if mater and slave fail at the same time, and just confirm the successful write to the application before. - -Note: Remote disaster recovery and no-downtime IDC migration are only supported by Enterprise Edition. **Hint: This function is not available yet** - -### Master/slave Selection - -Vnode maintains a Version number. When memory data is persisted, the version number will also be persisted. For each data update operation, whether it is collecting time-series data or metadata, this version number will be increased by one. - -When a vnode starts, the roles (master, slave) are uncertain, and the data is in an unsynchronized state. It’s necessary to establish TCP connections with other nodes in the virtual node group and exchange status, including version and its own roles. Through the exchange, the system implements a master-selection process. The rules are as follows: - -1. If there’s only one replica, it’s always master -2. When all replicas are online, the one with latest version is master -3. Over half of online nodes are virtual nodes, and some virtual node is slave, it will automatically become master -4. For 2 and 3, if multiple virtual nodes meet the requirement, the first vnode in virtual node group list will be selected as master - -See [TDengine 2.0 Data Replication Module Design](https://www.taosdata.com/cn/documentation/architecture/replica/) for more information on the data replication process. - -### Synchronous Replication - -For scenarios with higher data consistency requirements, asynchronous data replication is not applicable, because there is some small probability of data loss. So, TDengine provides a synchronous replication mechanism for users. When creating a database, in addition to specifying the number of replicas, user also needs to specify a new parameter “quorum”. If quorum is greater than one, it means that every time the Master forwards a message to the replica, it needs to wait for “quorum-1” reply confirms before informing the application that data has been successfully written in slave. If “quorum-1” reply confirms are not received within a certain period of time, the master vnode will return an error to the application. - -With synchronous replication, performance of system will decrease and latency will increase. Because metadata needs strong consistent, the default for data synchronization between mnodes is synchronous replication. - -Note: synchronous replication between vnodes is only supported in Enterprise Edition - -## Caching and Persistence - -### Caching - -TDengine adopts a time-driven cache management strategy (First-In-First-Out, FIFO), also known as a Write-driven Cache Management Mechanism. This strategy is different from the read-driven data caching mode (Least-Recent-Used, LRU), which directly put the most recently written data in the system buffer. When the buffer reaches a threshold, the earliest data are written to disk in batches. Generally speaking, for the use of IoT data, users are most concerned about the newly generated data, that is, the current status. TDengine takes full advantage of this feature to put the most recently arrived (current state) data in the buffer. - -TDengine provides millisecond-level data collecting capability to users through query functions. Putting the recently arrived data directly in the buffer can respond to users' analysis query for the latest piece or batch of data more quickly, and provide faster database query response capability as a whole. In this sense, **TDengine can be used as a data buffer by setting appropriate configuration parameters without deploying Redis or other additional cache systems**, which can effectively simplify the system architecture and reduce the operation costs. It should be noted that after the TDengine is restarted, the buffer of the system will be emptied, the previously cached data will be written to disk in batches, and the previously cached data will not be reloaded into the buffer as so in a proprietary key-value cache system. - -Each vnode has its own independent memory, and it is composed of multiple memory blocks of fixed size, and different vnodes are completely isolated. When writing data, similar to the writing of logs, data is sequentially added to memory, but each vnode maintains its own skip list for quick search. When more than one third of the memory block are used, the disk writing operation will start, and the subsequent writing operation is carried out in a new memory block. By this design, one third of the memory blocks in a vnode keep the latest data, so as to achieve the purpose of caching and quick search. The number of memory blocks of a vnode is determined by the configuration parameter “blocks”, and the size of memory blocks is determined by the configuration parameter “cache”. - -### Persistent Storage - -TDengine uses a data-driven method to write the data from buffer into hard disk for persistent storage. When the cached data in vnode reaches a certain volume, TDengine will also pull up the disk-writing thread to write the cached data into persistent storage in order not to block subsequent data writing. TDengine will open a new database log file when the data is written, and delete the old database log file after written successfully to avoid unlimited log growth. - -To make full use of the characteristics of time-series data, TDengine splits the data stored in persistent storage by a vnode into multiple files, each file only saves data for a fixed number of days, which is determined by the system configuration parameter “days”. By so, for the given start and end date of a query, you can locate the data files to open immediately without any index, thus greatly speeding up reading operations. - -For collected data, there is generally a retention period, which is determined by the system configuration parameter “keep”. Data files exceeding this set number of days will be automatically deleted by the system to free up storage space. - -Given “days” and “keep” parameters, the total number of data files in a vnode is: keep/days. The total number of data files should not be too large or too small. 10 to 100 is appropriate. Based on this principle, reasonable days can be set. In the current version, parameter “keep” can be modified, but parameter “days” cannot be modified once it is set. - -In each data file, the data of a table is stored by blocks. A table can have one or more data file blocks. In a file block, data is stored in columns, occupying a continuous storage space, thus greatly improving the reading speed. The size of file block is determined by the system parameter “maxRows” (the maximum number of records per block), and the default value is 4096. This value should not be too large or too small. If it is too large, the data locating in search will cost longer; if too small, the index of data block is too large, and the compression efficiency will be low with slower reading speed. - -Each data file (with a .data postfix) has a corresponding index file (with a .head postfix). The index file has summary information of a data block for each table, recording the offset of each data block in the data file, start and end time of data and other information, so as to lead system quickly locate the data to be found. Each data file also has a corresponding last file (with a .last postfix), which is designed to prevent data block fragmentation when written in disk. If the number of written records from a table does not reach the system configuration parameter “minRows” (minimum number of records per block), it will be stored in the last file first. When write to disk next time, the newly written records will be merged with the records in last file and then written into data file. - -When data is written to disk, it is decided whether to compress the data according to system configuration parameter “comp”. TDengine provides three compression options: no compression, one-stage compression and two-stage compression, corresponding to comp values of 0, 1 and 2 respectively. One-stage compression is carried out according to the type of data. Compression algorithms include delta-delta coding, simple 8B method, zig-zag coding, LZ4 and other algorithms. Two-stage compression is based on one-stage compression and compressed by general compression algorithm, which has higher compression ratio. - -### Tiered Storage - -By default, TDengine saves all data in /var/lib/taos directory, and the data files of each vnode are saved in a different directory under this directory. In order to expand the storage space, minimize the bottleneck of file reading and improve the data throughput rate, TDengine can configure the system parameter “dataDir” to allow multiple mounted hard disks to be used by system at the same time. In addition, TDengine also provides the function of tiered data storage, i.e. storage on different storage media according to the time stamps of data files. For example, the latest data is stored on SSD, the data for more than one week is stored on local hard disk, and the data for more than four weeks is stored on network storage device, thus reducing the storage cost and ensuring efficient data access. The movement of data on different storage media is automatically done by the system and completely transparent to applications. Tiered storage of data is also configured through the system parameter “dataDir”. - - - -dataDir format is as follows: - -1. dataDir data_path [tier_level] - -Where data_path is the folder path of mount point and tier_level is the media storage-tier. The higher the media storage-tier, means the older the data file. Multiple hard disks can be mounted at the same storage-tier, and data files on the same storage-tier are distributed on all hard disks within the tier. TDengine supports up to 3 tiers of storage, so tier_level values are 0, 1, and 2. When configuring dataDir, there must be only one mount path without specifying tier_level, which is called special mount disk (path). The mount path defaults to level 0 storage media and contains special file links, which cannot be removed, otherwise it will have a devastating impact on the written data. - - - -Suppose a physical node with six mountable hard disks/mnt/disk1,/mnt/disk2, …,/mnt/disk6, where disk1 and disk2 need to be designated as level 0 storage media, disk3 and disk4 are level 1 storage media, and disk5 and disk6 are level 2 storage media. Disk1 is a special mount disk, you can configure it in/etc/taos/taos.cfg as follows: - - - -1. dataDir /mnt/disk1/taos -2. dataDir /mnt/disk2/taos 0 -3. dataDir /mnt/disk3/taos 1 -4. dataDir /mnt/disk4/taos 1 -5. dataDir /mnt/disk5/taos 2 -6. dataDir /mnt/disk6/taos 2 - - - -Mounted disks can also be a non-local network disk, as long as the system can access it. - - - -Note: Tiered Storage is only supported in Enterprise Edition - -## Data Query - -TDengine provides a variety of query processing functions for tables and STables. In addition to common aggregation queries, TDengine also provides window queries and statistical aggregation functions for time-series data. The query processing of TDengine needs the collaboration of client, vnode and mnode. - -### Single Table Query - -The parsing and verification of SQL statements are completed on the client side. SQL statements are parsed and generate an Abstract Syntax Tree (AST), which is then checksummed. Then request metadata information (table metadata) for the table specified in the query from management node (mnode). - -According to the End Point information in metadata information, the query request is serialized and sent to the data node (dnode) where the table is located. After receiving the query, the dnode identifies the virtual node (vnode) pointed to and forwards the message to the query execution queue of the vnode. The query execution thread of vnode establishes the basic query execution environment, immediately returns the query request and starts executing the query at the same time. - -When client obtains query result, the worker thread in query execution queue of dnode will wait for the execution of vnode execution thread to complete before returning the query result to the requesting client. - -### Aggregation by Time Axis, Downsampling, Interpolation - -The remarkable feature that time-series data is different from ordinary data is that each record has a timestamp, so aggregating data with timestamps on the time axis is an important and unique function from common databases. From this point of view, it is similar to the window query of stream computing engine. - -The keyword “interval” is introduced into TDengine to split fixed length time windows on time axis, and the data are aggregated according to time windows, and the data within window range are aggregated as needed. For example: - - - -1. select count(*) from d1001 interval(1h); - - - -According to the data collected by device D1001, the number of records stored per hour is returned by a 1-hour time window. - - - -In application scenarios where query results need to be obtained continuously, if there is data missing in a given time interval, the data results in this interval will also be lost. TDengine provides a strategy to interpolate the results of timeline aggregation calculation. The results of time axis aggregation can be interpolated by using keyword Fill. For example: - - - -1. select count(*) from d1001 interval(1h) fill(prev); - - - -According to the data collected by device D1001, the number of records per hour is counted. If there is no data in a certain hour, statistical data of the previous hour is returned. TDengine provides forward interpolation (prev), linear interpolation (linear), NULL value populating (NULL), and specific value populating (value). - -### Multi-table Aggregation Query - -TDengine creates a separate table for each data collection point, but in practical applications, it is often necessary to aggregate data from different collection points. In order to perform aggregation operations efficiently, TDengine introduces the concept of STable. STable is used to represent a specific type of data collection point. It is a table set containing multiple tables. The schema of each table in the set is completely consistent, but each table has its own static tag. The tags can be multiple and be added, deleted and modified at any time. Applications can aggregate or statistically operate all or a subset of tables under a STABLE by specifying tag filters, thus greatly simplifying the development of applications. The process is shown in the following figure: - - - -Figure 5: Diagram of multi-table aggregation query - -1. Application sends a query condition to system; -2. taosc sends the STable name to Meta Node(management node); -3. Management node sends the vnode list owned by the STable back to taosc; -4. taosc sends the computing request together with tag filters to multiple data nodes corresponding to these vnodes; -5. Each vnode first finds out the set of tables within its own node that meet the tag filters from memory, then scans the stored time-series data, completes corresponding aggregation calculations, and returns result to taosc; -6. taosc finally aggregates the results returned by multiple data nodes and send them back to application. - -Since TDengine stores tag data and time-series data separately in vnode, by filtering tag data in memory, the set of tables that need to participate in aggregation operation is first found, which greatly reduces the volume of data scanned and improves aggregation calculation speed. At the same time, because the data is distributed in multiple vnodes/dnodes, the aggregation calculation operation is carried out concurrently in multiple vnodes, which further improves the aggregation speed. Aggregation functions for ordinary tables and most operations are applicable to STables. The syntax is exactly the same. Please see TAOS SQL for details. - -### Precomputation - -In order to effectively improve the performance of query processing, based-on the unchangeable feature of IoT data, statistical information of data stored in data block is recorded in the head of data block, including max value, min value, and sum. We call it a precomputing unit. If the query processing involves all the data of a whole data block, the pre-calculated results are directly used, and no need to read the data block contents at all. Since the amount of pre-calculated data is much smaller than the actual size of data block stored on disk, for query processing with disk IO as bottleneck, the use of pre-calculated results can greatly reduce the pressure of reading IO and accelerate the query process. The precomputation mechanism is similar to the index BRIN (Block Range Index) of PostgreSQL. \ No newline at end of file +TDengine recommends using multi-column model as much as possible because of higher insertion and storage efficiency. However, for some scenarios, types of collected metrics often change. In this case, if multi-column model is adopted, the structure definition of STable needs to be frequently modified so make the application complicated. To avoid that, single-column model is recommended. diff --git a/documentation20/en/05.insert/docs.md b/documentation20/en/05.insert/docs.md index b8169ad4343c1179830f5323bf036254607b8490..23576099cf6375a6a556dad531a862ce90accd5a 100644 --- a/documentation20/en/05.insert/docs.md +++ b/documentation20/en/05.insert/docs.md @@ -7,31 +7,19 @@ TDengine supports multiple interfaces to write data, including SQL, Prometheus, Applications insert data by executing SQL insert statements through C/C + +, JDBC, GO, or Python Connector, and users can manually enter SQL insert statements to insert data through TAOS Shell. For example, the following insert writes a record to table d1001: ```mysql -``` - INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31); - -``` ``` TDengine supports writing multiple records at a time. For example, the following command writes two records to table d1001: ```mysql -``` - INSERT INTO d1001 VALUES (1538548684000, 10.2, 220, 0.23) (1538548696650, 10.3, 218, 0.25); - -``` ``` TDengine also supports writing data to multiple tables at a time. For example, the following command writes two records to d1001 and one record to d1002: ```mysql -``` - INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31) (1538548695000, 12.6, 218, 0.33) d1002 VALUES (1538548696800, 12.3, 221, 0.31); - -``` ``` For the SQL INSERT Grammar, please refer to [Taos SQL insert](https://www.taosdata.com/en/documentation/taos-sql#insert)。 @@ -58,13 +46,9 @@ Users need to download the source code of [Bailongma](https://github.com/taosdat Bailongma project has a folder, blm_prometheus, which holds the prometheus writing API. The compiling process is as follows: ```bash -``` - cd blm_prometheus go build - -``` ``` If everything goes well, an executable of blm_prometheus will be generated in the corresponding directory. @@ -134,8 +118,6 @@ remote_write: The format of generated data by Prometheus is as follows: ```json - - { Timestamp: 1576466279341, Value: 37.000000, @@ -297,4 +279,4 @@ MQTT is a popular data transmission protocol in the IoT. TDengine can easily acc ## Direct Writing of HiveMQ Broker -[HiveMQ](https://www.hivemq.com/) is an MQTT agent that provides Free Personal and Enterprise Edition versions. It is mainly used for enterprises, emerging machine-to-machine(M2M) communication and internal transmission to meet scalability, easy management and security features. HiveMQ provides an open source plug-in development kit. You can store data to TDengine via HiveMQ extension-TDengine. Refer to the [HiveMQ extension-TDengine documentation](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README.md) for more details. \ No newline at end of file +[HiveMQ](https://www.hivemq.com/) is an MQTT agent that provides Free Personal and Enterprise Edition versions. It is mainly used for enterprises, emerging machine-to-machine(M2M) communication and internal transmission to meet scalability, easy management and security features. HiveMQ provides an open source plug-in development kit. You can store data to TDengine via HiveMQ extension-TDengine. Refer to the [HiveMQ extension-TDengine documentation](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README.md) for more details. diff --git a/documentation20/en/08.connector/docs.md b/documentation20/en/08.connector/docs.md index d7afd5cfd81e3bed283f1a081d98e615257b9a36..9913a2958f3b9d1cb981b623533e1b494bf88823 100644 --- a/documentation20/en/08.connector/docs.md +++ b/documentation20/en/08.connector/docs.md @@ -2,6 +2,8 @@ TDengine provides many connectors for development, including C/C++, JAVA, Python, RESTful, Go, Node.JS, etc. +![image-connector](page://images/connector.png) + At present, TDengine connectors support a wide range of platforms, including hardware platforms such as X64/X86/ARM64/ARM32/MIPS/Alpha, and development environments such as Linux/Win64/Win32. The comparison matrix is as follows: | **CPU** | **X64 64bit** | **X64 64bit** | **X64 64bit** | **X86 32bit** | **ARM64** | **ARM32** | **MIPS Godson** | **Alpha Whenwei** | **X64 TimecomTech** | @@ -30,11 +32,11 @@ The server should already have the TDengine server package installed. The connec **Linux** -**1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)** +**1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)** -- X64 hardware environment: TDengine-client-2.x.x.x-Linux-x64.tar.gz -- ARM64 hardware environment: TDengine-client-2.x.x.x-Linux-aarch64.tar.gz -- ARM32 hardware environment: TDengine-client-2.x.x.x-Linux-aarch32.tar.gz +* X64 hardware environment: TDengine-client-2.x.x.x-Linux-x64.tar.gz +* ARM64 hardware environment: TDengine-client-2.x.x.x-Linux-aarch64.tar.gz +* ARM32 hardware environment: TDengine-client-2.x.x.x-Linux-aarch32.tar.gz **2. Unzip the package** @@ -68,10 +70,10 @@ Edit the taos.cfg file (default path/etc/taos/taos.cfg) and change firstEP to En **Windows x64/x86** -- **1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)** +**1. Download from TAOS Data website(https://www.taosdata.com/cn/all-downloads/)** -- X64 hardware environment: TDengine-client-2.X.X.X-Windows-x64.exe -- X86 hardware environment: TDengine-client-2.X.X.X-Windows-x86.exe +* X64 hardware environment: TDengine-client-2.X.X.X-Windows-x64.exe +* X86 hardware environment: TDengine-client-2.X.X.X-Windows-x86.exe **2. Execute installation, select default vales as prompted to complete** @@ -187,11 +189,11 @@ Get version information of the client. Create a database connection and initialize the connection context. The parameters that need to be provided by user include: -- - host: FQDN used by TDengine to manage the master node - - user: User name - - pass: Password - - db: Database name. If user does not provide it, it can be connected normally, means user can create a new database through this connection. If user provides a database name, means the user has created the database and the database is used by default - - port: Port number +* host: FQDN used by TDengine to manage the master node +* user: User name +* pass: Password +* db: Database name. If user does not provide it, it can be connected normally, means user can create a new database through this connection. If user provides a database name, means the user has created the database and the database is used by default +* port: Port number A null return value indicates a failure. The application needs to save the returned parameters for subsequent API calls. @@ -278,20 +280,18 @@ Asynchronous APIs all need applications to provide corresponding callback functi Asynchronous APIs have relatively high requirements for users, who can selectively use them according to specific application scenarios. Here are three important asynchronous APIs: - `void taos_query_a(TAOS *taos, const char *sql, void (*fp)(void *param, TAOS_RES *, int code), void *param);` + Execute SQL statement asynchronously. -Execute SQL statement asynchronously. - -- - taos: The database connection returned by calling `taos_connect` - - sql: The SQL statement needed to execute - - fp: User-defined callback function, whose third parameter `code` is used to indicate whether the operation is successful, `0` for success, and negative number for failure (call `taos_errstr` to get the reason for failure). When defining the callback function, it mainly handles the second parameter `TAOS_RES *`, which is the result set returned by the query - - param：the parameter for the callback + * taos: The database connection returned by calling `taos_connect` + * sql: The SQL statement needed to execute + * fp: User-defined callback function, whose third parameter `code` is used to indicate whether the operation is successful, `0` for success, and negative number for failure (call `taos_errstr` to get the reason for failure). When defining the callback function, it mainly handles the second parameter `TAOS_RES *`, which is the result set returned by the query + * param：the parameter for the callback - `void taos_fetch_rows_a(TAOS_RES *res, void (*fp)(void *param, TAOS_RES *, int numOfRows), void *param);` + Get the result set of asynchronous queries in batch, which can only be used with `taos_query_a`. Within: -Get the result set of asynchronous queries in batch, which can only be used with `taos_query_a`. Within: - -- - res: The result set returned when backcall `taos_query_a` - - fp: Callback function. Its parameter `param` is a user-definable parameter construct passed to the callback function; `numOfRows` is the number of rows of data obtained (not a function of the entire query result set). In the callback function, applications can get each row of the batch records by calling `taos_fetch_rows` forward iteration. After reading all the records in a block, the application needs to continue calling `taos_fetch_rows_a` in the callback function to obtain the next batch of records for processing until the number of records returned (`numOfRows`) is zero (the result is returned) or the number of records is negative (the query fails). + * res: The result set returned when backcall `taos_query_a` + * fp: Callback function. Its parameter `param` is a user-definable parameter construct passed to the callback function; `numOfRows` is the number of rows of data obtained (not a function of the entire query result set). In the callback function, applications can get each row of the batch records by calling `taos_fetch_rows` forward iteration. After reading all the records in a block, the application needs to continue calling `taos_fetch_rows_a` in the callback function to obtain the next batch of records for processing until the number of records returned (`numOfRows`) is zero (the result is returned) or the number of records is negative (the query fails). The asynchronous APIs of TDengine all use non-blocking calling mode. Applications can use multithreading to open multiple tables at the same time, and can query or insert to each open table at the same time. It should be pointed out that the **application client must ensure that the operation on the same table is completely serialized**, that is, when the insertion or query operation on the same table is not completed (when no result returned), the second insertion or query operation cannot be performed. @@ -349,12 +349,12 @@ TDengine provides time-driven real-time stream computing APIs. You can perform v This API is used to create data streams where: -- - taos: Database connection established - - sql: SQL query statement (query statement only) - - fp: user-defined callback function pointer. After each stream computing is completed, TDengine passes the query result (TAOS_ROW), query status (TAOS_RES), and user-defined parameters (PARAM) to the callback function. In the callback function, the user can use taos_num_fields to obtain the number of columns in the result set, and taos_fetch_fields to obtain the type of data in each column of the result set. - - stime: The time when stream computing starts. If it is 0, it means starting from now. If it is not zero, it means starting from the specified time (the number of milliseconds from 1970/1/1 UTC time). - - param: It is a parameter provided by the application for callback. During callback, the parameter is provided to the application - - callback: The second callback function is called when the continuous query stop automatically. + * taos: Database connection established + * sql: SQL query statement (query statement only) + * fp: user-defined callback function pointer. After each stream computing is completed, TDengine passes the query result (TAOS_ROW), query status (TAOS_RES), and user-defined parameters (PARAM) to the callback function. In the callback function, the user can use taos_num_fields to obtain the number of columns in the result set, and taos_fetch_fields to obtain the type of data in each column of the result set. + * stime: The time when stream computing starts. If it is 0, it means starting from now. If it is not zero, it means starting from the specified time (the number of milliseconds from 1970/1/1 UTC time). + * param: It is a parameter provided by the application for callback. During callback, the parameter is provided to the application + * callback: The second callback function is called when the continuous query stop automatically. The return value is NULL, indicating creation failed; the return value is not NULL, indicating creation successful. @@ -370,22 +370,22 @@ The subscription API currently supports subscribing to one or more tables and co This function is for starting the subscription service, returning the subscription object in case of success, and NULL in case of failure. Its parameters are: -- - taos: Database connection established - - Restart: If the subscription already exists, do you want to start over or continue with the previous subscription - - Topic: Subject (that is, name) of the subscription. This parameter is the unique identification of the subscription - - sql: The query statement subscribed. This statement can only be a select statement. It should only query the original data, and can only query the data in positive time sequence - - fp: The callback function when the query result is received (the function prototype will be introduced later). It is only used when calling asynchronously, and this parameter should be passed to NULL when calling synchronously - - param: The additional parameter when calling the callback function, which is passed to the callback function as it is by the system API without any processing - - interval: Polling period in milliseconds. During asynchronous call, the callback function will be called periodically according to this parameter; In order to avoid affecting system performance, it is not recommended to set this parameter too small; When calling synchronously, if the interval between two calls to taos_consume is less than this period, the API will block until the interval exceeds this period. + * taos: Database connection established + * Restart: If the subscription already exists, do you want to start over or continue with the previous subscription + * Topic: Subject (that is, name) of the subscription. This parameter is the unique identification of the subscription + * sql: The query statement subscribed. This statement can only be a select statement. It should only query the original data, and can only query the data in positive time sequence + * fp: The callback function when the query result is received (the function prototype will be introduced later). It is only used when calling asynchronously, and this parameter should be passed to NULL when calling synchronously + * param: The additional parameter when calling the callback function, which is passed to the callback function as it is by the system API without any processing + * interval: Polling period in milliseconds. During asynchronous call, the callback function will be called periodically according to this parameter; In order to avoid affecting system performance, it is not recommended to set this parameter too small; When calling synchronously, if the interval between two calls to taos_consume is less than this period, the API will block until the interval exceeds this period. - `typedef void (*TAOS_SUBSCRIBE_CALLBACK)(TAOS_SUB* tsub, TAOS_RES *res, void* param, int code)` In asynchronous mode, the prototype of the callback function has the following parameters: -- - tsub: Subscription object - - res: Query the result set. Note that there may be no records in the result set - - param: Additional parameters supplied by the client when `taos_subscribe` is called - - code: Error code + * tsub: Subscription object + * res: Query the result set. Note that there may be no records in the result set + * param: Additional parameters supplied by the client when `taos_subscribe` is called + * code: Error code - `TAOS_RES *taos_consume(TAOS_SUB *tsub)` @@ -621,16 +621,16 @@ Description: Column types in column_meta: -- 1：BOOL -- 2：TINYINT -- 3：SMALLINT -- 4：INT -- 5：BIGINT -- 6：FLOAT -- 7：DOUBLE -- 8：BINARY -- 9：TIMESTAMP -- 10：NCHAR +* 1：BOOL +* 2：TINYINT +* 3：SMALLINT +* 4：INT +* 5：BIGINT +* 6：FLOAT +* 7：DOUBLE +* 8：BINARY +* 9：TIMESTAMP +* 10：NCHAR ### Custom authorization code @@ -868,7 +868,7 @@ This API is used to open DB and return an object of type \* DB. Generally, DRIVE The Node.js connector supports the following systems: -| **CPU Type** | **x64****（****64bit****）** | | | **aarch64** | **aarch32** | +| **CPU Type** | x64(64bit） | | | aarch64 | aarch32 | | -------------------- | ---------------------------- | ------- | ------- | ----------- | ----------- | | **OS Type** | Linux | Win64 | Win32 | Linux | Linux | | **Supported or Not** | **Yes** | **Yes** | **Yes** | **Yes** | **Yes** | @@ -1036,4 +1036,4 @@ promise2.then(function(result) { [node-example.js](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example.js) provides a code example that uses the NodeJS connector to create a table, insert weather data, and query the inserted data. -[node-example-raw.js](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example-raw.js) is also a code example that uses the NodeJS connector to create a table, insert weather data, and query the inserted data, but unlike the above, this example only uses cursor. \ No newline at end of file +[node-example-raw.js](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example-raw.js) is also a code example that uses the NodeJS connector to create a table, insert weather data, and query the inserted data, but unlike the above, this example only uses cursor. diff --git a/documentation20/en/10.cluster/docs.md b/documentation20/en/10.cluster/docs.md index 87c62e8e9891b06a071500009f0acf3075f6c3b4..d7d908ff424270d9aa33f89eefd36e73f6ab68b2 100644 --- a/documentation20/en/10.cluster/docs.md +++ b/documentation20/en/10.cluster/docs.md @@ -1,4 +1,4 @@ -# Installation and Management of TDengine Cluster +# TDengine Cluster Management Multiple TDengine servers, that is, multiple running instances of taosd, can form a cluster to ensure the highly reliable operation of TDengine and provide scale-out features. To understand cluster management in TDengine 2.0, it is necessary to understand the basic concepts of clustering. Please refer to the chapter "Overall Architecture of TDengine 2.0". And before installing the cluster, please follow the chapter ["Getting started"](https://www.taosdata.com/en/documentation/getting-started/) to install and experience the single node function. diff --git a/documentation20/en/11.administrator/docs.md b/documentation20/en/11.administrator/docs.md index dbcc2a7e1834cb4a5c435f8e782482804523e353..509b7fa62fc62c20ec271361124792c6ba0d461f 100644 --- a/documentation20/en/11.administrator/docs.md +++ b/documentation20/en/11.administrator/docs.md @@ -174,93 +174,78 @@ Client configuration parameters: - firstEp: end point of the first taosd instance in the actively connected cluster when taos is started, the default value is localhost: 6030. - secondEp: when taos starts, if not impossible to connect to firstEp, it will try to connect to secondEp. - locale - -Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API - -TDengine provides a special field type nchar for storing non-ASCII encoded wide characters such as Chinese, Japanese and Korean. The data written to the nchar field will be uniformly encoded in UCS4-LE format and sent to the server. It should be noted that the correctness of coding is guaranteed by the client. Therefore, if users want to normally use nchar fields to store non-ASCII characters such as Chinese, Japanese, Korean, etc., it’s needed to set the encoding format of the client correctly. - -The characters inputted by the client are all in the current default coding format of the operating system, mostly UTF-8 on Linux systems, and some Chinese system codes may be GB18030 or GBK, etc. The default encoding in the docker environment is POSIX. In the Chinese versions of Windows system, the code is CP936. The client needs to ensure that the character set it uses is correctly set, that is, the current encoded character set of the operating system running by the client, in order to ensure that the data in nchar is correctly converted into UCS4-LE encoding format. - -The naming rules of locale in Linux are: < language > _ < region >. < character set coding >, such as: zh_CN.UTF-8, zh stands for Chinese, CN stands for mainland region, and UTF-8 stands for character set. Character set encoding provides a description of encoding transformations for clients to correctly parse local strings. Linux system and Mac OSX system can determine the character encoding of the system by setting locale. Because the locale used by Windows is not the POSIX standard locale format, another configuration parameter charset is needed to specify the character encoding under Windows. You can also use charset to specify character encoding in Linux systems. + Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API + + TDengine provides a special field type nchar for storing non-ASCII encoded wide characters such as Chinese, Japanese and Korean. The data written to the nchar field will be uniformly encoded in UCS4-LE format and sent to the server. It should be noted that the correctness of coding is guaranteed by the client. Therefore, if users want to normally use nchar fields to store non-ASCII characters such as Chinese, Japanese, Korean, etc., it’s needed to set the encoding format of the client correctly. + + The characters inputted by the client are all in the current default coding format of the operating system, mostly UTF-8 on Linux systems, and some Chinese system codes may be GB18030 or GBK, etc. The default encoding in the docker environment is POSIX. In the Chinese versions of Windows system, the code is CP936. The client needs to ensure that the character set it uses is correctly set, that is, the current encoded character set of the operating system running by the client, in order to ensure that the data in nchar is correctly converted into UCS4-LE encoding format. + + The naming rules of locale in Linux are: < language > _ < region >. < character set coding >, such as: zh_CN.UTF-8, zh stands for Chinese, CN stands for mainland region, and UTF-8 stands for character set. Character set encoding provides a description of encoding transformations for clients to correctly parse local strings. Linux system and Mac OSX system can determine the character encoding of the system by setting locale. Because the locale used by Windows is not the POSIX standard locale format, another configuration parameter charset is needed to specify the character encoding under Windows. You can also use charset to specify character encoding in Linux systems. - charset -Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API - -If charset is not set in the configuration file, in Linux system, when taos starts up, it automatically reads the current locale information of the system, and parses and extracts the charset encoding format from the locale information. If the automatic reading of locale information fails, an attempt is made to read the charset configuration, and if the reading of the charset configuration also fails, the startup process is interrupted. - -In Linux system, locale information contains character encoding information, so it is unnecessary to set charset separately after setting locale of Linux system correctly. For example: - - ``` + Default value: obtained dynamically from the system. If the automatic acquisition fails, user needs to set it in the configuration file or through API + + If charset is not set in the configuration file, in Linux system, when taos starts up, it automatically reads the current locale information of the system, and parses and extracts the charset encoding format from the locale information. If the automatic reading of locale information fails, an attempt is made to read the charset configuration, and if the reading of the charset configuration also fails, the startup process is interrupted. + + In Linux system, locale information contains character encoding information, so it is unnecessary to set charset separately after setting locale of Linux system correctly. For example: + + ``` locale zh_CN.UTF-8 - ``` - -- On Windows systems, the current system encoding cannot be obtained from locale. If string encoding information cannot be read from the configuration file, taos defaults to CP936. It is equivalent to adding the following to the configuration file: - - ``` + ``` + On Windows systems, the current system encoding cannot be obtained from locale. If string encoding information cannot be read from the configuration file, taos defaults to CP936. It is equivalent to adding the following to the configuration file: + ``` charset CP936 - ``` - -- If you need to adjust the character encoding, check the encoding used by the current operating system and set it correctly in the configuration file. - -In Linux systems, if user sets both locale and charset encoding charset, and the locale and charset are inconsistent, the value set later will override the value set earlier. - -``` + ``` + If you need to adjust the character encoding, check the encoding used by the current operating system and set it correctly in the configuration file. + + In Linux systems, if user sets both locale and charset encoding charset, and the locale and charset are inconsistent, the value set later will override the value set earlier. + ``` locale zh_CN.UTF-8 charset GBK -``` - -- The valid value for charset is GBK. - -And the valid value for charset is UTF-8. - -The configuration parameters of log are exactly the same as those of server. + ``` + The valid value for charset is GBK. + + And the valid value for charset is UTF-8. + + The configuration parameters of log are exactly the same as those of server. - timezone -Default value: get the current time zone option dynamically from the system + Default value: get the current time zone option dynamically from the system -The time zone in which the client runs the system. In order to deal with the problem of data writing and query in multiple time zones, TDengine uses Unix Timestamp to record and store timestamps. The characteristics of UNIX timestamps determine that the generated timestamps are consistent at any time regardless of any time zone. It should be noted that UNIX timestamps are converted and recorded on the client side. In order to ensure that other forms of time on the client are converted into the correct Unix timestamp, the correct time zone needs to be set. + The time zone in which the client runs the system. In order to deal with the problem of data writing and query in multiple time zones, TDengine uses Unix Timestamp to record and store timestamps. The characteristics of UNIX timestamps determine that the generated timestamps are consistent at any time regardless of any time zone. It should be noted that UNIX timestamps are converted and recorded on the client side. In order to ensure that other forms of time on the client are converted into the correct Unix timestamp, the correct time zone needs to be set. -In Linux system, the client will automatically read the time zone information set by the system. Users can also set time zones in profiles in a number of ways. For example: - - ``` + In Linux system, the client will automatically read the time zone information set by the system. Users can also set time zones in profiles in a number of ways. For example: + ``` timezone UTC-8 timezone GMT-8 timezone Asia/Shanghai - ``` - -- All above are legal to set the format of the East Eight Zone. + ``` + + All above are legal to set the format of the East Eight Zone. + + The setting of time zone affects the content of non-Unix timestamp (timestamp string, parsing of keyword now) in query and writing SQL statements. For example: -The setting of time zone affects the content of non-Unix timestamp (timestamp string, parsing of keyword now) in query and writing SQL statements. For example: - - ```sql + ```sql SELECT count(*) FROM table_name WHERE TS<'2019-04-11 12:01:08'; - ``` - -- In East Eight Zone, the SQL statement is equivalent to - - ```sql + ``` + + In East Eight Zone, the SQL statement is equivalent to + ```sql SELECT count(*) FROM table_name WHERE TS<1554955268000; - ``` - -- - -In the UTC time zone, the SQL statement is equivalent to - - ```sql + ``` + + In the UTC time zone, the SQL statement is equivalent to + ```sql SELECT count(*) FROM table_name WHERE TS<1554984068000; - ``` - -- - -In order to avoid the uncertainty caused by using string time format, Unix timestamp can also be used directly. In addition, timestamp strings with time zones can also be used in SQL statements, such as: timestamp strings in RFC3339 format, 2013-04-12T15: 52: 01.123 +08:00, or ISO-8601 format timestamp strings 2013-04-12T15: 52: 01.123 +0800. The conversion of the above two strings into Unix timestamps is not affected by the time zone in which the system is located. - -When starting taos, you can also specify an end point for an instance of taosd from the command line, otherwise read from taos.cfg. + ``` + In order to avoid the uncertainty caused by using string time format, Unix timestamp can also be used directly. In addition, timestamp strings with time zones can also be used in SQL statements, such as: timestamp strings in RFC3339 format, 2013-04-12T15: 52: 01.123 +08:00, or ISO-8601 format timestamp strings 2013-04-12T15: 52: 01.123 +0800. The conversion of the above two strings into Unix timestamps is not affected by the time zone in which the system is located. + + When starting taos, you can also specify an end point for an instance of taosd from the command line, otherwise read from taos.cfg. - maxBinaryDisplayWidth - -The upper limit of the display width of binary and nchar fields in a shell, beyond which parts will be hidden. Default: 30. You can modify this option dynamically in the shell with the command set max_binary_display_width nn. + The upper limit of the display width of binary and nchar fields in a shell, beyond which parts will be hidden. Default: 30. You can modify this option dynamically in the shell with the command set max_binary_display_width nn. ## User Management @@ -508,4 +493,4 @@ At the moment, TDengine has nearly 200 internal reserved keywords, which cannot | CONCAT | GLOB | METRICS | SET | VIEW | | CONFIGS | GRANTS | MIN | SHOW | WAVG | | CONFLICT | GROUP | MINUS | SLASH | WHERE | -| CONNECTION | | | | | \ No newline at end of file +| CONNECTION | | | | | diff --git a/documentation20/en/12.taos-sql/docs.md b/documentation20/en/12.taos-sql/docs.md index a457b2324526c93eacdfa3c8e9f4c87229c8feae..f21508439253560b5b44e309872dd157fec848b7 100644 --- a/documentation20/en/12.taos-sql/docs.md +++ b/documentation20/en/12.taos-sql/docs.md @@ -63,7 +63,7 @@ In TDengine, the following 10 data types can be used in data model of an ordinar ## Database Management -- Create a Database +- **Create a Database** ```mysql CREATE DATABASE [IF NOT EXISTS] db_name [KEEP keep] [DAYS days] [UPDATE 1]; @@ -77,65 +77,64 @@ Note: 4. Maximum length of a SQL statement is 65480 characters; 5. Database has more storage-related configuration parameters, see System Management. -- Show current system parameters +- **Show current system parameters** -- ```mysql - SHOW VARIABLES; - ``` - -- Use a database - -Use/switch database + ```mysql + SHOW VARIABLES; + ``` -- Drop a database +- **Use a database** -Delete a database, all data tables included will be deleted. Please use with caution. + ```mysql + USE db_name; + ``` + Use/switch database -- Modify database parameters +- **Drop a database** + ```mysql + DROP DATABASE [IF EXISTS] db_name; + ``` + Delete a database, all data tables included will be deleted. Please use with caution. -- ```mysql - ALTER DATABASE db_name COMP 2; - ``` +- **Modify database parameters** -COMP parameter modifies the database file compression flag bit, with the default value of 2 and the value range is [0, 2]. 0 means no compression, 1 means one-stage compression, and 2 means two-stage compression. + ```mysql + ALTER DATABASE db_name COMP 2; + ``` + COMP parameter modifies the database file compression flag bit, with the default value of 2 and the value range is [0, 2]. 0 means no compression, 1 means one-stage compression, and 2 means two-stage compression. - ```mysql + ```mysql ALTER DATABASE db_name REPLICA 2; - ``` + ``` + REPLICA parameter refers to the number of replicas of the modified database, and the value range is [1, 3]. For use in a cluster, the number of replicas must be less than or equal to the number of DNODE. -- REPLICA parameter refers to the number of replicas of the modified database, and the value range is [1, 3]. For use in a cluster, the number of replicas must be less than or equal to the number of DNODE. + ```mysql + ALTER DATABASE db_name KEEP 365; + ``` + The KEEP parameter refers to the number of days to save a modified data file. The default value is 3650, and the value range is [days, 365000]. It must be greater than or equal to the days parameter value. ```mysql - ALTER DATABASE db_name KEEP 365; + ALTER DATABASE db_name QUORUM 2; ``` + QUORUM parameter refers to the number of confirmations required for successful data writing, and the value range is [1, 3]. For asynchronous replication, quorum is set to 1, and the virtual node with master role can confirm it by itself. For synchronous replication, it needs to be at least 2 or greater. In principle, Quorum > = 1 and Quorum < = replica number, which is required when starting a synchronization module instance. -- The KEEP parameter refers to the number of days to save a modified data file. The default value is 3650, and the value range is [days, 365000]. It must be greater than or equal to the days parameter value. - -```mysql - ALTER DATABASE db_name QUORUM 2; -``` - -- - -QUORUM parameter refers to the number of confirmations required for successful data writing, and the value range is [1, 3]. For asynchronous replication, quorum is set to 1, and the virtual node with master role can confirm it by itself. For synchronous replication, it needs to be at least 2 or greater. In principle, Quorum > = 1 and Quorum < = replica number, which is required when starting a synchronization module instance. - - ```mysql + ```mysql ALTER DATABASE db_name BLOCKS 100; - ``` - -- BLOCKS parameter is the number of cache-sized memory blocks in each VNODE (TSDB), so the memory size used for a VNODE equals roughly (cache * blocks). Value range is [3,1000]. + ``` + BLOCKS parameter is the number of cache-sized memory blocks in each VNODE (TSDB), so the memory size used for a VNODE equals roughly (cache * blocks). Value range is [3,1000]. - ```mysql + ```mysql ALTER DATABASE db_name CACHELAST 0; - ``` - -- CACHELAST parameter controls whether last_row of the data subtable is cached in memory. The default value is 0, and the value range is [0, 1]. Where 0 means not enabled and 1 means enabled. (supported from version 2.0. 11) - -Tips: After all the above parameters are modified, show databases can be used to confirm whether the modification is successful. - -- Show all databases in system + ``` + CACHELAST parameter controls whether last_row of the data subtable is cached in memory. The default value is 0, and the value range is [0, 1]. Where 0 means not enabled and 1 means enabled. (supported from version 2.0. 11) + + **Tips**: After all the above parameters are modified, show databases can be used to confirm whether the modification is successful. +- **Show all databases in system** + ```mysql + SHOW DATABASES; + ``` ## Table Management @@ -146,206 +145,217 @@ Note: 2. The max length of table name is 192; 3. The length of each row of the table cannot exceed 16k characters; 4. Sub-table names can only consist of letters, numbers, and underscores, and cannot begin with numbers -5. ​ 10)If the data type binary or nchar is used, the maximum number of bytes should be specified, such as binary (20), which means 20 bytes; - -- Create a table via STable +5. If the data type binary or nchar is used, the maximum number of bytes should be specified, such as binary (20), which means 20 bytes; -- ```mysql - CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name TAGS (tag_value1, ...); - ``` +- **Create a table via STable** -Use a STable as template and assign tag values to create a data table. + ```mysql + CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name TAGS (tag_value1, ...); + ``` + Use a STable as template and assign tag values to create a data table. - **Create a data table using STable as a template and specify a specific tags column** -- ```mysql - CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name (tag_name1, ...) TAGS (tag_value1, ...); - ``` - -Using the specified STable as a template, specify the values of some tags columns to create a data table. (Unspecified tags columns are set to null values.) - -Note: This method has been supported since version 2.0. 17. In previous versions, tags columns were not allowed to be specified, but the values of all tags columns must be explicitly given. - -- Create tables in batches - -- ```mysql - CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) tb_name2 USING stb_name TAGS (tag_value2, ...) ...; - ``` - -Create a large number of data tables in batches faster. (Server side 2.0. 14 and above) - -Note: - -1. The method of batch creating tables requires that the data table must use STable as a template. -2. On the premise of not exceeding the length limit of SQL statements, it is suggested that the number of tables in a single statement should be controlled between 1000 and 3000, which will obtain an ideal speed of table building. - -- Drop a table - -- Show all data table information under the current database + ```mysql + CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name (tag_name1, ...) TAGS (tag_value1, ...); + ``` + Using the specified STable as a template, specify the values of some tags columns to create a data table. (Unspecified tags columns are set to null values.) + Note: This method has been supported since version 2.0. 17. In previous versions, tags columns were not allowed to be specified, but the values of all tags columns must be explicitly given. -Show all data table information under the current database. +- **Create tables in batches** -Note: Wildcard characters can be used to match names in like. The maximum length of this wildcard character string cannot exceed 24 bytes. + ```mysql + CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) tb_name2 USING stb_name TAGS (tag_value2, ...) ...; + ``` + Create a large number of data tables in batches faster. (Server side 2.0. 14 and above) + + Note: + 1. The method of batch creating tables requires that the data table must use STable as a template. + 2. On the premise of not exceeding the length limit of SQL statements, it is suggested that the number of tables in a single statement should be controlled between 1000 and 3000, which will obtain an ideal speed of table building. -Wildcard matching: 1) '%' (percent sign) matches 0 to any number of characters; 2) '_' underscore matches one character. +- **Drop a table** + + ```mysql + DROP TABLE [IF EXISTS] tb_name; + ``` -- Modify display character width online +- **Show all data table information under the current database** -- Get schema information of a table + ```mysql + SHOW TABLES [LIKE tb_name_wildcar]; + ``` + Show all data table information under the current database. + Note: Wildcard characters can be used to match names in like. The maximum length of this wildcard character string cannot exceed 24 bytes. + Wildcard matching: 1) '%' (percent sign) matches 0 to any number of characters; 2) '_' underscore matches one character. -- Add a column to table +- **Modify display character width online** -- ```mysql - ALTER TABLE tb_name ADD COLUMN field_name data_type; - ``` + ```mysql + SET MAX_BINARY_DISPLAY_WIDTH ; + ``` -Note: +- **Get schema information of a table** + + ```mysql + DESCRIBE tb_name; + ``` -1. The maximum number of columns is 1024 and the minimum number is 2; -2. The maximum length of a column name is 64; +- **Add a column to table** -- Drop a column in table + ```mysql + ALTER TABLE tb_name ADD COLUMN field_name data_type; + ``` + Note: + 1. The maximum number of columns is 1024 and the minimum number is 2; + 2. The maximum length of a column name is 64; -- ```mysql - ALTER TABLE tb_name DROP COLUMN field_name; - ``` +- **Drop a column in table** -If the table is created through a STable, the operation of table schema changing can only be carried out on the STable. Moreover, the schema changes for the STable take effect for all tables created through the schema. For tables that are not created through STables, you can modify the table schema directly. + ```mysql + ALTER TABLE tb_name DROP COLUMN field_name; + ``` + If the table is created through a STable, the operation of table schema changing can only be carried out on the STable. Moreover, the schema changes for the STable take effect for all tables created through the schema. For tables that are not created through STables, you can modify the table schema directly. ## STable Management Note: In 2.0. 15.0 and later versions, STABLE reserved words are supported. That is, in the instruction description later in this section, the three instructions of CREATE, DROP and ALTER need to write TABLE instead of STABLE in the old version as the reserved word. -- Create a STable -Similiar to a standard table creation SQL, but you need to specify name and type of TAGS field. - -Note: - -1. Data types of TAGS column cannot be timestamp; -2. No duplicated TAGS column names; -3. Reversed word cannot be used as a TAGS column name; -4. The maximum number of TAGS is 128, and at least 1 TAG allowed, with a total length of no more than 16k characters. +- **Create a STable** -- Drop a STable + ```mysql + CREATE STABLE [IF NOT EXISTS] stb_name (timestamp_field_name TIMESTAMP, field1_name data_type1 [, field2_name data_type2 ...]) TAGS (tag1_name tag_type1, tag2_name tag_type2 [, tag3_name tag_type3]); + ``` + Similiar to a standard table creation SQL, but you need to specify name and type of TAGS field. + + Note: + + 1. Data types of TAGS column cannot be timestamp; + 2. No duplicated TAGS column names; + 3. Reversed word cannot be used as a TAGS column name; + 4. The maximum number of TAGS is 128, and at least 1 TAG allowed, with a total length of no more than 16k characters. -- ```mysql - DROP STABLE [IF EXISTS] stb_name; - ``` +- **Drop a STable** -Drop a STable automatically deletes all sub-tables created through the STable. + ```mysql + DROP STABLE [IF EXISTS] stb_name; + ``` + Drop a STable automatically deletes all sub-tables created through the STable. -- Show all STable information under the current database +- **Show all STable information under the current database** -- ```mysql - SHOW STABLES [LIKE tb_name_wildcard]; - ``` + ```mysql + SHOW STABLES [LIKE tb_name_wildcard]; + ``` + View all STables under the current database and relevant information, including name, creation time, column number, tag number, number of tables created through the STable, etc. -View all STables under the current database and relevant information, including name, creation time, column number, tag number, number of tables created through the STable, etc. +- **Obtain schema information of a STable** -- Obtain schema information of a STable + ```mysql + DESCRIBE stb_name; + ``` -- ```mysql - DESCRIBE stb_name; - ``` +- **Add column to STable** -- Add column to STable + ```mysql + ALTER STABLE stb_name ADD COLUMN field_name data_type; + ``` -- Drop column in STable +- **Drop column in STable** -- ```mysql - ALTER STABLE stb_name DROP COLUMN field_name; - ``` + ```mysql + ALTER STABLE stb_name DROP COLUMN field_name; + ``` ## TAG Management in STable -- Add a tag -Add a new tag to the STable and specify a type of the new tag. The total number of tags cannot exceed 128 and the total length does not exceed 16K characters. - -- Drop a tag - -- ```mysql - ALTER STABLE stb_name DROP TAG tag_name; - ``` - -Delete a tag of STable. After deleting the tag, all sub-tables under the STable will also automatically delete the same tag. +- **Add a tag** -- Modify a tag name + ```mysql + ALTER STABLE stb_name ADD TAG new_tag_name tag_type; + ``` + Add a new tag to the STable and specify a type of the new tag. The total number of tags cannot exceed 128 and the total length does not exceed 16K characters. -- ```mysql - ALTER STABLE stb_name CHANGE TAG old_tag_name new_tag_name; - ``` +- **Drop a tag** -Modify a tag name of STable. After modifying, all sub-tables under the STable will automatically update the new tag name. + ```mysql + ALTER STABLE stb_name DROP TAG tag_name; + ``` + Delete a tag of STable. After deleting the tag, all sub-tables under the STable will also automatically delete the same tag. -- Modify a tag value of sub-table +- **Modify a tag name** -- ```mysql - ALTER TABLE tb_name SET TAG tag_name=new_tag_value; - ``` + ```mysql + ALTER STABLE stb_name CHANGE TAG old_tag_name new_tag_name; + ``` + Modify a tag name of STable. After modifying, all sub-tables under the STable will automatically update the new tag name. -Note: Except that the operation of tag value updating is carried out for sub-tables, all other tag operations (adding tags, deleting tags, etc.) can only be applied to STable, and cannot be operated on a single sub-table. After adding a tag to a STable, all tables established based on that will automatically add a new tag, and the default value is NULL. +- **Modify a tag value of sub-table** + + ```mysql + ALTER TABLE tb_name SET TAG tag_name=new_tag_value; + ``` + Note: Except that the operation of tag value updating is carried out for sub-tables, all other tag operations (adding tags, deleting tags, etc.) can only be applied to STable, and cannot be operated on a single sub-table. After adding a tag to a STable, all tables established based on that will automatically add a new tag, and the default value is NULL. ## Data Writing -- Insert a record +- **Insert a record** ```mysql - INSERT INTO tb_name VALUES (field_value, ...); + INSERT INTO tb_name VALUES (field_value, ...); ``` + Insert a record into table tb_name. -Insert a record into table tb_name - -- Insert a record with data corresponding to a given column - -Insert a record into table tb_name, and the data corresponds to a given column. For columns that do not appear in the SQL statement, database will automatically populate them with NULL. Primary key (timestamp) cannot be NULL. - -- Insert multiple records - -- ```mysql - INSERT INTO tb_name VALUES (field1_value1, ...) (field1_value2, ...) ...; - ``` - -- Insert multiple records into table tb_name - -- Insert multiple records into a given column - -- ```mysql - INSERT INTO tb_name (field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...; - ``` - -Insert multiple records into a given column of table tb_name - -- Insert multiple records into multiple tables +- **Insert a record with data corresponding to a given column** + + ```mysql + INSERT INTO tb_name (field1_name, ...) VALUES (field1_value1, ...); + ``` + Insert a record into table tb_name, and the data corresponds to a given column. For columns that do not appear in the SQL statement, database will automatically populate them with NULL. Primary key (timestamp) cannot be NULL. -- ```mysql - INSERT INTO tb1_name VALUES (field1_value1, ...) (field1_value2, ...) ... - tb2_name VALUES (field1_value1, ...) (field1_value2, ...) ...; - ``` +- **Insert multiple records** -Insert multiple records into tables tb1_name and tb2_name at the same time + ```mysql + INSERT INTO tb_name VALUES (field1_value1, ...) (field1_value2, ...) ...; + ``` + Insert multiple records into table tb_name. -- Insert multiple records per column into multiple tables +- **Insert multiple records into a given column** + + ```mysql + INSERT INTO tb_name (field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...; + ``` + Insert multiple records into a given column of table tb_name. -- ```mysql - INSERT INTO tb1_name (tb1_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ... - tb2_name (tb2_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...; - ``` +- **Insert multiple records into multiple tables** + + ```mysql + INSERT INTO tb1_name VALUES (field1_value1, ...) (field1_value2, ...) ... + tb2_name VALUES (field1_value1, ...) (field1_value2, ...) ...; + ``` + Insert multiple records into tables tb1_name and tb2_name at the same time. -Insert multiple records per column into tables tb1_name and tb2_name at the same time +- **Insert multiple records per column into multiple tables** -Note: The timestamp of the oldest record allowed to be inserted is relative to the current server time, minus the configured keep value (days of data retention), and the timestamp of the latest record allowed to be inserted is relative to the current server time, plus the configured days value (interval of data storage in the data file, in days). Both keep and days can be specified when the database is created, and the default values are 3650 days and 10 days, respectively. + ```mysql + INSERT INTO tb1_name (tb1_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ... + tb2_name (tb2_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...; + ``` + Insert multiple records per column into tables tb1_name and tb2_name at the same time. + Note: The timestamp of the oldest record allowed to be inserted is relative to the current server time, minus the configured keep value (days of data retention), and the timestamp of the latest record allowed to be inserted is relative to the current server time, plus the configured days value (interval of data storage in the data file, in days). Both keep and days can be specified when the database is created, and the default values are 3650 days and 10 days, respectively. - Automatically create a table when inserting -- ```mysql - INSERT INTO tb_name USING stb_name TAGS (tag_value1, ...) VALUES (field_value1, ...); - ``` - -If user is not sure whether a table exists when writing data, the automatic table building syntax can be used to create a non-existent table when writing. If the table already exists, no new table will be created. When automatically creating a table, it is required to use the STable as a template and specify tags value for the data table. - -- Automatically create a table when inserting, and specify a given tags column + ```mysql + INSERT INTO tb_name USING stb_name TAGS (tag_value1, ...) VALUES (field_value1, ...); + ``` + If user is not sure whether a table exists when writing data, the automatic table building syntax can be used to create a non-existent table when writing. If the table already exists, no new table will be created. When automatically creating a table, it is required to use the STable as a template and specify tags value for the data table. -During automatic table creation, only the values of some tags columns can be specified, and the unspecified tags columns will be null. +- **Automatically create a table when inserting, and specify a given tags column** + + ```mysql + INSERT INTO tb_name USING stb_name (tag_name1, ...) TAGS (tag_value1, ...) VALUES (field_value1, ...); + ``` + During automatic table creation, only the values of some tags columns can be specified, and the unspecified tags columns will be null. **History writing**: The IMPORT or INSERT command can be used. The syntax and function of IMPORT are exactly the same as those of INSERT. @@ -427,7 +437,6 @@ taos> SELECT * FROM meters; 2018-10-03 14:38:15.000 | 12.60000 | 218 | 0.33000 | Beijing.Chaoyang | 2 | 2018-10-03 14:38:16.800 | 12.30000 | 221 | 0.31000 | Beijing.Chaoyang | 2 | Query OK, 9 row(s) in set (0.002022s) -​``` ``` Wildcards support table name prefixes, the two following SQL statements will return all columns: @@ -650,29 +659,34 @@ Query OK, 1 row(s) in set (0.001091s) ### SQL Example - For example, table tb1 is created with the following statement -- ```mysql - CREATE TABLE tb1 (ts TIMESTAMP, col1 INT, col2 FLOAT, col3 BINARY(50)); - ``` + + ```mysql + CREATE TABLE tb1 (ts TIMESTAMP, col1 INT, col2 FLOAT, col3 BINARY(50)); + ``` - Query all records of the last hour of tb1 + ```mysql + SELECT * FROM tb1 WHERE ts >= NOW - 1h; + ``` + - Look up table tb1 from 2018-06-01 08:00:00. 000 to 2018-06-02 08:00:00. 000, and col3 string is a record ending in'nny ', and the result is in descending order of timestamp: -- ```mysql - SELECT * FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND ts <= '2018-06-02 08:00:00.000' AND col3 LIKE '%nny' ORDER BY ts DESC; - ``` + ```mysql + SELECT * FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND ts <= '2018-06-02 08:00:00.000' AND col3 LIKE '%nny' ORDER BY ts DESC; + ``` - Query the sum of col1 and col2, and name it complex. The time is greater than 2018-06-01 08:00:00. 000, and col2 is greater than 1.2. As a result, only 10 records are outputted, starting from item 5 -- ```mysql - SELECT (col1 + col2) AS 'complex' FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND col2 > 1.2 LIMIT 10 OFFSET 5; - ``` + ```mysql + SELECT (col1 + col2) AS 'complex' FROM tb1 WHERE ts > '2018-06-01 08:00:00.000' AND col2 > 1.2 LIMIT 10 OFFSET 5; + ``` - Query the records of past 10 minutes, the value of col2 is greater than 3.14, and output the result to the file /home/testoutpu.csv. -- ```mysql - SELECT COUNT(*) FROM tb1 WHERE ts >= NOW - 10m AND col2 > 3.14 >> /home/testoutpu.csv; - ``` + ```mysql + SELECT COUNT(*) FROM tb1 WHERE ts >= NOW - 10m AND col2 > 3.14 >> /home/testoutpu.csv; + ``` @@ -681,150 +695,172 @@ Query OK, 1 row(s) in set (0.001091s) TDengine supports aggregations over data, they are listed below: - **COUNT** -Function: record the number of rows or non-null values in a column of statistics/STable. - -Returned result data type: long integer INT64. -Applicable Fields: Applied to all fields. - -Applied to: **table, STable**. - -Note: - -1. 1. You can use \* instead of specific fields, and use *() to return the total number of records. - 2. The query results for fields of the same table (excluding NULL values) are the same. - 3. If the statistic object is a specific column, return the number of records with non-NULL values in that column. - -Example: - -- ```mysql - taos> SELECT COUNT(*), COUNT(voltage) FROM meters; - count(*) | count(voltage) | - ================================================ - 9 | 9 | - Query OK, 1 row(s) in set (0.004475s) + ```mysql + SELECT COUNT([*|field_name]) FROM tb_name [WHERE clause]; + ``` + Function: record the number of rows or non-null values in a column of statistics/STable. + + Returned result data type: long integer INT64. + + Applicable Fields: Applied to all fields. + + Applied to: **table, STable**. + + Note: + 1. You can use \* instead of specific fields, and use *() to return the total number of records. + 2. The query results for fields of the same table (excluding NULL values) are the same. + 3. If the statistic object is a specific column, return the number of records with non-NULL values in that column. + + Example: + + ```mysql + taos> SELECT COUNT(*), COUNT(voltage) FROM meters; + count(*) | count(voltage) | + ================================================ + 9 | 9 | + Query OK, 1 row(s) in set (0.004475s) - taos> SELECT COUNT(*), COUNT(voltage) FROM d1001; - count(*) | count(voltage) | - ================================================ - 3 | 3 | - Query OK, 1 row(s) in set (0.001075s) + taos> SELECT COUNT(*), COUNT(voltage) FROM d1001; + count(*) | count(voltage) | + ================================================ + 3 | 3 | + Query OK, 1 row(s) in set (0.001075s) ``` - **AVG** -- ```mysql - SELECT AVG(field_name) FROM tb_name [WHERE clause]; + ```mysql + SELECT AVG(field_name) FROM tb_name [WHERE clause]; ``` - -Function: return the average value of a column in statistics/STable. - -Return Data Type: double. - -Applicable Fields: all types except timestamp, binary, nchar, bool. - -Applied to: **table,STable**. - -Example: - -- ```mysql - taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM meters; - avg(current) | avg(voltage) | avg(phase) | - ==================================================================================== - 11.466666751 | 220.444444444 | 0.293333333 | - Query OK, 1 row(s) in set (0.004135s) + Function: return the average value of a column in statistics/STable. - taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM d1001; - avg(current) | avg(voltage) | avg(phase) | - ==================================================================================== - 11.733333588 | 219.333333333 | 0.316666673 | - Query OK, 1 row(s) in set (0.000943s) - ``` + Return Data Type: double. + + Applicable Fields: all types except timestamp, binary, nchar, bool. + + Applied to: **table,STable**. + + Example: + + ```mysql + taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM meters; + avg(current) | avg(voltage) | avg(phase) | + ==================================================================================== + 11.466666751 | 220.444444444 | 0.293333333 | + Query OK, 1 row(s) in set (0.004135s) + + taos> SELECT AVG(current), AVG(voltage), AVG(phase) FROM d1001; + avg(current) | avg(voltage) | avg(phase) | + ==================================================================================== + 11.733333588 | 219.333333333 | 0.316666673 | + Query OK, 1 row(s) in set (0.000943s) + ``` - **TWA** - -- ```mysql - SELECT TWA(field_name) FROM tb_name WHERE clause; + + ```mysql + SELECT TWA(field_name) FROM tb_name WHERE clause; ``` - -Function: Time weighted average function. The time-weighted average of a column in a statistical table over a period of time. - -Return Data Type: double. - -Applicable Fields: all types except timestamp, binary, nchar, bool. - -Applied to: **table**. + + Function: Time weighted average function. The time-weighted average of a column in a statistical table over a period of time. + + Return Data Type: double. + + Applicable Fields: all types except timestamp, binary, nchar, bool. + + Applied to: **table**. - **SUM** -- ```mysql - SELECT SUM(field_name) FROM tb_name [WHERE clause]; + ```mysql + SELECT SUM(field_name) FROM tb_name [WHERE clause]; ``` + + Function: return the sum of a statistics/STable. + + Return Data Type: long integer INMT64 and Double. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Applied to: **table,STable**. + + Example: -Function: return the sum of a statistics/STable. - -Return Data Type: long integer INMT64 and Double. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Applied to: **table,STable**. - -Example: - -- ```mysql - taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM meters; - sum(current) | sum(voltage) | sum(phase) | - ================================================================================ - 103.200000763 | 1984 | 2.640000001 | - Query OK, 1 row(s) in set (0.001702s) + ```mysql + taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM meters; + sum(current) | sum(voltage) | sum(phase) | + ================================================================================ + 103.200000763 | 1984 | 2.640000001 | + Query OK, 1 row(s) in set (0.001702s) - taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM d1001; - sum(current) | sum(voltage) | sum(phase) | - ================================================================================ - 35.200000763 | 658 | 0.950000018 | - Query OK, 1 row(s) in set (0.000980s) + taos> SELECT SUM(current), SUM(voltage), SUM(phase) FROM d1001; + sum(current) | sum(voltage) | sum(phase) | + ================================================================================ + 35.200000763 | 658 | 0.950000018 | + Query OK, 1 row(s) in set (0.000980s) ``` - **STDDEV** - -- ```mysql - SELECT STDDEV(field_name) FROM tb_name [WHERE clause]; + + ```mysql + SELECT STDDEV(field_name) FROM tb_name [WHERE clause]; + ``` + + Function: Mean square deviation of a column in statistics table. + + Return Data Type: Double. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Applied to: **table**. (also support **STable** since version 2.0.15.1) + + Example: + + ```mysql + taos> SELECT STDDEV(current) FROM d1001; + stddev(current) | + ============================ + 1.020892909 | + Query OK, 1 row(s) in set (0.000915s) ``` - -Function: Mean square deviation of a column in statistics table. - -Return Data Type: Double. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Applied to: **table**. (also support **STable** since version 2.0.15.1) - -Example: - **LEASTSQUARES** - -Function: Value of a column in statistical table is a fitting straight equation of primary key (timestamp). Start_val is the initial value of independent variable, and step_val is the step size value of independent variable. - -Return Data Type: String expression (slope, intercept). - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: Independent variable is the timestamp, and dependent variable is the value of the column. - -Applied to: **table**. - -Example: + ```mysql + SELECT LEASTSQUARES(field_name, start_val, step_val) FROM tb_name [WHERE clause]; + ``` + Function: Value of a column in statistical table is a fitting straight equation of primary key (timestamp). Start_val is the initial value of independent variable, and step_val is the step size value of independent variable. + + Return Data Type: String expression (slope, intercept). + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: Independent variable is the timestamp, and dependent variable is the value of the column. + + Applied to: **table**. + + Example: + ```mysql + taos> SELECT LEASTSQUARES(current, 1, 1) FROM d1001; + leastsquares(current, 1, 1) | + ===================================================== + {slop:1.000000, intercept:9.733334} | + Query OK, 1 row(s) in set (0.000921s) + ``` ### Selector Functions - **MIN** -Function: return the minimum value of a specific column in statistics/STable. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Example: + ```mysql + SELECT MIN(field_name) FROM {tb_name | stb_name} [WHERE clause]; + ``` + Function: return the minimum value of a specific column in statistics/STable. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Example: ```mysql taos> SELECT MIN(current), MIN(voltage) FROM meters; @@ -842,86 +878,84 @@ Example: - **MAX** -- ```mysql - SELECT MAX(field_name) FROM { tb_name | stb_name } [WHERE clause]; - ``` - -Function: return the maximum value of a specific column in statistics/STable. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Example: + ```mysql + SELECT MAX(field_name) FROM { tb_name | stb_name } [WHERE clause]; + ``` + + Function: return the maximum value of a specific column in statistics/STable. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Example: -- ```mysql - taos> SELECT MAX(current), MAX(voltage) FROM meters; - max(current) | max(voltage) | - ====================================== - 13.40000 | 223 | - Query OK, 1 row(s) in set (0.001123s) + ```mysql + taos> SELECT MAX(current), MAX(voltage) FROM meters; + max(current) | max(voltage) | + ====================================== + 13.40000 | 223 | + Query OK, 1 row(s) in set (0.001123s) - taos> SELECT MAX(current), MAX(voltage) FROM d1001; - max(current) | max(voltage) | - ====================================== - 12.60000 | 221 | - Query OK, 1 row(s) in set (0.000987s) - ``` + taos> SELECT MAX(current), MAX(voltage) FROM d1001; + max(current) | max(voltage) | + ====================================== + 12.60000 | 221 | + Query OK, 1 row(s) in set (0.000987s) + ``` - **FIRST** -- ```mysql - SELECT FIRST(field_name) FROM { tb_name | stb_name } [WHERE clause]; - ``` - -Function: The first non-NULL value written into a column in statistics/STable. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types. - -Note: - -1. 1. To return the first (minimum timestamp) non-NULL value of each column, use FIRST (\*); + ```mysql + SELECT FIRST(field_name) FROM { tb_name | stb_name } [WHERE clause]; + ``` + + Function: The first non-NULL value written into a column in statistics/STable. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types. + + Note: + 1. To return the first (minimum timestamp) non-NULL value of each column, use FIRST (\*); 2. if all columns in the result set are NULL values, the return result of the column is also NULL; 3. If all columns in the result set are NULL values, no result is returned. - -Example: + + Example: ```mysql - taos> SELECT FIRST(*) FROM meters; - first(ts) | first(current) | first(voltage) | first(phase) | - ========================================================================================= - 2018-10-03 14:38:04.000 | 10.20000 | 220 | 0.23000 | - Query OK, 1 row(s) in set (0.004767s) - - taos> SELECT FIRST(current) FROM d1002; - first(current) | - ======================= + taos> SELECT FIRST(*) FROM meters; + first(ts) | first(current) | first(voltage) | first(phase) | + ========================================================================================= + 2018-10-03 14:38:04.000 | 10.20000 | 220 | 0.23000 | + Query OK, 1 row(s) in set (0.004767s) + + taos> SELECT FIRST(current) FROM d1002; + first(current) | + ======================= 10.20000 | - Query OK, 1 row(s) in set (0.001023s) + Query OK, 1 row(s) in set (0.001023s) ``` - - **LAST** -- ```mysql - SELECT LAST(field_name) FROM { tb_name | stb_name } [WHERE clause]; - ``` - -Function: The last non-NULL value written by the value of a column in statistics/STable. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types. - -Note: - -1. 1. To return the last (maximum timestamp) non-NULL value of each column, use LAST (\*); + ```mysql + SELECT LAST(field_name) FROM { tb_name | stb_name } [WHERE clause]; + ``` + + Function: The last non-NULL value written by the value of a column in statistics/STable. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types. + + Note: + 1. To return the last (maximum timestamp) non-NULL value of each column, use LAST (\*); 2. If a column in the result set has a NULL value, the returned result of the column is also NULL; if all columns in the result set have NULL values, no result is returned. - -Example: + + Example: ```mysql taos> SELECT LAST(*) FROM meters; @@ -937,195 +971,204 @@ Example: Query OK, 1 row(s) in set (0.000843s) ``` -- - - **TOP** + + ```mysql + SELECT TOP(field_name, K) FROM { tb_name | stb_name } [WHERE clause]; + ``` + Function: The top k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the largest, the one with smaller timestamp is returned. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: + 1. The range of *k* value is 1≤*k*≤100; + 2. System also returns the timestamp column associated with the record. + + Example: -Function: The top k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the largest, the one with smaller timestamp is returned. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: - -1. *The range of k value is* 1≤*k*≤100; -2. System also returns the timestamp column associated with the record. - -Example: - -- ```mysql - taos> SELECT TOP(current, 3) FROM meters; - ts | top(current, 3) | - ================================================= - 2018-10-03 14:38:15.000 | 12.60000 | - 2018-10-03 14:38:16.600 | 13.40000 | - 2018-10-03 14:38:16.800 | 12.30000 | - Query OK, 3 row(s) in set (0.001548s) + ```mysql + taos> SELECT TOP(current, 3) FROM meters; + ts | top(current, 3) | + ================================================= + 2018-10-03 14:38:15.000 | 12.60000 | + 2018-10-03 14:38:16.600 | 13.40000 | + 2018-10-03 14:38:16.800 | 12.30000 | + Query OK, 3 row(s) in set (0.001548s) - taos> SELECT TOP(current, 2) FROM d1001; - ts | top(current, 2) | - ================================================= - 2018-10-03 14:38:15.000 | 12.60000 | - 2018-10-03 14:38:16.800 | 12.30000 | - Query OK, 2 row(s) in set (0.000810s) - ``` + taos> SELECT TOP(current, 2) FROM d1001; + ts | top(current, 2) | + ================================================= + 2018-10-03 14:38:15.000 | 12.60000 | + 2018-10-03 14:38:16.800 | 12.30000 | + Query OK, 2 row(s) in set (0.000810s) + ``` - **BOTTOM** -Function: The last k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the smallest, the one with smaller timestamp is returned. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: + ```mysql + SELECT BOTTOM(field_name, K) FROM { tb_name | stb_name } [WHERE clause]; + ``` + Function: The last k non-NULL values of a column in statistics/STable. If there are more than k column values tied for the smallest, the one with smaller timestamp is returned. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except timestamp, binary, nchar, bool. -1. *The range of k value is* 1≤*k*≤100; -2. System also returns the timestamp column associated with the record. + Note: + 1. The range of *k* value is 1≤*k*≤100; + 2. System also returns the timestamp column associated with the record. -Example: + Example: -- ```mysql - taos> SELECT BOTTOM(voltage, 2) FROM meters; - ts | bottom(voltage, 2) | - =============================================== - 2018-10-03 14:38:15.000 | 218 | - 2018-10-03 14:38:16.650 | 218 | - Query OK, 2 row(s) in set (0.001332s) + ```mysql + taos> SELECT BOTTOM(voltage, 2) FROM meters; + ts | bottom(voltage, 2) | + =============================================== + 2018-10-03 14:38:15.000 | 218 | + 2018-10-03 14:38:16.650 | 218 | + Query OK, 2 row(s) in set (0.001332s) - taos> SELECT BOTTOM(current, 2) FROM d1001; - ts | bottom(current, 2) | - ================================================= - 2018-10-03 14:38:05.000 | 10.30000 | - 2018-10-03 14:38:16.800 | 12.30000 | - Query OK, 2 row(s) in set (0.000793s) + taos> SELECT BOTTOM(current, 2) FROM d1001; + ts | bottom(current, 2) | + ================================================= + 2018-10-03 14:38:05.000 | 10.30000 | + 2018-10-03 14:38:16.800 | 12.30000 | + Query OK, 2 row(s) in set (0.000793s) ``` - **PERCENTILE** + ```mysql + SELECT PERCENTILE(field_name, P) FROM { tb_name } [WHERE clause]; + ``` + Function: Percentile of the value of a column in statistical table. + + Return Data Type: Double. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: The range of P value is 0 ≤ P ≤ 100. P equals to MIN when, and equals MAX when it’s 100. + + Example: -Function: Percentile of the value of a column in statistical table. - -Return Data Type: Double. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: The range of P value is 0 ≤ P ≤ 100. P equals to MIN when, and equals MAX when it’s 100. - -Example: - - ```mysql + ```mysql taos> SELECT PERCENTILE(current, 20) FROM d1001; percentile(current, 20) | ============================ 11.100000191 | Query OK, 1 row(s) in set (0.000787s) - ``` - -- + ``` - **APERCENTILE** - -Function: The value percentile of a column in statistical table is similar to the PERCENTILE function, but returns approximate results. - -Return Data Type: Double. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: The range of P value is 0 ≤ P ≤ 100. P equals to MIN when, and equals MAX when it’s 100. APERCENTILE function is recommended, which performs far better than PERCENTILE function. + ```mysql + SELECT APERCENTILE(field_name, P) FROM { tb_name | stb_name } [WHERE clause]; + ``` + Function: The value percentile of a column in statistical table is similar to the PERCENTILE function, but returns approximate results. + + Return Data Type: Double. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: The range of *P* value is 0 ≤ *P* ≤ 100. *P* equals to MIN when, and equals MAX when it’s 100. APERCENTILE function is recommended, which performs far better than PERCENTILE function. - **LAST_ROW** + ```mysql + SELECT LAST_ROW(field_name) FROM { tb_name | stb_name }; + ``` + Function: Return the last record of a table (STtable). + + Return Data Type: Double. + + Applicable Fields: All types. + + Note: Unlike last function, last_row does not support time range restriction and forces the last record to be returned. + + Example: -Function: Return the last record of a table (STtable). - -Return Data Type: Double. - -Applicable Fields: All types. - -Note: Unlike last function, last_row does not support time range restriction and forces the last record to be returned. - -Example: - -- ```mysql - taos> SELECT LAST_ROW(current) FROM meters; - last_row(current) | - ======================= - 12.30000 | - Query OK, 1 row(s) in set (0.001238s) + ```mysql + taos> SELECT LAST_ROW(current) FROM meters; + last_row(current) | + ======================= + 12.30000 | + Query OK, 1 row(s) in set (0.001238s) - taos> SELECT LAST_ROW(current) FROM d1002; - last_row(current) | - ======================= - 10.30000 | + taos> SELECT LAST_ROW(current) FROM d1002; + last_row(current) | + ======================= + 10.30000 | Query OK, 1 row(s) in set (0.001042s) ``` - - ### Computing Functions - **DIFF** -Function: Return the value difference between a column and the previous column. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except timestamp, binary, nchar, bool. - -Note: The number of output result lines is the total number of lines in the range minus one, and there is no result output in the first line. - -Example: + ```mysql + SELECT DIFF(field_name) FROM tb_name [WHERE clause]; + ``` + Function: Return the value difference between a column and the previous column. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: The number of output result lines is the total number of lines in the range minus one, and there is no result output in the first line. + + Example: -- ```mysql - taos> SELECT DIFF(current) FROM d1001; - ts | diff(current) | - ================================================= - 2018-10-03 14:38:15.000 | 2.30000 | - 2018-10-03 14:38:16.800 | -0.30000 | - Query OK, 2 row(s) in set (0.001162s) - ``` + ```mysql + taos> SELECT DIFF(current) FROM d1001; + ts | diff(current) | + ================================================= + 2018-10-03 14:38:15.000 | 2.30000 | + 2018-10-03 14:38:16.800 | -0.30000 | + Query OK, 2 row(s) in set (0.001162s) + ``` - **SPREAD** -- ```mysql - SELECT SPREAD(field_name) FROM { tb_name | stb_name } [WHERE clause]; - ``` - -Function: Return the difference between the max value and the min value of a column in statistics /STable. - -Return Data Type: Same as applicable fields. - -Applicable Fields: All types except binary, nchar, bool. - -Note: Applicable for TIMESTAMP field, which indicates the time range of a record. - -Example: + ```mysql + SELECT SPREAD(field_name) FROM { tb_name | stb_name } [WHERE clause]; + ``` + Function: Return the difference between the max value and the min value of a column in statistics /STable. + + Return Data Type: Same as applicable fields. + + Applicable Fields: All types except binary, nchar, bool. + + Note: Applicable for TIMESTAMP field, which indicates the time range of a record. + + Example: -- ```mysql - taos> SELECT SPREAD(voltage) FROM meters; - spread(voltage) | - ============================ - 5.000000000 | - Query OK, 1 row(s) in set (0.001792s) + ```mysql + taos> SELECT SPREAD(voltage) FROM meters; + spread(voltage) | + ============================ + 5.000000000 | + Query OK, 1 row(s) in set (0.001792s) - taos> SELECT SPREAD(voltage) FROM d1001; - spread(voltage) | - ============================ - 3.000000000 | - Query OK, 1 row(s) in set (0.000836s) - ``` - -- Four Operations - -Function: Calculation results of addition, subtraction, multiplication, division and remainder of values in a column or among multiple columns in statistics/STable. - -Returned Data Type: Double. - -Applicable Fields: All types except timestamp, binary, nchar, bool. + taos> SELECT SPREAD(voltage) FROM d1001; + spread(voltage) | + ============================ + 3.000000000 | + Query OK, 1 row(s) in set (0.000836s) + ``` -Note: +- **Four Operations** -1. Calculation between two or more columns is supported, and the calculation priorities can be controlled by parentheses(); -2. The NULL field does not participate in the calculation. If a row involved in calculation contains NULL, the calculation result of the row is NULL. + ```mysql + SELECT field_name [+|-|*|/|%][Value|field_name] FROM { tb_name | stb_name } [WHERE clause]; + ``` + Function: Calculation results of addition, subtraction, multiplication, division and remainder of values in a column or among multiple columns in statistics/STable. + + Returned Data Type: Double. + + Applicable Fields: All types except timestamp, binary, nchar, bool. + + Note: + + 1. Calculation between two or more columns is supported, and the calculation priorities can be controlled by parentheses(); + 2. The NULL field does not participate in the calculation. If a row involved in calculation contains NULL, the calculation result of the row is NULL. ## Time-dimension Aggregation @@ -1151,18 +1194,18 @@ SELECT function_list FROM stb_name - WHERE statement specifies the start and end time of a query and other filters - FILL statement specifies a filling mode when data missed in a certain interval. Applicable filling modes include the following: - -- - Do not fill: NONE (default filingl mode). - - VALUE filling: Fixed value filling, where the filled value needs to be specified. For example: fill (VALUE, 1.23). - - NULL filling: Fill the data with NULL. For example: fill (NULL). - - PREV filling: Filling data with the previous non-NULL value. For example: fill (PREV). - - NEXT filling: Filling data with the next non-NULL value. For example: fill (NEXT). + + 1. Do not fill: NONE (default filingl mode). + 2. VALUE filling: Fixed value filling, where the filled value needs to be specified. For example: fill (VALUE, 1.23). + 3. NULL filling: Fill the data with NULL. For example: fill (NULL). + 4. PREV filling: Filling data with the previous non-NULL value. For example: fill (PREV). + 5. NEXT filling: Filling data with the next non-NULL value. For example: fill (NEXT). Note: -1. When using a FILL statement, a large number of filling outputs may be generated. Be sure to specify the time interval for the query. For each query, system can return no more than 10 million results with interpolation. -2. In a time-dimension aggregation, the time-series in returned results increases strictly monotonously. -3. If the query object is a STable, the aggregator function will act on the data of all tables under the STable that meet the value filters. If group by statement is not used in the query, the returned result increases strictly monotonously according to time-series; If group by statement is used to group in the query, each group in the returned result does not increase strictly monotonously according to time-series. + 1. When using a FILL statement, a large number of filling outputs may be generated. Be sure to specify the time interval for the query. For each query, system can return no more than 10 million results with interpolation. + 2. In a time-dimension aggregation, the time-series in returned results increases strictly monotonously. + 3. If the query object is a STable, the aggregator function will act on the data of all tables under the STable that meet the value filters. If group by statement is not used in the query, the returned result increases strictly monotonously according to time-series; If group by statement is used to group in the query, each group in the returned result does not increase strictly monotonously according to time-series. Example: The statement for building a database for smart meter is as follows: @@ -1200,4 +1243,4 @@ TAOS SQL supports join columns of two tables by Primary Key timestamp between th **Availability of is no null** -Is not null supports all types of columns. Non-null expression is < > "" and only applies to columns of non-numeric types. \ No newline at end of file +Is not null supports all types of columns. Non-null expression is < > "" and only applies to columns of non-numeric types. diff --git a/documentation20/webdocs/assets/Picture2.png b/documentation20/webdocs/assets/Picture2.png deleted file mode 100644 index 715a8bd37ee9fe7e96eacce4e7ff563fedeefbee..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/Picture2.png and /dev/null differ diff --git a/documentation20/webdocs/assets/clip_image001-2474914.png b/documentation20/webdocs/assets/clip_image001-2474914.png deleted file mode 100644 index eb369b1567c860b772e1bfdad64ff17aaac2534d..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/clip_image001-2474914.png and /dev/null differ diff --git a/documentation20/webdocs/assets/clip_image001-2474939.png b/documentation20/webdocs/assets/clip_image001-2474939.png deleted file mode 100644 index 53f00deea3a484986a5681ec9d00d8ae02e88fec..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/clip_image001-2474939.png and /dev/null differ diff --git a/documentation20/webdocs/assets/clip_image001-2474961.png b/documentation20/webdocs/assets/clip_image001-2474961.png deleted file mode 100644 index 20ae8d6f7724a4bddcf8c7eb3809d468aa4223ed..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/clip_image001-2474961.png and /dev/null differ diff --git a/documentation20/webdocs/assets/clip_image001-2474987.png b/documentation20/webdocs/assets/clip_image001-2474987.png deleted file mode 100644 index 3d09f7fc28e7a1fb7e3bb2b9b2bc7c20895e8bb4..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/clip_image001-2474987.png and /dev/null differ diff --git a/documentation20/webdocs/assets/clip_image001.png b/documentation20/webdocs/assets/clip_image001.png deleted file mode 100644 index 78b6d06a9562b802e80f0ed5fdb8963b5e525589..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/clip_image001.png and /dev/null differ diff --git a/documentation20/webdocs/assets/fig1.png b/documentation20/webdocs/assets/fig1.png deleted file mode 100644 index af9b74e0d1a872b8d93f71842dc0063bc8a86092..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/fig1.png and /dev/null differ diff --git a/documentation20/webdocs/assets/fig2.png b/documentation20/webdocs/assets/fig2.png deleted file mode 100644 index 3bae70ba86964c3c341b72ea1d3af04201f7c6c1..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/fig2.png and /dev/null differ diff --git a/documentation20/webdocs/assets/image-20190707124650780.png b/documentation20/webdocs/assets/image-20190707124650780.png deleted file mode 100644 index 9ebcac863e862d8b240c86dec29be1ebe7aa50f0..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/image-20190707124650780.png and /dev/null differ diff --git a/documentation20/webdocs/assets/image-20190707124818590.png b/documentation20/webdocs/assets/image-20190707124818590.png deleted file mode 100644 index dc1cb6325b2d4cd6f05c88b75b4d17ef85caa67f..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/image-20190707124818590.png and /dev/null differ diff --git a/documentation20/webdocs/assets/stable.png b/documentation20/webdocs/assets/stable.png deleted file mode 100644 index 0cefaab6a9a4cdd671c671f7c6186dea41415ff0..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/stable.png and /dev/null differ diff --git a/documentation20/webdocs/assets/structure.png b/documentation20/webdocs/assets/structure.png deleted file mode 100644 index 4fc8f47ab0a30d95b85ba1d85105726ed981e56e..0000000000000000000000000000000000000000 Binary files a/documentation20/webdocs/assets/structure.png and /dev/null differ diff --git a/documentation20/webdocs/markdowndocs/Connections with other Tools.md b/documentation20/webdocs/markdowndocs/Connections with other Tools.md deleted file mode 100644 index 8be05698497184aee2c41a60e32f39b636e2070e..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Connections with other Tools.md +++ /dev/null @@ -1,167 +0,0 @@ -# Connect with other tools - -## Telegraf - -TDengine is easy to integrate with [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/), an open-source server agent for collecting and sending metrics and events, without more development. - -### Install Telegraf - -At present, TDengine supports Telegraf newer than version 1.7.4. Users can go to the [download link] and choose the proper package to install on your system. - -### Configure Telegraf - -Telegraf is configured by changing items in the configuration file */etc/telegraf/telegraf.conf*. - - -In **output plugins** section，add _[[outputs.http]]_ iterm： - -- _url_: http://ip:6020/telegraf/udb, in which _ip_ is the IP address of any node in TDengine cluster. Port 6020 is the RESTful APT port used by TDengine. _udb_ is the name of the database to save data, which needs to create beforehand. -- _method_: "POST" -- _username_: username to login TDengine -- _password_: password to login TDengine -- _data_format_: "json" -- _json_timestamp_units_: "1ms" - -In **agent** part： - -- hostname: used to distinguish different machines. Need to be unique. -- metric_batch_size: 30，the maximum number of records allowed to write in Telegraf. The larger the value is, the less frequent requests are sent. For TDengine, the value should be less than 50. - -Please refer to the [Telegraf docs](https://docs.influxdata.com/telegraf/v1.11/) for more information. - -## Grafana - -[Grafana] is an open-source system for time-series data display. It is easy to integrate TDengine and Grafana to build a monitor system. Data saved in TDengine can be fetched and shown on the Grafana dashboard. - -### Install Grafana - -For now, TDengine only supports Grafana newer than version 5.2.4. Users can go to the [Grafana download page] for the proper package to download. - -### Configure Grafana - -TDengine Grafana plugin is in the _/usr/local/taos/connector/grafana_ directory. -Taking Centos 7.2 as an example, just copy TDengine directory to _/var/lib/grafana/plugins_ directory and restart Grafana. - -### Use Grafana - -Users can log in the Grafana server (username/password:admin/admin) through localhost:3000 to configure TDengine as the data source. As is shown in the picture below, TDengine as a data source option is shown in the box: - - -![img](../assets/clip_image001.png) - -When choosing TDengine as the data source, the Host in HTTP configuration should be configured as the IP address of any node of a TDengine cluster. The port should be set as 6020. For example, when TDengine and Grafana are on the same machine, it should be configured as _http://localhost:6020. - - -Besides, users also should set the username and password used to log into TDengine. Then click _Save&Test_ button to save. - -![img](../assets/clip_image001-2474914.png) - -Then, TDengine as a data source should show in the Grafana data source list. - -![img](../assets/clip_image001-2474939.png) - - -Then, users can create Dashboards in Grafana using TDengine as the data source: - - -![img](../assets/clip_image001-2474961.png) - - - -Click _Add Query_ button to add a query and input the SQL command you want to run in the _INPUT SQL_ text box. The SQL command should expect a two-row, multi-column result, such as _SELECT count(*) FROM sys.cpu WHERE ts>=from and ts<​to interval(interval)_, in which, _from_, _to_ and _inteval_ are TDengine inner variables representing query time range and time interval. - - -_ALIAS BY_ field is to set the query alias. Click _GENERATE SQL_ to send the command to TDengine: - -![img](../assets/clip_image001-2474987.png) - -Please refer to the [Grafana official document] for more information about Grafana. - - -## Matlab - -Matlab can connect to and retrieve data from TDengine by TDengine JDBC Driver. - -### MatLab and TDengine JDBC adaptation - -Several steps are required to adapt Matlab to TDengine. Taking adapting Matlab2017a on Windows10 as an example: - -1. Copy the file _JDBCDriver-1.0.0-dist.jar_ in TDengine package to the directory _${matlab_root}\MATLAB\R2017a\java\jar\toolbox_ -2. Copy the file _taos.lib_ in TDengine package to _${matlab_ root _dir}\MATLAB\R2017a\lib\win64_ -3. Add the .jar package just copied to the Matlab classpath. Append the line below as the end of the file of _${matlab_ root _dir}\MATLAB\R2017a\toolbox\local\classpath.txt_ - -​ `$matlabroot/java/jar/toolbox/JDBCDriver-1.0.0-dist.jar` - -4. Create a file called _javalibrarypath.txt_ in directory _${user_home}\AppData\Roaming\MathWorks\MATLAB\R2017a\_, and add the _taos.dll_ path in the file. For example, if the file _taos.dll_ is in the directory of _C:\Windows\System32_，then add the following line in file *javalibrarypath.txt*: - -​ `C:\Windows\System32` - -### TDengine operations in Matlab - -After correct configuration, open Matlab: - -- build a connection： - - `conn = database(‘db’, ‘root’, ‘taosdata’, ‘com.taosdata.jdbc.TSDBDriver’, ‘jdbc:TSDB://127.0.0.1:0/’)` - -- Query： - - `sql0 = [‘select * from tb’]` - - `data = select(conn, sql0);` - -- Insert a record: - - `sql1 = [‘insert into tb values (now, 1)’]` - - `exec(conn, sql1)` - -Please refer to the file _examples\Matlab\TDengineDemo.m_ for more information. - -## R - -Users can use R language to access the TDengine server with the JDBC interface. At first, install JDBC package in R: - -```R -install.packages('rJDBC', repos='http://cran.us.r-project.org') -``` - -Then use _library_ function to load the package: - -```R -library('RJDBC') -``` - -Then load the TDengine JDBC driver: - -```R -drv<-JDBC("com.taosdata.jdbc.TSDBDriver","JDBCDriver-1.0.0-dist.jar", identifier.quote="\"") -``` -If succeed, no error message will display. Then use the following command to try a database connection: - -```R -conn<-dbConnect(drv,"jdbc:TSDB://192.168.0.1:0/?user=root&password=taosdata","root","taosdata") -``` - -Please replace the IP address in the command above to the correct one. If no error message is shown, then the connection is established successfully. TDengine supports below functions in _RJDBC_ package: - - -- _dbWriteTable(conn, "test", iris, overwrite=FALSE, append=TRUE)_: write the data in a data frame _iris_ to the table _test_ in the TDengine server. Parameter _overwrite_ must be _false_. _append_ must be _TRUE_ and the schema of the data frame _iris_ should be the same as the table _test_. -- _dbGetQuery(conn, "select count(*) from test")_: run a query command -- _dbSendUpdate(conn, "use db")_: run any non-query command. -- _dbReadTable(conn, "test"_): read all the data in table _test_ -- _dbDisconnect(conn)_: close a connection -- _dbRemoveTable(conn, "test")_: remove table _test_ - -Below functions are **not supported** currently: -- _dbExistsTable(conn, "test")_: if talbe _test_ exists -- _dbListTables(conn)_: list all tables in the connection - - -[Telegraf]: www.taosdata.com -[download link]: https://portal.influxdata.com/downloads -[Telegraf document]: www.taosdata.com -[Grafana]: https://grafana.com -[Grafana download page]: https://grafana.com/grafana/download -[Grafana official document]: https://grafana.com/docs/ - diff --git a/documentation20/webdocs/markdowndocs/Connector.md b/documentation20/webdocs/markdowndocs/Connector.md deleted file mode 100644 index e5ba6d518542fa60f71708482a9e9b65c12d09ad..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Connector.md +++ /dev/null @@ -1,896 +0,0 @@ -# TDengine connectors - -TDengine provides many connectors for development, including C/C++, JAVA, Python, RESTful, Go, Node.JS, etc. - -NOTE: All APIs which require a SQL string as parameter, including but not limit to `taos_query`, `taos_query_a`, `taos_subscribe` in the C/C++ Connector and their counterparts in other connectors, can ONLY process one SQL statement at a time. If more than one SQL statements are provided, their behaviors are undefined. - -## C/C++ API - -C/C++ APIs are similar to the MySQL APIs. Applications should include TDengine head file _taos.h_ to use C/C++ APIs by adding the following line in code: -```C -#include -``` -Make sure TDengine library _libtaos.so_ is installed and use _-ltaos_ option to link the library when compiling. In most cases, if the return value of an API is integer, it return _0_ for success and other values as an error code for failure; if the return value is pointer, then _NULL_ is used for failure. - - -### Fundamental API - -Fundamentatal APIs prepare runtime environment for other APIs, for example, create a database connection. - -- `void taos_init()` - - Initialize the runtime environment for TDengine client. The API is not necessary since it is called int _taos_connect_ by default. - - -- `void taos_cleanup()` - - Cleanup runtime environment, client should call this API before exit. - - -- `int taos_options(TSDB_OPTION option, const void * arg, ...)` - - Set client options. The parameter _option_ supports values of _TSDB_OPTION_CONFIGDIR_ (configuration directory), _TSDB_OPTION_SHELL_ACTIVITY_TIMER_, _TSDB_OPTION_LOCALE_ (client locale) and _TSDB_OPTION_TIMEZONE_ (client timezone). - - -- `char* taos_get_client_info()` - - Retrieve version information of client. - - -- `TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, int port)` - - Open a connection to a TDengine server. The parameters are: - - * ip: IP address of the server - * user: username - * pass: password - * db: database to use, **NULL** for no database to use after connection. Otherwise, the database should exist before connection or a connection error is reported. - * port: port number to connect - - The handle returned by this API should be kept for future use. - - -- `char *taos_get_server_info(TAOS *taos)` - - Retrieve version information of server. - - -- `int taos_select_db(TAOS *taos, const char *db)` - - Set default database to `db`. - - -- `void taos_close(TAOS *taos)` - - Close a connection to a TDengine server by the handle returned by _taos_connect_` - - -### C/C++ sync API - -Sync APIs are those APIs waiting for responses from the server after sending a request. TDengine has the following sync APIs: - -- `TAOS_RES* taos_query(TAOS *taos, const char *sql)` - - The API used to run a SQL command. The command can be DQL, DML or DDL. The parameter _taos_ is the handle returned by _taos_connect_. Return value _NULL_ means failure. - - -- `int taos_result_precision(TAOS_RES *res)` - - Get the timestamp precision of the result set, return value _0_ means milli-second, _1_ mean micro-second and _2_ means nano-second. - - -- `TAOS_ROW taos_fetch_row(TAOS_RES *res)` - - Fetch a row of return results through _res_. - - -- `int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows)` - - Fetch multiple rows from the result set, return value is row count. - - -- `int taos_num_fields(TAOS_RES *res)` and `int taos_field_count(TAOS_RES* res)` - - These two APIs are identical, both return the number of fields in the return result. - - -- `int* taos_fetch_lengths(TAOS_RES *res)` - - Get the field lengths of the result set, return value is an array whose length is the field count. - - -- `int taos_affected_rows(TAOS_RES *res)` - - Get affected row count of the executed statement. - - -- `TAOS_FIELD *taos_fetch_fields(TAOS_RES *res)` - - Fetch the description of each field. The description includes the property of data type, field name, and bytes. The API should be used with _taos_num_fields_ to fetch a row of data. The structure of `TAOS_FIELD` is: - - ```c - typedef struct taosField { - char name[65]; // field name - uint8_t type; // data type - int16_t bytes; // length of the field in bytes - } TAOS_FIELD; - ``` - - -- `void taos_stop_query(TAOS_RES *res)` - - Stop the execution of a query. - - -- `void taos_free_result(TAOS_RES *res)` - - Free the resources used by a result set. Make sure to call this API after fetching results or memory leak would happen. - - -- `char *taos_errstr(TAOS_RES *res)` - - Return the reason of the last API call failure. The return value is a string. - - -- `int *taos_errno(TAOS_RES *res)` - - Return the error code of the last API call failure. The return value is an integer. - - -**Note**: The connection to a TDengine server is not multi-thread safe. So a connection can only be used by one thread. - - -### C/C++ async API - -In addition to sync APIs, TDengine also provides async APIs, which are more efficient. Async APIs are returned right away without waiting for a response from the server, allowing the application to continute with other tasks without blocking. So async APIs are more efficient, especially useful when in a poor network. - -All async APIs require callback functions. The callback functions have the format: -```C -void fp(void *param, TAOS_RES * res, TYPE param3) -``` -The first two parameters of the callback function are the same for all async APIs. The third parameter is different for different APIs. Generally, the first parameter is the handle provided to the API for action. The second parameter is a result handle. - -- `void taos_query_a(TAOS *taos, const char *sql, void (*fp)(void *param, TAOS_RES *, int code), void *param);` - - The async version of _taos_query_. - - * taos: the handle returned by _taos_connect_. - * sql: the SQL command to run. - * fp: user defined callback function. The third parameter of the callback function _code_ is _0_ (for success) or a negative number (for failure, call taos_errstr to get the error as a string). Applications mainly handle the second parameter, the returned result set. - * param: user provided parameter which is required by the callback function. - - -- `void taos_fetch_rows_a(TAOS_RES *res, void (*fp)(void *param, TAOS_RES *, int numOfRows), void *param);` - - The async API to fetch a batch of rows, which should only be used with a _taos_query_a_ call. - - * res: result handle returned by _taos_query_a_. - * fp: the callback function. _param_ is a user-defined structure to pass to _fp_. The parameter _numOfRows_ is the number of result rows in the current fetch cycle. In the callback function, applications should call _taos_fetch_row_ to get records from the result handle. After getting a batch of results, applications should continue to call _taos_fetch_rows_a_ API to handle the next batch, until the _numOfRows_ is _0_ (for no more data to fetch) or _-1_ (for failure). - - -- `void taos_fetch_row_a(TAOS_RES *res, void (*fp)(void *param, TAOS_RES *, TAOS_ROW row), void *param);` - - The async API to fetch a result row. - - * res: result handle. - * fp: the callback function. _param_ is a user-defined structure to pass to _fp_. The third parameter of the callback function is a single result row, which is different from that of _taos_fetch_rows_a_ API. With this API, it is not necessary to call _taos_fetch_row_ to retrieve each result row, which is handier than _taos_fetch_rows_a_ but less efficient. - - -Applications may apply operations on multiple tables. However, **it is important to make sure the operations on the same table are serialized**. That means after sending an insert request in a table to the server, no operations on the table are allowed before a response is received. - - -### C/C++ parameter binding API - -TDengine also provides parameter binding APIs, like MySQL, only question mark `?` can be used to represent a parameter in these APIs. - -- `TAOS_STMT* taos_stmt_init(TAOS *taos)` - - Create a TAOS_STMT to represent the prepared statement for other APIs. - -- `int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length)` - - Parse SQL statement _sql_ and bind result to _stmt_ , if _length_ larger than 0, its value is used to determine the length of _sql_, the API auto detects the actual length of _sql_ otherwise. - -- `int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_BIND *bind)` - - Bind values to parameters. _bind_ points to an array, the element count and sequence of the array must be identical as the parameters of the SQL statement. The usage of _TAOS_BIND_ is same as _MYSQL_BIND_ in MySQL, its definition is as below: - - ```c - typedef struct TAOS_BIND { - int buffer_type; - void * buffer; - unsigned long buffer_length; // not used in TDengine - unsigned long *length; - int * is_null; - int is_unsigned; // not used in TDengine - int * error; // not used in TDengine - } TAOS_BIND; - ``` - -- `int taos_stmt_add_batch(TAOS_STMT *stmt)` - - Add bound parameters to batch, client can call `taos_stmt_bind_param` again after calling this API. Note this API only support _insert_ / _import_ statements, it returns an error in other cases. - -- `int taos_stmt_execute(TAOS_STMT *stmt)` - - Execute the prepared statement. This API can only be called once for a statement at present. - -- `TAOS_RES* taos_stmt_use_result(TAOS_STMT *stmt)` - - Acquire the result set of an executed statement. The usage of the result is same as `taos_use_result`, `taos_free_result` must be called after one you are done with the result set to release resources. - -- `int taos_stmt_close(TAOS_STMT *stmt)` - - Close the statement, release all resources. - - -### C/C++ continuous query interface - -TDengine provides APIs for continuous query driven by time, which run queries periodically in the background. There are only two APIs: - - -- `TAOS_STREAM *taos_open_stream(TAOS *taos, const char *sqlstr, void (*fp)(void *param, TAOS_RES * res, TAOS_ROW row), int64_t stime, void *param, void (*callback)(void *));` - - The API is used to create a continuous query. - * _taos_: the connection handle returned by _taos_connect_. - * _sqlstr_: the SQL string to run. Only query commands are allowed. - * _fp_: the callback function to run after a query. TDengine passes query result `row`, query state `res` and user provided parameter `param` to this function. In this callback, `taos_num_fields` and `taos_fetch_fields` could be used to fetch field information. - * _param_: a parameter passed to _fp_ - * _stime_: the time of the stream starts in the form of epoch milliseconds. If _0_ is given, the start time is set as the current time. - * _callback_: a callback function to run when the continuous query stops automatically. - - The API is expected to return a handle for success. Otherwise, a NULL pointer is returned. - - -- `void taos_close_stream (TAOS_STREAM *tstr)` - - Close the continuous query by the handle returned by _taos_open_stream_. Make sure to call this API when the continuous query is not needed anymore. - - -### C/C++ subscription API - -For the time being, TDengine supports subscription on one or multiple tables. It is implemented through periodic pulling from a TDengine server. - -* `TAOS_SUB *taos_subscribe(TAOS* taos, int restart, const char* topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp, void *param, int interval)` - - The API is used to start a subscription session, it returns the subscription object on success and `NULL` in case of failure, the parameters are: - * **taos**: The database connnection, which must be established already. - * **restart**: `Zero` to continue a subscription if it already exits, other value to start from the beginning. - * **topic**: The unique identifier of a subscription. - * **sql**: A sql statement for data query, it can only be a `select` statement, can only query for raw data, and can only query data in ascending order of the timestamp field. - * **fp**: A callback function to receive query result, only used in asynchronization mode and should be `NULL` in synchronization mode, please refer below for its prototype. - * **param**: User provided additional parameter for the callback function. - * **interval**: Pulling interval in millisecond. Under asynchronization mode, API will call the callback function `fp` in this interval, system performance will be impacted if this interval is too short. Under synchronization mode, if the duration between two call to `taos_consume` is less than this interval, the second call blocks until the duration exceed this interval. - -* `typedef void (*TAOS_SUBSCRIBE_CALLBACK)(TAOS_SUB* tsub, TAOS_RES *res, void* param, int code)` - - Prototype of the callback function, the parameters are: - * tsub: The subscription object. - * res: The query result. - * param: User provided additional parameter when calling `taos_subscribe`. - * code: Error code in case of failures. - -* `TAOS_RES *taos_consume(TAOS_SUB *tsub)` - - The API used to get the new data from a TDengine server. It should be put in an loop. The parameter `tsub` is the handle returned by `taos_subscribe`. This API should only be called in synchronization mode. If the duration between two call to `taos_consume` is less than pulling interval, the second call blocks until the duration exceed the interval. The API returns the new rows if new data arrives, or empty rowset otherwise, and if there's an error, it returns `NULL`. - -* `void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress)` - - Stop a subscription session by the handle returned by `taos_subscribe`. If `keepProgress` is **not** zero, the subscription progress information is kept and can be reused in later call to `taos_subscribe`, the information is removed otherwise. - - -## Java Connector - -To Java delevopers, TDengine provides `taos-jdbcdriver` according to the JDBC(3.0) API. Users can find and download it through [Sonatype Repository][1]. - -Since the native language of TDengine is C, the necessary TDengine library should be checked before using the taos-jdbcdriver: - -* libtaos.so (Linux) - After TDengine is installed successfully, the library `libtaos.so` will be automatically copied to the `/usr/lib/`, which is the system's default search path. - -* taos.dll (Windows) - After TDengine client is installed, the library `taos.dll` will be automatically copied to the `C:/Windows/System32`, which is the system's default search path. - -> Note: Please make sure that [TDengine Windows client][14] has been installed if developing on Windows. Now although TDengine client would be defaultly installed together with TDengine server, it can also be installed [alone][15]. - -Since TDengine is time-series database, there are still some differences compared with traditional databases in using TDengine JDBC driver: -* TDengine doesn't allow to delete/modify a single record, and thus JDBC driver also has no such method. -* No support for transaction -* No support for union between tables -* No support for nested query，`There is at most one open ResultSet for each Connection. Thus, TSDB JDBC Driver will close current ResultSet if it is not closed and a new query begins`. - -## Version list of TAOS-JDBCDriver and required TDengine and JDK - -| taos-jdbcdriver | TDengine | JDK | -| --- | --- | --- | -| 2.0.2 | 2.0.0.x or higher | 1.8.x | -| 1.0.3 | 1.6.1.x or higher | 1.8.x | -| 1.0.2 | 1.6.1.x or higher | 1.8.x | -| 1.0.1 | 1.6.1.x or higher | 1.8.x | - -## DataType in TDengine and Java - -The datatypes in TDengine include timestamp, number, string and boolean, which are converted as follows in Java: - -| TDengine | Java | -| --- | --- | -| TIMESTAMP | java.sql.Timestamp | -| INT | java.lang.Integer | -| BIGINT | java.lang.Long | -| FLOAT | java.lang.Float | -| DOUBLE | java.lang.Double | -| SMALLINT, TINYINT |java.lang.Short | -| BOOL | java.lang.Boolean | -| BINARY, NCHAR | java.lang.String | - -## How to get TAOS-JDBC Driver - -### maven repository - -taos-jdbcdriver has been published to [Sonatype Repository][1]: -* [sonatype][8] -* [mvnrepository][9] -* [maven.aliyun][10] - -Using the following pom.xml for maven projects - -```xml - - - com.taosdata.jdbc - taos-jdbcdriver - 2.0.2 - - -``` - -### JAR file from the source code - -After downloading the [TDengine][3] source code, execute `mvn clean package` in the directory `src/connector/jdbc` and then the corresponding jar file is generated. - -## Usage - -### get the connection - -```java -Class.forName("com.taosdata.jdbc.TSDBDriver"); -String jdbcUrl = "jdbc:TAOS://127.0.0.1:6030/log?user=root&password=taosdata"; -Connection conn = DriverManager.getConnection(jdbcUrl); -``` -> `6030` is the default port and `log` is the default database for system monitor. - -A normal JDBC URL looks as follows: -`jdbc:TAOS://{host_ip}:{port}/[database_name]?[user={user}|&password={password}|&charset={charset}|&cfgdir={config_dir}|&locale={locale}|&timezone={timezone}]` - -values in `{}` are necessary while values in `[]` are optional。Each option in the above URL denotes: - -* user：user name for login, defaultly root。 -* password：password for login，defaultly taosdata。 -* charset：charset for client，defaultly system charset -* cfgdir：log directory for client, defaultly _/etc/taos/_ on Linux and _C:/TDengine/cfg_ on Windows。 -* locale：language for client，defaultly system locale。 -* timezone：timezone for client，defaultly system timezone。 - -The options above can be configures (`ordered by priority`): -1. JDBC URL - - As explained above. -2. java.sql.DriverManager.getConnection(String jdbcUrl, Properties connProps) -```java -public Connection getConn() throws Exception{ - Class.forName("com.taosdata.jdbc.TSDBDriver"); - String jdbcUrl = "jdbc:TAOS://127.0.0.1:0/log?user=root&password=taosdata"; - Properties connProps = new Properties(); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_USER, "root"); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_PASSWORD, "taosdata"); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_CONFIG_DIR, "/etc/taos"); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8"); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_LOCALE, "en_US.UTF-8"); - connProps.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8"); - Connection conn = DriverManager.getConnection(jdbcUrl, connProps); - return conn; -} -``` - -3. Configuration file (taos.cfg) - - Default configuration file is _/var/lib/taos/taos.cfg_ On Linux and _C:\TDengine\cfg\taos.cfg_ on Windows -```properties -# client default username -# defaultUser root - -# client default password -# defaultPass taosdata - -# default system charset -# charset UTF-8 - -# system locale -# locale en_US.UTF-8 -``` -> More options can refer to [client configuration][13] - -### Create databases and tables - -```java -Statement stmt = conn.createStatement(); - -// create database -stmt.executeUpdate("create database if not exists db"); - -// use database -stmt.executeUpdate("use db"); - -// create table -stmt.executeUpdate("create table if not exists tb (ts timestamp, temperature int, humidity float)"); -``` -> Note: if no step like `use db`, the name of database must be added as prefix like _db.tb_ when operating on tables - -### Insert data - -```java -// insert data -int affectedRows = stmt.executeUpdate("insert into tb values(now, 23, 10.3) (now + 1s, 20, 9.3)"); - -System.out.println("insert " + affectedRows + " rows."); -``` -> _now_ is the server time. -> _now+1s_ is 1 second later than current server time. The time unit includes: _a_(millisecond), _s_(second), _m_(minute), _h_(hour), _d_(day), _w_(week), _n_(month), _y_(year). - -### Query database - -```java -// query data -ResultSet resultSet = stmt.executeQuery("select * from tb"); - -Timestamp ts = null; -int temperature = 0; -float humidity = 0; -while(resultSet.next()){ - - ts = resultSet.getTimestamp(1); - temperature = resultSet.getInt(2); - humidity = resultSet.getFloat("humidity"); - - System.out.printf("%s, %d, %s\n", ts, temperature, humidity); -} -``` -> query is consistent with relational database. The subscript start with 1 when retrieving return results. It is recommended to use the column name to retrieve results. - -### Close all - -```java -resultSet.close(); -stmt.close(); -conn.close(); -``` -> `please make sure the connection is closed to avoid the error like connection leakage` - -## Using connection pool - -**HikariCP** - -* dependence in pom.xml： -```xml - - com.zaxxer - HikariCP - 3.4.1 - -``` - -* Examples： -```java - public static void main(String[] args) throws SQLException { - HikariConfig config = new HikariConfig(); - config.setJdbcUrl("jdbc:TAOS://127.0.0.1:6030/log"); - config.setUsername("root"); - config.setPassword("taosdata"); - - config.setMinimumIdle(3); //minimum number of idle connection - config.setMaximumPoolSize(10); //maximum number of connection in the pool - config.setConnectionTimeout(10000); //maximum wait milliseconds for get connection from pool - config.setIdleTimeout(60000); // max idle time for recycle idle connection - config.setConnectionTestQuery("describe log.dn"); //validation query - config.setValidationTimeout(3000); //validation query timeout - - HikariDataSource ds = new HikariDataSource(config); //create datasource - - Connection connection = ds.getConnection(); // get connection - Statement statement = connection.createStatement(); // get statement - - //query or insert - // ... - - connection.close(); // put back to conneciton pool -} -``` -> The close() method will not close the connection from HikariDataSource.getConnection(). Instead, the connection is put back to the connection pool. -> More instructions can refer to [User Guide][5] - -**Druid** - -* dependency in pom.xml： - -```xml - - com.alibaba - druid - 1.1.20 - -``` - -* Examples： -```java -public static void main(String[] args) throws Exception { - Properties properties = new Properties(); - properties.put("driverClassName","com.taosdata.jdbc.TSDBDriver"); - properties.put("url","jdbc:TAOS://127.0.0.1:6030/log"); - properties.put("username","root"); - properties.put("password","taosdata"); - - properties.put("maxActive","10"); //maximum number of connection in the pool - properties.put("initialSize","3");//initial number of connection - properties.put("maxWait","10000");//maximum wait milliseconds for get connection from pool - properties.put("minIdle","3");//minimum number of connection in the pool - - properties.put("timeBetweenEvictionRunsMillis","3000");// the interval milliseconds to test connection - - properties.put("minEvictableIdleTimeMillis","60000");//the minimum milliseconds to keep idle - properties.put("maxEvictableIdleTimeMillis","90000");//the maximum milliseconds to keep idle - - properties.put("validationQuery","describe log.dn"); //validation query - properties.put("testWhileIdle","true"); // test connection while idle - properties.put("testOnBorrow","false"); // don't need while testWhileIdle is true - properties.put("testOnReturn","false"); // don't need while testWhileIdle is true - - //create druid datasource - DataSource ds = DruidDataSourceFactory.createDataSource(properties); - Connection connection = ds.getConnection(); // get connection - Statement statement = connection.createStatement(); // get statement - - //query or insert - // ... - - connection.close(); // put back to conneciton pool -} -``` -> More instructions can refer to [User Guide][6] - -**Notice** -* TDengine `v1.6.4.1` provides a function `select server_status()` to check heartbeat. It is highly recommended to use this function for `Validation Query`. - -As follows，`1` will be returned if `select server_status()` is successfully executed。 -```shell -taos> select server_status(); -server_status()| -================ -1 | -Query OK, 1 row(s) in set (0.000141s) -``` - -## Python Connector - -### Install TDengine Python client - -Users can find python client packages in our source code directory _src/connector/python_. There are two directories corresponding two python versions. Please choose the correct package to install. Users can use _pip_ command to install: - -```cmd -pip install src/connector/python/python2/ -``` - -or - -``` -pip install src/connector/python/python3/ -``` - -If _pip_ command is not installed on the system, users can choose to install pip or just copy the _taos_ directory in the python client directory to the application directory to use. - -### Python client interfaces - -To use TDengine Python client, import TDengine module at first: - -```python -import taos -``` - -Users can get module information from Python help interface or refer to our [python code example](). We list the main classes and methods below: - -- _TDengineConnection_ class - - Run `help(taos.TDengineConnection)` in python terminal for details. - -- _TDengineCursor_ class - - Run `help(taos.TDengineCursor)` in python terminal for details. - -- connect method - - Open a connection. Run `help(taos.connect)` in python terminal for details. - -## RESTful Connector - -TDengine also provides RESTful API to satisfy developing on different platforms. Unlike other databases, TDengine RESTful API applies operations to the database through the SQL command in the body of HTTP POST request. What users are required to provide is just a URL. - - -For the time being, TDengine RESTful API uses a _\_ generated from username and password for identification. Safer identification methods will be provided in the future. - - -### HTTP URL encoding - -To use TDengine RESTful API, the URL should have the following encoding format: -``` -http://:/rest/sql -``` -- _ip_: IP address of any node in a TDengine cluster -- _PORT_: TDengine HTTP service port. It is 6020 by default. - -For example, the URL encoding _http://192.168.0.1:6020/rest/sql_ used to send HTTP request to a TDengine server with IP address as 192.168.0.1. - -It is required to add a token in an HTTP request header for identification. - -``` -Authorization: Basic -``` - -The HTTP request body contains the SQL command to run. If the SQL command contains a table name, it should also provide the database name it belongs to in the form of `.`. Otherwise, an error code is returned. - -For example, use _curl_ command to send a HTTP request: - -``` -curl -H 'Authorization: Basic ' -d '' :/rest/sql -``` - -or use - -``` -curl -u username:password -d '' :/rest/sql -``` - -where `TOKEN` is the encryted string of `{username}:{password}` using the Base64 algorithm, e.g. `root:taosdata` will be encoded as `cm9vdDp0YW9zZGF0YQ==` - -### HTTP response - -The HTTP resonse is in JSON format as below: - -``` -{ - "status": "succ", - "head": ["column1","column2", …], - "data": [ - ["2017-12-12 23:44:25.730", 1], - ["2017-12-12 22:44:25.728", 4] - ], - "rows": 2 -} -``` -Specifically, -- _status_: the result of the operation, success or failure -- _head_: description of returned result columns -- _data_: the returned data array. If no data is returned, only an _affected_rows_ field is listed -- _rows_: the number of rows returned - -### Example - -- Use _curl_ command to query all the data in table _t1_ of database _demo_: - - `curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'select * from demo.t1' 192.168.0.1:6020/rest/sql` - -The return value is like: - -``` -{ - "status": "succ", - "head": ["column1","column2","column3"], - "data": [ - ["2017-12-12 23:44:25.730", 1, 2.3], - ["2017-12-12 22:44:25.728", 4, 5.6] - ], - "rows": 2 -} -``` - -- Use HTTP to create a database： - - `curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'create database demo' 192.168.0.1:6020/rest/sql` - - The return value should be: - -``` -{ - "status": "succ", - "head": ["affected_rows"], - "data": [[1]], - "rows": 1, -} -``` - -## Go Connector - -TDengine provides a GO client package `taosSql`. `taosSql` implements a kind of interface of GO `database/sql/driver`. User can access TDengine by importing the package in their program with the following instructions, detailed usage please refer to `https://github.com/taosdata/driver-go/blob/develop/taosSql/driver_test.go` - -```Go -import ( - "database/sql" - _ github.com/taosdata/driver-go/taoSql“ -) -``` -### API - -* `sql.Open(DRIVER_NAME string, dataSourceName string) *DB` - - Open DB, generally DRIVER_NAME will be used as a constant with default value `taosSql`, dataSourceName is a combined String with format `user:password@/tcp(host:port)/dbname`. If user wants to access TDengine with multiple goroutine concurrently, the better way is to create an sql.Open object in each goroutine to access TDengine. - - **Note**: When calling this api, only a few initial work are done, instead the validity check happened during executing `Query` or `Exec`, at this time the connection will be created, and system will check if `user、password、host、port` is valid. Additionaly the most of features are implemented in the taosSql dependency lib `libtaos`, from this view, sql.Open is lightweight. - -* `func (db *DB) Exec(query string, args ...interface{}) (Result, error)` - - Execute non-Query related SQLs, the execution result is stored with type of Result. - - -* `func (db *DB) Query(query string, args ...interface{}) (*Rows, error)` - - Execute Query related SQLs, the execution result is *Raw, the detailed usage can refer GO interface `database/sql/driver` - -## Node.js Connector - -TDengine also provides a node.js connector package that is installable through [npm](https://www.npmjs.com/). The package is also in our source code at *src/connector/nodejs/*. The following instructions are also available [here](https://github.com/taosdata/tdengine/tree/master/src/connector/nodejs) - -To get started, just type in the following to install the connector through [npm](https://www.npmjs.com/). - -```cmd -npm install td-connector -``` - -It is highly suggested you use npm. If you don't have it installed, you can also just copy the nodejs folder from *src/connector/nodejs/* into your node project folder. - -To interact with TDengine, we make use of the [node-gyp](https://github.com/nodejs/node-gyp) library. To install, you will need to install the following depending on platform (the following instructions are quoted from node-gyp) - -### On Unix - -- `python` (`v2.7` recommended, `v3.x.x` is **not** supported) -- `make` -- A proper C/C++ compiler toolchain, like [GCC](https://gcc.gnu.org) - -### On macOS - -- `python` (`v2.7` recommended, `v3.x.x` is **not** supported) (already installed on macOS) - -- Xcode - - - You also need to install the - - ``` - Command Line Tools - ``` - - via Xcode. You can find this under the menu - - ``` - Xcode -> Preferences -> Locations - ``` - - (or by running - - ``` - xcode-select --install - ``` - - in your Terminal) - - - This step will install `gcc` and the related toolchain containing `make` - -### On Windows - -#### Option 1 - -Install all the required tools and configurations using Microsoft's [windows-build-tools](https://github.com/felixrieseberg/windows-build-tools) using `npm install --global --production windows-build-tools` from an elevated PowerShell or CMD.exe (run as Administrator). - -#### Option 2 - -Install tools and configuration manually: - -- Install Visual C++ Build Environment: [Visual Studio Build Tools](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools) (using "Visual C++ build tools" workload) or [Visual Studio 2017 Community](https://visualstudio.microsoft.com/pl/thank-you-downloading-visual-studio/?sku=Community) (using the "Desktop development with C++" workload) -- Install [Python 2.7](https://www.python.org/downloads/) (`v3.x.x` is not supported), and run `npm config set python python2.7` (or see below for further instructions on specifying the proper Python version and path.) -- Launch cmd, `npm config set msvs_version 2017` - -If the above steps didn't work for you, please visit [Microsoft's Node.js Guidelines for Windows](https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#compiling-native-addon-modules) for additional tips. - -To target native ARM64 Node.js on Windows 10 on ARM, add the components "Visual C++ compilers and libraries for ARM64" and "Visual C++ ATL for ARM64". - -### Usage - -The following is a short summary of the basic usage of the connector, the full api and documentation can be found [here](http://docs.taosdata.com/node) - -#### Connection - -To use the connector, first require the library ```td-connector```. Running the function ```taos.connect``` with the connection options passed in as an object will return a TDengine connection object. The required connection option is ```host```, other options if not set, will be the default values as shown below. - -A cursor also needs to be initialized in order to interact with TDengine from Node.js. - -```javascript -const taos = require('td-connector'); -var conn = taos.connect({host:"127.0.0.1", user:"root", password:"taosdata", config:"/etc/taos",port:0}) -var cursor = conn.cursor(); // Initializing a new cursor -``` - -To close a connection, run - -```javascript -conn.close(); -``` - -#### Queries - -We can now start executing simple queries through the ```cursor.query``` function, which returns a TaosQuery object. - -```javascript -var query = cursor.query('show databases;') -``` - -We can get the results of the queries through the ```query.execute()``` function, which returns a promise that resolves with a TaosResult object, which contains the raw data and additional functionalities such as pretty printing the results. - -```javascript -var promise = query.execute(); -promise.then(function(result) { - result.pretty(); //logs the results to the console as if you were in the taos shell -}); -``` - -You can also query by binding parameters to a query by filling in the question marks in a string as so. The query will automatically parse what was binded and convert it to the proper format for use with TDengine - -```javascript -var query = cursor.query('select * from meterinfo.meters where ts <= ? and areaid = ?;').bind(new Date(), 5); -query.execute().then(function(result) { - result.pretty(); -}) -``` - -The TaosQuery object can also be immediately executed upon creation by passing true as the second argument, returning a promise instead of a TaosQuery. - -```javascript -var promise = cursor.query('select * from meterinfo.meters where v1 = 30;', true) -promise.then(function(result) { - result.pretty(); -}) -``` -#### Async functionality - -Async queries can be performed using the same functions such as `cursor.execute`, `cursor.query`, but now with `_a` appended to them. - -Say you want to execute an two async query on two seperate tables, using `cursor.query_a`, you can do that and get a TaosQuery object, which upon executing with the `execute_a` function, returns a promise that resolves with a TaosResult object. - -```javascript -var promise1 = cursor.query_a('select count(*), avg(v1), avg(v2) from meter1;').execute_a() -var promise2 = cursor.query_a('select count(*), avg(v1), avg(v2) from meter2;').execute_a(); -promise1.then(function(result) { - result.pretty(); -}) -promise2.then(function(result) { - result.pretty(); -}) -``` - - -### Example - -An example of using the NodeJS connector to create a table with weather data and create and execute queries can be found [here](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example.js) (The preferred method for using the connector) - -An example of using the NodeJS connector to achieve the same things but without all the object wrappers that wrap around the data returned to achieve higher functionality can be found [here](https://github.com/taosdata/TDengine/tree/master/tests/examples/nodejs/node-example-raw.js) - -[1]: https://search.maven.org/artifact/com.taosdata.jdbc/taos-jdbcdriver -[2]: https://mvnrepository.com/artifact/com.taosdata.jdbc/taos-jdbcdriver -[3]: https://github.com/taosdata/TDengine -[4]: https://www.taosdata.com/blog/2019/12/03/jdbcdriver%e6%89%be%e4%b8%8d%e5%88%b0%e5%8a%a8%e6%80%81%e9%93%be%e6%8e%a5%e5%ba%93/ -[5]: https://github.com/brettwooldridge/HikariCP -[6]: https://github.com/alibaba/druid -[7]: https://github.com/taosdata/TDengine/issues -[8]: https://search.maven.org/artifact/com.taosdata.jdbc/taos-jdbcdriver -[9]: https://mvnrepository.com/artifact/com.taosdata.jdbc/taos-jdbcdriver -[10]: https://maven.aliyun.com/mvn/search -[11]: https://github.com/taosdata/TDengine/tree/develop/tests/examples/JDBC/SpringJdbcTemplate -[12]: https://github.com/taosdata/TDengine/tree/develop/tests/examples/JDBC/springbootdemo -[13]: https://www.taosdata.com/cn/documentation20/administrator/#%E5%AE%A2%E6%88%B7%E7%AB%AF%E9%85%8D%E7%BD%AE -[14]: https://www.taosdata.com/cn/documentation20/connector/#Windows -[15]: https://www.taosdata.com/cn/getting-started/#%E5%BF%AB%E9%80%9F%E4%B8%8A%E6%89%8B \ No newline at end of file diff --git a/documentation20/webdocs/markdowndocs/Contributor_License_Agreement.md b/documentation20/webdocs/markdowndocs/Contributor_License_Agreement.md deleted file mode 100644 index 8c158da4c5958384064b9993de6643be86b94fee..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Contributor_License_Agreement.md +++ /dev/null @@ -1,35 +0,0 @@ -# TaosData Contributor License Agreement - -This TaosData Contributor License Agreement (CLA) applies to any contribution you make to any TaosData projects. If you are representing your employing organization to sign this agreement, please warrant that you have the authority to grant the agreement. - -## Terms - -**"TaosData"**, **"we"**, **"our"** and **"us"** means TaosData, inc. - -**"You"** and **"your"** means you or the organization you are on behalf of to sign this agreement. - -**"Contribution"** means any original work you, or the organization you represent submit to TaosData for any project in any manner. - -## Copyright License - -All rights of your Contribution submitted to TaosData in any manner are granted to TaosData and recipients of software distributed by TaosData. You waive any rights that my affect our ownership of the copyright and grant to us a perpetual, worldwide, transferable, non-exclusive, no-charge, royalty-free, irrevocable, and sublicensable license to use, reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Contributions and any derivative work created based on a Contribution. - -## Patent License - -With respect to any patents you own or that you can license without payment to any third party, you grant to us and to any recipient of software distributed by us, a perpetual, worldwide, transferable, non-exclusive, no-charge, royalty-free, irrevocable patent license to make, have make, use, sell, offer to sell, import, and otherwise transfer the Contribution in whole or in part, alone or included in any product under any patent you own, or license from a third party, that is necessarily infringed by the Contribution or by combination of the Contribution with any Work. - -## Your Representations and Warranties - -You represent and warrant that: - -- the Contribution you submit is an original work that you can legally grant the rights set out in this agreement. - -- the Contribution you submit and licenses you granted does not and will not, infringe the rights of any third party. - -- you are not aware of any pending or threatened claims, suits, actions, or charges pertaining to the contributions. You also warrant to notify TaosData immediately if you become aware of any such actual or potential claims, suits, actions, allegations or charges. - -## Support - -You are not obligated to support your Contribution except you volunteer to provide support. If you want, you can provide for a fee. - -**I agree and accept on behalf of myself and behalf of my organization:** \ No newline at end of file diff --git a/documentation20/webdocs/markdowndocs/Documentation.md b/documentation20/webdocs/markdowndocs/Documentation.md deleted file mode 100644 index bdafd40f7c76425a4f9734a2561b2b9a945c757f..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Documentation.md +++ /dev/null @@ -1,87 +0,0 @@ -#Documentation - -TDengine is a highly efficient platform to store, query, and analyze time-series data. It works like a relational database, but you are strongly suggested to read through the following documentation before you experience it. - -##Getting Started - -- Quick Start: download, install and experience TDengine in a few seconds -- TDengine Shell: command-line interface to access TDengine server -- Major Features: insert/query, aggregation, cache, pub/sub, continuous query - -## Data Model and Architecture - -- Data Model: relational database model, but one table for one device with static tags -- Architecture: Management Module, Data Module, Client Module -- Writing Process: records recieved are written to WAL, cache, then ack is sent back to client -- Data Storage: records are sharded in the time range, and stored column by column - -##TAOS SQL - -- Data Types: support timestamp, int, float, double, binary, nchar, bool, and other types -- Database Management: add, drop, check databases -- Table Management: add, drop, check, alter tables -- Inserting Records: insert one or more records into tables, historical records can be imported -- Data Query: query data with time range and filter conditions, support limit/offset -- SQL Functions: support aggregation, selector, transformation functions -- Downsampling: aggregate data in successive time windows, support interpolation - -##STable: Super Table - -- What is a Super Table: an innovated way to aggregate tables -- Create a STable: it is like creating a standard table, but with tags defined -- Create a Table via STable: use STable as the template, with tags specified -- Aggregate Tables via STable: group tables together by specifying the tags filter condition -- Create Table Automatically: create tables automatically with a STable as a template -- Management of STables: create/delete/alter super table just like standard tables -- Management of Tags: add/delete/alter tags on super tables or tables - -##Advanced Features - -- Continuous Query: query executed by TDengine periodically with a sliding window -- Publisher/Subscriber: subscribe to the newly arrived data like a typical messaging system -- Caching: the newly arrived data of each device/table will always be cached - -##Connector - -- C/C++ Connector: primary method to connect to the server through libtaos client library -- Java Connector: driver for connecting to the server from Java applications using the JDBC API -- Python Connector: driver for connecting to the server from Python applications -- RESTful Connector: a simple way to interact with TDengine via HTTP -- Go Connector: driver for connecting to the server from Go applications -- Node.js Connector: driver for connecting to the server from node applications - -##Connections with Other Tools - -- Telegraf: pass the collected DevOps metrics to TDengine -- Grafana: query the data saved in TDengine and visualize them -- Matlab: access TDengine server from Matlab via JDBC -- R: access TDengine server from R via JDBC - -##Administrator - -- Directory and Files: files and directories related with TDengine -- Configuration on Server: customize IP port, cache size, file block size and other settings -- Configuration on Client: customize locale, default user and others -- User Management: add/delete users, change passwords -- Import Data: import data into TDengine from either script or CSV file -- Export Data: export data either from TDengine shell or from tool taosdump -- Management of Connections, Streams, Queries: check or kill the connections, queries -- System Monitor: collect the system metric, and log important operations - -##More on System Architecture - -- Storage Design: column-based storage with optimization on time-series data -- Query Design: an efficient way to query time-series data -- Technical blogs to delve into the inside of TDengine - -## More on IoT Big Data - -- [Characteristics of IoT Big Data](https://www.taosdata.com/blog/2019/07/09/characteristics-of-iot-big-data/) -- [Why don’t General Big Data Platforms Fit IoT Scenarios?](https://www.taosdata.com/blog/2019/07/09/why-does-the-general-big-data-platform-not-fit-iot-data-processing/) -- [Why TDengine is the Best Choice for IoT Big Data Processing?](https://www.taosdata.com/blog/2019/07/09/why-tdengine-is-the-best-choice-for-iot-big-data-processing/) - -##Tutorials & FAQ - -- FAQ: a list of frequently asked questions and answers -- Use cases: a few typical cases to explain how to use TDengine in IoT platform - diff --git a/documentation20/webdocs/markdowndocs/Getting Started.md b/documentation20/webdocs/markdowndocs/Getting Started.md deleted file mode 100644 index 4d34cb49f4a84ac6c9d63e47bc8230c150b9013e..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Getting Started.md +++ /dev/null @@ -1,151 +0,0 @@ -#Getting Started - -## Quick Start - -At the moment, TDengine only runs on Linux. You can set up and install it either from the source code or the packages. It takes only a few seconds from download to run it successfully. - -### Install from Source - -Please visit our [github page](https://github.com/taosdata/TDengine) for instructions on installation from the source code. - -### Install from Package - -Three different packages are provided, please pick up the one you like. - -For the time being, TDengine only supports installation on Linux systems using [`systemd`](https://en.wikipedia.org/wiki/Systemd) as the service manager. To check if your system has *systemd* package, use the _which systemctl_ command. - -```cmd -which systemctl -``` - -If the `systemd` package is not found, please [install from source code](#Install-from-Source). - -### Running TDengine - -After installation, start the TDengine service by the `systemctl` command. - -```cmd -systemctl start taosd -``` - -Then check if the server is working now. -```cmd -systemctl status taosd -``` - -If the service is running successfully, you can play around through TDengine shell `taos`, the command line interface tool located in directory /usr/local/bin/taos - -**Note: The _systemctl_ command needs the root privilege. Use _sudo_ if you are not the _root_ user.** - -##TDengine Shell -To launch TDengine shell, the command line interface, in a Linux terminal, type: - -```cmd -taos -``` - -The welcome message is printed if the shell connects to TDengine server successfully, otherwise, an error message will be printed (refer to our [FAQ](../faq) page for troubleshooting the connection error). The TDengine shell prompt is: - -```cmd -taos> -``` - -In the TDengine shell, you can create databases, create tables and insert/query data with SQL. Each query command ends with a semicolon. It works like MySQL, for example: - -```mysql -create database db; -use db; -create table t (ts timestamp, cdata int); -insert into t values ('2019-07-15 10:00:00', 10); -insert into t values ('2019-07-15 10:01:05', 20); -select * from t; - ts | speed | -=================================== - 19-07-15 10:00:00.000| 10| - 19-07-15 10:01:05.000| 20| -Query OK, 2 row(s) in set (0.001700s) -``` - -Besides the SQL commands, the system administrator can check system status, add or delete accounts, and manage the servers. - -###Shell Command Line Parameters - -You can run `taos` command with command line options to fit your needs. Some frequently used options are listed below: - -- -c, --config-dir: set the configuration directory. It is _/etc/taos_ by default -- -h, --host: set the IP address of the server it will connect to, Default is localhost -- -s, --commands: set the command to run without entering the shell -- -u, -- user: user name to connect to server. Default is root -- -p, --password: password. Default is 'taosdata' -- -?, --help: get a full list of supported options - -Examples: - -```cmd -taos -h 192.168.0.1 -s "use db; show tables;" -``` - -###Run Batch Commands - -Inside TDengine shell, you can run batch commands in a file with *source* command. - -``` -taos> source ; -``` - -### Tips - -- Use up/down arrow key to check the command history -- To change the default password, use "`alter user`" command -- ctrl+c to interrupt any queries -- To clean the cached schema of tables or STables, execute command `RESET QUERY CACHE` - -## Major Features - -The core functionality of TDengine is the time-series database. To reduce the development and management complexity, and to improve the system efficiency further, TDengine also provides caching, pub/sub messaging system, and stream computing functionalities. It provides a full stack for IoT big data platform. The detailed features are listed below: - -- SQL like query language used to insert or explore data - -- C/C++, Java(JDBC), Python, Go, RESTful, and Node.JS interfaces for development - -- Ad hoc queries/analysis via Python/R/Matlab or TDengine shell - -- Continuous queries to support sliding-window based stream computing - -- Super table to aggregate multiple time-streams efficiently with flexibility - -- Aggregation over a time window on one or multiple time-streams - -- Built-in messaging system to support publisher/subscriber model - -- Built-in cache for each time stream to make latest data available as fast as light speed - -- Transparent handling of historical data and real-time data - -- Integrating with Telegraf, Grafana and other tools seamlessly - -- A set of tools or configuration to manage TDengine - - -For enterprise edition, TDengine provides more advanced features below: - -- Linear scalability to deliver higher capacity/throughput - -- High availability to guarantee the carrier-grade service - -- Built-in replication between nodes which may span multiple geographical sites - -- Multi-tier storage to make historical data management simpler and cost-effective - -- Web-based management tools and other tools to make maintenance simpler - -TDengine is specially designed and optimized for time-series data processing in IoT, connected cars, Industrial IoT, IT infrastructure and application monitoring, and other scenarios. Compared with other solutions, it is 10x faster on insert/query speed. With a single-core machine, over 20K requestes can be processed, millions data points can be ingested, and over 10 million data points can be retrieved in a second. Via column-based storage and tuned compression algorithm for different data types, less than 1/10 storage space is required. - -## Explore More on TDengine - -Please read through the whole documentation to learn more about TDengine. - diff --git a/documentation20/webdocs/markdowndocs/More on System Architecture-ch.md b/documentation20/webdocs/markdowndocs/More on System Architecture-ch.md deleted file mode 100644 index 44d572268de04662c190a6a5975c784b38aad117..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/More on System Architecture-ch.md +++ /dev/null @@ -1,248 +0,0 @@ -# TDengine的技术设计 - -## 存储设计 - -TDengine的数据存储主要包含**元数据的存储**和**写入数据的存储**。以下章节详细介绍了TDengine各种数据的存储结构。 - -### 元数据的存储 - -TDengine中的元数据信息包括TDengine中的数据库，表，超级表等信息。元数据信息默认存放在 _/var/lib/taos/mgmt/_ 文件夹下。该文件夹的目录结构如下所示： -``` -/var/lib/taos/ - +--mgmt/ - +--db.db - +--meters.db - +--user.db - +--vgroups.db -``` -元数据在文件中按顺序排列。文件中的每条记录代表TDengine中的一个元数据机构（数据库、表等）。元数据文件只进行追加操作，即便是元数据的删除，也只是在数据文件中追加一条删除的记录。 - -### 写入数据的存储 - -TDengine中写入的数据在硬盘上是按时间维度进行分片的。同一个vnode中的表在同一时间范围内的数据都存放在同一文件组中，如下图中的v0f1804*文件。这一数据分片方式可以大大简化数据在时间维度的查询，提高查询速度。在默认配置下，硬盘上的每个文件存放10天数据。用户可根据需要调整数据库的 _daysPerFile_ 配置项进行配置。 数据在文件中是按块存储的。每个数据块只包含一张表的数据，且数据是按照时间主键递增排列的。数据在数据块中按列存储，这样使得同类型的数据存放在一起，可以大大提高压缩的比例，节省存储空间。TDengine对不同类型的数据采用了不同的压缩算法进行压缩，以达到最优的压缩结果。TDengine使用的压缩算法包括simple8B、delta-of-delta、RLE以及LZ4等。 - -TDengine的数据文件默认存放在 */var/lib/taos/data/* 下。而 */var/lib/taos/tsdb/* 文件夹下存放了vnode的信息、vnode中表的信息以及数据文件的链接等。其完整目录结构如下所示： -``` -/var/lib/taos/ - +--tsdb/ - | +--vnode0 - | +--meterObj.v0 - | +--db/ - | +--v0f1804.head->/var/lib/taos/data/vnode0/v0f1804.head1 - | +--v0f1804.data->/var/lib/taos/data/vnode0/v0f1804.data - | +--v0f1804.last->/var/lib/taos/data/vnode0/v0f1804.last1 - | +--v0f1805.head->/var/lib/taos/data/vnode0/v0f1805.head1 - | +--v0f1805.data->/var/lib/taos/data/vnode0/v0f1805.data - | +--v0f1805.last->/var/lib/taos/data/vnode0/v0f1805.last1 - | : - +--data/ - +--vnode0/ - +--v0f1804.head1 - +--v0f1804.data - +--v0f1804.last1 - +--v0f1805.head1 - +--v0f1805.data - +--v0f1805.last1 - : -``` - -#### meterObj文件 -每个vnode中只存在一个 _meterObj_ 文件。该文件中存储了vnode的基本信息（创建时间，配置信息，vnode的统计信息等）以及该vnode中表的信息。其结构如下所示： -``` -<文件开始> -[文件头] -[表记录1偏移量和长度] -[表记录2偏移量和长度] -... -[表记录N偏移量和长度] -[表记录1] -[表记录2] -... -[表记录N] -[表记录] -<文件结尾> -``` -其中，文件头大小为512字节，主要存放vnode的基本信息。每条表记录代表属于该vnode中的一张表在硬盘上的表示。 - -#### head文件 -head文件中存放了其对应的data文件中数据块的索引信息。该文件组织形式如下： -``` -<文件开始> -[文件头] -[表1偏移量] -[表2偏移量] -... -[表N偏移量] -[表1数据索引] -[表2数据索引] -... -[表N数据索引] -<文件结尾> -``` -文件开头的偏移量列表表示对应表的数据索引块的开始位置在文件中的偏移量。每张表的数据索引信息在head文件中都是连续存放的。这也使得TDengine在读取单表数据时，可以将该表所有的数据块索引一次性读入内存，大大提高读取速度。表的数据索引块组织如下： -``` -[索引块信息] -[数据块1索引] -[数据块2索引] -... -[数据块N索引] -``` -其中，索引块信息中记录了数据块的个数等描述信息。每个数据块索引对应一个在data文件或last文件中的一个单独的数据块。索引信息中记录了数据块存放的文件、数据块起始位置的偏移量、数据块中数据时间主键的范围等。索引块中的数据块索引是按照时间范围顺序排放的，这也就是说，索引块M对应的数据块中的数据时间范围都大于索引块M-1的。这种预先排序的存储方式使得在TDengine在进行按照时间戳进行查询时可以使用折半查找算法，大大提高查询速度。 - -#### data文件 -data文件中存放了真实的数据块。该文件只进行追加操作。其文件组织形式如下： -``` -<文件开始> -[文件头] -[数据块1] -[数据块2] -... -[数据块N] -<文件结尾> -``` -每个数据块只属于vnode中的一张表，且数据块中的数据按照时间主键排列。数据块中的数据按列组织排放，使得同一类型的数据排放在一起，方便压缩和读取。每个数据块的组织形式如下所示： -``` -[列1信息] -[列2信息] -... -[列N信息] -[列1数据] -[列2数据] -... -[列N数据] -``` -列信息中包含该列的类型，列的压缩算法，列数据在文件中的偏移量以及长度等。除此之外，列信息中也包含该内存块中该列数据的预计算结果，从而在过滤查询时根据预计算结果判定是否读取数据块，大大提高读取速度。 - -#### last文件 -为了防止数据块的碎片化，提高查询速度和压缩率，TDengine引入了last文件。当要落盘的数据块中的数据条数低于某个阈值时，TDengine会先将该数据块写入到last文件中进行暂时存储。当有新的数据需要落盘时，last文件中的数据会被读取出来与新数据组成新的数据块写入到data文件中。last文件的组织形式与data文件类似。 - -### TDengine数据存储小结 -TDengine通过其创新的架构和存储结构设计，有效提高了计算机资源的使用率。一方面，TDengine的虚拟化使得TDengine的水平扩展及备份非常容易。另一方面，TDengine将表中数据按时间主键排序存储且其列式存储的组织形式都使TDengine在写入、查询以及压缩方面拥有非常大的优势。 - - -## 查询处理 - -### 概述 - -TDengine提供了多种多样针对表和超级表的查询处理功能，除了常规的聚合查询之外，还提供针对时序数据的窗口查询、统计聚合等功能。TDengine的查询处理需要客户端、管理节点、数据节点协同完成。 各组件包含的与查询处理相关的功能和模块如下： - -客户端（Client App）。客户端包含TAOS SQL的解析（SQL Parser）和查询请求执行器（Query Executor），第二阶段聚合器（Result Merger），连续查询管理器（Continuous Query Manager）等主要功能模块构成。SQL解析器负责对SQL语句进行解析校验，并转化为抽象语法树，查询执行器负责将抽象语法树转化查询执行逻辑，并根据SQL语句查询条件，将其转换为针对管理节点元数据查询和针对数据节点的数据查询两级查询处理。由于TAOS SQL当前不提供复杂的嵌套查询和pipeline查询处理机制，所以不再需要查询计划优化、逻辑查询计划到物理查询计划转换等过程。第二阶段聚合器负责将各数据节点查询返回的独立结果进行二阶段聚合生成最后的结果。连续查询管理器则负责针对用户建立的连续查询进行管理，负责定时拉起查询请求并按需将结果写回TDengine或返回给客户应用。此外，客户端还负责查询失败后重试、取消查询请求、以及维持连接心跳、向管理节点上报查询状态等工作。 - -管理节点（Management Node）。管理节点保存了整个集群系统的全部数据的元数据信息，向客户端节点提供查询所需的数据的元数据，并根据集群的负载情况切分查询请求。通过超级表包含了通过该超级表创建的所有表的信息，因此查询处理器（Query Executor）负责针对标签（TAG）的查询处理，并将满足标签查询请求的表信息返回给客户端。此外，管理节点还维护集群的查询状态（Query Status Manager）维护，查询状态管理中在内存中临时保存有当前正在执行的全部查询，当客户端使用 *show queries* 命令的时候，将当前系统正在运行的查询信息返回客户端。 - -数据节点（Data Node）。数据节点保存了数据库中全部数据内容，并通过查询执行器、查询处理调度器、查询任务队列（Query Task Queue）进行查询处理的调度执行，从客户端接收到的查询处理请求都统一放置到处理队列中，查询执行器从队列中获得查询请求，并负责执行。通过查询优化器（Query Optimizer）对于查询进行基本的优化处理，以及通过数据节点的查询执行器（Query Executor）扫描符合条件的数据单元并返回计算结果。等接收客户端发出的查询请求，执行查询处理，并将结果返回。同时数据节点还需要响应来自管理节点的管理信息和命令，例如 *kill query* 命令以后，需要即刻停止执行的查询任务。 - -
-
图 1. 系统查询处理架构图(只包含查询相关组件)
- -### 普通查询处理 - -客户端、管理节点、数据节点协同完成TDengine的查询处理全流程。我们以一个具体的SQL查询为例，说明TDengine的查询处理流程。SQL语句向超级表*FOO_SUPER_TABLE*查询获取时间范围在2019年1月12日整天，标签TAG_LOC是'beijing'的表所包含的所有记录总数，SQL语句如下： - -```sql -SELECT COUNT(*) -FROM FOO_SUPER_TABLE -WHERE TAG_LOC = 'beijing' AND TS >= '2019-01-12 00:00:00' AND TS < '2019-01-13 00:00:00' -``` - -首先，客户端调用TAOS SQL解析器对SQL语句进行解析及合法性检查，然后生成语法树，并从中提取查询的对象 — 超级表 *FOO_SUPER_TABLE* ，然后解析器向管理节点（Management Node）请求其相应的元数据信息，并将过滤信息（TAG_LOC='beijing'）同时发送到管理节点。 - -管理节点接收元数据获取的请求，首先找到超级表 *FOO_SUPER_TABLE* 基础信息，然后应用查询条件来过滤通过该超级表创建的全部表，最后满足查询条件（TAG_LOC='beijing'），即 *TAG_LOC* 标签列是 'beijing' 的的通过其查询执行器将满足查询要求的对象（表或超级表）的元数据信息返回给客户端。 - -客户端获得了 *FOO_SUPER_TABLE* 的元数据信息后，查询执行器根据元数据中的数据分布，分别向保存有相应数据的节点发起查询请求，此时时间戳范围过滤条件（TS >= '2019-01-12 00:00:00' AND TS < '2019-01-13 00:00:00'）需要同时发送给全部的数据节点。 - -数据节点接收到发自客户端的查询，转化为内部结构并进行优化以后将其放入任务执行队列，等待查询执行器执行。当查询结果获得以后，将查询结果返回客户端。数据节点执行查询的过程均相互独立，完全只依赖于自身的数据和内容进行计算。 - -当所有查询涉及的数据节点返回结果后，客户端将每个数据节点查询的结果集再次进行聚合（针对本案例，即将所有结果再次进行累加），累加的结果即为最后的查询结果。第二阶段聚合并不是所有的查询都需要。例如，针对数据的列选取操作，实际上是不需要第二阶段聚合。 - -### REST查询处理 - -在 C/C++ 、Python接口、 JDBC 接口之外，TDengine 还提供基于 HTTP 协议的 REST 接口。不同于使用应用客户端开发程序进行的开发。当用户使用 REST 接口的时候，所有的查询处理过程都是在服务器端来完成，用户的应用服务不会参与数据库的计算过程，查询处理完成后结果通过 HTTP的 JSON 格式返回给用户。 - -
-
图 2. REST查询架构
- -当用户使用基于HTTP的REST查询接口，HTTP的请求首先与位于数据节点的HTTP连接器（ Connector），建立连接，然后通过REST的签名机制，使用Token来确保请求的可靠性。对于数据节点，HTTP连接器接收到请求后，调用内嵌的客户端程序发起查询请求，内嵌客户端将解析通过HTTP连接器传递过来的SQL语句，解析该SQL语句并按需向管理节点请求元数据信息，然后向本机或集群中其他节点发送查询请求，最后按需聚合计算结果。HTTP连接器接收到请求SQL以后，后续的流程处理与采用应用客户端方式的查询处理完全一致。最后，还需要将查询的结果转换为JSON格式字符串，并通过HTTP 响应返回给客户端。 - -可以看到，在处理HTTP流程的整个过程中，用户应用不再参与到查询处理的过程中，只负责通过HTTP协议发送SQL请求并接收JSON格式的结果。同时还需要注意的是，每个数据节点均内嵌了一个HTTP连接器和客户端程序，因此请求集群中任何一个数据节点，该数据节点均能够通过HTTP协议返回用户的查询结果。 - -### 技术特征 - -由于TDengine采用数据和标签分离存储的模式，能够极大地降低标签数据存储的冗余度。标签数据直接关联到每个表，并采用全内存的结构进行管理和维护标签数据,全内存的结构提供快速的查询处理，千万级别规模的标签数据查询可以在毫秒级别返回。首先针对标签数据的过滤可以有效地降低第二阶段的查询涉及的数据规模。为有效地提升查询处理的性能，针对物联网数据的不可更改的特点，TDengine采用在每个保存的数据块上，都记录下该数据块中数据的最大值、最小值、和等统计数据。如果查询处理涉及整个数据块的全部数据，则直接使用预计算结果，不再读取数据块的内容。由于预计算模块的大小远小于磁盘上存储的具体数据的大小，对于磁盘IO为瓶颈的查询处理，使用预计算结果可以极大地减小读取IO，并加速查询处理的流程。 - -由于TDengine采用按列存储数据。当从磁盘中读取数据块进行计算的时候，按照查询列信息读取该列数据，并不需要读取其他不相关的数据，可以最小化读取数据。此外，由于采用列存储结构，数据节点针对数据的扫描采用该列数据块进行，可以充分利用CPU L2高速缓存，极大地加速数据扫描的速度。此外，对于某些查询，并不会等全部查询结果生成后再返回结果。例如，列选取查询，当第一批查询结果获得以后，数据节点直接将其返回客户端。同时，在查询处理过程中，系统在数据节点接收到查询请求以后马上返回客户端查询确认信息，并同时拉起查询处理过程，并等待查询执行完成后才返回给用户查询有响应。 - -## TDengine集群设计 - -### 1：集群与主要逻辑单元 - -TDengine是基于硬件、软件系统不可靠、一定会有故障的假设进行设计的，是基于任何单台计算机都无足够能力处理海量数据的假设进行设计的。因此TDengine从研发的第一天起，就按照分布式高可靠架构进行设计，是完全去中心化的，是水平扩展的，这样任何单台或多台服务器宕机或软件错误都不影响系统的服务。通过节点虚拟化并辅以自动化负载均衡技术，TDengine能最大限度地利用异构集群中的计算和存储资源。而且只要数据副本数大于一，无论是硬软件的升级、还是IDC的迁移等都无需停止集群的服务，极大地保证系统的正常运行，并且降低了系统管理员和运维人员的工作量。 - -下面的示例图上有八个物理节点，每个物理节点被逻辑的划分为多个虚拟节点。下面对系统的基本概念进行介绍。 - - - -![assets/nodes.png](../assets/nodes.png) - -**物理节点(dnode)**：集群中的一物理服务器或云平台上的一虚拟机。为安全以及通讯效率，一个物理节点可配置两张网卡，或两个IP地址。其中一张网卡用于集群内部通讯，其IP地址为**privateIp**, 另外一张网卡用于与集群外部应用的通讯，其IP地址为**publicIp**。在一些云平台（如阿里云），对外的IP地址是映射过来的，因此publicIp还有一个对应的内部IP地址**internalIp**(与privateIp不同)。对于只有一个IP地址的物理节点，publicIp, privateIp以及internalIp都是同一个地址，没有任何区别。一个dnode上有而且只有一个taosd实例运行。 - -**虚拟数据节点(vnode)**：在物理节点之上的可独立运行的基础逻辑单元，时序数据写入、存储、查询等操作逻辑都在虚拟节点中进行(图中V)，采集的时序数据就存储在vnode上。一个vnode包含固定数量的表。当创建一张新表时，系统会检查是否需要创建新的vnode。一个物理节点上能创建的vnode的数量取决于物理节点的硬件资源。一个vnode只属于一个DB，但一个DB可以有多个vnode。 - -**虚拟数据节点组(vgroup)**: 位于不同物理节点的vnode可以组成一个虚拟数据节点组vnode group(如上图dnode0中的V0, dnode1中的V1, dnode6中的V2属于同一个虚拟节点组)。归属于同一个vgroup的虚拟节点采取master/slave的方式进行管理。写只能在master上进行，但采用asynchronous的方式将数据同步到slave，这样确保了一份数据在多个物理节点上有拷贝。如果master节点宕机，其他节点监测到后，将重新选举vgroup里的master, 新的master能继续处理数据请求，从而保证系统运行的可靠性。一个vgroup里虚拟节点个数就是数据的副本数。如果一个DB的副本数为N，系统必须有至少N个物理节点。副本数在创建DB时通过参数replica可以指定，缺省为1。使用TDengine, 数据的安全依靠多副本解决，因此不再需要昂贵的磁盘阵列等存储设备。 - -**虚拟管理节点(mnode)**：负责所有节点运行状态的监控和维护，以及节点之间的负载均衡（图中M）。同时，虚拟管理节点也负责元数据(包括用户、数据库、表、静态标签等)的存储和管理，因此也称为Meta Node。TDengine集群中可配置多个(最多不超过5个) mnode，它们自动构建成为一个管理节点集群(图中M0, M1, M2)。mnode间采用master/slave的机制进行管理，而且采取强一致方式进行数据同步。mnode集群的创建由系统自动完成，无需人工干预。每个dnode上至多有一个mnode，而且每个dnode都知道整个集群中所有mnode的IP地址。 - -**taosc**：一个软件模块，是TDengine给应用提供的驱动程序（driver）,内嵌于JDBC、ODBC driver中，或者C语言连接库里。应用都是通过taosc而不是直接来与整个集群进行交互的。这个模块负责获取并缓存元数据；将插入、查询等请求转发到正确的虚拟节点；在把结果返回给应用时，还需要负责最后一级的聚合、排序、过滤等操作。对于JDBC, ODBC, C/C++接口而言，这个模块是在应用所处的计算机上运行，但消耗的资源很小。为支持全分布式的REST接口，taosc在TDengine集群的每个dnode上都有一运行实例。 - -**对外服务地址**：TDengine集群可以容纳单台、多台甚至几千台物理节点。应用只需要向集群中任何一个物理节点的publicIp发起连接即可。启动CLI应用taos时，选项-h需要提供的就是publicIp。 - -**master/secondIp**：每一个dnode都需要配置一个masterIp。dnode启动后，将对配置的masterIp发起加入集群的连接请求。masterIp是已经创建的集群中的任何一个节点的privateIp，对于集群中的第一个节点，就是它自己的privateIp。为保证连接成功，每个dnode还可配置secondIp, 该IP地址也是已创建的集群中的任何一个节点的privateIp。如果一个节点连接masterIp失败,它将试图连接secondIp。 - -dnode启动后，会获知集群的mnode IP列表，并且定时向mnode发送状态信息。 - -vnode与mnode只是逻辑上的划分，都是执行程序taosd里的不同线程而已，无需安装不同的软件，做任何特殊的配置。最小的系统配置就是一个物理节点,vnode,mnode和taosc都存在而且都正常运行，但单一节点无法保证系统的高可靠。 - -### 2：一典型的操作流程 - -为解释vnode, mnode, taosc和应用之间的关系以及各自扮演的角色，下面对写入数据这个典型操作的流程进行剖析。 - - - -![Picture1](../assets/Picture2.png) - - - -1. 应用通过JDBC、ODBC或其他API接口发起插入数据的请求。 -2. taosc会检查缓存，看是有保存有该表的meta data。如果有，直接到第4步。如果没有，taosc将向mnode发出get meta-data请求。 -3. mnode将该表的meta-data返回给taosc。Meta-data包含有该表的schema, 而且还有该表所属的vgroup信息（vnode ID以及所在的dnode的IP地址，如果副本数为N，就有N组vnodeID/IP)。如果taosc迟迟得不到mnode回应，而且存在多个mnode,taosc将向下一个mnode发出请求。 -4. taosc向master vnode发起插入请求。 -5. vnode插入数据后，给taosc一个应答，表示插入成功。如果taosc迟迟得不到vnode的回应，taosc会认为该节点已经离线。这种情况下，如果被插入的数据库有多个副本，taosc将向vgroup里下一个vnode发出插入请求。 -6. taosc通知APP，写入成功。 - -对于第二和第三步，taosc启动时，并不知道mnode的IP地址，因此会直接向配置的集群对外服务的IP地址发起请求。如果接收到该请求的dnode并没有配置mnode，该dnode会在回复的消息中告知mnode的IP地址列表（如果有多个dnodes，mnode的IP地址可以有多个），这样taosc会重新向新的mnode的IP地址发出获取meta-data的请求。 - -对于第四和第五步，没有缓存的情况下，taosc无法知道虚拟节点组里谁是master，就假设第一个vnodeID/IP就是master,向它发出请求。如果接收到请求的vnode并不是master,它会在回复中告知谁是master，这样taosc就向建议的master vnode发出请求。一旦得到插入成功的回复，taosc会缓存住master节点的信息。 - -上述是插入数据的流程，查询、计算的流程也完全一致。taosc把这些复杂的流程全部封装屏蔽了，因此应用无需处理重定向、获取meta data等细节，完全是透明的。 - -通过taosc缓存机制，只有在第一次对一张表操作时，才需要访问mnode, 因此mnode不会成为系统瓶颈。但因为schema有可能变化，而且vgroup有可能发生改变（比如负载均衡发生），因此taosc需要定时自动刷新缓存。 - -### 3：数据分区 - -vnode(虚拟数据节点)保存采集的时序数据，而且查询、计算都在这些节点上进行。为便于负载均衡、数据恢复、支持异构环境，TDengine将一个物理节点根据其计算和存储资源切分为多个vnode。这些vnode的管理是TDengine自动完成的，对应用完全透明。 - -对于单独一个数据采集点，无论其数据量多大，一个vnode（或vnode group, 如果副本数大于1）有足够的计算资源和存储资源来处理（如果每秒生成一条16字节的记录，一年产生的原始数据不到0.5G），因此TDengine将一张表的所有数据都存放在一个vnode里，而不会让同一个采集点的数据分布到两个或多个dnode上。而且一个vnode可存储多张表的数据，一个vnode可容纳的表的数目由配置参数tables指定，缺省为2000。设计上，一个vnode里所有的表都属于同一个DB。因此一个数据库DB需要的vnode或vgroup的个数等于：数据库表的数目/tables。 - -创建DB时，系统并不会马上分配资源。但当创建一张表时，系统将看是否有已经分配的vnode, 而且是否有空位，如果有，立即在该有空位的vnode创建表。如果没有，系统将从集群中，根据当前的负载情况，在一个dnode上创建一新的vnode, 然后创建表。如果DB有多个副本，系统不是只创建一个vnode，而是一个vgroup(虚拟数据节点组)。系统对vnode的数目没有任何限制，仅仅受限于物理节点本身的计算和存储资源。 - -参数tables的设置需要考虑具体场景，创建DB时，可以个性化指定该参数。该参数不宜过大，也不宜过小。过小，极端情况，就是每个数据采集点一个vnode, 这样导致系统数据文件过多。过大，虚拟化带来的优势就会丧失。给定集群计算资源的情况下，整个系统vnode的个数应该是CPU核的数目的两倍以上。 - -### 4：负载均衡 - -每个dnode(物理节点)都定时向 mnode(虚拟管理节点)报告其状态（包括硬盘空间、内存大小、CPU、网络、虚拟节点个数等），因此mnode了解整个集群的状态。基于整体状态，当mnode发现某个dnode负载过重，它会将dnode上的一个或多个vnode挪到其他dnode。在挪动过程中，对外服务继续进行，数据插入、查询和计算操作都不受影响。负载均衡操作结束后，应用也无需重启，将自动连接新的vnode。 - -如果mnode一段时间没有收到dnode的状态报告，mnode会认为这个dnode已经离线。如果离线时间超过一定时长（时长由配置参数offlineThreshold决定），该dnode将被mnode强制剔除出集群。该dnode上的vnodes如果副本数大于一，系统将自动在其他dnode上创建新的副本，以保证数据的副本数。 - - - -**Note：**目前集群功能仅仅限于企业版 diff --git a/documentation20/webdocs/markdowndocs/More on System Architecture.md b/documentation20/webdocs/markdowndocs/More on System Architecture.md deleted file mode 100644 index d7a38b99a3ae5a630509f3ef0f0ffdc97d3aaaf1..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/More on System Architecture.md +++ /dev/null @@ -1,176 +0,0 @@ -# TDengine System Architecture - -## Storage Design - -TDengine data mainly include **metadata** and **data** that we will introduce in the following sections. - -### Metadata Storage - -Metadata include the information of databases, tables, etc. Metadata files are saved in _/var/lib/taos/mgmt/_ directory by default. The directory tree is as below: -``` -/var/lib/taos/ - +--mgmt/ - +--db.db - +--meters.db - +--user.db - +--vgroups.db -``` - -A metadata structure (database, table, etc.) is saved as a record in a metadata file. All metadata files are appended only, and even a drop operation adds a deletion record at the end of the file. - -### Data storage - -Data in TDengine are sharded according to the time range. Data of tables in the same vnode in a certain time range are saved in the same filegroup, such as files v0f1804*. This sharding strategy can effectively improve data searching speed. By default, a group of files contains data in 10 days, which can be configured by *daysPerFile* in the configuration file or by *DAYS* keyword in *CREATE DATABASE* clause. Data in files are blockwised. A data block only contains one table's data. Records in the same data block are sorted according to the primary timestamp, which helps to improve the compression rate and save storage. The compression algorithms used in TDengine include simple8B, delta-of-delta, RLE, LZ4, etc. - -By default, TDengine data are saved in */var/lib/taos/data/* directory. _/var/lib/taos/tsdb/_ directory contains vnode informations and data file linkes. - -``` -/var/lib/taos/ - +--tsdb/ - | +--vnode0 - | +--meterObj.v0 - | +--db/ - | +--v0f1804.head->/var/lib/taos/data/vnode0/v0f1804.head1 - | +--v0f1804.data->/var/lib/taos/data/vnode0/v0f1804.data - | +--v0f1804.last->/var/lib/taos/data/vnode0/v0f1804.last1 - | +--v0f1805.head->/var/lib/taos/data/vnode0/v0f1805.head1 - | +--v0f1805.data->/var/lib/taos/data/vnode0/v0f1805.data - | +--v0f1805.last->/var/lib/taos/data/vnode0/v0f1805.last1 - | : - +--data/ - +--vnode0/ - +--v0f1804.head1 - +--v0f1804.data - +--v0f1804.last1 - +--v0f1805.head1 - +--v0f1805.data - +--v0f1805.last1 - : -``` - -#### meterObj file -There are only one meterObj file in a vnode. Informations bout the vnode, such as created time, configuration information, vnode statistic informations are saved in this file. It has the structure like below: - -``` - -[file_header] -[table_record1_offset&length] -[table_record2_offset&length] -... -[table_recordN_offset&length] -[table_record1] -[table_record2] -... -[table_recordN] - -``` -The file header takes 512 bytes, which mainly contains informations about the vnode. Each table record is the representation of a table on disk. - -#### head file -The _head_ files contain the index of data blocks in the _data_ file. The inner organization is as below: -``` - -[file_header] -[table1_offset] -[table2_offset] -... -[tableN_offset] -[table1_index_block] -[table2_index_block] -... -[tableN_index_block] - -``` -The table offset array in the _head_ file saves the information about the offsets of each table index block. Indices on data blocks in the same table are saved continuously. This also makes it efficient to load data indices on the same table. The data index block has a structure like: - -``` -[index_block_info] -[block1_index] -[block2_index] -... -[blockN_index] -``` -The index block info part contains the information about the index block such as the number of index blocks, etc. Each block index corresponds to a real data block in the _data_ file or _last_ file. Information about the location of the real data block, the primary timestamp range of the data block, etc. are all saved in the block index part. The block indices are sorted in ascending order according to the primary timestamp. So we can apply algorithms such as the binary search on the data to efficiently search blocks according to time. - -#### data file -The _data_ files store the real data block. They are append-only. The organization is as: -``` - -[file_header] -[block1] -[block2] -... -[blockN] - -``` -A data block in _data_ files only belongs to a table in the vnode and the records in a data block are sorted in ascending order according to the primary timestamp key. Data blocks are column-oriented. Data in the same column are stored contiguously, which improves reading speed and compression rate because of their similarity. A data block has the following organization: - -``` -[column1_info] -[column2_info] -... -[columnN_info] -[column1_data] -[column2_data] -... -[columnN_data] -``` -The column info part includes information about column types, column compression algorithm, column data offset and length in the _data_ file, etc. Besides, pre-calculated results of the column data in the block are also in the column info part, which helps to improve reading speed by avoiding loading data block necessarily. - -#### last file -To avoid storage fragment and to import query speed and compression rate, TDengine introduces an extra file, the _last_ file. When the number of records in a data block is lower than a threshold, TDengine will flush the block to the _last_ file for temporary storage. When new data comes, the data in the _last_ file will be merged with the new data and form a larger data block and written to the _data_ file. The organization of the _last_ file is similar to the _data_ file. - -### Summary -The innovation in architecture and storage design of TDengine improves resource usage. On the one hand, the virtualization makes it easy to distribute resources between different vnodes and for future scaling. On the other hand, sorted and column-oriented storage makes TDengine have a great advantage in writing, querying and compression. - -## Query Design - -#### Introduction - -TDengine provides a variety of query functions for both tables and super tables. In addition to regular aggregate queries, it also provides time window based query and statistical aggregation for time series data. TDengine's query processing requires the client app, management node, and data node to work together. The functions and modules involved in query processing included in each component are as follows: - -Client (Client App). The client development kit, embed in a client application, consists of TAOS SQL parser and query executor, the second-stage aggregator (Result Merger), continuous query manager and other major functional modules. The SQL parser is responsible for parsing and verifying the SQL statement and converting it into an abstract syntax tree. The query executor is responsible for transforming the abstract syntax tree into the query execution logic and creates the metadata query according to the query condition of the SQL statement. Since TAOS SQL does not currently include complex nested queries and pipeline query processing mechanism, there is no longer need for query plan optimization and physical query plan conversions. The second-stage aggregator is responsible for performing the aggregation of the independent results returned by query involved data nodes at the client side to generate final results. The continuous query manager is dedicated to managing the continuous queries created by users, including issuing fixed-interval query requests and writing the results back to TDengine or returning to the client application as needed. Also, the client is also responsible for retrying after the query fails, canceling the query request, and maintaining the connection heartbeat and reporting the query status to the management node. - -Management Node. The management node keeps the metadata of all the data of the entire cluster system, provides the metadata of the data required for the query from the client node, and divides the query request according to the load condition of the cluster. The super table contains information about all the tables created according to the super table, so the query processor (Query Executor) of the management node is responsible for the query processing of the tags of tables and returns the table information satisfying the tag query. Besides, the management node maintains the query status of the cluster in the Query Status Manager component, in which the metadata of all queries that are currently executing are temporarily stored in-memory buffer. When the client issues *show queries* command to management node, current running queries information is returned to the client. - -Data Node. The data node, responsible for storing all data of the database, consists of query executor, query processing scheduler, query task queue, and other related components. Once the query requests from the client received, they are put into query task queue and waiting to be processed by query executor. The query executor extracts the query request from the query task queue and invokes the query optimizer to perform the basic optimization for the query execution plan. And then query executor scans the qualified data blocks in both cache and disk to obtain qualified data and return the calculated results. Besides, the data node also needs to respond to management information and commands from the management node. For example, after the *kill query* received from the management node, the query task needs to be stopped immediately. - -
-
Fig 1. System query processing architecture diagram (only query related components)
- -#### Query Process Design - -The client, the management node, and the data node cooperate to complete the entire query processing of TDengine. Let's take a concrete SQL query as an example to illustrate the whole query processing flow. The SQL statement is to query on super table *FOO_SUPER_TABLE* to get the total number of records generated on January 12, 2019, from the table, of which TAG_LOC equals to 'beijing'. The SQL statement is as follows: - -```sql -SELECT COUNT(*) -FROM FOO_SUPER_TABLE -WHERE TAG_LOC = 'beijing' AND TS >= '2019-01-12 00:00:00' AND TS < '2019-01-13 00:00:00' -``` - -First, the client invokes the TAOS SQL parser to parse and validate the SQL statement, then generates a syntax tree, and extracts the object of the query - the super table *FOO_SUPER_TABLE*, and then the parser sends requests with filtering information (TAG_LOC='beijing') to management node to get the corresponding metadata about *FOO_SUPER_TABLE*. - -Once the management node receives the request for metadata acquisition, first finds the super table *FOO_SUPER_TABLE* basic information, and then applies the query condition (TAG_LOC='beijing') to filter all the related tables created according to it. And finally, the query executor returns the metadata information that satisfies the query request to the client. - -After the client obtains the metadata information of *FOO_SUPER_TABLE*, the query executor initiates a query request with timestamp range filtering condition (TS >= '2019- 01-12 00:00:00' AND TS < '2019-01-13 00:00:00') to all nodes that hold the corresponding data according to the information about data distribution in metadata. - -The data node receives the query sent from the client, converts it into an internal structure and puts it into the query task queue to be executed by query executor after optimizing the execution plan. When the query result is obtained, the query result is returned to the client. It should be noted that the data nodes perform the query process independently of each other, and rely solely on their data and content for processing. - -When all data nodes involved in the query return results, the client aggregates the result sets from each data node. In this case, all results are accumulated to generate the final query result. The second stage of aggregation is not always required for all queries. For example, a column selection query does not require a second-stage aggregation at all. - -#### REST Query Process - -In addition to C/C++, Python, and JDBC interface, TDengine also provides a REST interface based on the HTTP protocol, which is different from using the client application programming interface. When the user uses the REST interface, all the query processing is completed on the server-side, and the user's application is not involved in query processing anymore. After the query processing is completed, the result is returned to the client through the HTTP JSON string. - -
-
Fig. 2 REST query architecture
- -When a client uses an HTTP-based REST query interface, the client first establishes a connection with the HTTP connector at the data node and then uses the token to ensure the reliability of the request through the REST signature mechanism. For the data node, after receiving the request, the HTTP connector invokes the embedded client program to initiate a query processing, and then the embedded client parses the SQL statement from the HTTP connector and requests the management node to get metadata as needed. After that, the embedded client sends query requests to the same data node or other nodes in the cluster and aggregates the calculation results on demand. Finally, you also need to convert the result of the query into a JSON format string and return it to the client via an HTTP response. After the HTTP connector receives the request SQL, the subsequent process processing is completely consistent with the query processing using the client application development kit. - -It should be noted that during the entire processing, the client application is no longer involved in, and is only responsible for sending SQL requests through the HTTP protocol and receiving the results in JSON format. Besides, each data node is embedded with an HTTP connector and a client, so any data node in the cluster received requests from a client, the data node can initiate the query and return the result to the client through the HTTP protocol, with transfer the request to other data nodes. - -#### Technology - -Because TDengine stores data and tags value separately, the tag value is kept in the management node and directly associated with each table instead of records, resulting in a great reduction of the data storage. Therefore, the tag value can be managed by a fully in-memory structure. First, the filtering of the tag data can drastically reduce the data size involved in the second phase of the query. The query processing for the data is performed at the data node. TDengine takes advantage of the immutable characteristics of IoT data by calculating the maximum, minimum, and other statistics of the data in one data block on each saved data block, to effectively improve the performance of query processing. If the query process involves all the data of the entire data block, the pre-computed result is used directly, and the content of the data block is no longer needed. Since the size of disk space required to store the pre-computation result is much smaller than the size of the specific data, the pre-computation result can greatly reduce the disk IO and speed up the query processing. - -TDengine employs column-oriented data storage techniques. When the data block is involved to be loaded from the disk for calculation, only the required column is read according to the query condition, and the read overhead can be minimized. The data of one column is stored in a contiguous memory block and therefore can make full use of the CPU L2 cache to greatly speed up the data scanning. Besides, TDengine utilizes the eagerly responding mechanism and returns a partial result before the complete result is acquired. For example, when the first batch of results is obtained, the data node immediately returns it directly to the client in case of a column select query. \ No newline at end of file diff --git a/documentation20/webdocs/markdowndocs/Super Table-ch.md b/documentation20/webdocs/markdowndocs/Super Table-ch.md deleted file mode 100644 index e5c77471570a76e608d59a0dca10462315460337..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Super Table-ch.md +++ /dev/null @@ -1,224 +0,0 @@ -# 超级表STable：多表聚合 - -TDengine要求每个数据采集点单独建表，这样能极大提高数据的插入/查询性能，但是导致系统中表的数量猛增，让应用对表的维护以及聚合、统计操作难度加大。为降低应用的开发难度，TDengine引入了超级表STable (Super Table)的概念。 - -## 什么是超级表 - -STable是同一类型数据采集点的抽象，是同类型采集实例的集合，包含多张数据结构一样的子表。每个STable为其子表定义了表结构和一组标签：表结构即表中记录的数据列及其数据类型；标签名和数据类型由STable定义，标签值记录着每个子表的静态信息，用以对子表进行分组过滤。子表本质上就是普通的表，由一个时间戳主键和若干个数据列组成，每行记录着具体的数据，数据查询操作与普通表完全相同；但子表与普通表的区别在于每个子表从属于一张超级表，并带有一组由STable定义的标签值。每种类型的采集设备可以定义一个STable。数据模型定义表的每列数据的类型，如温度、压力、电压、电流、GPS实时位置等，而标签信息属于Meta Data，如采集设备的序列号、型号、位置等，是静态的，是表的元数据。用户在创建表（数据采集点）时指定STable(采集类型)外，还可以指定标签的值，也可事后增加或修改。 - -TDengine扩展标准SQL语法用于定义STable，使用关键词tags指定标签信息。语法如下： - -```mysql -CREATE TABLE ( TIMESTAMP, field_name1 field_type,…) TAGS(tag_name tag_type, …) -``` - -其中tag_name是标签名，tag_type是标签的数据类型。标签可以使用时间戳之外的其他TDengine支持的数据类型，标签的个数最多为6个，名字不能与系统关键词相同，也不能与其他列名相同。如： - -```mysql -create table thermometer (ts timestamp, degree float) -tags (location binary(20), type int) -``` - -上述SQL创建了一个名为thermometer的STable，带有标签location和标签type。 - -为某个采集点创建表时，可以指定其所属的STable以及标签的值，语法如下： - -```mysql -CREATE TABLE USING TAGS (tag_value1,...) -``` - -沿用上面温度计的例子，使用超级表thermometer建立单个温度计数据表的语句如下： - -```mysql -create table t1 using thermometer tags (‘beijing’, 10) -``` - -上述SQL以thermometer为模板，创建了名为t1的表，这张表的Schema就是thermometer的Schema，但标签location值为‘beijing’，标签type值为10。 - -用户可以使用一个STable创建数量无上限的具有不同标签的表，从这个意义上理解，STable就是若干具有相同数据模型，不同标签的表的集合。与普通表一样，用户可以创建、删除、查看超级表STable，大部分适用于普通表的查询操作都可运用到STable上，包括各种聚合和投影选择函数。除此之外，可以设置标签的过滤条件，仅对STbale中部分表进行聚合查询，大大简化应用的开发。 - -TDengine对表的主键（时间戳）建立索引，暂时不提供针对数据模型中其他采集量（比如温度、压力值）的索引。每个数据采集点会采集若干数据记录，但每个采集点的标签仅仅是一条记录，因此数据标签在存储上没有冗余，且整体数据规模有限。TDengine将标签数据与采集的动态数据完全分离存储，而且针对STable的标签建立了高性能内存索引结构，为标签提供全方位的快速操作支持。用户可按照需求对其进行增删改查（Create，Retrieve，Update，Delete，CRUD）操作。 - -STable从属于库，一个STable只属于一个库，但一个库可以有一到多个STable, 一个STable可有多个子表。 - -## 超级表管理 - -- 创建超级表 - - ```mysql - CREATE TABLE ( TIMESTAMP, field_name1 field_type,…) TAGS(tag_name tag_type, …) - ``` - - 与创建表的SQL语法相似。但需指定TAGS字段的名称和类型。 - - 说明： - - 1. TAGS列总长度不能超过512 bytes； - 2. TAGS列的数据类型不能是timestamp和nchar类型； - 3. TAGS列名不能与其他列名相同; - 4. TAGS列名不能为预留关键字. - -- 显示已创建的超级表 - - ```mysql - show stables; - ``` - - 查看数据库内全部STable，及其相关信息，包括STable的名称、创建时间、列数量、标签（TAG）数量、通过该STable建表的数量。 - -- 删除超级表 - - ```mysql - DROP TABLE - ``` - - Note: 删除STable不会级联删除通过STable创建的表；相反删除STable时要求通过该STable创建的表都已经被删除。 - -- 查看属于某STable并满足查询条件的表 - - ```mysql - SELECT TBNAME,[TAG_NAME,…] FROM WHERE <[=|=<|>=|<>] values..> ([AND|OR] …) - ``` - - 查看属于某STable并满足查询条件的表。说明：TBNAME为关键词，显示通过STable建立的子表表名，查询过程中可以使用针对标签的条件。 - - ```mysql - SELECT COUNT(TBNAME) FROM WHERE <[=|=<|>=|<>] values..> ([AND|OR] …) - ``` - - 统计属于某个STable并满足查询条件的子表的数量 - -## 写数据时自动建子表 - -在某些特殊场景中，用户在写数据时并不确定某个设备的表是否存在，此时可使用自动建表语法来实现写入数据时里用超级表定义的表结构自动创建不存在的子表，若该表已存在则不会建立新表。注意：自动建表语句只能自动建立子表而不能建立超级表，这就要求超级表已经被事先定义好。自动建表语法跟insert/import语法非常相似，唯一区别是语句中增加了超级表和标签信息。具体语法如下： - -```mysql -INSERT INTO USING TAGS (, ...) VALUES (field_value, ...) (field_value, ...) ...; -``` - -向表tb_name中插入一条或多条记录，如果tb_name这张表不存在，则会用超级表stb_name定义的表结构以及用户指定的标签值(即tag1_value…)来创建名为tb_name新表，并将用户指定的值写入表中。如果tb_name已经存在，则建表过程会被忽略，系统也不会检查tb_name的标签是否与用户指定的标签值一致，也即不会更新已存在表的标签。 - -```mysql -INSERT INTO USING TAGS (, ...) VALUES (, ...) (, ...) ... USING TAGS(, ...) VALUES (, ...) ...; -``` - -向多张表tb1_name，tb2_name等插入一条或多条记录，并分别指定各自的超级表进行自动建表。 - -## STable中TAG管理 - -除了更新标签的值的操作是针对子表进行，其他所有的标签操作（添加标签、删除标签等）均只能作用于STable，不能对单个子表操作。对STable添加标签以后，依托于该STable建立的所有表将自动增加了一个标签，对于数值型的标签，新增加的标签的默认值是0. - -- 添加新的标签 - - ```mysql - ALTER TABLE ADD TAG - ``` - - 为STable增加一个新的标签，并指定新标签的类型。标签总数不能超过6个。 - -- 删除标签 - - ```mysql - ALTER TABLE DROP TAG - ``` - - 删除超级表的一个标签，从超级表删除某个标签后，该超级表下的所有子表也会自动删除该标签。 - - 说明：第一列标签不能删除，至少需要为STable保留一个标签。 - -- 修改标签名 - - ```mysql - ALTER TABLE CHANGE TAG - ``` - - 修改超级表的标签名，从超级表修改某个标签名后，该超级表下的所有子表也会自动更新该标签名。 - -- 修改子表的标签值 - - ```mysql - ALTER TABLE SET TAG = - ``` - -## STable多表聚合 - -针对所有的通过STable创建的子表进行多表聚合查询，支持按照全部的TAG值进行条件过滤，并可将结果按照TAGS中的值进行聚合，暂不支持针对binary类型的模糊匹配过滤。语法如下： - -```mysql -SELECT function,… - FROM - WHERE <[=|<=|>=|<>] values..> ([AND|OR] …) - INTERVAL ( [, offset]) - GROUP BY , … - ORDER BY - SLIMIT - SOFFSET - LIMIT - OFFSET -``` - -**说明**： - -超级表聚合查询，TDengine目前支持以下聚合\选择函数：sum、count、avg、first、last、min、max、top、bottom，以及针对全部或部分列的投影操作，使用方式与单表查询的计算过程相同。暂不支持其他类型的聚合计算和四则运算。当前所有的函数及计算过程均不支持嵌套的方式进行执行。 - - 不使用GROUP BY的查询将会对超级表下所有满足筛选条件的表按时间进行聚合，结果输出默认是按照时间戳单调递增输出，用户可以使用ORDER BY _c0 ASC|DESC选择查询结果时间戳的升降排序；使用GROUP BY 的聚合查询会按照tags进行分组，并对每个组内的数据分别进行聚合，输出结果为各个组的聚合结果，组间的排序可以由ORDER BY 语句指定，每个分组内部，时间序列是单调递增的。 - -使用SLIMIT/SOFFSET语句指定组间分页，即指定结果集中输出的最大组数以及对组起始的位置。使用LIMIT/OFFSET语句指定组内分页，即指定结果集中每个组内最多输出多少条记录以及记录起始的位置。 - -## STable使用示例 - -以温度传感器采集时序数据作为例，示范STable的使用。 在这个例子中，对每个温度计都会建立一张表，表名为温度计的ID，温度计读数的时刻记为ts，采集的值记为degree。通过tags给每个采集器打上不同的标签，其中记录温度计的地区和类型，以方便我们后面的查询。所有温度计的采集量都一样，因此我们用STable来定义表结构。 - -###定义STable表结构并使用它创建子表 - -创建STable语句如下： - -```mysql -CREATE TABLE thermometer (ts timestamp, degree double) -TAGS(location binary(20), type int) -``` - -假设有北京，天津和上海三个地区的采集器共4个，温度采集器有3种类型，我们就可以对每个采集器建表如下： - -```mysql -CREATE TABLE therm1 USING thermometer TAGS (’beijing’, 1); -CREATE TABLE therm2 USING thermometer TAGS (’beijing’, 2); -CREATE TABLE therm3 USING thermometer TAGS (’tianjin’, 1); -CREATE TABLE therm4 USING thermometer TAGS (’shanghai’, 3); -``` - -其中therm1，therm2，therm3，therm4是超级表thermometer四个具体的子表，也即普通的Table。以therm1为例，它表示采集器therm1的数据，表结构完全由thermometer定义，标签location=”beijing”, type=1表示therm1的地区是北京，类型是第1类的温度计。 - -###写入数据 - -注意，写入数据时不能直接对STable操作，而是要对每张子表进行操作。我们分别向四张表therm1，therm2， therm3， therm4写入一条数据，写入语句如下： - -```mysql -INSERT INTO therm1 VALUES (’2018-01-01 00:00:00.000’, 20); -INSERT INTO therm2 VALUES (’2018-01-01 00:00:00.000’, 21); -INSERT INTO therm3 VALUES (’2018-01-01 00:00:00.000’, 24); -INSERT INTO therm4 VALUES (’2018-01-01 00:00:00.000’, 23); -``` - -###按标签聚合查询 - -查询位于北京(beijing)和天津(tianjing)两个地区的温度传感器采样值的数量count(*)、平均温度avg(degree)、最高温度max(degree)、最低温度min(degree)，并将结果按所处地域(location)和传感器类型(type)进行聚合。 - -```mysql -SELECT COUNT(*), AVG(degree), MAX(degree), MIN(degree) -FROM thermometer -WHERE location=’beijing’ or location=’tianjing’ -GROUP BY location, type -``` - -###按时间周期聚合查询 - -查询仅位于北京以外地区的温度传感器最近24小时(24h)采样值的数量count(*)、平均温度avg(degree)、最高温度max(degree)和最低温度min(degree)，将采集结果按照10分钟为周期进行聚合，并将结果按所处地域(location)和传感器类型(type)再次进行聚合。 - -```mysql -SELECT COUNT(*), AVG(degree), MAX(degree), MIN(degree) -FROM thermometer -WHERE name<>’beijing’ and ts>=now-1d -INTERVAL(10M) -GROUP BY location, type -``` \ No newline at end of file diff --git a/documentation20/webdocs/markdowndocs/Super Table.md b/documentation20/webdocs/markdowndocs/Super Table.md deleted file mode 100644 index a213567f6d67ed351fac67b821f4db1929fa3a22..0000000000000000000000000000000000000000 --- a/documentation20/webdocs/markdowndocs/Super Table.md +++ /dev/null @@ -1,195 +0,0 @@ -# STable: Super Table - -"One Table for One Device" design can improve the insert/query performance significantly for a single device. But it has a side effect, the aggregation of multiple tables becomes hard. To reduce the complexity and improve the efficiency, TDengine introduced a new concept: STable (Super Table). - -## What is a Super Table - -STable is an abstract and a template for a type of device. A STable contains a set of devices (tables) that have the same schema or data structure. Besides the shared schema, a STable has a set of tags, like the model, serial number and so on. Tags are used to record the static attributes for the devices and are used to group a set of devices (tables) for aggregation. Tags are metadata of a table and can be added, deleted or changed. - -TDengine does not save tags as a part of the data points collected. Instead, tags are saved as metadata. Each table has a set of tags. To improve query performance, tags are all cached and indexed. One table can only belong to one STable, but one STable may contain many tables. - -Like a table, you can create, show, delete and describe STables. Most query operations on tables can be applied to STable too, including the aggregation and selector functions. For queries on a STable, if no tags filter, the operations are applied to all the tables created via this STable. If there is a tag filter, the operations are applied only to a subset of the tables which satisfy the tag filter conditions. It will be very convenient to use tags to put devices into different groups for aggregation. - -##Create a STable - -Similiar to creating a standard table, syntax is: - -```mysql -CREATE TABLE ( TIMESTAMP, field_name1 field_type,…) TAGS(tag_name tag_type, …) -``` - -New keyword "tags" is introduced, where tag_name is the tag name, and tag_type is the associated data type. - -Note： - -1. The bytes of all tags together shall be less than 512 -2. Tag's data type can not be time stamp or nchar -3. Tag name shall be different from the field name -4. Tag name shall not be the same as system keywords -5. Maximum number of tags is 6 - -For example: - -```mysql -create table thermometer (ts timestamp, degree float) -tags (location binary(20), type int) -``` - -The above statement creates a STable thermometer with two tag "location" and "type" - -##Create a Table via STable - -To create a table for a device, you can use a STable as its template and assign the tag values. The syntax is: - -```mysql -CREATE TABLE USING TAGS (tag_value1,...) -``` - -You can create any number of tables via a STable, and each table may have different tag values. For example, you create five tables via STable thermometer below: - -```mysql - create table t1 using thermometer tags (‘beijing’, 10); - create table t2 using thermometer tags (‘beijing’, 20); - create table t3 using thermometer tags (‘shanghai’, 10); - create table t4 using thermometer tags (‘shanghai’, 20); - create table t5 using thermometer tags (‘new york’, 10); -``` - -## Aggregate Tables via STable - -You can group a set of tables together by specifying the tags filter condition, then apply the aggregation operations. The result set can be grouped and ordered based on tag value. Syntax is： - -```mysql -SELECT function,… - FROM - WHERE <[=|<=|>=|<>] values..> ([AND|OR] …) - INTERVAL (