@@ -168,7 +168,8 @@ In a database, there can be one or more STables, but a STable belongs to only on
...
@@ -168,7 +168,8 @@ In a database, there can be one or more STables, but a STable belongs to only on
## Instance, URL, Token
## Instance, URL, Token
An instance is a running cluster of nodes of TDengine with one or more databases. An instance cannot span across multiple regions or multiple clouds, but a single account can have multiple instances. TDengine cloud provides a unique URL for each instance, and your applications need to use the generated, instance-specific tokens to access it for security purpose.
An instance is a running cluster of nodes of TDengine with one or more databases. An instance cannot span across multiple regions or multiple clouds, but a single account (organization) can have multiple instances. An account may invite multiple users into his/her organization to share the data, and each user can be configured with different access rights.
TDengine cloud provides a unique URL for each instance and uses tokens to authenticate the access. The token is generated by TDengine cloud for each user and for each instance. The token has a duration and can be reset by the user for each instance at any time for security purpose.
An account may have multiple users, and each user can be configured with different access rights. In addition, the token for an instance will be different for different users, and the token can be reset by the user.
@@ -42,6 +42,11 @@ For more details about `INSERT` please refer to [INSERT](https://docs.tdengine.c
...
@@ -42,6 +42,11 @@ For more details about `INSERT` please refer to [INSERT](https://docs.tdengine.c
## Connector Examples
## Connector Examples
:::note
Before executing the sample code in this section, you need to firstly establish connection to TDegnine cloud service, please refer to [Connect to TDengine Cloud Service](../../programming/connect/).
@@ -8,6 +8,10 @@ Telegraf is an open-source, metrics collection software. Telegraf can collect th
...
@@ -8,6 +8,10 @@ Telegraf is an open-source, metrics collection software. Telegraf can collect th
Telegraf's data can be written to TDengine by simply adding the output configuration of Telegraf to the URL corresponding to taosAdapter and modifying several configuration items. The presence of Telegraf data in TDengine can take advantage of TDengine's efficient storage query performance and clustering capabilities for time-series data.
Telegraf's data can be written to TDengine by simply adding the output configuration of Telegraf to the URL corresponding to taosAdapter and modifying several configuration items. The presence of Telegraf data in TDengine can take advantage of TDengine's efficient storage query performance and clustering capabilities for time-series data.
## Prerequisiteis
Before telegraf can write data into TDengine cloud service, you need to firstly manually create a database. Log in TDengine Cloud, click "Explorer" on the left navigation bar, then click the "+" button besides "Databases" to add a database named as "telegraf" using all default parameters.
## Install Telegraf
## Install Telegraf
Supposed that you use Ubuntu system:
Supposed that you use Ubuntu system:
...
@@ -63,8 +67,6 @@ telegraf --config telegraf.conf
...
@@ -63,8 +67,6 @@ telegraf --config telegraf.conf
## Verify
## Verify
- Log in TDengine Cloud, click "Explorer" on the left navigation bar, then click the "+" button besides "Databases" to add a database named as "telegraf" using all default parameters.
- Check weather database "telegraf" exist by executing:
- Check weather database "telegraf" exist by executing:
@@ -4,4 +4,9 @@ title: Write Data Into TDengine Cloud Service
...
@@ -4,4 +4,9 @@ title: Write Data Into TDengine Cloud Service
description:A number of ways for writing data into TDengine.
description:A number of ways for writing data into TDengine.
---
---
This chapter introduces a number of ways which can be used to write data into TDengine, users can use TDengine SQL to write data into TDengine cloud service, or use the [connectors](../programming/connector) provided by TDengine to program writing into TDengine. TDengine provides [taosBenchmark](../tools/taosbenchmark), which is a performance testing tool to write into TDengine, and taosX, which is a tool provided by TDengine enterprise edition, to sync data from one TDengine cloud service to another. Furthermore, 3rd party tools, like telegraf and prometheus, can also be used to write data into TDengine.
This chapter introduces a number of ways which can be used to write data into TDengine, users can use TDengine SQL to write data into TDengine cloud service, or use the [connectors](../programming/connector) provided by TDengine to program writing into TDengine. TDengine provides [taosBenchmark](../tools/taosbenchmark), which is a performance testing tool to write into TDengine, and taosX, which is a tool provided by TDengine enterprise edition, to sync data from one TDengine cloud service to another. Furthermore, 3rd party tools, like telegraf and prometheus, can also be used to write data into TDengine.
\ No newline at end of file
:::note
Because of privilege limitation on cloud, you need to firstly create database in the data explorer on cloud console before preparing to write data into TDengine cloud service. This limitation is applicable to any way of writing data.
@@ -123,6 +123,11 @@ For more details please refer to [Aggregate by Window](https://docs.tdengine.com
...
@@ -123,6 +123,11 @@ For more details please refer to [Aggregate by Window](https://docs.tdengine.com
## Connector Examples
## Connector Examples
:::note
Before executing the sample code in this section, you need to firstly establish connection to TDegnine cloud service, please refer to [Connect to TDengine Cloud Service](../../programming/connect/).
@@ -18,11 +18,7 @@ Users should not use taosdump to back up raw data, environment settings, hardwar
...
@@ -18,11 +18,7 @@ Users should not use taosdump to back up raw data, environment settings, hardwar
## Installation
## Installation
There are two ways to install taosdump:
Please refer to [Install taosTools](https://docs.tdengine.com/cloud/tools/taosdump/#installation).
- Install the taosTools official installer. Please find taosTools from [All download links](https://www.tdengine.com/all-downloads) page and download and install it.
- Compile taos-tools separately and install it. Please refer to the [taos-tools](https://github.com/taosdata/taos-tools) repository for details.
## Common usage scenarios
## Common usage scenarios
...
@@ -32,7 +28,7 @@ There are two ways to install taosdump:
...
@@ -32,7 +28,7 @@ There are two ways to install taosdump:
2. backup multiple specified databases: use `-D db1,db2,... ` parameters;
2. backup multiple specified databases: use `-D db1,db2,... ` parameters;
3. back up some super or normal tables in the specified database: use `-dbname stbname1 stbname2 tbname1 tbname2 ... ` parameters. Note that the first parameter of this input sequence is the database name, and only one database is supported. The second and subsequent parameters are the names of super or normal tables in that database, separated by spaces.
3. back up some super or normal tables in the specified database: use `-dbname stbname1 stbname2 tbname1 tbname2 ... ` parameters. Note that the first parameter of this input sequence is the database name, and only one database is supported. The second and subsequent parameters are the names of super or normal tables in that database, separated by spaces.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` command-line parameter.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` command-line parameter.
5. Loose mode backup: taosdump version 1.4.1 onwards provides `-n` and `-L` parameters for backing up data without using escape characters and "loose" mode, which can reduce the number of backups if table names, column names, tag names do not use escape characters. This can also reduce the backup data time and backup data footprint. If you are unsure about using `-n` and `-L` conditions, please use the default parameters for "strict" mode backup. See the [official documentation](/taos-sql/escape) for a description of escaped characters.
5. Loose mode backup: taosdump version 1.4.1 onwards provides `-n` and `-L` parameters for backing up data without using escape characters and "loose" mode, which can reduce the number of backups if table names, column names, tag names do not use escape characters. This can also reduce the backup data time and backup data footprint. If you are unsure about using `-n` and `-L` conditions, please use the default parameters for "strict" mode backup. See the [official documentation](https://docs.tdengine.com/taos-sql/escape/) for a description of escaped characters.
TDengine CLI will display a welcome message and version information if it successfully connected to the TDengine service. If it fails, TDengine CLI will print an error message. See [FAQ](/train-faq/faq) to solve the problem of terminal connection failure to the server. The TDengine CLI prompts as follows:
TDengine CLI will display a welcome message and version information if it successfully connected to the TDengine service. If it fails, TDengine CLI will print an error message. The TDengine CLI prompts as follows:
```
```
Welcome to the TDengine shell from Linux, Client Version:2.6.0.4
Welcome to the TDengine shell from Linux, Client Version:3.0.0.0
Copyright (c) 2022 by TAOS Data, Inc. All rights reserved.
Copyright (c) 2022 by TAOS Data, Inc. All rights reserved.
Successfully connect to cloud.tdengine.com:8085 in restful mode
Successfully connect to cloud.tdengine.com:8085 in restful mode
@@ -9,9 +9,11 @@ description: "taosBenchmark (once called taosdemo ) is a tool for testing the pe
...
@@ -9,9 +9,11 @@ description: "taosBenchmark (once called taosdemo ) is a tool for testing the pe
taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can be configured to generate user defined databases, supertables, subtables, and the time series data to populate these for performance benchmarking. taosBenchmark is highly configurable and some of the configurations include the time interval for inserting data, the number of working threads and the capability to insert disordered data. The installer provides taosdemo as a soft link to taosBenchmark for compatibility with past users.
taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can be configured to generate user defined databases, supertables, subtables, and the time series data to populate these for performance benchmarking. taosBenchmark is highly configurable and some of the configurations include the time interval for inserting data, the number of working threads and the capability to insert disordered data. The installer provides taosdemo as a soft link to taosBenchmark for compatibility with past users.
**Please be noted that in the context of TDengine cloud service, non privileged user can't create database using any tool, including taosBenchmark. The database needs to be firstly created in the data explorer in TDengine cloud service console. For any content about creating database in this document, the user needs to ignore and create the database manually inside TDengine cloud service.**
To use taosBenchmark, you need to download and install [taosTools](https://tdengine.com/assets-download/cloud/taosTools-2.1.3-Linux-x64.tar.gz). Before installing taosTools, please firstly download and install [TDengine CLI](https://docs.tdengine.com/cloud/tools/cli/#installation).
Decompress the package and install.
Decompress the package and install.
```
```
...
@@ -29,27 +31,6 @@ taosBenchmark supports the complete performance testing of TDengine by providing
...
@@ -29,27 +31,6 @@ taosBenchmark supports the complete performance testing of TDengine by providing
**Make sure that the TDengine cluster is running correctly before running taosBenchmark. **
**Make sure that the TDengine cluster is running correctly before running taosBenchmark. **
### Run without command-line arguments
Execute the following commands to quickly experience taosBenchmark's default configuration-based write performance testing of TDengine.
```bash
export TDENGINE_CLOUD_DSN="<DSN>"
taosBenchmark
```
When run without parameters, taosBenchmark connects to the TDengine cluster specified in `/etc/taos` by default and creates a database named `test`, a super table named `meters` under the test database, and 10,000 tables under the super table with 10,000 records written to each table. Note that if there is already a database named "test" this command will delete it first and create a new database.
### Run with command-line configuration parameters
The `-f <json_file>` argument cannot be used when running taosBenchmark with command-line parameters and controlling its behavior. Users must specify all configuration parameters from the command-line. The following is an example of testing taosBenchmark writing performance using the command-line approach.
```bash
taosBenchmark -I stmt -n 200 -t 100
```
Using the above command, `taosBenchmark` will create a database named `test`, create a super table `meters` in it, create 100 sub-tables in the super table and insert 200 records for each sub-table using parameter binding.
### Run with the configuration file
### Run with the configuration file
A sample configuration file is provided in the taosBenchmark installation package under `<install_directory>/examples/taosbenchmark-json`.
A sample configuration file is provided in the taosBenchmark installation package under `<install_directory>/examples/taosbenchmark-json`.
"sql":"select avg(current) from meters where location = 'beijing';",
"result":"./subscribe_res0.txt"
}
]
},
"super_table_query":{
"stblname":"meters",
"threads":1,
"mode":"sync",
"interval":1000,
"restart":"yes",
"keepProgress":"yes",
"sqls":[
{
"sql":"select phase from xxxx where groupid > 3;",
"result":"./subscribe_res1.txt"
}
]
}
}
</details>
```
## Command-line argument in detailed
## Command-line argument in detailed
...
@@ -142,9 +270,6 @@ taosBenchmark -f json-file
...
@@ -142,9 +270,6 @@ taosBenchmark -f json-file
-**-n/--records <recordNum\>** :
-**-n/--records <recordNum\>** :
The default value of the number of records inserted in each sub-table is 10000.
The default value of the number of records inserted in each sub-table is 10000.
-**-d/--database <dbName\>** :
The name of the database used, the default value is `test`.
-**-b/--data-type <colType\>** :
-**-b/--data-type <colType\>** :
specify the type of the data columns of the super table. It defaults to three columns of type FLOAT, INT, and FLOAT if not used.
specify the type of the data columns of the super table. It defaults to three columns of type FLOAT, INT, and FLOAT if not used.
...
@@ -198,9 +323,6 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
...
@@ -198,9 +323,6 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
-**-F/--prepare_rand <Num\>** :
-**-F/--prepare_rand <Num\>** :
Specify the number of unique values in the generated random data. A value of 1 means that all data are equal. The default value is 10000.
Specify the number of unique values in the generated random data. A value of 1 means that all data are equal. The default value is 10000.
-**-a/--replica <replicaNum\>** :
Specify the number of replicas when creating the database. The default value is 1.
-**-V/--version** :
-**-V/--version** :
Show version information only. Users should not use it with other parameters.
Show version information only. Users should not use it with other parameters.
...
@@ -228,14 +350,6 @@ The parameters listed in this section apply to all function modes.
...
@@ -228,14 +350,6 @@ The parameters listed in this section apply to all function modes.
`filetype` must be set to `insert` in the insertion scenario. See [General Configuration Parameters](#General Configuration Parameters)
`filetype` must be set to `insert` in the insertion scenario. See [General Configuration Parameters](#General Configuration Parameters)
#### Database related configuration parameters
The parameters related to database creation are configured in `dbinfo` in the json configuration file, as follows. The other parameters correspond to the database parameters specified when `create database` in [../../taos-sql/database].
-**name**: specify the name of the database.
-**drop**: indicate whether to delete the database before inserting. The default is true.
#### Stream processing related configuration parameters
#### Stream processing related configuration parameters
The parameters for creating streams are configured in `stream` in the json configuration file, as shown below.
The parameters for creating streams are configured in `stream` in the json configuration file, as shown below.
To use taosdump, you need to download and install [taosTools](https://tdengine.com/assets-download/cloud/taosTools-2.1.3-Linux-x64.tar.gz). Before installing taosTools, please firstly download and install [TDengine CLI](https://docs.tdengine.com/cloud/tools/cli/#installation).
2. backup multiple specified databases: use `-D db1,db2,... ` parameters;
2. backup multiple specified databases: use `-D db1,db2,... ` parameters;
3. back up some super or normal tables in the specified database: use `dbname stbname1 stbname2 tbname1 tbname2 ... ` parameters. Note that the first parameter of this input sequence is the database name, and only one database is supported. The second and subsequent parameters are the names of super or normal tables in that database, separated by spaces.
3. back up some super or normal tables in the specified database: use `dbname stbname1 stbname2 tbname1 tbname2 ... ` parameters. Note that the first parameter of this input sequence is the database name, and only one database is supported. The second and subsequent parameters are the names of super or normal tables in that database, separated by spaces.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` command-line parameter.
4. back up the system log database: TDengine clusters usually contain a system database named `log`. The data in this database is the data that TDengine runs itself, and the taosdump will not back up the log database by default. If users need to back up the log database, users can use the `-a` or `-allow-sys` command-line parameter.
5. Loose mode backup: taosdump version 1.4.1 onwards provides `-n` and `-L` parameters for backing up data without using escape characters and "loose" mode, which can reduce the number of backups if table names, column names, tag names do not use escape characters. This can also reduce the backup data time and backup data footprint. If you are unsure about using `-n` and `-L` conditions, please use the default parameters for "strict" mode backup. See the [official documentation](/taos-sql/escape) for a description of escaped characters.
5. Loose mode backup: taosdump version 1.4.1 onwards provides `-n` and `-L` parameters for backing up data without using escape characters and "loose" mode, which can reduce the number of backups if table names, column names, tag names do not use escape characters. This can also reduce the backup data time and backup data footprint. If you are unsure about using `-n` and `-L` conditions, please use the default parameters for "strict" mode backup. See the [official documentation](https://docs.tdengine.com/taos-sql/escape/) for a description of escaped characters.