提交 3519e917 编写于 作者: D dapan1121

Merge remote-tracking branch 'origin/main' into 3.0

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: check-json
- id: end-of-file-fixer
- id: trailing-whitespace
repos:
- repo: https://github.com/psf/black
rev: stable
hooks:
- id: black
repos:
- repo: https://github.com/pocc/pre-commit-hooks
rev: master
hooks:
- id: cppcheck
args: ["--error-exitcode=0"]
repos:
- repo: https://github.com/crate-ci/typos
rev: v1.15.7
hooks:
- id: typos
...@@ -33,7 +33,7 @@ The below SQL statement is used to insert one row into table "d1001". ...@@ -33,7 +33,7 @@ The below SQL statement is used to insert one row into table "d1001".
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31); INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31);
``` ```
`ts1` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert). `ts1` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detail, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
### Insert Multiple Rows ### Insert Multiple Rows
...@@ -43,7 +43,7 @@ Multiple rows can be inserted in a single SQL statement. The example below inser ...@@ -43,7 +43,7 @@ Multiple rows can be inserted in a single SQL statement. The example below inser
INSERT INTO d1001 VALUES (ts2, 10.2, 220, 0.23) (ts2, 10.3, 218, 0.25); INSERT INTO d1001 VALUES (ts2, 10.2, 220, 0.23) (ts2, 10.3, 218, 0.25);
``` ```
`ts1` and `ts2` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert). `ts1` and `ts2` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detail, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
### Insert into Multiple Tables ### Insert into Multiple Tables
...@@ -53,7 +53,7 @@ Data can be inserted into multiple tables in the same SQL statement. The example ...@@ -53,7 +53,7 @@ Data can be inserted into multiple tables in the same SQL statement. The example
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31) (ts2, 12.6, 218, 0.33) d1002 VALUES (ts3, 12.3, 221, 0.31); INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31) (ts2, 12.6, 218, 0.33) d1002 VALUES (ts3, 12.3, 221, 0.31);
``` ```
`ts1`, `ts2` and `ts3` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert). `ts1`, `ts2` and `ts3` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detail, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
For more details about `INSERT` please refer to [INSERT](/taos-sql/insert). For more details about `INSERT` please refer to [INSERT](/taos-sql/insert).
......
...@@ -17,7 +17,7 @@ When you create a user-defined function, you must implement standard interface f ...@@ -17,7 +17,7 @@ When you create a user-defined function, you must implement standard interface f
- For aggregate functions, implement the `aggfn_start`, `aggfn`, and `aggfn_finish` interface functions. - For aggregate functions, implement the `aggfn_start`, `aggfn`, and `aggfn_finish` interface functions.
- To initialize your function, implement the `udf_init` function. To terminate your function, implement the `udf_destroy` function. - To initialize your function, implement the `udf_init` function. To terminate your function, implement the `udf_destroy` function.
There are strict naming conventions for these interface functions. The names of the start, finish, init, and destroy interfaces must be <udf-name\>_start, <udf-name\>_finish, <udf-name\>_init, and <udf-name\>_destroy, respectively. Replace `scalarfn`, `aggfn`, and `udf` with the name of your user-defined function. There are strict naming conventions for these interface functions. The names of the start, finish, init, and destroy interfaces must be `_start`, `_finish`, `_init`, and `_destroy`, respectively. Replace `scalarfn`, `aggfn`, and `udf` with the name of your user-defined function.
### Implementing a Scalar Function in C ### Implementing a Scalar Function in C
The implementation of a scalar function is described as follows: The implementation of a scalar function is described as follows:
...@@ -318,7 +318,7 @@ The implementation of a scalar UDF is described as follows: ...@@ -318,7 +318,7 @@ The implementation of a scalar UDF is described as follows:
def process(input: datablock) -> tuple[output_type]: def process(input: datablock) -> tuple[output_type]:
``` ```
Description: this function prcesses datablock, which is the input; you can use datablock.data(row, col) to access the python object at location(row,col); the output is a tuple object consisted of objects of type outputtype Description: this function processes datablock, which is the input; you can use datablock.data(row, col) to access the python object at location(row,col); the output is a tuple object consisted of objects of type outputtype
#### Aggregate UDF Interface #### Aggregate UDF Interface
...@@ -356,7 +356,7 @@ def process(input: datablock) -> tuple[output_type]: ...@@ -356,7 +356,7 @@ def process(input: datablock) -> tuple[output_type]:
# return tuple object consisted of object of type outputtype # return tuple object consisted of object of type outputtype
``` ```
Note:process() must be implemeted, init() and destroy() must be defined too but they can do nothing. Note:process() must be implemented, init() and destroy() must be defined too but they can do nothing.
#### Aggregate Template #### Aggregate Template
...@@ -377,7 +377,7 @@ def finish(buf: bytes) -> output_type: ...@@ -377,7 +377,7 @@ def finish(buf: bytes) -> output_type:
#return obj of type outputtype #return obj of type outputtype
``` ```
Note: aggregate UDF requires init(), destroy(), start(), reduce() and finish() to be impemented. start() generates the initial result in buffer, then the input data is divided into multiple row data blocks, reduce() is invoked for each data block `inputs` and intermediate `buf`, finally finish() is invoked to generate final result from the intermediate result `buf`. Note: aggregate UDF requires init(), destroy(), start(), reduce() and finish() to be implemented. start() generates the initial result in buffer, then the input data is divided into multiple row data blocks, reduce() is invoked for each data block `inputs` and intermediate `buf`, finally finish() is invoked to generate final result from the intermediate result `buf`.
### Data Mapping between TDengine SQL and Python UDF ### Data Mapping between TDengine SQL and Python UDF
...@@ -559,7 +559,7 @@ Note: Prior to TDengine 3.0.5.0 (excluding), updating a UDF requires to restart ...@@ -559,7 +559,7 @@ Note: Prior to TDengine 3.0.5.0 (excluding), updating a UDF requires to restart
#### Sample 3: UDF with n arguments #### Sample 3: UDF with n arguments
A UDF which accepts n intergers, likee (x1, x2, ..., xn) and output the sum of the product of each value and its sequence number: 1 * x1 + 2 * x2 + ... + n * xn. If there is `null` in the input, then the result is `null`. The difference from sample 1 is that it can accept any number of columns as input and process each column. Assume the program is written in /root/udf/nsum.py: A UDF which accepts n integers, likee (x1, x2, ..., xn) and output the sum of the product of each value and its sequence number: 1 * x1 + 2 * x2 + ... + n * xn. If there is `null` in the input, then the result is `null`. The difference from sample 1 is that it can accept any number of columns as input and process each column. Assume the program is written in /root/udf/nsum.py:
```python ```python
def init(): def init():
...@@ -607,7 +607,7 @@ Query OK, 4 row(s) in set (0.010653s) ...@@ -607,7 +607,7 @@ Query OK, 4 row(s) in set (0.010653s)
#### Sample 4: Utilize 3rd party package #### Sample 4: Utilize 3rd party package
A UDF which accepts a timestamp and output the next closed Sunday. This sample requires to use third party package `moment`, you need to install it firslty. A UDF which accepts a timestamp and output the next closed Sunday. This sample requires to use third party package `moment`, you need to install it firstly.
```shell ```shell
pip3 install moment pip3 install moment
...@@ -701,7 +701,7 @@ Query OK, 4 row(s) in set (1.011474s) ...@@ -701,7 +701,7 @@ Query OK, 4 row(s) in set (1.011474s)
#### Sample 5: Aggregate Function #### Sample 5: Aggregate Function
An aggregate function which calculates the difference of the maximum and the minimum in a column. An aggregate funnction takes multiple rows as input and output only one data. The execution process of an aggregate UDF is like map-reduce, the framework divides the input into multiple parts, each mapper processes one block and the reducer aggregates the result of the mappers. The reduce() of Python UDF has the functionality of both map() and reduce(). The reduce() takes two arguments: the data to be processed; and the result of other tasks executing reduce(). For exmaple, assume the code is in `/root/udf/myspread.py`. An aggregate function which calculates the difference of the maximum and the minimum in a column. An aggregate funnction takes multiple rows as input and output only one data. The execution process of an aggregate UDF is like map-reduce, the framework divides the input into multiple parts, each mapper processes one block and the reducer aggregates the result of the mappers. The reduce() of Python UDF has the functionality of both map() and reduce(). The reduce() takes two arguments: the data to be processed; and the result of other tasks executing reduce(). For example, assume the code is in `/root/udf/myspread.py`.
```python ```python
import io import io
...@@ -755,7 +755,7 @@ In this example, we implemented an aggregate function, and added some logging. ...@@ -755,7 +755,7 @@ In this example, we implemented an aggregate function, and added some logging.
2. log() is the function for logging, it converts the input object to string and output with an end of line 2. log() is the function for logging, it converts the input object to string and output with an end of line
3. destroy() closes the log file \ 3. destroy() closes the log file \
4. start() returns the initial buffer for storing the intermediate result 4. start() returns the initial buffer for storing the intermediate result
5. reduce() processes each daa block and aggregates the result 5. reduce() processes each data block and aggregates the result
6. finish() converts the final buffer() to final result\ 6. finish() converts the final buffer() to final result\
Create the UDF. Create the UDF.
......
...@@ -672,7 +672,7 @@ If you input a specific column, the number of non-null values in the column is r ...@@ -672,7 +672,7 @@ If you input a specific column, the number of non-null values in the column is r
ELAPSED(ts_primary_key [, time_unit]) ELAPSED(ts_primary_key [, time_unit])
``` ```
**Description**: `elapsed` function can be used to calculate the continuous time length in which there is valid data. If it's used with `INTERVAL` clause, the returned result is the calculated time length within each time window. If it's used without `INTERVAL` caluse, the returned result is the calculated time length within the specified time range. Please be noted that the return value of `elapsed` is the number of `time_unit` in the calculated time length. **Description**: `elapsed` function can be used to calculate the continuous time length in which there is valid data. If it's used with `INTERVAL` clause, the returned result is the calculated time length within each time window. If it's used without `INTERVAL` clause, the returned result is the calculated time length within the specified time range. Please be noted that the return value of `elapsed` is the number of `time_unit` in the calculated time length.
**Return value type**: Double if the input value is not NULL; **Return value type**: Double if the input value is not NULL;
...@@ -999,18 +999,14 @@ SAMPLE(expr, k) ...@@ -999,18 +999,14 @@ SAMPLE(expr, k)
**Description**: _k_ sampling values of a specific column. The applicable range of _k_ is [1,1000]. **Description**: _k_ sampling values of a specific column. The applicable range of _k_ is [1,1000].
**Return value type**: Same as the column being operated plus the associated timestamp **Return value type**: Same as the column being operated
**Applicable data types**: Any data type except for tags of STable **Applicable data types**: Any data type
**Applicable nested query**: Inner query and Outer query **Applicable nested query**: Inner query and Outer query
**Applicable table types**: standard tables and supertables **Applicable table types**: standard tables and supertables
**More explanations**:
- This function cannot be used in expression calculation.
### TAIL ### TAIL
...@@ -1055,11 +1051,11 @@ TOP(expr, k) ...@@ -1055,11 +1051,11 @@ TOP(expr, k)
UNIQUE(expr) UNIQUE(expr)
``` ```
**Description**: The values that occur the first time in the specified column. The effect is similar to `distinct` keyword, but it can also be used to match tags or timestamp. The first occurrence of a timestamp or tag is used. **Description**: The values that occur the first time in the specified column. The effect is similar to `distinct` keyword.
**Return value type**:Same as the data type of the column being operated upon **Return value type**:Same as the data type of the column being operated upon
**Applicable column types**: Any data types except for timestamp **Applicable column types**: Any data types
**Applicable table types**: table, STable **Applicable table types**: table, STable
......
...@@ -21,7 +21,7 @@ part_list can be any scalar expression, such as a column, constant, scalar funct ...@@ -21,7 +21,7 @@ part_list can be any scalar expression, such as a column, constant, scalar funct
A PARTITION BY clause is processed as follows: A PARTITION BY clause is processed as follows:
- The PARTITION BY clause must occur after the WHERE clause - The PARTITION BY clause must occur after the WHERE clause
- The PARTITION BY caluse partitions the data according to the specified dimensions, then perform computation on each partition. The performed computation is determined by the rest of the statement - a window clause, GROUP BY clause, or SELECT clause. - The PARTITION BY clause partitions the data according to the specified dimensions, then perform computation on each partition. The performed computation is determined by the rest of the statement - a window clause, GROUP BY clause, or SELECT clause.
- The PARTITION BY clause can be used together with a window clause or GROUP BY clause. In this case, the window or GROUP BY clause takes effect on every partition. For example, the following statement partitions the table by the location tag, performs downsampling over a 10 minute window, and returns the maximum value: - The PARTITION BY clause can be used together with a window clause or GROUP BY clause. In this case, the window or GROUP BY clause takes effect on every partition. For example, the following statement partitions the table by the location tag, performs downsampling over a 10 minute window, and returns the maximum value:
```sql ```sql
......
...@@ -36,7 +36,8 @@ REST connection supports all platforms that can run Java. ...@@ -36,7 +36,8 @@ REST connection supports all platforms that can run Java.
| taos-jdbcdriver version | major changes | TDengine version | | taos-jdbcdriver version | major changes | TDengine version |
| :---------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------: | | :---------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------: |
| 3.2.1 | subscription add seek function | 3.0.5.0 or later | | 3.2.3 | Fixed resultSet data parsing failure in some cases | 3.0.5.0 or later |
| 3.2.2 | subscription add seek function | 3.0.5.0 or later |
| 3.2.1 | JDBC REST connection supports schemaless/prepareStatement over WebSocket | 3.0.3.0 or later | | 3.2.1 | JDBC REST connection supports schemaless/prepareStatement over WebSocket | 3.0.3.0 or later |
| 3.2.0 | This version has been deprecated | - | | 3.2.0 | This version has been deprecated | - |
| 3.1.0 | JDBC REST connection supports subscription over WebSocket | - | | 3.1.0 | JDBC REST connection supports subscription over WebSocket | - |
...@@ -284,9 +285,9 @@ The configuration parameters in the URL are as follows: ...@@ -284,9 +285,9 @@ The configuration parameters in the URL are as follows:
- batchfetch: true: pulls result sets in batches when executing queries; false: pulls result sets row by row. The default value is: false. batchfetch uses HTTP for data transfer. JDBC REST supports batch pulls. taos-jdbcdriver and TDengine transfer data via WebSocket connection. Compared with HTTP, WebSocket enables JDBC REST connection to support large data volume querying and improve query performance. - batchfetch: true: pulls result sets in batches when executing queries; false: pulls result sets row by row. The default value is: false. batchfetch uses HTTP for data transfer. JDBC REST supports batch pulls. taos-jdbcdriver and TDengine transfer data via WebSocket connection. Compared with HTTP, WebSocket enables JDBC REST connection to support large data volume querying and improve query performance.
- charset: specify the charset to parse the string, this parameter is valid only when set batchfetch to true. - charset: specify the charset to parse the string, this parameter is valid only when set batchfetch to true.
- batchErrorIgnore: true: when executing executeBatch of Statement, if one SQL execution fails in the middle, continue to execute the following SQL. false: no longer execute any statement after the failed SQL. The default value is: false. - batchErrorIgnore: true: when executing executeBatch of Statement, if one SQL execution fails in the middle, continue to execute the following SQL. false: no longer execute any statement after the failed SQL. The default value is: false.
- httpConnectTimeout: REST connection timeout in milliseconds, the default value is 5000 ms. - httpConnectTimeout: REST connection timeout in milliseconds, the default value is 60000 ms.
- httpSocketTimeout: socket timeout in milliseconds, the default value is 5000 ms. It only takes effect when batchfetch is false. - httpSocketTimeout: socket timeout in milliseconds, the default value is 60000 ms. It only takes effect when batchfetch is false.
- messageWaitTimeout: message transmission timeout in milliseconds, the default value is 3000 ms. It only takes effect when batchfetch is true. - messageWaitTimeout: message transmission timeout in milliseconds, the default value is 60000 ms. It only takes effect when batchfetch is true.
- useSSL: connecting Securely Using SSL. true: using SSL connection, false: not using SSL connection. - useSSL: connecting Securely Using SSL. true: using SSL connection, false: not using SSL connection.
- httpPoolSize: size of REST concurrent requests. The default value is 20. - httpPoolSize: size of REST concurrent requests. The default value is 20.
...@@ -352,9 +353,9 @@ The configuration parameters in properties are as follows. ...@@ -352,9 +353,9 @@ The configuration parameters in properties are as follows.
- TSDBDriver.PROPERTY_KEY_CHARSET: In the character set used by the client, the default value is the system character set. - TSDBDriver.PROPERTY_KEY_CHARSET: In the character set used by the client, the default value is the system character set.
- TSDBDriver.PROPERTY_KEY_LOCALE: this only takes effect when using JDBC native connection. Client language environment, the default value is system current locale. - TSDBDriver.PROPERTY_KEY_LOCALE: this only takes effect when using JDBC native connection. Client language environment, the default value is system current locale.
- TSDBDriver.PROPERTY_KEY_TIME_ZONE: only takes effect when using JDBC native connection. In the time zone used by the client, the default value is the system's current time zone. - TSDBDriver.PROPERTY_KEY_TIME_ZONE: only takes effect when using JDBC native connection. In the time zone used by the client, the default value is the system's current time zone.
- TSDBDriver.HTTP_CONNECT_TIMEOUT: REST connection timeout in milliseconds, the default value is 5000 ms. It only takes effect when using JDBC REST connection. - TSDBDriver.HTTP_CONNECT_TIMEOUT: REST connection timeout in milliseconds, the default value is 60000 ms. It only takes effect when using JDBC REST connection.
- TSDBDriver.HTTP_SOCKET_TIMEOUT: socket timeout in milliseconds, the default value is 5000 ms. It only takes effect when using JDBC REST connection and batchfetch is false. - TSDBDriver.HTTP_SOCKET_TIMEOUT: socket timeout in milliseconds, the default value is 60000 ms. It only takes effect when using JDBC REST connection and batchfetch is false.
- TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: message transmission timeout in milliseconds, the default value is 3000 ms. It only takes effect when using JDBC REST connection and batchfetch is true. - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: message transmission timeout in milliseconds, the default value is 60000 ms. It only takes effect when using JDBC REST connection and batchfetch is true.
- TSDBDriver.PROPERTY_KEY_USE_SSL: connecting Securely Using SSL. true: using SSL connection, false: not using SSL connection. It only takes effect when using JDBC REST connection. - TSDBDriver.PROPERTY_KEY_USE_SSL: connecting Securely Using SSL. true: using SSL connection, false: not using SSL connection. It only takes effect when using JDBC REST connection.
- TSDBDriver.HTTP_POOL_SIZE: size of REST concurrent requests. The default value is 20. - TSDBDriver.HTTP_POOL_SIZE: size of REST concurrent requests. The default value is 20.
For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only). For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only).
......
...@@ -31,42 +31,57 @@ REST connections are supported on all platforms that can run Go. ...@@ -31,42 +31,57 @@ REST connections are supported on all platforms that can run Go.
Please refer to [version support list](https://github.com/taosdata/driver-go#remind) Please refer to [version support list](https://github.com/taosdata/driver-go#remind)
## Supported features ## Handling exceptions
### Native connections If it is a TDengine error, you can get the error code and error information in the following ways.
```go
A "native connection" is established by the connector directly to the TDengine instance via the TDengine client driver (taosc). The supported functional features are: // import "github.com/taosdata/driver-go/v3/errors"
if err != nil {
* Normal queries tError, is := err.(*errors.TaosError)
* Continuous queries if is {
* Subscriptions fmt.Println("errorCode:", int(tError.Code))
* Schemaless interface fmt.Println("errorMessage:", tError.ErrStr)
* Parameter binding interface } else {
fmt.Println(err.Error())
### REST connection }
}
A "REST connection" is a connection between the application and the TDengine instance via the REST API provided by the taosAdapter component. The following features are supported: ```
* Normal queries ## TDengine DataType vs. Go DataType
* Continuous queries
| TDengine DataType | Go Type |
|-------------------|-----------|
| TIMESTAMP | time.Time |
| TINYINT | int8 |
| SMALLINT | int16 |
| INT | int32 |
| BIGINT | int64 |
| TINYINT UNSIGNED | uint8 |
| SMALLINT UNSIGNED | uint16 |
| INT UNSIGNED | uint32 |
| BIGINT UNSIGNED | uint64 |
| FLOAT | float32 |
| DOUBLE | float64 |
| BOOL | bool |
| BINARY | string |
| NCHAR | string |
| JSON | []byte |
**Note**: Only TAG supports JSON types
## Installation Steps ## Installation Steps
### Pre-installation preparation ### Pre-installation preparation
* Install Go development environment (Go 1.14 and above, GCC 4.8.5 and above) * Install Go development environment (Go 1.14 and above, GCC 4.8.5 and above)
- If you use the native connector, please install the TDengine client driver. Please refer to [Install Client Driver](/reference/connector/#install-client-driver) for specific steps * If you use the native connector, please install the TDengine client driver. Please refer to [Install Client Driver](/reference/connector/#install-client-driver) for specific steps
Configure the environment variables and check the command. Configure the environment variables and check the command.
* ```go env``` * ```go env```
* ```gcc -v``` * ```gcc -v```
### Use go get to install ### Install the connectors
`go get -u github.com/taosdata/driver-go/v3@latest`
### Manage with go mod
1. Initialize the project with the `go mod` command. 1. Initialize the project with the `go mod` command.
...@@ -98,8 +113,6 @@ Configure the environment variables and check the command. ...@@ -98,8 +113,6 @@ Configure the environment variables and check the command.
## Establishing a connection ## Establishing a connection
### Data source name (DSN)
Data source names have a standard format, e.g. [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php), but no type prefix (square brackets indicate optionally): Data source names have a standard format, e.g. [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php), but no type prefix (square brackets indicate optionally):
``` text ``` text
...@@ -111,9 +124,7 @@ DSN in full form. ...@@ -111,9 +124,7 @@ DSN in full form.
```text ```text
username:password@protocol(address)/dbname?param=value username:password@protocol(address)/dbname?param=value
``` ```
### Connecting via connector <Tabs defaultValue="rest" groupId="connect">
<Tabs defaultValue="rest">
<TabItem value="native" label="native connection"> <TabItem value="native" label="native connection">
_taosSql_ implements Go's `database/sql/driver` interface via cgo. You can use the [`database/sql`](https://golang.org/pkg/database/sql/) interface by simply introducing the driver. _taosSql_ implements Go's `database/sql/driver` interface via cgo. You can use the [`database/sql`](https://golang.org/pkg/database/sql/) interface by simply introducing the driver.
...@@ -209,340 +220,902 @@ func main() { ...@@ -209,340 +220,902 @@ func main() {
</TabItem> </TabItem>
</Tabs> </Tabs>
## Usage examples ### Specify the URL and Properties to get the connection
### Write data
#### SQL Write The Go connector does not support this feature
<GoInsert /> ### Priority of configuration parameters
#### InfluxDB line protocol write The Go connector does not support this feature
<GoInfluxLine /> ## Usage examples
#### OpenTSDB Telnet line protocol write ### Create database and tables
<GoOpenTSDBTelnet /> ```go
var taosDSN = "root:taosdata@tcp(localhost:6030)/"
taos, err := sql.Open("taosSql", taosDSN)
if err != nil {
log.Fatalln("failed to connect TDengine, err:", err)
}
defer taos.Close()
_, err := taos.Exec("CREATE DATABASE power")
if err != nil {
log.Fatalln("failed to create database, err:", err)
}
_, err = taos.Exec("CREATE STABLE power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)")
if err != nil {
log.Fatalln("failed to create stable, err:", err)
}
```
#### OpenTSDB JSON line protocol write ### Insert data
<GoOpenTSDBJson /> <GoInsert />
### Query data ### Querying data
<GoQuery /> <GoQuery />
### More sample programs ### execute SQL with reqId
* [sample program](https://github.com/taosdata/driver-go/tree/3.0/examples)
## Usage limitations This reqId can be used to request link tracing.
Since the REST interface is stateless, the `use db` syntax will not work. You need to put the db name into the SQL command, e.g. `create table if not exists tb1 (ts timestamp, a int)` to `create table if not exists test.tb1 (ts timestamp, a int)` otherwise it will report the error `[0x217] Database not specified or available`. ```go
db, err := sql.Open("taosSql", "root:taosdata@tcp(localhost:6030)/")
if err != nil {
panic(err)
}
defer db.Close()
ctx := context.WithValue(context.Background(), common.ReqIDKey, common.GetReqID())
_, err = db.ExecContext(ctx, "create database if not exists example_taos_sql")
if err != nil {
panic(err)
}
```
You can also put the db name in the DSN by changing `root:taosdata@http(localhost:6041)/` to `root:taosdata@http(localhost:6041)/test`. Executing the `create database` statement when the specified db does not exist will not report an error while executing other queries or writing against that db will report an error. ### Writing data via parameter binding
The complete example is as follows. <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="native connection">
```go ```go
package main package main
import ( import (
"database/sql"
"fmt"
"time" "time"
_ "github.com/taosdata/driver-go/v3/taosRestful" "github.com/taosdata/driver-go/v3/af"
"github.com/taosdata/driver-go/v3/common"
"github.com/taosdata/driver-go/v3/common/param"
) )
func main() { func main() {
var taosDSN = "root:taosdata@http(localhost:6041)/test" db, err := af.Open("", "root", "taosdata", "", 0)
taos, err := sql.Open("taosRestful", taosDSN)
if err != nil { if err != nil {
fmt.Println("failed to connect TDengine, err:", err) panic(err)
return
} }
defer taos.Close() defer db.Close()
taos.Exec("create database if not exists test") _, err = db.Exec("create database if not exists example_stmt")
taos.Exec("create table if not exists tb1 (ts timestamp, a int)")
_, err = taos.Exec("insert into tb1 values(now, 0)(now+1s,1)(now+2s,2)(now+3s,3)")
if err != nil { if err != nil {
fmt.Println("failed to insert, err:", err) panic(err)
return
} }
rows, err := taos.Query("select * from tb1") _, err = db.Exec("create table if not exists example_stmt.tb1(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")")
if err != nil { if err != nil {
fmt.Println("failed to select from table, err:", err) panic(err)
return
} }
stmt := db.InsertStmt()
defer rows.Close() err = stmt.Prepare("insert into example_stmt.tb1 values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
for rows.Next() { if err != nil {
var r struct { panic(err)
ts time.Time
a int
} }
err := rows.Scan(&r.ts, &r.a) now := time.Now()
params := make([]*param.Param, 14)
params[0] = param.NewParam(2).
AddTimestamp(now, common.PrecisionMilliSecond).
AddTimestamp(now.Add(time.Second), common.PrecisionMilliSecond)
params[1] = param.NewParam(2).AddBool(true).AddNull()
params[2] = param.NewParam(2).AddTinyint(2).AddNull()
params[3] = param.NewParam(2).AddSmallint(3).AddNull()
params[4] = param.NewParam(2).AddInt(4).AddNull()
params[5] = param.NewParam(2).AddBigint(5).AddNull()
params[6] = param.NewParam(2).AddUTinyint(6).AddNull()
params[7] = param.NewParam(2).AddUSmallint(7).AddNull()
params[8] = param.NewParam(2).AddUInt(8).AddNull()
params[9] = param.NewParam(2).AddUBigint(9).AddNull()
params[10] = param.NewParam(2).AddFloat(10).AddNull()
params[11] = param.NewParam(2).AddDouble(11).AddNull()
params[12] = param.NewParam(2).AddBinary([]byte("binary")).AddNull()
params[13] = param.NewParam(2).AddNchar("nchar").AddNull()
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(6).
AddNchar(5)
err = stmt.BindParam(params, paramTypes)
if err != nil { if err != nil {
fmt.Println("scan error:\n", err) panic(err)
return }
err = stmt.AddBatch()
if err != nil {
panic(err)
}
err = stmt.Execute()
if err != nil {
panic(err)
} }
fmt.Println(r.ts, r.a) err = stmt.Close()
if err != nil {
panic(err)
} }
// select * from example_stmt.tb1
} }
``` ```
## Frequently Asked Questions </TabItem>
<TabItem value="WebSocket" label="WebSocket connection">
1. bind interface in database/sql crashes
REST does not support parameter binding related interface. It is recommended to use `db.Exec` and `db.Query`.
2. error `[0x217] Database not specified or available` after executing other statements with `use db` statement
The execution of SQL command in the REST interface is not contextual, so using `use db` statement will not work, see the usage restrictions section above.
3. use `taosSql` without error but use `taosRestful` with error `[0x217] Database not specified or available`
Because the REST interface is stateless, using the `use db` statement will not take effect. See the usage restrictions section above.
4. `readBufferSize` parameter has no significant effect after being increased
Increasing `readBufferSize` will reduce the number of `syscall` calls when fetching results. If the query result is smaller, modifying this parameter will not improve performance significantly. If you increase the parameter value too much, the bottleneck will be parsing JSON data. If you need to optimize the query speed, you must adjust the value based on the actual situation to achieve the best query performance.
5. `disableCompression` parameter is set to `false` when the query efficiency is reduced
When set `disableCompression` parameter to `false`, the query result will be compressed by `gzip` and then transmitted, so you have to decompress the data by `gzip` after getting it.
6. `go get` command can't get the package, or timeout to get the package
Set Go proxy `go env -w GOPROXY=https://goproxy.cn,direct`.
## Common APIs
### database/sql API
* `sql.Open(DRIVER_NAME string, dataSourceName string) *DB`
Use This API to open a DB, returning an object of type \*DB.
:::info
This API is created successfully without checking permissions, but only when you execute a Query or Exec, and check if user/password/host/port is legal.
:::
* `func (db *DB) Exec(query string, args ...interface{}) (Result, error)`
`sql.Open` built-in method to execute non-query related SQL.
* `func (db *DB) Query(query string, args ...interface{}) (*Rows, error)`
`sql.Open` Built-in method to execute query statements.
### Advanced functions (af) API
The `af` package encapsulates TDengine advanced functions such as connection management, subscriptions, schemaless, parameter binding, etc.
#### Connection management
* `af.Open(host, user, pass, db string, port int) (*Connector, error)`
This API creates a connection to taosd via cgo.
* `func (conn *Connector) Close() error`
Closes the connection.
#### Subscribe
* `func NewConsumer(conf *tmq.ConfigMap) (*Consumer, error)`
Creates consumer group.
* `func (c *Consumer) Subscribe(topic string, rebalanceCb RebalanceCb) error`
Note: `rebalanceCb` is reserved for compatibility purpose
Subscribes a topic.
* `func (c *Consumer) SubscribeTopics(topics []string, rebalanceCb RebalanceCb) error`
Note: `rebalanceCb` is reserved for compatibility purpose
Subscribes to topics.
* `func (c *Consumer) Poll(timeoutMs int) tmq.Event`
Polling information.
* `func (c *Consumer) Commit() ([]tmq.TopicPartition, error)`
Note: `tmq.TopicPartition` is reserved for compatibility purpose
Commit information.
* `func (c *Consumer) Assignment() (partitions []tmq.TopicPartition, err error)`
Get Assignment(TDengine >= 3.0.5.0 and driver-go >= v3.5.0 are required).
* `func (c *Consumer) Seek(partition tmq.TopicPartition, ignoredTimeoutMs int) error`
Note: `ignoredTimeoutMs` is reserved for compatibility purpose
Seek offset(TDengine >= 3.0.5.0 and driver-go >= v3.5.0 are required).
* `func (c *Consumer) Unsubscribe() error`
Unsubscribe.
* `func (c *Consumer) Close() error`
Close consumer.
#### schemaless
* `func (conn *Connector) InfluxDBInsertLines(lines []string, precision string) error`
Write to InfluxDB line protocol. ```go
package main
* `func (conn *Connector) OpenTSDBInsertTelnetLines(lines []string) error` import (
"database/sql"
"fmt"
"time"
Write OpenTDSB telnet protocol data. "github.com/taosdata/driver-go/v3/common"
"github.com/taosdata/driver-go/v3/common/param"
_ "github.com/taosdata/driver-go/v3/taosRestful"
"github.com/taosdata/driver-go/v3/ws/stmt"
)
* `func (conn *Connector) OpenTSDBInsertJsonPayload(payload string) error` func main() {
db, err := sql.Open("taosRestful", "root:taosdata@http(localhost:6041)/")
if err != nil {
panic(err)
}
defer db.Close()
prepareEnv(db)
config := stmt.NewConfig("ws://127.0.0.1:6041/rest/stmt", 0)
config.SetConnectUser("root")
config.SetConnectPass("taosdata")
config.SetConnectDB("example_ws_stmt")
config.SetMessageTimeout(common.DefaultMessageTimeout)
config.SetWriteWait(common.DefaultWriteWait)
config.SetErrorHandler(func(connector *stmt.Connector, err error) {
panic(err)
})
config.SetCloseHandler(func() {
fmt.Println("stmt connector closed")
})
connector, err := stmt.NewConnector(config)
if err != nil {
panic(err)
}
now := time.Now()
{
stmt, err := connector.Init()
if err != nil {
panic(err)
}
err = stmt.Prepare("insert into ? using all_json tags(?) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
if err != nil {
panic(err)
}
err = stmt.SetTableName("tb1")
if err != nil {
panic(err)
}
err = stmt.SetTags(param.NewParam(1).AddJson([]byte(`{"tb":1}`)), param.NewColumnType(1).AddJson(0))
if err != nil {
panic(err)
}
params := []*param.Param{
param.NewParam(3).AddTimestamp(now, 0).AddTimestamp(now.Add(time.Second), 0).AddTimestamp(now.Add(time.Second*2), 0),
param.NewParam(3).AddBool(true).AddNull().AddBool(true),
param.NewParam(3).AddTinyint(1).AddNull().AddTinyint(1),
param.NewParam(3).AddSmallint(1).AddNull().AddSmallint(1),
param.NewParam(3).AddInt(1).AddNull().AddInt(1),
param.NewParam(3).AddBigint(1).AddNull().AddBigint(1),
param.NewParam(3).AddUTinyint(1).AddNull().AddUTinyint(1),
param.NewParam(3).AddUSmallint(1).AddNull().AddUSmallint(1),
param.NewParam(3).AddUInt(1).AddNull().AddUInt(1),
param.NewParam(3).AddUBigint(1).AddNull().AddUBigint(1),
param.NewParam(3).AddFloat(1).AddNull().AddFloat(1),
param.NewParam(3).AddDouble(1).AddNull().AddDouble(1),
param.NewParam(3).AddBinary([]byte("test_binary")).AddNull().AddBinary([]byte("test_binary")),
param.NewParam(3).AddNchar("test_nchar").AddNull().AddNchar("test_nchar"),
}
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0)
err = stmt.BindParam(params, paramTypes)
if err != nil {
panic(err)
}
err = stmt.AddBatch()
if err != nil {
panic(err)
}
err = stmt.Exec()
if err != nil {
panic(err)
}
affected := stmt.GetAffectedRows()
fmt.Println("all_json affected rows:", affected)
err = stmt.Close()
if err != nil {
panic(err)
}
}
{
stmt, err := connector.Init()
if err != nil {
panic(err)
}
err = stmt.Prepare("insert into ? using all_all tags(?,?,?,?,?,?,?,?,?,?,?,?,?,?) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
err = stmt.SetTableName("tb1")
if err != nil {
panic(err)
}
Writes OpenTSDB JSON protocol data. err = stmt.SetTableName("tb2")
if err != nil {
panic(err)
}
err = stmt.SetTags(
param.NewParam(14).
AddTimestamp(now, 0).
AddBool(true).
AddTinyint(2).
AddSmallint(2).
AddInt(2).
AddBigint(2).
AddUTinyint(2).
AddUSmallint(2).
AddUInt(2).
AddUBigint(2).
AddFloat(2).
AddDouble(2).
AddBinary([]byte("tb2")).
AddNchar("tb2"),
param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0),
)
if err != nil {
panic(err)
}
params := []*param.Param{
param.NewParam(3).AddTimestamp(now, 0).AddTimestamp(now.Add(time.Second), 0).AddTimestamp(now.Add(time.Second*2), 0),
param.NewParam(3).AddBool(true).AddNull().AddBool(true),
param.NewParam(3).AddTinyint(1).AddNull().AddTinyint(1),
param.NewParam(3).AddSmallint(1).AddNull().AddSmallint(1),
param.NewParam(3).AddInt(1).AddNull().AddInt(1),
param.NewParam(3).AddBigint(1).AddNull().AddBigint(1),
param.NewParam(3).AddUTinyint(1).AddNull().AddUTinyint(1),
param.NewParam(3).AddUSmallint(1).AddNull().AddUSmallint(1),
param.NewParam(3).AddUInt(1).AddNull().AddUInt(1),
param.NewParam(3).AddUBigint(1).AddNull().AddUBigint(1),
param.NewParam(3).AddFloat(1).AddNull().AddFloat(1),
param.NewParam(3).AddDouble(1).AddNull().AddDouble(1),
param.NewParam(3).AddBinary([]byte("test_binary")).AddNull().AddBinary([]byte("test_binary")),
param.NewParam(3).AddNchar("test_nchar").AddNull().AddNchar("test_nchar"),
}
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0)
err = stmt.BindParam(params, paramTypes)
if err != nil {
panic(err)
}
err = stmt.AddBatch()
if err != nil {
panic(err)
}
err = stmt.Exec()
if err != nil {
panic(err)
}
affected := stmt.GetAffectedRows()
fmt.Println("all_all affected rows:", affected)
err = stmt.Close()
if err != nil {
panic(err)
}
#### parameter binding }
}
* `func (conn *Connector) StmtExecute(sql string, params *param.Param) (res driver.Result, err error)` func prepareEnv(db *sql.DB) {
steps := []string{
"create database example_ws_stmt",
"create table example_ws_stmt.all_json(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")" +
"tags(t json)",
"create table example_ws_stmt.all_all(" +
"ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")" +
"tags(" +
"tts timestamp," +
"tc1 bool," +
"tc2 tinyint," +
"tc3 smallint," +
"tc4 int," +
"tc5 bigint," +
"tc6 tinyint unsigned," +
"tc7 smallint unsigned," +
"tc8 int unsigned," +
"tc9 bigint unsigned," +
"tc10 float," +
"tc11 double," +
"tc12 binary(20)," +
"tc13 nchar(20))",
}
for _, step := range steps {
_, err := db.Exec(step)
if err != nil {
panic(err)
}
}
}
Parameter bound single row insert. ```
* `func (conn *Connector) InsertStmt() *insertstmt.InsertStmt` </TabItem>
</Tabs>
Initialize the parameters.
* `func (stmt *InsertStmt) Prepare(sql string) error` ### Schemaless Writing
Parameter binding preprocessing SQL statement. <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="native connection">
* `func (stmt *InsertStmt) SetTableName(name string) error` ```go
import (
"fmt"
Bind the table name parameter. "github.com/taosdata/driver-go/v3/af"
)
* `func (stmt *InsertStmt) SetSubTableName(name string) error` func main() {
conn, err := af.Open("localhost", "root", "taosdata", "", 6030)
if err != nil {
fmt.Println("fail to connect, err:", err)
}
defer conn.Close()
_, err = conn.Exec("create database if not exists example")
if err != nil {
panic(err)
}
_, err = conn.Exec("use example")
if err != nil {
panic(err)
}
influxdbData := "st,t1=3i64,t2=4f64,t3=\"t3\" c1=3i64,c3=L\"passit\",c2=false,c4=4f64 1626006833639000000"
err = conn.InfluxDBInsertLines([]string{influxdbData}, "ns")
if err != nil {
panic(err)
}
telnetData := "stb0_0 1626006833 4 host=host0 interface=eth0"
err = conn.OpenTSDBInsertTelnetLines([]string{telnetData})
if err != nil {
panic(err)
}
jsonData := "{\"metric\": \"meter_current\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"
err = conn.OpenTSDBInsertJsonPayload(jsonData)
if err != nil {
panic(err)
}
}
```
Parameter binding to set the sub table name. </TabItem>
<TabItem value="WebSocket" label="WebSocket connection">
* `func (stmt *InsertStmt) BindParam(params []*param.Param, bindType *param.ColumnType) error` ```go
import (
"database/sql"
"log"
"time"
Parameter bind multiple rows of data. "github.com/taosdata/driver-go/v3/common"
_ "github.com/taosdata/driver-go/v3/taosWS"
"github.com/taosdata/driver-go/v3/ws/schemaless"
)
* `func (stmt *InsertStmt) AddBatch() error` func main() {
db, err := sql.Open("taosWS", "root:taosdata@ws(localhost:6041)/")
if err != nil {
log.Fatal(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists schemaless_ws")
if err != nil {
log.Fatal(err)
}
s, err := schemaless.NewSchemaless(schemaless.NewConfig("ws://localhost:6041/rest/schemaless", 1,
schemaless.SetDb("schemaless_ws"),
schemaless.SetReadTimeout(10*time.Second),
schemaless.SetWriteTimeout(10*time.Second),
schemaless.SetUser("root"),
schemaless.SetPassword("taosdata"),
schemaless.SetErrorHandler(func(err error) {
log.Fatal(err)
}),
))
if err != nil {
panic(err)
}
influxdbData := "st,t1=3i64,t2=4f64,t3=\"t3\" c1=3i64,c3=L\"passit\",c2=false,c4=4f64 1626006833639000000"
telnetData := "stb0_0 1626006833 4 host=host0 interface=eth0"
jsonData := "{\"metric\": \"meter_current\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"
Add to a parameter-bound batch. err = s.Insert(influxdbData, schemaless.InfluxDBLineProtocol, "ns", 0, common.GetReqID())
if err != nil {
panic(err)
}
err = s.Insert(telnetData, schemaless.OpenTSDBTelnetLineProtocol, "ms", 0, common.GetReqID())
if err != nil {
panic(err)
}
err = s.Insert(jsonData, schemaless.OpenTSDBJsonFormatProtocol, "ms", 0, common.GetReqID())
if err != nil {
panic(err)
}
}
```
* `func (stmt *InsertStmt) Execute() error` </TabItem>
</Tabs>
Execute a parameter binding.
* `func (stmt *InsertStmt) GetAffectedRows() int` ### Schemaless with reqId
Gets the number of affected rows inserted by the parameter binding. ```go
func (s *Schemaless) Insert(lines string, protocol int, precision string, ttl int, reqID int64) error
```
* `func (stmt *InsertStmt) Close() error` You can get the unique id by `common.GetReqID()`.
Closes the parameter binding. ### Data Subscription
### Subscribe via WebSocket The TDengine Go Connector supports subscription functionality with the following application API.
* `func NewConsumer(conf *tmq.ConfigMap) (*Consumer, error)` #### Create a Topic
Creates consumer group. ```go
db, err := af.Open("", "root", "taosdata", "", 0)
if err != nil {
panic(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists example_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic if not exists example_tmq_topic as DATABASE example_tmq")
if err != nil {
panic(err)
}
```
* `func (c *Consumer) Subscribe(topic string, rebalanceCb RebalanceCb) error` #### Create a Consumer
Note: `rebalanceCb` is reserved for compatibility purpose
Subscribes a topic. ```go
consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"group.id": "test",
"auto.offset.reset": "earliest",
"td.connect.ip": "127.0.0.1",
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"td.connect.port": "6030",
"client.id": "test_tmq_client",
"enable.auto.commit": "false",
"msg.with.table.name": "true",
})
if err != nil {
panic(err)
}
```
* `func (c *Consumer) SubscribeTopics(topics []string, rebalanceCb RebalanceCb) error` #### Subscribe to consume data
Note: `rebalanceCb` is reserved for compatibility purpose
Subscribes to topics. ```go
err = consumer.Subscribe("example_tmq_topic", nil)
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Fprintf(os.Stderr, "%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
```
* `func (c *Consumer) Poll(timeoutMs int) tmq.Event` #### Assignment subscription Offset
Polling information. ```go
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
```
* `func (c *Consumer) Commit() ([]tmq.TopicPartition, error)` #### Close subscriptions
Note: `tmq.TopicPartition` is reserved for compatibility purpose
Commit information. ```go
err = consumer.Close()
if err != nil {
panic(err)
}
```
* `func (c *Consumer) Assignment() (partitions []tmq.TopicPartition, err error)` #### Full Sample Code
Get Assignment(TDengine >= 3.0.5.0 and driver-go >= v3.5.0 are required). <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="native connection">
* `func (c *Consumer) Seek(partition tmq.TopicPartition, ignoredTimeoutMs int) error` ```go
Note: `ignoredTimeoutMs` is reserved for compatibility purpose package main
Seek offset(TDengine >= 3.0.5.0 and driver-go >= v3.5.0 are required). import (
"fmt"
"os"
* `func (c *Consumer) Unsubscribe() error` "github.com/taosdata/driver-go/v3/af"
"github.com/taosdata/driver-go/v3/af/tmq"
tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
)
Unsubscribe. func main() {
db, err := af.Open("", "root", "taosdata", "", 0)
if err != nil {
panic(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists example_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic if not exists example_tmq_topic as DATABASE example_tmq")
if err != nil {
panic(err)
}
if err != nil {
panic(err)
}
consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"group.id": "test",
"auto.offset.reset": "earliest",
"td.connect.ip": "127.0.0.1",
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"td.connect.port": "6030",
"client.id": "test_tmq_client",
"enable.auto.commit": "false",
"msg.with.table.name": "true",
})
if err != nil {
panic(err)
}
err = consumer.Subscribe("example_tmq_topic", nil)
if err != nil {
panic(err)
}
_, err = db.Exec("create table example_tmq.t1 (ts timestamp,v int)")
if err != nil {
panic(err)
}
_, err = db.Exec("insert into example_tmq.t1 values(now,1)")
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Fprintf(os.Stderr, "%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
* `func (c *Consumer) Close() error` partitions, err = consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
}
Close consumer. err = consumer.Close()
if err != nil {
panic(err)
}
}
```
For a complete example see [GitHub sample file](https://github.com/taosdata/driver-go/blob/main/examples/tmqoverws/main.go) </TabItem>
<TabItem value="WebSocket" label="WebSocket connection">
### parameter binding via WebSocket ```go
package main
* `func NewConnector(config *Config) (*Connector, error)` import (
"database/sql"
"fmt"
Create a connection. "github.com/taosdata/driver-go/v3/common"
tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
_ "github.com/taosdata/driver-go/v3/taosRestful"
"github.com/taosdata/driver-go/v3/ws/tmq"
)
* `func (c *Connector) Init() (*Stmt, error)` func main() {
db, err := sql.Open("taosRestful", "root:taosdata@http(localhost:6041)/")
if err != nil {
panic(err)
}
defer db.Close()
prepareEnv(db)
consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"ws.url": "ws://127.0.0.1:6041/rest/tmq",
"ws.message.channelLen": uint(0),
"ws.message.timeout": common.DefaultMessageTimeout,
"ws.message.writeWait": common.DefaultWriteWait,
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"group.id": "example",
"client.id": "example_consumer",
"auto.offset.reset": "earliest",
})
if err != nil {
panic(err)
}
err = consumer.Subscribe("example_ws_tmq_topic", nil)
if err != nil {
panic(err)
}
go func() {
_, err := db.Exec("create table example_ws_tmq.t_all(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")")
if err != nil {
panic(err)
}
_, err = db.Exec("insert into example_ws_tmq.t_all values(now,true,2,3,4,5,6,7,8,9,10.123,11.123,'binary','nchar')")
if err != nil {
panic(err)
}
}()
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Printf("%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
Initialize the parameters. partitions, err = consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
}
* `func (c *Connector) Close() error` err = consumer.Close()
if err != nil {
panic(err)
}
}
Close the connection. func prepareEnv(db *sql.DB) {
_, err := db.Exec("create database example_ws_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic example_ws_tmq_topic as database example_ws_tmq")
if err != nil {
panic(err)
}
}
```
* `func (s *Stmt) Prepare(sql string) error` </TabItem>
</Tabs>
Parameter binding preprocessing SQL statement. ### More sample programs
* `func (s *Stmt) SetTableName(name string) error` * [sample program](https://github.com/taosdata/driver-go/tree/3.0/examples)
Bind the table name parameter.
* `func (s *Stmt) SetTags(tags *param.Param, bindType *param.ColumnType) error` ## Frequently Asked Questions
Set tags. 1. bind interface in database/sql crashes
* `func (s *Stmt) BindParam(params []*param.Param, bindType *param.ColumnType) error` REST does not support parameter binding related interface. It is recommended to use `db.Exec` and `db.Query`.
Parameter bind multiple rows of data. 2. error `[0x217] Database not specified or available` after executing other statements with `use db` statement
* `func (s *Stmt) AddBatch() error` The execution of SQL command in the REST interface is not contextual, so using `use db` statement will not work, see the usage restrictions section above.
Add to a parameter-bound batch. 3. use `taosSql` without error but use `taosRestful` with error `[0x217] Database not specified or available`
* `func (s *Stmt) Exec() error` Because the REST interface is stateless, using the `use db` statement will not take effect. See the usage restrictions section above.
Execute a parameter binding. 4. `readBufferSize` parameter has no significant effect after being increased
* `func (s *Stmt) GetAffectedRows() int` Increasing `readBufferSize` will reduce the number of `syscall` calls when fetching results. If the query result is smaller, modifying this parameter will not improve performance significantly. If you increase the parameter value too much, the bottleneck will be parsing JSON data. If you need to optimize the query speed, you must adjust the value based on the actual situation to achieve the best query performance.
Gets the number of affected rows inserted by the parameter binding. 5. `disableCompression` parameter is set to `false` when the query efficiency is reduced
* `func (s *Stmt) Close() error` When set `disableCompression` parameter to `false`, the query result will be compressed by `gzip` and then transmitted, so you have to decompress the data by `gzip` after getting it.
Closes the parameter binding. 6. `go get` command can't get the package, or timeout to get the package
For a complete example see [GitHub sample file](https://github.com/taosdata/driver-go/blob/main/examples/stmtoverws/main.go) Set Go proxy `go env -w GOPROXY=https://goproxy.cn,direct`.
## API Reference ## API Reference
......
...@@ -31,21 +31,57 @@ Websocket connections are supported on all platforms that can run Go. ...@@ -31,21 +31,57 @@ Websocket connections are supported on all platforms that can run Go.
| connector-rust version | TDengine version | major features | | connector-rust version | TDengine version | major features |
| :----------------: | :--------------: | :--------------------------------------------------: | | :----------------: | :--------------: | :--------------------------------------------------: |
| v0.8.10 | 3.0.5.0 or later | TMQ: Get consuming progress and seek offset to consume. | | v0.8.12 | 3.0.5.0 or later | TMQ: Get consuming progress and seek offset to consume. |
| v0.8.0 | 3.0.4.0 | Support schemaless insert. | | v0.8.0 | 3.0.4.0 | Support schemaless insert. |
| v0.7.6 | 3.0.3.0 | Support req_id in query. | | v0.7.6 | 3.0.3.0 | Support req_id in query. |
| v0.6.0 | 3.0.0.0 | Base features. | | v0.6.0 | 3.0.0.0 | Base features. |
The Rust Connector is still under rapid development and is not guaranteed to be backward compatible before 1.0. We recommend using TDengine version 3.0 or higher to avoid known issues. The Rust Connector is still under rapid development and is not guaranteed to be backward compatible before 1.0. We recommend using TDengine version 3.0 or higher to avoid known issues.
## Installation ## Handling exceptions
After the error is reported, the specific information of the error can be obtained:
```rust
match conn.exec(sql) {
Ok(_) => {
Ok(())
}
Err(e) => {
eprintln!("ERROR: {:?}", e);
Err(e)
}
}
```
## TDengine DataType vs. Rust DataType
TDengine currently supports timestamp, number, character, Boolean type, and the corresponding type conversion with Rust is as follows:
| TDengine DataType | Rust DataType |
| ----------------- | ----------------- |
| TIMESTAMP | Timestamp |
| INT | i32 |
| BIGINT | i64 |
| FLOAT | f32 |
| DOUBLE | f64 |
| SMALLINT | i16 |
| TINYINT | i8 |
| BOOL | bool |
| BINARY | Vec<u8\> |
| NCHAR | String |
| JSON | serde_json::Value |
Note: Only TAG supports JSON types
## Installation Steps
### Pre-installation preparation ### Pre-installation preparation
* Install the Rust development toolchain * Install the Rust development toolchain
* If using the native connection, please install the TDengine client driver. Please refer to [install client driver](/reference/connector#install-client-driver) * If using the native connection, please install the TDengine client driver. Please refer to [install client driver](/reference/connector#install-client-driver)
### Add taos dependency ### Install the connectors
Depending on the connection method, add the [taos][taos] dependency in your Rust project as follows: Depending on the connection method, add the [taos][taos] dependency in your Rust project as follows:
...@@ -146,7 +182,8 @@ let builder = TaosBuilder::from_dsn("taos://localhost:6030")?; ...@@ -146,7 +182,8 @@ let builder = TaosBuilder::from_dsn("taos://localhost:6030")?;
let conn1 = builder.build(); let conn1 = builder.build();
// use websocket protocol. // use websocket protocol.
let conn2 = TaosBuilder::from_dsn("taos+ws://localhost:6041")?; let builder2 = TaosBuilder::from_dsn("taos+ws://localhost:6041")?;
let conn2 = builder2.build();
``` ```
After the connection is established, you can perform operations on your database. After the connection is established, you can perform operations on your database.
...@@ -228,41 +265,191 @@ There are two ways to query data: Using built-in types or the [serde](https://se ...@@ -228,41 +265,191 @@ There are two ways to query data: Using built-in types or the [serde](https://se
## Usage examples ## Usage examples
### Write data ### Create database and tables
#### SQL Write ```rust
use taos::*;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let dsn = "taos://localhost:6030";
let builder = TaosBuilder::from_dsn(dsn)?;
let taos = builder.build()?;
let db = "query";
// create database
taos.exec_many([
format!("DROP DATABASE IF EXISTS `{db}`"),
format!("CREATE DATABASE `{db}`"),
format!("USE `{db}`"),
])
.await?;
// create table
taos.exec_many([
// create super table
"CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) \
TAGS (`groupid` INT, `location` BINARY(16))",
// create child table
"CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')",
]).await?;
}
```
> The query is consistent with operating a relational database. When using subscripts to get the contents of the returned fields, you have to start from 1. However, we recommend using the field names to get the values of the fields in the result set.
### Insert data
<RustInsert /> <RustInsert />
#### STMT Write ### Query data
<RustQuery />
### execute SQL with req_id
This req_id can be used to request link tracing.
```rust
let rs = taos.query_with_req_id("select * from stable where tag1 is null", 1)?;
```
### Writing data via parameter binding
TDengine has significantly improved the bind APIs to support data writing (INSERT) scenarios. Writing data in this way avoids the resource consumption of SQL syntax parsing, resulting in significant write performance improvements in many cases.
Parameter binding details see [API Reference](#stmt-api)
<RustBind /> <RustBind />
#### Schemaless Write ### Schemaless Writing
TDengine supports schemaless writing. It is compatible with InfluxDB's Line Protocol, OpenTSDB's telnet line protocol, and OpenTSDB's JSON format protocol. For more information, see [Schemaless Writing](../../schemaless).
<RustSml /> <RustSml />
### Query data ### Schemaless with req_id
<RustQuery /> This req_id can be used to request link tracing.
## API Reference ```rust
let sml_data = SmlDataBuilder::default()
.protocol(SchemalessProtocol::Line)
.data(data)
.req_id(100u64)
.build()?;
client.put(&sml_data)?
```
### Data Subscription
TDengine starts subscriptions through [TMQ](../../../taos-sql/tmq/).
#### Create a Topic
### Connector Constructor ```rust
taos.exec_many([
// create topic for subscription
format!("CREATE TOPIC tmq_meters with META AS DATABASE {db}")
])
.await?;
```
You create a connector constructor by using a DSN. #### Create a Consumer
You create a TMQ connector by using a DSN.
```rust ```rust
let cfg = TaosBuilder::default().build()?; let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
``` ```
You use the builder object to create multiple connections. Create a consumer:
```rust ```rust
let conn: Taos = cfg.build(); let mut consumer = tmq.build()?;
``` ```
### Connection pooling #### Subscribe to consume data
A single consumer can subscribe to one or more topics.
```rust
consumer.subscribe(["tmq_meters"]).await?;
```
The TMQ is of [futures::Stream](https://docs.rs/futures/latest/futures/stream/index.html) type. You can use the corresponding API to consume each message in the queue and then use `.commit` to mark them as consumed.
```rust
{
let mut stream = consumer.stream();
while let Some((offset, message)) = stream.try_next().await? {
// get information from offset
// the topic
let topic = offset.topic();
// the vgroup id, like partition id in kafka.
let vgroup_id = offset.vgroup_id();
println!("* in vgroup id {vgroup_id} of topic {topic}\n");
if let Some(data) = message.into_data() {
while let Some(block) = data.fetch_raw_block().await? {
// one block for one table, get table name if needed
let name = block.table_name();
let records: Vec<Record> = block.deserialize().try_collect()?;
println!(
"** table: {}, got {} records: {:#?}\n",
name.unwrap(),
records.len(),
records
);
}
}
consumer.commit(offset).await?;
}
}
```
Get assignments:
Version requirements connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
let assignments = consumer.assignments().await.unwrap();
```
#### Assignment subscription Offset
Seek offset:
Version requirements connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
consumer.offset_seek(topic, vgroup_id, offset).await;
```
#### Close subscriptions
```rust
consumer.unsubscribe().await;
```
The following parameters can be configured for the TMQ DSN. Only `group.id` is mandatory.
- `group.id`: Within a consumer group, load balancing is implemented by consuming messages on an at-least-once basis.
- `client.id`: Subscriber client ID.
- `auto.offset.reset`: Initial point of subscription. *earliest* subscribes from the beginning, and *latest* subscribes from the newest message. The default is earliest. Note: This parameter is set per consumer group.
- `enable.auto.commit`: Automatically commits. This can be enabled when data consistency is not essential.
- `auto.commit.interval.ms`: Interval for automatic commits.
#### Full Sample Code
For more information, see [GitHub sample file](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/rust/nativeexample/examples/subscribe_demo.rs).
### Use with connection pool
In complex applications, we recommend enabling connection pools. [taos] implements connection pools based on [r2d2]. In complex applications, we recommend enabling connection pools. [taos] implements connection pools based on [r2d2].
...@@ -292,7 +479,17 @@ In the application code, use `pool.get()? ` to get a connection object [Taos]. ...@@ -292,7 +479,17 @@ In the application code, use `pool.get()? ` to get a connection object [Taos].
let taos = pool.get()?; let taos = pool.get()?;
``` ```
### Connectors ### More sample programs
The source code of the sample application is under `TDengine/examples/rust` :
[rust example](https://github.com/taosdata/TDengine/tree/3.0/examples/rust)
## Frequently Asked Questions
For additional troubleshooting, see [FAQ](../../../train-faq/faq).
## API Reference
The [Taos][struct.Taos] object provides an API to perform operations on multiple databases. The [Taos][struct.Taos] object provides an API to perform operations on multiple databases.
...@@ -378,9 +575,13 @@ Note that Rust asynchronous functions and an asynchronous runtime are required. ...@@ -378,9 +575,13 @@ Note that Rust asynchronous functions and an asynchronous runtime are required.
- `.create_database(database: &str)`: Executes the `CREATE DATABASE` statement. - `.create_database(database: &str)`: Executes the `CREATE DATABASE` statement.
- `.use_database(database: &str)`: Executes the `USE` statement. - `.use_database(database: &str)`: Executes the `USE` statement.
In addition, this structure is also the entry point for [Parameter Binding](#Parameter Binding Interface) and [Line Protocol Interface](#Line Protocol Interface). Please refer to the specific API descriptions for usage. In addition, this structure is also the entry point for Parameter Binding and Line Protocol Interface. Please refer to the specific API descriptions for usage.
### Bind Interface <p>
<a id="stmt-api" style={{color:'#141414'}}>
Bind Interface
</a>
</p>
Similar to the C interface, Rust provides the bind interface's wrapping. First, the [Taos][struct.taos] object creates a parameter binding object [Stmt] for an SQL statement. Similar to the C interface, Rust provides the bind interface's wrapping. First, the [Taos][struct.taos] object creates a parameter binding object [Stmt] for an SQL statement.
...@@ -391,7 +592,7 @@ stmt.prepare("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?; ...@@ -391,7 +592,7 @@ stmt.prepare("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?;
The bind object provides a set of interfaces for implementing parameter binding. The bind object provides a set of interfaces for implementing parameter binding.
#### `.set_tbname(name)` `.set_tbname(name)`
To bind table names. To bind table names.
...@@ -400,7 +601,7 @@ let mut stmt = taos.stmt("insert into ? values(? ,?)")?; ...@@ -400,7 +601,7 @@ let mut stmt = taos.stmt("insert into ? values(? ,?)")?;
stmt.set_tbname("d0")?; stmt.set_tbname("d0")?;
``` ```
#### `.set_tags(&[tag])` `.set_tags(&[tag])`
Bind sub-table table names and tag values when the SQL statement uses a super table. Bind sub-table table names and tag values when the SQL statement uses a super table.
...@@ -410,7 +611,7 @@ stmt.set_tbname("d0")?; ...@@ -410,7 +611,7 @@ stmt.set_tbname("d0")?;
stmt.set_tags(&[Value::VarChar("taos".to_string())])?; stmt.set_tags(&[Value::VarChar("taos".to_string())])?;
``` ```
#### `.bind(&[column])` `.bind(&[column])`
Bind value types. Use the [ColumnView] structure to create and bind the required types. Bind value types. Use the [ColumnView] structure to create and bind the required types.
...@@ -434,7 +635,7 @@ let params = vec![ ...@@ -434,7 +635,7 @@ let params = vec![
let rows = stmt.bind(&params)?.add_batch()?.execute()?; let rows = stmt.bind(&params)?.add_batch()?.execute()?;
``` ```
#### `.execute()` `.execute()`
Execute SQL. [Stmt] objects can be reused, re-binded, and executed after execution. Before execution, ensure that all data has been added to the queue with `.add_batch`. Execute SQL. [Stmt] objects can be reused, re-binded, and executed after execution. Before execution, ensure that all data has been added to the queue with `.add_batch`.
...@@ -449,92 +650,6 @@ stmt.execute()?; ...@@ -449,92 +650,6 @@ stmt.execute()?;
For a working example, see [GitHub](https://github.com/taosdata/taos-connector-rust/blob/main/examples/bind.rs). For a working example, see [GitHub](https://github.com/taosdata/taos-connector-rust/blob/main/examples/bind.rs).
### Subscriptions
TDengine starts subscriptions through [TMQ](../../../taos-sql/tmq/).
You create a TMQ connector by using a DSN.
```rust
let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
```
Create a consumer:
```rust
let mut consumer = tmq.build()?;
```
A single consumer can subscribe to one or more topics.
```rust
consumer.subscribe(["tmq_meters"]).await?;
```
The TMQ is of [futures::Stream](https://docs.rs/futures/latest/futures/stream/index.html) type. You can use the corresponding API to consume each message in the queue and then use `.commit` to mark them as consumed.
```rust
{
let mut stream = consumer.stream();
while let Some((offset, message)) = stream.try_next().await? {
// get information from offset
// the topic
let topic = offset.topic();
// the vgroup id, like partition id in kafka.
let vgroup_id = offset.vgroup_id();
println!("* in vgroup id {vgroup_id} of topic {topic}\n");
if let Some(data) = message.into_data() {
while let Some(block) = data.fetch_raw_block().await? {
// one block for one table, get table name if needed
let name = block.table_name();
let records: Vec<Record> = block.deserialize().try_collect()?;
println!(
"** table: {}, got {} records: {:#?}\n",
name.unwrap(),
records.len(),
records
);
}
}
consumer.commit(offset).await?;
}
}
```
Get assignments:
Version requirements connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
let assignments = consumer.assignments().await.unwrap();
```
Seek offset:
Version requirements connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
consumer.offset_seek(topic, vgroup_id, offset).await;
```
Unsubscribe:
```rust
consumer.unsubscribe().await;
```
The following parameters can be configured for the TMQ DSN. Only `group.id` is mandatory.
- `group.id`: Within a consumer group, load balancing is implemented by consuming messages on an at-least-once basis.
- `client.id`: Subscriber client ID.
- `auto.offset.reset`: Initial point of subscription. *earliest* subscribes from the beginning, and *latest* subscribes from the newest message. The default is earliest. Note: This parameter is set per consumer group.
- `enable.auto.commit`: Automatically commits. This can be enabled when data consistency is not essential.
- `auto.commit.interval.ms`: Interval for automatic commits.
For more information, see [GitHub sample file](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/rust/nativeexample/examples/subscribe_demo.rs).
For information about other structure APIs, see the [Rust documentation](https://docs.rs/taos). For information about other structure APIs, see the [Rust documentation](https://docs.rs/taos).
......
...@@ -24,6 +24,16 @@ The source code for the Python connector is hosted on [GitHub](https://github.co ...@@ -24,6 +24,16 @@ The source code for the Python connector is hosted on [GitHub](https://github.co
We recommend using the latest version of `taospy`, regardless of the version of TDengine. We recommend using the latest version of `taospy`, regardless of the version of TDengine.
|Python Connector Version|major changes|
|:-------------------:|:----:|
|2.7.9|support for getting assignment and seek function on subscription|
|2.7.8|add `execute_many` method|
|Python Websocket Connector Version|major changes|
|:----------------------------:|:-----:|
|0.2.5|1. support for getting assignment and seek function on subscription <br/> 2. support schemaless <br/> 3. support STMT|
|0.2.4|support `unsubscribe` on subscription|
## Handling Exceptions ## Handling Exceptions
There are 4 types of exception in python connector. There are 4 types of exception in python connector.
......
...@@ -17,7 +17,7 @@ TDengine 支持通过 C/Python 语言进行 UDF 定义。接下来结合示例 ...@@ -17,7 +17,7 @@ TDengine 支持通过 C/Python 语言进行 UDF 定义。接下来结合示例
- 聚合函数需要实现聚合接口函数 aggfn_start , aggfn , aggfn_finish。 - 聚合函数需要实现聚合接口函数 aggfn_start , aggfn , aggfn_finish。
- 如果需要初始化,实现 udf_init;如果需要清理工作,实现udf_destroy。 - 如果需要初始化,实现 udf_init;如果需要清理工作,实现udf_destroy。
接口函数的名称是 UDF 名称,或者是 UDF 名称和特定后缀(_start, _finish, _init, _destroy)的连接。列表中的scalarfn,aggfn, udf需要替换成udf函数名。 接口函数的名称是 UDF 名称,或者是 UDF 名称和特定后缀(`_start`, `_finish`, `_init`, `_destroy`)的连接。列表中的scalarfn,aggfn, udf需要替换成udf函数名。
### 用 C 语言实现标量函数 ### 用 C 语言实现标量函数
标量函数实现模板如下 标量函数实现模板如下
......
...@@ -36,14 +36,15 @@ REST 连接支持所有能运行 Java 的平台。 ...@@ -36,14 +36,15 @@ REST 连接支持所有能运行 Java 的平台。
| taos-jdbcdriver 版本 | 主要变化 | TDengine 版本 | | taos-jdbcdriver 版本 | 主要变化 | TDengine 版本 |
| :------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------: | | :------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------: |
| 3.2.3 | 修复 ResultSet 在一些情况数据解析失败 | - |
| 3.2.2 | 新增功能:数据订阅支持 seek 功能。 | 3.0.5.0 及更高版本 | | 3.2.2 | 新增功能:数据订阅支持 seek 功能。 | 3.0.5.0 及更高版本 |
| 3.2.1 | 新增功能:WebSocket 连接支持 schemaless 与 prepareStatement 写入。变更:consumer poll 返回结果集为 ConsumerRecord,可通过 value() 获取指定结果集数据。 | 3.0.3.0 及更高版本 | | 3.2.1 | 新增功能:WebSocket 连接支持 schemaless 与 prepareStatement 写入。变更:consumer poll 返回结果集为 ConsumerRecord,可通过 value() 获取指定结果集数据。 | 3.0.3.0 及更高版本 |
| 3.2.0 | 存在连接问题,不推荐使用 | - | | 3.2.0 | 存在连接问题,不推荐使用 | - |
| 3.1.0 | WebSocket 连接支持订阅功能 | - | | 3.1.0 | WebSocket 连接支持订阅功能 | - |
| 3.0.1 - 3.0.4 | 修复一些情况下结果集数据解析错误的问题。3.0.1 在 JDK 11 环境编译,JDK 8 环境下建议使用其他版本 | - | | 3.0.1 - 3.0.4 | 修复一些情况下结果集数据解析错误的问题。3.0.1 在 JDK 11 环境编译,JDK 8 环境下建议使用其他版本 | - |
| 3.0.0 | 支持 TDengine 3.0 | 3.0.0.0 及更高版本 | | 3.0.0 | 支持 TDengine 3.0 | 3.0.0.0 及更高版本 |
| 2.0.42 | 修 WebSocket 连接中 wasNull 接口返回值 | - | | 2.0.42 | 修 WebSocket 连接中 wasNull 接口返回值 | - |
| 2.0.41 | 修 REST 连接中用户名和密码转码方式 | - | | 2.0.41 | 修 REST 连接中用户名和密码转码方式 | - |
| 2.0.39 - 2.0.40 | 增加 REST 连接/请求 超时设置 | - | | 2.0.39 - 2.0.40 | 增加 REST 连接/请求 超时设置 | - |
| 2.0.38 | JDBC REST 连接增加批量拉取功能 | - | | 2.0.38 | JDBC REST 连接增加批量拉取功能 | - |
| 2.0.37 | 增加对 json tag 支持 | - | | 2.0.37 | 增加对 json tag 支持 | - |
...@@ -287,9 +288,9 @@ url 中的配置参数如下: ...@@ -287,9 +288,9 @@ url 中的配置参数如下:
- batchfetch: true:在执行查询时批量拉取结果集;false:逐行拉取结果集。默认值为:false。逐行拉取结果集使用 HTTP 方式进行数据传输。JDBC REST 连接支持批量拉取数据功能。taos-jdbcdriver 与 TDengine 之间通过 WebSocket 连接进行数据传输。相较于 HTTP,WebSocket 可以使 JDBC REST 连接支持大数据量查询,并提升查询性能。 - batchfetch: true:在执行查询时批量拉取结果集;false:逐行拉取结果集。默认值为:false。逐行拉取结果集使用 HTTP 方式进行数据传输。JDBC REST 连接支持批量拉取数据功能。taos-jdbcdriver 与 TDengine 之间通过 WebSocket 连接进行数据传输。相较于 HTTP,WebSocket 可以使 JDBC REST 连接支持大数据量查询,并提升查询性能。
- charset: 当开启批量拉取数据时,指定解析字符串数据的字符集。 - charset: 当开启批量拉取数据时,指定解析字符串数据的字符集。
- batchErrorIgnore:true:在执行 Statement 的 executeBatch 时,如果中间有一条 SQL 执行失败,继续执行下面的 SQL 了。false:不再执行失败 SQL 后的任何语句。默认值为:false。 - batchErrorIgnore:true:在执行 Statement 的 executeBatch 时,如果中间有一条 SQL 执行失败,继续执行下面的 SQL 了。false:不再执行失败 SQL 后的任何语句。默认值为:false。
- httpConnectTimeout: 连接超时时间,单位 ms, 默认值为 5000。 - httpConnectTimeout: 连接超时时间,单位 ms, 默认值为 60000。
- httpSocketTimeout: socket 超时时间,单位 ms,默认值为 5000。仅在 batchfetch 设置为 false 时生效。 - httpSocketTimeout: socket 超时时间,单位 ms,默认值为 60000。仅在 batchfetch 设置为 false 时生效。
- messageWaitTimeout: 消息超时时间, 单位 ms, 默认值为 3000。 仅在 batchfetch 设置为 true 时生效。 - messageWaitTimeout: 消息超时时间, 单位 ms, 默认值为 60000。 仅在 batchfetch 设置为 true 时生效。
- useSSL: 连接中是否使用 SSL。 - useSSL: 连接中是否使用 SSL。
- httpPoolSize: REST 并发请求大小,默认 20。 - httpPoolSize: REST 并发请求大小,默认 20。
...@@ -355,9 +356,9 @@ properties 中的配置参数如下: ...@@ -355,9 +356,9 @@ properties 中的配置参数如下:
- TSDBDriver.PROPERTY_KEY_CHARSET:客户端使用的字符集,默认值为系统字符集。 - TSDBDriver.PROPERTY_KEY_CHARSET:客户端使用的字符集,默认值为系统字符集。
- TSDBDriver.PROPERTY_KEY_LOCALE:仅在使用 JDBC 原生连接时生效。 客户端语言环境,默认值系统当前 locale。 - TSDBDriver.PROPERTY_KEY_LOCALE:仅在使用 JDBC 原生连接时生效。 客户端语言环境,默认值系统当前 locale。
- TSDBDriver.PROPERTY_KEY_TIME_ZONE:仅在使用 JDBC 原生连接时生效。 客户端使用的时区,默认值为系统当前时区。 - TSDBDriver.PROPERTY_KEY_TIME_ZONE:仅在使用 JDBC 原生连接时生效。 客户端使用的时区,默认值为系统当前时区。
- TSDBDriver.HTTP_CONNECT_TIMEOUT: 连接超时时间,单位 ms, 默认值为 5000。仅在 REST 连接时生效。 - TSDBDriver.HTTP_CONNECT_TIMEOUT: 连接超时时间,单位 ms, 默认值为 60000。仅在 REST 连接时生效。
- TSDBDriver.HTTP_SOCKET_TIMEOUT: socket 超时时间,单位 ms,默认值为 5000。仅在 REST 连接且 batchfetch 设置为 false 时生效。 - TSDBDriver.HTTP_SOCKET_TIMEOUT: socket 超时时间,单位 ms,默认值为 60000。仅在 REST 连接且 batchfetch 设置为 false 时生效。
- TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms, 默认值为 3000。 仅在 REST 连接且 batchfetch 设置为 true 时生效。 - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms, 默认值为 60000。 仅在 REST 连接且 batchfetch 设置为 true 时生效。
- TSDBDriver.PROPERTY_KEY_USE_SSL: 连接中是否使用 SSL。仅在 REST 连接时生效。 - TSDBDriver.PROPERTY_KEY_USE_SSL: 连接中是否使用 SSL。仅在 REST 连接时生效。
- TSDBDriver.HTTP_POOL_SIZE: REST 并发请求大小,默认 20。 - TSDBDriver.HTTP_POOL_SIZE: REST 并发请求大小,默认 20。
此外对 JDBC 原生连接,通过指定 URL 和 Properties 还可以指定其他参数,比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。 此外对 JDBC 原生连接,通过指定 URL 和 Properties 还可以指定其他参数,比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。
......
...@@ -32,24 +32,44 @@ REST 连接支持所有能运行 Go 的平台。 ...@@ -32,24 +32,44 @@ REST 连接支持所有能运行 Go 的平台。
请参考[版本支持列表](https://github.com/taosdata/driver-go#remind) 请参考[版本支持列表](https://github.com/taosdata/driver-go#remind)
## 支持的功能特性 ## 处理异常
### 原生连接 如果是 TDengine 错误可以通过以下方式获取错误码和错误信息。
“原生连接”指连接器通过 TDengine 客户端驱动(taosc)直接与 TDengine 运行实例建立的连接。支持的功能特性有: ```go
// import "github.com/taosdata/driver-go/v3/errors"
* 普通查询 if err != nil {
* 连续查询 tError, is := err.(*errors.TaosError)
* 订阅 if is {
* schemaless 接口 fmt.Println("errorCode:", int(tError.Code))
* 参数绑定接口 fmt.Println("errorMessage:", tError.ErrStr)
} else {
### REST 连接 fmt.Println(err.Error())
}
"REST 连接"指连接器通过 taosAdapter 组件提供的 REST API TDengine 运行实例建立的连接。支持的功能特性有: }
```
* 普通查询 ## TDengine DataType Go DataType
* 连续查询
| TDengine DataType | Go Type |
|-------------------|-----------|
| TIMESTAMP | time.Time |
| TINYINT | int8 |
| SMALLINT | int16 |
| INT | int32 |
| BIGINT | int64 |
| TINYINT UNSIGNED | uint8 |
| SMALLINT UNSIGNED | uint16 |
| INT UNSIGNED | uint32 |
| BIGINT UNSIGNED | uint64 |
| FLOAT | float32 |
| DOUBLE | float64 |
| BOOL | bool |
| BINARY | string |
| NCHAR | string |
| JSON | []byte |
**注意**JSON 类型仅在 tag 中支持。
## 安装步骤 ## 安装步骤
...@@ -63,11 +83,7 @@ REST 连接支持所有能运行 Go 的平台。 ...@@ -63,11 +83,7 @@ REST 连接支持所有能运行 Go 的平台。
* ```go env``` * ```go env```
* ```gcc -v``` * ```gcc -v```
### 使用 go get 安装 ### 安装连接器
`go get -u github.com/taosdata/driver-go/v3@latest`
### 使用 go mod 管理
1. 使用 `go mod` 命令初始化项目: 1. 使用 `go mod` 命令初始化项目:
...@@ -99,8 +115,6 @@ REST 连接支持所有能运行 Go 的平台。 ...@@ -99,8 +115,6 @@ REST 连接支持所有能运行 Go 的平台。
## 建立连接 ## 建立连接
### 数据源名称(DSN
数据源名称具有通用格式,例如 [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php),但没有类型前缀(方括号表示可选): 数据源名称具有通用格式,例如 [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php),但没有类型前缀(方括号表示可选):
``` text ``` text
...@@ -113,9 +127,7 @@ REST 连接支持所有能运行 Go 的平台。 ...@@ -113,9 +127,7 @@ REST 连接支持所有能运行 Go 的平台。
username:password@protocol(address)/dbname?param=value username:password@protocol(address)/dbname?param=value
``` ```
### 使用连接器进行连接 <Tabs defaultValue="rest" groupId="connect">
<Tabs defaultValue="rest">
<TabItem value="native" label="原生连接"> <TabItem value="native" label="原生连接">
_taosSql_ 通过 cgo 实现了 Go `database/sql/driver` 接口。只需要引入驱动就可以使用 [`database/sql`](https://golang.org/pkg/database/sql/) 的接口。 _taosSql_ 通过 cgo 实现了 Go `database/sql/driver` 接口。只需要引入驱动就可以使用 [`database/sql`](https://golang.org/pkg/database/sql/) 的接口。
...@@ -213,332 +225,900 @@ func main() { ...@@ -213,332 +225,900 @@ func main() {
</TabItem> </TabItem>
</Tabs> </Tabs>
## 使用示例 ### 指定 URL Properties 获取连接
### 写入数据
#### SQL 写入 Go 连接器不支持此功能
<GoInsert /> ### 配置参数的优先级
#### InfluxDB 行协议写入 Go 连接器不支持此功能
<GoInfluxLine /> ## 使用示例
#### OpenTSDB Telnet 行协议写入 ### 创建数据库和表
<GoOpenTSDBTelnet /> ```go
var taosDSN = "root:taosdata@tcp(localhost:6030)/"
taos, err := sql.Open("taosSql", taosDSN)
if err != nil {
log.Fatalln("failed to connect TDengine, err:", err)
}
defer taos.Close()
_, err := taos.Exec("CREATE DATABASE power")
if err != nil {
log.Fatalln("failed to create database, err:", err)
}
_, err = taos.Exec("CREATE STABLE power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)")
if err != nil {
log.Fatalln("failed to create stable, err:", err)
}
```
#### OpenTSDB JSON 行协议写入 ### 插入数据
<GoOpenTSDBJson /> <GoInsert />
### 查询数据 ### 查询数据
<GoQuery /> <GoQuery />
### 更多示例程序 ### 执行带有 reqId SQL
* [示例程序](https://github.com/taosdata/driver-go/tree/3.0/examples)
* [视频教程](https://www.taosdata.com/blog/2020/11/11/1951.html)
## 使用限制 reqId 可用于请求链路追踪。
由于 REST 接口无状态所以 `use db` 语法不会生效,需要将 db 名称放到 SQL 语句中,如:`create table if not exists tb1 (ts timestamp, a int)`改为`create table if not exists test.tb1 (ts timestamp, a int)`否则将报错`[0x217] Database not specified or available` ```go
db, err := sql.Open("taosSql", "root:taosdata@tcp(localhost:6030)/")
if err != nil {
panic(err)
}
defer db.Close()
ctx := context.WithValue(context.Background(), common.ReqIDKey, common.GetReqID())
_, err = db.ExecContext(ctx, "create database if not exists example_taos_sql")
if err != nil {
panic(err)
}
```
也可以将 db 名称放到 DSN 中,将 `root:taosdata@http(localhost:6041)/` 改为 `root:taosdata@http(localhost:6041)/test`。当指定的 db 不存在时执行 `create database` 语句不会报错,而执行针对该 db 的其他查询或写入操作会报错。 ### 通过参数绑定写入数据
完整示例如下: <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="原生连接">
```go ```go
package main package main
import ( import (
"database/sql"
"fmt"
"time" "time"
_ "github.com/taosdata/driver-go/v3/taosRestful" "github.com/taosdata/driver-go/v3/af"
"github.com/taosdata/driver-go/v3/common"
"github.com/taosdata/driver-go/v3/common/param"
) )
func main() { func main() {
var taosDSN = "root:taosdata@http(localhost:6041)/test" db, err := af.Open("", "root", "taosdata", "", 0)
taos, err := sql.Open("taosRestful", taosDSN)
if err != nil { if err != nil {
fmt.Println("failed to connect TDengine, err:", err) panic(err)
return
} }
defer taos.Close() defer db.Close()
taos.Exec("create database if not exists test") _, err = db.Exec("create database if not exists example_stmt")
taos.Exec("create table if not exists tb1 (ts timestamp, a int)")
_, err = taos.Exec("insert into tb1 values(now, 0)(now+1s,1)(now+2s,2)(now+3s,3)")
if err != nil { if err != nil {
fmt.Println("failed to insert, err:", err) panic(err)
return
} }
rows, err := taos.Query("select * from tb1") _, err = db.Exec("create table if not exists example_stmt.tb1(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")")
if err != nil { if err != nil {
fmt.Println("failed to select from table, err:", err) panic(err)
return
} }
stmt := db.InsertStmt()
defer rows.Close() err = stmt.Prepare("insert into example_stmt.tb1 values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
for rows.Next() { if err != nil {
var r struct { panic(err)
ts time.Time
a int
} }
err := rows.Scan(&r.ts, &r.a) now := time.Now()
params := make([]*param.Param, 14)
params[0] = param.NewParam(2).
AddTimestamp(now, common.PrecisionMilliSecond).
AddTimestamp(now.Add(time.Second), common.PrecisionMilliSecond)
params[1] = param.NewParam(2).AddBool(true).AddNull()
params[2] = param.NewParam(2).AddTinyint(2).AddNull()
params[3] = param.NewParam(2).AddSmallint(3).AddNull()
params[4] = param.NewParam(2).AddInt(4).AddNull()
params[5] = param.NewParam(2).AddBigint(5).AddNull()
params[6] = param.NewParam(2).AddUTinyint(6).AddNull()
params[7] = param.NewParam(2).AddUSmallint(7).AddNull()
params[8] = param.NewParam(2).AddUInt(8).AddNull()
params[9] = param.NewParam(2).AddUBigint(9).AddNull()
params[10] = param.NewParam(2).AddFloat(10).AddNull()
params[11] = param.NewParam(2).AddDouble(11).AddNull()
params[12] = param.NewParam(2).AddBinary([]byte("binary")).AddNull()
params[13] = param.NewParam(2).AddNchar("nchar").AddNull()
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(6).
AddNchar(5)
err = stmt.BindParam(params, paramTypes)
if err != nil { if err != nil {
fmt.Println("scan error:\n", err) panic(err)
return }
err = stmt.AddBatch()
if err != nil {
panic(err)
} }
fmt.Println(r.ts, r.a) err = stmt.Execute()
if err != nil {
panic(err)
}
err = stmt.Close()
if err != nil {
panic(err)
} }
// select * from example_stmt.tb1
} }
``` ```
## 常见问题 </TabItem>
<TabItem value="WebSocket" label="WebSocket 连接">
1. database/sql stmt(参数绑定)相关接口崩溃
REST 不支持参数绑定相关接口,建议使用`db.Exec``db.Query`
2. 使用 `use db` 语句后执行其他语句报错 `[0x217] Database not specified or available`
REST 接口中 SQL 语句的执行无上下文关联,使用 `use db` 语句不会生效,解决办法见上方使用限制章节。
3. 使用 taosSql 不报错使用 taosRestful 报错 `[0x217] Database not specified or available`
因为 REST 接口无状态,使用 `use db` 语句不会生效,解决办法见上方使用限制章节。
4. `readBufferSize` 参数调大后无明显效果
`readBufferSize` 调大后会减少获取结果时 `syscall` 的调用。如果查询结果的数据量不大,修改该参数不会带来明显提升,如果该参数修改过大,瓶颈会在解析 JSON 数据。如果需要优化查询速度,需要根据实际情况调整该值来达到查询效果最优。
5. `disableCompression` 参数设置为 `false` 时查询效率降低
`disableCompression` 参数设置为 `false` 时查询结果会使用 `gzip` 压缩后传输,拿到数据后要先进行 `gzip` 解压。
6. `go get` 命令无法获取包,或者获取包超时
设置 Go 代理 `go env -w GOPROXY=https://goproxy.cn,direct`
## 常用 API
### database/sql API
* `sql.Open(DRIVER_NAME string, dataSourceName string) *DB`
API 用来打开 DB,返回一个类型为 \*DB 的对象。
:::info
API 成功创建的时候,并没有做权限等检查,只有在真正执行 Query 或者 Exec 的时候才能真正的去创建连接,并同时检查 user/password/host/port 是不是合法。
:::
* `func (db *DB) Exec(query string, args ...interface{}) (Result, error)`
`sql.Open` 内置的方法,用来执行非查询相关 SQL
* `func (db *DB) Query(query string, args ...interface{}) (*Rows, error)`
`sql.Open` 内置的方法,用来执行查询语句。
### 高级功能(afAPI
`af` 包封装了连接管理、订阅、schemaless、参数绑定等 TDengine 高级功能。
#### 连接管理
* `af.Open(host, user, pass, db string, port int) (*Connector, error)`
API 通过 cgo 创建与 taosd 的连接。
* `func (conn *Connector) Close() error`
关闭与 taosd 的连接。
#### 订阅
* `func NewConsumer(conf *tmq.ConfigMap) (*Consumer, error)`
创建消费者。
* `func (c *Consumer) Subscribe(topic string, rebalanceCb RebalanceCb) error`
注意:出于兼容目的保留 `rebalanceCb` 参数,当前未使用
订阅单个主题。
* `func (c *Consumer) SubscribeTopics(topics []string, rebalanceCb RebalanceCb) error`
注意:出于兼容目的保留 `rebalanceCb` 参数,当前未使用
订阅主题。
* `func (c *Consumer) Poll(timeoutMs int) tmq.Event`
轮询消息。
* `func (c *Consumer) Commit() ([]tmq.TopicPartition, error)`
注意:出于兼容目的保留 `tmq.TopicPartition` 参数,当前未使用
提交消息。
* `func (c *Consumer) Assignment() (partitions []tmq.TopicPartition, err error)`
获取消费进度。(需要 TDengine >= 3.0.5.0 driver-go >= v3.5.0)
* `func (c *Consumer) Seek(partition tmq.TopicPartition, ignoredTimeoutMs int) error`
注意:出于兼容目的保留 `ignoredTimeoutMs` 参数,当前未使用
按照指定的进度消费。(需要 TDengine >= 3.0.5.0 driver-go >= v3.5.0)
* `func (c *Consumer) Close() error`
关闭连接。
#### schemaless
* `func (conn *Connector) InfluxDBInsertLines(lines []string, precision string) error`
写入 InfluxDB 行协议。
* `func (conn *Connector) OpenTSDBInsertTelnetLines(lines []string) error` ```go
package main
写入 OpenTDSB telnet 协议数据。 import (
"database/sql"
"fmt"
"time"
* `func (conn *Connector) OpenTSDBInsertJsonPayload(payload string) error` "github.com/taosdata/driver-go/v3/common"
"github.com/taosdata/driver-go/v3/common/param"
_ "github.com/taosdata/driver-go/v3/taosRestful"
"github.com/taosdata/driver-go/v3/ws/stmt"
)
写入 OpenTSDB JSON 协议数据。 func main() {
db, err := sql.Open("taosRestful", "root:taosdata@http(localhost:6041)/")
if err != nil {
panic(err)
}
defer db.Close()
prepareEnv(db)
config := stmt.NewConfig("ws://127.0.0.1:6041/rest/stmt", 0)
config.SetConnectUser("root")
config.SetConnectPass("taosdata")
config.SetConnectDB("example_ws_stmt")
config.SetMessageTimeout(common.DefaultMessageTimeout)
config.SetWriteWait(common.DefaultWriteWait)
config.SetErrorHandler(func(connector *stmt.Connector, err error) {
panic(err)
})
config.SetCloseHandler(func() {
fmt.Println("stmt connector closed")
})
connector, err := stmt.NewConnector(config)
if err != nil {
panic(err)
}
now := time.Now()
{
stmt, err := connector.Init()
if err != nil {
panic(err)
}
err = stmt.Prepare("insert into ? using all_json tags(?) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
if err != nil {
panic(err)
}
err = stmt.SetTableName("tb1")
if err != nil {
panic(err)
}
err = stmt.SetTags(param.NewParam(1).AddJson([]byte(`{"tb":1}`)), param.NewColumnType(1).AddJson(0))
if err != nil {
panic(err)
}
params := []*param.Param{
param.NewParam(3).AddTimestamp(now, 0).AddTimestamp(now.Add(time.Second), 0).AddTimestamp(now.Add(time.Second*2), 0),
param.NewParam(3).AddBool(true).AddNull().AddBool(true),
param.NewParam(3).AddTinyint(1).AddNull().AddTinyint(1),
param.NewParam(3).AddSmallint(1).AddNull().AddSmallint(1),
param.NewParam(3).AddInt(1).AddNull().AddInt(1),
param.NewParam(3).AddBigint(1).AddNull().AddBigint(1),
param.NewParam(3).AddUTinyint(1).AddNull().AddUTinyint(1),
param.NewParam(3).AddUSmallint(1).AddNull().AddUSmallint(1),
param.NewParam(3).AddUInt(1).AddNull().AddUInt(1),
param.NewParam(3).AddUBigint(1).AddNull().AddUBigint(1),
param.NewParam(3).AddFloat(1).AddNull().AddFloat(1),
param.NewParam(3).AddDouble(1).AddNull().AddDouble(1),
param.NewParam(3).AddBinary([]byte("test_binary")).AddNull().AddBinary([]byte("test_binary")),
param.NewParam(3).AddNchar("test_nchar").AddNull().AddNchar("test_nchar"),
}
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0)
err = stmt.BindParam(params, paramTypes)
if err != nil {
panic(err)
}
err = stmt.AddBatch()
if err != nil {
panic(err)
}
err = stmt.Exec()
if err != nil {
panic(err)
}
affected := stmt.GetAffectedRows()
fmt.Println("all_json affected rows:", affected)
err = stmt.Close()
if err != nil {
panic(err)
}
}
{
stmt, err := connector.Init()
if err != nil {
panic(err)
}
err = stmt.Prepare("insert into ? using all_all tags(?,?,?,?,?,?,?,?,?,?,?,?,?,?) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
err = stmt.SetTableName("tb1")
if err != nil {
panic(err)
}
#### 参数绑定 err = stmt.SetTableName("tb2")
if err != nil {
panic(err)
}
err = stmt.SetTags(
param.NewParam(14).
AddTimestamp(now, 0).
AddBool(true).
AddTinyint(2).
AddSmallint(2).
AddInt(2).
AddBigint(2).
AddUTinyint(2).
AddUSmallint(2).
AddUInt(2).
AddUBigint(2).
AddFloat(2).
AddDouble(2).
AddBinary([]byte("tb2")).
AddNchar("tb2"),
param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0),
)
if err != nil {
panic(err)
}
params := []*param.Param{
param.NewParam(3).AddTimestamp(now, 0).AddTimestamp(now.Add(time.Second), 0).AddTimestamp(now.Add(time.Second*2), 0),
param.NewParam(3).AddBool(true).AddNull().AddBool(true),
param.NewParam(3).AddTinyint(1).AddNull().AddTinyint(1),
param.NewParam(3).AddSmallint(1).AddNull().AddSmallint(1),
param.NewParam(3).AddInt(1).AddNull().AddInt(1),
param.NewParam(3).AddBigint(1).AddNull().AddBigint(1),
param.NewParam(3).AddUTinyint(1).AddNull().AddUTinyint(1),
param.NewParam(3).AddUSmallint(1).AddNull().AddUSmallint(1),
param.NewParam(3).AddUInt(1).AddNull().AddUInt(1),
param.NewParam(3).AddUBigint(1).AddNull().AddUBigint(1),
param.NewParam(3).AddFloat(1).AddNull().AddFloat(1),
param.NewParam(3).AddDouble(1).AddNull().AddDouble(1),
param.NewParam(3).AddBinary([]byte("test_binary")).AddNull().AddBinary([]byte("test_binary")),
param.NewParam(3).AddNchar("test_nchar").AddNull().AddNchar("test_nchar"),
}
paramTypes := param.NewColumnType(14).
AddTimestamp().
AddBool().
AddTinyint().
AddSmallint().
AddInt().
AddBigint().
AddUTinyint().
AddUSmallint().
AddUInt().
AddUBigint().
AddFloat().
AddDouble().
AddBinary(0).
AddNchar(0)
err = stmt.BindParam(params, paramTypes)
if err != nil {
panic(err)
}
err = stmt.AddBatch()
if err != nil {
panic(err)
}
err = stmt.Exec()
if err != nil {
panic(err)
}
affected := stmt.GetAffectedRows()
fmt.Println("all_all affected rows:", affected)
err = stmt.Close()
if err != nil {
panic(err)
}
* `func (conn *Connector) StmtExecute(sql string, params *param.Param) (res driver.Result, err error)` }
}
参数绑定单行插入。 func prepareEnv(db *sql.DB) {
steps := []string{
"create database example_ws_stmt",
"create table example_ws_stmt.all_json(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")" +
"tags(t json)",
"create table example_ws_stmt.all_all(" +
"ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")" +
"tags(" +
"tts timestamp," +
"tc1 bool," +
"tc2 tinyint," +
"tc3 smallint," +
"tc4 int," +
"tc5 bigint," +
"tc6 tinyint unsigned," +
"tc7 smallint unsigned," +
"tc8 int unsigned," +
"tc9 bigint unsigned," +
"tc10 float," +
"tc11 double," +
"tc12 binary(20)," +
"tc13 nchar(20))",
}
for _, step := range steps {
_, err := db.Exec(step)
if err != nil {
panic(err)
}
}
}
* `func (conn *Connector) InsertStmt() *insertstmt.InsertStmt` ```
初始化参数。 </TabItem>
</Tabs>
* `func (stmt *InsertStmt) Prepare(sql string) error` ### 无模式写入
参数绑定预处理 SQL 语句。 <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="原生连接">
* `func (stmt *InsertStmt) SetTableName(name string) error` ```go
import (
"fmt"
参数绑定设置表名。 "github.com/taosdata/driver-go/v3/af"
)
* `func (stmt *InsertStmt) SetSubTableName(name string) error` func main() {
conn, err := af.Open("localhost", "root", "taosdata", "", 6030)
if err != nil {
fmt.Println("fail to connect, err:", err)
}
defer conn.Close()
_, err = conn.Exec("create database if not exists example")
if err != nil {
panic(err)
}
_, err = conn.Exec("use example")
if err != nil {
panic(err)
}
influxdbData := "st,t1=3i64,t2=4f64,t3=\"t3\" c1=3i64,c3=L\"passit\",c2=false,c4=4f64 1626006833639000000"
err = conn.InfluxDBInsertLines([]string{influxdbData}, "ns")
if err != nil {
panic(err)
}
telnetData := "stb0_0 1626006833 4 host=host0 interface=eth0"
err = conn.OpenTSDBInsertTelnetLines([]string{telnetData})
if err != nil {
panic(err)
}
jsonData := "{\"metric\": \"meter_current\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"
err = conn.OpenTSDBInsertJsonPayload(jsonData)
if err != nil {
panic(err)
}
}
```
参数绑定设置子表名。 </TabItem>
<TabItem value="WebSocket" label="WebSocket 连接">
* `func (stmt *InsertStmt) BindParam(params []*param.Param, bindType *param.ColumnType) error` ```go
import (
"database/sql"
"log"
"time"
参数绑定多行数据。 "github.com/taosdata/driver-go/v3/common"
_ "github.com/taosdata/driver-go/v3/taosWS"
"github.com/taosdata/driver-go/v3/ws/schemaless"
)
* `func (stmt *InsertStmt) AddBatch() error` func main() {
db, err := sql.Open("taosWS", "root:taosdata@ws(localhost:6041)/")
if err != nil {
log.Fatal(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists schemaless_ws")
if err != nil {
log.Fatal(err)
}
s, err := schemaless.NewSchemaless(schemaless.NewConfig("ws://localhost:6041/rest/schemaless", 1,
schemaless.SetDb("schemaless_ws"),
schemaless.SetReadTimeout(10*time.Second),
schemaless.SetWriteTimeout(10*time.Second),
schemaless.SetUser("root"),
schemaless.SetPassword("taosdata"),
schemaless.SetErrorHandler(func(err error) {
log.Fatal(err)
}),
))
if err != nil {
panic(err)
}
influxdbData := "st,t1=3i64,t2=4f64,t3=\"t3\" c1=3i64,c3=L\"passit\",c2=false,c4=4f64 1626006833639000000"
telnetData := "stb0_0 1626006833 4 host=host0 interface=eth0"
jsonData := "{\"metric\": \"meter_current\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"
添加到参数绑定批处理。 err = s.Insert(influxdbData, schemaless.InfluxDBLineProtocol, "ns", 0, common.GetReqID())
if err != nil {
panic(err)
}
err = s.Insert(telnetData, schemaless.OpenTSDBTelnetLineProtocol, "ms", 0, common.GetReqID())
if err != nil {
panic(err)
}
err = s.Insert(jsonData, schemaless.OpenTSDBJsonFormatProtocol, "ms", 0, common.GetReqID())
if err != nil {
panic(err)
}
}
```
* `func (stmt *InsertStmt) Execute() error` </TabItem>
</Tabs>
执行参数绑定。 ### 执行带有 reqId 的无模式写入
* `func (stmt *InsertStmt) GetAffectedRows() int` ```go
func (s *Schemaless) Insert(lines string, protocol int, precision string, ttl int, reqID int64) error
```
获取参数绑定插入受影响行数 可以通过 `common.GetReqID()` 获取唯一 id
* `func (stmt *InsertStmt) Close() error` ### 数据订阅
结束参数绑定。 TDengine Go 连接器支持订阅功能,应用 API 如下:
### 通过 WebSocket 订阅 #### 创建 Topic
* `func NewConsumer(conf *tmq.ConfigMap) (*Consumer, error)` ```go
db, err := af.Open("", "root", "taosdata", "", 0)
if err != nil {
panic(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists example_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic if not exists example_tmq_topic as DATABASE example_tmq")
if err != nil {
panic(err)
}
```
创建消费者。 #### 创建 Consumer
* `func (c *Consumer) Subscribe(topic string, rebalanceCb RebalanceCb) error` ```go
注意:出于兼容目的保留 `rebalanceCb` 参数,当前未使用 consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"group.id": "test",
"auto.offset.reset": "earliest",
"td.connect.ip": "127.0.0.1",
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"td.connect.port": "6030",
"client.id": "test_tmq_client",
"enable.auto.commit": "false",
"msg.with.table.name": "true",
})
if err != nil {
panic(err)
}
```
订阅单个主题。 #### 订阅消费数据
* `func (c *Consumer) SubscribeTopics(topics []string, rebalanceCb RebalanceCb) error` ```go
注意:出于兼容目的保留 `rebalanceCb` 参数,当前未使用 err = consumer.Subscribe("example_tmq_topic", nil)
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Fprintf(os.Stderr, "%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
```
订阅主题。 #### 指定订阅 Offset
* `func (c *Consumer) Poll(timeoutMs int) tmq.Event` ```go
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
```
轮询消息。 #### 关闭订阅
* `func (c *Consumer) Commit() ([]tmq.TopicPartition, error)` ```go
注意:出于兼容目的保留 `tmq.TopicPartition` 参数,当前未使用 err = consumer.Close()
if err != nil {
panic(err)
}
```
提交消息。 #### 完整示例
* `func (c *Consumer) Assignment() (partitions []tmq.TopicPartition, err error)` <Tabs defaultValue="native" groupId="connect">
<TabItem value="native" label="原生连接">
获取消费进度。(需要 TDengine >= 3.0.5.0 driver-go >= v3.5.0) ```go
package main
* `func (c *Consumer) Seek(partition tmq.TopicPartition, ignoredTimeoutMs int) error` import (
注意:出于兼容目的保留 `ignoredTimeoutMs` 参数,当前未使用 "fmt"
"os"
按照指定的进度消费。(需要 TDengine >= 3.0.5.0 driver-go >= v3.5.0) "github.com/taosdata/driver-go/v3/af"
"github.com/taosdata/driver-go/v3/af/tmq"
tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
)
* `func (c *Consumer) Close() error` func main() {
db, err := af.Open("", "root", "taosdata", "", 0)
if err != nil {
panic(err)
}
defer db.Close()
_, err = db.Exec("create database if not exists example_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic if not exists example_tmq_topic as DATABASE example_tmq")
if err != nil {
panic(err)
}
if err != nil {
panic(err)
}
consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"group.id": "test",
"auto.offset.reset": "earliest",
"td.connect.ip": "127.0.0.1",
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"td.connect.port": "6030",
"client.id": "test_tmq_client",
"enable.auto.commit": "false",
"msg.with.table.name": "true",
})
if err != nil {
panic(err)
}
err = consumer.Subscribe("example_tmq_topic", nil)
if err != nil {
panic(err)
}
_, err = db.Exec("create table example_tmq.t1 (ts timestamp,v int)")
if err != nil {
panic(err)
}
_, err = db.Exec("insert into example_tmq.t1 values(now,1)")
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Fprintf(os.Stderr, "%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
关闭连接。 partitions, err = consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
}
完整订阅示例参见 [GitHub 示例文件](https://github.com/taosdata/driver-go/blob/main/examples/tmqoverws/main.go) err = consumer.Close()
if err != nil {
panic(err)
}
}
```
### 通过 WebSocket 进行参数绑定 </TabItem>
<TabItem value="WebSocket" label="WebSocket 连接">
* `func NewConnector(config *Config) (*Connector, error)` ```go
package main
创建连接。 import (
"database/sql"
"fmt"
* `func (c *Connector) Init() (*Stmt, error)` "github.com/taosdata/driver-go/v3/common"
tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
_ "github.com/taosdata/driver-go/v3/taosRestful"
"github.com/taosdata/driver-go/v3/ws/tmq"
)
初始化参数。 func main() {
db, err := sql.Open("taosRestful", "root:taosdata@http(localhost:6041)/")
if err != nil {
panic(err)
}
defer db.Close()
prepareEnv(db)
consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
"ws.url": "ws://127.0.0.1:6041/rest/tmq",
"ws.message.channelLen": uint(0),
"ws.message.timeout": common.DefaultMessageTimeout,
"ws.message.writeWait": common.DefaultWriteWait,
"td.connect.user": "root",
"td.connect.pass": "taosdata",
"group.id": "example",
"client.id": "example_consumer",
"auto.offset.reset": "earliest",
})
if err != nil {
panic(err)
}
err = consumer.Subscribe("example_ws_tmq_topic", nil)
if err != nil {
panic(err)
}
go func() {
_, err := db.Exec("create table example_ws_tmq.t_all(ts timestamp," +
"c1 bool," +
"c2 tinyint," +
"c3 smallint," +
"c4 int," +
"c5 bigint," +
"c6 tinyint unsigned," +
"c7 smallint unsigned," +
"c8 int unsigned," +
"c9 bigint unsigned," +
"c10 float," +
"c11 double," +
"c12 binary(20)," +
"c13 nchar(20)" +
")")
if err != nil {
panic(err)
}
_, err = db.Exec("insert into example_ws_tmq.t_all values(now,true,2,3,4,5,6,7,8,9,10.123,11.123,'binary','nchar')")
if err != nil {
panic(err)
}
}()
for i := 0; i < 5; i++ {
ev := consumer.Poll(500)
if ev != nil {
switch e := ev.(type) {
case *tmqcommon.DataMessage:
fmt.Printf("get message:%v\n", e)
case tmqcommon.Error:
fmt.Printf("%% Error: %v: %v\n", e.Code(), e)
panic(e)
}
consumer.Commit()
}
}
partitions, err := consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
err = consumer.Seek(tmqcommon.TopicPartition{
Topic: partitions[i].Topic,
Partition: partitions[i].Partition,
Offset: 0,
}, 0)
if err != nil {
panic(err)
}
}
* `func (c *Connector) Close() error` partitions, err = consumer.Assignment()
if err != nil {
panic(err)
}
for i := 0; i < len(partitions); i++ {
fmt.Println(partitions[i])
}
关闭连接。 err = consumer.Close()
if err != nil {
panic(err)
}
}
* `func (s *Stmt) Prepare(sql string) error` func prepareEnv(db *sql.DB) {
_, err := db.Exec("create database example_ws_tmq WAL_RETENTION_PERIOD 86400")
if err != nil {
panic(err)
}
_, err = db.Exec("create topic example_ws_tmq_topic as database example_ws_tmq")
if err != nil {
panic(err)
}
}
```
参数绑定预处理 SQL 语句。 </TabItem>
</Tabs>
* `func (s *Stmt) SetTableName(name string) error` ### 更多示例程序
参数绑定设置表名。 * [示例程序](https://github.com/taosdata/driver-go/tree/3.0/examples)
* [视频教程](https://www.taosdata.com/blog/2020/11/11/1951.html)
* `func (s *Stmt) SetTags(tags *param.Param, bindType *param.ColumnType) error` ## 常见问题
参数绑定设置标签。 1. database/sql stmt(参数绑定)相关接口崩溃
* `func (s *Stmt) BindParam(params []*param.Param, bindType *param.ColumnType) error` REST 不支持参数绑定相关接口,建议使用`db.Exec``db.Query`
参数绑定多行数据。 2. 使用 `use db` 语句后执行其他语句报错 `[0x217] Database not specified or available`
* `func (s *Stmt) AddBatch() error` REST 接口中 SQL 语句的执行无上下文关联,使用 `use db` 语句不会生效,解决办法见上方使用限制章节。
添加到参数绑定批处理。 3. 使用 taosSql 不报错使用 taosRestful 报错 `[0x217] Database not specified or available`
* `func (s *Stmt) Exec() error` 因为 REST 接口无状态,使用 `use db` 语句不会生效,解决办法见上方使用限制章节。
执行参数绑定。 4. `readBufferSize` 参数调大后无明显效果
* `func (s *Stmt) GetAffectedRows() int` `readBufferSize` 调大后会减少获取结果时 `syscall` 的调用。如果查询结果的数据量不大,修改该参数不会带来明显提升,如果该参数修改过大,瓶颈会在解析 JSON 数据。如果需要优化查询速度,需要根据实际情况调整该值来达到查询效果最优。
获取参数绑定插入受影响行数。 5. `disableCompression` 参数设置为 `false` 时查询效率降低
* `func (s *Stmt) Close() error` `disableCompression` 参数设置为 `false` 时查询结果会使用 `gzip` 压缩后传输,拿到数据后要先进行 `gzip` 解压。
结束参数绑定。 6. `go get` 命令无法获取包,或者获取包超时
完整参数绑定示例参见 [GitHub 示例文件](https://github.com/taosdata/driver-go/blob/main/examples/stmtoverws/main.go) 设置 Go 代理 `go env -w GOPROXY=https://goproxy.cn,direct`
## API 参考 ## API 参考
......
...@@ -30,21 +30,57 @@ Websocket 连接支持所有能运行 Rust 的平台。 ...@@ -30,21 +30,57 @@ Websocket 连接支持所有能运行 Rust 的平台。
| Rust 连接器版本 | TDengine 版本 | 主要功能 | | Rust 连接器版本 | TDengine 版本 | 主要功能 |
| :----------------: | :--------------: | :--------------------------------------------------: | | :----------------: | :--------------: | :--------------------------------------------------: |
| v0.8.10 | 3.0.5.0 or later | 消息订阅:获取消费进度及按照指定进度开始消费。 | | v0.8.12 | 3.0.5.0 or later | 消息订阅:获取消费进度及按照指定进度开始消费。 |
| v0.8.0 | 3.0.4.0 | 支持无模式写入。 | | v0.8.0 | 3.0.4.0 | 支持无模式写入。 |
| v0.7.6 | 3.0.3.0 | 支持在请求中使用 req_id。 | | v0.7.6 | 3.0.3.0 | 支持在请求中使用 req_id。 |
| v0.6.0 | 3.0.0.0 | 基础功能。 | | v0.6.0 | 3.0.0.0 | 基础功能。 |
Rust 连接器仍然在快速开发中,1.0 之前无法保证其向后兼容。建议使用 3.0 版本以上的 TDengine,以避免已知问题。 Rust 连接器仍然在快速开发中,1.0 之前无法保证其向后兼容。建议使用 3.0 版本以上的 TDengine,以避免已知问题。
## 安装 ## 处理错误
在报错后,可以获取到错误的具体信息:
```rust
match conn.exec(sql) {
Ok(_) => {
Ok(())
}
Err(e) => {
eprintln!("ERROR: {:?}", e);
Err(e)
}
}
```
## TDengine DataType 和 Rust DataType
TDengine 目前支持时间戳、数字、字符、布尔类型,与 Rust 对应类型转换如下:
| TDengine DataType | Rust DataType |
| ----------------- | ----------------- |
| TIMESTAMP | Timestamp |
| INT | i32 |
| BIGINT | i64 |
| FLOAT | f32 |
| DOUBLE | f64 |
| SMALLINT | i16 |
| TINYINT | i8 |
| BOOL | bool |
| BINARY | Vec<u8\> |
| NCHAR | String |
| JSON | serde_json::Value |
**注意**:JSON 类型仅在 tag 中支持。
## 安装步骤
### 安装前准备 ### 安装前准备
* 安装 Rust 开发工具链 * 安装 Rust 开发工具链
* 如果使用原生连接,请安装 TDengine 客户端驱动,具体步骤请参考[安装客户端驱动](../#安装客户端驱动) * 如果使用原生连接,请安装 TDengine 客户端驱动,具体步骤请参考[安装客户端驱动](../#安装客户端驱动)
### 添加 taos 依赖 ### 安装连接器
根据选择的连接方式,按照如下说明在 [Rust](https://rust-lang.org) 项目中添加 [taos][taos] 依赖: 根据选择的连接方式,按照如下说明在 [Rust](https://rust-lang.org) 项目中添加 [taos][taos] 依赖:
...@@ -151,7 +187,8 @@ let builder = TaosBuilder::from_dsn("taos://localhost:6030")?; ...@@ -151,7 +187,8 @@ let builder = TaosBuilder::from_dsn("taos://localhost:6030")?;
let conn1 = builder.build(); let conn1 = builder.build();
// use websocket protocol. // use websocket protocol.
let conn2 = TaosBuilder::from_dsn("taos+ws://localhost:6041")?; let builder2 = TaosBuilder::from_dsn("taos+ws://localhost:6041")?;
let conn2 = builder2.build();
``` ```
建立连接后,您可以进行相关数据库操作: 建立连接后,您可以进行相关数据库操作:
...@@ -233,41 +270,191 @@ async fn demo(taos: &Taos, db: &str) -> Result<(), Error> { ...@@ -233,41 +270,191 @@ async fn demo(taos: &Taos, db: &str) -> Result<(), Error> {
## 使用示例 ## 使用示例
### 写入数据 ### 创建数据库和表
#### SQL 写入 ```rust
use taos::*;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let dsn = "taos://localhost:6030";
let builder = TaosBuilder::from_dsn(dsn)?;
let taos = builder.build()?;
let db = "query";
// create database
taos.exec_many([
format!("DROP DATABASE IF EXISTS `{db}`"),
format!("CREATE DATABASE `{db}`"),
format!("USE `{db}`"),
])
.await?;
// create table
taos.exec_many([
// create super table
"CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) \
TAGS (`groupid` INT, `location` BINARY(16))",
// create child table
"CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')",
]).await?;
}
```
> **注意**:如果不使用 `use db` 指定数据库,则后续对表的操作都需要增加数据库名称作为前缀,如 db.tb。
### 插入数据
<RustInsert /> <RustInsert />
#### STMT 写入 ### 查询数据
<RustQuery />
### 执行带有 req_id 的 SQL
此 req_id 可用于请求链路追踪。
```rust
let rs = taos.query_with_req_id("select * from stable where tag1 is null", 1)?;
```
### 通过参数绑定写入数据
TDengine 的 Rust 连接器实现了参数绑定方式对数据写入(INSERT)场景的支持。采用这种方式写入数据时,能避免 SQL 语法解析的资源消耗,从而在很多情况下显著提升写入性能。
参数绑定接口详见[API参考](#stmt-api)
<RustBind /> <RustBind />
#### Schemaless 写入 ### 无模式写入
TDengine 支持无模式写入功能。无模式写入兼容 InfluxDB 的 行协议(Line Protocol)、OpenTSDB 的 telnet 行协议和 OpenTSDB 的 JSON 格式协议。详情请参见[无模式写入](../../reference/schemaless/)。
<RustSml /> <RustSml />
### 查询数据 ### 执行带有 req_id 的无模式写入
<RustQuery /> 此 req_id 可用于请求链路追踪。
## API 参考 ```rust
let sml_data = SmlDataBuilder::default()
.protocol(SchemalessProtocol::Line)
.data(data)
.req_id(100u64)
.build()?;
client.put(&sml_data)?
```
### 数据订阅
TDengine 通过消息队列 [TMQ](../../../taos-sql/tmq/) 启动一个订阅。
#### 创建 Topic
### 连接构造器 ```rust
taos.exec_many([
// create topic for subscription
format!("CREATE TOPIC tmq_meters with META AS DATABASE {db}")
])
.await?;
```
通过 DSN 来构建一个连接器构造器。 #### 创建 Consumer
从 DSN 开始,构建一个 TMQ 连接器。
```rust ```rust
let cfg = TaosBuilder::default().build()?; let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
``` ```
使用 `builder` 对象创建多个连接 创建消费者
```rust ```rust
let conn: Taos = cfg.build(); let mut consumer = tmq.build()?;
``` ```
### 连接池 #### 订阅消费数据
消费者可订阅一个或多个 `TOPIC`。
```rust
consumer.subscribe(["tmq_meters"]).await?;
```
TMQ 消息队列是一个 [futures::Stream](https://docs.rs/futures/latest/futures/stream/index.html) 类型,可以使用相应 API 对每个消息进行消费,并通过 `.commit` 进行已消费标记。
```rust
{
let mut stream = consumer.stream();
while let Some((offset, message)) = stream.try_next().await? {
// get information from offset
// the topic
let topic = offset.topic();
// the vgroup id, like partition id in kafka.
let vgroup_id = offset.vgroup_id();
println!("* in vgroup id {vgroup_id} of topic {topic}\n");
if let Some(data) = message.into_data() {
while let Some(block) = data.fetch_raw_block().await? {
// one block for one table, get table name if needed
let name = block.table_name();
let records: Vec<Record> = block.deserialize().try_collect()?;
println!(
"** table: {}, got {} records: {:#?}\n",
name.unwrap(),
records.len(),
records
);
}
}
consumer.commit(offset).await?;
}
}
```
获取消费进度:
版本要求 connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
let assignments = consumer.assignments().await.unwrap();
```
#### 指定订阅 Offset
按照指定的进度消费:
版本要求 connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
consumer.offset_seek(topic, vgroup_id, offset).await;
```
#### 关闭订阅
```rust
consumer.unsubscribe().await;
```
对于 TMQ DSN, 有以下配置项可以进行设置,需要注意的是,`group.id` 是必须的。
- `group.id`: 同一个消费者组,将以至少消费一次的方式进行消息负载均衡。
- `client.id`: 可选的订阅客户端识别项。
- `auto.offset.reset`: 可选初始化订阅起点, *earliest* 为从头开始订阅, *latest* 为仅从最新数据开始订阅,默认为从头订阅。注意,此选项在同一个 `group.id` 中仅生效一次。
- `enable.auto.commit`: 当设置为 `true` 时,将启用自动标记模式,当对数据一致性不敏感时,可以启用此方式。
- `auto.commit.interval.ms`: 自动标记的时间间隔。
#### 完整示例
完整订阅示例参见 [GitHub 示例文件](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/rust/nativeexample/examples/subscribe_demo.rs).
### 与连接池使用
在复杂应用中,建议启用连接池。[taos] 的连接池默认(异步模式)使用 [deadpool] 实现。 在复杂应用中,建议启用连接池。[taos] 的连接池默认(异步模式)使用 [deadpool] 实现。
...@@ -295,7 +482,17 @@ let pool: Pool<TaosBuilder> = Pool::builder(Manager::from_dsn(self.dsn.clone()). ...@@ -295,7 +482,17 @@ let pool: Pool<TaosBuilder> = Pool::builder(Manager::from_dsn(self.dsn.clone()).
let taos = pool.get()?; let taos = pool.get()?;
``` ```
### 连接 ### 更多示例程序
示例程序源码位于 `TDengine/examples/rust` 下:
请参考:[rust example](https://github.com/taosdata/TDengine/tree/3.0/examples/rust)
## 常见问题
请参考 [FAQ](../../../train-faq/faq)
## API 参考
[Taos][struct.Taos] 对象提供了多个数据库操作的 API: [Taos][struct.Taos] 对象提供了多个数据库操作的 API:
...@@ -381,9 +578,13 @@ let taos = pool.get()?; ...@@ -381,9 +578,13 @@ let taos = pool.get()?;
- `.create_database(database: &str)`: 执行 `CREATE DATABASE` 语句。 - `.create_database(database: &str)`: 执行 `CREATE DATABASE` 语句。
- `.use_database(database: &str)`: 执行 `USE` 语句。 - `.use_database(database: &str)`: 执行 `USE` 语句。
除此之外,该结构也是 [参数绑定](#参数绑定接口) 和 [行协议接口](#行协议接口) 的入口,使用方法请参考具体的 API 说明。 除此之外,该结构也是参数绑定和行协议接口的入口,使用方法请参考具体的 API 说明。
### 参数绑定接口 <p>
<a id="stmt-api" style={{color:'#141414'}}>
参数绑定接口
</a>
</p>
与 C 接口类似,Rust 提供参数绑定接口。首先,通过 [Taos][struct.Taos] 对象创建一个 SQL 语句的参数绑定对象 [Stmt]: 与 C 接口类似,Rust 提供参数绑定接口。首先,通过 [Taos][struct.Taos] 对象创建一个 SQL 语句的参数绑定对象 [Stmt]:
...@@ -394,7 +595,7 @@ stmt.prepare("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?; ...@@ -394,7 +595,7 @@ stmt.prepare("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?;
参数绑定对象提供了一组接口用于实现参数绑定: 参数绑定对象提供了一组接口用于实现参数绑定:
#### `.set_tbname(name)` `.set_tbname(name)`
用于绑定表名。 用于绑定表名。
...@@ -403,7 +604,7 @@ let mut stmt = taos.stmt("insert into ? values(? ,?)")?; ...@@ -403,7 +604,7 @@ let mut stmt = taos.stmt("insert into ? values(? ,?)")?;
stmt.set_tbname("d0")?; stmt.set_tbname("d0")?;
``` ```
#### `.set_tags(&[tag])` `.set_tags(&[tag])`
当 SQL 语句使用超级表时,用于绑定子表表名和标签值: 当 SQL 语句使用超级表时,用于绑定子表表名和标签值:
...@@ -413,7 +614,7 @@ stmt.set_tbname("d0")?; ...@@ -413,7 +614,7 @@ stmt.set_tbname("d0")?;
stmt.set_tags(&[Value::VarChar("涛思".to_string())])?; stmt.set_tags(&[Value::VarChar("涛思".to_string())])?;
``` ```
#### `.bind(&[column])` `.bind(&[column])`
用于绑定值类型。使用 [ColumnView] 结构体构建需要的类型并绑定: 用于绑定值类型。使用 [ColumnView] 结构体构建需要的类型并绑定:
...@@ -437,7 +638,7 @@ let params = vec![ ...@@ -437,7 +638,7 @@ let params = vec![
let rows = stmt.bind(&params)?.add_batch()?.execute()?; let rows = stmt.bind(&params)?.add_batch()?.execute()?;
``` ```
#### `.execute()` `.execute()`
执行 SQL。[Stmt] 对象可以复用,在执行后可以重新绑定并执行。执行前请确保所有数据已通过 `.add_batch` 加入到执行队列中。 执行 SQL。[Stmt] 对象可以复用,在执行后可以重新绑定并执行。执行前请确保所有数据已通过 `.add_batch` 加入到执行队列中。
...@@ -452,92 +653,6 @@ stmt.execute()?; ...@@ -452,92 +653,6 @@ stmt.execute()?;
一个可运行的示例请见 [GitHub 上的示例](https://github.com/taosdata/taos-connector-rust/blob/main/examples/bind.rs)。 一个可运行的示例请见 [GitHub 上的示例](https://github.com/taosdata/taos-connector-rust/blob/main/examples/bind.rs)。
### 订阅
TDengine 通过消息队列 [TMQ](../../../taos-sql/tmq/) 启动一个订阅。
从 DSN 开始,构建一个 TMQ 连接器。
```rust
let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
```
创建消费者:
```rust
let mut consumer = tmq.build()?;
```
消费者可订阅一个或多个 `TOPIC`。
```rust
consumer.subscribe(["tmq_meters"]).await?;
```
TMQ 消息队列是一个 [futures::Stream](https://docs.rs/futures/latest/futures/stream/index.html) 类型,可以使用相应 API 对每个消息进行消费,并通过 `.commit` 进行已消费标记。
```rust
{
let mut stream = consumer.stream();
while let Some((offset, message)) = stream.try_next().await? {
// get information from offset
// the topic
let topic = offset.topic();
// the vgroup id, like partition id in kafka.
let vgroup_id = offset.vgroup_id();
println!("* in vgroup id {vgroup_id} of topic {topic}\n");
if let Some(data) = message.into_data() {
while let Some(block) = data.fetch_raw_block().await? {
// one block for one table, get table name if needed
let name = block.table_name();
let records: Vec<Record> = block.deserialize().try_collect()?;
println!(
"** table: {}, got {} records: {:#?}\n",
name.unwrap(),
records.len(),
records
);
}
}
consumer.commit(offset).await?;
}
}
```
获取消费进度:
版本要求 connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
let assignments = consumer.assignments().await.unwrap();
```
按照指定的进度消费:
版本要求 connector-rust >= v0.8.8, TDengine >= 3.0.5.0
```rust
consumer.offset_seek(topic, vgroup_id, offset).await;
```
停止订阅:
```rust
consumer.unsubscribe().await;
```
对于 TMQ DSN, 有以下配置项可以进行设置,需要注意的是,`group.id` 是必须的。
- `group.id`: 同一个消费者组,将以至少消费一次的方式进行消息负载均衡。
- `client.id`: 可选的订阅客户端识别项。
- `auto.offset.reset`: 可选初始化订阅起点, *earliest* 为从头开始订阅, *latest* 为仅从最新数据开始订阅,默认为从头订阅。注意,此选项在同一个 `group.id` 中仅生效一次。
- `enable.auto.commit`: 当设置为 `true` 时,将启用自动标记模式,当对数据一致性不敏感时,可以启用此方式。
- `auto.commit.interval.ms`: 自动标记的时间间隔。
完整订阅示例参见 [GitHub 示例文件](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/rust/nativeexample/examples/subscribe_demo.rs).
其他相关结构体 API 使用说明请移步 Rust 文档托管网页:<https://docs.rs/taos>。 其他相关结构体 API 使用说明请移步 Rust 文档托管网页:<https://docs.rs/taos>。
......
...@@ -25,6 +25,16 @@ Python 连接器的源码托管在 [GitHub](https://github.com/taosdata/taos-con ...@@ -25,6 +25,16 @@ Python 连接器的源码托管在 [GitHub](https://github.com/taosdata/taos-con
无论使用什么版本的 TDengine 都建议使用最新版本的 `taospy`。 无论使用什么版本的 TDengine 都建议使用最新版本的 `taospy`。
|Python Connector 版本|主要变化|
|:-------------------:|:----:|
|2.7.9|数据订阅支持获取消费进度和重置消费进度|
|2.7.8|新增 `execute_many`|
|Python Websocket Connector 版本|主要变化|
|:----------------------------:|:-----:|
|0.2.5|1. 数据订阅支持获取消费进度和重置消费进度 <br/> 2. 支持 schemaless <br/> 3. 支持 STMT|
|0.2.4|数据订阅新增取消订阅方法|
## 处理异常 ## 处理异常
Python 连接器可能会产生 4 种异常: Python 连接器可能会产生 4 种异常:
...@@ -549,7 +559,7 @@ consumer = Consumer({"group.id": "local", "td.connect.ip": "127.0.0.1"}) ...@@ -549,7 +559,7 @@ consumer = Consumer({"group.id": "local", "td.connect.ip": "127.0.0.1"})
#### 订阅 topics #### 订阅 topics
Comsumer API 的 `subscribe` 方法用于订阅 topics,consumer 支持同时订阅多个 topic。 Consumer API 的 `subscribe` 方法用于订阅 topics,consumer 支持同时订阅多个 topic。
```python ```python
consumer.subscribe(['topic1', 'topic2']) consumer.subscribe(['topic1', 'topic2'])
...@@ -631,7 +641,7 @@ consumer = taosws.(conf={"group.id": "local", "td.connect.websocket.scheme": "ws ...@@ -631,7 +641,7 @@ consumer = taosws.(conf={"group.id": "local", "td.connect.websocket.scheme": "ws
#### 订阅 topics #### 订阅 topics
Comsumer API 的 `subscribe` 方法用于订阅 topics,consumer 支持同时订阅多个 topic。 Consumer API 的 `subscribe` 方法用于订阅 topics,consumer 支持同时订阅多个 topic。
```python ```python
consumer.subscribe(['topic1', 'topic2']) consumer.subscribe(['topic1', 'topic2'])
......
...@@ -991,18 +991,14 @@ SAMPLE(expr, k) ...@@ -991,18 +991,14 @@ SAMPLE(expr, k)
**功能说明**: 获取数据的 k 个采样值。参数 k 的合法输入范围是 1≤ k ≤ 1000。 **功能说明**: 获取数据的 k 个采样值。参数 k 的合法输入范围是 1≤ k ≤ 1000。
**返回结果类型**: 同原始数据类型, 返回结果中带有该行记录的时间戳 **返回结果类型**: 同原始数据类型。
**适用数据类型**在超级表查询中使用时,不能应用在标签之上 **适用数据类型**全部类型字段
**嵌套子查询支持**: 适用于内层查询和外层查询。 **嵌套子查询支持**: 适用于内层查询和外层查询。
**适用于**:表和超级表。 **适用于**:表和超级表。
**使用说明**
- 不能参与表达式计算;该函数可以应用在普通表和超级表上;
### TAIL ### TAIL
...@@ -1047,11 +1043,11 @@ TOP(expr, k) ...@@ -1047,11 +1043,11 @@ TOP(expr, k)
UNIQUE(expr) UNIQUE(expr)
``` ```
**功能说明**:返回该列的数值首次出现的值。该函数功能与 distinct 相似,但是可以匹配标签和时间戳信息。可以针对除时间列以外的字段进行查询,可以匹配标签和时间戳,其中的标签和时间戳是第一次出现时刻的标签和时间戳 **功能说明**:返回该列数据首次出现的值。该函数功能与 distinct 相似
**返回数据类型**:同应用的字段。 **返回数据类型**:同应用的字段。
**适用数据类型**适合于除时间类型以外的字段。 **适用数据类型**全部类型字段。
**适用于**: 表和超级表。 **适用于**: 表和超级表。
......
...@@ -164,6 +164,8 @@ extern char tsSmlTagName[]; ...@@ -164,6 +164,8 @@ extern char tsSmlTagName[];
// extern bool tsSmlDataFormat; // extern bool tsSmlDataFormat;
// extern int32_t tsSmlBatchSize; // extern int32_t tsSmlBatchSize;
extern int32_t tmqMaxTopicNum;
// wal // wal
extern int64_t tsWalFsyncDataSizeLimit; extern int64_t tsWalFsyncDataSizeLimit;
......
...@@ -145,7 +145,7 @@ enum { ...@@ -145,7 +145,7 @@ enum {
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_DROP_TOPIC, "drop-topic", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_DROP_TOPIC, "drop-topic", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_SUBSCRIBE, "subscribe", SCMSubscribeReq, SCMSubscribeRsp) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_SUBSCRIBE, "subscribe", SCMSubscribeReq, SCMSubscribeRsp)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_ASK_EP, "ask-ep", SMqAskEpReq, SMqAskEpRsp) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_ASK_EP, "ask-ep", SMqAskEpReq, SMqAskEpRsp)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_CONSUMER_LOST, "consumer-lost", SMqConsumerLostMsg, NULL) // TD_DEF_MSG_TYPE(TDMT_MND_TMQ_CONSUMER_LOST, "consumer-lost", SMqConsumerLostMsg, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_CONSUMER_RECOVER, "consumer-recover", SMqConsumerRecoverMsg, NULL) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_CONSUMER_RECOVER, "consumer-recover", SMqConsumerRecoverMsg, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_HB, "consumer-hb", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_HB, "consumer-hb", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_TMQ_DO_REBALANCE, "do-rebalance", SMqDoRebalanceMsg, NULL) TD_DEF_MSG_TYPE(TDMT_MND_TMQ_DO_REBALANCE, "do-rebalance", SMqDoRebalanceMsg, NULL)
......
...@@ -111,6 +111,12 @@ int32_t udfStartUdfd(int32_t startDnodeId); ...@@ -111,6 +111,12 @@ int32_t udfStartUdfd(int32_t startDnodeId);
*/ */
int32_t udfStopUdfd(); int32_t udfStopUdfd();
/**
* get udfd pid
*
*/
int32_t udfGetUdfdPid(int32_t* pUdfdPid);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif
......
...@@ -769,6 +769,8 @@ int32_t* taosGetErrno(); ...@@ -769,6 +769,8 @@ int32_t* taosGetErrno();
#define TSDB_CODE_TMQ_CONSUMER_MISMATCH TAOS_DEF_ERROR_CODE(0, 0x4001) #define TSDB_CODE_TMQ_CONSUMER_MISMATCH TAOS_DEF_ERROR_CODE(0, 0x4001)
#define TSDB_CODE_TMQ_CONSUMER_CLOSED TAOS_DEF_ERROR_CODE(0, 0x4002) #define TSDB_CODE_TMQ_CONSUMER_CLOSED TAOS_DEF_ERROR_CODE(0, 0x4002)
#define TSDB_CODE_TMQ_CONSUMER_ERROR TAOS_DEF_ERROR_CODE(0, 0x4003) #define TSDB_CODE_TMQ_CONSUMER_ERROR TAOS_DEF_ERROR_CODE(0, 0x4003)
#define TSDB_CODE_TMQ_TOPIC_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x4004)
#define TSDB_CODE_TMQ_GROUP_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x4005)
// stream // stream
#define TSDB_CODE_STREAM_TASK_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x4100) #define TSDB_CODE_STREAM_TASK_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x4100)
......
...@@ -749,6 +749,9 @@ static int32_t smlSendMetaMsg(SSmlHandle *info, SName *pName, SArray *pColumns, ...@@ -749,6 +749,9 @@ static int32_t smlSendMetaMsg(SSmlHandle *info, SName *pName, SArray *pColumns,
pReq.suid = pTableMeta->uid; pReq.suid = pTableMeta->uid;
pReq.source = TD_REQ_FROM_TAOX; pReq.source = TD_REQ_FROM_TAOX;
pSql = (action == SCHEMA_ACTION_ADD_COLUMN) ? "sml_add_column" : "sml_modify_column_size"; pSql = (action == SCHEMA_ACTION_ADD_COLUMN) ? "sml_add_column" : "sml_modify_column_size";
} else{
uError("SML:0x%" PRIx64 " invalid action:%d", info->id, action);
goto end;
} }
code = buildRequest(info->taos->id, pSql, strlen(pSql), NULL, false, &pRequest, 0); code = buildRequest(info->taos->id, pSql, strlen(pSql), NULL, false, &pRequest, 0);
......
...@@ -939,8 +939,6 @@ int stmtClose(TAOS_STMT* stmt) { ...@@ -939,8 +939,6 @@ int stmtClose(TAOS_STMT* stmt) {
stmtCleanSQLInfo(pStmt); stmtCleanSQLInfo(pStmt);
taosMemoryFree(stmt); taosMemoryFree(stmt);
STMT_DLOG_E("stmt freed");
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
......
...@@ -652,7 +652,7 @@ static void asyncCommitOffset(tmq_t* tmq, const TAOS_RES* pRes, int32_t type, tm ...@@ -652,7 +652,7 @@ static void asyncCommitOffset(tmq_t* tmq, const TAOS_RES* pRes, int32_t type, tm
int32_t j = 0; int32_t j = 0;
int32_t numOfVgroups = taosArrayGetSize(pTopic->vgs); int32_t numOfVgroups = taosArrayGetSize(pTopic->vgs);
for (j = 0; j < numOfVgroups; j++) { for (j = 0; j < numOfVgroups; j++) {
SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); SMqClientVg* pVg = (SMqClientVg*)taosArrayGet(pTopic->vgs, j);
if (pVg->vgId == vgId) { if (pVg->vgId == vgId) {
break; break;
} }
...@@ -666,7 +666,7 @@ static void asyncCommitOffset(tmq_t* tmq, const TAOS_RES* pRes, int32_t type, tm ...@@ -666,7 +666,7 @@ static void asyncCommitOffset(tmq_t* tmq, const TAOS_RES* pRes, int32_t type, tm
return; return;
} }
SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); SMqClientVg* pVg = (SMqClientVg*)taosArrayGet(pTopic->vgs, j);
if (pVg->offsetInfo.currentOffset.type > 0 && !tOffsetEqual(&pVg->offsetInfo.currentOffset, &pVg->offsetInfo.committedOffset)) { if (pVg->offsetInfo.currentOffset.type > 0 && !tOffsetEqual(&pVg->offsetInfo.currentOffset, &pVg->offsetInfo.committedOffset)) {
code = doSendCommitMsg(tmq, pVg, pTopic->topicName, pParamSet, j, numOfVgroups, type); code = doSendCommitMsg(tmq, pVg, pTopic->topicName, pParamSet, j, numOfVgroups, type);
...@@ -742,13 +742,15 @@ static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* us ...@@ -742,13 +742,15 @@ static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* us
static void generateTimedTask(int64_t refId, int32_t type) { static void generateTimedTask(int64_t refId, int32_t type) {
tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId); tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId);
if (tmq != NULL) { if(tmq == NULL) return;
int8_t* pTaskType = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0); int8_t* pTaskType = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0);
if(pTaskType == NULL) return;
*pTaskType = type; *pTaskType = type;
taosWriteQitem(tmq->delayedTask, pTaskType); taosWriteQitem(tmq->delayedTask, pTaskType);
tsem_post(&tmq->rspSem); tsem_post(&tmq->rspSem);
taosReleaseRef(tmqMgmt.rsetId, refId); taosReleaseRef(tmqMgmt.rsetId, refId);
}
} }
void tmqAssignAskEpTask(void* param, void* tmrId) { void tmqAssignAskEpTask(void* param, void* tmrId) {
...@@ -763,19 +765,19 @@ void tmqAssignDelayedCommitTask(void* param, void* tmrId) { ...@@ -763,19 +765,19 @@ void tmqAssignDelayedCommitTask(void* param, void* tmrId) {
taosMemoryFree(param); taosMemoryFree(param);
} }
void tmqAssignDelayedReportTask(void* param, void* tmrId) { //void tmqAssignDelayedReportTask(void* param, void* tmrId) {
int64_t refId = *(int64_t*)param; // int64_t refId = *(int64_t*)param;
tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId); // tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId);
if (tmq != NULL) { // if (tmq != NULL) {
int8_t* pTaskType = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0); // int8_t* pTaskType = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0);
*pTaskType = TMQ_DELAYED_TASK__REPORT; // *pTaskType = TMQ_DELAYED_TASK__REPORT;
taosWriteQitem(tmq->delayedTask, pTaskType); // taosWriteQitem(tmq->delayedTask, pTaskType);
tsem_post(&tmq->rspSem); // tsem_post(&tmq->rspSem);
} // }
//
taosReleaseRef(tmqMgmt.rsetId, refId); // taosReleaseRef(tmqMgmt.rsetId, refId);
taosMemoryFree(param); // taosMemoryFree(param);
} //}
int32_t tmqHbCb(void* param, SDataBuf* pMsg, int32_t code) { int32_t tmqHbCb(void* param, SDataBuf* pMsg, int32_t code) {
if (pMsg) { if (pMsg) {
......
...@@ -290,7 +290,7 @@ static const SSysDbTableSchema subscriptionSchema[] = { ...@@ -290,7 +290,7 @@ static const SSysDbTableSchema subscriptionSchema[] = {
{.name = "topic_name", .bytes = TSDB_TOPIC_FNAME_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "topic_name", .bytes = TSDB_TOPIC_FNAME_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "consumer_group", .bytes = TSDB_CGROUP_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "consumer_group", .bytes = TSDB_CGROUP_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "vgroup_id", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false}, {.name = "vgroup_id", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "consumer_id", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = false}, {.name = "consumer_id", .bytes = 32, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "offset", .bytes = TSDB_OFFSET_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "offset", .bytes = TSDB_OFFSET_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "rows", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = false}, {.name = "rows", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = false},
}; };
...@@ -352,7 +352,7 @@ static const SSysDbTableSchema connectionsSchema[] = { ...@@ -352,7 +352,7 @@ static const SSysDbTableSchema connectionsSchema[] = {
static const SSysDbTableSchema consumerSchema[] = { static const SSysDbTableSchema consumerSchema[] = {
{.name = "consumer_id", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = false}, {.name = "consumer_id", .bytes = 32, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "consumer_group", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "consumer_group", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "client_id", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "client_id", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
{.name = "status", .bytes = 20 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false}, {.name = "status", .bytes = 20 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY, .sysInfo = false},
......
...@@ -105,6 +105,8 @@ char tsSmlChildTableName[TSDB_TABLE_NAME_LEN] = ""; // user defined child table ...@@ -105,6 +105,8 @@ char tsSmlChildTableName[TSDB_TABLE_NAME_LEN] = ""; // user defined child table
// bool tsSmlDataFormat = false; // bool tsSmlDataFormat = false;
// int32_t tsSmlBatchSize = 10000; // int32_t tsSmlBatchSize = 10000;
// tmq
int32_t tmqMaxTopicNum = 20;
// query // query
int32_t tsQueryPolicy = 1; int32_t tsQueryPolicy = 1;
int32_t tsQueryRspPolicy = 0; int32_t tsQueryRspPolicy = 0;
...@@ -511,6 +513,8 @@ static int32_t taosAddServerCfg(SConfig *pCfg) { ...@@ -511,6 +513,8 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
if (cfgAddString(pCfg, "telemetryServer", tsTelemServer, 0) != 0) return -1; if (cfgAddString(pCfg, "telemetryServer", tsTelemServer, 0) != 0) return -1;
if (cfgAddInt32(pCfg, "telemetryPort", tsTelemPort, 1, 65056, 0) != 0) return -1; if (cfgAddInt32(pCfg, "telemetryPort", tsTelemPort, 1, 65056, 0) != 0) return -1;
if (cfgAddInt32(pCfg, "tmqMaxTopicNum", tmqMaxTopicNum, 1, 10000, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "transPullupInterval", tsTransPullupInterval, 1, 10000, 1) != 0) return -1; if (cfgAddInt32(pCfg, "transPullupInterval", tsTransPullupInterval, 1, 10000, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "mqRebalanceInterval", tsMqRebalanceInterval, 1, 10000, 1) != 0) return -1; if (cfgAddInt32(pCfg, "mqRebalanceInterval", tsMqRebalanceInterval, 1, 10000, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "ttlUnit", tsTtlUnit, 1, 86400 * 365, 1) != 0) return -1; if (cfgAddInt32(pCfg, "ttlUnit", tsTtlUnit, 1, 86400 * 365, 1) != 0) return -1;
...@@ -882,6 +886,8 @@ static int32_t taosSetServerCfg(SConfig *pCfg) { ...@@ -882,6 +886,8 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
tstrncpy(tsTelemServer, cfgGetItem(pCfg, "telemetryServer")->str, TSDB_FQDN_LEN); tstrncpy(tsTelemServer, cfgGetItem(pCfg, "telemetryServer")->str, TSDB_FQDN_LEN);
tsTelemPort = (uint16_t)cfgGetItem(pCfg, "telemetryPort")->i32; tsTelemPort = (uint16_t)cfgGetItem(pCfg, "telemetryPort")->i32;
tmqMaxTopicNum= cfgGetItem(pCfg, "tmqMaxTopicNum")->i32;
tsTransPullupInterval = cfgGetItem(pCfg, "transPullupInterval")->i32; tsTransPullupInterval = cfgGetItem(pCfg, "transPullupInterval")->i32;
tsMqRebalanceInterval = cfgGetItem(pCfg, "mqRebalanceInterval")->i32; tsMqRebalanceInterval = cfgGetItem(pCfg, "mqRebalanceInterval")->i32;
tsTtlUnit = cfgGetItem(pCfg, "ttlUnit")->i32; tsTtlUnit = cfgGetItem(pCfg, "ttlUnit")->i32;
......
...@@ -969,7 +969,7 @@ void taosFormatUtcTime(char* buf, int32_t bufLen, int64_t t, int32_t precision) ...@@ -969,7 +969,7 @@ void taosFormatUtcTime(char* buf, int32_t bufLen, int64_t t, int32_t precision)
default: default:
fractionLen = 0; fractionLen = 0;
ASSERT(false); return;
} }
if (taosLocalTime(&quot, &ptm, buf) == NULL) { if (taosLocalTime(&quot, &ptm, buf) == NULL) {
......
...@@ -25,14 +25,15 @@ extern "C" { ...@@ -25,14 +25,15 @@ extern "C" {
enum { enum {
MQ_CONSUMER_STATUS_REBALANCE = 1, MQ_CONSUMER_STATUS_REBALANCE = 1,
// MQ_CONSUMER_STATUS__MODIFY_IN_REB, // this value is not used anymore // MQ_CONSUMER_STATUS__MODIFY_IN_REB, // this value is not used anymore
MQ_CONSUMER_STATUS__READY, MQ_CONSUMER_STATUS_READY,
MQ_CONSUMER_STATUS__LOST, MQ_CONSUMER_STATUS_LOST,
// MQ_CONSUMER_STATUS__LOST_IN_REB, // this value is not used anymore // MQ_CONSUMER_STATUS__LOST_IN_REB, // this value is not used anymore
MQ_CONSUMER_STATUS__LOST_REBD, // MQ_CONSUMER_STATUS__LOST_REBD,
}; };\
int32_t mndInitConsumer(SMnode *pMnode); int32_t mndInitConsumer(SMnode *pMnode);
void mndCleanupConsumer(SMnode *pMnode); void mndCleanupConsumer(SMnode *pMnode);
void mndDropConsumerFromSdb(SMnode *pMnode, int64_t consumerId);
SMqConsumerObj *mndAcquireConsumer(SMnode *pMnode, int64_t consumerId); SMqConsumerObj *mndAcquireConsumer(SMnode *pMnode, int64_t consumerId);
void mndReleaseConsumer(SMnode *pMnode, SMqConsumerObj *pConsumer); void mndReleaseConsumer(SMnode *pMnode, SMqConsumerObj *pConsumer);
......
...@@ -137,12 +137,12 @@ typedef enum { ...@@ -137,12 +137,12 @@ typedef enum {
} EDndReason; } EDndReason;
typedef enum { typedef enum {
CONSUMER_UPDATE__TOUCH = 1, // rebalance req do not need change consume topic CONSUMER_UPDATE_REB_MODIFY_NOTOPIC = 1, // topic do not need modified after rebalance
CONSUMER_UPDATE__ADD, CONSUMER_UPDATE_REB_MODIFY_TOPIC, // topic need modified after rebalance
CONSUMER_UPDATE__REMOVE, CONSUMER_UPDATE_REB_MODIFY_REMOVE, // topic need removed after rebalance
CONSUMER_UPDATE__LOST, // CONSUMER_UPDATE_TIMER_LOST,
CONSUMER_UPDATE__RECOVER, CONSUMER_UPDATE_RECOVER,
CONSUMER_UPDATE__REBALANCE, // subscribe req need change consume topic CONSUMER_UPDATE_SUB_MODIFY, // modify after subscribe req
} ECsmUpdateType; } ECsmUpdateType;
typedef struct { typedef struct {
...@@ -549,7 +549,7 @@ typedef struct { ...@@ -549,7 +549,7 @@ typedef struct {
// data for display // data for display
int32_t pid; int32_t pid;
SEpSet ep; SEpSet ep;
int64_t upTime; int64_t createTime;
int64_t subscribeTime; int64_t subscribeTime;
int64_t rebalanceTime; int64_t rebalanceTime;
...@@ -560,7 +560,7 @@ typedef struct { ...@@ -560,7 +560,7 @@ typedef struct {
} SMqConsumerObj; } SMqConsumerObj;
SMqConsumerObj* tNewSMqConsumerObj(int64_t consumerId, char cgroup[TSDB_CGROUP_LEN]); SMqConsumerObj* tNewSMqConsumerObj(int64_t consumerId, char cgroup[TSDB_CGROUP_LEN]);
void tDeleteSMqConsumerObj(SMqConsumerObj* pConsumer); void tDeleteSMqConsumerObj(SMqConsumerObj* pConsumer, bool delete);
int32_t tEncodeSMqConsumerObj(void** buf, const SMqConsumerObj* pConsumer); int32_t tEncodeSMqConsumerObj(void** buf, const SMqConsumerObj* pConsumer);
void* tDecodeSMqConsumerObj(const void* buf, SMqConsumerObj* pConsumer, int8_t sver); void* tDecodeSMqConsumerObj(const void* buf, SMqConsumerObj* pConsumer, int8_t sver);
......
...@@ -26,6 +26,7 @@ ...@@ -26,6 +26,7 @@
#define MND_CONSUMER_VER_NUMBER 2 #define MND_CONSUMER_VER_NUMBER 2
#define MND_CONSUMER_RESERVE_SIZE 64 #define MND_CONSUMER_RESERVE_SIZE 64
#define MND_MAX_GROUP_PER_TOPIC 100
#define MND_CONSUMER_LOST_HB_CNT 6 #define MND_CONSUMER_LOST_HB_CNT 6
#define MND_CONSUMER_LOST_CLEAR_THRESHOLD 43200 #define MND_CONSUMER_LOST_CLEAR_THRESHOLD 43200
...@@ -63,7 +64,7 @@ int32_t mndInitConsumer(SMnode *pMnode) { ...@@ -63,7 +64,7 @@ int32_t mndInitConsumer(SMnode *pMnode) {
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_HB, mndProcessMqHbReq); mndSetMsgHandle(pMnode, TDMT_MND_TMQ_HB, mndProcessMqHbReq);
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_ASK_EP, mndProcessAskEpReq); mndSetMsgHandle(pMnode, TDMT_MND_TMQ_ASK_EP, mndProcessAskEpReq);
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_TIMER, mndProcessMqTimerMsg); mndSetMsgHandle(pMnode, TDMT_MND_TMQ_TIMER, mndProcessMqTimerMsg);
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_CONSUMER_LOST, mndProcessConsumerLostMsg); // mndSetMsgHandle(pMnode, TDMT_MND_TMQ_CONSUMER_LOST, mndProcessConsumerLostMsg);
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_CONSUMER_RECOVER, mndProcessConsumerRecoverMsg); mndSetMsgHandle(pMnode, TDMT_MND_TMQ_CONSUMER_RECOVER, mndProcessConsumerRecoverMsg);
mndSetMsgHandle(pMnode, TDMT_MND_TMQ_LOST_CONSUMER_CLEAR, mndProcessConsumerClearMsg); mndSetMsgHandle(pMnode, TDMT_MND_TMQ_LOST_CONSUMER_CLEAR, mndProcessConsumerClearMsg);
...@@ -75,6 +76,22 @@ int32_t mndInitConsumer(SMnode *pMnode) { ...@@ -75,6 +76,22 @@ int32_t mndInitConsumer(SMnode *pMnode) {
void mndCleanupConsumer(SMnode *pMnode) {} void mndCleanupConsumer(SMnode *pMnode) {}
void mndDropConsumerFromSdb(SMnode *pMnode, int64_t consumerId){
SMqConsumerClearMsg *pClearMsg = rpcMallocCont(sizeof(SMqConsumerClearMsg));
if (pClearMsg == NULL) {
mError("consumer:0x%"PRIx64" failed to clear consumer due to out of memory. alloc size:%d", consumerId, (int32_t)sizeof(SMqConsumerClearMsg));
return;
}
pClearMsg->consumerId = consumerId;
SRpcMsg rpcMsg = {
.msgType = TDMT_MND_TMQ_LOST_CONSUMER_CLEAR, .pCont = pClearMsg, .contLen = sizeof(SMqConsumerClearMsg)};
mInfo("consumer:0x%" PRIx64 " drop from sdb", consumerId);
tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
return;
}
bool mndRebTryStart() { bool mndRebTryStart() {
int32_t old = atomic_val_compare_exchange_32(&mqRebInExecCnt, 0, 1); int32_t old = atomic_val_compare_exchange_32(&mqRebInExecCnt, 0, 1);
mDebug("tq timer, rebalance counter old val:%d", old); mDebug("tq timer, rebalance counter old val:%d", old);
...@@ -105,50 +122,48 @@ void mndRebCntDec() { ...@@ -105,50 +122,48 @@ void mndRebCntDec() {
} }
} }
static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) { //static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) {
SMnode *pMnode = pMsg->info.node; // SMnode *pMnode = pMsg->info.node;
SMqConsumerLostMsg *pLostMsg = pMsg->pCont; // SMqConsumerLostMsg *pLostMsg = pMsg->pCont;
SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId); // SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId);
if (pConsumer == NULL) { // if (pConsumer == NULL) {
return 0; // return 0;
} // }
//
mInfo("process consumer lost msg, consumer:0x%" PRIx64 " status:%d(%s)", pLostMsg->consumerId, pConsumer->status, // mInfo("process consumer lost msg, consumer:0x%" PRIx64 " status:%d(%s)", pLostMsg->consumerId, pConsumer->status,
mndConsumerStatusName(pConsumer->status)); // mndConsumerStatusName(pConsumer->status));
//
if (pConsumer->status != MQ_CONSUMER_STATUS__READY) { // if (pConsumer->status != MQ_CONSUMER_STATUS_READY) {
mndReleaseConsumer(pMnode, pConsumer); // mndReleaseConsumer(pMnode, pConsumer);
return -1; // return -1;
} // }
//
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup); // SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup);
pConsumerNew->updateType = CONSUMER_UPDATE__LOST; // pConsumerNew->updateType = CONSUMER_UPDATE_TIMER_LOST;
//
mndReleaseConsumer(pMnode, pConsumer); // mndReleaseConsumer(pMnode, pConsumer);
//
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_NOTHING, pMsg, "lost-csm"); // STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_NOTHING, pMsg, "lost-csm");
if (pTrans == NULL) { // if (pTrans == NULL) {
goto FAIL; // goto FAIL;
} // }
//
if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) { // if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) {
goto FAIL; // goto FAIL;
} // }
//
if (mndTransPrepare(pMnode, pTrans) != 0) { // if (mndTransPrepare(pMnode, pTrans) != 0) {
goto FAIL; // goto FAIL;
} // }
//
tDeleteSMqConsumerObj(pConsumerNew); // tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew); // mndTransDrop(pTrans);
mndTransDrop(pTrans); // return 0;
return 0; //FAIL:
FAIL: // tDeleteSMqConsumerObj(pConsumerNew, true);
tDeleteSMqConsumerObj(pConsumerNew); // mndTransDrop(pTrans);
taosMemoryFree(pConsumerNew); // return -1;
mndTransDrop(pTrans); //}
return -1;
}
static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) { static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) {
SMnode *pMnode = pMsg->info.node; SMnode *pMnode = pMsg->info.node;
...@@ -162,14 +177,14 @@ static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) { ...@@ -162,14 +177,14 @@ static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) {
mInfo("receive consumer recover msg, consumer:0x%" PRIx64 " status:%d(%s)", pRecoverMsg->consumerId, mInfo("receive consumer recover msg, consumer:0x%" PRIx64 " status:%d(%s)", pRecoverMsg->consumerId,
pConsumer->status, mndConsumerStatusName(pConsumer->status)); pConsumer->status, mndConsumerStatusName(pConsumer->status));
if (pConsumer->status != MQ_CONSUMER_STATUS__LOST_REBD) { if (pConsumer->status != MQ_CONSUMER_STATUS_LOST) {
mndReleaseConsumer(pMnode, pConsumer); mndReleaseConsumer(pMnode, pConsumer);
terrno = TSDB_CODE_MND_CONSUMER_NOT_READY; terrno = TSDB_CODE_MND_CONSUMER_NOT_READY;
return -1; return -1;
} }
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup); SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup);
pConsumerNew->updateType = CONSUMER_UPDATE__RECOVER; pConsumerNew->updateType = CONSUMER_UPDATE_RECOVER;
mndReleaseConsumer(pMnode, pConsumer); mndReleaseConsumer(pMnode, pConsumer);
...@@ -181,13 +196,13 @@ static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) { ...@@ -181,13 +196,13 @@ static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) {
if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) goto FAIL; if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) goto FAIL;
if (mndTransPrepare(pMnode, pTrans) != 0) goto FAIL; if (mndTransPrepare(pMnode, pTrans) != 0) goto FAIL;
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return 0; return 0;
FAIL: FAIL:
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return -1; return -1;
} }
...@@ -206,13 +221,13 @@ static int32_t mndProcessConsumerClearMsg(SRpcMsg *pMsg) { ...@@ -206,13 +221,13 @@ static int32_t mndProcessConsumerClearMsg(SRpcMsg *pMsg) {
mInfo("consumer:0x%" PRIx64 " needs to be cleared, status %s", pClearMsg->consumerId, mInfo("consumer:0x%" PRIx64 " needs to be cleared, status %s", pClearMsg->consumerId,
mndConsumerStatusName(pConsumer->status)); mndConsumerStatusName(pConsumer->status));
if (pConsumer->status != MQ_CONSUMER_STATUS__LOST_REBD) { // if (pConsumer->status != MQ_CONSUMER_STATUS_LOST) {
mndReleaseConsumer(pMnode, pConsumer); // mndReleaseConsumer(pMnode, pConsumer);
return -1; // return -1;
} // }
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup); SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup);
pConsumerNew->updateType = CONSUMER_UPDATE__LOST; // pConsumerNew->updateType = CONSUMER_UPDATE_TIMER_LOST;
mndReleaseConsumer(pMnode, pConsumer); mndReleaseConsumer(pMnode, pConsumer);
...@@ -223,14 +238,14 @@ static int32_t mndProcessConsumerClearMsg(SRpcMsg *pMsg) { ...@@ -223,14 +238,14 @@ static int32_t mndProcessConsumerClearMsg(SRpcMsg *pMsg) {
if (mndSetConsumerDropLogs(pMnode, pTrans, pConsumerNew) != 0) goto FAIL; if (mndSetConsumerDropLogs(pMnode, pTrans, pConsumerNew) != 0) goto FAIL;
if (mndTransPrepare(pMnode, pTrans) != 0) goto FAIL; if (mndTransPrepare(pMnode, pTrans) != 0) goto FAIL;
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return 0; return 0;
FAIL: FAIL:
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return -1; return -1;
} }
...@@ -297,46 +312,14 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) { ...@@ -297,46 +312,14 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) {
int32_t hbStatus = atomic_add_fetch_32(&pConsumer->hbStatus, 1); int32_t hbStatus = atomic_add_fetch_32(&pConsumer->hbStatus, 1);
int32_t status = atomic_load_32(&pConsumer->status); int32_t status = atomic_load_32(&pConsumer->status);
mDebug("check for consumer:0x%" PRIx64 " status:%d(%s), sub-time:%" PRId64 ", uptime:%" PRId64 ", hbstatus:%d", mDebug("check for consumer:0x%" PRIx64 " status:%d(%s), sub-time:%" PRId64 ", createTime:%" PRId64 ", hbstatus:%d",
pConsumer->consumerId, status, mndConsumerStatusName(status), pConsumer->subscribeTime, pConsumer->upTime, pConsumer->consumerId, status, mndConsumerStatusName(status), pConsumer->subscribeTime, pConsumer->createTime,
hbStatus); hbStatus);
if (status == MQ_CONSUMER_STATUS__READY) { if (status == MQ_CONSUMER_STATUS_READY) {
if (hbStatus > MND_CONSUMER_LOST_HB_CNT) { if (taosArrayGetSize(pConsumer->assignedTopics) == 0) { // unsubscribe or close
SMqConsumerLostMsg *pLostMsg = rpcMallocCont(sizeof(SMqConsumerLostMsg)); mndDropConsumerFromSdb(pMnode, pConsumer->consumerId);
if (pLostMsg == NULL) { } else if (hbStatus > MND_CONSUMER_LOST_HB_CNT) {
mError("consumer:0x%"PRIx64" failed to transfer consumer status to lost due to out of memory. alloc size:%d",
pConsumer->consumerId, (int32_t)sizeof(SMqConsumerLostMsg));
continue;
}
pLostMsg->consumerId = pConsumer->consumerId;
SRpcMsg rpcMsg = {
.msgType = TDMT_MND_TMQ_CONSUMER_LOST, .pCont = pLostMsg, .contLen = sizeof(SMqConsumerLostMsg)};
mDebug("consumer:0x%"PRIx64" hb not received beyond threshold %d, set to lost", pConsumer->consumerId,
MND_CONSUMER_LOST_HB_CNT);
tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
}
} else if (status == MQ_CONSUMER_STATUS__LOST_REBD) {
// if the client is lost longer than one day, clear it. Otherwise, do nothing about the lost consumers.
if (hbStatus > MND_CONSUMER_LOST_CLEAR_THRESHOLD) {
SMqConsumerClearMsg *pClearMsg = rpcMallocCont(sizeof(SMqConsumerClearMsg));
if (pClearMsg == NULL) {
mError("consumer:0x%"PRIx64" failed to clear consumer due to out of memory. alloc size:%d",
pConsumer->consumerId, (int32_t)sizeof(SMqConsumerClearMsg));
continue;
}
pClearMsg->consumerId = pConsumer->consumerId;
SRpcMsg rpcMsg = {
.msgType = TDMT_MND_TMQ_LOST_CONSUMER_CLEAR, .pCont = pClearMsg, .contLen = sizeof(SMqConsumerClearMsg)};
mDebug("consumer:0x%" PRIx64 " lost beyond threshold %d, clear it", pConsumer->consumerId,
MND_CONSUMER_LOST_CLEAR_THRESHOLD);
tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
}
} else if (status == MQ_CONSUMER_STATUS__LOST) {
taosRLockLatch(&pConsumer->lock); taosRLockLatch(&pConsumer->lock);
int32_t topicNum = taosArrayGetSize(pConsumer->currentTopics); int32_t topicNum = taosArrayGetSize(pConsumer->currentTopics);
for (int32_t i = 0; i < topicNum; i++) { for (int32_t i = 0; i < topicNum; i++) {
...@@ -347,6 +330,11 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) { ...@@ -347,6 +330,11 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) {
taosArrayPush(pRebSub->removedConsumers, &pConsumer->consumerId); taosArrayPush(pRebSub->removedConsumers, &pConsumer->consumerId);
} }
taosRUnLockLatch(&pConsumer->lock); taosRUnLockLatch(&pConsumer->lock);
}
} else if (status == MQ_CONSUMER_STATUS_LOST) {
if (hbStatus > MND_CONSUMER_LOST_CLEAR_THRESHOLD) { // clear consumer if lost a day
mndDropConsumerFromSdb(pMnode, pConsumer->consumerId);
}
} else { // MQ_CONSUMER_STATUS_REBALANCE } else { // MQ_CONSUMER_STATUS_REBALANCE
taosRLockLatch(&pConsumer->lock); taosRLockLatch(&pConsumer->lock);
...@@ -413,7 +401,7 @@ static int32_t mndProcessMqHbReq(SRpcMsg *pMsg) { ...@@ -413,7 +401,7 @@ static int32_t mndProcessMqHbReq(SRpcMsg *pMsg) {
int32_t status = atomic_load_32(&pConsumer->status); int32_t status = atomic_load_32(&pConsumer->status);
if (status == MQ_CONSUMER_STATUS__LOST_REBD) { if (status == MQ_CONSUMER_STATUS_LOST) {
mInfo("try to recover consumer:0x%" PRIx64 "", consumerId); mInfo("try to recover consumer:0x%" PRIx64 "", consumerId);
SMqConsumerRecoverMsg *pRecoverMsg = rpcMallocCont(sizeof(SMqConsumerRecoverMsg)); SMqConsumerRecoverMsg *pRecoverMsg = rpcMallocCont(sizeof(SMqConsumerRecoverMsg));
...@@ -475,7 +463,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) { ...@@ -475,7 +463,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
mError("consumer:0x%" PRIx64 " group:%s not consistent with data in sdb, saved cgroup:%s", consumerId, req.cgroup, mError("consumer:0x%" PRIx64 " group:%s not consistent with data in sdb, saved cgroup:%s", consumerId, req.cgroup,
pConsumer->cgroup); pConsumer->cgroup);
terrno = TSDB_CODE_MND_CONSUMER_NOT_EXIST; terrno = TSDB_CODE_MND_CONSUMER_NOT_EXIST;
return -1; goto FAIL;
} }
atomic_store_32(&pConsumer->hbStatus, 0); atomic_store_32(&pConsumer->hbStatus, 0);
...@@ -483,7 +471,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) { ...@@ -483,7 +471,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
// 1. check consumer status // 1. check consumer status
int32_t status = atomic_load_32(&pConsumer->status); int32_t status = atomic_load_32(&pConsumer->status);
if (status == MQ_CONSUMER_STATUS__LOST_REBD) { if (status == MQ_CONSUMER_STATUS_LOST) {
mInfo("try to recover consumer:0x%" PRIx64, consumerId); mInfo("try to recover consumer:0x%" PRIx64, consumerId);
SMqConsumerRecoverMsg *pRecoverMsg = rpcMallocCont(sizeof(SMqConsumerRecoverMsg)); SMqConsumerRecoverMsg *pRecoverMsg = rpcMallocCont(sizeof(SMqConsumerRecoverMsg));
...@@ -497,10 +485,10 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) { ...@@ -497,10 +485,10 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &pRpcMsg); tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &pRpcMsg);
} }
if (status != MQ_CONSUMER_STATUS__READY) { if (status != MQ_CONSUMER_STATUS_READY) {
mInfo("consumer:0x%" PRIx64 " not ready, status: %s", consumerId, mndConsumerStatusName(status)); mInfo("consumer:0x%" PRIx64 " not ready, status: %s", consumerId, mndConsumerStatusName(status));
terrno = TSDB_CODE_MND_CONSUMER_NOT_READY; terrno = TSDB_CODE_MND_CONSUMER_NOT_READY;
return -1; goto FAIL;
} }
int32_t serverEpoch = atomic_load_32(&pConsumer->epoch); int32_t serverEpoch = atomic_load_32(&pConsumer->epoch);
...@@ -582,7 +570,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) { ...@@ -582,7 +570,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
void *buf = rpcMallocCont(tlen); void *buf = rpcMallocCont(tlen);
if (buf == NULL) { if (buf == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return -1; goto FAIL;
} }
SMqRspHead* pHead = buf; SMqRspHead* pHead = buf;
...@@ -669,6 +657,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) { ...@@ -669,6 +657,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
char *cgroup = subscribe.cgroup; char *cgroup = subscribe.cgroup;
SMqConsumerObj *pExistedConsumer = NULL; SMqConsumerObj *pExistedConsumer = NULL;
SMqConsumerObj *pConsumerNew = NULL; SMqConsumerObj *pConsumerNew = NULL;
STrans *pTrans = NULL;
int32_t code = -1; int32_t code = -1;
SArray *pTopicList = subscribe.topicNames; SArray *pTopicList = subscribe.topicNames;
...@@ -676,9 +665,23 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) { ...@@ -676,9 +665,23 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
taosArrayRemoveDuplicate(pTopicList, taosArrayCompareString, freeItem); taosArrayRemoveDuplicate(pTopicList, taosArrayCompareString, freeItem);
int32_t newTopicNum = taosArrayGetSize(pTopicList); int32_t newTopicNum = taosArrayGetSize(pTopicList);
for(int i = 0; i < newTopicNum; i++){
SMqSubscribeObj *pSub = mndAcquireSubscribe(pMnode, (const char*)cgroup, (const char*)taosArrayGetP(pTopicList, i));
if(pSub == NULL) continue;
taosRLockLatch(&pSub->lock);
if(taosHashGetSize(pSub->consumerHash) > MND_MAX_GROUP_PER_TOPIC){
terrno = TSDB_CODE_TMQ_GROUP_OUT_OF_RANGE;
code = terrno;
taosRUnLockLatch(&pSub->lock);
mndReleaseSubscribe(pMnode, pSub);
goto _over;
}
taosRUnLockLatch(&pSub->lock);
mndReleaseSubscribe(pMnode, pSub);
}
// check topic existence // check topic existence
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_NOTHING, pMsg, "subscribe"); pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_NOTHING, pMsg, "subscribe");
if (pTrans == NULL) { if (pTrans == NULL) {
goto _over; goto _over;
} }
...@@ -701,8 +704,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) { ...@@ -701,8 +704,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
pConsumerNew->autoCommitInterval = subscribe.autoCommitInterval; pConsumerNew->autoCommitInterval = subscribe.autoCommitInterval;
pConsumerNew->resetOffsetCfg = subscribe.resetOffsetCfg; pConsumerNew->resetOffsetCfg = subscribe.resetOffsetCfg;
// set the update type // pConsumerNew->updateType = CONSUMER_UPDATE_SUB_MODIFY; // use insert logic
pConsumerNew->updateType = CONSUMER_UPDATE__REBALANCE;
taosArrayDestroy(pConsumerNew->assignedTopics); taosArrayDestroy(pConsumerNew->assignedTopics);
pConsumerNew->assignedTopics = taosArrayDup(pTopicList, topicNameDup); pConsumerNew->assignedTopics = taosArrayDup(pTopicList, topicNameDup);
...@@ -721,7 +723,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) { ...@@ -721,7 +723,7 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
" cgroup:%s, current status:%d(%s), subscribe topic num: %d", " cgroup:%s, current status:%d(%s), subscribe topic num: %d",
consumerId, subscribe.cgroup, status, mndConsumerStatusName(status), newTopicNum); consumerId, subscribe.cgroup, status, mndConsumerStatusName(status), newTopicNum);
if (status != MQ_CONSUMER_STATUS__READY) { if (status != MQ_CONSUMER_STATUS_READY) {
terrno = TSDB_CODE_MND_CONSUMER_NOT_READY; terrno = TSDB_CODE_MND_CONSUMER_NOT_READY;
goto _over; goto _over;
} }
...@@ -732,11 +734,11 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) { ...@@ -732,11 +734,11 @@ int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
} }
// set the update type // set the update type
pConsumerNew->updateType = CONSUMER_UPDATE__REBALANCE; pConsumerNew->updateType = CONSUMER_UPDATE_SUB_MODIFY;
taosArrayDestroy(pConsumerNew->assignedTopics); taosArrayDestroy(pConsumerNew->assignedTopics);
pConsumerNew->assignedTopics = taosArrayDup(pTopicList, topicNameDup); pConsumerNew->assignedTopics = taosArrayDup(pTopicList, topicNameDup);
int32_t oldTopicNum = (pExistedConsumer->currentTopics) ? taosArrayGetSize(pExistedConsumer->currentTopics) : 0; int32_t oldTopicNum = taosArrayGetSize(pExistedConsumer->currentTopics);
int32_t i = 0, j = 0; int32_t i = 0, j = 0;
while (i < oldTopicNum || j < newTopicNum) { while (i < oldTopicNum || j < newTopicNum) {
...@@ -791,10 +793,7 @@ _over: ...@@ -791,10 +793,7 @@ _over:
mndReleaseConsumer(pMnode, pExistedConsumer); mndReleaseConsumer(pMnode, pExistedConsumer);
} }
if (pConsumerNew) { tDeleteSMqConsumerObj(pConsumerNew, true);
tDeleteSMqConsumerObj(pConsumerNew);
taosMemoryFree(pConsumerNew);
}
// TODO: replace with destroy subscribe msg // TODO: replace with destroy subscribe msg
taosArrayDestroyP(subscribe.topicNames, (FDelete)taosMemoryFree); taosArrayDestroyP(subscribe.topicNames, (FDelete)taosMemoryFree);
...@@ -894,17 +893,17 @@ CM_DECODE_OVER: ...@@ -894,17 +893,17 @@ CM_DECODE_OVER:
} }
static int32_t mndConsumerActionInsert(SSdb *pSdb, SMqConsumerObj *pConsumer) { static int32_t mndConsumerActionInsert(SSdb *pSdb, SMqConsumerObj *pConsumer) {
mDebug("consumer:0x%" PRIx64 " cgroup:%s status:%d(%s) epoch:%d load from sdb, perform insert action", mInfo("consumer:0x%" PRIx64 " sub insert, cgroup:%s status:%d(%s) epoch:%d",
pConsumer->consumerId, pConsumer->cgroup, pConsumer->status, mndConsumerStatusName(pConsumer->status), pConsumer->consumerId, pConsumer->cgroup, pConsumer->status, mndConsumerStatusName(pConsumer->status),
pConsumer->epoch); pConsumer->epoch);
pConsumer->subscribeTime = pConsumer->upTime; pConsumer->subscribeTime = taosGetTimestampMs();
return 0; return 0;
} }
static int32_t mndConsumerActionDelete(SSdb *pSdb, SMqConsumerObj *pConsumer) { static int32_t mndConsumerActionDelete(SSdb *pSdb, SMqConsumerObj *pConsumer) {
mDebug("consumer:0x%" PRIx64 " perform delete action, status:(%d)%s", pConsumer->consumerId, pConsumer->status, mInfo("consumer:0x%" PRIx64 " perform delete action, status:(%d)%s", pConsumer->consumerId, pConsumer->status,
mndConsumerStatusName(pConsumer->status)); mndConsumerStatusName(pConsumer->status));
tDeleteSMqConsumerObj(pConsumer); tDeleteSMqConsumerObj(pConsumer, false);
return 0; return 0;
} }
...@@ -913,10 +912,9 @@ static void updateConsumerStatus(SMqConsumerObj *pConsumer) { ...@@ -913,10 +912,9 @@ static void updateConsumerStatus(SMqConsumerObj *pConsumer) {
if (taosArrayGetSize(pConsumer->rebNewTopics) == 0 && taosArrayGetSize(pConsumer->rebRemovedTopics) == 0) { if (taosArrayGetSize(pConsumer->rebNewTopics) == 0 && taosArrayGetSize(pConsumer->rebRemovedTopics) == 0) {
if (status == MQ_CONSUMER_STATUS_REBALANCE) { if (status == MQ_CONSUMER_STATUS_REBALANCE) {
pConsumer->status = MQ_CONSUMER_STATUS__READY; pConsumer->status = MQ_CONSUMER_STATUS_READY;
} else if (status == MQ_CONSUMER_STATUS__LOST) { } else if (status == MQ_CONSUMER_STATUS_READY) {
ASSERT(taosArrayGetSize(pConsumer->currentTopics) == 0); pConsumer->status = MQ_CONSUMER_STATUS_LOST;
pConsumer->status = MQ_CONSUMER_STATUS__LOST_REBD;
} }
} }
} }
...@@ -930,7 +928,7 @@ static void removeFromNewTopicList(SMqConsumerObj *pConsumer, const char *pTopic ...@@ -930,7 +928,7 @@ static void removeFromNewTopicList(SMqConsumerObj *pConsumer, const char *pTopic
taosArrayRemove(pConsumer->rebNewTopics, i); taosArrayRemove(pConsumer->rebNewTopics, i);
taosMemoryFree(p); taosMemoryFree(p);
mDebug("consumer:0x%" PRIx64 " remove new topic:%s in the topic list, remain newTopics:%d", pConsumer->consumerId, mInfo("consumer:0x%" PRIx64 " remove new topic:%s in the topic list, remain newTopics:%d", pConsumer->consumerId,
pTopic, (int)taosArrayGetSize(pConsumer->rebNewTopics)); pTopic, (int)taosArrayGetSize(pConsumer->rebNewTopics));
break; break;
} }
...@@ -946,7 +944,7 @@ static void removeFromRemoveTopicList(SMqConsumerObj *pConsumer, const char *pTo ...@@ -946,7 +944,7 @@ static void removeFromRemoveTopicList(SMqConsumerObj *pConsumer, const char *pTo
taosArrayRemove(pConsumer->rebRemovedTopics, i); taosArrayRemove(pConsumer->rebRemovedTopics, i);
taosMemoryFree(p); taosMemoryFree(p);
mDebug("consumer:0x%" PRIx64 " remove topic:%s in the removed topic list, remain removedTopics:%d", mInfo("consumer:0x%" PRIx64 " remove topic:%s in the removed topic list, remain removedTopics:%d",
pConsumer->consumerId, pTopic, (int)taosArrayGetSize(pConsumer->rebRemovedTopics)); pConsumer->consumerId, pTopic, (int)taosArrayGetSize(pConsumer->rebRemovedTopics));
break; break;
} }
...@@ -961,7 +959,7 @@ static void removeFromCurrentTopicList(SMqConsumerObj *pConsumer, const char *pT ...@@ -961,7 +959,7 @@ static void removeFromCurrentTopicList(SMqConsumerObj *pConsumer, const char *pT
taosArrayRemove(pConsumer->currentTopics, i); taosArrayRemove(pConsumer->currentTopics, i);
taosMemoryFree(topic); taosMemoryFree(topic);
mDebug("consumer:0x%" PRIx64 " remove topic:%s in the current topic list, remain currentTopics:%d", mInfo("consumer:0x%" PRIx64 " remove topic:%s in the current topic list, remain currentTopics:%d",
pConsumer->consumerId, pTopic, (int)taosArrayGetSize(pConsumer->currentTopics)); pConsumer->consumerId, pTopic, (int)taosArrayGetSize(pConsumer->currentTopics));
break; break;
} }
...@@ -984,47 +982,46 @@ static bool existInCurrentTopicList(const SMqConsumerObj* pConsumer, const char* ...@@ -984,47 +982,46 @@ static bool existInCurrentTopicList(const SMqConsumerObj* pConsumer, const char*
} }
static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, SMqConsumerObj *pNewConsumer) { static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, SMqConsumerObj *pNewConsumer) {
mDebug("consumer:0x%" PRIx64 " perform update action, update type:%d, subscribe-time:%" PRId64 ", uptime:%" PRId64, mInfo("consumer:0x%" PRIx64 " perform update action, update type:%d, subscribe-time:%" PRId64 ", createTime:%" PRId64,
pOldConsumer->consumerId, pNewConsumer->updateType, pOldConsumer->subscribeTime, pOldConsumer->upTime); pOldConsumer->consumerId, pNewConsumer->updateType, pOldConsumer->subscribeTime, pOldConsumer->createTime);
taosWLockLatch(&pOldConsumer->lock); taosWLockLatch(&pOldConsumer->lock);
if (pNewConsumer->updateType == CONSUMER_UPDATE__REBALANCE) { if (pNewConsumer->updateType == CONSUMER_UPDATE_SUB_MODIFY) {
TSWAP(pOldConsumer->rebNewTopics, pNewConsumer->rebNewTopics); TSWAP(pOldConsumer->rebNewTopics, pNewConsumer->rebNewTopics);
TSWAP(pOldConsumer->rebRemovedTopics, pNewConsumer->rebRemovedTopics); TSWAP(pOldConsumer->rebRemovedTopics, pNewConsumer->rebRemovedTopics);
TSWAP(pOldConsumer->assignedTopics, pNewConsumer->assignedTopics); TSWAP(pOldConsumer->assignedTopics, pNewConsumer->assignedTopics);
pOldConsumer->subscribeTime = pNewConsumer->upTime; pOldConsumer->subscribeTime = taosGetTimestampMs();
pOldConsumer->status = MQ_CONSUMER_STATUS_REBALANCE; pOldConsumer->status = MQ_CONSUMER_STATUS_REBALANCE;
} else if (pNewConsumer->updateType == CONSUMER_UPDATE__LOST) { mInfo("consumer:0x%" PRIx64 " sub update, modify existed consumer",pOldConsumer->consumerId);
int32_t sz = taosArrayGetSize(pOldConsumer->currentTopics); // } else if (pNewConsumer->updateType == CONSUMER_UPDATE_TIMER_LOST) {
for (int32_t i = 0; i < sz; i++) { // int32_t sz = taosArrayGetSize(pOldConsumer->currentTopics);
char *topic = taosStrdup(taosArrayGetP(pOldConsumer->currentTopics, i)); // for (int32_t i = 0; i < sz; i++) {
taosArrayPush(pOldConsumer->rebRemovedTopics, &topic); // char *topic = taosStrdup(taosArrayGetP(pOldConsumer->currentTopics, i));
} // taosArrayPush(pOldConsumer->rebRemovedTopics, &topic);
// }
pOldConsumer->rebalanceTime = pNewConsumer->upTime; //
// int32_t prevStatus = pOldConsumer->status;
int32_t prevStatus = pOldConsumer->status; // pOldConsumer->status = MQ_CONSUMER_STATUS_LOST;
pOldConsumer->status = MQ_CONSUMER_STATUS__LOST; // mInfo("consumer:0x%" PRIx64 " timer update, timer lost. state %s -> %s, reb-time:%" PRId64 ", reb-removed-topics:%d",
mDebug("consumer:0x%" PRIx64 " state %s -> %s, reb-time:%" PRId64 ", reb-removed-topics:%d", // pOldConsumer->consumerId, mndConsumerStatusName(prevStatus), mndConsumerStatusName(pOldConsumer->status),
pOldConsumer->consumerId, mndConsumerStatusName(prevStatus), mndConsumerStatusName(pOldConsumer->status), // pOldConsumer->rebalanceTime, (int)taosArrayGetSize(pOldConsumer->rebRemovedTopics));
pOldConsumer->rebalanceTime, (int)taosArrayGetSize(pOldConsumer->rebRemovedTopics)); } else if (pNewConsumer->updateType == CONSUMER_UPDATE_RECOVER) {
} else if (pNewConsumer->updateType == CONSUMER_UPDATE__RECOVER) {
int32_t sz = taosArrayGetSize(pOldConsumer->assignedTopics); int32_t sz = taosArrayGetSize(pOldConsumer->assignedTopics);
for (int32_t i = 0; i < sz; i++) { for (int32_t i = 0; i < sz; i++) {
char *topic = taosStrdup(taosArrayGetP(pOldConsumer->assignedTopics, i)); char *topic = taosStrdup(taosArrayGetP(pOldConsumer->assignedTopics, i));
taosArrayPush(pOldConsumer->rebNewTopics, &topic); taosArrayPush(pOldConsumer->rebNewTopics, &topic);
} }
pOldConsumer->rebalanceTime = pNewConsumer->upTime;
pOldConsumer->status = MQ_CONSUMER_STATUS_REBALANCE; pOldConsumer->status = MQ_CONSUMER_STATUS_REBALANCE;
} else if (pNewConsumer->updateType == CONSUMER_UPDATE__TOUCH) { mInfo("consumer:0x%" PRIx64 " timer update, timer recover",pOldConsumer->consumerId);
} else if (pNewConsumer->updateType == CONSUMER_UPDATE_REB_MODIFY_NOTOPIC) {
atomic_add_fetch_32(&pOldConsumer->epoch, 1); atomic_add_fetch_32(&pOldConsumer->epoch, 1);
pOldConsumer->rebalanceTime = pNewConsumer->upTime; pOldConsumer->rebalanceTime = taosGetTimestampMs();
mInfo("consumer:0x%" PRIx64 " reb update, only rebalance time", pOldConsumer->consumerId);
} else if (pNewConsumer->updateType == CONSUMER_UPDATE__ADD) { } else if (pNewConsumer->updateType == CONSUMER_UPDATE_REB_MODIFY_TOPIC) {
char *pNewTopic = taosStrdup(taosArrayGetP(pNewConsumer->rebNewTopics, 0)); char *pNewTopic = taosStrdup(taosArrayGetP(pNewConsumer->rebNewTopics, 0));
// check if exist in current topic // check if exist in current topic
...@@ -1033,6 +1030,7 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, ...@@ -1033,6 +1030,7 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer,
// add to current topic // add to current topic
bool existing = existInCurrentTopicList(pOldConsumer, pNewTopic); bool existing = existInCurrentTopicList(pOldConsumer, pNewTopic);
if (existing) { if (existing) {
mError("consumer:0x%" PRIx64 "new topic:%s should not in currentTopics", pOldConsumer->consumerId, pNewTopic);
taosMemoryFree(pNewTopic); taosMemoryFree(pNewTopic);
} else { // added into current topic list } else { // added into current topic list
taosArrayPush(pOldConsumer->currentTopics, &pNewTopic); taosArrayPush(pOldConsumer->currentTopics, &pNewTopic);
...@@ -1044,17 +1042,17 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, ...@@ -1044,17 +1042,17 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer,
updateConsumerStatus(pOldConsumer); updateConsumerStatus(pOldConsumer);
// the re-balance is triggered when the new consumer is launched. // the re-balance is triggered when the new consumer is launched.
pOldConsumer->rebalanceTime = pNewConsumer->upTime; pOldConsumer->rebalanceTime = taosGetTimestampMs();
atomic_add_fetch_32(&pOldConsumer->epoch, 1); atomic_add_fetch_32(&pOldConsumer->epoch, 1);
mDebug("consumer:0x%" PRIx64 " state (%d)%s -> (%d)%s, new epoch:%d, reb-time:%" PRId64 mInfo("consumer:0x%" PRIx64 " reb update add, state (%d)%s -> (%d)%s, new epoch:%d, reb-time:%" PRId64
", current topics:%d, newTopics:%d, removeTopics:%d", ", current topics:%d, newTopics:%d, removeTopics:%d",
pOldConsumer->consumerId, status, mndConsumerStatusName(status), pOldConsumer->status, pOldConsumer->consumerId, status, mndConsumerStatusName(status), pOldConsumer->status,
mndConsumerStatusName(pOldConsumer->status), pOldConsumer->epoch, pOldConsumer->rebalanceTime, mndConsumerStatusName(pOldConsumer->status), pOldConsumer->epoch, pOldConsumer->rebalanceTime,
(int)taosArrayGetSize(pOldConsumer->currentTopics), (int)taosArrayGetSize(pOldConsumer->rebNewTopics), (int)taosArrayGetSize(pOldConsumer->currentTopics), (int)taosArrayGetSize(pOldConsumer->rebNewTopics),
(int)taosArrayGetSize(pOldConsumer->rebRemovedTopics)); (int)taosArrayGetSize(pOldConsumer->rebRemovedTopics));
} else if (pNewConsumer->updateType == CONSUMER_UPDATE__REMOVE) { } else if (pNewConsumer->updateType == CONSUMER_UPDATE_REB_MODIFY_REMOVE) {
char *removedTopic = taosArrayGetP(pNewConsumer->rebRemovedTopics, 0); char *removedTopic = taosArrayGetP(pNewConsumer->rebRemovedTopics, 0);
// remove from removed topic // remove from removed topic
...@@ -1067,10 +1065,10 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, ...@@ -1067,10 +1065,10 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer,
int32_t status = pOldConsumer->status; int32_t status = pOldConsumer->status;
updateConsumerStatus(pOldConsumer); updateConsumerStatus(pOldConsumer);
pOldConsumer->rebalanceTime = pNewConsumer->upTime; pOldConsumer->rebalanceTime = taosGetTimestampMs();
atomic_add_fetch_32(&pOldConsumer->epoch, 1); atomic_add_fetch_32(&pOldConsumer->epoch, 1);
mDebug("consumer:0x%" PRIx64 " state (%d)%s -> (%d)%s, new epoch:%d, reb-time:%" PRId64 mInfo("consumer:0x%" PRIx64 " reb update remove, state (%d)%s -> (%d)%s, new epoch:%d, reb-time:%" PRId64
", current topics:%d, newTopics:%d, removeTopics:%d", ", current topics:%d, newTopics:%d, removeTopics:%d",
pOldConsumer->consumerId, status, mndConsumerStatusName(status), pOldConsumer->status, pOldConsumer->consumerId, status, mndConsumerStatusName(status), pOldConsumer->status,
mndConsumerStatusName(pOldConsumer->status), pOldConsumer->epoch, pOldConsumer->rebalanceTime, mndConsumerStatusName(pOldConsumer->status), pOldConsumer->epoch, pOldConsumer->rebalanceTime,
...@@ -1133,8 +1131,12 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock * ...@@ -1133,8 +1131,12 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *
int32_t cols = 0; int32_t cols = 0;
// consumer id // consumer id
char consumerIdHex[32] = {0};
sprintf(varDataVal(consumerIdHex), "0x%"PRIx64, pConsumer->consumerId);
varDataSetLen(consumerIdHex, strlen(varDataVal(consumerIdHex)));
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataSetVal(pColInfo, numOfRows, (const char *)&pConsumer->consumerId, false); colDataSetVal(pColInfo, numOfRows, (const char *)consumerIdHex, false);
// consumer group // consumer group
char cgroup[TSDB_CGROUP_LEN + VARSTR_HEADER_SIZE] = {0}; char cgroup[TSDB_CGROUP_LEN + VARSTR_HEADER_SIZE] = {0};
...@@ -1175,7 +1177,7 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock * ...@@ -1175,7 +1177,7 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *
// up time // up time
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataSetVal(pColInfo, numOfRows, (const char *)&pConsumer->upTime, false); colDataSetVal(pColInfo, numOfRows, (const char *)&pConsumer->createTime, false);
// subscribe time // subscribe time
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
...@@ -1216,10 +1218,9 @@ static void mndCancelGetNextConsumer(SMnode *pMnode, void *pIter) { ...@@ -1216,10 +1218,9 @@ static void mndCancelGetNextConsumer(SMnode *pMnode, void *pIter) {
static const char *mndConsumerStatusName(int status) { static const char *mndConsumerStatusName(int status) {
switch (status) { switch (status) {
case MQ_CONSUMER_STATUS__READY: case MQ_CONSUMER_STATUS_READY:
return "ready"; return "ready";
case MQ_CONSUMER_STATUS__LOST: case MQ_CONSUMER_STATUS_LOST:
case MQ_CONSUMER_STATUS__LOST_REBD:
return "lost"; return "lost";
case MQ_CONSUMER_STATUS_REBALANCE: case MQ_CONSUMER_STATUS_REBALANCE:
return "rebalancing"; return "rebalancing";
......
...@@ -218,7 +218,7 @@ void *tDecodeSMqVgEp(const void *buf, SMqVgEp *pVgEp, int8_t sver) { ...@@ -218,7 +218,7 @@ void *tDecodeSMqVgEp(const void *buf, SMqVgEp *pVgEp, int8_t sver) {
return (void *)buf; return (void *)buf;
} }
SMqConsumerObj *tNewSMqConsumerObj(int64_t consumerId, char cgroup[TSDB_CGROUP_LEN]) { SMqConsumerObj *tNewSMqConsumerObj(int64_t consumerId, char* cgroup) {
SMqConsumerObj *pConsumer = taosMemoryCalloc(1, sizeof(SMqConsumerObj)); SMqConsumerObj *pConsumer = taosMemoryCalloc(1, sizeof(SMqConsumerObj));
if (pConsumer == NULL) { if (pConsumer == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
...@@ -249,16 +249,20 @@ SMqConsumerObj *tNewSMqConsumerObj(int64_t consumerId, char cgroup[TSDB_CGROUP_L ...@@ -249,16 +249,20 @@ SMqConsumerObj *tNewSMqConsumerObj(int64_t consumerId, char cgroup[TSDB_CGROUP_L
return NULL; return NULL;
} }
pConsumer->upTime = taosGetTimestampMs(); pConsumer->createTime = taosGetTimestampMs();
return pConsumer; return pConsumer;
} }
void tDeleteSMqConsumerObj(SMqConsumerObj *pConsumer) { void tDeleteSMqConsumerObj(SMqConsumerObj *pConsumer, bool delete) {
if(pConsumer == NULL) return;
taosArrayDestroyP(pConsumer->currentTopics, (FDelete)taosMemoryFree); taosArrayDestroyP(pConsumer->currentTopics, (FDelete)taosMemoryFree);
taosArrayDestroyP(pConsumer->rebNewTopics, (FDelete)taosMemoryFree); taosArrayDestroyP(pConsumer->rebNewTopics, (FDelete)taosMemoryFree);
taosArrayDestroyP(pConsumer->rebRemovedTopics, (FDelete)taosMemoryFree); taosArrayDestroyP(pConsumer->rebRemovedTopics, (FDelete)taosMemoryFree);
taosArrayDestroyP(pConsumer->assignedTopics, (FDelete)taosMemoryFree); taosArrayDestroyP(pConsumer->assignedTopics, (FDelete)taosMemoryFree);
if(delete){
taosMemoryFree(pConsumer);
}
} }
int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) { int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) {
...@@ -273,7 +277,7 @@ int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) { ...@@ -273,7 +277,7 @@ int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) {
tlen += taosEncodeFixedI32(buf, pConsumer->pid); tlen += taosEncodeFixedI32(buf, pConsumer->pid);
tlen += taosEncodeSEpSet(buf, &pConsumer->ep); tlen += taosEncodeSEpSet(buf, &pConsumer->ep);
tlen += taosEncodeFixedI64(buf, pConsumer->upTime); tlen += taosEncodeFixedI64(buf, pConsumer->createTime);
tlen += taosEncodeFixedI64(buf, pConsumer->subscribeTime); tlen += taosEncodeFixedI64(buf, pConsumer->subscribeTime);
tlen += taosEncodeFixedI64(buf, pConsumer->rebalanceTime); tlen += taosEncodeFixedI64(buf, pConsumer->rebalanceTime);
...@@ -343,7 +347,7 @@ void *tDecodeSMqConsumerObj(const void *buf, SMqConsumerObj *pConsumer, int8_t s ...@@ -343,7 +347,7 @@ void *tDecodeSMqConsumerObj(const void *buf, SMqConsumerObj *pConsumer, int8_t s
buf = taosDecodeFixedI32(buf, &pConsumer->pid); buf = taosDecodeFixedI32(buf, &pConsumer->pid);
buf = taosDecodeSEpSet(buf, &pConsumer->ep); buf = taosDecodeSEpSet(buf, &pConsumer->ep);
buf = taosDecodeFixedI64(buf, &pConsumer->upTime); buf = taosDecodeFixedI64(buf, &pConsumer->createTime);
buf = taosDecodeFixedI64(buf, &pConsumer->subscribeTime); buf = taosDecodeFixedI64(buf, &pConsumer->subscribeTime);
buf = taosDecodeFixedI64(buf, &pConsumer->rebalanceTime); buf = taosDecodeFixedI64(buf, &pConsumer->rebalanceTime);
......
...@@ -233,7 +233,6 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) { ...@@ -233,7 +233,6 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) {
} }
code = -1; code = -1;
taosIp2String(pReq->info.conn.clientIp, ip); taosIp2String(pReq->info.conn.clientIp, ip);
if (mndCheckOperPrivilege(pMnode, pReq->info.conn.user, MND_OPER_CONNECT) != 0) { if (mndCheckOperPrivilege(pMnode, pReq->info.conn.user, MND_OPER_CONNECT) != 0) {
mGError("user:%s, failed to login from %s since %s", pReq->info.conn.user, ip, terrstr()); mGError("user:%s, failed to login from %s since %s", pReq->info.conn.user, ip, terrstr());
...@@ -271,6 +270,7 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) { ...@@ -271,6 +270,7 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) {
} }
} }
_CONNECT:
pConn = mndCreateConn(pMnode, pReq->info.conn.user, connReq.connType, pReq->info.conn.clientIp, pConn = mndCreateConn(pMnode, pReq->info.conn.user, connReq.connType, pReq->info.conn.clientIp,
pReq->info.conn.clientPort, connReq.pid, connReq.app, connReq.startTime); pReq->info.conn.clientPort, connReq.pid, connReq.app, connReq.startTime);
if (pConn == NULL) { if (pConn == NULL) {
......
...@@ -160,10 +160,10 @@ static int32_t mndBuildSubChangeReq(void **pBuf, int32_t *pLen, SMqSubscribeObj ...@@ -160,10 +160,10 @@ static int32_t mndBuildSubChangeReq(void **pBuf, int32_t *pLen, SMqSubscribeObj
static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubscribeObj *pSub, static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubscribeObj *pSub,
const SMqRebOutputVg *pRebVg, SSubplan* pPlan) { const SMqRebOutputVg *pRebVg, SSubplan* pPlan) {
// if (pRebVg->oldConsumerId == pRebVg->newConsumerId) { if (pRebVg->oldConsumerId == pRebVg->newConsumerId) {
// terrno = TSDB_CODE_MND_INVALID_SUB_OPTION; terrno = TSDB_CODE_MND_INVALID_SUB_OPTION;
// return -1; return -1;
// } }
void *buf; void *buf;
int32_t tlen; int32_t tlen;
...@@ -175,7 +175,7 @@ static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubsc ...@@ -175,7 +175,7 @@ static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubsc
SVgObj *pVgObj = mndAcquireVgroup(pMnode, vgId); SVgObj *pVgObj = mndAcquireVgroup(pMnode, vgId);
if (pVgObj == NULL) { if (pVgObj == NULL) {
taosMemoryFree(buf); taosMemoryFree(buf);
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_MND_VGROUP_NOT_EXIST;
return -1; return -1;
} }
...@@ -296,17 +296,17 @@ static void addUnassignedVgroups(SMqRebOutputObj *pOutput, SHashObj *pHash) { ...@@ -296,17 +296,17 @@ static void addUnassignedVgroups(SMqRebOutputObj *pOutput, SHashObj *pHash) {
} }
} }
static void putNoTransferToOutput(SMqRebOutputObj *pOutput, SMqConsumerEp *pConsumerEp){ //static void putNoTransferToOutput(SMqRebOutputObj *pOutput, SMqConsumerEp *pConsumerEp){
for(int i = 0; i < taosArrayGetSize(pConsumerEp->vgs); i++){ // for(int i = 0; i < taosArrayGetSize(pConsumerEp->vgs); i++){
SMqVgEp *pVgEp = (SMqVgEp *)taosArrayGetP(pConsumerEp->vgs, i); // SMqVgEp *pVgEp = (SMqVgEp *)taosArrayGetP(pConsumerEp->vgs, i);
SMqRebOutputVg outputVg = { // SMqRebOutputVg outputVg = {
.oldConsumerId = pConsumerEp->consumerId, // .oldConsumerId = pConsumerEp->consumerId,
.newConsumerId = pConsumerEp->consumerId, // .newConsumerId = pConsumerEp->consumerId,
.pVgEp = pVgEp, // .pVgEp = pVgEp,
}; // };
taosArrayPush(pOutput->rebVgs, &outputVg); // taosArrayPush(pOutput->rebVgs, &outputVg);
} // }
} //}
static void transferVgroupsForConsumers(SMqRebOutputObj *pOutput, SHashObj *pHash, int32_t minVgCnt, static void transferVgroupsForConsumers(SMqRebOutputObj *pOutput, SHashObj *pHash, int32_t minVgCnt,
int32_t imbConsumerNum) { int32_t imbConsumerNum) {
...@@ -357,7 +357,7 @@ static void transferVgroupsForConsumers(SMqRebOutputObj *pOutput, SHashObj *pHas ...@@ -357,7 +357,7 @@ static void transferVgroupsForConsumers(SMqRebOutputObj *pOutput, SHashObj *pHas
} }
} }
} }
putNoTransferToOutput(pOutput, pConsumerEp); // putNoTransferToOutput(pOutput, pConsumerEp);
} }
} }
...@@ -576,50 +576,44 @@ static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOu ...@@ -576,50 +576,44 @@ static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOu
return -1; return -1;
} }
char topic[TSDB_TOPIC_FNAME_LEN] = {0};
char cgroup[TSDB_CGROUP_LEN] = {0};
mndSplitSubscribeKey(pOutput->pSub->key, topic, cgroup, true);
// 3. commit log: consumer to update status and epoch // 3. commit log: consumer to update status and epoch
// 3.1 set touched consumer // 3.1 set touched consumer
int32_t consumerNum = taosArrayGetSize(pOutput->modifyConsumers); int32_t consumerNum = taosArrayGetSize(pOutput->modifyConsumers);
for (int32_t i = 0; i < consumerNum; i++) { for (int32_t i = 0; i < consumerNum; i++) {
int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->modifyConsumers, i); int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->modifyConsumers, i);
SMqConsumerObj *pConsumerOld = mndAcquireConsumer(pMnode, consumerId); SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(consumerId, cgroup);
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumerOld->consumerId, pConsumerOld->cgroup); pConsumerNew->updateType = CONSUMER_UPDATE_REB_MODIFY_NOTOPIC;
pConsumerNew->updateType = CONSUMER_UPDATE__TOUCH;
mndReleaseConsumer(pMnode, pConsumerOld);
if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) { if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) {
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return -1; return -1;
} }
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
} }
// 3.2 set new consumer // 3.2 set new consumer
consumerNum = taosArrayGetSize(pOutput->newConsumers); consumerNum = taosArrayGetSize(pOutput->newConsumers);
for (int32_t i = 0; i < consumerNum; i++) { for (int32_t i = 0; i < consumerNum; i++) {
int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->newConsumers, i); int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->newConsumers, i);
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(consumerId, cgroup);
pConsumerNew->updateType = CONSUMER_UPDATE_REB_MODIFY_TOPIC;
SMqConsumerObj *pConsumerOld = mndAcquireConsumer(pMnode, consumerId); char* topicTmp = taosStrdup(topic);
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumerOld->consumerId, pConsumerOld->cgroup); taosArrayPush(pConsumerNew->rebNewTopics, &topicTmp);
pConsumerNew->updateType = CONSUMER_UPDATE__ADD;
char *topic = taosMemoryCalloc(1, TSDB_TOPIC_FNAME_LEN);
char cgroup[TSDB_CGROUP_LEN];
mndSplitSubscribeKey(pOutput->pSub->key, topic, cgroup, true);
taosArrayPush(pConsumerNew->rebNewTopics, &topic);
mndReleaseConsumer(pMnode, pConsumerOld);
if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) { if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) {
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return -1; return -1;
} }
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
} }
// 3.3 set removed consumer // 3.3 set removed consumer
...@@ -627,24 +621,19 @@ static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOu ...@@ -627,24 +621,19 @@ static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOu
for (int32_t i = 0; i < consumerNum; i++) { for (int32_t i = 0; i < consumerNum; i++) {
int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->removedConsumers, i); int64_t consumerId = *(int64_t *)taosArrayGet(pOutput->removedConsumers, i);
SMqConsumerObj *pConsumerOld = mndAcquireConsumer(pMnode, consumerId); SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(consumerId, cgroup);
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumerOld->consumerId, pConsumerOld->cgroup); pConsumerNew->updateType = CONSUMER_UPDATE_REB_MODIFY_REMOVE;
pConsumerNew->updateType = CONSUMER_UPDATE__REMOVE;
char *topic = taosMemoryCalloc(1, TSDB_TOPIC_FNAME_LEN); char* topicTmp = taosStrdup(topic);
char cgroup[TSDB_CGROUP_LEN]; taosArrayPush(pConsumerNew->rebRemovedTopics, &topicTmp);
mndSplitSubscribeKey(pOutput->pSub->key, topic, cgroup, true);
taosArrayPush(pConsumerNew->rebRemovedTopics, &topic);
mndReleaseConsumer(pMnode, pConsumerOld);
if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) { if (mndSetConsumerCommitLogs(pMnode, pTrans, pConsumerNew) != 0) {
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
mndTransDrop(pTrans); mndTransDrop(pTrans);
return -1; return -1;
} }
tDeleteSMqConsumerObj(pConsumerNew); tDeleteSMqConsumerObj(pConsumerNew, true);
taosMemoryFree(pConsumerNew);
} }
// 4. TODO commit log: modification log // 4. TODO commit log: modification log
...@@ -798,6 +787,20 @@ static int32_t mndProcessDropCgroupReq(SRpcMsg *pMsg) { ...@@ -798,6 +787,20 @@ static int32_t mndProcessDropCgroupReq(SRpcMsg *pMsg) {
return -1; return -1;
} }
void *pIter = NULL;
SMqConsumerObj *pConsumer;
while (1) {
pIter = sdbFetch(pMnode->pSdb, SDB_CONSUMER, pIter, (void **)&pConsumer);
if (pIter == NULL) {
break;
}
if (strcmp(dropReq.cgroup, pConsumer->cgroup) == 0) {
mndDropConsumerFromSdb(pMnode, pConsumer->consumerId);
}
sdbRelease(pMnode->pSdb, pConsumer);
}
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_NOTHING, pMsg, "drop-cgroup"); STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_NOTHING, pMsg, "drop-cgroup");
if (pTrans == NULL) { if (pTrans == NULL) {
mError("cgroup: %s on topic:%s, failed to drop since %s", dropReq.cgroup, dropReq.topic, terrstr()); mError("cgroup: %s on topic:%s, failed to drop since %s", dropReq.cgroup, dropReq.topic, terrstr());
......
...@@ -569,6 +569,11 @@ static int32_t mndProcessCreateTopicReq(SRpcMsg *pReq) { ...@@ -569,6 +569,11 @@ static int32_t mndProcessCreateTopicReq(SRpcMsg *pReq) {
SMqTopicObj *pTopic = NULL; SMqTopicObj *pTopic = NULL;
SDbObj *pDb = NULL; SDbObj *pDb = NULL;
SCMCreateTopicReq createTopicReq = {0}; SCMCreateTopicReq createTopicReq = {0};
if (sdbGetSize(pMnode->pSdb, SDB_TOPIC) >= tmqMaxTopicNum){
terrno = TSDB_CODE_TMQ_TOPIC_OUT_OF_RANGE;
mError("topic num out of range");
return code;
}
if (tDeserializeSCMCreateTopicReq(pReq->pCont, pReq->contLen, &createTopicReq) != 0) { if (tDeserializeSCMCreateTopicReq(pReq->pCont, pReq->contLen, &createTopicReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG; terrno = TSDB_CODE_INVALID_MSG;
...@@ -681,7 +686,11 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) { ...@@ -681,7 +686,11 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) {
break; break;
} }
if (pConsumer->status == MQ_CONSUMER_STATUS__LOST_REBD) continue; if (pConsumer->status == MQ_CONSUMER_STATUS_LOST){
mndDropConsumerFromSdb(pMnode, pConsumer->consumerId);
mndReleaseConsumer(pMnode, pConsumer);
continue;
}
int32_t sz = taosArrayGetSize(pConsumer->assignedTopics); int32_t sz = taosArrayGetSize(pConsumer->assignedTopics);
for (int32_t i = 0; i < sz; i++) { for (int32_t i = 0; i < sz; i++) {
......
...@@ -335,6 +335,7 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d ...@@ -335,6 +335,7 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
tagArray = taosArrayInit(1, sizeof(STagVal)); tagArray = taosArrayInit(1, sizeof(STagVal));
if (!tagArray) { if (!tagArray) {
tdDestroySVCreateTbReq(pCreateTbReq); tdDestroySVCreateTbReq(pCreateTbReq);
taosMemoryFreeClear(pCreateTbReq);
goto _end; goto _end;
} }
STagVal tagVal = { STagVal tagVal = {
...@@ -350,6 +351,7 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d ...@@ -350,6 +351,7 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
tagArray = taosArrayDestroy(tagArray); tagArray = taosArrayDestroy(tagArray);
if (pTag == NULL) { if (pTag == NULL) {
tdDestroySVCreateTbReq(pCreateTbReq); tdDestroySVCreateTbReq(pCreateTbReq);
taosMemoryFreeClear(pCreateTbReq);
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _end; goto _end;
} }
......
...@@ -2415,6 +2415,10 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { ...@@ -2415,6 +2415,10 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) {
} }
static int32_t firstLastTransferInfoImpl(SFirstLastRes* pInput, SFirstLastRes* pOutput, bool isFirst) { static int32_t firstLastTransferInfoImpl(SFirstLastRes* pInput, SFirstLastRes* pOutput, bool isFirst) {
if (!pInput->hasResult) {
return TSDB_CODE_FAILED;
}
if (pOutput->hasResult) { if (pOutput->hasResult) {
if (isFirst) { if (isFirst) {
if (pInput->ts > pOutput->ts) { if (pInput->ts > pOutput->ts) {
......
...@@ -474,8 +474,8 @@ double* tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num) { ...@@ -474,8 +474,8 @@ double* tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num) {
} }
ASSERTS(total <= numOfElem && total + pHisto->elems[j + 1].num > numOfElem, ASSERTS(total <= numOfElem && total + pHisto->elems[j + 1].num > numOfElem,
"tHistogramUniform Error, total:%d, numOfElem:%d, elems[%d].num:%d", "tHistogramUniform Error, total:%ld, numOfElem:%ld, elems[%d].num:%ld",
total, numOfElem, j + 1, pHisto->elems[j + 1].num); total, (int64_t)numOfElem, j + 1, pHisto->elems[j + 1].num);
double delta = numOfElem - total; double delta = numOfElem - total;
if (fabs(delta) < FLT_EPSILON) { if (fabs(delta) < FLT_EPSILON) {
......
...@@ -39,6 +39,7 @@ static SFilePage *loadDataFromFilePage(tMemBucket *pMemBucket, int32_t slotIdx) ...@@ -39,6 +39,7 @@ static SFilePage *loadDataFromFilePage(tMemBucket *pMemBucket, int32_t slotIdx)
if (p != NULL) { if (p != NULL) {
pIdList = *(SArray **)p; pIdList = *(SArray **)p;
} else { } else {
taosMemoryFree(buffer);
return NULL; return NULL;
} }
...@@ -48,6 +49,7 @@ static SFilePage *loadDataFromFilePage(tMemBucket *pMemBucket, int32_t slotIdx) ...@@ -48,6 +49,7 @@ static SFilePage *loadDataFromFilePage(tMemBucket *pMemBucket, int32_t slotIdx)
SFilePage *pg = getBufPage(pMemBucket->pBuffer, *pageId); SFilePage *pg = getBufPage(pMemBucket->pBuffer, *pageId);
if (pg == NULL) { if (pg == NULL) {
taosMemoryFree(buffer);
return NULL; return NULL;
} }
......
...@@ -255,6 +255,18 @@ int32_t udfStopUdfd() { ...@@ -255,6 +255,18 @@ int32_t udfStopUdfd() {
return 0; return 0;
} }
int32_t udfGetUdfdPid(int32_t* pUdfdPid) {
SUdfdData *pData = &udfdGlobal;
if (pData->spawnErr) {
return pData->spawnErr;
}
uv_pid_t pid = uv_process_get_pid(&pData->process);
if (pUdfdPid) {
*pUdfdPid = (int32_t)pid;
}
return TSDB_CODE_SUCCESS;
}
//============================================================================================== //==============================================================================================
/* Copyright (c) 2013, Ben Noordhuis <info@bnoordhuis.nl> /* Copyright (c) 2013, Ben Noordhuis <info@bnoordhuis.nl>
* The QUEUE is copied from queue.h under libuv * The QUEUE is copied from queue.h under libuv
......
...@@ -965,40 +965,6 @@ int32_t udfdFillUdfInfoFromMNode(void *clientRpc, char *udfName, SUdf *udf) { ...@@ -965,40 +965,6 @@ int32_t udfdFillUdfInfoFromMNode(void *clientRpc, char *udfName, SUdf *udf) {
return code; return code;
} }
int32_t udfdConnectToMnode() {
SConnectReq connReq = {0};
connReq.connType = CONN_TYPE__UDFD;
tstrncpy(connReq.app, "udfd", sizeof(connReq.app));
tstrncpy(connReq.user, TSDB_DEFAULT_USER, sizeof(connReq.user));
char pass[TSDB_PASSWORD_LEN + 1] = {0};
taosEncryptPass_c((uint8_t *)(TSDB_DEFAULT_PASS), strlen(TSDB_DEFAULT_PASS), pass);
tstrncpy(connReq.passwd, pass, sizeof(connReq.passwd));
connReq.pid = taosGetPId();
connReq.startTime = taosGetTimestampMs();
strcpy(connReq.sVer, version);
int32_t contLen = tSerializeSConnectReq(NULL, 0, &connReq);
void *pReq = rpcMallocCont(contLen);
tSerializeSConnectReq(pReq, contLen, &connReq);
SUdfdRpcSendRecvInfo *msgInfo = taosMemoryCalloc(1, sizeof(SUdfdRpcSendRecvInfo));
msgInfo->rpcType = UDFD_RPC_MNODE_CONNECT;
uv_sem_init(&msgInfo->resultSem, 0);
SRpcMsg rpcMsg = {0};
rpcMsg.msgType = TDMT_MND_CONNECT;
rpcMsg.pCont = pReq;
rpcMsg.contLen = contLen;
rpcMsg.info.ahandle = msgInfo;
rpcSendRequest(global.clientRpc, &global.mgmtEp.epSet, &rpcMsg, NULL);
uv_sem_wait(&msgInfo->resultSem);
int32_t code = msgInfo->code;
uv_sem_destroy(&msgInfo->resultSem);
taosMemoryFree(msgInfo);
return code;
}
static bool udfdRpcRfp(int32_t code, tmsg_t msgType) { static bool udfdRpcRfp(int32_t code, tmsg_t msgType) {
if (code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_RPC_BROKEN_LINK || code == TSDB_CODE_SYN_NOT_LEADER || if (code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_RPC_BROKEN_LINK || code == TSDB_CODE_SYN_NOT_LEADER ||
code == TSDB_CODE_RPC_SOMENODE_NOT_CONNECTED || code == TSDB_CODE_SYN_RESTORING || code == TSDB_CODE_RPC_SOMENODE_NOT_CONNECTED || code == TSDB_CODE_SYN_RESTORING ||
...@@ -1378,23 +1344,6 @@ static int32_t udfdRun() { ...@@ -1378,23 +1344,6 @@ static int32_t udfdRun() {
return 0; return 0;
} }
void udfdConnectMnodeThreadFunc(void *args) {
int32_t retryMnodeTimes = 0;
int32_t code = 0;
while (retryMnodeTimes++ <= TSDB_MAX_REPLICA) {
uv_sleep(100 * (1 << retryMnodeTimes));
code = udfdConnectToMnode();
if (code == 0) {
break;
}
fnError("udfd can not connect to mnode, code: %s. retry", tstrerror(code));
}
if (code != 0) {
fnError("udfd can not connect to mnode");
}
}
int32_t udfdInitResidentFuncs() { int32_t udfdInitResidentFuncs() {
if (strlen(tsUdfdResFuncs) == 0) { if (strlen(tsUdfdResFuncs) == 0) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
...@@ -1497,9 +1446,6 @@ int main(int argc, char *argv[]) { ...@@ -1497,9 +1446,6 @@ int main(int argc, char *argv[]) {
udfdInitResidentFuncs(); udfdInitResidentFuncs();
uv_thread_t mnodeConnectThread;
uv_thread_create(&mnodeConnectThread, udfdConnectMnodeThreadFunc, NULL);
udfdRun(); udfdRun();
removeListeningPipe(); removeListeningPipe();
......
...@@ -3746,10 +3746,10 @@ int32_t fltSclBuildRangeFromBlockSma(SFltSclColumnRange *colRange, SColumnDataAg ...@@ -3746,10 +3746,10 @@ int32_t fltSclBuildRangeFromBlockSma(SFltSclColumnRange *colRange, SColumnDataAg
taosArrayPush(points, &startPt); taosArrayPush(points, &startPt);
taosArrayPush(points, &endPt); taosArrayPush(points, &endPt);
} }
SFltSclDatum min; SFltSclDatum min = {0};
fltSclBuildDatumFromBlockSmaValue(&min, colRange->colNode->node.resType.type, pAgg->min); fltSclBuildDatumFromBlockSmaValue(&min, colRange->colNode->node.resType.type, pAgg->min);
SFltSclPoint minPt = {.excl = false, .start = true, .val = min}; SFltSclPoint minPt = {.excl = false, .start = true, .val = min};
SFltSclDatum max; SFltSclDatum max = {0};
fltSclBuildDatumFromBlockSmaValue(&max, colRange->colNode->node.resType.type, pAgg->max); fltSclBuildDatumFromBlockSmaValue(&max, colRange->colNode->node.resType.type, pAgg->max);
SFltSclPoint maxPt = {.excl = false, .start = false, .val = max}; SFltSclPoint maxPt = {.excl = false, .start = false, .val = max};
taosArrayPush(points, &minPt); taosArrayPush(points, &minPt);
......
...@@ -74,7 +74,6 @@ void streamSchedByTimer(void* param, void* tmrId) { ...@@ -74,7 +74,6 @@ void streamSchedByTimer(void* param, void* tmrId) {
atomic_store_8(&pTask->triggerStatus, TASK_TRIGGER_STATUS__INACTIVE); atomic_store_8(&pTask->triggerStatus, TASK_TRIGGER_STATUS__INACTIVE);
if (tAppendDataToInputQueue(pTask, (SStreamQueueItem*)trigger) < 0) { if (tAppendDataToInputQueue(pTask, (SStreamQueueItem*)trigger) < 0) {
taosFreeQitem(trigger);
taosTmrReset(streamSchedByTimer, (int32_t)pTask->triggerParam, pTask, streamEnv.timer, &pTask->timer); taosTmrReset(streamSchedByTimer, (int32_t)pTask->triggerParam, pTask, streamEnv.timer, &pTask->timer);
return; return;
} }
......
...@@ -360,7 +360,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot, ...@@ -360,7 +360,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot,
SRowBuffPos* pPos = *(SRowBuffPos**)pNode->data; SRowBuffPos* pPos = *(SRowBuffPos**)pNode->data;
ASSERT(pPos->pRowBuff && pFileState->rowSize > 0); ASSERT(pPos->pRowBuff && pFileState->rowSize > 0);
if (streamStateGetBatchSize(batch) >= BATCH_LIMIT) { if (streamStateGetBatchSize(batch) >= BATCH_LIMIT) {
code = streamStatePutBatch_rocksdb(pFileState->pFileStore, batch); streamStatePutBatch_rocksdb(pFileState->pFileStore, batch);
streamStateClearBatch(batch); streamStateClearBatch(batch);
} }
...@@ -373,7 +373,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot, ...@@ -373,7 +373,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot,
taosMemoryFree(buf); taosMemoryFree(buf);
if (streamStateGetBatchSize(batch) > 0) { if (streamStateGetBatchSize(batch) > 0) {
code = streamStatePutBatch_rocksdb(pFileState->pFileStore, batch); streamStatePutBatch_rocksdb(pFileState->pFileStore, batch);
} }
streamStateClearBatch(batch); streamStateClearBatch(batch);
...@@ -385,7 +385,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot, ...@@ -385,7 +385,7 @@ int32_t flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot,
int32_t len = 0; int32_t len = 0;
sprintf(keyBuf, "%s:%" PRId64 "", taskKey, ((SStreamState*)pFileState->pFileStore)->checkPointId); sprintf(keyBuf, "%s:%" PRId64 "", taskKey, ((SStreamState*)pFileState->pFileStore)->checkPointId);
streamFileStateEncode(&pFileState->flushMark, &valBuf, &len); streamFileStateEncode(&pFileState->flushMark, &valBuf, &len);
code = streamStatePutBatch(pFileState->pFileStore, "default", batch, keyBuf, valBuf, len, 0); streamStatePutBatch(pFileState->pFileStore, "default", batch, keyBuf, valBuf, len, 0);
taosMemoryFree(valBuf); taosMemoryFree(valBuf);
} }
{ {
...@@ -489,7 +489,7 @@ int32_t recoverSnapshot(SStreamFileState* pFileState) { ...@@ -489,7 +489,7 @@ int32_t recoverSnapshot(SStreamFileState* pFileState) {
break; break;
} }
memcpy(pNewPos->pRowBuff, pVal, pVLen); memcpy(pNewPos->pRowBuff, pVal, pVLen);
code = tSimpleHashPut(pFileState->rowBuffMap, pNewPos->pKey, pFileState->rowSize, &pNewPos, POINTER_BYTES); code = tSimpleHashPut(pFileState->rowBuffMap, pNewPos->pKey, pFileState->keyLen, &pNewPos, POINTER_BYTES);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
destroyRowBuffPos(pNewPos); destroyRowBuffPos(pNewPos);
break; break;
......
...@@ -631,6 +631,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_INVALID_MSG, "Invalid message") ...@@ -631,6 +631,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_INVALID_MSG, "Invalid message")
TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_MISMATCH, "Consumer mismatch") TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_MISMATCH, "Consumer mismatch")
TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_CLOSED, "Consumer closed") TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_CLOSED, "Consumer closed")
TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_ERROR, "Consumer error, to see log") TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_ERROR, "Consumer error, to see log")
TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_TOPIC_OUT_OF_RANGE, "Topic num out of range")
TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_GROUP_OUT_OF_RANGE, "Group num out of range 100")
// stream // stream
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_NOT_EXIST, "Stream task not exist") TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_NOT_EXIST, "Stream task not exist")
......
...@@ -129,6 +129,7 @@ ...@@ -129,6 +129,7 @@
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-19201.py ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-19201.py
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-21561.py ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-21561.py
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TS-3404.py ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TS-3404.py
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TS-3581.py
,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/balance_vgroups_r1.py -N 6 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/balance_vgroups_r1.py -N 6
,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosShell.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosShell.py
......
...@@ -8,6 +8,9 @@ system sh/deploy.sh -n dnode1 -i 1 ...@@ -8,6 +8,9 @@ system sh/deploy.sh -n dnode1 -i 1
system sh/cfg.sh -n dnode1 -c udf -v 1 system sh/cfg.sh -n dnode1 -c udf -v 1
system sh/exec.sh -n dnode1 -s start system sh/exec.sh -n dnode1 -s start
sql connect sql connect
sql alter user root pass 'taosdata2'
system sh/exec.sh -n dnode1 -s stop
system sh/exec.sh -n dnode1 -s start
print ======== step1 udf print ======== step1 udf
system sh/compile_udf.sh system sh/compile_udf.sh
......
import taos
import sys
import time
import socket
import os
import threading
from util.log import *
from util.sql import *
from util.cases import *
from util.dnodes import *
class TDTestCase:
hostname = socket.gethostname()
def init(self, conn, logSql, replicaVar=1):
self.replicaVar = int(replicaVar)
tdLog.debug(f"start to excute {__file__}")
#tdSql.init(conn.cursor())
tdSql.init(conn.cursor(), logSql) # output sql.txt file
def getBuildPath(self):
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
projPath = selfPath[:selfPath.find("community")]
else:
projPath = selfPath[:selfPath.find("tests")]
for root, dirs, files in os.walk(projPath):
if ("taosd" in files or "taosd.exe" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
if ("packaging" not in rootRealPath):
buildPath = root[:len(root) - len("/build/bin")]
break
return buildPath
def create_tables(self):
tdSql.execute(f'''CREATE STABLE `dwd_log_master` (`ts` TIMESTAMP, `dim_ip` NCHAR(64)) TAGS (`group_id` BIGINT, `st_hour` NCHAR(2), `org_id` NCHAR(32),
`dev_manufacturer_name` NCHAR(64), `dev_manufacturer_id` INT, `dev_category_name` NCHAR(64), `dev_category_id` INT, `dev_feature_name` NCHAR(64),
`dev_feature_id` INT, `dev_ip` NCHAR(64), `black_list` TINYINT, `white_list` TINYINT)''')
tdSql.execute(f'''CREATE TABLE `dwd_log_master_475021043` USING `dwd_log_master` (`group_id`, `st_hour`, `org_id`, `dev_manufacturer_name`, `dev_manufacturer_id`,
`dev_category_name`, `dev_category_id`, `dev_feature_name`, `dev_feature_id`, `dev_ip`, `black_list`, `white_list`) TAGS
(475021043, "14", NULL, NULL, NULL, NULL, NULL, NULL, NULL, "172.18.22.230", NULL, NULL)''')
def insert_data(self):
tdLog.debug("start to insert data ............")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:30.000','192.168.192.102')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:31.000','172.18.23.249')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:32.000','192.168.200.231')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:33.000','172.18.22.231')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:34.000','192.168.210.231')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:35.000','192.168.192.100')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:36.000','192.168.192.231')")
tdSql.execute(f"INSERT INTO `dwd_log_master_475021043` VALUES ('2023-06-26 14:38:37.000','172.18.23.231')")
tdLog.debug("insert data ............ [OK]")
def run(self):
tdSql.prepare()
self.create_tables()
self.insert_data()
tdLog.printNoPrefix("======== test TS-3581")
for i in range(100):
tdSql.query(f"select first(ts), last(ts), count(*) from dwd_log_master;")
tdSql.checkRows(1)
print(tdSql.queryResult)
tdSql.checkData(0, 0, '2023-06-26 14:38:30.000')
return
def stop(self):
tdSql.close()
tdLog.success(f"{__file__} successfully executed")
tdCases.addLinux(__file__, TDTestCase())
tdCases.addWindows(__file__, TDTestCase())
...@@ -17,6 +17,9 @@ ...@@ -17,6 +17,9 @@
#include <taosws.h> #include <taosws.h>
#include <shellInt.h> #include <shellInt.h>
// save current database name
char curDBName[128] = ""; // TDB_MAX_DBNAME_LEN is 24, put large
int shell_conn_ws_server(bool first) { int shell_conn_ws_server(bool first) {
char cuttedDsn[SHELL_WS_DSN_BUFF] = {0}; char cuttedDsn[SHELL_WS_DSN_BUFF] = {0};
int dsnLen = strlen(shell.args.dsn); int dsnLen = strlen(shell.args.dsn);
...@@ -59,6 +62,14 @@ int shell_conn_ws_server(bool first) { ...@@ -59,6 +62,14 @@ int shell_conn_ws_server(bool first) {
fprintf(stdout, "successfully connected to cloud service\n"); fprintf(stdout, "successfully connected to cloud service\n");
} }
fflush(stdout); fflush(stdout);
// switch to current database if have
if(curDBName[0] !=0) {
char command[256];
sprintf(command, "use %s;", curDBName);
shellRunSingleCommandWebsocketImp(command);
}
return 0; return 0;
} }
...@@ -290,7 +301,46 @@ void shellRunSingleCommandWebsocketImp(char *command) { ...@@ -290,7 +301,46 @@ void shellRunSingleCommandWebsocketImp(char *command) {
if (shellRegexMatch(command, "^\\s*use\\s+[a-zA-Z0-9_]+\\s*;\\s*$", if (shellRegexMatch(command, "^\\s*use\\s+[a-zA-Z0-9_]+\\s*;\\s*$",
REG_EXTENDED | REG_ICASE)) { REG_EXTENDED | REG_ICASE)) {
fprintf(stdout, "Database changed.\r\n\r\n");
// copy dbname to curDBName
char *p = command;
bool firstStart = false;
bool firstEnd = false;
int i = 0;
while (*p != 0) {
if (*p != ' ') {
// not blank
if (!firstStart) {
firstStart = true;
} else if (firstEnd) {
if(*p == ';' && *p != '\\') {
break;
}
// database name
curDBName[i++] = *p;
if(i + 4 > sizeof(curDBName)) {
// DBName is too long, reset zero and break
i = 0;
break;
}
}
} else {
// blank
if(firstStart == true && firstEnd == false){
firstEnd = true;
}
if(firstStart && firstEnd && i > 0){
// blank after database name
break;
}
}
// move next
p++;
}
// append end
curDBName[i] = 0;
fprintf(stdout, "Database changed to %s.\r\n\r\n", curDBName);
fflush(stdout); fflush(stdout);
ws_free_result(res); ws_free_result(res);
return; return;
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册