Merge branch '3.0' into enh/taos.ref

docs/en/14-reference/05-taosbenchmark.md

@@ -7,7 +7,7 @@ description: "taosBenchmark (once called taosdemo ) is a tool for testing the pe
 
 ## Introduction
 
-taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can flexibly control the number and type of databases, supertables, tag columns, number and type of data columns, and sub-tables, and types of databases, super tables, the number and types of data columns, the number of sub-tables, the amount of data per sub-table, the time interval for inserting data, the number of working threads, whether and how to insert disordered data, and so on. The installer provides taosdemo as a soft link to taosBenchmark for compatibility and for the convenience of past users.
+taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can be configured to generate user defined databases, supertables, subtables, and the time series data to populate these for performance benchmarking. taosBenchmark is highly configurable and some of the configurations include the time interval for inserting data, the number of working threads and the capability to insert disordered data. The installer provides taosdemo as a soft link to taosBenchmark for compatibility with past users.
 
 ## Installation
 
@@ -21,9 +21,13 @@ There are two ways to install taosBenchmark:
 
 ### Configuration and running methods
 
-TaosBenchmark needs to be executed on the terminal of the operating system, it supports two configuration methods: [Command-line arguments](#Command-line arguments in detailed) and [JSON configuration file](#Configuration file arguments in detailed). These two methods are mutually exclusive. Users can use `-f <json file>` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
+TaosBenchmark needs to be executed on the terminal of the operating system, it supports two configuration methods: [Command-line arguments](#command-line-arguments-in-detail) and [JSON configuration file](#configuration-file-parameters-in-detail). These two methods are mutually exclusive. Users can use `-f <json file>` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
 
+<<<<<<< HEAD
 taosBenchmark supports complete performance testing of TDengine. taosBenchmark supports the TDengine functions in three categories: write, query, and subscribe. These three functions are mutually exclusive, and users can select only one of them each time taosBenchmark runs. It is important to note that the type of functionality to be tested is not configurable when using the command-line configuration method, which can only test writing performance. To test the query and subscription performance of the TDengine, you must use the configuration file method and specify the function type to test via the parameter `filetype` in the configuration file.
+=======
+taosBenchmark supports the complete performance testing of TDengine by providing functionally to write, query, and subscribe. These three functions are mutually exclusive, users can only select one of them each time taosBenchmark runs. The query and subscribe functionalities are only configurable using a json configuration file by specifying the parameter `filetype`, while write can be performed through both the command-line and a configuration file.
+>>>>>>> 108548b4d6 (docs: typo)
 
 **Make sure that the TDengine cluster is running correctly before running taosBenchmark. **
 
@@ -57,9 +61,8 @@ Use the following command-line to run taosBenchmark and control its behavior via
 taosBenchmark -f <json file>
 ```
 
-**Here are a few examples of configuration files:**
-
-#### Example of inserting a scenario JSON configuration file
+#### Configuration file examples
+##### Example of inserting a scenario JSON configuration file
 
 <details>
 <summary>insert.json</summary>
@@ -70,7 +73,7 @@ taosBenchmark -f <json file>
 
 </details>
 
-#### Query Scenario JSON Profile Example
+##### Query Scenario JSON Profile Example
 
 <details>
 <summary>query.json</summary>
@@ -81,7 +84,7 @@ taosBenchmark -f <json file>
 
 </details>
 
-#### Subscription JSON configuration example
+##### Subscription JSON configuration example
 
 <details>
 <summary>subscribe.json</summary>
@@ -172,7 +175,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
   Switch parameter specifying whether to use escape characters in the super table and sub-table names. By default is not used.
 
 - **-C/--chinese** :
-  Switch specifying whether to use Unicode Chinese characters in nchar and binary. By default is not used.
+  specify whether to use Unicode Chinese characters in nchar and binary, the default is no.
 
 - **-N/--normal-table** :
   This parameter indicates that taosBenchmark will create only normal tables instead of super tables. The default value is false. It can be used if the insert mode is taosc, stmt, and rest.
@@ -373,7 +376,7 @@ The configuration parameters for querying the sub-tables or the normal tables ar
 
 - **sqls**.
   - **sql**: the SQL command to be executed.
-  - **result**: the file to save the query result. If it is unspecified, taosBenchark will not save the result.
+  - **result**: the file to save the query result. If it is unspecified, taosBenchmark will not save the result.
 
 #### Configuration parameters of query super table
 

docs/en/20-third-party/01-grafana.mdx

@@ -31,38 +31,41 @@ TDengine currently supports Grafana versions 7.5 and above. Users can go to the
 ### Install Grafana Plugin and Configure Data Source
 
 <Tabs defaultValue="script">
-<TabItem value="script" label="Using Script">
+<TabItem value="gui" label="With GUI">
 
-Set the url and authorization environment variables by `export` or a [`.env`(dotenv) file](https://hexdocs.pm/dotenvy/dotenv-file-format.html):
+Under Grafana 8, plugin catalog allows you to [browse and manage plugins within Grafana](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog) (but for Grafana 7.x, use **With Script** or **Install & Configure Manually**). Find the page at **Configurations > Plugins**, search **TDengine** and click it to install.
 
-```sh
-export TDENGINE_API=http://tdengine.local:6041
-# user + password
-export TDENGINE_USER=user
-export TDENGINE_PASSWORD=password
-
-# Other useful variables
-# - If to install TDengine data source, default is true
-export TDENGINE_DS_ENABLED=false
-# - Data source name to be created, default is TDengine
-export TDENGINE_DS_NAME=TDengine
-# - Data source organization id, default is 1
-export GF_ORG_ID=1
-# - Data source is editable in admin ui or not, default is 0 (false)
-export TDENGINE_EDITABLE=1
-```
+![Search tdengine in grafana plugins](./grafana/grafana-plugin-search-tdengine.png)
+
+Installation may cost some minutes, then you can **Create a TDengine data source**:
+
+![Install and configure Grafana data source](./grafana/grafana-install-and-config.png)
+
+Then you can add a TDengine data source by filling up the configuration options.
+
+![TDengine Database Grafana plugin add data source](./grafana/grafana-data-source.png)
+
+You can create dashboards with TDengine now.
+
+</TabItem>
+<TabItem value="script" label="With Script">
 
-Run `install.sh`:
+On a server with Grafana installed, run `install.sh` with TDengine url and username/passwords will install TDengine data source plugin and add a data source named TDengine. This is the recommended way for Grafana 7.x or [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) users.
 
 ```sh
-bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
+bash -c "$(curl -fsSL \
+  https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" -- \
+  -a http://localhost:6041 \
+  -u root \
+  -p taosdata
 ```
 
-With this script, TDengine data source plugin and the Grafana data source will be installed and created automatically with Grafana provisioning configurations. Save the script and type `./install.sh --help` for the full usage of the script.
+Restart Grafana service and open Grafana in web-browser, usually <http://localhost:3000>.
 
-And then, restart Grafana service and open Grafana in web-browser, usually <http://localhost:3000>.
+Save the script and type `./install.sh --help` for the full usage of the script.
 
 </TabItem>
+
 <TabItem value="manual" label="Install & Configure Manually">
 
 Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation.
@@ -115,6 +118,73 @@ Click `Save & Test` to test. You should see a success message if the test worked
 
 ![TDengine Database TDinsight plugin add database 4](./grafana/add_datasource4.webp)
 
+</TabItem>
+<TabItem value="container" label="Container">
+
+Please refer to [Install plugins in the Docker container](https://grafana.com/docs/grafana/next/setup-grafana/installation/docker/#install-plugins-in-the-docker-container). This will install `tdengine-datasource` plugin when Grafana container starts:
+
+```bash
+docker run -d \
+  -p 3000:3000 \
+  --name=grafana \
+  -e "GF_INSTALL_PLUGINS=tdengine-datasource" \
+  grafana/grafana
+```
+
+You can setup a zero-configuration stack for TDengine + Grafana by [docker-compose](https://docs.docker.com/compose/) and [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) file：
+
+1. Save the provisioning configuration file to `tdengine.yml`.
+
+    ```yml
+    apiVersion: 1
+    datasources:
+    - name: TDengine
+      type: tdengine-datasource
+      orgId: 1
+      url: "$TDENGINE_API"
+      isDefault: true
+      secureJsonData:
+        url: "$TDENGINE_URL"
+        basicAuth: "$TDENGINE_BASIC_AUTH"
+        token: "$TDENGINE_CLOUD_TOKEN"
+      version: 1
+      editable: true
+    ```
+
+2. Write `docker-compose.yml` with [TDengine](https://hub.docker.com/r/tdengine/tdengine) and [Grafana](https://hub.docker.com/r/grafana/grafana) image.
+
+    ```yml
+    version: "3.7"
+
+    services:
+      tdengine:
+        image: tdengine/tdengine:2.6.0.2
+        environment:
+          TAOS_FQDN: tdengine
+        volumes:
+          - tdengine-data:/var/lib/taos/
+      grafana:
+        image: grafana/grafana:8.5.6
+        volumes:
+          - ./tdengine.yml/:/etc/grafana/provisioning/tdengine.yml
+          - grafana-data:/var/lib/grafana
+        environment:
+          # install tdengine plugin at start
+          GF_INSTALL_PLUGINS: "tdengine-datasource"
+          TDENGINE_URL: "http://tdengine:6041"
+          #printf "$TDENGINE_USER:$TDENGINE_PASSWORD" | base64
+          TDENGINE_BASIC_AUTH: "cm9vdDp0YmFzZTEyNQ=="
+        ports:
+          - 3000:3000
+    volumes:
+      grafana-data:
+      tdengine-data:
+    ```
+
+3. Start TDengine and Grafana services: `docker-compose up -d`.
+
+Open Grafana <http://localhost:3000>, and you can add dashboard with TDengine now.
+
 </TabItem>
 </Tabs>
 

docs/zh/20-third-party/01-grafana.mdx

@@ -29,39 +29,41 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/
 ### 安装 Grafana Plugin 并配置数据源
 
 <Tabs defaultValue="script">
-<TabItem value="script" label="使用安装脚本">
+<TabItem value="gui" label="图形化界面安装">
 
-将集群信息设置为环境变量；也可以使用 `.env` 文件，请参考 [dotenv](https://hexdocs.pm/dotenvy/dotenv-file-format.html)：
+使用 Grafana 最新版本（8.5+），您可以在 Grafana 中[浏览和管理插件](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog)（对于 7.x 版本，请使用 **安装脚本** 或 **手动安装并配置** 方式）。在 Grafana 管理界面中的 **Configurations > Plugins** 页面直接搜索并按照提示安装 TDengine。
 
-```sh
-export TDENGINE_API=http://tdengine.local:6041
-# user + password
-export TDENGINE_USER=user
-export TDENGINE_PASSWORD=password
-
-# 其他环境变量：
-# - 是否安装数据源，默认为 true，表示安装
-export TDENGINE_DS_ENABLED=false
-# - 数据源名称，默认为 TDengine
-export TDENGINE_DS_NAME=TDengine
-# - 数据源所属组织 ID，默认为 1
-export GF_ORG_ID=1
-# - 数据源是否可通过管理面板编辑，默认为 0，表示不可编辑
-export TDENGINE_EDITABLE=1
-```
+![Search tdengine in grafana plugins](grafana-plugin-search-tdengine.png)
+
+如图示即安装完毕，按照指示 **Create a TDengine data source** 添加数据源。
+
+![Install and configure Grafana data source](grafana-install-and-config.png)
+
+输入 TDengine 相关配置，完成数据源配置。
+
+![TDengine Database Grafana plugin add data source](./grafana-data-source.png)
+
+配置完毕，现在可以使用 TDengine 创建 Dashboard 了。
+
+</TabItem>
+<TabItem value="script" label="使用安装脚本">
 
-运行安装脚本：
+对于使用 Grafana 7.x 版本或使用 [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) 配置的用户，可以在 Grafana 服务器上使用安装脚本自动安装插件即添加数据源 Provisioning 配置文件。
 
 ```sh
-bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
+bash -c "$(curl -fsSL \
+  https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" -- \
+  -a http://localhost:6041 \
+  -u root \
+  -p taosdata
 ```
 
-该脚本将自动安装 Grafana 插件并配置数据源。安装完毕后，需要重启 Grafana 服务后生效。
+安装完毕后，需要重启 Grafana 服务后方可生效。
 
 保存该脚本并执行 `./install.sh --help` 可查看详细帮助文档。
 
 </TabItem>
-<TabItem value="manual" label="手动安装并配置">
+<TabItem value="manual" label="手动安装">
 
 使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。
 
@@ -113,6 +115,73 @@ GF_INSTALL_PLUGINS=tdengine-datasource
 
 ![TDengine Database Grafana plugin add data source](./add_datasource4.webp)
 
+</TabItem>
+<TabItem value="container" label="K8s/Docker 容器">
+
+参考 [Grafana 容器化安装说明](https://grafana.com/docs/grafana/next/setup-grafana/installation/docker/#install-plugins-in-the-docker-container)。使用如下命令启动一个容器，并自动安装 TDengine 插件：
+
+```bash
+docker run -d \
+  -p 3000:3000 \
+  --name=grafana \
+  -e "GF_INSTALL_PLUGINS=tdengine-datasource" \
+  grafana/grafana
+```
+
+使用 docker-compose，配置 Grafana Provisioning 自动化配置，体验 TDengine + Grafana 组合的零配置启动：
+
+1. 保存该文件为 `tdengine.yml`。
+
+    ```yml
+    apiVersion: 1
+    datasources:
+    - name: TDengine
+      type: tdengine-datasource
+      orgId: 1
+      url: "$TDENGINE_API"
+      isDefault: true
+      secureJsonData:
+        url: "$TDENGINE_URL"
+        basicAuth: "$TDENGINE_BASIC_AUTH"
+        token: "$TDENGINE_CLOUD_TOKEN"
+      version: 1
+      editable: true
+    ```
+
+2. 保存该文件为 `docker-compose.yml`。
+
+    ```yml
+    version: "3.7"
+
+    services:
+      tdengine:
+        image: tdengine/tdengine:2.6.0.2
+        environment:
+          TAOS_FQDN: tdengine
+        volumes:
+          - tdengine-data:/var/lib/taos/
+      grafana:
+        image: grafana/grafana:8.5.6
+        volumes:
+          - ./tdengine.yml/:/etc/grafana/provisioning/tdengine.yml
+          - grafana-data:/var/lib/grafana
+        environment:
+          # install tdengine plugin at start
+          GF_INSTALL_PLUGINS: "tdengine-datasource"
+          TDENGINE_URL: "http://tdengine:6041"
+          #printf "$TDENGINE_USER:$TDENGINE_PASSWORD" | base64
+          TDENGINE_BASIC_AUTH: "cm9vdDp0YmFzZTEyNQ=="
+        ports:
+          - 3000:3000
+    volumes:
+      grafana-data:
+      tdengine-data:
+    ```
+
+3. 使用 docker-compose 命令启动 TDengine + Grafana ：`docker-compose up -d`。
+
+打开 Grafana <http://localhost:3000>，现在可以添加 Dashboard 了。
+
 </TabItem>
 </Tabs>
 

examples/c/stream_demo.c

@@ -32,6 +32,13 @@ int32_t init_env() {
   }
   taos_free_result(pRes);
 
+  pRes = taos_query(pConn, "create database if not exists abc2 vgroups 20");
+  if (taos_errno(pRes) != 0) {
+    printf("error in create db, reason:%s\n", taos_errstr(pRes));
+    return -1;
+  }
+  taos_free_result(pRes);
+
   pRes = taos_query(pConn, "use abc1");
   if (taos_errno(pRes) != 0) {
     printf("error in use db, reason:%s\n", taos_errstr(pRes));
@@ -81,9 +88,9 @@ int32_t create_stream() {
   /*const char* sql = "select min(k), max(k), sum(k) as sum_of_k from st1";*/
   /*const char* sql = "select sum(k) from tu1 interval(10m)";*/
   /*pRes = tmq_create_stream(pConn, "stream1", "out1", sql);*/
-  pRes = taos_query(
-      pConn,
-      "create stream stream1 trigger max_delay 10s into outstb as select _wstartts, sum(k) from st1 interval(10m)");
+  pRes = taos_query(pConn,
+                    "create stream stream1 trigger at_once into abc2.outstb as select _wstartts, sum(k) from st1  "
+                    "partition by tbname interval(10m) ");
   if (taos_errno(pRes) != 0) {
     printf("failed to create stream stream1, reason:%s\n", taos_errstr(pRes));
     return -1;

examples/c/tmq.c

@@ -47,7 +47,7 @@ int32_t init_env() {
     return -1;
   }
 
-  TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 2");
+  TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 1");
   if (taos_errno(pRes) != 0) {
     printf("error in create db, reason:%s\n", taos_errstr(pRes));
     return -1;
@@ -146,8 +146,8 @@ int32_t create_topic() {
   return 0;
 }
 
-void tmq_commit_cb_print(tmq_t* tmq, tmq_resp_err_t resp, tmq_topic_vgroup_list_t* offsets, void* param) {
-  printf("commit %d tmq %p offsets %p param %p\n", resp, tmq, offsets, param);
+void tmq_commit_cb_print(tmq_t* tmq, int32_t code, void* param) {
+  printf("commit %d tmq %p param %p\n", code, tmq, param);
 }
 
 tmq_t* build_consumer() {
@@ -167,7 +167,7 @@ tmq_t* build_consumer() {
   tmq_conf_set(conf, "td.connect.user", "root");
   tmq_conf_set(conf, "td.connect.pass", "taosdata");
   tmq_conf_set(conf, "msg.with.table.name", "true");
-  tmq_conf_set(conf, "enable.auto.commit", "false");
+  tmq_conf_set(conf, "enable.auto.commit", "true");
   tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL);
   tmq_t* tmq = tmq_consumer_new(conf, NULL, 0);
   assert(tmq);
@@ -183,10 +183,10 @@ tmq_list_t* build_topic_list() {
 }
 
 void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) {
-  tmq_resp_err_t err;
+  int32_t code;
 
-  if ((err = tmq_subscribe(tmq, topics))) {
-    fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err));
+  if ((code = tmq_subscribe(tmq, topics))) {
+    fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(code));
     printf("subscribe err\n");
     return;
   }
@@ -201,12 +201,13 @@ void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) {
       taos_free_result(tmqmessage);
       /*} else {*/
       /*break;*/
+      /*tmq_commit_sync(tmq, NULL);*/
     }
   }
 
-  err = tmq_consumer_close(tmq);
-  if (err)
-    fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(err));
+  code = tmq_consumer_close(tmq);
+  if (code)
+    fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(code));
   else
     fprintf(stderr, "%% Consumer closed\n");
 }
@@ -214,11 +215,11 @@ void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) {
 void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) {
   static const int MIN_COMMIT_COUNT = 1;
 
-  int            msg_count = 0;
-  tmq_resp_err_t err;
+  int     msg_count = 0;
+  int32_t code;
 
-  if ((err = tmq_subscribe(tmq, topics))) {
-    fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err));
+  if ((code = tmq_subscribe(tmq, topics))) {
+    fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(code));
     return;
   }
 
@@ -239,14 +240,14 @@ void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) {
       msg_process(tmqmessage);
       taos_free_result(tmqmessage);
 
-      /*tmq_commit_async(tmq, NULL, tmq_commit_cb_print, NULL);*/
+      /*tmq_commit_sync(tmq, NULL);*/
       /*if ((++msg_count % MIN_COMMIT_COUNT) == 0) tmq_commit(tmq, NULL, 0);*/
     }
   }
 
-  err = tmq_consumer_close(tmq);
-  if (err)
-    fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(err));
+  code = tmq_consumer_close(tmq);
+  if (code)
+    fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(code));
   else
     fprintf(stderr, "%% Consumer closed\n");
 }

include/client/taos.h

@@ -126,47 +126,47 @@ typedef struct setConfRet {
   char              retMsg[RET_MSG_LENGTH];
 } setConfRet;
 
-DLL_EXPORT void        taos_cleanup(void);
-DLL_EXPORT int         taos_options(TSDB_OPTION option, const void *arg, ...);
-DLL_EXPORT setConfRet  taos_set_config(const char *config);
-DLL_EXPORT int         taos_init(void);
-DLL_EXPORT TAOS       *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port);
-DLL_EXPORT TAOS       *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port);
-DLL_EXPORT void        taos_close(TAOS *taos);
-
-const char            *taos_data_type(int type);
-
-DLL_EXPORT TAOS_STMT  *taos_stmt_init(TAOS *taos);
-DLL_EXPORT int         taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length);
-DLL_EXPORT int         taos_stmt_set_tbname_tags(TAOS_STMT *stmt, const char *name, TAOS_MULTI_BIND *tags);
-DLL_EXPORT int         taos_stmt_set_tbname(TAOS_STMT *stmt, const char *name);
-DLL_EXPORT int         taos_stmt_set_tags(TAOS_STMT *stmt, TAOS_MULTI_BIND *tags);
-DLL_EXPORT int         taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name);
-DLL_EXPORT int         taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
-DLL_EXPORT int         taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
-
-DLL_EXPORT int         taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
-DLL_EXPORT int         taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
-DLL_EXPORT int         taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes);
-DLL_EXPORT int         taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
-DLL_EXPORT int         taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
-DLL_EXPORT int         taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx);
-DLL_EXPORT int         taos_stmt_add_batch(TAOS_STMT *stmt);
-DLL_EXPORT int         taos_stmt_execute(TAOS_STMT *stmt);
-DLL_EXPORT TAOS_RES   *taos_stmt_use_result(TAOS_STMT *stmt);
-DLL_EXPORT int         taos_stmt_close(TAOS_STMT *stmt);
-DLL_EXPORT char       *taos_stmt_errstr(TAOS_STMT *stmt);
-DLL_EXPORT int         taos_stmt_affected_rows(TAOS_STMT *stmt);
-DLL_EXPORT int         taos_stmt_affected_rows_once(TAOS_STMT *stmt);
-
-DLL_EXPORT TAOS_RES   *taos_query(TAOS *taos, const char *sql);
-
-DLL_EXPORT TAOS_ROW    taos_fetch_row(TAOS_RES *res);
-DLL_EXPORT int         taos_result_precision(TAOS_RES *res);  // get the time precision of result
-DLL_EXPORT void        taos_free_result(TAOS_RES *res);
-DLL_EXPORT int         taos_field_count(TAOS_RES *res);
-DLL_EXPORT int         taos_num_fields(TAOS_RES *res);
-DLL_EXPORT int         taos_affected_rows(TAOS_RES *res);
+DLL_EXPORT void       taos_cleanup(void);
+DLL_EXPORT int        taos_options(TSDB_OPTION option, const void *arg, ...);
+DLL_EXPORT setConfRet taos_set_config(const char *config);
+DLL_EXPORT int        taos_init(void);
+DLL_EXPORT TAOS      *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port);
+DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port);
+DLL_EXPORT void  taos_close(TAOS *taos);
+
+const char *taos_data_type(int type);
+
+DLL_EXPORT TAOS_STMT *taos_stmt_init(TAOS *taos);
+DLL_EXPORT int        taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length);
+DLL_EXPORT int        taos_stmt_set_tbname_tags(TAOS_STMT *stmt, const char *name, TAOS_MULTI_BIND *tags);
+DLL_EXPORT int        taos_stmt_set_tbname(TAOS_STMT *stmt, const char *name);
+DLL_EXPORT int        taos_stmt_set_tags(TAOS_STMT *stmt, TAOS_MULTI_BIND *tags);
+DLL_EXPORT int        taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name);
+DLL_EXPORT int        taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
+DLL_EXPORT int        taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
+
+DLL_EXPORT int       taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
+DLL_EXPORT int       taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
+DLL_EXPORT int       taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes);
+DLL_EXPORT int       taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
+DLL_EXPORT int       taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
+DLL_EXPORT int       taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx);
+DLL_EXPORT int       taos_stmt_add_batch(TAOS_STMT *stmt);
+DLL_EXPORT int       taos_stmt_execute(TAOS_STMT *stmt);
+DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt);
+DLL_EXPORT int       taos_stmt_close(TAOS_STMT *stmt);
+DLL_EXPORT char     *taos_stmt_errstr(TAOS_STMT *stmt);
+DLL_EXPORT int       taos_stmt_affected_rows(TAOS_STMT *stmt);
+DLL_EXPORT int       taos_stmt_affected_rows_once(TAOS_STMT *stmt);
+
+DLL_EXPORT TAOS_RES *taos_query(TAOS *taos, const char *sql);
+
+DLL_EXPORT TAOS_ROW taos_fetch_row(TAOS_RES *res);
+DLL_EXPORT int      taos_result_precision(TAOS_RES *res);  // get the time precision of result
+DLL_EXPORT void     taos_free_result(TAOS_RES *res);
+DLL_EXPORT int      taos_field_count(TAOS_RES *res);
+DLL_EXPORT int      taos_num_fields(TAOS_RES *res);
+DLL_EXPORT int      taos_affected_rows(TAOS_RES *res);
 
 DLL_EXPORT TAOS_FIELD *taos_fetch_fields(TAOS_RES *res);
 DLL_EXPORT int         taos_select_db(TAOS *taos, const char *db);
@@ -181,8 +181,8 @@ DLL_EXPORT int        *taos_get_column_data_offset(TAOS_RES *res, int columnInde
 DLL_EXPORT int         taos_validate_sql(TAOS *taos, const char *sql);
 DLL_EXPORT void        taos_reset_current_db(TAOS *taos);
 
-DLL_EXPORT int        *taos_fetch_lengths(TAOS_RES *res);
-DLL_EXPORT TAOS_ROW   *taos_result_block(TAOS_RES *res);
+DLL_EXPORT int      *taos_fetch_lengths(TAOS_RES *res);
+DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res);
 
 DLL_EXPORT const char *taos_get_server_info(TAOS *taos);
 DLL_EXPORT const char *taos_get_client_info();
@@ -192,8 +192,8 @@ DLL_EXPORT int         taos_errno(TAOS_RES *tres);
 
 DLL_EXPORT void        taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param);
 DLL_EXPORT void        taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param);
-DLL_EXPORT void        taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param);
-DLL_EXPORT const void *taos_get_raw_block(TAOS_RES* res);
+DLL_EXPORT void        taos_fetch_raw_block_a(TAOS_RES *res, __taos_async_fn_t fp, void *param);
+DLL_EXPORT const void *taos_get_raw_block(TAOS_RES *res);
 
 // Shuduo: temporary enable for app build
 #if 1
@@ -209,21 +209,11 @@ DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLi
 
 /* --------------------------TMQ INTERFACE------------------------------- */
 
-enum {
-  TMQ_RESP_ERR__FAIL = -1,
-  TMQ_RESP_ERR__SUCCESS = 0,
-};
-
-typedef int32_t tmq_resp_err_t;
-
-typedef struct tmq_t                   tmq_t;
-typedef struct tmq_topic_vgroup_t      tmq_topic_vgroup_t;
-typedef struct tmq_topic_vgroup_list_t tmq_topic_vgroup_list_t;
-
+typedef struct tmq_t      tmq_t;
 typedef struct tmq_conf_t tmq_conf_t;
 typedef struct tmq_list_t tmq_list_t;
 
-typedef void(tmq_commit_cb(tmq_t *, tmq_resp_err_t, tmq_topic_vgroup_list_t *, void *param));
+typedef void(tmq_commit_cb(tmq_t *, int32_t code, void *param));
 
 DLL_EXPORT tmq_list_t *tmq_list_new();
 DLL_EXPORT int32_t     tmq_list_append(tmq_list_t *, const char *);
@@ -233,25 +223,17 @@ DLL_EXPORT char      **tmq_list_to_c_array(const tmq_list_t *);
 
 DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen);
 
-DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t);
+DLL_EXPORT const char *tmq_err2str(int32_t code);
 
 /* ------------------------TMQ CONSUMER INTERFACE------------------------ */
 
-DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list);
-DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq);
-DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics);
-
-// timeout: -1 means infinitely waiting
+DLL_EXPORT int32_t   tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list);
+DLL_EXPORT int32_t   tmq_unsubscribe(tmq_t *tmq);
+DLL_EXPORT int32_t   tmq_subscription(tmq_t *tmq, tmq_list_t **topics);
 DLL_EXPORT TAOS_RES *tmq_consumer_poll(tmq_t *tmq, int64_t timeout);
-
-DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq);
-DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets);
-DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param);
-
-#if 0
-DLL_EXPORT tmq_resp_err_t tmq_commit(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, int32_t async);
-DLL_EXPORT tmq_resp_err_t tmq_seek(tmq_t *tmq, const tmq_topic_vgroup_t *offset);
-#endif
+DLL_EXPORT int32_t   tmq_consumer_close(tmq_t *tmq);
+DLL_EXPORT int32_t   tmq_commit_sync(tmq_t *tmq, const TAOS_RES *msg);
+DLL_EXPORT void      tmq_commit_async(tmq_t *tmq, const TAOS_RES *msg, tmq_commit_cb *cb, void *param);
 
 /* ----------------------TMQ CONFIGURATION INTERFACE---------------------- */
 
@@ -275,11 +257,6 @@ DLL_EXPORT const char *tmq_get_db_name(TAOS_RES *res);
 DLL_EXPORT int32_t     tmq_get_vgroup_id(TAOS_RES *res);
 DLL_EXPORT const char *tmq_get_table_name(TAOS_RES *res);
 
-#if 0
-DLL_EXPORT int64_t     tmq_get_request_offset(tmq_message_t *message);
-DLL_EXPORT int64_t     tmq_get_response_offset(tmq_message_t *message);
-#endif
-
 /* ------------------------------ TMQ END -------------------------------- */
 
 #if 1  // Shuduo: temporary enable for app build

include/common/tcommon.h

@@ -35,6 +35,7 @@ enum {
   TMQ_MSG_TYPE__DUMMY = 0,
   TMQ_MSG_TYPE__POLL_RSP,
   TMQ_MSG_TYPE__EP_RSP,
+  TMQ_MSG_TYPE__END_RSP,
 };
 
 typedef enum EStreamType {

include/common/tmsg.h

@@ -429,8 +429,10 @@ STSchema* tdGetSTSChemaFromSSChema(SSchema** pSchema, int32_t nCols);
 typedef struct {
   char    name[TSDB_TABLE_FNAME_LEN];
   int8_t  igExists;
-  float   xFilesFactor;
-  int32_t delay;
+  int64_t delay1;
+  int64_t delay2;
+  int64_t watermark1;
+  int64_t watermark2;
   int32_t ttl;
   int32_t numOfColumns;
   int32_t numOfTags;

include/common/ttokendef.h

@@ -127,134 +127,133 @@
 #define TK_BLOB          109
 #define TK_VARBINARY     110
 #define TK_DECIMAL       111
-#define TK_FILE_FACTOR   112
-#define TK_NK_FLOAT      113
+#define TK_MAX_DELAY     112
+#define TK_WATERMARK     113
 #define TK_ROLLUP        114
 #define TK_TTL           115
 #define TK_SMA           116
-#define TK_SHOW          117
-#define TK_DATABASES     118
-#define TK_TABLES        119
-#define TK_STABLES       120
-#define TK_MNODES        121
-#define TK_MODULES       122
-#define TK_QNODES        123
-#define TK_FUNCTIONS     124
-#define TK_INDEXES       125
-#define TK_ACCOUNTS      126
-#define TK_APPS          127
-#define TK_CONNECTIONS   128
-#define TK_LICENCE       129
-#define TK_GRANTS        130
-#define TK_QUERIES       131
-#define TK_SCORES        132
-#define TK_TOPICS        133
-#define TK_VARIABLES     134
-#define TK_BNODES        135
-#define TK_SNODES        136
-#define TK_CLUSTER       137
-#define TK_TRANSACTIONS  138
-#define TK_LIKE          139
-#define TK_INDEX         140
-#define TK_FULLTEXT      141
-#define TK_FUNCTION      142
-#define TK_INTERVAL      143
-#define TK_TOPIC         144
-#define TK_AS            145
-#define TK_CONSUMER      146
-#define TK_GROUP         147
-#define TK_DESC          148
-#define TK_DESCRIBE      149
-#define TK_RESET         150
-#define TK_QUERY         151
-#define TK_CACHE         152
-#define TK_EXPLAIN       153
-#define TK_ANALYZE       154
-#define TK_VERBOSE       155
-#define TK_NK_BOOL       156
-#define TK_RATIO         157
-#define TK_COMPACT       158
-#define TK_VNODES        159
-#define TK_IN            160
-#define TK_OUTPUTTYPE    161
-#define TK_AGGREGATE     162
-#define TK_BUFSIZE       163
-#define TK_STREAM        164
-#define TK_INTO          165
-#define TK_TRIGGER       166
-#define TK_AT_ONCE       167
-#define TK_WINDOW_CLOSE  168
-#define TK_MAX_DELAY     169
-#define TK_WATERMARK     170
-#define TK_KILL          171
-#define TK_CONNECTION    172
-#define TK_TRANSACTION   173
-#define TK_BALANCE       174
-#define TK_VGROUP        175
-#define TK_MERGE         176
-#define TK_REDISTRIBUTE  177
-#define TK_SPLIT         178
-#define TK_SYNCDB        179
-#define TK_DELETE        180
-#define TK_NULL          181
-#define TK_NK_QUESTION   182
-#define TK_NK_ARROW      183
-#define TK_ROWTS         184
-#define TK_TBNAME        185
-#define TK_QSTARTTS      186
-#define TK_QENDTS        187
-#define TK_WSTARTTS      188
-#define TK_WENDTS        189
-#define TK_WDURATION     190
-#define TK_CAST          191
-#define TK_NOW           192
-#define TK_TODAY         193
-#define TK_TIMEZONE      194
-#define TK_COUNT         195
-#define TK_FIRST         196
-#define TK_LAST          197
-#define TK_LAST_ROW      198
-#define TK_BETWEEN       199
-#define TK_IS            200
-#define TK_NK_LT         201
-#define TK_NK_GT         202
-#define TK_NK_LE         203
-#define TK_NK_GE         204
-#define TK_NK_NE         205
-#define TK_MATCH         206
-#define TK_NMATCH        207
-#define TK_CONTAINS      208
-#define TK_JOIN          209
-#define TK_INNER         210
-#define TK_SELECT        211
-#define TK_DISTINCT      212
-#define TK_WHERE         213
-#define TK_PARTITION     214
-#define TK_BY            215
-#define TK_SESSION       216
-#define TK_STATE_WINDOW  217
-#define TK_SLIDING       218
-#define TK_FILL          219
-#define TK_VALUE         220
-#define TK_NONE          221
-#define TK_PREV          222
-#define TK_LINEAR        223
-#define TK_NEXT          224
-#define TK_HAVING        225
-#define TK_ORDER         226
-#define TK_SLIMIT        227
-#define TK_SOFFSET       228
-#define TK_LIMIT         229
-#define TK_OFFSET        230
-#define TK_ASC           231
-#define TK_NULLS         232
-#define TK_ID            233
-#define TK_NK_BITNOT     234
-#define TK_INSERT        235
-#define TK_VALUES        236
-#define TK_IMPORT        237
-#define TK_NK_SEMI       238
-#define TK_FILE          239
+#define TK_FIRST         117
+#define TK_LAST          118
+#define TK_SHOW          119
+#define TK_DATABASES     120
+#define TK_TABLES        121
+#define TK_STABLES       122
+#define TK_MNODES        123
+#define TK_MODULES       124
+#define TK_QNODES        125
+#define TK_FUNCTIONS     126
+#define TK_INDEXES       127
+#define TK_ACCOUNTS      128
+#define TK_APPS          129
+#define TK_CONNECTIONS   130
+#define TK_LICENCE       131
+#define TK_GRANTS        132
+#define TK_QUERIES       133
+#define TK_SCORES        134
+#define TK_TOPICS        135
+#define TK_VARIABLES     136
+#define TK_BNODES        137
+#define TK_SNODES        138
+#define TK_CLUSTER       139
+#define TK_TRANSACTIONS  140
+#define TK_LIKE          141
+#define TK_INDEX         142
+#define TK_FULLTEXT      143
+#define TK_FUNCTION      144
+#define TK_INTERVAL      145
+#define TK_TOPIC         146
+#define TK_AS            147
+#define TK_CONSUMER      148
+#define TK_GROUP         149
+#define TK_DESC          150
+#define TK_DESCRIBE      151
+#define TK_RESET         152
+#define TK_QUERY         153
+#define TK_CACHE         154
+#define TK_EXPLAIN       155
+#define TK_ANALYZE       156
+#define TK_VERBOSE       157
+#define TK_NK_BOOL       158
+#define TK_RATIO         159
+#define TK_NK_FLOAT      160
+#define TK_COMPACT       161
+#define TK_VNODES        162
+#define TK_IN            163
+#define TK_OUTPUTTYPE    164
+#define TK_AGGREGATE     165
+#define TK_BUFSIZE       166
+#define TK_STREAM        167
+#define TK_INTO          168
+#define TK_TRIGGER       169
+#define TK_AT_ONCE       170
+#define TK_WINDOW_CLOSE  171
+#define TK_KILL          172
+#define TK_CONNECTION    173
+#define TK_TRANSACTION   174
+#define TK_BALANCE       175
+#define TK_VGROUP        176
+#define TK_MERGE         177
+#define TK_REDISTRIBUTE  178
+#define TK_SPLIT         179
+#define TK_SYNCDB        180
+#define TK_DELETE        181
+#define TK_NULL          182
+#define TK_NK_QUESTION   183
+#define TK_NK_ARROW      184
+#define TK_ROWTS         185
+#define TK_TBNAME        186
+#define TK_QSTARTTS      187
+#define TK_QENDTS        188
+#define TK_WSTARTTS      189
+#define TK_WENDTS        190
+#define TK_WDURATION     191
+#define TK_CAST          192
+#define TK_NOW           193
+#define TK_TODAY         194
+#define TK_TIMEZONE      195
+#define TK_COUNT         196
+#define TK_LAST_ROW      197
+#define TK_BETWEEN       198
+#define TK_IS            199
+#define TK_NK_LT         200
+#define TK_NK_GT         201
+#define TK_NK_LE         202
+#define TK_NK_GE         203
+#define TK_NK_NE         204
+#define TK_MATCH         205
+#define TK_NMATCH        206
+#define TK_CONTAINS      207
+#define TK_JOIN          208
+#define TK_INNER         209
+#define TK_SELECT        210
+#define TK_DISTINCT      211
+#define TK_WHERE         212
+#define TK_PARTITION     213
+#define TK_BY            214
+#define TK_SESSION       215
+#define TK_STATE_WINDOW  216
+#define TK_SLIDING       217
+#define TK_FILL          218
+#define TK_VALUE         219
+#define TK_NONE          220
+#define TK_PREV          221
+#define TK_LINEAR        222
+#define TK_NEXT          223
+#define TK_HAVING        224
+#define TK_ORDER         225
+#define TK_SLIMIT        226
+#define TK_SOFFSET       227
+#define TK_LIMIT         228
+#define TK_OFFSET        229
+#define TK_ASC           230
+#define TK_NULLS         231
+#define TK_ID            232
+#define TK_NK_BITNOT     233
+#define TK_INSERT        234
+#define TK_VALUES        235
+#define TK_IMPORT        236
+#define TK_NK_SEMI       237
+#define TK_FILE          238
 
 #define TK_NK_SPACE   300
 #define TK_NK_COMMENT 301

include/libs/executor/executor.h

@@ -159,11 +159,14 @@ int64_t qGetQueriedTableUid(qTaskInfo_t tinfo);
  */
 int32_t qGetQualifiedTableIdList(void* pTableList, const char* tagCond, int32_t tagCondLen, SArray* pTableIdList);
 
-
 void qProcessFetchRsp(void* parent, struct SRpcMsg* pMsg, struct SEpSet* pEpSet);
 
 int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t* resNum, SExplainExecInfo** pRes);
 
+int32_t qSerializeTaskStatus(qTaskInfo_t tinfo, char** pOutput, int32_t* len);
+
+int32_t qDeserializeTaskStatus(qTaskInfo_t tinfo, const char* pInput, int32_t len);
+
 #ifdef __cplusplus
 }
 #endif

include/libs/function/function.h

@@ -158,10 +158,10 @@ typedef struct tExprNode {
   int32_t nodeType;
   union {
     struct {// function node
-      char              functionName[FUNCTIONS_NAME_MAX_LENGTH];  // todo refactor
-      int32_t           functionId;
-      int32_t           num;
-      struct SFunctionNode    *pFunctNode;
+      char     functionName[FUNCTIONS_NAME_MAX_LENGTH];  // todo refactor
+      int32_t  functionId;
+      int32_t  num;
+      struct SFunctionNode  *pFunctNode;
     } _function;
 
     struct {

include/libs/nodes/cmdnodes.h

@@ -88,8 +88,12 @@ typedef struct SAlterDatabaseStmt {
 typedef struct STableOptions {
   ENodeType  type;
   char       comment[TSDB_TB_COMMENT_LEN];
-  double     filesFactor;
-  int32_t    delay;
+  SNodeList* pMaxDelay;
+  int64_t    maxDelay1;
+  int64_t    maxDelay2;
+  SNodeList* pWatermark;
+  int64_t    watermark1;
+  int64_t    watermark2;
   SNodeList* pRollupFuncs;
   int32_t    ttl;
   SNodeList* pSma;
@@ -204,11 +208,18 @@ typedef struct SShowStmt {
   SNode*    pTbNamePattern;  // SValueNode
 } SShowStmt;
 
-typedef struct SShowCreatStmt {
+typedef struct SShowCreateDatabaseStmt {
   ENodeType type;
   char      dbName[TSDB_DB_NAME_LEN];
-  char      tableName[TSDB_TABLE_NAME_LEN];
-} SShowCreatStmt;
+  void*     pCfg;  // SDbCfgInfo
+} SShowCreateDatabaseStmt;
+
+typedef struct SShowCreateTableStmt {
+  ENodeType   type;
+  char        dbName[TSDB_DB_NAME_LEN];
+  char        tableName[TSDB_TABLE_NAME_LEN];
+  STableMeta* pMeta;
+} SShowCreateTableStmt;
 
 typedef enum EIndexType { INDEX_TYPE_SMA = 1, INDEX_TYPE_FULLTEXT } EIndexType;
 

include/libs/nodes/nodes.h

@@ -228,6 +228,7 @@ typedef enum ENodeType {
   QUERY_NODE_PHYSICAL_PLAN_FILL,
   QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION,
   QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION,
+  QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION,
   QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION,
   QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE,
   QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE,

include/libs/nodes/plannodes.h

@@ -130,13 +130,17 @@ typedef struct SMergeLogicNode {
 
 typedef enum EWindowType { WINDOW_TYPE_INTERVAL = 1, WINDOW_TYPE_SESSION, WINDOW_TYPE_STATE } EWindowType;
 
-typedef enum EIntervalAlgorithm {
+typedef enum EWindowAlgorithm {
   INTERVAL_ALGO_HASH = 1,
   INTERVAL_ALGO_MERGE,
   INTERVAL_ALGO_STREAM_FINAL,
   INTERVAL_ALGO_STREAM_SEMI,
   INTERVAL_ALGO_STREAM_SINGLE,
-} EIntervalAlgorithm;
+  SESSION_ALGO_STREAM_SEMI,
+  SESSION_ALGO_STREAM_FINAL,
+  SESSION_ALGO_STREAM_SINGLE,
+  SESSION_ALGO_MERGE,
+} EWindowAlgorithm;
 
 typedef struct SWindowLogicNode {
   SLogicNode         node;
@@ -153,7 +157,7 @@ typedef struct SWindowLogicNode {
   int8_t             triggerType;
   int64_t            watermark;
   double             filesFactor;
-  EIntervalAlgorithm intervalAlgo;
+  EWindowAlgorithm   windowAlgo;
 } SWindowLogicNode;
 
 typedef struct SFillLogicNode {
@@ -371,6 +375,8 @@ typedef struct SSessionWinodwPhysiNode {
 } SSessionWinodwPhysiNode;
 
 typedef SSessionWinodwPhysiNode SStreamSessionWinodwPhysiNode;
+typedef SSessionWinodwPhysiNode SStreamSemiSessionWinodwPhysiNode;
+typedef SSessionWinodwPhysiNode SStreamFinalSessionWinodwPhysiNode;
 
 typedef struct SStateWinodwPhysiNode {
   SWinodwPhysiNode window;

include/libs/sync/sync.h

@@ -62,6 +62,7 @@ typedef struct SSyncCfg {
 
 typedef struct SFsmCbMeta {
   SyncIndex  index;
+  SyncIndex  lastConfigIndex;
   bool       isWeak;
   int32_t    code;
   ESyncState state;
@@ -75,6 +76,7 @@ typedef struct SReConfigCbMeta {
   int32_t   code;
   SyncIndex index;
   SyncTerm  term;
+  SyncIndex lastConfigIndex;
   SyncTerm  currentTerm;
   SSyncCfg  oldCfg;
   SSyncCfg  newCfg;

include/libs/transport/trpc.h

@@ -23,6 +23,7 @@ extern "C" {
 #include <stdint.h>
 #include "taosdef.h"
 #include "tmsg.h"
+#include "ttrace.h"
 
 #define TAOS_CONN_SERVER 0
 #define TAOS_CONN_CLIENT 1
@@ -41,10 +42,12 @@ typedef struct {
 
 typedef struct SRpcHandleInfo {
   // rpc info
-  void *  handle;         // rpc handle returned to app
-  int64_t refId;          // refid, used by server
-  int32_t noResp;         // has response or not(default 0, 0: resp, 1: no resp);
-  int32_t persistHandle;  // persist handle or not
+  void *   handle;         // rpc handle returned to app
+  int64_t  refId;          // refid, used by server
+  int32_t  noResp;         // has response or not(default 0, 0: resp, 1: no resp);
+  int32_t  persistHandle;  // persist handle or not
+  STraceId traceId;
+  // int64_t traceId;
 
   // app info
   void *ahandle;  // app handle set by client

include/os/osTime.h

@@ -23,22 +23,22 @@ extern "C" {
 // If the error is in a third-party library, place this header file under the third-party library header file.
 // When you want to use this feature, you should find or add the same function in the following section.
 #ifndef ALLOW_FORBID_FUNC
-    #define strptime STRPTIME_FUNC_TAOS_FORBID
-    #define gettimeofday GETTIMEOFDAY_FUNC_TAOS_FORBID
-    #define localtime   LOCALTIME_FUNC_TAOS_FORBID
-    #define localtime_s LOCALTIMES_FUNC_TAOS_FORBID
-    #define localtime_r LOCALTIMER_FUNC_TAOS_FORBID
-    #define time TIME_FUNC_TAOS_FORBID
-    #define mktime MKTIME_FUNC_TAOS_FORBID
+#define strptime     STRPTIME_FUNC_TAOS_FORBID
+#define gettimeofday GETTIMEOFDAY_FUNC_TAOS_FORBID
+#define localtime    LOCALTIME_FUNC_TAOS_FORBID
+#define localtime_s  LOCALTIMES_FUNC_TAOS_FORBID
+#define localtime_r  LOCALTIMER_FUNC_TAOS_FORBID
+#define time         TIME_FUNC_TAOS_FORBID
+#define mktime       MKTIME_FUNC_TAOS_FORBID
 #endif
 
 #ifdef WINDOWS
 
-  #define CLOCK_REALTIME 	0
+#define CLOCK_REALTIME 0
 
-  #define MILLISECOND_PER_SECOND (1000i64)
+#define MILLISECOND_PER_SECOND (1000i64)
 #else
-  #define MILLISECOND_PER_SECOND ((int64_t)1000L)
+#define MILLISECOND_PER_SECOND ((int64_t)1000L)
 #endif
 
 #define MILLISECOND_PER_MINUTE (MILLISECOND_PER_SECOND * 60)
@@ -82,13 +82,13 @@ static FORCE_INLINE int64_t taosGetTimestampNs() {
   return (int64_t)systemTime.tv_sec * 1000000000L + (int64_t)systemTime.tv_nsec;
 }
 
-char *taosStrpTime(const char *buf, const char *fmt, struct tm *tm);
+char *     taosStrpTime(const char *buf, const char *fmt, struct tm *tm);
 struct tm *taosLocalTime(const time_t *timep, struct tm *result);
-time_t taosTime(time_t *t);
-time_t taosMktime(struct tm *timep);
+time_t     taosTime(time_t *t);
+time_t     taosMktime(struct tm *timep);
 
 #ifdef __cplusplus
 }
 #endif
 
-#endif  /*_TD_OS_TIME_H_*/
+#endif /*_TD_OS_TIME_H_*/

include/util/taoserror.h

@@ -431,7 +431,8 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_TQ_META_KEY_NOT_IN_TXN        TAOS_DEF_ERROR_CODE(0, 0x0A09)
 #define TSDB_CODE_TQ_META_KEY_DUP_IN_TXN        TAOS_DEF_ERROR_CODE(0, 0x0A0A)
 #define TSDB_CODE_TQ_GROUP_NOT_SET              TAOS_DEF_ERROR_CODE(0, 0x0A0B)
-#define TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND     TAOS_DEF_ERROR_CODE(0, 0x0A0B)
+#define TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND     TAOS_DEF_ERROR_CODE(0, 0x0A0C)
+#define TSDB_CODE_TQ_NO_COMMITTED_OFFSET        TAOS_DEF_ERROR_CODE(0, 0x0A0D)
 
 // wal
 #define TSDB_CODE_WAL_APP_ERROR                 TAOS_DEF_ERROR_CODE(0, 0x1000)
@@ -441,108 +442,6 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_WAL_OUT_OF_MEMORY             TAOS_DEF_ERROR_CODE(0, 0x1004)
 #define TSDB_CODE_WAL_LOG_NOT_EXIST             TAOS_DEF_ERROR_CODE(0, 0x1005)
 
-// http
-#define TSDB_CODE_HTTP_SERVER_OFFLINE           TAOS_DEF_ERROR_CODE(0, 0x1100)  //"http server is not online"
-#define TSDB_CODE_HTTP_UNSUPPORT_URL            TAOS_DEF_ERROR_CODE(0, 0x1101)  //"url is not support"
-#define TSDB_CODE_HTTP_INVALID_URL              TAOS_DEF_ERROR_CODE(0, 0x1102)  //invalid url format"
-#define TSDB_CODE_HTTP_NO_ENOUGH_MEMORY         TAOS_DEF_ERROR_CODE(0, 0x1103)  //"no enough memory"
-#define TSDB_CODE_HTTP_REQUSET_TOO_BIG          TAOS_DEF_ERROR_CODE(0, 0x1104)  //"request size is too big"
-#define TSDB_CODE_HTTP_NO_AUTH_INFO             TAOS_DEF_ERROR_CODE(0, 0x1105)  //"no auth info input"
-#define TSDB_CODE_HTTP_NO_MSG_INPUT             TAOS_DEF_ERROR_CODE(0, 0x1106)  //"request is empty"
-#define TSDB_CODE_HTTP_NO_SQL_INPUT             TAOS_DEF_ERROR_CODE(0, 0x1107)  //"no sql input"
-#define TSDB_CODE_HTTP_NO_EXEC_USEDB            TAOS_DEF_ERROR_CODE(0, 0x1108)  //"no need to execute use db cmd"
-#define TSDB_CODE_HTTP_SESSION_FULL             TAOS_DEF_ERROR_CODE(0, 0x1109)  //"session list was full"
-#define TSDB_CODE_HTTP_GEN_TAOSD_TOKEN_ERR      TAOS_DEF_ERROR_CODE(0, 0x110A)  //"generate taosd token error"
-#define TSDB_CODE_HTTP_INVALID_MULTI_REQUEST    TAOS_DEF_ERROR_CODE(0, 0x110B)  //"size of multi request is 0"
-#define TSDB_CODE_HTTP_CREATE_GZIP_FAILED       TAOS_DEF_ERROR_CODE(0, 0x110C)  //"failed to create gzip"
-#define TSDB_CODE_HTTP_FINISH_GZIP_FAILED       TAOS_DEF_ERROR_CODE(0, 0x110D)  //"failed to finish gzip"
-#define TSDB_CODE_HTTP_LOGIN_FAILED             TAOS_DEF_ERROR_CODE(0, 0x110E)  //"failed to login"
-
-#define TSDB_CODE_HTTP_INVALID_VERSION          TAOS_DEF_ERROR_CODE(0, 0x1120)  //"invalid http version"
-#define TSDB_CODE_HTTP_INVALID_CONTENT_LENGTH   TAOS_DEF_ERROR_CODE(0, 0x1121)  //"invalid content length"
-#define TSDB_CODE_HTTP_INVALID_AUTH_TYPE        TAOS_DEF_ERROR_CODE(0, 0x1122)  //"invalid type of Authorization"
-#define TSDB_CODE_HTTP_INVALID_AUTH_FORMAT      TAOS_DEF_ERROR_CODE(0, 0x1123)  //"invalid format of Authorization"
-#define TSDB_CODE_HTTP_INVALID_BASIC_AUTH       TAOS_DEF_ERROR_CODE(0, 0x1124)  //"invalid basic Authorization"
-#define TSDB_CODE_HTTP_INVALID_TAOSD_AUTH       TAOS_DEF_ERROR_CODE(0, 0x1125)  //"invalid taosd Authorization"
-#define TSDB_CODE_HTTP_PARSE_METHOD_FAILED      TAOS_DEF_ERROR_CODE(0, 0x1126)  //"failed to parse method"
-#define TSDB_CODE_HTTP_PARSE_TARGET_FAILED      TAOS_DEF_ERROR_CODE(0, 0x1127)  //"failed to parse target"
-#define TSDB_CODE_HTTP_PARSE_VERSION_FAILED     TAOS_DEF_ERROR_CODE(0, 0x1128)  //"failed to parse http version"
-#define TSDB_CODE_HTTP_PARSE_SP_FAILED          TAOS_DEF_ERROR_CODE(0, 0x1129)  //"failed to parse sp"
-#define TSDB_CODE_HTTP_PARSE_STATUS_FAILED      TAOS_DEF_ERROR_CODE(0, 0x112A)  //"failed to parse status"
-#define TSDB_CODE_HTTP_PARSE_PHRASE_FAILED      TAOS_DEF_ERROR_CODE(0, 0x112B)  //"failed to parse phrase"
-#define TSDB_CODE_HTTP_PARSE_CRLF_FAILED        TAOS_DEF_ERROR_CODE(0, 0x112C)  //"failed to parse crlf"
-#define TSDB_CODE_HTTP_PARSE_HEADER_FAILED      TAOS_DEF_ERROR_CODE(0, 0x112D)  //"failed to parse header"
-#define TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED  TAOS_DEF_ERROR_CODE(0, 0x112E)  //"failed to parse header key"
-#define TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED  TAOS_DEF_ERROR_CODE(0, 0x112F)  //"failed to parse header val"
-#define TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED  TAOS_DEF_ERROR_CODE(0, 0x1130)  //"failed to parse chunk size"
-#define TSDB_CODE_HTTP_PARSE_CHUNK_FAILED       TAOS_DEF_ERROR_CODE(0, 0x1131)  //"failed to parse chunk"
-#define TSDB_CODE_HTTP_PARSE_END_FAILED         TAOS_DEF_ERROR_CODE(0, 0x1132)  //"failed to parse end section"
-#define TSDB_CODE_HTTP_PARSE_INVALID_STATE      TAOS_DEF_ERROR_CODE(0, 0x1134)  //"invalid parse state"
-#define TSDB_CODE_HTTP_PARSE_ERROR_STATE        TAOS_DEF_ERROR_CODE(0, 0x1135)  //"failed to parse error section"
-
-#define TSDB_CODE_HTTP_GC_QUERY_NULL            TAOS_DEF_ERROR_CODE(0, 0x1150)  //"query size is 0"
-#define TSDB_CODE_HTTP_GC_QUERY_SIZE            TAOS_DEF_ERROR_CODE(0, 0x1151)  //"query size can not more than 100"
-#define TSDB_CODE_HTTP_GC_REQ_PARSE_ERROR       TAOS_DEF_ERROR_CODE(0, 0x1152)  //"parse grafana json error"
-
-#define TSDB_CODE_HTTP_TG_DB_NOT_INPUT          TAOS_DEF_ERROR_CODE(0, 0x1160)  //"database name can not be null"
-#define TSDB_CODE_HTTP_TG_DB_TOO_LONG           TAOS_DEF_ERROR_CODE(0, 0x1161)  //"database name too long"
-#define TSDB_CODE_HTTP_TG_INVALID_JSON          TAOS_DEF_ERROR_CODE(0, 0x1162)  //"invalid telegraf json fromat"
-#define TSDB_CODE_HTTP_TG_METRICS_NULL          TAOS_DEF_ERROR_CODE(0, 0x1163)  //"metrics size is 0"
-#define TSDB_CODE_HTTP_TG_METRICS_SIZE          TAOS_DEF_ERROR_CODE(0, 0x1164)  //"metrics size can not more than 1K"
-#define TSDB_CODE_HTTP_TG_METRIC_NULL           TAOS_DEF_ERROR_CODE(0, 0x1165)  //"metric name not find"
-#define TSDB_CODE_HTTP_TG_METRIC_TYPE           TAOS_DEF_ERROR_CODE(0, 0x1166)  //"metric name type should be string"
-#define TSDB_CODE_HTTP_TG_METRIC_NAME_NULL      TAOS_DEF_ERROR_CODE(0, 0x1167)  //"metric name length is 0"
-#define TSDB_CODE_HTTP_TG_METRIC_NAME_LONG      TAOS_DEF_ERROR_CODE(0, 0x1168)  //"metric name length too long"
-#define TSDB_CODE_HTTP_TG_TIMESTAMP_NULL        TAOS_DEF_ERROR_CODE(0, 0x1169)  //"timestamp not find"
-#define TSDB_CODE_HTTP_TG_TIMESTAMP_TYPE        TAOS_DEF_ERROR_CODE(0, 0x116A)  //"timestamp type should be integer"
-#define TSDB_CODE_HTTP_TG_TIMESTAMP_VAL_NULL    TAOS_DEF_ERROR_CODE(0, 0x116B)  //"timestamp value smaller than 0"
-#define TSDB_CODE_HTTP_TG_TAGS_NULL             TAOS_DEF_ERROR_CODE(0, 0x116C)  //"tags not find"
-#define TSDB_CODE_HTTP_TG_TAGS_SIZE_0           TAOS_DEF_ERROR_CODE(0, 0x116D)  //"tags size is 0"
-#define TSDB_CODE_HTTP_TG_TAGS_SIZE_LONG        TAOS_DEF_ERROR_CODE(0, 0x116E)  //"tags size too long"
-#define TSDB_CODE_HTTP_TG_TAG_NULL              TAOS_DEF_ERROR_CODE(0, 0x116F)  //"tag is null"
-#define TSDB_CODE_HTTP_TG_TAG_NAME_NULL         TAOS_DEF_ERROR_CODE(0, 0x1170)  //"tag name is null"
-#define TSDB_CODE_HTTP_TG_TAG_NAME_SIZE         TAOS_DEF_ERROR_CODE(0, 0x1171)  //"tag name length too long"
-#define TSDB_CODE_HTTP_TG_TAG_VALUE_TYPE        TAOS_DEF_ERROR_CODE(0, 0x1172)  //"tag value type should be number or string"
-#define TSDB_CODE_HTTP_TG_TAG_VALUE_NULL        TAOS_DEF_ERROR_CODE(0, 0x1173)  //"tag value is null"
-#define TSDB_CODE_HTTP_TG_TABLE_NULL            TAOS_DEF_ERROR_CODE(0, 0x1174)  //"table is null"
-#define TSDB_CODE_HTTP_TG_TABLE_SIZE            TAOS_DEF_ERROR_CODE(0, 0x1175)  //"table name length too long"
-#define TSDB_CODE_HTTP_TG_FIELDS_NULL           TAOS_DEF_ERROR_CODE(0, 0x1176)  //"fields not find"
-#define TSDB_CODE_HTTP_TG_FIELDS_SIZE_0         TAOS_DEF_ERROR_CODE(0, 0x1177)  //"fields size is 0"
-#define TSDB_CODE_HTTP_TG_FIELDS_SIZE_LONG      TAOS_DEF_ERROR_CODE(0, 0x1178)  //"fields size too long"
-#define TSDB_CODE_HTTP_TG_FIELD_NULL            TAOS_DEF_ERROR_CODE(0, 0x1179)  //"field is null"
-#define TSDB_CODE_HTTP_TG_FIELD_NAME_NULL       TAOS_DEF_ERROR_CODE(0, 0x117A)  //"field name is null"
-#define TSDB_CODE_HTTP_TG_FIELD_NAME_SIZE       TAOS_DEF_ERROR_CODE(0, 0x117B)  //"field name length too long"
-#define TSDB_CODE_HTTP_TG_FIELD_VALUE_TYPE      TAOS_DEF_ERROR_CODE(0, 0x117C)  //"field value type should be number or string"
-#define TSDB_CODE_HTTP_TG_FIELD_VALUE_NULL      TAOS_DEF_ERROR_CODE(0, 0x117D)  //"field value is null"
-#define TSDB_CODE_HTTP_TG_HOST_NOT_STRING       TAOS_DEF_ERROR_CODE(0, 0x117E)  //"host type should be string"
-#define TSDB_CODE_HTTP_TG_STABLE_NOT_EXIST      TAOS_DEF_ERROR_CODE(0, 0x117F)  //"stable not exist"
-
-#define TSDB_CODE_HTTP_OP_DB_NOT_INPUT          TAOS_DEF_ERROR_CODE(0, 0x1190)  //"database name can not be null"
-#define TSDB_CODE_HTTP_OP_DB_TOO_LONG           TAOS_DEF_ERROR_CODE(0, 0x1191)  //"database name too long"
-#define TSDB_CODE_HTTP_OP_INVALID_JSON          TAOS_DEF_ERROR_CODE(0, 0x1192)  //"invalid opentsdb json fromat"
-#define TSDB_CODE_HTTP_OP_METRICS_NULL          TAOS_DEF_ERROR_CODE(0, 0x1193)  //"metrics size is 0"
-#define TSDB_CODE_HTTP_OP_METRICS_SIZE          TAOS_DEF_ERROR_CODE(0, 0x1194)  //"metrics size can not more than 10K"
-#define TSDB_CODE_HTTP_OP_METRIC_NULL           TAOS_DEF_ERROR_CODE(0, 0x1195)  //"metric name not find"
-#define TSDB_CODE_HTTP_OP_METRIC_TYPE           TAOS_DEF_ERROR_CODE(0, 0x1196)  //"metric name type should be string"
-#define TSDB_CODE_HTTP_OP_METRIC_NAME_NULL      TAOS_DEF_ERROR_CODE(0, 0x1197)  //"metric name length is 0"
-#define TSDB_CODE_HTTP_OP_METRIC_NAME_LONG      TAOS_DEF_ERROR_CODE(0, 0x1198)  //"metric name length can not more than 22"
-#define TSDB_CODE_HTTP_OP_TIMESTAMP_NULL        TAOS_DEF_ERROR_CODE(0, 0x1199)  //"timestamp not find"
-#define TSDB_CODE_HTTP_OP_TIMESTAMP_TYPE        TAOS_DEF_ERROR_CODE(0, 0x119A)  //"timestamp type should be integer"
-#define TSDB_CODE_HTTP_OP_TIMESTAMP_VAL_NULL    TAOS_DEF_ERROR_CODE(0, 0x119B)  //"timestamp value smaller than 0"
-#define TSDB_CODE_HTTP_OP_TAGS_NULL             TAOS_DEF_ERROR_CODE(0, 0x119C)  //"tags not find"
-#define TSDB_CODE_HTTP_OP_TAGS_SIZE_0           TAOS_DEF_ERROR_CODE(0, 0x119D)  //"tags size is 0"
-#define TSDB_CODE_HTTP_OP_TAGS_SIZE_LONG        TAOS_DEF_ERROR_CODE(0, 0x119E)  //"tags size too long"
-#define TSDB_CODE_HTTP_OP_TAG_NULL              TAOS_DEF_ERROR_CODE(0, 0x119F)  //"tag is null"
-#define TSDB_CODE_HTTP_OP_TAG_NAME_NULL         TAOS_DEF_ERROR_CODE(0, 0x11A0)  //"tag name is null"
-#define TSDB_CODE_HTTP_OP_TAG_NAME_SIZE         TAOS_DEF_ERROR_CODE(0, 0x11A1)  //"tag name length too long"
-#define TSDB_CODE_HTTP_OP_TAG_VALUE_TYPE        TAOS_DEF_ERROR_CODE(0, 0x11A2)  //"tag value type should be boolean number or string"
-#define TSDB_CODE_HTTP_OP_TAG_VALUE_NULL        TAOS_DEF_ERROR_CODE(0, 0x11A3)  //"tag value is null"
-#define TSDB_CODE_HTTP_OP_TAG_VALUE_TOO_LONG    TAOS_DEF_ERROR_CODE(0, 0x11A4)  //"tag value can not more than 64"
-#define TSDB_CODE_HTTP_OP_VALUE_NULL            TAOS_DEF_ERROR_CODE(0, 0x11A5)  //"value not find"
-#define TSDB_CODE_HTTP_OP_VALUE_TYPE            TAOS_DEF_ERROR_CODE(0, 0x11A6)  //"value type should be boolean number or string"
-
-#define TSDB_CODE_HTTP_REQUEST_JSON_ERROR       TAOS_DEF_ERROR_CODE(0, 0x1F00)  //"http request json error"
-
 // tfs
 #define TSDB_CODE_FS_APP_ERROR                  TAOS_DEF_ERROR_CODE(0, 0x2200)
 #define TSDB_CODE_FS_INVLD_CFG                  TAOS_DEF_ERROR_CODE(0, 0x2201)
@@ -664,6 +563,7 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC   TAOS_DEF_ERROR_CODE(0, 0x2659)
 #define TSDB_CODE_PAR_STREAM_NOT_ALLOWED_FUNC   TAOS_DEF_ERROR_CODE(0, 0x265A)
 #define TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC TAOS_DEF_ERROR_CODE(0, 0x265B)
+#define TSDB_CODE_PAR_INVALID_TABLE_OPTION      TAOS_DEF_ERROR_CODE(0, 0x265C)
 
 //planner
 #define TSDB_CODE_PLAN_INTERNAL_ERROR           TAOS_DEF_ERROR_CODE(0, 0x2700)
@@ -712,6 +612,8 @@ int32_t* taosGetErrno();
 //index
 #define TSDB_CODE_INDEX_REBUILDING               TAOS_DEF_ERROR_CODE(0, 0x3200)
 
+//tmq
+#define TSDB_CODE_TMQ_INVALID_MSG                TAOS_DEF_ERROR_CODE(0, 0x4000)
 
 #ifdef __cplusplus
 }

include/util/tdef.h

@@ -344,14 +344,13 @@ typedef enum ELogicConditionType {
 #define TSDB_DB_SCHEMALESS_OFF          0
 #define TSDB_DEFAULT_DB_SCHEMALESS      TSDB_DB_SCHEMALESS_OFF
 
-// #define TSDB_MIN_ROLLUP_DELAY     1
-// #define TSDB_MAX_ROLLUP_DELAY     10
-// #define TSDB_DEFAULT_ROLLUP_DELAY 1
-#define TSDB_MIN_ROLLUP_FILE_FACTOR     0
-#define TSDB_MAX_ROLLUP_FILE_FACTOR     10
-#define TSDB_DEFAULT_ROLLUP_FILE_FACTOR 0.1
-#define TSDB_MIN_TABLE_TTL              0
-#define TSDB_DEFAULT_TABLE_TTL          0
+#define TSDB_MIN_ROLLUP_MAX_DELAY     1  // unit millisecond
+#define TSDB_MAX_ROLLUP_MAX_DELAY     (15 * 60 * 1000)
+#define TSDB_MIN_ROLLUP_WATERMARK     0  // unit millisecond
+#define TSDB_MAX_ROLLUP_WATERMARK     (15 * 60 * 1000)
+#define TSDB_DEFAULT_ROLLUP_WATERMARK 5000
+#define TSDB_MIN_TABLE_TTL            0
+#define TSDB_DEFAULT_TABLE_TTL        0
 
 #define TSDB_MIN_EXPLAIN_RATIO     0
 #define TSDB_MAX_EXPLAIN_RATIO     1

include/util/ttrace.h

+/*
+ * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
+ *
+ * This program is free software: you can use, redistribute, and/or modify
+ * it under the terms of the GNU Affero General Public License, version 3
+ * or later ("AGPL"), as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef _TD_TRACE_H_
+#define _TD_TRACE_H_
+
+#include <stdint.h>
+#include <stdlib.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#pragma(push, 1)
+
+typedef struct STraceId {
+  int64_t rootId;
+  int64_t msgId;
+} STraceId;
+
+#pragma(pop)
+
+#define TRACE_SET_ROOTID(traceId, root) \
+  do {                                  \
+    (traceId)->rootId = root;           \
+  } while (0);
+
+#define TRACE_GET_ROOTID(traceId) (traceId)->rootId
+
+#define TRACE_SET_MSGID(traceId, mId) \
+  do {                                \
+    (traceId)->msgId = mId;           \
+  } while (0)
+
+#define TRACE_GET_MSGID(traceId) (traceId)->msgId
+
+#define TRACE_TO_STR(traceId, buf)                                                \
+  do {                                                                            \
+    sprintf(buf, "0x%" PRIx64 ":0x%" PRIx64 "", traceId->rootId, traceId->msgId); \
+  } while (0)
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

source/common/src/systable.c

@@ -220,7 +220,8 @@ static const SSysDbTableSchema transSchema[] = {
     {.name = "id", .bytes = 4, .type = TSDB_DATA_TYPE_INT},
     {.name = "create_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP},
     {.name = "stage", .bytes = TSDB_TRANS_STAGE_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR},
-    {.name = "db", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR},
+    {.name = "db1", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR},
+    {.name = "db2", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR},
     {.name = "failed_times", .bytes = 4, .type = TSDB_DATA_TYPE_INT},
     {.name = "last_exec_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP},
     {.name = "last_action_info",

source/common/src/tdatablock.c

@@ -1713,6 +1713,7 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks
 }
 
 char* buildCtbNameByGroupId(const char* stbName, uint64_t groupId) {
+  ASSERT(stbName[0] != 0);
   SArray* tags = taosArrayInit(0, sizeof(void*));
   SSmlKv* pTag = taosMemoryCalloc(1, sizeof(SSmlKv));
   pTag->key = "group_id";

source/common/src/tmsg.c

@@ -204,7 +204,7 @@ static int32_t tSerializeSClientHbReq(SEncoder *pEncoder, const SClientHbReq *pR
     if (tEncodeU64(pEncoder, pReq->app.summary.numOfSlowQueries) < 0) return -1;
     if (tEncodeU64(pEncoder, pReq->app.summary.totalRequests) < 0) return -1;
     if (tEncodeU64(pEncoder, pReq->app.summary.currentRequests) < 0) return -1;
-    
+
     int32_t queryNum = 0;
     if (pReq->query) {
       queryNum = 1;
@@ -288,7 +288,7 @@ static int32_t tDeserializeSClientHbReq(SDecoder *pDecoder, SClientHbReq *pReq)
           if (tDecodeI64(pDecoder, &desc.useconds) < 0) return -1;
           if (tDecodeI64(pDecoder, &desc.stime) < 0) return -1;
           if (tDecodeI64(pDecoder, &desc.reqRid) < 0) return -1;
-          if (tDecodeI8(pDecoder, (int8_t*)&desc.stableQuery) < 0) return -1;
+          if (tDecodeI8(pDecoder, (int8_t *)&desc.stableQuery) < 0) return -1;
           if (tDecodeCStrTo(pDecoder, desc.fqdn) < 0) return -1;
           if (tDecodeI32(pDecoder, &desc.subPlanNum) < 0) return -1;
 
@@ -496,8 +496,10 @@ int32_t tSerializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pReq
   if (tStartEncode(&encoder) < 0) return -1;
   if (tEncodeCStr(&encoder, pReq->name) < 0) return -1;
   if (tEncodeI8(&encoder, pReq->igExists) < 0) return -1;
-  if (tEncodeFloat(&encoder, pReq->xFilesFactor) < 0) return -1;
-  if (tEncodeI32(&encoder, pReq->delay) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->delay1) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->delay2) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->watermark1) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->watermark2) < 0) return -1;
   if (tEncodeI32(&encoder, pReq->ttl) < 0) return -1;
   if (tEncodeI32(&encoder, pReq->numOfColumns) < 0) return -1;
   if (tEncodeI32(&encoder, pReq->numOfTags) < 0) return -1;
@@ -544,8 +546,10 @@ int32_t tDeserializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pR
   if (tStartDecode(&decoder) < 0) return -1;
   if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1;
   if (tDecodeI8(&decoder, &pReq->igExists) < 0) return -1;
-  if (tDecodeFloat(&decoder, &pReq->xFilesFactor) < 0) return -1;
-  if (tDecodeI32(&decoder, &pReq->delay) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->delay1) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->delay2) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->watermark1) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->watermark2) < 0) return -1;
   if (tDecodeI32(&decoder, &pReq->ttl) < 0) return -1;
   if (tDecodeI32(&decoder, &pReq->numOfColumns) < 0) return -1;
   if (tDecodeI32(&decoder, &pReq->numOfTags) < 0) return -1;

source/dnode/mgmt/mgmt_dnode/src/dmWorker.c

@@ -101,7 +101,8 @@ void dmStopMonitorThread(SDnodeMgmt *pMgmt) {
 static void dmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   SDnodeMgmt *pMgmt = pInfo->ahandle;
   int32_t     code = -1;
-  dTrace("msg:%p, will be processed in dnode queue, type:%s", pMsg, TMSG_INFO(pMsg->msgType));
+  STraceId *  trace = &pMsg->info.traceId;
+  dGTrace("msg:%p, will be processed in dnode queue, type:%s", pMsg, TMSG_INFO(pMsg->msgType));
 
   switch (pMsg->msgType) {
     case TDMT_DND_CONFIG_DNODE:

source/dnode/mgmt/mgmt_mnode/src/mmWorker.c

@@ -48,7 +48,8 @@ static inline void mmSendRsp(SRpcMsg *pMsg, int32_t code) {
 static void mmProcessRpcMsg(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   SMnodeMgmt *pMgmt = pInfo->ahandle;
   int32_t     code = -1;
-  dTrace("msg:%p, get from mnode queue", pMsg);
+  STraceId *  trace = &pMsg->info.traceId;
+  dGTrace("msg:%p, get from mnode queue", pMsg);
 
   switch (pMsg->msgType) {
     case TDMT_MON_MM_INFO:

source/dnode/mgmt/mgmt_vnode/src/vmWorker.c

@@ -31,7 +31,8 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   SVnodeMgmt *pMgmt = pInfo->ahandle;
   int32_t     code = -1;
 
-  dTrace("msg:%p, get from vnode-mgmt queue", pMsg);
+  STraceId *trace = &pMsg->info.traceId;
+  dGTrace("msg:%p, get from vnode-mgmt queue", pMsg);
   switch (pMsg->msgType) {
     case TDMT_MON_VM_INFO:
       code = vmProcessGetMonitorInfoReq(pMgmt, pMsg);
@@ -97,7 +98,7 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
 
 static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
   SVnodeObj *pVnode = pInfo->ahandle;
-  SRpcMsg   *pMsg = NULL;
+  SRpcMsg *  pMsg = NULL;
 
   for (int32_t i = 0; i < numOfMsgs; ++i) {
     if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
@@ -118,7 +119,7 @@ static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOf
 
 static void vmProcessMergeQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
   SVnodeObj *pVnode = pInfo->ahandle;
-  SRpcMsg   *pMsg = NULL;
+  SRpcMsg *  pMsg = NULL;
 
   for (int32_t i = 0; i < numOfMsgs; ++i) {
     if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;

source/dnode/mgmt/node_mgmt/src/dmTransport.c

@@ -40,8 +40,9 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
   SMgmtWrapper *pWrapper = NULL;
   SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)];
 
-  dTrace("msg:%s is received, handle:%p len:%d code:0x%x app:%p refId:%" PRId64, TMSG_INFO(pRpc->msgType),
-         pRpc->info.handle, pRpc->contLen, pRpc->code, pRpc->info.ahandle, pRpc->info.refId);
+  STraceId *trace = &pRpc->info.traceId;
+  dGTrace("msg:%s is received, handle:%p len:%d code:0x%x app:%p refId:%" PRId64, TMSG_INFO(pRpc->msgType),
+          pRpc->info.handle, pRpc->contLen, pRpc->code, pRpc->info.ahandle, pRpc->info.refId);
 
   if (pRpc->msgType == TDMT_DND_NET_TEST) {
     dmProcessNetTestReq(pDnode, pRpc);

source/dnode/mgmt/node_util/inc/dmUtil.h

@@ -34,13 +34,13 @@
 
 #include "dnode.h"
 #include "mnode.h"
-#include "qnode.h"
 #include "monitor.h"
+#include "qnode.h"
 #include "sync.h"
 #include "wal.h"
 
 #include "libs/function/function.h"
-
+// clang-format off
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -51,6 +51,7 @@ extern "C" {
 #define dInfo(...)  { if (dDebugFlag & DEBUG_INFO)  { taosPrintLog("DND ", DEBUG_INFO, 255, __VA_ARGS__); }}
 #define dDebug(...) { if (dDebugFlag & DEBUG_DEBUG) { taosPrintLog("DND ", DEBUG_DEBUG, dDebugFlag, __VA_ARGS__); }}
 #define dTrace(...) { if (dDebugFlag & DEBUG_TRACE) { taosPrintLog("DND ", DEBUG_TRACE, dDebugFlag, __VA_ARGS__); }}
+#define dGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dTrace(param ",GTID: %s", __VA_ARGS__, buf);} while(0)
 
 typedef enum {
   DNODE = 0,
@@ -184,3 +185,4 @@ void    dmSetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet);
 #endif
 
 #endif /*_TD_DM_INT_H_*/
+// clang-format on

source/dnode/mnode/impl/inc/mndDef.h

@@ -124,7 +124,8 @@ typedef struct {
   int32_t        lastErrorNo;
   tmsg_t         lastMsgType;
   SEpSet         lastEpset;
-  char           dbname[TSDB_DB_FNAME_LEN];
+  char           dbname1[TSDB_DB_FNAME_LEN];
+  char           dbname2[TSDB_DB_FNAME_LEN];
   int32_t        startFunc;
   int32_t        stopFunc;
   int32_t        paramLen;

source/dnode/mnode/impl/inc/mndInt.h

@@ -40,6 +40,8 @@ extern "C" {
 #define mInfo(...)  { if (mDebugFlag & DEBUG_INFO)  { taosPrintLog("MND ", DEBUG_INFO, 255, __VA_ARGS__); }}
 #define mDebug(...) { if (mDebugFlag & DEBUG_DEBUG) { taosPrintLog("MND ", DEBUG_DEBUG, mDebugFlag, __VA_ARGS__); }}
 #define mTrace(...) { if (mDebugFlag & DEBUG_TRACE) { taosPrintLog("MND ", DEBUG_TRACE, mDebugFlag, __VA_ARGS__); }}
+#define mGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); mTrace(param ", GTID: %s", __VA_ARGS__, buf);} while(0)
+
 // clang-format on
 
 #define SYSTABLE_SCH_TABLE_NAME_LEN ((TSDB_TABLE_NAME_LEN - 1) + VARSTR_HEADER_SIZE)
@@ -54,7 +56,7 @@ typedef void (*ShowFreeIterFp)(SMnode *pMnode, void *pIter);
 typedef struct SQWorker SQHandle;
 
 typedef struct {
-  const char  *name;
+  const char * name;
   MndInitFp    initFp;
   MndCleanupFp cleanupFp;
 } SMnodeStep;
@@ -63,7 +65,7 @@ typedef struct {
   int64_t        showId;
   ShowRetrieveFp retrieveFps[TSDB_MGMT_TABLE_MAX];
   ShowFreeIterFp freeIterFps[TSDB_MGMT_TABLE_MAX];
-  SCacheObj     *cache;
+  SCacheObj *    cache;
 } SShowMgmt;
 
 typedef struct {
@@ -100,14 +102,14 @@ typedef struct SMnode {
   bool           stopped;
   bool           restored;
   bool           deploy;
-  char          *path;
+  char *         path;
   int64_t        checkTime;
-  SSdb          *pSdb;
-  SArray        *pSteps;
-  SQHandle      *pQuery;
-  SHashObj      *infosMeta;
-  SHashObj      *perfsMeta;
-  SWal          *pWal;
+  SSdb *         pSdb;
+  SArray *       pSteps;
+  SQHandle *     pQuery;
+  SHashObj *     infosMeta;
+  SHashObj *     perfsMeta;
+  SWal *         pWal;
   SShowMgmt      showMgmt;
   SProfileMgmt   profileMgmt;
   STelemMgmt     telemMgmt;

source/dnode/mnode/impl/inc/mndTrans.h

@@ -68,7 +68,7 @@ int32_t mndTransAppendRedoAction(STrans *pTrans, STransAction *pAction);
 int32_t mndTransAppendUndoAction(STrans *pTrans, STransAction *pAction);
 void    mndTransSetRpcRsp(STrans *pTrans, void *pCont, int32_t contLen);
 void    mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *param, int32_t paramLen);
-void    mndTransSetDbName(STrans *pTrans, const char *dbname);
+void    mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2);
 void    mndTransSetSerial(STrans *pTrans);
 
 int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans);

source/dnode/mnode/impl/src/mndDb.c

@@ -477,7 +477,7 @@ static int32_t mndCreateDb(SMnode *pMnode, SRpcMsg *pReq, SCreateDbReq *pCreate,
 
   mDebug("trans:%d, used to create db:%s", pTrans->id, pCreate->db);
 
-  mndTransSetDbName(pTrans, dbObj.name);
+  mndTransSetDbName(pTrans, dbObj.name, NULL);
   if (mndSetCreateDbRedoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
   if (mndSetCreateDbUndoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
   if (mndSetCreateDbCommitLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
@@ -668,7 +668,7 @@ static int32_t mndAlterDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pOld, SDbObj *p
   mDebug("trans:%d, used to alter db:%s", pTrans->id, pOld->name);
 
   int32_t code = -1;
-  mndTransSetDbName(pTrans, pOld->name);
+  mndTransSetDbName(pTrans, pOld->name, NULL);
   if (mndSetAlterDbRedoLogs(pMnode, pTrans, pOld, pNew) != 0) goto _OVER;
   if (mndSetAlterDbCommitLogs(pMnode, pTrans, pOld, pNew) != 0) goto _OVER;
   if (mndSetAlterDbRedoActions(pMnode, pTrans, pOld, pNew) != 0) goto _OVER;
@@ -921,7 +921,7 @@ static int32_t mndDropDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb) {
   if (pTrans == NULL) goto _OVER;
 
   mDebug("trans:%d, used to drop db:%s", pTrans->id, pDb->name);
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
 
   if (mndSetDropDbRedoLogs(pMnode, pTrans, pDb) != 0) goto _OVER;
   if (mndSetDropDbCommitLogs(pMnode, pTrans, pDb) != 0) goto _OVER;

source/dnode/mnode/impl/src/mndDnode.c

@@ -761,11 +761,12 @@ static int32_t mndRetrieveDnodes(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
     pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
     colDataAppend(pColInfo, numOfRows, (const char *)&pDnode->createdTime, false);
 
-    char b[tListLen(offlineReason) + VARSTR_HEADER_SIZE] = {0};
+    char *b = taosMemoryCalloc(VARSTR_HEADER_SIZE + strlen(offlineReason[pDnode->offlineReason]) + 1, 1);
     STR_TO_VARSTR(b, online ? "" : offlineReason[pDnode->offlineReason]);
 
     pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
     colDataAppend(pColInfo, numOfRows, b, false);
+    taosMemoryFreeClear(b);
 
     numOfRows++;
     sdbRelease(pSdb, pDnode);

source/dnode/mnode/impl/src/mndMain.c

@@ -58,21 +58,21 @@ static void *mndBuildTimerMsg(int32_t *pContLen) {
 
 static void mndPullupTrans(SMnode *pMnode) {
   int32_t contLen = 0;
-  void   *pReq = mndBuildTimerMsg(&contLen);
+  void *  pReq = mndBuildTimerMsg(&contLen);
   SRpcMsg rpcMsg = {.msgType = TDMT_MND_TRANS_TIMER, .pCont = pReq, .contLen = contLen};
   tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
 }
 
 static void mndCalMqRebalance(SMnode *pMnode) {
   int32_t contLen = 0;
-  void   *pReq = mndBuildTimerMsg(&contLen);
+  void *  pReq = mndBuildTimerMsg(&contLen);
   SRpcMsg rpcMsg = {.msgType = TDMT_MND_MQ_TIMER, .pCont = pReq, .contLen = contLen};
   tmsgPutToQueue(&pMnode->msgCb, READ_QUEUE, &rpcMsg);
 }
 
 static void mndPullupTelem(SMnode *pMnode) {
   int32_t contLen = 0;
-  void   *pReq = mndBuildTimerMsg(&contLen);
+  void *  pReq = mndBuildTimerMsg(&contLen);
   SRpcMsg rpcMsg = {.msgType = TDMT_MND_TELEM_TIMER, .pCont = pReq, .contLen = contLen};
   tmsgPutToQueue(&pMnode->msgCb, READ_QUEUE, &rpcMsg);
 }
@@ -378,7 +378,7 @@ void mndStop(SMnode *pMnode) {
 }
 
 int32_t mndProcessSyncMsg(SRpcMsg *pMsg) {
-  SMnode    *pMnode = pMsg->info.node;
+  SMnode *   pMnode = pMsg->info.node;
   SSyncMgmt *pMgmt = &pMnode->syncMgmt;
   int32_t    code = 0;
 
@@ -510,8 +510,7 @@ static int32_t mndCheckMnodeState(SRpcMsg *pMsg) {
 
   if (IsReq(pMsg) && pMsg->msgType != TDMT_MND_MQ_TIMER && pMsg->msgType != TDMT_MND_TELEM_TIMER &&
       pMsg->msgType != TDMT_MND_TRANS_TIMER) {
-    mError("msg:%p, failed to check mnode state since %s, app:%p type:%s", pMsg, terrstr(), pMsg->info.ahandle,
-           TMSG_INFO(pMsg->msgType));
+    mError("msg:%p, failed to check mnode state since %s, type:%s", pMsg, terrstr(), TMSG_INFO(pMsg->msgType));
 
     SEpSet epSet = {0};
     mndGetMnodeEpSet(pMsg->info.node, &epSet);
@@ -534,13 +533,14 @@ static int32_t mndCheckMsgContent(SRpcMsg *pMsg) {
   if (!IsReq(pMsg)) return 0;
   if (pMsg->contLen != 0 && pMsg->pCont != NULL) return 0;
 
-  mError("msg:%p, failed to check msg content, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
+  mError("msg:%p, failed to check msg, cont:%p contLen:%d, app:%p type:%s", pMsg, pMsg->pCont, pMsg->contLen,
+         pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
   terrno = TSDB_CODE_INVALID_MSG_LEN;
   return -1;
 }
 
 int32_t mndProcessRpcMsg(SRpcMsg *pMsg) {
-  SMnode  *pMnode = pMsg->info.node;
+  SMnode * pMnode = pMsg->info.node;
   MndMsgFp fp = pMnode->msgFp[TMSG_INDEX(pMsg->msgType)];
   if (fp == NULL) {
     mError("msg:%p, failed to get msg handle, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
@@ -551,14 +551,15 @@ int32_t mndProcessRpcMsg(SRpcMsg *pMsg) {
   if (mndCheckMsgContent(pMsg) != 0) return -1;
   if (mndCheckMnodeState(pMsg) != 0) return -1;
 
-  mTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
+  STraceId *trace = &pMsg->info.traceId;
+  mGTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
   int32_t code = (*fp)(pMsg);
   mndReleaseRpcRef(pMnode);
 
   if (code == TSDB_CODE_ACTION_IN_PROGRESS) {
     mTrace("msg:%p, won't response immediately since in progress", pMsg);
   } else if (code == 0) {
-    mTrace("msg:%p, successfully processed and response", pMsg);
+    mTrace("msg:%p, successfully processed", pMsg);
   } else {
     mError("msg:%p, failed to process since %s, app:%p type:%s", pMsg, terrstr(), pMsg->info.ahandle,
            TMSG_INFO(pMsg->msgType));
@@ -592,7 +593,7 @@ int32_t mndGetMonitorInfo(SMnode *pMnode, SMonClusterInfo *pClusterInfo, SMonVgr
                           SMonGrantInfo *pGrantInfo) {
   if (mndAcquireRpcRef(pMnode) != 0) return -1;
 
-  SSdb   *pSdb = pMnode->pSdb;
+  SSdb *  pSdb = pMnode->pSdb;
   int64_t ms = taosGetTimestampMs();
 
   pClusterInfo->dnodes = taosArrayInit(sdbGetSize(pSdb, SDB_DNODE), sizeof(SMonDnodeDesc));
@@ -668,7 +669,7 @@ int32_t mndGetMonitorInfo(SMnode *pMnode, SMonClusterInfo *pClusterInfo, SMonVgr
     pGrantInfo->timeseries_used += pVgroup->numOfTimeSeries;
     tstrncpy(desc.status, "unsynced", sizeof(desc.status));
     for (int32_t i = 0; i < pVgroup->replica; ++i) {
-      SVnodeGid     *pVgid = &pVgroup->vnodeGid[i];
+      SVnodeGid *    pVgid = &pVgroup->vnodeGid[i];
       SMonVnodeDesc *pVnDesc = &desc.vnodes[i];
       pVnDesc->dnode_id = pVgid->dnodeId;
       tstrncpy(pVnDesc->vnode_role, syncStr(pVgid->role), sizeof(pVnDesc->vnode_role));

source/dnode/mnode/impl/src/mndScheduler.c

@@ -105,7 +105,7 @@ int32_t mndPersistTaskDeployReq(STrans* pTrans, SStreamTask* pTask, const SEpSet
   int32_t size = encoder.pos;
   int32_t tlen = sizeof(SMsgHead) + size;
   tEncoderClear(&encoder);
-  void* buf = taosMemoryMalloc(tlen);
+  void* buf = taosMemoryCalloc(1, tlen);
   if (buf == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
     return -1;
@@ -157,6 +157,7 @@ int32_t mndAddDispatcherToInnerTask(SMnode* pMnode, STrans* pTrans, SStreamObj*
     }
     sdbRelease(pMnode->pSdb, pDb);
 
+    memcpy(pTask->shuffleDispatcher.stbFullName, pStream->targetSTbName, TSDB_TABLE_FNAME_LEN);
     SArray* pVgs = pTask->shuffleDispatcher.dbInfo.pVgroupInfos;
     int32_t sz = taosArrayGetSize(pVgs);
     SArray* sinkLv = taosArrayGetP(pStream->tasks, 0);
@@ -166,6 +167,7 @@ int32_t mndAddDispatcherToInnerTask(SMnode* pMnode, STrans* pTrans, SStreamObj*
       for (int32_t j = 0; j < sinkLvSize; j++) {
         SStreamTask* pLastLevelTask = taosArrayGetP(sinkLv, j);
         if (pLastLevelTask->nodeId == pVgInfo->vgId) {
+          ASSERT(pVgInfo->vgId > 0);
           pVgInfo->taskId = pLastLevelTask->taskId;
           ASSERT(pVgInfo->taskId != 0);
           break;

source/dnode/mnode/impl/src/mndSma.c

@@ -530,7 +530,10 @@ static int32_t mndCreateSma(SMnode *pMnode, SRpcMsg *pReq, SMCreateSmaReq *pCrea
   smaObj.dbUid = pStb->dbUid;
   smaObj.intervalUnit = pCreate->intervalUnit;
   smaObj.slidingUnit = pCreate->slidingUnit;
+#if 0
   smaObj.timezone = pCreate->timezone;
+#endif
+  smaObj.timezone = tsTimezone; // use timezone of server
   smaObj.interval = pCreate->interval;
   smaObj.offset = pCreate->offset;
   smaObj.sliding = pCreate->sliding;
@@ -609,7 +612,7 @@ static int32_t mndCreateSma(SMnode *pMnode, SRpcMsg *pReq, SMCreateSmaReq *pCrea
   int32_t code = -1;
   STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB, pReq);
   if (pTrans == NULL) goto _OVER;
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
   mndTransSetSerial(pTrans);
   mDebug("trans:%d, used to create sma:%s", pTrans->id, pCreate->name);
 
@@ -852,7 +855,7 @@ static int32_t mndDropSma(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SSmaObj *p
   if (pTrans == NULL) goto _OVER;
 
   mDebug("trans:%d, used to drop sma:%s", pTrans->id, pSma->name);
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
 
   if (mndSetDropSmaRedoLogs(pMnode, pTrans, pSma) != 0) goto _OVER;
   if (mndSetDropSmaVgroupRedoLogs(pMnode, pTrans, pVgroup) != 0) goto _OVER;

source/dnode/mnode/impl/src/mndStb.c

@@ -669,8 +669,8 @@ int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreat
   pDst->tagVer = 1;
   pDst->colVer = 1;
   pDst->nextColId = 1;
-  pDst->xFilesFactor = pCreate->xFilesFactor;
-  pDst->delay = pCreate->delay;
+  // pDst->xFilesFactor = pCreate->xFilesFactor;
+  // pDst->delay = pCreate->delay;
   pDst->ttl = pCreate->ttl;
   pDst->numOfColumns = pCreate->numOfColumns;
   pDst->numOfTags = pCreate->numOfTags;
@@ -758,7 +758,7 @@ _OVER:
 }
 
 int32_t mndAddStbToTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb) {
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
   if (mndSetCreateStbRedoLogs(pMnode, pTrans, pDb, pStb) != 0) return -1;
   if (mndSetCreateStbUndoLogs(pMnode, pTrans, pDb, pStb) != 0) return -1;
   if (mndSetCreateStbCommitLogs(pMnode, pTrans, pDb, pStb) != 0) return -1;
@@ -1396,7 +1396,7 @@ static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *p
   if (pTrans == NULL) goto _OVER;
 
   mDebug("trans:%d, used to alter stb:%s", pTrans->id, pAlter->name);
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
 
   if (needRsp) {
     void   *pCont = NULL;
@@ -1537,7 +1537,7 @@ static int32_t mndDropStb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *p
   if (pTrans == NULL) goto _OVER;
 
   mDebug("trans:%d, used to drop stb:%s", pTrans->id, pStb->name);
-  mndTransSetDbName(pTrans, pDb->name);
+  mndTransSetDbName(pTrans, pDb->name, NULL);
 
   if (mndSetDropStbRedoLogs(pMnode, pTrans, pStb) != 0) goto _OVER;
   if (mndSetDropStbCommitLogs(pMnode, pTrans, pStb) != 0) goto _OVER;

source/dnode/mnode/impl/src/mndStream.c

@@ -613,9 +613,9 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
     goto _OVER;
   }
 
-  mndTransSetDbName(pTrans, createStreamReq.sourceDB);
+  mndTransSetDbName(pTrans, createStreamReq.sourceDB, NULL);
   // TODO
-  /*mndTransSetDbName(pTrans, streamObj.targetDb);*/
+  /*mndTransSetDbName(pTrans, streamObj.targetDb, NULL);*/
   mDebug("trans:%d, used to create stream:%s", pTrans->id, createStreamReq.name);
 
   // build stream obj from request

source/dnode/mnode/impl/src/mndSubscribe.c

@@ -403,7 +403,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
 
 static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOutputObj *pOutput) {
   STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pMsg);
-  mndTransSetDbName(pTrans, pOutput->pSub->dbName);
+  mndTransSetDbName(pTrans, pOutput->pSub->dbName, NULL);
   if (pTrans == NULL) return -1;
 
   // make txn:

source/dnode/mnode/impl/src/mndSync.c

@@ -46,13 +46,14 @@ void mndSyncCommitMsg(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbM
 
   int32_t transId = sdbGetIdFromRaw(pMnode->pSdb, pRaw);
   pMgmt->errCode = cbMeta.code;
-  mDebug("trans:%d, is proposed, saved:%d code:0x%x, index:%" PRId64 " term:%" PRId64 " role:%s raw:%p", transId,
-         pMgmt->transId, cbMeta.code, cbMeta.index, cbMeta.term, syncStr(cbMeta.state), pRaw);
+  mDebug("trans:%d, is proposed, saved:%d code:0x%x, apply index:%" PRId64 " term:%" PRIu64 " config:%" PRId64
+         " role:%s raw:%p",
+         transId, pMgmt->transId, cbMeta.code, cbMeta.index, cbMeta.term, cbMeta.lastConfigIndex, syncStr(cbMeta.state),
+         pRaw);
 
   if (pMgmt->errCode == 0) {
     sdbWriteWithoutFree(pMnode->pSdb, pRaw);
-    sdbSetApplyIndex(pMnode->pSdb, cbMeta.index);
-    sdbSetApplyTerm(pMnode->pSdb, cbMeta.term);
+    sdbSetApplyInfo(pMnode->pSdb, cbMeta.index, cbMeta.term, cbMeta.lastConfigIndex);
   }
 
   if (pMgmt->transId == transId) {
@@ -68,36 +69,19 @@ void mndSyncCommitMsg(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbM
       mndReleaseTrans(pMnode, pTrans);
     }
 
-    if (cbMeta.index - sdbGetApplyIndex(pMnode->pSdb) > 100) {
-      SSnapshotMeta sMeta = {0};
-      // if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) {
-      if (syncGetSnapshotMetaByIndex(pMnode->syncMgmt.sync, cbMeta.index, &sMeta) == 0) {
-        sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex);
-      }
-      sdbWriteFile(pMnode->pSdb);
-    }
+    sdbWriteFile(pMnode->pSdb, SDB_WRITE_DELTA);
   }
 }
 
 int32_t mndSyncGetSnapshot(struct SSyncFSM *pFsm, SSnapshot *pSnapshot) {
   SMnode *pMnode = pFsm->data;
-  pSnapshot->lastApplyIndex = sdbGetCommitIndex(pMnode->pSdb);
-  pSnapshot->lastApplyTerm = sdbGetCommitTerm(pMnode->pSdb);
-  pSnapshot->lastConfigIndex = sdbGetCurConfig(pMnode->pSdb);
+  sdbGetCommitInfo(pMnode->pSdb, &pSnapshot->lastApplyIndex, &pSnapshot->lastApplyTerm, &pSnapshot->lastConfigIndex);
   return 0;
 }
 
 void mndRestoreFinish(struct SSyncFSM *pFsm) {
   SMnode *pMnode = pFsm->data;
 
-  SSnapshotMeta sMeta = {0};
-  // if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) {
-
-  SyncIndex snapshotIndex = sdbGetApplyIndex(pMnode->pSdb);
-  if (syncGetSnapshotMetaByIndex(pMnode->syncMgmt.sync, snapshotIndex, &sMeta) == 0) {
-    sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex);
-  }
-
   if (!pMnode->deploy) {
     mInfo("mnode sync restore finished, and will handle outstanding transactions");
     mndTransPullup(pMnode);

source/dnode/mnode/impl/src/mndTopic.c

@@ -566,7 +566,7 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) {
 #endif
 
   STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pReq);
-  mndTransSetDbName(pTrans, pTopic->db);
+  mndTransSetDbName(pTrans, pTopic->db, NULL);
   if (pTrans == NULL) {
     mError("topic:%s, failed to drop since %s", pTopic->name, terrstr());
     return -1;

source/dnode/mnode/impl/src/mndTrans.c

@@ -22,8 +22,8 @@
 #include "mndSync.h"
 #include "mndUser.h"
 
-#define TRANS_VER_NUMBER 1
-#define TRANS_ARRAY_SIZE 8
+#define TRANS_VER_NUMBER   1
+#define TRANS_ARRAY_SIZE   8
 #define TRANS_RESERVE_SIZE 64
 
 static SSdbRaw *mndTransActionEncode(STrans *pTrans);
@@ -52,7 +52,7 @@ static bool    mndTransPerformCommitActionStage(SMnode *pMnode, STrans *pTrans);
 static bool    mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans);
 static bool    mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans);
 static bool    mndTransPerfromFinishedStage(SMnode *pMnode, STrans *pTrans);
-static bool    mndCantExecuteTransAction(SMnode *pMnode) { return !pMnode->deploy && !mndIsMaster(pMnode); }
+static bool    mndCannotExecuteTransAction(SMnode *pMnode) { return !pMnode->deploy && !mndIsMaster(pMnode); }
 
 static void    mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans);
 static int32_t mndProcessTransReq(SRpcMsg *pReq);
@@ -122,7 +122,8 @@ static SSdbRaw *mndTransActionEncode(STrans *pTrans) {
   SDB_SET_INT8(pRaw, dataPos, pTrans->conflict, _OVER)
   SDB_SET_INT8(pRaw, dataPos, pTrans->exec, _OVER)
   SDB_SET_INT64(pRaw, dataPos, pTrans->createdTime, _OVER)
-  SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname, TSDB_DB_FNAME_LEN, _OVER)
+  SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
+  SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
   SDB_SET_INT32(pRaw, dataPos, pTrans->redoActionPos, _OVER)
 
   int32_t redoActionNum = taosArrayGetSize(pTrans->redoActions);
@@ -270,7 +271,8 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
   pTrans->conflict = conflict;
   pTrans->exec = exec;
   SDB_GET_INT64(pRaw, dataPos, &pTrans->createdTime, _OVER)
-  SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname, TSDB_DB_FNAME_LEN, _OVER)
+  SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
+  SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
   SDB_GET_INT32(pRaw, dataPos, &pTrans->redoActionPos, _OVER)
   SDB_GET_INT32(pRaw, dataPos, &redoActionNum, _OVER)
   SDB_GET_INT32(pRaw, dataPos, &undoActionNum, _OVER)
@@ -521,9 +523,10 @@ static int32_t mndTransActionUpdate(SSdb *pSdb, STrans *pOld, STrans *pNew) {
   }
 
   if (pOld->stage == TRN_STAGE_ROLLBACK) {
-    pOld->stage = TRN_STAGE_FINISHED;
-    mTrace("trans:%d, stage from rollback to finished since perform update action", pNew->id);
+    pOld->stage = TRN_STAGE_REDO_ACTION;
+    mTrace("trans:%d, stage from rollback to undoAction since perform update action", pNew->id);
   }
+
   return 0;
 }
 
@@ -649,7 +652,14 @@ void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *
   pTrans->paramLen = paramLen;
 }
 
-void mndTransSetDbName(STrans *pTrans, const char *dbname) { memcpy(pTrans->dbname, dbname, TSDB_DB_FNAME_LEN); }
+void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2) {
+  if (dbname1 != NULL) {
+    memcpy(pTrans->dbname1, dbname1, TSDB_DB_FNAME_LEN);
+  }
+  if (dbname2 != NULL) {
+    memcpy(pTrans->dbname2, dbname2, TSDB_DB_FNAME_LEN);
+  }
+}
 
 void mndTransSetSerial(STrans *pTrans) { pTrans->exec = TRN_EXEC_SERIAL; }
 
@@ -661,7 +671,7 @@ static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) {
   }
   sdbSetRawStatus(pRaw, SDB_STATUS_READY);
 
-  mDebug("trans:%d, sync to other mnodes", pTrans->id);
+  mDebug("trans:%d, sync to other mnodes, stage:%s", pTrans->id, mndTransStr(pTrans->stage));
   int32_t code = mndSyncPropose(pMnode, pRaw, pTrans->id);
   if (code != 0) {
     mError("trans:%d, failed to sync since %s", pTrans->id, terrstr());
@@ -674,6 +684,12 @@ static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) {
   return 0;
 }
 
+static bool mndCheckDbConflict(const char *db, STrans *pTrans) {
+  if (db[0] == 0) return false;
+  if (strcmp(db, pTrans->dbname1) == 0 || strcmp(db, pTrans->dbname2) == 0) return true;
+  return false;
+}
+
 static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) {
   STrans *pTrans = NULL;
   void   *pIter = NULL;
@@ -688,14 +704,21 @@ static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) {
     if (pNew->conflict == TRN_CONFLICT_GLOBAL) conflict = true;
     if (pNew->conflict == TRN_CONFLICT_DB) {
       if (pTrans->conflict == TRN_CONFLICT_GLOBAL) conflict = true;
-      if (pTrans->conflict == TRN_CONFLICT_DB && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true;
-      if (pTrans->conflict == TRN_CONFLICT_DB_INSIDE && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true;
+      if (pTrans->conflict == TRN_CONFLICT_DB || pTrans->conflict == TRN_CONFLICT_DB_INSIDE) {
+        if (mndCheckDbConflict(pNew->dbname1, pTrans)) conflict = true;
+        if (mndCheckDbConflict(pNew->dbname2, pTrans)) conflict = true;
+      }
     }
     if (pNew->conflict == TRN_CONFLICT_DB_INSIDE) {
       if (pTrans->conflict == TRN_CONFLICT_GLOBAL) conflict = true;
-      if (pTrans->conflict == TRN_CONFLICT_DB && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true;
+      if (pTrans->conflict == TRN_CONFLICT_DB) {
+        if (mndCheckDbConflict(pNew->dbname1, pTrans)) conflict = true;
+        if (mndCheckDbConflict(pNew->dbname2, pTrans)) conflict = true;
+      }
     }
-    mError("trans:%d, can't execute since conflict with trans:%d, db:%s", pNew->id, pTrans->id, pTrans->dbname);
+
+    mError("trans:%d, can't execute since conflict with trans:%d, db1:%s db2:%s", pNew->id, pTrans->id, pTrans->dbname1,
+           pTrans->dbname2);
     sdbRelease(pMnode->pSdb, pTrans);
   }
 
@@ -704,7 +727,7 @@ static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) {
 
 int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans) {
   if (pTrans->conflict == TRN_CONFLICT_DB || pTrans->conflict == TRN_CONFLICT_DB_INSIDE) {
-    if (strlen(pTrans->dbname) == 0) {
+    if (strlen(pTrans->dbname1) == 0 && strlen(pTrans->dbname2) == 0) {
       terrno = TSDB_CODE_MND_TRANS_CONFLICT;
       mError("trans:%d, failed to prepare conflict db not set", pTrans->id);
       return -1;
@@ -915,7 +938,7 @@ static int32_t mndTransWriteSingleLog(SMnode *pMnode, STrans *pTrans, STransActi
 
 static int32_t mndTransSendSingleMsg(SMnode *pMnode, STrans *pTrans, STransAction *pAction) {
   if (pAction->msgSent) return 0;
-  if (mndCantExecuteTransAction(pMnode)) return -1;
+  if (mndCannotExecuteTransAction(pMnode)) return -1;
 
   int64_t signature = pTrans->id;
   signature = (signature << 32);
@@ -1115,7 +1138,7 @@ static int32_t mndTransExecuteRedoActionsSerial(SMnode *pMnode, STrans *pTrans)
       pTrans->lastEpset = pAction->epSet;
     }
 
-    if (mndCantExecuteTransAction(pMnode)) break;
+    if (mndCannotExecuteTransAction(pMnode)) break;
 
     if (code == 0) {
       pTrans->code = 0;
@@ -1158,7 +1181,7 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) {
     code = mndTransExecuteRedoActions(pMnode, pTrans);
   }
 
-  if (mndCantExecuteTransAction(pMnode)) return false;
+  if (mndCannotExecuteTransAction(pMnode)) return false;
 
   if (code == 0) {
     pTrans->code = 0;
@@ -1171,8 +1194,8 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) {
   } else {
     pTrans->code = terrno;
     if (pTrans->policy == TRN_POLICY_ROLLBACK) {
-      pTrans->stage = TRN_STAGE_UNDO_ACTION;
-      mError("trans:%d, stage from redoAction to undoAction since %s", pTrans->id, terrstr());
+      pTrans->stage = TRN_STAGE_ROLLBACK;
+      mError("trans:%d, stage from redoAction to rollback since %s", pTrans->id, terrstr());
       continueExec = true;
     } else {
       pTrans->failedTimes++;
@@ -1185,7 +1208,7 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) {
 }
 
 static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans) {
-  if (mndCantExecuteTransAction(pMnode)) return false;
+  if (mndCannotExecuteTransAction(pMnode)) return false;
 
   bool    continueExec = true;
   int32_t code = mndTransCommit(pMnode, pTrans);
@@ -1197,16 +1220,9 @@ static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans) {
     continueExec = true;
   } else {
     pTrans->code = terrno;
-    if (pTrans->policy == TRN_POLICY_ROLLBACK) {
-      pTrans->stage = TRN_STAGE_UNDO_ACTION;
-      mError("trans:%d, stage from commit to undoAction since %s, failedTimes:%d", pTrans->id, terrstr(),
-             pTrans->failedTimes);
-      continueExec = true;
-    } else {
-      pTrans->failedTimes++;
-      mError("trans:%d, stage keep on commit since %s, failedTimes:%d", pTrans->id, terrstr(), pTrans->failedTimes);
-      continueExec = false;
-    }
+    pTrans->failedTimes++;
+    mError("trans:%d, stage keep on commit since %s, failedTimes:%d", pTrans->id, terrstr(), pTrans->failedTimes);
+    continueExec = false;
   }
 
   return continueExec;
@@ -1235,11 +1251,9 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans) {
   bool    continueExec = true;
   int32_t code = mndTransExecuteUndoActions(pMnode, pTrans);
 
-  if (mndCantExecuteTransAction(pMnode)) return false;
-
   if (code == 0) {
-    pTrans->stage = TRN_STAGE_ROLLBACK;
-    mDebug("trans:%d, stage from undoAction to rollback", pTrans->id);
+    pTrans->stage = TRN_STAGE_FINISHED;
+    mDebug("trans:%d, stage from undoAction to finished", pTrans->id);
     continueExec = true;
   } else if (code == TSDB_CODE_ACTION_IN_PROGRESS) {
     mDebug("trans:%d, stage keep on undoAction since %s", pTrans->id, tstrerror(code));
@@ -1254,14 +1268,14 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans) {
 }
 
 static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans) {
-  if (mndCantExecuteTransAction(pMnode)) return false;
+  if (mndCannotExecuteTransAction(pMnode)) return false;
 
   bool    continueExec = true;
   int32_t code = mndTransRollback(pMnode, pTrans);
 
   if (code == 0) {
-    pTrans->stage = TRN_STAGE_FINISHED;
-    mDebug("trans:%d, stage from rollback to finished", pTrans->id);
+    pTrans->stage = TRN_STAGE_UNDO_ACTION;
+    mDebug("trans:%d, stage from rollback to undoAction", pTrans->id);
     continueExec = true;
   } else {
     pTrans->failedTimes++;
@@ -1309,12 +1323,12 @@ void mndTransExecute(SMnode *pMnode, STrans *pTrans) {
       case TRN_STAGE_COMMIT_ACTION:
         continueExec = mndTransPerformCommitActionStage(pMnode, pTrans);
         break;
-      case TRN_STAGE_UNDO_ACTION:
-        continueExec = mndTransPerformUndoActionStage(pMnode, pTrans);
-        break;
       case TRN_STAGE_ROLLBACK:
         continueExec = mndTransPerformRollbackStage(pMnode, pTrans);
         break;
+      case TRN_STAGE_UNDO_ACTION:
+        continueExec = mndTransPerformUndoActionStage(pMnode, pTrans);
+        break;
       case TRN_STAGE_FINISHED:
         continueExec = mndTransPerfromFinishedStage(pMnode, pTrans);
         break;
@@ -1416,13 +1430,8 @@ void mndTransPullup(SMnode *pMnode) {
     mndReleaseTrans(pMnode, pTrans);
   }
 
-  SSnapshotMeta sMeta = {0};
-  // if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) {
-  SyncIndex snapshotIndex = sdbGetApplyIndex(pMnode->pSdb);
-  if (syncGetSnapshotMetaByIndex(pMnode->syncMgmt.sync, snapshotIndex, &sMeta) == 0) {
-    sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex);
-  }
-  sdbWriteFile(pMnode->pSdb);
+  // todo, set to SDB_WRITE_DELTA
+  sdbWriteFile(pMnode->pSdb, 0);
   taosArrayDestroy(pArray);
 }
 
@@ -1451,10 +1460,15 @@ static int32_t mndRetrieveTrans(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBl
     pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
     colDataAppend(pColInfo, numOfRows, (const char *)stage, false);
 
-    char dbname[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
-    STR_WITH_MAXSIZE_TO_VARSTR(dbname, mndGetDbStr(pTrans->dbname), pShow->pMeta->pSchemas[cols].bytes);
+    char dbname1[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
+    STR_WITH_MAXSIZE_TO_VARSTR(dbname1, mndGetDbStr(pTrans->dbname1), pShow->pMeta->pSchemas[cols].bytes);
+    pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
+    colDataAppend(pColInfo, numOfRows, (const char *)dbname1, false);
+
+    char dbname2[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
+    STR_WITH_MAXSIZE_TO_VARSTR(dbname2, mndGetDbStr(pTrans->dbname2), pShow->pMeta->pSchemas[cols].bytes);
     pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
-    colDataAppend(pColInfo, numOfRows, (const char *)dbname, false);
+    colDataAppend(pColInfo, numOfRows, (const char *)dbname2, false);
 
     pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
     colDataAppend(pColInfo, numOfRows, (const char *)&pTrans->failedTimes, false);

source/dnode/mnode/impl/src/mndVgroup.c

@@ -1058,7 +1058,7 @@ int32_t mndSetMoveVgroupsInfoToTrans(SMnode *pMnode, STrans *pTrans, int32_t del
 
 static int32_t mndAddIncVgroupReplicaToTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup,
                                              int32_t newDnodeId) {
-  mDebug("vgId:%d, will add 1 vnode, replica:%d, dnode:%d", pVgroup->vgId, pVgroup->replica, newDnodeId);
+  mDebug("vgId:%d, will add 1 vnode, replica:%d dnode:%d", pVgroup->vgId, pVgroup->replica, newDnodeId);
 
   SVnodeGid *pGid = &pVgroup->vnodeGid[pVgroup->replica];
   pVgroup->replica++;
@@ -1074,7 +1074,7 @@ static int32_t mndAddIncVgroupReplicaToTrans(SMnode *pMnode, STrans *pTrans, SDb
 
 static int32_t mndAddDecVgroupReplicaFromTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup,
                                                int32_t delDnodeId) {
-  mDebug("vgId:%d, will remove 1 vnode, replica:%d, dnode:%d", pVgroup->vgId, pVgroup->replica, delDnodeId);
+  mDebug("vgId:%d, will remove 1 vnode, replica:%d dnode:%d", pVgroup->vgId, pVgroup->replica, delDnodeId);
 
   SVnodeGid *pGid = NULL;
   SVnodeGid  delGid = {0};
@@ -1116,7 +1116,8 @@ static int32_t mndRedistributeVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb,
   memcpy(&newVg, pVgroup, sizeof(SVgObj));
   mInfo("vgId:%d, vgroup info before redistribute, replica:%d", newVg.vgId, newVg.replica);
   for (int32_t i = 0; i < newVg.replica; ++i) {
-    mInfo("vgId:%d, vnode:%d dnode:%d", newVg.vgId, i, newVg.vnodeGid[i].dnodeId);
+    mInfo("vgId:%d, vnode:%d dnode:%d role:%s", newVg.vgId, i, newVg.vnodeGid[i].dnodeId,
+          syncStr(newVg.vnodeGid[i].role));
   }
 
   if (pNew1 != NULL && pOld1 != NULL) {
@@ -1198,7 +1199,7 @@ static int32_t mndProcessRedistributeVgroupMsg(SRpcMsg *pReq) {
     goto _OVER;
   }
 
-  mInfo("vgId:%d, start to redistribute to dnode %d:%d:%d", req.vgId, req.dnodeId1, req.dnodeId2, req.dnodeId3);
+  mInfo("vgId:%d, start to redistribute vgroup to dnode %d:%d:%d", req.vgId, req.dnodeId1, req.dnodeId2, req.dnodeId3);
 
   if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_REDISTRIBUTE_VGROUP) != 0) goto _OVER;
 

source/dnode/mnode/impl/test/sdb/sdbTest.cpp

@@ -493,8 +493,11 @@ TEST_F(MndTestSdb, 01_Write_Str) {
   ASSERT_EQ(sdbGetSize(pSdb, SDB_USER), 2);
   ASSERT_EQ(sdbGetMaxId(pSdb, SDB_USER), -1);
   ASSERT_EQ(sdbGetTableVer(pSdb, SDB_USER), 2);
-  sdbSetApplyIndex(pSdb, -1);
-  ASSERT_EQ(sdbGetApplyIndex(pSdb), -1);
+  sdbSetApplyInfo(pSdb, -1, -1, -1);
+  // int64_t index, config;
+  // int64_t term;
+  // sdbGetCommitInfo(pSdb, &index, &term, &config);
+  // ASSERT_EQ(index, -1);
   ASSERT_EQ(mnode.insertTimes, 2);
   ASSERT_EQ(mnode.deleteTimes, 0);
 
@@ -700,11 +703,12 @@ TEST_F(MndTestSdb, 01_Write_Str) {
   }
 
   // write version
-  sdbSetApplyIndex(pSdb, 0);
-  sdbSetApplyIndex(pSdb, 1);
-  ASSERT_EQ(sdbGetApplyIndex(pSdb), 1);
-  ASSERT_EQ(sdbWriteFile(pSdb), 0);
-  ASSERT_EQ(sdbWriteFile(pSdb), 0);
+  sdbSetApplyInfo(pSdb, 0, 0, 0);
+  sdbSetApplyInfo(pSdb, 1, 0, 0);
+  // sdbGetApplyInfo(pSdb, &index, &term, &config);
+  // ASSERT_EQ(index, 1);
+  ASSERT_EQ(sdbWriteFile(pSdb, 0), 0);
+  ASSERT_EQ(sdbWriteFile(pSdb, 0), 0);
 
   sdbCleanup(pSdb);
   ASSERT_EQ(mnode.insertTimes, 7);
@@ -772,7 +776,11 @@ TEST_F(MndTestSdb, 01_Read_Str) {
   ASSERT_EQ(sdbGetSize(pSdb, SDB_USER), 2);
   ASSERT_EQ(sdbGetMaxId(pSdb, SDB_USER), -1);
   ASSERT_EQ(sdbGetTableVer(pSdb, SDB_USER), 5);
-  ASSERT_EQ(sdbGetApplyIndex(pSdb), 1);
+
+  int64_t index, config;
+  int64_t term;
+  sdbGetCommitInfo(pSdb, &index, &term, &config);
+  ASSERT_EQ(index, 1);
   ASSERT_EQ(mnode.insertTimes, 4);
   ASSERT_EQ(mnode.deleteTimes, 0);
 

source/dnode/mnode/impl/test/trans/trans2.cpp

@@ -128,7 +128,7 @@ class MndTestTrans2 : public ::testing::Test {
     mndTransSetCb(pTrans, TRANS_START_FUNC_TEST, TRANS_STOP_FUNC_TEST, param, strlen(param) + 1);
 
     if (pDb != NULL) {
-      mndTransSetDbName(pTrans, pDb->name);
+      mndTransSetDbName(pTrans, pDb->name, NULL);
     }
 
     int32_t code = mndTransPrepare(pMnode, pTrans);
@@ -201,7 +201,7 @@ class MndTestTrans2 : public ::testing::Test {
     }
 
     if (pDb != NULL) {
-      mndTransSetDbName(pTrans, pDb->name);
+      mndTransSetDbName(pTrans, pDb->name, NULL);
     }
 
     int32_t code = mndTransPrepare(pMnode, pTrans);

source/dnode/mnode/sdb/inc/sdb.h

@@ -37,6 +37,8 @@ extern "C" {
 #define mTrace(...) { if (mDebugFlag & DEBUG_TRACE) { taosPrintLog("MND ", DEBUG_TRACE, mDebugFlag, __VA_ARGS__); }}
 // clang-format on
 
+#define SDB_WRITE_DELTA 100
+
 #define SDB_GET_VAL(pData, dataPos, val, pos, func, type) \
   {                                                       \
     if (func(pRaw, dataPos, val) != 0) {                  \
@@ -169,11 +171,12 @@ typedef struct SSdb {
   SWal          *pWal;
   char          *currDir;
   char          *tmpDir;
-  int64_t        lastCommitVer;
-  int64_t        lastCommitTerm;
-  int64_t        curVer;
-  int64_t        curTerm;
-  int64_t        curConfig;
+  int64_t        commitIndex;
+  int64_t        commitTerm;
+  int64_t        commitConfig;
+  int64_t        applyIndex;
+  int64_t        applyTerm;
+  int64_t        applyConfig;
   int64_t        tableVer[SDB_MAX];
   int64_t        maxId[SDB_MAX];
   EKeyType       keyTypes[SDB_MAX];
@@ -257,7 +260,7 @@ int32_t sdbReadFile(SSdb *pSdb);
  * @param pSdb The sdb object.
  * @return int32_t 0 for success, -1 for failure.
  */
-int32_t sdbWriteFile(SSdb *pSdb);
+int32_t sdbWriteFile(SSdb *pSdb, int32_t delta);
 
 /**
  * @brief Parse and write raw data to sdb, then free the pRaw object
@@ -361,14 +364,8 @@ int64_t sdbGetTableVer(SSdb *pSdb, ESdbType type);
  * @param index The update value of the apply index.
  * @return int32_t The current index of sdb
  */
-void    sdbSetApplyIndex(SSdb *pSdb, int64_t index);
-void    sdbSetApplyTerm(SSdb *pSdb, int64_t term);
-void    sdbSetCurConfig(SSdb *pSdb, int64_t config);
-int64_t sdbGetApplyIndex(SSdb *pSdb);
-int64_t sdbGetApplyTerm(SSdb *pSdb);
-int64_t sdbGetCommitIndex(SSdb *pSdb);
-int64_t sdbGetCommitTerm(SSdb *pSdb);
-int64_t sdbGetCurConfig(SSdb *pSdb);
+void sdbSetApplyInfo(SSdb *pSdb, int64_t index, int64_t term, int64_t config);
+void sdbGetCommitInfo(SSdb *pSdb, int64_t *index, int64_t *term, int64_t *config);
 
 SSdbRaw *sdbAllocRaw(ESdbType type, int8_t sver, int32_t dataLen);
 void     sdbFreeRaw(SSdbRaw *pRaw);

source/dnode/mnode/sdb/src/sdb.c

@@ -53,11 +53,12 @@ SSdb *sdbInit(SSdbOpt *pOption) {
   }
 
   pSdb->pWal = pOption->pWal;
-  pSdb->curVer = -1;
-  pSdb->curTerm = -1;
-  pSdb->lastCommitVer = -1;
-  pSdb->lastCommitTerm = -1;
-  pSdb->curConfig = -1;
+  pSdb->applyIndex = -1;
+  pSdb->applyTerm = -1;
+  pSdb->applyConfig = -1;
+  pSdb->commitIndex = -1;
+  pSdb->commitTerm = -1;
+  pSdb->commitConfig = -1;
   pSdb->pMnode = pOption->pMnode;
   taosThreadMutexInit(&pSdb->filelock, NULL);
   mDebug("sdb init successfully");
@@ -67,7 +68,7 @@ SSdb *sdbInit(SSdbOpt *pOption) {
 void sdbCleanup(SSdb *pSdb) {
   mDebug("start to cleanup sdb");
 
-  sdbWriteFile(pSdb);
+  sdbWriteFile(pSdb, 0);
 
   if (pSdb->currDir != NULL) {
     taosMemoryFreeClear(pSdb->currDir);
@@ -159,23 +160,20 @@ static int32_t sdbCreateDir(SSdb *pSdb) {
   return 0;
 }
 
-void sdbSetApplyIndex(SSdb *pSdb, int64_t index) { pSdb->curVer = index; }
-
-void sdbSetApplyTerm(SSdb *pSdb, int64_t term) { pSdb->curTerm = term; }
-
-void sdbSetCurConfig(SSdb *pSdb, int64_t config) {
-  if (pSdb->curConfig != config) {
-    mDebug("mnode sync config set from %" PRId64 " to %" PRId64, pSdb->curConfig, config);
-    pSdb->curConfig = config;
-  }
+void sdbSetApplyInfo(SSdb *pSdb, int64_t index, int64_t term, int64_t config) {
+  mTrace("mnode apply info changed, from index:%" PRId64 " term:%" PRId64 " config:%" PRId64 ", to index:%" PRId64
+         " term:%" PRId64 " config:%" PRId64,
+         pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, index, term, config);
+  pSdb->applyIndex = index;
+  pSdb->applyTerm = term;
+  pSdb->applyConfig = config;
 }
 
-int64_t sdbGetApplyIndex(SSdb *pSdb) { return pSdb->curVer; }
-
-int64_t sdbGetApplyTerm(SSdb *pSdb) { return pSdb->curTerm; }
-
-int64_t sdbGetCommitIndex(SSdb *pSdb) { return pSdb->lastCommitVer; }
-
-int64_t sdbGetCommitTerm(SSdb *pSdb) { return pSdb->lastCommitTerm; }
-
-int64_t sdbGetCurConfig(SSdb *pSdb) { return pSdb->curConfig; }
+void sdbGetCommitInfo(SSdb *pSdb, int64_t *index, int64_t *term, int64_t *config) {
+  *index = pSdb->commitIndex;
+  *term = pSdb->commitTerm;
+  *config = pSdb->commitConfig;
+  mTrace("mnode current info, apply index:%" PRId64 " term:%" PRId64 " config:%" PRId64 ", commit index:%" PRId64
+         " term:%" PRId64 " config:%" PRId64,
+         pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, *index, *term, *config);
+}
\ No newline at end of file

source/dnode/mnode/sdb/src/sdbFile.c

@@ -67,10 +67,12 @@ static void sdbResetData(SSdb *pSdb) {
     mDebug("sdb:%s is reset", sdbTableName(i));
   }
 
-  pSdb->curVer = -1;
-  pSdb->curTerm = -1;
-  pSdb->lastCommitVer = -1;
-  pSdb->lastCommitTerm = -1;
+  pSdb->applyIndex = -1;
+  pSdb->applyTerm = -1;
+  pSdb->applyConfig = -1;
+  pSdb->commitIndex = -1;
+  pSdb->commitTerm = -1;
+  pSdb->commitConfig = -1;
   mDebug("sdb reset successfully");
 }
 
@@ -90,7 +92,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) {
     return -1;
   }
 
-  ret = taosReadFile(pFile, &pSdb->curVer, sizeof(int64_t));
+  ret = taosReadFile(pFile, &pSdb->applyIndex, sizeof(int64_t));
   if (ret < 0) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
@@ -100,7 +102,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) {
     return -1;
   }
 
-  ret = taosReadFile(pFile, &pSdb->curTerm, sizeof(int64_t));
+  ret = taosReadFile(pFile, &pSdb->applyTerm, sizeof(int64_t));
   if (ret < 0) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
@@ -110,7 +112,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) {
     return -1;
   }
 
-  ret = taosReadFile(pFile, &pSdb->curConfig, sizeof(int64_t));
+  ret = taosReadFile(pFile, &pSdb->applyConfig, sizeof(int64_t));
   if (ret < 0) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
@@ -173,17 +175,17 @@ static int32_t sdbWriteFileHead(SSdb *pSdb, TdFilePtr pFile) {
     return -1;
   }
 
-  if (taosWriteFile(pFile, &pSdb->curVer, sizeof(int64_t)) != sizeof(int64_t)) {
+  if (taosWriteFile(pFile, &pSdb->applyIndex, sizeof(int64_t)) != sizeof(int64_t)) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
   }
 
-  if (taosWriteFile(pFile, &pSdb->curTerm, sizeof(int64_t)) != sizeof(int64_t)) {
+  if (taosWriteFile(pFile, &pSdb->applyTerm, sizeof(int64_t)) != sizeof(int64_t)) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
   }
 
-  if (taosWriteFile(pFile, &pSdb->curConfig, sizeof(int64_t)) != sizeof(int64_t)) {
+  if (taosWriteFile(pFile, &pSdb->applyConfig, sizeof(int64_t)) != sizeof(int64_t)) {
     terrno = TAOS_SYSTEM_ERROR(errno);
     return -1;
   }
@@ -300,11 +302,12 @@ static int32_t sdbReadFileImp(SSdb *pSdb) {
   }
 
   code = 0;
-  pSdb->lastCommitVer = pSdb->curVer;
-  pSdb->lastCommitTerm = pSdb->curTerm;
+  pSdb->commitIndex = pSdb->applyIndex;
+  pSdb->commitTerm = pSdb->applyTerm;
+  pSdb->commitConfig = pSdb->applyConfig;
   memcpy(pSdb->tableVer, tableVer, sizeof(tableVer));
-  mDebug("read sdb file:%s successfully, index:%" PRId64 " term:%" PRId64 " config:%" PRId64, file, pSdb->lastCommitVer,
-         pSdb->lastCommitTerm, pSdb->curConfig);
+  mDebug("read sdb file:%s successfully, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64, file,
+         pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig);
 
 _OVER:
   taosCloseFile(&pFile);
@@ -336,9 +339,10 @@ static int32_t sdbWriteFileImp(SSdb *pSdb) {
   char curfile[PATH_MAX] = {0};
   snprintf(curfile, sizeof(curfile), "%s%ssdb.data", pSdb->currDir, TD_DIRSEP);
 
-  mDebug("start to write sdb file, current ver:%" PRId64 " term:%" PRId64 ", commit ver:%" PRId64 " term:%" PRId64
-         " file:%s",
-         pSdb->curVer, pSdb->curTerm, pSdb->lastCommitVer, pSdb->lastCommitTerm, curfile);
+  mDebug("start to write sdb file, apply index:%" PRId64 " term:%" PRId64 " config:%" PRId64 ", commit index:%" PRId64
+         " term:%" PRId64 " config:%" PRId64 ", file:%s",
+         pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig,
+         curfile);
 
   TdFilePtr pFile = taosOpenFile(tmpfile, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_TRUNC);
   if (pFile == NULL) {
@@ -430,25 +434,30 @@ static int32_t sdbWriteFileImp(SSdb *pSdb) {
   if (code != 0) {
     mError("failed to write sdb file:%s since %s", curfile, tstrerror(code));
   } else {
-    pSdb->lastCommitVer = pSdb->curVer;
-    pSdb->lastCommitTerm = pSdb->curTerm;
-    mDebug("write sdb file successfully, index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s",
-           pSdb->lastCommitVer, pSdb->lastCommitTerm, pSdb->curConfig, curfile);
+    pSdb->commitIndex = pSdb->applyIndex;
+    pSdb->commitTerm = pSdb->applyTerm;
+    pSdb->commitConfig = pSdb->applyConfig;
+    mDebug("write sdb file successfully, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s",
+           pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig, curfile);
   }
 
   terrno = code;
   return code;
 }
 
-int32_t sdbWriteFile(SSdb *pSdb) {
+int32_t sdbWriteFile(SSdb *pSdb, int32_t delta) {
   int32_t code = 0;
-  if (pSdb->curVer == pSdb->lastCommitVer) {
+  if (pSdb->applyIndex == pSdb->commitIndex) {
+    return 0;
+  }
+
+  if (pSdb->applyIndex - pSdb->commitIndex < delta) {
     return 0;
   }
 
   taosThreadMutexLock(&pSdb->filelock);
   if (pSdb->pWal != NULL) {
-    code = walBeginSnapshot(pSdb->pWal, pSdb->curVer);
+    code = walBeginSnapshot(pSdb->pWal, pSdb->applyIndex);
   }
   if (code == 0) {
     code = sdbWriteFileImp(pSdb);
@@ -470,7 +479,7 @@ int32_t sdbDeploy(SSdb *pSdb) {
     return -1;
   }
 
-  if (sdbWriteFile(pSdb) != 0) {
+  if (sdbWriteFile(pSdb, 0) != 0) {
     return -1;
   }
 
@@ -522,9 +531,9 @@ int32_t sdbStartRead(SSdb *pSdb, SSdbIter **ppIter) {
   snprintf(datafile, sizeof(datafile), "%s%ssdb.data", pSdb->currDir, TD_DIRSEP);
 
   taosThreadMutexLock(&pSdb->filelock);
-  int64_t commitIndex = pSdb->lastCommitVer;
-  int64_t commitTerm = pSdb->lastCommitTerm;
-  int64_t curConfig = pSdb->curConfig;
+  int64_t commitIndex = pSdb->commitIndex;
+  int64_t commitTerm = pSdb->commitTerm;
+  int64_t commitConfig = pSdb->commitConfig;
   if (taosCopyFile(datafile, pIter->name) < 0) {
     taosThreadMutexUnlock(&pSdb->filelock);
     terrno = TAOS_SYSTEM_ERROR(errno);
@@ -543,8 +552,8 @@ int32_t sdbStartRead(SSdb *pSdb, SSdbIter **ppIter) {
   }
 
   *ppIter = pIter;
-  mInfo("sdbiter:%p, is created to read snapshot, index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s", pIter,
-        commitIndex, commitTerm, curConfig, pIter->name);
+  mInfo("sdbiter:%p, is created to read snapshot, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s",
+        pIter, commitIndex, commitTerm, commitConfig, pIter->name);
   return 0;
 }
 

source/dnode/vnode/src/inc/vnd.h

@@ -18,6 +18,7 @@
 
 #include "sync.h"
 #include "syncTools.h"
+#include "ttrace.h"
 #include "vnodeInt.h"
 
 #ifdef __cplusplus
@@ -31,6 +32,7 @@ extern "C" {
 #define vInfo(...)  do { if (vDebugFlag & DEBUG_INFO)  { taosPrintLog("VND ", DEBUG_INFO, 255, __VA_ARGS__); }}            while(0)
 #define vDebug(...) do { if (vDebugFlag & DEBUG_DEBUG) { taosPrintLog("VND ", DEBUG_DEBUG, vDebugFlag, __VA_ARGS__); }}    while(0)
 #define vTrace(...) do { if (vDebugFlag & DEBUG_TRACE) { taosPrintLog("VND ", DEBUG_TRACE, vDebugFlag, __VA_ARGS__); }}    while(0)
+#define vGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vTrace(param " GTID: %s", __VA_ARGS__, buf);}      while(0)//#define vDye(...)   do
 // clang-format on
 
 // vnodeCfg.c
@@ -89,4 +91,4 @@ void    vnodeSyncClose(SVnode* pVnode);
 }
 #endif
 
-#endif /*_TD_VND_H_*/
+#endif /*_TD_VND_H_*/
\ No newline at end of file

source/dnode/vnode/src/meta/metaQuery.c

@@ -628,7 +628,7 @@ int32_t metaFilteTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
   void *  tagData = NULL;
 
   if (param->val == NULL) {
-    metaError("vgId:%d failed to filter NULL data", TD_VID(pMeta->pVnode));
+    metaError("vgId:%d, failed to filter NULL data", TD_VID(pMeta->pVnode));
     return -1;
   } else {
     if (IS_VAR_DATA_TYPE(param->type)) {

source/dnode/vnode/src/tq/tq.c

@@ -124,17 +124,20 @@ int32_t tqProcessOffsetCommitReq(STQ* pTq, char* msg, int32_t msgLen) {
   tDecoderClear(&decoder);
 
   if (offset.type == TMQ_OFFSET__SNAPSHOT) {
-    tqDebug("receive offset commit msg to %s, offset(type:snapshot) uid: %ld, ts: %ld", offset.subKey, offset.uid,
-            offset.ts);
+    tqDebug("receive offset commit msg to %s on vg %d, offset(type:snapshot) uid: %ld, ts: %ld", offset.subKey,
+            pTq->pVnode->config.vgId, offset.uid, offset.ts);
   } else if (offset.type == TMQ_OFFSET__LOG) {
-    tqDebug("receive offset commit msg to %s, offset(type:log) version: %ld", offset.subKey, offset.version);
+    tqDebug("receive offset commit msg to %s on vg %d, offset(type:log) version: %ld", offset.subKey,
+            pTq->pVnode->config.vgId, offset.version);
   } else {
     ASSERT(0);
   }
-
-  if (tqOffsetWrite(pTq->pOffsetStore, &offset) < 0) {
-    ASSERT(0);
-    return -1;
+  STqOffset* pOffset = tqOffsetRead(pTq->pOffsetStore, offset.subKey);
+  if (pOffset == NULL || pOffset->version < offset.version) {
+    if (tqOffsetWrite(pTq->pOffsetStore, &offset) < 0) {
+      ASSERT(0);
+      return -1;
+    }
   }
 
   return 0;
@@ -149,16 +152,33 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
   int32_t     code = 0;
 
   // get offset to fetch message
-  if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__EARLIEAST) {
-    fetchOffset = walGetFirstVer(pTq->pWal);
-  } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__LATEST) {
-    fetchOffset = walGetCommittedVer(pTq->pWal);
-  } else {
+  if (pReq->currentOffset >= 0) {
     fetchOffset = pReq->currentOffset + 1;
+  } else {
+    STqOffset* pOffset = tqOffsetRead(pTq->pOffsetStore, pReq->subKey);
+    if (pOffset != NULL) {
+      ASSERT(pOffset->type == TMQ_OFFSET__LOG);
+      tqDebug("consumer %ld, restore offset of %s on vg %d, offset(type:log) version: %ld", consumerId, pReq->subKey,
+              pTq->pVnode->config.vgId, pOffset->version);
+      fetchOffset = pOffset->version + 1;
+    } else {
+      if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__EARLIEAST) {
+        fetchOffset = walGetFirstVer(pTq->pWal);
+      } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__LATEST) {
+        fetchOffset = walGetCommittedVer(pTq->pWal);
+      } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__NONE) {
+        tqError("tmq poll: no offset committed for consumer %ld in vg %d, subkey %s", consumerId,
+                pTq->pVnode->config.vgId, pReq->subKey);
+        terrno = TSDB_CODE_TQ_NO_COMMITTED_OFFSET;
+        return -1;
+      }
+      tqDebug("consumer %ld, restore offset of %s on vg %d failed, config is %ld, set to %ld", consumerId, pReq->subKey,
+              pTq->pVnode->config.vgId, pReq->currentOffset, fetchOffset);
+    }
   }
 
-  tqDebug("tmq poll: consumer %ld (epoch %d) recv poll req in vg %d, req %ld %ld", consumerId, pReq->epoch,
-          TD_VID(pTq->pVnode), pReq->currentOffset, fetchOffset);
+  tqDebug("tmq poll: consumer %ld (epoch %d) recv poll req in vg %d, req offset %ld fetch offset %ld", consumerId,
+          pReq->epoch, TD_VID(pTq->pVnode), pReq->currentOffset, fetchOffset);
 
   STqHandle* pHandle = taosHashGet(pTq->handles, pReq->subKey, strlen(pReq->subKey));
   /*ASSERT(pHandle);*/

source/dnode/vnode/src/tq/tqCommit.c

@@ -15,7 +15,4 @@
 
 #include "tq.h"
 
-int tqCommit(STQ* pTq) {
-  // do nothing
-  return 0;
-}
+int tqCommit(STQ* pTq) { return tqOffsetSnapshot(pTq->pOffsetStore); }

source/dnode/vnode/src/tq/tqOffset.c

@@ -92,6 +92,8 @@ STqOffset* tqOffsetRead(STqOffsetStore* pStore, const char* subscribeKey) {
 }
 
 int32_t tqOffsetWrite(STqOffsetStore* pStore, const STqOffset* pOffset) {
+  ASSERT(pOffset->type == TMQ_OFFSET__LOG);
+  ASSERT(pOffset->version >= 0);
   return taosHashPut(pStore->pHash, pOffset->subKey, strlen(pOffset->subKey), pOffset, sizeof(STqOffset));
 }
 
@@ -129,7 +131,7 @@ int32_t tqOffsetSnapshot(STqOffsetStore* pStore) {
     tEncodeSTqOffset(&encoder, pOffset);
     // write file
     int64_t writeLen;
-    if ((writeLen = taosWriteFile(pFile, buf, totLen)) != bodyLen) {
+    if ((writeLen = taosWriteFile(pFile, buf, totLen)) != totLen) {
       ASSERT(0);
       tqError("write offset incomplete, len %d, write len %ld", bodyLen, writeLen);
       taosHashCancelIterate(pStore->pHash, pIter);

source/dnode/vnode/src/tsdb/tsdbCommit.c

@@ -111,7 +111,7 @@ int32_t tsdbBegin(STsdb *pTsdb) {
 
 int32_t tsdbCommit(STsdb *pTsdb) {
   if (!pTsdb) return 0;
-  
+
   int32_t    code = 0;
   SCommitH   commith = {0};
   SDFileSet *pSet = NULL;
@@ -495,7 +495,9 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
       break;
     }
 
-    if (pIter && pIter->pTable && (!pIdx || (pIter->pTable->suid <= pIdx->suid || pIter->pTable->uid <= pIdx->uid))) {
+    if (pIter && pIter->pTable &&
+        (!pIdx || ((pIter->pTable->suid < pIdx->suid) ||
+                   ((pIter->pTable->suid == pIdx->suid) && (pIter->pTable->uid <= pIdx->uid))))) {
       if (tsdbCommitToTable(pCommith, mIter) < 0) {
         tsdbCloseCommitFile(pCommith, true);
         // revert the file change
@@ -503,7 +505,7 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
         return -1;
       }
 
-      if (pIdx && (pIter->pTable->uid == pIdx->uid)) {
+      if (pIdx && ((pIter->pTable->uid == pIdx->uid) && (pIter->pTable->suid == pIdx->suid))) {
         ++fIter;
       }
       ++mIter;
@@ -518,6 +520,8 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
         return -1;
       }
       ++fIter;
+    } else {
+      ASSERT(0);
     }
   }
 

source/dnode/vnode/src/vnd/vnodeSvr.c

@@ -774,9 +774,6 @@ static int32_t vnodeProcessCreateTSmaReq(SVnode *pVnode, int64_t version, void *
     goto _err;
   }
 
-  // record current timezone of server side
-  req.timezoneInt = tsTimezone;
-
   if (tdProcessTSmaCreate(pVnode->pSma, version, (const char *)&req) < 0) {
     if (pRsp) pRsp->code = terrno;
     goto _err;

source/dnode/vnode/src/vnd/vnodeSync.c

@@ -61,6 +61,7 @@ static int32_t vnodeSetStandBy(SVnode *pVnode) {
     return -1;
   }
 
+  vInfo("vgId:%d, start to transfer leader", TD_VID(pVnode));
   if (syncLeaderTransfer(pVnode->sync) != 0) {
     vError("vgId:%d, failed to transfer leader since:%s", TD_VID(pVnode), terrstr());
     return -1;
@@ -83,8 +84,8 @@ static int32_t vnodeProcessAlterReplicaReq(SVnode *pVnode, SRpcMsg *pMsg) {
     terrno = TSDB_CODE_INVALID_MSG;
     return TSDB_CODE_INVALID_MSG;
   }
-
-  vInfo("vgId:%d, start to alter vnode replica to %d, handle:%p", TD_VID(pVnode), req.replica, pMsg->info.handle);
+  STraceId *trace = &pMsg->info.traceId;
+  vGTrace("vgId:%d, start to alter vnode replica to %d, handle:%p", TD_VID(pVnode), req.replica, pMsg->info.handle);
   SSyncCfg cfg = {.replicaNum = req.replica, .myIndex = req.selfIndex};
   for (int32_t r = 0; r < req.replica; ++r) {
     SNodeInfo *pNode = &cfg.nodeInfo[r];
@@ -118,14 +119,15 @@ static int32_t vnodeProcessAlterReplicaReq(SVnode *pVnode, SRpcMsg *pMsg) {
 }
 
 void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
-  SVnode  *pVnode = pInfo->ahandle;
+  SVnode * pVnode = pInfo->ahandle;
   int32_t  vgId = pVnode->config.vgId;
   int32_t  code = 0;
   SRpcMsg *pMsg = NULL;
 
   for (int32_t m = 0; m < numOfMsgs; m++) {
     if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
-    vTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle);
+    STraceId *trace = &pMsg->info.traceId;
+    vGTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle);
 
     if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) {
       code = vnodeProcessAlterReplicaReq(pVnode, pMsg);
@@ -148,10 +150,10 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
         newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps;
       }
 
-      vTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps,
-             newEpSet.inUse);
+      vGTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps,
+              newEpSet.inUse);
       for (int32_t i = 0; i < newEpSet.numOfEps; ++i) {
-        vTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port);
+        vGTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port);
       }
 
       SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info};
@@ -163,7 +165,7 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
       tmsgSendRsp(&rsp);
     }
 
-    vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code);
+    vGTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code);
     rpcFreeCont(pMsg->pCont);
     taosFreeQitem(pMsg);
   }
@@ -172,15 +174,16 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
 }
 
 void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
-  SVnode  *pVnode = pInfo->ahandle;
+  SVnode * pVnode = pInfo->ahandle;
   int32_t  vgId = pVnode->config.vgId;
   int32_t  code = 0;
   SRpcMsg *pMsg = NULL;
 
   for (int32_t i = 0; i < numOfMsgs; ++i) {
     if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
-    vTrace("vgId:%d, msg:%p get from vnode-apply queue, type:%s handle:%p", vgId, pMsg, TMSG_INFO(pMsg->msgType),
-           pMsg->info.handle);
+    STraceId *trace = &pMsg->info.traceId;
+    vGTrace("vgId:%d, msg:%p get from vnode-apply queue, type:%s handle:%p", vgId, pMsg, TMSG_INFO(pMsg->msgType),
+            pMsg->info.handle);
 
     SRpcMsg rsp = {.code = pMsg->code, .info = pMsg->info};
     if (rsp.code == 0) {
@@ -195,7 +198,7 @@ void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
       tmsgSendRsp(&rsp);
     }
 
-    vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code);
+    vGTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code);
     rpcFreeCont(pMsg->pCont);
     taosFreeQitem(pMsg);
   }
@@ -217,8 +220,9 @@ int32_t vnodeProcessSyncReq(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) {
     char *syncNodeStr = sync2SimpleStr(pVnode->sync);
     snprintf(logBuf, sizeof(logBuf), "==vnodeProcessSyncReq== msgType:%d, syncNode: %s", pMsg->msgType, syncNodeStr);
     static int64_t vndTick = 0;
+    STraceId *     trace = &pMsg->info.traceId;
     if (++vndTick % 10 == 1) {
-      vTrace("sync trace msg:%s, %s", TMSG_INFO(pMsg->msgType), syncNodeStr);
+      vGTrace("sync trace msg:%s, %s", TMSG_INFO(pMsg->msgType), syncNodeStr);
     }
     syncRpcMsgLog2(logBuf, pMsg);
     taosMemoryFree(syncNodeStr);
@@ -297,7 +301,7 @@ int32_t vnodeProcessSyncReq(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) {
     ret = -1;
   }
 
-  if (ret != 0) {
+  if (ret != 0 && terrno == 0) {
     terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
   }
   return ret;
@@ -333,8 +337,9 @@ static void vnodeSyncReconfig(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SReCon
   syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info);
   rpcMsg.info.conn.applyIndex = cbMeta.index;
 
-  vInfo("vgId:%d, alter vnode replica is confirmed, type:%s contLen:%d seq:%" PRIu64 " handle:%p", TD_VID(pVnode),
-        TMSG_INFO(pMsg->msgType), pMsg->contLen, cbMeta.seqNum, rpcMsg.info.handle);
+  STraceId *trace = (STraceId *)&pMsg->info.traceId;
+  vGTrace("vgId:%d, alter vnode replica is confirmed, type:%s contLen:%d seq:%" PRIu64 " handle:%p", TD_VID(pVnode),
+          TMSG_INFO(pMsg->msgType), pMsg->contLen, cbMeta.seqNum, rpcMsg.info.handle);
   if (rpcMsg.info.handle != NULL) {
     tmsgSendRsp(&rpcMsg);
   }
@@ -343,7 +348,7 @@ static void vnodeSyncReconfig(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SReCon
 }
 
 static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
-  SVnode   *pVnode = pFsm->data;
+  SVnode *  pVnode = pFsm->data;
   SSnapshot snapshot = {0};
   SyncIndex beginIndex = SYNC_INDEX_INVALID;
   char      logBuf[256] = {0};

source/libs/command/src/command.c

@@ -47,7 +47,8 @@ static SSDataBlock* buildDescResultDataBlock() {
   taosArrayPush(pBlock->pDataBlock, &infoData);
 
   infoData.info.type = TSDB_DATA_TYPE_INT;
-  infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes;;
+  infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes;
+
   taosArrayPush(pBlock->pDataBlock, &infoData);
 
   infoData.info.type = TSDB_DATA_TYPE_VARCHAR;
@@ -63,7 +64,7 @@ static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, S
 
   // field
   SColumnInfoData* pCol1 = taosArrayGet(pBlock->pDataBlock, 0);
-  char buf[DESCRIBE_RESULT_FIELD_LEN] = {0};
+  char             buf[DESCRIBE_RESULT_FIELD_LEN] = {0};
   for (int32_t i = 0; i < numOfRows; ++i) {
     STR_TO_VARSTR(buf, pMeta->schema[i].name);
     colDataAppend(pCol1, i, buf, false);
@@ -92,8 +93,8 @@ static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, S
 }
 
 static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) {
-  SDescribeStmt* pDesc = (SDescribeStmt*) pStmt;
-  int32_t numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta);
+  SDescribeStmt* pDesc = (SDescribeStmt*)pStmt;
+  int32_t        numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta);
 
   SSDataBlock* pBlock = buildDescResultDataBlock();
   setDescResultIntoDataBlock(pBlock, numOfRows, pDesc->pMeta);
@@ -120,9 +121,15 @@ static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) {
   return TSDB_CODE_SUCCESS;
 }
 
-static int32_t execResetQueryCache() {
-  return catalogClearCache();
-}
+static int32_t execResetQueryCache() { return catalogClearCache(); }
+
+static int32_t execShowCreateDatabase(SShowCreateDatabaseStmt* pStmt) { return TSDB_CODE_FAILED; }
+
+static int32_t execShowCreateTable(SShowCreateTableStmt* pStmt) { return TSDB_CODE_FAILED; }
+
+static int32_t execShowCreateSTable(SShowCreateTableStmt* pStmt) { return TSDB_CODE_FAILED; }
+
+static int32_t execAlterLocal(SAlterLocalStmt* pStmt) { return TSDB_CODE_FAILED; }
 
 int32_t qExecCommand(SNode* pStmt, SRetrieveTableRsp** pRsp) {
   switch (nodeType(pStmt)) {
@@ -130,6 +137,14 @@ int32_t qExecCommand(SNode* pStmt, SRetrieveTableRsp** pRsp) {
       return execDescribe(pStmt, pRsp);
     case QUERY_NODE_RESET_QUERY_CACHE_STMT:
       return execResetQueryCache();
+    case QUERY_NODE_SHOW_CREATE_DATABASE_STMT:
+      return execShowCreateDatabase((SShowCreateDatabaseStmt*)pStmt);
+    case QUERY_NODE_SHOW_CREATE_TABLE_STMT:
+      return execShowCreateTable((SShowCreateTableStmt*)pStmt);
+    case QUERY_NODE_SHOW_CREATE_STABLE_STMT:
+      return execShowCreateSTable((SShowCreateTableStmt*)pStmt);
+    case QUERY_NODE_ALTER_LOCAL_STMT:
+      return execAlterLocal((SAlterLocalStmt*)pStmt);
     default:
       break;
   }

source/libs/executor/inc/executil.h

@@ -15,7 +15,9 @@
 #ifndef TDENGINE_QUERYUTIL_H
 #define TDENGINE_QUERYUTIL_H
 
-#include <libs/function/function.h>
+#include "function.h"
+#include "nodes.h"
+#include "plannodes.h"
 #include "tbuffer.h"
 #include "tcommon.h"
 #include "tpagedbuf.h"
@@ -77,7 +79,7 @@ typedef struct SResultRowInfo {
 struct SqlFunctionCtx;
 
 size_t  getResultRowSize(struct SqlFunctionCtx* pCtx, int32_t numOfOutput);
-int32_t initResultRowInfo(SResultRowInfo* pResultRowInfo, int32_t size);
+void    initResultRowInfo(SResultRowInfo* pResultRowInfo);
 void    cleanupResultRowInfo(SResultRowInfo* pResultRowInfo);
 
 void    closeAllResultRows(SResultRowInfo* pResultRowInfo);
@@ -86,7 +88,7 @@ void    initResultRow(SResultRow *pResultRow);
 void    closeResultRow(SResultRow* pResultRow);
 bool    isResultRowClosed(SResultRow* pResultRow);
 
-struct SResultRowEntryInfo* getResultCell(const SResultRow* pRow, int32_t index, const int32_t* offset);
+struct SResultRowEntryInfo* getResultEntryInfo(const SResultRow* pRow, int32_t index, const int32_t* offset);
 
 static FORCE_INLINE SResultRow *getResultRowByPos(SDiskbasedBuf* pBuf, SResultRowPosition* pos) {
   SFilePage*  bufPage = (SFilePage*) getBufPage(pBuf, pos->pageId);
@@ -98,9 +100,27 @@ void    initGroupedResultInfo(SGroupResInfo* pGroupResInfo, SHashObj* pHashmap,
 void    initMultiResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayList);
 
 void    cleanupGroupResInfo(SGroupResInfo* pGroupResInfo);
-bool    hashRemainDataInGroupInfo(SGroupResInfo* pGroupResInfo);
+bool    hasDataInGroupInfo(SGroupResInfo* pGroupResInfo);
 
-bool    incNextGroup(SGroupResInfo* pGroupResInfo);
 int32_t getNumOfTotalRes(SGroupResInfo* pGroupResInfo);
 
+SSDataBlock* createResDataBlock(SDataBlockDescNode* pNode);
+
+int32_t getTableList(void* metaHandle, SScanPhysiNode* pScanNode, STableListInfo* pListInfo, SNode* pTagCond);
+SArray* createSortInfo(SNodeList* pNodeList);
+SArray* extractPartitionColInfo(SNodeList* pNodeList);
+SArray* extractColMatchInfo(SNodeList* pNodeList, SDataBlockDescNode* pOutputNodeList, int32_t* numOfOutputCols, int32_t type);
+
+SExprInfo* createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, int32_t* numOfExprs);
+
+SqlFunctionCtx* createSqlFunctionCtx(SExprInfo* pExprInfo, int32_t numOfOutput, int32_t** rowEntryInfoOffset);
+void    relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray* pCols);
+void    initExecTimeWindowInfo(SColumnInfoData* pColData, STimeWindow* pQueryWindow);
+
+SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode);
+SColumn   extractColumnFromColumnNode(SColumnNode* pColNode);
+
+int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode);
+void    cleanupQueryTableDataCond(SQueryTableDataCond* pCond);
+
 #endif  // TDENGINE_QUERYUTIL_H

source/libs/executor/inc/tsort.h

@@ -63,7 +63,7 @@ typedef int32_t (*_sort_merge_compar_fn_t)(const void* p1, const void* p2, void*
  * @param type
  * @return
  */
-SSortHandle* tsortCreateSortHandle(SArray* pOrderInfo, SArray* pIndexMap, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr);
+SSortHandle* tsortCreateSortHandle(SArray* pOrderInfo, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr);
 
 /**
  *

source/libs/executor/src/executorMain.c

@@ -219,4 +219,23 @@ int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t *resNum, SExplainExecInfo
   return getOperatorExplainExecInfo(pTaskInfo->pRoot, pRes, &capacity, resNum);  
 }
 
+int32_t qSerializeTaskStatus(qTaskInfo_t tinfo, char** pOutput, int32_t* len) {
+  SExecTaskInfo* pTaskInfo = (struct SExecTaskInfo*)tinfo;
+  if (pTaskInfo->pRoot == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
+
+  return encodeOperator(pTaskInfo->pRoot, pOutput, len);
+}
+
+int32_t qDeserializeTaskStatus(qTaskInfo_t tinfo, const char* pInput, int32_t len) {
+  SExecTaskInfo* pTaskInfo = (struct SExecTaskInfo*) tinfo;
+
+  if (pTaskInfo == NULL || pInput == NULL || len == 0) {
+    return TSDB_CODE_INVALID_PARA;
+  }
+
+  return decodeOperator(pTaskInfo->pRoot, pInput, len);
+}
+
 

source/libs/executor/src/groupoperator.c

@@ -26,15 +26,18 @@
 #include "ttypes.h"
 #include "executorInt.h"
 
+static void*    getCurrentDataGroupInfo(const SPartitionOperatorInfo* pInfo, SDataGroupInfo** pGroupInfo, int32_t len);
 static int32_t* setupColumnOffset(const SSDataBlock* pBlock, int32_t rowCapacity);
-static void* getCurrentDataGroupInfo(const SPartitionOperatorInfo* pInfo, SDataGroupInfo** pGroupInfo, int32_t len);
+static int32_t  setGroupResultOutputBuf(SOperatorInfo* pOperator, SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t bytes,
+                                        int32_t groupId, SDiskbasedBuf* pBuf, SAggSupporter* pAggSup);
 
 static void destroyGroupOperatorInfo(void* param, int32_t numOfOutput) {
   SGroupbyOperatorInfo* pInfo = (SGroupbyOperatorInfo*)param;
-  doDestroyBasicInfo(&pInfo->binfo, numOfOutput);
+  cleanupBasicInfo(&pInfo->binfo);
   taosMemoryFreeClear(pInfo->keyBuf);
   taosArrayDestroy(pInfo->pGroupCols);
   taosArrayDestroy(pInfo->pGroupColVals);
+  cleanupExecSupp(&pInfo->scalarSup);
 }
 
 static int32_t initGroupOptrInfo(SArray** pGroupColVals, int32_t* keyLen, char** keyBuf, const SArray* pGroupColList) {
@@ -214,7 +217,7 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
   SExecTaskInfo*        pTaskInfo = pOperator->pTaskInfo;
   SGroupbyOperatorInfo* pInfo = pOperator->info;
 
-  SqlFunctionCtx* pCtx = pInfo->binfo.pCtx;
+  SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx;
   int32_t         numOfGroupCols = taosArrayGetSize(pInfo->pGroupCols);
   //  if (type == TSDB_DATA_TYPE_FLOAT || type == TSDB_DATA_TYPE_DOUBLE) {
   // qError("QInfo:0x%"PRIx64" group by not supported on double/float columns, abort", GET_TASKID(pRuntimeEnv));
@@ -248,16 +251,16 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
     }
 
     len = buildGroupKeys(pInfo->keyBuf, pInfo->pGroupColVals);
-    int32_t ret = setGroupResultOutputBuf(&(pInfo->binfo), pOperator->numOfExprs, pInfo->keyBuf, TSDB_DATA_TYPE_VARCHAR, len, 0, pInfo->aggSup.pResultBuf, pTaskInfo, &pInfo->aggSup);
+    int32_t ret = setGroupResultOutputBuf(pOperator, &(pInfo->binfo), pOperator->exprSupp.numOfExprs, pInfo->keyBuf, len, 0, pInfo->aggSup.pResultBuf, &pInfo->aggSup);
     if (ret != TSDB_CODE_SUCCESS) {  // null data, too many state code
       longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR);
     }
 
     int32_t rowIndex = j - num;
-    doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->numOfExprs, TSDB_ORDER_ASC);
+    doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->exprSupp.numOfExprs, TSDB_ORDER_ASC);
 
     // assign the group keys or user input constant values if required
-    doAssignGroupKeys(pCtx, pOperator->numOfExprs, pBlock->info.rows, rowIndex);
+    doAssignGroupKeys(pCtx, pOperator->exprSupp.numOfExprs, pBlock->info.rows, rowIndex);
     recordNewGroupKeys(pInfo->pGroupCols, pInfo->pGroupColVals, pBlock, j);
     num = 1;
   }
@@ -265,15 +268,15 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
   if (num > 0) {
     len = buildGroupKeys(pInfo->keyBuf, pInfo->pGroupColVals);
     int32_t ret =
-        setGroupResultOutputBuf(&(pInfo->binfo), pOperator->numOfExprs, pInfo->keyBuf, TSDB_DATA_TYPE_VARCHAR, len,
-                                   0, pInfo->aggSup.pResultBuf, pTaskInfo, &pInfo->aggSup);
+        setGroupResultOutputBuf(pOperator, &(pInfo->binfo), pOperator->exprSupp.numOfExprs, pInfo->keyBuf, len,
+                                   0, pInfo->aggSup.pResultBuf, &pInfo->aggSup);
     if (ret != TSDB_CODE_SUCCESS) {
       longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR);
     }
 
     int32_t rowIndex = pBlock->info.rows - num;
-    doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->numOfExprs, TSDB_ORDER_ASC);
-    doAssignGroupKeys(pCtx, pOperator->numOfExprs, pBlock->info.rows, rowIndex);
+    doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->exprSupp.numOfExprs, TSDB_ORDER_ASC);
+    doAssignGroupKeys(pCtx, pOperator->exprSupp.numOfExprs, pBlock->info.rows, rowIndex);
   }
 }
 
@@ -291,7 +294,7 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
     doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf);
 
     size_t rows = pRes->info.rows;
-    if (rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) {
+    if (rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) {
       doSetOperatorCompleted(pOperator);
     }
 
@@ -317,11 +320,11 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
     }
 
     // the pDataBlock are always the same one, no need to call this again
-    setInputDataBlock(pOperator, pInfo->binfo.pCtx, pBlock, order, scanFlag, true);
+    setInputDataBlock(pOperator, pOperator->exprSupp.pCtx, pBlock, order, scanFlag, true);
 
     // there is an scalar expression that needs to be calculated right before apply the group aggregation.
-    if (pInfo->scalarSup.pScalarExprInfo != NULL) {
-      pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pScalarExprInfo, pBlock, pBlock, pInfo->scalarSup.pScalarFuncCtx, pInfo->scalarSup.numOfScalarExpr, NULL);
+    if (pInfo->scalarSup.pExprInfo != NULL) {
+      pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pExprInfo, pBlock, pBlock, pInfo->scalarSup.pCtx, pInfo->scalarSup.numOfExprs, NULL);
       if (pTaskInfo->code != TSDB_CODE_SUCCESS) {
         longjmp(pTaskInfo->env, pTaskInfo->code);
       }
@@ -355,7 +358,7 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
     doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf);
     doFilter(pInfo->pCondition, pRes);
 
-    bool hasRemain = hashRemainDataInGroupInfo(&pInfo->groupResInfo);
+    bool hasRemain = hasDataInGroupInfo(&pInfo->groupResInfo);
     if (!hasRemain) {
       doSetOperatorCompleted(pOperator);
       break;
@@ -384,9 +387,9 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SExprInfo* pEx
   pInfo->pGroupCols      = pGroupColList;
   pInfo->pCondition      = pCondition;
 
-  pInfo->scalarSup.pScalarExprInfo = pScalarExprInfo;
-  pInfo->scalarSup.numOfScalarExpr = numOfScalarExpr;
-  pInfo->scalarSup.pScalarFuncCtx  = createSqlFunctionCtx(pScalarExprInfo, numOfScalarExpr, &pInfo->scalarSup.rowCellInfoOffset);
+  pInfo->scalarSup.pExprInfo = pScalarExprInfo;
+  pInfo->scalarSup.numOfExprs = numOfScalarExpr;
+  pInfo->scalarSup.pCtx  = createSqlFunctionCtx(pScalarExprInfo, numOfScalarExpr, &pInfo->scalarSup.rowEntryInfoOffset);
 
   int32_t code = initGroupOptrInfo(&pInfo->pGroupColVals, &pInfo->groupKeyLen, &pInfo->keyBuf, pGroupColList);
   if (code != TSDB_CODE_SUCCESS) {
@@ -394,15 +397,16 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SExprInfo* pEx
   }
 
   initResultSizeInfo(pOperator, 4096);
-  initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResultBlock, pInfo->groupKeyLen, pTaskInfo->id.str);
-  initResultRowInfo(&pInfo->binfo.resultRowInfo, 8);
+  initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, pInfo->groupKeyLen, pTaskInfo->id.str);
+  initBasicInfo(&pInfo->binfo, pResultBlock);
+  initResultRowInfo(&pInfo->binfo.resultRowInfo);
 
   pOperator->name         = "GroupbyAggOperator";
   pOperator->blocking     = true;
   pOperator->status       = OP_NOT_OPENED;
   // pOperator->operatorType = OP_Groupby;
-  pOperator->pExpr        = pExprInfo;
-  pOperator->numOfExprs   = numOfCols;
+  pOperator->exprSupp.pExprInfo    = pExprInfo;
+  pOperator->exprSupp.numOfExprs   = numOfCols;
   pOperator->info         = pInfo;
   pOperator->pTaskInfo    = pTaskInfo;
 
@@ -439,9 +443,9 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
 
     // group id
 
-    size_t numOfCols = pOperator->numOfExprs;
+    size_t numOfCols = pOperator->exprSupp.numOfExprs;
     for(int32_t i = 0; i < numOfCols; ++i) {
-      SExprInfo* pExpr = &pOperator->pExpr[i];
+      SExprInfo* pExpr = &pOperator->exprSupp.pExprInfo[i];
       int32_t slotId = pExpr->base.pParam[0].pCol->slotId;
 
       SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, slotId);
@@ -643,8 +647,8 @@ static SSDataBlock* hashPartition(SOperatorInfo* pOperator) {
     }
 
     // there is an scalar expression that needs to be calculated right before apply the group aggregation.
-    if (pInfo->scalarSupp.pScalarExprInfo != NULL) {
-      pTaskInfo->code = projectApplyFunctions(pInfo->scalarSupp.pScalarExprInfo, pBlock, pBlock, pInfo->scalarSupp.pScalarFuncCtx, pInfo->scalarSupp.numOfScalarExpr, NULL);
+    if (pInfo->scalarSup.pExprInfo != NULL) {
+      pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pExprInfo, pBlock, pBlock, pInfo->scalarSup.pCtx, pInfo->scalarSup.numOfExprs, NULL);
       if (pTaskInfo->code != TSDB_CODE_SUCCESS) {
         longjmp(pTaskInfo->env, pTaskInfo->code);
       }
@@ -662,16 +666,20 @@ static SSDataBlock* hashPartition(SOperatorInfo* pOperator) {
 
 static void destroyPartitionOperatorInfo(void* param, int32_t numOfOutput) {
   SPartitionOperatorInfo* pInfo = (SPartitionOperatorInfo*)param;
-  doDestroyBasicInfo(&pInfo->binfo, numOfOutput);
+  cleanupBasicInfo(&pInfo->binfo);
   taosArrayDestroy(pInfo->pGroupCols);
+
   for(int i = 0; i < taosArrayGetSize(pInfo->pGroupColVals); i++){
     SGroupKeys key = *(SGroupKeys*)taosArrayGet(pInfo->pGroupColVals, i);
     taosMemoryFree(key.pData);
   }
+
   taosArrayDestroy(pInfo->pGroupColVals);
   taosMemoryFree(pInfo->keyBuf);
   taosHashCleanup(pInfo->pGroupSet);
   taosMemoryFree(pInfo->columnOffset);
+
+  cleanupExecSupp(&pInfo->scalarSup);
 }
 
 SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartitionPhysiNode* pPartNode, SExecTaskInfo* pTaskInfo) {
@@ -689,10 +697,10 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition
   pInfo->pGroupCols = extractPartitionColInfo(pPartNode->pPartitionKeys);
 
   if (pPartNode->pExprs != NULL) {
-    pInfo->scalarSupp.numOfScalarExpr = 0;
-    pInfo->scalarSupp.pScalarExprInfo = createExprInfo(pPartNode->pExprs, NULL, &pInfo->scalarSupp.numOfScalarExpr);
-    pInfo->scalarSupp.pScalarFuncCtx = createSqlFunctionCtx(
-        pInfo->scalarSupp.pScalarExprInfo, pInfo->scalarSupp.numOfScalarExpr, &pInfo->scalarSupp.rowCellInfoOffset);
+    pInfo->scalarSup.numOfExprs = 0;
+    pInfo->scalarSup.pExprInfo = createExprInfo(pPartNode->pExprs, NULL, &pInfo->scalarSup.numOfExprs);
+    pInfo->scalarSup.pCtx = createSqlFunctionCtx(
+        pInfo->scalarSup.pExprInfo, pInfo->scalarSup.numOfExprs, &pInfo->scalarSup.rowEntryInfoOffset);
   }
 
   _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
@@ -722,8 +730,8 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition
   pOperator->status       = OP_NOT_OPENED;
   pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_PARTITION;
   pInfo->binfo.pRes       = pResBlock;
-  pOperator->numOfExprs   = numOfCols;
-  pOperator->pExpr        = pExprInfo;
+  pOperator->exprSupp.numOfExprs   = numOfCols;
+  pOperator->exprSupp.pExprInfo    = pExprInfo;
   pOperator->info         = pInfo;
   pOperator->pTaskInfo    = pTaskInfo;
 
@@ -738,4 +746,18 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition
   taosMemoryFreeClear(pInfo);
   taosMemoryFreeClear(pOperator);
   return NULL;
+}
+
+int32_t setGroupResultOutputBuf(SOperatorInfo* pOperator, SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t bytes,
+                                int32_t groupId, SDiskbasedBuf* pBuf, SAggSupporter* pAggSup) {
+  SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
+  SResultRowInfo* pResultRowInfo = &binfo->resultRowInfo;
+  SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx;
+
+  SResultRow* pResultRow =
+      doSetResultOutBufByKey(pBuf, pResultRowInfo, (char*)pData, bytes, true, groupId, pTaskInfo, false, pAggSup);
+  assert(pResultRow != NULL);
+
+  setResultRowInitCtx(pResultRow, pCtx, numOfCols, pOperator->exprSupp.rowEntryInfoOffset);
+  return TSDB_CODE_SUCCESS;
 }
\ No newline at end of file

source/libs/executor/src/joinoperator.c

@@ -28,27 +28,32 @@ static SSDataBlock* doMergeJoin(struct SOperatorInfo* pOperator);
 static void         destroyMergeJoinOperator(void* param, int32_t numOfOutput);
 static void         extractTimeCondition(SJoinOperatorInfo* Info, SLogicConditionNode* pLogicConditionNode);
 
-SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SExprInfo* pExprInfo,
-                                           int32_t numOfCols, SSDataBlock* pResBlock, SNode* pOnCondition,
-                                           SExecTaskInfo* pTaskInfo) {
+SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SJoinPhysiNode* pJoinNode,
+    SExecTaskInfo* pTaskInfo) {
   SJoinOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SJoinOperatorInfo));
   SOperatorInfo*     pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
   if (pOperator == NULL || pInfo == NULL) {
     goto _error;
   }
 
+  SSDataBlock*    pResBlock = createResDataBlock(pJoinNode->node.pOutputDataBlockDesc);
+
+  int32_t numOfCols = 0;
+  SExprInfo* pExprInfo = createExprInfo(pJoinNode->pTargets, NULL, &numOfCols);
+
   initResultSizeInfo(pOperator, 4096);
 
-  pInfo->pRes = pResBlock;
-  pOperator->name = "MergeJoinOperator";
+  pInfo->pRes             = pResBlock;
+  pOperator->name         = "MergeJoinOperator";
   pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN;
-  pOperator->blocking = false;
-  pOperator->status = OP_NOT_OPENED;
-  pOperator->pExpr = pExprInfo;
-  pOperator->numOfExprs = numOfCols;
-  pOperator->info = pInfo;
-  pOperator->pTaskInfo = pTaskInfo;
-
+  pOperator->blocking     = false;
+  pOperator->status       = OP_NOT_OPENED;
+  pOperator->exprSupp.pExprInfo   = pExprInfo;
+  pOperator->exprSupp.numOfExprs  = numOfCols;
+  pOperator->info         = pInfo;
+  pOperator->pTaskInfo    = pTaskInfo;
+
+  SNode* pOnCondition = pJoinNode->pOnConditions;
   if (nodeType(pOnCondition) == QUERY_NODE_OPERATOR) {
     SOperatorNode* pNode = (SOperatorNode*)pOnCondition;
     setJoinColumnInfo(&pInfo->leftCol, (SColumnNode*)pNode->pLeft);
@@ -127,10 +132,10 @@ SSDataBlock* doMergeJoin(struct SOperatorInfo* pOperator) {
     // only the timestamp match support for ordinary table
     ASSERT(pLeftCol->info.type == TSDB_DATA_TYPE_TIMESTAMP);
     if (*(int64_t*)pLeftVal == *(int64_t*)pRightVal) {
-      for (int32_t i = 0; i < pOperator->numOfExprs; ++i) {
+      for (int32_t i = 0; i < pOperator->exprSupp.numOfExprs; ++i) {
         SColumnInfoData* pDst = taosArrayGet(pRes->pDataBlock, i);
 
-        SExprInfo* pExprInfo = &pOperator->pExpr[i];
+        SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[i];
 
         int32_t blockId = pExprInfo->base.pParam[0].pCol->dataBlockId;
         int32_t slotId = pExprInfo->base.pParam[0].pCol->slotId;

source/libs/executor/src/sortoperator.c

@@ -22,41 +22,52 @@ static int32_t      getExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain
 
 static void destroyOrderOperatorInfo(void* param, int32_t numOfOutput);
 
-SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pResBlock, SArray* pSortInfo,
-                                      SExprInfo* pExprInfo, int32_t numOfCols, SArray* pColMatchColInfo,
-                                      SExecTaskInfo* pTaskInfo) {
+SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortPhyNode, SExecTaskInfo* pTaskInfo) {
   SSortOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SSortOperatorInfo));
   SOperatorInfo*     pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
-  int32_t            rowSize = pResBlock->info.rowSize;
-
-  if (pInfo == NULL || pOperator == NULL || rowSize > 100 * 1024 * 1024) {
+  if (pInfo == NULL || pOperator == NULL/* || rowSize > 100 * 1024 * 1024*/) {
     goto _error;
   }
 
-  pOperator->pExpr = pExprInfo;
-  pOperator->numOfExprs = numOfCols;
-  pInfo->binfo.pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pInfo->binfo.rowCellInfoOffset);
+  SDataBlockDescNode* pDescNode = pSortPhyNode->node.pOutputDataBlockDesc;
+
+  int32_t      numOfCols = 0;
+  SSDataBlock* pResBlock = createResDataBlock(pDescNode);
+  SExprInfo*   pExprInfo = createExprInfo(pSortPhyNode->pExprs, NULL, &numOfCols);
+
+  int32_t numOfOutputCols = 0;
+  SArray* pColMatchColInfo =
+      extractColMatchInfo(pSortPhyNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID);
+
+  pOperator->exprSupp.pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pOperator->exprSupp.rowEntryInfoOffset);
   pInfo->binfo.pRes = pResBlock;
 
   initResultSizeInfo(pOperator, 1024);
 
-  pInfo->pSortInfo        = pSortInfo;
+  pInfo->pSortInfo        = createSortInfo(pSortPhyNode->pSortKeys);;
   pInfo->pColMatchInfo    = pColMatchColInfo;
   pOperator->name         = "SortOperator";
   pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT;
   pOperator->blocking     = true;
   pOperator->status       = OP_NOT_OPENED;
   pOperator->info         = pInfo;
+  pOperator->exprSupp.pExprInfo        = pExprInfo;
+  pOperator->exprSupp.numOfExprs   = numOfCols;
+  pOperator->pTaskInfo    = pTaskInfo;
 
   // lazy evaluation for the following parameter since the input datablock is not known till now.
-  //  pInfo->bufPageSize  = rowSize < 1024 ? 1024 * 2 : rowSize * 2;  // there are headers, so pageSize = rowSize +
-  //  header pInfo->sortBufSize  = pInfo->bufPageSize * 16;  // TODO dynamic set the available sort buffer
+  //  pInfo->bufPageSize  = rowSize < 1024 ? 1024 * 2 : rowSize * 2;
+  //  there are headers, so pageSize = rowSize + header pInfo->sortBufSize  = pInfo->bufPageSize * 16;
+  // TODO dynamic set the available sort buffer
 
-  pOperator->pTaskInfo = pTaskInfo;
   pOperator->fpSet = createOperatorFpSet(doOpenSortOperator, doSort, NULL, NULL, destroyOrderOperatorInfo, NULL, NULL,
                                          getExplainExecInfo);
 
   int32_t code = appendDownstream(pOperator, &downstream, 1);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto _error;
+  }
+
   return pOperator;
 
 _error:
@@ -134,9 +145,9 @@ SSDataBlock* loadNextDataBlock(void* param) {
 void applyScalarFunction(SSDataBlock* pBlock, void* param) {
   SOperatorInfo*     pOperator = param;
   SSortOperatorInfo* pSort = pOperator->info;
-  if (pOperator->pExpr != NULL) {
+  if (pOperator->exprSupp.pExprInfo != NULL) {
     int32_t code =
-        projectApplyFunctions(pOperator->pExpr, pBlock, pBlock, pSort->binfo.pCtx, pOperator->numOfExprs, NULL);
+        projectApplyFunctions(pOperator->exprSupp.pExprInfo, pBlock, pBlock, pOperator->exprSupp.pCtx, pOperator->exprSupp.numOfExprs, NULL);
     if (code != TSDB_CODE_SUCCESS) {
       longjmp(pOperator->pTaskInfo->env, code);
     }
@@ -154,7 +165,7 @@ int32_t doOpenSortOperator(SOperatorInfo* pOperator) {
   pInfo->startTs = taosGetTimestampUs();
 
   //  pInfo->binfo.pRes is not equalled to the input datablock.
-  pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, pInfo->pColMatchInfo, SORT_SINGLESOURCE_SORT, -1, -1,
+  pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_SINGLESOURCE_SORT, -1, -1,
                                              NULL, pTaskInfo->id.str);
 
   tsortSetFetchRawDataFp(pInfo->pSortHandle, loadNextDataBlock, applyScalarFunction, pOperator);
@@ -248,7 +259,7 @@ int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) {
 
   int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize;
 
-  pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, pInfo->pColMatchInfo, SORT_MULTISOURCE_MERGE,
+  pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_MULTISOURCE_MERGE,
                                              pInfo->bufPageSize, numOfBufPage, pInfo->pInputBlock, pTaskInfo->id.str);
 
   tsortSetFetchRawDataFp(pInfo->pSortHandle, loadNextDataBlock, NULL, NULL);
@@ -392,10 +403,8 @@ SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams,
 
   pInfo->bufPageSize   = getProperSortPageSize(rowSize);
 
-  uint32_t numOfSources = taosArrayGetSize(pSortInfo);
-  numOfSources = TMAX(4, numOfSources);
-
-  pInfo->sortBufSize    = numOfSources * pInfo->bufPageSize;
+  // one additional is reserved for merged result.
+  pInfo->sortBufSize   = pInfo->bufPageSize * (numStreams + 1);
 
   pOperator->fpSet =
       createOperatorFpSet(doOpenMultiwaySortMergeOperator, doMultiwaySortMerge, NULL, NULL,

source/libs/executor/src/tsort.c

@@ -71,7 +71,7 @@ SSDataBlock* tsortGetSortedDataBlock(const SSortHandle* pSortHandle) {
  * @param type
  * @return
  */
-SSortHandle* tsortCreateSortHandle(SArray* pSortInfo, SArray* pIndexMap, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr) {
+SSortHandle* tsortCreateSortHandle(SArray* pSortInfo, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr) {
   SSortHandle* pSortHandle = taosMemoryCalloc(1, sizeof(SSortHandle));
 
   pSortHandle->type       = type;
@@ -195,6 +195,11 @@ static int32_t doAddToBuf(SSDataBlock* pDataBlock, SSortHandle* pHandle) {
   return doAddNewExternalMemSource(pHandle->pBuf, pHandle->pOrderedSource, pBlock, &pHandle->sourceId);
 }
 
+static void setCurrentSourceIsDone(SSortSource* pSource, SSortHandle* pHandle) {
+  pSource->src.rowIndex = -1;
+  ++pHandle->numOfCompletedSources;
+}
+
 static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int32_t startIndex, int32_t endIndex, SSortHandle* pHandle) {
   cmpParam->pSources  = taosArrayGet(pSources, startIndex);
   cmpParam->numOfSources = (endIndex - startIndex + 1);
@@ -205,8 +210,10 @@ static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int
     for (int32_t i = 0; i < cmpParam->numOfSources; ++i) {
       SSortSource* pSource = cmpParam->pSources[i];
 
+      // set current source is done
       if (taosArrayGetSize(pSource->pageIdList) == 0) {
-        return TSDB_CODE_SUCCESS;
+        setCurrentSourceIsDone(pSource, pHandle);
+        continue;
       }
 
       SPageInfo* pPgInfo = *(SPageInfo**)taosArrayGet(pSource->pageIdList, pSource->pageIndex);
@@ -233,10 +240,9 @@ static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int
       SSortSource* pSource = cmpParam->pSources[i];
       pSource->src.pBlock = pHandle->fetchfp(pSource->param);
 
-      // set current source id done
+      // set current source is done
       if (pSource->src.pBlock == NULL) {
-        pSource->src.rowIndex = -1;
-        ++pHandle->numOfCompletedSources;
+        setCurrentSourceIsDone(pSource, pHandle);
       }
     }
   }

source/libs/executor/test/sortTests.cpp

@@ -209,7 +209,7 @@ TEST(testCase, inMem_sort_Test) {
   SArray* orderInfo = taosArrayInit(1, sizeof(SBlockOrderInfo));
   taosArrayPush(orderInfo, &oi);
 
-  SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_SINGLESOURCE_SORT, 1024, 5, NULL, "test_abc");
+  SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_SINGLESOURCE_SORT, 1024, 5, NULL, "test_abc");
   tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL);
 
   _info* pInfo = (_info*) taosMemoryCalloc(1, sizeof(_info));
@@ -298,7 +298,7 @@ TEST(testCase, external_mem_sort_Test) {
     SArray* orderInfo = taosArrayInit(1, sizeof(SBlockOrderInfo));
     taosArrayPush(orderInfo, &oi);
 
-    SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_SINGLESOURCE_SORT, 128, 3, NULL, "test_abc");
+    SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_SINGLESOURCE_SORT, 128, 3, NULL, "test_abc");
     tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL);
 
     SSortSource* ps = static_cast<SSortSource*>(taosMemoryCalloc(1, sizeof(SSortSource)));
@@ -365,7 +365,7 @@ TEST(testCase, ordered_merge_sort_Test) {
     taosArrayPush(pBlock->pDataBlock, &colInfo);
   }
 
-  SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_MULTISOURCE_MERGE, 1024, 5, pBlock,"test_abc");
+  SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_MULTISOURCE_MERGE, 1024, 5, pBlock,"test_abc");
   tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL);
   tsortSetComparFp(phandle, docomp);
 

source/libs/nodes/src/nodesCloneFuncs.c

@@ -427,7 +427,7 @@ static SNode* logicWindowCopy(const SWindowLogicNode* pSrc, SWindowLogicNode* pD
   COPY_SCALAR_FIELD(triggerType);
   COPY_SCALAR_FIELD(watermark);
   COPY_SCALAR_FIELD(filesFactor);
-  COPY_SCALAR_FIELD(intervalAlgo);
+  COPY_SCALAR_FIELD(windowAlgo);
   return (SNode*)pDst;
 }
 
@@ -538,6 +538,12 @@ static SNode* physiIntervalCopy(const SIntervalPhysiNode* pSrc, SIntervalPhysiNo
   return (SNode*)pDst;
 }
 
+static SNode* physiSessionCopy(const SSessionWinodwPhysiNode* pSrc, SSessionWinodwPhysiNode* pDst) {
+  COPY_BASE_OBJECT_FIELD(window, physiWindowCopy);
+  COPY_SCALAR_FIELD(gap);
+  return (SNode*)pDst;
+}
+
 static SNode* dataBlockDescCopy(const SDataBlockDescNode* pSrc, SDataBlockDescNode* pDst) {
   COPY_SCALAR_FIELD(dataBlockId);
   CLONE_NODE_LIST_FIELD(pSlots);
@@ -678,6 +684,9 @@ SNode* nodesCloneNode(const SNode* pNode) {
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL:
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_INTERVAL:
       return physiIntervalCopy((const SIntervalPhysiNode*)pNode, (SIntervalPhysiNode*)pDst);
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION:
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION:
+      return physiSessionCopy((const SSessionWinodwPhysiNode*)pNode, (SSessionWinodwPhysiNode*)pDst);
     default:
       break;
   }

source/libs/nodes/src/nodesCodeFuncs.c

@@ -244,6 +244,10 @@ const char* nodesNodeName(ENodeType type) {
       return "PhysiSessionWindow";
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION:
       return "PhysiStreamSessionWindow";
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION:
+      return "PhysiStreamSemiSessionWindow";
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION:
+      return "PhysiStreamFinalSessionWindow";
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE:
       return "PhysiStateWindow";
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE:
@@ -3996,6 +4000,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
       return physiFillNodeToJson(pObj, pJson);
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION:
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION:
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION:
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION:
       return physiSessionWindowNodeToJson(pObj, pJson);
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE:
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE:
@@ -4129,6 +4135,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
       return jsonToPhysiFillNode(pJson, pObj);
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION:
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION:
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION:
+    case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION:
       return jsonToPhysiSessionWindowNode(pJson, pObj);
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE:
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE:

source/libs/parser/src/parTokenizer.c

@@ -81,7 +81,7 @@ static SKeyword keywordTable[] = {
     {"DURATION",      TK_DURATION},
     {"EXISTS",        TK_EXISTS},
     {"EXPLAIN",       TK_EXPLAIN},
-    {"FILE_FACTOR",   TK_FILE_FACTOR},
+    // {"FILE_FACTOR",   TK_FILE_FACTOR},
     {"FILL",          TK_FILL},
     {"FIRST",         TK_FIRST},
     {"FLOAT",         TK_FLOAT},