Merge remote-tracking branch 'origin/3.0' into fix/liao_cov

CONTRIBUTING-CN.md

@@ -7,25 +7,18 @@
 - 任何用户都可以通过 **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)** 向我们报告错误。请您对所遇到的问题进行**详细描述**，最好提供重现错误的详细步骤。
 - 欢迎提供包含由 Bug 生成的日志文件的附录。
 
-## 需要强调的代码提交规则
-
-- 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。
-- 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。
-- 如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。
-- 将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。
-
-## 贡献指南
-
-1. 请用友好的语气书写。
-
-2. **主动语态**总体上优于被动语态。主动语态中的句子会突出执行动作的人，而不是被动语态突出动作的接受者。
-
-3. 文档写作建议
-
-- 正确拼写产品名称 “TDengine”。 “TD” 用大写字母，“TD” 和 “engine” 之间没有空格 **（正确拼写：TDengine）**。
-- 在句号或其他标点符号后只留一个空格。
-
-4. 尽量**使用简单句**，而不是复杂句。
+## 代码提交规则
+
+1. 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。
+2. 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。
+   如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。
+   将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。
+3. 将TDengine仓库库fork到自己的账户中并创建分支(branch)。
+   注意：默认分支`main`不能直接接受PR，请基于开发分支`3.0`创建自己的分支。
+   注意：修改文档的分支要以`docs/`为开头，以免进行不必要的测试。
+4. 创建pull request，将自己的分支合并到开发分支`3.0`，我们开发团队将尽快审核。
+
+如遇任何问题，请添加官方微信TDengineECO。我们的团队会帮忙解决。
 
 ## 给贡献者的礼品
 

CONTRIBUTING.md

-# Contributing
+# Contributing to TDengine
 
-We appreciate contributions from all developers. Feel free to follow us, fork the repository, report bugs, and even submit your code on GitHub. However, we would like developers to follow the guidelines in this document to ensure effective cooperation.
+TDengine Community Edition is free, open-source software. Its development is led by the TDengine Team, but we welcome contributions from all community members and open-source developers. This document describes how you can contribute, no matter whether you're a user or a developer yourself.
 
-## Reporting a bug
+## Bug reports
 
-- Any users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. We would appreciate if you could provide **a detailed description** of the problem you encountered, including steps to reproduce it.
+All users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. To ensure that the development team can locate and resolve the issue that you experienced, please include the following in your bug report:
 
-- Attaching log files caused by the bug is really appreciated.
+- A detailed description of the issue, including the steps to reproduce it.
+- Any log files that may be relevant to the issue.
 
-## Guidelines for committing code
+## Code contributions
 
-- You must agree to the **Contributor License Agreement(CLA) before submitting your code patch**. Follow the **[TAOSData CLA](https://cla-assistant.io/taosdata/TDengine)** link to read through and sign the agreement. If you do not accept the agreement, your contributions cannot be accepted.
+Developers are encouraged to submit patches to the project, and all contributions, from minor documentation changes to bug fixes, are appreciated by our team. To ensure that your code can be merged successfully and improve the experience for other community members, we ask that you go through the following procedure before submitting a pull request: 
 
-- Please solve an issue or add a feature registered in the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**.
-- If no corresponding issue or feature is found in the issue tracker, please **create one**.
-- When submitting your code to our repository, please create a pull request with the **issue number** included.
+1. Read and accept the terms of the TAOS Data Contributor License Agreement (CLA) located at [https://cla-assistant.io/taosdata/TDengine](https://cla-assistant.io/taosdata/TDengine).
 
-## Guidelines for communicating
+2. For bug fixes, search the [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) to check whether the bug has already been filed.
+   - If the bug that you want to fix already exists in the issue tracker, review the previous discussion before submitting your patch.
+   - If the bug that you want to fix does not exist in the issue tracker, click **New issue** and file a report.
+   - Ensure that you note the issue number in your pull request when you submit your patch.
+	
+3. Fork our repository to your GitHub account and create a branch for your patch.
+   **Important:** The `main` branch is for stable versions and cannot accept patches directly. For all code and documentation changes, create your own branch from the development branch `3.0` and not from `main`.
+   Note: For a documentation change, ensure that the branch name starts with `docs/` so that the change can be merged without running tests.
+   
+4. Create a pull request to merge your changes into the development branch `3.0`, and our team members will review the request as soon as possible.
 
-1. Please be **nice and polite** in the description.
-2. **Active voice is better than passive voice in general**. Sentences in the active voice will highlight who is performing the action rather than the recipient of the action highlighted by the passive voice.
-3. Documentation writing advice
+If you encounter any difficulties or problems in contributing your code, you can join our [Discord server](https://discord.com/invite/VZdSuUg4pS) and receive assistance from members of the TDengine Team.
 
-- Spell the product name "TDengine" correctly. "TD" is written in capital letters, and there is no space between "TD" and "engine" (**Correct spelling: TDengine**).
-- Please **capitalize the first letter** of every sentence.
-- Leave **only one space** after periods or other punctuation marks.
-- Use **American spelling**.
-- When possible, **use second person** rather than first person (e.g.“You are recommended to use a reverse proxy such as Nginx.” rather than “We recommend to use a reverse proxy such as Nginx.”).
+## Expressing our thanks
 
-5. Use **simple sentences**, rather than complex sentences.
-
-## Gifts for the contributors
-
-Developers, as long as you contribute to TDengine, whether it's code contributions to fix bugs or feature requests, or documentation changes, **you are eligible for a very special Contributor Souvenir Gift!**
-
-**You can choose one of the following gifts:**
+To thank community members for your support, we are offering a free gift to any developer who submits at least one contribution. You can choose one of the following items:
 
 <p align="left">
   <img
@@ -53,12 +49,10 @@ Developers, as long as you contribute to TDengine, whether it's code contributio
     width="200"
     />
 
-The TDengine community is committed to making TDengine accepted and used by more developers.
-
-Just fill out the **Contributor Submission Form** to choose your desired gift.
-
-- [Contributor Submission Form](https://page.ma.scrmtech.com/form/index?pf_uid=27715_2095&id=12100)
+If you would like to claim your gift, send an email to [developer@tdengine.com](mailto:developer@tdengine.com?subject=Claiming&20my%20developer%20gift) including the following information:
 
-## Contact us
+- Your GitHub account name
+- Your name and mailing address
+- Your preferred gift
 
-If you have any problems or questions that need help from us, please feel free to add our WeChat account: TDengineECO.
+Note: Limit one per person.
\ No newline at end of file

Jenkinsfile2

@@ -303,7 +303,7 @@ def pre_test_build_win() {
         set CL=/MP8
         echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cmake"
         time /t
-        cmake .. -G "NMake Makefiles JOM" -DBUILD_TEST=true || exit 7
+        cmake .. -G "NMake Makefiles JOM" -DBUILD_TEST=true -DBUILD_TOOLS=true || exit 7
         echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jom -j 6"
         time /t
         jom -j 6 || exit 8

cmake/taostools_CMakeLists.txt.in

@@ -2,7 +2,7 @@
 # taos-tools
 ExternalProject_Add(taos-tools
         GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
-        GIT_TAG 23e2b73
+        GIT_TAG efa2a5f
         SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
         BINARY_DIR ""
         #BUILD_IN_SOURCE TRUE

docs/en/05-get-started/index.md

@@ -20,7 +20,18 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
 <DocCardList items={useCurrentSidebarCategory().items}/>
 ```
 
-### Join TDengine Community
+## Study TDengine Knowledge Map
+
+The TDengine Knowledge Map covers the various knowledge points of TDengine, revealing the invocation relationships and data flow between various conceptual entities. Learning and understanding the TDengine Knowledge Map will help you quickly master the TDengine knowledge system.
+
+<figure>
+<center>
+<a href="pathname:///img/tdengine-map.svg" target="_blank"><img src="/img/tdengine-map.svg" width="80%" /></a>
+<figcaption>Diagram 1. TDengine Knowledge Map</figcaption>
+</center>
+</figure>
+
+## Join TDengine Community
 
 <table width="100%">
 <tr align="center" style={{border:0}}>

docs/en/12-taos-sql/06-select.md

@@ -301,7 +301,7 @@ SELECT TIMEZONE();
 ### Syntax
 
 ```txt
-WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_
+WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_
 ```
 
 ### Specification

docs/en/14-reference/03-connector/index.mdx

@@ -26,14 +26,13 @@ Using REST connection can support a broader range of operating systems as it doe
 
 TDengine version updates often add new features, and the connector versions in the list are the best-fit versions of the connector.
 
-| **TDengine Versions** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** |
-| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- |
-| **3.0.0.0 and later**    | 3.0.0    | current version   | 3.0 branch     | 3.0.0         | 3.0.0           | current version |
-| **2.4.0.14 and up** | 2.0.38 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
-| **2.4.0.6 and up** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
-| **2.4.0.4 - 2.4.0.5** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
-| **2.2.x.x ** | 2.0.36 | current version | master branch | n/a | 2.0.7 - 2.0.9 | current version |
-| **2.0.x.x ** | 2.0.34 | current version | master branch | n/a | 2.0.1 - 2.0.6 | current version |
+| **TDengine Versions**       | **Java**       | **Python**     | **Go**         | **C#**        | **Node.js**     | **Rust**        |
+| --------------------------- | -------------- | -------------- | -------------- | ------------- | --------------- | --------------- |
+| **3.0.0.0 and later**       | 3.0.2 +        | latest version | 3.0 branch     | 3.0.0         | 3.0.0           | current version |
+| **2.4.0.14 and up **        | 2.0.38         | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
+| **2.4.0.4 - 2.4.0.13 **     | 2.0.37         | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
+| **2.2.x.x **                | 2.0.36         | latest version | master branch  | n/a           | 2.0.7 - 2.0.9   | current version |
+| **2.0.x.x **                | 2.0.34         | latest version | master branch  | n/a           | 2.0.1 - 2.0.6   | current version |
 
 ## Functional Features
 

docs/zh/08-connector/index.md

@@ -26,14 +26,13 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速
 
 TDengine 版本更新往往会增加新的功能特性，列表中的连接器版本为连接器最佳适配版本。
 
-| **TDengine 版本**     | **Java** | **Python** | **Go**       | **C#**        | **Node.js**     | **Rust** |
-| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- |
-| **3.0.0.0 及以上**    | 3.0.0    | 当前版本   | 3.0 分支     | 3.0.0         | 3.0.0           | 当前版本 |
-| **2.4.0.14 及以上**   | 2.0.38   | 当前版本   | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
-| **2.4.0.6 及以上**    | 2.0.37   | 当前版本   | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
-| **2.4.0.4 - 2.4.0.5** | 2.0.37   | 当前版本   | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
-| **2.2.x.x **          | 2.0.36   | 当前版本   | master 分支  | n/a           | 2.0.7 - 2.0.9   | 当前版本 |
-| **2.0.x.x **          | 2.0.34   | 当前版本   | master 分支  | n/a           | 2.0.1 - 2.0.6   | 当前版本 |
+| **TDengine 版本**      | **Java**  | **Python** | **Go**       | **C#**        | **Node.js**     | **Rust** |
+| ---------------------- | --------- | ---------- | ------------ | ------------- | --------------- | -------- |
+| **3.0.0.0 及以上**     | 3.0.2以上 | 当前版本   | 3.0 分支     | 3.0.0         | 3.0.0           | 当前版本 |
+| **2.4.0.14 及以上**    | 2.0.38    | 当前版本   | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
+| **2.4.0.4 - 2.4.0.13** | 2.0.37    | 当前版本   | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
+| **2.2.x.x **           | 2.0.36    | 当前版本   | master 分支  | n/a           | 2.0.7 - 2.0.9   | 当前版本 |
+| **2.0.x.x **           | 2.0.34    | 当前版本   | master 分支  | n/a           | 2.0.1 - 2.0.6   | 当前版本 |
 
 ## 功能特性
 

docs/zh/12-taos-sql/06-select.md

@@ -302,7 +302,7 @@ SELECT TIMEZONE();
 ### 语法
 
 ```txt
-WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_
+WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_
 ```
 
 ### 正则表达式规范

include/common/tmsg.h

@@ -1615,15 +1615,21 @@ typedef struct SSubQueryMsg {
   uint64_t taskId;
   int64_t  refId;
   int32_t  execId;
+  int32_t  msgMask;
   int8_t   taskType;
   int8_t   explain;
   int8_t   needFetch;
-  uint32_t sqlLen;  // the query sql,
-  uint32_t phyLen;
-  int32_t  msgMask;
-  char     msg[];
+  uint32_t sqlLen;
+  char    *sql;
+  uint32_t msgLen;
+  char    *msg;
 } SSubQueryMsg;
 
+int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq);
+int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq);
+void tFreeSSubQueryMsg(SSubQueryMsg *pReq);
+
+
 typedef struct {
   SMsgHead header;
   uint64_t sId;
@@ -1732,6 +1738,13 @@ typedef struct {
   int32_t  execId;
 } STaskDropReq;
 
+int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq);
+int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq);
+
+int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp);
+int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp);
+
+
 typedef struct {
   int32_t code;
 } STaskDropRsp;

include/libs/sync/sync.h

@@ -36,7 +36,7 @@ extern "C" {
 #define SYNC_DEL_WAL_MS              (1000 * 60)
 #define SYNC_ADD_QUORUM_COUNT        3
 #define SYNC_MNODE_LOG_RETENTION     10000
-#define SYNC_VNODE_LOG_RETENTION     100
+#define SYNC_VNODE_LOG_RETENTION     20
 #define SNAPSHOT_MAX_CLOCK_SKEW_MS   1000 * 10
 #define SNAPSHOT_WAIT_MS             1000 * 30
 

packaging/MPtestJenkinsfile

@@ -40,27 +40,32 @@ pipeline {
         choice(
             name: 'sourcePath',
             choices: ['nas','web'], 
-            description: 'choice which way to download the installation pacakge;web is Office Web and nas means taos nas server '
+            description: 'Choice which way to download the installation pacakge;web is Office Web and nas means taos nas server '
+        )
+        choice(
+            name: 'verMode',
+            choices: ['all','community','enterprise'], 
+            description: 'Choice  which types of package you want do check '
         )
         string (
             name:'version', 
-            defaultValue:'3.0.1.6', 
-            description: 'release version number,eg: 3.0.0.1 or 3.0.0.'
+            defaultValue:'3.0.1.7', 
+            description: 'Release version number,eg: 3.0.0.1 or 3.0.0.'
         )
         string (
             name:'baseVersion', 
-            defaultValue:'3.0.1.6', 
-            description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
+            defaultValue:'3.0.1.7', 
+            description: 'The number of baseVerison is generally not modified.Now it is 3.0.0.1'
         )
         string (
             name:'toolsVersion', 
             defaultValue:'2.2.7', 
-            description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
+            description: 'Release version number,eg:2.2.0'
         )
         string (
             name:'toolsBaseVersion', 
             defaultValue:'2.1.2', 
-            description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
+            description: 'The number of baseVerison is generally not modified.Now it is 2.1.2'
         )
     }
     environment{
@@ -68,10 +73,10 @@ pipeline {
         TDINTERNAL_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal'
         TDENGINE_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal/community'
         BRANCH_NAME = '3.0'
-
-        TD_SERVER_TAR = "TDengine-server-${version}-Linux-x64.tar.gz"
+        
+        TD_SERVER_TAR = "${preServerPackag}-${version}-Linux-x64.tar.gz"
         BASE_TD_SERVER_TAR = "TDengine-server-${baseVersion}-Linux-x64.tar.gz"
-
+    
         TD_SERVER_ARM_TAR = "TDengine-server-${version}-Linux-arm64.tar.gz"
         BASE_TD_SERVER_ARM_TAR = "TDengine-server-${baseVersion}-Linux-arm64.tar.gz"
 
@@ -108,19 +113,28 @@ pipeline {
                         timeout(time: 30, unit: 'MINUTES'){
                             sync_source("${BRANCH_NAME}")
                             sh '''
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi
+                            verModeList=${verMode}                          
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m ${verModeSin} -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done 
+                            '''  
+
+                            sh '''                              
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
-                            python3 checkPackageRuning.py  
-                            '''
-                            sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
-                            '''
-                            sh '''
+                            bash testpackage.sh -f server -m community -f server  -l true -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                            python3 checkPackageRuning.py
+                            '''   
+
+                            sh ''' 
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t deb 
+                            python3 checkPackageRuning.py
                             '''
                         }
                     }
@@ -131,22 +145,30 @@ pipeline {
                         timeout(time: 30, unit: 'MINUTES'){
                             sync_source("${BRANCH_NAME}")
                             sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server  ${sourcePath} 
-                            python3 checkPackageRuning.py  
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi
+                            verModeList=${verMode}
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m ${verModeSin} -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done 
                             '''
+
                             sh '''
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l true -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                            python3 checkPackageRuning.py
                             '''
+
                             sh '''
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server   ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t deb 
+                            python3 checkPackageRuning.py
                             dpkg -r tdengine
-                            '''
-
+                            '''                                
                         }
                     }
                 }
@@ -156,19 +178,28 @@ pipeline {
                         timeout(time: 30, unit: 'MINUTES'){
                             sync_source("${BRANCH_NAME}")
                             sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi                            
+                            verModeList=${verMode}
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m community -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done 
                             '''
-                            sh '''
+
+                            sh '''                            
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l true -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                            python3 checkPackageRuning.py
                             '''
+
                             sh '''
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t rpm 
+                            python3 checkPackageRuning.py
                             '''
                         }
                     }
@@ -179,21 +210,30 @@ pipeline {
                         timeout(time: 30, unit: 'MINUTES'){
                             sync_source("${BRANCH_NAME}")
                             sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging 
-                            bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server   ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi                            
+                            verModeList=${verMode}
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m ${verModeSin} -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done
                             '''
+
                             sh '''
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l true -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                            python3 checkPackageRuning.py
                             '''
+
                             sh '''
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server  ${sourcePath}
-                            python3 checkPackageRuning.py  
+                            bash testpackage.sh -f server -m community -f server  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t rpm 
+                            python3 checkPackageRuning.py
                             sudo rpm -e tdengine 
-                            '''              
+                            '''
                         }
                     }
                 }                  
@@ -203,9 +243,16 @@ pipeline {
                         timeout(time: 30, unit: 'MINUTES'){
                             sync_source("${BRANCH_NAME}")
                             sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_SERVER_ARM_TAR} ${version} ${BASE_TD_SERVER_ARM_TAR} ${baseVersion} server ${sourcePath} 
-                            python3 checkPackageRuning.py  
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi                            
+                            verModeList=${verMode}
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m ${verModeSin} -f server  -l false -c arm64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done 
                             '''
                         }
                     }
@@ -219,8 +266,16 @@ pipeline {
                     steps {
                         timeout(time: 30, unit: 'MINUTES'){
                             sh '''
-                            cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_CLIENT_TAR} ${version} ${BASE_TD_CLIENT_TAR} ${baseVersion} client ${sourcePath}
+                            if [ "${verMode}" = "all" ];then
+                                verMode="community enterprise"
+                            fi  
+                            verModeList=${verMode}
+                            for verModeSin in ${verModeList}
+                            do
+                                cd ${TDENGINE_ROOT_DIR}/packaging
+                                bash testpackage.sh -f server -m ${verModeSin} -f client  -l false -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar 
+                                python3 checkPackageRuning.py
+                            done 
                             python3 checkPackageRuning.py  192.168.0.21
                             '''
                         }
@@ -231,8 +286,10 @@ pipeline {
                     steps {
                         timeout(time: 30, unit: 'MINUTES'){
                             sh '''
+                            verModeList=community
                             cd ${TDENGINE_ROOT_DIR}/packaging
-                            bash testpackage.sh ${TD_CLIENT_LITE_TAR} ${version} ${BASE_TD_CLIENT_LITE_TAR} ${baseVersion} client  ${sourcePath}
+                            bash testpackage.sh -f server -m ${verModeSin} -f client  -l true -c x64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar
+                            python3 checkPackageRuning.py
                             python3 checkPackageRuning.py  192.168.0.24
                             '''         
                         }
@@ -245,8 +302,16 @@ pipeline {
             steps {
                 timeout(time: 30, unit: 'MINUTES'){
                     sh '''
-                    cd ${TDENGINE_ROOT_DIR}/packaging
-                    bash testpackage.sh ${TD_CLIENT_ARM_TAR} ${version} ${BASE_TD_CLIENT_ARM_TAR} ${baseVersion} client  ${sourcePath}
+                    if [ "${verMode}" = "all" ];then
+                        verMode="community enterprise"
+                    fi  
+                    verModeList=${verMode}
+                    for verModeSin in ${verModeList}
+                    do
+                        cd ${TDENGINE_ROOT_DIR}/packaging
+                        bash testpackage.sh -f server -m ${verModeSin} -f client  -l false -c arm64  -v ${version} -o ${baseVersion} -s ${sourcePath}  -t tar
+                        python3 checkPackageRuning.py
+                    done 
                     python3 checkPackageRuning.py  192.168.0.21
                     '''  
                 }

packaging/release.sh

@@ -221,12 +221,12 @@ if [[ "$cpuType" == "x64" ]] || [[ "$cpuType" == "aarch64" ]] || [[ "$cpuType" =
     # community-version compile
     cmake ../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DPAGMODE=${pagMode} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
   elif [ "$verMode" == "cloud" ]; then
-    cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
+    cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
   elif [ "$verMode" == "cluster" ]; then
     if [[ "$dbName" != "taos" ]]; then
       replace_enterprise_$dbName
     fi
-    cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
+    cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
   fi
 else
   echo "input cpuType=${cpuType} error!!!"

packaging/testpackage.sh

 #!/bin/sh
+
+
+function usage() {
+    echo "$0"
+    echo -e "\t -f test file type,server/client/tools/"
+    echo -e "\t -m pacakage version Type,community/enterprise"
+    echo -e "\t -l package type,lite or not"
+    echo -e "\t -c operation type,x64/arm64"
+    echo -e "\t -v pacakage version,3.0.1.7"
+    echo -e "\t -o pacakage version,3.0.1.7"
+    echo -e "\t -s source Path,web/nas"
+    echo -e "\t -t package Type,tar/rpm/deb"
+    echo -e "\t -h help"
+}
+
+
 #parameter
 scriptDir=$(dirname $(readlink -f $0))
-packgeName=$1
-version=$2
-originPackageName=$3
-originversion=$4
-testFile=$5
-# sourcePath:web/nas
-sourcePath=$6
+version="3.0.1.7"
+originversion="3.0.1.7"
+testFile="server"
+verMode="communtity"
+sourcePath="nas"
+cpuType="x64"
+lite="true"
+packageType="tar"
 subFile="taos.tar.gz"
-
+while getopts "m:c:f:l:s:o:t:v:h" opt; do
+    case $opt in
+        m)
+            verMode=$OPTARG
+            ;;
+        v)
+            version=$OPTARG
+            ;;
+        f)
+            testFile=$OPTARG
+            ;;
+        l)
+            lite=$OPTARG
+            ;;
+        s)
+            sourcePath=$OPTARG
+            ;;
+        o)
+            originversion=$OPTARG
+            ;;
+        c)
+            cpuType=$OPTARG
+            ;;
+        t)
+            packageType=$OPTARG
+            ;;
+        h)
+            usage
+            exit 0
+            ;;
+        ?)
+            echo "Invalid option: -$OPTARG"
+            usage
+            exit 0
+            ;;
+    esac
+done
+
+
+
+echo "testFile:${testFile},verMode:${verMode},lite:${lite},cpuType:${cpuType},packageType:${packageType},version-${version},originversion:${originversion},sourcePath:${sourcePath}"
 # Color setting
 RED='\033[41;30m'
 GREEN='\033[1;32m'
@@ -21,20 +78,40 @@ BLUE_DARK='\033[0;34m'
 GREEN_UNDERLINE='\033[4;32m'
 NC='\033[0m'
 
-if [ ${testFile} = "server" ];then
-    tdPath="TDengine-server-${version}"
-    originTdpPath="TDengine-server-${originversion}"
+if [[ ${verMode} = "enterprise" ]];then
+    prePackag="TDengine-enterprise-${testFile}"
+elif [ ${verMode} = "community" ];then
+    prePackag="TDengine-${testFile}"
+fi
+if [ ${lite} = "true" ];then
+    packageLite="-Lite"
+elif [ ${lite} = "false"  ];then
+    packageLite=""
+fi
+if [[ "$packageType" = "tar" ]] ;then
+    packageType="tar.gz"
+fi
+
+tdPath="${prePackag}-${version}"
+originTdpPath="${prePackag}-${originversion}"
+
+packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}"
+originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}"
+
+if [ "$testFile" == "server" ] ;then
     installCmd="install.sh"
 elif [ ${testFile} = "client" ];then
-    tdPath="TDengine-client-${version}"
-    originTdpPath="TDengine-client-${originversion}"
     installCmd="install_client.sh"    
 elif [ ${testFile} = "tools" ];then
     tdPath="taosTools-${version}"
     originTdpPath="taosTools-${originversion}"
+    packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}"
+    originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}"    
     installCmd="install-taostools.sh"
 fi
 
+
+echo "tdPath:${tdPath},originTdpPath:${originTdpPath},packgeName:${packgeName},originPackageName:${originPackageName}"
 function cmdInstall {
 command=$1
 if command -v ${command} ;then
@@ -76,16 +153,16 @@ file=$1
 versionPath=$2
 sourceP=$3
 nasServerIP="192.168.1.131"
-packagePath="/nas/TDengine/v${versionPath}/community"
+packagePath="/nas/TDengine/v${versionPath}/${verMode}"
 if [ -f  ${file}  ];then
     echoColor  YD "${file} already exists ,it will delete it and download  it again "
     rm -rf ${file}
 fi
 
-if [ ${sourceP} = 'web' ];then
+if [[ ${sourceP} = 'web' ]];then
     echoColor  BD "====download====:wget https://www.taosdata.com/assets-download/3.0/${file}"
     wget https://www.taosdata.com/assets-download/3.0/${file}
-elif [ ${sourceP} = 'nas' ];then
+elif [[ ${sourceP} = 'nas' ]];then
     echoColor  BD "====download====:scp root@${nasServerIP}:${packagePath}/${file} ."
     scp root@${nasServerIP}:${packagePath}/${file} .
 fi

source/client/src/clientMain.c

@@ -1106,6 +1106,8 @@ int taos_get_table_vgId(TAOS *taos, const char *db, const char *table, int *vgId
     return terrno;
   }
 
+  pRequest->syncQuery = true;
+
   STscObj  *pTscObj = pRequest->pTscObj;
   SCatalog *pCtg = NULL;
   code = catalogGetHandle(pTscObj->pAppInfo->clusterId, &pCtg);

source/common/src/tdatablock.c

@@ -36,7 +36,7 @@ int32_t colDataGetFullLength(const SColumnInfoData* pColumnInfoData, int32_t num
   if (IS_VAR_DATA_TYPE(pColumnInfoData->info.type)) {
     return pColumnInfoData->varmeta.length + sizeof(int32_t) * numOfRows;
   } else {
-    return pColumnInfoData->info.bytes * numOfRows + BitmapLen(numOfRows);
+    return ((pColumnInfoData->info.type == TSDB_DATA_TYPE_NULL) ? 0 : pColumnInfoData->info.bytes * numOfRows) + BitmapLen(numOfRows);
   }
 }
 
@@ -1960,6 +1960,7 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf)
           memset(pBuf, 0, sizeof(pBuf));
           char*   pData = colDataGetVarData(pColInfoData, j);
           int32_t dataSize = TMIN(sizeof(pBuf), varDataLen(pData));
+          dataSize = TMIN(dataSize, 50);
           memcpy(pBuf, varDataVal(pData), dataSize);
           len += snprintf(dumpBuf + len, size - len, " %15s |", pBuf);
           if (len >= size - 1) return dumpBuf;

source/common/src/tglobal.c

@@ -277,7 +277,9 @@ static int32_t taosAddServerLogCfg(SConfig *pCfg) {
 static int32_t taosAddClientCfg(SConfig *pCfg) {
   char    defaultFqdn[TSDB_FQDN_LEN] = {0};
   int32_t defaultServerPort = 6030;
-  if (taosGetFqdn(defaultFqdn) != 0) return -1;
+  if (taosGetFqdn(defaultFqdn) != 0) {
+    strcpy(defaultFqdn, "localhost");
+  }
 
   if (cfgAddString(pCfg, "firstEp", "", 1) != 0) return -1;
   if (cfgAddString(pCfg, "secondEp", "", 1) != 0) return -1;

source/common/src/tmsg.c

@@ -4643,6 +4643,178 @@ int32_t tDeserializeSMqHbReq(void *buf, int32_t bufLen, SMqHbReq *pReq) {
   return 0;
 }
 
+int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) {
+  int32_t headLen = sizeof(SMsgHead);
+  if (buf != NULL) {
+    buf = (char *)buf + headLen;
+    bufLen -= headLen;
+  }
+
+  SEncoder encoder = {0};
+  tEncoderInit(&encoder, buf, bufLen);
+  if (tStartEncode(&encoder) < 0) return -1;
+
+  if (tEncodeU64(&encoder, pReq->sId) < 0) return -1;
+  if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1;
+  if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->refId) < 0) return -1;
+  if (tEncodeI32(&encoder, pReq->execId) < 0) return -1;
+  if (tEncodeI32(&encoder, pReq->msgMask) < 0) return -1;
+  if (tEncodeI8(&encoder, pReq->taskType) < 0) return -1;
+  if (tEncodeI8(&encoder, pReq->explain) < 0) return -1;
+  if (tEncodeI8(&encoder, pReq->needFetch) < 0) return -1;
+  if (tEncodeU32(&encoder, pReq->sqlLen) < 0) return -1;
+  if (tEncodeCStrWithLen(&encoder, pReq->sql, pReq->sqlLen) < 0) return -1;
+  if (tEncodeU32(&encoder, pReq->msgLen) < 0) return -1;
+  if (tEncodeBinary(&encoder, (uint8_t*)pReq->msg, pReq->msgLen) < 0) return -1;
+
+  tEndEncode(&encoder);
+
+  int32_t tlen = encoder.pos;
+  tEncoderClear(&encoder);
+
+  if (buf != NULL) {
+    SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen);
+    pHead->vgId = htonl(pReq->header.vgId);
+    pHead->contLen = htonl(tlen + headLen);
+  }
+
+  return tlen + headLen;
+}
+
+int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) {
+  int32_t headLen = sizeof(SMsgHead);
+
+  SMsgHead *pHead = buf;
+  pHead->vgId = pReq->header.vgId;
+  pHead->contLen = pReq->header.contLen;
+
+  SDecoder decoder = {0};
+  tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen);
+
+  if (tStartDecode(&decoder) < 0) return -1;
+
+  if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1;
+  if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1;
+  if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1;
+  if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1;
+  if (tDecodeI32(&decoder, &pReq->msgMask) < 0) return -1;
+  if (tDecodeI8(&decoder, &pReq->taskType) < 0) return -1;
+  if (tDecodeI8(&decoder, &pReq->explain) < 0) return -1;
+  if (tDecodeI8(&decoder, &pReq->needFetch) < 0) return -1;
+  if (tDecodeU32(&decoder, &pReq->sqlLen) < 0) return -1;
+  if (tDecodeCStrAlloc(&decoder, &pReq->sql) < 0) return -1;
+  if (tDecodeU32(&decoder, &pReq->msgLen) < 0) return -1;
+  if (tDecodeBinaryAlloc(&decoder, (void**)&pReq->msg, NULL) < 0) return -1;
+  
+  tEndDecode(&decoder);
+
+  tDecoderClear(&decoder);
+  return 0;
+}
+
+void tFreeSSubQueryMsg(SSubQueryMsg *pReq) {
+  if (NULL == pReq) {
+    return;
+  }
+
+  taosMemoryFreeClear(pReq->sql);
+  taosMemoryFreeClear(pReq->msg);
+}
+
+int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) {
+  int32_t headLen = sizeof(SMsgHead);
+  if (buf != NULL) {
+    buf = (char *)buf + headLen;
+    bufLen -= headLen;
+  }
+
+  SEncoder encoder = {0};
+  tEncoderInit(&encoder, buf, bufLen);
+  if (tStartEncode(&encoder) < 0) return -1;
+
+  if (tEncodeU64(&encoder, pReq->sId) < 0) return -1;
+  if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1;
+  if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->refId) < 0) return -1;
+  if (tEncodeI32(&encoder, pReq->execId) < 0) return -1;
+
+  tEndEncode(&encoder);
+
+  int32_t tlen = encoder.pos;
+  tEncoderClear(&encoder);
+
+  if (buf != NULL) {
+    SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen);
+    pHead->vgId = htonl(pReq->header.vgId);
+    pHead->contLen = htonl(tlen + headLen);
+  }
+
+  return tlen + headLen;
+}
+
+int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) {
+  int32_t headLen = sizeof(SMsgHead);
+
+  SMsgHead *pHead = buf;
+  pHead->vgId = pReq->header.vgId;
+  pHead->contLen = pReq->header.contLen;
+
+  SDecoder decoder = {0};
+  tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen);
+
+  if (tStartDecode(&decoder) < 0) return -1;
+
+  if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1;
+  if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1;
+  if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1;
+  if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1;
+  
+  tEndDecode(&decoder);
+
+  tDecoderClear(&decoder);
+  return 0;
+}
+
+int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) {
+  SEncoder encoder = {0};
+  tEncoderInit(&encoder, buf, bufLen);
+  if (tStartEncode(&encoder) < 0) return -1;
+
+  if (tEncodeI32(&encoder, pRsp->code) < 0) return -1;
+  if (tEncodeCStr(&encoder, pRsp->tbFName) < 0) return -1;
+  if (tEncodeI32(&encoder, pRsp->sversion) < 0) return -1;
+  if (tEncodeI32(&encoder, pRsp->tversion) < 0) return -1;
+  if (tEncodeI64(&encoder, pRsp->affectedRows) < 0) return -1;
+
+  tEndEncode(&encoder);
+
+  int32_t tlen = encoder.pos;
+  tEncoderClear(&encoder);
+
+  return tlen;
+}
+
+int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) {
+  SDecoder decoder = {0};
+  tDecoderInit(&decoder, (char *)buf, bufLen);
+
+  if (tStartDecode(&decoder) < 0) return -1;
+
+  if (tDecodeI32(&decoder, &pRsp->code) < 0) return -1;
+  if (tDecodeCStrTo(&decoder, pRsp->tbFName) < 0) return -1;
+  if (tDecodeI32(&decoder, &pRsp->sversion) < 0) return -1;
+  if (tDecodeI32(&decoder, &pRsp->tversion) < 0) return -1;
+  if (tDecodeI64(&decoder, &pRsp->affectedRows) < 0) return -1;
+  
+  tEndDecode(&decoder);
+
+  tDecoderClear(&decoder);
+  return 0;
+}
+
 
 int32_t tSerializeSSchedulerHbReq(void *buf, int32_t bufLen, SSchedulerHbReq *pReq) {
   int32_t headLen = sizeof(SMsgHead);

source/dnode/mgmt/mgmt_vnode/src/vmWorker.c

@@ -148,8 +148,8 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
 
   SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
   if (pVnode == NULL) {
-    dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d", pHead->vgId, pMsg, terrstr(),
-            TMSG_INFO(pMsg->msgType), qtype);
+    dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d contLen:%d", pHead->vgId, pMsg, terrstr(),
+            TMSG_INFO(pMsg->msgType), qtype, pHead->contLen);
     return terrno != 0 ? terrno : -1;
   }
 

source/dnode/vnode/inc/vnode.h

@@ -239,6 +239,7 @@ int32_t tqReaderSetDataMsg(STqReader *pReader, const SSubmitReq *pMsg, int64_t v
 bool    tqNextDataBlock(STqReader *pReader);
 bool    tqNextDataBlockFilterOut(STqReader *pReader, SHashObj *filterOutUids);
 int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader);
+int32_t tqRetrieveTaosxBlock(STqReader *pReader, SArray *blocks, SArray *schemas);
 
 int32_t vnodeEnqueueStreamMsg(SVnode *pVnode, SRpcMsg *pMsg);
 

source/dnode/vnode/src/sma/smaTimeRange.c

@@ -203,10 +203,13 @@ static int32_t tdProcessTSmaInsertImpl(SSma *pSma, int64_t indexUid, const char
     goto _err;
   }
 
-  SBatchDeleteReq deleteReq;
+  SBatchDeleteReq deleteReq = {0};
   SSubmitReq     *pSubmitReq =
       tqBlockToSubmit(pSma->pVnode, (const SArray *)msg, pTsmaStat->pTSchema, &pTsmaStat->pTSma->schemaTag, true,
                       pTsmaStat->pTSma->dstTbUid, pTsmaStat->pTSma->dstTbName, &deleteReq);
+  // TODO deleteReq
+  taosArrayDestroy(deleteReq.deleteReqs);
+  
 
   if (!pSubmitReq) {
     smaError("vgId:%d, failed to gen submit blk while tsma insert for smaIndex %" PRIi64 " since %s", SMA_VID(pSma),

source/dnode/vnode/src/tq/tq.c

@@ -671,7 +671,6 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) {
         SSubmitReq* pCont = (SSubmitReq*)&pHead->body;
 
         if (tqTaosxScanLog(pTq, pHandle, pCont, &taosxRsp) < 0) {
-          /*ASSERT(0);*/
         }
         if (taosxRsp.blockNum > 0 /* threshold */) {
           tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer);

source/dnode/vnode/src/tq/tqExec.c

@@ -44,7 +44,7 @@ static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, SMqDataRsp* pRs
   return 0;
 }
 
-static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) {
+static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t n) {
   SMetaReader mr = {0};
   metaReaderInit(&mr, pTq->pVnode->pMeta, 0);
   // TODO add reference to gurantee success
@@ -52,8 +52,10 @@ static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) {
     metaReaderClear(&mr);
     return -1;
   }
-  char* tbName = strdup(mr.me.name);
-  taosArrayPush(pRsp->blockTbName, &tbName);
+  for (int32_t i = 0; i < n; i++) {
+    char* tbName = strdup(mr.me.name);
+    taosArrayPush(pRsp->blockTbName, &tbName);
+  }
   metaReaderClear(&mr);
   return 0;
 }
@@ -111,7 +113,7 @@ int32_t tqScanData(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, STqOffs
   if (pRsp->withTbName) {
     if (pRsp->rspOffset.type == TMQ_OFFSET__LOG) {
       int64_t uid = pExec->pExecReader->msgIter.uid;
-      tqAddTbNameToRsp(pTq, uid, pRsp);
+      tqAddTbNameToRsp(pTq, uid, pRsp, 1);
     } else {
       pRsp->withTbName = false;
     }
@@ -155,7 +157,7 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, STaosxRsp* pRsp, SMqMeta
         int64_t uid = 0;
         if (pOffset->type == TMQ_OFFSET__LOG) {
           uid = pExec->pExecReader->msgIter.uid;
-          if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
+          if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, 1) < 0) {
             continue;
           }
         } else {
@@ -225,18 +227,30 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
   STqExecHandle* pExec = &pHandle->execHandle;
   ASSERT(pExec->subType != TOPIC_SUB_TYPE__COLUMN);
 
+  SArray* pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
+  SArray* pSchemas = taosArrayInit(0, sizeof(void*));
+
   if (pExec->subType == TOPIC_SUB_TYPE__TABLE) {
     STqReader* pReader = pExec->pExecReader;
     tqReaderSetDataMsg(pReader, pReq, 0);
     while (tqNextDataBlock(pReader)) {
-      SSDataBlock block = {0};
-      if (tqRetrieveDataBlock(&block, pReader) < 0) {
+      /*SSDataBlock block = {0};*/
+      /*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/
+      /*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/
+      /*}*/
+
+      taosArrayClear(pBlocks);
+      taosArrayClear(pSchemas);
+      if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) {
         if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;
       }
       if (pRsp->withTbName) {
         int64_t uid = pExec->pExecReader->msgIter.uid;
-        if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
-          blockDataFreeRes(&block);
+        if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) {
+          taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes);
+          taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper);
+          pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
+          pSchemas = taosArrayInit(0, sizeof(void*));
           continue;
         }
       }
@@ -255,25 +269,37 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
           pRsp->createTableNum++;
         }
       }
-      tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),
-                          pTq->pVnode->config.tsdbCfg.precision);
-      blockDataFreeRes(&block);
-      tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);
-      pRsp->blockNum++;
+      for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) {
+        SSDataBlock* pBlock = taosArrayGet(pBlocks, i);
+        tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock),
+                            pTq->pVnode->config.tsdbCfg.precision);
+        blockDataFreeRes(pBlock);
+        SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i);
+        taosArrayPush(pRsp->blockSchema, &pSW);
+        pRsp->blockNum++;
+      }
     }
   } else if (pExec->subType == TOPIC_SUB_TYPE__DB) {
     STqReader* pReader = pExec->pExecReader;
     tqReaderSetDataMsg(pReader, pReq, 0);
     while (tqNextDataBlockFilterOut(pReader, pExec->execDb.pFilterOutTbUid)) {
-      SSDataBlock block = {0};
-      if (tqRetrieveDataBlock(&block, pReader) < 0) {
+      /*SSDataBlock block = {0};*/
+      /*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/
+      /*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/
+      /*}*/
+      taosArrayClear(pBlocks);
+      taosArrayClear(pSchemas);
+      if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) {
         if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;
       }
       if (pRsp->withTbName) {
         int64_t uid = pExec->pExecReader->msgIter.uid;
-        if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
-          blockDataFreeRes(&block);
-          continue;
+        if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) {
+          taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes);
+          taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper);
+          pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
+          pSchemas = taosArrayInit(0, sizeof(void*));
+          return -1;
         }
       }
       if (pHandle->fetchMeta) {
@@ -291,14 +317,26 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
           pRsp->createTableNum++;
         }
       }
-      tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),
-                          pTq->pVnode->config.tsdbCfg.precision);
-      blockDataFreeRes(&block);
-      tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);
-      pRsp->blockNum++;
+      /*tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),*/
+      /*pTq->pVnode->config.tsdbCfg.precision);*/
+      /*blockDataFreeRes(&block);*/
+      /*tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);*/
+      /*pRsp->blockNum++;*/
+      for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) {
+        SSDataBlock* pBlock = taosArrayGet(pBlocks, i);
+        tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock),
+                            pTq->pVnode->config.tsdbCfg.precision);
+        blockDataFreeRes(pBlock);
+        SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i);
+        taosArrayPush(pRsp->blockSchema, &pSW);
+        pRsp->blockNum++;
+      }
     }
   }
 
+  taosArrayDestroy(pBlocks);
+  taosArrayDestroy(pSchemas);
+
   if (pRsp->blockNum == 0) {
     return -1;
   }

source/dnode/vnode/src/tq/tqPush.c

@@ -308,9 +308,8 @@ int tqPushMsg(STQ* pTq, void* msg, int32_t msgLen, tmsg_t msgType, int64_t ver)
   }
 
   if (vnodeIsRoleLeader(pTq->pVnode)) {
+    if (taosHashGetSize(pTq->pStreamMeta->pTasks) == 0) return 0;
     if (msgType == TDMT_VND_SUBMIT) {
-      if (taosHashGetSize(pTq->pStreamMeta->pTasks) == 0) return 0;
-
       void* data = taosMemoryMalloc(msgLen);
       if (data == NULL) {
         terrno = TSDB_CODE_OUT_OF_MEMORY;

source/dnode/vnode/src/tq/tqRead.c

@@ -556,7 +556,7 @@ FAIL:
   return -1;
 }
 
-int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* schemas) {
+int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas) {
   int32_t sversion = htonl(pReader->pBlock->sversion);
 
   if (pReader->cachedSchemaSuid == 0 || pReader->cachedSchemaVer != sversion ||
@@ -592,9 +592,10 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
   int32_t colAtMost = pSchemaWrapper->nCols;
 
   int32_t curRow = 0;
+  int32_t lastRow = 0;
 
   char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols);
-  if (assigned) return -1;
+  if (assigned == NULL) return -1;
 
   tInitSubmitBlkIter(&pReader->msgIter, pReader->pBlock, &pReader->blkIter);
   STSRowIter iter = {0};
@@ -605,11 +606,13 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
     bool buildNew = false;
     tdSTSRowIterReset(&iter, row);
 
+    tqDebug("vgId:%d, row of block %d", pReader->pWalReader->pWal->cfg.vgId, curRow);
     for (int32_t i = 0; i < colAtMost; i++) {
       SCellVal sVal = {0};
       if (!tdSTSRowIterFetch(&iter, pSchemaWrapper->pSchema[i].colId, pSchemaWrapper->pSchema[i].type, &sVal)) {
         break;
       }
+      tqDebug("vgId:%d, %d col, type %d", pReader->pWalReader->pWal->cfg.vgId, i, sVal.valType);
       if (curRow == 0) {
         assigned[i] = sVal.valType != TD_VTYPE_NONE;
         buildNew = true;
@@ -623,27 +626,42 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
     }
 
     if (buildNew) {
-      SSDataBlock    block;
-      SSchemaWrapper sw;
-      if (tqMaskBlock(&sw, &block, pSchemaWrapper, assigned) < 0) {
+      if (taosArrayGetSize(blocks) > 0) {
+        SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
+        pLastBlock->info.rows = curRow - lastRow;
+        lastRow = curRow;
+      }
+      SSDataBlock*    pBlock = createDataBlock();
+      SSchemaWrapper* pSW = taosMemoryCalloc(1, sizeof(SSchemaWrapper));
+      if (tqMaskBlock(pSW, pBlock, pSchemaWrapper, assigned) < 0) {
+        blockDataDestroy(pBlock);
         goto FAIL;
       }
+      SSDataBlock block = {0};
+      assignOneDataBlock(&block, pBlock);
+      blockDataDestroy(pBlock);
+
+      tqDebug("vgId:%d, build new block, col %d", pReader->pWalReader->pWal->cfg.vgId,
+              (int32_t)taosArrayGetSize(block.pDataBlock));
 
       taosArrayPush(blocks, &block);
-      taosArrayPush(schemas, &sw);
+      taosArrayPush(schemas, &pSW);
     }
 
     SSDataBlock* pBlock = taosArrayGetLast(blocks);
     pBlock->info.uid = pReader->msgIter.uid;
-    pBlock->info.rows = pReader->msgIter.numOfRows;
+    pBlock->info.rows = 0;
     pBlock->info.version = pReader->pMsg->version;
 
+    tqDebug("vgId:%d, taosx scan, block num: %d", pReader->pWalReader->pWal->cfg.vgId,
+            (int32_t)taosArrayGetSize(blocks));
+
     if (blockDataEnsureCapacity(pBlock, pReader->msgIter.numOfRows - curRow) < 0) {
       terrno = TSDB_CODE_OUT_OF_MEMORY;
       goto FAIL;
     }
 
-    tdSTSRowIterInit(&iter, pTschema);
+    tdSTSRowIterReset(&iter, row);
     for (int32_t i = 0; i < taosArrayGetSize(pBlock->pDataBlock); i++) {
       SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, i);
       SCellVal         sVal = {0};
@@ -654,12 +672,16 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
 
       ASSERT(sVal.valType != TD_VTYPE_NONE);
 
-      if (colDataAppend(pColData, curRow, sVal.val, sVal.valType != TD_VTYPE_NORM) < 0) {
+      if (colDataAppend(pColData, curRow, sVal.val, sVal.valType == TD_VTYPE_NULL) < 0) {
         goto FAIL;
       }
+      tqDebug("vgId:%d, row %d col %d append %d", pReader->pWalReader->pWal->cfg.vgId, curRow, i,
+              sVal.valType == TD_VTYPE_NULL);
     }
     curRow++;
   }
+  SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
+  pLastBlock->info.rows = curRow - lastRow;
 
   taosMemoryFree(assigned);
   return 0;

source/dnode/vnode/src/tq/tqSink.c

@@ -236,6 +236,7 @@ SSubmitReq* tqBlockToSubmit(SVnode* pVnode, const SArray* pBlocks, const STSchem
     SSDataBlock* pDataBlock = taosArrayGet(pBlocks, i);
     if (pDataBlock->info.type == STREAM_DELETE_RESULT) {
       pDeleteReq->suid = suid;
+      pDeleteReq->deleteReqs = taosArrayInit(0, sizeof(SSingleDeleteReq));
       tqBuildDeleteReq(pVnode, stbFullName, pDataBlock, pDeleteReq);
       continue;
     }
@@ -349,7 +350,6 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
           .contLen = len + sizeof(SMsgHead),
       };
       if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) {
-        rpcFreeCont(serializedDeleteReq);
         tqDebug("failed to put delete req into write-queue since %s", terrstr());
       }
     } else {
@@ -476,12 +476,12 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
 
       cap += sizeof(SSubmitBlk) + schemaLen + rows * maxLen;
 
-      SSubmitReq* ret = rpcMallocCont(cap);
-      ret->header.vgId = pVnode->config.vgId;
-      ret->length = sizeof(SSubmitReq);
-      ret->numOfBlocks = htonl(1);
+      SSubmitReq* pSubmit = rpcMallocCont(cap);
+      pSubmit->header.vgId = pVnode->config.vgId;
+      pSubmit->length = sizeof(SSubmitReq);
+      pSubmit->numOfBlocks = htonl(1);
 
-      SSubmitBlk* blkHead = POINTER_SHIFT(ret, sizeof(SSubmitReq));
+      SSubmitBlk* blkHead = POINTER_SHIFT(pSubmit, sizeof(SSubmitReq));
 
       blkHead->numOfRows = htonl(pDataBlock->info.rows);
       blkHead->sversion = htonl(pTSchema->version);
@@ -531,17 +531,16 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
       }
       blkHead->dataLen = htonl(dataLen);
 
-      ret->length += sizeof(SSubmitBlk) + schemaLen + dataLen;
-      ret->length = htonl(ret->length);
+      pSubmit->length += sizeof(SSubmitBlk) + schemaLen + dataLen;
+      pSubmit->length = htonl(pSubmit->length);
 
       SRpcMsg msg = {
           .msgType = TDMT_VND_SUBMIT,
-          .pCont = ret,
-          .contLen = ntohl(ret->length),
+          .pCont = pSubmit,
+          .contLen = ntohl(pSubmit->length),
       };
 
       if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) {
-        rpcFreeCont(ret);
         tqDebug("failed to put into write-queue since %s", terrstr());
       }
     }

source/dnode/vnode/src/tsdb/tsdbRead.c

@@ -4115,9 +4115,13 @@ int32_t tsdbRetrieveDatablockSMA(STsdbReader* pReader, SColumnDataAgg*** pBlockS
     } else if (pAgg->colId < pSup->colIds[j]) {
       i += 1;
     } else if (pSup->colIds[j] < pAgg->colId) {
+      if (pSup->colIds[j] == PRIMARYKEY_TIMESTAMP_COL_ID) {
+        taosArrayPush(pNewAggList, &pSup->tsColAgg);
+      } else {
       // all date in this block are null
-      SColumnDataAgg nullColAgg = {.colId = pSup->colIds[j], .numOfNull = pBlock->nRow};
-      taosArrayPush(pNewAggList, &nullColAgg);
+        SColumnDataAgg nullColAgg = {.colId = pSup->colIds[j], .numOfNull = pBlock->nRow};
+        taosArrayPush(pNewAggList, &nullColAgg);
+      }
       j += 1;
     }
   }

source/dnode/vnode/src/vnd/vnodeSvr.c

@@ -609,6 +609,7 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t version, void *pR
 _exit:
   for (int32_t iReq = 0; iReq < req.nReqs; iReq++) {
     pCreateReq = req.pReqs + iReq;
+    taosMemoryFree(pCreateReq->comment);
     taosArrayDestroy(pCreateReq->ctb.tagName);
   }
   taosArrayDestroyEx(rsp.pArray, tFreeSVCreateTbRsp);

source/libs/executor/inc/executorimpl.h

@@ -647,6 +647,23 @@ typedef struct SStreamPartitionOperatorInfo {
   SSDataBlock*          pDelRes;
 } SStreamPartitionOperatorInfo;
 
+typedef struct SStreamFillSupporter {
+  int32_t        type;  // fill type
+  SInterval      interval;
+  SResultRowData prev;
+  SResultRowData cur;
+  SResultRowData next;
+  SResultRowData nextNext;
+  SFillColInfo*  pAllColInfo;  // fill exprs and not fill exprs
+  SExprSupp      notFillExprSup;
+  int32_t        numOfAllCols;  // number of all exprs, including the tags columns
+  int32_t        numOfFillCols;
+  int32_t        numOfNotFillCols;
+  int32_t        rowSize;
+  SSHashObj*     pResMap;
+  bool           hasDelete;
+} SStreamFillSupporter;
+
 typedef struct SStreamFillOperatorInfo {
   SStreamFillSupporter* pFillSup;
   SSDataBlock*          pRes;

source/libs/executor/inc/tfill.h

@@ -111,22 +111,6 @@ typedef struct SStreamFillInfo {
   int32_t                delIndex;
 } SStreamFillInfo;
 
-typedef struct SStreamFillSupporter {
-  int32_t        type;  // fill type
-  SInterval      interval;
-  SResultRowData prev;
-  SResultRowData cur;
-  SResultRowData next;
-  SResultRowData nextNext;
-  SFillColInfo*  pAllColInfo;   // fill exprs and not fill exprs
-  int32_t        numOfAllCols;  // number of all exprs, including the tags columns
-  int32_t        numOfFillCols;
-  int32_t        numOfNotFillCols;
-  int32_t        rowSize;
-  SSHashObj*     pResMap;
-  bool           hasDelete;
-} SStreamFillSupporter;
-
 int64_t getNumOfResultsAfterFillGap(SFillInfo* pFillInfo, int64_t ekey, int32_t maxNumOfRows);
 
 void                 taosFillSetStartInfo(struct SFillInfo* pFillInfo, int32_t numOfRows, TSKEY endKey);

source/libs/executor/inc/tsort.h

@@ -36,12 +36,13 @@ typedef struct SMultiMergeSource {
 
 typedef struct SSortSource {
   SMultiMergeSource src;
-  union {
-    struct {
-      SArray* pageIdList;
-      int32_t pageIndex;
-    };
+  struct {
+    SArray* pageIdList;
+    int32_t pageIndex;
+  };
+  struct {
     void* param;
+    bool  onlyRef;
   };
 
 } SSortSource;

source/libs/executor/src/dataInserter.c

@@ -250,6 +250,8 @@ static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput,
     return code;
   }
 
+  taosArrayClear(pInserter->pDataBlocks);
+  
   code = sendSubmitRequest(pInserter, pMsg, pInserter->pParam->readHandle->pMsgCb->clientRpc, &pInserter->pNode->epSet);
   if (code) {
     return code;

source/libs/executor/src/executorimpl.c

@@ -1864,6 +1864,8 @@ void destroyExprInfo(SExprInfo* pExpr, int32_t numOfExprs) {
     for (int32_t j = 0; j < pExprInfo->base.numOfParams; ++j) {
       if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_COLUMN) {
         taosMemoryFreeClear(pExprInfo->base.pParam[j].pCol);
+      } else if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_VALUE) {
+        taosVariantDestroy(&pExprInfo->base.pParam[j].param);
       }
     }
 

source/libs/executor/src/scanoperator.c

@@ -2666,6 +2666,7 @@ int32_t startGroupTableMergeScan(SOperatorInfo* pOperator) {
     SSortSource*                    ps = taosMemoryCalloc(1, sizeof(SSortSource));
     STableMergeScanSortSourceParam* param = taosArrayGet(pInfo->sortSourceParams, i);
     ps->param = param;
+    ps->onlyRef = true;
     tsortAddSource(pInfo->pSortHandle, ps);
   }
 

source/libs/executor/src/sortoperator.c

@@ -188,10 +188,10 @@ int32_t doOpenSortOperator(SOperatorInfo* pOperator) {
 
   SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource));
   ps->param = pOperator->pDownstream[0];
+  ps->onlyRef = true;
   tsortAddSource(pInfo->pSortHandle, ps);
 
   int32_t code = tsortOpen(pInfo->pSortHandle);
-  taosMemoryFreeClear(ps);
 
   if (code != TSDB_CODE_SUCCESS) {
     T_LONG_JMP(pTaskInfo->env, terrno);
@@ -389,10 +389,10 @@ int32_t beginSortGroup(SOperatorInfo* pOperator) {
   param->childOpInfo = pOperator->pDownstream[0];
   param->grpSortOpInfo = pInfo;
   ps->param = param;
+  ps->onlyRef = false;
   tsortAddSource(pInfo->pCurrSortHandle, ps);
 
   int32_t code = tsortOpen(pInfo->pCurrSortHandle);
-  taosMemoryFreeClear(ps);
 
   if (code != TSDB_CODE_SUCCESS) {
     T_LONG_JMP(pTaskInfo->env, terrno);
@@ -483,6 +483,9 @@ void destroyGroupSortOperatorInfo(void* param) {
   taosArrayDestroy(pInfo->pSortInfo);
   taosArrayDestroy(pInfo->matchInfo.pList);
 
+  tsortDestroySortHandle(pInfo->pCurrSortHandle);
+  pInfo->pCurrSortHandle = NULL;
+
   taosMemoryFreeClear(param);
 }
 
@@ -575,6 +578,7 @@ int32_t doOpenMultiwayMergeOperator(SOperatorInfo* pOperator) {
   for (int32_t i = 0; i < pOperator->numOfDownstream; ++i) {
     SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource));
     ps->param = pOperator->pDownstream[i];
+    ps->onlyRef = true;
     tsortAddSource(pInfo->pSortHandle, ps);
   }
 

source/libs/executor/src/tfill.c

@@ -709,6 +709,7 @@ void* destroyStreamFillSupporter(SStreamFillSupporter* pFillSup) {
   pFillSup->pResMap = NULL;
   releaseOutputBuf(NULL, NULL, (SResultRow*)pFillSup->cur.pRowVal);
   pFillSup->cur.pRowVal = NULL;
+  cleanupExprSupp(&pFillSup->notFillExprSup);
 
   taosMemoryFree(pFillSup);
   return NULL;
@@ -1417,25 +1418,13 @@ static void doApplyStreamScalarCalculation(SOperatorInfo* pOperator, SSDataBlock
   blockDataEnsureCapacity(pDstBlock, pSrcBlock->info.rows);
   setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false);
   projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL);
-  pDstBlock->info.groupId = pSrcBlock->info.groupId;
-
-  SColumnInfoData* pDst = taosArrayGet(pDstBlock->pDataBlock, pInfo->primaryTsCol);
-  SColumnInfoData* pSrc = taosArrayGet(pSrcBlock->pDataBlock, pInfo->primarySrcSlotId);
-  colDataAssign(pDst, pSrc, pDstBlock->info.rows, &pDstBlock->info);
-
-  int32_t numOfNotFill = pInfo->pFillSup->numOfAllCols - pInfo->pFillSup->numOfFillCols;
-  for (int32_t i = 0; i < numOfNotFill; ++i) {
-    SFillColInfo* pCol = &pInfo->pFillSup->pAllColInfo[i + pInfo->pFillSup->numOfFillCols];
-    ASSERT(pCol->notFillCol);
 
-    SExprInfo* pExpr = pCol->pExpr;
-    int32_t    srcSlotId = pExpr->base.pParam[0].pCol->slotId;
-    int32_t    dstSlotId = pExpr->base.resSchema.slotId;
+  pDstBlock->info.rows = 0;
+  pSup = &pInfo->pFillSup->notFillExprSup;
+  setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false);
+  projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL);
+  pDstBlock->info.groupId = pSrcBlock->info.groupId;
 
-    SColumnInfoData* pDst1 = taosArrayGet(pDstBlock->pDataBlock, dstSlotId);
-    SColumnInfoData* pSrc1 = taosArrayGet(pSrcBlock->pDataBlock, srcSlotId);
-    colDataAssign(pDst1, pSrc1, pDstBlock->info.rows, &pDstBlock->info);
-  }
   blockDataUpdateTsWindow(pDstBlock, pInfo->primaryTsCol);
 }
 
@@ -1577,6 +1566,14 @@ static SStreamFillSupporter* initStreamFillSup(SStreamFillPhysiNode* pPhyFillNod
     destroyStreamFillSupporter(pFillSup);
     return NULL;
   }
+  
+  SExprInfo* noFillExpr = createExprInfo(pPhyFillNode->pNotFillExprs, NULL, &numOfNotFillCols);
+  code = initExprSupp(&pFillSup->notFillExprSup, noFillExpr, numOfNotFillCols);
+  if (code != TSDB_CODE_SUCCESS) {
+    destroyStreamFillSupporter(pFillSup);
+    return NULL;
+  }
+
   _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
   pFillSup->pResMap = tSimpleHashInit(16, hashFn);
   pFillSup->hasDelete = false;

source/libs/executor/src/timewindowoperator.c

@@ -1618,6 +1618,7 @@ void destroyStreamFinalIntervalOperatorInfo(void* param) {
   nodesDestroyNode((SNode*)pInfo->pPhyNode);
   colDataDestroy(&pInfo->twAggSup.timeWindowData);
   cleanupGroupResInfo(&pInfo->groupResInfo);
+  cleanupExprSupp(&pInfo->scalarSupp);
 
   taosMemoryFreeClear(param);
 }
@@ -2703,6 +2704,7 @@ static void rebuildIntervalWindow(SOperatorInfo* pOperator, SArray* pWinArray, S
                    pChildSup->rowEntryInfoOffset, &pChInfo->aggSup);
       updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &parentWin, true);
       compactFunctions(pSup->pCtx, pChildSup->pCtx, numOfOutput, pTaskInfo, &pInfo->twAggSup.timeWindowData);
+      releaseOutputBuf(pChInfo->pState, pWinRes, pChResult);
     }
     if (num > 0 && pUpdatedMap) {
       saveWinResultInfo(pCurResult->win.skey, pWinRes->groupId, pUpdatedMap);
@@ -5362,15 +5364,6 @@ SOperatorInfo* createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhys
   pInfo->primaryTsIndex = ((SColumnNode*)pIntervalPhyNode->window.pTspk)->slotId;
   initResultSizeInfo(&pOperator->resultInfo, 4096);
 
-  if (pIntervalPhyNode->window.pExprs != NULL) {
-    int32_t    numOfScalar = 0;
-    SExprInfo* pScalarExprInfo = createExprInfo(pIntervalPhyNode->window.pExprs, NULL, &numOfScalar);
-    code = initExprSupp(&pInfo->scalarSupp, pScalarExprInfo, numOfScalar);
-    if (code != TSDB_CODE_SUCCESS) {
-      goto _error;
-    }
-  }
-
   size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES;
   code = initAggInfo(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str);
   if (code != TSDB_CODE_SUCCESS) {

source/libs/executor/src/tsort.c

@@ -110,6 +110,22 @@ static int32_t sortComparCleanup(SMsortComparParam* cmpParam) {
   return TSDB_CODE_SUCCESS;
 }
 
+void tsortClearOrderdSource(SArray *pOrderedSource) {
+  for (size_t i = 0; i < taosArrayGetSize(pOrderedSource); i++) {
+    SSortSource** pSource = taosArrayGet(pOrderedSource, i);
+    if (NULL == *pSource) {
+      continue;
+    }
+    
+    if ((*pSource)->param && !(*pSource)->onlyRef) {
+      taosMemoryFree((*pSource)->param);  
+    }
+    taosMemoryFreeClear(*pSource);
+  }
+
+  taosArrayClear(pOrderedSource);
+}
+
 void tsortDestroySortHandle(SSortHandle* pSortHandle) {
   if (pSortHandle == NULL) {
     return;
@@ -123,10 +139,8 @@ void tsortDestroySortHandle(SSortHandle* pSortHandle) {
   destroyDiskbasedBuf(pSortHandle->pBuf);
   taosMemoryFreeClear(pSortHandle->idStr);
   blockDataDestroy(pSortHandle->pDataBlock);
-  for (size_t i = 0; i < taosArrayGetSize(pSortHandle->pOrderedSource); i++) {
-    SSortSource** pSource = taosArrayGet(pSortHandle->pOrderedSource, i);
-    taosMemoryFreeClear(*pSource);
-  }
+
+  tsortClearOrderdSource(pSortHandle->pOrderedSource);
   taosArrayDestroy(pSortHandle->pOrderedSource);
   taosMemoryFreeClear(pSortHandle);
 }
@@ -561,7 +575,7 @@ static int32_t doInternalMergeSort(SSortHandle* pHandle) {
       }
     }
 
-    taosArrayClear(pHandle->pOrderedSource);
+    tsortClearOrderdSource(pHandle->pOrderedSource);
     taosArrayAddAll(pHandle->pOrderedSource, pResList);
     taosArrayDestroy(pResList);
 
@@ -598,8 +612,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
   size_t sortBufSize = pHandle->numOfPages * pHandle->pageSize;
 
   if (pHandle->type == SORT_SINGLESOURCE_SORT) {
-    SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0);
-    taosArrayClear(pHandle->pOrderedSource);
+    SSortSource** pSource = taosArrayGet(pHandle->pOrderedSource, 0);
+    SSortSource* source = *pSource;
+    *pSource = NULL;
+    
+    tsortClearOrderdSource(pHandle->pOrderedSource);
 
     while (1) {
       SSDataBlock* pBlock = pHandle->fetchfp(source->param);
@@ -623,6 +640,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
 
       int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock);
       if (code != 0) {
+        if (source->param && !source->onlyRef) {
+          taosMemoryFree(source->param);
+        }
+        taosMemoryFree(source);
         return code;
       }
 
@@ -632,6 +653,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
         int64_t p = taosGetTimestampUs();
         code = blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo);
         if (code != 0) {
+          if (source->param && !source->onlyRef) {
+            taosMemoryFree(source->param);
+          }
+          taosMemoryFree(source);
           return code;
         }
 
@@ -642,6 +667,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
       }
     }
 
+    if (source->param && !source->onlyRef) {
+      taosMemoryFree(source->param);
+    }
+    taosMemoryFree(source);
+
     if (pHandle->pDataBlock != NULL && pHandle->pDataBlock->info.rows > 0) {
       size_t size = blockDataGetSize(pHandle->pDataBlock);
 

source/libs/qworker/src/qwMsg.c

@@ -65,19 +65,37 @@ int32_t qwBuildAndSendErrorRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t c
 int32_t qwBuildAndSendQueryRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t code, SQWTaskCtx *ctx) {
   STbVerInfo     *tbInfo = ctx ? &ctx->tbInfo : NULL;
   int64_t         affectedRows = ctx ? ctx->affectedRows : 0;
-  SQueryTableRsp *pRsp = (SQueryTableRsp *)rpcMallocCont(sizeof(SQueryTableRsp));
-  pRsp->code = htonl(code);
-  pRsp->affectedRows = htobe64(affectedRows);
+  SQueryTableRsp  rsp = {0};
+  rsp.code = code;
+  rsp.affectedRows = affectedRows;
+
   if (tbInfo) {
-    strcpy(pRsp->tbFName, tbInfo->tbFName);
-    pRsp->sversion = htonl(tbInfo->sversion);
-    pRsp->tversion = htonl(tbInfo->tversion);
+    strcpy(rsp.tbFName, tbInfo->tbFName);
+    rsp.sversion = tbInfo->sversion;
+    rsp.tversion = tbInfo->tversion;
+  }
+
+  int32_t msgSize = tSerializeSQueryTableRsp(NULL, 0, &rsp);
+  if (msgSize < 0) {
+    qError("tSerializeSQueryTableRsp failed");
+    QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+  
+  void *pRsp = rpcMallocCont(msgSize);
+  if (NULL == pRsp) {
+    qError("rpcMallocCont %d failed", msgSize);
+    QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+
+  if (tSerializeSQueryTableRsp(pRsp, msgSize, &rsp) < 0) {
+    qError("tSerializeSQueryTableRsp %d failed", msgSize);
+    QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
   }
 
   SRpcMsg rpcRsp = {
       .msgType = rspType,
       .pCont = pRsp,
-      .contLen = sizeof(*pRsp),
+      .contLen = msgSize,
       .code = code,
       .info = *pConn,
   };
@@ -182,23 +200,37 @@ int32_t qwBuildAndSendDropRsp(SRpcHandleInfo *pConn, int32_t code) {
 #endif
 
 int32_t qwBuildAndSendDropMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
-  STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq));
-  if (NULL == req) {
-    QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq));
+  STaskDropReq qMsg;
+  qMsg.header.vgId = mgmt->nodeId;
+  qMsg.header.contLen = 0;
+  qMsg.sId = sId;
+  qMsg.queryId = qId;
+  qMsg.taskId = tId;
+  qMsg.refId = rId;
+  qMsg.execId = eId;
+  
+  int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
+  if (msgSize < 0) {
+    QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
+    QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+  
+  void *msg = rpcMallocCont(msgSize);
+  if (NULL == msg) {
+    QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize);
+    QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+  
+  if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
+    QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
+    rpcFreeCont(msg);
     QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
   }
-
-  req->header.vgId = mgmt->nodeId;
-  req->sId = sId;
-  req->queryId = qId;
-  req->taskId = tId;
-  req->refId = rId;
-  req->execId = eId;
 
   SRpcMsg pNewMsg = {
       .msgType = TDMT_SCH_DROP_TASK,
-      .pCont = req,
-      .contLen = sizeof(STaskDropReq),
+      .pCont = msg,
+      .contLen = msgSize,
       .code = 0,
       .info = *pConn,
   };
@@ -247,22 +279,37 @@ int32_t qwBuildAndSendCQueryMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
 }
 
 int32_t qwRegisterQueryBrokenLinkArg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
-  STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq));
-  if (NULL == req) {
-    QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq));
+  STaskDropReq qMsg;
+  qMsg.header.vgId = mgmt->nodeId;
+  qMsg.header.contLen = 0;
+  qMsg.sId = sId;
+  qMsg.queryId = qId;
+  qMsg.taskId = tId;
+  qMsg.refId = rId;
+  qMsg.execId = eId;
+  
+  int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
+  if (msgSize < 0) {
+    QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
+    QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+  
+  void *msg = rpcMallocCont(msgSize);
+  if (NULL == msg) {
+    QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize);
+    QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+  }
+  
+  if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
+    QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
+    rpcFreeCont(msg);
     QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
   }
-
-  req->header.vgId = htonl(mgmt->nodeId);
-  req->sId = htobe64(sId);
-  req->queryId = htobe64(qId);
-  req->taskId = htobe64(tId);
-  req->refId = htobe64(rId);
 
   SRpcMsg brokenMsg = {
       .msgType = TDMT_SCH_DROP_TASK,
-      .pCont = req,
-      .contLen = sizeof(STaskDropReq),
+      .pCont = msg,
+      .contLen = msgSize,
       .code = TSDB_CODE_RPC_BROKEN_LINK,
       .info = *pConn,
   };
@@ -312,40 +359,33 @@ int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg, bool chkGran
   }
 
   int32_t       code = 0;
-  SSubQueryMsg *msg = pMsg->pCont;
   SQWorker     *mgmt = (SQWorker *)qWorkerMgmt;
-
-  if (NULL == msg || pMsg->contLen <= sizeof(*msg)) {
-    QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
+  SSubQueryMsg  msg = {0};
+  if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
+    QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
     QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
   }
 
-  msg->sId = be64toh(msg->sId);
-  msg->queryId = be64toh(msg->queryId);
-  msg->taskId = be64toh(msg->taskId);
-  msg->refId = be64toh(msg->refId);
-  msg->execId = ntohl(msg->execId);
-  msg->phyLen = ntohl(msg->phyLen);
-  msg->sqlLen = ntohl(msg->sqlLen);
-  msg->msgMask = ntohl(msg->msgMask);
-
-  if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg->msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
-    QW_ELOG("query failed cause of grant expired, msgMask:%d", msg->msgMask);
+  if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg.msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
+    QW_ELOG("query failed cause of grant expired, msgMask:%d", msg.msgMask);
+    tFreeSSubQueryMsg(&msg);
     QW_ERR_RET(TSDB_CODE_GRANT_EXPIRED);
   }
 
-  uint64_t sId = msg->sId;
-  uint64_t qId = msg->queryId;
-  uint64_t tId = msg->taskId;
-  int64_t  rId = msg->refId;
-  int32_t  eId = msg->execId;
+  uint64_t sId = msg.sId;
+  uint64_t qId = msg.queryId;
+  uint64_t tId = msg.taskId;
+  int64_t  rId = msg.refId;
+  int32_t  eId = msg.execId;
 
   SQWMsg qwMsg = {
-      .msgType = pMsg->msgType, .msg = msg->msg + msg->sqlLen, .msgLen = msg->phyLen, .connInfo = pMsg->info};
+      .msgType = pMsg->msgType, .msg = msg.msg, .msgLen = msg.msgLen, .connInfo = pMsg->info};
 
-  QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p", pMsg->info.handle);
-  QW_ERR_RET(qwPreprocessQuery(QW_FPARAMS(), &qwMsg));
-  QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p", pMsg->info.handle);
+  QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p, SQL:%s", pMsg->info.handle, msg.sql);
+  code = qwPreprocessQuery(QW_FPARAMS(), &qwMsg);
+  QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p, code:%x", pMsg->info.handle, code);
+
+  tFreeSSubQueryMsg(&msg);
 
   return TSDB_CODE_SUCCESS;
 }
@@ -355,19 +395,25 @@ int32_t qWorkerAbortPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) {
     QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
   }
 
-  SSubQueryMsg *msg = pMsg->pCont;
   SQWorker     *mgmt = (SQWorker *)qWorkerMgmt;
+  SSubQueryMsg msg = {0};
+  if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
+    QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
+    QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
+  }
 
-  uint64_t sId = msg->sId;
-  uint64_t qId = msg->queryId;
-  uint64_t tId = msg->taskId;
-  int64_t  rId = msg->refId;
-  int32_t  eId = msg->execId;
+  uint64_t sId = msg.sId;
+  uint64_t qId = msg.queryId;
+  uint64_t tId = msg.taskId;
+  int64_t  rId = msg.refId;
+  int32_t  eId = msg.execId;
 
   QW_SCH_TASK_DLOG("Abort prerocessQuery start, handle:%p", pMsg->info.handle);
   qwAbortPrerocessQuery(QW_FPARAMS());
   QW_SCH_TASK_DLOG("Abort prerocessQuery end, handle:%p", pMsg->info.handle);
 
+  tFreeSSubQueryMsg(&msg);
+
   return TSDB_CODE_SUCCESS;
 }
 
@@ -377,42 +423,41 @@ int32_t qWorkerProcessQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int
   }
 
   int32_t       code = 0;
-  SSubQueryMsg *msg = pMsg->pCont;
   SQWorker     *mgmt = (SQWorker *)qWorkerMgmt;
 
   qwUpdateTimeInQueue(mgmt, ts, QUERY_QUEUE);
   QW_STAT_INC(mgmt->stat.msgStat.queryProcessed, 1);
 
-  if (NULL == msg || pMsg->contLen <= sizeof(*msg)) {
-    QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
+  SSubQueryMsg  msg = {0};
+  if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
+    QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
     QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
   }
 
-  uint64_t sId = msg->sId;
-  uint64_t qId = msg->queryId;
-  uint64_t tId = msg->taskId;
-  int64_t  rId = msg->refId;
-  int32_t  eId = msg->execId;
+  uint64_t sId = msg.sId;
+  uint64_t qId = msg.queryId;
+  uint64_t tId = msg.taskId;
+  int64_t  rId = msg.refId;
+  int32_t  eId = msg.execId;
 
   SQWMsg qwMsg = {.node = node,
-                  .msg = msg->msg + msg->sqlLen,
-                  .msgLen = msg->phyLen,
+                  .msg = msg.msg,
+                  .msgLen = msg.msgLen,
                   .connInfo = pMsg->info,
                   .msgType = pMsg->msgType};
-  qwMsg.msgInfo.explain = msg->explain;
-  qwMsg.msgInfo.taskType = msg->taskType;
-  qwMsg.msgInfo.needFetch = msg->needFetch;
+  qwMsg.msgInfo.explain = msg.explain;
+  qwMsg.msgInfo.taskType = msg.taskType;
+  qwMsg.msgInfo.needFetch = msg.needFetch;
 
-  char *sql = strndup(msg->msg, msg->sqlLen);
   QW_SCH_TASK_DLOG("processQuery start, node:%p, type:%s, handle:%p, SQL:%s", node, TMSG_INFO(pMsg->msgType),
-                   pMsg->info.handle, sql);
-  QW_ERR_JRET(qwProcessQuery(QW_FPARAMS(), &qwMsg, sql));
+                   pMsg->info.handle, msg.sql);
+  code = qwProcessQuery(QW_FPARAMS(), &qwMsg, msg.sql);
+  msg.sql = NULL;
+  QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%x", node, code);
 
-_return:
-
-  QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%d", node, code);
+  tFreeSSubQueryMsg(&msg);
 
-  return code;
+  return TSDB_CODE_SUCCESS;
 }
 
 int32_t qWorkerProcessCQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts) {
@@ -548,28 +593,22 @@ int32_t qWorkerProcessDropMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int6
   }
 
   int32_t       code = 0;
-  STaskDropReq *msg = pMsg->pCont;
   SQWorker     *mgmt = (SQWorker *)qWorkerMgmt;
 
   qwUpdateTimeInQueue(mgmt, ts, FETCH_QUEUE);
   QW_STAT_INC(mgmt->stat.msgStat.dropProcessed, 1);
 
-  if (NULL == msg || pMsg->contLen < sizeof(*msg)) {
-    QW_ELOG("invalid task drop msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
+  STaskDropReq  msg = {0};
+  if (tDeserializeSTaskDropReq(pMsg->pCont, pMsg->contLen, &msg) < 0) {
+    QW_ELOG("tDeserializeSTaskDropReq failed, contLen:%d", pMsg->contLen);
     QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
   }
 
-  msg->sId = be64toh(msg->sId);
-  msg->queryId = be64toh(msg->queryId);
-  msg->taskId = be64toh(msg->taskId);
-  msg->refId = be64toh(msg->refId);
-  msg->execId = ntohl(msg->execId);
-
-  uint64_t sId = msg->sId;
-  uint64_t qId = msg->queryId;
-  uint64_t tId = msg->taskId;
-  int64_t  rId = msg->refId;
-  int32_t  eId = msg->execId;
+  uint64_t sId = msg.sId;
+  uint64_t qId = msg.queryId;
+  uint64_t tId = msg.taskId;
+  int64_t  rId = msg.refId;
+  int32_t  eId = msg.execId;
 
   SQWMsg qwMsg = {.node = node, .msg = NULL, .msgLen = 0, .code = pMsg->code, .connInfo = pMsg->info};
 

source/libs/qworker/test/qworkerTests.cpp

@@ -114,7 +114,7 @@ void qwtBuildQueryReqMsg(SRpcMsg *queryRpc) {
   qwtqueryMsg.queryId = htobe64(atomic_add_fetch_64(&qwtTestQueryId, 1));
   qwtqueryMsg.sId = htobe64(1);
   qwtqueryMsg.taskId = htobe64(1);
-  qwtqueryMsg.phyLen = htonl(100);
+  qwtqueryMsg.msgLen = htonl(100);
   qwtqueryMsg.sqlLen = 0;
   queryRpc->msgType = TDMT_SCH_QUERY;
   queryRpc->pCont = &qwtqueryMsg;
@@ -131,12 +131,29 @@ void qwtBuildFetchReqMsg(SResFetchReq *fetchMsg, SRpcMsg *fetchRpc) {
 }
 
 void qwtBuildDropReqMsg(STaskDropReq *dropMsg, SRpcMsg *dropRpc) {
-  dropMsg->sId = htobe64(1);
-  dropMsg->queryId = htobe64(atomic_load_64(&qwtTestQueryId));
-  dropMsg->taskId = htobe64(1);
+  dropMsg->sId = 1;
+  dropMsg->queryId = atomic_load_64(&qwtTestQueryId);
+  dropMsg->taskId = 1;
+  
+  int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, dropMsg);
+  if (msgSize < 0) {
+    return;
+  }
+  
+  char *msg = (char*)taosMemoryCalloc(1, msgSize);
+  if (NULL == msg) {
+    return;
+  }
+  
+  if (tSerializeSTaskDropReq(msg, msgSize, dropMsg) < 0) {
+    taosMemoryFree(msg);
+    return;
+  }
+
+
   dropRpc->msgType = TDMT_SCH_DROP_TASK;
-  dropRpc->pCont = dropMsg;
-  dropRpc->contLen = sizeof(STaskDropReq);
+  dropRpc->pCont = msg;
+  dropRpc->contLen = msgSize;
 }
 
 int32_t qwtStringToPlan(const char *str, SSubplan **subplan) {

source/libs/scalar/src/sclfunc.c

@@ -24,6 +24,8 @@ static double tlog2(double v, double base) {
     return a;
   } else if (isnan(b) || isinf(b)) {
     return b;
+  } else if (b == 0) {
+    return INFINITY;
   } else {
     return a / b;
   }

source/libs/scheduler/src/schRemote.c

@@ -334,17 +334,17 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa
         SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_INPUT);
       }
 
-      SQueryTableRsp *rsp = (SQueryTableRsp *)msg;
-      rsp->code = ntohl(rsp->code);
-      rsp->sversion = ntohl(rsp->sversion);
-      rsp->tversion = ntohl(rsp->tversion);
-      rsp->affectedRows = be64toh(rsp->affectedRows);
-
-      SCH_ERR_JRET(rsp->code);
+      SQueryTableRsp rsp = {0};
+      if (tDeserializeSQueryTableRsp(msg, msgSize, &rsp) < 0) {
+        SCH_TASK_ELOG("tDeserializeSQueryTableRsp failed, msgSize:%d", msgSize);
+        SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_MSG);
+      }
+      
+      SCH_ERR_JRET(rsp.code);
 
-      SCH_ERR_JRET(schSaveJobExecRes(pJob, rsp));
+      SCH_ERR_JRET(schSaveJobExecRes(pJob, &rsp));
 
-      atomic_add_fetch_32(&pJob->resNumOfRows, rsp->affectedRows);
+      atomic_add_fetch_32(&pJob->resNumOfRows, rsp.affectedRows);
 
       taosMemoryFreeClear(msg);
 
@@ -1042,30 +1042,40 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
     case TDMT_SCH_MERGE_QUERY: {
       SCH_ERR_RET(schMakeQueryRpcCtx(pJob, pTask, &rpcCtx));
 
-      uint32_t len = strlen(pJob->sql);
-      msgSize = sizeof(SSubQueryMsg) + pTask->msgLen + len;
+      SSubQueryMsg qMsg;
+      qMsg.header.vgId = addr->nodeId;
+      qMsg.header.contLen = 0;
+      qMsg.sId = schMgmt.sId;
+      qMsg.queryId = pJob->queryId;
+      qMsg.taskId = pTask->taskId;
+      qMsg.refId = pJob->refId;
+      qMsg.execId = pTask->execId;
+      qMsg.msgMask = (pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0;
+      qMsg.taskType = TASK_TYPE_TEMP;
+      qMsg.explain = SCH_IS_EXPLAIN_JOB(pJob);
+      qMsg.needFetch = SCH_TASK_NEED_FETCH(pTask);
+      qMsg.sqlLen = strlen(pJob->sql);
+      qMsg.sql = pJob->sql;
+      qMsg.msgLen = pTask->msgLen;
+      qMsg.msg = pTask->msg;
+
+      msgSize = tSerializeSSubQueryMsg(NULL, 0, &qMsg);
+      if (msgSize < 0) {
+        SCH_TASK_ELOG("tSerializeSSubQueryMsg get size, msgSize:%d", msgSize);
+        SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+      }
+      
       msg = taosMemoryCalloc(1, msgSize);
       if (NULL == msg) {
         SCH_TASK_ELOG("calloc %d failed", msgSize);
         SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
       }
 
-      SSubQueryMsg *pMsg = msg;
-      pMsg->header.vgId = htonl(addr->nodeId);
-      pMsg->sId = htobe64(schMgmt.sId);
-      pMsg->queryId = htobe64(pJob->queryId);
-      pMsg->taskId = htobe64(pTask->taskId);
-      pMsg->refId = htobe64(pJob->refId);
-      pMsg->execId = htonl(pTask->execId);
-      pMsg->taskType = TASK_TYPE_TEMP;
-      pMsg->explain = SCH_IS_EXPLAIN_JOB(pJob);
-      pMsg->needFetch = SCH_TASK_NEED_FETCH(pTask);
-      pMsg->phyLen = htonl(pTask->msgLen);
-      pMsg->sqlLen = htonl(len);
-      pMsg->msgMask = htonl((pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0);
-
-      memcpy(pMsg->msg, pJob->sql, len);
-      memcpy(pMsg->msg + len, pTask->msg, pTask->msgLen);
+      if (tSerializeSSubQueryMsg(msg, msgSize, &qMsg) < 0) {
+        SCH_TASK_ELOG("tSerializeSSubQueryMsg failed, msgSize:%d", msgSize);
+        taosMemoryFree(msg);
+        SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+      }
 
       persistHandle = true;
       SCH_SET_TASK_HANDLE(pTask, rpcAllocHandle());
@@ -1092,22 +1102,32 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
       break;
     }
     case TDMT_SCH_DROP_TASK: {
-      msgSize = sizeof(STaskDropReq);
+      STaskDropReq qMsg;
+      qMsg.header.vgId = addr->nodeId;
+      qMsg.header.contLen = 0;
+      qMsg.sId = schMgmt.sId;
+      qMsg.queryId = pJob->queryId;
+      qMsg.taskId = pTask->taskId;
+      qMsg.refId = pJob->refId;
+      qMsg.execId = pTask->execId;
+
+      msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
+      if (msgSize < 0) {
+        SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
+        SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+      }
+      
       msg = taosMemoryCalloc(1, msgSize);
       if (NULL == msg) {
         SCH_TASK_ELOG("calloc %d failed", msgSize);
         SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
       }
 
-      STaskDropReq *pMsg = msg;
-
-      pMsg->header.vgId = htonl(addr->nodeId);
-
-      pMsg->sId = htobe64(schMgmt.sId);
-      pMsg->queryId = htobe64(pJob->queryId);
-      pMsg->taskId = htobe64(pTask->taskId);
-      pMsg->refId = htobe64(pJob->refId);
-      pMsg->execId = htonl(pTask->execId);
+      if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
+        SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
+        taosMemoryFree(msg);
+        SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
+      }
       break;
     }
     case TDMT_SCH_QUERY_HEARTBEAT: {

source/libs/sync/src/syncAppendEntriesReply.c

@@ -67,6 +67,9 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
       if (pMsg->matchIndex > oldMatchIndex) {
         syncIndexMgrSetIndex(ths->pMatchIndex, &(pMsg->srcId), pMsg->matchIndex);
         syncMaybeAdvanceCommitIndex(ths);
+
+        // maybe update minMatchIndex
+        ths->minMatchIndex = syncMinMatchIndex(ths);
       }
       syncIndexMgrSetIndex(ths->pNextIndex, &(pMsg->srcId), pMsg->matchIndex + 1);
 

source/libs/sync/src/syncMain.c

@@ -243,6 +243,18 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) {
     goto _DEL_WAL;
 
   } else {
+    lastApplyIndex -= SYNC_VNODE_LOG_RETENTION;
+
+    SyncIndex beginIndex = pSyncNode->pLogStore->syncLogBeginIndex(pSyncNode->pLogStore);
+    SyncIndex endIndex = pSyncNode->pLogStore->syncLogEndIndex(pSyncNode->pLogStore);
+    bool      isEmpty = pSyncNode->pLogStore->syncLogIsEmpty(pSyncNode->pLogStore);
+
+    if (isEmpty || !(lastApplyIndex >= beginIndex && lastApplyIndex <= endIndex)) {
+      sNTrace(pSyncNode, "new-snapshot-index:%" PRId64 ", empty:%d, do not delete wal", lastApplyIndex, isEmpty);
+      syncNodeRelease(pSyncNode);
+      return 0;
+    }
+
     // vnode
     if (pSyncNode->replicaNum > 1) {
       // multi replicas
@@ -300,26 +312,31 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) {
 _DEL_WAL:
 
   do {
-    SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex);
-
-    if (snapshottingIndex == SYNC_INDEX_INVALID) {
-      atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex);
-      pSyncNode->snapshottingTime = taosGetTimestampMs();
+    SSyncLogStoreData* pData = pSyncNode->pLogStore->data;
+    SyncIndex          snapshotVer = walGetSnapshotVer(pData->pWal);
+    SyncIndex          walCommitVer = walGetCommittedVer(pData->pWal);
+    SyncIndex          wallastVer = walGetLastVer(pData->pWal);
+    if (lastApplyIndex <= walCommitVer) {
+      SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex);
+
+      if (snapshottingIndex == SYNC_INDEX_INVALID) {
+        atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex);
+        pSyncNode->snapshottingTime = taosGetTimestampMs();
+
+        code = walBeginSnapshot(pData->pWal, lastApplyIndex);
+        if (code == 0) {
+          sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64,
+                  pSyncNode->snapshottingIndex, lastApplyIndex);
+        } else {
+          sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64,
+                  terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex);
+          atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID);
+        }
 
-      SSyncLogStoreData* pData = pSyncNode->pLogStore->data;
-      code = walBeginSnapshot(pData->pWal, lastApplyIndex);
-      if (code == 0) {
-        sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64,
-                pSyncNode->snapshottingIndex, lastApplyIndex);
       } else {
-        sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64,
-                terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex);
-        atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID);
+        sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64,
+                snapshottingIndex, lastApplyIndex);
       }
-
-    } else {
-      sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64,
-              snapshottingIndex, lastApplyIndex);
     }
   } while (0);
 

source/libs/sync/src/syncRaftLog.c

@@ -375,7 +375,17 @@ static int32_t raftLogGetLastEntry(SSyncLogStore* pLogStore, SSyncRaftEntry** pp
 int32_t raftLogUpdateCommitIndex(SSyncLogStore* pLogStore, SyncIndex index) {
   SSyncLogStoreData* pData = pLogStore->data;
   SWal*              pWal = pData->pWal;
-  // ASSERT(walCommit(pWal, index) == 0);
+
+  // need not update
+  SyncIndex snapshotVer = walGetSnapshotVer(pWal);
+  SyncIndex walCommitVer = walGetCommittedVer(pWal);
+  SyncIndex wallastVer = walGetLastVer(pWal);
+
+  if (index < snapshotVer || index > wallastVer) {
+    // ignore
+    return 0;
+  }
+
   int32_t code = walCommit(pWal, index);
   if (code != 0) {
     int32_t     err = terrno;

source/libs/sync/src/syncTimeout.c

@@ -62,18 +62,20 @@ static int32_t syncNodeTimerRoutine(SSyncNode* ths) {
     syncNodeCleanConfigIndex(ths);
   }
 
-  // end timeout wal snapshot
   int64_t timeNow = taosGetTimestampMs();
-  if (timeNow - ths->snapshottingIndex > SYNC_DEL_WAL_MS &&
-      atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
-    SSyncLogStoreData* pData = ths->pLogStore->data;
-    int32_t            code = walEndSnapshot(pData->pWal);
-    if (code != 0) {
-      sNError(ths, "timer wal snapshot end error since:%s", terrstr());
-      return -1;
-    } else {
-      sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex));
-      atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID);
+  if (atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
+    // end timeout wal snapshot
+    if (timeNow - ths->snapshottingTime > SYNC_DEL_WAL_MS &&
+        atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
+      SSyncLogStoreData* pData = ths->pLogStore->data;
+      int32_t            code = walEndSnapshot(pData->pWal);
+      if (code != 0) {
+        sNError(ths, "timer wal snapshot end error since:%s", terrstr());
+        return -1;
+      } else {
+        sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex));
+        atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID);
+      }
     }
   }
 

source/libs/sync/src/syncUtil.c

@@ -239,11 +239,11 @@ void syncPrintNodeLog(const char* flags, ELogLevel level, int32_t dflag, SSyncNo
                  "vgId:%d, sync %s "
                  "%s"
                  ", tm:%" PRIu64 ", cmt:%" PRId64 ", fst:%" PRId64 ", lst:%" PRId64 ", min:%" PRId64 ", snap:%" PRId64
-                 ", snap-tm:%" PRIu64 ", sby:%d, aq:%d, bch:%d, r-num:%d, lcfg:%" PRId64
+                 ", snap-tm:%" PRIu64 ", sby:%d, aq:%d, snaping:%" PRId64 ", r-num:%d, lcfg:%" PRId64
                  ", chging:%d, rsto:%d, dquorum:%d, elt:%" PRId64 ", hb:%" PRId64 ", %s, %s",
                  pNode->vgId, syncStr(pNode->state), eventLog, currentTerm, pNode->commitIndex, logBeginIndex,
                  logLastIndex, pNode->minMatchIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
-                 pNode->pRaftCfg->isStandBy, aqItems, pNode->pRaftCfg->batchSize, pNode->replicaNum,
+                 pNode->pRaftCfg->isStandBy, aqItems, pNode->snapshottingIndex, pNode->replicaNum,
                  pNode->pRaftCfg->lastConfigIndex, pNode->changing, pNode->restoreFinish, quorum,
                  pNode->electTimerLogicClock, pNode->heartbeatTimerLogicClockUser, peerStr, cfgStr);
   }

source/libs/transport/src/transCli.c

@@ -791,6 +791,7 @@ void cliSend(SCliConn* pConn) {
 
   int            msgLen = transMsgLenFromCont(pMsg->contLen);
   STransMsgHead* pHead = transHeadFromCont(pMsg->pCont);
+
   pHead->ahandle = pCtx != NULL ? (uint64_t)pCtx->ahandle : 0;
   pHead->noResp = REQUEST_NO_RESP(pMsg) ? 1 : 0;
   pHead->persist = REQUEST_PERSIS_HANDLE(pMsg) ? 1 : 0;
@@ -820,10 +821,15 @@ void cliSend(SCliConn* pConn) {
     uv_timer_start((uv_timer_t*)pConn->timer, cliReadTimeoutCb, TRANS_READ_TIMEOUT, 0);
   }
 
-  if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) {
-    msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead);
-    pHead->msgLen = (int32_t)htonl((uint32_t)msgLen);
+  if (pHead->comp == 0) {
+    if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) {
+      msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead);
+      pHead->msgLen = (int32_t)htonl((uint32_t)msgLen);
+    }
+  } else {
+    msgLen = (int32_t)ntohl((uint32_t)(pHead->msgLen));
   }
+
   tGDebug("%s conn %p %s is sent to %s, local info %s, len:%d", CONN_GET_INST_LABEL(pConn), pConn,
           TMSG_INFO(pHead->msgType), pConn->dst, pConn->src, msgLen);
 

source/libs/transport/src/transSvr.c

@@ -398,7 +398,7 @@ static int uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) {
   pHead->magicNum = htonl(TRANS_MAGIC_NUM);
 
   // handle invalid drop_task resp, TD-20098
-  if (pMsg->msgType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) {
+  if (pConn->inType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) {
     transQueuePop(&pConn->srvMsgs);
     destroySmsg(smsg);
     return -1;

source/libs/wal/src/walWrite.c

@@ -324,23 +324,36 @@ int32_t walEndSnapshot(SWal *pWal) {
   // find files safe to delete
   SWalFileInfo *pInfo = taosArraySearch(pWal->fileInfoSet, &tmp, compareWalFileInfo, TD_LE);
   if (pInfo) {
+    SWalFileInfo *pLastFileInfo = taosArrayGetLast(pWal->fileInfoSet);
+    wDebug("vgId:%d, wal search found file info: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
+           pInfo->lastVer);
     if (ver >= pInfo->lastVer) {
-      pInfo--;
+      pInfo++;
+      wDebug("vgId:%d, wal remove advance one file: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
+             pInfo->lastVer);
     }
-    if (POINTER_DISTANCE(pInfo, pWal->fileInfoSet->pData) > 0) {
-      wDebug("vgId:%d, wal end remove for %" PRId64, pWal->cfg.vgId, pInfo->firstVer);
+    if (pInfo <= pLastFileInfo) {
+      wDebug("vgId:%d, wal end remove for first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
+             pInfo->lastVer);
     } else {
       wDebug("vgId:%d, wal no remove", pWal->cfg.vgId);
     }
     // iterate files, until the searched result
     for (SWalFileInfo *iter = pWal->fileInfoSet->pData; iter < pInfo; iter++) {
-      if ((pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize) ||
-          (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts)) {
+      wDebug("vgId:%d, wal check remove file %" PRId64 "(file size %" PRId64 " close ts %" PRId64
+             "), new tot size %" PRId64,
+             pWal->cfg.vgId, iter->firstVer, iter->fileSize, iter->closeTs, newTotSize);
+      if (((pWal->cfg.retentionSize == 0) || (pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize)) ||
+          ((pWal->cfg.retentionPeriod == 0) ||
+           (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts))) {
         // delete according to file size or close time
+        wDebug("vgId:%d, check pass", pWal->cfg.vgId);
         deleteCnt++;
         newTotSize -= iter->fileSize;
       }
+      wDebug("vgId:%d, check not pass", pWal->cfg.vgId);
     }
+    wDebug("vgId:%d, wal should delete %d files", pWal->cfg.vgId, deleteCnt);
     int32_t actualDelete = 0;
     char    fnameStr[WAL_FILE_LEN];
     // remove file

source/util/src/tpagedbuf.c

@@ -107,7 +107,7 @@ static uint64_t allocatePositionInFile(SDiskbasedBuf* pBuf, size_t size) {
 
 static void setPageNotInBuf(SPageInfo* pPageInfo) { pPageInfo->pData = NULL; }
 
-static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + 2; }
+static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + sizeof(SFilePage); }
 
 /**
  *   +--------------------------+-------------------+--------------+

tests/parallel_test/cases.task

@@ -216,7 +216,7 @@
 ,,y,script,./test.sh -f tsim/stream/drop_stream.sim
 ,,y,script,./test.sh -f tsim/stream/fillHistoryBasic1.sim
 ,,y,script,./test.sh -f tsim/stream/fillHistoryBasic2.sim
-,,n,script,./test.sh -f tsim/stream/fillHistoryBasic3.sim
+,,y,script,./test.sh -f tsim/stream/fillHistoryBasic3.sim
 ,,y,script,./test.sh -f tsim/stream/distributeInterval0.sim
 ,,y,script,./test.sh -f tsim/stream/distributeIntervalRetrive0.sim
 ,,y,script,./test.sh -f tsim/stream/distributeSession0.sim
@@ -227,7 +227,7 @@
 ,,y,script,./test.sh -f tsim/stream/triggerSession0.sim
 ,,y,script,./test.sh -f tsim/stream/partitionby.sim
 ,,y,script,./test.sh -f tsim/stream/partitionby1.sim
-,,n,script,./test.sh -f tsim/stream/schedSnode.sim
+,,y,script,./test.sh -f tsim/stream/schedSnode.sim
 ,,y,script,./test.sh -f tsim/stream/windowClose.sim
 ,,y,script,./test.sh -f tsim/stream/ignoreExpiredData.sim
 ,,y,script,./test.sh -f tsim/stream/sliding.sim
@@ -416,208 +416,208 @@
 ,,,system-test,python3 ./test.py -f 0-others/sysinfo.py
 ,,,system-test,python3 ./test.py -f 0-others/user_control.py
 ,,,system-test,python3 ./test.py -f 0-others/fsync.py
-,,,system-test,python3 ./test.py  -f 0-others/compatibility.py
+,,,system-test,python3 ./test.py -f 0-others/compatibility.py
 ,,,system-test,python3 ./test.py -f 1-insert/alter_database.py
 ,,,system-test,python3 ./test.py -f 1-insert/influxdb_line_taosc_insert.py
-,,,system-test,python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py
 ,,,system-test,python3 ./test.py -f 1-insert/opentsdb_json_taosc_insert.py
 ,,,system-test,python3 ./test.py -f 1-insert/test_stmt_muti_insert_query.py
 ,,,system-test,python3 ./test.py -f 1-insert/test_stmt_set_tbname_tag.py
-,,,system-test,python3 ./test.py -f 1-insert/alter_stable.py
-,,,system-test,python3 ./test.py -f 1-insert/alter_table.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_stable.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_table.py
 ,,,system-test,python3 ./test.py -f 1-insert/boundary.py
 ,,,system-test,python3 ./test.py -f 1-insert/insertWithMoreVgroup.py
 ,,,system-test,python3 ./test.py -f 1-insert/table_comment.py
 ,,,system-test,python3 ./test.py -f 1-insert/time_range_wise.py
 ,,,system-test,python3 ./test.py -f 1-insert/block_wise.py
 ,,,system-test,python3 ./test.py -f 1-insert/create_retentions.py
-,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py
 ,,,system-test,python3 ./test.py -f 1-insert/mutil_stage.py
-,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py -R
-,,,system-test,python3 ./test.py -f 1-insert/update_data_muti_rows.py
-,,,system-test,python3 ./test.py -f 1-insert/db_tb_name_check.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/update_data_muti_rows.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/db_tb_name_check.py
 ,,,system-test,python3 ./test.py -f 1-insert/database_pre_suf.py
 ,,,system-test,python3 ./test.py -f 1-insert/InsertFuturets.py
-,,,system-test,python3 ./test.py -f 0-others/show.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/show.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py
-,,,system-test,python3 ./test.py -f 2-query/abs.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py
-,,,system-test,python3 ./test.py -f 2-query/and_or_for_byte.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py
-,,,system-test,python3 ./test.py -f 2-query/apercentile.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py
-,,,system-test,python3 ./test.py -f 2-query/arccos.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py
-,,,system-test,python3 ./test.py -f 2-query/arcsin.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py
-,,,system-test,python3 ./test.py -f 2-query/arctan.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -R
 ,,,system-test,python3 ./test.py -f 2-query/avg.py
 ,,,system-test,python3 ./test.py -f 2-query/avg.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py
-,,,system-test,python3 ./test.py -f 2-query/between.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py
-,,,system-test,python3 ./test.py -f 2-query/bottom.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py
-,,,system-test,python3 ./test.py -f 2-query/cast.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py
-,,,system-test,python3 ./test.py -f 2-query/ceil.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py
-,,,system-test,python3 ./test.py -f 2-query/char_length.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -R
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py
-,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -R
 ,,,system-test,python3 ./test.py -f 2-query/concat.py
 ,,,system-test,python3 ./test.py -f 2-query/concat.py -R
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -R
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -R
-,,,system-test,python3 ./test.py -f 2-query/cos.py
-,,,system-test,python3 ./test.py -f 2-query/cos.py -R
-,,,system-test,python3 ./test.py -f 2-query/count_partition.py
-,,,system-test,python3 ./test.py -f 2-query/count_partition.py -R
-,,,system-test,python3 ./test.py -f 2-query/count.py
-,,,system-test,python3 ./test.py -f 2-query/count.py -R
-,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py
-,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R
-,,,system-test,python3 ./test.py -f 2-query/db.py
-,,,system-test,python3 ./test.py -f 2-query/db.py -R
-,,,system-test,python3 ./test.py -f 2-query/diff.py
-,,,system-test,python3 ./test.py -f 2-query/diff.py -R
-,,,system-test,python3 ./test.py -f 2-query/distinct.py
-,,,system-test,python3 ./test.py -f 2-query/distinct.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -R
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -R
-,,,system-test,python3 ./test.py -f 2-query/explain.py
-,,,system-test,python3 ./test.py -f 2-query/explain.py -R
-,,,system-test,python3 ./test.py -f 2-query/first.py
-,,,system-test,python3 ./test.py -f 2-query/first.py -R
-,,,system-test,python3 ./test.py -f 2-query/floor.py
-,,,system-test,python3 ./test.py -f 2-query/floor.py -R
-,,,system-test,python3 ./test.py -f 2-query/function_null.py
-,,,system-test,python3 ./test.py -f 2-query/function_null.py -R
-,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py
-,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -R
-,,,system-test,python3 ./test.py -f 2-query/histogram.py
-,,,system-test,python3 ./test.py -f 2-query/histogram.py -R
-,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py
-,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -R
 ,,,system-test,python3 ./test.py -f 2-query/interp.py
 ,,,system-test,python3 ./test.py -f 2-query/interp.py -R
-,,,system-test,python3 ./test.py -f 2-query/irate.py
-,,,system-test,python3 ./test.py -f 2-query/irate.py -R
-,,,system-test,python3 ./test.py -f 2-query/join.py
-,,,system-test,python3 ./test.py -f 2-query/join.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -R
 ,,,system-test,python3 ./test.py -f 2-query/last_row.py
 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -R
-,,,system-test,python3 ./test.py -f 2-query/last.py
-,,,system-test,python3 ./test.py -f 2-query/last.py -R
-,,,system-test,python3 ./test.py -f 2-query/leastsquares.py
-,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -R
-,,,system-test,python3 ./test.py -f 2-query/length.py
-,,,system-test,python3 ./test.py -f 2-query/length.py -R
-,,,system-test,python3 ./test.py -f 2-query/log.py
-,,,system-test,python3 ./test.py -f 2-query/log.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -R
 ,,,system-test,python3 ./test.py -f 2-query/lower.py
 ,,,system-test,python3 ./test.py -f 2-query/lower.py -R
 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py
 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -R
-,,,system-test,python3 ./test.py -f 2-query/mavg.py
-,,,system-test,python3 ./test.py -f 2-query/mavg.py -R
-,,,system-test,python3 ./test.py -f 2-query/max_partition.py
-,,,system-test,python3 ./test.py -f 2-query/max_partition.py -R
-,,,system-test,python3 ./test.py -f 2-query/max.py
-,,,system-test,python3 ./test.py -f 2-query/max.py -R
-,,,system-test,python3 ./test.py -f 2-query/min.py
-,,,system-test,python3 ./test.py -f 2-query/min.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -R
 ,,,system-test,python3 ./test.py -f 2-query/mode.py
 ,,,system-test,python3 ./test.py -f 2-query/mode.py -R
-,,,system-test,python3 ./test.py -f 2-query/Now.py
-,,,system-test,python3 ./test.py -f 2-query/Now.py -R
-,,,system-test,python3 ./test.py -f 2-query/percentile.py
-,,,system-test,python3 ./test.py -f 2-query/percentile.py -R
-,,,system-test,python3 ./test.py -f 2-query/pow.py
-,,,system-test,python3 ./test.py -f 2-query/pow.py -R
-,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py
-,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R
-,,,system-test,python3 ./test.py -f 2-query/round.py
-,,,system-test,python3 ./test.py -f 2-query/round.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -R
 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py
 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -R
-,,,system-test,python3 ./test.py -f 2-query/sample.py
-,,,system-test,python3 ./test.py -f 2-query/sample.py -R
-,,,system-test,python3 ./test.py -f 2-query/sin.py
-,,,system-test,python3 ./test.py -f 2-query/sin.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -R
 ,,,system-test,python3 ./test.py -f 2-query/smaTest.py
 ,,,system-test,python3 ./test.py -f 2-query/smaTest.py -R
 ,,,system-test,python3 ./test.py -f 2-query/sml.py
 ,,,system-test,python3 ./test.py -f 2-query/sml.py -R
-,,,system-test,python3 ./test.py -f 2-query/spread.py
-,,,system-test,python3 ./test.py -f 2-query/spread.py -R
-,,,system-test,python3 ./test.py -f 2-query/sqrt.py
-,,,system-test,python3 ./test.py -f 2-query/sqrt.py -R
-,,,system-test,python3 ./test.py -f 2-query/statecount.py
-,,,system-test,python3 ./test.py -f 2-query/statecount.py -R
-,,,system-test,python3 ./test.py -f 2-query/stateduration.py
-,,,system-test,python3 ./test.py -f 2-query/stateduration.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -R
 ,,,system-test,python3 ./test.py -f 2-query/substr.py
 ,,,system-test,python3 ./test.py -f 2-query/substr.py -R
-,,,system-test,python3 ./test.py -f 2-query/sum.py
-,,,system-test,python3 ./test.py -f 2-query/sum.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py -R
 ,,,system-test,python3 ./test.py -f 2-query/tail.py
 ,,,system-test,python3 ./test.py -f 2-query/tail.py -R
-,,,system-test,python3 ./test.py -f 2-query/tan.py
-,,,system-test,python3 ./test.py -f 2-query/tan.py -R
-,,,system-test,python3 ./test.py -f 2-query/Timediff.py
-,,,system-test,python3 ./test.py -f 2-query/Timediff.py -R
-,,,system-test,python3 ./test.py -f 2-query/timetruncate.py
-,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -R
-,,,system-test,python3 ./test.py -f 2-query/timezone.py
-,,,system-test,python3 ./test.py -f 2-query/timezone.py -R
-,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py
-,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -R
-,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py
-,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -R
-,,,system-test,python3 ./test.py -f 2-query/Today.py
-,,,system-test,python3 ./test.py -f 2-query/Today.py -R
-,,,system-test,python3 ./test.py -f 2-query/top.py
-,,,system-test,python3 ./test.py -f 2-query/top.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -R
 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py
 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -R
-,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py
-,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -R
-,,,system-test,python3 ./test.py -f 2-query/twa.py
-,,,system-test,python3 ./test.py -f 2-query/twa.py -R
-,,,system-test,python3 ./test.py -f 2-query/union.py
-,,,system-test,python3 ./test.py -f 2-query/union.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py -R
 ,,,system-test,python3 ./test.py -f 2-query/unique.py
 ,,,system-test,python3 ./test.py -f 2-query/unique.py -R
-,,,system-test,python3 ./test.py -f 2-query/upper.py
-,,,system-test,python3 ./test.py -f 2-query/upper.py -R
-,,,system-test,python3 ./test.py -f 2-query/varchar.py
-,,,system-test,python3 ./test.py -f 2-query/varchar.py -R
-,,,system-test,python3 ./test.py -f 2-query/case_when.py 
-,,,system-test,python3 ./test.py -f 2-query/case_when.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -R
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py 
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -R
 ,,,system-test,python3 ./test.py -f 1-insert/update_data.py
 ,,,system-test,python3 ./test.py -f 1-insert/tb_100w_data_order.py
 ,,,system-test,python3 ./test.py -f 1-insert/delete_stable.py
 ,,,system-test,python3 ./test.py -f 1-insert/delete_childtable.py
 ,,,system-test,python3 ./test.py -f 1-insert/delete_normaltable.py
-,,,system-test,python3 ./test.py -f 1-insert/keep_expired.py
-,,,system-test,python3 ./test.py -f 2-query/join2.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/keep_expired.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py
 ,,,system-test,python3 ./test.py -f 2-query/union1.py
 ,,,system-test,python3 ./test.py -f 2-query/concat2.py
 ,,,system-test,python3 ./test.py -f 2-query/json_tag.py
@@ -627,9 +627,9 @@
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_time.py
 ,,,system-test,python3 ./test.py -f 2-query/stablity.py
 ,,,system-test,python3 ./test.py -f 2-query/stablity_1.py
-,,,system-test,python3 ./test.py -f 2-query/elapsed.py
-,,,system-test,python3 ./test.py -f 2-query/csum.py
-,,,system-test,python3 ./test.py -f 2-query/function_diff.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/elapsed.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/csum.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_diff.py
 ,,,system-test,python3 ./test.py -f 2-query/queryQnode.py
 ,,,system-test,python3 ./test.py -f 6-cluster/5dnode1mnode.py
 ,,,system-test,python3 ./test.py -f 6-cluster/5dnode2mnode.py  -N 5
@@ -666,7 +666,7 @@
 ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas.py -N 4 -M 1
 ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas_querys.py -N 4 -M 1
 ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_vgroups.py  -N 4 -M 1
-,,,system-test,python3 ./test.py -f 7-tmq/create_wrong_topic.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/create_wrong_topic.py
 ,,,system-test,python3 ./test.py -f 7-tmq/dropDbR3ConflictTransaction.py -N 3
 ,,,system-test,python3 ./test.py -f 7-tmq/basic5.py
 ,,,system-test,python3 ./test.py -f 7-tmq/subscribeDb.py
@@ -704,6 +704,7 @@
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqConsFromTsdb1-mutilVg-mutilCtb.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqAutoCreateTbl.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart.py
+,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart1.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-1ctb.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdateWithConsume.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-multiCtb-snapshot0.py
@@ -724,18 +725,18 @@
 ,,,system-test,python3 ./test.py -f 7-tmq/stbTagFilter-multiCtb.py
 ,,,system-test,python3 ./test.py -f 99-TDcase/TD-19201.py
 ,,,system-test,python3 ./test.py -f 7-tmq/tmqSubscribeStb-r3.py -N 5
-,,,system-test,python3 ./test.py -f 2-query/between.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/length.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/join.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/cast.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/union1.py -Q 2
@@ -743,45 +744,45 @@
 ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/max.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/min.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/count.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/last.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/first.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/top.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/round.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/log.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/interp.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 2
@@ -792,44 +793,44 @@
 ,,,system-test,python3 ./test.py -f 2-query/avg.py   -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/csum.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/mavg.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/sample.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/unique.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/stateduration.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/statecount.py  -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/tail.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/twa.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/irate.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/function_null.py  -Q 2
-,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py  -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 2
 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 2
-,,,system-test,python3 ./test.py -f 2-query/between.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 2
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/length.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/upper.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/lower.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/join.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/join2.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/cast.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/union1.py -Q  3
@@ -837,45 +838,45 @@
 ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/spread.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/explain.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/Now.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/Today.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/max.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/min.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/count.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3
-,,,system-test,python3 ./test.py -f 2-query/last.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/first.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/diff.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/top.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/abs.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/floor.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/round.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/log.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/pow.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/sin.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/cos.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/tan.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 3
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 3
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 3
@@ -885,43 +886,43 @@
 ,,,system-test,python3 ./test.py -f 2-query/avg.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/csum.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/sample.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/unique.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q  3
 ,,,system-test,python3 ./test.py -f 2-query/tail.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/twa.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/irate.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q  3
-,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 3
-,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q  3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 3
 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 3
 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 3
-,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 3
 ,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 3
-,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 3
-,,,system-test,python3 ./test.py -f 2-query/between.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 3
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/length.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/upper.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/lower.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/join.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/join2.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q  4
@@ -930,45 +931,45 @@
 ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/spread.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/explain.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/Now.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/Today.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/max.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/min.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/count.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4
-,,,system-test,python3 ./test.py -f 2-query/last.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/first.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/diff.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/top.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/abs.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/floor.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/round.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/log.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/pow.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/sin.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/cos.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/tan.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/json_tag.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 4
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 4
 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 4
@@ -978,33 +979,34 @@
 ,,,system-test,python3 ./test.py -f 2-query/avg.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/csum.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/sample.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/unique.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q  4
 ,,,system-test,python3 ./test.py -f 2-query/tail.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/twa.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/irate.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q  4
-,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 4
-,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q  4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 4
 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 4
 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 4
-,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 4
 ,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 4
-,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 4
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 4
 
 #develop test
 ,,,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/auto_create_table_json.py

tests/pytest/auto_crash_gen.py

+import os
+import socket
+import requests
+
+# -*- coding: utf-8 -*-
+import os ,sys
+import random
+import argparse
+import subprocess
+import time
+import platform
+
+# valgrind mode ?
+valgrind_mode =  False
+
+msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
+
+# formal
+hostname = socket.gethostname()
+
+group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
+
+def get_msg(text):
+    return {
+        "msg_type": "post",
+        "content": {
+            "post": {
+                "zh_cn": {
+                    "title": "Crash_gen Monitor",
+                    "content": [
+                        [{
+                            "tag": "text",
+                            "text": text
+                        }
+                        ]]
+                }
+            }
+        }
+    }
+
+
+def send_msg(json):
+    headers = {
+        'Content-Type': 'application/json'
+    }
+
+    req = requests.post(url=group_url, headers=headers, json=json)
+    inf = req.json()
+    if "StatusCode" in inf and inf["StatusCode"] == 0:
+        pass
+    else:
+        print(inf)
+
+
+# set path about run instance
+
+core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+core_path = "/".join(core_path.split("/")[:-1])
+print(" =======   core path is %s ======== " %core_path)
+if not os.path.exists(core_path):
+    os.mkdir(core_path)
+
+base_dir = os.path.dirname(os.path.realpath(__file__))
+if base_dir.find("community")>0:
+    repo = "community"
+elif base_dir.find("TDengine")>0:
+    repo = "TDengine"
+else:
+    repo ="TDengine"
+print("base_dir:",base_dir)
+home_dir = base_dir[:base_dir.find(repo)]
+print("home_dir:",home_dir)
+run_dir = os.path.join(home_dir,'run_dir')
+run_dir = os.path.abspath(run_dir)
+print("run dir is *** :",run_dir)
+if not os.path.exists(run_dir):
+    os.mkdir(run_dir)
+run_log_file = run_dir+'/crash_gen_run.log'
+crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
+exit_status_logs =  os.path.join(run_dir, 'crash_exit.log')
+
+def get_path():
+    buildPath=''
+    selfPath = os.path.dirname(os.path.realpath(__file__))
+    if ("community" in selfPath):
+        projPath = selfPath[:selfPath.find("community")]
+    else:
+        projPath = selfPath[:selfPath.find("tests")]
+
+    for root, dirs, files in os.walk(projPath):
+        if ("taosd" in files):
+            rootRealPath = os.path.dirname(os.path.realpath(root))
+            if ("packaging" not in rootRealPath):
+                buildPath = root[:len(root) - len("/build/bin")]
+                break
+    return buildPath
+
+# generate crash_gen start script randomly
+
+def random_args(args_list):
+    nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
+    ]  # record bool type arguments
+    strs_args_list = ["--connector-type"]  # record str type arguments
+
+    args_list["--auto-start-service"]= False
+    args_list["--continue-on-exception"]=True
+    # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
+    connect_types=['native']
+    # args_list["--connector-type"]=connect_types[random.randint(0,2)]
+    args_list["--connector-type"]= connect_types[0]
+    args_list["--max-dbs"]= random.randint(1,10)
+    
+    # dnodes = [1,3] # set single dnodes;
+    
+    # args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
+    # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
+    args_list["--debug"]=False
+    args_list["--per-thread-db-connection"]=True
+    args_list["--track-memory-leaks"]=False
+
+    args_list["--max-steps"]=random.randint(500,2000)
+
+    # args_list["--ignore-errors"]=[]   ## can add error codes for detail
+
+    
+    args_list["--run-tdengine"]= False
+    args_list["--use-shadow-db"]= False
+    args_list["--dynamic-db-table-names"]= True
+    args_list["--verify-data"]= False
+    args_list["--record-ops"] = False
+
+    for key in bools_args_list:
+        set_bool_value = [True,False]
+        if key == "--auto-start-service" :
+            continue
+        elif key =="--run-tdengine":
+            continue
+        elif key == "--ignore-errors":
+            continue
+        elif key == "--debug":
+            continue
+        elif key == "--per-thread-db-connection":
+            continue
+        elif key == "--continue-on-exception":
+            continue
+        elif key == "--use-shadow-db":
+            continue
+        elif key =="--track-memory-leaks":
+            continue
+        elif key == "--dynamic-db-table-names":
+            continue
+        elif key == "--verify-data":
+            continue
+        elif key == "--record-ops":
+            continue
+        else:
+            args_list[key]=set_bool_value[random.randint(0,1)]
+
+    if args_list["--larger-data"]:
+        threads = [16,32]
+    else:
+        threads = [32,64,128,256] 
+    args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
+
+    return args_list
+
+def limits(args_list):
+    if args_list["--use-shadow-db"]==True:
+        if args_list["--max-dbs"] > 1:
+            print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
+            args_list["--max-dbs"]=1
+        else:
+            pass
+
+    # env is start by test frame , not crash_gen instance
+    
+    # elif args_list["--num-replicas"]==0:
+    #     print(" make sure num-replicas is at least 1 ")
+    #     args_list["--num-replicas"]=1
+    # elif args_list["--num-replicas"]==1:
+    #     pass
+
+    # elif args_list["--num-replicas"]>1:
+    #     if not args_list["--auto-start-service"]:
+    #         print("it should be deployed by crash_gen auto-start-service for multi replicas")
+            
+    # else:
+    #     pass
+         
+    return  args_list
+
+def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
+    arguments = ""
+    for k ,v in args_list.items():
+        if k == "--ignore-errors":
+            if v:
+                arguments+=(k+"="+str(v)+" ")
+            else:
+                arguments+=""
+        elif  k in bools_args_list and v==True:
+            arguments+=(k+" ")
+        elif k in bools_args_list and v==False:
+            arguments+=""
+        else:
+            arguments+=(k+"="+str(v)+" ")
+    
+    if valgrind :
+         
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments)
+
+    else:
+
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments)
+
+    return crash_gen_cmd
+
+def start_taosd():
+    build_path = get_path()
+    if repo == "community":
+        start_path = build_path[:-5]+"community/tests/system-test/"
+    elif repo == "TDengine":
+        start_path = build_path[:-5]+"/tests/system-test/"
+    else:
+        pass
+
+    start_cmd = 'cd %s && python3 test.py >>/dev/null '%(start_path)
+    os.system(start_cmd)
+
+def get_cmds(args_list):
+    # build_path = get_path()
+    # if repo == "community":
+    #     crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    # elif repo == "TDengine":
+    #     crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    # else:
+    #     pass
+
+    # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550  '%(crash_gen_path)
+
+    crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
+    return crash_gen_cmd
+
+def run_crash_gen(crash_cmds):
+
+    # prepare env of taosd
+    start_taosd()
+
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    result_file = os.path.join(crash_gen_path, 'valgrind.out')
+
+
+    # run crash_gen and back logs
+    os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
+    os.system("%s >>%s "%(crash_cmds,result_file))
+
+
+def check_status():
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    result_file = os.path.join(crash_gen_path, 'valgrind.out')
+    run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
+
+    core_check = subprocess.Popen('ls -l  %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+
+    if int(core_check.strip().rstrip()) > 0:
+        # it means core files has occured
+        return 3
+    
+    if "Crash_Gen is now exiting with status code: 1" in run_code:
+        return 1
+    elif "Crash_Gen is now exiting with status code: 0" in run_code:
+        return 0
+    else:
+        return 2 
+
+
+def main():
+
+    args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
+    "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
+    "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , 
+    "--continue-on-exception":False }
+
+    args = random_args(args_list)
+    args = limits(args)
+
+
+    build_path = get_path()
+    os.system("pip install git+https://github.com/taosdata/taos-connector-python.git")
+    if repo =="community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo =="TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    
+    if os.path.exists(crash_gen_path+"crash_gen.sh"):
+        print(" make sure crash_gen.sh is ready")
+    else:
+        print( " crash_gen.sh is not exists ")
+        sys.exit(1)
+    
+    git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
+    
+    # crash_cmds = get_cmds()
+    
+    crash_cmds = get_cmds(args)
+    # clean run_dir
+    os.system('rm -rf %s'%run_dir )
+    if not os.path.exists(run_dir):
+        os.mkdir(run_dir)
+    print(crash_cmds)
+    run_crash_gen(crash_cmds)
+    status = check_status()
+    
+    print("exit status : ", status)
+    
+    if status ==4:
+        print('======== crash_gen found memory bugs   ========')
+    if status ==5:
+        print('======== crash_gen found memory errors   ========')
+    if status >0:
+        print('======== crash_gen run failed and not exit as expected ========')
+    else:
+        print('======== crash_gen run sucess and exit as expected ========')
+
+
+    if status!=0 :
+
+        try:
+            text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n  " + f" and git commit :  {git_commit}"
+            send_msg(get_msg(text))  
+        except Exception as e:
+            print("exception:", e)
+        exit(status)
+        
+
+if __name__ == '__main__':
+    main()
+
+

tests/pytest/auto_crash_gen_valgrind.py

+#!/usr/bin/python3
+
+
+import os
+import socket
+import requests
+
+# -*- coding: utf-8 -*-
+import os ,sys
+import random
+import argparse
+import subprocess
+import time
+import platform
+
+# valgrind mode ?
+valgrind_mode =  True
+
+msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
+
+# formal
+hostname = socket.gethostname()
+
+group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
+
+def get_msg(text):
+    return {
+        "msg_type": "post",
+        "content": {
+            "post": {
+                "zh_cn": {
+                    "title": "Crash_gen Monitor",
+                    "content": [
+                        [{
+                            "tag": "text",
+                            "text": text
+                        }
+                        ]]
+                }
+            }
+        }
+    }
+
+
+def send_msg(json):
+    headers = {
+        'Content-Type': 'application/json'
+    }
+
+    req = requests.post(url=group_url, headers=headers, json=json)
+    inf = req.json()
+    if "StatusCode" in inf and inf["StatusCode"] == 0:
+        pass
+    else:
+        print(inf)
+
+
+# set path about run instance
+
+core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+core_path = "/".join(core_path.split("/")[:-1])
+print(" =======   core path is %s ======== " %core_path)
+if not os.path.exists(core_path):
+    os.mkdir(core_path)
+
+base_dir = os.path.dirname(os.path.realpath(__file__))
+if base_dir.find("community")>0:
+    repo = "community"
+elif base_dir.find("TDengine")>0:
+    repo = "TDengine"
+else:
+    repo ="TDengine"
+print("base_dir:",base_dir)
+home_dir = base_dir[:base_dir.find(repo)]
+print("home_dir:",home_dir)
+run_dir = os.path.join(home_dir,'run_dir')
+run_dir = os.path.abspath(run_dir)
+print("run dir is *** :",run_dir)
+if not os.path.exists(run_dir):
+    os.mkdir(run_dir)
+run_log_file = run_dir+'/crash_gen_run.log'
+crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
+exit_status_logs =  os.path.join(run_dir, 'crash_exit.log')
+
+def get_path():
+    buildPath=''
+    selfPath = os.path.dirname(os.path.realpath(__file__))
+    if ("community" in selfPath):
+        projPath = selfPath[:selfPath.find("community")]
+    else:
+        projPath = selfPath[:selfPath.find("tests")]
+
+    for root, dirs, files in os.walk(projPath):
+        if ("taosd" in files):
+            rootRealPath = os.path.dirname(os.path.realpath(root))
+            if ("packaging" not in rootRealPath):
+                buildPath = root[:len(root) - len("/build/bin")]
+                break
+    return buildPath
+
+# generate crash_gen start script randomly
+
+def random_args(args_list):
+    nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
+    ]  # record bool type arguments
+    strs_args_list = ["--connector-type"]  # record str type arguments
+
+    args_list["--auto-start-service"]= False
+    args_list["--continue-on-exception"]=True
+    # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
+    connect_types=['native']
+    # args_list["--connector-type"]=connect_types[random.randint(0,2)]
+    args_list["--connector-type"]= connect_types[0]
+    args_list["--max-dbs"]= random.randint(1,10)
+    
+    # dnodes = [1,3] # set single dnodes;
+    
+    # args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
+    # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
+    args_list["--debug"]=False
+    args_list["--per-thread-db-connection"]=True
+    args_list["--track-memory-leaks"]=False
+
+    args_list["--max-steps"]=random.randint(200,500)
+    
+    threads = [16,32]
+
+    args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
+    # args_list["--ignore-errors"]=[]   ## can add error codes for detail
+
+    
+    args_list["--run-tdengine"]= False
+    args_list["--use-shadow-db"]= False
+    args_list["--dynamic-db-table-names"]= True
+    args_list["--verify-data"]= False
+    args_list["--record-ops"] = False
+
+    for key in bools_args_list:
+        set_bool_value = [True,False]
+        if key == "--auto-start-service" :
+            continue
+        elif key =="--run-tdengine":
+            continue
+        elif key == "--ignore-errors":
+            continue
+        elif key == "--debug":
+            continue
+        elif key == "--per-thread-db-connection":
+            continue
+        elif key == "--continue-on-exception":
+            continue
+        elif key == "--use-shadow-db":
+            continue
+        elif key =="--track-memory-leaks":
+            continue
+        elif key == "--dynamic-db-table-names":
+            continue
+        elif key == "--verify-data":
+            continue
+        elif key == "--record-ops":
+            continue
+        elif key == "--larger-data":
+            continue
+        else:
+            args_list[key]=set_bool_value[random.randint(0,1)]
+    return args_list
+
+def limits(args_list):
+    if args_list["--use-shadow-db"]==True:
+        if args_list["--max-dbs"] > 1:
+            print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
+            args_list["--max-dbs"]=1
+        else:
+            pass
+
+    # env is start by test frame , not crash_gen instance
+    
+    # elif args_list["--num-replicas"]==0:
+    #     print(" make sure num-replicas is at least 1 ")
+    #     args_list["--num-replicas"]=1
+    # elif args_list["--num-replicas"]==1:
+    #     pass
+
+    # elif args_list["--num-replicas"]>1:
+    #     if not args_list["--auto-start-service"]:
+    #         print("it should be deployed by crash_gen auto-start-service for multi replicas")
+            
+    # else:
+    #     pass
+         
+    return  args_list
+
+def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
+    arguments = ""
+    for k ,v in args_list.items():
+        if k == "--ignore-errors":
+            if v:
+                arguments+=(k+"="+str(v)+" ")
+            else:
+                arguments+=""
+        elif  k in bools_args_list and v==True:
+            arguments+=(k+" ")
+        elif k in bools_args_list and v==False:
+            arguments+=""
+        else:
+            arguments+=(k+"="+str(v)+" ")
+    
+    if valgrind :
+         
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments)
+
+    else:
+
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments)
+
+    return crash_gen_cmd
+
+
+def start_taosd():
+    build_path = get_path()
+    if repo == "community":
+        start_path = build_path[:-5]+"community/tests/system-test/"
+    elif repo == "TDengine":
+        start_path = build_path[:-5]+"/tests/system-test/"
+    else:
+        pass
+
+    start_cmd = 'cd %s && python3 test.py  '%(start_path)
+    os.system(start_cmd +">>/dev/null")
+
+def get_cmds(args_list):
+    # build_path = get_path()
+    # if repo == "community":
+    #     crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    # elif repo == "TDengine":
+    #     crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    # else:
+    #     pass
+
+    # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550  '%(crash_gen_path)
+
+    crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
+    return crash_gen_cmd
+
+def run_crash_gen(crash_cmds):
+
+    # prepare env of taosd
+    start_taosd()
+    # run crash_gen and back logs
+    os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
+    # os.system("cp %s %s"%(crash_gen_cmds_file, core_path))
+    os.system("%s"%(crash_cmds))
+
+def check_status():
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    result_file = os.path.join(crash_gen_path, 'valgrind.out')
+    run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
+
+    core_check = subprocess.Popen('ls -l  %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+
+    if int(core_check.strip().rstrip()) > 0:
+        # it means core files has occured
+        return 3
+    
+    mem_status = check_memory()
+    if mem_status >0:
+        return mem_status
+    if "Crash_Gen is now exiting with status code: 1" in run_code:
+        return 1
+    elif "Crash_Gen is now exiting with status code: 0" in run_code:
+        return 0
+    else:
+        return 2 
+    
+
+def check_memory():
+
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    '''
+    invalid read, invalid write
+    '''
+    back_path = os.path.join(core_path,"valgrind_report")
+    if not os.path.exists(back_path):
+        os.mkdir(back_path)
+
+    stderr_file = os.path.join(crash_gen_path , "valgrind.err")
+    
+    status = 0
+
+    grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 4
+    
+    grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 4
+        
+    grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 5
+    
+    return status
+
+def main():
+
+    args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
+    "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
+    "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , 
+    "--continue-on-exception":False }
+
+    args = random_args(args_list)
+    args = limits(args)
+
+    build_path = get_path()
+    os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null")
+    if repo =="community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo =="TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    
+    if os.path.exists(crash_gen_path+"crash_gen.sh"):
+        print(" make sure crash_gen.sh is ready")
+    else:
+        print( " crash_gen.sh is not exists ")
+        sys.exit(1)
+    
+    git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
+
+    # crash_cmds = get_cmds()
+    
+    crash_cmds = get_cmds(args)
+
+    # clean run_dir
+    os.system('rm -rf %s'%run_dir )
+    if not os.path.exists(run_dir):
+        os.mkdir(run_dir)
+    print(crash_cmds)
+    run_crash_gen(crash_cmds)
+    status = check_status()
+    # back_path = os.path.join(core_path,"valgrind_report")
+    
+    print("exit status : ", status)
+    
+    if status ==4:
+        print('======== crash_gen found memory bugs   ========')
+    if status ==5:
+        print('======== crash_gen found memory errors   ========')
+    if status >0:
+        print('======== crash_gen run failed and not exit as expected ========')
+    else:
+        print('======== crash_gen run sucess and exit as expected ========')
+
+    if status!=0 :
+
+        try:
+            text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n  " + f" and git commit : {git_commit}"
+            send_msg(get_msg(text))  
+        except Exception as e:
+            print("exception:", e)
+        exit(status)
+        
+
+if __name__ == '__main__':
+    main()
+
+

tests/pytest/auto_crash_gen_valgrind_cluster.py

+#!/usr/bin/python3
+
+
+import os
+import socket
+import requests
+
+# -*- coding: utf-8 -*-
+import os ,sys
+import random
+import argparse
+import subprocess
+import time
+import platform
+
+# valgrind mode ?
+valgrind_mode =  True
+
+msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
+
+# formal
+hostname = socket.gethostname()
+
+group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
+
+def get_msg(text):
+    return {
+        "msg_type": "post",
+        "content": {
+            "post": {
+                "zh_cn": {
+                    "title": "Crash_gen Monitor",
+                    "content": [
+                        [{
+                            "tag": "text",
+                            "text": text
+                        }
+                        ]]
+                }
+            }
+        }
+    }
+
+
+def send_msg(json):
+    headers = {
+        'Content-Type': 'application/json'
+    }
+
+    req = requests.post(url=group_url, headers=headers, json=json)
+    inf = req.json()
+    if "StatusCode" in inf and inf["StatusCode"] == 0:
+        pass
+    else:
+        print(inf)
+
+
+# set path about run instance
+
+core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+core_path = "/".join(core_path.split("/")[:-1])
+print(" =======   core path is %s ======== " %core_path)
+if not os.path.exists(core_path):
+    os.mkdir(core_path)
+
+base_dir = os.path.dirname(os.path.realpath(__file__))
+if base_dir.find("community")>0:
+    repo = "community"
+elif base_dir.find("TDengine")>0:
+    repo = "TDengine"
+else:
+    repo ="TDengine"
+print("base_dir:",base_dir)
+home_dir = base_dir[:base_dir.find(repo)]
+print("home_dir:",home_dir)
+run_dir = os.path.join(home_dir,'run_dir')
+run_dir = os.path.abspath(run_dir)
+print("run dir is *** :",run_dir)
+if not os.path.exists(run_dir):
+    os.mkdir(run_dir)
+run_log_file = run_dir+'/crash_gen_run.log'
+crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
+exit_status_logs =  os.path.join(run_dir, 'crash_exit.log')
+
+def get_path():
+    buildPath=''
+    selfPath = os.path.dirname(os.path.realpath(__file__))
+    if ("community" in selfPath):
+        projPath = selfPath[:selfPath.find("community")]
+    else:
+        projPath = selfPath[:selfPath.find("tests")]
+
+    for root, dirs, files in os.walk(projPath):
+        if ("taosd" in files):
+            rootRealPath = os.path.dirname(os.path.realpath(root))
+            if ("packaging" not in rootRealPath):
+                buildPath = root[:len(root) - len("/build/bin")]
+                break
+    return buildPath
+
+# generate crash_gen start script randomly
+
+def random_args(args_list):
+    nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
+    ]  # record bool type arguments
+    strs_args_list = ["--connector-type"]  # record str type arguments
+
+    args_list["--auto-start-service"]= False
+    args_list["--continue-on-exception"]=True
+    # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
+    connect_types=['native']
+    # args_list["--connector-type"]=connect_types[random.randint(0,2)]
+    args_list["--connector-type"]= connect_types[0]
+    args_list["--max-dbs"]= random.randint(1,10)
+    
+    # dnodes = [1,3] # set single dnodes;
+    
+    # args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
+    # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
+    args_list["--debug"]=False
+    args_list["--per-thread-db-connection"]=True
+    args_list["--track-memory-leaks"]=False
+
+    args_list["--max-steps"]=random.randint(200,500)
+    
+    threads = [16,32]
+
+    args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
+    # args_list["--ignore-errors"]=[]   ## can add error codes for detail
+
+    
+    args_list["--run-tdengine"]= False
+    args_list["--use-shadow-db"]= False
+    args_list["--dynamic-db-table-names"]= True
+    args_list["--verify-data"]= False
+    args_list["--record-ops"] = False
+
+    for key in bools_args_list:
+        set_bool_value = [True,False]
+        if key == "--auto-start-service" :
+            continue
+        elif key =="--run-tdengine":
+            continue
+        elif key == "--ignore-errors":
+            continue
+        elif key == "--debug":
+            continue
+        elif key == "--per-thread-db-connection":
+            continue
+        elif key == "--continue-on-exception":
+            continue
+        elif key == "--use-shadow-db":
+            continue
+        elif key =="--track-memory-leaks":
+            continue
+        elif key == "--dynamic-db-table-names":
+            continue
+        elif key == "--verify-data":
+            continue
+        elif key == "--record-ops":
+            continue
+        elif key == "--larger-data":
+            continue
+        else:
+            args_list[key]=set_bool_value[random.randint(0,1)]
+    return args_list
+
+def limits(args_list):
+    if args_list["--use-shadow-db"]==True:
+        if args_list["--max-dbs"] > 1:
+            print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
+            args_list["--max-dbs"]=1
+        else:
+            pass
+
+    # env is start by test frame , not crash_gen instance
+    
+    # elif args_list["--num-replicas"]==0:
+    #     print(" make sure num-replicas is at least 1 ")
+    #     args_list["--num-replicas"]=1
+    # elif args_list["--num-replicas"]==1:
+    #     pass
+
+    # elif args_list["--num-replicas"]>1:
+    #     if not args_list["--auto-start-service"]:
+    #         print("it should be deployed by crash_gen auto-start-service for multi replicas")
+            
+    # else:
+    #     pass
+         
+    return  args_list
+
+def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+
+    bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
+    "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
+    arguments = ""
+    for k ,v in args_list.items():
+        if k == "--ignore-errors":
+            if v:
+                arguments+=(k+"="+str(v)+" ")
+            else:
+                arguments+=""
+        elif  k in bools_args_list and v==True:
+            arguments+=(k+" ")
+        elif k in bools_args_list and v==False:
+            arguments+=""
+        else:
+            arguments+=(k+"="+str(v)+" ")
+    
+    if valgrind :
+         
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  --valgrind   -i 3 %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0707,0x0203 '%(crash_gen_path ,arguments)
+
+    else:
+
+        crash_gen_cmd = 'cd %s && ./crash_gen.sh  -i 3 %s  -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0014,0x0707,0x0203'%(crash_gen_path ,arguments)
+
+    return crash_gen_cmd
+
+
+def start_taosd():
+    build_path = get_path()
+    if repo == "community":
+        start_path = build_path[:-5]+"community/tests/system-test/"
+    elif repo == "TDengine":
+        start_path = build_path[:-5]+"/tests/system-test/"
+    else:
+        pass
+
+    start_cmd = 'cd %s && python3 test.py -N 4 -M 1 '%(start_path)
+    os.system(start_cmd +">>/dev/null")
+
+def get_cmds(args_list):
+    # build_path = get_path()
+    # if repo == "community":
+    #     crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    # elif repo == "TDengine":
+    #     crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    # else:
+    #     pass
+
+    # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550  '%(crash_gen_path)
+
+    crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
+    return crash_gen_cmd
+
+def run_crash_gen(crash_cmds):
+
+    # prepare env of taosd
+    start_taosd()
+    # run crash_gen and back logs
+    os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
+    # os.system("cp %s %s"%(crash_gen_cmds_file, core_path))
+    os.system("%s"%(crash_cmds))
+
+def check_status():
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    result_file = os.path.join(crash_gen_path, 'valgrind.out')
+    run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
+
+    core_check = subprocess.Popen('ls -l  %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+
+    if int(core_check.strip().rstrip()) > 0:
+        # it means core files has occured
+        return 3
+    
+    mem_status = check_memory()
+    if mem_status >0:
+        return mem_status
+    if "Crash_Gen is now exiting with status code: 1" in run_code:
+        return 1
+    elif "Crash_Gen is now exiting with status code: 0" in run_code:
+        return 0
+    else:
+        return 2 
+    
+
+def check_memory():
+
+    build_path = get_path()
+    if repo == "community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo == "TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    '''
+    invalid read, invalid write
+    '''
+    back_path = os.path.join(core_path,"valgrind_report")
+    if not os.path.exists(back_path):
+        os.mkdir(back_path)
+
+    stderr_file = os.path.join(crash_gen_path , "valgrind.err")
+    
+    status = 0
+
+    grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 4
+    
+    grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 4
+        
+    grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
+    
+    if grep_res:
+        # os.system("cp %s %s"%(stderr_file , back_path))
+        status = 5
+    
+    return status
+
+def main():
+
+    args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
+    "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
+    "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , 
+    "--continue-on-exception":False }
+
+    args = random_args(args_list)
+    args = limits(args)
+
+    build_path = get_path()
+    os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null")
+    if repo =="community":
+        crash_gen_path = build_path[:-5]+"community/tests/pytest/"
+    elif repo =="TDengine":
+        crash_gen_path = build_path[:-5]+"/tests/pytest/"
+    else:
+        pass
+    
+    if os.path.exists(crash_gen_path+"crash_gen.sh"):
+        print(" make sure crash_gen.sh is ready")
+    else:
+        print( " crash_gen.sh is not exists ")
+        sys.exit(1)
+    
+    git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
+
+    # crash_cmds = get_cmds()
+    
+    crash_cmds = get_cmds(args)
+
+    # clean run_dir
+    os.system('rm -rf %s'%run_dir )
+    if not os.path.exists(run_dir):
+        os.mkdir(run_dir)
+    print(crash_cmds)
+    run_crash_gen(crash_cmds)
+    status = check_status()
+    # back_path = os.path.join(core_path,"valgrind_report")
+    
+    print("exit status : ", status)
+    
+    if status ==4:
+        print('======== crash_gen found memory bugs   ========')
+    if status ==5:
+        print('======== crash_gen found memory errors   ========')
+    if status >0:
+        print('======== crash_gen run failed and not exit as expected ========')
+    else:
+        print('======== crash_gen run sucess and exit as expected ========')
+
+    if status!=0 :
+
+        try:
+            text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n  " + f" and git commit : {git_commit}"
+            send_msg(get_msg(text))  
+        except Exception as e:
+            print("exception:", e)
+        exit(status)
+        
+
+if __name__ == '__main__':
+    main()
+
+

tests/pytest/auto_run_regular.sh

+#!/bin/bash
+
+# set LD_LIBRARY_PATH
+export PATH=$PATH:/home/TDengine/debug/build/bin
+export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
+ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
+
+# run crash_gen auto script
+python3 /home/TDengine/tests/pytest/auto_crash_gen.py
\ No newline at end of file

tests/pytest/auto_run_valgrind.sh

+#!/bin/bash
+
+# set LD_LIBRARY_PATH
+export PATH=$PATH:/home/TDengine/debug/build/bin
+export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
+ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
+
+# run crash_gen auto script
+python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind.py
\ No newline at end of file

tests/pytest/auto_run_valgrind_cluster.sh

+#!/bin/bash
+
+# set LD_LIBRARY_PATH
+export PATH=$PATH:/home/TDengine/debug/build/bin
+export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
+ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
+ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
+
+# run crash_gen auto script
+python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind_cluster.py
\ No newline at end of file

tests/pytest/docker_exec_service.sh

+#!/bin/bash
+
+mkdir -p /data/wz/crash_gen_logs/
+logdir='/data/wz/crash_gen_logs/'
+date_tag=`date +%Y%m%d-%H%M%S`
+hostname='vm_valgrind_'
+
+for i in {1..50}
+do 
+        echo $i
+        # create docker and start crash_gen
+        log_dir=${logdir}${hostname}${date_tag}_${i}
+        docker run -d  --hostname=${hostname}${date_tag}_${i} --name ${hostname}${date_tag}_${i} --privileged  -v ${log_dir}:/corefile/ -- crash_gen:v1.0 sleep 99999999999999 
+                echo create docker ${hostname}${date_tag}_${i}
+                docker exec -d ${hostname}${date_tag}_${i} sh -c 'rm -rf /home/taos-connector-python'
+                docker cp /data/wz/TDengine ${hostname}${date_tag}_${i}:/home/TDengine
+                docker cp /data/wz/taos-connector-python ${hostname}${date_tag}_${i}:/home/taos-connector-python
+                echo copy TDengine in container done!
+        docker exec  ${hostname}${date_tag}_${i}  sh -c 'sh /home/TDengine/tests/pytest/auto_run_valgrind.sh '
+        if [ $? -eq 0 ] 
+        then
+                echo crash_gen exit as expect , run success
+
+                # # clear docker which success
+                docker stop ${hostname}${date_tag}_${i}
+                docker rm -f ${hostname}${date_tag}_${i}
+        else
+                docker stop ${hostname}${date_tag}_${i}
+                echo crash_gen exit error , run failed
+        fi
+done
\ No newline at end of file

tests/system-test/7-tmq/tmqDnodeRestart1.py

@@ -149,6 +149,8 @@ class TDTestCase:
 
         tmqCom.waitSubscriptionExit(tdSql, topicFromStb)
         tdSql.query("drop topic %s"%topicFromStb)
+        
+        tmqCom.stopTmqSimProcess(processorName="tmq_sim")
 
         tdLog.printNoPrefix("======== test case 1 end ...... ")
 
@@ -178,6 +180,8 @@ class TDTestCase:
         paraDict['vgroups'] = self.vgroups
         paraDict['ctbNum'] = self.ctbNum
         paraDict['rowsPerTbl'] = self.rowsPerTbl
+        
+        tmqCom.initConsumerTable()
    
         tdLog.info("create topics from stb")
         topicFromDb = 'topic_db'
@@ -203,10 +207,10 @@ class TDTestCase:
         tmqCom.getStartCommitNotifyFromTmqsim('cdb',1)
 
         tdLog.info("create some new child table and insert data for latest mode")
-        paraDict["batchNum"] = 100
+        paraDict["batchNum"] = 10
         paraDict["ctbPrefix"] = 'newCtb'
-        paraDict["ctbNum"]     = 10
-        paraDict["rowsPerTbl"] = 10
+        paraDict["ctbNum"]     = 100
+        paraDict["rowsPerTbl"] = 100
         tmqCom.insert_data_with_autoCreateTbl(tdSql,paraDict["dbName"],paraDict["stbName"],paraDict["ctbPrefix"],paraDict["ctbNum"],paraDict["rowsPerTbl"],paraDict["batchNum"])
 
         tdLog.info("================= restart dnode ===========================")

tests/system-test/pytest.sh

@@ -77,13 +77,14 @@ echo "Preload AsanSo:" $?
 $* -a 2> $AsanFile
 
 unset LD_PRELOAD
-AsanFileLen=`cat $AsanFile | wc -l`
-while [ $AsanFileLen -lt 10 ]
+for ((i=1;i<=20;i++))
 do
-  sleep 1
-  `cat $AsanFile | wc -l`
+  AsanFileLen=`cat $AsanFile | wc -l`
+  echo "AsanFileLen:" $AsanFileLen
+  if [ $AsanFileLen -gt 10 ]; then
+    break
+  fi
 done 
-echo "AsanFileLen:" $AsanFileLen
 
 AsanFileSuccessLen=`grep -w successfully $AsanFile | wc -l`
 echo "AsanFileSuccessLen:" $AsanFileSuccessLen

tools/CMakeLists.txt

@@ -101,19 +101,20 @@ ELSE ()
             BUILD_COMMAND
             COMMAND set CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client
             COMMAND set CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib
-            COMMAND go build -a -o taosadapter.exe -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
-            COMMAND go build -a -o taosadapter-debug.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+            COMMAND go build -a -o taosadapter.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+#            COMMAND go build -a -o taosadapter.exe -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+#            COMMAND go build -a -o taosadapter-debug.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
 
             INSTALL_COMMAND
-            COMMAND cmake -E echo "Comparessing taosadapter.exe"
-            COMMAND cmake -E time upx taosadapter.exe
+#            COMMAND cmake -E echo "Comparessing taosadapter.exe"
+#            COMMAND cmake -E time upx taosadapter.exe
             COMMAND cmake -E echo "Copy taosadapter.exe"
             COMMAND cmake -E copy taosadapter.exe ${CMAKE_BINARY_DIR}/build/bin/taosadapter.exe
             COMMAND cmake -E make_directory ${CMAKE_BINARY_DIR}/test/cfg/
             COMMAND cmake -E echo "Copy taosadapter.toml"
             COMMAND cmake -E copy ./example/config/taosadapter.toml ${CMAKE_BINARY_DIR}/test/cfg/
-            COMMAND cmake -E echo "Copy taosadapter-debug.exe"
-            COMMAND cmake -E copy taosadapter-debug.exe ${CMAKE_BINARY_DIR}/build/bin
+#            COMMAND cmake -E echo "Copy taosadapter-debug.exe"
+#            COMMAND cmake -E copy taosadapter-debug.exe ${CMAKE_BINARY_DIR}/build/bin
             )
     ELSE (TD_WINDOWS)
         MESSAGE("Building taosAdapter on non-Windows")
@@ -128,19 +129,20 @@ ELSE ()
             PATCH_COMMAND
             COMMAND     git clean -f -d
             BUILD_COMMAND
-            COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
-            COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -o taosadapter-debug -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+            COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+#            COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
+#            COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -o taosadapter-debug -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}"
             INSTALL_COMMAND
-            COMMAND cmake -E echo "Comparessing taosadapter.exe"
-            COMMAND upx taosadapter || :
+#            COMMAND cmake -E echo "Comparessing taosadapter.exe"
+#            COMMAND upx taosadapter || :
             COMMAND cmake -E echo "Copy taosadapter"
             COMMAND cmake -E copy taosadapter ${CMAKE_BINARY_DIR}/build/bin
             COMMAND cmake -E make_directory ${CMAKE_BINARY_DIR}/test/cfg/
             COMMAND cmake -E echo "Copy taosadapter.toml"
             COMMAND cmake -E copy ./example/config/taosadapter.toml ${CMAKE_BINARY_DIR}/test/cfg/
             COMMAND cmake -E copy ./taosadapter.service ${CMAKE_BINARY_DIR}/test/cfg/
-            COMMAND cmake -E echo "Copy taosadapter-debug"
-            COMMAND cmake -E copy taosadapter-debug ${CMAKE_BINARY_DIR}/build/bin
+#            COMMAND cmake -E echo "Copy taosadapter-debug"
+#            COMMAND cmake -E copy taosadapter-debug ${CMAKE_BINARY_DIR}/build/bin
             )
     ENDIF (TD_WINDOWS)
 ENDIF ()