diff --git a/CONTRIBUTING-CN.md b/CONTRIBUTING-CN.md index 19f3000d45b53a4fcdab9d0e0ebbf2da7e2d94f8..efaa2077fe05a7c9f4b08098fcb74da8ad3a97cd 100644 --- a/CONTRIBUTING-CN.md +++ b/CONTRIBUTING-CN.md @@ -7,25 +7,18 @@ - 任何用户都可以通过 **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)** 向我们报告错误。请您对所遇到的问题进行**详细描述**，最好提供重现错误的详细步骤。 - 欢迎提供包含由 Bug 生成的日志文件的附录。 -## 需要强调的代码提交规则 - -- 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。 -- 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。 -- 如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。 -- 将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。 - -## 贡献指南 - -1. 请用友好的语气书写。 - -2. **主动语态**总体上优于被动语态。主动语态中的句子会突出执行动作的人，而不是被动语态突出动作的接受者。 - -3. 文档写作建议 - -- 正确拼写产品名称 “TDengine”。 “TD” 用大写字母，“TD” 和 “engine” 之间没有空格 **（正确拼写：TDengine）**。 -- 在句号或其他标点符号后只留一个空格。 - -4. 尽量**使用简单句**，而不是复杂句。 +## 代码提交规则 + +1. 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。 +2. 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。 + 如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。 + 将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。 +3. 将TDengine仓库库fork到自己的账户中并创建分支(branch)。 + 注意：默认分支`main`不能直接接受PR，请基于开发分支`3.0`创建自己的分支。 + 注意：修改文档的分支要以`docs/`为开头，以免进行不必要的测试。 +4. 创建pull request，将自己的分支合并到开发分支`3.0`，我们开发团队将尽快审核。 + +如遇任何问题，请添加官方微信TDengineECO。我们的团队会帮忙解决。 ## 给贡献者的礼品 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5be84bec3483ac2f79f43941465df3b50047e661..058c624e10876c3c077f4bd1d0a455a97f75f39c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,40 +1,36 @@ -# Contributing +# Contributing to TDengine -We appreciate contributions from all developers. Feel free to follow us, fork the repository, report bugs, and even submit your code on GitHub. However, we would like developers to follow the guidelines in this document to ensure effective cooperation. +TDengine Community Edition is free, open-source software. Its development is led by the TDengine Team, but we welcome contributions from all community members and open-source developers. This document describes how you can contribute, no matter whether you're a user or a developer yourself. -## Reporting a bug +## Bug reports -- Any users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. We would appreciate if you could provide **a detailed description** of the problem you encountered, including steps to reproduce it. +All users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. To ensure that the development team can locate and resolve the issue that you experienced, please include the following in your bug report: -- Attaching log files caused by the bug is really appreciated. +- A detailed description of the issue, including the steps to reproduce it. +- Any log files that may be relevant to the issue. -## Guidelines for committing code +## Code contributions -- You must agree to the **Contributor License Agreement(CLA) before submitting your code patch**. Follow the **[TAOSData CLA](https://cla-assistant.io/taosdata/TDengine)** link to read through and sign the agreement. If you do not accept the agreement, your contributions cannot be accepted. +Developers are encouraged to submit patches to the project, and all contributions, from minor documentation changes to bug fixes, are appreciated by our team. To ensure that your code can be merged successfully and improve the experience for other community members, we ask that you go through the following procedure before submitting a pull request: -- Please solve an issue or add a feature registered in the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. -- If no corresponding issue or feature is found in the issue tracker, please **create one**. -- When submitting your code to our repository, please create a pull request with the **issue number** included. +1. Read and accept the terms of the TAOS Data Contributor License Agreement (CLA) located at [https://cla-assistant.io/taosdata/TDengine](https://cla-assistant.io/taosdata/TDengine). -## Guidelines for communicating +2. For bug fixes, search the [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) to check whether the bug has already been filed. + - If the bug that you want to fix already exists in the issue tracker, review the previous discussion before submitting your patch. + - If the bug that you want to fix does not exist in the issue tracker, click **New issue** and file a report. + - Ensure that you note the issue number in your pull request when you submit your patch. + +3. Fork our repository to your GitHub account and create a branch for your patch. + **Important:** The `main` branch is for stable versions and cannot accept patches directly. For all code and documentation changes, create your own branch from the development branch `3.0` and not from `main`. + Note: For a documentation change, ensure that the branch name starts with `docs/` so that the change can be merged without running tests. + +4. Create a pull request to merge your changes into the development branch `3.0`, and our team members will review the request as soon as possible. -1. Please be **nice and polite** in the description. -2. **Active voice is better than passive voice in general**. Sentences in the active voice will highlight who is performing the action rather than the recipient of the action highlighted by the passive voice. -3. Documentation writing advice +If you encounter any difficulties or problems in contributing your code, you can join our [Discord server](https://discord.com/invite/VZdSuUg4pS) and receive assistance from members of the TDengine Team. -- Spell the product name "TDengine" correctly. "TD" is written in capital letters, and there is no space between "TD" and "engine" (**Correct spelling: TDengine**). -- Please **capitalize the first letter** of every sentence. -- Leave **only one space** after periods or other punctuation marks. -- Use **American spelling**. -- When possible, **use second person** rather than first person (e.g.“You are recommended to use a reverse proxy such as Nginx.” rather than “We recommend to use a reverse proxy such as Nginx.”). +## Expressing our thanks -5. Use **simple sentences**, rather than complex sentences. - -## Gifts for the contributors - -Developers, as long as you contribute to TDengine, whether it's code contributions to fix bugs or feature requests, or documentation changes, **you are eligible for a very special Contributor Souvenir Gift!** - -**You can choose one of the following gifts:** +To thank community members for your support, we are offering a free gift to any developer who submits at least one contribution. You can choose one of the following items:

-The TDengine community is committed to making TDengine accepted and used by more developers. - -Just fill out the **Contributor Submission Form** to choose your desired gift. - -- [Contributor Submission Form](https://page.ma.scrmtech.com/form/index?pf_uid=27715_2095&id=12100) +If you would like to claim your gift, send an email to [developer@tdengine.com](mailto:developer@tdengine.com?subject=Claiming&20my%20developer%20gift) including the following information: -## Contact us +- Your GitHub account name +- Your name and mailing address +- Your preferred gift -If you have any problems or questions that need help from us, please feel free to add our WeChat account: TDengineECO. +Note: Limit one per person. \ No newline at end of file diff --git a/cmake/taostools_CMakeLists.txt.in b/cmake/taostools_CMakeLists.txt.in index 82a70521258416e8fc08e1f5c5f7c1282d8149cf..b996ffcd17c7b8070a6fa0023bc6df2bb60c618e 100644 --- a/cmake/taostools_CMakeLists.txt.in +++ b/cmake/taostools_CMakeLists.txt.in @@ -2,7 +2,7 @@ # taos-tools ExternalProject_Add(taos-tools GIT_REPOSITORY https://github.com/taosdata/taos-tools.git - GIT_TAG 23e2b73 + GIT_TAG e00ebd9 SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools" BINARY_DIR "" #BUILD_IN_SOURCE TRUE diff --git a/docs/en/05-get-started/index.md b/docs/en/05-get-started/index.md index d80ec022686d4cf1941258a9651af706c8c11c7c..251581e98fbe3246d5c92cbc1d63cea432717f9e 100644 --- a/docs/en/05-get-started/index.md +++ b/docs/en/05-get-started/index.md @@ -20,7 +20,18 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; ``` -### Join TDengine Community +## Study TDengine Knowledge Map + +The TDengine Knowledge Map covers the various knowledge points of TDengine, revealing the invocation relationships and data flow between various conceptual entities. Learning and understanding the TDengine Knowledge Map will help you quickly master the TDengine knowledge system. + +

+
+ +
Diagram 1. TDengine Knowledge Map
+
+
+ +## Join TDengine Community diff --git a/docs/en/12-taos-sql/06-select.md b/docs/en/12-taos-sql/06-select.md index 0c55578efa0896d5b66799c8cf0a0da3cca51266..c087a9e9fb2f0af921aa031d41d124c66fbb0ae7 100644 --- a/docs/en/12-taos-sql/06-select.md +++ b/docs/en/12-taos-sql/06-select.md @@ -301,7 +301,7 @@ SELECT TIMEZONE(); ### Syntax ```txt -WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_ +WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_ ``` ### Specification diff --git a/docs/en/14-reference/03-connector/index.mdx b/docs/en/14-reference/03-connector/index.mdx index 4fd9c452d8602a2b131e725dfebf85627981ad0f..54031db618b45e3fb314b973f9837b4ce9b4062f 100644 --- a/docs/en/14-reference/03-connector/index.mdx +++ b/docs/en/14-reference/03-connector/index.mdx @@ -26,14 +26,13 @@ Using REST connection can support a broader range of operating systems as it doe TDengine version updates often add new features, and the connector versions in the list are the best-fit versions of the connector. -| **TDengine Versions** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** | -| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- | -| **3.0.0.0 and later** | 3.0.0 | current version | 3.0 branch | 3.0.0 | 3.0.0 | current version | -| **2.4.0.14 and up** | 2.0.38 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version | -| **2.4.0.6 and up** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version | -| **2.4.0.4 - 2.4.0.5** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version | -| **2.2.x.x ** | 2.0.36 | current version | master branch | n/a | 2.0.7 - 2.0.9 | current version | -| **2.0.x.x ** | 2.0.34 | current version | master branch | n/a | 2.0.1 - 2.0.6 | current version | +| **TDengine Versions** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** | +| --------------------------- | -------------- | -------------- | -------------- | ------------- | --------------- | --------------- | +| **3.0.0.0 and later** | 3.0.2 + | latest version | 3.0 branch | 3.0.0 | 3.0.0 | current version | +| **2.4.0.14 and up ** | 2.0.38 | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version | +| **2.4.0.4 - 2.4.0.13 ** | 2.0.37 | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version | +| **2.2.x.x ** | 2.0.36 | latest version | master branch | n/a | 2.0.7 - 2.0.9 | current version | +| **2.0.x.x ** | 2.0.34 | latest version | master branch | n/a | 2.0.1 - 2.0.6 | current version | ## Functional Features diff --git a/docs/zh/08-connector/index.md b/docs/zh/08-connector/index.md index e00e0b2fa2c63163e70e210077f46e6b92cdd3a2..eecf564b905ddbf1e930acbb196cfb80869b67e9 100644 --- a/docs/zh/08-connector/index.md +++ b/docs/zh/08-connector/index.md @@ -26,14 +26,13 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速 TDengine 版本更新往往会增加新的功能特性，列表中的连接器版本为连接器最佳适配版本。 -| **TDengine 版本** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** | -| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- | -| **3.0.0.0 及以上** | 3.0.0 | 当前版本 | 3.0 分支 | 3.0.0 | 3.0.0 | 当前版本 | -| **2.4.0.14 及以上** | 2.0.38 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 | -| **2.4.0.6 及以上** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 | -| **2.4.0.4 - 2.4.0.5** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 | -| **2.2.x.x ** | 2.0.36 | 当前版本 | master 分支 | n/a | 2.0.7 - 2.0.9 | 当前版本 | -| **2.0.x.x ** | 2.0.34 | 当前版本 | master 分支 | n/a | 2.0.1 - 2.0.6 | 当前版本 | +| **TDengine 版本** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** | +| ---------------------- | --------- | ---------- | ------------ | ------------- | --------------- | -------- | +| **3.0.0.0 及以上** | 3.0.2以上 | 当前版本 | 3.0 分支 | 3.0.0 | 3.0.0 | 当前版本 | +| **2.4.0.14 及以上** | 2.0.38 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 | +| **2.4.0.4 - 2.4.0.13** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 | +| **2.2.x.x ** | 2.0.36 | 当前版本 | master 分支 | n/a | 2.0.7 - 2.0.9 | 当前版本 | +| **2.0.x.x ** | 2.0.34 | 当前版本 | master 分支 | n/a | 2.0.1 - 2.0.6 | 当前版本 | ## 功能特性 diff --git a/docs/zh/12-taos-sql/06-select.md b/docs/zh/12-taos-sql/06-select.md index 9d4faae23a82fb8b328fe0df39c82789a7cc2ad2..3b681f401c826a8b93ba446ad2804cb37c8c7bf7 100644 --- a/docs/zh/12-taos-sql/06-select.md +++ b/docs/zh/12-taos-sql/06-select.md @@ -302,7 +302,7 @@ SELECT TIMEZONE(); ### 语法 ```txt -WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_ +WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_ ``` ### 正则表达式规范 diff --git a/include/common/tmsg.h b/include/common/tmsg.h index 556c4a9a1ff6e82491c17dedbb8938814b5ae103..02d4c2279c543aab42f1d0dd933a282d99ef7386 100644 --- a/include/common/tmsg.h +++ b/include/common/tmsg.h @@ -1615,15 +1615,21 @@ typedef struct SSubQueryMsg { uint64_t taskId; int64_t refId; int32_t execId; + int32_t msgMask; int8_t taskType; int8_t explain; int8_t needFetch; - uint32_t sqlLen; // the query sql, - uint32_t phyLen; - int32_t msgMask; - char msg[]; + uint32_t sqlLen; + char *sql; + uint32_t msgLen; + char *msg; } SSubQueryMsg; +int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq); +int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq); +void tFreeSSubQueryMsg(SSubQueryMsg *pReq); + + typedef struct { SMsgHead header; uint64_t sId; @@ -1732,6 +1738,13 @@ typedef struct { int32_t execId; } STaskDropReq; +int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq); +int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq); + +int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp); +int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp); + + typedef struct { int32_t code; } STaskDropRsp; diff --git a/include/libs/sync/sync.h b/include/libs/sync/sync.h index 3a808ac6f331f8e774ff37a0e28db1a7c7e6d188..513ba8cb3432de72911be0a56a8c5dc87c05dc60 100644 --- a/include/libs/sync/sync.h +++ b/include/libs/sync/sync.h @@ -36,7 +36,7 @@ extern "C" { #define SYNC_DEL_WAL_MS (1000 * 60) #define SYNC_ADD_QUORUM_COUNT 3 #define SYNC_MNODE_LOG_RETENTION 10000 -#define SYNC_VNODE_LOG_RETENTION 100 +#define SYNC_VNODE_LOG_RETENTION 20 #define SNAPSHOT_MAX_CLOCK_SKEW_MS 1000 * 10 #define SNAPSHOT_WAIT_MS 1000 * 30 diff --git a/packaging/MPtestJenkinsfile b/packaging/MPtestJenkinsfile index 5dc8024f29ddc0f09727d167aceb3f2516886488..dad5b7f129440e1fcf38524c95097b9bf8c1fd03 100644 --- a/packaging/MPtestJenkinsfile +++ b/packaging/MPtestJenkinsfile @@ -40,27 +40,32 @@ pipeline { choice( name: 'sourcePath', choices: ['nas','web'], - description: 'choice which way to download the installation pacakge;web is Office Web and nas means taos nas server ' + description: 'Choice which way to download the installation pacakge;web is Office Web and nas means taos nas server ' + ) + choice( + name: 'verMode', + choices: ['all','community','enterprise'], + description: 'Choice which types of package you want do check ' ) string ( name:'version', - defaultValue:'3.0.1.6', - description: 'release version number,eg: 3.0.0.1 or 3.0.0.' + defaultValue:'3.0.1.7', + description: 'Release version number,eg: 3.0.0.1 or 3.0.0.' ) string ( name:'baseVersion', - defaultValue:'3.0.1.6', - description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1' + defaultValue:'3.0.1.7', + description: 'The number of baseVerison is generally not modified.Now it is 3.0.0.1' ) string ( name:'toolsVersion', defaultValue:'2.2.7', - description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1' + description: 'Release version number,eg:2.2.0' ) string ( name:'toolsBaseVersion', defaultValue:'2.1.2', - description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1' + description: 'The number of baseVerison is generally not modified.Now it is 2.1.2' ) } environment{ @@ -68,10 +73,10 @@ pipeline { TDINTERNAL_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal' TDENGINE_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal/community' BRANCH_NAME = '3.0' - - TD_SERVER_TAR = "TDengine-server-${version}-Linux-x64.tar.gz" + + TD_SERVER_TAR = "${preServerPackag}-${version}-Linux-x64.tar.gz" BASE_TD_SERVER_TAR = "TDengine-server-${baseVersion}-Linux-x64.tar.gz" - + TD_SERVER_ARM_TAR = "TDengine-server-${version}-Linux-arm64.tar.gz" BASE_TD_SERVER_ARM_TAR = "TDengine-server-${baseVersion}-Linux-arm64.tar.gz" @@ -108,19 +113,28 @@ pipeline { timeout(time: 30, unit: 'MINUTES'){ sync_source("${BRANCH_NAME}") sh ''' + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done + ''' + + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py - ''' - sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py - ''' - sh ''' + bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + ''' + + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t deb + python3 checkPackageRuning.py ''' } } @@ -131,22 +145,30 @@ pipeline { timeout(time: 30, unit: 'MINUTES'){ sync_source("${BRANCH_NAME}") sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done ''' + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py ''' + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t deb + python3 checkPackageRuning.py dpkg -r tdengine - ''' - + ''' } } } @@ -156,19 +178,28 @@ pipeline { timeout(time: 30, unit: 'MINUTES'){ sync_source("${BRANCH_NAME}") sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done ''' - sh ''' + + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py ''' + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t rpm + python3 checkPackageRuning.py ''' } } @@ -179,21 +210,30 @@ pipeline { timeout(time: 30, unit: 'MINUTES'){ sync_source("${BRANCH_NAME}") sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done ''' + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py ''' + sh ''' cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t rpm + python3 checkPackageRuning.py sudo rpm -e tdengine - ''' + ''' } } } @@ -203,9 +243,16 @@ pipeline { timeout(time: 30, unit: 'MINUTES'){ sync_source("${BRANCH_NAME}") sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_SERVER_ARM_TAR} ${version} ${BASE_TD_SERVER_ARM_TAR} ${baseVersion} server ${sourcePath} - python3 checkPackageRuning.py + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c arm64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done ''' } } @@ -219,8 +266,16 @@ pipeline { steps { timeout(time: 30, unit: 'MINUTES'){ sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_CLIENT_TAR} ${version} ${BASE_TD_CLIENT_TAR} ${baseVersion} client ${sourcePath} + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f client -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done python3 checkPackageRuning.py 192.168.0.21 ''' } @@ -231,8 +286,10 @@ pipeline { steps { timeout(time: 30, unit: 'MINUTES'){ sh ''' + verModeList=community cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_CLIENT_LITE_TAR} ${version} ${BASE_TD_CLIENT_LITE_TAR} ${baseVersion} client ${sourcePath} + bash testpackage.sh -f server -m ${verModeSin} -f client -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py python3 checkPackageRuning.py 192.168.0.24 ''' } @@ -245,8 +302,16 @@ pipeline { steps { timeout(time: 30, unit: 'MINUTES'){ sh ''' - cd ${TDENGINE_ROOT_DIR}/packaging - bash testpackage.sh ${TD_CLIENT_ARM_TAR} ${version} ${BASE_TD_CLIENT_ARM_TAR} ${baseVersion} client ${sourcePath} + if [ "${verMode}" = "all" ];then + verMode="community enterprise" + fi + verModeList=${verMode} + for verModeSin in ${verModeList} + do + cd ${TDENGINE_ROOT_DIR}/packaging + bash testpackage.sh -f server -m ${verModeSin} -f client -l false -c arm64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar + python3 checkPackageRuning.py + done python3 checkPackageRuning.py 192.168.0.21 ''' } diff --git a/packaging/release.sh b/packaging/release.sh index c07331a0df0ce202bf304ba070c2fd239fb5dfec..a3334e734dbf0efabbf14229fdaa39b331e0a85f 100755 --- a/packaging/release.sh +++ b/packaging/release.sh @@ -221,12 +221,12 @@ if [[ "$cpuType" == "x64" ]] || [[ "$cpuType" == "aarch64" ]] || [[ "$cpuType" = # community-version compile cmake ../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DPAGMODE=${pagMode} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro} elif [ "$verMode" == "cloud" ]; then - cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro} + cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro} elif [ "$verMode" == "cluster" ]; then if [[ "$dbName" != "taos" ]]; then replace_enterprise_$dbName fi - cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro} + cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro} fi else echo "input cpuType=${cpuType} error!!!" diff --git a/packaging/testpackage.sh b/packaging/testpackage.sh index b0e33835ddf5316d4d8374baaa657176642af181..846c8d160fe446d5a99a210ab33a1e53785573b6 100755 --- a/packaging/testpackage.sh +++ b/packaging/testpackage.sh @@ -1,15 +1,72 @@ #!/bin/sh + + +function usage() { + echo "$0" + echo -e "\t -f test file type,server/client/tools/" + echo -e "\t -m pacakage version Type,community/enterprise" + echo -e "\t -l package type,lite or not" + echo -e "\t -c operation type,x64/arm64" + echo -e "\t -v pacakage version,3.0.1.7" + echo -e "\t -o pacakage version,3.0.1.7" + echo -e "\t -s source Path,web/nas" + echo -e "\t -t package Type,tar/rpm/deb" + echo -e "\t -h help" +} + + #parameter scriptDir=$(dirname $(readlink -f $0)) -packgeName=$1 -version=$2 -originPackageName=$3 -originversion=$4 -testFile=$5 -# sourcePath:web/nas -sourcePath=$6 +version="3.0.1.7" +originversion="3.0.1.7" +testFile="server" +verMode="communtity" +sourcePath="nas" +cpuType="x64" +lite="true" +packageType="tar" subFile="taos.tar.gz" - +while getopts "m:c:f:l:s:o:t:v:h" opt; do + case $opt in + m) + verMode=$OPTARG + ;; + v) + version=$OPTARG + ;; + f) + testFile=$OPTARG + ;; + l) + lite=$OPTARG + ;; + s) + sourcePath=$OPTARG + ;; + o) + originversion=$OPTARG + ;; + c) + cpuType=$OPTARG + ;; + t) + packageType=$OPTARG + ;; + h) + usage + exit 0 + ;; + ?) + echo "Invalid option: -$OPTARG" + usage + exit 0 + ;; + esac +done + + + +echo "testFile:${testFile},verMode:${verMode},lite:${lite},cpuType:${cpuType},packageType:${packageType},version-${version},originversion:${originversion},sourcePath:${sourcePath}" # Color setting RED='\033[41;30m' GREEN='\033[1;32m' @@ -21,20 +78,40 @@ BLUE_DARK='\033[0;34m' GREEN_UNDERLINE='\033[4;32m' NC='\033[0m' -if [ ${testFile} = "server" ];then - tdPath="TDengine-server-${version}" - originTdpPath="TDengine-server-${originversion}" +if [[ ${verMode} = "enterprise" ]];then + prePackag="TDengine-enterprise-${testFile}" +elif [ ${verMode} = "community" ];then + prePackag="TDengine-${testFile}" +fi +if [ ${lite} = "true" ];then + packageLite="-Lite" +elif [ ${lite} = "false" ];then + packageLite="" +fi +if [[ "$packageType" = "tar" ]] ;then + packageType="tar.gz" +fi + +tdPath="${prePackag}-${version}" +originTdpPath="${prePackag}-${originversion}" + +packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}" +originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}" + +if [ "$testFile" == "server" ] ;then installCmd="install.sh" elif [ ${testFile} = "client" ];then - tdPath="TDengine-client-${version}" - originTdpPath="TDengine-client-${originversion}" installCmd="install_client.sh" elif [ ${testFile} = "tools" ];then tdPath="taosTools-${version}" originTdpPath="taosTools-${originversion}" + packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}" + originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}" installCmd="install-taostools.sh" fi + +echo "tdPath:${tdPath},originTdpPath:${originTdpPath},packgeName:${packgeName},originPackageName:${originPackageName}" function cmdInstall { command=$1 if command -v ${command} ;then @@ -76,16 +153,16 @@ file=$1 versionPath=$2 sourceP=$3 nasServerIP="192.168.1.131" -packagePath="/nas/TDengine/v${versionPath}/community" +packagePath="/nas/TDengine/v${versionPath}/${verMode}" if [ -f ${file} ];then echoColor YD "${file} already exists ,it will delete it and download it again " rm -rf ${file} fi -if [ ${sourceP} = 'web' ];then +if [[ ${sourceP} = 'web' ]];then echoColor BD "====download====:wget https://www.taosdata.com/assets-download/3.0/${file}" wget https://www.taosdata.com/assets-download/3.0/${file} -elif [ ${sourceP} = 'nas' ];then +elif [[ ${sourceP} = 'nas' ]];then echoColor BD "====download====:scp root@${nasServerIP}:${packagePath}/${file} ." scp root@${nasServerIP}:${packagePath}/${file} . fi diff --git a/source/client/src/clientMain.c b/source/client/src/clientMain.c index 0aa88382febc2204a69a8198fc3ed5f1848dee70..423cfcdc5b1f5c30f16cf1dcb50ce1b2bb28c53e 100644 --- a/source/client/src/clientMain.c +++ b/source/client/src/clientMain.c @@ -1106,6 +1106,8 @@ int taos_get_table_vgId(TAOS *taos, const char *db, const char *table, int *vgId return terrno; } + pRequest->syncQuery = true; + STscObj *pTscObj = pRequest->pTscObj; SCatalog *pCtg = NULL; code = catalogGetHandle(pTscObj->pAppInfo->clusterId, &pCtg); diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index b5fffea73d8b238237d69f96ad5039a3c72bd221..607db72ec67e81f5fee5ab40bb4a7e2383400c62 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -36,7 +36,7 @@ int32_t colDataGetFullLength(const SColumnInfoData* pColumnInfoData, int32_t num if (IS_VAR_DATA_TYPE(pColumnInfoData->info.type)) { return pColumnInfoData->varmeta.length + sizeof(int32_t) * numOfRows; } else { - return pColumnInfoData->info.bytes * numOfRows + BitmapLen(numOfRows); + return ((pColumnInfoData->info.type == TSDB_DATA_TYPE_NULL) ? 0 : pColumnInfoData->info.bytes * numOfRows) + BitmapLen(numOfRows); } } @@ -1960,6 +1960,7 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf) memset(pBuf, 0, sizeof(pBuf)); char* pData = colDataGetVarData(pColInfoData, j); int32_t dataSize = TMIN(sizeof(pBuf), varDataLen(pData)); + dataSize = TMIN(dataSize, 50); memcpy(pBuf, varDataVal(pData), dataSize); len += snprintf(dumpBuf + len, size - len, " %15s |", pBuf); if (len >= size - 1) return dumpBuf; diff --git a/source/common/src/tglobal.c b/source/common/src/tglobal.c index 27dcbd5be3d4fd3ccab8475ba26d69edde9050fc..f2d8b9aa7c3460c035db7dfcee82f76eb62749d5 100644 --- a/source/common/src/tglobal.c +++ b/source/common/src/tglobal.c @@ -277,7 +277,9 @@ static int32_t taosAddServerLogCfg(SConfig *pCfg) { static int32_t taosAddClientCfg(SConfig *pCfg) { char defaultFqdn[TSDB_FQDN_LEN] = {0}; int32_t defaultServerPort = 6030; - if (taosGetFqdn(defaultFqdn) != 0) return -1; + if (taosGetFqdn(defaultFqdn) != 0) { + strcpy(defaultFqdn, "localhost"); + } if (cfgAddString(pCfg, "firstEp", "", 1) != 0) return -1; if (cfgAddString(pCfg, "secondEp", "", 1) != 0) return -1; diff --git a/source/common/src/tmsg.c b/source/common/src/tmsg.c index 52d9c5b1991001ecf25dc53465d64bdb14166a49..35900638dd9f3b1c03f13744ce1ceb0f48a11129 100644 --- a/source/common/src/tmsg.c +++ b/source/common/src/tmsg.c @@ -4643,6 +4643,178 @@ int32_t tDeserializeSMqHbReq(void *buf, int32_t bufLen, SMqHbReq *pReq) { return 0; } +int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) { + int32_t headLen = sizeof(SMsgHead); + if (buf != NULL) { + buf = (char *)buf + headLen; + bufLen -= headLen; + } + + SEncoder encoder = {0}; + tEncoderInit(&encoder, buf, bufLen); + if (tStartEncode(&encoder) < 0) return -1; + + if (tEncodeU64(&encoder, pReq->sId) < 0) return -1; + if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1; + if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1; + if (tEncodeI64(&encoder, pReq->refId) < 0) return -1; + if (tEncodeI32(&encoder, pReq->execId) < 0) return -1; + if (tEncodeI32(&encoder, pReq->msgMask) < 0) return -1; + if (tEncodeI8(&encoder, pReq->taskType) < 0) return -1; + if (tEncodeI8(&encoder, pReq->explain) < 0) return -1; + if (tEncodeI8(&encoder, pReq->needFetch) < 0) return -1; + if (tEncodeU32(&encoder, pReq->sqlLen) < 0) return -1; + if (tEncodeCStrWithLen(&encoder, pReq->sql, pReq->sqlLen) < 0) return -1; + if (tEncodeU32(&encoder, pReq->msgLen) < 0) return -1; + if (tEncodeBinary(&encoder, (uint8_t*)pReq->msg, pReq->msgLen) < 0) return -1; + + tEndEncode(&encoder); + + int32_t tlen = encoder.pos; + tEncoderClear(&encoder); + + if (buf != NULL) { + SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen); + pHead->vgId = htonl(pReq->header.vgId); + pHead->contLen = htonl(tlen + headLen); + } + + return tlen + headLen; +} + +int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) { + int32_t headLen = sizeof(SMsgHead); + + SMsgHead *pHead = buf; + pHead->vgId = pReq->header.vgId; + pHead->contLen = pReq->header.contLen; + + SDecoder decoder = {0}; + tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen); + + if (tStartDecode(&decoder) < 0) return -1; + + if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1; + if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1; + if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1; + if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1; + if (tDecodeI32(&decoder, &pReq->msgMask) < 0) return -1; + if (tDecodeI8(&decoder, &pReq->taskType) < 0) return -1; + if (tDecodeI8(&decoder, &pReq->explain) < 0) return -1; + if (tDecodeI8(&decoder, &pReq->needFetch) < 0) return -1; + if (tDecodeU32(&decoder, &pReq->sqlLen) < 0) return -1; + if (tDecodeCStrAlloc(&decoder, &pReq->sql) < 0) return -1; + if (tDecodeU32(&decoder, &pReq->msgLen) < 0) return -1; + if (tDecodeBinaryAlloc(&decoder, (void**)&pReq->msg, NULL) < 0) return -1; + + tEndDecode(&decoder); + + tDecoderClear(&decoder); + return 0; +} + +void tFreeSSubQueryMsg(SSubQueryMsg *pReq) { + if (NULL == pReq) { + return; + } + + taosMemoryFreeClear(pReq->sql); + taosMemoryFreeClear(pReq->msg); +} + +int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) { + int32_t headLen = sizeof(SMsgHead); + if (buf != NULL) { + buf = (char *)buf + headLen; + bufLen -= headLen; + } + + SEncoder encoder = {0}; + tEncoderInit(&encoder, buf, bufLen); + if (tStartEncode(&encoder) < 0) return -1; + + if (tEncodeU64(&encoder, pReq->sId) < 0) return -1; + if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1; + if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1; + if (tEncodeI64(&encoder, pReq->refId) < 0) return -1; + if (tEncodeI32(&encoder, pReq->execId) < 0) return -1; + + tEndEncode(&encoder); + + int32_t tlen = encoder.pos; + tEncoderClear(&encoder); + + if (buf != NULL) { + SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen); + pHead->vgId = htonl(pReq->header.vgId); + pHead->contLen = htonl(tlen + headLen); + } + + return tlen + headLen; +} + +int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) { + int32_t headLen = sizeof(SMsgHead); + + SMsgHead *pHead = buf; + pHead->vgId = pReq->header.vgId; + pHead->contLen = pReq->header.contLen; + + SDecoder decoder = {0}; + tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen); + + if (tStartDecode(&decoder) < 0) return -1; + + if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1; + if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1; + if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1; + if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1; + + tEndDecode(&decoder); + + tDecoderClear(&decoder); + return 0; +} + +int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) { + SEncoder encoder = {0}; + tEncoderInit(&encoder, buf, bufLen); + if (tStartEncode(&encoder) < 0) return -1; + + if (tEncodeI32(&encoder, pRsp->code) < 0) return -1; + if (tEncodeCStr(&encoder, pRsp->tbFName) < 0) return -1; + if (tEncodeI32(&encoder, pRsp->sversion) < 0) return -1; + if (tEncodeI32(&encoder, pRsp->tversion) < 0) return -1; + if (tEncodeI64(&encoder, pRsp->affectedRows) < 0) return -1; + + tEndEncode(&encoder); + + int32_t tlen = encoder.pos; + tEncoderClear(&encoder); + + return tlen; +} + +int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) { + SDecoder decoder = {0}; + tDecoderInit(&decoder, (char *)buf, bufLen); + + if (tStartDecode(&decoder) < 0) return -1; + + if (tDecodeI32(&decoder, &pRsp->code) < 0) return -1; + if (tDecodeCStrTo(&decoder, pRsp->tbFName) < 0) return -1; + if (tDecodeI32(&decoder, &pRsp->sversion) < 0) return -1; + if (tDecodeI32(&decoder, &pRsp->tversion) < 0) return -1; + if (tDecodeI64(&decoder, &pRsp->affectedRows) < 0) return -1; + + tEndDecode(&decoder); + + tDecoderClear(&decoder); + return 0; +} + int32_t tSerializeSSchedulerHbReq(void *buf, int32_t bufLen, SSchedulerHbReq *pReq) { int32_t headLen = sizeof(SMsgHead); diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c index 24240e82efc2a535b7da096888a63fd43f8ce3f7..4aa07cad98533645b00fb17c9d249920e390fe65 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c @@ -148,8 +148,8 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId); if (pVnode == NULL) { - dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d", pHead->vgId, pMsg, terrstr(), - TMSG_INFO(pMsg->msgType), qtype); + dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d contLen:%d", pHead->vgId, pMsg, terrstr(), + TMSG_INFO(pMsg->msgType), qtype, pHead->contLen); return terrno != 0 ? terrno : -1; } diff --git a/source/dnode/vnode/inc/vnode.h b/source/dnode/vnode/inc/vnode.h index 370103c222b5d4a34b3865f7727b3e5f5035b745..27084700b0b3b3a01bdfdf5064094fcd2a484c54 100644 --- a/source/dnode/vnode/inc/vnode.h +++ b/source/dnode/vnode/inc/vnode.h @@ -239,6 +239,7 @@ int32_t tqReaderSetDataMsg(STqReader *pReader, const SSubmitReq *pMsg, int64_t v bool tqNextDataBlock(STqReader *pReader); bool tqNextDataBlockFilterOut(STqReader *pReader, SHashObj *filterOutUids); int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader); +int32_t tqRetrieveTaosxBlock(STqReader *pReader, SArray *blocks, SArray *schemas); int32_t vnodeEnqueueStreamMsg(SVnode *pVnode, SRpcMsg *pMsg); diff --git a/source/dnode/vnode/src/sma/smaTimeRange.c b/source/dnode/vnode/src/sma/smaTimeRange.c index f5ba7b501412841a943ab75dcf481cf999dc14f9..a2c9484693a720243ccb205f0d311289c86f254e 100644 --- a/source/dnode/vnode/src/sma/smaTimeRange.c +++ b/source/dnode/vnode/src/sma/smaTimeRange.c @@ -203,10 +203,13 @@ static int32_t tdProcessTSmaInsertImpl(SSma *pSma, int64_t indexUid, const char goto _err; } - SBatchDeleteReq deleteReq; + SBatchDeleteReq deleteReq = {0}; SSubmitReq *pSubmitReq = tqBlockToSubmit(pSma->pVnode, (const SArray *)msg, pTsmaStat->pTSchema, &pTsmaStat->pTSma->schemaTag, true, pTsmaStat->pTSma->dstTbUid, pTsmaStat->pTSma->dstTbName, &deleteReq); + // TODO deleteReq + taosArrayDestroy(deleteReq.deleteReqs); + if (!pSubmitReq) { smaError("vgId:%d, failed to gen submit blk while tsma insert for smaIndex %" PRIi64 " since %s", SMA_VID(pSma), diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 092128fe6efb417250acb3d4ba0b9b7edfc1d901..5e35e34b87bd83b23ed0428e2aab0f661c38ad0f 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -671,7 +671,6 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) { SSubmitReq* pCont = (SSubmitReq*)&pHead->body; if (tqTaosxScanLog(pTq, pHandle, pCont, &taosxRsp) < 0) { - /*ASSERT(0);*/ } if (taosxRsp.blockNum > 0 /* threshold */) { tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer); diff --git a/source/dnode/vnode/src/tq/tqExec.c b/source/dnode/vnode/src/tq/tqExec.c index 48c14bc758bc4ee6271b20a731b7f0a25be2ea4b..3887f72740491701744b3fb9753e128e2d074428 100644 --- a/source/dnode/vnode/src/tq/tqExec.c +++ b/source/dnode/vnode/src/tq/tqExec.c @@ -44,7 +44,7 @@ static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, SMqDataRsp* pRs return 0; } -static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) { +static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t n) { SMetaReader mr = {0}; metaReaderInit(&mr, pTq->pVnode->pMeta, 0); // TODO add reference to gurantee success @@ -52,8 +52,10 @@ static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) { metaReaderClear(&mr); return -1; } - char* tbName = strdup(mr.me.name); - taosArrayPush(pRsp->blockTbName, &tbName); + for (int32_t i = 0; i < n; i++) { + char* tbName = strdup(mr.me.name); + taosArrayPush(pRsp->blockTbName, &tbName); + } metaReaderClear(&mr); return 0; } @@ -111,7 +113,7 @@ int32_t tqScanData(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, STqOffs if (pRsp->withTbName) { if (pRsp->rspOffset.type == TMQ_OFFSET__LOG) { int64_t uid = pExec->pExecReader->msgIter.uid; - tqAddTbNameToRsp(pTq, uid, pRsp); + tqAddTbNameToRsp(pTq, uid, pRsp, 1); } else { pRsp->withTbName = false; } @@ -155,7 +157,7 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, STaosxRsp* pRsp, SMqMeta int64_t uid = 0; if (pOffset->type == TMQ_OFFSET__LOG) { uid = pExec->pExecReader->msgIter.uid; - if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) { + if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, 1) < 0) { continue; } } else { @@ -225,18 +227,30 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp STqExecHandle* pExec = &pHandle->execHandle; ASSERT(pExec->subType != TOPIC_SUB_TYPE__COLUMN); + SArray* pBlocks = taosArrayInit(0, sizeof(SSDataBlock)); + SArray* pSchemas = taosArrayInit(0, sizeof(void*)); + if (pExec->subType == TOPIC_SUB_TYPE__TABLE) { STqReader* pReader = pExec->pExecReader; tqReaderSetDataMsg(pReader, pReq, 0); while (tqNextDataBlock(pReader)) { - SSDataBlock block = {0}; - if (tqRetrieveDataBlock(&block, pReader) < 0) { + /*SSDataBlock block = {0};*/ + /*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/ + /*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/ + /*}*/ + + taosArrayClear(pBlocks); + taosArrayClear(pSchemas); + if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) { if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue; } if (pRsp->withTbName) { int64_t uid = pExec->pExecReader->msgIter.uid; - if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) { - blockDataFreeRes(&block); + if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) { + taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes); + taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper); + pBlocks = taosArrayInit(0, sizeof(SSDataBlock)); + pSchemas = taosArrayInit(0, sizeof(void*)); continue; } } @@ -255,25 +269,37 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp pRsp->createTableNum++; } } - tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock), - pTq->pVnode->config.tsdbCfg.precision); - blockDataFreeRes(&block); - tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp); - pRsp->blockNum++; + for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) { + SSDataBlock* pBlock = taosArrayGet(pBlocks, i); + tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock), + pTq->pVnode->config.tsdbCfg.precision); + blockDataFreeRes(pBlock); + SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i); + taosArrayPush(pRsp->blockSchema, &pSW); + pRsp->blockNum++; + } } } else if (pExec->subType == TOPIC_SUB_TYPE__DB) { STqReader* pReader = pExec->pExecReader; tqReaderSetDataMsg(pReader, pReq, 0); while (tqNextDataBlockFilterOut(pReader, pExec->execDb.pFilterOutTbUid)) { - SSDataBlock block = {0}; - if (tqRetrieveDataBlock(&block, pReader) < 0) { + /*SSDataBlock block = {0};*/ + /*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/ + /*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/ + /*}*/ + taosArrayClear(pBlocks); + taosArrayClear(pSchemas); + if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) { if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue; } if (pRsp->withTbName) { int64_t uid = pExec->pExecReader->msgIter.uid; - if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) { - blockDataFreeRes(&block); - continue; + if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) { + taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes); + taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper); + pBlocks = taosArrayInit(0, sizeof(SSDataBlock)); + pSchemas = taosArrayInit(0, sizeof(void*)); + return -1; } } if (pHandle->fetchMeta) { @@ -291,14 +317,26 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp pRsp->createTableNum++; } } - tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock), - pTq->pVnode->config.tsdbCfg.precision); - blockDataFreeRes(&block); - tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp); - pRsp->blockNum++; + /*tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),*/ + /*pTq->pVnode->config.tsdbCfg.precision);*/ + /*blockDataFreeRes(&block);*/ + /*tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);*/ + /*pRsp->blockNum++;*/ + for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) { + SSDataBlock* pBlock = taosArrayGet(pBlocks, i); + tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock), + pTq->pVnode->config.tsdbCfg.precision); + blockDataFreeRes(pBlock); + SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i); + taosArrayPush(pRsp->blockSchema, &pSW); + pRsp->blockNum++; + } } } + taosArrayDestroy(pBlocks); + taosArrayDestroy(pSchemas); + if (pRsp->blockNum == 0) { return -1; } diff --git a/source/dnode/vnode/src/tq/tqRead.c b/source/dnode/vnode/src/tq/tqRead.c index 5c5ba1205e1229ae0ece42bbfb67ab2212e22ada..afb7ac39de8ff5e6582c96ef627371646571dfe4 100644 --- a/source/dnode/vnode/src/tq/tqRead.c +++ b/source/dnode/vnode/src/tq/tqRead.c @@ -556,7 +556,7 @@ FAIL: return -1; } -int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* schemas) { +int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas) { int32_t sversion = htonl(pReader->pBlock->sversion); if (pReader->cachedSchemaSuid == 0 || pReader->cachedSchemaVer != sversion || @@ -592,9 +592,10 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch int32_t colAtMost = pSchemaWrapper->nCols; int32_t curRow = 0; + int32_t lastRow = 0; char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols); - if (assigned) return -1; + if (assigned == NULL) return -1; tInitSubmitBlkIter(&pReader->msgIter, pReader->pBlock, &pReader->blkIter); STSRowIter iter = {0}; @@ -605,11 +606,13 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch bool buildNew = false; tdSTSRowIterReset(&iter, row); + tqDebug("vgId:%d, row of block %d", pReader->pWalReader->pWal->cfg.vgId, curRow); for (int32_t i = 0; i < colAtMost; i++) { SCellVal sVal = {0}; if (!tdSTSRowIterFetch(&iter, pSchemaWrapper->pSchema[i].colId, pSchemaWrapper->pSchema[i].type, &sVal)) { break; } + tqDebug("vgId:%d, %d col, type %d", pReader->pWalReader->pWal->cfg.vgId, i, sVal.valType); if (curRow == 0) { assigned[i] = sVal.valType != TD_VTYPE_NONE; buildNew = true; @@ -623,27 +626,42 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch } if (buildNew) { - SSDataBlock block; - SSchemaWrapper sw; - if (tqMaskBlock(&sw, &block, pSchemaWrapper, assigned) < 0) { + if (taosArrayGetSize(blocks) > 0) { + SSDataBlock* pLastBlock = taosArrayGetLast(blocks); + pLastBlock->info.rows = curRow - lastRow; + lastRow = curRow; + } + SSDataBlock* pBlock = createDataBlock(); + SSchemaWrapper* pSW = taosMemoryCalloc(1, sizeof(SSchemaWrapper)); + if (tqMaskBlock(pSW, pBlock, pSchemaWrapper, assigned) < 0) { + blockDataDestroy(pBlock); goto FAIL; } + SSDataBlock block = {0}; + assignOneDataBlock(&block, pBlock); + blockDataDestroy(pBlock); + + tqDebug("vgId:%d, build new block, col %d", pReader->pWalReader->pWal->cfg.vgId, + (int32_t)taosArrayGetSize(block.pDataBlock)); taosArrayPush(blocks, &block); - taosArrayPush(schemas, &sw); + taosArrayPush(schemas, &pSW); } SSDataBlock* pBlock = taosArrayGetLast(blocks); pBlock->info.uid = pReader->msgIter.uid; - pBlock->info.rows = pReader->msgIter.numOfRows; + pBlock->info.rows = 0; pBlock->info.version = pReader->pMsg->version; + tqDebug("vgId:%d, taosx scan, block num: %d", pReader->pWalReader->pWal->cfg.vgId, + (int32_t)taosArrayGetSize(blocks)); + if (blockDataEnsureCapacity(pBlock, pReader->msgIter.numOfRows - curRow) < 0) { terrno = TSDB_CODE_OUT_OF_MEMORY; goto FAIL; } - tdSTSRowIterInit(&iter, pTschema); + tdSTSRowIterReset(&iter, row); for (int32_t i = 0; i < taosArrayGetSize(pBlock->pDataBlock); i++) { SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, i); SCellVal sVal = {0}; @@ -654,12 +672,16 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch ASSERT(sVal.valType != TD_VTYPE_NONE); - if (colDataAppend(pColData, curRow, sVal.val, sVal.valType != TD_VTYPE_NORM) < 0) { + if (colDataAppend(pColData, curRow, sVal.val, sVal.valType == TD_VTYPE_NULL) < 0) { goto FAIL; } + tqDebug("vgId:%d, row %d col %d append %d", pReader->pWalReader->pWal->cfg.vgId, curRow, i, + sVal.valType == TD_VTYPE_NULL); } curRow++; } + SSDataBlock* pLastBlock = taosArrayGetLast(blocks); + pLastBlock->info.rows = curRow - lastRow; taosMemoryFree(assigned); return 0; diff --git a/source/dnode/vnode/src/tq/tqSink.c b/source/dnode/vnode/src/tq/tqSink.c index b2624d1bc1d2e31623f91aad84d55c26f28cabb1..30cde5d475fcc02c65c9b535b4b6986857c8324e 100644 --- a/source/dnode/vnode/src/tq/tqSink.c +++ b/source/dnode/vnode/src/tq/tqSink.c @@ -236,6 +236,7 @@ SSubmitReq* tqBlockToSubmit(SVnode* pVnode, const SArray* pBlocks, const STSchem SSDataBlock* pDataBlock = taosArrayGet(pBlocks, i); if (pDataBlock->info.type == STREAM_DELETE_RESULT) { pDeleteReq->suid = suid; + pDeleteReq->deleteReqs = taosArrayInit(0, sizeof(SSingleDeleteReq)); tqBuildDeleteReq(pVnode, stbFullName, pDataBlock, pDeleteReq); continue; } @@ -349,7 +350,6 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d .contLen = len + sizeof(SMsgHead), }; if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) { - rpcFreeCont(serializedDeleteReq); tqDebug("failed to put delete req into write-queue since %s", terrstr()); } } else { @@ -476,12 +476,12 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d cap += sizeof(SSubmitBlk) + schemaLen + rows * maxLen; - SSubmitReq* ret = rpcMallocCont(cap); - ret->header.vgId = pVnode->config.vgId; - ret->length = sizeof(SSubmitReq); - ret->numOfBlocks = htonl(1); + SSubmitReq* pSubmit = rpcMallocCont(cap); + pSubmit->header.vgId = pVnode->config.vgId; + pSubmit->length = sizeof(SSubmitReq); + pSubmit->numOfBlocks = htonl(1); - SSubmitBlk* blkHead = POINTER_SHIFT(ret, sizeof(SSubmitReq)); + SSubmitBlk* blkHead = POINTER_SHIFT(pSubmit, sizeof(SSubmitReq)); blkHead->numOfRows = htonl(pDataBlock->info.rows); blkHead->sversion = htonl(pTSchema->version); @@ -531,17 +531,16 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d } blkHead->dataLen = htonl(dataLen); - ret->length += sizeof(SSubmitBlk) + schemaLen + dataLen; - ret->length = htonl(ret->length); + pSubmit->length += sizeof(SSubmitBlk) + schemaLen + dataLen; + pSubmit->length = htonl(pSubmit->length); SRpcMsg msg = { .msgType = TDMT_VND_SUBMIT, - .pCont = ret, - .contLen = ntohl(ret->length), + .pCont = pSubmit, + .contLen = ntohl(pSubmit->length), }; if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) { - rpcFreeCont(ret); tqDebug("failed to put into write-queue since %s", terrstr()); } } diff --git a/source/dnode/vnode/src/tsdb/tsdbRead.c b/source/dnode/vnode/src/tsdb/tsdbRead.c index 117729b3bd8350a305d97cfb30f15ec9e8bd6e68..4b332c3e85b9daa8dd44095dd68bda74caa45808 100644 --- a/source/dnode/vnode/src/tsdb/tsdbRead.c +++ b/source/dnode/vnode/src/tsdb/tsdbRead.c @@ -4115,9 +4115,13 @@ int32_t tsdbRetrieveDatablockSMA(STsdbReader* pReader, SColumnDataAgg*** pBlockS } else if (pAgg->colId < pSup->colIds[j]) { i += 1; } else if (pSup->colIds[j] < pAgg->colId) { + if (pSup->colIds[j] == PRIMARYKEY_TIMESTAMP_COL_ID) { + taosArrayPush(pNewAggList, &pSup->tsColAgg); + } else { // all date in this block are null - SColumnDataAgg nullColAgg = {.colId = pSup->colIds[j], .numOfNull = pBlock->nRow}; - taosArrayPush(pNewAggList, &nullColAgg); + SColumnDataAgg nullColAgg = {.colId = pSup->colIds[j], .numOfNull = pBlock->nRow}; + taosArrayPush(pNewAggList, &nullColAgg); + } j += 1; } } diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index 5c8c166833a7df33c342dabd065ab3ceac1fec72..c75d1ffded37b3651e539a3353c48b9ba35d2f68 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -609,6 +609,7 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t version, void *pR _exit: for (int32_t iReq = 0; iReq < req.nReqs; iReq++) { pCreateReq = req.pReqs + iReq; + taosMemoryFree(pCreateReq->comment); taosArrayDestroy(pCreateReq->ctb.tagName); } taosArrayDestroyEx(rsp.pArray, tFreeSVCreateTbRsp); diff --git a/source/libs/executor/inc/executorimpl.h b/source/libs/executor/inc/executorimpl.h index f85bddcc31a8f2daa3833bf4321234ae505a7ada..b4420348604beb0f31923828dc9807b32994fac7 100644 --- a/source/libs/executor/inc/executorimpl.h +++ b/source/libs/executor/inc/executorimpl.h @@ -647,6 +647,23 @@ typedef struct SStreamPartitionOperatorInfo { SSDataBlock* pDelRes; } SStreamPartitionOperatorInfo; +typedef struct SStreamFillSupporter { + int32_t type; // fill type + SInterval interval; + SResultRowData prev; + SResultRowData cur; + SResultRowData next; + SResultRowData nextNext; + SFillColInfo* pAllColInfo; // fill exprs and not fill exprs + SExprSupp notFillExprSup; + int32_t numOfAllCols; // number of all exprs, including the tags columns + int32_t numOfFillCols; + int32_t numOfNotFillCols; + int32_t rowSize; + SSHashObj* pResMap; + bool hasDelete; +} SStreamFillSupporter; + typedef struct SStreamFillOperatorInfo { SStreamFillSupporter* pFillSup; SSDataBlock* pRes; diff --git a/source/libs/executor/inc/tfill.h b/source/libs/executor/inc/tfill.h index 2d8df81dbd5318b80258a2b71d16acf3f9880a8b..b0017fef5037ed030e3295019436a1d5fd48296e 100644 --- a/source/libs/executor/inc/tfill.h +++ b/source/libs/executor/inc/tfill.h @@ -111,22 +111,6 @@ typedef struct SStreamFillInfo { int32_t delIndex; } SStreamFillInfo; -typedef struct SStreamFillSupporter { - int32_t type; // fill type - SInterval interval; - SResultRowData prev; - SResultRowData cur; - SResultRowData next; - SResultRowData nextNext; - SFillColInfo* pAllColInfo; // fill exprs and not fill exprs - int32_t numOfAllCols; // number of all exprs, including the tags columns - int32_t numOfFillCols; - int32_t numOfNotFillCols; - int32_t rowSize; - SSHashObj* pResMap; - bool hasDelete; -} SStreamFillSupporter; - int64_t getNumOfResultsAfterFillGap(SFillInfo* pFillInfo, int64_t ekey, int32_t maxNumOfRows); void taosFillSetStartInfo(struct SFillInfo* pFillInfo, int32_t numOfRows, TSKEY endKey); diff --git a/source/libs/executor/inc/tsort.h b/source/libs/executor/inc/tsort.h index 51440a7f59567daaf3d76929628946a1d4c0544a..cff568aebc0c26fb2e9c22c7a027fa9edb4dd81a 100644 --- a/source/libs/executor/inc/tsort.h +++ b/source/libs/executor/inc/tsort.h @@ -36,12 +36,13 @@ typedef struct SMultiMergeSource { typedef struct SSortSource { SMultiMergeSource src; - union { - struct { - SArray* pageIdList; - int32_t pageIndex; - }; + struct { + SArray* pageIdList; + int32_t pageIndex; + }; + struct { void* param; + bool onlyRef; }; } SSortSource; diff --git a/source/libs/executor/src/dataInserter.c b/source/libs/executor/src/dataInserter.c index 78afdd16b79bc45879b50aa031b2008cb8fcfba2..09ca1d27b914c1ff888cd6e5c7990e6642343f4d 100644 --- a/source/libs/executor/src/dataInserter.c +++ b/source/libs/executor/src/dataInserter.c @@ -250,6 +250,8 @@ static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput, return code; } + taosArrayClear(pInserter->pDataBlocks); + code = sendSubmitRequest(pInserter, pMsg, pInserter->pParam->readHandle->pMsgCb->clientRpc, &pInserter->pNode->epSet); if (code) { return code; diff --git a/source/libs/executor/src/executorimpl.c b/source/libs/executor/src/executorimpl.c index 9b1bc81dbe0c55282ca42d1b58323da0d602fd62..4319dd379a5ffcbfe674efb04ccde6136baf29d1 100644 --- a/source/libs/executor/src/executorimpl.c +++ b/source/libs/executor/src/executorimpl.c @@ -1864,6 +1864,8 @@ void destroyExprInfo(SExprInfo* pExpr, int32_t numOfExprs) { for (int32_t j = 0; j < pExprInfo->base.numOfParams; ++j) { if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_COLUMN) { taosMemoryFreeClear(pExprInfo->base.pParam[j].pCol); + } else if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_VALUE) { + taosVariantDestroy(&pExprInfo->base.pParam[j].param); } } diff --git a/source/libs/executor/src/scanoperator.c b/source/libs/executor/src/scanoperator.c index 0173dd1c6cac1ef0c2313d1c2e5281eeb88e9544..de2a7b9daccf526aada12d74504b394a9defa554 100644 --- a/source/libs/executor/src/scanoperator.c +++ b/source/libs/executor/src/scanoperator.c @@ -2666,6 +2666,7 @@ int32_t startGroupTableMergeScan(SOperatorInfo* pOperator) { SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource)); STableMergeScanSortSourceParam* param = taosArrayGet(pInfo->sortSourceParams, i); ps->param = param; + ps->onlyRef = true; tsortAddSource(pInfo->pSortHandle, ps); } diff --git a/source/libs/executor/src/sortoperator.c b/source/libs/executor/src/sortoperator.c index 0efa27dd46e6347920e83a4074b8117000c0302e..6201dfc9cbc5e010a6d9b15d108ca135600e00d5 100644 --- a/source/libs/executor/src/sortoperator.c +++ b/source/libs/executor/src/sortoperator.c @@ -188,10 +188,10 @@ int32_t doOpenSortOperator(SOperatorInfo* pOperator) { SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource)); ps->param = pOperator->pDownstream[0]; + ps->onlyRef = true; tsortAddSource(pInfo->pSortHandle, ps); int32_t code = tsortOpen(pInfo->pSortHandle); - taosMemoryFreeClear(ps); if (code != TSDB_CODE_SUCCESS) { T_LONG_JMP(pTaskInfo->env, terrno); @@ -389,10 +389,10 @@ int32_t beginSortGroup(SOperatorInfo* pOperator) { param->childOpInfo = pOperator->pDownstream[0]; param->grpSortOpInfo = pInfo; ps->param = param; + ps->onlyRef = false; tsortAddSource(pInfo->pCurrSortHandle, ps); int32_t code = tsortOpen(pInfo->pCurrSortHandle); - taosMemoryFreeClear(ps); if (code != TSDB_CODE_SUCCESS) { T_LONG_JMP(pTaskInfo->env, terrno); @@ -483,6 +483,9 @@ void destroyGroupSortOperatorInfo(void* param) { taosArrayDestroy(pInfo->pSortInfo); taosArrayDestroy(pInfo->matchInfo.pList); + tsortDestroySortHandle(pInfo->pCurrSortHandle); + pInfo->pCurrSortHandle = NULL; + taosMemoryFreeClear(param); } @@ -575,6 +578,7 @@ int32_t doOpenMultiwayMergeOperator(SOperatorInfo* pOperator) { for (int32_t i = 0; i < pOperator->numOfDownstream; ++i) { SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource)); ps->param = pOperator->pDownstream[i]; + ps->onlyRef = true; tsortAddSource(pInfo->pSortHandle, ps); } diff --git a/source/libs/executor/src/tfill.c b/source/libs/executor/src/tfill.c index c41376b2dcaf74f2eea572590d6ee41820122483..9908f3581854f6d1e65b4ad3bbeb0c65b4858084 100644 --- a/source/libs/executor/src/tfill.c +++ b/source/libs/executor/src/tfill.c @@ -709,6 +709,7 @@ void* destroyStreamFillSupporter(SStreamFillSupporter* pFillSup) { pFillSup->pResMap = NULL; releaseOutputBuf(NULL, NULL, (SResultRow*)pFillSup->cur.pRowVal); pFillSup->cur.pRowVal = NULL; + cleanupExprSupp(&pFillSup->notFillExprSup); taosMemoryFree(pFillSup); return NULL; @@ -1417,25 +1418,13 @@ static void doApplyStreamScalarCalculation(SOperatorInfo* pOperator, SSDataBlock blockDataEnsureCapacity(pDstBlock, pSrcBlock->info.rows); setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false); projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL); - pDstBlock->info.groupId = pSrcBlock->info.groupId; - - SColumnInfoData* pDst = taosArrayGet(pDstBlock->pDataBlock, pInfo->primaryTsCol); - SColumnInfoData* pSrc = taosArrayGet(pSrcBlock->pDataBlock, pInfo->primarySrcSlotId); - colDataAssign(pDst, pSrc, pDstBlock->info.rows, &pDstBlock->info); - - int32_t numOfNotFill = pInfo->pFillSup->numOfAllCols - pInfo->pFillSup->numOfFillCols; - for (int32_t i = 0; i < numOfNotFill; ++i) { - SFillColInfo* pCol = &pInfo->pFillSup->pAllColInfo[i + pInfo->pFillSup->numOfFillCols]; - ASSERT(pCol->notFillCol); - SExprInfo* pExpr = pCol->pExpr; - int32_t srcSlotId = pExpr->base.pParam[0].pCol->slotId; - int32_t dstSlotId = pExpr->base.resSchema.slotId; + pDstBlock->info.rows = 0; + pSup = &pInfo->pFillSup->notFillExprSup; + setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false); + projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL); + pDstBlock->info.groupId = pSrcBlock->info.groupId; - SColumnInfoData* pDst1 = taosArrayGet(pDstBlock->pDataBlock, dstSlotId); - SColumnInfoData* pSrc1 = taosArrayGet(pSrcBlock->pDataBlock, srcSlotId); - colDataAssign(pDst1, pSrc1, pDstBlock->info.rows, &pDstBlock->info); - } blockDataUpdateTsWindow(pDstBlock, pInfo->primaryTsCol); } @@ -1577,6 +1566,14 @@ static SStreamFillSupporter* initStreamFillSup(SStreamFillPhysiNode* pPhyFillNod destroyStreamFillSupporter(pFillSup); return NULL; } + + SExprInfo* noFillExpr = createExprInfo(pPhyFillNode->pNotFillExprs, NULL, &numOfNotFillCols); + code = initExprSupp(&pFillSup->notFillExprSup, noFillExpr, numOfNotFillCols); + if (code != TSDB_CODE_SUCCESS) { + destroyStreamFillSupporter(pFillSup); + return NULL; + } + _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY); pFillSup->pResMap = tSimpleHashInit(16, hashFn); pFillSup->hasDelete = false; diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index fc78f8a20dd6c13fcc30462ba511f7ec6f9d9c24..dd02ce9cd443350cad61bf801af01d5333ed186c 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -1618,6 +1618,7 @@ void destroyStreamFinalIntervalOperatorInfo(void* param) { nodesDestroyNode((SNode*)pInfo->pPhyNode); colDataDestroy(&pInfo->twAggSup.timeWindowData); cleanupGroupResInfo(&pInfo->groupResInfo); + cleanupExprSupp(&pInfo->scalarSupp); taosMemoryFreeClear(param); } @@ -2703,6 +2704,7 @@ static void rebuildIntervalWindow(SOperatorInfo* pOperator, SArray* pWinArray, S pChildSup->rowEntryInfoOffset, &pChInfo->aggSup); updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &parentWin, true); compactFunctions(pSup->pCtx, pChildSup->pCtx, numOfOutput, pTaskInfo, &pInfo->twAggSup.timeWindowData); + releaseOutputBuf(pChInfo->pState, pWinRes, pChResult); } if (num > 0 && pUpdatedMap) { saveWinResultInfo(pCurResult->win.skey, pWinRes->groupId, pUpdatedMap); @@ -5362,15 +5364,6 @@ SOperatorInfo* createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhys pInfo->primaryTsIndex = ((SColumnNode*)pIntervalPhyNode->window.pTspk)->slotId; initResultSizeInfo(&pOperator->resultInfo, 4096); - if (pIntervalPhyNode->window.pExprs != NULL) { - int32_t numOfScalar = 0; - SExprInfo* pScalarExprInfo = createExprInfo(pIntervalPhyNode->window.pExprs, NULL, &numOfScalar); - code = initExprSupp(&pInfo->scalarSupp, pScalarExprInfo, numOfScalar); - if (code != TSDB_CODE_SUCCESS) { - goto _error; - } - } - size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; code = initAggInfo(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); if (code != TSDB_CODE_SUCCESS) { diff --git a/source/libs/executor/src/tsort.c b/source/libs/executor/src/tsort.c index 9c10b51b1f2968500071cb0d14400ed631b7d887..1c31b550c64223b2afaf10499ff5512685b2ca68 100644 --- a/source/libs/executor/src/tsort.c +++ b/source/libs/executor/src/tsort.c @@ -110,6 +110,22 @@ static int32_t sortComparCleanup(SMsortComparParam* cmpParam) { return TSDB_CODE_SUCCESS; } +void tsortClearOrderdSource(SArray *pOrderedSource) { + for (size_t i = 0; i < taosArrayGetSize(pOrderedSource); i++) { + SSortSource** pSource = taosArrayGet(pOrderedSource, i); + if (NULL == *pSource) { + continue; + } + + if ((*pSource)->param && !(*pSource)->onlyRef) { + taosMemoryFree((*pSource)->param); + } + taosMemoryFreeClear(*pSource); + } + + taosArrayClear(pOrderedSource); +} + void tsortDestroySortHandle(SSortHandle* pSortHandle) { if (pSortHandle == NULL) { return; @@ -123,10 +139,8 @@ void tsortDestroySortHandle(SSortHandle* pSortHandle) { destroyDiskbasedBuf(pSortHandle->pBuf); taosMemoryFreeClear(pSortHandle->idStr); blockDataDestroy(pSortHandle->pDataBlock); - for (size_t i = 0; i < taosArrayGetSize(pSortHandle->pOrderedSource); i++) { - SSortSource** pSource = taosArrayGet(pSortHandle->pOrderedSource, i); - taosMemoryFreeClear(*pSource); - } + + tsortClearOrderdSource(pSortHandle->pOrderedSource); taosArrayDestroy(pSortHandle->pOrderedSource); taosMemoryFreeClear(pSortHandle); } @@ -561,7 +575,7 @@ static int32_t doInternalMergeSort(SSortHandle* pHandle) { } } - taosArrayClear(pHandle->pOrderedSource); + tsortClearOrderdSource(pHandle->pOrderedSource); taosArrayAddAll(pHandle->pOrderedSource, pResList); taosArrayDestroy(pResList); @@ -598,8 +612,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) { size_t sortBufSize = pHandle->numOfPages * pHandle->pageSize; if (pHandle->type == SORT_SINGLESOURCE_SORT) { - SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0); - taosArrayClear(pHandle->pOrderedSource); + SSortSource** pSource = taosArrayGet(pHandle->pOrderedSource, 0); + SSortSource* source = *pSource; + *pSource = NULL; + + tsortClearOrderdSource(pHandle->pOrderedSource); while (1) { SSDataBlock* pBlock = pHandle->fetchfp(source->param); @@ -623,6 +640,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) { int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock); if (code != 0) { + if (source->param && !source->onlyRef) { + taosMemoryFree(source->param); + } + taosMemoryFree(source); return code; } @@ -632,6 +653,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) { int64_t p = taosGetTimestampUs(); code = blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo); if (code != 0) { + if (source->param && !source->onlyRef) { + taosMemoryFree(source->param); + } + taosMemoryFree(source); return code; } @@ -642,6 +667,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) { } } + if (source->param && !source->onlyRef) { + taosMemoryFree(source->param); + } + taosMemoryFree(source); + if (pHandle->pDataBlock != NULL && pHandle->pDataBlock->info.rows > 0) { size_t size = blockDataGetSize(pHandle->pDataBlock); diff --git a/source/libs/qworker/src/qwMsg.c b/source/libs/qworker/src/qwMsg.c index 7e7f71b1764f79f6c6b52a77d8e981d285f12add..bb8a7cd140fa22992127d8c40f71d5fdbb0f251b 100644 --- a/source/libs/qworker/src/qwMsg.c +++ b/source/libs/qworker/src/qwMsg.c @@ -65,19 +65,37 @@ int32_t qwBuildAndSendErrorRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t c int32_t qwBuildAndSendQueryRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t code, SQWTaskCtx *ctx) { STbVerInfo *tbInfo = ctx ? &ctx->tbInfo : NULL; int64_t affectedRows = ctx ? ctx->affectedRows : 0; - SQueryTableRsp *pRsp = (SQueryTableRsp *)rpcMallocCont(sizeof(SQueryTableRsp)); - pRsp->code = htonl(code); - pRsp->affectedRows = htobe64(affectedRows); + SQueryTableRsp rsp = {0}; + rsp.code = code; + rsp.affectedRows = affectedRows; + if (tbInfo) { - strcpy(pRsp->tbFName, tbInfo->tbFName); - pRsp->sversion = htonl(tbInfo->sversion); - pRsp->tversion = htonl(tbInfo->tversion); + strcpy(rsp.tbFName, tbInfo->tbFName); + rsp.sversion = tbInfo->sversion; + rsp.tversion = tbInfo->tversion; + } + + int32_t msgSize = tSerializeSQueryTableRsp(NULL, 0, &rsp); + if (msgSize < 0) { + qError("tSerializeSQueryTableRsp failed"); + QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + void *pRsp = rpcMallocCont(msgSize); + if (NULL == pRsp) { + qError("rpcMallocCont %d failed", msgSize); + QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + if (tSerializeSQueryTableRsp(pRsp, msgSize, &rsp) < 0) { + qError("tSerializeSQueryTableRsp %d failed", msgSize); + QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } SRpcMsg rpcRsp = { .msgType = rspType, .pCont = pRsp, - .contLen = sizeof(*pRsp), + .contLen = msgSize, .code = code, .info = *pConn, }; @@ -182,23 +200,37 @@ int32_t qwBuildAndSendDropRsp(SRpcHandleInfo *pConn, int32_t code) { #endif int32_t qwBuildAndSendDropMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) { - STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq)); - if (NULL == req) { - QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq)); + STaskDropReq qMsg; + qMsg.header.vgId = mgmt->nodeId; + qMsg.header.contLen = 0; + qMsg.sId = sId; + qMsg.queryId = qId; + qMsg.taskId = tId; + qMsg.refId = rId; + qMsg.execId = eId; + + int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg); + if (msgSize < 0) { + QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize); + QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + void *msg = rpcMallocCont(msgSize); + if (NULL == msg) { + QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize); + QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) { + QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize); + rpcFreeCont(msg); QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - - req->header.vgId = mgmt->nodeId; - req->sId = sId; - req->queryId = qId; - req->taskId = tId; - req->refId = rId; - req->execId = eId; SRpcMsg pNewMsg = { .msgType = TDMT_SCH_DROP_TASK, - .pCont = req, - .contLen = sizeof(STaskDropReq), + .pCont = msg, + .contLen = msgSize, .code = 0, .info = *pConn, }; @@ -247,22 +279,37 @@ int32_t qwBuildAndSendCQueryMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) { } int32_t qwRegisterQueryBrokenLinkArg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) { - STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq)); - if (NULL == req) { - QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq)); + STaskDropReq qMsg; + qMsg.header.vgId = mgmt->nodeId; + qMsg.header.contLen = 0; + qMsg.sId = sId; + qMsg.queryId = qId; + qMsg.taskId = tId; + qMsg.refId = rId; + qMsg.execId = eId; + + int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg); + if (msgSize < 0) { + QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize); + QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + void *msg = rpcMallocCont(msgSize); + if (NULL == msg) { + QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize); + QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + + if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) { + QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize); + rpcFreeCont(msg); QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - - req->header.vgId = htonl(mgmt->nodeId); - req->sId = htobe64(sId); - req->queryId = htobe64(qId); - req->taskId = htobe64(tId); - req->refId = htobe64(rId); SRpcMsg brokenMsg = { .msgType = TDMT_SCH_DROP_TASK, - .pCont = req, - .contLen = sizeof(STaskDropReq), + .pCont = msg, + .contLen = msgSize, .code = TSDB_CODE_RPC_BROKEN_LINK, .info = *pConn, }; @@ -312,40 +359,33 @@ int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg, bool chkGran } int32_t code = 0; - SSubQueryMsg *msg = pMsg->pCont; SQWorker *mgmt = (SQWorker *)qWorkerMgmt; - - if (NULL == msg || pMsg->contLen <= sizeof(*msg)) { - QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen); + SSubQueryMsg msg = {0}; + if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) { + QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen); QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } - msg->sId = be64toh(msg->sId); - msg->queryId = be64toh(msg->queryId); - msg->taskId = be64toh(msg->taskId); - msg->refId = be64toh(msg->refId); - msg->execId = ntohl(msg->execId); - msg->phyLen = ntohl(msg->phyLen); - msg->sqlLen = ntohl(msg->sqlLen); - msg->msgMask = ntohl(msg->msgMask); - - if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg->msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) { - QW_ELOG("query failed cause of grant expired, msgMask:%d", msg->msgMask); + if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg.msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) { + QW_ELOG("query failed cause of grant expired, msgMask:%d", msg.msgMask); + tFreeSSubQueryMsg(&msg); QW_ERR_RET(TSDB_CODE_GRANT_EXPIRED); } - uint64_t sId = msg->sId; - uint64_t qId = msg->queryId; - uint64_t tId = msg->taskId; - int64_t rId = msg->refId; - int32_t eId = msg->execId; + uint64_t sId = msg.sId; + uint64_t qId = msg.queryId; + uint64_t tId = msg.taskId; + int64_t rId = msg.refId; + int32_t eId = msg.execId; SQWMsg qwMsg = { - .msgType = pMsg->msgType, .msg = msg->msg + msg->sqlLen, .msgLen = msg->phyLen, .connInfo = pMsg->info}; + .msgType = pMsg->msgType, .msg = msg.msg, .msgLen = msg.msgLen, .connInfo = pMsg->info}; - QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p", pMsg->info.handle); - QW_ERR_RET(qwPreprocessQuery(QW_FPARAMS(), &qwMsg)); - QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p", pMsg->info.handle); + QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p, SQL:%s", pMsg->info.handle, msg.sql); + code = qwPreprocessQuery(QW_FPARAMS(), &qwMsg); + QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p, code:%x", pMsg->info.handle, code); + + tFreeSSubQueryMsg(&msg); return TSDB_CODE_SUCCESS; } @@ -355,19 +395,25 @@ int32_t qWorkerAbortPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) { QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } - SSubQueryMsg *msg = pMsg->pCont; SQWorker *mgmt = (SQWorker *)qWorkerMgmt; + SSubQueryMsg msg = {0}; + if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) { + QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen); + QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); + } - uint64_t sId = msg->sId; - uint64_t qId = msg->queryId; - uint64_t tId = msg->taskId; - int64_t rId = msg->refId; - int32_t eId = msg->execId; + uint64_t sId = msg.sId; + uint64_t qId = msg.queryId; + uint64_t tId = msg.taskId; + int64_t rId = msg.refId; + int32_t eId = msg.execId; QW_SCH_TASK_DLOG("Abort prerocessQuery start, handle:%p", pMsg->info.handle); qwAbortPrerocessQuery(QW_FPARAMS()); QW_SCH_TASK_DLOG("Abort prerocessQuery end, handle:%p", pMsg->info.handle); + tFreeSSubQueryMsg(&msg); + return TSDB_CODE_SUCCESS; } @@ -377,42 +423,41 @@ int32_t qWorkerProcessQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int } int32_t code = 0; - SSubQueryMsg *msg = pMsg->pCont; SQWorker *mgmt = (SQWorker *)qWorkerMgmt; qwUpdateTimeInQueue(mgmt, ts, QUERY_QUEUE); QW_STAT_INC(mgmt->stat.msgStat.queryProcessed, 1); - if (NULL == msg || pMsg->contLen <= sizeof(*msg)) { - QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen); + SSubQueryMsg msg = {0}; + if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) { + QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen); QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } - uint64_t sId = msg->sId; - uint64_t qId = msg->queryId; - uint64_t tId = msg->taskId; - int64_t rId = msg->refId; - int32_t eId = msg->execId; + uint64_t sId = msg.sId; + uint64_t qId = msg.queryId; + uint64_t tId = msg.taskId; + int64_t rId = msg.refId; + int32_t eId = msg.execId; SQWMsg qwMsg = {.node = node, - .msg = msg->msg + msg->sqlLen, - .msgLen = msg->phyLen, + .msg = msg.msg, + .msgLen = msg.msgLen, .connInfo = pMsg->info, .msgType = pMsg->msgType}; - qwMsg.msgInfo.explain = msg->explain; - qwMsg.msgInfo.taskType = msg->taskType; - qwMsg.msgInfo.needFetch = msg->needFetch; + qwMsg.msgInfo.explain = msg.explain; + qwMsg.msgInfo.taskType = msg.taskType; + qwMsg.msgInfo.needFetch = msg.needFetch; - char *sql = strndup(msg->msg, msg->sqlLen); QW_SCH_TASK_DLOG("processQuery start, node:%p, type:%s, handle:%p, SQL:%s", node, TMSG_INFO(pMsg->msgType), - pMsg->info.handle, sql); - QW_ERR_JRET(qwProcessQuery(QW_FPARAMS(), &qwMsg, sql)); + pMsg->info.handle, msg.sql); + code = qwProcessQuery(QW_FPARAMS(), &qwMsg, msg.sql); + msg.sql = NULL; + QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%x", node, code); -_return: - - QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%d", node, code); + tFreeSSubQueryMsg(&msg); - return code; + return TSDB_CODE_SUCCESS; } int32_t qWorkerProcessCQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts) { @@ -548,28 +593,22 @@ int32_t qWorkerProcessDropMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int6 } int32_t code = 0; - STaskDropReq *msg = pMsg->pCont; SQWorker *mgmt = (SQWorker *)qWorkerMgmt; qwUpdateTimeInQueue(mgmt, ts, FETCH_QUEUE); QW_STAT_INC(mgmt->stat.msgStat.dropProcessed, 1); - if (NULL == msg || pMsg->contLen < sizeof(*msg)) { - QW_ELOG("invalid task drop msg, msg:%p, msgLen:%d", msg, pMsg->contLen); + STaskDropReq msg = {0}; + if (tDeserializeSTaskDropReq(pMsg->pCont, pMsg->contLen, &msg) < 0) { + QW_ELOG("tDeserializeSTaskDropReq failed, contLen:%d", pMsg->contLen); QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } - msg->sId = be64toh(msg->sId); - msg->queryId = be64toh(msg->queryId); - msg->taskId = be64toh(msg->taskId); - msg->refId = be64toh(msg->refId); - msg->execId = ntohl(msg->execId); - - uint64_t sId = msg->sId; - uint64_t qId = msg->queryId; - uint64_t tId = msg->taskId; - int64_t rId = msg->refId; - int32_t eId = msg->execId; + uint64_t sId = msg.sId; + uint64_t qId = msg.queryId; + uint64_t tId = msg.taskId; + int64_t rId = msg.refId; + int32_t eId = msg.execId; SQWMsg qwMsg = {.node = node, .msg = NULL, .msgLen = 0, .code = pMsg->code, .connInfo = pMsg->info}; diff --git a/source/libs/qworker/test/qworkerTests.cpp b/source/libs/qworker/test/qworkerTests.cpp index 6078a2a3ac44b226e722c253521031afc0fb4eda..8a48977c777af8d794a38ed1721319c4b0646953 100644 --- a/source/libs/qworker/test/qworkerTests.cpp +++ b/source/libs/qworker/test/qworkerTests.cpp @@ -114,7 +114,7 @@ void qwtBuildQueryReqMsg(SRpcMsg *queryRpc) { qwtqueryMsg.queryId = htobe64(atomic_add_fetch_64(&qwtTestQueryId, 1)); qwtqueryMsg.sId = htobe64(1); qwtqueryMsg.taskId = htobe64(1); - qwtqueryMsg.phyLen = htonl(100); + qwtqueryMsg.msgLen = htonl(100); qwtqueryMsg.sqlLen = 0; queryRpc->msgType = TDMT_SCH_QUERY; queryRpc->pCont = &qwtqueryMsg; @@ -131,12 +131,29 @@ void qwtBuildFetchReqMsg(SResFetchReq *fetchMsg, SRpcMsg *fetchRpc) { } void qwtBuildDropReqMsg(STaskDropReq *dropMsg, SRpcMsg *dropRpc) { - dropMsg->sId = htobe64(1); - dropMsg->queryId = htobe64(atomic_load_64(&qwtTestQueryId)); - dropMsg->taskId = htobe64(1); + dropMsg->sId = 1; + dropMsg->queryId = atomic_load_64(&qwtTestQueryId); + dropMsg->taskId = 1; + + int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, dropMsg); + if (msgSize < 0) { + return; + } + + char *msg = (char*)taosMemoryCalloc(1, msgSize); + if (NULL == msg) { + return; + } + + if (tSerializeSTaskDropReq(msg, msgSize, dropMsg) < 0) { + taosMemoryFree(msg); + return; + } + + dropRpc->msgType = TDMT_SCH_DROP_TASK; - dropRpc->pCont = dropMsg; - dropRpc->contLen = sizeof(STaskDropReq); + dropRpc->pCont = msg; + dropRpc->contLen = msgSize; } int32_t qwtStringToPlan(const char *str, SSubplan **subplan) { diff --git a/source/libs/scalar/src/sclfunc.c b/source/libs/scalar/src/sclfunc.c index 336bc6d533687d8ecaffa6e1c19c7f3c43357754..5496c5d1abc86c9631070c713b2a97b163040902 100644 --- a/source/libs/scalar/src/sclfunc.c +++ b/source/libs/scalar/src/sclfunc.c @@ -24,6 +24,8 @@ static double tlog2(double v, double base) { return a; } else if (isnan(b) || isinf(b)) { return b; + } else if (b == 0) { + return INFINITY; } else { return a / b; } diff --git a/source/libs/scheduler/src/schRemote.c b/source/libs/scheduler/src/schRemote.c index a6a2a6c301bfc0e079fedd88fe247ba16264b034..b0bc0df8506b2c8bd704fde26f103dcab2b6a61f 100644 --- a/source/libs/scheduler/src/schRemote.c +++ b/source/libs/scheduler/src/schRemote.c @@ -334,17 +334,17 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_INPUT); } - SQueryTableRsp *rsp = (SQueryTableRsp *)msg; - rsp->code = ntohl(rsp->code); - rsp->sversion = ntohl(rsp->sversion); - rsp->tversion = ntohl(rsp->tversion); - rsp->affectedRows = be64toh(rsp->affectedRows); - - SCH_ERR_JRET(rsp->code); + SQueryTableRsp rsp = {0}; + if (tDeserializeSQueryTableRsp(msg, msgSize, &rsp) < 0) { + SCH_TASK_ELOG("tDeserializeSQueryTableRsp failed, msgSize:%d", msgSize); + SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_MSG); + } + + SCH_ERR_JRET(rsp.code); - SCH_ERR_JRET(schSaveJobExecRes(pJob, rsp)); + SCH_ERR_JRET(schSaveJobExecRes(pJob, &rsp)); - atomic_add_fetch_32(&pJob->resNumOfRows, rsp->affectedRows); + atomic_add_fetch_32(&pJob->resNumOfRows, rsp.affectedRows); taosMemoryFreeClear(msg); @@ -1042,30 +1042,40 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr, case TDMT_SCH_MERGE_QUERY: { SCH_ERR_RET(schMakeQueryRpcCtx(pJob, pTask, &rpcCtx)); - uint32_t len = strlen(pJob->sql); - msgSize = sizeof(SSubQueryMsg) + pTask->msgLen + len; + SSubQueryMsg qMsg; + qMsg.header.vgId = addr->nodeId; + qMsg.header.contLen = 0; + qMsg.sId = schMgmt.sId; + qMsg.queryId = pJob->queryId; + qMsg.taskId = pTask->taskId; + qMsg.refId = pJob->refId; + qMsg.execId = pTask->execId; + qMsg.msgMask = (pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0; + qMsg.taskType = TASK_TYPE_TEMP; + qMsg.explain = SCH_IS_EXPLAIN_JOB(pJob); + qMsg.needFetch = SCH_TASK_NEED_FETCH(pTask); + qMsg.sqlLen = strlen(pJob->sql); + qMsg.sql = pJob->sql; + qMsg.msgLen = pTask->msgLen; + qMsg.msg = pTask->msg; + + msgSize = tSerializeSSubQueryMsg(NULL, 0, &qMsg); + if (msgSize < 0) { + SCH_TASK_ELOG("tSerializeSSubQueryMsg get size, msgSize:%d", msgSize); + SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + msg = taosMemoryCalloc(1, msgSize); if (NULL == msg) { SCH_TASK_ELOG("calloc %d failed", msgSize); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - SSubQueryMsg *pMsg = msg; - pMsg->header.vgId = htonl(addr->nodeId); - pMsg->sId = htobe64(schMgmt.sId); - pMsg->queryId = htobe64(pJob->queryId); - pMsg->taskId = htobe64(pTask->taskId); - pMsg->refId = htobe64(pJob->refId); - pMsg->execId = htonl(pTask->execId); - pMsg->taskType = TASK_TYPE_TEMP; - pMsg->explain = SCH_IS_EXPLAIN_JOB(pJob); - pMsg->needFetch = SCH_TASK_NEED_FETCH(pTask); - pMsg->phyLen = htonl(pTask->msgLen); - pMsg->sqlLen = htonl(len); - pMsg->msgMask = htonl((pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0); - - memcpy(pMsg->msg, pJob->sql, len); - memcpy(pMsg->msg + len, pTask->msg, pTask->msgLen); + if (tSerializeSSubQueryMsg(msg, msgSize, &qMsg) < 0) { + SCH_TASK_ELOG("tSerializeSSubQueryMsg failed, msgSize:%d", msgSize); + taosMemoryFree(msg); + SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } persistHandle = true; SCH_SET_TASK_HANDLE(pTask, rpcAllocHandle()); @@ -1092,22 +1102,32 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr, break; } case TDMT_SCH_DROP_TASK: { - msgSize = sizeof(STaskDropReq); + STaskDropReq qMsg; + qMsg.header.vgId = addr->nodeId; + qMsg.header.contLen = 0; + qMsg.sId = schMgmt.sId; + qMsg.queryId = pJob->queryId; + qMsg.taskId = pTask->taskId; + qMsg.refId = pJob->refId; + qMsg.execId = pTask->execId; + + msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg); + if (msgSize < 0) { + SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize); + SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } + msg = taosMemoryCalloc(1, msgSize); if (NULL == msg) { SCH_TASK_ELOG("calloc %d failed", msgSize); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - STaskDropReq *pMsg = msg; - - pMsg->header.vgId = htonl(addr->nodeId); - - pMsg->sId = htobe64(schMgmt.sId); - pMsg->queryId = htobe64(pJob->queryId); - pMsg->taskId = htobe64(pTask->taskId); - pMsg->refId = htobe64(pJob->refId); - pMsg->execId = htonl(pTask->execId); + if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) { + SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize); + taosMemoryFree(msg); + SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + } break; } case TDMT_SCH_QUERY_HEARTBEAT: { diff --git a/source/libs/sync/src/syncAppendEntriesReply.c b/source/libs/sync/src/syncAppendEntriesReply.c index 13ea250155cca006170725af991ba5da69d65d67..c602788b19843111dc02fdb041b2746a4c0cb3f4 100644 --- a/source/libs/sync/src/syncAppendEntriesReply.c +++ b/source/libs/sync/src/syncAppendEntriesReply.c @@ -67,6 +67,9 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) { if (pMsg->matchIndex > oldMatchIndex) { syncIndexMgrSetIndex(ths->pMatchIndex, &(pMsg->srcId), pMsg->matchIndex); syncMaybeAdvanceCommitIndex(ths); + + // maybe update minMatchIndex + ths->minMatchIndex = syncMinMatchIndex(ths); } syncIndexMgrSetIndex(ths->pNextIndex, &(pMsg->srcId), pMsg->matchIndex + 1); diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index a427d7aa0cbd2689faa88482ecf6eb7465704dc7..88b8ba7e2531bd386fdd224ef1651c5992a6dc7d 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -243,6 +243,18 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) { goto _DEL_WAL; } else { + lastApplyIndex -= SYNC_VNODE_LOG_RETENTION; + + SyncIndex beginIndex = pSyncNode->pLogStore->syncLogBeginIndex(pSyncNode->pLogStore); + SyncIndex endIndex = pSyncNode->pLogStore->syncLogEndIndex(pSyncNode->pLogStore); + bool isEmpty = pSyncNode->pLogStore->syncLogIsEmpty(pSyncNode->pLogStore); + + if (isEmpty || !(lastApplyIndex >= beginIndex && lastApplyIndex <= endIndex)) { + sNTrace(pSyncNode, "new-snapshot-index:%" PRId64 ", empty:%d, do not delete wal", lastApplyIndex, isEmpty); + syncNodeRelease(pSyncNode); + return 0; + } + // vnode if (pSyncNode->replicaNum > 1) { // multi replicas @@ -300,26 +312,31 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) { _DEL_WAL: do { - SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex); - - if (snapshottingIndex == SYNC_INDEX_INVALID) { - atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex); - pSyncNode->snapshottingTime = taosGetTimestampMs(); + SSyncLogStoreData* pData = pSyncNode->pLogStore->data; + SyncIndex snapshotVer = walGetSnapshotVer(pData->pWal); + SyncIndex walCommitVer = walGetCommittedVer(pData->pWal); + SyncIndex wallastVer = walGetLastVer(pData->pWal); + if (lastApplyIndex <= walCommitVer) { + SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex); + + if (snapshottingIndex == SYNC_INDEX_INVALID) { + atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex); + pSyncNode->snapshottingTime = taosGetTimestampMs(); + + code = walBeginSnapshot(pData->pWal, lastApplyIndex); + if (code == 0) { + sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64, + pSyncNode->snapshottingIndex, lastApplyIndex); + } else { + sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64, + terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex); + atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID); + } - SSyncLogStoreData* pData = pSyncNode->pLogStore->data; - code = walBeginSnapshot(pData->pWal, lastApplyIndex); - if (code == 0) { - sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64, - pSyncNode->snapshottingIndex, lastApplyIndex); } else { - sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64, - terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex); - atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID); + sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64, + snapshottingIndex, lastApplyIndex); } - - } else { - sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64, - snapshottingIndex, lastApplyIndex); } } while (0); diff --git a/source/libs/sync/src/syncRaftLog.c b/source/libs/sync/src/syncRaftLog.c index db0b6d1d02550e5b2ec2319617c490acf45ebb46..2f824b6b3bc9f525a95fca8a48e1e350a0c52fb5 100644 --- a/source/libs/sync/src/syncRaftLog.c +++ b/source/libs/sync/src/syncRaftLog.c @@ -375,7 +375,17 @@ static int32_t raftLogGetLastEntry(SSyncLogStore* pLogStore, SSyncRaftEntry** pp int32_t raftLogUpdateCommitIndex(SSyncLogStore* pLogStore, SyncIndex index) { SSyncLogStoreData* pData = pLogStore->data; SWal* pWal = pData->pWal; - // ASSERT(walCommit(pWal, index) == 0); + + // need not update + SyncIndex snapshotVer = walGetSnapshotVer(pWal); + SyncIndex walCommitVer = walGetCommittedVer(pWal); + SyncIndex wallastVer = walGetLastVer(pWal); + + if (index < snapshotVer || index > wallastVer) { + // ignore + return 0; + } + int32_t code = walCommit(pWal, index); if (code != 0) { int32_t err = terrno; diff --git a/source/libs/sync/src/syncTimeout.c b/source/libs/sync/src/syncTimeout.c index 3d4583aadbb59208137210ce59ac83046300d171..151e5cdf462e8e1246b7331e9b43fa802e0e7d0a 100644 --- a/source/libs/sync/src/syncTimeout.c +++ b/source/libs/sync/src/syncTimeout.c @@ -62,18 +62,20 @@ static int32_t syncNodeTimerRoutine(SSyncNode* ths) { syncNodeCleanConfigIndex(ths); } - // end timeout wal snapshot int64_t timeNow = taosGetTimestampMs(); - if (timeNow - ths->snapshottingIndex > SYNC_DEL_WAL_MS && - atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) { - SSyncLogStoreData* pData = ths->pLogStore->data; - int32_t code = walEndSnapshot(pData->pWal); - if (code != 0) { - sNError(ths, "timer wal snapshot end error since:%s", terrstr()); - return -1; - } else { - sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex)); - atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID); + if (atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) { + // end timeout wal snapshot + if (timeNow - ths->snapshottingTime > SYNC_DEL_WAL_MS && + atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) { + SSyncLogStoreData* pData = ths->pLogStore->data; + int32_t code = walEndSnapshot(pData->pWal); + if (code != 0) { + sNError(ths, "timer wal snapshot end error since:%s", terrstr()); + return -1; + } else { + sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex)); + atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID); + } } } diff --git a/source/libs/sync/src/syncUtil.c b/source/libs/sync/src/syncUtil.c index b50336cd63e5526e99155bb858df11666783723c..4fc7dd245dc198881198419e9b1f87f2d7c46266 100644 --- a/source/libs/sync/src/syncUtil.c +++ b/source/libs/sync/src/syncUtil.c @@ -239,11 +239,11 @@ void syncPrintNodeLog(const char* flags, ELogLevel level, int32_t dflag, SSyncNo "vgId:%d, sync %s " "%s" ", tm:%" PRIu64 ", cmt:%" PRId64 ", fst:%" PRId64 ", lst:%" PRId64 ", min:%" PRId64 ", snap:%" PRId64 - ", snap-tm:%" PRIu64 ", sby:%d, aq:%d, bch:%d, r-num:%d, lcfg:%" PRId64 + ", snap-tm:%" PRIu64 ", sby:%d, aq:%d, snaping:%" PRId64 ", r-num:%d, lcfg:%" PRId64 ", chging:%d, rsto:%d, dquorum:%d, elt:%" PRId64 ", hb:%" PRId64 ", %s, %s", pNode->vgId, syncStr(pNode->state), eventLog, currentTerm, pNode->commitIndex, logBeginIndex, logLastIndex, pNode->minMatchIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm, - pNode->pRaftCfg->isStandBy, aqItems, pNode->pRaftCfg->batchSize, pNode->replicaNum, + pNode->pRaftCfg->isStandBy, aqItems, pNode->snapshottingIndex, pNode->replicaNum, pNode->pRaftCfg->lastConfigIndex, pNode->changing, pNode->restoreFinish, quorum, pNode->electTimerLogicClock, pNode->heartbeatTimerLogicClockUser, peerStr, cfgStr); } diff --git a/source/libs/transport/src/transCli.c b/source/libs/transport/src/transCli.c index 4fb00b1a6d5a2ee1d7709dbc9ec5b5c24994a416..5b8f059e04ff1745ad0dba18297025d6431cf19e 100644 --- a/source/libs/transport/src/transCli.c +++ b/source/libs/transport/src/transCli.c @@ -791,6 +791,7 @@ void cliSend(SCliConn* pConn) { int msgLen = transMsgLenFromCont(pMsg->contLen); STransMsgHead* pHead = transHeadFromCont(pMsg->pCont); + pHead->ahandle = pCtx != NULL ? (uint64_t)pCtx->ahandle : 0; pHead->noResp = REQUEST_NO_RESP(pMsg) ? 1 : 0; pHead->persist = REQUEST_PERSIS_HANDLE(pMsg) ? 1 : 0; @@ -820,10 +821,15 @@ void cliSend(SCliConn* pConn) { uv_timer_start((uv_timer_t*)pConn->timer, cliReadTimeoutCb, TRANS_READ_TIMEOUT, 0); } - if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) { - msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead); - pHead->msgLen = (int32_t)htonl((uint32_t)msgLen); + if (pHead->comp == 0) { + if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) { + msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead); + pHead->msgLen = (int32_t)htonl((uint32_t)msgLen); + } + } else { + msgLen = (int32_t)ntohl((uint32_t)(pHead->msgLen)); } + tGDebug("%s conn %p %s is sent to %s, local info %s, len:%d", CONN_GET_INST_LABEL(pConn), pConn, TMSG_INFO(pHead->msgType), pConn->dst, pConn->src, msgLen); diff --git a/source/libs/transport/src/transSvr.c b/source/libs/transport/src/transSvr.c index 8dd3628c5fdbe43ea5c3ec3a2c43bf74cb3754ed..f093d84db6db31098703bf87b5e0c8ac42c929b3 100644 --- a/source/libs/transport/src/transSvr.c +++ b/source/libs/transport/src/transSvr.c @@ -398,7 +398,7 @@ static int uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) { pHead->magicNum = htonl(TRANS_MAGIC_NUM); // handle invalid drop_task resp, TD-20098 - if (pMsg->msgType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) { + if (pConn->inType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) { transQueuePop(&pConn->srvMsgs); destroySmsg(smsg); return -1; diff --git a/source/libs/wal/src/walWrite.c b/source/libs/wal/src/walWrite.c index 6cd127fed4eb354f79c7f444cde1cfd95caaa1ee..c5c8173f63a3b5b624a61b45813fdb9ffd7618f9 100644 --- a/source/libs/wal/src/walWrite.c +++ b/source/libs/wal/src/walWrite.c @@ -324,23 +324,36 @@ int32_t walEndSnapshot(SWal *pWal) { // find files safe to delete SWalFileInfo *pInfo = taosArraySearch(pWal->fileInfoSet, &tmp, compareWalFileInfo, TD_LE); if (pInfo) { + SWalFileInfo *pLastFileInfo = taosArrayGetLast(pWal->fileInfoSet); + wDebug("vgId:%d, wal search found file info: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer, + pInfo->lastVer); if (ver >= pInfo->lastVer) { - pInfo--; + pInfo++; + wDebug("vgId:%d, wal remove advance one file: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer, + pInfo->lastVer); } - if (POINTER_DISTANCE(pInfo, pWal->fileInfoSet->pData) > 0) { - wDebug("vgId:%d, wal end remove for %" PRId64, pWal->cfg.vgId, pInfo->firstVer); + if (pInfo <= pLastFileInfo) { + wDebug("vgId:%d, wal end remove for first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer, + pInfo->lastVer); } else { wDebug("vgId:%d, wal no remove", pWal->cfg.vgId); } // iterate files, until the searched result for (SWalFileInfo *iter = pWal->fileInfoSet->pData; iter < pInfo; iter++) { - if ((pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize) || - (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts)) { + wDebug("vgId:%d, wal check remove file %" PRId64 "(file size %" PRId64 " close ts %" PRId64 + "), new tot size %" PRId64, + pWal->cfg.vgId, iter->firstVer, iter->fileSize, iter->closeTs, newTotSize); + if (((pWal->cfg.retentionSize == 0) || (pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize)) || + ((pWal->cfg.retentionPeriod == 0) || + (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts))) { // delete according to file size or close time + wDebug("vgId:%d, check pass", pWal->cfg.vgId); deleteCnt++; newTotSize -= iter->fileSize; } + wDebug("vgId:%d, check not pass", pWal->cfg.vgId); } + wDebug("vgId:%d, wal should delete %d files", pWal->cfg.vgId, deleteCnt); int32_t actualDelete = 0; char fnameStr[WAL_FILE_LEN]; // remove file diff --git a/source/util/src/tpagedbuf.c b/source/util/src/tpagedbuf.c index c81888eb95c0a76fd8a6c36172ba54348bd6566c..79ea10552ccaf1fb44fa9847695e6424539c37ed 100644 --- a/source/util/src/tpagedbuf.c +++ b/source/util/src/tpagedbuf.c @@ -107,7 +107,7 @@ static uint64_t allocatePositionInFile(SDiskbasedBuf* pBuf, size_t size) { static void setPageNotInBuf(SPageInfo* pPageInfo) { pPageInfo->pData = NULL; } -static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + 2; } +static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + sizeof(SFilePage); } /** * +--------------------------+-------------------+--------------+ diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 890412f8eb0091a1065d47a1837c01d4511c6b72..937ddb7a5e7e60cf1a719f3ff1f62a33485f0437 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -216,7 +216,7 @@ ,,y,script,./test.sh -f tsim/stream/drop_stream.sim ,,y,script,./test.sh -f tsim/stream/fillHistoryBasic1.sim ,,y,script,./test.sh -f tsim/stream/fillHistoryBasic2.sim -,,n,script,./test.sh -f tsim/stream/fillHistoryBasic3.sim +,,y,script,./test.sh -f tsim/stream/fillHistoryBasic3.sim ,,y,script,./test.sh -f tsim/stream/distributeInterval0.sim ,,y,script,./test.sh -f tsim/stream/distributeIntervalRetrive0.sim ,,y,script,./test.sh -f tsim/stream/distributeSession0.sim @@ -227,7 +227,7 @@ ,,y,script,./test.sh -f tsim/stream/triggerSession0.sim ,,y,script,./test.sh -f tsim/stream/partitionby.sim ,,y,script,./test.sh -f tsim/stream/partitionby1.sim -,,n,script,./test.sh -f tsim/stream/schedSnode.sim +,,y,script,./test.sh -f tsim/stream/schedSnode.sim ,,y,script,./test.sh -f tsim/stream/windowClose.sim ,,y,script,./test.sh -f tsim/stream/ignoreExpiredData.sim ,,y,script,./test.sh -f tsim/stream/sliding.sim @@ -416,208 +416,208 @@ ,,,system-test,python3 ./test.py -f 0-others/sysinfo.py ,,,system-test,python3 ./test.py -f 0-others/user_control.py ,,,system-test,python3 ./test.py -f 0-others/fsync.py -,,,system-test,python3 ./test.py -f 0-others/compatibility.py +,,,system-test,python3 ./test.py -f 0-others/compatibility.py ,,,system-test,python3 ./test.py -f 1-insert/alter_database.py ,,,system-test,python3 ./test.py -f 1-insert/influxdb_line_taosc_insert.py -,,,system-test,python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py ,,,system-test,python3 ./test.py -f 1-insert/opentsdb_json_taosc_insert.py ,,,system-test,python3 ./test.py -f 1-insert/test_stmt_muti_insert_query.py ,,,system-test,python3 ./test.py -f 1-insert/test_stmt_set_tbname_tag.py -,,,system-test,python3 ./test.py -f 1-insert/alter_stable.py -,,,system-test,python3 ./test.py -f 1-insert/alter_table.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_stable.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_table.py ,,,system-test,python3 ./test.py -f 1-insert/boundary.py ,,,system-test,python3 ./test.py -f 1-insert/insertWithMoreVgroup.py ,,,system-test,python3 ./test.py -f 1-insert/table_comment.py ,,,system-test,python3 ./test.py -f 1-insert/time_range_wise.py ,,,system-test,python3 ./test.py -f 1-insert/block_wise.py ,,,system-test,python3 ./test.py -f 1-insert/create_retentions.py -,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py ,,,system-test,python3 ./test.py -f 1-insert/mutil_stage.py -,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py -R -,,,system-test,python3 ./test.py -f 1-insert/update_data_muti_rows.py -,,,system-test,python3 ./test.py -f 1-insert/db_tb_name_check.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/update_data_muti_rows.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/db_tb_name_check.py ,,,system-test,python3 ./test.py -f 1-insert/database_pre_suf.py ,,,system-test,python3 ./test.py -f 1-insert/InsertFuturets.py -,,,system-test,python3 ./test.py -f 0-others/show.py +,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/show.py ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -,,,system-test,python3 ./test.py -f 2-query/abs.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py -,,,system-test,python3 ./test.py -f 2-query/and_or_for_byte.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -,,,system-test,python3 ./test.py -f 2-query/apercentile.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -,,,system-test,python3 ./test.py -f 2-query/arccos.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -,,,system-test,python3 ./test.py -f 2-query/arcsin.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -,,,system-test,python3 ./test.py -f 2-query/arctan.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -R ,,,system-test,python3 ./test.py -f 2-query/avg.py ,,,system-test,python3 ./test.py -f 2-query/avg.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -,,,system-test,python3 ./test.py -f 2-query/between.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -,,,system-test,python3 ./test.py -f 2-query/bottom.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -,,,system-test,python3 ./test.py -f 2-query/cast.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -,,,system-test,python3 ./test.py -f 2-query/ceil.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -,,,system-test,python3 ./test.py -f 2-query/char_length.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -R ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -R ,,,system-test,python3 ./test.py -f 2-query/concat.py ,,,system-test,python3 ./test.py -f 2-query/concat.py -R ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -R ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -R -,,,system-test,python3 ./test.py -f 2-query/cos.py -,,,system-test,python3 ./test.py -f 2-query/cos.py -R -,,,system-test,python3 ./test.py -f 2-query/count_partition.py -,,,system-test,python3 ./test.py -f 2-query/count_partition.py -R -,,,system-test,python3 ./test.py -f 2-query/count.py -,,,system-test,python3 ./test.py -f 2-query/count.py -R -,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R -,,,system-test,python3 ./test.py -f 2-query/db.py -,,,system-test,python3 ./test.py -f 2-query/db.py -R -,,,system-test,python3 ./test.py -f 2-query/diff.py -,,,system-test,python3 ./test.py -f 2-query/diff.py -R -,,,system-test,python3 ./test.py -f 2-query/distinct.py -,,,system-test,python3 ./test.py -f 2-query/distinct.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -R -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -R -,,,system-test,python3 ./test.py -f 2-query/explain.py -,,,system-test,python3 ./test.py -f 2-query/explain.py -R -,,,system-test,python3 ./test.py -f 2-query/first.py -,,,system-test,python3 ./test.py -f 2-query/first.py -R -,,,system-test,python3 ./test.py -f 2-query/floor.py -,,,system-test,python3 ./test.py -f 2-query/floor.py -R -,,,system-test,python3 ./test.py -f 2-query/function_null.py -,,,system-test,python3 ./test.py -f 2-query/function_null.py -R -,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -R -,,,system-test,python3 ./test.py -f 2-query/histogram.py -,,,system-test,python3 ./test.py -f 2-query/histogram.py -R -,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -R ,,,system-test,python3 ./test.py -f 2-query/interp.py ,,,system-test,python3 ./test.py -f 2-query/interp.py -R -,,,system-test,python3 ./test.py -f 2-query/irate.py -,,,system-test,python3 ./test.py -f 2-query/irate.py -R -,,,system-test,python3 ./test.py -f 2-query/join.py -,,,system-test,python3 ./test.py -f 2-query/join.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -R ,,,system-test,python3 ./test.py -f 2-query/last_row.py ,,,system-test,python3 ./test.py -f 2-query/last_row.py -R -,,,system-test,python3 ./test.py -f 2-query/last.py -,,,system-test,python3 ./test.py -f 2-query/last.py -R -,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -R -,,,system-test,python3 ./test.py -f 2-query/length.py -,,,system-test,python3 ./test.py -f 2-query/length.py -R -,,,system-test,python3 ./test.py -f 2-query/log.py -,,,system-test,python3 ./test.py -f 2-query/log.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -R ,,,system-test,python3 ./test.py -f 2-query/lower.py ,,,system-test,python3 ./test.py -f 2-query/lower.py -R ,,,system-test,python3 ./test.py -f 2-query/ltrim.py ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -R -,,,system-test,python3 ./test.py -f 2-query/mavg.py -,,,system-test,python3 ./test.py -f 2-query/mavg.py -R -,,,system-test,python3 ./test.py -f 2-query/max_partition.py -,,,system-test,python3 ./test.py -f 2-query/max_partition.py -R -,,,system-test,python3 ./test.py -f 2-query/max.py -,,,system-test,python3 ./test.py -f 2-query/max.py -R -,,,system-test,python3 ./test.py -f 2-query/min.py -,,,system-test,python3 ./test.py -f 2-query/min.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -R ,,,system-test,python3 ./test.py -f 2-query/mode.py ,,,system-test,python3 ./test.py -f 2-query/mode.py -R -,,,system-test,python3 ./test.py -f 2-query/Now.py -,,,system-test,python3 ./test.py -f 2-query/Now.py -R -,,,system-test,python3 ./test.py -f 2-query/percentile.py -,,,system-test,python3 ./test.py -f 2-query/percentile.py -R -,,,system-test,python3 ./test.py -f 2-query/pow.py -,,,system-test,python3 ./test.py -f 2-query/pow.py -R -,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R -,,,system-test,python3 ./test.py -f 2-query/round.py -,,,system-test,python3 ./test.py -f 2-query/round.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -R ,,,system-test,python3 ./test.py -f 2-query/rtrim.py ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -R -,,,system-test,python3 ./test.py -f 2-query/sample.py -,,,system-test,python3 ./test.py -f 2-query/sample.py -R -,,,system-test,python3 ./test.py -f 2-query/sin.py -,,,system-test,python3 ./test.py -f 2-query/sin.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -R ,,,system-test,python3 ./test.py -f 2-query/smaTest.py ,,,system-test,python3 ./test.py -f 2-query/smaTest.py -R ,,,system-test,python3 ./test.py -f 2-query/sml.py ,,,system-test,python3 ./test.py -f 2-query/sml.py -R -,,,system-test,python3 ./test.py -f 2-query/spread.py -,,,system-test,python3 ./test.py -f 2-query/spread.py -R -,,,system-test,python3 ./test.py -f 2-query/sqrt.py -,,,system-test,python3 ./test.py -f 2-query/sqrt.py -R -,,,system-test,python3 ./test.py -f 2-query/statecount.py -,,,system-test,python3 ./test.py -f 2-query/statecount.py -R -,,,system-test,python3 ./test.py -f 2-query/stateduration.py -,,,system-test,python3 ./test.py -f 2-query/stateduration.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -R ,,,system-test,python3 ./test.py -f 2-query/substr.py ,,,system-test,python3 ./test.py -f 2-query/substr.py -R -,,,system-test,python3 ./test.py -f 2-query/sum.py -,,,system-test,python3 ./test.py -f 2-query/sum.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py -R ,,,system-test,python3 ./test.py -f 2-query/tail.py ,,,system-test,python3 ./test.py -f 2-query/tail.py -R -,,,system-test,python3 ./test.py -f 2-query/tan.py -,,,system-test,python3 ./test.py -f 2-query/tan.py -R -,,,system-test,python3 ./test.py -f 2-query/Timediff.py -,,,system-test,python3 ./test.py -f 2-query/Timediff.py -R -,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -R -,,,system-test,python3 ./test.py -f 2-query/timezone.py -,,,system-test,python3 ./test.py -f 2-query/timezone.py -R -,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -R -,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -R -,,,system-test,python3 ./test.py -f 2-query/Today.py -,,,system-test,python3 ./test.py -f 2-query/Today.py -R -,,,system-test,python3 ./test.py -f 2-query/top.py -,,,system-test,python3 ./test.py -f 2-query/top.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -R ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -R -,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -R -,,,system-test,python3 ./test.py -f 2-query/twa.py -,,,system-test,python3 ./test.py -f 2-query/twa.py -R -,,,system-test,python3 ./test.py -f 2-query/union.py -,,,system-test,python3 ./test.py -f 2-query/union.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py -R ,,,system-test,python3 ./test.py -f 2-query/unique.py ,,,system-test,python3 ./test.py -f 2-query/unique.py -R -,,,system-test,python3 ./test.py -f 2-query/upper.py -,,,system-test,python3 ./test.py -f 2-query/upper.py -R -,,,system-test,python3 ./test.py -f 2-query/varchar.py -,,,system-test,python3 ./test.py -f 2-query/varchar.py -R -,,,system-test,python3 ./test.py -f 2-query/case_when.py -,,,system-test,python3 ./test.py -f 2-query/case_when.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -R +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -R ,,,system-test,python3 ./test.py -f 1-insert/update_data.py ,,,system-test,python3 ./test.py -f 1-insert/tb_100w_data_order.py ,,,system-test,python3 ./test.py -f 1-insert/delete_stable.py ,,,system-test,python3 ./test.py -f 1-insert/delete_childtable.py ,,,system-test,python3 ./test.py -f 1-insert/delete_normaltable.py -,,,system-test,python3 ./test.py -f 1-insert/keep_expired.py -,,,system-test,python3 ./test.py -f 2-query/join2.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/keep_expired.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py ,,,system-test,python3 ./test.py -f 2-query/union1.py ,,,system-test,python3 ./test.py -f 2-query/concat2.py ,,,system-test,python3 ./test.py -f 2-query/json_tag.py @@ -627,9 +627,9 @@ ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_time.py ,,,system-test,python3 ./test.py -f 2-query/stablity.py ,,,system-test,python3 ./test.py -f 2-query/stablity_1.py -,,,system-test,python3 ./test.py -f 2-query/elapsed.py -,,,system-test,python3 ./test.py -f 2-query/csum.py -,,,system-test,python3 ./test.py -f 2-query/function_diff.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/elapsed.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/csum.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_diff.py ,,,system-test,python3 ./test.py -f 2-query/queryQnode.py ,,,system-test,python3 ./test.py -f 6-cluster/5dnode1mnode.py ,,,system-test,python3 ./test.py -f 6-cluster/5dnode2mnode.py -N 5 @@ -666,7 +666,7 @@ ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas.py -N 4 -M 1 ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas_querys.py -N 4 -M 1 ,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_vgroups.py -N 4 -M 1 -,,,system-test,python3 ./test.py -f 7-tmq/create_wrong_topic.py +,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/create_wrong_topic.py ,,,system-test,python3 ./test.py -f 7-tmq/dropDbR3ConflictTransaction.py -N 3 ,,,system-test,python3 ./test.py -f 7-tmq/basic5.py ,,,system-test,python3 ./test.py -f 7-tmq/subscribeDb.py @@ -704,6 +704,7 @@ ,,,system-test,python3 ./test.py -f 7-tmq/tmqConsFromTsdb1-mutilVg-mutilCtb.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqAutoCreateTbl.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart.py +,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart1.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-1ctb.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdateWithConsume.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-multiCtb-snapshot0.py @@ -724,18 +725,18 @@ ,,,system-test,python3 ./test.py -f 7-tmq/stbTagFilter-multiCtb.py ,,,system-test,python3 ./test.py -f 99-TDcase/TD-19201.py ,,,system-test,python3 ./test.py -f 7-tmq/tmqSubscribeStb-r3.py -N 5 -,,,system-test,python3 ./test.py -f 2-query/between.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/length.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/join.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/cast.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/union1.py -Q 2 @@ -743,45 +744,45 @@ ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/max.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/min.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/count.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/last.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/first.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/top.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/round.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/log.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 2 @@ -792,44 +793,44 @@ ,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 2 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 2 -,,,system-test,python3 ./test.py -f 2-query/between.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/length.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/join.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/cast.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/union1.py -Q 3 @@ -837,45 +838,45 @@ ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/max.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/min.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/count.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/last.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/first.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/top.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/round.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/log.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 3 @@ -885,43 +886,43 @@ ,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 3 ,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 3 -,,,system-test,python3 ./test.py -f 2-query/between.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/length.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/join.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/union.py -Q 4 @@ -930,45 +931,45 @@ ,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/max.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/min.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/count.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/last.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/first.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/top.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/round.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/log.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/json_tag.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 4 @@ -978,33 +979,34 @@ ,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 4 ,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 4 -,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 4 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 4 #develop test ,,,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/auto_create_table_json.py diff --git a/tests/pytest/auto_crash_gen.py b/tests/pytest/auto_crash_gen.py new file mode 100755 index 0000000000000000000000000000000000000000..9c134e6d64a343dfa3a20afdf3ceec3d6e8d6855 --- /dev/null +++ b/tests/pytest/auto_crash_gen.py @@ -0,0 +1,365 @@ +import os +import socket +import requests + +# -*- coding: utf-8 -*- +import os ,sys +import random +import argparse +import subprocess +import time +import platform + +# valgrind mode ? +valgrind_mode = False + +msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" } + +# formal +hostname = socket.gethostname() + +group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9' + +def get_msg(text): + return { + "msg_type": "post", + "content": { + "post": { + "zh_cn": { + "title": "Crash_gen Monitor", + "content": [ + [{ + "tag": "text", + "text": text + } + ]] + } + } + } + } + + +def send_msg(json): + headers = { + 'Content-Type': 'application/json' + } + + req = requests.post(url=group_url, headers=headers, json=json) + inf = req.json() + if "StatusCode" in inf and inf["StatusCode"] == 0: + pass + else: + print(inf) + + +# set path about run instance + +core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") +core_path = "/".join(core_path.split("/")[:-1]) +print(" ======= core path is %s ======== " %core_path) +if not os.path.exists(core_path): + os.mkdir(core_path) + +base_dir = os.path.dirname(os.path.realpath(__file__)) +if base_dir.find("community")>0: + repo = "community" +elif base_dir.find("TDengine")>0: + repo = "TDengine" +else: + repo ="TDengine" +print("base_dir:",base_dir) +home_dir = base_dir[:base_dir.find(repo)] +print("home_dir:",home_dir) +run_dir = os.path.join(home_dir,'run_dir') +run_dir = os.path.abspath(run_dir) +print("run dir is *** :",run_dir) +if not os.path.exists(run_dir): + os.mkdir(run_dir) +run_log_file = run_dir+'/crash_gen_run.log' +crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh') +exit_status_logs = os.path.join(run_dir, 'crash_exit.log') + +def get_path(): + buildPath='' + selfPath = os.path.dirname(os.path.realpath(__file__)) + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + +# generate crash_gen start script randomly + +def random_args(args_list): + nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception" + ] # record bool type arguments + strs_args_list = ["--connector-type"] # record str type arguments + + args_list["--auto-start-service"]= False + args_list["--continue-on-exception"]=True + # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test" + connect_types=['native'] + # args_list["--connector-type"]=connect_types[random.randint(0,2)] + args_list["--connector-type"]= connect_types[0] + args_list["--max-dbs"]= random.randint(1,10) + + # dnodes = [1,3] # set single dnodes; + + # args_list["--num-dnodes"]= random.sample(dnodes,1)[0] + # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"]) + args_list["--debug"]=False + args_list["--per-thread-db-connection"]=True + args_list["--track-memory-leaks"]=False + + args_list["--max-steps"]=random.randint(500,2000) + + # args_list["--ignore-errors"]=[] ## can add error codes for detail + + + args_list["--run-tdengine"]= False + args_list["--use-shadow-db"]= False + args_list["--dynamic-db-table-names"]= True + args_list["--verify-data"]= False + args_list["--record-ops"] = False + + for key in bools_args_list: + set_bool_value = [True,False] + if key == "--auto-start-service" : + continue + elif key =="--run-tdengine": + continue + elif key == "--ignore-errors": + continue + elif key == "--debug": + continue + elif key == "--per-thread-db-connection": + continue + elif key == "--continue-on-exception": + continue + elif key == "--use-shadow-db": + continue + elif key =="--track-memory-leaks": + continue + elif key == "--dynamic-db-table-names": + continue + elif key == "--verify-data": + continue + elif key == "--record-ops": + continue + else: + args_list[key]=set_bool_value[random.randint(0,1)] + + if args_list["--larger-data"]: + threads = [16,32] + else: + threads = [32,64,128,256] + args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug + + return args_list + +def limits(args_list): + if args_list["--use-shadow-db"]==True: + if args_list["--max-dbs"] > 1: + print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1") + args_list["--max-dbs"]=1 + else: + pass + + # env is start by test frame , not crash_gen instance + + # elif args_list["--num-replicas"]==0: + # print(" make sure num-replicas is at least 1 ") + # args_list["--num-replicas"]=1 + # elif args_list["--num-replicas"]==1: + # pass + + # elif args_list["--num-replicas"]>1: + # if not args_list["--auto-start-service"]: + # print("it should be deployed by crash_gen auto-start-service for multi replicas") + + # else: + # pass + + return args_list + +def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"] + arguments = "" + for k ,v in args_list.items(): + if k == "--ignore-errors": + if v: + arguments+=(k+"="+str(v)+" ") + else: + arguments+="" + elif k in bools_args_list and v==True: + arguments+=(k+" ") + elif k in bools_args_list and v==False: + arguments+="" + else: + arguments+=(k+"="+str(v)+" ") + + if valgrind : + + crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments) + + else: + + crash_gen_cmd = 'cd %s && ./crash_gen.sh %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments) + + return crash_gen_cmd + +def start_taosd(): + build_path = get_path() + if repo == "community": + start_path = build_path[:-5]+"community/tests/system-test/" + elif repo == "TDengine": + start_path = build_path[:-5]+"/tests/system-test/" + else: + pass + + start_cmd = 'cd %s && python3 test.py >>/dev/null '%(start_path) + os.system(start_cmd) + +def get_cmds(args_list): + # build_path = get_path() + # if repo == "community": + # crash_gen_path = build_path[:-5]+"community/tests/pytest/" + # elif repo == "TDengine": + # crash_gen_path = build_path[:-5]+"/tests/pytest/" + # else: + # pass + + # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path) + + crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode) + return crash_gen_cmd + +def run_crash_gen(crash_cmds): + + # prepare env of taosd + start_taosd() + + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + result_file = os.path.join(crash_gen_path, 'valgrind.out') + + + # run crash_gen and back logs + os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file)) + os.system("%s >>%s "%(crash_cmds,result_file)) + + +def check_status(): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + result_file = os.path.join(crash_gen_path, 'valgrind.out') + run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs)) + + core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if int(core_check.strip().rstrip()) > 0: + # it means core files has occured + return 3 + + if "Crash_Gen is now exiting with status code: 1" in run_code: + return 1 + elif "Crash_Gen is now exiting with status code: 0" in run_code: + return 0 + else: + return 2 + + +def main(): + + args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[], + "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False, + "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , + "--continue-on-exception":False } + + args = random_args(args_list) + args = limits(args) + + + build_path = get_path() + os.system("pip install git+https://github.com/taosdata/taos-connector-python.git") + if repo =="community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo =="TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + if os.path.exists(crash_gen_path+"crash_gen.sh"): + print(" make sure crash_gen.sh is ready") + else: + print( " crash_gen.sh is not exists ") + sys.exit(1) + + git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16] + + # crash_cmds = get_cmds() + + crash_cmds = get_cmds(args) + # clean run_dir + os.system('rm -rf %s'%run_dir ) + if not os.path.exists(run_dir): + os.mkdir(run_dir) + print(crash_cmds) + run_crash_gen(crash_cmds) + status = check_status() + + print("exit status : ", status) + + if status ==4: + print('======== crash_gen found memory bugs ========') + if status ==5: + print('======== crash_gen found memory errors ========') + if status >0: + print('======== crash_gen run failed and not exit as expected ========') + else: + print('======== crash_gen run sucess and exit as expected ========') + + + if status!=0 : + + try: + text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}" + send_msg(get_msg(text)) + except Exception as e: + print("exception:", e) + exit(status) + + +if __name__ == '__main__': + main() + + diff --git a/tests/pytest/auto_crash_gen_valgrind.py b/tests/pytest/auto_crash_gen_valgrind.py new file mode 100755 index 0000000000000000000000000000000000000000..ce87fec684bdfaa4941a9ffcc01e984143aab559 --- /dev/null +++ b/tests/pytest/auto_crash_gen_valgrind.py @@ -0,0 +1,399 @@ +#!/usr/bin/python3 + + +import os +import socket +import requests + +# -*- coding: utf-8 -*- +import os ,sys +import random +import argparse +import subprocess +import time +import platform + +# valgrind mode ? +valgrind_mode = True + +msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" } + +# formal +hostname = socket.gethostname() + +group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9' + +def get_msg(text): + return { + "msg_type": "post", + "content": { + "post": { + "zh_cn": { + "title": "Crash_gen Monitor", + "content": [ + [{ + "tag": "text", + "text": text + } + ]] + } + } + } + } + + +def send_msg(json): + headers = { + 'Content-Type': 'application/json' + } + + req = requests.post(url=group_url, headers=headers, json=json) + inf = req.json() + if "StatusCode" in inf and inf["StatusCode"] == 0: + pass + else: + print(inf) + + +# set path about run instance + +core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") +core_path = "/".join(core_path.split("/")[:-1]) +print(" ======= core path is %s ======== " %core_path) +if not os.path.exists(core_path): + os.mkdir(core_path) + +base_dir = os.path.dirname(os.path.realpath(__file__)) +if base_dir.find("community")>0: + repo = "community" +elif base_dir.find("TDengine")>0: + repo = "TDengine" +else: + repo ="TDengine" +print("base_dir:",base_dir) +home_dir = base_dir[:base_dir.find(repo)] +print("home_dir:",home_dir) +run_dir = os.path.join(home_dir,'run_dir') +run_dir = os.path.abspath(run_dir) +print("run dir is *** :",run_dir) +if not os.path.exists(run_dir): + os.mkdir(run_dir) +run_log_file = run_dir+'/crash_gen_run.log' +crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh') +exit_status_logs = os.path.join(run_dir, 'crash_exit.log') + +def get_path(): + buildPath='' + selfPath = os.path.dirname(os.path.realpath(__file__)) + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + +# generate crash_gen start script randomly + +def random_args(args_list): + nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception" + ] # record bool type arguments + strs_args_list = ["--connector-type"] # record str type arguments + + args_list["--auto-start-service"]= False + args_list["--continue-on-exception"]=True + # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test" + connect_types=['native'] + # args_list["--connector-type"]=connect_types[random.randint(0,2)] + args_list["--connector-type"]= connect_types[0] + args_list["--max-dbs"]= random.randint(1,10) + + # dnodes = [1,3] # set single dnodes; + + # args_list["--num-dnodes"]= random.sample(dnodes,1)[0] + # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"]) + args_list["--debug"]=False + args_list["--per-thread-db-connection"]=True + args_list["--track-memory-leaks"]=False + + args_list["--max-steps"]=random.randint(200,500) + + threads = [16,32] + + args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug + # args_list["--ignore-errors"]=[] ## can add error codes for detail + + + args_list["--run-tdengine"]= False + args_list["--use-shadow-db"]= False + args_list["--dynamic-db-table-names"]= True + args_list["--verify-data"]= False + args_list["--record-ops"] = False + + for key in bools_args_list: + set_bool_value = [True,False] + if key == "--auto-start-service" : + continue + elif key =="--run-tdengine": + continue + elif key == "--ignore-errors": + continue + elif key == "--debug": + continue + elif key == "--per-thread-db-connection": + continue + elif key == "--continue-on-exception": + continue + elif key == "--use-shadow-db": + continue + elif key =="--track-memory-leaks": + continue + elif key == "--dynamic-db-table-names": + continue + elif key == "--verify-data": + continue + elif key == "--record-ops": + continue + elif key == "--larger-data": + continue + else: + args_list[key]=set_bool_value[random.randint(0,1)] + return args_list + +def limits(args_list): + if args_list["--use-shadow-db"]==True: + if args_list["--max-dbs"] > 1: + print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1") + args_list["--max-dbs"]=1 + else: + pass + + # env is start by test frame , not crash_gen instance + + # elif args_list["--num-replicas"]==0: + # print(" make sure num-replicas is at least 1 ") + # args_list["--num-replicas"]=1 + # elif args_list["--num-replicas"]==1: + # pass + + # elif args_list["--num-replicas"]>1: + # if not args_list["--auto-start-service"]: + # print("it should be deployed by crash_gen auto-start-service for multi replicas") + + # else: + # pass + + return args_list + +def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"] + arguments = "" + for k ,v in args_list.items(): + if k == "--ignore-errors": + if v: + arguments+=(k+"="+str(v)+" ") + else: + arguments+="" + elif k in bools_args_list and v==True: + arguments+=(k+" ") + elif k in bools_args_list and v==False: + arguments+="" + else: + arguments+=(k+"="+str(v)+" ") + + if valgrind : + + crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments) + + else: + + crash_gen_cmd = 'cd %s && ./crash_gen.sh %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments) + + return crash_gen_cmd + + +def start_taosd(): + build_path = get_path() + if repo == "community": + start_path = build_path[:-5]+"community/tests/system-test/" + elif repo == "TDengine": + start_path = build_path[:-5]+"/tests/system-test/" + else: + pass + + start_cmd = 'cd %s && python3 test.py '%(start_path) + os.system(start_cmd +">>/dev/null") + +def get_cmds(args_list): + # build_path = get_path() + # if repo == "community": + # crash_gen_path = build_path[:-5]+"community/tests/pytest/" + # elif repo == "TDengine": + # crash_gen_path = build_path[:-5]+"/tests/pytest/" + # else: + # pass + + # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path) + + crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode) + return crash_gen_cmd + +def run_crash_gen(crash_cmds): + + # prepare env of taosd + start_taosd() + # run crash_gen and back logs + os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file)) + # os.system("cp %s %s"%(crash_gen_cmds_file, core_path)) + os.system("%s"%(crash_cmds)) + +def check_status(): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + result_file = os.path.join(crash_gen_path, 'valgrind.out') + run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs)) + + core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if int(core_check.strip().rstrip()) > 0: + # it means core files has occured + return 3 + + mem_status = check_memory() + if mem_status >0: + return mem_status + if "Crash_Gen is now exiting with status code: 1" in run_code: + return 1 + elif "Crash_Gen is now exiting with status code: 0" in run_code: + return 0 + else: + return 2 + + +def check_memory(): + + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + ''' + invalid read, invalid write + ''' + back_path = os.path.join(core_path,"valgrind_report") + if not os.path.exists(back_path): + os.mkdir(back_path) + + stderr_file = os.path.join(crash_gen_path , "valgrind.err") + + status = 0 + + grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 4 + + grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 4 + + grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 5 + + return status + +def main(): + + args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[], + "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False, + "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , + "--continue-on-exception":False } + + args = random_args(args_list) + args = limits(args) + + build_path = get_path() + os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null") + if repo =="community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo =="TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + if os.path.exists(crash_gen_path+"crash_gen.sh"): + print(" make sure crash_gen.sh is ready") + else: + print( " crash_gen.sh is not exists ") + sys.exit(1) + + git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16] + + # crash_cmds = get_cmds() + + crash_cmds = get_cmds(args) + + # clean run_dir + os.system('rm -rf %s'%run_dir ) + if not os.path.exists(run_dir): + os.mkdir(run_dir) + print(crash_cmds) + run_crash_gen(crash_cmds) + status = check_status() + # back_path = os.path.join(core_path,"valgrind_report") + + print("exit status : ", status) + + if status ==4: + print('======== crash_gen found memory bugs ========') + if status ==5: + print('======== crash_gen found memory errors ========') + if status >0: + print('======== crash_gen run failed and not exit as expected ========') + else: + print('======== crash_gen run sucess and exit as expected ========') + + if status!=0 : + + try: + text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}" + send_msg(get_msg(text)) + except Exception as e: + print("exception:", e) + exit(status) + + +if __name__ == '__main__': + main() + + diff --git a/tests/pytest/auto_crash_gen_valgrind_cluster.py b/tests/pytest/auto_crash_gen_valgrind_cluster.py new file mode 100755 index 0000000000000000000000000000000000000000..f4afa80afe999a30d46619b7bba6905a43e820c4 --- /dev/null +++ b/tests/pytest/auto_crash_gen_valgrind_cluster.py @@ -0,0 +1,399 @@ +#!/usr/bin/python3 + + +import os +import socket +import requests + +# -*- coding: utf-8 -*- +import os ,sys +import random +import argparse +import subprocess +import time +import platform + +# valgrind mode ? +valgrind_mode = True + +msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" } + +# formal +hostname = socket.gethostname() + +group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9' + +def get_msg(text): + return { + "msg_type": "post", + "content": { + "post": { + "zh_cn": { + "title": "Crash_gen Monitor", + "content": [ + [{ + "tag": "text", + "text": text + } + ]] + } + } + } + } + + +def send_msg(json): + headers = { + 'Content-Type': 'application/json' + } + + req = requests.post(url=group_url, headers=headers, json=json) + inf = req.json() + if "StatusCode" in inf and inf["StatusCode"] == 0: + pass + else: + print(inf) + + +# set path about run instance + +core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") +core_path = "/".join(core_path.split("/")[:-1]) +print(" ======= core path is %s ======== " %core_path) +if not os.path.exists(core_path): + os.mkdir(core_path) + +base_dir = os.path.dirname(os.path.realpath(__file__)) +if base_dir.find("community")>0: + repo = "community" +elif base_dir.find("TDengine")>0: + repo = "TDengine" +else: + repo ="TDengine" +print("base_dir:",base_dir) +home_dir = base_dir[:base_dir.find(repo)] +print("home_dir:",home_dir) +run_dir = os.path.join(home_dir,'run_dir') +run_dir = os.path.abspath(run_dir) +print("run dir is *** :",run_dir) +if not os.path.exists(run_dir): + os.mkdir(run_dir) +run_log_file = run_dir+'/crash_gen_run.log' +crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh') +exit_status_logs = os.path.join(run_dir, 'crash_exit.log') + +def get_path(): + buildPath='' + selfPath = os.path.dirname(os.path.realpath(__file__)) + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + +# generate crash_gen start script randomly + +def random_args(args_list): + nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception" + ] # record bool type arguments + strs_args_list = ["--connector-type"] # record str type arguments + + args_list["--auto-start-service"]= False + args_list["--continue-on-exception"]=True + # connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test" + connect_types=['native'] + # args_list["--connector-type"]=connect_types[random.randint(0,2)] + args_list["--connector-type"]= connect_types[0] + args_list["--max-dbs"]= random.randint(1,10) + + # dnodes = [1,3] # set single dnodes; + + # args_list["--num-dnodes"]= random.sample(dnodes,1)[0] + # args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"]) + args_list["--debug"]=False + args_list["--per-thread-db-connection"]=True + args_list["--track-memory-leaks"]=False + + args_list["--max-steps"]=random.randint(200,500) + + threads = [16,32] + + args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug + # args_list["--ignore-errors"]=[] ## can add error codes for detail + + + args_list["--run-tdengine"]= False + args_list["--use-shadow-db"]= False + args_list["--dynamic-db-table-names"]= True + args_list["--verify-data"]= False + args_list["--record-ops"] = False + + for key in bools_args_list: + set_bool_value = [True,False] + if key == "--auto-start-service" : + continue + elif key =="--run-tdengine": + continue + elif key == "--ignore-errors": + continue + elif key == "--debug": + continue + elif key == "--per-thread-db-connection": + continue + elif key == "--continue-on-exception": + continue + elif key == "--use-shadow-db": + continue + elif key =="--track-memory-leaks": + continue + elif key == "--dynamic-db-table-names": + continue + elif key == "--verify-data": + continue + elif key == "--record-ops": + continue + elif key == "--larger-data": + continue + else: + args_list[key]=set_bool_value[random.randint(0,1)] + return args_list + +def limits(args_list): + if args_list["--use-shadow-db"]==True: + if args_list["--max-dbs"] > 1: + print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1") + args_list["--max-dbs"]=1 + else: + pass + + # env is start by test frame , not crash_gen instance + + # elif args_list["--num-replicas"]==0: + # print(" make sure num-replicas is at least 1 ") + # args_list["--num-replicas"]=1 + # elif args_list["--num-replicas"]==1: + # pass + + # elif args_list["--num-replicas"]>1: + # if not args_list["--auto-start-service"]: + # print("it should be deployed by crash_gen auto-start-service for multi replicas") + + # else: + # pass + + return args_list + +def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names", + "--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"] + arguments = "" + for k ,v in args_list.items(): + if k == "--ignore-errors": + if v: + arguments+=(k+"="+str(v)+" ") + else: + arguments+="" + elif k in bools_args_list and v==True: + arguments+=(k+" ") + elif k in bools_args_list and v==False: + arguments+="" + else: + arguments+=(k+"="+str(v)+" ") + + if valgrind : + + crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -i 3 %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0707,0x0203 '%(crash_gen_path ,arguments) + + else: + + crash_gen_cmd = 'cd %s && ./crash_gen.sh -i 3 %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0014,0x0707,0x0203'%(crash_gen_path ,arguments) + + return crash_gen_cmd + + +def start_taosd(): + build_path = get_path() + if repo == "community": + start_path = build_path[:-5]+"community/tests/system-test/" + elif repo == "TDengine": + start_path = build_path[:-5]+"/tests/system-test/" + else: + pass + + start_cmd = 'cd %s && python3 test.py -N 4 -M 1 '%(start_path) + os.system(start_cmd +">>/dev/null") + +def get_cmds(args_list): + # build_path = get_path() + # if repo == "community": + # crash_gen_path = build_path[:-5]+"community/tests/pytest/" + # elif repo == "TDengine": + # crash_gen_path = build_path[:-5]+"/tests/pytest/" + # else: + # pass + + # crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path) + + crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode) + return crash_gen_cmd + +def run_crash_gen(crash_cmds): + + # prepare env of taosd + start_taosd() + # run crash_gen and back logs + os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file)) + # os.system("cp %s %s"%(crash_gen_cmds_file, core_path)) + os.system("%s"%(crash_cmds)) + +def check_status(): + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + result_file = os.path.join(crash_gen_path, 'valgrind.out') + run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs)) + + core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if int(core_check.strip().rstrip()) > 0: + # it means core files has occured + return 3 + + mem_status = check_memory() + if mem_status >0: + return mem_status + if "Crash_Gen is now exiting with status code: 1" in run_code: + return 1 + elif "Crash_Gen is now exiting with status code: 0" in run_code: + return 0 + else: + return 2 + + +def check_memory(): + + build_path = get_path() + if repo == "community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo == "TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + ''' + invalid read, invalid write + ''' + back_path = os.path.join(core_path,"valgrind_report") + if not os.path.exists(back_path): + os.mkdir(back_path) + + stderr_file = os.path.join(crash_gen_path , "valgrind.err") + + status = 0 + + grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 4 + + grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 4 + + grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + + if grep_res: + # os.system("cp %s %s"%(stderr_file , back_path)) + status = 5 + + return status + +def main(): + + args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[], + "--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False, + "--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False , + "--continue-on-exception":False } + + args = random_args(args_list) + args = limits(args) + + build_path = get_path() + os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null") + if repo =="community": + crash_gen_path = build_path[:-5]+"community/tests/pytest/" + elif repo =="TDengine": + crash_gen_path = build_path[:-5]+"/tests/pytest/" + else: + pass + + if os.path.exists(crash_gen_path+"crash_gen.sh"): + print(" make sure crash_gen.sh is ready") + else: + print( " crash_gen.sh is not exists ") + sys.exit(1) + + git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16] + + # crash_cmds = get_cmds() + + crash_cmds = get_cmds(args) + + # clean run_dir + os.system('rm -rf %s'%run_dir ) + if not os.path.exists(run_dir): + os.mkdir(run_dir) + print(crash_cmds) + run_crash_gen(crash_cmds) + status = check_status() + # back_path = os.path.join(core_path,"valgrind_report") + + print("exit status : ", status) + + if status ==4: + print('======== crash_gen found memory bugs ========') + if status ==5: + print('======== crash_gen found memory errors ========') + if status >0: + print('======== crash_gen run failed and not exit as expected ========') + else: + print('======== crash_gen run sucess and exit as expected ========') + + if status!=0 : + + try: + text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}" + send_msg(get_msg(text)) + except Exception as e: + print("exception:", e) + exit(status) + + +if __name__ == '__main__': + main() + + diff --git a/tests/pytest/auto_run_regular.sh b/tests/pytest/auto_run_regular.sh new file mode 100755 index 0000000000000000000000000000000000000000..27e80132698994d99dbad7abc4134f0b240cb7db --- /dev/null +++ b/tests/pytest/auto_run_regular.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# set LD_LIBRARY_PATH +export PATH=$PATH:/home/TDengine/debug/build/bin +export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null +ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null + +# run crash_gen auto script +python3 /home/TDengine/tests/pytest/auto_crash_gen.py \ No newline at end of file diff --git a/tests/pytest/auto_run_valgrind.sh b/tests/pytest/auto_run_valgrind.sh new file mode 100755 index 0000000000000000000000000000000000000000..c7154e867c97b1349209b3ff3b9b40168d4c51b4 --- /dev/null +++ b/tests/pytest/auto_run_valgrind.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# set LD_LIBRARY_PATH +export PATH=$PATH:/home/TDengine/debug/build/bin +export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null +ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null + +# run crash_gen auto script +python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind.py \ No newline at end of file diff --git a/tests/pytest/auto_run_valgrind_cluster.sh b/tests/pytest/auto_run_valgrind_cluster.sh new file mode 100755 index 0000000000000000000000000000000000000000..62bc22e923a204538c2d467b6c71826a2a4a3db6 --- /dev/null +++ b/tests/pytest/auto_run_valgrind_cluster.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# set LD_LIBRARY_PATH +export PATH=$PATH:/home/TDengine/debug/build/bin +export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null +ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null +ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null + +# run crash_gen auto script +python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind_cluster.py \ No newline at end of file diff --git a/tests/pytest/docker_exec_service.sh b/tests/pytest/docker_exec_service.sh new file mode 100644 index 0000000000000000000000000000000000000000..4156a0bae515ba88a83a2954dc38117aa6d47f67 --- /dev/null +++ b/tests/pytest/docker_exec_service.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +mkdir -p /data/wz/crash_gen_logs/ +logdir='/data/wz/crash_gen_logs/' +date_tag=`date +%Y%m%d-%H%M%S` +hostname='vm_valgrind_' + +for i in {1..50} +do + echo $i + # create docker and start crash_gen + log_dir=${logdir}${hostname}${date_tag}_${i} + docker run -d --hostname=${hostname}${date_tag}_${i} --name ${hostname}${date_tag}_${i} --privileged -v ${log_dir}:/corefile/ -- crash_gen:v1.0 sleep 99999999999999 + echo create docker ${hostname}${date_tag}_${i} + docker exec -d ${hostname}${date_tag}_${i} sh -c 'rm -rf /home/taos-connector-python' + docker cp /data/wz/TDengine ${hostname}${date_tag}_${i}:/home/TDengine + docker cp /data/wz/taos-connector-python ${hostname}${date_tag}_${i}:/home/taos-connector-python + echo copy TDengine in container done! + docker exec ${hostname}${date_tag}_${i} sh -c 'sh /home/TDengine/tests/pytest/auto_run_valgrind.sh ' + if [ $? -eq 0 ] + then + echo crash_gen exit as expect , run success + + # # clear docker which success + docker stop ${hostname}${date_tag}_${i} + docker rm -f ${hostname}${date_tag}_${i} + else + docker stop ${hostname}${date_tag}_${i} + echo crash_gen exit error , run failed + fi +done \ No newline at end of file diff --git a/tests/system-test/7-tmq/tmqDnodeRestart1.py b/tests/system-test/7-tmq/tmqDnodeRestart1.py index 714f10b362f9a901252fcc63c790977e266b76d5..982cc0a6313f8482031290acd89ea9f556dd5be3 100644 --- a/tests/system-test/7-tmq/tmqDnodeRestart1.py +++ b/tests/system-test/7-tmq/tmqDnodeRestart1.py @@ -149,6 +149,8 @@ class TDTestCase: tmqCom.waitSubscriptionExit(tdSql, topicFromStb) tdSql.query("drop topic %s"%topicFromStb) + + tmqCom.stopTmqSimProcess(processorName="tmq_sim") tdLog.printNoPrefix("======== test case 1 end ...... ") @@ -178,6 +180,8 @@ class TDTestCase: paraDict['vgroups'] = self.vgroups paraDict['ctbNum'] = self.ctbNum paraDict['rowsPerTbl'] = self.rowsPerTbl + + tmqCom.initConsumerTable() tdLog.info("create topics from stb") topicFromDb = 'topic_db' @@ -203,10 +207,10 @@ class TDTestCase: tmqCom.getStartCommitNotifyFromTmqsim('cdb',1) tdLog.info("create some new child table and insert data for latest mode") - paraDict["batchNum"] = 100 + paraDict["batchNum"] = 10 paraDict["ctbPrefix"] = 'newCtb' - paraDict["ctbNum"] = 10 - paraDict["rowsPerTbl"] = 10 + paraDict["ctbNum"] = 100 + paraDict["rowsPerTbl"] = 100 tmqCom.insert_data_with_autoCreateTbl(tdSql,paraDict["dbName"],paraDict["stbName"],paraDict["ctbPrefix"],paraDict["ctbNum"],paraDict["rowsPerTbl"],paraDict["batchNum"]) tdLog.info("================= restart dnode ===========================") diff --git a/tests/system-test/pytest.sh b/tests/system-test/pytest.sh index 3f0305834288993bf5a9d430d65b78f6b46696cc..148f80eca32e516be27deae6eb7d06ebab4549e0 100755 --- a/tests/system-test/pytest.sh +++ b/tests/system-test/pytest.sh @@ -77,13 +77,14 @@ echo "Preload AsanSo:" $? $* -a 2> $AsanFile unset LD_PRELOAD -AsanFileLen=`cat $AsanFile | wc -l` -while [ $AsanFileLen -lt 10 ] +for ((i=1;i<=20;i++)) do - sleep 1 - `cat $AsanFile | wc -l` + AsanFileLen=`cat $AsanFile | wc -l` + echo "AsanFileLen:" $AsanFileLen + if [ $AsanFileLen -gt 10 ]; then + break + fi done -echo "AsanFileLen:" $AsanFileLen AsanFileSuccessLen=`grep -w successfully $AsanFile | wc -l` echo "AsanFileSuccessLen:" $AsanFileSuccessLen diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt index 0f3c383b9edd6639ba82a09b4d2041cacadcb658..d61d25602b081d9cc6e7609f7e807add5a71e597 100644 --- a/tools/CMakeLists.txt +++ b/tools/CMakeLists.txt @@ -101,19 +101,20 @@ ELSE () BUILD_COMMAND COMMAND set CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client COMMAND set CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib - COMMAND go build -a -o taosadapter.exe -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" - COMMAND go build -a -o taosadapter-debug.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" + COMMAND go build -a -o taosadapter.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" +# COMMAND go build -a -o taosadapter.exe -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" +# COMMAND go build -a -o taosadapter-debug.exe -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" INSTALL_COMMAND - COMMAND cmake -E echo "Comparessing taosadapter.exe" - COMMAND cmake -E time upx taosadapter.exe +# COMMAND cmake -E echo "Comparessing taosadapter.exe" +# COMMAND cmake -E time upx taosadapter.exe COMMAND cmake -E echo "Copy taosadapter.exe" COMMAND cmake -E copy taosadapter.exe ${CMAKE_BINARY_DIR}/build/bin/taosadapter.exe COMMAND cmake -E make_directory ${CMAKE_BINARY_DIR}/test/cfg/ COMMAND cmake -E echo "Copy taosadapter.toml" COMMAND cmake -E copy ./example/config/taosadapter.toml ${CMAKE_BINARY_DIR}/test/cfg/ - COMMAND cmake -E echo "Copy taosadapter-debug.exe" - COMMAND cmake -E copy taosadapter-debug.exe ${CMAKE_BINARY_DIR}/build/bin +# COMMAND cmake -E echo "Copy taosadapter-debug.exe" +# COMMAND cmake -E copy taosadapter-debug.exe ${CMAKE_BINARY_DIR}/build/bin ) ELSE (TD_WINDOWS) MESSAGE("Building taosAdapter on non-Windows") @@ -128,19 +129,20 @@ ELSE () PATCH_COMMAND COMMAND git clean -f -d BUILD_COMMAND - COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib go build -a -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" - COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib go build -a -o taosadapter-debug -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" + COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib go build -a -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" +# COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib go build -a -ldflags "-s -w -X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" +# COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib go build -a -o taosadapter-debug -ldflags "-X github.com/taosdata/taosadapter/v3/version.Version=${taos_version} -X github.com/taosdata/taosadapter/v3/version.CommitID=${taosadapter_commit_sha1}" INSTALL_COMMAND - COMMAND cmake -E echo "Comparessing taosadapter.exe" - COMMAND upx taosadapter || : +# COMMAND cmake -E echo "Comparessing taosadapter.exe" +# COMMAND upx taosadapter || : COMMAND cmake -E echo "Copy taosadapter" COMMAND cmake -E copy taosadapter ${CMAKE_BINARY_DIR}/build/bin COMMAND cmake -E make_directory ${CMAKE_BINARY_DIR}/test/cfg/ COMMAND cmake -E echo "Copy taosadapter.toml" COMMAND cmake -E copy ./example/config/taosadapter.toml ${CMAKE_BINARY_DIR}/test/cfg/ COMMAND cmake -E copy ./taosadapter.service ${CMAKE_BINARY_DIR}/test/cfg/ - COMMAND cmake -E echo "Copy taosadapter-debug" - COMMAND cmake -E copy taosadapter-debug ${CMAKE_BINARY_DIR}/build/bin +# COMMAND cmake -E echo "Copy taosadapter-debug" +# COMMAND cmake -E copy taosadapter-debug ${CMAKE_BINARY_DIR}/build/bin ) ENDIF (TD_WINDOWS) ENDIF ()