Merge pull request #7 from wu-champion/develop

更新分支1380

README-CN.md

+[![Build Status](https://travis-ci.org/taosdata/TDengine.svg?branch=master)](https://travis-ci.org/taosdata/TDengine)
+[![Build status](https://ci.appveyor.com/api/projects/status/kf3pwh2or5afsgl9/branch/master?svg=true)](https://ci.appveyor.com/project/sangshuduo/tdengine-2n8ge/branch/master)
+[![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=develop)](https://coveralls.io/github/taosdata/TDengine?branch=develop)
+[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
+[![tdengine](https://snapcraft.io//tdengine/badge.svg)](https://snapcraft.io/tdengine)
+
+[![TDengine](TDenginelogo.png)](https://www.taosdata.com)
+
+简体中文 | [English](./README.md) 
+
+# TDengine 简介
+
+TDengine是涛思数据专为物联网、车联网、工业互联网、IT运维等设计和优化的大数据平台。除核心的快10倍以上的时序数据库功能外，还提供缓存、数据订阅、流式计算等功能，最大程度减少研发和运维的复杂度，且核心代码，包括集群功能全部开源（开源协议，AGPL v3.0）。
+
+- 10 倍以上性能提升。定义了创新的数据存储结构，单核每秒就能处理至少2万次请求，插入数百万个数据点，读出一千万以上数据点，比现有通用数据库快了十倍以上。
+- 硬件或云服务成本降至1/5。由于超强性能，计算资源不到通用大数据方案的1/5；通过列式存储和先进的压缩算法，存储空间不到通用数据库的1/10。
+- 全栈时序数据处理引擎。将数据库、消息队列、缓存、流式计算等功能融合一起，应用无需再集成Kafka/Redis/HBase/Spark等软件，大幅降低应用开发和维护成本。
+- 强大的分析功能。无论是十年前还是一秒钟前的数据，指定时间范围即可查询。数据可在时间轴上或多个设备上进行聚合。即席查询可通过Shell/Python/R/Matlab随时进行。
+- 与第三方工具无缝连接。不用一行代码，即可与Telegraf, Grafana, EMQ X, Prometheus, Matlab, R集成。后续还将支持MQTT, OPC, Hadoop，Spark等, BI工具也将无缝连接。
+- 零运维成本、零学习成本。安装、集群一秒搞定，无需分库分表，实时备份。标准SQL，支持JDBC,RESTful，支持Python/Java/C/C++/Go/Node.JS, 与MySQL相似，零学习成本。
+
+# 文档
+
+TDengine是一个高效的存储、查询、分析时序大数据的平台，专为物联网、车联网、工业互联网、运维监测等优化而设计。您可以像使用关系型数据库MySQL一样来使用它，但建议您在使用前仔细阅读一遍下面的文档，特别是 [数据模型](https://www.taosdata.com/cn/documentation/architecture) 与 [数据建模](https://www.taosdata.com/cn/documentation/model)。除本文档之外，欢迎 [下载产品白皮书](https://www.taosdata.com/downloads/TDengine%20White%20Paper.pdf)。
+
+# 生成
+
+TDengine目前2.0版服务器仅能在Linux系统上安装和运行，后续会支持Windows、macOS等系统。客户端可以在Windows或Linux上安装和运行。任何OS的应用也可以选择RESTful接口连接服务器taosd。CPU支持X64/ARM64/MIPS64/Alpha64，后续会支持ARM32、RISC-V等CPU架构。用户可根据需求选择通过[源码](https://www.taosdata.com/cn/getting-started/#通过源码安装)或者[安装包](https://www.taosdata.com/cn/getting-started/#通过安装包安装)来安装。本快速指南仅适用于通过源码安装。
+
+## 安装工具
+
+### Ubuntu 16.04 及以上版本 & Debian：
+
+```bash
+sudo apt-get install -y gcc cmake build-essential git
+```
+
+### Ubuntu 14.04：
+
+```bash
+sudo apt-get install -y gcc cmake3 build-essential git binutils-2.26
+export PATH=/usr/lib/binutils-2.26/bin:$PATH
+```
+
+编译或打包 JDBC 驱动源码，需安装 Java JDK 8 或以上版本和 Apache Maven 2.7 或以上版本。
+
+安装 OpenJDK 8：
+
+```bash
+sudo apt-get install -y openjdk-8-jdk
+```
+
+安装 Apache Maven：
+
+```bash
+sudo apt-get install -y  maven
+```
+
+### CentOS 7：
+
+```bash
+sudo yum install -y gcc gcc-c++ make cmake git
+```
+
+安装 OpenJDK 8：
+
+```bash
+sudo yum install -y java-1.8.0-openjdk
+```
+
+安装 Apache Maven：
+
+```bash
+sudo yum install -y maven
+```
+
+### CentOS 8 & Fedora:
+
+```bash
+sudo dnf install -y gcc gcc-c++ make cmake epel-release git
+```
+
+安装 OpenJDK 8：
+
+```bash
+sudo dnf install -y java-1.8.0-openjdk
+```
+
+安装 Apache Maven：
+
+```bash
+sudo dnf install -y maven
+```
+
+## 获取源码
+
+首先，你需要从 GitHub 克隆源码：
+
+```bash
+git clone https://github.com/taosdata/TDengine.git
+cd TDengine
+```
+
+Go 连接器和 Grafana 插件在其他独立仓库，如果安装它们的话，需要在 TDengine 目录下通过此命令安装：
+
+```bash
+git submodule update --init --recursive
+```
+
+## 生成 TDengine
+
+### Linux 系统
+
+```bash
+mkdir debug && cd debug
+cmake .. && cmake --build .
+```
+
+在X86-64、X86、arm64 和 arm32 平台上，TDengine 生成脚本可以自动检测机器架构。也可以手动配置 CPUTYPE 参数来指定 CPU 类型，如 aarch64 或 aarch32 等。
+
+aarch64：
+
+```bash
+cmake .. -DCPUTYPE=aarch64 && cmake --build .
+```
+
+aarch32：
+
+```bash
+cmake .. -DCPUTYPE=aarch32 && cmake --build .
+```
+
+### Windows 系统
+
+如果你使用的是 Visual Studio 2013 版本：
+
+打开 cmd.exe，执行 vcvarsall.bat 时，为 64 位操作系统指定“x86_amd64”，为 32 位操作系统指定“x86”。
+
+```bash
+mkdir debug && cd debug
+"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" < x86_amd64 | x86 >
+cmake .. -G "NMake Makefiles"
+nmake
+```
+
+如果你使用的是 Visual Studio 2019 或 2017 版本：
+
+打开cmd.exe，执行 vcvarsall.bat 时，为 64 位操作系统指定“x64”，为 32 位操作系统指定“x86”。
+
+```bash
+mkdir debug && cd debug
+"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" < x64 | x86 >
+cmake .. -G "NMake Makefiles"
+nmake
+```
+
+你也可以从开始菜单中找到"Visual Studio < 2019 | 2017 >"菜单项，根据你的系统选择"x64 Native Tools Command Prompt for VS < 2019 | 2017 >"或"x86 Native Tools Command Prompt for VS < 2019 | 2017 >"，打开命令行窗口，执行：
+
+```bash
+mkdir debug && cd debug
+cmake .. -G "NMake Makefiles"
+nmake
+```
+
+### Mac OS X 系统
+
+安装 Xcode 命令行工具和 cmake. 在 Catalina 和 Big Sur 操作系统上，需要安装 XCode 11.4+ 版本。
+
+```bash
+mkdir debug && cd debug
+cmake .. && cmake --build .
+```
+
+# 安装
+
+如果你不想安装，可以直接在shell中运行。生成完成后，安装 TDengine：
+```bash
+make install
+```
+
+用户可以在[文件目录结构](https://www.taosdata.com/cn/documentation/administrator#directories)中了解更多在操作系统中生成的目录或文件。
+
+安装成功后，在终端中启动 TDengine 服务：
+
+```bash
+taosd
+```
+
+用户可以使用 TDengine Shell 来连接 TDengine 服务，在终端中，输入：
+
+```bash
+taos
+```
+
+如果 TDengine Shell 连接服务成功，将会打印出欢迎消息和版本信息。如果失败，则会打印出错误消息。
+
+## 快速运行
+
+TDengine 生成后，在终端执行以下命令：
+
+```bash
+./build/bin/taosd -c test/cfg
+```
+
+在另一个终端，使用 TDengine shell 连接服务器：
+
+```bash
+./build/bin/taos -c test/cfg
+```
+
+"-c test/cfg"指定系统配置文件所在目录。
+
+# 体验 TDengine
+
+在TDengine终端中，用户可以通过SQL命令来创建/删除数据库、表等，并进行插入查询操作。
+
+```bash
+create database demo;
+use demo;
+create table t (ts timestamp, speed int);
+insert into t values ('2019-07-15 00:00:00', 10);
+insert into t values ('2019-07-15 01:00:00', 20);
+select * from t;
+          ts          |   speed   |
+===================================
+ 19-07-15 00:00:00.000|         10|
+ 19-07-15 01:00:00.000|         20|
+Query OK, 2 row(s) in set (0.001700s)
+```
+
+# 应用开发
+
+## 官方连接器
+
+TDengine 提供了丰富的应用程序开发接口，其中包括C/C++、Java、Python、Go、Node.js、C# 、RESTful 等，便于用户快速开发应用：
+
+- Java
+
+- C/C++
+
+- Python
+
+- Go
+
+- RESTful API
+
+- Node.js
+
+## 第三方连接器
+
+TDengine 社区生态中也有一些非常友好的第三方连接器，可以通过以下链接访问它们的源码。
+
+- [Rust Connector](https://github.com/taosdata/TDengine/tree/master/tests/examples/rust)
+- [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos)
+- [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/tests/examples/lua)
+
+# 运行和添加测试例
+
+TDengine 的测试框架和所有测试例全部开源。
+
+点击[这里](tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md)，了解如何运行测试例和添加新的测试例。
+
+# 成为社区贡献者
+点击[这里](https://www.taosdata.com/cn/contributor/)，了解如何成为 TDengine 的贡献者。
+
+#加入技术交流群
+TDengine官方社群「物联网大数据群」对外开放，欢迎您加入讨论。搜索微信号 "tdengine"，加小T为好友，即可入群。

README.md

@@ -6,6 +6,8 @@
 
 [![TDengine](TDenginelogo.png)](https://www.taosdata.com)
 
+English | [简体中文](./README-CN.md) 
+
 # What is TDengine？
 
 TDengine is an open-sourced big data platform under [GNU AGPL v3.0](http://www.gnu.org/licenses/agpl-3.0.html), designed and optimized for the Internet of Things (IoT), Connected Cars, Industrial IoT, and IT Infrastructure and Application Monitoring. Besides the 10x faster time-series database, it provides caching, stream computing, message queuing and other functionalities to reduce the complexity and cost of development and operation.
@@ -160,24 +162,13 @@ mkdir debug && cd debug
 cmake .. && cmake --build .
 ```
 
-# Quick Run
-
-# Quick Run
-To quickly start a TDengine server after building, run the command below in terminal:
-```bash
-./build/bin/taosd -c test/cfg
-```
-In another terminal, use the TDengine shell to connect the server:
-```bash
-./build/bin/taos -c test/cfg
-```
-option "-c test/cfg" specifies the system configuration file directory. 
-
 # Installing
+
 After building successfully, TDengine can be installed by:
 ```bash
 sudo make install
 ```
+
 Users can find more information about directories installed on the system in the [directory and files](https://www.taosdata.com/en/documentation/administrator/#Directory-and-Files) section. Since version 2.0, installing from source code will also configure service management for TDengine.
 Users can also choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) for it.
 
@@ -193,6 +184,20 @@ taos
 
 If TDengine shell connects the server successfully, welcome messages and version info are printed. Otherwise, an error message is shown.
 
+## Quick Run
+
+If you don't want to run TDengine as a service, you can run it in current shell. For example, to quickly start a TDengine server after building, run the command below in terminal:
+```bash
+./build/bin/taosd -c test/cfg
+```
+
+In another terminal, use the TDengine shell to connect the server:
+```bash
+./build/bin/taos -c test/cfg
+```
+
+option "-c test/cfg" specifies the system configuration file directory. 
+
 # Try TDengine
 It is easy to run SQL commands from TDengine shell which is the same as other SQL databases.
 ```sql

documentation20/cn/03.architecture/docs.md

@@ -145,7 +145,7 @@ TDengine 建议用数据采集点的名字(如上表中的D1001)来做表名。
 
 在TDengine的设计里，**表用来代表一个具体的数据采集点，超级表用来代表一组相同类型的数据采集点集合**。当为某个具体数据采集点创建表时，用户使用超级表的定义做模板，同时指定该具体采集点（表）的标签值。与传统的关系型数据库相比，表（一个数据采集点）是带有静态标签的，而且这些标签可以事后增加、删除、修改。**一张超级表包含有多张表，这些表具有相同的时序数据schema，但带有不同的标签值**。
 
-当对多个具有相同数据类型的数据采集点进行聚合操作时，TDengine将先把满足标签过滤条件的表从超级表的中查找出来，然后再扫描这些表的时序数据，进行聚合操作，这样能将需要扫描的数据集大幅减少，从而大幅提高聚合计算的性能。
+当对多个具有相同数据类型的数据采集点进行聚合操作时，TDengine会先把满足标签过滤条件的表从超级表中找出来，然后再扫描这些表的时序数据，进行聚合操作，这样需要扫描的数据集会大幅减少，从而显著提高聚合计算的性能。
 
 ## <a class="anchor" id="cluster"></a>集群与基本逻辑单元
 

documentation20/cn/12.taos-sql/docs.md

@@ -125,7 +125,7 @@ TDengine缺省的时间戳是毫秒精度，但通过修改配置参数enableMic
     ```mysql
     ALTER DATABASE db_name CACHELAST 0;
     ```
-    CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11 版本开始支持）
+    CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11 版本开始支持，修改后需要重启服务器生效。）
 
     **Tips**: 以上所有参数修改后都可以用show databases来确认是否修改成功。
 

documentation20/cn/13.faq/docs.md

@@ -16,13 +16,13 @@
 
 ## 1. TDengine2.0之前的版本升级到2.0及以上的版本应该注意什么？☆☆☆
 
-2.0版本在之前版本的基础上，进行了完全的重构，配置文件和数据文件是不兼容的。在升级之前务必进行如下操作：
+2.0版在之前版本的基础上，进行了完全的重构，配置文件和数据文件是不兼容的。在升级之前务必进行如下操作：
 
-1. 删除配置文件，执行 <code> sudo rm -rf /etc/taos/taos.cfg </code>
-2. 删除日志文件，执行 <code> sudo rm -rf /var/log/taos/ </code>
-3. 确保数据已经不再需要的前提下，删除数据文件，执行 <code> sudo rm -rf /var/lib/taos/ </code>
-4. 安装最新稳定版本的TDengine
-5. 如果数据需要迁移数据或者数据文件损坏，请联系涛思数据官方技术支持团队，进行协助解决
+1. 删除配置文件，执行 `sudo rm -rf /etc/taos/taos.cfg`
+2. 删除日志文件，执行 `sudo rm -rf /var/log/taos/`
+3. 确保数据已经不再需要的前提下，删除数据文件，执行 `sudo rm -rf /var/lib/taos/`
+4. 安装最新稳定版本的 TDengine
+5. 如果需要迁移数据或者数据文件损坏，请联系涛思数据官方技术支持团队，进行协助解决
 
 ## 2. Windows平台下JDBCDriver找不到动态链接库，怎么办？
 

packaging/tools/remove.sh

@@ -213,10 +213,10 @@ fi
 
 if echo $osinfo | grep -qwi "ubuntu" ; then
 #  echo "this is ubuntu system"
-   ${csudo} rm -f /var/lib/dpkg/info/tdengine* || :
+   ${csudo} dpkg --force-all  -P tdengine || :
 elif echo $osinfo | grep -qwi "debian" ; then
 #  echo "this is debian system"
-   ${csudo} rm -f /var/lib/dpkg/info/tdengine* || :
+   ${csudo} dpkg --force-all  -P tdengine || :
 elif  echo $osinfo | grep -qwi "centos" ; then
 #  echo "this is centos system"
   ${csudo} rpm -e --noscripts tdengine || :

src/balance/src/bnMain.c

@@ -425,7 +425,7 @@ static bool bnMonitorVgroups() {
 
   while (1) {
     pIter = mnodeGetNextVgroup(pIter, &pVgroup);
-    if (pVgroup == NULL) break;
+    if (pVgroup == NULL || pVgroup->pDb == NULL) break;
 
     int32_t dbReplica = pVgroup->pDb->cfg.replications;
     int32_t vgReplica = pVgroup->numOfVnodes;

src/client/src/tscAsync.c

@@ -281,7 +281,7 @@ void tscQueueAsyncError(void(*fp), void *param, int32_t code) {
 }
 
 static void tscAsyncResultCallback(SSchedMsg *pMsg) {
-  SSqlObj* pSql = pMsg->ahandle;
+  SSqlObj* pSql = (SSqlObj*)taosAcquireRef(tscObjRef, (int64_t)pMsg->ahandle);
   if (pSql == NULL || pSql->signature != pSql) {
     tscDebug("%p SqlObj is freed, not add into queue async res", pSql);
     return;
@@ -292,23 +292,26 @@ static void tscAsyncResultCallback(SSchedMsg *pMsg) {
 
   SSqlRes *pRes = &pSql->res;
   if (pSql->fp == NULL || pSql->fetchFp == NULL){
+    taosReleaseRef(tscObjRef, pSql->self);
     return;
   }
 
   pSql->fp = pSql->fetchFp;
   (*pSql->fp)(pSql->param, pSql, pRes->code);
+  taosReleaseRef(tscObjRef, pSql->self);
 }
 
 void tscAsyncResultOnError(SSqlObj* pSql) { 
   SSchedMsg schedMsg = {0};
   schedMsg.fp = tscAsyncResultCallback;
-  schedMsg.ahandle = pSql;
+  schedMsg.ahandle = (void *)pSql->self;
   schedMsg.thandle = (void *)1;
   schedMsg.msg = 0;
   taosScheduleTask(tscQhandle, &schedMsg);
 }
 
 
+
 int tscSendMsgToServer(SSqlObj *pSql);
 
 void tscTableMetaCallBack(void *param, TAOS_RES *res, int code) {

src/client/src/tscSQLParser.c

@@ -2223,7 +2223,7 @@ int32_t addExprAndResultField(SSqlCmd* pCmd, SQueryInfo* pQueryInfo, int32_t col
             bool multiColOutput = taosArrayGetSize(pItem->pNode->pParam) > 1;
             setResultColName(name, pItem, cvtFunc.originFuncId, &pParamElem->pNode->colInfo, multiColOutput);
 
-            if (setExprInfoForFunctions(pCmd, pQueryInfo, pSchema, cvtFunc, name, colIndex + i, &index, finalResult) != 0) {
+            if (setExprInfoForFunctions(pCmd, pQueryInfo, pSchema, cvtFunc, name, colIndex++, &index, finalResult) != 0) {
               return TSDB_CODE_TSC_INVALID_SQL;
             }
           }
@@ -3092,18 +3092,26 @@ static int32_t doExtractColumnFilterInfo(SSqlCmd* pCmd, SQueryInfo* pQueryInfo,
   }
 
   int32_t retVal = TSDB_CODE_SUCCESS;
+
+  int32_t bufLen = 0;
+  if (IS_NUMERIC_TYPE(pRight->value.nType)) {
+    bufLen = 60;
+  } else {
+    bufLen = pRight->value.nLen + 1;
+  }
+
   if (pExpr->tokenId == TK_LE || pExpr->tokenId == TK_LT) {
     retVal = tVariantDump(&pRight->value, (char*)&pColumnFilter->upperBndd, colType, false);
 
   // TK_GT,TK_GE,TK_EQ,TK_NE are based on the pColumn->lowerBndd
   } else if (colType == TSDB_DATA_TYPE_BINARY) {
-    pColumnFilter->pz = (int64_t)calloc(1, pRight->value.nLen + TSDB_NCHAR_SIZE);
+    pColumnFilter->pz = (int64_t)calloc(1, bufLen * TSDB_NCHAR_SIZE);
     pColumnFilter->len = pRight->value.nLen;
     retVal = tVariantDump(&pRight->value, (char*)pColumnFilter->pz, colType, false);
 
   } else if (colType == TSDB_DATA_TYPE_NCHAR) {
     // pRight->value.nLen + 1 is larger than the actual nchar string length
-    pColumnFilter->pz = (int64_t)calloc(1, (pRight->value.nLen + 1) * TSDB_NCHAR_SIZE);
+    pColumnFilter->pz = (int64_t)calloc(1, bufLen * TSDB_NCHAR_SIZE);
     retVal = tVariantDump(&pRight->value, (char*)pColumnFilter->pz, colType, false);
     size_t len = twcslen((wchar_t*)pColumnFilter->pz);
     pColumnFilter->len = len * TSDB_NCHAR_SIZE;
@@ -4138,13 +4146,21 @@ static int32_t validateTagCondExpr(SSqlCmd* pCmd, tExprNode *p) {
     }
     
     int32_t retVal = TSDB_CODE_SUCCESS;
+
+    int32_t bufLen = 0;
+    if (IS_NUMERIC_TYPE(vVariant->nType)) {
+      bufLen = 60;  // The maximum length of string that a number is converted to.
+    } else {
+      bufLen = vVariant->nLen + 1;
+    }
+
     if (schemaType == TSDB_DATA_TYPE_BINARY) {
-      char *tmp = calloc(1, vVariant->nLen + TSDB_NCHAR_SIZE);
+      char *tmp = calloc(1, bufLen * TSDB_NCHAR_SIZE);
       retVal = tVariantDump(vVariant, tmp, schemaType, false);
       free(tmp);
     } else if (schemaType == TSDB_DATA_TYPE_NCHAR) {
       // pRight->value.nLen + 1 is larger than the actual nchar string length
-      char *tmp = calloc(1, (vVariant->nLen + 1) * TSDB_NCHAR_SIZE);
+      char *tmp = calloc(1, bufLen * TSDB_NCHAR_SIZE);
       retVal = tVariantDump(vVariant, tmp, schemaType, false);
       free(tmp);
     } else {
@@ -4155,7 +4171,7 @@ static int32_t validateTagCondExpr(SSqlCmd* pCmd, tExprNode *p) {
     if (retVal != TSDB_CODE_SUCCESS) {
       return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg2);
     }
-  }while (0);
+  } while (0);
 
   return TSDB_CODE_SUCCESS;
 }

src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulResultSet.java

@@ -136,7 +136,8 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
             throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
 
         columnIndex = getTrueColumnIndex(columnIndex);
-        return resultSet.get(pos).get(columnIndex).toString();
+        Object value = resultSet.get(pos).get(columnIndex);
+        return value == null ? null : value.toString();
     }
 
     @Override

src/inc/tp.h

@@ -22,7 +22,7 @@ extern "C" {
 
 int32_t tpInit();
 void    tpCleanUp();
-void    tpUpdateTs(int32_t *seq, void *pMsg);
+void    tpUpdateTs(int32_t vgId, int64_t *seq, void *pMsg);
 
 #ifdef __cplusplus
 }

src/inc/tsdb.h

@@ -25,6 +25,8 @@
 #include "tdataformat.h"
 #include "tname.h"
 #include "hash.h"
+#include "tlockfree.h"
+#include "tlist.h"
 
 #ifdef __cplusplus
 extern "C" {
@@ -172,10 +174,32 @@ typedef struct STsdbQueryCond {
   int32_t      type;              // data block load type:
 } STsdbQueryCond;
 
+typedef struct STableData STableData;
+typedef struct {
+  T_REF_DECLARE()
+  SRWLatch     latch;
+  TSKEY        keyFirst;
+  TSKEY        keyLast;
+  int64_t      numOfRows;
+  int32_t      maxTables;
+  STableData **tData;
+  SList *      actList;
+  SList *      extraBuffList;
+  SList *      bufBlockList;
+  int64_t      pointsAdd;   // TODO
+  int64_t      storageAdd;  // TODO
+} SMemTable;
+
+typedef struct {
+  SMemTable* mem;
+  SMemTable* imem;
+  SMemTable  mtable;
+  SMemTable* omem;
+} SMemSnapshot;
+
 typedef struct SMemRef {
   int32_t      ref;
-  void *  mem;
-  void *  imem;
+  SMemSnapshot snapshot;
 } SMemRef;
 
 typedef struct SDataBlockInfo {

src/inc/ttokendef.h

@@ -197,13 +197,11 @@
 #define TK_PREV                           178
 #define TK_LINEAR                         179
 #define TK_IMPORT                         180
-#define TK_METRIC                         181
-#define TK_TBNAME                         182
-#define TK_JOIN                           183
-#define TK_METRICS                        184
-#define TK_INSERT                         185
-#define TK_INTO                           186
-#define TK_VALUES                         187
+#define TK_TBNAME                         181
+#define TK_JOIN                           182
+#define TK_INSERT                         183
+#define TK_INTO                           184
+#define TK_VALUES                         185
 
 
 #define TK_SPACE                          300

src/kit/taosdemo/insert-interlace.json

+{
+    "filetype": "insert",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "thread_count": 4,
+    "thread_count_create_tbl": 4,
+    "result_file": "./insert_res.txt",
+    "confirm_parameter_prompt": "no",
+    "insert_interval": 1000,
+    "num_of_records_per_req": 100,
+    "max_sql_len": 1024000,
+    "databases": [{
+        "dbinfo": {
+            "name": "db",
+            "drop": "yes",
+            "replica": 1,
+            "days": 10,
+            "cache": 16,
+            "blocks": 8,
+            "precision": "ms",
+            "keep": 365,
+            "minRows": 100,
+            "maxRows": 4096,
+            "comp":2,
+            "walLevel":1,
+            "cachelast":0,
+            "quorum":1,
+            "fsync":3000,
+            "update": 0
+        },
+        "super_tables": [{
+            "name": "stb",
+            "child_table_exists":"no",
+            "childtable_count": 100,
+            "childtable_prefix": "stb_",
+            "auto_create_table": "no",
+            "data_source": "rand",
+            "insert_mode": "taosc",
+            "insert_rows": 1000,
+            "multi_thread_write_one_tbl": "no",
+            "rows_per_tbl": 20,
+            "max_sql_len": 1024000,
+            "disorder_ratio": 0,
+            "disorder_range": 1000,
+            "timestamp_step": 1,
+            "start_timestamp": "2020-10-01 00:00:00.000",
+            "sample_format": "csv",
+            "sample_file": "./sample.csv",
+            "tags_file": "",
+            "columns": [{"type": "INT"}],
+            "tags": [{"type": "TINYINT", "count":1}]
+        }]
+    }]
+}

src/mnode/inc/mnodeVgroup.h

@@ -44,7 +44,7 @@ void    mnodeDropVgroup(SVgObj *pVgroup, void *ahandle);
 void    mnodeAlterVgroup(SVgObj *pVgroup, void *ahandle);
 int32_t mnodeGetAvailableVgroup(struct SMnodeMsg *pMsg, SVgObj **pVgroup, int32_t *sid);
 
-void    mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable);
+int32_t mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable, bool needCheck);
 void    mnodeRemoveTableFromVgroup(SVgObj *pVgroup, SCTableObj *pTable);
 void    mnodeSendDropVnodeMsg(int32_t vgId, SRpcEpSet *epSet, void *ahandle);
 void    mnodeSendCreateVgroupMsg(SVgObj *pVgroup, void *ahandle);

src/mnode/src/mnodeDb.c

@@ -55,7 +55,7 @@ int32_t mnodeProcessAlterDbMsg(SMnodeMsg *pMsg);
 #ifndef _TOPIC
 int32_t tpInit() { return 0; }
 void    tpCleanUp() {}
-void    tpUpdateTs(int32_t *seq, void *pMsg) {}
+void    tpUpdateTs(int32_t vgId, int64_t *seq, void *pMsg) {}
 #endif
 
 static void mnodeDestroyDb(SDbObj *pDb) {

src/mnode/src/mnodeSdb.c

@@ -552,7 +552,7 @@ static int32_t sdbInsertHash(SSdbTable *pTable, SSdbRow *pRow) {
 
   int32_t code = (*pTable->fpInsert)(pRow);
   if (code != TSDB_CODE_SUCCESS) {
-    sdbError("vgId:1, sdb:%s, failed to insert key:%s to hash, remove it", pTable->name,
+    sdbError("vgId:1, sdb:%s, failed to perform insert action for key:%s, remove it", pTable->name,
              sdbGetRowStr(pTable, pRow->pObj));
     sdbDeleteHash(pTable, pRow);
   }

src/mnode/src/mnodeTable.c

@@ -108,10 +108,12 @@ static int32_t mnodeChildTableActionDestroy(SSdbRow *pRow) {
 
 static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
   SCTableObj *pTable = pRow->pObj;
+  int32_t code = 0;
 
   SVgObj *pVgroup = mnodeGetVgroup(pTable->vgId);
   if (pVgroup == NULL) {
     mError("ctable:%s, not in vgId:%d", pTable->info.tableId, pTable->vgId);
+    code = -1;
   }
 
   SDbObj *pDb = NULL;
@@ -119,6 +121,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
     pDb = mnodeGetDb(pVgroup->dbName);
     if (pDb == NULL) {
       mError("ctable:%s, vgId:%d not in db:%s", pTable->info.tableId, pVgroup->vgId, pVgroup->dbName);
+      code = -1;
     }
   }
 
@@ -127,6 +130,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
     pAcct = mnodeGetAcct(pDb->acct);
     if (pAcct == NULL) {
       mError("ctable:%s, acct:%s not exists", pTable->info.tableId, pDb->acct);
+      code = -1;
     }
   }
 
@@ -139,6 +143,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
       if (pAcct) pAcct->acctInfo.numOfTimeSeries += (pTable->superTable->numOfColumns - 1);
     } else {
       mError("table:%s:%p, correspond stable not found suid:%" PRIu64, pTable->info.tableId, pTable, pTable->suid);
+      code = -1;
     }
   } else {
     grantAdd(TSDB_GRANT_TIMESERIES, pTable->numOfColumns - 1);
@@ -146,18 +151,31 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
   }
 
   if (pDb) mnodeAddTableIntoDb(pDb);
-  if (pVgroup) mnodeAddTableIntoVgroup(pVgroup, pTable);
+  if (pVgroup) {
+    if (mnodeAddTableIntoVgroup(pVgroup, pTable, pRow->pMsg == NULL) != 0) {
+      mError("table:%s, vgId:%d tid:%d, failed to perform insert action, uid:%" PRIu64 " suid:%" PRIu64,
+             pTable->info.tableId, pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
+      code = -1;
+    }
+  }
 
   mnodeDecVgroupRef(pVgroup);
   mnodeDecDbRef(pDb);
   mnodeDecAcctRef(pAcct);
 
-  return TSDB_CODE_SUCCESS;
+  if (code == 0) {
+    mTrace("table:%s, vgId:%d tid:%d, perform insert action, uid:%" PRIu64 " suid:%" PRIu64, pTable->info.tableId,
+           pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
+  }
+
+  return code;
 }
 
 static int32_t mnodeChildTableActionDelete(SSdbRow *pRow) {
   SCTableObj *pTable = pRow->pObj;
   if (pTable->vgId == 0) {
+    mError("table:%s, vgId:%d tid:%d, failed to perform delete action, uid:%" PRIu64 " suid:%" PRIu64,
+           pTable->info.tableId, pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
     return TSDB_CODE_MND_VGROUP_NOT_EXIST;
   }
 
@@ -188,6 +206,8 @@ static int32_t mnodeChildTableActionDelete(SSdbRow *pRow) {
   mnodeDecDbRef(pDb);
   mnodeDecAcctRef(pAcct);
 
+  mTrace("table:%s, vgId:%d tid:%d, perform delete action, uid:%" PRIu64 " suid:%" PRIu64, pTable->info.tableId,
+         pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -399,13 +419,13 @@ static void mnodeAddTableIntoStable(SSTableObj *pStable, SCTableObj *pCtable) {
 
   if (pStable->vgHash == NULL) {
     pStable->vgHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_ENTRY_LOCK);
-    mDebug("table:%s, create hash:%p", pStable->info.tableId, pStable->vgHash);
+    mDebug("stable:%s, create vgId hash:%p", pStable->info.tableId, pStable->vgHash);
   }
 
   if (pStable->vgHash != NULL) {
     if (taosHashGet(pStable->vgHash, &pCtable->vgId, sizeof(pCtable->vgId)) == NULL) {
       taosHashPut(pStable->vgHash, &pCtable->vgId, sizeof(pCtable->vgId), &pCtable->vgId, sizeof(pCtable->vgId));
-      mDebug("table:%s, vgId:%d is put into stable hash:%p, sizeOfVgList:%d", pStable->info.tableId, pCtable->vgId,
+      mDebug("stable:%s, vgId:%d is put into stable vgId hash:%p, sizeOfVgList:%d", pStable->info.tableId, pCtable->vgId,
              pStable->vgHash, taosHashGetSize(pStable->vgHash));
     }
   }
@@ -443,19 +463,21 @@ static int32_t mnodeSuperTableActionDestroy(SSdbRow *pRow) {
 
 static int32_t mnodeSuperTableActionInsert(SSdbRow *pRow) {
   SSTableObj *pStable = pRow->pObj;
-  SDbObj *pDb = mnodeGetDbByTableName(pStable->info.tableId);
+  SDbObj *    pDb = mnodeGetDbByTableName(pStable->info.tableId);
   if (pDb != NULL && pDb->status == TSDB_DB_STATUS_READY) {
     mnodeAddSuperTableIntoDb(pDb);
   }
   mnodeDecDbRef(pDb);
 
   taosHashPut(tsSTableUidHash, &pStable->uid, sizeof(int64_t), &pStable, sizeof(int64_t));
+
+  mTrace("stable:%s, perform insert action, uid:%" PRIu64, pStable->info.tableId, pStable->uid);
   return TSDB_CODE_SUCCESS;
 }
 
 static int32_t mnodeSuperTableActionDelete(SSdbRow *pRow) {
   SSTableObj *pStable = pRow->pObj;
-  SDbObj *pDb = mnodeGetDbByTableName(pStable->info.tableId);
+  SDbObj *    pDb = mnodeGetDbByTableName(pStable->info.tableId);
   if (pDb != NULL) {
     mnodeRemoveSuperTableFromDb(pDb);
     mnodeDropAllChildTablesInStable((SSTableObj *)pStable);
@@ -463,6 +485,8 @@ static int32_t mnodeSuperTableActionDelete(SSdbRow *pRow) {
   mnodeDecDbRef(pDb);
 
   taosHashRemove(tsSTableUidHash, &pStable->uid, sizeof(int64_t));
+
+  mTrace("stable:%s, perform delete action, uid:%" PRIu64, pStable->info.tableId, pStable->uid);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -1889,8 +1913,7 @@ static int32_t mnodeDoCreateChildTableCb(SMnodeMsg *pMsg, int32_t code) {
 
       if (pMsg->pBatchMasterMsg) {
         ++pMsg->pBatchMasterMsg->successed;
-	if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
-	    >= pMsg->pBatchMasterMsg->expected) {
+        if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
           dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, code);
         }
 
@@ -1911,8 +1934,7 @@ static int32_t mnodeDoCreateChildTableCb(SMnodeMsg *pMsg, int32_t code) {
     if (pMsg->pBatchMasterMsg) {
       ++pMsg->pBatchMasterMsg->received;
       pMsg->pBatchMasterMsg->code = code;
-      if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
-	  >= pMsg->pBatchMasterMsg->expected) {
+      if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
         dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, code);
       }
 
@@ -2662,8 +2684,7 @@ static void mnodeProcessCreateChildTableRsp(SRpcMsg *rpcMsg) {
 
     if (pMsg->pBatchMasterMsg) {
       ++pMsg->pBatchMasterMsg->successed;
-      if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
-	  >= pMsg->pBatchMasterMsg->expected) {
+      if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
         dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, rpcMsg->code);
       }
 

src/mnode/src/mnodeVgroup.c

@@ -443,6 +443,7 @@ int32_t mnodeGetAvailableVgroup(SMnodeMsg *pMsg, SVgObj **ppVgroup, int32_t *pSi
       mDebug("msg:%p, app:%p db:%s, no enough sid in vgId:%d", pMsg, pMsg->rpcMsg.ahandle, pDb->name, pVgroup->vgId);
       continue;
     }
+    mTrace("vgId:%d, alloc tid:%d", pVgroup->vgId, sid);
 
     *pSid = sid;
     *ppVgroup = pVgroup;
@@ -507,6 +508,7 @@ int32_t mnodeGetAvailableVgroup(SMnodeMsg *pMsg, SVgObj **ppVgroup, int32_t *pSi
   pDb->vgListIndex = 0;
   pthread_mutex_unlock(&pDb->mutex);
 
+  mTrace("vgId:%d, alloc tid:%d", pVgroup->vgId, sid);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -832,26 +834,37 @@ static int32_t mnodeRetrieveVgroups(SShowObj *pShow, char *data, int32_t rows, v
   return numOfRows;
 }
 
-void mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable) {
+int32_t mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable, bool needCheck) {
   int32_t idPoolSize = taosIdPoolMaxSize(pVgroup->idPool);
   if (pTable->tid > idPoolSize) {
     mnodeAllocVgroupIdPool(pVgroup);
   }
 
   if (pTable->tid >= 1) {
-    taosIdPoolMarkStatus(pVgroup->idPool, pTable->tid);
+    if (taosIdPoolMarkStatus(pVgroup->idPool, pTable->tid) || !needCheck) {
       pVgroup->numOfTables++;
+      mTrace("table:%s, vgId:%d tid:%d, mark tid used, uid:%" PRIu64, pTable->info.tableId, pTable->vgId, pTable->tid,
+             pTable->uid);
       // The create vgroup message may be received later than the create table message
       // and the writing order in sdb is therefore uncertain
       // which will cause the reference count of the vgroup to be incorrect when restarting
       // mnodeIncVgroupRef(pVgroup);
+    } else {
+      mError("table:%s, vgId:%d tid:%d, failed to mark tid, uid:%" PRIu64, pTable->info.tableId, pTable->vgId,
+             pTable->tid, pTable->uid);
+      return -1;
+    }
   }
+
+  return 0;
 }
 
 void mnodeRemoveTableFromVgroup(SVgObj *pVgroup, SCTableObj *pTable) {
   if (pTable->tid >= 1) {
     taosFreeId(pVgroup->idPool, pTable->tid);
     pVgroup->numOfTables--;
+    mTrace("table:%s, vgId:%d tid:%d, put tid back uid:%" PRIu64, pTable->info.tableId, pTable->vgId, pTable->tid,
+           pTable->uid);
     // The create vgroup message may be received later than the create table message
     // and the writing order in sdb is therefore uncertain
     // which will cause the reference count of the vgroup to be incorrect when restarting

src/query/src/qExecutor.c

@@ -1840,7 +1840,7 @@ static void doFreeQueryHandle(SQueryRuntimeEnv* pRuntimeEnv) {
   pRuntimeEnv->pQueryHandle = NULL;
 
   SMemRef* pMemRef = &pQuery->memRef;
-  assert(pMemRef->ref == 0 && pMemRef->imem == NULL && pMemRef->mem == NULL);
+  assert(pMemRef->ref == 0 && pMemRef->snapshot.imem == NULL && pMemRef->snapshot.mem == NULL);
 }
 
 static void teardownQueryRuntimeEnv(SQueryRuntimeEnv *pRuntimeEnv) {

src/query/src/qTokenizer.c

@@ -206,10 +206,8 @@ static SKeyword keywordTable[] = {
     {"PREV",         TK_PREV},
     {"LINEAR",       TK_LINEAR},
     {"IMPORT",       TK_IMPORT},
-    {"METRIC",       TK_METRIC},
     {"TBNAME",       TK_TBNAME},
     {"JOIN",         TK_JOIN},
-    {"METRICS",      TK_METRICS},
     {"STABLE",       TK_STABLE},
     {"FILE",         TK_FILE},
     {"VNODES",       TK_VNODES},

src/query/src/queryMain.c

@@ -430,7 +430,7 @@ void qQueryMgmtNotifyClosed(void* pQMgmt) {
   }
 
   SQueryMgmt* pQueryMgmt = pQMgmt;
-  qDebug("vgId:%d, set querymgmt closed, wait for all queries cancelled", pQueryMgmt->vgId);
+  qInfo("vgId:%d, set querymgmt closed, wait for all queries cancelled", pQueryMgmt->vgId);
 
   pthread_mutex_lock(&pQueryMgmt->lock);
   pQueryMgmt->closed = true;
@@ -445,7 +445,7 @@ void qQueryMgmtReOpen(void *pQMgmt) {
   }
 
   SQueryMgmt *pQueryMgmt = pQMgmt;
-  qDebug("vgId:%d, set querymgmt reopen", pQueryMgmt->vgId);
+  qInfo("vgId:%d, set querymgmt reopen", pQueryMgmt->vgId);
 
   pthread_mutex_lock(&pQueryMgmt->lock);
   pQueryMgmt->closed = false;

src/tsdb/inc/tsdbMemTable.h

@@ -31,29 +31,14 @@ typedef struct {
   SSkipListIterator *pIter;
 } SCommitIter;
 
-typedef struct {
+struct STableData {
   uint64_t   uid;
   TSKEY      keyFirst;
   TSKEY      keyLast;
   int64_t    numOfRows;
   SSkipList* pData;
   T_REF_DECLARE()
-} STableData;
-
-typedef struct {
-  T_REF_DECLARE()
-  SRWLatch     latch;
-  TSKEY        keyFirst;
-  TSKEY        keyLast;
-  int64_t      numOfRows;
-  int32_t      maxTables;
-  STableData** tData;
-  SList*       actList;
-  SList*       extraBuffList;
-  SList*       bufBlockList;
-  int64_t      pointsAdd;   // TODO
-  int64_t      storageAdd;  // TODO
-} SMemTable;
+};
 
 enum { TSDB_UPDATE_META, TSDB_DROP_META };
 
@@ -77,8 +62,8 @@ typedef struct {
 
 int   tsdbRefMemTable(STsdbRepo* pRepo, SMemTable* pMemTable);
 int   tsdbUnRefMemTable(STsdbRepo* pRepo, SMemTable* pMemTable);
-int   tsdbTakeMemSnapshot(STsdbRepo* pRepo, SMemTable** pMem, SMemTable** pIMem, SArray* pATable);
-void  tsdbUnTakeMemSnapShot(STsdbRepo* pRepo, SMemTable* pMem, SMemTable* pIMem);
+int   tsdbTakeMemSnapshot(STsdbRepo* pRepo, SMemSnapshot* pSnapshot, SArray* pATable);
+void  tsdbUnTakeMemSnapShot(STsdbRepo* pRepo, SMemSnapshot* pSnapshot);
 void* tsdbAllocBytes(STsdbRepo* pRepo, int bytes);
 int   tsdbAsyncCommit(STsdbRepo* pRepo);
 int   tsdbLoadDataFromCache(STable* pTable, SSkipListIterator* pIter, TSKEY maxKey, int maxRowsToRead, SDataCols* pCols,

src/tsdb/src/tsdbMeta.c

@@ -213,9 +213,9 @@ void *tsdbGetTableTagVal(const void* pTable, int32_t colId, int16_t type, int16_
   char *val = tdGetKVRowValOfCol(((STable*)pTable)->tagVal, colId);
   assert(type == pCol->type && bytes == pCol->bytes);
 
-  if (val != NULL && IS_VAR_DATA_TYPE(type)) {
-    assert(varDataLen(val) < pCol->bytes);
-  }
+  // if (val != NULL && IS_VAR_DATA_TYPE(type)) {
+  //   assert(varDataLen(val) < pCol->bytes);
+  // }
 
   return val;
 }

src/util/inc/tidpool.h

@@ -34,7 +34,7 @@ void taosIdPoolCleanUp(void *handle);
 
 int taosIdPoolNumOfUsed(void *handle);
 
-void taosIdPoolMarkStatus(void *handle, int id);
+bool taosIdPoolMarkStatus(void *handle, int id);
 
 #ifdef __cplusplus
 }

src/util/inc/tsocket.h

@@ -28,7 +28,7 @@ int32_t taosReadn(SOCKET sock, char *buffer, int32_t len);
 int32_t taosWriteMsg(SOCKET fd, void *ptr, int32_t nbytes);
 int32_t taosReadMsg(SOCKET fd, void *ptr, int32_t nbytes);
 int32_t taosNonblockwrite(SOCKET fd, char *ptr, int32_t nbytes);
-int32_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len);
+int64_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len);
 int32_t taosSetNonblocking(SOCKET sock, int32_t on);
 
 SOCKET  taosOpenUdpSocket(uint32_t localIp, uint16_t localPort);

tests/pytest/tools/insert-tblimit-tboffset0.json

@@ -44,7 +44,6 @@
             "childtable_offset": 0,
             "multi_thread_write_one_tbl": "no",
             "number_of_tbl_in_one_sql": 0,
-            "rows_per_tbl": 100,
             "max_sql_len": 1024000,
             "disorder_ratio": 0,
             "disorder_range": 1000,