diff --git a/documentation20/cn/02.getting-started/docs.md b/documentation20/cn/02.getting-started/docs.md
index a98159d8c408c7140b1a24f2534e575b38d1c56a..6eb58a1433ed0d43b313a9dc979ae5873ba00e8f 100644
--- a/documentation20/cn/02.getting-started/docs.md
+++ b/documentation20/cn/02.getting-started/docs.md
@@ -24,7 +24,7 @@ TDengine的安装非常简单，从下载到安装成功仅仅只要几秒钟。
## 轻松启动
-安装成功后，用户可使用`systemctl`命令来启动TDengine的服务进程。
+安装成功后，用户可使用 `systemctl` 命令来启动 TDengine 的服务进程。
```bash
$ systemctl start taosd
@@ -35,21 +35,22 @@ $ systemctl start taosd
$ systemctl status taosd
```
-如果TDengine服务正常工作，那么您可以通过TDengine的命令行程序`taos`来访问并体验TDengine。
+如果 TDengine 服务正常工作，那么您可以通过 TDengine 的命令行程序 `taos` 来访问并体验 TDengine。
**注意：**
-- systemctl命令需要 _root_ 权限来运行，如果您非 _root_ 用户，请在命令前添加 sudo
-- 为更好的获得产品反馈，改善产品，TDengine会采集基本的使用信息，但您可以修改系统配置文件taos.cfg里的配置参数telemetryReporting, 将其设为0，就可将其关闭。
-- TDengine采用FQDN(一般就是hostname)作为节点的ID，为保证正常运行，需要给运行taosd的服务器配置好hostname，在客户端应用运行的机器配置好DNS服务或hosts文件，保证FQDN能够解析。
+- systemctl 命令需要 _root_ 权限来运行，如果您非 _root_ 用户，请在命令前添加 sudo 。
+- 为更好的获得产品反馈，改善产品，TDengine 会采集基本的使用信息，但您可以修改系统配置文件 taos.cfg 里的配置参数 telemetryReporting, 将其设为 0，就可将其关闭。
+- TDengine 采用 FQDN (一般就是 hostname )作为节点的 ID，为保证正常运行，需要给运行 taosd 的服务器配置好 hostname，在客户端应用运行的机器配置好 DNS 服务或 hosts 文件，保证 FQDN 能够解析。
+- `systemctl stop taosd` 指令在执行后并不会马上停止 TDengine 服务，而是会等待系统中必要的落盘工作正常完成。在数据量很大的情况下，这可能会消耗较长时间。
-* TDengine 支持在使用[`systemd`](https://en.wikipedia.org/wiki/Systemd)做进程服务管理的linux系统上安装，用`which systemctl`命令来检测系统中是否存在`systemd`包:
+* TDengine 支持在使用 [`systemd`](https://en.wikipedia.org/wiki/Systemd) 做进程服务管理的 linux 系统上安装，用 `which systemctl` 命令来检测系统中是否存在 `systemd` 包:
```bash
$ which systemctl
```
- 如果系统中不支持systemd，也可以用手动运行 /usr/local/taos/bin/taosd 方式启动 TDengine 服务。
+ 如果系统中不支持 systemd，也可以用手动运行 /usr/local/taos/bin/taosd 方式启动 TDengine 服务。
## TDengine命令行程序
diff --git a/documentation20/cn/11.administrator/docs.md b/documentation20/cn/11.administrator/docs.md
index bfa0456c7d4e80d1fd9336d7c4b7b9ca829b278e..ee2f34ff11ef5e69195c01d0b605383c3257f48c 100644
--- a/documentation20/cn/11.administrator/docs.md
+++ b/documentation20/cn/11.administrator/docs.md
@@ -129,7 +129,7 @@ taosd -C
- blocks：每个VNODE（TSDB）中有多少cache大小的内存块。因此一个VNODE的用的内存大小粗略为（cache * blocks）。单位为块，默认值：4。（可通过 alter database 修改）
- replica：副本个数，取值范围：1-3。单位为个，默认值：1。（可通过 alter database 修改）
- precision：时间戳精度标识，ms表示毫秒，us表示微秒。默认值：ms。
-- cacheLast：是否在内存中缓存子表 last_row，0：关闭；1：开启。默认值：0。（可通过 alter database 修改）（从 2.0.11 版本开始支持此参数）
+- cacheLast：是否在内存中缓存子表的最近数据，0：关闭；1：缓存子表最近一行数据；2：缓存子表每一列的最近的非NULL值，3：同时打开缓存最近行和列功能，默认值：0。（可通过 alter database 修改）（从 2.0.11 版本开始支持此参数）
对于一个应用场景，可能有多种数据特征的数据并存，最佳的设计是将具有相同数据特征的表放在一个库里，这样一个应用有多个库，而每个库可以配置不同的存储参数，从而保证系统有最优的性能。TDengine允许应用在创建库时指定上述存储参数，如果指定，该参数就将覆盖对应的系统配置参数。举例，有下述SQL：
diff --git a/documentation20/cn/12.taos-sql/01.error-code/docs.md b/documentation20/cn/12.taos-sql/01.error-code/docs.md
index 0a0d694ee3da417ec0162cd9e4f8ddda9ab044f1..867aa18715f87a1dfc9ea36203d32382bb726e30 100644
--- a/documentation20/cn/12.taos-sql/01.error-code/docs.md
+++ b/documentation20/cn/12.taos-sql/01.error-code/docs.md
@@ -26,7 +26,7 @@
| TSDB_CODE_COM_OUT_OF_MEMORY | 0 | 0x0102 | "Out of memory" | -2147483390 |
| TSDB_CODE_COM_INVALID_CFG_MSG | 0 | 0x0103 | "Invalid config message" | -2147483389 |
| TSDB_CODE_COM_FILE_CORRUPTED | 0 | 0x0104 | "Data file corrupted" | -2147483388 |
-| TSDB_CODE_TSC_INVALID_SQL | 0 | 0x0200 | "Invalid SQL statement" | -2147483136 |
+| TSDB_CODE_TSC_INVALID_OPERATION | 0 | 0x0200 | "Invalid SQL statement" | -2147483136 |
| TSDB_CODE_TSC_INVALID_QHANDLE | 0 | 0x0201 | "Invalid qhandle" | -2147483135 |
| TSDB_CODE_TSC_INVALID_TIME_STAMP | 0 | 0x0202 | "Invalid combination of client/service time" | -2147483134 |
| TSDB_CODE_TSC_INVALID_VALUE | 0 | 0x0203 | "Invalid value in client" | -2147483133 |
diff --git a/documentation20/cn/12.taos-sql/docs.md b/documentation20/cn/12.taos-sql/docs.md
index bcf80d8fa20e613ff8955afa13036687da9d8a59..af78074eb25391d605bfca9067d24e1626296891 100644
--- a/documentation20/cn/12.taos-sql/docs.md
+++ b/documentation20/cn/12.taos-sql/docs.md
@@ -76,7 +76,7 @@ TDengine 缺省的时间戳是毫秒精度，但通过修改配置参数 enableM
4) 一条SQL 语句的最大长度为65480个字符；
- 5) 数据库还有更多与存储相关的配置参数，请参见 [服务端配置](https://www.taosdata.com/cn/documentation/taos-sql#management) 章节。
+ 5) 数据库还有更多与存储相关的配置参数，请参见 [服务端配置](https://www.taosdata.com/cn/documentation/administrator#config) 章节。
- **显示系统当前参数**
@@ -126,7 +126,7 @@ TDengine 缺省的时间戳是毫秒精度，但通过修改配置参数 enableM
```mysql
ALTER DATABASE db_name CACHELAST 0;
```
- CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11 版本开始支持，修改后需要重启服务器生效。）
+ CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11.0 版本开始支持。从 2.1.1.0 版本开始，修改此参数后无需重启服务器即可生效。）
**Tips**: 以上所有参数修改后都可以用show databases来确认是否修改成功。
@@ -399,7 +399,12 @@ TDengine 缺省的时间戳是毫秒精度，但通过修改配置参数 enableM
INSERT INTO tb1_name (tb1_field1_name, ...) [USING stb1_name TAGS (tag_value1, ...)] VALUES (field1_value1, ...) (field1_value2, ...) ...
tb2_name (tb2_field1_name, ...) [USING stb2_name TAGS (tag_value2, ...)] VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
- 以自动建表的方式，同时向表tb1_name和tb2_name中按列分别插入多条记录。
+ 以自动建表的方式，同时向表tb1_name和tb2_name中按列分别插入多条记录。
+ 从 2.0.20.5 版本开始，子表的列名可以不跟在子表名称后面，而是可以放在 TAGS 和 VALUES 之间，例如像下面这样写：
+ ```mysql
+ INSERT INTO tb1_name [USING stb1_name TAGS (tag_value1, ...)] (tb1_field1_name, ...) VALUES (field1_value1, ...) (field1_value2, ...) ...;
+ ```
+ 注意：虽然两种写法都可以，但并不能在一条 SQL 语句中混用，否则会报语法错误。
**历史记录写入**：可使用IMPORT或者INSERT命令，IMPORT的语法，功能与INSERT完全一样。
diff --git a/src/client/inc/tscUtil.h b/src/client/inc/tscUtil.h
index 3459bf0b9a922f05bfb74b872d72540be972b6ae..4c36b872c12453d66957d9b5ccf4f8257f530672 100644
--- a/src/client/inc/tscUtil.h
+++ b/src/client/inc/tscUtil.h
@@ -94,6 +94,8 @@ typedef struct SVgroupTableInfo {
SArray *itemList; // SArray
} SVgroupTableInfo;
+int32_t converToStr(char *str, int type, void *buf, int32_t bufSize, int32_t *len);
+
int32_t tscCreateDataBlock(size_t initialSize, int32_t rowSize, int32_t startOffset, SName* name, STableMeta* pTableMeta, STableDataBlocks** dataBlocks);
void tscDestroyDataBlock(STableDataBlocks* pDataBlock, bool removeMeta);
void tscSortRemoveDataBlockDupRows(STableDataBlocks* dataBuf);
diff --git a/src/client/jni/com_taosdata_jdbc_TSDBJNIConnector.h b/src/client/jni/com_taosdata_jdbc_TSDBJNIConnector.h
index 04bccc1a4a9e81c8dd9d70521f9916c304df3a53..d16b672f38fe1f7d9c36990f0eab28253f2d4d1b 100644
--- a/src/client/jni/com_taosdata_jdbc_TSDBJNIConnector.h
+++ b/src/client/jni/com_taosdata_jdbc_TSDBJNIConnector.h
@@ -218,11 +218,19 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_executeBatchImp(J
/*
* Class: com_taosdata_jdbc_TSDBJNIConnector
- * Method: executeBatchImp
+ * Method: closeStmt
* Signature: (JJ)I
*/
JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_closeStmt(JNIEnv *env, jobject jobj, jlong stmt, jlong con);
+/**
+ * Class: com_taosdata_jdbc_TSDBJNIConnector
+ * Method: setTableNameTagsImp
+ * Signature: (JLjava/lang/String;I[B[B[B[BJ)I
+ */
+JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_setTableNameTagsImp
+ (JNIEnv *, jobject, jlong, jstring, jint, jbyteArray, jbyteArray, jbyteArray, jbyteArray, jlong);
+
#ifdef __cplusplus
}
#endif
diff --git a/src/client/src/TSDBJNIConnector.c b/src/client/src/TSDBJNIConnector.c
index da7da17aa3ee29b31acbe2470edf052f1d9cb15b..324c436dce442f78b6daea314506c5c6b465f97f 100644
--- a/src/client/src/TSDBJNIConnector.c
+++ b/src/client/src/TSDBJNIConnector.c
@@ -749,7 +749,6 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_setBindTableNameI
}
jniDebug("jobj:%p, conn:%p, set stmt bind table name:%s", jobj, tsconn, name);
-
(*env)->ReleaseStringUTFChars(env, jname, name);
return JNI_SUCCESS;
}
@@ -762,7 +761,7 @@ JNIEXPORT jlong JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_bindColDataImp(J
return JNI_CONNECTION_NULL;
}
- TAOS_STMT* pStmt = (TAOS_STMT*) stmt;
+ TAOS_STMT *pStmt = (TAOS_STMT *)stmt;
if (pStmt == NULL) {
jniError("jobj:%p, conn:%p, invalid stmt", jobj, tscon);
return JNI_SQL_NULL;
@@ -777,14 +776,14 @@ JNIEXPORT jlong JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_bindColDataImp(J
}
len = (*env)->GetArrayLength(env, lengthList);
- char *lengthArray = (char*) calloc(1, len);
- (*env)->GetByteArrayRegion(env, lengthList, 0, len, (jbyte*) lengthArray);
+ char *lengthArray = (char *)calloc(1, len);
+ (*env)->GetByteArrayRegion(env, lengthList, 0, len, (jbyte *)lengthArray);
if ((*env)->ExceptionCheck(env)) {
}
len = (*env)->GetArrayLength(env, nullList);
- char *nullArray = (char*) calloc(1, len);
- (*env)->GetByteArrayRegion(env, nullList, 0, len, (jbyte*) nullArray);
+ char *nullArray = (char *)calloc(1, len);
+ (*env)->GetByteArrayRegion(env, nullList, 0, len, (jbyte *)nullArray);
if ((*env)->ExceptionCheck(env)) {
}
@@ -799,22 +798,10 @@ JNIEXPORT jlong JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_bindColDataImp(J
b->length = (int32_t*)lengthArray;
// set the length and is_null array
- switch(dataType) {
- case TSDB_DATA_TYPE_INT:
- case TSDB_DATA_TYPE_TINYINT:
- case TSDB_DATA_TYPE_SMALLINT:
- case TSDB_DATA_TYPE_TIMESTAMP:
- case TSDB_DATA_TYPE_BIGINT: {
- int32_t bytes = tDataTypes[dataType].bytes;
- for(int32_t i = 0; i < numOfRows; ++i) {
- b->length[i] = bytes;
- }
- break;
- }
-
- case TSDB_DATA_TYPE_NCHAR:
- case TSDB_DATA_TYPE_BINARY: {
- // do nothing
+ if (!IS_VAR_DATA_TYPE(dataType)) {
+ int32_t bytes = tDataTypes[dataType].bytes;
+ for (int32_t i = 0; i < numOfRows; ++i) {
+ b->length[i] = bytes;
}
}
@@ -878,3 +865,74 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_closeStmt(JNIEnv
jniDebug("jobj:%p, conn:%p, stmt closed", jobj, tscon);
return JNI_SUCCESS;
}
+
+JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_setTableNameTagsImp(JNIEnv *env, jobject jobj,
+ jlong stmt, jstring tableName, jint numOfTags, jbyteArray tags, jbyteArray typeList, jbyteArray lengthList, jbyteArray nullList, jlong conn) {
+ TAOS *tsconn = (TAOS *)conn;
+ if (tsconn == NULL) {
+ jniError("jobj:%p, connection already closed", jobj);
+ return JNI_CONNECTION_NULL;
+ }
+
+ TAOS_STMT* pStmt = (TAOS_STMT*) stmt;
+ if (pStmt == NULL) {
+ jniError("jobj:%p, conn:%p, invalid stmt handle", jobj, tsconn);
+ return JNI_SQL_NULL;
+ }
+
+ jsize len = (*env)->GetArrayLength(env, tags);
+ char *tagsData = (char *)calloc(1, len);
+ (*env)->GetByteArrayRegion(env, tags, 0, len, (jbyte *)tagsData);
+ if ((*env)->ExceptionCheck(env)) {
+ // todo handle error
+ }
+
+ len = (*env)->GetArrayLength(env, lengthList);
+ int64_t *lengthArray = (int64_t*) calloc(1, len);
+ (*env)->GetByteArrayRegion(env, lengthList, 0, len, (jbyte*) lengthArray);
+ if ((*env)->ExceptionCheck(env)) {
+ }
+
+ len = (*env)->GetArrayLength(env, typeList);
+ char *typeArray = (char*) calloc(1, len);
+ (*env)->GetByteArrayRegion(env, typeList, 0, len, (jbyte*) typeArray);
+ if ((*env)->ExceptionCheck(env)) {
+ }
+
+ len = (*env)->GetArrayLength(env, nullList);
+ int32_t *nullArray = (int32_t*) calloc(1, len);
+ (*env)->GetByteArrayRegion(env, nullList, 0, len, (jbyte*) nullArray);
+ if ((*env)->ExceptionCheck(env)) {
+ }
+
+ const char *name = (*env)->GetStringUTFChars(env, tableName, NULL);
+ char* curTags = tagsData;
+
+ TAOS_BIND *tagsBind = calloc(numOfTags, sizeof(TAOS_BIND));
+ for(int32_t i = 0; i < numOfTags; ++i) {
+ tagsBind[i].buffer_type = typeArray[i];
+ tagsBind[i].buffer = curTags;
+ tagsBind[i].is_null = &nullArray[i];
+ tagsBind[i].length = (uintptr_t*) &lengthArray[i];
+
+ curTags += lengthArray[i];
+ }
+
+ int32_t code = taos_stmt_set_tbname_tags((void*)stmt, name, tagsBind);
+
+ int32_t nTags = (int32_t) numOfTags;
+ jniDebug("jobj:%p, conn:%p, set table name:%s, numOfTags:%d", jobj, tsconn, name, nTags);
+
+ tfree(tagsData);
+ tfree(lengthArray);
+ tfree(typeArray);
+ tfree(nullArray);
+ (*env)->ReleaseStringUTFChars(env, tableName, name);
+
+ if (code != TSDB_CODE_SUCCESS) {
+ jniError("jobj:%p, conn:%p, code:%s", jobj, tsconn, tstrerror(code));
+ return JNI_TDENGINE_ERROR;
+ }
+
+ return JNI_SUCCESS;
+}
diff --git a/src/client/src/tscPrepare.c b/src/client/src/tscPrepare.c
index f99257b296ea4261971f829c2573b5b76f3e0cc0..e3ff7669161146a9736f40cb86af9c341cdc37b6 100644
--- a/src/client/src/tscPrepare.c
+++ b/src/client/src/tscPrepare.c
@@ -46,9 +46,13 @@ typedef struct SNormalStmt {
typedef struct SMultiTbStmt {
bool nameSet;
+ bool tagSet;
uint64_t currentUid;
uint32_t tbNum;
SStrToken tbname;
+ SStrToken stbname;
+ SStrToken values;
+ SArray *tags;
SHashObj *pTableHash;
SHashObj *pTableBlockHashList; // data block for each table
} SMultiTbStmt;
@@ -1199,6 +1203,184 @@ static int insertBatchStmtExecute(STscStmt* pStmt) {
return pStmt->pSql->res.code;
}
+
+int stmtParseInsertTbTags(SSqlObj* pSql, STscStmt* pStmt) {
+ SSqlCmd *pCmd = &pSql->cmd;
+ int32_t ret = TSDB_CODE_SUCCESS;
+
+ if ((ret = tsInsertInitialCheck(pSql)) != TSDB_CODE_SUCCESS) {
+ return ret;
+ }
+
+ int32_t index = 0;
+ SStrToken sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n == 0) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ if (sToken.n == 1 && sToken.type == TK_QUESTION) {
+ pStmt->multiTbInsert = true;
+ pStmt->mtb.tbname = sToken;
+ pStmt->mtb.nameSet = false;
+ if (pStmt->mtb.pTableHash == NULL) {
+ pStmt->mtb.pTableHash = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, false);
+ }
+
+ if (pStmt->mtb.pTableBlockHashList == NULL) {
+ pStmt->mtb.pTableBlockHashList = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, false);
+ }
+
+ pStmt->mtb.tagSet = true;
+
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n > 0 && sToken.type == TK_VALUES) {
+ return TSDB_CODE_SUCCESS;
+ }
+
+ if (sToken.n <= 0 || sToken.type != TK_USING) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n <= 0 || ((sToken.type != TK_ID) && (sToken.type != TK_STRING))) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+ pStmt->mtb.stbname = sToken;
+
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n <= 0 || sToken.type != TK_TAGS) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n <= 0 || sToken.type != TK_LP) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ pStmt->mtb.tags = taosArrayInit(4, sizeof(SStrToken));
+
+ int32_t loopCont = 1;
+
+ while (loopCont) {
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n <= 0) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ switch (sToken.type) {
+ case TK_RP:
+ loopCont = 0;
+ break;
+ case TK_VALUES:
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ case TK_QUESTION:
+ pStmt->mtb.tagSet = false; //continue
+ default:
+ taosArrayPush(pStmt->mtb.tags, &sToken);
+ break;
+ }
+ }
+
+ if (taosArrayGetSize(pStmt->mtb.tags) <= 0) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ sToken = tStrGetToken(pCmd->curSql, &index, false);
+ if (sToken.n <= 0 || sToken.type != TK_VALUES) {
+ return TSDB_CODE_TSC_INVALID_OPERATION;
+ }
+
+ pStmt->mtb.values = sToken;
+ }
+
+ return TSDB_CODE_SUCCESS;
+}
+
+
+
+
+int stmtGenInsertStatement(SSqlObj* pSql, STscStmt* pStmt, const char* name, TAOS_BIND* tags) {
+ size_t tagNum = taosArrayGetSize(pStmt->mtb.tags);
+ size_t size = 1048576;
+ char *str = calloc(1, size);
+ size_t len = 0;
+ int32_t ret = 0;
+ int32_t j = 0;
+
+ while (1) {
+ len = (size_t)snprintf(str, size - 1, "insert into %s using %.*s tags(", name, pStmt->mtb.stbname.n, pStmt->mtb.stbname.z);
+ if (len >= (size -1)) {
+ size *= 2;
+ free(str);
+ str = calloc(1, size);
+ continue;
+ }
+
+ j = 0;
+
+ for (size_t i = 0; i < tagNum && len < (size - 1); ++i) {
+ SStrToken *t = taosArrayGet(pStmt->mtb.tags, i);
+ if (t->type == TK_QUESTION) {
+ int32_t l = 0;
+ if (i > 0) {
+ str[len++] = ',';
+ }
+
+ if (tags[j].is_null && (*tags[j].is_null)) {
+ ret = converToStr(str + len, TSDB_DATA_TYPE_NULL, NULL, -1, &l);
+ } else {
+ if (tags[j].buffer == NULL) {
+ free(str);
+ tscError("empty");
+ return TSDB_CODE_TSC_APP_ERROR;
+ }
+
+ ret = converToStr(str + len, tags[j].buffer_type, tags[j].buffer, tags[j].length ? (int32_t)*tags[j].length : -1, &l);
+ }
+
+ ++j;
+
+ if (ret) {
+ free(str);
+ return ret;
+ }
+
+ len += l;
+ } else {
+ len += (size_t)snprintf(str + len, size - len - 1, i > 0 ? ",%.*s" : "%.*s", t->n, t->z);
+ }
+ }
+
+ if (len >= (size - 1)) {
+ size *= 2;
+ free(str);
+ str = calloc(1, size);
+ continue;
+ }
+
+ strcat(str, ") ");
+ len += 2;
+
+ if ((len + strlen(pStmt->mtb.values.z)) >= (size - 1)) {
+ size *= 2;
+ free(str);
+ str = calloc(1, size);
+ continue;
+ }
+
+ strcat(str, pStmt->mtb.values.z);
+
+ break;
+ }
+
+ free(pSql->sqlstr);
+ pSql->sqlstr = str;
+
+ return TSDB_CODE_SUCCESS;
+}
+
+
+
////////////////////////////////////////////////////////////////////////////////
// interface functions
@@ -1291,34 +1473,15 @@ int taos_stmt_prepare(TAOS_STMT* stmt, const char* sql, unsigned long length) {
registerSqlObj(pSql);
- int32_t ret = TSDB_CODE_SUCCESS;
-
- if ((ret = tsInsertInitialCheck(pSql)) != TSDB_CODE_SUCCESS) {
+ int32_t ret = stmtParseInsertTbTags(pSql, pStmt);
+ if (ret != TSDB_CODE_SUCCESS) {
return ret;
}
- int32_t index = 0;
- SStrToken sToken = tStrGetToken(pCmd->insertParam.sql, &index, false);
-
- if (sToken.n == 0) {
- return TSDB_CODE_TSC_INVALID_OPERATION;
- }
-
- if (sToken.n == 1 && sToken.type == TK_QUESTION) {
- pStmt->multiTbInsert = true;
- pStmt->mtb.tbname = sToken;
- pStmt->mtb.nameSet = false;
- if (pStmt->mtb.pTableHash == NULL) {
- pStmt->mtb.pTableHash = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, false);
- }
- if (pStmt->mtb.pTableBlockHashList == NULL) {
- pStmt->mtb.pTableBlockHashList = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, false);
- }
-
+ if (pStmt->multiTbInsert) {
return TSDB_CODE_SUCCESS;
}
- pStmt->multiTbInsert = false;
memset(&pStmt->mtb, 0, sizeof(pStmt->mtb));
int32_t code = tsParseSql(pSql, true);
@@ -1335,7 +1498,7 @@ int taos_stmt_prepare(TAOS_STMT* stmt, const char* sql, unsigned long length) {
return normalStmtPrepare(pStmt);
}
-int taos_stmt_set_tbname(TAOS_STMT* stmt, const char* name) {
+int taos_stmt_set_tbname_tags(TAOS_STMT* stmt, const char* name, TAOS_BIND* tags) {
STscStmt* pStmt = (STscStmt*)stmt;
SSqlObj* pSql = pStmt->pSql;
SSqlCmd* pCmd = &pSql->cmd;
@@ -1383,8 +1546,22 @@ int taos_stmt_set_tbname(TAOS_STMT* stmt, const char* name) {
return TSDB_CODE_SUCCESS;
}
- pStmt->mtb.tbname = tscReplaceStrToken(&pSql->sqlstr, &pStmt->mtb.tbname, name);
+ if (pStmt->mtb.tagSet) {
+ pStmt->mtb.tbname = tscReplaceStrToken(&pSql->sqlstr, &pStmt->mtb.tbname, name);
+ } else {
+ if (tags == NULL) {
+ tscError("No tags set");
+ return TSDB_CODE_TSC_APP_ERROR;
+ }
+
+ int32_t ret = stmtGenInsertStatement(pSql, pStmt, name, tags);
+ if (ret != TSDB_CODE_SUCCESS) {
+ return ret;
+ }
+ }
+
pStmt->mtb.nameSet = true;
+ pStmt->mtb.tagSet = true;
tscDebug("0x%"PRIx64" SQL: %s", pSql->self, pSql->sqlstr);
@@ -1432,6 +1609,12 @@ int taos_stmt_set_tbname(TAOS_STMT* stmt, const char* name) {
return code;
}
+
+int taos_stmt_set_tbname(TAOS_STMT* stmt, const char* name) {
+ return taos_stmt_set_tbname_tags(stmt, name, NULL);
+}
+
+
int taos_stmt_close(TAOS_STMT* stmt) {
STscStmt* pStmt = (STscStmt*)stmt;
if (!pStmt->isInsert) {
@@ -1449,7 +1632,8 @@ int taos_stmt_close(TAOS_STMT* stmt) {
taosHashCleanup(pStmt->mtb.pTableHash);
pStmt->mtb.pTableBlockHashList = tscDestroyBlockHashTable(pStmt->mtb.pTableBlockHashList, true);
taosHashCleanup(pStmt->pSql->cmd.insertParam.pTableBlockHashList);
- pStmt->pSql->cmd.insertParam.pTableNameList = NULL;
+ pStmt->pSql->cmd.insertParam.pTableBlockHashList = NULL;
+ taosArrayDestroy(pStmt->mtb.tags);
}
}
diff --git a/src/client/src/tscUtil.c b/src/client/src/tscUtil.c
index cc8e7fded1df57c966cfbcead4b0ebd06133e2b6..c114bfd9b7887e3f86717f03da6f3ae34b38aad5 100644
--- a/src/client/src/tscUtil.c
+++ b/src/client/src/tscUtil.c
@@ -32,6 +32,67 @@
static void freeQueryInfoImpl(SQueryInfo* pQueryInfo);
+int32_t converToStr(char *str, int type, void *buf, int32_t bufSize, int32_t *len) {
+ int32_t n = 0;
+
+ switch (type) {
+ case TSDB_DATA_TYPE_NULL:
+ n = sprintf(str, "null");
+ break;
+
+ case TSDB_DATA_TYPE_BOOL:
+ n = sprintf(str, (*(int8_t*)buf) ? "true" : "false");
+ break;
+
+ case TSDB_DATA_TYPE_TINYINT:
+ n = sprintf(str, "%d", *(int8_t*)buf);
+ break;
+
+ case TSDB_DATA_TYPE_SMALLINT:
+ n = sprintf(str, "%d", *(int16_t*)buf);
+ break;
+
+ case TSDB_DATA_TYPE_INT:
+ n = sprintf(str, "%d", *(int32_t*)buf);
+ break;
+
+ case TSDB_DATA_TYPE_BIGINT:
+ case TSDB_DATA_TYPE_TIMESTAMP:
+ n = sprintf(str, "%" PRId64, *(int64_t*)buf);
+ break;
+
+ case TSDB_DATA_TYPE_FLOAT:
+ n = sprintf(str, "%f", GET_FLOAT_VAL(buf));
+ break;
+
+ case TSDB_DATA_TYPE_DOUBLE:
+ n = sprintf(str, "%f", GET_DOUBLE_VAL(buf));
+ break;
+
+ case TSDB_DATA_TYPE_BINARY:
+ case TSDB_DATA_TYPE_NCHAR:
+ if (bufSize < 0) {
+ tscError("invalid buf size");
+ return TSDB_CODE_TSC_INVALID_VALUE;
+ }
+
+ *str = '"';
+ memcpy(str + 1, buf, bufSize);
+ *(str + bufSize + 1) = '"';
+ n = bufSize + 2;
+ break;
+
+ default:
+ tscError("unsupported type:%d", type);
+ return TSDB_CODE_TSC_INVALID_VALUE;
+ }
+
+ *len = n;
+
+ return TSDB_CODE_SUCCESS;
+}
+
+
static void tscStrToLower(char *str, int32_t n) {
if (str == NULL || n <= 0) { return;}
for (int32_t i = 0; i < n; i++) {
diff --git a/src/common/inc/tdataformat.h b/src/common/inc/tdataformat.h
index 88d5b850102cc2ecc796ead0fd75408da428c431..fcae7a415fae8bc1b5897bf6865f1cc2004ba731 100644
--- a/src/common/inc/tdataformat.h
+++ b/src/common/inc/tdataformat.h
@@ -234,6 +234,7 @@ typedef struct SDataCol {
int len; // column data length
VarDataOffsetT *dataOff; // For binary and nchar data, the offset in the data column
void * pData; // Actual data pointer
+ TSKEY ts; // only used in last NULL column
} SDataCol;
static FORCE_INLINE void dataColReset(SDataCol *pDataCol) { pDataCol->len = 0; }
diff --git a/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBJNIConnector.java b/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBJNIConnector.java
index d6934b8e46c6a9dc3821ce06da1397532c747564..179bad2cedea7759b05f507525b584029bfd3ef9 100755
--- a/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBJNIConnector.java
+++ b/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBJNIConnector.java
@@ -310,6 +310,16 @@ public class TSDBJNIConnector {
private native int setBindTableNameImp(long stmt, String name, long conn);
+ public void setBindTableNameAndTags(long stmt, String tableName, int numOfTags, ByteBuffer tags, ByteBuffer typeList, ByteBuffer lengthList, ByteBuffer nullList) throws SQLException {
+ int code = setTableNameTagsImp(stmt, tableName, numOfTags, tags.array(), typeList.array(), lengthList.array(),
+ nullList.array(), this.taos);
+ if (code != TSDBConstants.JNI_SUCCESS) {
+ throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN, "failed to bind table name and corresponding tags");
+ }
+ }
+
+ private native int setTableNameTagsImp(long stmt, String name, int numOfTags, byte[] tags, byte[] typeList, byte[] lengthList, byte[] nullList, long conn);
+
public void bindColumnDataArray(long stmt, ByteBuffer colDataList, ByteBuffer lengthList, ByteBuffer isNullList, int type, int bytes, int numOfRows,int columnIndex) throws SQLException {
int code = bindColDataImp(stmt, colDataList.array(), lengthList.array(), isNullList.array(), type, bytes, numOfRows, columnIndex, this.taos);
if (code != TSDBConstants.JNI_SUCCESS) {
diff --git a/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBPreparedStatement.java b/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBPreparedStatement.java
index 71e07252a34003c3fcc4dceae80e897030e55926..f6810237c097a62a4c8f0f63d6e435bfb0354125 100644
--- a/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBPreparedStatement.java
+++ b/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBPreparedStatement.java
@@ -41,6 +41,9 @@ public class TSDBPreparedStatement extends TSDBStatement implements PreparedStat
private boolean isPrepared;
private ArrayList colData;
+ private ArrayList tableTags;
+ private int tagValueLength;
+
private String tableName;
private long nativeStmtHandle = 0;
@@ -63,8 +66,8 @@ public class TSDBPreparedStatement extends TSDBStatement implements PreparedStat
if (parameterCnt > 1) {
// the table name is also a parameter, so ignore it.
- this.colData = new ArrayList(parameterCnt - 1);
- this.colData.addAll(Collections.nCopies(parameterCnt - 1, null));
+ this.colData = new ArrayList();
+ this.tableTags = new ArrayList();
}
}
@@ -562,11 +565,109 @@ public class TSDBPreparedStatement extends TSDBStatement implements PreparedStat
}
};
+ private static class TableTagInfo {
+ private boolean isNull;
+ private Object value;
+ private int type;
+ public TableTagInfo(Object value, int type) {
+ this.value = value;
+ this.type = type;
+ }
+
+ public static TableTagInfo createNullTag(int type) {
+ TableTagInfo info = new TableTagInfo(null, type);
+ info.isNull = true;
+ return info;
+ }
+ };
+
public void setTableName(String name) {
this.tableName = name;
}
+ private void ensureTagCapacity(int index) {
+ if (this.tableTags.size() < index + 1) {
+ int delta = index + 1 - this.tableTags.size();
+ this.tableTags.addAll(Collections.nCopies(delta, null));
+ }
+ }
+
+ public void setTagNull(int index, int type) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, TableTagInfo.createNullTag(type));
+ }
+
+ public void setTagBoolean(int index, boolean value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_BOOL));
+ this.tagValueLength += Byte.BYTES;
+ }
+
+ public void setTagInt(int index, int value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_INT));
+ this.tagValueLength += Integer.BYTES;
+ }
+
+ public void setTagByte(int index, byte value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_TINYINT));
+ this.tagValueLength += Byte.BYTES;
+ }
+
+ public void setTagShort(int index, short value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_SMALLINT));
+ this.tagValueLength += Short.BYTES;
+ }
+
+ public void setTagLong(int index, long value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_BIGINT));
+ this.tagValueLength += Long.BYTES;
+ }
+
+ public void setTagTimestamp(int index, long value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP));
+ this.tagValueLength += Long.BYTES;
+ }
+
+ public void setTagFloat(int index, float value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_FLOAT));
+ this.tagValueLength += Float.BYTES;
+ }
+
+ public void setTagDouble(int index, double value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_DOUBLE));
+ this.tagValueLength += Double.BYTES;
+ }
+
+ public void setTagString(int index, String value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_BINARY));
+ this.tagValueLength += value.getBytes().length;
+ }
+
+ public void setTagNString(int index, String value) {
+ ensureTagCapacity(index);
+ this.tableTags.set(index, new TableTagInfo(value, TSDBConstants.TSDB_DATA_TYPE_NCHAR));
+
+ String charset = TaosGlobalConfig.getCharset();
+ try {
+ this.tagValueLength += value.getBytes(charset).length;
+ } catch (UnsupportedEncodingException e) {
+ e.printStackTrace();
+ }
+ }
+
public void setValueImpl(int columnIndex, ArrayList list, int type, int bytes) throws SQLException {
+ if (this.colData.size() == 0) {
+ this.colData.addAll(Collections.nCopies(this.parameters.length - 1 - this.tableTags.size(), null));
+
+ }
ColumnInfo col = (ColumnInfo) this.colData.get(columnIndex);
if (col == null) {
ColumnInfo p = new ColumnInfo();
@@ -641,7 +742,122 @@ public class TSDBPreparedStatement extends TSDBStatement implements PreparedStat
TSDBJNIConnector connector = ((TSDBConnection) this.getConnection()).getConnector();
this.nativeStmtHandle = connector.prepareStmt(rawSql);
- connector.setBindTableName(this.nativeStmtHandle, this.tableName);
+
+ if (this.tableTags == null) {
+ connector.setBindTableName(this.nativeStmtHandle, this.tableName);
+ } else {
+ int num = this.tableTags.size();
+ ByteBuffer tagDataList = ByteBuffer.allocate(this.tagValueLength);
+ tagDataList.order(ByteOrder.LITTLE_ENDIAN);
+
+ ByteBuffer typeList = ByteBuffer.allocate(num);
+ typeList.order(ByteOrder.LITTLE_ENDIAN);
+
+ ByteBuffer lengthList = ByteBuffer.allocate(num * Long.BYTES);
+ lengthList.order(ByteOrder.LITTLE_ENDIAN);
+
+ ByteBuffer isNullList = ByteBuffer.allocate(num * Integer.BYTES);
+ isNullList.order(ByteOrder.LITTLE_ENDIAN);
+
+ for (int i = 0; i < num; ++i) {
+ TableTagInfo tag = this.tableTags.get(i);
+ if (tag.isNull) {
+ typeList.put((byte) tag.type);
+ isNullList.putInt(1);
+ lengthList.putLong(0);
+ continue;
+ }
+
+ switch (tag.type) {
+ case TSDBConstants.TSDB_DATA_TYPE_INT: {
+ Integer val = (Integer) tag.value;
+ tagDataList.putInt(val);
+ lengthList.putLong(Integer.BYTES);
+ break;
+ }
+ case TSDBConstants.TSDB_DATA_TYPE_TINYINT: {
+ Byte val = (Byte) tag.value;
+ tagDataList.put(val);
+ lengthList.putLong(Byte.BYTES);
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_BOOL: {
+ Boolean val = (Boolean) tag.value;
+ tagDataList.put((byte) (val ? 1 : 0));
+ lengthList.putLong(Byte.BYTES);
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_SMALLINT: {
+ Short val = (Short) tag.value;
+ tagDataList.putShort(val);
+ lengthList.putLong(Short.BYTES);
+
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
+ case TSDBConstants.TSDB_DATA_TYPE_BIGINT: {
+ Long val = (Long) tag.value;
+ tagDataList.putLong(val == null ? 0 : val);
+ lengthList.putLong(Long.BYTES);
+
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_FLOAT: {
+ Float val = (Float) tag.value;
+ tagDataList.putFloat(val == null ? 0 : val);
+ lengthList.putLong(Float.BYTES);
+
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: {
+ Double val = (Double) tag.value;
+ tagDataList.putDouble(val == null ? 0 : val);
+ lengthList.putLong(Double.BYTES);
+
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
+ case TSDBConstants.TSDB_DATA_TYPE_BINARY: {
+ String charset = TaosGlobalConfig.getCharset();
+ String val = (String) tag.value;
+
+ byte[] b = null;
+ try {
+ if (tag.type == TSDBConstants.TSDB_DATA_TYPE_BINARY) {
+ b = val.getBytes();
+ } else {
+ b = val.getBytes(charset);
+ }
+ } catch (UnsupportedEncodingException e) {
+ e.printStackTrace();
+ }
+
+ tagDataList.put(b);
+ lengthList.putLong(b.length);
+ break;
+ }
+
+ case TSDBConstants.TSDB_DATA_TYPE_UTINYINT:
+ case TSDBConstants.TSDB_DATA_TYPE_USMALLINT:
+ case TSDBConstants.TSDB_DATA_TYPE_UINT:
+ case TSDBConstants.TSDB_DATA_TYPE_UBIGINT: {
+ throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN, "not support data types");
+ }
+ }
+
+ typeList.put((byte) tag.type);
+ isNullList.putInt(tag.isNull? 1 : 0);
+ }
+
+ connector.setBindTableNameAndTags(this.nativeStmtHandle, this.tableName, this.tableTags.size(), tagDataList,
+ typeList, lengthList, isNullList);
+ }
ColumnInfo colInfo = (ColumnInfo) this.colData.get(0);
if (colInfo == null) {
diff --git a/src/dnode/src/dnodeMain.c b/src/dnode/src/dnodeMain.c
index ee4dfcf85210d389651224796727d5b72ea8b9ac..64da11f3123b5eafd1de2ae9253fe5fc3090ad80 100644
--- a/src/dnode/src/dnodeMain.c
+++ b/src/dnode/src/dnodeMain.c
@@ -86,6 +86,17 @@ static SStep tsDnodeSteps[] = {
{"dnode-telemetry", dnodeInitTelemetry, dnodeCleanupTelemetry},
};
+static SStep tsDnodeCompactSteps[] = {
+ {"dnode-tfile", tfInit, tfCleanup},
+ {"dnode-storage", dnodeInitStorage, dnodeCleanupStorage},
+ {"dnode-eps", dnodeInitEps, dnodeCleanupEps},
+ {"dnode-wal", walInit, walCleanUp},
+ {"dnode-mread", dnodeInitMRead, NULL},
+ {"dnode-mwrite", dnodeInitMWrite, NULL},
+ {"dnode-mpeer", dnodeInitMPeer, NULL},
+ {"dnode-modules", dnodeInitModules, dnodeCleanupModules},
+};
+
static int dnodeCreateDir(const char *dir) {
if (mkdir(dir, 0755) != 0 && errno != EEXIST) {
return -1;
@@ -95,13 +106,23 @@ static int dnodeCreateDir(const char *dir) {
}
static void dnodeCleanupComponents() {
- int32_t stepSize = sizeof(tsDnodeSteps) / sizeof(SStep);
- dnodeStepCleanup(tsDnodeSteps, stepSize);
+ if (!tsCompactMnodeWal) {
+ int32_t stepSize = sizeof(tsDnodeSteps) / sizeof(SStep);
+ dnodeStepCleanup(tsDnodeSteps, stepSize);
+ } else {
+ int32_t stepSize = sizeof(tsDnodeCompactSteps) / sizeof(SStep);
+ dnodeStepCleanup(tsDnodeCompactSteps, stepSize);
+ }
}
static int32_t dnodeInitComponents() {
- int32_t stepSize = sizeof(tsDnodeSteps) / sizeof(SStep);
- return dnodeStepInit(tsDnodeSteps, stepSize);
+ if (!tsCompactMnodeWal) {
+ int32_t stepSize = sizeof(tsDnodeSteps) / sizeof(SStep);
+ return dnodeStepInit(tsDnodeSteps, stepSize);
+ } else {
+ int32_t stepSize = sizeof(tsDnodeCompactSteps) / sizeof(SStep);
+ return dnodeStepInit(tsDnodeCompactSteps, stepSize);
+ }
}
static int32_t dnodeInitTmr() {
@@ -219,14 +240,20 @@ static int32_t dnodeInitStorage() {
if (tsCompactMnodeWal == 1) {
sprintf(tsMnodeTmpDir, "%s/mnode_tmp", tsDataDir);
- tfsRmdir(tsMnodeTmpDir);
+ if (taosDirExist(tsMnodeTmpDir)) {
+ dError("mnode_tmp dir already exist in %s,quit compact job", tsMnodeTmpDir);
+ return -1;
+ }
if (dnodeCreateDir(tsMnodeTmpDir) < 0) {
dError("failed to create dir: %s, reason: %s", tsMnodeTmpDir, strerror(errno));
return -1;
}
sprintf(tsMnodeBakDir, "%s/mnode_bak", tsDataDir);
- //tfsRmdir(tsMnodeBakDir);
+ if (taosDirExist(tsMnodeBakDir)) {
+ dError("mnode_bak dir already exist in %s,quit compact job", tsMnodeBakDir);
+ return -1;
+ }
}
//TODO(dengyihao): no need to init here
if (dnodeCreateDir(tsMnodeDir) < 0) {
diff --git a/src/inc/taos.h b/src/inc/taos.h
index 6dd695b320f71ff65b97715aed108c7d1b4e2f7e..d27828fc364fd795e238b33df464fdc3548b60e2 100644
--- a/src/inc/taos.h
+++ b/src/inc/taos.h
@@ -112,6 +112,7 @@ typedef struct TAOS_MULTI_BIND {
TAOS_STMT *taos_stmt_init(TAOS *taos);
int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length);
+int taos_stmt_set_tbname_tags(TAOS_STMT* stmt, const char* name, TAOS_BIND* tags);
int taos_stmt_set_tbname(TAOS_STMT* stmt, const char* name);
int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
int taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
diff --git a/src/inc/taosdef.h b/src/inc/taosdef.h
index 979559c7e725079082380f63df52c6a4a119aab9..c00fde18162a1c7006706286ac26bc5cd6e7e547 100644
--- a/src/inc/taosdef.h
+++ b/src/inc/taosdef.h
@@ -33,6 +33,8 @@ extern "C" {
#endif
#define TSWINDOW_INITIALIZER ((STimeWindow) {INT64_MIN, INT64_MAX})
+#define TSWINDOW_DESC_INITIALIZER ((STimeWindow) {INT64_MAX, INT64_MIN})
+
#define TSKEY_INITIAL_VAL INT64_MIN
// Bytes for each type.
@@ -298,7 +300,7 @@ do { \
#define TSDB_DEFAULT_DB_UPDATE_OPTION 0
#define TSDB_MIN_DB_CACHE_LAST_ROW 0
-#define TSDB_MAX_DB_CACHE_LAST_ROW 1
+#define TSDB_MAX_DB_CACHE_LAST_ROW 3
#define TSDB_DEFAULT_CACHE_LAST_ROW 0
#define TSDB_MIN_FSYNC_PERIOD 0
diff --git a/src/inc/taoserror.h b/src/inc/taoserror.h
index b239b0ebbf7d854788ba49e847b5f5b6d0c51865..09f632ea32077b3e9bd3fcddcff418f280192c3f 100644
--- a/src/inc/taoserror.h
+++ b/src/inc/taoserror.h
@@ -74,7 +74,7 @@ int32_t* taosGetErrno();
#define TSDB_CODE_REF_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x010A) //"Ref is not there")
//client
-#define TSDB_CODE_TSC_INVALID_OPERATION TAOS_DEF_ERROR_CODE(0, 0x0200) //"Invalid SQL statement")
+#define TSDB_CODE_TSC_INVALID_OPERATION TAOS_DEF_ERROR_CODE(0, 0x0200) //"Invalid Operation")
#define TSDB_CODE_TSC_INVALID_QHANDLE TAOS_DEF_ERROR_CODE(0, 0x0201) //"Invalid qhandle")
#define TSDB_CODE_TSC_INVALID_TIME_STAMP TAOS_DEF_ERROR_CODE(0, 0x0202) //"Invalid combination of client/service time")
#define TSDB_CODE_TSC_INVALID_VALUE TAOS_DEF_ERROR_CODE(0, 0x0203) //"Invalid value in client")
diff --git a/src/inc/tsdb.h b/src/inc/tsdb.h
index 1ba5131f6db82cf7f0fdd512b7dbf94bd68914dc..21d55d5d0fea0312b07e17eee2cbfc91e2175fc1 100644
--- a/src/inc/tsdb.h
+++ b/src/inc/tsdb.h
@@ -69,9 +69,13 @@ typedef struct {
int8_t precision;
int8_t compression;
int8_t update;
- int8_t cacheLastRow;
+ int8_t cacheLastRow; // 0:no cache, 1: cache last row, 2: cache last NULL column 3: 1&2
} STsdbCfg;
+#define CACHE_NO_LAST(c) ((c)->cacheLastRow == 0)
+#define CACHE_LAST_ROW(c) (((c)->cacheLastRow & 1) > 0)
+#define CACHE_LAST_NULL_COLUMN(c) (((c)->cacheLastRow & 2) > 0)
+
// --------- TSDB REPOSITORY USAGE STATISTICS
typedef struct {
int64_t totalStorage; // total bytes occupie
@@ -261,6 +265,12 @@ TsdbQueryHandleT *tsdbQueryTables(STsdbRepo *tsdb, STsdbQueryCond *pCond, STable
TsdbQueryHandleT tsdbQueryLastRow(STsdbRepo *tsdb, STsdbQueryCond *pCond, STableGroupInfo *tableInfo, uint64_t qId,
SMemRef *pRef);
+
+TsdbQueryHandleT tsdbQueryCacheLast(STsdbRepo *tsdb, STsdbQueryCond *pCond, STableGroupInfo *groupList, uint64_t qId, SMemRef* pMemRef);
+
+bool isTsdbCacheLastRow(TsdbQueryHandleT* pQueryHandle);
+
+
/**
* get the queried table object list
* @param pHandle
diff --git a/src/kit/taosdemo/taosdemo.c b/src/kit/taosdemo/taosdemo.c
index 888dc4a04e4fc31d1b775e2f078f123899a61fa5..6f5d407a7348b2fa7768f0266c5dcd79a6b4653f 100644
--- a/src/kit/taosdemo/taosdemo.c
+++ b/src/kit/taosdemo/taosdemo.c
@@ -53,6 +53,8 @@
#include "taoserror.h"
#include "tutil.h"
+#define STMT_IFACE_ENABLED 1
+
#define REQ_EXTRA_BUF_LEN 1024
#define RESP_BUF_LEN 4096
@@ -253,12 +255,13 @@ typedef struct SColumn_S {
typedef struct SSuperTable_S {
char sTblName[MAX_TB_NAME_SIZE+1];
+ char dataSource[MAX_TB_NAME_SIZE+1]; // rand_gen or sample
+ char childTblPrefix[MAX_TB_NAME_SIZE];
+ char insertMode[MAX_TB_NAME_SIZE]; // taosc, rest
+ uint16_t childTblExists;
int64_t childTblCount;
- bool childTblExists; // 0: no, 1: yes
uint64_t batchCreateTableNum; // 0: no batch, > 0: batch table number in one sql
uint8_t autoCreateTable; // 0: create sub table, 1: auto create sub table
- char childTblPrefix[MAX_TB_NAME_SIZE];
- char dataSource[MAX_TB_NAME_SIZE+1]; // rand_gen or sample
uint16_t iface; // 0: taosc, 1: rest, 2: stmt
int64_t childTblLimit;
uint64_t childTblOffset;
@@ -377,7 +380,7 @@ typedef struct SDbs_S {
typedef struct SpecifiedQueryInfo_S {
uint64_t queryInterval; // 0: unlimit > 0 loop/s
uint32_t concurrent;
- uint64_t sqlCount;
+ int sqlCount;
uint32_t asyncMode; // 0: sync, 1: async
uint64_t subscribeInterval; // ms
uint64_t queryTimes;
@@ -386,6 +389,7 @@ typedef struct SpecifiedQueryInfo_S {
char sql[MAX_QUERY_SQL_COUNT][MAX_QUERY_SQL_LENGTH+1];
char result[MAX_QUERY_SQL_COUNT][MAX_FILE_NAME_LEN+1];
int resubAfterConsume[MAX_QUERY_SQL_COUNT];
+ int endAfterConsume[MAX_QUERY_SQL_COUNT];
TAOS_SUB* tsub[MAX_QUERY_SQL_COUNT];
char topic[MAX_QUERY_SQL_COUNT][32];
int consumed[MAX_QUERY_SQL_COUNT];
@@ -404,10 +408,11 @@ typedef struct SuperQueryInfo_S {
uint64_t queryTimes;
int64_t childTblCount;
char childTblPrefix[MAX_TB_NAME_SIZE];
- uint64_t sqlCount;
+ int sqlCount;
char sql[MAX_QUERY_SQL_COUNT][MAX_QUERY_SQL_LENGTH+1];
char result[MAX_QUERY_SQL_COUNT][MAX_FILE_NAME_LEN+1];
int resubAfterConsume;
+ int endAfterConsume;
TAOS_SUB* tsub[MAX_QUERY_SQL_COUNT];
char* childTblName;
@@ -688,7 +693,11 @@ static void printHelp() {
printf("%s%s%s%s\n", indent, "-p", indent,
"The TCP/IP port number to use for the connection. Default is 0.");
printf("%s%s%s%s\n", indent, "-I", indent,
+#if STMT_IFACE_ENABLED == 1
"The interface (taosc, rest, and stmt) taosdemo uses. Default is 'taosc'.");
+#else
+ "The interface (taosc, rest) taosdemo uses. Default is 'taosc'.");
+#endif
printf("%s%s%s%s\n", indent, "-d", indent,
"Destination database. Default is 'test'.");
printf("%s%s%s%s\n", indent, "-a", indent,
@@ -788,8 +797,10 @@ static void parse_args(int argc, char *argv[], SArguments *arguments) {
arguments->iface = TAOSC_IFACE;
} else if (0 == strcasecmp(argv[i], "rest")) {
arguments->iface = REST_IFACE;
+#if STMT_IFACE_ENABLED == 1
} else if (0 == strcasecmp(argv[i], "stmt")) {
arguments->iface = STMT_IFACE;
+#endif
} else {
errorPrint("%s", "\n\t-I need a valid string following!\n");
exit(EXIT_FAILURE);
@@ -826,7 +837,7 @@ static void parse_args(int argc, char *argv[], SArguments *arguments) {
if ((argc == i+1)
|| (!isStringNumber(argv[i+1]))) {
printHelp();
- errorPrint("%s", "\n\t-q need a number following!\nQuery mode -- 0: SYNC, 1: ASYNC. Default is SYNC.\n");
+ errorPrint("%s", "\n\t-q need a number following!\nQuery mode -- 0: SYNC, not-0: ASYNC. Default is SYNC.\n");
exit(EXIT_FAILURE);
}
arguments->async_mode = atoi(argv[++i]);
@@ -1054,6 +1065,19 @@ static void parse_args(int argc, char *argv[], SArguments *arguments) {
}
}
+ int columnCount;
+ for (columnCount = 0; columnCount < MAX_NUM_DATATYPE; columnCount ++) {
+ if (g_args.datatype[columnCount] == NULL) {
+ break;
+ }
+ }
+
+ if (0 == columnCount) {
+ perror("data type error!");
+ exit(-1);
+ }
+ g_args.num_of_CPR = columnCount;
+
if (((arguments->debug_print) && (arguments->metaFile == NULL))
|| arguments->verbose_print) {
printf("###################################################################\n");
@@ -1076,7 +1100,7 @@ static void parse_args(int argc, char *argv[], SArguments *arguments) {
}
printf("# Insertion interval: %"PRIu64"\n",
arguments->insert_interval);
- printf("# Number of records per req: %ud\n",
+ printf("# Number of records per req: %u\n",
arguments->num_of_RPR);
printf("# Max SQL length: %"PRIu64"\n",
arguments->max_sql_len);
@@ -1368,7 +1392,7 @@ static int printfInsertMeta() {
g_Dbs.threadCountByCreateTbl);
printf("top insert interval: \033[33m%"PRIu64"\033[0m\n",
g_args.insert_interval);
- printf("number of records per req: \033[33m%ud\033[0m\n",
+ printf("number of records per req: \033[33m%u\033[0m\n",
g_args.num_of_RPR);
printf("max sql length: \033[33m%"PRIu64"\033[0m\n",
g_args.max_sql_len);
@@ -1454,7 +1478,8 @@ static int printfInsertMeta() {
if (PRE_CREATE_SUBTBL == g_Dbs.db[i].superTbls[j].autoCreateTable) {
printf(" autoCreateTable: \033[33m%s\033[0m\n", "no");
- } else if (AUTO_CREATE_SUBTBL == g_Dbs.db[i].superTbls[j].autoCreateTable) {
+ } else if (AUTO_CREATE_SUBTBL ==
+ g_Dbs.db[i].superTbls[j].autoCreateTable) {
printf(" autoCreateTable: \033[33m%s\033[0m\n", "yes");
} else {
printf(" autoCreateTable: \033[33m%s\033[0m\n", "error");
@@ -1494,7 +1519,7 @@ static int printfInsertMeta() {
printf(" multiThreadWriteOneTbl: \033[33myes\033[0m\n");
}
*/
- printf(" interlaceRows: \033[33m%ud\033[0m\n",
+ printf(" interlaceRows: \033[33m%u\033[0m\n",
g_Dbs.db[i].superTbls[j].interlaceRows);
if (g_Dbs.db[i].superTbls[j].interlaceRows > 0) {
@@ -1572,7 +1597,7 @@ static void printfInsertMetaToFile(FILE* fp) {
fprintf(fp, "resultFile: %s\n", g_Dbs.resultFile);
fprintf(fp, "thread num of insert data: %d\n", g_Dbs.threadCount);
fprintf(fp, "thread num of create table: %d\n", g_Dbs.threadCountByCreateTbl);
- fprintf(fp, "number of records per req: %ud\n", g_args.num_of_RPR);
+ fprintf(fp, "number of records per req: %u\n", g_args.num_of_RPR);
fprintf(fp, "max sql length: %"PRIu64"\n", g_args.max_sql_len);
fprintf(fp, "database count: %d\n", g_Dbs.dbCount);
@@ -1669,7 +1694,7 @@ static void printfInsertMetaToFile(FILE* fp) {
(g_Dbs.db[i].superTbls[j].iface==REST_IFACE)?"rest":"stmt");
fprintf(fp, " insertRows: %"PRId64"\n",
g_Dbs.db[i].superTbls[j].insertRows);
- fprintf(fp, " interlace rows: %ud\n",
+ fprintf(fp, " interlace rows: %u\n",
g_Dbs.db[i].superTbls[j].interlaceRows);
if (g_Dbs.db[i].superTbls[j].interlaceRows > 0) {
fprintf(fp, " stable insert interval: %"PRIu64"\n",
@@ -1682,7 +1707,7 @@ static void printfInsertMetaToFile(FILE* fp) {
fprintf(fp, " multiThreadWriteOneTbl: yes\n");
}
*/
- fprintf(fp, " interlaceRows: %ud\n",
+ fprintf(fp, " interlaceRows: %u\n",
g_Dbs.db[i].superTbls[j].interlaceRows);
fprintf(fp, " disorderRange: %d\n",
g_Dbs.db[i].superTbls[j].disorderRange);
@@ -1758,7 +1783,7 @@ static void printfQueryMeta() {
if ((SUBSCRIBE_TEST == g_args.test_mode) || (QUERY_TEST == g_args.test_mode)) {
printf("specified table query info: \n");
- printf("sqlCount: \033[33m%"PRIu64"\033[0m\n",
+ printf("sqlCount: \033[33m%d\033[0m\n",
g_queryInfo.specifiedQueryInfo.sqlCount);
if (g_queryInfo.specifiedQueryInfo.sqlCount > 0) {
printf("specified tbl query times:\n");
@@ -1778,15 +1803,15 @@ static void printfQueryMeta() {
printf("keepProgress: \033[33m%d\033[0m\n",
g_queryInfo.specifiedQueryInfo.subscribeKeepProgress);
- for (uint64_t i = 0; i < g_queryInfo.specifiedQueryInfo.sqlCount; i++) {
- printf(" sql[%"PRIu64"]: \033[33m%s\033[0m\n",
+ for (int i = 0; i < g_queryInfo.specifiedQueryInfo.sqlCount; i++) {
+ printf(" sql[%d]: \033[33m%s\033[0m\n",
i, g_queryInfo.specifiedQueryInfo.sql[i]);
}
printf("\n");
}
printf("super table query info:\n");
- printf("sqlCount: \033[33m%"PRIu64"\033[0m\n",
+ printf("sqlCount: \033[33m%d\033[0m\n",
g_queryInfo.superQueryInfo.sqlCount);
if (g_queryInfo.superQueryInfo.sqlCount > 0) {
@@ -2885,8 +2910,8 @@ static void* createTable(void *sarg)
int buff_len;
buff_len = BUFFER_SIZE / 8;
- char *buffer = calloc(buff_len, 1);
- if (buffer == NULL) {
+ pThreadInfo->buffer = calloc(buff_len, 1);
+ if (pThreadInfo->buffer == NULL) {
errorPrint("%s() LN%d, Memory allocated failed!\n", __func__, __LINE__);
exit(-1);
}
@@ -2901,7 +2926,7 @@ static void* createTable(void *sarg)
for (uint64_t i = pThreadInfo->start_table_from;
i <= pThreadInfo->end_table_to; i++) {
if (0 == g_Dbs.use_metric) {
- snprintf(buffer, buff_len,
+ snprintf(pThreadInfo->buffer, buff_len,
"create table if not exists %s.%s%"PRIu64" %s;",
pThreadInfo->db_name,
g_args.tb_prefix, i,
@@ -2910,13 +2935,13 @@ static void* createTable(void *sarg)
if (superTblInfo == NULL) {
errorPrint("%s() LN%d, use metric, but super table info is NULL\n",
__func__, __LINE__);
- free(buffer);
+ free(pThreadInfo->buffer);
exit(-1);
} else {
if (0 == len) {
batchNum = 0;
- memset(buffer, 0, buff_len);
- len += snprintf(buffer + len,
+ memset(pThreadInfo->buffer, 0, buff_len);
+ len += snprintf(pThreadInfo->buffer + len,
buff_len - len, "create table ");
}
char* tagsValBuf = NULL;
@@ -2928,10 +2953,10 @@ static void* createTable(void *sarg)
i % superTblInfo->tagSampleCount);
}
if (NULL == tagsValBuf) {
- free(buffer);
+ free(pThreadInfo->buffer);
return NULL;
}
- len += snprintf(buffer + len,
+ len += snprintf(pThreadInfo->buffer + len,
buff_len - len,
"if not exists %s.%s%"PRIu64" using %s.%s tags %s ",
pThreadInfo->db_name, superTblInfo->childTblPrefix,
@@ -2948,9 +2973,10 @@ static void* createTable(void *sarg)
}
len = 0;
- if (0 != queryDbExec(pThreadInfo->taos, buffer, NO_INSERT_TYPE, false)){
- errorPrint( "queryDbExec() failed. buffer:\n%s\n", buffer);
- free(buffer);
+ if (0 != queryDbExec(pThreadInfo->taos, pThreadInfo->buffer,
+ NO_INSERT_TYPE, false)){
+ errorPrint( "queryDbExec() failed. buffer:\n%s\n", pThreadInfo->buffer);
+ free(pThreadInfo->buffer);
return NULL;
}
@@ -2963,12 +2989,13 @@ static void* createTable(void *sarg)
}
if (0 != len) {
- if (0 != queryDbExec(pThreadInfo->taos, buffer, NO_INSERT_TYPE, false)) {
- errorPrint( "queryDbExec() failed. buffer:\n%s\n", buffer);
+ if (0 != queryDbExec(pThreadInfo->taos, pThreadInfo->buffer,
+ NO_INSERT_TYPE, false)) {
+ errorPrint( "queryDbExec() failed. buffer:\n%s\n", pThreadInfo->buffer);
}
}
- free(buffer);
+ free(pThreadInfo->buffer);
return NULL;
}
@@ -3046,61 +3073,61 @@ static void createChildTables() {
char tblColsBuf[MAX_SQL_SIZE];
int len;
- for (int i = 0; i < g_Dbs.dbCount; i++) {
- if (g_Dbs.use_metric) {
- if (g_Dbs.db[i].superTblCount > 0) {
- // with super table
- for (uint64_t j = 0; j < g_Dbs.db[i].superTblCount; j++) {
- if ((AUTO_CREATE_SUBTBL == g_Dbs.db[i].superTbls[j].autoCreateTable)
- || (TBL_ALREADY_EXISTS == g_Dbs.db[i].superTbls[j].childTblExists)) {
- continue;
- }
+ for (int i = 0; i < g_Dbs.dbCount; i++) {
+ if (g_Dbs.use_metric) {
+ if (g_Dbs.db[i].superTblCount > 0) {
+ // with super table
+ for (int j = 0; j < g_Dbs.db[i].superTblCount; j++) {
+ if ((AUTO_CREATE_SUBTBL == g_Dbs.db[i].superTbls[j].autoCreateTable)
+ || (TBL_ALREADY_EXISTS == g_Dbs.db[i].superTbls[j].childTblExists)) {
+ continue;
+ }
+ verbosePrint("%s() LN%d: %s\n", __func__, __LINE__,
+ g_Dbs.db[i].superTbls[j].colsOfCreateChildTable);
+ uint64_t startFrom = 0;
+ g_totalChildTables += g_Dbs.db[i].superTbls[j].childTblCount;
+
+ verbosePrint("%s() LN%d: create %"PRId64" child tables from %"PRIu64"\n",
+ __func__, __LINE__, g_totalChildTables, startFrom);
+
+ startMultiThreadCreateChildTable(
+ g_Dbs.db[i].superTbls[j].colsOfCreateChildTable,
+ g_Dbs.threadCountByCreateTbl,
+ startFrom,
+ g_Dbs.db[i].superTbls[j].childTblCount,
+ g_Dbs.db[i].dbName, &(g_Dbs.db[i].superTbls[j]));
+ }
+ }
+ } else {
+ // normal table
+ len = snprintf(tblColsBuf, MAX_SQL_SIZE, "(TS TIMESTAMP");
+ for (int j = 0; j < g_args.num_of_CPR; j++) {
+ if ((strncasecmp(g_args.datatype[j], "BINARY", strlen("BINARY")) == 0)
+ || (strncasecmp(g_args.datatype[j],
+ "NCHAR", strlen("NCHAR")) == 0)) {
+ snprintf(tblColsBuf + len, MAX_SQL_SIZE - len,
+ ", COL%d %s(%d)", j, g_args.datatype[j], g_args.len_of_binary);
+ } else {
+ snprintf(tblColsBuf + len, MAX_SQL_SIZE - len,
+ ", COL%d %s", j, g_args.datatype[j]);
+ }
+ len = strlen(tblColsBuf);
+ }
- verbosePrint("%s() LN%d: %s\n", __func__, __LINE__,
- g_Dbs.db[i].superTbls[j].colsOfCreateChildTable);
- uint64_t tableFrom = 0;
- g_totalChildTables += g_Dbs.db[i].superTbls[j].childTblCount;
-
- verbosePrint("%s() LN%d: create %"PRId64" child tables from %"PRIu64"\n",
- __func__, __LINE__, g_totalChildTables, tableFrom);
- startMultiThreadCreateChildTable(
- g_Dbs.db[i].superTbls[j].colsOfCreateChildTable,
- g_Dbs.threadCountByCreateTbl,
- tableFrom,
- g_Dbs.db[i].superTbls[j].childTblCount,
- g_Dbs.db[i].dbName, &(g_Dbs.db[i].superTbls[j]));
+ snprintf(tblColsBuf + len, MAX_SQL_SIZE - len, ")");
+
+ verbosePrint("%s() LN%d: dbName: %s num of tb: %"PRId64" schema: %s\n",
+ __func__, __LINE__,
+ g_Dbs.db[i].dbName, g_args.num_of_tables, tblColsBuf);
+ startMultiThreadCreateChildTable(
+ tblColsBuf,
+ g_Dbs.threadCountByCreateTbl,
+ 0,
+ g_args.num_of_tables,
+ g_Dbs.db[i].dbName,
+ NULL);
}
- }
- } else {
- // normal table
- len = snprintf(tblColsBuf, MAX_SQL_SIZE, "(TS TIMESTAMP");
- for (int j = 0; j < g_args.num_of_CPR; j++) {
- if ((strncasecmp(g_args.datatype[j], "BINARY", strlen("BINARY")) == 0)
- || (strncasecmp(g_args.datatype[j],
- "NCHAR", strlen("NCHAR")) == 0)) {
- snprintf(tblColsBuf + len, MAX_SQL_SIZE - len,
- ", COL%d %s(%d)", j, g_args.datatype[j], g_args.len_of_binary);
- } else {
- snprintf(tblColsBuf + len, MAX_SQL_SIZE - len,
- ", COL%d %s", j, g_args.datatype[j]);
- }
- len = strlen(tblColsBuf);
- }
-
- snprintf(tblColsBuf + len, MAX_SQL_SIZE - len, ")");
-
- verbosePrint("%s() LN%d: dbName: %s num of tb: %"PRId64" schema: %s\n",
- __func__, __LINE__,
- g_Dbs.db[i].dbName, g_args.num_of_tables, tblColsBuf);
- startMultiThreadCreateChildTable(
- tblColsBuf,
- g_Dbs.threadCountByCreateTbl,
- 0,
- g_args.num_of_tables,
- g_Dbs.db[i].dbName,
- NULL);
}
- }
}
/*
@@ -3561,9 +3588,9 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
// rows per table need be less than insert batch
if (g_args.interlace_rows > g_args.num_of_RPR) {
- printf("NOTICE: interlace rows value %ud > num_of_records_per_req %ud\n\n",
+ printf("NOTICE: interlace rows value %u > num_of_records_per_req %u\n\n",
g_args.interlace_rows, g_args.num_of_RPR);
- printf(" interlace rows value will be set to num_of_records_per_req %ud\n\n",
+ printf(" interlace rows value will be set to num_of_records_per_req %u\n\n",
g_args.num_of_RPR);
prompt();
g_args.interlace_rows = g_args.num_of_RPR;
@@ -3794,36 +3821,40 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
// dbinfo
cJSON *stbName = cJSON_GetObjectItem(stbInfo, "name");
- if (!stbName || stbName->type != cJSON_String || stbName->valuestring == NULL) {
+ if (!stbName || stbName->type != cJSON_String
+ || stbName->valuestring == NULL) {
errorPrint("%s() LN%d, failed to read json, stb name not found\n",
__func__, __LINE__);
goto PARSE_OVER;
}
- tstrncpy(g_Dbs.db[i].superTbls[j].sTblName, stbName->valuestring, MAX_TB_NAME_SIZE);
+ tstrncpy(g_Dbs.db[i].superTbls[j].sTblName, stbName->valuestring,
+ MAX_TB_NAME_SIZE);
cJSON *prefix = cJSON_GetObjectItem(stbInfo, "childtable_prefix");
if (!prefix || prefix->type != cJSON_String || prefix->valuestring == NULL) {
printf("ERROR: failed to read json, childtable_prefix not found\n");
goto PARSE_OVER;
}
- tstrncpy(g_Dbs.db[i].superTbls[j].childTblPrefix, prefix->valuestring, MAX_DB_NAME_SIZE);
+ tstrncpy(g_Dbs.db[i].superTbls[j].childTblPrefix, prefix->valuestring,
+ MAX_DB_NAME_SIZE);
- cJSON *autoCreateTbl = cJSON_GetObjectItem(stbInfo, "auto_create_table"); // yes, no, null
+ cJSON *autoCreateTbl = cJSON_GetObjectItem(stbInfo, "auto_create_table");
if (autoCreateTbl
&& autoCreateTbl->type == cJSON_String
&& autoCreateTbl->valuestring != NULL) {
- if (0 == strncasecmp(autoCreateTbl->valuestring, "yes", 3)) {
- g_Dbs.db[i].superTbls[j].autoCreateTable = AUTO_CREATE_SUBTBL;
- } else if (0 == strncasecmp(autoCreateTbl->valuestring, "no", 2)) {
- g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
- } else {
- g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
- }
+ if ((0 == strncasecmp(autoCreateTbl->valuestring, "yes", 3))
+ && (TBL_ALREADY_EXISTS != g_Dbs.db[i].superTbls[j].childTblExists)) {
+ g_Dbs.db[i].superTbls[j].autoCreateTable = AUTO_CREATE_SUBTBL;
+ } else if (0 == strncasecmp(autoCreateTbl->valuestring, "no", 2)) {
+ g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
+ } else {
+ g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
+ }
} else if (!autoCreateTbl) {
- g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
+ g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
} else {
- printf("ERROR: failed to read json, auto_create_table not found\n");
- goto PARSE_OVER;
+ printf("ERROR: failed to read json, auto_create_table not found\n");
+ goto PARSE_OVER;
}
cJSON* batchCreateTbl = cJSON_GetObjectItem(stbInfo, "batch_create_tbl_num");
@@ -3857,6 +3888,10 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
goto PARSE_OVER;
}
+ if (TBL_ALREADY_EXISTS == g_Dbs.db[i].superTbls[j].childTblExists) {
+ g_Dbs.db[i].superTbls[j].autoCreateTable = PRE_CREATE_SUBTBL;
+ }
+
cJSON* count = cJSON_GetObjectItem(stbInfo, "childtable_count");
if (!count || count->type != cJSON_Number || 0 >= count->valueint) {
errorPrint("%s() LN%d, failed to read json, childtable_count input mistake\n",
@@ -3885,8 +3920,10 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
g_Dbs.db[i].superTbls[j].iface= TAOSC_IFACE;
} else if (0 == strcasecmp(stbIface->valuestring, "rest")) {
g_Dbs.db[i].superTbls[j].iface= REST_IFACE;
+#if STMT_IFACE_ENABLED == 1
} else if (0 == strcasecmp(stbIface->valuestring, "stmt")) {
g_Dbs.db[i].superTbls[j].iface= STMT_IFACE;
+#endif
} else {
errorPrint("%s() LN%d, failed to read json, insert_mode %s not recognized\n",
__func__, __LINE__, stbIface->valuestring);
@@ -3914,7 +3951,8 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
cJSON* childTbl_offset = cJSON_GetObjectItem(stbInfo, "childtable_offset");
if ((childTbl_offset) && (g_Dbs.db[i].drop != true)
&& (g_Dbs.db[i].superTbls[j].childTblExists == TBL_ALREADY_EXISTS)) {
- if (childTbl_offset->type != cJSON_Number || 0 > childTbl_offset->valueint) {
+ if ((childTbl_offset->type != cJSON_Number)
+ || (0 > childTbl_offset->valueint)) {
printf("ERROR: failed to read json, childtable_offset\n");
goto PARSE_OVER;
}
@@ -3970,7 +4008,8 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
}
cJSON *tagsFile = cJSON_GetObjectItem(stbInfo, "tags_file");
- if (tagsFile && tagsFile->type == cJSON_String && tagsFile->valuestring != NULL) {
+ if ((tagsFile && tagsFile->type == cJSON_String)
+ && (tagsFile->valuestring != NULL)) {
tstrncpy(g_Dbs.db[i].superTbls[j].tagsFile,
tagsFile->valuestring, MAX_FILE_NAME_LEN);
if (0 == g_Dbs.db[i].superTbls[j].tagsFile[0]) {
@@ -4030,10 +4069,10 @@ static bool getMetaFromInsertJsonFile(cJSON* root) {
g_Dbs.db[i].superTbls[j].interlaceRows = stbInterlaceRows->valueint;
// rows per table need be less than insert batch
if (g_Dbs.db[i].superTbls[j].interlaceRows > g_args.num_of_RPR) {
- printf("NOTICE: db[%d].superTbl[%d]'s interlace rows value %ud > num_of_records_per_req %ud\n\n",
+ printf("NOTICE: db[%d].superTbl[%d]'s interlace rows value %u > num_of_records_per_req %u\n\n",
i, j, g_Dbs.db[i].superTbls[j].interlaceRows,
g_args.num_of_RPR);
- printf(" interlace rows value will be set to num_of_records_per_req %ud\n\n",
+ printf(" interlace rows value will be set to num_of_records_per_req %u\n\n",
g_args.num_of_RPR);
prompt();
g_Dbs.db[i].superTbls[j].interlaceRows = g_args.num_of_RPR;
@@ -4250,7 +4289,7 @@ static bool getMetaFromQueryJsonFile(cJSON* root) {
if (concurrent && concurrent->type == cJSON_Number) {
if (concurrent->valueint <= 0) {
errorPrint(
- "%s() LN%d, query sqlCount %"PRIu64" or concurrent %d is not correct.\n",
+ "%s() LN%d, query sqlCount %d or concurrent %d is not correct.\n",
__func__, __LINE__,
g_queryInfo.specifiedQueryInfo.sqlCount,
g_queryInfo.specifiedQueryInfo.concurrent);
@@ -4349,16 +4388,27 @@ static bool getMetaFromQueryJsonFile(cJSON* root) {
tstrncpy(g_queryInfo.specifiedQueryInfo.sql[j],
sqlStr->valuestring, MAX_QUERY_SQL_LENGTH);
+ cJSON* endAfterConsume =
+ cJSON_GetObjectItem(specifiedQuery, "endAfterConsume");
+ if (endAfterConsume
+ && endAfterConsume->type == cJSON_Number) {
+ g_queryInfo.specifiedQueryInfo.endAfterConsume[j]
+ = endAfterConsume->valueint;
+ } else if (!endAfterConsume) {
+ // default value is -1, which mean infinite loop
+ g_queryInfo.specifiedQueryInfo.endAfterConsume[j] = -1;
+ }
+
cJSON* resubAfterConsume =
cJSON_GetObjectItem(specifiedQuery, "resubAfterConsume");
- if (resubAfterConsume
- && resubAfterConsume->type == cJSON_Number) {
+ if ((resubAfterConsume)
+ && (resubAfterConsume->type == cJSON_Number)
+ && (resubAfterConsume->valueint >= 0)) {
g_queryInfo.specifiedQueryInfo.resubAfterConsume[j]
= resubAfterConsume->valueint;
} else if (!resubAfterConsume) {
- //printf("failed to read json, subscribe interval no found\n");
- //goto PARSE_OVER;
- g_queryInfo.specifiedQueryInfo.resubAfterConsume[j] = 1;
+ // default value is -1, which mean do not resub
+ g_queryInfo.specifiedQueryInfo.resubAfterConsume[j] = -1;
}
cJSON *result = cJSON_GetObjectItem(sql, "result");
@@ -4502,16 +4552,27 @@ static bool getMetaFromQueryJsonFile(cJSON* root) {
g_queryInfo.superQueryInfo.subscribeKeepProgress = 0;
}
+ cJSON* superEndAfterConsume =
+ cJSON_GetObjectItem(superQuery, "endAfterConsume");
+ if (superEndAfterConsume
+ && superEndAfterConsume->type == cJSON_Number) {
+ g_queryInfo.superQueryInfo.endAfterConsume =
+ superEndAfterConsume->valueint;
+ } else if (!superEndAfterConsume) {
+ // default value is -1, which mean do not resub
+ g_queryInfo.superQueryInfo.endAfterConsume = -1;
+ }
+
cJSON* superResubAfterConsume =
cJSON_GetObjectItem(superQuery, "resubAfterConsume");
- if (superResubAfterConsume
- && superResubAfterConsume->type == cJSON_Number) {
+ if ((superResubAfterConsume)
+ && (superResubAfterConsume->type == cJSON_Number)
+ && (superResubAfterConsume->valueint >= 0)) {
g_queryInfo.superQueryInfo.resubAfterConsume =
superResubAfterConsume->valueint;
} else if (!superResubAfterConsume) {
- //printf("failed to read json, subscribe interval no found\n");
- ////goto PARSE_OVER;
- g_queryInfo.superQueryInfo.resubAfterConsume = 1;
+ // default value is -1, which mean do not resub
+ g_queryInfo.superQueryInfo.resubAfterConsume = -1;
}
// supert table sqls
@@ -4773,44 +4834,34 @@ static int64_t generateData(char *recBuf, char **data_type,
memset(recBuf, 0, MAX_DATA_SIZE);
char *pstr = recBuf;
pstr += sprintf(pstr, "(%" PRId64, timestamp);
- int c = 0;
- for (; c < MAX_NUM_DATATYPE; c++) {
- if (data_type[c] == NULL) {
- break;
- }
- }
+ int columnCount = g_args.num_of_CPR;
- if (0 == c) {
- perror("data type error!");
- exit(-1);
- }
-
- for (int i = 0; i < c; i++) {
- if (strcasecmp(data_type[i % c], "TINYINT") == 0) {
+ for (int i = 0; i < columnCount; i++) {
+ if (strcasecmp(data_type[i % columnCount], "TINYINT") == 0) {
pstr += sprintf(pstr, ",%d", rand_tinyint() );
- } else if (strcasecmp(data_type[i % c], "SMALLINT") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "SMALLINT") == 0) {
pstr += sprintf(pstr, ",%d", rand_smallint());
- } else if (strcasecmp(data_type[i % c], "INT") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "INT") == 0) {
pstr += sprintf(pstr, ",%d", rand_int());
- } else if (strcasecmp(data_type[i % c], "BIGINT") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "BIGINT") == 0) {
pstr += sprintf(pstr, ",%" PRId64, rand_bigint());
- } else if (strcasecmp(data_type[i % c], "TIMESTAMP") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "TIMESTAMP") == 0) {
pstr += sprintf(pstr, ",%" PRId64, rand_bigint());
- } else if (strcasecmp(data_type[i % c], "FLOAT") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "FLOAT") == 0) {
pstr += sprintf(pstr, ",%10.4f", rand_float());
- } else if (strcasecmp(data_type[i % c], "DOUBLE") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "DOUBLE") == 0) {
double t = rand_double();
pstr += sprintf(pstr, ",%20.8f", t);
- } else if (strcasecmp(data_type[i % c], "BOOL") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "BOOL") == 0) {
bool b = rand_bool() & 1;
pstr += sprintf(pstr, ",%s", b ? "true" : "false");
- } else if (strcasecmp(data_type[i % c], "BINARY") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "BINARY") == 0) {
char *s = malloc(lenOfBinary);
rand_string(s, lenOfBinary);
pstr += sprintf(pstr, ",\"%s\"", s);
free(s);
- } else if (strcasecmp(data_type[i % c], "NCHAR") == 0) {
+ } else if (strcasecmp(data_type[i % columnCount], "NCHAR") == 0) {
char *s = malloc(lenOfBinary);
rand_string(s, lenOfBinary);
pstr += sprintf(pstr, ",\"%s\"", s);
@@ -4857,74 +4908,93 @@ static int prepareSampleDataForSTable(SSuperTable *superTblInfo) {
return 0;
}
-static int64_t execInsert(threadInfo *pThreadInfo, uint64_t k)
+static int32_t execInsert(threadInfo *pThreadInfo, uint32_t k)
{
- int affectedRows;
- SSuperTable* superTblInfo = pThreadInfo->superTblInfo;
+ int32_t affectedRows;
+ SSuperTable* superTblInfo = pThreadInfo->superTblInfo;
- verbosePrint("[%d] %s() LN%d %s\n", pThreadInfo->threadID,
+ verbosePrint("[%d] %s() LN%d %s\n", pThreadInfo->threadID,
__func__, __LINE__, pThreadInfo->buffer);
- if (superTblInfo) {
- if (superTblInfo->iface == TAOSC_IFACE) {
- affectedRows = queryDbExec(
- pThreadInfo->taos,
- pThreadInfo->buffer, INSERT_TYPE, false);
- } else if (superTblInfo->iface == REST_IFACE) {
- if (0 != postProceSql(g_Dbs.host, &g_Dbs.serv_addr, g_Dbs.port,
- pThreadInfo->buffer, NULL /* not set result file */)) {
- affectedRows = -1;
- printf("========restful return fail, threadID[%d]\n",
- pThreadInfo->threadID);
- } else {
- affectedRows = k;
- }
- } else if (superTblInfo->iface == STMT_IFACE) {
- debugPrint("%s() LN%d, stmt=%p", __func__, __LINE__, pThreadInfo->stmt);
- if (0 != taos_stmt_execute(pThreadInfo->stmt)) {
- errorPrint("%s() LN%d, failied to execute insert statement\n",
- __func__, __LINE__);
- exit(-1);
- }
- affectedRows = k;
- } else {
- errorPrint("%s() LN%d: unknown insert mode: %d\n",
- __func__, __LINE__, superTblInfo->iface);
- affectedRows = 0;
+ uint16_t iface;
+ if (superTblInfo)
+ iface = superTblInfo->iface;
+ else
+ iface = g_args.iface;
+
+ debugPrint("[%d] %s() LN%d %s\n", pThreadInfo->threadID,
+ __func__, __LINE__,
+ (g_args.iface==TAOSC_IFACE)?
+ "taosc":(g_args.iface==REST_IFACE)?"rest":"stmt");
+
+ switch(iface) {
+ case TAOSC_IFACE:
+ affectedRows = queryDbExec(
+ pThreadInfo->taos,
+ pThreadInfo->buffer, INSERT_TYPE, false);
+ break;
+
+ case REST_IFACE:
+ if (0 != postProceSql(g_Dbs.host, &g_Dbs.serv_addr, g_Dbs.port,
+ pThreadInfo->buffer, pThreadInfo)) {
+ affectedRows = -1;
+ printf("========restful return fail, threadID[%d]\n",
+ pThreadInfo->threadID);
+ } else {
+ affectedRows = k;
+ }
+ break;
+
+#if STMT_IFACE_ENABLED == 1
+ case STMT_IFACE:
+ debugPrint("%s() LN%d, stmt=%p", __func__, __LINE__, pThreadInfo->stmt);
+ if (0 != taos_stmt_execute(pThreadInfo->stmt)) {
+ errorPrint("%s() LN%d, failied to execute insert statement\n",
+ __func__, __LINE__);
+ exit(-1);
+ }
+ affectedRows = k;
+ break;
+#endif
+
+ default:
+ errorPrint("%s() LN%d: unknown insert mode: %d\n",
+ __func__, __LINE__, superTblInfo->iface);
+ affectedRows = 0;
}
- } else {
- affectedRows = queryDbExec(pThreadInfo->taos, pThreadInfo->buffer, INSERT_TYPE, false);
- }
- return affectedRows;
+ return affectedRows;
}
static void getTableName(char *pTblName,
threadInfo* pThreadInfo, uint64_t tableSeq)
{
- SSuperTable* superTblInfo = pThreadInfo->superTblInfo;
- if ((superTblInfo)
- && (AUTO_CREATE_SUBTBL != superTblInfo->autoCreateTable)) {
- if (superTblInfo->childTblLimit > 0) {
- snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s",
- superTblInfo->childTblName +
- (tableSeq - superTblInfo->childTblOffset) * TSDB_TABLE_NAME_LEN);
+ SSuperTable* superTblInfo = pThreadInfo->superTblInfo;
+ if (superTblInfo) {
+ if (AUTO_CREATE_SUBTBL != superTblInfo->autoCreateTable) {
+ if (superTblInfo->childTblLimit > 0) {
+ snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s",
+ superTblInfo->childTblName +
+ (tableSeq - superTblInfo->childTblOffset) * TSDB_TABLE_NAME_LEN);
+ } else {
+ verbosePrint("[%d] %s() LN%d: from=%"PRIu64" count=%"PRId64" seq=%"PRIu64"\n",
+ pThreadInfo->threadID, __func__, __LINE__,
+ pThreadInfo->start_table_from,
+ pThreadInfo->ntables, tableSeq);
+ snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s",
+ superTblInfo->childTblName + tableSeq * TSDB_TABLE_NAME_LEN);
+ }
+ } else {
+ snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s%"PRIu64"",
+ superTblInfo->childTblPrefix, tableSeq);
+ }
} else {
-
- verbosePrint("[%d] %s() LN%d: from=%"PRIu64" count=%"PRId64" seq=%"PRIu64"\n",
- pThreadInfo->threadID, __func__, __LINE__,
- pThreadInfo->start_table_from,
- pThreadInfo->ntables, tableSeq);
- snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s",
- superTblInfo->childTblName + tableSeq * TSDB_TABLE_NAME_LEN);
+ snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s%"PRIu64"",
+ g_args.tb_prefix, tableSeq);
}
- } else {
- snprintf(pTblName, TSDB_TABLE_NAME_LEN, "%s%"PRIu64"",
- g_args.tb_prefix, tableSeq);
- }
}
-static int64_t generateDataTailWithoutStb(
+static int32_t generateDataTailWithoutStb(
uint32_t batch, char* buffer,
int64_t remainderBufLen, int64_t insertRows,
uint64_t recordFrom, int64_t startTime,
@@ -4935,7 +5005,7 @@ static int64_t generateDataTailWithoutStb(
verbosePrint("%s() LN%d batch=%d\n", __func__, __LINE__, batch);
- int64_t k = 0;
+ int32_t k = 0;
for (k = 0; k < batch;) {
char data[MAX_DATA_SIZE];
memset(data, 0, MAX_DATA_SIZE);
@@ -4947,7 +5017,7 @@ static int64_t generateDataTailWithoutStb(
retLen = generateData(data, data_type,
startTime + getTSRandTail(
- (int64_t)DEFAULT_TIMESTAMP_STEP, k,
+ (int64_t) DEFAULT_TIMESTAMP_STEP, k,
g_args.disorderRatio,
g_args.disorderRange),
lenOfBinary);
@@ -4960,7 +5030,7 @@ static int64_t generateDataTailWithoutStb(
len += retLen;
remainderBufLen -= retLen;
- verbosePrint("%s() LN%d len=%"PRIu64" k=%"PRIu64" \nbuffer=%s\n",
+ verbosePrint("%s() LN%d len=%"PRIu64" k=%d \nbuffer=%s\n",
__func__, __LINE__, len, k, buffer);
recordFrom ++;
@@ -5006,7 +5076,7 @@ static int32_t generateStbDataTail(
} else {
tsRand = false;
}
- verbosePrint("%s() LN%d batch=%ud\n", __func__, __LINE__, batch);
+ verbosePrint("%s() LN%d batch=%u\n", __func__, __LINE__, batch);
int32_t k = 0;
for (k = 0; k < batch;) {
@@ -5040,7 +5110,7 @@ static int32_t generateStbDataTail(
len += retLen;
remainderBufLen -= retLen;
- verbosePrint("%s() LN%d len=%"PRIu64" k=%ud \nbuffer=%s\n",
+ verbosePrint("%s() LN%d len=%"PRIu64" k=%u \nbuffer=%s\n",
__func__, __LINE__, len, k, buffer);
recordFrom ++;
@@ -5168,7 +5238,7 @@ static int32_t generateStbInterlaceData(
int64_t dataLen = 0;
- verbosePrint("[%d] %s() LN%d i=%"PRIu64" batchPerTblTimes=%ud batchPerTbl = %ud\n",
+ verbosePrint("[%d] %s() LN%d i=%"PRIu64" batchPerTblTimes=%u batchPerTbl = %u\n",
pThreadInfo->threadID, __func__, __LINE__,
i, batchPerTblTimes, batchPerTbl);
@@ -5186,7 +5256,7 @@ static int32_t generateStbInterlaceData(
pstr += dataLen;
*pRemainderBufLen -= dataLen;
} else {
- debugPrint("%s() LN%d, generated data tail: %ud, not equal batch per table: %ud\n",
+ debugPrint("%s() LN%d, generated data tail: %u, not equal batch per table: %u\n",
__func__, __LINE__, k, batchPerTbl);
pstr -= headLen;
pstr[0] = '\0';
@@ -5197,10 +5267,11 @@ static int32_t generateStbInterlaceData(
}
static int64_t generateInterlaceDataWithoutStb(
- char *tableName, uint32_t batchPerTbl,
+ char *tableName, uint32_t batch,
uint64_t tableSeq,
char *dbName, char *buffer,
int64_t insertRows,
+ int64_t startTime,
uint64_t *pRemainderBufLen)
{
assert(buffer);
@@ -5219,18 +5290,17 @@ static int64_t generateInterlaceDataWithoutStb(
int64_t dataLen = 0;
- int64_t startTime = 1500000000000;
- int64_t k = generateDataTailWithoutStb(
- batchPerTbl, pstr, *pRemainderBufLen, insertRows, 0,
+ int32_t k = generateDataTailWithoutStb(
+ batch, pstr, *pRemainderBufLen, insertRows, 0,
startTime,
&dataLen);
- if (k == batchPerTbl) {
+ if (k == batch) {
pstr += dataLen;
*pRemainderBufLen -= dataLen;
} else {
- debugPrint("%s() LN%d, generated data tail: %"PRIu64", not equal batch per table: %ud\n",
- __func__, __LINE__, k, batchPerTbl);
+ debugPrint("%s() LN%d, generated data tail: %d, not equal batch per table: %u\n",
+ __func__, __LINE__, k, batch);
pstr -= headLen;
pstr[0] = '\0';
k = 0;
@@ -5239,32 +5309,239 @@ static int64_t generateInterlaceDataWithoutStb(
return k;
}
-static int32_t prepareStbStmt(SSuperTable *stbInfo,
+#if STMT_IFACE_ENABLED == 1
+static int32_t prepareStmtBindArrayByType(TAOS_BIND *bind,
+ char *dataType, int32_t dataLen, char **ptr)
+{
+ if (0 == strncasecmp(dataType,
+ "BINARY", strlen("BINARY"))) {
+ if (dataLen > TSDB_MAX_BINARY_LEN) {
+ errorPrint( "binary length overflow, max size:%u\n",
+ (uint32_t)TSDB_MAX_BINARY_LEN);
+ return -1;
+ }
+ char *bind_binary = (char *)*ptr;
+ rand_string(bind_binary, dataLen);
+
+ bind->buffer_type = TSDB_DATA_TYPE_BINARY;
+ bind->buffer_length = dataLen;
+ bind->buffer = bind_binary;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "NCHAR", strlen("NCHAR"))) {
+ if (dataLen > TSDB_MAX_BINARY_LEN) {
+ errorPrint( "nchar length overflow, max size:%u\n",
+ (uint32_t)TSDB_MAX_BINARY_LEN);
+ return -1;
+ }
+ char *bind_nchar = (char *)*ptr;
+ rand_string(bind_nchar, dataLen);
+
+ bind->buffer_type = TSDB_DATA_TYPE_NCHAR;
+ bind->buffer_length = strlen(bind_nchar);
+ bind->buffer = bind_nchar;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "INT", strlen("INT"))) {
+ int32_t *bind_int = (int32_t *)*ptr;
+
+ *bind_int = rand_int();
+ bind->buffer_type = TSDB_DATA_TYPE_INT;
+ bind->buffer_length = sizeof(int32_t);
+ bind->buffer = bind_int;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "BIGINT", strlen("BIGINT"))) {
+ int64_t *bind_bigint = (int64_t *)*ptr;
+
+ *bind_bigint = rand_bigint();
+ bind->buffer_type = TSDB_DATA_TYPE_BIGINT;
+ bind->buffer_length = sizeof(int64_t);
+ bind->buffer = bind_bigint;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "FLOAT", strlen("FLOAT"))) {
+ float *bind_float = (float *) *ptr;
+
+ *bind_float = rand_float();
+ bind->buffer_type = TSDB_DATA_TYPE_FLOAT;
+ bind->buffer_length = sizeof(float);
+ bind->buffer = bind_float;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "DOUBLE", strlen("DOUBLE"))) {
+ double *bind_double = (double *)*ptr;
+
+ *bind_double = rand_double();
+ bind->buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ bind->buffer_length = sizeof(double);
+ bind->buffer = bind_double;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "SMALLINT", strlen("SMALLINT"))) {
+ int16_t *bind_smallint = (int16_t *)*ptr;
+
+ *bind_smallint = rand_smallint();
+ bind->buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ bind->buffer_length = sizeof(int16_t);
+ bind->buffer = bind_smallint;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "TINYINT", strlen("TINYINT"))) {
+ int8_t *bind_tinyint = (int8_t *)*ptr;
+
+ *bind_tinyint = rand_tinyint();
+ bind->buffer_type = TSDB_DATA_TYPE_TINYINT;
+ bind->buffer_length = sizeof(int8_t);
+ bind->buffer = bind_tinyint;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "BOOL", strlen("BOOL"))) {
+ int8_t *bind_bool = (int8_t *)*ptr;
+
+ *bind_bool = rand_bool();
+ bind->buffer_type = TSDB_DATA_TYPE_BOOL;
+ bind->buffer_length = sizeof(int8_t);
+ bind->buffer = bind_bool;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else if (0 == strncasecmp(dataType,
+ "TIMESTAMP", strlen("TIMESTAMP"))) {
+ int64_t *bind_ts2 = (int64_t *) *ptr;
+
+ *bind_ts2 = rand_bigint();
+ bind->buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ bind->buffer_length = sizeof(int64_t);
+ bind->buffer = bind_ts2;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ *ptr += bind->buffer_length;
+ } else {
+ errorPrint( "No support data type: %s\n",
+ dataType);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int32_t prepareStmtWithoutStb(
TAOS_STMT *stmt,
- char *tableName, uint32_t batch, uint64_t insertRows,
- uint64_t recordFrom,
- int64_t startTime, char *buffer)
+ char *tableName,
+ uint32_t batch,
+ int64_t insertRows,
+ int64_t recordFrom,
+ int64_t startTime)
{
- uint32_t k;
- int ret;
- char *pstr = buffer;
- pstr += sprintf(pstr, "INSERT INTO %s values(?", tableName);
+ int ret = taos_stmt_set_tbname(stmt, tableName);
+ if (ret != 0) {
+ errorPrint("failed to execute taos_stmt_set_tbname(%s). return 0x%x. reason: %s\n",
+ tableName, ret, taos_errstr(NULL));
+ return ret;
+ }
+
+ char **data_type = g_args.datatype;
+
+ char *bindArray = malloc(sizeof(TAOS_BIND) * (g_args.num_of_CPR + 1));
+ if (bindArray == NULL) {
+ errorPrint("Failed to allocate %d bind params\n",
+ (g_args.num_of_CPR + 1));
+ return -1;
+ }
- for (int i = 0; i < stbInfo->columnCount; i++) {
- pstr += sprintf(pstr, ",?");
+ int32_t k = 0;
+ for (k = 0; k < batch;) {
+ /* columnCount + 1 (ts) */
+ char data[MAX_DATA_SIZE];
+ memset(data, 0, MAX_DATA_SIZE);
+
+ char *ptr = data;
+ TAOS_BIND *bind = (TAOS_BIND *)(bindArray + 0);
+
+ int64_t *bind_ts;
+
+ bind_ts = (int64_t *)ptr;
+ bind->buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ *bind_ts = startTime + getTSRandTail(
+ (int64_t)DEFAULT_TIMESTAMP_STEP, k,
+ g_args.disorderRatio,
+ g_args.disorderRange);
+ bind->buffer_length = sizeof(int64_t);
+ bind->buffer = bind_ts;
+ bind->length = &bind->buffer_length;
+ bind->is_null = NULL;
+
+ ptr += bind->buffer_length;
+
+ for (int i = 0; i < g_args.num_of_CPR; i ++) {
+ bind = (TAOS_BIND *)((char *)bindArray + (sizeof(TAOS_BIND) * (i + 1)));
+ if ( -1 == prepareStmtBindArrayByType(
+ bind,
+ data_type[i],
+ g_args.len_of_binary,
+ &ptr)) {
+ return -1;
+ }
+ }
+ taos_stmt_bind_param(stmt, (TAOS_BIND *)bindArray);
+ // if msg > 3MB, break
+ taos_stmt_add_batch(stmt);
+
+ k++;
+ recordFrom ++;
+ if (recordFrom >= insertRows) {
+ break;
+ }
}
- pstr += sprintf(pstr, ")");
- ret = taos_stmt_prepare(stmt, buffer, 0);
- if (ret != 0){
- errorPrint("failed to execute taos_stmt_prepare. return 0x%x. reason: %s\n",
- ret, taos_errstr(NULL));
+ free(bindArray);
+ return k;
+}
+
+static int32_t prepareStbStmt(SSuperTable *stbInfo,
+ TAOS_STMT *stmt,
+ char *tableName, uint32_t batch,
+ uint64_t insertRows,
+ uint64_t recordFrom,
+ int64_t startTime, char *buffer)
+{
+ int ret = taos_stmt_set_tbname(stmt, tableName);
+ if (ret != 0) {
+ errorPrint("failed to execute taos_stmt_set_tbname(%s). return 0x%x. reason: %s\n",
+ tableName, ret, taos_errstr(NULL));
return ret;
}
char *bindArray = malloc(sizeof(TAOS_BIND) * (stbInfo->columnCount + 1));
if (bindArray == NULL) {
- errorPrint("Failed to allocate %d bind params\n", batch);
+ errorPrint("Failed to allocate %d bind params\n",
+ (stbInfo->columnCount + 1));
return -1;
}
@@ -5274,6 +5551,8 @@ static int32_t prepareStbStmt(SSuperTable *stbInfo,
} else {
tsRand = false;
}
+
+ uint32_t k;
for (k = 0; k < batch;) {
/* columnCount + 1 (ts) */
char data[MAX_DATA_SIZE];
@@ -5303,135 +5582,12 @@ static int32_t prepareStbStmt(SSuperTable *stbInfo,
for (int i = 0; i < stbInfo->columnCount; i ++) {
bind = (TAOS_BIND *)((char *)bindArray + (sizeof(TAOS_BIND) * (i + 1)));
- if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "BINARY", strlen("BINARY"))) {
- if (stbInfo->columns[i].dataLen > TSDB_MAX_BINARY_LEN) {
- errorPrint( "binary length overflow, max size:%u\n",
- (uint32_t)TSDB_MAX_BINARY_LEN);
- return -1;
- }
- char *bind_binary = (char *)ptr;
- rand_string(bind_binary, stbInfo->columns[i].dataLen);
-
- bind->buffer_type = TSDB_DATA_TYPE_BINARY;
- bind->buffer_length = stbInfo->columns[i].dataLen;
- bind->buffer = bind_binary;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
-
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "NCHAR", strlen("NCHAR"))) {
- if (stbInfo->columns[i].dataLen > TSDB_MAX_BINARY_LEN) {
- errorPrint( "nchar length overflow, max size:%u\n",
- (uint32_t)TSDB_MAX_BINARY_LEN);
- return -1;
- }
- char *bind_nchar = (char *)ptr;
- rand_string(bind_nchar, stbInfo->columns[i].dataLen);
-
- bind->buffer_type = TSDB_DATA_TYPE_NCHAR;
- bind->buffer_length = strlen(bind_nchar);
- bind->buffer = bind_nchar;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
-
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "INT", strlen("INT"))) {
- int32_t *bind_int = (int32_t *)ptr;
-
- *bind_int = rand_int();
- bind->buffer_type = TSDB_DATA_TYPE_INT;
- bind->buffer_length = sizeof(int32_t);
- bind->buffer = bind_int;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
-
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "BIGINT", strlen("BIGINT"))) {
- int64_t *bind_bigint = (int64_t *)ptr;
-
- *bind_bigint = rand_bigint();
- bind->buffer_type = TSDB_DATA_TYPE_BIGINT;
- bind->buffer_length = sizeof(int64_t);
- bind->buffer = bind_bigint;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "FLOAT", strlen("FLOAT"))) {
- float *bind_float = (float *)ptr;
-
- *bind_float = rand_float();
- bind->buffer_type = TSDB_DATA_TYPE_FLOAT;
- bind->buffer_length = sizeof(float);
- bind->buffer = bind_float;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "DOUBLE", strlen("DOUBLE"))) {
- double *bind_double = (double *)ptr;
-
- *bind_double = rand_double();
- bind->buffer_type = TSDB_DATA_TYPE_DOUBLE;
- bind->buffer_length = sizeof(double);
- bind->buffer = bind_double;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "SMALLINT", strlen("SMALLINT"))) {
- int16_t *bind_smallint = (int16_t *)ptr;
-
- *bind_smallint = rand_smallint();
- bind->buffer_type = TSDB_DATA_TYPE_SMALLINT;
- bind->buffer_length = sizeof(int16_t);
- bind->buffer = bind_smallint;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "TINYINT", strlen("TINYINT"))) {
- int8_t *bind_tinyint = (int8_t *)ptr;
-
- *bind_tinyint = rand_tinyint();
- bind->buffer_type = TSDB_DATA_TYPE_TINYINT;
- bind->buffer_length = sizeof(int8_t);
- bind->buffer = bind_tinyint;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "BOOL", strlen("BOOL"))) {
- int8_t *bind_bool = (int8_t *)ptr;
-
- *bind_bool = rand_bool();
- bind->buffer_type = TSDB_DATA_TYPE_BOOL;
- bind->buffer_length = sizeof(int8_t);
- bind->buffer = bind_bool;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
-
- ptr += bind->buffer_length;
- } else if (0 == strncasecmp(stbInfo->columns[i].dataType,
- "TIMESTAMP", strlen("TIMESTAMP"))) {
- int64_t *bind_ts2 = (int64_t *)ptr;
-
- *bind_ts2 = rand_bigint();
- bind->buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
- bind->buffer_length = sizeof(int64_t);
- bind->buffer = bind_ts2;
- bind->length = &bind->buffer_length;
- bind->is_null = NULL;
-
- ptr += bind->buffer_length;
- } else {
- errorPrint( "No support data type: %s\n",
- stbInfo->columns[i].dataType);
- return -1;
+ if ( -1 == prepareStmtBindArrayByType(
+ bind,
+ stbInfo->columns[i].dataType,
+ stbInfo->columns[i].dataLen,
+ &ptr)) {
+ return -1;
}
}
taos_stmt_bind_param(stmt, (TAOS_BIND *)bindArray);
@@ -5445,8 +5601,10 @@ static int32_t prepareStbStmt(SSuperTable *stbInfo,
}
}
+ free(bindArray);
return k;
}
+#endif
static int32_t generateStbProgressiveData(
SSuperTable *superTblInfo,
@@ -5482,12 +5640,7 @@ static int32_t generateStbProgressiveData(
pSamplePos, &dataLen);
}
-static int64_t prepareStmtWithoutStb(char *tableName)
-{
- return -1;
-}
-
-static int64_t generateProgressiveDataWithoutStb(
+static int32_t generateProgressiveDataWithoutStb(
char *tableName,
/* int64_t tableSeq, */
threadInfo *pThreadInfo, char *buffer,
@@ -5520,7 +5673,7 @@ static int64_t generateProgressiveDataWithoutStb(
static void printStatPerThread(threadInfo *pThreadInfo)
{
- fprintf(stderr, "====thread[%d] completed total inserted rows: %"PRIu64 ", total affected rows: %"PRIu64". %.2f records/second====\n",
+ fprintf(stderr, "====thread[%d] completed total inserted rows: %"PRIu64 ", total affected rows: %"PRIu64". %.2f records/second====\n",
pThreadInfo->threadID,
pThreadInfo->totalInsertRows,
pThreadInfo->totalAffectedRows,
@@ -5638,22 +5791,53 @@ static void* syncWriteInterlace(threadInfo *pThreadInfo) {
int32_t generated;
if (superTblInfo) {
- generated = generateStbInterlaceData(
- superTblInfo,
- tableName, batchPerTbl, i,
- batchPerTblTimes,
- tableSeq,
- pThreadInfo, pstr,
- insertRows,
- startTime,
- &remainderBufLen);
+ if (superTblInfo->iface == STMT_IFACE) {
+#if STMT_IFACE_ENABLED == 1
+ generated = prepareStbStmt(superTblInfo,
+ pThreadInfo->stmt,
+ tableName,
+ batchPerTbl,
+ insertRows, i,
+ startTime,
+ pThreadInfo->buffer);
+#else
+ generated = -1;
+#endif
+ } else {
+ generated = generateStbInterlaceData(
+ superTblInfo,
+ tableName, batchPerTbl, i,
+ batchPerTblTimes,
+ tableSeq,
+ pThreadInfo, pstr,
+ insertRows,
+ startTime,
+ &remainderBufLen);
+ }
} else {
- generated = generateInterlaceDataWithoutStb(
- tableName, batchPerTbl,
- tableSeq,
- pThreadInfo->db_name, pstr,
- insertRows,
- &remainderBufLen);
+ if (g_args.iface == STMT_IFACE) {
+ debugPrint("[%d] %s() LN%d, tableName:%s, batch:%d startTime:%"PRId64"\n",
+ pThreadInfo->threadID,
+ __func__, __LINE__,
+ tableName, batchPerTbl, startTime);
+#if STMT_IFACE_ENABLED == 1
+ generated = prepareStmtWithoutStb(
+ pThreadInfo->stmt, tableName,
+ batchPerTbl,
+ insertRows, i,
+ startTime);
+#else
+ generated = -1;
+#endif
+ } else {
+ generated = generateInterlaceDataWithoutStb(
+ tableName, batchPerTbl,
+ tableSeq,
+ pThreadInfo->db_name, pstr,
+ insertRows,
+ startTime,
+ &remainderBufLen);
+ }
}
debugPrint("[%d] %s() LN%d, generated records is %d\n",
@@ -5668,9 +5852,10 @@ static void* syncWriteInterlace(threadInfo *pThreadInfo) {
tableSeq ++;
recOfBatch += batchPerTbl;
+
pstr += (oldRemainderLen - remainderBufLen);
-// startTime += batchPerTbl * superTblInfo->timeStampStep;
pThreadInfo->totalInsertRows += batchPerTbl;
+
verbosePrint("[%d] %s() LN%d batchPerTbl=%d recOfBatch=%d\n",
pThreadInfo->threadID, __func__, __LINE__,
batchPerTbl, recOfBatch);
@@ -5824,10 +6009,16 @@ static void* syncWriteProgressive(threadInfo *pThreadInfo) {
int32_t generated;
if (superTblInfo) {
if (superTblInfo->iface == STMT_IFACE) {
- generated = prepareStbStmt(superTblInfo,
+#if STMT_IFACE_ENABLED == 1
+ generated = prepareStbStmt(
+ superTblInfo,
pThreadInfo->stmt,
- tableName, g_args.num_of_RPR,
+ tableName,
+ g_args.num_of_RPR,
insertRows, i, start_time, pstr);
+#else
+ generated = -1;
+#endif
} else {
generated = generateStbProgressiveData(
superTblInfo,
@@ -5838,7 +6029,16 @@ static void* syncWriteProgressive(threadInfo *pThreadInfo) {
}
} else {
if (g_args.iface == STMT_IFACE) {
- generated = prepareStmtWithoutStb(tableName);
+#if STMT_IFACE_ENABLED == 1
+ generated = prepareStmtWithoutStb(
+ pThreadInfo->stmt,
+ tableName,
+ g_args.num_of_RPR,
+ insertRows, i,
+ start_time);
+#else
+ generated = -1;
+#endif
} else {
generated = generateProgressiveDataWithoutStb(
tableName,
@@ -5859,13 +6059,13 @@ static void* syncWriteProgressive(threadInfo *pThreadInfo) {
startTs = taosGetTimestampMs();
- int64_t affectedRows = execInsert(pThreadInfo, generated);
+ int32_t affectedRows = execInsert(pThreadInfo, generated);
endTs = taosGetTimestampMs();
uint64_t delay = endTs - startTs;
performancePrint("%s() LN%d, insert execution time is %"PRId64"ms\n",
__func__, __LINE__, delay);
- verbosePrint("[%d] %s() LN%d affectedRows=%"PRId64"\n",
+ verbosePrint("[%d] %s() LN%d affectedRows=%d\n",
pThreadInfo->threadID,
__func__, __LINE__, affectedRows);
@@ -5875,7 +6075,7 @@ static void* syncWriteProgressive(threadInfo *pThreadInfo) {
pThreadInfo->totalDelay += delay;
if (affectedRows < 0) {
- errorPrint("%s() LN%d, affected rows: %"PRId64"\n",
+ errorPrint("%s() LN%d, affected rows: %d\n",
__func__, __LINE__, affectedRows);
goto free_of_progressive;
}
@@ -6045,15 +6245,6 @@ static int convertHostToServAddr(char *host, uint16_t port, struct sockaddr_in *
static void startMultiThreadInsertData(int threads, char* db_name,
char* precision,SSuperTable* superTblInfo) {
- //TAOS* taos;
- //if (0 == strncasecmp(superTblInfo->iface, "taosc", 5)) {
- // taos = taos_connect(g_Dbs.host, g_Dbs.user, g_Dbs.password, db_name, g_Dbs.port);
- // if (NULL == taos) {
- // printf("connect to server fail, reason: %s\n", taos_errstr(NULL));
- // exit(-1);
- // }
- //}
-
int32_t timePrec = TSDB_TIME_PRECISION_MILLI;
if (0 != precision[0]) {
if (0 == strncasecmp(precision, "ms", 2)) {
@@ -6232,7 +6423,17 @@ static void startMultiThreadInsertData(int threads, char* db_name,
exit(-1);
}
- if ((superTblInfo) && (superTblInfo->iface == STMT_IFACE)) {
+#if STMT_IFACE_ENABLED == 1
+ if ((g_args.iface == STMT_IFACE)
+ || ((superTblInfo) && (superTblInfo->iface == STMT_IFACE))) {
+
+ int columnCount;
+ if (superTblInfo) {
+ columnCount = superTblInfo->columnCount;
+ } else {
+ columnCount = g_args.num_of_CPR;
+ }
+
pThreadInfo->stmt = taos_stmt_init(pThreadInfo->taos);
if (NULL == pThreadInfo->stmt) {
errorPrint(
@@ -6243,7 +6444,26 @@ static void startMultiThreadInsertData(int threads, char* db_name,
free(infos);
exit(-1);
}
+
+ char buffer[3000];
+ char *pstr = buffer;
+ pstr += sprintf(pstr, "INSERT INTO ? values(?");
+
+ for (int col = 0; col < columnCount; col ++) {
+ pstr += sprintf(pstr, ",?");
+ }
+ pstr += sprintf(pstr, ")");
+
+ int ret = taos_stmt_prepare(pThreadInfo->stmt, buffer, 0);
+ if (ret != 0){
+ errorPrint("failed to execute taos_stmt_prepare. return 0x%x. reason: %s\n",
+ ret, taos_errstr(NULL));
+ free(pids);
+ free(infos);
+ exit(-1);
+ }
}
+#endif
} else {
pThreadInfo->taos = NULL;
}
@@ -6284,9 +6504,11 @@ static void startMultiThreadInsertData(int threads, char* db_name,
tsem_destroy(&(pThreadInfo->lock_sem));
+#if STMT_IFACE_ENABLED == 1
if (pThreadInfo->stmt) {
taos_stmt_close(pThreadInfo->stmt);
}
+#endif
tsem_destroy(&(pThreadInfo->lock_sem));
taos_close(pThreadInfo->taos);
@@ -6563,7 +6785,6 @@ static int insertTestProcess() {
}
}
- taosMsleep(1000);
// create sub threads for inserting data
//start = taosGetTimestampMs();
for (int i = 0; i < g_Dbs.dbCount; i++) {
@@ -6642,7 +6863,7 @@ static void *specifiedTableQuery(void *sarg) {
uint64_t lastPrintTime = taosGetTimestampMs();
uint64_t startTs = taosGetTimestampMs();
- if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq] != NULL) {
+ if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq][0] != '\0') {
sprintf(pThreadInfo->filePath, "%s-%d",
g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq],
pThreadInfo->threadID);
@@ -6743,7 +6964,7 @@ static void *superTableQuery(void *sarg) {
for (int j = 0; j < g_queryInfo.superQueryInfo.sqlCount; j++) {
memset(sqlstr,0,sizeof(sqlstr));
replaceChildTblName(g_queryInfo.superQueryInfo.sql[j], sqlstr, i);
- if (g_queryInfo.superQueryInfo.result[j] != NULL) {
+ if (g_queryInfo.superQueryInfo.result[j][0] != '\0') {
sprintf(pThreadInfo->filePath, "%s-%d",
g_queryInfo.superQueryInfo.result[j],
pThreadInfo->threadID);
@@ -7082,7 +7303,10 @@ static void *superSubscribe(void *sarg) {
uint64_t st = 0, et = 0;
- while(1) {
+ while ((g_queryInfo.superQueryInfo.endAfterConsume == -1)
+ || (g_queryInfo.superQueryInfo.endAfterConsume <
+ consumed[pThreadInfo->end_table_to - pThreadInfo->start_table_from])) {
+
for (uint64_t i = pThreadInfo->start_table_from;
i <= pThreadInfo->end_table_to; i++) {
tsubSeq = i - pThreadInfo->start_table_from;
@@ -7111,7 +7335,7 @@ static void *superSubscribe(void *sarg) {
}
consumed[tsubSeq] ++;
- if ((g_queryInfo.superQueryInfo.subscribeKeepProgress)
+ if ((g_queryInfo.superQueryInfo.resubAfterConsume != -1)
&& (consumed[tsubSeq] >=
g_queryInfo.superQueryInfo.resubAfterConsume)) {
printf("keepProgress:%d, resub super table query: %"PRIu64"\n",
@@ -7174,7 +7398,7 @@ static void *specifiedSubscribe(void *sarg) {
"taosdemo-subscribe-%"PRIu64"-%d",
pThreadInfo->querySeq,
pThreadInfo->threadID);
- if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq] != NULL) {
+ if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq][0] != '\0') {
sprintf(pThreadInfo->filePath, "%s-%d",
g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq],
pThreadInfo->threadID);
@@ -7193,7 +7417,10 @@ static void *specifiedSubscribe(void *sarg) {
// start loop to consume result
g_queryInfo.specifiedQueryInfo.consumed[pThreadInfo->threadID] = 0;
- while(1) {
+ while((g_queryInfo.specifiedQueryInfo.endAfterConsume[pThreadInfo->querySeq] == -1)
+ || (g_queryInfo.specifiedQueryInfo.consumed[pThreadInfo->threadID] <
+ g_queryInfo.specifiedQueryInfo.endAfterConsume[pThreadInfo->querySeq])) {
+
if (ASYNC_MODE == g_queryInfo.specifiedQueryInfo.asyncMode) {
continue;
}
@@ -7201,15 +7428,18 @@ static void *specifiedSubscribe(void *sarg) {
g_queryInfo.specifiedQueryInfo.res[pThreadInfo->threadID] = taos_consume(
g_queryInfo.specifiedQueryInfo.tsub[pThreadInfo->threadID]);
if (g_queryInfo.specifiedQueryInfo.res[pThreadInfo->threadID]) {
- if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq][0] != 0) {
+ if (g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq][0]
+ != 0) {
sprintf(pThreadInfo->filePath, "%s-%d",
g_queryInfo.specifiedQueryInfo.result[pThreadInfo->querySeq],
pThreadInfo->threadID);
- fetchResult(g_queryInfo.specifiedQueryInfo.res[pThreadInfo->threadID], pThreadInfo);
+ fetchResult(
+ g_queryInfo.specifiedQueryInfo.res[pThreadInfo->threadID],
+ pThreadInfo);
}
g_queryInfo.specifiedQueryInfo.consumed[pThreadInfo->threadID] ++;
- if ((g_queryInfo.specifiedQueryInfo.subscribeKeepProgress)
+ if ((g_queryInfo.specifiedQueryInfo.resubAfterConsume[pThreadInfo->querySeq] != -1)
&& (g_queryInfo.specifiedQueryInfo.consumed[pThreadInfo->threadID] >=
g_queryInfo.specifiedQueryInfo.resubAfterConsume[pThreadInfo->querySeq])) {
printf("keepProgress:%d, resub specified query: %"PRIu64"\n",
@@ -7218,16 +7448,17 @@ static void *specifiedSubscribe(void *sarg) {
g_queryInfo.specifiedQueryInfo.consumed[pThreadInfo->threadID] = 0;
taos_unsubscribe(g_queryInfo.specifiedQueryInfo.tsub[pThreadInfo->threadID],
g_queryInfo.specifiedQueryInfo.subscribeKeepProgress);
- g_queryInfo.specifiedQueryInfo.tsub[pThreadInfo->threadID] = subscribeImpl(
- SPECIFIED_CLASS,
- pThreadInfo,
- g_queryInfo.specifiedQueryInfo.sql[pThreadInfo->querySeq],
- g_queryInfo.specifiedQueryInfo.topic[pThreadInfo->threadID],
- g_queryInfo.specifiedQueryInfo.subscribeRestart,
- g_queryInfo.specifiedQueryInfo.subscribeInterval);
+ g_queryInfo.specifiedQueryInfo.tsub[pThreadInfo->threadID] =
+ subscribeImpl(
+ SPECIFIED_CLASS,
+ pThreadInfo,
+ g_queryInfo.specifiedQueryInfo.sql[pThreadInfo->querySeq],
+ g_queryInfo.specifiedQueryInfo.topic[pThreadInfo->threadID],
+ g_queryInfo.specifiedQueryInfo.subscribeRestart,
+ g_queryInfo.specifiedQueryInfo.subscribeInterval);
if (NULL == g_queryInfo.specifiedQueryInfo.tsub[pThreadInfo->threadID]) {
- taos_close(pThreadInfo->taos);
- return NULL;
+ taos_close(pThreadInfo->taos);
+ return NULL;
}
}
}
@@ -7276,12 +7507,12 @@ static int subscribeTestProcess() {
//==== create threads for query for specified table
if (g_queryInfo.specifiedQueryInfo.sqlCount <= 0) {
- debugPrint("%s() LN%d, sepcified query sqlCount %"PRIu64".\n",
+ debugPrint("%s() LN%d, sepcified query sqlCount %d.\n",
__func__, __LINE__,
g_queryInfo.specifiedQueryInfo.sqlCount);
} else {
if (g_queryInfo.specifiedQueryInfo.concurrent <= 0) {
- errorPrint("%s() LN%d, sepcified query sqlCount %"PRIu64".\n",
+ errorPrint("%s() LN%d, sepcified query sqlCount %d.\n",
__func__, __LINE__,
g_queryInfo.specifiedQueryInfo.sqlCount);
exit(-1);
@@ -7314,7 +7545,7 @@ static int subscribeTestProcess() {
//==== create threads for super table query
if (g_queryInfo.superQueryInfo.sqlCount <= 0) {
- debugPrint("%s() LN%d, super table query sqlCount %"PRIu64".\n",
+ debugPrint("%s() LN%d, super table query sqlCount %d.\n",
__func__, __LINE__,
g_queryInfo.superQueryInfo.sqlCount);
} else {
@@ -7440,7 +7671,7 @@ static void setParaFromArg(){
g_Dbs.threadCountByCreateTbl = g_args.num_of_threads;
g_Dbs.dbCount = 1;
- g_Dbs.db[0].drop = 1;
+ g_Dbs.db[0].drop = true;
tstrncpy(g_Dbs.db[0].dbName, g_args.database, MAX_DB_NAME_SIZE);
g_Dbs.db[0].dbCfg.replica = g_args.replica;
diff --git a/src/mnode/src/mnodeMain.c b/src/mnode/src/mnodeMain.c
index d3511a4e62b56ab70050f51ad606a7297a2be6a1..8ce798c8ec2271ecb236d49278f07f974e4bb043 100644
--- a/src/mnode/src/mnodeMain.c
+++ b/src/mnode/src/mnodeMain.c
@@ -121,7 +121,7 @@ int32_t mnodeStartSystem() {
int32_t mnodeInitSystem() {
mnodeInitTimer();
- if (mnodeNeedStart()) {
+ if (mnodeNeedStart() || tsCompactMnodeWal) {
return mnodeStartSystem();
}
return 0;
diff --git a/src/mnode/src/mnodeSdb.c b/src/mnode/src/mnodeSdb.c
index 84e4f33ca7efe2cafca9ec64d8b7872cac827403..ac3e5d86ecfd57b3b06f49b39bcb991ff06423d5 100644
--- a/src/mnode/src/mnodeSdb.c
+++ b/src/mnode/src/mnodeSdb.c
@@ -690,7 +690,7 @@ static int32_t sdbProcessWrite(void *wparam, void *hparam, int32_t qtype, void *
pthread_mutex_unlock(&tsSdbMgmt.mutex);
// from app, row is created
- if (pRow != NULL) {
+ if (pRow != NULL && tsCompactMnodeWal != 1) {
// forward to peers
pRow->processedCount = 0;
int32_t syncCode = syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC, false);
@@ -713,7 +713,9 @@ static int32_t sdbProcessWrite(void *wparam, void *hparam, int32_t qtype, void *
actStr[action], sdbGetKeyStr(pTable, pHead->cont), pHead->version);
// even it is WAL/FWD, it shall be called to update version in sync
- syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC, false);
+ if (tsCompactMnodeWal != 1) {
+ syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC, false);
+ }
// from wal or forward msg, row not created, should add into hash
if (action == SDB_ACTION_INSERT) {
diff --git a/src/mnode/src/mnodeTable.c b/src/mnode/src/mnodeTable.c
index db822c2b43b37c87cd5db5bd4f0a2d4caea9b03c..5710601e5c057282637bd6c67ac5d7b1e41d7341 100644
--- a/src/mnode/src/mnodeTable.c
+++ b/src/mnode/src/mnodeTable.c
@@ -3375,7 +3375,7 @@ static int32_t mnodeCompactSuperTables() {
.rowSize = sizeof(SSTableObj) + schemaSize,
};
- mInfo("compact super %" PRIu64, pTable->uid);
+ //mInfo("compact super %" PRIu64, pTable->uid);
sdbInsertCompactRow(&row);
}
@@ -3401,7 +3401,7 @@ static int32_t mnodeCompactChildTables() {
.pTable = tsChildTableSdb,
};
- mInfo("compact child %" PRIu64 ":%d", pTable->uid, pTable->tid);
+ //mInfo("compact child %" PRIu64 ":%d", pTable->uid, pTable->tid);
sdbInsertCompactRow(&row);
}
diff --git a/src/os/inc/osDir.h b/src/os/inc/osDir.h
index bdc65e452068cf34fd3969ada6ac69fbf5035217..7afe1264512bbffd34d9278ffb37034a473b827f 100644
--- a/src/os/inc/osDir.h
+++ b/src/os/inc/osDir.h
@@ -21,6 +21,7 @@ extern "C" {
#endif
void taosRemoveDir(char *rootDir);
+bool taosDirExist(const char* dirname);
int32_t taosMkDir(const char *pathname, mode_t mode);
void taosRemoveOldLogFiles(char *rootDir, int32_t keepDays);
int32_t taosRename(char *oldName, char *newName);
diff --git a/src/os/src/detail/osDir.c b/src/os/src/detail/osDir.c
index 2a2e2519b5dadf94ee8641064735e92267394a87..c467c64872d4b660af5cebb19b017f1528b55055 100644
--- a/src/os/src/detail/osDir.c
+++ b/src/os/src/detail/osDir.c
@@ -45,6 +45,10 @@ void taosRemoveDir(char *rootDir) {
uInfo("dir:%s is removed", rootDir);
}
+bool taosDirExist(const char* dirname) {
+ return access(dirname, F_OK) == 0;
+}
+
int taosMkDir(const char *path, mode_t mode) {
int code = mkdir(path, 0755);
if (code < 0 && errno == EEXIST) code = 0;
diff --git a/src/os/src/detail/osTime.c b/src/os/src/detail/osTime.c
index d9d070218e2a64e5f2535c073db4852b0a8960ef..67e0c2642e42f66229c437e603f7062edf571f34 100644
--- a/src/os/src/detail/osTime.c
+++ b/src/os/src/detail/osTime.c
@@ -87,12 +87,12 @@ static int32_t (*parseLocaltimeFp[]) (char* timestr, int64_t* time, int32_t time
int32_t taosGetTimestampSec() { return (int32_t)time(NULL); }
-int32_t taosParseTime(char* timestr, int64_t* time, int32_t len, int32_t timePrec, int8_t daylight) {
+int32_t taosParseTime(char* timestr, int64_t* time, int32_t len, int32_t timePrec, int8_t day_light) {
/* parse datatime string in with tz */
if (strnchr(timestr, 'T', len, false) != NULL) {
return parseTimeWithTz(timestr, time, timePrec);
} else {
- return (*parseLocaltimeFp[daylight])(timestr, time, timePrec);
+ return (*parseLocaltimeFp[day_light])(timestr, time, timePrec);
}
}
diff --git a/src/plugins/http/src/httpUtil.c b/src/plugins/http/src/httpUtil.c
index 399a33954d5670a7e928f856a898dcde1e4ac4eb..a8031d3fd85714d43735c5f18a60c6c388a7d69b 100644
--- a/src/plugins/http/src/httpUtil.c
+++ b/src/plugins/http/src/httpUtil.c
@@ -237,6 +237,11 @@ void httpFreeMultiCmds(HttpContext *pContext) {
JsonBuf *httpMallocJsonBuf(HttpContext *pContext) {
if (pContext->jsonBuf == NULL) {
pContext->jsonBuf = (JsonBuf *)malloc(sizeof(JsonBuf));
+ if (pContext->jsonBuf == NULL) {
+ return NULL;
+ }
+
+ memset(pContext->jsonBuf, 0, sizeof(JsonBuf));
}
if (!pContext->jsonBuf->pContext) {
diff --git a/src/query/src/qExecutor.c b/src/query/src/qExecutor.c
index 348d493a526b702e08915fe0ee5b548e5f82a987..4a0fb06b4878cb6ff3b6027cff4741fa2ea25658 100644
--- a/src/query/src/qExecutor.c
+++ b/src/query/src/qExecutor.c
@@ -33,6 +33,8 @@
#define SET_MASTER_SCAN_FLAG(runtime) ((runtime)->scanFlag = MASTER_SCAN)
#define SET_REVERSE_SCAN_FLAG(runtime) ((runtime)->scanFlag = REVERSE_SCAN)
+#define TSWINDOW_IS_EQUAL(t1, t2) (((t1).skey == (t2).skey) && ((t1).ekey == (t2).ekey))
+
#define SWITCH_ORDER(n) (((n) = ((n) == TSDB_ORDER_ASC) ? TSDB_ORDER_DESC : TSDB_ORDER_ASC))
#define SDATA_BLOCK_INITIALIZER (SDataBlockInfo) {{0}, 0}
@@ -1997,6 +1999,37 @@ static bool isFirstLastRowQuery(SQueryAttr *pQueryAttr) {
return false;
}
+static bool isCachedLastQuery(SQueryAttr *pQueryAttr) {
+ for (int32_t i = 0; i < pQueryAttr->numOfOutput; ++i) {
+ int32_t functionID = pQueryAttr->pExpr1[i].base.functionId;
+ if (functionID == TSDB_FUNC_LAST || functionID == TSDB_FUNC_LAST_DST) {
+ continue;
+ }
+
+ return false;
+ }
+
+ if (pQueryAttr->order.order != TSDB_ORDER_DESC || !TSWINDOW_IS_EQUAL(pQueryAttr->window, TSWINDOW_DESC_INITIALIZER)) {
+ return false;
+ }
+
+ if (pQueryAttr->groupbyColumn) {
+ return false;
+ }
+
+ if (pQueryAttr->interval.interval > 0) {
+ return false;
+ }
+
+ if (pQueryAttr->numOfFilterCols > 0 || pQueryAttr->havingNum > 0) {
+ return false;
+ }
+
+ return true;
+}
+
+
+
/**
* The following 4 kinds of query are treated as the tags query
* tagprj, tid_tag query, count(tbname), 'abc' (user defined constant value column) query
@@ -4022,6 +4055,8 @@ static int32_t setupQueryHandle(void* tsdb, SQueryRuntimeEnv* pRuntimeEnv, int64
}
}
}
+ } else if (isCachedLastQuery(pQueryAttr)) {
+ pRuntimeEnv->pQueryHandle = tsdbQueryCacheLast(tsdb, &cond, &pQueryAttr->tableGroupInfo, qId, &pQueryAttr->memRef);
} else if (pQueryAttr->pointInterpQuery) {
pRuntimeEnv->pQueryHandle = tsdbQueryRowsInExternalWindow(tsdb, &cond, &pQueryAttr->tableGroupInfo, qId, &pQueryAttr->memRef);
} else {
@@ -4267,7 +4302,7 @@ static SSDataBlock* doTableScan(void* param, bool *newgroup) {
}
if (++pTableScanInfo->current >= pTableScanInfo->times) {
- if (pTableScanInfo->reverseTimes <= 0) {
+ if (pTableScanInfo->reverseTimes <= 0 || isTsdbCacheLastRow(pTableScanInfo->pQueryHandle)) {
return NULL;
} else {
break;
diff --git a/src/tsdb/inc/tsdbCommit.h b/src/tsdb/inc/tsdbCommit.h
index 5e740081d187466435860a3a6c066412419ab571..6bd5dc432584bf1e7f8ed4274efdf014edf54e50 100644
--- a/src/tsdb/inc/tsdbCommit.h
+++ b/src/tsdb/inc/tsdbCommit.h
@@ -33,6 +33,7 @@ void tsdbGetRtnSnap(STsdbRepo *pRepo, SRtn *pRtn);
int tsdbEncodeKVRecord(void **buf, SKVRecord *pRecord);
void *tsdbDecodeKVRecord(void *buf, SKVRecord *pRecord);
void *tsdbCommitData(STsdbRepo *pRepo);
+int tsdbApplyRtn(STsdbRepo *pRepo);
static FORCE_INLINE int tsdbGetFidLevel(int fid, SRtn *pRtn) {
if (fid >= pRtn->maxFid) {
diff --git a/src/tsdb/inc/tsdbMeta.h b/src/tsdb/inc/tsdbMeta.h
index 7484071ce3c3bdb9e5aad449f22fc36d1809271a..45bbd5a7c6911fed4ea7309a77d3ac144109a34b 100644
--- a/src/tsdb/inc/tsdbMeta.h
+++ b/src/tsdb/inc/tsdbMeta.h
@@ -36,6 +36,12 @@ typedef struct STable {
char* sql;
void* cqhandle;
SRWLatch latch; // TODO: implementa latch functions
+
+ SDataCol *lastCols;
+ int16_t maxColNum;
+ int16_t restoreColumnNum;
+ bool hasRestoreLastColumn;
+ int lastColSVersion;
T_REF_DECLARE()
} STable;
@@ -78,6 +84,11 @@ void tsdbUnRefTable(STable* pTable);
void tsdbUpdateTableSchema(STsdbRepo* pRepo, STable* pTable, STSchema* pSchema, bool insertAct);
int tsdbRestoreTable(STsdbRepo* pRepo, void* cont, int contLen);
void tsdbOrgMeta(STsdbRepo* pRepo);
+int tsdbInitColIdCacheWithSchema(STable* pTable, STSchema* pSchema);
+int16_t tsdbGetLastColumnsIndexByColId(STable* pTable, int16_t colId);
+int tsdbUpdateLastColSchema(STable *pTable, STSchema *pNewSchema);
+STSchema* tsdbGetTableLatestSchema(STable *pTable);
+void tsdbFreeLastColumns(STable* pTable);
static FORCE_INLINE int tsdbCompareSchemaVersion(const void *key1, const void *key2) {
if (*(int16_t *)key1 < schemaVersion(*(STSchema **)key2)) {
diff --git a/src/tsdb/inc/tsdbint.h b/src/tsdb/inc/tsdbint.h
index 4d62164df9920c916e7a01cd67496711707b76e1..945b74af353bcf771c870a8e66fff7eb5f51c614 100644
--- a/src/tsdb/inc/tsdbint.h
+++ b/src/tsdb/inc/tsdbint.h
@@ -75,6 +75,9 @@ struct STsdbRepo {
STsdbCfg save_config; // save apply config
bool config_changed; // config changed flag
pthread_mutex_t save_mutex; // protect save config
+
+ uint8_t hasCachedLastRow;
+ uint8_t hasCachedLastColumn;
STsdbAppH appH;
STsdbStat stat;
@@ -83,6 +86,7 @@ struct STsdbRepo {
SMemTable* mem;
SMemTable* imem;
STsdbFS* fs;
+ SRtn rtn;
tsem_t readyToCommit;
pthread_mutex_t mutex;
bool repoLocked;
@@ -100,6 +104,7 @@ int tsdbUnlockRepo(STsdbRepo* pRepo);
STsdbMeta* tsdbGetMeta(STsdbRepo* pRepo);
int tsdbCheckCommit(STsdbRepo* pRepo);
int tsdbRestoreInfo(STsdbRepo* pRepo);
+int tsdbCacheLastData(STsdbRepo *pRepo, STsdbCfg* oldCfg);
void tsdbGetRootDir(int repoid, char dirName[]);
void tsdbGetDataDir(int repoid, char dirName[]);
diff --git a/src/tsdb/src/tsdbCommit.c b/src/tsdb/src/tsdbCommit.c
index acf34278fc253c82c26eb1f2cabc2ad91f2aac66..f2bbe347bd0bfa1f8f7ae5feb016dbfea13adfd7 100644
--- a/src/tsdb/src/tsdbCommit.c
+++ b/src/tsdb/src/tsdbCommit.c
@@ -86,10 +86,12 @@ static void tsdbCloseCommitFile(SCommitH *pCommith, bool hasError);
static bool tsdbCanAddSubBlock(SCommitH *pCommith, SBlock *pBlock, SMergeInfo *pInfo);
static void tsdbLoadAndMergeFromCache(SDataCols *pDataCols, int *iter, SCommitIter *pCommitIter, SDataCols *pTarget,
TSKEY maxKey, int maxRows, int8_t update);
-static int tsdbApplyRtn(STsdbRepo *pRepo);
static int tsdbApplyRtnOnFSet(STsdbRepo *pRepo, SDFileSet *pSet, SRtn *pRtn);
void *tsdbCommitData(STsdbRepo *pRepo) {
+ if (pRepo->imem == NULL) {
+ return NULL;
+ }
tsdbStartCommit(pRepo);
// Commit to update meta file
@@ -1428,7 +1430,7 @@ static bool tsdbCanAddSubBlock(SCommitH *pCommith, SBlock *pBlock, SMergeInfo *p
return false;
}
-static int tsdbApplyRtn(STsdbRepo *pRepo) {
+int tsdbApplyRtn(STsdbRepo *pRepo) {
SRtn rtn;
SFSIter fsiter;
STsdbFS * pfs = REPO_FS(pRepo);
diff --git a/src/tsdb/src/tsdbCommitQueue.c b/src/tsdb/src/tsdbCommitQueue.c
index e753a3211e5a789548ee6fc6f7caca6ccd3b59c3..abea79bc4f46129055af868af459c147dd93793f 100644
--- a/src/tsdb/src/tsdbCommitQueue.c
+++ b/src/tsdb/src/tsdbCommitQueue.c
@@ -113,11 +113,15 @@ int tsdbScheduleCommit(STsdbRepo *pRepo) {
}
static void tsdbApplyRepoConfig(STsdbRepo *pRepo) {
+ pthread_mutex_lock(&pRepo->save_mutex);
+
pRepo->config_changed = false;
STsdbCfg * pSaveCfg = &pRepo->save_config;
-
+ STsdbCfg oldCfg;
int32_t oldTotalBlocks = pRepo->config.totalBlocks;
+ memcpy(&oldCfg, &(pRepo->config), sizeof(STsdbCfg));
+
pRepo->config.compression = pRepo->save_config.compression;
pRepo->config.keep = pRepo->save_config.keep;
pRepo->config.keep1 = pRepo->save_config.keep1;
@@ -125,10 +129,12 @@ static void tsdbApplyRepoConfig(STsdbRepo *pRepo) {
pRepo->config.cacheLastRow = pRepo->save_config.cacheLastRow;
pRepo->config.totalBlocks = pRepo->save_config.totalBlocks;
- tsdbInfo("vgId:%d apply new config: compression(%d), keep(%d,%d,%d), totalBlocks(%d), cacheLastRow(%d),totalBlocks(%d)",
+ pthread_mutex_unlock(&pRepo->save_mutex);
+
+ tsdbInfo("vgId:%d apply new config: compression(%d), keep(%d,%d,%d), totalBlocks(%d), cacheLastRow(%d->%d),totalBlocks(%d->%d)",
REPO_ID(pRepo),
pSaveCfg->compression, pSaveCfg->keep,pSaveCfg->keep1, pSaveCfg->keep2,
- pSaveCfg->totalBlocks, pSaveCfg->cacheLastRow, pSaveCfg->totalBlocks);
+ pSaveCfg->totalBlocks, oldCfg.cacheLastRow, pSaveCfg->cacheLastRow, oldTotalBlocks, pSaveCfg->totalBlocks);
int err = tsdbExpendPool(pRepo, oldTotalBlocks);
if (!TAOS_SUCCEEDED(err)) {
@@ -136,6 +142,12 @@ static void tsdbApplyRepoConfig(STsdbRepo *pRepo) {
REPO_ID(pRepo), oldTotalBlocks, pSaveCfg->totalBlocks, tstrerror(err));
}
+ if (oldCfg.cacheLastRow != pRepo->config.cacheLastRow) {
+ if (tsdbLockRepo(pRepo) < 0) return;
+ tsdbCacheLastData(pRepo, &oldCfg);
+ tsdbUnlockRepo(pRepo);
+ }
+
}
static void *tsdbLoopCommit(void *arg) {
@@ -165,10 +177,8 @@ static void *tsdbLoopCommit(void *arg) {
pRepo = ((SCommitReq *)pNode->data)->pRepo;
// check if need to apply new config
- if (pRepo->config_changed) {
- pthread_mutex_lock(&pRepo->save_mutex);
+ if (pRepo->config_changed) {
tsdbApplyRepoConfig(pRepo);
- pthread_mutex_unlock(&pRepo->save_mutex);
}
tsdbCommitData(pRepo);
diff --git a/src/tsdb/src/tsdbFS.c b/src/tsdb/src/tsdbFS.c
index fd9b5e77e3fba01d49fc6f8f962730f1b8fbc9ec..54372ae8c28d91a72243256a74a8fb53c317eab2 100644
--- a/src/tsdb/src/tsdbFS.c
+++ b/src/tsdb/src/tsdbFS.c
@@ -33,7 +33,9 @@ static int tsdbScanDataDir(STsdbRepo *pRepo);
static bool tsdbIsTFileInFS(STsdbFS *pfs, const TFILE *pf);
static int tsdbRestoreCurrent(STsdbRepo *pRepo);
static int tsdbComparTFILE(const void *arg1, const void *arg2);
-static void tsdbScanAndTryFixDFilesHeader(STsdbRepo *pRepo);
+static void tsdbScanAndTryFixDFilesHeader(STsdbRepo *pRepo, int32_t *nExpired);
+static int tsdbProcessExpiredFS(STsdbRepo *pRepo);
+static int tsdbCreateMeta(STsdbRepo *pRepo);
// ================== CURRENT file header info
static int tsdbEncodeFSHeader(void **buf, SFSHeader *pHeader) {
@@ -212,6 +214,8 @@ STsdbFS *tsdbNewFS(STsdbCfg *pCfg) {
return NULL;
}
+ pfs->intxn = false;
+
pfs->nstatus = tsdbNewFSStatus(maxFSet);
if (pfs->nstatus == NULL) {
tsdbFreeFS(pfs);
@@ -234,22 +238,84 @@ void *tsdbFreeFS(STsdbFS *pfs) {
return NULL;
}
+static int tsdbProcessExpiredFS(STsdbRepo *pRepo) {
+ tsdbStartFSTxn(pRepo, 0, 0);
+ if (tsdbCreateMeta(pRepo) < 0) {
+ tsdbError("vgId:%d failed to create meta since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+
+ if (tsdbApplyRtn(pRepo) < 0) {
+ tsdbEndFSTxnWithError(REPO_FS(pRepo));
+ tsdbError("vgId:%d failed to apply rtn since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+ if (tsdbEndFSTxn(pRepo) < 0) {
+ tsdbError("vgId:%d failed to end fs txn since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+ return 0;
+}
+
+static int tsdbCreateMeta(STsdbRepo *pRepo) {
+ STsdbFS *pfs = REPO_FS(pRepo);
+ SMFile * pOMFile = pfs->cstatus->pmf;
+ SMFile mf;
+ SDiskID did;
+
+ if (pOMFile != NULL) {
+ // keep the old meta file
+ tsdbUpdateMFile(pfs, pOMFile);
+ return 0;
+ }
+
+ // Create a new meta file
+ did.level = TFS_PRIMARY_LEVEL;
+ did.id = TFS_PRIMARY_ID;
+ tsdbInitMFile(&mf, did, REPO_ID(pRepo), FS_TXN_VERSION(REPO_FS(pRepo)));
+
+ if (tsdbCreateMFile(&mf, true) < 0) {
+ tsdbError("vgId:%d failed to create META file since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+
+ tsdbInfo("vgId:%d meta file %s is created", REPO_ID(pRepo), TSDB_FILE_FULL_NAME(&mf));
+
+ if (tsdbUpdateMFileHeader(&mf) < 0) {
+ tsdbError("vgId:%d failed to update META file header since %s, revert it", REPO_ID(pRepo), tstrerror(terrno));
+ tsdbApplyMFileChange(&mf, pOMFile);
+ return -1;
+ }
+
+ TSDB_FILE_FSYNC(&mf);
+ tsdbCloseMFile(&mf);
+ tsdbUpdateMFile(pfs, &mf);
+
+ return 0;
+}
+
int tsdbOpenFS(STsdbRepo *pRepo) {
STsdbFS *pfs = REPO_FS(pRepo);
char current[TSDB_FILENAME_LEN] = "\0";
+ int nExpired = 0;
ASSERT(pfs != NULL);
tsdbGetTxnFname(REPO_ID(pRepo), TSDB_TXN_CURR_FILE, current);
+ tsdbGetRtnSnap(pRepo, &pRepo->rtn);
if (access(current, F_OK) == 0) {
if (tsdbOpenFSFromCurrent(pRepo) < 0) {
tsdbError("vgId:%d failed to open FS since %s", REPO_ID(pRepo), tstrerror(terrno));
return -1;
}
- tsdbScanAndTryFixDFilesHeader(pRepo);
+ tsdbScanAndTryFixDFilesHeader(pRepo, &nExpired);
+ if (nExpired > 0) {
+ tsdbProcessExpiredFS(pRepo);
+ }
} else {
+ // should skip expired fileset inside of the function
if (tsdbRestoreCurrent(pRepo) < 0) {
tsdbError("vgId:%d failed to restore current file since %s", REPO_ID(pRepo), tstrerror(terrno));
return -1;
@@ -1110,6 +1176,11 @@ static int tsdbRestoreDFileSet(STsdbRepo *pRepo) {
ASSERT(tvid == REPO_ID(pRepo));
+ if (tfid < pRepo->rtn.minFid) { // skip file expired
+ ++index;
+ continue;
+ }
+
if (ftype == 0) {
fset.fid = tfid;
} else {
@@ -1206,7 +1277,7 @@ static int tsdbComparTFILE(const void *arg1, const void *arg2) {
}
}
-static void tsdbScanAndTryFixDFilesHeader(STsdbRepo *pRepo) {
+static void tsdbScanAndTryFixDFilesHeader(STsdbRepo *pRepo, int32_t *nExpired) {
STsdbFS * pfs = REPO_FS(pRepo);
SFSStatus *pStatus = pfs->cstatus;
SDFInfo info;
@@ -1214,7 +1285,9 @@ static void tsdbScanAndTryFixDFilesHeader(STsdbRepo *pRepo) {
for (size_t i = 0; i < taosArrayGetSize(pStatus->df); i++) {
SDFileSet fset;
tsdbInitDFileSetEx(&fset, (SDFileSet *)taosArrayGet(pStatus->df, i));
-
+ if (fset.fid < pRepo->rtn.minFid) {
+ ++*nExpired;
+ }
tsdbDebug("vgId:%d scan DFileSet %d header", REPO_ID(pRepo), fset.fid);
if (tsdbOpenDFileSet(&fset, O_RDWR) < 0) {
diff --git a/src/tsdb/src/tsdbMain.c b/src/tsdb/src/tsdbMain.c
index fd02a3c8b97d7506209d92661143a158d8d94951..afbedd5b2fd231606902db104916e4ff4f10ba67 100644
--- a/src/tsdb/src/tsdbMain.c
+++ b/src/tsdb/src/tsdbMain.c
@@ -26,6 +26,8 @@ static STsdbRepo *tsdbNewRepo(STsdbCfg *pCfg, STsdbAppH *pAppH);
static void tsdbFreeRepo(STsdbRepo *pRepo);
static void tsdbStartStream(STsdbRepo *pRepo);
static void tsdbStopStream(STsdbRepo *pRepo);
+static int tsdbRestoreLastColumns(STsdbRepo *pRepo, STable *pTable, SReadH* pReadh);
+static int tsdbRestoreLastRow(STsdbRepo *pRepo, STable *pTable, SReadH* pReadh, SBlockIdx *pIdx);
// Function declaration
int32_t tsdbCreateRepo(int repoid) {
@@ -267,6 +269,10 @@ int32_t tsdbConfigRepo(STsdbRepo *repo, STsdbCfg *pCfg) {
repo->config_changed = true;
pthread_mutex_unlock(&repo->save_mutex);
+
+ // schedule a commit msg then the new config will be applied immediatly
+ tsdbAsyncCommit(repo);
+
return 0;
#if 0
STsdbRepo *pRepo = (STsdbRepo *)repo;
@@ -511,8 +517,10 @@ static int32_t tsdbCheckAndSetDefaultCfg(STsdbCfg *pCfg) {
if (pCfg->update != 0) pCfg->update = 1;
// update cacheLastRow
- if (pCfg->cacheLastRow != 0) pCfg->cacheLastRow = 1;
-
+ if (pCfg->cacheLastRow != 0) {
+ if (pCfg->cacheLastRow > 3)
+ pCfg->cacheLastRow = 1;
+ }
return 0;
}
@@ -545,6 +553,8 @@ static STsdbRepo *tsdbNewRepo(STsdbCfg *pCfg, STsdbAppH *pAppH) {
return NULL;
}
pRepo->config_changed = false;
+ atomic_store_8(&pRepo->hasCachedLastRow, 0);
+ atomic_store_8(&pRepo->hasCachedLastColumn, 0);
code = tsem_init(&(pRepo->readyToCommit), 0, 1);
if (code != 0) {
@@ -614,13 +624,180 @@ static void tsdbStopStream(STsdbRepo *pRepo) {
}
}
+static int tsdbRestoreLastColumns(STsdbRepo *pRepo, STable *pTable, SReadH* pReadh) {
+ //tsdbInfo("tsdbRestoreLastColumns of table %s", pTable->name->data);
+
+ STSchema *pSchema = tsdbGetTableLatestSchema(pTable);
+ if (pSchema == NULL) {
+ tsdbError("tsdbGetTableLatestSchema of table %s fail", pTable->name->data);
+ return 0;
+ }
+
+ SBlock* pBlock;
+ int numColumns;
+ int32_t blockIdx;
+ SDataStatis* pBlockStatis = NULL;
+ SDataRow row = NULL;
+ // restore last column data with last schema
+
+ int err = 0;
+
+ numColumns = schemaNCols(pSchema);
+ if (numColumns <= pTable->restoreColumnNum) {
+ pTable->hasRestoreLastColumn = true;
+ return 0;
+ }
+ if (pTable->lastColSVersion != schemaVersion(pSchema)) {
+ if (tsdbInitColIdCacheWithSchema(pTable, pSchema) < 0) {
+ return -1;
+ }
+ }
+
+ row = taosTMalloc(dataRowMaxBytesFromSchema(pSchema));
+ if (row == NULL) {
+ terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
+ err = -1;
+ goto out;
+ }
+ tdInitDataRow(row, pSchema);
+
+ // first load block index info
+ if (tsdbLoadBlockInfo(pReadh, NULL) < 0) {
+ err = -1;
+ goto out;
+ }
+
+ pBlockStatis = calloc(numColumns, sizeof(SDataStatis));
+ if (pBlockStatis == NULL) {
+ terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
+ err = -1;
+ goto out;
+ }
+ memset(pBlockStatis, 0, numColumns * sizeof(SDataStatis));
+ for(int32_t i = 0; i < numColumns; ++i) {
+ STColumn *pCol = schemaColAt(pSchema, i);
+ pBlockStatis[i].colId = pCol->colId;
+ }
+
+ // load block from backward
+ SBlockIdx *pIdx = pReadh->pBlkIdx;
+ blockIdx = (int32_t)(pIdx->numOfBlocks - 1);
+
+ while (numColumns > pTable->restoreColumnNum && blockIdx >= 0) {
+ bool loadStatisData = false;
+ pBlock = pReadh->pBlkInfo->blocks + blockIdx;
+ blockIdx -= 1;
+
+ // load block data
+ if (tsdbLoadBlockData(pReadh, pBlock, NULL) < 0) {
+ err = -1;
+ goto out;
+ }
+
+ // file block with sub-blocks has no statistics data
+ if (pBlock->numOfSubBlocks <= 1) {
+ tsdbLoadBlockStatis(pReadh, pBlock);
+ tsdbGetBlockStatis(pReadh, pBlockStatis, (int)numColumns);
+ loadStatisData = true;
+ }
+
+ for (int16_t i = 0; i < numColumns && numColumns > pTable->restoreColumnNum; ++i) {
+ STColumn *pCol = schemaColAt(pSchema, i);
+ // ignore loaded columns
+ if (pTable->lastCols[i].bytes != 0) {
+ continue;
+ }
+
+ // ignore block which has no not-null colId column
+ if (loadStatisData && pBlockStatis[i].numOfNull == pBlock->numOfRows) {
+ continue;
+ }
+
+ // OK,let's load row from backward to get not-null column
+ for (int32_t rowId = pBlock->numOfRows - 1; rowId >= 0; rowId--) {
+ SDataCol *pDataCol = pReadh->pDCols[0]->cols + i;
+ tdAppendColVal(row, tdGetColDataOfRow(pDataCol, rowId), pCol->type, pCol->bytes, pCol->offset);
+ //SDataCol *pDataCol = readh.pDCols[0]->cols + j;
+ void* value = tdGetRowDataOfCol(row, (int8_t)pCol->type, TD_DATA_ROW_HEAD_SIZE + pCol->offset);
+ if (isNull(value, pCol->type)) {
+ continue;
+ }
+
+ int16_t idx = tsdbGetLastColumnsIndexByColId(pTable, pCol->colId);
+ if (idx == -1) {
+ tsdbError("tsdbRestoreLastColumns restore vgId:%d,table:%s cache column %d fail", REPO_ID(pRepo), pTable->name->data, pCol->colId);
+ continue;
+ }
+ // save not-null column
+ SDataCol *pLastCol = &(pTable->lastCols[idx]);
+ pLastCol->pData = malloc(pCol->bytes);
+ pLastCol->bytes = pCol->bytes;
+ pLastCol->colId = pCol->colId;
+ memcpy(pLastCol->pData, value, pCol->bytes);
+
+ // save row ts(in column 0)
+ pDataCol = pReadh->pDCols[0]->cols + 0;
+ pCol = schemaColAt(pSchema, 0);
+ tdAppendColVal(row, tdGetColDataOfRow(pDataCol, rowId), pCol->type, pCol->bytes, pCol->offset);
+ pLastCol->ts = dataRowKey(row);
+
+ pTable->restoreColumnNum += 1;
+
+ tsdbDebug("tsdbRestoreLastColumns restore vgId:%d,table:%s cache column %d, %" PRId64, REPO_ID(pRepo), pTable->name->data, pLastCol->colId, pLastCol->ts);
+ break;
+ }
+ }
+ }
+
+out:
+ taosTZfree(row);
+ tfree(pBlockStatis);
+
+ if (err == 0 && numColumns <= pTable->restoreColumnNum) {
+ pTable->hasRestoreLastColumn = true;
+ }
+
+ return err;
+}
+
+static int tsdbRestoreLastRow(STsdbRepo *pRepo, STable *pTable, SReadH* pReadh, SBlockIdx *pIdx) {
+ ASSERT(pTable->lastRow == NULL);
+ if (tsdbLoadBlockInfo(pReadh, NULL) < 0) {
+ return -1;
+ }
+
+ SBlock* pBlock = pReadh->pBlkInfo->blocks + pIdx->numOfBlocks - 1;
+
+ if (tsdbLoadBlockData(pReadh, pBlock, NULL) < 0) {
+ return -1;
+ }
+
+ // Get the data in row
+
+ STSchema *pSchema = tsdbGetTableSchema(pTable);
+ pTable->lastRow = taosTMalloc(dataRowMaxBytesFromSchema(pSchema));
+ if (pTable->lastRow == NULL) {
+ terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
+ return -1;
+ }
+
+ tdInitDataRow(pTable->lastRow, pSchema);
+ for (int icol = 0; icol < schemaNCols(pSchema); icol++) {
+ STColumn *pCol = schemaColAt(pSchema, icol);
+ SDataCol *pDataCol = pReadh->pDCols[0]->cols + icol;
+ tdAppendColVal(pTable->lastRow, tdGetColDataOfRow(pDataCol, pBlock->numOfRows - 1), pCol->type, pCol->bytes,
+ pCol->offset);
+ }
+
+ return 0;
+}
+
int tsdbRestoreInfo(STsdbRepo *pRepo) {
SFSIter fsiter;
SReadH readh;
SDFileSet *pSet;
STsdbMeta *pMeta = pRepo->tsdbMeta;
STsdbCfg * pCfg = REPO_CFG(pRepo);
- SBlock * pBlock;
if (tsdbInitReadH(&readh, pRepo) < 0) {
return -1;
@@ -628,6 +805,14 @@ int tsdbRestoreInfo(STsdbRepo *pRepo) {
tsdbFSIterInit(&fsiter, REPO_FS(pRepo), TSDB_FS_ITER_BACKWARD);
+ if (CACHE_LAST_NULL_COLUMN(pCfg)) {
+ for (int i = 1; i < pMeta->maxTables; i++) {
+ STable *pTable = pMeta->tables[i];
+ if (pTable == NULL) continue;
+ pTable->restoreColumnNum = 0;
+ }
+ }
+
while ((pSet = tsdbFSIterNext(&fsiter)) != NULL) {
if (tsdbSetAndOpenReadFSet(&readh, pSet) < 0) {
tsdbDestroyReadH(&readh);
@@ -643,6 +828,8 @@ int tsdbRestoreInfo(STsdbRepo *pRepo) {
STable *pTable = pMeta->tables[i];
if (pTable == NULL) continue;
+ //tsdbInfo("tsdbRestoreInfo restore vgId:%d,table:%s", REPO_ID(pRepo), pTable->name->data);
+
if (tsdbSetReadTable(&readh, pTable) < 0) {
tsdbDestroyReadH(&readh);
return -1;
@@ -653,42 +840,155 @@ int tsdbRestoreInfo(STsdbRepo *pRepo) {
if (pIdx && lastKey < pIdx->maxKey) {
pTable->lastKey = pIdx->maxKey;
- if (pCfg->cacheLastRow) {
- if (tsdbLoadBlockInfo(&readh, NULL) < 0) {
- tsdbDestroyReadH(&readh);
- return -1;
- }
-
- pBlock = readh.pBlkInfo->blocks + pIdx->numOfBlocks - 1;
-
- if (tsdbLoadBlockData(&readh, pBlock, NULL) < 0) {
- tsdbDestroyReadH(&readh);
- return -1;
- }
-
- // Get the data in row
- ASSERT(pTable->lastRow == NULL);
- STSchema *pSchema = tsdbGetTableSchema(pTable);
- pTable->lastRow = taosTMalloc(dataRowMaxBytesFromSchema(pSchema));
- if (pTable->lastRow == NULL) {
- terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
- tsdbDestroyReadH(&readh);
- return -1;
- }
-
- tdInitDataRow(pTable->lastRow, pSchema);
- for (int icol = 0; icol < schemaNCols(pSchema); icol++) {
- STColumn *pCol = schemaColAt(pSchema, icol);
- SDataCol *pDataCol = readh.pDCols[0]->cols + icol;
- tdAppendColVal(pTable->lastRow, tdGetColDataOfRow(pDataCol, pBlock->numOfRows - 1), pCol->type, pCol->bytes,
- pCol->offset);
- }
+ if (CACHE_LAST_ROW(pCfg) && tsdbRestoreLastRow(pRepo, pTable, &readh, pIdx) != 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
}
}
+ // restore NULL columns
+ if (pIdx && CACHE_LAST_NULL_COLUMN(pCfg) && !pTable->hasRestoreLastColumn) {
+ if (tsdbRestoreLastColumns(pRepo, pTable, &readh) != 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+ }
}
}
tsdbDestroyReadH(&readh);
+ if (CACHE_LAST_ROW(pCfg)) {
+ atomic_store_8(&pRepo->hasCachedLastRow, 1);
+ }
+ if (CACHE_LAST_NULL_COLUMN(pCfg)) {
+ atomic_store_8(&pRepo->hasCachedLastColumn, 1);
+ }
+
return 0;
}
+
+int tsdbCacheLastData(STsdbRepo *pRepo, STsdbCfg* oldCfg) {
+ bool cacheLastRow = false, cacheLastCol = false;
+ SFSIter fsiter;
+ SReadH readh;
+ SDFileSet *pSet;
+ STsdbMeta *pMeta = pRepo->tsdbMeta;
+ int tableNum = 0;
+ int maxTableIdx = 0;
+ int cacheLastRowTableNum = 0;
+ int cacheLastColTableNum = 0;
+
+ bool need_free_last_row = CACHE_LAST_ROW(oldCfg) && !CACHE_LAST_ROW(&(pRepo->config));
+ bool need_free_last_col = CACHE_LAST_NULL_COLUMN(oldCfg) && !CACHE_LAST_NULL_COLUMN(&(pRepo->config));
+
+ if (CACHE_LAST_ROW(&(pRepo->config)) || CACHE_LAST_NULL_COLUMN(&(pRepo->config))) {
+ tsdbInfo("tsdbCacheLastData cache last data since cacheLast option changed");
+ cacheLastRow = !CACHE_LAST_ROW(oldCfg) && CACHE_LAST_ROW(&(pRepo->config));
+ cacheLastCol = !CACHE_LAST_NULL_COLUMN(oldCfg) && CACHE_LAST_NULL_COLUMN(&(pRepo->config));
+ }
+
+ // calc max table idx and table num
+ for (int i = 1; i < pMeta->maxTables; i++) {
+ STable *pTable = pMeta->tables[i];
+ if (pTable == NULL) continue;
+ tableNum += 1;
+ maxTableIdx = i;
+ if (cacheLastCol) {
+ pTable->restoreColumnNum = 0;
+ }
+ }
+
+ // if close last option,need to free data
+ if (need_free_last_row || need_free_last_col) {
+ if (need_free_last_row) {
+ atomic_store_8(&pRepo->hasCachedLastRow, 0);
+ }
+ if (need_free_last_col) {
+ atomic_store_8(&pRepo->hasCachedLastColumn, 0);
+ }
+ tsdbInfo("free cache last data since cacheLast option changed");
+ for (int i = 1; i < maxTableIdx; i++) {
+ STable *pTable = pMeta->tables[i];
+ if (pTable == NULL) continue;
+ if (need_free_last_row) {
+ taosTZfree(pTable->lastRow);
+ pTable->lastRow = NULL;
+ pTable->lastKey = TSKEY_INITIAL_VAL;
+ }
+ if (need_free_last_col) {
+ tsdbFreeLastColumns(pTable);
+ }
+ }
+ }
+
+ if (!cacheLastRow && !cacheLastCol) {
+ return 0;
+ }
+
+ cacheLastRowTableNum = cacheLastRow ? tableNum : 0;
+ cacheLastColTableNum = cacheLastCol ? tableNum : 0;
+
+ if (tsdbInitReadH(&readh, pRepo) < 0) {
+ return -1;
+ }
+
+ tsdbFSIterInit(&fsiter, REPO_FS(pRepo), TSDB_FS_ITER_BACKWARD);
+
+ while ((pSet = tsdbFSIterNext(&fsiter)) != NULL && (cacheLastRowTableNum > 0 || cacheLastColTableNum > 0)) {
+ if (tsdbSetAndOpenReadFSet(&readh, pSet) < 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+
+ if (tsdbLoadBlockIdx(&readh) < 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+
+ for (int i = 1; i <= maxTableIdx; i++) {
+ STable *pTable = pMeta->tables[i];
+ if (pTable == NULL) continue;
+
+ //tsdbInfo("tsdbRestoreInfo restore vgId:%d,table:%s", REPO_ID(pRepo), pTable->name->data);
+
+ if (tsdbSetReadTable(&readh, pTable) < 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+
+ SBlockIdx *pIdx = readh.pBlkIdx;
+
+ if (pIdx && cacheLastRowTableNum > 0 && pTable->lastRow == NULL) {
+ pTable->lastKey = pIdx->maxKey;
+
+ if (tsdbRestoreLastRow(pRepo, pTable, &readh, pIdx) != 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+ cacheLastRowTableNum -= 1;
+ }
+
+ // restore NULL columns
+ if (pIdx && cacheLastColTableNum > 0 && !pTable->hasRestoreLastColumn) {
+ if (tsdbRestoreLastColumns(pRepo, pTable, &readh) != 0) {
+ tsdbDestroyReadH(&readh);
+ return -1;
+ }
+ if (pTable->hasRestoreLastColumn) {
+ cacheLastColTableNum -= 1;
+ }
+ }
+ }
+ }
+
+ tsdbDestroyReadH(&readh);
+
+ if (cacheLastRow) {
+ atomic_store_8(&pRepo->hasCachedLastRow, 1);
+ }
+ if (cacheLastCol) {
+ atomic_store_8(&pRepo->hasCachedLastColumn, 1);
+ }
+
+ return 0;
+}
\ No newline at end of file
diff --git a/src/tsdb/src/tsdbMemTable.c b/src/tsdb/src/tsdbMemTable.c
index c6fcf55686850432a90d82e1f50c856f29c8eac0..79dbb8be5d8719d7a9df7fe71695d013dffdefc1 100644
--- a/src/tsdb/src/tsdbMemTable.c
+++ b/src/tsdb/src/tsdbMemTable.c
@@ -274,7 +274,7 @@ void *tsdbAllocBytes(STsdbRepo *pRepo, int bytes) {
int tsdbAsyncCommit(STsdbRepo *pRepo) {
tsem_wait(&(pRepo->readyToCommit));
- ASSERT(pRepo->imem == NULL);
+ //ASSERT(pRepo->imem == NULL);
if (pRepo->mem == NULL) {
tsem_post(&(pRepo->readyToCommit));
return 0;
@@ -964,6 +964,49 @@ static void tsdbFreeRows(STsdbRepo *pRepo, void **rows, int rowCounter) {
}
}
+static void updateTableLatestColumn(STsdbRepo *pRepo, STable *pTable, SDataRow row) {
+ tsdbDebug("vgId:%d updateTableLatestColumn, %s row version:%d", REPO_ID(pRepo), pTable->name->data, dataRowVersion(row));
+
+ STSchema* pSchema = tsdbGetTableLatestSchema(pTable);
+ if (tsdbUpdateLastColSchema(pTable, pSchema) < 0) {
+ return;
+ }
+
+ pSchema = tsdbGetTableSchemaByVersion(pTable, dataRowVersion(row));
+ if (pSchema == NULL) {
+ return;
+ }
+
+ SDataCol *pLatestCols = pTable->lastCols;
+
+ for (int16_t j = 0; j < schemaNCols(pSchema); j++) {
+ STColumn *pTCol = schemaColAt(pSchema, j);
+ // ignore not exist colId
+ int16_t idx = tsdbGetLastColumnsIndexByColId(pTable, pTCol->colId);
+ if (idx == -1) {
+ continue;
+ }
+
+ void* value = tdGetRowDataOfCol(row, (int8_t)pTCol->type, TD_DATA_ROW_HEAD_SIZE + pSchema->columns[j].offset);
+ if (isNull(value, pTCol->type)) {
+ continue;
+ }
+
+ SDataCol *pDataCol = &(pLatestCols[idx]);
+ if (pDataCol->pData == NULL) {
+ pDataCol->pData = malloc(pSchema->columns[j].bytes);
+ pDataCol->bytes = pSchema->columns[j].bytes;
+ } else if (pDataCol->bytes < pSchema->columns[j].bytes) {
+ pDataCol->pData = realloc(pDataCol->pData, pSchema->columns[j].bytes);
+ pDataCol->bytes = pSchema->columns[j].bytes;
+ }
+
+ memcpy(pDataCol->pData, value, pDataCol->bytes);
+ //tsdbInfo("updateTableLatestColumn vgId:%d cache column %d for %d,%s", REPO_ID(pRepo), j, pDataCol->bytes, (char*)pDataCol->pData);
+ pDataCol->ts = dataRowKey(row);
+ }
+}
+
static int tsdbUpdateTableLatestInfo(STsdbRepo *pRepo, STable *pTable, SDataRow row) {
STsdbCfg *pCfg = &pRepo->config;
@@ -977,7 +1020,7 @@ static int tsdbUpdateTableLatestInfo(STsdbRepo *pRepo, STable *pTable, SDataRow
}
if (tsdbGetTableLastKeyImpl(pTable) < dataRowKey(row)) {
- if (pCfg->cacheLastRow || pTable->lastRow != NULL) {
+ if (CACHE_LAST_ROW(pCfg) || pTable->lastRow != NULL) {
SDataRow nrow = pTable->lastRow;
if (taosTSizeof(nrow) < dataRowLen(row)) {
SDataRow orow = nrow;
@@ -1002,7 +1045,10 @@ static int tsdbUpdateTableLatestInfo(STsdbRepo *pRepo, STable *pTable, SDataRow
} else {
pTable->lastKey = dataRowKey(row);
}
- }
+ if (CACHE_LAST_NULL_COLUMN(pCfg)) {
+ updateTableLatestColumn(pRepo, pTable, row);
+ }
+ }
return 0;
}
diff --git a/src/tsdb/src/tsdbMeta.c b/src/tsdb/src/tsdbMeta.c
index e6cbc4da9e6a3d2000e395e8ae8b9a52a01d1f6c..324a7c79c5b7dbfa69bbdf240301c3c710f90b59 100644
--- a/src/tsdb/src/tsdbMeta.c
+++ b/src/tsdb/src/tsdbMeta.c
@@ -589,6 +589,131 @@ void tsdbUnRefTable(STable *pTable) {
}
}
+void tsdbFreeLastColumns(STable* pTable) {
+ if (pTable->lastCols == NULL) {
+ return;
+ }
+
+ for (int i = 0; i < pTable->maxColNum; ++i) {
+ if (pTable->lastCols[i].bytes == 0) {
+ continue;
+ }
+ tfree(pTable->lastCols[i].pData);
+ pTable->lastCols[i].bytes = 0;
+ pTable->lastCols[i].pData = NULL;
+ }
+ tfree(pTable->lastCols);
+ pTable->lastCols = NULL;
+ pTable->maxColNum = 0;
+ pTable->lastColSVersion = -1;
+ pTable->restoreColumnNum = 0;
+}
+
+int16_t tsdbGetLastColumnsIndexByColId(STable* pTable, int16_t colId) {
+ if (pTable->lastCols == NULL) {
+ return -1;
+ }
+ for (int16_t i = 0; i < pTable->maxColNum; ++i) {
+ if (pTable->lastCols[i].colId == colId) {
+ return i;
+ }
+ }
+
+ return -1;
+}
+
+int tsdbInitColIdCacheWithSchema(STable* pTable, STSchema* pSchema) {
+ ASSERT(pTable->lastCols == NULL);
+
+ int16_t numOfColumn = pSchema->numOfCols;
+
+ pTable->lastCols = (SDataCol*)malloc(numOfColumn * sizeof(SDataCol));
+ if (pTable->lastCols == NULL) {
+ return -1;
+ }
+
+ for (int16_t i = 0; i < numOfColumn; ++i) {
+ STColumn *pCol = schemaColAt(pSchema, i);
+ SDataCol* pDataCol = &(pTable->lastCols[i]);
+ pDataCol->bytes = 0;
+ pDataCol->pData = NULL;
+ pDataCol->colId = pCol->colId;
+ }
+
+ pTable->lastColSVersion = schemaVersion(pSchema);
+ pTable->maxColNum = numOfColumn;
+ pTable->restoreColumnNum = 0;
+ return 0;
+}
+
+STSchema* tsdbGetTableLatestSchema(STable *pTable) {
+ return tsdbGetTableSchemaByVersion(pTable, -1);
+}
+
+int tsdbUpdateLastColSchema(STable *pTable, STSchema *pNewSchema) {
+ if (pTable->lastColSVersion == schemaVersion(pNewSchema)) {
+ return 0;
+ }
+
+ tsdbInfo("tsdbUpdateLastColSchema:%s,%d->%d", pTable->name->data, pTable->lastColSVersion, schemaVersion(pNewSchema));
+
+ int16_t numOfCols = pNewSchema->numOfCols;
+ SDataCol *lastCols = (SDataCol*)malloc(numOfCols * sizeof(SDataCol));
+ if (lastCols == NULL) {
+ return -1;
+ }
+
+ TSDB_WLOCK_TABLE(pTable);
+
+ for (int16_t i = 0; i < numOfCols; ++i) {
+ STColumn *pCol = schemaColAt(pNewSchema, i);
+ int16_t idx = tsdbGetLastColumnsIndexByColId(pTable, pCol->colId);
+
+ SDataCol* pDataCol = &(lastCols[i]);
+ if (idx != -1) {
+ // move col data to new last column array
+ SDataCol* pOldDataCol = &(pTable->lastCols[idx]);
+ memcpy(pDataCol, pOldDataCol, sizeof(SDataCol));
+ } else {
+ // init new colid data
+ pDataCol->colId = pCol->colId;
+ pDataCol->bytes = 0;
+ pDataCol->pData = NULL;
+ }
+ }
+
+ SDataCol *oldLastCols = pTable->lastCols;
+ int16_t oldLastColNum = pTable->maxColNum;
+
+ pTable->lastColSVersion = schemaVersion(pNewSchema);
+ pTable->lastCols = lastCols;
+ pTable->maxColNum = numOfCols;
+
+ if (oldLastCols == NULL) {
+ TSDB_WUNLOCK_TABLE(pTable);
+ return 0;
+ }
+
+ // free old schema last column datas
+ for (int16_t i = 0; i < oldLastColNum; ++i) {
+ SDataCol* pDataCol = &(oldLastCols[i]);
+ if (pDataCol->bytes == 0) {
+ continue;
+ }
+ int16_t idx = tsdbGetLastColumnsIndexByColId(pTable, pDataCol->colId);
+ if (idx != -1) {
+ continue;
+ }
+
+ // free not exist column data
+ tfree(pDataCol->pData);
+ }
+ TSDB_WUNLOCK_TABLE(pTable);
+ tfree(oldLastCols);
+
+ return 0;
+}
+
void tsdbUpdateTableSchema(STsdbRepo *pRepo, STable *pTable, STSchema *pSchema, bool insertAct) {
ASSERT(TABLE_TYPE(pTable) != TSDB_STREAM_TABLE && TABLE_TYPE(pTable) != TSDB_SUPER_TABLE);
STsdbMeta *pMeta = pRepo->tsdbMeta;
@@ -672,6 +797,10 @@ static STable *tsdbNewTable() {
pTable->lastKey = TSKEY_INITIAL_VAL;
+ pTable->lastCols = NULL;
+ pTable->restoreColumnNum = 0;
+ pTable->maxColNum = 0;
+ pTable->lastColSVersion = -1;
return pTable;
}
@@ -785,8 +914,10 @@ static void tsdbFreeTable(STable *pTable) {
kvRowFree(pTable->tagVal);
tSkipListDestroy(pTable->pIndex);
- taosTZfree(pTable->lastRow);
+ taosTZfree(pTable->lastRow);
tfree(pTable->sql);
+
+ tsdbFreeLastColumns(pTable);
free(pTable);
}
}
diff --git a/src/tsdb/src/tsdbRead.c b/src/tsdb/src/tsdbRead.c
index 9df25409de62f9b2579d43800fd6a5709bd41f1a..0a39dba021c92510e9e6d5f1420fe88792645cad 100644
--- a/src/tsdb/src/tsdbRead.c
+++ b/src/tsdb/src/tsdbRead.c
@@ -62,6 +62,7 @@ typedef struct SLoadCompBlockInfo {
int32_t fileId;
} SLoadCompBlockInfo;
+
typedef struct STableCheckInfo {
STableId tableId;
TSKEY lastKey;
@@ -107,7 +108,7 @@ typedef struct STsdbQueryHandle {
SArray* pTableCheckInfo; // SArray
int32_t activeIndex;
bool checkFiles; // check file stage
- bool cachelastrow; // check if last row cached
+ int8_t cachelastrow; // check if last row cached
bool loadExternalRow; // load time window external data rows
bool currentLoadExternalRows; // current load external rows
int32_t loadType; // block load type
@@ -117,7 +118,6 @@ typedef struct STsdbQueryHandle {
SFSIter fileIter;
SReadH rhelper;
STableBlockInfo* pDataBlockInfo;
-
SDataCols *pDataCols; // in order to hold current file data block
int32_t allocSize; // allocated data block size
SMemRef *pMemRef;
@@ -138,6 +138,7 @@ typedef struct STableGroupSupporter {
static STimeWindow updateLastrowForEachGroup(STableGroupInfo *groupList);
static int32_t checkForCachedLastRow(STsdbQueryHandle* pQueryHandle, STableGroupInfo *groupList);
+static int32_t checkForCachedLast(STsdbQueryHandle* pQueryHandle);
static int32_t tsdbGetCachedLastRow(STable* pTable, SDataRow* pRes, TSKEY* lastKey);
static void changeQueryHandleForInterpQuery(TsdbQueryHandleT pHandle);
@@ -512,6 +513,8 @@ void tsdbResetQueryHandleForNewTable(TsdbQueryHandleT queryHandle, STsdbQueryCon
pQueryHandle->next = doFreeColumnInfoData(pQueryHandle->next);
}
+
+
TsdbQueryHandleT tsdbQueryLastRow(STsdbRepo *tsdb, STsdbQueryCond *pCond, STableGroupInfo *groupList, uint64_t qId, SMemRef* pMemRef) {
pCond->twindow = updateLastrowForEachGroup(groupList);
@@ -528,10 +531,30 @@ TsdbQueryHandleT tsdbQueryLastRow(STsdbRepo *tsdb, STsdbQueryCond *pCond, STable
}
assert(pCond->order == TSDB_ORDER_ASC && pCond->twindow.skey <= pCond->twindow.ekey);
- pQueryHandle->type = TSDB_QUERY_TYPE_LAST;
+ if (pQueryHandle->cachelastrow) {
+ pQueryHandle->type = TSDB_QUERY_TYPE_LAST;
+ }
+
+ return pQueryHandle;
+}
+
+
+TsdbQueryHandleT tsdbQueryCacheLast(STsdbRepo *tsdb, STsdbQueryCond *pCond, STableGroupInfo *groupList, uint64_t qId, SMemRef* pMemRef) {
+ STsdbQueryHandle *pQueryHandle = (STsdbQueryHandle*) tsdbQueryTables(tsdb, pCond, groupList, qId, pMemRef);
+ int32_t code = checkForCachedLast(pQueryHandle);
+ if (code != TSDB_CODE_SUCCESS) { // set the numOfTables to be 0
+ terrno = code;
+ return NULL;
+ }
+
+ if (pQueryHandle->cachelastrow) {
+ pQueryHandle->type = TSDB_QUERY_TYPE_LAST;
+ }
+
return pQueryHandle;
}
+
SArray* tsdbGetQueriedTableList(TsdbQueryHandleT *pHandle) {
assert(pHandle != NULL);
@@ -2460,6 +2483,159 @@ static bool loadCachedLastRow(STsdbQueryHandle* pQueryHandle) {
return false;
}
+
+
+static bool loadCachedLast(STsdbQueryHandle* pQueryHandle) {
+ // the last row is cached in buffer, return it directly.
+ // here note that the pQueryHandle->window must be the TS_INITIALIZER
+ int32_t tgNumOfCols = (int32_t)QH_GET_NUM_OF_COLS(pQueryHandle);
+ size_t numOfTables = taosArrayGetSize(pQueryHandle->pTableCheckInfo);
+ int32_t numOfRows = 0;
+ assert(numOfTables > 0 && tgNumOfCols > 0);
+ SQueryFilePos* cur = &pQueryHandle->cur;
+ TSKEY priKey = TSKEY_INITIAL_VAL;
+ int32_t priIdx = -1;
+ SColumnInfoData* pColInfo = NULL;
+
+ while (++pQueryHandle->activeIndex < numOfTables) {
+ STableCheckInfo* pCheckInfo = taosArrayGet(pQueryHandle->pTableCheckInfo, pQueryHandle->activeIndex);
+ STable* pTable = pCheckInfo->pTableObj;
+ char* pData = NULL;
+
+ int32_t numOfCols = pTable->maxColNum;
+
+ if (pTable->lastCols == NULL || pTable->maxColNum <= 0) {
+ tsdbWarn("no last cached for table, uid:%" PRIu64 ",tid:%d", pTable->tableId.uid, pTable->tableId.tid);
+ continue;
+ }
+
+ int32_t i = 0, j = 0;
+ while(i < tgNumOfCols && j < numOfCols) {
+ pColInfo = taosArrayGet(pQueryHandle->pColumns, i);
+ if (pTable->lastCols[j].colId < pColInfo->info.colId) {
+ j++;
+ continue;
+ } else if (pTable->lastCols[j].colId > pColInfo->info.colId) {
+ i++;
+ continue;
+ }
+
+ pData = (char*)pColInfo->pData + numOfRows * pColInfo->info.bytes;
+
+ if (pTable->lastCols[j].bytes > 0) {
+ void* value = pTable->lastCols[j].pData;
+ switch (pColInfo->info.type) {
+ case TSDB_DATA_TYPE_BINARY:
+ case TSDB_DATA_TYPE_NCHAR:
+ memcpy(pData, value, varDataTLen(value));
+ break;
+ case TSDB_DATA_TYPE_NULL:
+ case TSDB_DATA_TYPE_BOOL:
+ case TSDB_DATA_TYPE_TINYINT:
+ case TSDB_DATA_TYPE_UTINYINT:
+ *(uint8_t *)pData = *(uint8_t *)value;
+ break;
+ case TSDB_DATA_TYPE_SMALLINT:
+ case TSDB_DATA_TYPE_USMALLINT:
+ *(uint16_t *)pData = *(uint16_t *)value;
+ break;
+ case TSDB_DATA_TYPE_INT:
+ case TSDB_DATA_TYPE_UINT:
+ *(uint32_t *)pData = *(uint32_t *)value;
+ break;
+ case TSDB_DATA_TYPE_BIGINT:
+ case TSDB_DATA_TYPE_UBIGINT:
+ *(uint64_t *)pData = *(uint64_t *)value;
+ break;
+ case TSDB_DATA_TYPE_FLOAT:
+ SET_FLOAT_PTR(pData, value);
+ break;
+ case TSDB_DATA_TYPE_DOUBLE:
+ SET_DOUBLE_PTR(pData, value);
+ break;
+ case TSDB_DATA_TYPE_TIMESTAMP:
+ if (pColInfo->info.colId == PRIMARYKEY_TIMESTAMP_COL_INDEX) {
+ priKey = tdGetKey(*(TKEY *)value);
+ priIdx = i;
+
+ i++;
+ j++;
+ continue;
+ } else {
+ *(TSKEY *)pData = *(TSKEY *)value;
+ }
+ break;
+ default:
+ memcpy(pData, value, pColInfo->info.bytes);
+ }
+
+ for (int32_t n = 0; n < tgNumOfCols; ++n) {
+ if (n == i) {
+ continue;
+ }
+
+ pColInfo = taosArrayGet(pQueryHandle->pColumns, n);
+ pData = (char*)pColInfo->pData + numOfRows * pColInfo->info.bytes;;
+
+ if (pColInfo->info.colId == PRIMARYKEY_TIMESTAMP_COL_INDEX) {
+ *(TSKEY *)pData = pTable->lastCols[j].ts;
+ continue;
+ }
+
+ if (pColInfo->info.type == TSDB_DATA_TYPE_BINARY || pColInfo->info.type == TSDB_DATA_TYPE_NCHAR) {
+ setVardataNull(pData, pColInfo->info.type);
+ } else {
+ setNull(pData, pColInfo->info.type, pColInfo->info.bytes);
+ }
+ }
+
+ numOfRows++;
+ assert(numOfRows < pQueryHandle->outputCapacity);
+ }
+
+ i++;
+ j++;
+ }
+
+ // leave the real ts column as the last row, because last function only (not stable) use the last row as res
+ if (priKey != TSKEY_INITIAL_VAL) {
+ pColInfo = taosArrayGet(pQueryHandle->pColumns, priIdx);
+ pData = (char*)pColInfo->pData + numOfRows * pColInfo->info.bytes;
+
+ *(TSKEY *)pData = priKey;
+
+ for (int32_t n = 0; n < tgNumOfCols; ++n) {
+ if (n == priIdx) {
+ continue;
+ }
+
+ pColInfo = taosArrayGet(pQueryHandle->pColumns, n);
+ pData = (char*)pColInfo->pData + numOfRows * pColInfo->info.bytes;;
+
+ assert (pColInfo->info.colId != PRIMARYKEY_TIMESTAMP_COL_INDEX);
+
+ if (pColInfo->info.type == TSDB_DATA_TYPE_BINARY || pColInfo->info.type == TSDB_DATA_TYPE_NCHAR) {
+ setVardataNull(pData, pColInfo->info.type);
+ } else {
+ setNull(pData, pColInfo->info.type, pColInfo->info.bytes);
+ }
+ }
+
+ numOfRows++;
+ }
+
+ if (numOfRows > 0) {
+ cur->rows = numOfRows;
+ cur->mixBlock = true;
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
+
static bool loadDataBlockFromTableSeq(STsdbQueryHandle* pQueryHandle) {
size_t numOfTables = taosArrayGetSize(pQueryHandle->pTableCheckInfo);
assert(numOfTables > 0);
@@ -2496,8 +2672,12 @@ bool tsdbNextDataBlock(TsdbQueryHandleT pHandle) {
int64_t stime = taosGetTimestampUs();
int64_t elapsedTime = stime;
- if (pQueryHandle->type == TSDB_QUERY_TYPE_LAST && pQueryHandle->cachelastrow) {
- return loadCachedLastRow(pQueryHandle);
+ if (pQueryHandle->type == TSDB_QUERY_TYPE_LAST) {
+ if (pQueryHandle->cachelastrow == 1) {
+ return loadCachedLastRow(pQueryHandle);
+ } else if (pQueryHandle->cachelastrow == 2) {
+ return loadCachedLast(pQueryHandle);
+ }
}
if (pQueryHandle->loadType == BLOCK_LOAD_TABLE_SEQ_ORDER) {
@@ -2695,6 +2875,10 @@ int32_t tsdbGetCachedLastRow(STable* pTable, SDataRow* pRes, TSKEY* lastKey) {
return TSDB_CODE_SUCCESS;
}
+bool isTsdbCacheLastRow(TsdbQueryHandleT* pQueryHandle) {
+ return ((STsdbQueryHandle *)pQueryHandle)->cachelastrow > 0;
+}
+
int32_t checkForCachedLastRow(STsdbQueryHandle* pQueryHandle, STableGroupInfo *groupList) {
assert(pQueryHandle != NULL && groupList != NULL);
@@ -2706,11 +2890,15 @@ int32_t checkForCachedLastRow(STsdbQueryHandle* pQueryHandle, STableGroupInfo *g
STableKeyInfo* pInfo = (STableKeyInfo*)taosArrayGet(group, 0);
- int32_t code = tsdbGetCachedLastRow(pInfo->pTable, &pRow, &key);
- if (code != TSDB_CODE_SUCCESS) {
- pQueryHandle->cachelastrow = false;
- } else {
- pQueryHandle->cachelastrow = (pRow != NULL);
+ int32_t code = 0;
+
+ if (((STable*)pInfo->pTable)->lastRow) {
+ code = tsdbGetCachedLastRow(pInfo->pTable, &pRow, &key);
+ if (code != TSDB_CODE_SUCCESS) {
+ pQueryHandle->cachelastrow = 0;
+ } else {
+ pQueryHandle->cachelastrow = 1;
+ }
}
// update the tsdb query time range
@@ -2724,6 +2912,26 @@ int32_t checkForCachedLastRow(STsdbQueryHandle* pQueryHandle, STableGroupInfo *g
return code;
}
+int32_t checkForCachedLast(STsdbQueryHandle* pQueryHandle) {
+ assert(pQueryHandle != NULL);
+
+ int32_t code = 0;
+
+ if (pQueryHandle->pTsdb && atomic_load_8(&pQueryHandle->pTsdb->hasCachedLastColumn)){
+ pQueryHandle->cachelastrow = 2;
+ }
+
+ // update the tsdb query time range
+ if (pQueryHandle->cachelastrow) {
+ pQueryHandle->window = TSWINDOW_INITIALIZER;
+ pQueryHandle->checkFiles = false;
+ pQueryHandle->activeIndex = -1; // start from -1
+ }
+
+ return code;
+}
+
+
STimeWindow updateLastrowForEachGroup(STableGroupInfo *groupList) {
STimeWindow window = {INT64_MAX, INT64_MIN};
diff --git a/src/tsdb/src/tsdbSync.c b/src/tsdb/src/tsdbSync.c
index 5a2756537e6a37c8b4a0e7c3e1f81199523df2eb..edcb84d091eb4a1bcb4cb23835a3c889eee35d54 100644
--- a/src/tsdb/src/tsdbSync.c
+++ b/src/tsdb/src/tsdbSync.c
@@ -424,24 +424,42 @@ static int32_t tsdbSyncRecvDFileSetArray(SSyncH *pSynch) {
}
if (tsdbSendDecision(pSynch, false) < 0) {
- tsdbError("vgId:%d, filed to send decision since %s", REPO_ID(pRepo), tstrerror(terrno));
+ tsdbError("vgId:%d, failed to send decision since %s", REPO_ID(pRepo), tstrerror(terrno));
return -1;
}
} else {
// Need to copy from remote
- tsdbInfo("vgId:%d, fileset:%d will be received", REPO_ID(pRepo), pSynch->pdf->fid);
-
- // Notify remote to send there file here
- if (tsdbSendDecision(pSynch, true) < 0) {
- tsdbError("vgId:%d, failed to send decision since %s", REPO_ID(pRepo), tstrerror(terrno));
- return -1;
+ int fidLevel = tsdbGetFidLevel(pSynch->pdf->fid, &(pSynch->rtn));
+ if (fidLevel < 0) { // expired fileset
+ tsdbInfo("vgId:%d, fileset:%d will be skipped as expired", REPO_ID(pRepo), pSynch->pdf->fid);
+ if (tsdbSendDecision(pSynch, false) < 0) {
+ tsdbError("vgId:%d, failed to send decision since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+ // Move forward
+ if (tsdbRecvDFileSetInfo(pSynch) < 0) {
+ tsdbError("vgId:%d, failed to recv fileset since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
+ if (pLSet) {
+ pLSet = tsdbFSIterNext(&fsiter);
+ }
+ // Next loop
+ continue;
+ } else {
+ tsdbInfo("vgId:%d, fileset:%d will be received", REPO_ID(pRepo), pSynch->pdf->fid);
+ // Notify remote to send there file here
+ if (tsdbSendDecision(pSynch, true) < 0) {
+ tsdbError("vgId:%d, failed to send decision since %s", REPO_ID(pRepo), tstrerror(terrno));
+ return -1;
+ }
}
// Create local files and copy from remote
SDiskID did;
SDFileSet fset;
- tfsAllocDisk(tsdbGetFidLevel(pSynch->pdf->fid, &(pSynch->rtn)), &(did.level), &(did.id));
+ tfsAllocDisk(fidLevel, &(did.level), &(did.id));
if (did.level == TFS_UNDECIDED_LEVEL) {
terrno = TSDB_CODE_TDB_NO_AVAIL_DISK;
tsdbError("vgId:%d, failed allc disk since %s", REPO_ID(pRepo), tstrerror(terrno));
@@ -548,6 +566,13 @@ static int32_t tsdbSyncSendDFileSet(SSyncH *pSynch, SDFileSet *pSet) {
STsdbRepo *pRepo = pSynch->pRepo;
bool toSend = false;
+ // skip expired fileset
+ if (pSet && tsdbGetFidLevel(pSet->fid, &(pSynch->rtn)) < 0) {
+ tsdbInfo("vgId:%d, don't sync send since fileset:%d smaller than minFid:%d", REPO_ID(pRepo), pSet->fid,
+ pSynch->rtn.minFid);
+ return 0;
+ }
+
if (tsdbSendDFileSetInfo(pSynch, pSet) < 0) {
tsdbError("vgId:%d, failed to send fileset:%d info since %s", REPO_ID(pRepo), pSet ? pSet->fid : -1, tstrerror(terrno));
return -1;
diff --git a/src/wal/src/walWrite.c b/src/wal/src/walWrite.c
index f865870d478882df64bd2308b87f448c4602c847..b884546a08674fef7c2c42f104c5200e0d5efff8 100644
--- a/src/wal/src/walWrite.c
+++ b/src/wal/src/walWrite.c
@@ -430,6 +430,8 @@ static int32_t walRestoreWalFile(SWal *pWal, void *pVnode, FWalWrite writeFp, ch
pWal->vgId, fileId, pHead->version, pWal->version, pHead->len, offset);
pWal->version = pHead->version;
+
+ //wInfo("writeFp: %ld", offset);
(*writeFp)(pVnode, pHead, TAOS_QTYPE_WAL, NULL);
}
diff --git a/tests/pytest/fulltest.sh b/tests/pytest/fulltest.sh
index d8e2a31e70bd14b8feef6c8a45df671473e68537..be2cfee04b2e2405804fad8d24b77759b6f79b13 100755
--- a/tests/pytest/fulltest.sh
+++ b/tests/pytest/fulltest.sh
@@ -38,6 +38,8 @@ python3 ./test.py -f table/boundary.py
python3 ./test.py -f table/create.py
python3 ./test.py -f table/del_stable.py
+#stable
+python3 ./test.py -f stable/insert.py
# tag
python3 ./test.py -f tag_lite/filter.py
diff --git a/tests/pytest/stable/insert.py b/tests/pytest/stable/insert.py
index 0ef816da8d02fc6dcb48954435aad51342248e56..ef5635c77ce04ddd33354e1754dc70b2c9f8b6a5 100644
--- a/tests/pytest/stable/insert.py
+++ b/tests/pytest/stable/insert.py
@@ -72,6 +72,9 @@ class TDTestCase:
tdSql.query("describe db.stb")
tdSql.checkRows(3)
+ tdSql.error("drop stable if exists db.dev_01")
+ tdSql.error("drop stable if exists db.dev_02")
+
tdSql.execute("alter stable db.stb add tag t1 int")
tdSql.query("describe db.stb")
tdSql.checkRows(4)
@@ -80,6 +83,13 @@ class TDTestCase:
tdSql.query("show stables")
tdSql.checkRows(1)
+ tdSql.error("drop stable if exists db.dev_001")
+ tdSql.error("drop stable if exists db.dev_002")
+
+ for i in range(10):
+ tdSql.execute("drop stable if exists db.stb")
+ tdSql.query("show stables")
+ tdSql.checkRows(1)
def stop(self):
tdSql.close()
diff --git a/tests/script/api/batchprepare.c b/tests/script/api/batchprepare.c
index db48632cb7814229db4a973cc6b41e00d6e8ada4..92253be78c89e7f6420cead370e2e365534f152b 100644
--- a/tests/script/api/batchprepare.c
+++ b/tests/script/api/batchprepare.c
@@ -840,6 +840,1021 @@ int stmt_func3(TAOS_STMT *stmt) {
}
+
+//1 tables 10 records
+int stmt_funcb_autoctb1(TAOS_STMT *stmt) {
+ struct {
+ int64_t *ts;
+ int8_t b[10];
+ int8_t v1[10];
+ int16_t v2[10];
+ int32_t v4[10];
+ int64_t v8[10];
+ float f4[10];
+ double f8[10];
+ char bin[10][40];
+ } v = {0};
+
+ v.ts = malloc(sizeof(int64_t) * 1 * 10);
+
+ int *lb = malloc(10 * sizeof(int));
+
+ TAOS_BIND *tags = calloc(1, sizeof(TAOS_BIND) * 9 * 1);
+ TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * 1*10);
+
+// int one_null = 1;
+ int one_not_null = 0;
+
+ char* is_null = malloc(sizeof(char) * 10);
+ char* no_null = malloc(sizeof(char) * 10);
+
+ for (int i = 0; i < 10; ++i) {
+ lb[i] = 40;
+ no_null[i] = 0;
+ is_null[i] = (i % 10 == 2) ? 1 : 0;
+ v.b[i] = (int8_t)(i % 2);
+ v.v1[i] = (int8_t)((i+1) % 2);
+ v.v2[i] = (int16_t)i;
+ v.v4[i] = (int32_t)(i+1);
+ v.v8[i] = (int64_t)(i+2);
+ v.f4[i] = (float)(i+3);
+ v.f8[i] = (double)(i+4);
+ memset(v.bin[i], '0'+i%10, 40);
+ }
+
+ for (int i = 0; i < 10; i+=10) {
+ params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ params[i+0].buffer_length = sizeof(int64_t);
+ params[i+0].buffer = &v.ts[10*i/10];
+ params[i+0].length = NULL;
+ params[i+0].is_null = no_null;
+ params[i+0].num = 10;
+
+ params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ params[i+1].buffer_length = sizeof(int8_t);
+ params[i+1].buffer = v.b;
+ params[i+1].length = NULL;
+ params[i+1].is_null = is_null;
+ params[i+1].num = 10;
+
+ params[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ params[i+2].buffer_length = sizeof(int8_t);
+ params[i+2].buffer = v.v1;
+ params[i+2].length = NULL;
+ params[i+2].is_null = is_null;
+ params[i+2].num = 10;
+
+ params[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ params[i+3].buffer_length = sizeof(int16_t);
+ params[i+3].buffer = v.v2;
+ params[i+3].length = NULL;
+ params[i+3].is_null = is_null;
+ params[i+3].num = 10;
+
+ params[i+4].buffer_type = TSDB_DATA_TYPE_INT;
+ params[i+4].buffer_length = sizeof(int32_t);
+ params[i+4].buffer = v.v4;
+ params[i+4].length = NULL;
+ params[i+4].is_null = is_null;
+ params[i+4].num = 10;
+
+ params[i+5].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ params[i+5].buffer_length = sizeof(int64_t);
+ params[i+5].buffer = v.v8;
+ params[i+5].length = NULL;
+ params[i+5].is_null = is_null;
+ params[i+5].num = 10;
+
+ params[i+6].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ params[i+6].buffer_length = sizeof(float);
+ params[i+6].buffer = v.f4;
+ params[i+6].length = NULL;
+ params[i+6].is_null = is_null;
+ params[i+6].num = 10;
+
+ params[i+7].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ params[i+7].buffer_length = sizeof(double);
+ params[i+7].buffer = v.f8;
+ params[i+7].length = NULL;
+ params[i+7].is_null = is_null;
+ params[i+7].num = 10;
+
+ params[i+8].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+8].buffer_length = 40;
+ params[i+8].buffer = v.bin;
+ params[i+8].length = lb;
+ params[i+8].is_null = is_null;
+ params[i+8].num = 10;
+
+ params[i+9].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+9].buffer_length = 40;
+ params[i+9].buffer = v.bin;
+ params[i+9].length = lb;
+ params[i+9].is_null = is_null;
+ params[i+9].num = 10;
+
+ }
+
+ int64_t tts = 1591060628000;
+ for (int i = 0; i < 10; ++i) {
+ v.ts[i] = tts + i;
+ }
+
+
+ for (int i = 0; i < 1; ++i) {
+ tags[i+0].buffer_type = TSDB_DATA_TYPE_INT;
+ tags[i+0].buffer = v.v4;
+ tags[i+0].is_null = &one_not_null;
+ tags[i+0].length = NULL;
+
+ tags[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ tags[i+1].buffer = v.b;
+ tags[i+1].is_null = &one_not_null;
+ tags[i+1].length = NULL;
+
+ tags[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ tags[i+2].buffer = v.v1;
+ tags[i+2].is_null = &one_not_null;
+ tags[i+2].length = NULL;
+
+ tags[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ tags[i+3].buffer = v.v2;
+ tags[i+3].is_null = &one_not_null;
+ tags[i+3].length = NULL;
+
+ tags[i+4].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ tags[i+4].buffer = v.v8;
+ tags[i+4].is_null = &one_not_null;
+ tags[i+4].length = NULL;
+
+ tags[i+5].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ tags[i+5].buffer = v.f4;
+ tags[i+5].is_null = &one_not_null;
+ tags[i+5].length = NULL;
+
+ tags[i+6].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ tags[i+6].buffer = v.f8;
+ tags[i+6].is_null = &one_not_null;
+ tags[i+6].length = NULL;
+
+ tags[i+7].buffer_type = TSDB_DATA_TYPE_BINARY;
+ tags[i+7].buffer = v.bin;
+ tags[i+7].is_null = &one_not_null;
+ tags[i+7].length = (uintptr_t *)lb;
+
+ tags[i+8].buffer_type = TSDB_DATA_TYPE_NCHAR;
+ tags[i+8].buffer = v.bin;
+ tags[i+8].is_null = &one_not_null;
+ tags[i+8].length = (uintptr_t *)lb;
+ }
+
+
+ unsigned long long starttime = getCurrentTime();
+
+ char *sql = "insert into ? using stb1 tags(?,?,?,?,?,?,?,?,?) values(?,?,?,?,?,?,?,?,?,?)";
+ int code = taos_stmt_prepare(stmt, sql, 0);
+ if (code != 0){
+ printf("failed to execute taos_stmt_prepare. code:0x%x\n", code);
+ exit(1);
+ }
+
+ int id = 0;
+ for (int zz = 0; zz < 1; zz++) {
+ char buf[32];
+ sprintf(buf, "m%d", zz);
+ code = taos_stmt_set_tbname_tags(stmt, buf, tags);
+ if (code != 0){
+ printf("failed to execute taos_stmt_set_tbname_tags. code:0x%x\n", code);
+ }
+
+ taos_stmt_bind_param_batch(stmt, params + id * 10);
+ taos_stmt_add_batch(stmt);
+ }
+
+ if (taos_stmt_execute(stmt) != 0) {
+ printf("failed to execute insert statement.\n");
+ exit(1);
+ }
+
+ ++id;
+
+ unsigned long long endtime = getCurrentTime();
+ printf("insert total %d records, used %u seconds, avg:%u useconds\n", 10, (endtime-starttime)/1000000UL, (endtime-starttime)/(10));
+
+ free(v.ts);
+ free(lb);
+ free(params);
+ free(is_null);
+ free(no_null);
+
+ return 0;
+}
+
+
+
+
+//1 tables 10 records
+int stmt_funcb_autoctb2(TAOS_STMT *stmt) {
+ struct {
+ int64_t *ts;
+ int8_t b[10];
+ int8_t v1[10];
+ int16_t v2[10];
+ int32_t v4[10];
+ int64_t v8[10];
+ float f4[10];
+ double f8[10];
+ char bin[10][40];
+ } v = {0};
+
+ v.ts = malloc(sizeof(int64_t) * 1 * 10);
+
+ int *lb = malloc(10 * sizeof(int));
+
+ TAOS_BIND *tags = calloc(1, sizeof(TAOS_BIND) * 9 * 1);
+ TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * 1*10);
+
+// int one_null = 1;
+ int one_not_null = 0;
+
+ char* is_null = malloc(sizeof(char) * 10);
+ char* no_null = malloc(sizeof(char) * 10);
+
+ for (int i = 0; i < 10; ++i) {
+ lb[i] = 40;
+ no_null[i] = 0;
+ is_null[i] = (i % 10 == 2) ? 1 : 0;
+ v.b[i] = (int8_t)(i % 2);
+ v.v1[i] = (int8_t)((i+1) % 2);
+ v.v2[i] = (int16_t)i;
+ v.v4[i] = (int32_t)(i+1);
+ v.v8[i] = (int64_t)(i+2);
+ v.f4[i] = (float)(i+3);
+ v.f8[i] = (double)(i+4);
+ memset(v.bin[i], '0'+i%10, 40);
+ }
+
+ for (int i = 0; i < 10; i+=10) {
+ params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ params[i+0].buffer_length = sizeof(int64_t);
+ params[i+0].buffer = &v.ts[10*i/10];
+ params[i+0].length = NULL;
+ params[i+0].is_null = no_null;
+ params[i+0].num = 10;
+
+ params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ params[i+1].buffer_length = sizeof(int8_t);
+ params[i+1].buffer = v.b;
+ params[i+1].length = NULL;
+ params[i+1].is_null = is_null;
+ params[i+1].num = 10;
+
+ params[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ params[i+2].buffer_length = sizeof(int8_t);
+ params[i+2].buffer = v.v1;
+ params[i+2].length = NULL;
+ params[i+2].is_null = is_null;
+ params[i+2].num = 10;
+
+ params[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ params[i+3].buffer_length = sizeof(int16_t);
+ params[i+3].buffer = v.v2;
+ params[i+3].length = NULL;
+ params[i+3].is_null = is_null;
+ params[i+3].num = 10;
+
+ params[i+4].buffer_type = TSDB_DATA_TYPE_INT;
+ params[i+4].buffer_length = sizeof(int32_t);
+ params[i+4].buffer = v.v4;
+ params[i+4].length = NULL;
+ params[i+4].is_null = is_null;
+ params[i+4].num = 10;
+
+ params[i+5].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ params[i+5].buffer_length = sizeof(int64_t);
+ params[i+5].buffer = v.v8;
+ params[i+5].length = NULL;
+ params[i+5].is_null = is_null;
+ params[i+5].num = 10;
+
+ params[i+6].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ params[i+6].buffer_length = sizeof(float);
+ params[i+6].buffer = v.f4;
+ params[i+6].length = NULL;
+ params[i+6].is_null = is_null;
+ params[i+6].num = 10;
+
+ params[i+7].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ params[i+7].buffer_length = sizeof(double);
+ params[i+7].buffer = v.f8;
+ params[i+7].length = NULL;
+ params[i+7].is_null = is_null;
+ params[i+7].num = 10;
+
+ params[i+8].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+8].buffer_length = 40;
+ params[i+8].buffer = v.bin;
+ params[i+8].length = lb;
+ params[i+8].is_null = is_null;
+ params[i+8].num = 10;
+
+ params[i+9].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+9].buffer_length = 40;
+ params[i+9].buffer = v.bin;
+ params[i+9].length = lb;
+ params[i+9].is_null = is_null;
+ params[i+9].num = 10;
+
+ }
+
+ int64_t tts = 1591060628000;
+ for (int i = 0; i < 10; ++i) {
+ v.ts[i] = tts + i;
+ }
+
+
+ for (int i = 0; i < 1; ++i) {
+ tags[i+0].buffer_type = TSDB_DATA_TYPE_INT;
+ tags[i+0].buffer = v.v4;
+ tags[i+0].is_null = &one_not_null;
+ tags[i+0].length = NULL;
+
+ tags[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ tags[i+1].buffer = v.b;
+ tags[i+1].is_null = &one_not_null;
+ tags[i+1].length = NULL;
+
+ tags[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ tags[i+2].buffer = v.v1;
+ tags[i+2].is_null = &one_not_null;
+ tags[i+2].length = NULL;
+
+ tags[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ tags[i+3].buffer = v.v2;
+ tags[i+3].is_null = &one_not_null;
+ tags[i+3].length = NULL;
+
+ tags[i+4].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ tags[i+4].buffer = v.v8;
+ tags[i+4].is_null = &one_not_null;
+ tags[i+4].length = NULL;
+
+ tags[i+5].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ tags[i+5].buffer = v.f4;
+ tags[i+5].is_null = &one_not_null;
+ tags[i+5].length = NULL;
+
+ tags[i+6].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ tags[i+6].buffer = v.f8;
+ tags[i+6].is_null = &one_not_null;
+ tags[i+6].length = NULL;
+
+ tags[i+7].buffer_type = TSDB_DATA_TYPE_BINARY;
+ tags[i+7].buffer = v.bin;
+ tags[i+7].is_null = &one_not_null;
+ tags[i+7].length = (uintptr_t *)lb;
+
+ tags[i+8].buffer_type = TSDB_DATA_TYPE_NCHAR;
+ tags[i+8].buffer = v.bin;
+ tags[i+8].is_null = &one_not_null;
+ tags[i+8].length = (uintptr_t *)lb;
+ }
+
+
+ unsigned long long starttime = getCurrentTime();
+
+ char *sql = "insert into ? using stb1 tags(1,true,2,3,4,5.0,6.0,'a','b') values(?,?,?,?,?,?,?,?,?,?)";
+ int code = taos_stmt_prepare(stmt, sql, 0);
+ if (code != 0){
+ printf("failed to execute taos_stmt_prepare. code:0x%x\n", code);
+ exit(1);
+ }
+
+ int id = 0;
+ for (int zz = 0; zz < 1; zz++) {
+ char buf[32];
+ sprintf(buf, "m%d", zz);
+ code = taos_stmt_set_tbname_tags(stmt, buf, tags);
+ if (code != 0){
+ printf("failed to execute taos_stmt_set_tbname_tags. code:0x%x\n", code);
+ }
+
+ taos_stmt_bind_param_batch(stmt, params + id * 10);
+ taos_stmt_add_batch(stmt);
+ }
+
+ if (taos_stmt_execute(stmt) != 0) {
+ printf("failed to execute insert statement.\n");
+ exit(1);
+ }
+
+ ++id;
+
+ unsigned long long endtime = getCurrentTime();
+ printf("insert total %d records, used %u seconds, avg:%u useconds\n", 10, (endtime-starttime)/1000000UL, (endtime-starttime)/(10));
+
+ free(v.ts);
+ free(lb);
+ free(params);
+ free(is_null);
+ free(no_null);
+
+ return 0;
+}
+
+
+
+
+
+//1 tables 10 records
+int stmt_funcb_autoctb3(TAOS_STMT *stmt) {
+ struct {
+ int64_t *ts;
+ int8_t b[10];
+ int8_t v1[10];
+ int16_t v2[10];
+ int32_t v4[10];
+ int64_t v8[10];
+ float f4[10];
+ double f8[10];
+ char bin[10][40];
+ } v = {0};
+
+ v.ts = malloc(sizeof(int64_t) * 1 * 10);
+
+ int *lb = malloc(10 * sizeof(int));
+
+ TAOS_BIND *tags = calloc(1, sizeof(TAOS_BIND) * 9 * 1);
+ TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * 1*10);
+
+// int one_null = 1;
+ int one_not_null = 0;
+
+ char* is_null = malloc(sizeof(char) * 10);
+ char* no_null = malloc(sizeof(char) * 10);
+
+ for (int i = 0; i < 10; ++i) {
+ lb[i] = 40;
+ no_null[i] = 0;
+ is_null[i] = (i % 10 == 2) ? 1 : 0;
+ v.b[i] = (int8_t)(i % 2);
+ v.v1[i] = (int8_t)((i+1) % 2);
+ v.v2[i] = (int16_t)i;
+ v.v4[i] = (int32_t)(i+1);
+ v.v8[i] = (int64_t)(i+2);
+ v.f4[i] = (float)(i+3);
+ v.f8[i] = (double)(i+4);
+ memset(v.bin[i], '0'+i%10, 40);
+ }
+
+ for (int i = 0; i < 10; i+=10) {
+ params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ params[i+0].buffer_length = sizeof(int64_t);
+ params[i+0].buffer = &v.ts[10*i/10];
+ params[i+0].length = NULL;
+ params[i+0].is_null = no_null;
+ params[i+0].num = 10;
+
+ params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ params[i+1].buffer_length = sizeof(int8_t);
+ params[i+1].buffer = v.b;
+ params[i+1].length = NULL;
+ params[i+1].is_null = is_null;
+ params[i+1].num = 10;
+
+ params[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ params[i+2].buffer_length = sizeof(int8_t);
+ params[i+2].buffer = v.v1;
+ params[i+2].length = NULL;
+ params[i+2].is_null = is_null;
+ params[i+2].num = 10;
+
+ params[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ params[i+3].buffer_length = sizeof(int16_t);
+ params[i+3].buffer = v.v2;
+ params[i+3].length = NULL;
+ params[i+3].is_null = is_null;
+ params[i+3].num = 10;
+
+ params[i+4].buffer_type = TSDB_DATA_TYPE_INT;
+ params[i+4].buffer_length = sizeof(int32_t);
+ params[i+4].buffer = v.v4;
+ params[i+4].length = NULL;
+ params[i+4].is_null = is_null;
+ params[i+4].num = 10;
+
+ params[i+5].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ params[i+5].buffer_length = sizeof(int64_t);
+ params[i+5].buffer = v.v8;
+ params[i+5].length = NULL;
+ params[i+5].is_null = is_null;
+ params[i+5].num = 10;
+
+ params[i+6].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ params[i+6].buffer_length = sizeof(float);
+ params[i+6].buffer = v.f4;
+ params[i+6].length = NULL;
+ params[i+6].is_null = is_null;
+ params[i+6].num = 10;
+
+ params[i+7].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ params[i+7].buffer_length = sizeof(double);
+ params[i+7].buffer = v.f8;
+ params[i+7].length = NULL;
+ params[i+7].is_null = is_null;
+ params[i+7].num = 10;
+
+ params[i+8].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+8].buffer_length = 40;
+ params[i+8].buffer = v.bin;
+ params[i+8].length = lb;
+ params[i+8].is_null = is_null;
+ params[i+8].num = 10;
+
+ params[i+9].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+9].buffer_length = 40;
+ params[i+9].buffer = v.bin;
+ params[i+9].length = lb;
+ params[i+9].is_null = is_null;
+ params[i+9].num = 10;
+
+ }
+
+ int64_t tts = 1591060628000;
+ for (int i = 0; i < 10; ++i) {
+ v.ts[i] = tts + i;
+ }
+
+
+ for (int i = 0; i < 1; ++i) {
+ tags[i+0].buffer_type = TSDB_DATA_TYPE_BOOL;
+ tags[i+0].buffer = v.b;
+ tags[i+0].is_null = &one_not_null;
+ tags[i+0].length = NULL;
+
+ tags[i+1].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ tags[i+1].buffer = v.v2;
+ tags[i+1].is_null = &one_not_null;
+ tags[i+1].length = NULL;
+
+ tags[i+2].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ tags[i+2].buffer = v.f4;
+ tags[i+2].is_null = &one_not_null;
+ tags[i+2].length = NULL;
+
+ tags[i+3].buffer_type = TSDB_DATA_TYPE_BINARY;
+ tags[i+3].buffer = v.bin;
+ tags[i+3].is_null = &one_not_null;
+ tags[i+3].length = (uintptr_t *)lb;
+ }
+
+
+ unsigned long long starttime = getCurrentTime();
+
+ char *sql = "insert into ? using stb1 tags(1,?,2,?,4,?,6.0,?,'b') values(?,?,?,?,?,?,?,?,?,?)";
+ int code = taos_stmt_prepare(stmt, sql, 0);
+ if (code != 0){
+ printf("failed to execute taos_stmt_prepare. code:0x%x\n", code);
+ exit(1);
+ }
+
+ int id = 0;
+ for (int zz = 0; zz < 1; zz++) {
+ char buf[32];
+ sprintf(buf, "m%d", zz);
+ code = taos_stmt_set_tbname_tags(stmt, buf, tags);
+ if (code != 0){
+ printf("failed to execute taos_stmt_set_tbname_tags. code:0x%x\n", code);
+ }
+
+ taos_stmt_bind_param_batch(stmt, params + id * 10);
+ taos_stmt_add_batch(stmt);
+ }
+
+ if (taos_stmt_execute(stmt) != 0) {
+ printf("failed to execute insert statement.\n");
+ exit(1);
+ }
+
+ ++id;
+
+ unsigned long long endtime = getCurrentTime();
+ printf("insert total %d records, used %u seconds, avg:%u useconds\n", 10, (endtime-starttime)/1000000UL, (endtime-starttime)/(10));
+
+ free(v.ts);
+ free(lb);
+ free(params);
+ free(is_null);
+ free(no_null);
+
+ return 0;
+}
+
+
+
+
+
+
+//1 tables 10 records
+int stmt_funcb_autoctb_e1(TAOS_STMT *stmt) {
+ struct {
+ int64_t *ts;
+ int8_t b[10];
+ int8_t v1[10];
+ int16_t v2[10];
+ int32_t v4[10];
+ int64_t v8[10];
+ float f4[10];
+ double f8[10];
+ char bin[10][40];
+ } v = {0};
+
+ v.ts = malloc(sizeof(int64_t) * 1 * 10);
+
+ int *lb = malloc(10 * sizeof(int));
+
+ TAOS_BIND *tags = calloc(1, sizeof(TAOS_BIND) * 9 * 1);
+ TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * 1*10);
+
+// int one_null = 1;
+ int one_not_null = 0;
+
+ char* is_null = malloc(sizeof(char) * 10);
+ char* no_null = malloc(sizeof(char) * 10);
+
+ for (int i = 0; i < 10; ++i) {
+ lb[i] = 40;
+ no_null[i] = 0;
+ is_null[i] = (i % 10 == 2) ? 1 : 0;
+ v.b[i] = (int8_t)(i % 2);
+ v.v1[i] = (int8_t)((i+1) % 2);
+ v.v2[i] = (int16_t)i;
+ v.v4[i] = (int32_t)(i+1);
+ v.v8[i] = (int64_t)(i+2);
+ v.f4[i] = (float)(i+3);
+ v.f8[i] = (double)(i+4);
+ memset(v.bin[i], '0'+i%10, 40);
+ }
+
+ for (int i = 0; i < 10; i+=10) {
+ params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ params[i+0].buffer_length = sizeof(int64_t);
+ params[i+0].buffer = &v.ts[10*i/10];
+ params[i+0].length = NULL;
+ params[i+0].is_null = no_null;
+ params[i+0].num = 10;
+
+ params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ params[i+1].buffer_length = sizeof(int8_t);
+ params[i+1].buffer = v.b;
+ params[i+1].length = NULL;
+ params[i+1].is_null = is_null;
+ params[i+1].num = 10;
+
+ params[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ params[i+2].buffer_length = sizeof(int8_t);
+ params[i+2].buffer = v.v1;
+ params[i+2].length = NULL;
+ params[i+2].is_null = is_null;
+ params[i+2].num = 10;
+
+ params[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ params[i+3].buffer_length = sizeof(int16_t);
+ params[i+3].buffer = v.v2;
+ params[i+3].length = NULL;
+ params[i+3].is_null = is_null;
+ params[i+3].num = 10;
+
+ params[i+4].buffer_type = TSDB_DATA_TYPE_INT;
+ params[i+4].buffer_length = sizeof(int32_t);
+ params[i+4].buffer = v.v4;
+ params[i+4].length = NULL;
+ params[i+4].is_null = is_null;
+ params[i+4].num = 10;
+
+ params[i+5].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ params[i+5].buffer_length = sizeof(int64_t);
+ params[i+5].buffer = v.v8;
+ params[i+5].length = NULL;
+ params[i+5].is_null = is_null;
+ params[i+5].num = 10;
+
+ params[i+6].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ params[i+6].buffer_length = sizeof(float);
+ params[i+6].buffer = v.f4;
+ params[i+6].length = NULL;
+ params[i+6].is_null = is_null;
+ params[i+6].num = 10;
+
+ params[i+7].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ params[i+7].buffer_length = sizeof(double);
+ params[i+7].buffer = v.f8;
+ params[i+7].length = NULL;
+ params[i+7].is_null = is_null;
+ params[i+7].num = 10;
+
+ params[i+8].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+8].buffer_length = 40;
+ params[i+8].buffer = v.bin;
+ params[i+8].length = lb;
+ params[i+8].is_null = is_null;
+ params[i+8].num = 10;
+
+ params[i+9].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+9].buffer_length = 40;
+ params[i+9].buffer = v.bin;
+ params[i+9].length = lb;
+ params[i+9].is_null = is_null;
+ params[i+9].num = 10;
+
+ }
+
+ int64_t tts = 1591060628000;
+ for (int i = 0; i < 10; ++i) {
+ v.ts[i] = tts + i;
+ }
+
+
+ for (int i = 0; i < 1; ++i) {
+ tags[i+0].buffer_type = TSDB_DATA_TYPE_BOOL;
+ tags[i+0].buffer = v.b;
+ tags[i+0].is_null = &one_not_null;
+ tags[i+0].length = NULL;
+
+ tags[i+1].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ tags[i+1].buffer = v.v2;
+ tags[i+1].is_null = &one_not_null;
+ tags[i+1].length = NULL;
+
+ tags[i+2].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ tags[i+2].buffer = v.f4;
+ tags[i+2].is_null = &one_not_null;
+ tags[i+2].length = NULL;
+
+ tags[i+3].buffer_type = TSDB_DATA_TYPE_BINARY;
+ tags[i+3].buffer = v.bin;
+ tags[i+3].is_null = &one_not_null;
+ tags[i+3].length = (uintptr_t *)lb;
+ }
+
+
+ unsigned long long starttime = getCurrentTime();
+
+ char *sql = "insert into ? using stb1 (id1,id2,id3,id4,id5,id6,id7,id8,id9) tags(1,?,2,?,4,?,6.0,?,'b') values(?,?,?,?,?,?,?,?,?,?)";
+ int code = taos_stmt_prepare(stmt, sql, 0);
+ if (code != 0){
+ printf("failed to execute taos_stmt_prepare. code:0x%x\n", code);
+ return -1;
+ }
+
+ int id = 0;
+ for (int zz = 0; zz < 1; zz++) {
+ char buf[32];
+ sprintf(buf, "m%d", zz);
+ code = taos_stmt_set_tbname_tags(stmt, buf, tags);
+ if (code != 0){
+ printf("failed to execute taos_stmt_set_tbname_tags. code:0x%x\n", code);
+ }
+
+ taos_stmt_bind_param_batch(stmt, params + id * 10);
+ taos_stmt_add_batch(stmt);
+ }
+
+ if (taos_stmt_execute(stmt) != 0) {
+ printf("failed to execute insert statement.\n");
+ exit(1);
+ }
+
+ ++id;
+
+ unsigned long long endtime = getCurrentTime();
+ printf("insert total %d records, used %u seconds, avg:%u useconds\n", 10, (endtime-starttime)/1000000UL, (endtime-starttime)/(10));
+
+ free(v.ts);
+ free(lb);
+ free(params);
+ free(is_null);
+ free(no_null);
+
+ return 0;
+}
+
+
+
+
+
+//1 tables 10 records
+int stmt_funcb_autoctb_e2(TAOS_STMT *stmt) {
+ struct {
+ int64_t *ts;
+ int8_t b[10];
+ int8_t v1[10];
+ int16_t v2[10];
+ int32_t v4[10];
+ int64_t v8[10];
+ float f4[10];
+ double f8[10];
+ char bin[10][40];
+ } v = {0};
+
+ v.ts = malloc(sizeof(int64_t) * 1 * 10);
+
+ int *lb = malloc(10 * sizeof(int));
+
+ TAOS_BIND *tags = calloc(1, sizeof(TAOS_BIND) * 9 * 1);
+ TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * 1*10);
+
+// int one_null = 1;
+ int one_not_null = 0;
+
+ char* is_null = malloc(sizeof(char) * 10);
+ char* no_null = malloc(sizeof(char) * 10);
+
+ for (int i = 0; i < 10; ++i) {
+ lb[i] = 40;
+ no_null[i] = 0;
+ is_null[i] = (i % 10 == 2) ? 1 : 0;
+ v.b[i] = (int8_t)(i % 2);
+ v.v1[i] = (int8_t)((i+1) % 2);
+ v.v2[i] = (int16_t)i;
+ v.v4[i] = (int32_t)(i+1);
+ v.v8[i] = (int64_t)(i+2);
+ v.f4[i] = (float)(i+3);
+ v.f8[i] = (double)(i+4);
+ memset(v.bin[i], '0'+i%10, 40);
+ }
+
+ for (int i = 0; i < 10; i+=10) {
+ params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
+ params[i+0].buffer_length = sizeof(int64_t);
+ params[i+0].buffer = &v.ts[10*i/10];
+ params[i+0].length = NULL;
+ params[i+0].is_null = no_null;
+ params[i+0].num = 10;
+
+ params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ params[i+1].buffer_length = sizeof(int8_t);
+ params[i+1].buffer = v.b;
+ params[i+1].length = NULL;
+ params[i+1].is_null = is_null;
+ params[i+1].num = 10;
+
+ params[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ params[i+2].buffer_length = sizeof(int8_t);
+ params[i+2].buffer = v.v1;
+ params[i+2].length = NULL;
+ params[i+2].is_null = is_null;
+ params[i+2].num = 10;
+
+ params[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ params[i+3].buffer_length = sizeof(int16_t);
+ params[i+3].buffer = v.v2;
+ params[i+3].length = NULL;
+ params[i+3].is_null = is_null;
+ params[i+3].num = 10;
+
+ params[i+4].buffer_type = TSDB_DATA_TYPE_INT;
+ params[i+4].buffer_length = sizeof(int32_t);
+ params[i+4].buffer = v.v4;
+ params[i+4].length = NULL;
+ params[i+4].is_null = is_null;
+ params[i+4].num = 10;
+
+ params[i+5].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ params[i+5].buffer_length = sizeof(int64_t);
+ params[i+5].buffer = v.v8;
+ params[i+5].length = NULL;
+ params[i+5].is_null = is_null;
+ params[i+5].num = 10;
+
+ params[i+6].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ params[i+6].buffer_length = sizeof(float);
+ params[i+6].buffer = v.f4;
+ params[i+6].length = NULL;
+ params[i+6].is_null = is_null;
+ params[i+6].num = 10;
+
+ params[i+7].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ params[i+7].buffer_length = sizeof(double);
+ params[i+7].buffer = v.f8;
+ params[i+7].length = NULL;
+ params[i+7].is_null = is_null;
+ params[i+7].num = 10;
+
+ params[i+8].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+8].buffer_length = 40;
+ params[i+8].buffer = v.bin;
+ params[i+8].length = lb;
+ params[i+8].is_null = is_null;
+ params[i+8].num = 10;
+
+ params[i+9].buffer_type = TSDB_DATA_TYPE_BINARY;
+ params[i+9].buffer_length = 40;
+ params[i+9].buffer = v.bin;
+ params[i+9].length = lb;
+ params[i+9].is_null = is_null;
+ params[i+9].num = 10;
+
+ }
+
+ int64_t tts = 1591060628000;
+ for (int i = 0; i < 10; ++i) {
+ v.ts[i] = tts + i;
+ }
+
+
+ for (int i = 0; i < 1; ++i) {
+ tags[i+0].buffer_type = TSDB_DATA_TYPE_INT;
+ tags[i+0].buffer = v.v4;
+ tags[i+0].is_null = &one_not_null;
+ tags[i+0].length = NULL;
+
+ tags[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
+ tags[i+1].buffer = v.b;
+ tags[i+1].is_null = &one_not_null;
+ tags[i+1].length = NULL;
+
+ tags[i+2].buffer_type = TSDB_DATA_TYPE_TINYINT;
+ tags[i+2].buffer = v.v1;
+ tags[i+2].is_null = &one_not_null;
+ tags[i+2].length = NULL;
+
+ tags[i+3].buffer_type = TSDB_DATA_TYPE_SMALLINT;
+ tags[i+3].buffer = v.v2;
+ tags[i+3].is_null = &one_not_null;
+ tags[i+3].length = NULL;
+
+ tags[i+4].buffer_type = TSDB_DATA_TYPE_BIGINT;
+ tags[i+4].buffer = v.v8;
+ tags[i+4].is_null = &one_not_null;
+ tags[i+4].length = NULL;
+
+ tags[i+5].buffer_type = TSDB_DATA_TYPE_FLOAT;
+ tags[i+5].buffer = v.f4;
+ tags[i+5].is_null = &one_not_null;
+ tags[i+5].length = NULL;
+
+ tags[i+6].buffer_type = TSDB_DATA_TYPE_DOUBLE;
+ tags[i+6].buffer = v.f8;
+ tags[i+6].is_null = &one_not_null;
+ tags[i+6].length = NULL;
+
+ tags[i+7].buffer_type = TSDB_DATA_TYPE_BINARY;
+ tags[i+7].buffer = v.bin;
+ tags[i+7].is_null = &one_not_null;
+ tags[i+7].length = (uintptr_t *)lb;
+
+ tags[i+8].buffer_type = TSDB_DATA_TYPE_NCHAR;
+ tags[i+8].buffer = v.bin;
+ tags[i+8].is_null = &one_not_null;
+ tags[i+8].length = (uintptr_t *)lb;
+ }
+
+
+ unsigned long long starttime = getCurrentTime();
+
+ char *sql = "insert into ? using stb1 tags(?,?,?,?,?,?,?,?,?) values(?,?,?,?,?,?,?,?,?,?)";
+ int code = taos_stmt_prepare(stmt, sql, 0);
+ if (code != 0){
+ printf("failed to execute taos_stmt_prepare. code:0x%x\n", code);
+ exit(1);
+ }
+
+ int id = 0;
+ for (int zz = 0; zz < 1; zz++) {
+ char buf[32];
+ sprintf(buf, "m%d", zz);
+ code = taos_stmt_set_tbname_tags(stmt, buf, NULL);
+ if (code != 0){
+ printf("failed to execute taos_stmt_set_tbname_tags. code:0x%x\n", code);
+ return -1;
+ }
+
+ taos_stmt_bind_param_batch(stmt, params + id * 10);
+ taos_stmt_add_batch(stmt);
+ }
+
+ if (taos_stmt_execute(stmt) != 0) {
+ printf("failed to execute insert statement.\n");
+ exit(1);
+ }
+
+ ++id;
+
+ unsigned long long endtime = getCurrentTime();
+ printf("insert total %d records, used %u seconds, avg:%u useconds\n", 10, (endtime-starttime)/1000000UL, (endtime-starttime)/(10));
+
+ free(v.ts);
+ free(lb);
+ free(params);
+ free(is_null);
+ free(no_null);
+
+ return 0;
+}
+
+
+
//300 tables 60 records
int stmt_funcb1(TAOS_STMT *stmt) {
struct {
@@ -2540,14 +3555,14 @@ int sql_s_perf1(TAOS *taos) {
}
-void prepare(TAOS *taos, int bigsize) {
+void prepare(TAOS *taos, int bigsize, int createChildTable) {
TAOS_RES *result;
int code;
result = taos_query(taos, "drop database demo");
taos_free_result(result);
- result = taos_query(taos, "create database demo");
+ result = taos_query(taos, "create database demo keep 36500");
code = taos_errno(result);
if (code != 0) {
printf("failed to create database, reason:%s\n", taos_errstr(result));
@@ -2558,15 +3573,34 @@ void prepare(TAOS *taos, int bigsize) {
result = taos_query(taos, "use demo");
taos_free_result(result);
-
- // create table
- for (int i = 0 ; i < 300; i++) {
+
+ if (createChildTable) {
+ // create table
+ for (int i = 0 ; i < 300; i++) {
+ char buf[1024];
+ if (bigsize) {
+ sprintf(buf, "create table m%d (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, bin binary(40), bin2 binary(40))", i) ;
+ } else {
+ sprintf(buf, "create table m%d (ts timestamp, b int)", i) ;
+ }
+ result = taos_query(taos, buf);
+ code = taos_errno(result);
+ if (code != 0) {
+ printf("failed to create table, reason:%s\n", taos_errstr(result));
+ taos_free_result(result);
+ exit(1);
+ }
+ taos_free_result(result);
+ }
+ } else {
char buf[1024];
if (bigsize) {
- sprintf(buf, "create table m%d (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, bin binary(40), bin2 binary(40))", i) ;
+ sprintf(buf, "create stable stb1 (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, bin binary(40), bin2 binary(40))"
+ " tags(id1 int, id2 bool, id3 tinyint, id4 smallint, id5 bigint, id6 float, id7 double, id8 binary(40), id9 nchar(40))") ;
} else {
- sprintf(buf, "create table m%d (ts timestamp, b int)", i) ;
+ sprintf(buf, "create stable stb1 (ts timestamp, b int) tags(id1 int, id2 bool, id3 tinyint, id4 smallint, id5 bigint, id6 float, id7 double, id8 binary(40), id9 nchar(40))") ;
}
+
result = taos_query(taos, buf);
code = taos_errno(result);
if (code != 0) {
@@ -2593,7 +3627,7 @@ void preparem(TAOS *taos, int bigsize, int idx) {
result = taos_query(taos, sql);
taos_free_result(result);
- sprintf(sql, "create database %s", dbname);
+ sprintf(sql, "create database %s keep 36500", dbname);
result = taos_query(taos, sql);
code = taos_errno(result);
if (code != 0) {
@@ -2640,8 +3674,8 @@ void* runcase(void *par) {
(void)idx;
-#if 0
- prepare(taos, 1);
+#if 1
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2664,8 +3698,8 @@ void* runcase(void *par) {
#endif
-#if 0
- prepare(taos, 1);
+#if 1
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2679,8 +3713,8 @@ void* runcase(void *par) {
#endif
-#if 0
- prepare(taos, 1);
+#if 1
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2699,7 +3733,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2720,12 +3754,11 @@ void* runcase(void *par) {
printf("check result end\n");
taos_stmt_close(stmt);
- return NULL;
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2749,7 +3782,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2773,7 +3806,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2798,7 +3831,7 @@ void* runcase(void *par) {
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2813,11 +3846,88 @@ void* runcase(void *par) {
check_result(taos, "m299", 0, 180000);
printf("check result end\n");
taos_stmt_close(stmt);
+
+#endif
+
+#if 1
+ prepare(taos, 1, 0);
+
+ stmt = taos_stmt_init(taos);
+
+ printf("1t+10r+bm+autoctb start\n");
+ stmt_funcb_autoctb1(stmt);
+ printf("1t+10r+bm+autoctb end\n");
+ printf("check result start\n");
+ check_result(taos, "m0", 1, 10);
+ printf("check result end\n");
+ taos_stmt_close(stmt);
+
+#endif
+
+#if 1
+ prepare(taos, 1, 0);
+
+ stmt = taos_stmt_init(taos);
+
+ printf("1t+10r+bm+autoctb start\n");
+ stmt_funcb_autoctb2(stmt);
+ printf("1t+10r+bm+autoctb end\n");
+ printf("check result start\n");
+ check_result(taos, "m0", 1, 10);
+ printf("check result end\n");
+ taos_stmt_close(stmt);
+
+#endif
+
+
+#if 1
+ prepare(taos, 1, 0);
+
+ stmt = taos_stmt_init(taos);
+
+ printf("1t+10r+bm+autoctb start\n");
+ stmt_funcb_autoctb3(stmt);
+ printf("1t+10r+bm+autoctb end\n");
+ printf("check result start\n");
+ check_result(taos, "m0", 1, 10);
+ printf("check result end\n");
+ taos_stmt_close(stmt);
+
+#endif
+
+#if 1
+ prepare(taos, 1, 0);
+
+ stmt = taos_stmt_init(taos);
+
+ printf("1t+10r+bm+autoctb+e1 start\n");
+ stmt_funcb_autoctb_e1(stmt);
+ printf("1t+10r+bm+autoctb+e1 end\n");
+ printf("check result start\n");
+ //check_result(taos, "m0", 1, 0);
+ printf("check result end\n");
+ taos_stmt_close(stmt);
+
+#endif
+
+#if 1
+ prepare(taos, 1, 0);
+
+ stmt = taos_stmt_init(taos);
+
+ printf("1t+10r+bm+autoctb+e2 start\n");
+ stmt_funcb_autoctb_e2(stmt);
+ printf("1t+10r+bm+autoctb+e2 end\n");
+ printf("check result start\n");
+ //check_result(taos, "m0", 1, 0);
+ printf("check result end\n");
+ taos_stmt_close(stmt);
+
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2836,7 +3946,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2856,7 +3966,7 @@ void* runcase(void *par) {
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2874,7 +3984,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2890,7 +4000,7 @@ void* runcase(void *par) {
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2908,7 +4018,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2927,7 +4037,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2952,7 +4062,7 @@ void* runcase(void *par) {
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
stmt = taos_stmt_init(taos);
@@ -2972,7 +4082,7 @@ void* runcase(void *par) {
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
(void)stmt;
printf("120t+60r+sql start\n");
@@ -2988,7 +4098,7 @@ void* runcase(void *par) {
#endif
#if 1
- prepare(taos, 1);
+ prepare(taos, 1, 1);
(void)stmt;
printf("1t+60r+sql start\n");
diff --git a/tests/script/general/parser/last_cache.sim b/tests/script/general/parser/last_cache.sim
new file mode 100644
index 0000000000000000000000000000000000000000..9d7da9ddbabbd668382740a230ecc9582c27fe84
--- /dev/null
+++ b/tests/script/general/parser/last_cache.sim
@@ -0,0 +1,71 @@
+system sh/stop_dnodes.sh
+system sh/deploy.sh -n dnode1 -i 1
+system sh/cfg.sh -n dnode1 -c walLevel -v 0
+system sh/cfg.sh -n dnode1 -c maxtablesPerVnode -v 4
+system sh/exec.sh -n dnode1 -s start
+
+sleep 100
+sql connect
+print ======================== dnode1 start
+
+$db = testdb
+
+sql create database $db cachelast 2
+sql use $db
+
+sql create stable st2 (ts timestamp, f1 int, f2 double, f3 binary(10), f4 timestamp) tags (id int)
+
+sql create table tb1 using st2 tags (1);
+sql create table tb2 using st2 tags (2);
+sql create table tb3 using st2 tags (3);
+sql create table tb4 using st2 tags (4);
+sql create table tb5 using st2 tags (1);
+sql create table tb6 using st2 tags (2);
+sql create table tb7 using st2 tags (3);
+sql create table tb8 using st2 tags (4);
+sql create table tb9 using st2 tags (5);
+sql create table tba using st2 tags (5);
+sql create table tbb using st2 tags (5);
+sql create table tbc using st2 tags (5);
+sql create table tbd using st2 tags (5);
+sql create table tbe using st2 tags (5);
+
+sql insert into tb1 values ("2021-05-09 10:10:10", 1, 2.0, '3', -1000)
+sql insert into tb1 values ("2021-05-10 10:10:11", 4, 5.0, NULL, -2000)
+sql insert into tb1 values ("2021-05-12 10:10:12", 6,NULL, NULL, -3000)
+
+sql insert into tb2 values ("2021-05-09 10:11:13",-1,-2.0,'-3', -1001)
+sql insert into tb2 values ("2021-05-10 10:11:14",-4,-5.0, NULL, -2001)
+sql insert into tb2 values ("2021-05-11 10:11:15",-6, -7, '-8', -3001)
+
+sql insert into tb3 values ("2021-05-09 10:12:17", 7, 8.0, '9' , -1002)
+sql insert into tb3 values ("2021-05-09 10:12:17",10,11.0, NULL, -2002)
+sql insert into tb3 values ("2021-05-09 10:12:18",12,NULL, NULL, -3002)
+
+sql insert into tb4 values ("2021-05-09 10:12:19",13,14.0,'15' , -1003)
+sql insert into tb4 values ("2021-05-10 10:12:20",16,17.0, NULL, -2003)
+sql insert into tb4 values ("2021-05-11 10:12:21",18,NULL, NULL, -3003)
+
+sql insert into tb5 values ("2021-05-09 10:12:22",19, 20, '21', -1004)
+sql insert into tb6 values ("2021-05-11 10:12:23",22, 23, NULL, -2004)
+sql insert into tb7 values ("2021-05-10 10:12:24",24,NULL, '25', -3004)
+sql insert into tb8 values ("2021-05-11 10:12:25",26,NULL, '27', -4004)
+
+sql insert into tb9 values ("2021-05-09 10:12:26",28, 29, '30', -1005)
+sql insert into tba values ("2021-05-10 10:12:27",31, 32, NULL, -2005)
+sql insert into tbb values ("2021-05-10 10:12:28",33,NULL, '35', -3005)
+sql insert into tbc values ("2021-05-11 10:12:29",36, 37, NULL, -4005)
+sql insert into tbd values ("2021-05-11 10:12:29",NULL,NULL,NULL,NULL )
+
+run general/parser/last_cache_query.sim
+
+system sh/exec.sh -n dnode1 -s stop -x SIGINT
+
+system sh/exec.sh -n dnode1 -s start
+
+run general/parser/last_cache_query.sim
+
+system sh/exec.sh -n dnode1 -s stop -x SIGINT
+
+
+
diff --git a/tests/script/general/parser/last_cache_query.sim b/tests/script/general/parser/last_cache_query.sim
new file mode 100644
index 0000000000000000000000000000000000000000..2acd00058592c3d4f70600aad1b2ee41a688f1e1
--- /dev/null
+++ b/tests/script/general/parser/last_cache_query.sim
@@ -0,0 +1,416 @@
+
+sleep 100
+sql connect
+
+$db = testdb
+
+sql use $db
+
+print "test tb1"
+
+sql select last(ts) from tb1
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+
+
+sql select last(f1) from tb1
+if $rows != 1 then
+ return -1
+endi
+if $data00 != 6 then
+ print $data00
+ return -1
+endi
+
+sql select last(*) from tb1
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 5.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 3 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+
+
+sql select last(tb1.*,ts,f4) from tb1
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 5.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 3 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+if $data05 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data06 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+
+
+
+
+print "test tb2"
+
+sql select last(ts) from tb2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-11 10:11:15.000@ then
+ print $data00
+ return -1
+endi
+
+
+sql select last(f1) from tb2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != -6 then
+ print $data00
+ return -1
+endi
+
+sql select last(*) from tb2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-11 10:11:15.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != -6 then
+ return -1
+endi
+if $data02 != -7.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != -8 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:56.999@ then
+ if $data04 != @70-01-01 07:59:57.-01@ then
+ return -1
+ endi
+endi
+
+
+sql select last(tb2.*,ts,f4) from tb2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-11 10:11:15.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != -6 then
+ return -1
+endi
+if $data02 != -7.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != -8 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:56.999@ then
+ if $data04 != @70-01-01 07:59:57.-01@ then
+ return -1
+ endi
+endi
+if $data05 != @21-05-11 10:11:15.000@ then
+ print $data00
+ return -1
+endi
+if $data06 != @70-01-01 07:59:56.999@ then
+ if $data04 != @70-01-01 07:59:57.-01@ then
+ return -1
+ endi
+endi
+
+
+
+
+
+
+
+print "test tbd"
+sql select last(*) from tbd
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-11 10:12:29.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != NULL then
+ return -1
+endi
+if $data02 != NULL then
+ print $data02
+ return -1
+endi
+if $data03 != NULL then
+ return -1
+endi
+if $data04 != NULL then
+ return -1
+endi
+
+
+
+print "test tbe"
+sql select last(*) from tbe
+if $rows != 0 then
+ return -1
+endi
+
+
+
+
+
+print "test stable"
+sql select last(ts) from st2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+
+
+sql select last(f1) from st2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != 6 then
+ print $data00
+ return -1
+endi
+
+sql select last(*) from st2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 37.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 27 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+
+
+sql select last(st2.*,ts,f4) from st2
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 37.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 27 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+if $data05 != @21-05-12 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data06 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+
+
+sql select last(*) from st2 group by id
+if $rows != 5 then
+ return -1
+endi
+if $data00 != @21-05-12 10:10:12.000@ then
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 5.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 21 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+if $data05 != 1 then
+ return -1
+endi
+if $data10 != @21-05-11 10:12:23.000@ then
+ return -1
+endi
+if $data11 != 22 then
+ return -1
+endi
+if $data12 != 23.000000000 then
+ print $data02
+ return -1
+endi
+if $data13 != -8 then
+ return -1
+endi
+if $data14 != @70-01-01 07:59:57.996@ then
+if $data14 != @70-01-01 07:59:58.-04@ then
+ print $data14
+ return -1
+endi
+endi
+if $data15 != 2 then
+ return -1
+endi
+if $data20 != @21-05-10 10:12:24.000@ then
+ return -1
+endi
+if $data21 != 24 then
+ return -1
+endi
+if $data22 != 8.000000000 then
+ print $data02
+ return -1
+endi
+if $data23 != 25 then
+ return -1
+endi
+if $data24 != @70-01-01 07:59:56.996@ then
+if $data24 != @70-01-01 07:59:57.-04@ then
+ return -1
+endi
+endi
+if $data25 != 3 then
+ return -1
+endi
+if $data30 != @21-05-11 10:12:25.000@ then
+ return -1
+endi
+if $data31 != 26 then
+ return -1
+endi
+if $data32 != 17.000000000 then
+ print $data02
+ return -1
+endi
+if $data33 != 27 then
+ return -1
+endi
+if $data34 != @70-01-01 07:59:55.996@ then
+if $data34 != @70-01-01 07:59:56.-04@ then
+ return -1
+endi
+endi
+if $data35 != 4 then
+ return -1
+endi
+if $data40 != @21-05-11 10:12:29.000@ then
+ return -1
+endi
+if $data41 != 36 then
+ return -1
+endi
+if $data42 != 37.000000000 then
+ print $data02
+ return -1
+endi
+if $data43 != 35 then
+ return -1
+endi
+if $data44 != @70-01-01 07:59:55.995@ then
+if $data44 != @70-01-01 07:59:56.-05@ then
+ return -1
+endi
+endi
+if $data45 != 5 then
+ return -1
+endi
+
+
+print "test tbn"
+sql create table tbn (ts timestamp, f1 int, f2 double, f3 binary(10), f4 timestamp)
+sql insert into tbn values ("2021-05-09 10:10:10", 1, 2.0, '3', -1000)
+sql insert into tbn values ("2021-05-10 10:10:11", 4, 5.0, NULL, -2000)
+sql insert into tbn values ("2021-05-12 10:10:12", 6,NULL, NULL, -3000)
+sql insert into tbn values ("2021-05-13 10:10:12", NULL,NULL, NULL,NULL)
+
+sql select last(*) from tbn;
+if $rows != 1 then
+ return -1
+endi
+if $data00 != @21-05-13 10:10:12.000@ then
+ print $data00
+ return -1
+endi
+if $data01 != 6 then
+ return -1
+endi
+if $data02 != 5.000000000 then
+ print $data02
+ return -1
+endi
+if $data03 != 3 then
+ return -1
+endi
+if $data04 != @70-01-01 07:59:57.000@ then
+ return -1
+endi
+
diff --git a/tests/script/general/parser/testSuite.sim b/tests/script/general/parser/testSuite.sim
index f263593b3d70fe35144ed2667b879c6c89601fb7..afb76c799ecc2c7c55058cd4f0e287975c0d7d45 100644
--- a/tests/script/general/parser/testSuite.sim
+++ b/tests/script/general/parser/testSuite.sim
@@ -58,4 +58,6 @@ run general/parser/having.sim
run general/parser/having_child.sim
run general/parser/slimit_alter_tags.sim
run general/parser/binary_escapeCharacter.sim
+run general/parser/between_and.sim
+run general/parser/last_cache.sim