未验证 提交 07163f60 编写于 作者: S shenglian-zhou 提交者: GitHub

Merge branch 'develop' into szhou/fix/td-10799

......@@ -83,7 +83,7 @@ TDengine是一个高效的存储、查询、分析时序大数据的平台,专
* [Windows客户端](https://www.taosdata.com/blog/2019/07/26/514.html):自行编译windows客户端,Windows环境的各种连接器都需要它
* [Rust Connector](/connector/rust): Rust语言下通过libtaos客户端或RESTful接口,连接TDengine服务器。
## [TDengine 组件与工具](/cn/documentation/)
## TDengine 组件与工具
* [taosAdapter 用户手册](/tools/adapter)
* [TDinsight 用户手册](/tools/insight)
......
......@@ -20,7 +20,7 @@ INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31) (1538548695000, 12.6,
详细的SQL INSERT语法规则请见 [TAOS SQL 的数据写入](https://www.taosdata.com/cn/documentation/taos-sql#insert) 章节。
**Tips:**
**Tips:**
- 要提高写入效率,需要批量写入。一批写入的记录条数越多,插入效率就越高。但一条记录不能超过16K,一条SQL语句总长度不能超过1M 。
- TDengine支持多线程同时写入,要进一步提高写入速度,一个客户端需要打开20个以上的线程同时写。但线程数达到一定数量后,无法再提高,甚至还会下降,因为线程频繁切换,带来额外开销。
......@@ -56,7 +56,7 @@ tag_set 中的所有的数据自动转化为 nchar 数据类型,并不需要
* 对空格、等号(=)、逗号(,)、双引号("),前面需要使用反斜杠(\)进行转义。(都指的是英文半角符号)
* 数值类型将通过后缀来区分数据类型:
| **序号** | **后缀** | **映射类型** | **大小(字节)** |
| **序号** | **后缀** | **映射类型** | **大小(字节)** |
| -- | ------- | ---------| ------ |
| 1 | 无或f64 | double | 8 |
| 2 | f32 | float | 4 |
......@@ -231,16 +231,16 @@ prometheus产生的数据格式如下:
```json
{
Timestamp: 1576466279341,
Value: 37.000000,
Value: 37.000000,
apiserver_request_latencies_bucket {
component="apiserver",
instance="192.168.99.116:8443",
job="kubernetes-apiservers",
le="125000",
resource="persistentvolumes",
component="apiserver",
instance="192.168.99.116:8443",
job="kubernetes-apiservers",
le="125000",
resource="persistentvolumes",
scope="cluster",
verb="LIST",
version="v1"
verb="LIST",
version="v1"
}
}
```
......@@ -251,6 +251,7 @@ select * from apiserver_request_latencies_bucket;
```
## <a class="anchor" id="telegraf"></a> Telegraf 直接写入(通过 taosAdapter)
安装 Telegraf 请参考[官方文档](https://portal.influxdata.com/downloads/)
TDengine 新版本(2.3.0.0+)包含一个 taosAdapter 独立程序,负责接收包括 Telegraf 的多种应用的数据写入。
......@@ -276,6 +277,7 @@ sudo systemctl start telegraf
taosAdapter 相关配置参数请参考 taosadapter --help 命令输出以及相关文档。
## <a class="anchor" id="collectd"></a> collectd 直接写入(通过 taosAdapter)
安装 collectd,请参考[官方文档](https://collectd.org/download.shtml)
TDengine 新版本(2.3.0.0+)包含一个 taosAdapter 独立程序,负责接收包括 collectd 的多种应用的数据写入。
......@@ -294,6 +296,7 @@ sudo systemctl start collectd
taosAdapter 相关配置参数请参考 taosadapter --help 命令输出以及相关文档。
## <a class="anchor" id="statsd"></a> StatsD 直接写入(通过 taosAdapter)
安装 StatsD
请参考[官方文档](https://github.com/statsd/statsd)
......@@ -316,6 +319,30 @@ port: 8125
taosAdapter 相关配置参数请参考 taosadapter --help 命令输出以及相关文档。
icinga2 可以收集监控和性能数据并写入 OpenTSDB,taosAdapter 可以支持接收 icinga2 的数据并写入到 TDengine 中。
## <a class="anchor" id="icinga2"></a> icinga2 直接写入(通过 taosAdapter)
* 参考链接 https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer 使能 opentsdb-writer
* 使能 taosAdapter 配置项 opentsdb_telnet.enable
* 修改配置文件 /etc/icinga2/features-enabled/opentsdb.conf
```
object OpenTsdbWriter "opentsdb" {
host = "host to taosAdapter"
port = 6048
}
```
taosAdapter 相关配置参数请参考 taosadapter --help 命令输出以及相关文档。
## <a class="anchor" id="tcollector"></a> TCollector 直接写入(通过 taosAdapter)
TCollector 是一个在客户侧收集本地收集器并发送数据到 OpenTSDB 的进程,taosAdaapter 可以支持接收 TCollector 的数据并写入到 TDengine 中。
使能 taosAdapter 配置项 opentsdb_telnet.enable
修改 TCollector 配置文件,修改 OpenTSDB 宿主机地址为 taosAdapter 被部署的地址,并修改端口号为 taosAdapter 使用的端口(默认6049)。
taosAdapter 相关配置参数请参考 taosadapter --help 命令输出以及相关文档。
## <a class="anchor" id="taosadapter2-telegraf"></a> 使用 Bailongma 2.0 接入 Telegraf 数据写入
......
......@@ -1851,3 +1851,24 @@ TDengine 中的表(列)名命名规则如下:
```mysql
select jtag->'key' from (select jtag from stable) where jtag->'key'>0
```
## 转义字符说明
- 转义字符表
| 字符序列 | **代表的字符** |
| :--------: | ------- |
| `\'` | 单引号' |
| `\"` | 双引号" |
| \n | 换行符 |
| \r | 回车符 |
| \t | tab符 |
| `\\` | 斜杠\ |
| `\%` | % 规则见下 |
| `\%` | _ 规则见下 |
- 转义字符使用规则
1. 标识符里有转义字符(数据库名、表名、列名)
1. 普通标识符: 直接提示错误的标识符,因为标识符规定必须是数字、字母和下划线,并且不能以数字开头。
2. 反引号``标识符: 保持原样,不转义
2. 数据里有转义字符
1. 遇到上面定义的转义字符会转义(%和_见下面说明),如果没有匹配的转义字符会忽略掉转义符\。
2. 对于%和_,因为在like里这两个字符是通配符,所以在模式匹配like里用`\%`%和`\_`表示字符里本身的%和_,如果在like模式匹配上下文之外使用`\%`或`\_`,则它们的计算结果为字符串`\%`和`\_`,而不是%和_。
\ No newline at end of file
......@@ -2,7 +2,7 @@
TDengine supports multiple ways to write data, including SQL, Prometheus, Telegraf, collectd, StatsD, EMQ MQTT Broker, HiveMQ Broker, CSV file, etc. Kafka, OPC and other interfaces will be provided in the future. Data can be inserted in one single record or in batches, data from one or multiple data collection points can be inserted at the same time. TDengine supports multi-thread insertion, out-of-order data insertion, and also historical data insertion.
## <a class="anchor" id="sql"></a> Data Writing via SQL
## <a class="anchor" id="sql"></a> Data Writing via SQL
Applications insert data by executing SQL insert statements through C/C++, Java, Go, C#, Python, Node.js Connectors, and users can manually enter SQL insert statements to insert data through TAOS Shell. For example, the following insert writes a record to table d1001:
......@@ -119,16 +119,16 @@ The format of generated data by Prometheus is as follows:
```json
{
Timestamp: 1576466279341,
Value: 37.000000,
Value: 37.000000,
apiserver_request_latencies_bucket {
component="apiserver",
instance="192.168.99.116:8443",
job="kubernetes-apiservers",
le="125000",
component="apiserver",
instance="192.168.99.116:8443",
job="kubernetes-apiservers",
le="125000",
resource="persistentvolumes", s
cope="cluster",
verb="LIST",
version=“v1"
verb="LIST",
version=“v1"
}
}
```
......@@ -167,13 +167,13 @@ Now you can query the metrics data of Telegraf from TDengine.
Please find taosAdapter configuration and usage from `taosadapter --help` output.
## <a class="anchor" id="collectd"></a> collectd 直接写入(通过 taosAdapter)
## <a class="anchor" id="collectd"></a> Data Writing via collectd and taosAdapter
Please refer to [official document](https://collectd.org/download.shtml) for collectd installation.
TDengine version 2.3.0.0+ includes a stand-alone application taosAdapter in charge of receive data insertion from collectd.
Configuration:
Please add following words in /etc/collectd/collectd.conf. Please fill the value 'host' and 'port' with what the TDengine and taosAdapter using.
Please add following words in /etc/collectd/collectd.conf. Please fill the value 'host' and 'port' with what the TDengine and taosAdapter using.
```
LoadPlugin network
<Plugin network>
......@@ -186,12 +186,12 @@ sudo systemctl start collectd
```
Please find taosAdapter configuration and usage from `taosadapter --help` output.
## <a class="anchor" id="statsd"></a> StatsD 直接写入(通过 taosAdapter)
## <a class="anchor" id="statsd"></a> Data Writting via StatsD and taosAdapter
Please refer to [official document](https://github.com/statsd/statsd) for StatsD installation.
TDengine version 2.3.0.0+ includes a stand-alone application taosAdapter in charge of receive data insertion from StatsD.
Please add following words in the config.js file. Please fill the value to 'host' and 'port' with what the TDengine and taosAdapter using.
Please add following words in the config.js file. Please fill the value to 'host' and 'port' with what the TDengine and taosAdapter using.
```
add "./backends/repeater" to backends section.
add { host:'<TDengine server/cluster host>', port: <port for StatsD>} to repeater section.
......@@ -206,8 +206,30 @@ port: 8125
}
```
## <a class="anchor" id="cinga2"></a> Data Writting via icinga2 and taosAdapter
Use icinga2 to collect check result metrics and performance data
* Follow the doc to enable opentsdb-writer https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer
* Enable taosAdapter configuration opentsdb_telnet.enable
* Modify the configuration file /etc/icinga2/features-enabled/opentsdb.conf
```
object OpenTsdbWriter "opentsdb" {
host = "host to taosAdapter"
port = 6048
}
```
Please find taosAdapter configuration and usage from `taosadapter --help` output.
## <a class="anchor" id="tcollector"></a> Data Writting via TCollector and taosAdapter
TCollector is a client-side process that gathers data from local collectors and pushes the data to OpenTSDB. You run it on all your hosts, and it does the work of sending each host’s data to the TSD (OpenTSDB backend process).
* Enable taosAdapter configuration opentsdb_telnet.enable
* Modify the TCollector configuration file, modify the OpenTSDB host to the host where taosAdapter is deployed, and modify the port to 6049
Please find taosAdapter configuration and usage from `taosadapter --help` output.
## <a class="anchor" id="taosadapter2-telegraf"></a> Insert data via Bailongma 2.0 and Telegraf
......
......@@ -1335,3 +1335,24 @@ Is not null supports all types of columns. Non-null expression is < > "" and onl
select jtag->'key' from (select jtag from stable) where jtag->'key'>0
```
## Escape character description
- Special Character Escape Sequences
| Escape Sequence | **Character Represented by Sequence** |
| :--------: | ------------------- |
| `\'` | A single quote (') character |
| `\"` | A double quote (") character |
| \n | A newline (linefeed) character |
| \r | A carriage return character |
| \t | A tab character |
| `\\` | A backslash (\) character |
| `\%` | A % character; see note following the table |
| `\_` | A _ character; see note following the table |
- Escape character usage rules
- The escape characters that in a identifier (database name, table name, column name)
1. Normal identifier: The wrong identifier is prompted directly, because the identifier must be numbers, letters and underscores, and cannot start with a number.
2. Backquote`` identifier: Keep it as it is.
- The escape characters that in a data
3. The escape character defined above will be escaped (% and _ see the description below). If there is no matching escape character, the escape character will be ignored.
4. The `\%` and `\_` sequences are used to search for literal instances of % and _ in pattern-matching contexts where they would otherwise be interpreted as wildcard characters.If you use `\%` or `\_` outside of pattern-matching contexts, they evaluate to the strings `\%` and `\_`, not to % and _.
\ No newline at end of file
......@@ -199,7 +199,7 @@ function install_bin() {
[ -x ${install_main_dir}/bin/taos ] && ${csudo}ln -s ${install_main_dir}/bin/taos ${bin_link_dir}/taos || :
[ -x ${install_main_dir}/bin/taosd ] && ${csudo}ln -s ${install_main_dir}/bin/taosd ${bin_link_dir}/taosd || :
[ -x ${install_main_dir}/bin/taosadapter ] && ${csudo}ln -s ${install_main_dir}/bin/taosadapter ${bin_link_dir}/taosadapter || :
[ -x ${install_main_dir}/bin/taosdemo ] && ${csudo}ln -s ${install_main_dir}/bin/taosdemo ${bin_link_dir}/taosdemo || :
[ -x ${install_main_dir}/bin/taosBenchmark ] && ${csudo}ln -s ${install_main_dir}/bin/taosBenchmark ${bin_link_dir}/taosdemo || :
[ -x ${install_main_dir}/bin/taosdump ] && ${csudo}ln -s ${install_main_dir}/bin/taosdump ${bin_link_dir}/taosdump || :
[ -x ${install_main_dir}/bin/remove.sh ] && ${csudo}ln -s ${install_main_dir}/bin/remove.sh ${bin_link_dir}/rmtaos || :
[ -x ${install_main_dir}/bin/set_core.sh ] && ${csudo}ln -s ${install_main_dir}/bin/set_core.sh ${bin_link_dir}/set_core || :
......
......@@ -123,9 +123,9 @@ if [ -n "${taostools_bin_files}" ]; then
mkdir -p ${taostools_install_dir}/bin \
&& cp ${taostools_bin_files} ${taostools_install_dir}/bin \
&& chmod a+x ${taostools_install_dir}/bin/* || :
[ -f ${taostools_install_dir}/bin/taosBenchmark ] && \
ln -sf ${taostools_install_dir}/bin/taosBenchmark \
${taostools_install_dir}/bin/taosdemo
# [ -f ${taostools_install_dir}/bin/taosBenchmark ] && \
# ln -sf ${taostools_install_dir}/bin/taosBenchmark \
# ${taostools_install_dir}/bin/taosdemo
if [ -f ${top_dir}/src/kit/taos-tools/packaging/tools/install-taostools.sh ]; then
cp ${top_dir}/src/kit/taos-tools/packaging/tools/install-taostools.sh \
......
......@@ -258,8 +258,6 @@ void tscColumnListCopyAll(SArray* dst, const SArray* src);
void convertQueryResult(SSqlRes* pRes, SQueryInfo* pQueryInfo, uint64_t objId, bool convertNchar, bool convertJson);
void tscDequoteAndTrimToken(SStrToken* pToken);
void tscRmEscapeAndTrimToken(SStrToken* pToken);
int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded);
void tscIncStreamExecutionCount(void* pStream);
......
......@@ -52,3 +52,4 @@ taos_stmt_bind_single_param_batch
taos_is_null
taos_insert_lines
taos_schemaless_insert
taos_result_block
......@@ -481,32 +481,12 @@ int tsParseOneRow(char **str, STableDataBlocks *pDataBlocks, int16_t timePrec, i
// Remove quotation marks
if (TK_STRING == sToken.type) {
// delete escape character: \\, \', \"
char delim = sToken.z[0];
int32_t cnt = 0;
int32_t j = 0;
if (sToken.n >= TSDB_MAX_BYTES_PER_ROW) {
return tscSQLSyntaxErrMsg(pInsertParam->msg, "too long string", sToken.z);
}
for (uint32_t k = 1; k < sToken.n - 1; ++k) {
if (sToken.z[k] == '\\' || (sToken.z[k] == delim && sToken.z[k + 1] == delim)) {
tmpTokenBuf[j] = sToken.z[k + 1];
cnt++;
j++;
k++;
continue;
}
tmpTokenBuf[j] = sToken.z[k];
j++;
}
tmpTokenBuf[j] = 0;
strncpy(tmpTokenBuf, sToken.z, sToken.n);
sToken.n = stringProcess(tmpTokenBuf, sToken.n);
sToken.z = tmpTokenBuf;
sToken.n -= 2 + cnt;
}
bool isPrimaryKey = (colIndex == PRIMARYKEY_TIMESTAMP_COL_INDEX);
......@@ -1057,10 +1037,12 @@ static int32_t tscCheckIfCreateTable(char **sqlstr, SSqlObj *pSql, char** boundC
break;
}
char* tmp = NULL;
// Remove quotation marks
if (TK_STRING == sToken.type) {
sToken.z++;
sToken.n -= 2;
tmp = strndup(sToken.z, sToken.n);
sToken.n = stringProcess(tmp, sToken.n);
sToken.z = tmp;
}
char tagVal[TSDB_MAX_TAGS_LEN] = {0};
......@@ -1068,6 +1050,7 @@ static int32_t tscCheckIfCreateTable(char **sqlstr, SSqlObj *pSql, char** boundC
if (code != TSDB_CODE_SUCCESS) {
tdDestroyKVRowBuilder(&kvRowBuilder);
tscDestroyBoundColumnInfo(&spd);
tfree(tmp);
return code;
}
......@@ -1078,18 +1061,18 @@ static int32_t tscCheckIfCreateTable(char **sqlstr, SSqlObj *pSql, char** boundC
if(sToken.n > TSDB_MAX_JSON_TAGS_LEN/TSDB_NCHAR_SIZE){
tdDestroyKVRowBuilder(&kvRowBuilder);
tscDestroyBoundColumnInfo(&spd);
tfree(tmp);
return tscSQLSyntaxErrMsg(pInsertParam->msg, "json tag too long", NULL);
}
char* json = strndup(sToken.z, sToken.n);
code = parseJsontoTagData(json, &kvRowBuilder, pInsertParam->msg, pTagSchema[spd.boundedColumns[0]].colId);
code = parseJsontoTagData(sToken.z, &kvRowBuilder, pInsertParam->msg, pTagSchema[spd.boundedColumns[0]].colId);
if (code != TSDB_CODE_SUCCESS) {
tdDestroyKVRowBuilder(&kvRowBuilder);
tscDestroyBoundColumnInfo(&spd);
tfree(json);
tfree(tmp);
return code;
}
tfree(json);
}
tfree(tmp);
}
tscDestroyBoundColumnInfo(&spd);
......@@ -1246,12 +1229,8 @@ static int32_t parseBoundColumns(SInsertStatementParam *pInsertParam, SParsedDat
strncpy(tmpTokenBuf, sToken.z, sToken.n);
sToken.z = tmpTokenBuf;
if (TK_STRING == sToken.type) {
tscDequoteAndTrimToken(&sToken);
}
if (TK_ID == sToken.type) {
tscRmEscapeAndTrimToken(&sToken);
if (TK_STRING == sToken.type || TK_ID == sToken.type) {
sToken.n = stringProcess(sToken.z, sToken.n);
}
if (sToken.type == TK_RP) {
......@@ -1371,7 +1350,7 @@ _clean:
static int32_t getFileFullPath(SStrToken* pToken, char* output) {
char path[PATH_MAX] = {0};
strncpy(path, pToken->z, pToken->n);
strdequote(path);
stringProcess(path, (int32_t)strlen(path));
wordexp_t full_path;
if (wordexp(path, &full_path, 0) != 0) {
......
......@@ -20,7 +20,7 @@
#include "tscParseLine.h"
typedef struct {
char sTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE];
char sTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE];
SHashObj* tagHash;
SHashObj* fieldHash;
SArray* tags; //SArray<SSchema>
......@@ -68,13 +68,13 @@ typedef enum {
} ESchemaAction;
typedef struct {
char sTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE];
char sTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE];
SArray* tags; //SArray<SSchema>
SArray* fields; //SArray<SSchema>
} SCreateSTableActionInfo;
typedef struct {
char sTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE];
char sTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE];
SSchema* field;
} SAlterSTableActionInfo;
......@@ -161,14 +161,14 @@ static int32_t getSmlMd5ChildTableName(TAOS_SML_DATA_POINT* point, char* tableNa
}
SStringBuilder sb; memset(&sb, 0, sizeof(sb));
char sTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char sTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
strncpy(sTableName, point->stableName, strlen(point->stableName));
//strtolower(sTableName, point->stableName);
taosStringBuilderAppendString(&sb, sTableName);
for (int j = 0; j < point->tagNum; ++j) {
taosStringBuilderAppendChar(&sb, ',');
TAOS_SML_KV* tagKv = point->tags + j;
char tagName[TSDB_COL_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char tagName[TSDB_COL_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
strncpy(tagName, tagKv->key, strlen(tagKv->key));
//strtolower(tagName, tagKv->key);
taosStringBuilderAppendString(&sb, tagName);
......@@ -192,8 +192,8 @@ static int32_t getSmlMd5ChildTableName(TAOS_SML_DATA_POINT* point, char* tableNa
static int32_t buildSmlChildTableName(TAOS_SML_DATA_POINT* point, SSmlLinesInfo* info) {
tscDebug("SML:0x%"PRIx64" taos_sml_insert build child table name", info->id);
char childTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE];
int32_t tableNameLen = TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE;
char childTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE];
int32_t tableNameLen = TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE;
getSmlMd5ChildTableName(point, childTableName, &tableNameLen, info);
point->childTableName = calloc(1, tableNameLen+1);
strncpy(point->childTableName, childTableName, tableNameLen);
......@@ -251,15 +251,15 @@ static int32_t buildDataPointSchemas(TAOS_SML_DATA_POINT* points, int numPoint,
size_t nameLen = strlen(tsSmlTagNullName);
strncpy(tagNullName, tsSmlTagNullName, nameLen);
addEscapeCharToString(tagNullName, (int32_t)nameLen);
size_t* pTagNullIdx = taosHashGet(pStableSchema->tagHash, tagNullName, nameLen + TS_ESCAPE_CHAR_SIZE);
size_t* pTagNullIdx = taosHashGet(pStableSchema->tagHash, tagNullName, nameLen + TS_BACKQUOTE_CHAR_SIZE);
if (!pTagNullIdx) {
SSchema tagNull = {0};
tagNull.type = TSDB_DATA_TYPE_NCHAR;
tagNull.bytes = TSDB_NCHAR_SIZE + VARSTR_HEADER_SIZE;
strncpy(tagNull.name, tagNullName, nameLen + TS_ESCAPE_CHAR_SIZE);
strncpy(tagNull.name, tagNullName, nameLen + TS_BACKQUOTE_CHAR_SIZE);
taosArrayPush(pStableSchema->tags, &tagNull);
size_t tagNullIdx = taosArrayGetSize(pStableSchema->tags) - 1;
taosHashPut(pStableSchema->tagHash, tagNull.name, nameLen + TS_ESCAPE_CHAR_SIZE, &tagNullIdx, sizeof(tagNullIdx));
taosHashPut(pStableSchema->tagHash, tagNull.name, nameLen + TS_BACKQUOTE_CHAR_SIZE, &tagNullIdx, sizeof(tagNullIdx));
}
}
......@@ -295,7 +295,7 @@ static int32_t buildDataPointSchemas(TAOS_SML_DATA_POINT* points, int numPoint,
static int32_t generateSchemaAction(SSchema* pointColField, SHashObj* dbAttrHash, SArray* dbAttrArray, bool isTag, char sTableName[],
SSchemaAction* action, bool* actionNeeded, SSmlLinesInfo* info) {
char fieldName[TSDB_COL_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char fieldName[TSDB_COL_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
strcpy(fieldName, pointColField->name);
size_t* pDbIndex = taosHashGet(dbAttrHash, fieldName, strlen(fieldName));
......@@ -315,7 +315,7 @@ static int32_t generateSchemaAction(SSchema* pointColField, SHashObj* dbAttrHash
action->action = SCHEMA_ACTION_CHANGE_COLUMN_SIZE;
}
memset(&action->alterSTable, 0, sizeof(SAlterSTableActionInfo));
memcpy(action->alterSTable.sTableName, sTableName, TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE);
memcpy(action->alterSTable.sTableName, sTableName, TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE);
action->alterSTable.field = pointColField;
*actionNeeded = true;
}
......@@ -326,7 +326,7 @@ static int32_t generateSchemaAction(SSchema* pointColField, SHashObj* dbAttrHash
action->action = SCHEMA_ACTION_ADD_COLUMN;
}
memset(&action->alterSTable, 0, sizeof(SAlterSTableActionInfo));
memcpy(action->alterSTable.sTableName, sTableName, TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE);
memcpy(action->alterSTable.sTableName, sTableName, TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE);
action->alterSTable.field = pointColField;
*actionNeeded = true;
}
......@@ -572,7 +572,7 @@ static int32_t getSuperTableMetaFromLocalCache(TAOS* taos, char* tableName, STab
pSql->fp = NULL;
registerSqlObj(pSql);
char tableNameBuf[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char tableNameBuf[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
memcpy(tableNameBuf, tableName, strlen(tableName));
SStrToken tableToken = {.z = tableNameBuf, .n = (uint32_t)strlen(tableName), .type = TK_ID};
tGetToken(tableNameBuf, &tableToken.type);
......@@ -689,7 +689,7 @@ static int32_t modifyDBSchemas(TAOS* taos, SArray* stableSchemas, SSmlLinesInfo*
SSchemaAction schemaAction = {0};
schemaAction.action = SCHEMA_ACTION_CREATE_STABLE;
memset(&schemaAction.createSTable, 0, sizeof(SCreateSTableActionInfo));
memcpy(schemaAction.createSTable.sTableName, pointSchema->sTableName, TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE);
memcpy(schemaAction.createSTable.sTableName, pointSchema->sTableName, TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE);
schemaAction.createSTable.tags = pointSchema->tags;
schemaAction.createSTable.fields = pointSchema->fields;
applySchemaAction(taos, &schemaAction, info);
......@@ -726,7 +726,7 @@ static int32_t modifyDBSchemas(TAOS* taos, SArray* stableSchemas, SSmlLinesInfo*
SSchema* pointColTs = taosArrayGet(pointSchema->fields, 0);
SSchema* dbColTs = taosArrayGet(dbSchema.fields, 0);
memcpy(pointColTs->name, dbColTs->name, TSDB_COL_NAME_LEN + TS_ESCAPE_CHAR_SIZE);
memcpy(pointColTs->name, dbColTs->name, TSDB_COL_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE);
for (int j = 1; j < pointFieldSize; ++j) {
SSchema* pointCol = taosArrayGet(pointSchema->fields, j);
......@@ -1398,7 +1398,7 @@ char* addEscapeCharToString(char *str, int32_t len) {
return NULL;
}
memmove(str + 1, str, len);
str[0] = str[len + 1] = TS_ESCAPE_CHAR;
str[0] = str[len + 1] = TS_BACKQUOTE_CHAR;
str[len + 2] = '\0';
return str;
}
......@@ -2129,7 +2129,7 @@ static int32_t parseSmlKey(TAOS_SML_KV *pKV, const char **index, SHashObj *pHash
return TSDB_CODE_TSC_LINE_SYNTAX_ERROR;
}
pKV->key = calloc(len + TS_ESCAPE_CHAR_SIZE + 1, 1);
pKV->key = calloc(len + TS_BACKQUOTE_CHAR_SIZE + 1, 1);
memcpy(pKV->key, key, len + 1);
addEscapeCharToString(pKV->key, len);
tscDebug("SML:0x%"PRIx64" Key:%s|len:%d", info->id, pKV->key, len);
......@@ -2227,7 +2227,7 @@ static int32_t parseSmlMeasurement(TAOS_SML_DATA_POINT *pSml, const char **index
const char *cur = *index;
int16_t len = 0;
pSml->stableName = calloc(TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE, 1);
pSml->stableName = calloc(TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE, 1);
if (pSml->stableName == NULL){
return TSDB_CODE_TSC_OUT_OF_MEMORY;
}
......@@ -2313,7 +2313,7 @@ static int32_t parseSmlKvPairs(TAOS_SML_KV **pKVs, int *num_kvs,
}
size_t childTableNameLen = strlen(tsSmlChildTableName);
char childTableName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char childTableName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
if (childTableNameLen != 0) {
memcpy(childTableName, tsSmlChildTableName, childTableNameLen);
addEscapeCharToString(childTableName, (int32_t)(childTableNameLen));
......@@ -2332,7 +2332,7 @@ static int32_t parseSmlKvPairs(TAOS_SML_KV **pKVs, int *num_kvs,
}
if (!isField && childTableNameLen != 0 && strcasecmp(pkv->key, childTableName) == 0) {
smlData->childTableName = malloc(pkv->length + TS_ESCAPE_CHAR_SIZE + 1);
smlData->childTableName = malloc(pkv->length + TS_BACKQUOTE_CHAR_SIZE + 1);
memcpy(smlData->childTableName, pkv->value, pkv->length);
addEscapeCharToString(smlData->childTableName, (int32_t)pkv->length);
free(pkv->key);
......
......@@ -37,7 +37,7 @@ static int32_t parseTelnetMetric(TAOS_SML_DATA_POINT *pSml, const char **index,
const char *cur = *index;
uint16_t len = 0;
pSml->stableName = tcalloc(TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE, 1);
pSml->stableName = tcalloc(TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE, 1);
if (pSml->stableName == NULL) {
return TSDB_CODE_TSC_OUT_OF_MEMORY;
}
......@@ -125,7 +125,7 @@ static int32_t parseTelnetTimeStamp(TAOS_SML_KV **pTS, int *num_kvs, const char
}
tfree(value);
(*pTS)->key = tcalloc(sizeof(key) + TS_ESCAPE_CHAR_SIZE, 1);
(*pTS)->key = tcalloc(sizeof(key) + TS_BACKQUOTE_CHAR_SIZE, 1);
memcpy((*pTS)->key, key, sizeof(key));
addEscapeCharToString((*pTS)->key, (int32_t)strlen(key));
......@@ -196,7 +196,7 @@ static int32_t parseTelnetMetricValue(TAOS_SML_KV **pKVs, int *num_kvs, const ch
}
tfree(value);
pVal->key = tcalloc(sizeof(key) + TS_ESCAPE_CHAR_SIZE, 1);
pVal->key = tcalloc(sizeof(key) + TS_BACKQUOTE_CHAR_SIZE, 1);
memcpy(pVal->key, key, sizeof(key));
addEscapeCharToString(pVal->key, (int32_t)strlen(pVal->key));
*num_kvs += 1;
......@@ -240,7 +240,7 @@ static int32_t parseTelnetTagKey(TAOS_SML_KV *pKV, const char **index, SHashObj
return TSDB_CODE_TSC_DUP_TAG_NAMES;
}
pKV->key = tcalloc(len + TS_ESCAPE_CHAR_SIZE + 1, 1);
pKV->key = tcalloc(len + TS_BACKQUOTE_CHAR_SIZE + 1, 1);
memcpy(pKV->key, key, len + 1);
addEscapeCharToString(pKV->key, len);
//tscDebug("OTD:0x%"PRIx64" Key:%s|len:%d", info->id, pKV->key, len);
......@@ -307,7 +307,7 @@ static int32_t parseTelnetTagKvs(TAOS_SML_KV **pKVs, int *num_kvs,
pkv = *pKVs;
size_t childTableNameLen = strlen(tsSmlChildTableName);
char childTbName[TSDB_TABLE_NAME_LEN + TS_ESCAPE_CHAR_SIZE] = {0};
char childTbName[TSDB_TABLE_NAME_LEN + TS_BACKQUOTE_CHAR_SIZE] = {0};
if (childTableNameLen != 0) {
memcpy(childTbName, tsSmlChildTableName, childTableNameLen);
addEscapeCharToString(childTbName, (int32_t)(childTableNameLen));
......@@ -324,7 +324,7 @@ static int32_t parseTelnetTagKvs(TAOS_SML_KV **pKVs, int *num_kvs,
return ret;
}
if (childTableNameLen != 0 && strcasecmp(pkv->key, childTbName) == 0) {
*childTableName = tcalloc(pkv->length + TS_ESCAPE_CHAR_SIZE + 1, 1);
*childTableName = tcalloc(pkv->length + TS_BACKQUOTE_CHAR_SIZE + 1, 1);
memcpy(*childTableName, pkv->value, pkv->length);
(*childTableName)[pkv->length] = '\0';
addEscapeCharToString(*childTableName, pkv->length);
......@@ -500,7 +500,7 @@ static int32_t parseMetricFromJSON(cJSON *root, TAOS_SML_DATA_POINT* pSml, SSmlL
return TSDB_CODE_TSC_INVALID_TABLE_ID_LENGTH;
}
pSml->stableName = tcalloc(stableLen + TS_ESCAPE_CHAR_SIZE + 1, sizeof(char));
pSml->stableName = tcalloc(stableLen + TS_BACKQUOTE_CHAR_SIZE + 1, sizeof(char));
if (pSml->stableName == NULL){
return TSDB_CODE_TSC_OUT_OF_MEMORY;
}
......@@ -879,7 +879,7 @@ static int32_t parseMetricValueFromJSON(cJSON *root, TAOS_SML_KV **pKVs, int *nu
return ret;
}
pVal->key = tcalloc(sizeof(key) + TS_ESCAPE_CHAR_SIZE, 1);
pVal->key = tcalloc(sizeof(key) + TS_BACKQUOTE_CHAR_SIZE, 1);
memcpy(pVal->key, key, sizeof(key));
addEscapeCharToString(pVal->key, (int32_t)strlen(pVal->key));
......@@ -910,7 +910,7 @@ static int32_t parseTagsFromJSON(cJSON *root, TAOS_SML_KV **pKVs, int *num_kvs,
return TSDB_CODE_TSC_INVALID_JSON;
}
size_t idLen = strlen(id->valuestring);
*childTableName = tcalloc(idLen + TS_ESCAPE_CHAR_SIZE + 1, sizeof(char));
*childTableName = tcalloc(idLen + TS_BACKQUOTE_CHAR_SIZE + 1, sizeof(char));
memcpy(*childTableName, id->valuestring, idLen);
addEscapeCharToString(*childTableName, (int32_t)idLen);
......@@ -948,7 +948,7 @@ static int32_t parseTagsFromJSON(cJSON *root, TAOS_SML_KV **pKVs, int *num_kvs,
tscError("OTD:0x%"PRIx64" Tag key cannot exceeds %d characters in JSON", info->id, TSDB_COL_NAME_LEN - 1);
return TSDB_CODE_TSC_INVALID_COLUMN_LENGTH;
}
pkv->key = tcalloc(keyLen + TS_ESCAPE_CHAR_SIZE + 1, sizeof(char));
pkv->key = tcalloc(keyLen + TS_BACKQUOTE_CHAR_SIZE + 1, sizeof(char));
strncpy(pkv->key, tag->string, keyLen);
addEscapeCharToString(pkv->key, (int32_t)keyLen);
//value
......
......@@ -322,7 +322,7 @@ static int32_t invalidOperationMsg(char* dstBuffer, const char* errMsg) {
static int convertTimestampStrToInt64(tVariant *pVar, int32_t precision) {
int64_t time = 0;
strdequote(pVar->pz);
stringProcess(pVar->pz, pVar->nLen);
char* seg = strnchr(pVar->pz, '-', pVar->nLen, false);
if (seg != NULL) {
......@@ -359,7 +359,7 @@ static int32_t handlePassword(SSqlCmd* pCmd, SStrToken* pPwd) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg3);
}
strdequote(pPwd->z);
stringProcess(pPwd->z, pPwd->n);
pPwd->n = (uint32_t)strtrim(pPwd->z); // trim space before and after passwords
if (pPwd->n <= 0) {
......@@ -477,7 +477,7 @@ int32_t handleUserDefinedFunc(SSqlObj* pSql, struct SSqlInfo* pInfo) {
if (validateColumnName(createInfo->name.z) != TSDB_CODE_SUCCESS) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg1);
}
strdequote(createInfo->name.z);
stringProcess(createInfo->name.z, createInfo->name.n);
if (strlen(createInfo->name.z) >= TSDB_FUNC_NAME_LEN) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg1);
......@@ -485,7 +485,7 @@ int32_t handleUserDefinedFunc(SSqlObj* pSql, struct SSqlInfo* pInfo) {
createInfo->path.z[createInfo->path.n] = 0;
strdequote(createInfo->path.z);
stringProcess(createInfo->path.z, createInfo->path.n);
if (strlen(createInfo->path.z) >= PATH_MAX) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg2);
......@@ -543,7 +543,7 @@ int32_t handleUserDefinedFunc(SSqlObj* pSql, struct SSqlInfo* pInfo) {
t0->z[t0->n] = 0;
strdequote(t0->z);
stringProcess(t0->z, t0->n);
if (strlen(t0->z) >= TSDB_FUNC_NAME_LEN) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg1);
......@@ -628,7 +628,7 @@ int32_t tscValidateSqlInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
}
} else if (pInfo->type == TSDB_SQL_DROP_DNODE) {
if (pzName->type == TK_STRING) {
pzName->n = strdequote(pzName->z);
pzName->n = stringProcess(pzName->z, pzName->n);
}
strncpy(pCmd->payload, pzName->z, pzName->n);
} else { // drop user/account
......@@ -718,7 +718,7 @@ int32_t tscValidateSqlInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
SStrToken* id = taosArrayGet(pInfo->pMiscInfo->a, 0);
if (id->type == TK_STRING) {
id->n = strdequote(id->z);
id->n = stringProcess(id->z, id->n);
}
break;
}
......@@ -834,7 +834,7 @@ int32_t tscValidateSqlInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
SStrToken* t0 = taosArrayGet(pMiscInfo->a, 0);
SStrToken* t1 = taosArrayGet(pMiscInfo->a, 1);
t0->n = strdequote(t0->z);
t0->n = stringProcess(t0->z, t0->n);
strncpy(pCfg->ep, t0->z, t0->n);
if (validateEp(pCfg->ep) != TSDB_CODE_SUCCESS) {
......@@ -3441,7 +3441,7 @@ static int16_t doGetColumnIndex(SQueryInfo* pQueryInfo, int32_t index, SStrToken
pToken->z = tmpTokenBuf;
if (pToken->type == TK_ID) {
tscRmEscapeAndTrimToken(pToken);
pToken->n = stringProcess(pToken->z, pToken->n);
}
for (int16_t i = 0; i < numOfCols; ++i) {
......@@ -3602,11 +3602,11 @@ int32_t setShowInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
// show table/stable like 'xxxx', set the like pattern for show tables
SStrToken* pPattern = &pShowInfo->pattern;
if (pPattern->type != 0) {
if (pPattern->type == TK_ID && pPattern->z[0] == TS_ESCAPE_CHAR) {
if (pPattern->type == TK_ID && pPattern->z[0] == TS_BACKQUOTE_CHAR) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg7);
}
pPattern->n = strdequote(pPattern->z);
pPattern->n = stringProcess(pPattern->z, pPattern->n);
if (pPattern->n <= 0) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg6);
......@@ -3624,7 +3624,7 @@ int32_t setShowInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
}
if (pShowInfo->prefix.type == TK_STRING) {
pShowInfo->prefix.n = strdequote(pShowInfo->prefix.z);
pShowInfo->prefix.n = stringProcess(pShowInfo->prefix.z, pShowInfo->prefix.n);
}
}
return TSDB_CODE_SUCCESS;
......@@ -4938,7 +4938,7 @@ static int32_t validateNullExpr(tSqlExpr* pExpr, STableMeta* pTableMeta, int32_t
}
char *v = strndup(pRight->exprToken.z, pRight->exprToken.n);
int32_t len = strRmquote(v, pRight->exprToken.n);
int32_t len = stringProcess(v, pRight->exprToken.n);
if (len > 0) {
uint32_t type = 0;
tGetToken(v, &type);
......@@ -6065,7 +6065,7 @@ int32_t getTimeRange(STimeWindow* win, tSqlExpr* pRight, int32_t optr, int16_t t
int64_t val = 0;
bool parsed = false;
if (pRight->value.nType == TSDB_DATA_TYPE_BINARY) {
pRight->value.nLen = strdequote(pRight->value.pz);
pRight->value.nLen = stringProcess(pRight->value.pz, pRight->value.nLen);
char* seg = strnchr(pRight->value.pz, '-', pRight->value.nLen, false);
if (seg != NULL) {
......@@ -7021,7 +7021,7 @@ int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
//handle Escape character backstick
bool inEscape = false;
if (name.z[0] == TS_ESCAPE_CHAR && name.z[name.n - 1] == TS_ESCAPE_CHAR) {
if (name.z[0] == TS_BACKQUOTE_CHAR && name.z[name.n - 1] == TS_BACKQUOTE_CHAR) {
inEscape = true;
name.type = TK_ID;
}
......@@ -7040,7 +7040,7 @@ int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
int32_t nameLen = pItem->pVar.nLen;
if (inEscape) {
memmove(name1, name1 + 1, nameLen);
name1[nameLen - TS_ESCAPE_CHAR_SIZE] = '\0';
name1[nameLen - TS_BACKQUOTE_CHAR_SIZE] = '\0';
}
TAOS_FIELD f = tscCreateField(TSDB_DATA_TYPE_INT, name1, tDataTypes[TSDB_DATA_TYPE_INT].bytes);
......@@ -7061,7 +7061,7 @@ int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
//handle Escape character backstick
bool inEscape = false;
if (name.z[0] == TS_ESCAPE_CHAR && name.z[name.n - 1] == TS_ESCAPE_CHAR) {
if (name.z[0] == TS_BACKQUOTE_CHAR && name.z[name.n - 1] == TS_BACKQUOTE_CHAR) {
inEscape = true;
name.type = TK_ID;
}
......@@ -7102,8 +7102,8 @@ int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
if (inEscape) {
memmove(name.z, name.z + 1, name.n);
name.z[name.n - TS_ESCAPE_CHAR_SIZE] = '\0';
name.n -= TS_ESCAPE_CHAR_SIZE;
name.z[name.n - TS_BACKQUOTE_CHAR_SIZE] = '\0';
name.n -= TS_BACKQUOTE_CHAR_SIZE;
}
TAOS_FIELD f = tscCreateField(pColSchema->type, name.z, pItem->bytes);
......@@ -7121,10 +7121,10 @@ int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
SColumnIndex columnIndex = COLUMN_INDEX_INITIALIZER;
SStrToken name = {.type = TK_STRING, .z = pItem->name, .n = (uint32_t)strlen(pItem->name)};
//handle Escape character backstick
if (name.z[0] == TS_ESCAPE_CHAR && name.z[name.n - 1] == TS_ESCAPE_CHAR) {
if (name.z[0] == TS_BACKQUOTE_CHAR && name.z[name.n - 1] == TS_BACKQUOTE_CHAR) {
memmove(name.z, name.z + 1, name.n);
name.z[name.n - TS_ESCAPE_CHAR_SIZE] = '\0';
name.n -= TS_ESCAPE_CHAR_SIZE;
name.z[name.n - TS_BACKQUOTE_CHAR_SIZE] = '\0';
name.n -= TS_BACKQUOTE_CHAR_SIZE;
}
if (getColumnIndexByName(&name, pQueryInfo, &columnIndex, tscGetErrorMsgPayload(pCmd)) != TSDB_CODE_SUCCESS) {
return invalidOperationMsg(pMsg, msg17);
......@@ -7321,7 +7321,7 @@ int32_t validateDNodeConfig(SMiscInfo* pOptions) {
SStrToken* pValToken = taosArrayGet(pOptions->a, 2);
int32_t vnodeId = 0;
int32_t dnodeId = 0;
strdequote(pValToken->z);
stringProcess(pValToken->z, pValToken->n);
bool parseOk = taosCheckBalanceCfgOptions(pValToken->z, &vnodeId, &dnodeId);
if (!parseOk) {
return TSDB_CODE_TSC_INVALID_OPERATION; // options value is invalid
......@@ -7423,7 +7423,7 @@ int32_t validateColumnName(char* name) {
}
if (token.type == TK_STRING) {
strdequote(token.z);
token.n = stringProcess(token.z, token.n);
strntolower(token.z, token.z, token.n);
token.n = (uint32_t)strtrim(token.z);
......@@ -7434,7 +7434,7 @@ int32_t validateColumnName(char* name) {
return validateColumnName(token.z);
} else if (token.type == TK_ID) {
strRmquoteEscape(name, token.n);
stringProcess(name, token.n);
return TSDB_CODE_SUCCESS;
} else {
if (isNumber(&token)) {
......@@ -7585,7 +7585,7 @@ static int32_t setTimePrecision(SSqlCmd* pCmd, SCreateDbMsg* pMsg, SCreateDbInfo
SStrToken* pToken = &pCreateDbInfo->precision;
if (pToken->n > 0) {
pToken->n = strdequote(pToken->z);
pToken->n = stringProcess(pToken->z, pToken->n);
if (strncmp(pToken->z, TSDB_TIME_PRECISION_MILLI_STR, pToken->n) == 0 &&
strlen(TSDB_TIME_PRECISION_MILLI_STR) == pToken->n) {
......@@ -8640,12 +8640,8 @@ int32_t doCheckForCreateFromStable(SSqlObj* pSql, SSqlInfo* pInfo) {
strncpy(tmpTokenBuf, sToken->z, sToken->n);
sToken->z = tmpTokenBuf;
if (TK_STRING == sToken->type) {
tscDequoteAndTrimToken(sToken);
}
if (TK_ID == sToken->type) {
tscRmEscapeAndTrimToken(sToken);
if (TK_STRING == sToken->type || TK_ID == sToken->type) {
sToken->n = stringProcess(sToken->z, sToken->n);
}
tVariantListItem* pItem = taosArrayGet(pValList, i);
......@@ -9584,8 +9580,6 @@ static int32_t doLoadAllTableMeta(SSqlObj* pSql, SQueryInfo* pQueryInfo, SSqlNod
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg1);
}
tscDequoteAndTrimToken(oriName);
bool dbIncluded = false;
char buf[TSDB_TABLE_FNAME_LEN];
SStrToken sTblToken;
......@@ -9607,7 +9601,6 @@ static int32_t doLoadAllTableMeta(SSqlObj* pSql, SQueryInfo* pQueryInfo, SSqlNod
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg2);
}
tscDequoteAndTrimToken(aliasName);
if (tscValidateName(aliasName, false, NULL) != TSDB_CODE_SUCCESS || aliasName->n >= TSDB_TABLE_NAME_LEN) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg3);
}
......
......@@ -137,7 +137,7 @@ static SSqlObj *taosConnectImpl(const char *ip, const char *user, const char *pa
char tmp[TSDB_DB_NAME_LEN] = {0};
tstrncpy(tmp, db, sizeof(tmp));
strdequote(tmp);
stringProcess(tmp, (int32_t)strlen(tmp));
strtolower(pObj->db, tmp);
}
......@@ -547,6 +547,28 @@ int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows) {
return pRes->numOfRows;
}
TAOS_ROW *taos_result_block(TAOS_RES *res) {
SSqlObj *pSql = (SSqlObj *)res;
if (pSql == NULL || pSql->signature != pSql) {
terrno = TSDB_CODE_TSC_DISCONNECTED;
return NULL;
}
SSqlCmd *pCmd = &pSql->cmd;
SSqlRes *pRes = &pSql->res;
if (pCmd == NULL ||
pRes == NULL ||
pRes->qId == 0 ||
pRes->code == TSDB_CODE_TSC_QUERY_CANCELLED ||
pCmd->command == TSDB_SQL_RETRIEVE_EMPTY_RESULT ||
pCmd->command == TSDB_SQL_INSERT) {
return NULL;
}
return &pRes->urow;
}
int taos_select_db(TAOS *taos, const char *db) {
char sql[256] = {0};
......
......@@ -2918,7 +2918,7 @@ void tscColumnListDestroy(SArray* pColumnList) {
*
*/
static int32_t validateQuoteToken(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded) {
tscDequoteAndTrimToken(pToken);
if(pToken->z[0] != TS_BACKQUOTE_CHAR) pToken->n = stringProcess(pToken->z, pToken->n);
int32_t k = tGetToken(pToken->z, &pToken->type);
......@@ -2932,94 +2932,6 @@ static int32_t validateQuoteToken(SStrToken* pToken, bool escapeEnabled, bool *d
return TSDB_CODE_SUCCESS;
}
void tscDequoteAndTrimToken(SStrToken* pToken) {
uint32_t first = 0, last = pToken->n;
// trim leading spaces
while (first < last) {
char c = pToken->z[first];
if (c != ' ' && c != '\t') {
break;
}
first++;
}
// trim ending spaces
while (first < last) {
char c = pToken->z[last - 1];
if (c != ' ' && c != '\t') {
break;
}
last--;
}
// there are still at least two characters
if (first < last - 1) {
char c = pToken->z[first];
// dequote
if ((c == '\'' || c == '"') && c == pToken->z[last - 1]) {
first++;
last--;
}
}
// left shift the string and pad spaces
for (uint32_t i = 0; i + first < last; i++) {
pToken->z[i] = pToken->z[first + i];
}
for (uint32_t i = last - first; i < pToken->n; i++) {
pToken->z[i] = ' ';
}
// adjust token length
pToken->n = last - first;
}
void tscRmEscapeAndTrimToken(SStrToken* pToken) {
uint32_t first = 0, last = pToken->n;
// trim leading spaces
while (first < last) {
char c = pToken->z[first];
if (c != ' ' && c != '\t') {
break;
}
first++;
}
// trim ending spaces
while (first < last) {
char c = pToken->z[last - 1];
if (c != ' ' && c != '\t') {
break;
}
last--;
}
// there are still at least two characters
if (first < last - 1) {
char c = pToken->z[first];
// dequote
if ((c == '`') && c == pToken->z[last - 1]) {
first++;
last--;
}
}
// left shift the string and pad spaces
for (uint32_t i = 0; i + first < last; i++) {
pToken->z[i] = pToken->z[first + i];
}
for (uint32_t i = last - first; i < pToken->n; i++) {
pToken->z[i] = ' ';
}
// adjust token length
pToken->n = last - first;
}
int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded) {
if (pToken == NULL || pToken->z == NULL
|| (pToken->type != TK_STRING && pToken->type != TK_ID)) {
......@@ -3027,7 +2939,7 @@ int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded)
}
if ((!escapeEnabled) && pToken->type == TK_ID) {
if (pToken->z[0] == TS_ESCAPE_CHAR) {
if (pToken->z[0] == TS_BACKQUOTE_CHAR) {
return TSDB_CODE_TSC_INVALID_OPERATION;
}
}
......@@ -3045,7 +2957,7 @@ int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded)
if (pToken->type == TK_STRING) {
tscDequoteAndTrimToken(pToken);
if(pToken->z[0] != TS_BACKQUOTE_CHAR) pToken->n = stringProcess(pToken->z, pToken->n);
// tscStrToLower(pToken->z, pToken->n);
strntolower(pToken->z, pToken->z, pToken->n);
//pToken->n = (uint32_t)strtrim(pToken->z);
......@@ -3065,7 +2977,7 @@ int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded)
return tscValidateName(pToken, escapeEnabled, NULL);
}
} else if (pToken->type == TK_ID) {
tscRmEscapeAndTrimToken(pToken);
if(pToken->z[0] == TS_BACKQUOTE_CHAR) pToken->n = stringProcess(pToken->z, pToken->n);
if (pToken->n == 0) {
return TSDB_CODE_TSC_INVALID_OPERATION;
......@@ -3126,7 +3038,7 @@ int32_t tscValidateName(SStrToken* pToken, bool escapeEnabled, bool *dbIncluded)
}
if (escapeEnabled && pToken->type == TK_ID) {
tscRmEscapeAndTrimToken(pToken);
if(pToken->z[0] == TS_BACKQUOTE_CHAR) pToken->n = stringProcess(pToken->z, pToken->n);
}
// re-build the whole name string
......
......@@ -289,7 +289,7 @@ char Compressor[32] = "ZSTD_COMPRESSOR"; // ZSTD_COMPRESSOR or GZIP_COMPRES
#endif
// long query death-lock
int8_t tsDeadLockKillQuery = 0;
int8_t tsDeadLockKillQuery = 1;
// default JSON string type
char tsDefaultJSONStrType[7] = "nchar";
......
......@@ -50,7 +50,7 @@ SSchema tGetUserSpecifiedColumnSchema(tVariant* pVal, SStrToken* exprStr, const
} else {
size_t tlen = MIN(sizeof(s.name), exprStr->n + 1);
tstrncpy(s.name, exprStr->z, tlen);
strdequote(s.name);
stringProcess(s.name, (int32_t)strlen(s.name));
}
return s;
......@@ -163,7 +163,7 @@ char *tableNameGetPosition(SStrToken* pToken, char target) {
return pToken->z + i;
}
if (*(pToken->z + i) == TS_ESCAPE_CHAR) {
if (*(pToken->z + i) == TS_BACKQUOTE_CHAR) {
if (!inQuote) {
inEscape = !inEscape;
}
......@@ -223,7 +223,7 @@ void extractTableNameFromToken(SStrToken* pToken, SStrToken* pTable) {
char* r = tableNameGetPosition(pToken, sep);
if (r != NULL) { // record the table name token
if (pToken->z[0] == TS_ESCAPE_CHAR && *(r - 1) == TS_ESCAPE_CHAR) {
if (pToken->z[0] == TS_BACKQUOTE_CHAR && *(r - 1) == TS_BACKQUOTE_CHAR) {
pTable->n = (uint32_t)(r - pToken->z - 2);
pTable->z = pToken->z + 1;
} else {
......
......@@ -87,7 +87,7 @@ void tVariantCreateExt(tVariant *pVar, SStrToken *token, int32_t optrType, bool
case TSDB_DATA_TYPE_BINARY: {
pVar->pz = strndup(token->z, token->n);
pVar->nLen = needRmquoteEscape ? strRmquoteEscape(pVar->pz, token->n) : token->n;
pVar->nLen = needRmquoteEscape ? stringProcess(pVar->pz, token->n) : token->n;
break;
}
case TSDB_DATA_TYPE_TIMESTAMP: {
......
......@@ -287,7 +287,7 @@ static void dnodeSendStatusMsg(void *handle, void *tmrId) {
dnodeGetCfg(&pStatus->dnodeId, pStatus->clusterId);
pStatus->dnodeId = htonl(dnodeGetDnodeId());
pStatus->version = htonl(tsVersion);
pStatus->version = htonl(tsVersion >> 8);
pStatus->lastReboot = htonl(tsRebootTime);
pStatus->numOfCores = htons((uint16_t) tsNumOfCores);
pStatus->diskAvailable = tsAvailDataDirGB;
......
......@@ -179,6 +179,7 @@ DLL_EXPORT bool taos_is_null(TAOS_RES *res, int32_t row, int32_t col);
DLL_EXPORT bool taos_is_update_query(TAOS_RES *res);
DLL_EXPORT int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows);
DLL_EXPORT int* taos_fetch_lengths(TAOS_RES *res);
DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res);
DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql);
DLL_EXPORT void taos_reset_current_db(TAOS *taos);
......
......@@ -108,8 +108,8 @@ extern const int32_t TYPE_BYTES[16];
#define TSDB_ERR -1
#define TS_PATH_DELIMITER "."
#define TS_ESCAPE_CHAR '`'
#define TS_ESCAPE_CHAR_SIZE 2
#define TS_BACKQUOTE_CHAR '`'
#define TS_BACKQUOTE_CHAR_SIZE 2
#define TSDB_TIME_PRECISION_MILLI 0
#define TSDB_TIME_PRECISION_MICRO 1
......
......@@ -51,8 +51,8 @@ void getPrevCharSize(const char *str, int pos, int *size, int *width) {
if (str[pos] > 0 || countPrefixOnes((unsigned char )str[pos]) > 1) break;
}
int rc = mbtowc(&wc, str + pos, MB_CUR_MAX);
assert(rc == *size);
mbtowc(&wc, str + pos, MB_CUR_MAX);
// assert(rc == *size); // it will be core, if str is encode by utf8 and taos charset is gbk
*width = wcwidth(wc);
}
......
......@@ -81,9 +81,9 @@ extern TAOS *taos_connect_auth(const char *ip, const char *user, const char *aut
TAOS *shellInit(SShellArguments *_args) {
printf("\n");
if (!_args->is_use_passwd) {
#ifdef TD_WINDOWS
#ifdef WINDOWS
strcpy(tsOsName, "Windows");
#elif defined(TD_DARWIN)
#elif defined(DARWIN)
strcpy(tsOsName, "Darwin");
#endif
printf(CLIENT_VERSION, tsOsName, taos_get_client_info());
......@@ -239,64 +239,27 @@ int32_t shellRunCommand(TAOS* con, char* command) {
}
}
bool esc = false;
char quote = 0, *cmd = command, *p = command;
char quote = 0, *cmd = command;
for (char c = *command++; c != 0; c = *command++) {
if (esc) {
switch (c) {
case 'n':
c = '\n';
break;
case 'r':
c = '\r';
break;
case 't':
c = '\t';
break;
case 'G':
*p++ = '\\';
break;
case '\'':
case '"':
case '`':
if (quote) {
*p++ = '\\';
}
break;
}
*p++ = c;
esc = false;
if (c == '\\' && (*command == '\'' || *command == '"' || *command == '`')) {
command ++;
continue;
}
if (c == '\\') {
if (quote != 0 && (*command == '_' || *command == '%' || *command == '\\')) {
//DO nothing
} else {
esc = true;
continue;
}
}
if (quote == c) {
quote = 0;
} else if (quote == 0 && (c == '\'' || c == '"' || c == '`')) {
quote = c;
}
*p++ = c;
if (c == ';' && quote == 0) {
c = *p;
*p = 0;
} else if (c == ';' && quote == 0) {
c = *command;
*command = 0;
if (shellRunSingleCommand(con, cmd) < 0) {
return -1;
}
*p = c;
p = cmd;
*command = c;
cmd = command;
}
}
*p = 0;
return shellRunSingleCommand(con, cmd);
}
......@@ -411,7 +374,14 @@ int regex_match(const char *s, const char *reg, int cflags) {
} else if (reti == REG_NOMATCH) {
regfree(&regex);
return 0;
} else {
}
#ifdef DARWIN
else if (reti == REG_ILLSEQ){
regfree(&regex);
return 0;
}
#endif
else {
regerror(reti, &regex, msgbuf, sizeof(msgbuf));
fprintf(stderr, "Regex match failed: %s\n", msgbuf);
regfree(&regex);
......@@ -609,20 +579,25 @@ static void shellPrintNChar(const char *str, int length, int width) {
if (bytes <= 0) {
break;
}
pos += bytes;
if (pos > length) {
break;
}
int w = 0;
#ifdef WINDOWS
int w = bytes;
w = bytes;
#else
int w = wcwidth(wc);
if(*(str + pos) == '\t' || *(str + pos) == '\n' || *(str + pos) == '\r'){
w = bytes;
}else{
w = wcwidth(wc);
}
#endif
if (w <= 0) {
continue;
}
pos += bytes;
if (pos > length) {
break;
}
if (width <= 0) {
printf("%lc", wc);
continue;
......
Subproject commit 00792e2c1a1ee61952e49dbcc33f28ebe4e2c45d
Subproject commit 5c944c1c93c63112890c8c65e1f57cfa6bd89179
......@@ -530,7 +530,7 @@ static int32_t mnodeProcessDnodeStatusMsg(SMnodeMsg *pMsg) {
pStatus->numOfCores = htons(pStatus->numOfCores);
uint32_t _version = htonl(pStatus->version);
if (_version != tsVersion) {
if (_version != tsVersion >> 8) {
pDnode = mnodeGetDnodeByEp(pStatus->dnodeEp);
if (pDnode != NULL && pDnode->status != TAOS_DN_STATUS_READY) {
pDnode->offlineReason = TAOS_DN_OFF_VERSION_NOT_MATCH;
......
......@@ -120,7 +120,7 @@ static void taosGetSystemLocale() {
SGlobalCfg *cfg_charset = taosGetConfigOption("charset");
if (cfg_charset && cfg_charset->cfgStatus < TAOS_CFG_CSTATUS_DEFAULT) {
strcpy(tsCharset, "cp936");
strcpy(tsCharset, "UTF-8");
cfg_charset->cfgStatus = TAOS_CFG_CSTATUS_DEFAULT;
uInfo("charset not configured, set to default:%s", tsCharset);
}
......
......@@ -25,10 +25,11 @@
#include "tlist.h"
#include "qUdf.h"
#define MAX_FUNC_NAME 64
#define USER_FUNC_NAME "funcName"
#define USER_FUNC_NAME_LIMIT 48
/* define in this way to let others know that these two macros are logically related */
#define MAX_FUNC_NAME (USER_FUNC_NAME_LIMIT + 16)
enum ScriptState {
SCRIPT_STATE_INIT,
......@@ -44,7 +45,9 @@ typedef struct {
} ScriptEnv;
typedef struct ScriptCtx {
char funcName[USER_FUNC_NAME_LIMIT];
// one-more-space-for-null-terminator to support function name
// at most USER_FUNC_NAME_LIMIT bytes long actually
char funcName[USER_FUNC_NAME_LIMIT+1];
int8_t state;
ScriptEnv *pEnv;
int8_t isAgg; // agg function or not
......
......@@ -1808,11 +1808,17 @@ static bool functionNeedToExecute(SQueryRuntimeEnv *pRuntimeEnv, SQLFunctionCtx
}
if (functionId == TSDB_FUNC_FIRST_DST || functionId == TSDB_FUNC_FIRST) {
// if param[2] is set value, input data come from client, order is no relation with pQueryAttr->order, so always return true
if(pCtx->param[2].nType == TSDB_DATA_TYPE_INT)
return true;
return QUERY_IS_ASC_QUERY(pQueryAttr);
}
// denote the order type
if ((functionId == TSDB_FUNC_LAST_DST || functionId == TSDB_FUNC_LAST)) {
// if param[2] is set value, input data come from client, order is no relation with pQueryAttr->order , so always return true
if(pCtx->param[2].nType == TSDB_DATA_TYPE_INT)
return true;
return pCtx->param[0].i64 == pQueryAttr->order.order;
}
......
......@@ -91,8 +91,12 @@ void taosValueToLuaType(lua_State *lua, int32_t type, char *val) {
}
int taosLoadScriptInit(void* pInit) {
ScriptCtx *pCtx = pInit;
char funcName[MAX_FUNC_NAME] = {0};
sprintf(funcName, "%s_init", pCtx->funcName);
char funcName[MAX_FUNC_NAME+1] = {0}; // one-more-space-for-null-terminator
int n = snprintf(funcName, sizeof(funcName), "%s_init", pCtx->funcName);
if (n<0 || (size_t)n>=sizeof(funcName)) {
// FIXME: what internal error-code to set?
return -1;
}
lua_State* lua = pCtx->pEnv->lua_state;
lua_getglobal(lua, funcName);
......@@ -105,8 +109,12 @@ int taosLoadScriptInit(void* pInit) {
void taosLoadScriptNormal(void *pInit, char *pInput, int16_t iType, int16_t iBytes, int32_t numOfRows,
int64_t *ptsList, int64_t key, char* pOutput, char *ptsOutput, int32_t *numOfOutput, int16_t oType, int16_t oBytes) {
ScriptCtx* pCtx = pInit;
char funcName[MAX_FUNC_NAME] = {0};
sprintf(funcName, "%s_add", pCtx->funcName);
char funcName[MAX_FUNC_NAME+1] = {0}; // one-more-space-for-null-terminator
int n = snprintf(funcName, sizeof(funcName), "%s_add", pCtx->funcName);
if (n<0 || (size_t)n>=sizeof(funcName)) {
// FIXME: since prototype of this function does NOT return anything
assert(0); // TODO: assert has no effect in case when compiling with NDEBUG set
}
lua_State* lua = pCtx->pEnv->lua_state;
lua_getglobal(lua, funcName);
......@@ -142,8 +150,12 @@ void taosLoadScriptNormal(void *pInit, char *pInput, int16_t iType, int16_t iByt
void taosLoadScriptMerge(void *pInit, char* data, int32_t numOfRows, char* pOutput, int32_t* numOfOutput) {
ScriptCtx *pCtx = pInit;
char funcName[MAX_FUNC_NAME] = {0};
sprintf(funcName, "%s_merge", pCtx->funcName);
char funcName[MAX_FUNC_NAME+1] = {0}; // one-more-space-for-null-terminator
int n = snprintf(funcName, sizeof(funcName), "%s_merge", pCtx->funcName);
if (n<0 || (size_t)n>=sizeof(funcName)) {
// FIXME: since prototype of this function does NOT return anything
assert(0); // TODO: assert has no effect in case when compiling with NDEBUG set
}
lua_State* lua = pCtx->pEnv->lua_state;
lua_getglobal(lua, funcName);
......@@ -166,8 +178,12 @@ void taosLoadScriptMerge(void *pInit, char* data, int32_t numOfRows, char* pOutp
//do not support agg now
void taosLoadScriptFinalize(void *pInit,int64_t key, char *pOutput, int32_t* numOfOutput) {
ScriptCtx *pCtx = pInit;
char funcName[MAX_FUNC_NAME] = {0};
sprintf(funcName, "%s_finalize", pCtx->funcName);
char funcName[MAX_FUNC_NAME+1] = {0}; // one-more-space-for-null-terminator
int n = snprintf(funcName, sizeof(funcName), "%s_finalize", pCtx->funcName);
if (n<0 || (size_t)n>=sizeof(funcName)) {
// FIXME: since prototype of this function does NOT return anything
assert(0); // TODO: assert has no effect in case when compiling with NDEBUG set
}
lua_State* lua = pCtx->pEnv->lua_state;
lua_getglobal(lua, funcName);
......@@ -401,19 +417,23 @@ void addScriptEnvToPool(ScriptEnv *pEnv) {
bool hasBaseFuncDefinedInScript(lua_State *lua, const char *funcPrefix, int32_t len) {
bool ret = true;
char funcName[MAX_FUNC_NAME];
memcpy(funcName, funcPrefix, len);
char funcName[MAX_FUNC_NAME+1] = {0}; // one-more-space-for-null-terminator
const char *base[] = {"_init", "_add"};
for (int i = 0; (i < sizeof(base)/sizeof(base[0])) && (ret == true); i++) {
memcpy(funcName + len, base[i], strlen(base[i]));
memset(funcName + len + strlen(base[i]), 0, MAX_FUNC_NAME - len - strlen(base[i]));
int n = snprintf(funcName, sizeof(funcName), "%.*s%s", len, funcPrefix, base[i]);
if (n<0 || (size_t)n>=sizeof(funcName)) {
// FIXME: what internal error-code to set?
return false;
}
lua_getglobal(lua, funcName);
ret = lua_isfunction(lua, -1); // exsit function or not
lua_pop(lua, 1);
if (!ret) // if it's not lua-function
break;
}
return ret;
}
}
bool isValidScript(char *script, int32_t len) {
ScriptEnv *pEnv = getScriptEnvFromPool(); //
......@@ -432,7 +452,7 @@ bool isValidScript(char *script, int32_t len) {
}
lua_getglobal(lua, USER_FUNC_NAME);
const char *name = lua_tostring(lua, -1);
if (name == NULL || strlen(name) >= USER_FUNC_NAME_LIMIT) {
if (name == NULL || strlen(name) > USER_FUNC_NAME_LIMIT) {
lua_pop(lua, 1);
addScriptEnvToPool(pEnv);
qError("error at %s name: %s, len = %d", script, name, (int)(strlen(name)));
......
......@@ -96,7 +96,7 @@ SArray *tSqlExprListAppend(SArray *pList, tSqlExpr *pNode, SStrToken *pDistinct,
strncpy(item.aliasName, pToken->z, pToken->n);
item.aliasName[pToken->n] = 0;
strdequote(item.aliasName);
stringProcess(item.aliasName, (int32_t)strlen(item.aliasName));
}
taosArrayPush(pList, &item);
......
......@@ -25,9 +25,8 @@ extern "C" {
#include "tcrc32c.h"
#include "taosdef.h"
int32_t strdequote(char *src);
int32_t strRmquote(char *z, int32_t len);
int32_t strRmquoteEscape(char *z, int32_t len);
int32_t strDealWithEscape(char *z, int32_t len);
int32_t stringProcess(char *z, int32_t len);
size_t strtrim(char *src);
char * tstrstr(char *src, char *dst, bool ignoreInEsc);
char * strnchr(char *haystack, char needle, int32_t len, bool skipquote);
......
......@@ -390,6 +390,7 @@ int WCSPatternMatch(const uint32_t *patterStr, const uint32_t *str, size_t size,
uint32_t c, c1;
uint32_t matchOne = (uint32_t) L'_'; // "_"
uint32_t matchAll = (uint32_t) L'%'; // "%"
uint32_t escape = (uint32_t) L'\\'; // "\"
int32_t i = 0;
int32_t j = 0;
......@@ -427,6 +428,8 @@ int WCSPatternMatch(const uint32_t *patterStr, const uint32_t *str, size_t size,
c1 = str[j++];
if (j <= size) {
if (c == escape && patterStr[i] == matchOne && c1 == matchOne) { i++; continue; }
if (c == escape && patterStr[i] == matchAll && c1 == matchAll) { i++; continue; }
if (c == c1 || towlower(c) == towlower(c1) || (c == matchOne && c1 != 0)) {
continue;
}
......
......@@ -448,6 +448,13 @@ uint32_t tGetToken(char* z, uint32_t* tokenId) {
}
case '`': {
for (i = 1; z[i]; i++) {
// if(isprint(z[i]) == 0){
// break;
// }
// if (z[i] == '`' && z[i+1] == '`') {
// i++;
// continue;
// }
if (z[i] == '`') {
i++;
*tokenId = TK_ID;
......
......@@ -26,74 +26,81 @@ bool isInteger(double x){
return (x == truncated);
}
int32_t strdequote(char *z) {
int32_t strDealWithEscape(char *z, int32_t len){
if (z == NULL) {
return 0;
}
int32_t quote = z[0];
if (quote != '\'' && quote != '"') {
return (int32_t)strlen(z);
}
int32_t i = 1, j = 0;
while (z[i] != 0) {
if (z[i] == quote) {
if (z[i + 1] == quote) {
z[j++] = (char)quote;
i++;
} else {
z[j++] = 0;
return (j - 1);
int32_t j = 0;
for (int32_t i = 0; i < len; i++) {
if (z[i] == '\\') { // deal with escape character
if(z[i+1] == 'n'){
z[j++] = '\n';
}else if(z[i+1] == 'r'){
z[j++] = '\r';
}else if(z[i+1] == 't'){
z[j++] = '\t';
}else if(z[i+1] == '\\'){
z[j++] = '\\';
}else if(z[i+1] == '\''){
z[j++] = '\'';
}else if(z[i+1] == '"'){
z[j++] = '"';
}else if(z[i+1] == '%'){
z[j++] = z[i];
z[j++] = z[i+1];
}else if(z[i+1] == '_'){
z[j++] = z[i];
z[j++] = z[i+1];
}else{
z[j++] = z[i+1];
}
} else {
z[j++] = z[i];
i++;
continue;
}
i++;
z[j++] = z[i];
}
return j + 1; // only one quote, do nothing
z[j] = 0;
return j;
}
// delete escape character: \\, \', \"
int32_t strRmquote(char *z, int32_t len){
char delim = 0;
int32_t cnt = 0;
int32_t j = 0;
for (size_t k = 0; k < len; ++k) {
if (!delim && (z[k] == '\'' || z[k] == '"')){ // find the start ' or "
delim = z[k];
}
if ((z[k] == '\\' && z[k + 1] == '_') || (z[k] == '\\' && z[k + 1] == '%')) {
//match '_' '%' self
}else if(z[k] == '\\'){
z[j] = z[k + 1];
cnt++;
j++;
k++;
continue;
}else if(z[k] == delim){
continue;
/*
* remove the quotation marks at both ends
* "fsd" => fsd
* "f""sd" =>f"sd
* 'fsd' => fsd
* 'f''sd' =>f'sd
* `fsd => fsd
* `f``sd` =>f`sd
*/
static int32_t strdequote(char *z, int32_t n){
if(z == NULL || n < 2) return n;
int32_t quote = z[0];
z[0] = 0;
z[n - 1] = 0;
int32_t i = 1, j = 0;
while (i < n) {
if (i < n - 1 && z[i] == quote && z[i + 1] == quote) { // two consecutive quotation marks keep one
z[j++] = (char)quote;
i += 2;
} else {
z[j++] = z[i++];
}
z[j] = z[k];
j++;
}
z[j] = 0;
return j;
z[j - 1] = 0;
return j - 1;
}
int32_t strRmquoteEscape(char *z, int32_t len) {
if (len <= 0) return len;
int32_t stringProcess(char *z, int32_t len) {
if (z == NULL || len < 2) return len;
if (z[0] == '\'' || z[0] == '\"') {
return strRmquote(z, len);
} else if (len > 1 && z[0] == TS_ESCAPE_CHAR && z[len - 1] == TS_ESCAPE_CHAR) {
memmove(z, z + 1, len - 2);
z[len - 2] = '\0';
return len - 2;
if ((z[0] == '\'' && z[len - 1] == '\'')|| (z[0] == '"' && z[len - 1] == '"')) {
int32_t n = strdequote(z, len);
return strDealWithEscape(z, n);
} else if (z[0] == TS_BACKQUOTE_CHAR && z[len - 1] == TS_BACKQUOTE_CHAR) {
return strdequote(z, len);
}
return len;
......@@ -134,7 +141,6 @@ size_t strtrim(char *z) {
} else if (j != i) {
z[i] = 0;
}
return i;
}
......@@ -190,9 +196,9 @@ char *tstrstr(char *src, char *dst, bool ignoreInEsc) {
bool inEsc = false;
char escChar = 0;
char *str = src, *res = NULL;
for (int32_t i = 0; i < len; ++i) {
if (src[i] == TS_ESCAPE_CHAR || src[i] == '\'' || src[i] == '\"') {
if (src[i] == TS_BACKQUOTE_CHAR || src[i] == '\'' || src[i] == '\"') {
if (!inEsc) {
escChar = src[i];
src[i] = 0;
......@@ -209,7 +215,7 @@ char *tstrstr(char *src, char *dst, bool ignoreInEsc) {
str = src + i + 1;
}
inEsc = !inEsc;
continue;
}
......@@ -218,8 +224,6 @@ char *tstrstr(char *src, char *dst, bool ignoreInEsc) {
return str ? strstr(str, dst) : NULL;
}
char* strtolower(char *dst, const char *src) {
int esc = 0;
char quote = 0, *p = dst, c;
......
......@@ -6,56 +6,80 @@
#include "taos.h"
#include "tutil.h"
TEST(testCase, str_rmquote_test) {
char t1[] = "\"\".dd";
int32_t len = strRmquote(t1, strlen(t1));
TEST(testCase, str_escape_test) {
char t1[] = "\"\\\".dd";
int32_t len = strDealWithEscape(t1, strlen(t1));
printf("t1:%s, len:%d\n", t1, len);
EXPECT_EQ(3, len);
EXPECT_STRCASEEQ(t1, ".dd");
EXPECT_EQ(5, len);
EXPECT_STRCASEEQ(t1, "\"\".dd");
char t2[] = "\"fsd\\\"fs\".dd";
len = strRmquote(t2, strlen(t2));
char t2[] = "'\\\'.dd";
len = strDealWithEscape(t2, strlen(t2));
printf("t2:%s, len:%d\n", t2, len);
EXPECT_EQ(9, len);
EXPECT_STRCASEEQ(t2, "fsd\"fs.dd");
EXPECT_EQ(5, len);
EXPECT_STRCASEEQ(t2, "''.dd");
char t3[] = "fs\\_d\\%.d\\d";
len = strRmquote(t3, strlen(t3));
char t3[] = "\\\\.dd";
len = strDealWithEscape(t3, strlen(t3));
printf("t3:%s, len:%d\n", t3, len);
EXPECT_EQ(10, len);
EXPECT_STRCASEEQ(t3, "fs\\_d\\%.dd");
EXPECT_EQ(4, len);
EXPECT_STRCASEEQ(t3, "\\.dd");
char t4[] = "\"fs\\_d\\%\".dd";
len = strRmquote(t4, strlen(t4));
char t4[] = "'\\n.dd";
len = strDealWithEscape(t4, strlen(t4));
printf("t4:%s, len:%d\n", t4, len);
EXPECT_EQ(10, len);
EXPECT_STRCASEEQ(t4, "fs\\_d\\%.dd");
char t5[] = "\"fs\\_d\\%\"";
len = strRmquote(t5, strlen(t5));
printf("t5:%s, len:%d\n", t5, len);
EXPECT_EQ(7, len);
EXPECT_STRCASEEQ(t5, "fs\\_d\\%");
char t6[] = "'fs\\_d\\%'";
len = strRmquote(t6, strlen(t6));
printf("t6:%s, len:%d\n", t6, len);
EXPECT_EQ(7, len);
EXPECT_STRCASEEQ(t6, "fs\\_d\\%");
EXPECT_EQ(4, len);
EXPECT_STRCASEEQ(t4, "\n.dd");
// char t2[] = "\"fsd\\\"fs\".dd";
// len = strDealWithEscape(t2, strlen(t2));
// printf("t2:%s, len:%d\n", t2, len);
// EXPECT_EQ(11, len);
// EXPECT_STRCASEEQ(t2, "\"fsd\"fs\".dd");
//
// char t3[] = "fs\\_d\\%.d\\d";
// len = strRmquote(t3, strlen(t3));
// printf("t3:%s, len:%d\n", t3, len);
// EXPECT_EQ(10, len);
// EXPECT_STRCASEEQ(t3, "fs\\_d\\%.dd");
//
// char t4[] = "\"fs\\_d\\%\".dd";
// len = strRmquote(t4, strlen(t4));
// printf("t4:%s, len:%d\n", t4, len);
// EXPECT_EQ(10, len);
// EXPECT_STRCASEEQ(t4, "fs\\_d\\%.dd");
//
// char t5[] = "\"fs\\_d\\%\"";
// len = strRmquote(t5, strlen(t5));
// printf("t5:%s, len:%d\n", t5, len);
// EXPECT_EQ(7, len);
// EXPECT_STRCASEEQ(t5, "fs\\_d\\%");
//
// char t6[] = "'fs\\_d\\%'";
// len = strRmquote(t6, strlen(t6));
// printf("t6:%s, len:%d\n", t6, len);
// EXPECT_EQ(7, len);
// EXPECT_STRCASEEQ(t6, "fs\\_d\\%");
}
TEST(testCase, string_dequote_test) {
char t1[] = "'abc'";
int32_t len = strdequote(t1);
char t1[] = "'ab''c'";
int32_t len = stringProcess(t1, strlen(t1));
EXPECT_EQ(3, len);
EXPECT_STRCASEEQ(t1, "abc");
EXPECT_EQ(4, len);
EXPECT_STRCASEEQ(t1, "ab'c");
char t2[] = "\"ab\"\"c\"";
len = stringProcess(t2, strlen(t2));
EXPECT_EQ(4, len);
EXPECT_STRCASEEQ(t1, "ab\"c");
char t2[] = "\"abc\"";
len = strdequote(t2);
char t3[] = "`ab``c`";
len = stringProcess(t3, strlen(t3));
EXPECT_EQ(3, len);
EXPECT_STRCASEEQ(t1, "abc");
EXPECT_STRCASEEQ(t1, "ab`c");
char t21[] = " abc ";
int32_t lx = strtrim(t21);
......
......@@ -8,10 +8,9 @@
3. mkdir debug; cd debug; cmake ..; make ; sudo make install
4. pip install ../src/connector/python ; pip3 install
../src/connector/python
4. cd ../tests && pip3 install -r requirements.txt
5. pip install numpy; pip3 install numpy fabric2 psutil pandas(numpy is required only if you need to run querySort.py)
> Note: Both Python2 and Python3 are currently supported by the Python test
> framework. Since Python2 is no longer officially supported by Python Software
......
###################################################################
# Copyright (c) 2021 by TAOS Technologies, Inc.
# All rights reserved.
#
# This file is proprietary and confidential to TAOS Technologies.
# No part of this file may be reproduced, stored, transmitted,
# disclosed or used in any form or by any means other than as
# expressly provided by the written permission from Jianhui Tao
#
###################################################################
# -*- coding: utf-8 -*-
import sys
from util.log import *
from util.cases import *
from util.sql import *
class TDTestCase:
def caseDescription(self):
'''
case1: [TD-12251] json type containing single quotes cannot be inserted
case2: [TD-12334] '\' escape unknown
case3: [TD-11071] escape table creation problem
case5: [TD-12815] like wildcards (% _) are not supported nchar type
'''
return
def init(self, conn, logSql):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), logSql)
self._conn = conn
def run(self):
print("running {}".format(__file__))
tdSql.execute("drop database if exists escape")
tdSql.execute("create database if not exists escape")
tdSql.execute('use escape')
# [TD-12251]
tdSql.execute('create stable st (ts timestamp,t int) tags(metrics json)')
tdSql.execute(r"insert into t1 using st tags('{\"a\":\"a\",\"b\":\"\'a\'=b\"}') values(now,1)")
tdSql.query('select * from st')
tdSql.checkData(0, 2, '''{"a":"a","b":"'a'=b"}''')
# [TD-12334]
tdSql.execute('create table car (ts timestamp, s int) tags(j int)')
tdSql.execute(r'create table `zz\ ` using car tags(11)')
tdSql.execute(r'create table `zz\\ ` using car tags(11)')
tdSql.execute(r'create table `zz\\\ ` using car tags(11)')
tdSql.query(r'select tbname from car where tbname like "zz\\\\ "')
tdSql.checkRows(1)
tdSql.checkData(0, 0, r"zz\\ ")
tdSql.query(r'show tables like "zz\\\\ "')
tdSql.checkRows(1)
tdSql.checkData(0, 0, r"zz\\ ")
tdSql.query(r'show tables like "zz\\ "')
tdSql.checkRows(1)
# [TD-11071]
tdSql.execute('create table es (ts timestamp, s int) tags(j int)')
tdSql.execute(r'create table `zz\t` using es tags(11)')
tdSql.execute(r'create table `zz\\n` using es tags(11)')
tdSql.execute(r'create table `zz\r\ ` using es tags(11)')
tdSql.execute(r'create table ` ` using es tags(11)')
tdSql.query(r'select tbname from es')
tdSql.checkData(0, 0, r'zz\t')
tdSql.checkData(1, 0, r'zz\\n')
tdSql.checkData(2, 0, r'zz\r\ ')
tdSql.checkData(3, 0, r' ')
# [TD-6232]
tdSql.execute('create table tt(ts timestamp, i nchar(128))')
tdSql.execute(r"insert into tt values(1591060628000, '\t')")
tdSql.execute(r"insert into tt values(1591060638000, '\n')")
tdSql.execute(r"insert into tt values(1591060648000, '\r')")
tdSql.execute(r"insert into tt values(1591060658000, '\\')")
tdSql.execute(r"insert into tt values(1591060668000, '\"')")
tdSql.execute(r"insert into tt values(1591060678000, '\'')")
tdSql.execute(r"insert into tt values(1591060688000, '\%')")
tdSql.execute(r"insert into tt values(1591060698000, '\_')")
tdSql.execute(r"insert into tt values(1591060708000, '\9')")
tdSql.query(r"select * from tt where i='\t'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\n'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\r'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\\'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\"'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\''")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\%'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\_'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='\9'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i='9'")
tdSql.checkRows(1)
# [TD-12815] like wildcard(%, _) are not supported nchar
tdSql.execute(r"insert into tt values(1591070708000, 'h%d')")
tdSql.execute(r"insert into tt values(1591080708000, 'h_j')")
tdSql.query(r"select * from tt where i like 'h\%d'")
tdSql.checkRows(1)
tdSql.query(r"select * from tt where i like 'h\_j'")
tdSql.checkRows(1)
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())
......@@ -4,3 +4,4 @@ python3 ./test.py -f 2-query/session_two_stage.py
python3 ./test.py -f 2-query/timeline_agg_func_groupby.py
python3 ./test.py -f 2-query/ts_2016.py
python3 ./test.py -f 2-query/function_mavg.py
python3 ./test.py -f 2-query/escape.py
......@@ -28,7 +28,7 @@ import taos
if __name__ == "__main__":
fileName = "all"
deployPath = ""
masterIp = ""
......@@ -55,7 +55,7 @@ if __name__ == "__main__":
tdLog.printNoPrefix('-w taos on windows')
sys.exit(0)
if key in ['-r', '--restart']:
if key in ['-r', '--restart']:
restart = True
if key in ['-f', '--file']:
......@@ -117,7 +117,7 @@ if __name__ == "__main__":
time.sleep(2)
tdLog.info('stop All dnodes')
if masterIp == "":
host = '127.0.0.1'
else:
......@@ -129,11 +129,11 @@ if __name__ == "__main__":
tdLog.info("Procedures for testing self-deployment")
td_clinet = TDSimClient("C:\\TDengine")
td_clinet.deploy()
remote_conn = Connection("root@%s"%host)
remote_conn = Connection("root@%s" % host)
with remote_conn.cd('/var/lib/jenkins/workspace/TDinternal/community/tests/pytest'):
remote_conn.run("python3 ./test.py")
conn = taos.connect(
host="%s"%(host),
host="%s" % (host),
config=td_clinet.cfgDir)
tdCases.runOneWindows(conn, fileName)
else:
......@@ -146,22 +146,21 @@ if __name__ == "__main__":
try:
if key_word in open(fileName).read():
is_test_framework = 1
except:
except BaseException:
pass
if is_test_framework:
moduleName = fileName.replace(".py", "").replace("/", ".")
uModule = importlib.import_module(moduleName)
try:
ucase = uModule.TDTestCase()
tdDnodes.deploy(1,ucase.updatecfgDict)
except :
tdDnodes.deploy(1,{})
tdDnodes.deploy(1, ucase.updatecfgDict)
except BaseException:
tdDnodes.deploy(1, {})
else:
pass
tdDnodes.deploy(1,{})
tdDnodes.start(1)
tdDnodes.deploy(1, {})
tdDnodes.start(1)
tdCases.logSql(logSql)
......@@ -179,18 +178,20 @@ if __name__ == "__main__":
if fileName == "all":
tdCases.runAllLinux(conn)
else:
tdCases.runOneWindows(conn, fileName)
tdCases.runOneLinux(conn, fileName)
if restart:
if fileName == "all":
tdLog.info("not need to query ")
else:
else:
sp = fileName.rsplit(".", 1)
if len(sp) == 2 and sp[1] == "py":
tdDnodes.stopAll()
tdDnodes.start(1)
time.sleep(1)
conn = taos.connect( host, config=tdDnodes.getSimCfgPath())
tdLog.info("Procedures for tdengine deployed in %s" % (host))
time.sleep(1)
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
tdLog.info(
"Procedures for tdengine deployed in %s" %
(host))
tdLog.info("query test after taosd restart")
tdCases.runOneLinux(conn, sp[0] + "_" + "restart.py")
else:
......
......@@ -154,7 +154,22 @@ class TDTestCase:
sql = "select last(*) from ( select sum(i1) from st where ts>='2017-07-14 11:40:00' and ts<'2017-07-14 12:40:00' interval(10m) order by ts desc );" # desc
tdSql.waitedQuery(sql, 1, WAITS)
tdSql.checkData(0, 1, 192419100)
# add parent query order by
# first
sql = "select first(*) from (select first(i1) from st interval(10m) order by ts asc) order by ts desc;"
tdSql.waitedQuery(sql, 1, WAITS)
tdSql.checkData(0, 1, 0)
sql = "select first(*) from (select first(i1) from st interval(10m) order by ts desc) order by ts asc;"
tdSql.waitedQuery(sql, 1, WAITS)
tdSql.checkData(0, 1, 0)
# last
sql = "select last(*) from (select first(i1) from st interval(10m) order by ts asc) order by ts desc;"
tdSql.waitedQuery(sql, 1, WAITS)
tdSql.checkData(0, 1, 229400)
sql = "select last(*) from (select first(i1) from st interval(10m) order by ts desc) order by ts asc;"
tdSql.waitedQuery(sql, 1, WAITS)
tdSql.checkData(0, 1, 229400)
#
# add case with filename
......
......@@ -54,7 +54,7 @@ if __name__ == "__main__":
tdLog.printNoPrefix('-w taos on windows')
sys.exit(0)
if key in ['-r', '--restart']:
if key in ['-r', '--restart']:
restart = True
if key in ['-f', '--file']:
......@@ -116,7 +116,7 @@ if __name__ == "__main__":
time.sleep(2)
tdLog.info('stop All dnodes')
if masterIp == "":
host = '127.0.0.1'
else:
......@@ -157,7 +157,7 @@ if __name__ == "__main__":
with remote_conn.cd('/var/lib/jenkins/workspace/TDinternal/community/tests/pytest'):
remote_conn.run("python3 ./test.py")
conn = taos.connect(
host="%s"%(host),
host="%s" % (host),
config=td_clinet.cfgDir)
tdCases.runOneWindows(conn, fileName)
else:
......@@ -170,23 +170,21 @@ if __name__ == "__main__":
try:
if key_word in open(fileName).read():
is_test_framework = 1
except:
except BaseException:
pass
if is_test_framework:
moduleName = fileName.replace(".py", "").replace(os.sep, ".")
uModule = importlib.import_module(moduleName)
try:
ucase = uModule.TDTestCase()
tdDnodes.deploy(1,ucase.updatecfgDict)
except :
tdDnodes.deploy(1,{})
tdDnodes.deploy(1, ucase.updatecfgDict)
except BaseException:
tdDnodes.deploy(1, {})
else:
pass
tdDnodes.deploy(1,{})
tdDnodes.deploy(1, {})
tdDnodes.start(1)
tdCases.logSql(logSql)
if testCluster:
......@@ -203,18 +201,20 @@ if __name__ == "__main__":
if fileName == "all":
tdCases.runAllLinux(conn)
else:
tdCases.runOneWindows(conn, fileName)
tdCases.runOneLinux(conn, fileName)
if restart:
if fileName == "all":
tdLog.info("not need to query ")
else:
else:
sp = fileName.rsplit(".", 1)
if len(sp) == 2 and sp[1] == "py":
tdDnodes.stopAll()
tdDnodes.start(1)
time.sleep(1)
conn = taos.connect( host, config=tdDnodes.getSimCfgPath())
tdLog.info("Procedures for tdengine deployed in %s" % (host))
time.sleep(1)
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
tdLog.info(
"Procedures for tdengine deployed in %s" %
(host))
tdLog.info("query test after taosd restart")
tdCases.runOneLinux(conn, sp[0] + "_" + "restart.py")
else:
......
......@@ -225,7 +225,7 @@ class TDDnode:
# self.cfg("logDir",self.logDir)
# print(updatecfgDict)
isFirstDir = 1
if updatecfgDict[0] and updatecfgDict[0][0]:
if bool(updatecfgDict) and updatecfgDict[0] and updatecfgDict[0][0]:
print(updatecfgDict[0][0])
for key,value in updatecfgDict[0][0].items():
if value == 'dataDir' :
......
../src/connector/python
numpy
fabric2
psutil
pandas
\ No newline at end of file
......@@ -21,14 +21,16 @@ exe:
gcc $(CFLAGS) ./stmt.c -o $(ROOT)stmt $(LFLAGS)
gcc $(CFLAGS) ./clientcfgtest.c -o $(ROOT)clientcfgtest $(LFLAGS)
gcc $(CFLAGS) ./openTSDBTest.c -o $(ROOT)openTSDBTest $(LFLAGS)
gcc $(CFLAGS) ./resultBlock.c -o $(ROOT)resultBlock $(LFLAGS)
clean:
rm $(ROOT)batchprepare
rm $(ROOT)stmtBatchTest
rm $(ROOT)stmtTest
rm $(ROOT)stmt
rm $(ROOT)stmt_function
rm $(ROOT)clientcfgtest
rm $(ROOT)openTSDBTest
rm $(ROOT)stmt
rm $(ROOT)resultBlock
#include "taoserror.h"
#include "cJSON.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <taos.h>
#include <unistd.h>
#include <inttypes.h>
static void prepare_data(TAOS* taos) {
TAOS_RES* result;
result = taos_query(taos, "drop database if exists test;");
taos_free_result(result);
usleep(100000);
result = taos_query(taos, "create database test precision 'ms';");
taos_free_result(result);
usleep(100000);
taos_select_db(taos, "test");
result = taos_query(taos, "create table meters(ts timestamp, c0 tinyint, c1 smallint, c2 int, c3 bigint, c4 float, c5 double, c6 bool, c7 binary(10), c8 nchar(10)) tags (t0 int, t1 float, t2 double, t3 bool, t4 binary(10), t5 nchar(10));");
taos_free_result(result);
result = taos_query(taos, "create table tb0 using meters tags(0, 0.0, 0.0, true, \"tag0\", \"标签0\");");
taos_free_result(result);
result = taos_query(taos, "create table tb1 using meters tags(1, 1.0, 1.0, true, \"tag1\", \"标签1\");");
taos_free_result(result);
result = taos_query(taos, "create table tb2 using meters tags(2, 2.0, 2.0, true, \"tag2\", \"标签2\");");
taos_free_result(result);
result = taos_query(taos, "create table tb3 using meters tags(3, 3.0, 3.0, true, \"tag3\", \"标签3\");");
taos_free_result(result);
result = taos_query(taos, "create table tb4 using meters tags(4, 4.0, 4.0, true, \"tag4\", \"标签4\");");
taos_free_result(result);
result = taos_query(taos, "create table tb5 using meters tags(5, 5.0, 5.0, true, \"tag5\", \"标签5\");");
taos_free_result(result);
result = taos_query(taos, "create table tb6 using meters tags(6, 6.0, 6.0, true, \"tag6\", \"标签6\");");
taos_free_result(result);
result = taos_query(taos, "create table tb7 using meters tags(7, 7.0, 7.0, true, \"tag7\", \"标签7\");");
taos_free_result(result);
result = taos_query(taos, "create table tb8 using meters tags(8, 8.0, 8.0, true, \"tag8\", \"标签8\");");
taos_free_result(result);
result = taos_query(taos, "create table tb9 using meters tags(9, 9.0, 9.0, true, \"tag9\", \"标签9\");");
taos_free_result(result);
result = taos_query(taos,
"insert into tb0 values('2020-01-01 00:00:00.000', 11, 11, 11, 11, 11.0, 11.0, false, \"col11\", \"值11\")"
" ('2020-01-01 00:01:00.000', 12, 12, 12, 12, 12.0, 12.0, false, \"col12\", \"值12\")"
" ('2020-01-01 00:02:00.000', 13, 13, 13, 13, 13.0, 13.0, false, \"col13\", \"值13\")"
" tb1 values('2020-01-01 00:00:00.000', 21, 21, 21, 21, 21.0, 21.0, false, \"col21\", \"值21\")"
" tb2 values('2020-01-01 00:00:00.000', 31, 31, 31, 31, 31.0, 31.0, false, \"col31\", \"值31\")"
" tb3 values('2020-01-01 00:01:02.000', 41, 41, 41, 41, 41.0, 41.0, false, \"col41\", \"值41\")"
" tb4 values('2020-01-01 00:01:02.000', 51, 51, 51, 51, 51.0, 51.0, false, \"col51\", \"值51\")"
" tb5 values('2020-01-01 00:01:02.000', 61, 61, 61, 61, 61.0, 61.0, false, \"col61\", \"值61\")"
" tb6 values('2020-01-01 00:01:02.000', 71, 71, 71, 71, 71.0, 71.0, false, \"col71\", \"值71\")"
" tb7 values('2020-01-01 00:01:02.000', 81, 81, 81, 81, 81.0, 81.0, false, \"col81\", \"值81\")"
" tb8 values('2020-01-01 00:01:02.000', 91, 91, 91, 91, 91.0, 91.0, false, \"col91\", \"值91\")"
" tb9 values('2020-01-01 00:01:02.000', 101, 101, 101, 101, 101.0, 101.0, false, \"col101\", \"值101\")");
int affected = taos_affected_rows(result);
if (affected != 12) {
printf("\033[31m%d rows affected by last insert statement, but it should be 12\033[0m\n", affected);
}
taos_free_result(result);
// super tables subscription
usleep(1000000);
}
static int print_result(TAOS_RES* res, int32_t rows) {
TAOS_ROW* block_ptr = NULL;
int num_fields = taos_num_fields(res);
TAOS_FIELD* fields = taos_fetch_fields(res);
block_ptr = taos_result_block(res);
TAOS_ROW col = *block_ptr;
for (int k = 0; k < rows; k++) {
char str[256] = {0};
int32_t len = 0;
for (int i = 0; i < num_fields; ++i) {
if (i > 0) {
str[len++] = ' ';
}
switch (fields[i].type) {
case TSDB_DATA_TYPE_TINYINT:
len += sprintf(str + len, "%d", *(((int8_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_UTINYINT:
len += sprintf(str + len, "%u", *(((uint8_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_SMALLINT:
len += sprintf(str + len, "%d", *(((int16_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_USMALLINT:
len += sprintf(str + len, "%u", *(((uint16_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_INT:
len += sprintf(str + len, "%d", *(((int32_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_UINT:
len += sprintf(str + len, "%u", *(((uint32_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_BIGINT:
len += sprintf(str + len, "%" PRId64, *(((int64_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_UBIGINT:
len += sprintf(str + len, "%" PRIu64, *(((uint64_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_FLOAT: {
len += sprintf(str + len, "%f", *(((float *)col[i]) + k));
} break;
case TSDB_DATA_TYPE_DOUBLE: {
len += sprintf(str + len, "%lf", *(((double *)col[i]) + k));
} break;
case TSDB_DATA_TYPE_BINARY:
case TSDB_DATA_TYPE_NCHAR: {
int32_t charLen = *(int16_t *)col[i];
int32_t charBytes = (fields[i].type == TSDB_DATA_TYPE_BINARY) ? sizeof(char) : sizeof(wchar_t);
int32_t offset = k * (sizeof(int16_t) + fields[i].bytes * charBytes);
memcpy(str + len, (char *)col[i] + sizeof(int16_t) + offset, charLen);
len += charLen;
} break;
case TSDB_DATA_TYPE_TIMESTAMP:
len += sprintf(str + len, "%" PRId64, *(((int64_t *)col[i]) + k));
break;
case TSDB_DATA_TYPE_BOOL:
len += sprintf(str + len, "%d", *(((int8_t *)col[i]) + k));
default:
break;
}
}
puts(str);
}
}
void fetch_cb(void *param, TAOS_RES* tres, int32_t numOfRows) {
if (tres == NULL) {
printf("result not available!\n");
return;
}
if (numOfRows > 0) {
printf("%d rows async retrieved\n", numOfRows);
print_result(tres, numOfRows);
taos_fetch_rows_a(tres, fetch_cb, param);
} else {
if (numOfRows < 0) {
printf("\033[31masync retrieve failed, code: %d\033[0m\n", numOfRows);
} else {
printf("async retrieve completed\n");
}
taos_free_result(tres);
}
}
void query_cb(void* param, TAOS_RES* tres, int32_t code) {
if (code == 0 && tres) {
taos_fetch_rows_a(tres, fetch_cb, param);
} else {
printf("\033[31masync query failed, code: %d\033[0m\n", code);
}
}
int main(int argc, char *argv[]) {
const char* host = "127.0.0.1";
const char* user = "root";
const char* passwd = "taosdata";
taos_options(TSDB_OPTION_TIMEZONE, "GMT-8");
TAOS* taos = taos_connect(host, user, passwd, "", 0);
if (taos == NULL) {
printf("\033[31mfailed to connect to db, reason:%s\033[0m\n", taos_errstr(taos));
exit(1);
}
char* info = taos_get_server_info(taos);
printf("server info: %s\n", info);
info = taos_get_client_info(taos);
printf("client info: %s\n", info);
printf("************ Prepare data *************\n");
prepare_data(taos);
printf("************ Async query *************\n");
taos_query_a(taos, "select * from meters", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb0", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb1", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb2", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb3", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb4", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb5", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb6", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb7", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb8", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select * from tb9", query_cb, NULL);
usleep(1000000);
taos_query_a(taos, "select count(*) from meters", query_cb, NULL);
usleep(1000000);
printf("done\n");
taos_close(taos);
taos_cleanup();
}
......@@ -31,7 +31,7 @@ class TDTestCase:
def caseDescription(self):
'''
case1 <authors>: wenzhouwww[TD-11943] :
case1 <wenzhouwww>: [TD-11943] :
this test case is an test case for unexpected coredump about taosd ;
root cause : the pExpr2 of sql select tbname, max(col)+5 from child_table has two functions, col_proj and scalar_expr.
for function col_proj (tbname column), it is a tag during master scan stage, the input data is not set.
......
......@@ -30,7 +30,7 @@ class TDTestCase:
def caseDescription(self):
'''
case1 <authors>: wenzhouwww[TD-11978] :
case1 <wenzhouwww>: [TD-11978] :
this test case is an test case for unexpected coredump about taoshell ;
root cause : The function does not determine whether the input is empty
'''
......
......@@ -92,15 +92,12 @@ class TDTestCase:
taosd_pid = int(subprocess.getstatusoutput('ps aux|grep "taosd" |grep -v "grep"|awk \'{print $2}\'')[1])
sleep(10)
cmd = "top -H -p %d -n 1"%taosd_pid
sys_output = subprocess.check_output(cmd, shell=True).decode("utf-8")
print(sys_output)
cmd_insert = "%staosBenchmark -y -n 10 -t 10 -S 10000 > /dev/null 2>&1 & " % (build_path)
cmd_insert = "%staosBenchmark -y -n 10 -t 10 -S 10000 " % (build_path)
os.system(cmd_insert)
sleep(5)
tdSql.query("select count(*) from meters")
tdSql.checkData(0,0,10)
tdSql.query("select count(*) from test.meters")
tdSql.checkData(0,0,100)
def stop(self):
tdSql.close()
......
......@@ -369,7 +369,7 @@ class TDTestCase:
print(conn1)
for i in range(20):
for i in range(2):
try:
taos_cmd1 = "taos -f 2-query/TD-12204.py.sql"
_ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8")
......@@ -377,7 +377,7 @@ class TDTestCase:
print(i)
print(conn1)
for i in range(10):
for i in range(5):
cur1.execute('use db ;')
sql = 'select * from stable_1 where t_smallint between 0 and 32767 and t_float between 0 and 3.4E38 and t_nchar is not null and q_smallint between 0 and 32767 and q_nchar is not null and t_binary is not null and q_tinyint is not null and ts < now +1s order by ts ;;;'
......
###################################################################
# Copyright (c) 2020 by TAOS Technologies, Inc.
# All rights reserved.
#
# This file is proprietary and confidential to TAOS Technologies.
# No part of this file may be reproduced, stored, transmitted,
# disclosed or used in any form or by any means other than as
# expressly provided by the written permission from Jianhui Tao
#
###################################################################
# -*- coding: utf-8 -*-
from posixpath import split
import sys
import os
from util.log import *
from util.cases import *
from util.sql import *
from util.dnodes import *
import subprocess
class TDTestCase:
def init(self, conn, logSql):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), logSql)
def getBuildPath(self):
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
projPath = selfPath[:selfPath.find("community")]
else:
projPath = selfPath[:selfPath.find("tests")]
for root, dirs, files in os.walk(projPath):
if ("taosd" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
if ("packaging" not in rootRealPath):
buildPath = root[:len(root) - len("/build/bin")]
break
return buildPath
def caseDescription(self):
'''
case1 <wenzhouwww>: [TD-12275] :
this test case is an long query crash for elapsed function .
'''
return
def run(self):
tdSql.prepare()
build_path = self.getBuildPath()+"/build/bin/"
prepare_cmd = "%staosBenchmark -t 100 -n 100000 -S 10000 -y " % (build_path)
# only taos -s for shell can generate this issue
print(prepare_cmd)
_ = subprocess.check_output(prepare_cmd, shell=True).decode("utf-8")
cmd1 = "taos -s 'select elapsed(ts) from test.meters interval(10s) sliding(5s) group by tbname' "
print(cmd1)
_ = subprocess.check_output(cmd1, shell=True).decode("utf-8")
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())
......@@ -31,7 +31,7 @@ class TDTestCase:
def caseDescription(self):
'''
case1 <authors>: wenzhouwww[TD-12388] :
case1 <wenzhouwww>: [TD-12388] :
this test case is an test case for unit time params about elapsed function.
'''
......
......@@ -140,8 +140,8 @@ class TDTestCase:
tablenames = ["sub_table1_1","sub_table1_2","sub_table1_3","sub_table2_1","sub_table2_2","sub_table2_3","regular_table_1","regular_table_2","regular_table_3"]
abnormal_list = ["()","(NULL)","(*)","(abc)","( , )","(NULL,*)","( ,NULL)","(%)","(+)","(*,)","(*, /)","(ts,10)","(ts,*)" "(ts,tbname*10)","(ts,tagname)","(ts,now-2d+3m)","(ts,2d+3m-2s)",
"(ts,2d+3m-2s,)","(ts+1d,10s)","(ts+10d,NULL)" ,"(ts,now -1m%1d)","(ts+10d)","(ts+10d,_c0)","(ts+10d,)","(ts,%)","(ts, , )","(ts,abc)","(ts,/)","(ts,*)","(ts,now)","(ts,now+1d)","(ts,_c0)","(ts,1s,100)",
abnormal_list = ["()","(NULL)","(*)","(abc)","( , )","(NULL,*)","( ,NULL)","(%)","(+)","(*,)","(*, /)","(ts,10)","(ts,*)" "(ts,tbname*10)","(ts,tagname)","(ts,now-2d+3m)",
"(ts,2d+3m-2s,NULL)","(ts+1d,10s)","(ts+10d,NULL)" ,"(ts,now -1m%1d)","(ts+10d)","(ts+10d,_c0)","(ts+10d,)","(ts,%)","(ts, , m)","(ts,abc)","(ts,/)","(ts,*)","(ts,now)","(ts,now+1d)","(ts,_c0)","(ts,1s,100)",
"(ts,1s,abc)","(ts,1s,_c0)","(ts,1s,*)","(ts,1s,NULL)","(ts,,_c0)","(ts,tbname)","(ts,tbname,ts)","(ts,0,tbname)","('2021-11-18 00:00:10')","('2021-11-18 00:00:10', 1s)",
"('2021-11-18T00:00:10+0800', '1s')","('2021-11-18T00:00:10Z', '1s')","('2021-11-18T00:00:10+0800', 10000000d,)","('ts', ,2021-11-18T00:00:10+0800, )"]
......@@ -1597,7 +1597,7 @@ class TDTestCase:
def run(self):
tdSql.prepare()
self.prepare_data()
# self.abnormal_common_test()
self.abnormal_common_test()
self.abnormal_use_test()
self.query_filter()
self.query_interval()
......
{
"filetype": "insert",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"thread_count": 16,
"thread_count_create_tbl": 4,
"result_file": "./insert_res.txt",
"confirm_parameter_prompt": "no",
"insert_interval": 0,
"chinese":"no",
"databases": [{
"dbinfo": {
"name": "test_TD11483",
"drop": "yes",
"replica": 1,
"days": 10,
"cache": 16,
"blocks": 8,
"precision": "ms",
"keep": 36500,
"minRows": 100,
"maxRows": 4096,
"comp":2,
"walLevel":1,
"cachelast":0,
"quorum":1,
"fsync":3000,
"update": 0
},
"super_tables": [{
"name": "stb",
"child_table_exists":"no",
"childtable_count": 10,
"childtable_prefix": "stb_",
"escape_character": "yes",
"auto_create_table": "no",
"batch_create_tbl_num": 5,
"data_source": "rand",
"insert_mode": "taosc",
"line_protocol": "line",
"insert_rows": 100,
"childtable_limit": 10,
"childtable_offset":100,
"interlace_rows": 0,
"insert_interval":0,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 10000,
"start_timestamp": "2010-10-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"use_sameple_ts": "no",
"tags_file": "",
"columns": [{"type": "INT"}, {"type": "DOUBLE", "count":2}, {"type": "BINARY", "len": 16, "count":3}, {"type": "BINARY", "len": 32, "count":6}],
"tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":1}]
}]
}]
}
......@@ -44,6 +44,7 @@ class TDTestCase:
else:
projPath = selfPath[:selfPath.find("tests")]
buildPath = ""
for root, dirs, files in os.walk(projPath):
if ("taosdump" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
......
......@@ -2,8 +2,40 @@ python3 ./test.py -f 2-query/TD-11256.py
# python3 ./test.py -f 2-query/TD-11389.py
python3 ./test.py -f 2-query/TD-11945_crash.py
python3 ./test.py -f 2-query/TD-12340-12342.py
python3 ./test.py -f 2-query/TD-11561.py
python3 ./test.py -f 2-query/TD-12204.py
python3 ./test.py -f 2-query/TD-11483.py
python3 ./test.py -f 2-query/TD-11943.py
python3 ./test.py -f 2-query/TD-11969.py
python3 ./test.py -f 2-query/TD-11978.py
python3 ./test.py -f 2-query/TD-12014.py
python3 ./test.py -f 2-query/TD-12145.py
python3 ./test.py -f 2-query/TD-12164.py
python3 ./test.py -f 2-query/TD-12165.py
python3 ./test.py -f 2-query/TD-12191.py
python3 ./test.py -f 2-query/TD-12228.py
python3 ./test.py -f 2-query/TD-12229.py
python3 ./test.py -f 2-query/TD-12276.py
python3 ./test.py -f 2-query/TD-12344.py
#python3 ./test.py -f 2-query/TD-12388.py
#python3 ./test.py -f 2-query/TD-12593.py
#python3 ./test.py -f 2-query/TD-12594.py
python3 ./test.py -f 2-query/TD-12614.py
python3 ./test.py -f 2-query/function_elapsed.py
python3 ./test.py -f 5-taos-tools/basic.py
python3 ./test.py -f 5-taos-tools/TD-12478.py
python3 ./test.py -f 5-taos-tools/dump_col_tag.py
python3 ./test.py -f 5-taos-tools/taosdump/taosdumpTestColTag.py
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册