diff --git a/docs-cn/20-third-party/01-grafana.mdx b/docs-cn/20-third-party/01-grafana.mdx index 40b5c0ff4f2de8ff9eeb3afa61728ca7a899f5ea..2daaebc32d1888a8197ebfefdb284f307174b0d6 100644 --- a/docs-cn/20-third-party/01-grafana.mdx +++ b/docs-cn/20-third-party/01-grafana.mdx @@ -12,12 +12,38 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/ - TDengine 集群已经部署并正常运行 - taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter) +记录以下信息： + +- TDengine 集群 REST API 地址，如：`http://tdengine.local:6041`。 +- TDengine 集群认证信息，可使用用户名及密码。 + ## 安装 Grafana -目前 TDengine 支持 Grafana 7.0 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：。 +目前 TDengine 支持 Grafana 7.5 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：。 ## 配置 Grafana +### 安装方式一：安装脚本 + +将集群信息设置为环境变量（也可以使用 [`.env`(dotenv) 文件](https://hexdocs.pm/dotenvy/dotenv-file-format.html) ）： + +```sh +export TDENGINE_API=http://tdengine.local:6041 +# user + password +export TDENGINE_USER=user +export TDENGINE_PASSWORD=password +``` + +运行安装脚本： + +```sh +bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" +``` + +该脚本将自动安装 Grafana 插件并配置数据源。安装完毕后，需要重启 Grafana 服务后生效。 + +### 安装方式二：手动安装 TDengine 数据源插件 + 使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。 ```bash @@ -48,11 +74,7 @@ sudo unzip tdengine-datasource-$GF_VERSION.zip -d /var/lib/grafana/plugins/ GF_INSTALL_PLUGINS=tdengine-datasource ``` -## 使用 Grafana - -### 配置数据源 - -用户可以直接通过 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示： +之后，用户可以直接通过 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示： ![TDengine Database Grafana plugin add data source](./add_datasource1.webp) @@ -92,4 +114,11 @@ GF_INSTALL_PLUGINS=tdengine-datasource ### 导入 Dashboard -在 2.3.3.0 及以上版本，您可以导入 TDinsight Dashboard (Grafana Dashboard ID： [15167](https://grafana.com/grafana/dashboards/15167)) 作为 TDengine 集群的监控可视化工具。安装和使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。 +在数据源配置页面，您可以为该数据源导入 TDinsight 面板，作为 TDengine 集群的监控可视化工具。该 Dashboard 已发布在 Grafana：[Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167)) 。其他安装方式和相关使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。 + +使用 TDengine 作为数据源的其他面板，可以[在此搜索](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource)。以下是一份不完全列表： + +- [15146](https://grafana.com/grafana/dashboards/15146): 监控多个 TDengine 集群 +- [15155](https://grafana.com/grafana/dashboards/15155): TDengine 告警示例 +- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight +- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf 采集节点信息的数据展示 diff --git a/docs-en/20-third-party/01-grafana.mdx b/docs-en/20-third-party/01-grafana.mdx index 1a84e02c665d2e49deca35a20b137b205736def5..1e1b49703ecaa9b640d7eadcfa5cb9a26c572758 100644 --- a/docs-en/20-third-party/01-grafana.mdx +++ b/docs-en/20-third-party/01-grafana.mdx @@ -14,12 +14,40 @@ In order for Grafana to add the TDengine data source successfully, the following 1. The TDengine cluster is deployed and functioning properly 2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details. +Record these values: + +- TDengine REST API url: `http://tdengine.local:6041`. +- TDengine cluster authorization, with user + password. + ## Installing Grafana -TDengine currently supports Grafana versions 7.0 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: . +TDengine currently supports Grafana versions 7.5 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: . ## Configuring Grafana +### Option 1: Install with `install.sh` + +Set the url and authorization environment variables by `export` or a [`.env`(dotenv) file](https://hexdocs.pm/dotenvy/dotenv-file-format.html): + +```sh +export TDENGINE_API=http://tdengine.local:6041 +# user + password +export TDENGINE_USER=user +export TDENGINE_PASSWORD=password +``` + +Run `install.sh`: + +```sh +bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" +``` + +With this script, TDengine data source plugin and the Grafana data source will be installed and created automatically with Grafana provisioning configurations. + +And then, restart Grafana service and open Grafana in web-browser, usually . + +### Option 2: Install Plugin Manually + Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation. ```bash @@ -50,11 +78,7 @@ If Grafana is running in a Docker environment, the TDengine plugin can be automa GF_INSTALL_PLUGINS=tdengine-datasource ``` -## Using Grafana - -### Configuring Data Sources - -Users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure. +Now users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure. ![TDengine Database TDinsight plugin add datasource 1](./grafana/add_datasource1.webp) @@ -94,4 +118,11 @@ Follow the default prompt to query the average system memory usage for the speci ### Importing the Dashboard -In version 2.3.3.0 and above, you can import the TDinsight Dashboard (Grafana Dashboard ID: [15168](https://grafana.com/grafana/dashboards/15167)) as a monitoring visualization tool for TDengine clusters. You can find installation and usage instructions in the TDinsight User Manual (/reference/tdinsight/). +You can install TDinsight dashboard in data source configuration page (like `http://localhost:3000/datasources/edit/1/dashboards`) as a monitoring visualization tool for TDengine cluster. The dashboard is published in Grafana as [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167). Check the [TDinsight User Manual](/reference/tdinsight/) for the details. + +For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list: + +- [15146](https://grafana.com/grafana/dashboards/15146): Monitor multiple TDengine clusters. +- [15155](https://grafana.com/grafana/dashboards/15155): TDengine alert demo. +- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight. +- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf node metrics dashboard using TDengine data source. diff --git a/include/client/taos.h b/include/client/taos.h index 7f317b3a9d0f47768e106b1745fa1ec7325b449a..b4e5a41ccfdb82857377b9b2c281d69375047d59 100644 --- a/include/client/taos.h +++ b/include/client/taos.h @@ -103,10 +103,10 @@ typedef void (*__taos_async_fn_t)(void *param, TAOS_RES *, int code); typedef struct TAOS_MULTI_BIND { int buffer_type; - void * buffer; + void *buffer; uintptr_t buffer_length; - int32_t * length; - char * is_null; + int32_t *length; + char *is_null; int num; } TAOS_MULTI_BIND; @@ -130,7 +130,7 @@ DLL_EXPORT void taos_cleanup(void); DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...); DLL_EXPORT setConfRet taos_set_config(const char *config); DLL_EXPORT int taos_init(void); -DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port); +DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port); DLL_EXPORT TAOS *taos_connect_l(const char *ip, int ipLen, const char *user, int userLen, const char *pass, int passLen, const char *db, int dbLen, uint16_t port); DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port); @@ -147,17 +147,17 @@ DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); -DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert); -DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums); -DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes); -DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); -DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); -DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx); -DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt); -DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert); +DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums); +DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes); +DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); +DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); +DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx); +DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt); DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt); -DLL_EXPORT char * taos_stmt_errstr(TAOS_STMT *stmt); +DLL_EXPORT char *taos_stmt_errstr(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt); @@ -179,11 +179,11 @@ DLL_EXPORT bool taos_is_update_query(TAOS_RES *res); DLL_EXPORT int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows); DLL_EXPORT int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows); DLL_EXPORT int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData); -DLL_EXPORT int * taos_get_column_data_offset(TAOS_RES *res, int columnIndex); +DLL_EXPORT int *taos_get_column_data_offset(TAOS_RES *res, int columnIndex); DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql); DLL_EXPORT void taos_reset_current_db(TAOS *taos); -DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res); +DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res); DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res); DLL_EXPORT const char *taos_get_server_info(TAOS *taos); @@ -194,6 +194,7 @@ DLL_EXPORT int taos_errno(TAOS_RES *tres); DLL_EXPORT void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param); DLL_EXPORT void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param); +DLL_EXPORT void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param); // Shuduo: temporary enable for app build #if 1 @@ -204,17 +205,17 @@ DLL_EXPORT TAOS_RES *taos_consume(TAOS_SUB *tsub); DLL_EXPORT void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress); #endif -DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList); +DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList); DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision); /* --------------------------TMQ INTERFACE------------------------------- */ -enum tmq_resp_err_t { +enum { TMQ_RESP_ERR__FAIL = -1, TMQ_RESP_ERR__SUCCESS = 0, }; -typedef enum tmq_resp_err_t tmq_resp_err_t; +typedef int32_t tmq_resp_err_t; typedef struct tmq_t tmq_t; typedef struct tmq_topic_vgroup_t tmq_topic_vgroup_t; @@ -229,7 +230,7 @@ DLL_EXPORT tmq_list_t *tmq_list_new(); DLL_EXPORT int32_t tmq_list_append(tmq_list_t *, const char *); DLL_EXPORT void tmq_list_destroy(tmq_list_t *); DLL_EXPORT int32_t tmq_list_get_size(const tmq_list_t *); -DLL_EXPORT char ** tmq_list_to_c_array(const tmq_list_t *); +DLL_EXPORT char **tmq_list_to_c_array(const tmq_list_t *); DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen); @@ -240,7 +241,7 @@ DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t); DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list); DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq); DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics); -DLL_EXPORT TAOS_RES * tmq_consumer_poll(tmq_t *tmq, int64_t timeout); +DLL_EXPORT TAOS_RES *tmq_consumer_poll(tmq_t *tmq, int64_t timeout); DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq); DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets); DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param); @@ -260,7 +261,7 @@ enum tmq_conf_res_t { typedef enum tmq_conf_res_t tmq_conf_res_t; -DLL_EXPORT tmq_conf_t * tmq_conf_new(); +DLL_EXPORT tmq_conf_t *tmq_conf_new(); DLL_EXPORT tmq_conf_res_t tmq_conf_set(tmq_conf_t *conf, const char *key, const char *value); DLL_EXPORT void tmq_conf_destroy(tmq_conf_t *conf); DLL_EXPORT void tmq_conf_set_auto_commit_cb(tmq_conf_t *conf, tmq_commit_cb *cb, void *param); diff --git a/include/common/tdatablock.h b/include/common/tdatablock.h index ab76cb404afa3480e7bb2cab15b40dfe8b2df8af..af46535c948a2aa0be0282f580b68ade9955c8fa 100644 --- a/include/common/tdatablock.h +++ b/include/common/tdatablock.h @@ -206,7 +206,7 @@ SSDataBlock* blockDataExtractBlock(SSDataBlock* pBlock, int32_t startIndex, int3 size_t blockDataGetSize(const SSDataBlock* pBlock); size_t blockDataGetRowSize(SSDataBlock* pBlock); double blockDataGetSerialRowSize(const SSDataBlock* pBlock); -size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock); +size_t blockDataGetSerialMetaSize(uint32_t numOfCols); int32_t blockDataSort(SSDataBlock* pDataBlock, SArray* pOrderInfo); int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullFirst); @@ -238,7 +238,7 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pSchema, bool const char* stbFullName, int32_t vgId); static FORCE_INLINE int32_t blockGetEncodeSize(const SSDataBlock* pBlock) { - return blockDataGetSerialMetaSize(pBlock) + blockDataGetSize(pBlock); + return blockDataGetSerialMetaSize(pBlock->info.numOfCols) + blockDataGetSize(pBlock); } static FORCE_INLINE int32_t blockCompressColData(SColumnInfoData* pColRes, int32_t numOfRows, char* data, diff --git a/include/common/tmsg.h b/include/common/tmsg.h index 4a3c4b0c3f236557ffe4e28ff7cedce3e82093fa..5453c8d0ea19e0bbc33c651f3a9d3acf73814b03 100644 --- a/include/common/tmsg.h +++ b/include/common/tmsg.h @@ -2500,6 +2500,7 @@ typedef struct { int64_t sliding; int64_t dstTbUid; int32_t dstVgId; // for stream + SEpSet epSet; char* expr; } STableIndexInfo; diff --git a/include/libs/catalog/catalog.h b/include/libs/catalog/catalog.h index 21fee3cc3e17cd3c9bf970f70c469f905472b1d8..4d074db08db3ecd1aa9f12d7f4a8e6bd41df9e42 100644 --- a/include/libs/catalog/catalog.h +++ b/include/libs/catalog/catalog.h @@ -71,16 +71,21 @@ typedef struct SCatalogReq { bool forceUpdate; } SCatalogReq; +typedef struct SMetaRes { + int32_t code; + void* pRes; +} SMetaRes; + typedef struct SMetaData { - SArray *pDbVgroup; // SArray*> - SArray *pDbCfg; // SArray - SArray *pDbInfo; // SArray - SArray *pTableMeta; // SArray - SArray *pTableHash; // SArray - SArray *pUdfList; // SArray - SArray *pIndex; // SArray - SArray *pUser; // SArray - SArray *pQnodeList; // SArray + SArray *pDbVgroup; // pRes = SArray* + SArray *pDbCfg; // pRes = SDbCfgInfo* + SArray *pDbInfo; // pRes = SDbInfo* + SArray *pTableMeta; // pRes = STableMeta* + SArray *pTableHash; // pRes = SVgroupInfo* + SArray *pUdfList; // pRes = SFuncInfo* + SArray *pIndex; // pRes = SIndexInfo* + SArray *pUser; // pRes = bool* + SArray *pQnodeList; // pRes = SQueryNodeAddr* } SMetaData; typedef struct SCatalogCfg { diff --git a/include/libs/function/function.h b/include/libs/function/function.h index e8cb363e08fa65385d36762face331f5de5cf1eb..810991f770a87c7697b97129f1a5a637c63abe61 100644 --- a/include/libs/function/function.h +++ b/include/libs/function/function.h @@ -197,6 +197,7 @@ typedef struct SAggFunctionInfo { struct SScalarParam { SColumnInfoData *columnData; SHashObj *pHashFilter; + int32_t hashValueType; void *param; // other parameter, such as meta handle from vnode, to extract table name/tag value int32_t numOfRows; }; diff --git a/include/libs/function/functionMgt.h b/include/libs/function/functionMgt.h index e6512b5bac7dc14315e7d9f55c476c9879ebe6eb..77376a05d91b8a1e1167e8474d2f0e572fa9e78e 100644 --- a/include/libs/function/functionMgt.h +++ b/include/libs/function/functionMgt.h @@ -127,6 +127,8 @@ typedef enum EFunctionType { FUNCTION_TYPE_APERCENTILE_MERGE, FUNCTION_TYPE_SPREAD_PARTIAL, FUNCTION_TYPE_SPREAD_MERGE, + FUNCTION_TYPE_HISTOGRAM_PARTIAL, + FUNCTION_TYPE_HISTOGRAM_MERGE, // user defined funcion FUNCTION_TYPE_UDF = 10000 diff --git a/include/libs/index/index.h b/include/libs/index/index.h index bd601f1d9fa5b9cb1dc287b14970d1d143dd99d3..9e71c941d36b6646a34aed8a69be4cb08864934e 100644 --- a/include/libs/index/index.h +++ b/include/libs/index/index.h @@ -136,14 +136,14 @@ int indexRebuild(SIndex* index, SIndexOpts* opt); * @param index (output, index json object) * @return error code */ -int tIndexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index); +int indexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index); /* * close index * @param index (input, index to be closed) * @return void */ -void tIndexJsonClose(SIndexJson* index); +void indexJsonClose(SIndexJson* index); /* * insert terms into index @@ -152,7 +152,7 @@ void tIndexJsonClose(SIndexJson* index); * @param uid (input, uid of terms) * @return error code */ -int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid); +int indexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid); /* * search index * @param index (input, index object) @@ -161,7 +161,7 @@ int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid); * @return error code */ -int tIndexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result); +int indexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result); /* * @param * @param diff --git a/include/libs/qcom/query.h b/include/libs/qcom/query.h index a717a6b333fcd70db5344d07930967f09b8a194c..94e68ad48a13250b4536f6548a384dbdf4debbd0 100644 --- a/include/libs/qcom/query.h +++ b/include/libs/qcom/query.h @@ -210,8 +210,8 @@ extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t #define NEED_CLIENT_RM_TBLMETA_ERROR(_code) \ ((_code) == TSDB_CODE_PAR_TABLE_NOT_EXIST || (_code) == TSDB_CODE_VND_TB_NOT_EXIST || \ (_code) == TSDB_CODE_PAR_INVALID_COLUMNS_NUM || (_code) == TSDB_CODE_PAR_INVALID_COLUMN || \ - (_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code == TSDB_CODE_PAR_VALUE_TOO_LONG) || \ - (_code == TSDB_CODE_PAR_INVALID_DROP_COL)) + (_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code) == TSDB_CODE_PAR_VALUE_TOO_LONG || \ + (_code) == TSDB_CODE_PAR_INVALID_DROP_COL || ((_code) == TSDB_CODE_TDB_INVALID_TABLE_ID)) #define NEED_CLIENT_REFRESH_VG_ERROR(_code) \ ((_code) == TSDB_CODE_VND_HASH_MISMATCH || (_code) == TSDB_CODE_VND_INVALID_VGROUP_ID) #define NEED_CLIENT_REFRESH_TBLMETA_ERROR(_code) ((_code) == TSDB_CODE_TDB_TABLE_RECREATED) diff --git a/include/libs/stream/tstream.h b/include/libs/stream/tstream.h index ddd7e1cd02b69c74469da66623372c535d873c5b..6b5eb3b491fdbbd441eed703d5327efe67afcb64 100644 --- a/include/libs/stream/tstream.h +++ b/include/libs/stream/tstream.h @@ -339,6 +339,7 @@ typedef struct { int32_t sourceTaskId; int32_t sourceVg; int32_t sourceChildId; + int32_t upstreamNodeId; #if 0 int64_t sourceVer; #endif diff --git a/include/libs/sync/syncTools.h b/include/libs/sync/syncTools.h index 4b160c9e6163946edd6fee236ca99f4c665a0f15..bd396edf55130cb9426a84a6eb764e16418cd279 100644 --- a/include/libs/sync/syncTools.h +++ b/include/libs/sync/syncTools.h @@ -33,7 +33,7 @@ SSyncNode* syncNodeAcquire(int64_t rid); void syncNodeRelease(SSyncNode* pNode); int32_t syncGetRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg); -int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg); +int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcHandleInfo* pInfo); void syncSetMsgCb(int64_t rid, const SMsgCb* msgcb); char* sync2SimpleStr(int64_t rid); diff --git a/include/libs/transport/trpc.h b/include/libs/transport/trpc.h index 839194da94e5a184ab11b446077e334f085d68b5..fd57eef83ae19c00de8415cafff7764ef34ecdc4 100644 --- a/include/libs/transport/trpc.h +++ b/include/libs/transport/trpc.h @@ -33,7 +33,10 @@ extern int32_t tsRpcHeadSize; typedef struct { uint32_t clientIp; uint16_t clientPort; - char user[TSDB_USER_LEN]; + union { + char user[TSDB_USER_LEN]; + int64_t applyIndex; + }; } SRpcConnInfo; typedef struct SRpcHandleInfo { @@ -43,6 +46,7 @@ typedef struct SRpcHandleInfo { int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp); int32_t persistHandle; // persist handle or not + SRpcConnInfo connInfo; // app info void *ahandle; // app handle set by client void *wrapper; // wrapper handle diff --git a/include/util/taoserror.h b/include/util/taoserror.h index 5a83d1525e654a5ed9be6c315d1d94fea1ea2b68..ae0191e6d206f945b0bf4e72eca3ebdaf7ca4f6b 100644 --- a/include/util/taoserror.h +++ b/include/util/taoserror.h @@ -339,9 +339,9 @@ int32_t* taosGetErrno(); #define TSDB_CODE_TDB_TAG_VER_OUT_OF_DATE TAOS_DEF_ERROR_CODE(0, 0x060D) #define TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x060E) #define TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP TAOS_DEF_ERROR_CODE(0, 0x060F) -#define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0600) -#define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0601) -#define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0602) +#define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0610) +#define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0611) +#define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0612) #define TSDB_CODE_TDB_FILE_ALREADY_EXISTS TAOS_DEF_ERROR_CODE(0, 0x0613) #define TSDB_CODE_TDB_TABLE_RECONFIGURE TAOS_DEF_ERROR_CODE(0, 0x0614) #define TSDB_CODE_TDB_IVD_CREATE_TABLE_INFO TAOS_DEF_ERROR_CODE(0, 0x0615) diff --git a/include/util/tdef.h b/include/util/tdef.h index f6169af1aae0429d3793f4b6e0dcc7783f5a3545..1fa8ad19126f2fe69b4500eac2d9dfdd2665ba17 100644 --- a/include/util/tdef.h +++ b/include/util/tdef.h @@ -356,8 +356,8 @@ typedef enum ELogicConditionType { #define TSDB_DEFAULT_EXPLAIN_VERBOSE false -#define TSDB_EXPLAIN_RESULT_ROW_SIZE 1024 -#define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY PLAN" +#define TSDB_EXPLAIN_RESULT_ROW_SIZE 512 +#define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY_PLAN" #define TSDB_MAX_FIELD_LEN 16384 #define TSDB_MAX_BINARY_LEN (TSDB_MAX_FIELD_LEN - TSDB_KEYSIZE) // keep 16384 diff --git a/source/client/inc/clientInt.h b/source/client/inc/clientInt.h index 0630704d93df9c564fcc91d9debfa1be397c7427..6aa83e95754b946613a2226ce390804714e701f9 100644 --- a/source/client/inc/clientInt.h +++ b/source/client/inc/clientInt.h @@ -45,7 +45,7 @@ extern "C" { #define ERROR_MSG_BUF_DEFAULT_SIZE 512 #define HEARTBEAT_INTERVAL 1500 // ms -#define SYNC_ON_TOP_OF_ASYNC 0 +#define SYNC_ON_TOP_OF_ASYNC 1 enum { RES_TYPE__QUERY = 1, @@ -144,7 +144,7 @@ typedef struct STscObj { int32_t numOfReqs; // number of sqlObj bound to this connection SAppInstInfo* pAppInfo; SHashObj* pRequests; - int8_t schemalessType; + int8_t schemalessType; // todo remove it, this attribute should be move to request } STscObj; typedef struct SResultColumn { @@ -171,6 +171,7 @@ typedef struct SReqResultInfo { uint32_t current; bool completed; int32_t precision; + bool convertUcs4; int32_t payloadLen; } SReqResultInfo; @@ -222,7 +223,7 @@ typedef struct SSyncQueryParam { SRequestObj* pRequest; } SSyncQueryParam; -void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4); +void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4); void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4); void doSetOneRowPtr(SReqResultInfo* pResultInfo); diff --git a/source/client/src/clientEnv.c b/source/client/src/clientEnv.c index 75e18843609d5225cf93674e28f02f5df08c6dd3..171f06c2575af50ca7b514c70b9b30ddb601b46b 100644 --- a/source/client/src/clientEnv.c +++ b/source/client/src/clientEnv.c @@ -191,6 +191,8 @@ void *createRequest(STscObj *pObj, int32_t type) { pRequest->requestId = generateRequestId(); pRequest->metric.start = taosGetTimestampUs(); + pRequest->body.resInfo.convertUcs4 = true; // convert ucs4 by default + pRequest->type = type; pRequest->pTscObj = pObj; pRequest->msgBuf = taosMemoryCalloc(1, ERROR_MSG_BUF_DEFAULT_SIZE); diff --git a/source/client/src/clientImpl.c b/source/client/src/clientImpl.c index 58c31548b05c104e9185c7658db67235b4fc6691..90251218540230aaf1e7583b6d989cb06520ec87 100644 --- a/source/client/src/clientImpl.c +++ b/source/client/src/clientImpl.c @@ -580,7 +580,8 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) { STscObj* pTscObj = pRequest->pTscObj; if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code)) { - tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); + tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code), + pRequest->retry, pRequest->requestId); pRequest->prevCode = code; doAsyncQuery(pRequest, true); return; @@ -592,6 +593,7 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) { pRequest->code = code; } + tscDebug("schedulerExecCb request type %s", TMSG_INFO(pRequest->type)); if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) { removeMeta(pTscObj, pRequest->tableList); } @@ -695,6 +697,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { if (TSDB_CODE_SUCCESS == code) { schedulerAsyncExecJob(pAppInfo->pTransporter, pNodeList, pRequest->body.pDag, &pRequest->body.queryJob, pRequest->sqlstr, pRequest->metric.start, schedulerExecCb, pRequest); + } else { + pRequest->body.queryFp(pRequest->body.param, pRequest, code); } //todo not to be released here @@ -702,7 +706,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { break; } case QUERY_EXEC_MODE_EMPTY_RESULT: - pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT; + pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT; + pRequest->body.queryFp(pRequest->body.param, pRequest, 0); break; default: break; @@ -1109,7 +1114,7 @@ static void syncFetchFn(void* param, TAOS_RES* res, int32_t numOfRows) { tsem_post(&pParam->sem); } -void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) { +void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) { assert(pRequest != NULL); SReqResultInfo* pResultInfo = &pRequest->body.resInfo; @@ -1121,6 +1126,10 @@ void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUc } SSyncQueryParam* pParam = pRequest->body.param; + + // convert ucs4 to native multi-bytes string + pResultInfo->convertUcs4 = convertUcs4; + taos_fetch_rows_a(pRequest, syncFetchFn, pParam); tsem_wait(&pParam->sem); } @@ -1339,6 +1348,17 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 uint64_t groupId = *(uint64_t*)p; p += sizeof(uint64_t); + // check fields + for(int32_t i = 0; i < numOfCols; ++i) { + int16_t type = *(int16_t*) p; + p += sizeof(int16_t); + + int32_t bytes = *(int32_t*) p; + p += sizeof(int32_t); + +// ASSERT(type == pFields[i].type && bytes == pFields[i].bytes); + } + int32_t* colLength = (int32_t*)p; p += sizeof(int32_t) * numOfCols; diff --git a/source/client/src/clientMain.c b/source/client/src/clientMain.c index d192b65bf0a628cb4ee0846c5528a8e6a56b3ca3..5e442c4bf1194aa2c53e9021d9f728b42ef1d3ee 100644 --- a/source/client/src/clientMain.c +++ b/source/client/src/clientMain.c @@ -219,14 +219,13 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { if (TD_RES_QUERY(res)) { SRequestObj *pRequest = (SRequestObj *)res; +#if SYNC_ON_TOP_OF_ASYNC + return doAsyncFetchRows(pRequest, true, true); +#else if (pRequest->type == TSDB_SQL_RETRIEVE_EMPTY_RESULT || pRequest->type == TSDB_SQL_INSERT || pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0) { return NULL; } - -#if SYNC_ON_TOP_OF_ASYNC - return doAsyncFetchRow(pRequest, true, true); -#else return doFetchRows(pRequest, true, true); #endif @@ -489,6 +488,7 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) { if (res == NULL) { return 0; } + if (TD_RES_QUERY(res)) { SRequestObj *pRequest = (SRequestObj *)res; @@ -500,7 +500,11 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) { return 0; } - doFetchRows(pRequest, false, true); +#if SYNC_ON_TOP_OF_ASYNC + doAsyncFetchRows(pRequest, false, true); +#else + doFetchRows(pRequest, true, true); +#endif // TODO refactor SReqResultInfo *pResultInfo = &pRequest->body.resInfo; @@ -548,7 +552,11 @@ int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData) { return 0; } +#if SYNC_ON_TOP_OF_ASYNC + doAsyncFetchRows(pRequest, false, false); +#else doFetchRows(pRequest, false, false); +#endif SReqResultInfo *pResultInfo = &pRequest->body.resInfo; @@ -625,8 +633,10 @@ void retrieveMetaCallback(SMetaData* pResultMeta, void* param, int32_t code) { taosMemoryFree(pWrapper); launchAsyncQuery(pRequest, pQuery); } else { + tscDebug("error happens, code:%d", code); if (NEED_CLIENT_HANDLE_ERROR(code)) { - tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); + tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code), + pRequest->retry, pRequest->requestId); pRequest->prevCode = code; doAsyncQuery(pRequest, true); return; @@ -691,6 +701,7 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) { .pTransporter = pTscObj->pAppInfo->pTransporter, .pStmtCb = NULL, .pUser = pTscObj->user, + .schemalessType = pTscObj->schemalessType, .isSuperUser = (0 == strcmp(pTscObj->user, TSDB_DEFAULT_USER)), .async = true,}; return TSDB_CODE_SUCCESS; @@ -699,13 +710,14 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) { void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) { SParseContext* pCxt = NULL; STscObj *pTscObj = pRequest->pTscObj; + int32_t code = 0; if (pRequest->retry++ > REQUEST_TOTAL_EXEC_TIMES) { - pRequest->code = pRequest->prevCode; + code = pRequest->prevCode; goto _error; } - int32_t code = createParseContext(pRequest, &pCxt); + code = createParseContext(pRequest, &pCxt); if (code != TSDB_CODE_SUCCESS) { goto _error; } @@ -742,7 +754,7 @@ void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) { } _error: - tscError("0x%"PRIx64" error happens, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); + tscError("0x%"PRIx64" error happens, code:%d - %s, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code), pRequest->requestId); terrno = code; pRequest->code = code; pRequest->body.queryFp(pRequest->body.param, pRequest, code); @@ -763,11 +775,11 @@ static void fetchCallback(void* pResult, void* param, int32_t code) { } if (pRequest->code != TSDB_CODE_SUCCESS) { - pRequest->code = code; pRequest->body.fetchFp(pRequest->body.param, pRequest, 0); + return; } - pRequest->code = setQueryResultFromRsp(&pRequest->body.resInfo, (SRetrieveTableRsp*)pResultInfo->pData, true, false); + pRequest->code = setQueryResultFromRsp(pResultInfo, (SRetrieveTableRsp*)pResultInfo->pData, pResultInfo->convertUcs4, false); if (pRequest->code != TSDB_CODE_SUCCESS) { pResultInfo->numOfRows = 0; pRequest->code = code; @@ -807,6 +819,13 @@ void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) { schedulerAsyncFetchRows(pRequest->body.queryJob, fetchCallback, pRequest); } +void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param) { + ASSERT(res != NULL && fp != NULL); + SRequestObj *pRequest = res; + pRequest->body.resInfo.convertUcs4 = false; + taos_fetch_rows_a(res, fp, param); +} + TAOS_SUB *taos_subscribe(TAOS *taos, int restart, const char *topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp, void *param, int interval) { // TODO diff --git a/source/client/src/tmq.c b/source/client/src/tmq.c index ef37eed284ccdd11b5b8a32451104327e73433c8..b5011a19a29d2025ad84bfe662ed4e2050da8655 100644 --- a/source/client/src/tmq.c +++ b/source/client/src/tmq.c @@ -323,7 +323,7 @@ static int32_t tmqMakeTopicVgKey(char* dst, const char* topicName, int32_t vg) { int32_t tmqCommitCb(void* param, const SDataBuf* pMsg, int32_t code) { SMqCommitCbParam* pParam = (SMqCommitCbParam*)param; - pParam->rspErr = code == 0 ? TMQ_RESP_ERR__SUCCESS : TMQ_RESP_ERR__FAIL; + pParam->rspErr = code; if (pParam->async) { if (pParam->automatic && pParam->tmq->commitCb) { pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets, @@ -432,12 +432,13 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_ code = pParam->rspErr; tsem_destroy(&pParam->rspSem); taosMemoryFree(pParam); + } else { + code = 0; } // avoid double free if msg is sent buf = NULL; - code = 0; END: if (buf) taosMemoryFree(buf); /*if (pParam) taosMemoryFree(pParam);*/ @@ -445,9 +446,9 @@ END: if (code != 0 && async) { if (automatic) { - tmq->commitCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam); + tmq->commitCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam); } else { - userCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, userParam); + userCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, userParam); } } @@ -493,7 +494,6 @@ int32_t tmqHandleAllDelayedTask(tmq_t* tmq) { tmqAskEp(tmq, true); taosTmrReset(tmqAssignDelayedHbTask, 1000, tmq, tmqMgmt.timer, &tmq->hbTimer); } else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) { - /*tmq_commit(tmq, NULL, true);*/ tmqCommitInner(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam); taosTmrReset(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, tmq, tmqMgmt.timer, &tmq->commitTimer); } else if (*pTaskType == TMQ_DELAYED_TASK__REPORT) { @@ -666,94 +666,6 @@ FAIL: tmq_resp_err_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) { return tmqCommitInner(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam); -#if 0 - // TODO: add read write lock - SRequestObj* pRequest = NULL; - tmq_resp_err_t resp = TMQ_RESP_ERR__SUCCESS; - // build msg - // send to mnode - SMqCMCommitOffsetReq req; - SArray* pOffsets = NULL; - - if (offsets == NULL) { - pOffsets = taosArrayInit(0, sizeof(SMqOffset)); - for (int i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { - SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); - for (int j = 0; j < taosArrayGetSize(pTopic->vgs); j++) { - SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); - SMqOffset offset; - strcpy(offset.topicName, pTopic->topicName); - strcpy(offset.cgroup, tmq->groupId); - offset.vgId = pVg->vgId; - offset.offset = pVg->currentOffset; - taosArrayPush(pOffsets, &offset); - } - } - req.num = pOffsets->size; - req.offsets = pOffsets->pData; - } else { - req.num = taosArrayGetSize(&offsets->container); - req.offsets = (SMqOffset*)offsets->container.pData; - } - - SEncoder encoder; - - tEncoderInit(&encoder, NULL, 0); - tEncodeSMqCMCommitOffsetReq(&encoder, &req); - int32_t tlen = encoder.pos; - void* buf = taosMemoryMalloc(tlen); - if (buf == NULL) { - tEncoderClear(&encoder); - return -1; - } - tEncoderClear(&encoder); - - tEncoderInit(&encoder, buf, tlen); - tEncodeSMqCMCommitOffsetReq(&encoder, &req); - tEncoderClear(&encoder); - - pRequest = createRequest(tmq->pTscObj, NULL, NULL, TDMT_MND_MQ_COMMIT_OFFSET); - if (pRequest == NULL) { - tscError("failed to malloc request"); - } - - SMqCommitCbParam* pParam = taosMemoryCalloc(1, sizeof(SMqCommitCbParam)); - if (pParam == NULL) { - return -1; - } - pParam->tmq = tmq; - tsem_init(&pParam->rspSem, 0, 0); - pParam->async = async; - pParam->offsets = pOffsets; - - pRequest->body.requestMsg = (SDataBuf){ - .pData = buf, - .len = tlen, - .handle = NULL, - }; - - SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest); - sendInfo->requestObjRefId = 0; - sendInfo->param = pParam; - sendInfo->fp = tmqCommitCb; - SEpSet epSet = getEpSet_s(&tmq->pTscObj->pAppInfo->mgmtEp); - - int64_t transporterId = 0; - asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo); - - if (!async) { - tsem_wait(&pParam->rspSem); - resp = pParam->rspErr; - tsem_destroy(&pParam->rspSem); - taosMemoryFree(pParam); - - if (pOffsets) { - taosArrayDestroy(pOffsets); - } - } - - return resp; -#endif } tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { @@ -858,93 +770,6 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para conf->commitCbUserParam = param; } -#if 0 -TAOS_RES* tmq_create_stream(TAOS* taos, const char* streamName, const char* tbName, const char* sql) { - STscObj* pTscObj = (STscObj*)taos; - SRequestObj* pRequest = NULL; - SQuery* pQueryNode = NULL; - char* astStr = NULL; - int32_t sqlLen; - - terrno = TSDB_CODE_SUCCESS; - if (taos == NULL || streamName == NULL || sql == NULL) { - tscError("invalid parameters for creating stream, connObj:%p, stream name:%s, sql:%s", taos, streamName, sql); - terrno = TSDB_CODE_TSC_INVALID_INPUT; - goto _return; - } - sqlLen = strlen(sql); - - if (strlen(tbName) >= TSDB_TABLE_NAME_LEN) { - tscError("output tb name too long, max length:%d", TSDB_TABLE_NAME_LEN - 1); - terrno = TSDB_CODE_TSC_INVALID_INPUT; - goto _return; - } - - if (sqlLen > TSDB_MAX_ALLOWED_SQL_LEN) { - tscError("sql string exceeds max length:%d", TSDB_MAX_ALLOWED_SQL_LEN); - terrno = TSDB_CODE_TSC_EXCEED_SQL_LIMIT; - goto _return; - } - - tscDebug("start to create stream: %s", streamName); - - int32_t code = 0; - CHECK_CODE_GOTO(buildRequest(pTscObj, sql, sqlLen, &pRequest), _return); - CHECK_CODE_GOTO(parseSql(pRequest, false, &pQueryNode, NULL), _return); - CHECK_CODE_GOTO(nodesNodeToString(pQueryNode->pRoot, false, &astStr, NULL), _return); - - /*printf("%s\n", pStr);*/ - - SName name = {.acctId = pTscObj->acctId, .type = TSDB_TABLE_NAME_T}; - strcpy(name.dbname, pRequest->pDb); - strcpy(name.tname, streamName); - - SCMCreateStreamReq req = { - .igExists = 1, - .ast = astStr, - .sql = (char*)sql, - }; - tNameExtractFullName(&name, req.name); - strcpy(req.targetStbFullName, tbName); - - int tlen = tSerializeSCMCreateStreamReq(NULL, 0, &req); - void* buf = taosMemoryMalloc(tlen); - if (buf == NULL) { - goto _return; - } - - tSerializeSCMCreateStreamReq(buf, tlen, &req); - - pRequest->body.requestMsg = (SDataBuf){ - .pData = buf, - .len = tlen, - .handle = NULL, - }; - pRequest->type = TDMT_MND_CREATE_STREAM; - - SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest); - SEpSet epSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp); - - int64_t transporterId = 0; - asyncSendMsgToServer(pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo); - - tsem_wait(&pRequest->body.rspSem); - -_return: - taosMemoryFreeClear(astStr); - qDestroyQuery(pQueryNode); - /*if (sendInfo != NULL) {*/ - /*destroySendMsgInfo(sendInfo);*/ - /*}*/ - - if (pRequest != NULL && terrno != TSDB_CODE_SUCCESS) { - pRequest->code = terrno; - } - - return pRequest; -} -#endif - #if 0 int32_t tmqGetSkipLogNum(tmq_message_t* tmq_message) { if (tmq_message == NULL) return 0; @@ -1474,16 +1299,16 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) { tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { if (tmq->status == TMQ_CONSUMER_STATUS__READY) { tmq_resp_err_t rsp = tmq_commit_sync(tmq, NULL); - if (rsp == TMQ_RESP_ERR__FAIL) { - return TMQ_RESP_ERR__FAIL; + if (rsp != TMQ_RESP_ERR__SUCCESS) { + return rsp; } tmq_list_t* lst = tmq_list_new(); rsp = tmq_subscribe(tmq, lst); tmq_list_destroy(lst); - if (rsp == TMQ_RESP_ERR__FAIL) { - return TMQ_RESP_ERR__FAIL; + if (rsp != TMQ_RESP_ERR__SUCCESS) { + return rsp; } } // TODO: free resources @@ -1493,8 +1318,11 @@ tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { const char* tmq_err2str(tmq_resp_err_t err) { if (err == TMQ_RESP_ERR__SUCCESS) { return "success"; + } else if (err == TMQ_RESP_ERR__FAIL) { + return "fail"; + } else { + return tstrerror(err); } - return "fail"; } const char* tmq_get_topic_name(TAOS_RES* res) { @@ -1536,10 +1364,11 @@ const char* tmq_get_table_name(TAOS_RES* res) { } return NULL; } -DLL_EXPORT void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) { + +void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) { tmqCommitInner(tmq, offsets, 0, 1, cb, param); } -DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) { +tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) { return tmqCommitInner(tmq, offsets, 0, 0, NULL, NULL); } diff --git a/source/client/test/clientTests.cpp b/source/client/test/clientTests.cpp index f16baba0e7e9f2788bcdb697596f4887cba39afd..f9825c50ffaf77133702cf9b1afaae0242f5bbb5 100644 --- a/source/client/test/clientTests.cpp +++ b/source/client/test/clientTests.cpp @@ -780,31 +780,28 @@ TEST(testCase, async_api_test) { taos_query(pConn, "use test"); - TAOS_RES* pRes = taos_query(pConn, "select * from t1"); - - taos_query(pConn, "alter table t1 add column b int"); - pRes = taos_query(pConn, "insert into t1 values(now, 1, 2)"); + TAOS_RES* pRes = taos_query(pConn, "desc abc1.tu"); if (taos_errno(pRes) != 0) { printf("failed, reason:%s\n", taos_errstr(pRes)); } -// int32_t n = 0; -// TAOS_ROW pRow = NULL; -// TAOS_FIELD* pFields = taos_fetch_fields(pRes); -// int32_t numOfFields = taos_num_fields(pRes); -// -// char str[512] = {0}; -// while ((pRow = taos_fetch_row(pRes)) != NULL) { -// int32_t* length = taos_fetch_lengths(pRes); -// for(int32_t i = 0; i < numOfFields; ++i) { -// printf("(%d):%d " , i, length[i]); -// } -// printf("\n"); -// -// int32_t code = taos_print_row(str, pRow, pFields, numOfFields); -// printf("%s\n", str); -// memset(str, 0, sizeof(str)); -// } + int32_t n = 0; + TAOS_ROW pRow = NULL; + TAOS_FIELD* pFields = taos_fetch_fields(pRes); + int32_t numOfFields = taos_num_fields(pRes); + + char str[512] = {0}; + while ((pRow = taos_fetch_row(pRes)) != NULL) { + int32_t* length = taos_fetch_lengths(pRes); + for(int32_t i = 0; i < numOfFields; ++i) { + printf("(%d):%d " , i, length[i]); + } + printf("\n"); + + int32_t code = taos_print_row(str, pRow, pFields, numOfFields); + printf("%s\n", str); + memset(str, 0, sizeof(str)); + } taos_query_a(pConn, "alter table test.m1 comment 'abcde' ", queryCallback, pConn); getchar(); diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index 704a319512a7611476a6149e909df7cbcb9483bb..4ebedc98c110a15d2d80eeb20617a71b20536981 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -682,9 +682,10 @@ size_t blockDataGetRowSize(SSDataBlock* pBlock) { * @param pBlock * @return */ -size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock) { - // | total rows/total length | block group id | each column length | - return sizeof(int32_t) + sizeof(uint64_t) + pBlock->info.numOfCols * sizeof(int32_t); +size_t blockDataGetSerialMetaSize(uint32_t numOfCols) { + // | total rows/total length | block group id | column schema | each column length | + return sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t)) + + numOfCols * sizeof(int32_t); } double blockDataGetSerialRowSize(const SSDataBlock* pBlock) { @@ -1217,6 +1218,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) { pBlock->info.numOfCols = numOfCols; pBlock->info.hasVarCol = pDataBlock->info.hasVarCol; pBlock->info.rowSize = pDataBlock->info.rowSize; + pBlock->info.groupId = pDataBlock->info.groupId; for (int32_t i = 0; i < numOfCols; ++i) { SColumnInfoData colInfo = {0}; @@ -1246,7 +1248,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) { } size_t blockDataGetCapacityInRow(const SSDataBlock* pBlock, size_t pageSize) { - int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock); + int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock->info.numOfCols); int32_t rowSize = pBlock->info.rowSize; @@ -1885,34 +1887,44 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols, int8_t needCompress) { + // todo extract method int32_t* actualLen = (int32_t*)data; data += sizeof(int32_t); uint64_t* groupId = (uint64_t*)data; data += sizeof(uint64_t); + for (int32_t i = 0; i < numOfCols; ++i) { + SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i); + *((int16_t*)data) = pColInfoData->info.type; + data += sizeof(int16_t); + + *((int32_t*)data) = pColInfoData->info.bytes; + data += sizeof(int32_t); + } + int32_t* colSizes = (int32_t*)data; data += numOfCols * sizeof(int32_t); - *dataLen = (numOfCols * sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t)); + *dataLen = blockDataGetSerialMetaSize(numOfCols); int32_t numOfRows = pBlock->info.rows; for (int32_t col = 0; col < numOfCols; ++col) { SColumnInfoData* pColRes = (SColumnInfoData*)taosArrayGet(pBlock->pDataBlock, col); // copy the null bitmap + size_t metaSize = 0; if (IS_VAR_DATA_TYPE(pColRes->info.type)) { - size_t metaSize = numOfRows * sizeof(int32_t); + metaSize = numOfRows * sizeof(int32_t); memcpy(data, pColRes->varmeta.offset, metaSize); - data += metaSize; - (*dataLen) += metaSize; } else { - int32_t len = BitmapLen(numOfRows); - memcpy(data, pColRes->nullbitmap, len); - data += len; - (*dataLen) += len; + metaSize = BitmapLen(numOfRows); + memcpy(data, pColRes->nullbitmap, metaSize); } + data += metaSize; + (*dataLen) += metaSize; + if (needCompress) { colSizes[col] = blockCompressColData(pColRes, numOfRows, data, needCompress); data += colSizes[col]; @@ -1933,6 +1945,8 @@ void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData) { blockDataEnsureCapacity(pBlock, numOfRows); + pBlock->info.rows = numOfRows; + const char* pStart = pData; int32_t dataLen = *(int32_t*)pStart; @@ -1941,6 +1955,29 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t pBlock->info.groupId = *(uint64_t*)pStart; pStart += sizeof(uint64_t); + if (pBlock->pDataBlock == NULL) { + pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData)); + taosArraySetSize(pBlock->pDataBlock, numOfCols); + } + + pBlock->info.numOfCols = taosArrayGetSize(pBlock->pDataBlock); + ASSERT(pBlock->pDataBlock->size >= numOfCols); + + for (int32_t i = 0; i < numOfCols; ++i) { + SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i); + pColInfoData->info.type = *(int16_t*)pStart; + pStart += sizeof(int16_t); + + pColInfoData->info.bytes = *(int32_t*)pStart; + pStart += sizeof(int32_t); + + if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) { + pBlock->info.hasVarCol = true; + } + } + + blockDataEnsureCapacity(pBlock, numOfRows); + int32_t* colLen = (int32_t*)pStart; pStart += sizeof(int32_t) * numOfCols; @@ -1961,11 +1998,17 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t pColInfoData->pData = taosMemoryMalloc(colLen[i]); } } else { + if (pColInfoData->nullbitmap == NULL) { + pColInfoData->nullbitmap = taosMemoryCalloc(1, BitmapLen(numOfRows)); + } memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows)); pStart += BitmapLen(numOfRows); } if (colLen[i] > 0) { + if (pColInfoData->pData == NULL) { + pColInfoData->pData = taosMemoryCalloc(1, colLen[i]); + } memcpy(pColInfoData->pData, pStart, colLen[i]); } diff --git a/source/common/src/tdataformat.c b/source/common/src/tdataformat.c index 9aa94b0216ee219e0bc294ae3988a11b14f3ffe4..84a2cf05441b81f3fc1a4e95ecfce92e21387c7d 100644 --- a/source/common/src/tdataformat.c +++ b/source/common/src/tdataformat.c @@ -208,7 +208,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S if (iColumn == 0) { ASSERT(pColVal->cid == pTColumn->colId); ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP); - ASSERT(pTColumn->colId == 0); + ASSERT(pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID); iColVal++; } else { @@ -244,7 +244,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S } } - ASSERT(flags); + // ASSERT(flags); // only 1 column(ts) // decide uint32_t nData = 0; @@ -268,7 +268,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S nDataT = BIT2_SIZE(pTSchema->numOfCols - 1) + pTSchema->flen + ntv; break; default: - ASSERT(0); + break; // only ts column + // ASSERT(0); } uint8_t tflags = 0; @@ -283,7 +284,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S tflags |= TSROW_KV_BIG; } - if (nDataT < nDataK) { + if (nDataT <= nDataK) { nData = nDataT; } else { nData = nDataK; @@ -352,7 +353,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ntv = 0; iColVal = 1; - if (flags & 0xf0 == 0) { + if ((flags & 0xf0) == 0) { switch (flags & 0xf) { case TSROW_HAS_VAL: pf = (*ppRow)->pData; @@ -373,7 +374,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ptv = pf + pTSchema->flen; break; default: - ASSERT(0); + // ASSERT(0); + break; } } else { pTSKVRow = (STSKVRow *)(*ppRow)->pData; @@ -417,13 +419,13 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S } _set_none: - if (flags & 0xf0 == 0) { + if ((flags & 0xf0) == 0) { setBitMap(pb, 0, iColumn - 1, flags); } continue; _set_null: - if (flags & 0xf0 == 0) { + if ((flags & 0xf0) == 0) { setBitMap(pb, 1, iColumn - 1, flags); } else { SET_IDX(pidx, pTSKVRow->nCols, nkv, flags); @@ -433,7 +435,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S continue; _set_value: - if (flags & 0xf0 == 0) { + if ((flags & 0xf0) == 0) { setBitMap(pb, 2, iColumn - 1, flags); if (IS_VAR_DATA_TYPE(pTColumn->type)) { @@ -489,13 +491,13 @@ void tTSRowFree(STSRow2 *pRow) { } void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) { - uint8_t isTuple = (pRow->flags & 0xf0 == 0) ? 1 : 0; + uint8_t isTuple = ((pRow->flags & 0xf0) == 0) ? 1 : 0; STColumn *pTColumn = &pTSchema->columns[iCol]; uint8_t flags = pRow->flags & (uint8_t)0xf; SValue value; ASSERT(iCol < pTSchema->numOfCols); - ASSERT(flags); + // ASSERT(flags); // only 1 ts column ASSERT(pRow->sver == pTSchema->version); if (iCol == 0) { @@ -505,7 +507,7 @@ void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal if (flags == TSROW_HAS_NONE) { goto _return_none; - } else if (flags == TSROW_HAS_NONE) { + } else if (flags == TSROW_HAS_NULL) { goto _return_null; } diff --git a/source/common/src/tmsg.c b/source/common/src/tmsg.c index 51c40827b5f4854f19013ebedc2159c6990097d9..b9b4556255f5b4a2b9dbe5ff22fb2037f5c4e875 100644 --- a/source/common/src/tmsg.c +++ b/source/common/src/tmsg.c @@ -2427,6 +2427,7 @@ int32_t tSerializeSTableIndexInfo(SEncoder *pEncoder, STableIndexInfo* pInfo) { if (tEncodeI64(pEncoder, pInfo->sliding) < 0) return -1; if (tEncodeI64(pEncoder, pInfo->dstTbUid) < 0) return -1; if (tEncodeI32(pEncoder, pInfo->dstVgId) < 0) return -1; + if (tEncodeSEpSet(pEncoder, &pInfo->epSet) < 0) return -1; if (tEncodeCStr(pEncoder, pInfo->expr) < 0) return -1; return 0; } @@ -2459,6 +2460,7 @@ int32_t tDeserializeSTableIndexInfo(SDecoder *pDecoder, STableIndexInfo *pInfo) if (tDecodeI64(pDecoder, &pInfo->sliding) < 0) return -1; if (tDecodeI64(pDecoder, &pInfo->dstTbUid) < 0) return -1; if (tDecodeI32(pDecoder, &pInfo->dstVgId) < 0) return -1; + if (tDecodeSEpSet(pDecoder, &pInfo->epSet) < 0) return -1; if (tDecodeCStrAlloc(pDecoder, &pInfo->expr) < 0) return -1; return 0; diff --git a/source/common/test/CMakeLists.txt b/source/common/test/CMakeLists.txt index 0535b08be7ddae7433fc42481a4197168f525bb4..eb79e79afa1568633ea0c34faa2b9350816bcfb5 100644 --- a/source/common/test/CMakeLists.txt +++ b/source/common/test/CMakeLists.txt @@ -5,7 +5,12 @@ MESSAGE(STATUS "build parser unit test") SET(CMAKE_CXX_STANDARD 11) AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST) -ADD_EXECUTABLE(commonTest ${SOURCE_LIST}) +ADD_EXECUTABLE(commonTest "") +TARGET_SOURCES( + commonTest + PRIVATE + "commonTests.cpp" +) TARGET_LINK_LIBRARIES( commonTest PUBLIC os util common gtest @@ -24,7 +29,12 @@ target_sources( PRIVATE "dataformatTest.cpp" ) -target_link_libraries(dataformatTest gtest gtest_main util) +target_link_libraries(dataformatTest gtest gtest_main util common) +target_include_directories( + dataformatTest + PUBLIC "${TD_SOURCE_DIR}/include/common" + PUBLIC "${TD_SOURCE_DIR}/include/util" +) # tmsg test # add_executable(tmsgTest "") diff --git a/source/common/test/dataformatTest.cpp b/source/common/test/dataformatTest.cpp index 3497014c22ad053ee184d8f1852686c956ee03c1..81e91da0d67e92b0fd1ca6bdca6fc9a9a2745386 100644 --- a/source/common/test/dataformatTest.cpp +++ b/source/common/test/dataformatTest.cpp @@ -1 +1,481 @@ -#include "gtest/gtest.h" \ No newline at end of file +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +#include + +#include +#include +#include +#include +#include + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wwrite-strings" +#pragma GCC diagnostic ignored "-Wunused-function" +#pragma GCC diagnostic ignored "-Wunused-variable" +#pragma GCC diagnostic ignored "-Wsign-compare" + +#define NONE_CSTR "no" +#define NULL_CSTR "nu" +#define NONE_LEN 2 +#define NULL_LEN 2 +const static int16_t MAX_COLS = 14; + +int main(int argc, char **argv) { + testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +} + +// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool +STSchema *genSTSchema(int16_t nCols) { + EXPECT_LE(nCols, MAX_COLS); + SSchema *pSchema = (SSchema *)taosMemoryCalloc(nCols, sizeof(SSchema)); + EXPECT_NE(pSchema, nullptr); + + for (int16_t i = 0; i < nCols; ++i) { + pSchema[i].colId = PRIMARYKEY_TIMESTAMP_COL_ID + i; + char colName[TSDB_COL_NAME_LEN] = {0}; + snprintf(colName, TSDB_COL_NAME_LEN, "c%" PRIi16, i); + strncpy(pSchema[i].name, colName, TSDB_COL_NAME_LEN); + + switch (i) { + case 0: { + pSchema[0].type = TSDB_DATA_TYPE_TIMESTAMP; + pSchema[0].bytes = TYPE_BYTES[pSchema[0].type]; + } break; + case 1: { + pSchema[1].type = TSDB_DATA_TYPE_INT; + pSchema[1].bytes = TYPE_BYTES[pSchema[1].type]; + ; + } break; + case 2: { + pSchema[2].type = TSDB_DATA_TYPE_BIGINT; + pSchema[2].bytes = TYPE_BYTES[pSchema[2].type]; + } break; + case 3: { + pSchema[3].type = TSDB_DATA_TYPE_FLOAT; + pSchema[3].bytes = TYPE_BYTES[pSchema[3].type]; + } break; + case 4: { + pSchema[4].type = TSDB_DATA_TYPE_DOUBLE; + pSchema[4].bytes = TYPE_BYTES[pSchema[4].type]; + } break; + case 5: { + pSchema[5].type = TSDB_DATA_TYPE_BINARY; + pSchema[5].bytes = 12; + } break; + case 6: { + pSchema[6].type = TSDB_DATA_TYPE_NCHAR; + pSchema[6].bytes = 42; + } break; + case 7: { + pSchema[7].type = TSDB_DATA_TYPE_TINYINT; + pSchema[7].bytes = TYPE_BYTES[pSchema[7].type]; + } break; + case 8: { + pSchema[8].type = TSDB_DATA_TYPE_SMALLINT; + pSchema[8].bytes = TYPE_BYTES[pSchema[8].type]; + } break; + case 9: { + pSchema[9].type = TSDB_DATA_TYPE_BOOL; + pSchema[9].bytes = TYPE_BYTES[pSchema[9].type]; + } break; + case 10: { + pSchema[10].type = TSDB_DATA_TYPE_UTINYINT; + pSchema[10].bytes = TYPE_BYTES[pSchema[10].type]; + } break; + case 11: { + pSchema[11].type = TSDB_DATA_TYPE_USMALLINT; + pSchema[11].bytes = TYPE_BYTES[pSchema[11].type]; + } break; + case 12: { + pSchema[12].type = TSDB_DATA_TYPE_UINT; + pSchema[12].bytes = TYPE_BYTES[pSchema[12].type]; + } break; + case 13: { + pSchema[13].type = TSDB_DATA_TYPE_UBIGINT; + pSchema[13].bytes = TYPE_BYTES[pSchema[13].type]; + } break; + + default: + ASSERT(0); + break; + } + } + + STSchema *pResult = NULL; + pResult = tdGetSTSChemaFromSSChema(&pSchema, nCols); + + taosMemoryFree(pSchema); + return pResult; +} + +// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool +static int32_t genTestData(const char **data, int16_t nCols, SArray **pArray) { + if (!(*pArray)) { + *pArray = taosArrayInit(nCols, sizeof(SColVal)); + if (!(*pArray)) return -1; + } + + for (int16_t i = 0; i < nCols; ++i) { + SColVal colVal = {0}; + colVal.cid = PRIMARYKEY_TIMESTAMP_COL_ID + i; + if (strncasecmp(data[i], NONE_CSTR, NONE_LEN) == 0) { + colVal.isNone = 1; + taosArrayPush(*pArray, &colVal); + continue; + } else if (strncasecmp(data[i], NULL_CSTR, NULL_LEN) == 0) { + colVal.isNull = 1; + taosArrayPush(*pArray, &colVal); + continue; + } + + switch (i) { + case 0: + sscanf(data[i], "%" PRIi64, &colVal.value.ts); + break; + case 1: { + sscanf(data[i], "%" PRIi32, &colVal.value.i32); + } break; + case 2: + sscanf(data[i], "%" PRIi64, &colVal.value.i64); + break; + case 3: + sscanf(data[i], "%f", &colVal.value.f); + break; + case 4: + sscanf(data[i], "%lf", &colVal.value.d); + break; + case 5: { + int16_t dataLen = strlen(data[i]) + 1; + colVal.value.nData = dataLen < 10 ? dataLen : 10; + colVal.value.pData = (uint8_t *)data[i]; + } break; + case 6: { + int16_t dataLen = strlen(data[i]) + 1; + colVal.value.nData = dataLen < 40 ? dataLen : 40; + colVal.value.pData = (uint8_t *)data[i]; // just for test, not real nchar + } break; + case 7: + case 9: { + int32_t d8; + sscanf(data[i], "%" PRId32, &d8); + colVal.value.i8 = (int8_t)d8; + } break; + case 8: { + int32_t d16; + sscanf(data[i], "%" PRId32, &d16); + colVal.value.i16 = (int16_t)d16; + } break; + case 10: { + uint32_t u8; + sscanf(data[i], "%" PRId32, &u8); + colVal.value.u8 = (uint8_t)u8; + } break; + case 11: { + uint32_t u16; + sscanf(data[i], "%" PRId32, &u16); + colVal.value.u16 = (uint16_t)u16; + } break; + case 12: { + sscanf(data[i], "%" PRIu32, &colVal.value.u32); + } break; + case 13: { + sscanf(data[i], "%" PRIu64, &colVal.value.u64); + } break; + default: + ASSERT(0); + } + taosArrayPush(*pArray, &colVal); + } + return 0; +} + +int32_t debugPrintSColVal(SColVal *cv, int8_t type) { + if (cv->isNone) { + printf("None "); + return 0; + } + if (cv->isNull) { + printf("Null "); + return 0; + } + switch (type) { + case TSDB_DATA_TYPE_BOOL: + printf("%s ", cv->value.i8 == 0 ? "false" : "true"); + break; + case TSDB_DATA_TYPE_TINYINT: + printf("%" PRIi8 " ", cv->value.i8); + break; + case TSDB_DATA_TYPE_SMALLINT: + printf("%" PRIi16 " ", cv->value.i16); + break; + case TSDB_DATA_TYPE_INT: + printf("%" PRIi32 " ", cv->value.i32); + break; + case TSDB_DATA_TYPE_BIGINT: + printf("%" PRIi64 " ", cv->value.i64); + break; + case TSDB_DATA_TYPE_FLOAT: + printf("%f ", cv->value.f); + break; + case TSDB_DATA_TYPE_DOUBLE: + printf("%lf ", cv->value.d); + break; + case TSDB_DATA_TYPE_VARCHAR: { + char tv[15] = {0}; + snprintf(tv, 15, "%s", cv->value.pData); + printf("%s ", tv); + } break; + case TSDB_DATA_TYPE_TIMESTAMP: + printf("%" PRIi64 " ", cv->value.i64); + break; + case TSDB_DATA_TYPE_NCHAR: { + char tv[15] = {0}; + snprintf(tv, 15, "%s", cv->value.pData); + printf("%s ", tv); + } break; + case TSDB_DATA_TYPE_UTINYINT: + printf("%" PRIu8 " ", cv->value.u8); + break; + case TSDB_DATA_TYPE_USMALLINT: + printf("%" PRIu16 " ", cv->value.u16); + break; + case TSDB_DATA_TYPE_UINT: + printf("%" PRIu32 " ", cv->value.u32); + break; + case TSDB_DATA_TYPE_UBIGINT: + printf("%" PRIu64 " ", cv->value.u64); + break; + case TSDB_DATA_TYPE_JSON: + printf("JSON "); + break; + case TSDB_DATA_TYPE_VARBINARY: + printf("VARBIN "); + break; + case TSDB_DATA_TYPE_DECIMAL: + printf("DECIMAL "); + break; + case TSDB_DATA_TYPE_BLOB: + printf("BLOB "); + break; + case TSDB_DATA_TYPE_MEDIUMBLOB: + printf("MedBLOB "); + break; + // case TSDB_DATA_TYPE_BINARY: + // printf("BINARY "); + // break; + case TSDB_DATA_TYPE_MAX: + printf("UNDEF "); + break; + default: + printf("UNDEF "); + break; + } + return 0; +} + +void debugPrintTSRow(STSRow2 *row, STSchema *pTSchema, const char *tags, int32_t ln) { + printf("%s:%d %s:v%d:%d ", tags, ln, (row->flags & 0xf0) ? "KV" : "TP", row->sver, row->nData); + for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) { + SColVal cv = {0}; + tTSRowGet(row, pTSchema, i, &cv); + debugPrintSColVal(&cv, pTSchema->columns[i].type); + } + printf("\n"); + fflush(stdout); +} + +static int32_t checkSColVal(const char *rawVal, SColVal *cv, int8_t type) { + ASSERT(rawVal); + + if (cv->isNone) { + EXPECT_STRCASEEQ(rawVal, NONE_CSTR); + return 0; + } + if (cv->isNull) { + EXPECT_STRCASEEQ(rawVal, NULL_CSTR); + return 0; + } + + SValue rawSVal = {0}; + switch (type) { + case TSDB_DATA_TYPE_BOOL: + case TSDB_DATA_TYPE_TINYINT: { + int32_t d8; + sscanf(rawVal, "%" PRId32, &d8); + EXPECT_EQ(cv->value.i8, (int8_t)d8); + } break; + case TSDB_DATA_TYPE_SMALLINT: { + int32_t d16; + sscanf(rawVal, "%" PRId32, &d16); + EXPECT_EQ(cv->value.i16, (int16_t)d16); + } break; + case TSDB_DATA_TYPE_INT: { + sscanf(rawVal, "%" PRId32, &rawSVal.i32); + EXPECT_EQ(cv->value.i32, rawSVal.i32); + } break; + case TSDB_DATA_TYPE_BIGINT: { + sscanf(rawVal, "%" PRIi64, &rawSVal.i64); + EXPECT_EQ(cv->value.i64, rawSVal.i64); + } break; + case TSDB_DATA_TYPE_FLOAT: { + sscanf(rawVal, "%f", &rawSVal.f); + EXPECT_FLOAT_EQ(cv->value.f, rawSVal.f); + } break; + case TSDB_DATA_TYPE_DOUBLE: { + sscanf(rawVal, "%lf", &rawSVal.d); + EXPECT_DOUBLE_EQ(cv->value.d, rawSVal.d); + } break; + case TSDB_DATA_TYPE_VARCHAR: { + EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData); + } break; + case TSDB_DATA_TYPE_TIMESTAMP: { + sscanf(rawVal, "%" PRIi64, &rawSVal.ts); + EXPECT_DOUBLE_EQ(cv->value.ts, rawSVal.ts); + } break; + case TSDB_DATA_TYPE_NCHAR: { + EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData); // informal nchar comparsion + } break; + case TSDB_DATA_TYPE_UTINYINT: { + uint32_t u8; + sscanf(rawVal, "%" PRIu32, &u8); + EXPECT_EQ(cv->value.u8, (uint8_t)u8); + } break; + case TSDB_DATA_TYPE_USMALLINT: { + uint32_t u16; + sscanf(rawVal, "%" PRIu32, &u16); + EXPECT_EQ(cv->value.u16, (uint16_t)u16); + } break; + case TSDB_DATA_TYPE_UINT: { + sscanf(rawVal, "%" PRIu32, &rawSVal.u32); + EXPECT_EQ(cv->value.u32, rawSVal.u32); + } break; + case TSDB_DATA_TYPE_UBIGINT: { + sscanf(rawVal, "%" PRIu64, &rawSVal.u64); + EXPECT_EQ(cv->value.u64, rawSVal.u64); + } break; + case TSDB_DATA_TYPE_JSON: + printf("JSON "); + break; + case TSDB_DATA_TYPE_VARBINARY: + printf("VARBIN "); + break; + case TSDB_DATA_TYPE_DECIMAL: + printf("DECIMAL "); + break; + case TSDB_DATA_TYPE_BLOB: + printf("BLOB "); + break; + case TSDB_DATA_TYPE_MEDIUMBLOB: + printf("MedBLOB "); + break; + // case TSDB_DATA_TYPE_BINARY: + // printf("BINARY "); + // break; + case TSDB_DATA_TYPE_MAX: + printf("UNDEF "); + break; + default: + printf("UNDEF "); + break; + } + return 0; +} + +static void checkTSRow(const char **data, STSRow2 *row, STSchema *pTSchema) { + for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) { + SColVal cv = {0}; + tTSRowGet(row, pTSchema, i, &cv); + checkSColVal(data[i], &cv, pTSchema->columns[i].type); + } +} + +TEST(testCase, AllNormTest) { + int16_t nCols = 1; + STSRow2 *row = nullptr; + SArray *pArray = taosArrayInit(nCols, sizeof(SColVal)); + EXPECT_NE(pArray, nullptr); + + STSchema *pTSchema = genSTSchema(nCols); + EXPECT_NE(pTSchema, nullptr); + + // ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, + // c9 bool + char *data[10] = {"1653694220000", "10", "20", "10.1", "10.1", "binary10", "nchar10", "10", "10", "1"}; + + genTestData((const char **)&data, nCols, &pArray); + + tTSRowNew(NULL, pArray, pTSchema, &row); + + debugPrintTSRow(row, pTSchema, __func__, __LINE__); + checkTSRow((const char **)&data, row, pTSchema); + + taosArrayDestroy(pArray); + taosMemoryFree(pTSchema); +} + +#if 1 +TEST(testCase, NoneTest) { + const static int nCols = 14; + const static int nRows = 20; + STSRow2 *row = nullptr; + SArray *pArray = taosArrayInit(nCols, sizeof(SColVal)); + EXPECT_NE(pArray, nullptr); + + STSchema *pTSchema = genSTSchema(nCols); + EXPECT_NE(pTSchema, nullptr); + + // ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, + // c9 bool c10 tinyint unsigned, c11 smallint unsigned, c12 int unsigned, c13 bigint unsigned + const char *data[nRows][nCols] = { + {"1653694220000", "no", "20", "10.1", "10.1", "binary10", "no", "10", "10", "nu", "10", "20", "30", "40"}, + {"1653694220001", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no"}, + {"1653694220002", "no", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "nu"}, + {"1653694220003", "nu", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "no"}, + {"1653694220004", "no", "20", "no", "no", "no", "nchar10", "no", "no", "no", "no", "no", "no", "no"}, + {"1653694220005", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"}, + {"1653694220006", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"}, + {"1653694220007", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"}, + {"1653694220008", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"}, + {"1653694220009", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"}, + {"1653694220010", "-1", "-1", "-1", "-1", "binary10", "nu", "-1", "0", "0", "0", "0", "0", "0"}, + {"1653694220011", "-2147483648", "nu", "nu", "nu", "biy10", "nu", "nu", "32767", "no", "nu", "nu", "nu", "100"}, + {"1653694220012", "2147483647", "nu", "nu", "nu", "ary10", "nu", "nu", "-32768", "no", "nu", "nu", "nu", "100"}, + {"1653694220013", "no", "-9223372036854775818", "nu", "nu", "b1", "nu", "nu", "10", "no", "nu", "nu", "nu", "nu"}, + {"1653694220014", "no", "nu", "nu", "nu", "b0", "nu", "nu", "10", "no", "nu", "nu", "nu", "9223372036854775808"}, + {"1653694220015", "no", "nu", "nu", "nu", "binary30", "char4", "nu", "10", "no", "nu", "nu", "nu", + "18446744073709551615"}, + {"1653694220016", "2147483647", "nu", "nu", "nu", "bin50", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"}, + {"1653694220017", "2147483646", "0", "0", "0", "binary10", "0", "0", "0", "0", "255", "0", "0", "0"}, + {"1653694220018", "no", "-9223372036854775808", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu", + "4294967295", "100"}, + {"1653694220019", "no", "9223372036854775807", "nu", "nu", "bin10", "nu", "nu", "10", "no", "254", "nu", "nu", + "no"}}; + + + for (int r = 0; r < nRows; ++r) { + genTestData((const char **)&data[r], nCols, &pArray); + tTSRowNew(NULL, pArray, pTSchema, &row); + debugPrintTSRow(row, pTSchema, __func__, __LINE__); // debug print + checkTSRow((const char **)&data[r], row, pTSchema); // check + taosMemoryFreeClear(row); + taosArrayClear(pArray); + } + + taosArrayDestroy(pArray); + taosMemoryFree(pTSchema); +} +#endif \ No newline at end of file diff --git a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h index adc18fdced4777fcfdc731993ce758e301c34ac9..7e5379b0f8661cd25eed2bd8b0ed6db5e6d5a99b 100644 --- a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h +++ b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h @@ -34,6 +34,7 @@ typedef struct SVnodeMgmt { SQWorkerPool fetchPool; SWWorkerPool syncPool; SWWorkerPool writePool; + SWWorkerPool applyPool; SWWorkerPool mergePool; SSingleWorker mgmtWorker; SSingleWorker monitorWorker; @@ -52,10 +53,9 @@ typedef struct { typedef struct { int32_t vgId; - int32_t refCount; int32_t vgVersion; + int32_t refCount; int8_t dropped; - int8_t accessState; char *path; SVnode *pImpl; STaosQueue *pWriteQ; diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c index 4c5a32536f13440e9abfaec0bf79eee6301aa761..90e4e30261b8699e6c0f680087aa753ac5569f6c 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c @@ -323,7 +323,6 @@ SArray *vmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_MON_VM_INFO, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_MON_VM_LOAD, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER; - // Requests handled by VNODE if (dmSetMgmtHandle(pArray, TDMT_VND_SUBMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_CONTINUE, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER; @@ -353,7 +352,9 @@ SArray *vmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c index 23927255bb8c920bfaf9de12e91bbffe4bd1de39..9e4e7713f2b1f9e63129f02279ed3742b1b55937 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c @@ -49,10 +49,9 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) { } pVnode->vgId = pCfg->vgId; - pVnode->refCount = 0; pVnode->vgVersion = pCfg->vgVersion; + pVnode->refCount = 0; pVnode->dropped = 0; - pVnode->accessState = TSDB_VN_ALL_ACCCESS; pVnode->path = tstrdup(pCfg->path); pVnode->pImpl = pImpl; @@ -96,7 +95,7 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { dDebug("vgId:%d, vnode is closed", pVnode->vgId); if (pVnode->dropped) { - dDebug("vgId:%d, vnode is destroyed for dropped:%d", pVnode->vgId, pVnode->dropped); + dInfo("vgId:%d, vnode is destroyed, dropped:%d", pVnode->vgId, pVnode->dropped); snprintf(path, TSDB_FILENAME_LEN, "vnode%svnode%d", TD_DIRSEP, pVnode->vgId); vnodeDestroy(path, pMgmt->pTfs); } diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c index 427b3d5c94e49883f7349c133d6d8bfc6111a59a..88831384d4e719f2a64ac321671888e4507a20bc 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c @@ -16,10 +16,8 @@ #define _DEFAULT_SOURCE #include "vmInt.h" -#include "sync.h" -#include "syncTools.h" - static inline void vmSendRsp(SRpcMsg *pMsg, int32_t code) { + if (pMsg->info.handle == NULL) return; SRpcMsg rsp = { .code = code, .pCont = pMsg->info.rsp, @@ -53,9 +51,9 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { } if (IsReq(pMsg)) { - if (code != 0 && terrno != 0) { - dError("msg:%p failed to process since %s", pMsg, terrstr()); - code = terrno; + if (code != 0) { + if (terrno != 0) code = terrno; + dError("msg:%p, failed to process since %s", pMsg, terrstr()); } vmSendRsp(pMsg, code); } @@ -97,110 +95,6 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { taosFreeQitem(pMsg); } -static void vmProcessWriteQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { - int32_t code = 0; - SRpcMsg *pMsg = NULL; - SVnodeObj *pVnode = pInfo->ahandle; - int64_t sync = vnodeGetSyncHandle(pVnode->pImpl); - SArray *pArray = taosArrayInit(numOfMsgs, sizeof(SRpcMsg **)); - - for (int32_t m = 0; m < numOfMsgs; m++) { - if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; - dTrace("vgId:%d, msg:%p get from vnode-write queue", pVnode->vgId, pMsg); - - if (taosArrayPush(pArray, &pMsg) == NULL) { - dError("vgId:%d, failed to push msg:%p to vnode-write array", pVnode->vgId, pMsg); - vmSendRsp(pMsg, TSDB_CODE_OUT_OF_MEMORY); - } - } - - for (int32_t m = 0; m < taosArrayGetSize(pArray); m++) { - pMsg = *(SRpcMsg **)taosArrayGet(pArray, m); - code = vnodePreprocessReq(pVnode->pImpl, pMsg); - - if (code == TSDB_CODE_ACTION_IN_PROGRESS) { - dTrace("vgId:%d, msg:%p in progress and no rsp", pVnode->vgId, pMsg); - continue; - } - - if (pMsg->msgType != TDMT_VND_ALTER_REPLICA) { - code = syncPropose(sync, pMsg, false); - } - - if (code == TAOS_SYNC_PROPOSE_SUCCESS) { - dTrace("vgId:%d, msg:%p is proposed and no rsp", pVnode->vgId, pMsg); - continue; - } else if (code == TAOS_SYNC_PROPOSE_NOT_LEADER) { - SEpSet newEpSet = {0}; - syncGetEpSet(sync, &newEpSet); - SEp *pEp = &newEpSet.eps[newEpSet.inUse]; - if (pEp->port == tsServerPort && strcmp(pEp->fqdn, tsLocalFqdn) == 0) { - newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps; - } - - dTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", pVnode->vgId, pMsg, - newEpSet.numOfEps, newEpSet.inUse); - for (int32_t i = 0; i < newEpSet.numOfEps; ++i) { - dTrace("vgId:%d, msg:%p ep:%s:%u", pVnode->vgId, pMsg, newEpSet.eps[i].fqdn, newEpSet.eps[i].port); - } - - SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info}; - tmsgSendRedirectRsp(&rsp, &newEpSet); - } else { - dError("vgId:%d, msg:%p failed to propose write since %s, code:0x%x", pVnode->vgId, pMsg, tstrerror(code), code); - vmSendRsp(pMsg, code); - } - } - - for (int32_t i = 0; i < numOfMsgs; i++) { - pMsg = *(SRpcMsg **)taosArrayGet(pArray, i); - dTrace("vgId:%d, msg:%p is freed", pVnode->vgId, pMsg); - rpcFreeCont(pMsg->pCont); - taosFreeQitem(pMsg); - } - - taosArrayDestroy(pArray); -} - -static void vmProcessApplyQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { - SVnodeObj *pVnode = pInfo->ahandle; - SRpcMsg *pMsg = NULL; - - for (int32_t i = 0; i < numOfMsgs; ++i) { - if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; - dTrace("vgId:%d, msg:%p get from vnode-apply queue", pVnode->vgId, pMsg); - - // init response rpc msg - SRpcMsg rsp = {0}; - - // get original rpc msg - assert(pMsg->msgType == TDMT_SYNC_APPLY_MSG); - SyncApplyMsg *pSyncApplyMsg = syncApplyMsgFromRpcMsg2(pMsg); - syncApplyMsgLog2("==vmProcessApplyQueue==", pSyncApplyMsg); - SRpcMsg originalRpcMsg; - syncApplyMsg2OriginalRpcMsg(pSyncApplyMsg, &originalRpcMsg); - - // apply data into tsdb - if (vnodeProcessWriteReq(pVnode->pImpl, &originalRpcMsg, pSyncApplyMsg->fsmMeta.index, &rsp) < 0) { - rsp.code = terrno; - dError("vgId:%d, msg:%p failed to apply since %s", pVnode->vgId, pMsg, terrstr()); - } - - syncApplyMsgDestroy(pSyncApplyMsg); - rpcFreeCont(originalRpcMsg.pCont); - - // if leader, send response - if (pMsg->info.handle != NULL) { - rsp.info = pMsg->info; - tmsgSendRsp(&rsp); - } - - dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, rsp.code); - rpcFreeCont(pMsg->pCont); - taosFreeQitem(pMsg); - } -} - static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { SVnodeObj *pVnode = pInfo->ahandle; SRpcMsg *pMsg = NULL; @@ -211,11 +105,9 @@ static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOf int32_t code = vnodeProcessSyncReq(pVnode->pImpl, pMsg, NULL); if (code != 0) { + if (terrno != 0) code = terrno; dError("vgId:%d, msg:%p failed to sync since %s", pVnode->vgId, pMsg, terrstr()); - if (pMsg->info.handle != NULL) { - if (terrno != 0) code = terrno; - vmSendRsp(pMsg, code); - } + vmSendRsp(pMsg, code); } dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, code); @@ -234,8 +126,8 @@ static void vmProcessMergeQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numO int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo); if (code != 0) { - dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr()); if (terrno != 0) code = terrno; + dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr()); vmSendRsp(pMsg, code); } @@ -254,7 +146,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId); if (pVnode == NULL) { - dError("vgId:%d, failed to put msg:%p into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(), + dError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(), TMSG_INFO(pMsg->msgType)); return terrno != 0 ? terrno : -1; } @@ -322,7 +214,7 @@ int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { if (pMsg == NULL) return -1; SMsgHead *pHead = pRpc->pCont; - dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pMsg->msgType)); + dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pRpc->msgType)); pHead->contLen = htonl(pHead->contLen); pHead->vgId = htonl(pHead->vgId); @@ -362,9 +254,9 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) { } int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { - pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessWriteQueue); + pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode->pImpl, (FItems)vnodeProposeMsg); pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue); - pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessApplyQueue); + pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyMsg); pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue); pVnode->pFetchQ = tQWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItem)vmProcessFetchQueue); pVnode->pMergeQ = tWWorkerAllocQueue(&pMgmt->mergePool, pVnode, (FItems)vmProcessMergeQueue); @@ -381,8 +273,8 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ); + tWWorkerFreeQueue(&pMgmt->applyPool, pVnode->pApplyQ); tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ); - tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pApplyQ); tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ); tQWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ); tWWorkerFreeQueue(&pMgmt->mergePool, pVnode->pMergeQ); @@ -413,6 +305,11 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) { pWPool->max = tsNumOfVnodeWriteThreads; if (tWWorkerInit(pWPool) != 0) return -1; + SWWorkerPool *pAPool = &pMgmt->applyPool; + pAPool->name = "vnode-apply"; + pAPool->max = tsNumOfVnodeWriteThreads; + if (tWWorkerInit(pAPool) != 0) return -1; + SWWorkerPool *pSPool = &pMgmt->syncPool; pSPool->name = "vnode-sync"; pSPool->max = tsNumOfVnodeSyncThreads; @@ -449,6 +346,7 @@ void vmStopWorker(SVnodeMgmt *pMgmt) { tSingleWorkerCleanup(&pMgmt->monitorWorker); tSingleWorkerCleanup(&pMgmt->mgmtWorker); tWWorkerCleanup(&pMgmt->writePool); + tWWorkerCleanup(&pMgmt->applyPool); tWWorkerCleanup(&pMgmt->syncPool); tQWorkerCleanup(&pMgmt->queryPool); tQWorkerCleanup(&pMgmt->fetchPool); diff --git a/source/dnode/mgmt/node_mgmt/src/dmTransport.c b/source/dnode/mgmt/node_mgmt/src/dmTransport.c index 0ad1c508b275c94bbf313abd641e659e5abb37c0..b66e5593702707a9506226b1cbc7d458b0d0c1f1 100644 --- a/source/dnode/mgmt/node_mgmt/src/dmTransport.c +++ b/source/dnode/mgmt/node_mgmt/src/dmTransport.c @@ -22,17 +22,17 @@ static void dmSendRsp(SRpcMsg *pMsg); static void dmBuildMnodeRedirectRsp(SDnode *pDnode, SRpcMsg *pMsg); static inline int32_t dmBuildNodeMsg(SRpcMsg *pMsg, SRpcMsg *pRpc) { - SRpcConnInfo connInfo = {0}; - if (IsReq(pRpc) && rpcGetConnInfo(pRpc->info.handle, &connInfo) != 0) { - terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; - dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle); - return -1; - } + SRpcConnInfo *pConnInfo = &(pRpc->info.connInfo); + // if (IsReq(pRpc)) { + // terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; + // dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle); + // return -1; + //} memcpy(pMsg, pRpc, sizeof(SRpcMsg)); - memcpy(pMsg->conn.user, connInfo.user, TSDB_USER_LEN); - pMsg->conn.clientIp = connInfo.clientIp; - pMsg->conn.clientPort = connInfo.clientPort; + memcpy(pMsg->conn.user, pConnInfo->user, TSDB_USER_LEN); + pMsg->conn.clientIp = pConnInfo->clientIp; + pMsg->conn.clientPort = pConnInfo->clientPort; return 0; } @@ -49,9 +49,9 @@ int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) { } static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) { - SDnodeTrans *pTrans = &pDnode->trans; + SDnodeTrans * pTrans = &pDnode->trans; int32_t code = -1; - SRpcMsg *pMsg = NULL; + SRpcMsg * pMsg = NULL; SMgmtWrapper *pWrapper = NULL; SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)]; @@ -167,11 +167,11 @@ int32_t dmInitMsgHandle(SDnode *pDnode) { for (EDndNodeType ntype = DNODE; ntype < NODE_END; ++ntype) { SMgmtWrapper *pWrapper = &pDnode->wrappers[ntype]; - SArray *pArray = (*pWrapper->func.getHandlesFp)(); + SArray * pArray = (*pWrapper->func.getHandlesFp)(); if (pArray == NULL) return -1; for (int32_t i = 0; i < taosArrayGetSize(pArray); ++i) { - SMgmtHandle *pMgmt = taosArrayGet(pArray, i); + SMgmtHandle * pMgmt = taosArrayGet(pArray, i); SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pMgmt->msgType)]; if (pMgmt->needCheckVgId) { pHandle->needCheckVgId = pMgmt->needCheckVgId; diff --git a/source/dnode/mnode/impl/src/mndConsumer.c b/source/dnode/mnode/impl/src/mndConsumer.c index 1f8bf0699322ffdaad5c479b3c8fec3451645527..7f3c0a9fa8b07e8d60251615caf35227206fae90 100644 --- a/source/dnode/mnode/impl/src/mndConsumer.c +++ b/source/dnode/mnode/impl/src/mndConsumer.c @@ -92,6 +92,15 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) { SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId); ASSERT(pConsumer); + + mInfo("receive consumer lost msg, consumer id %ld, status %s", pLostMsg->consumerId, + mndConsumerStatusName(pConsumer->status)); + + if (pConsumer->status != MQ_CONSUMER_STATUS__READY) { + mndReleaseConsumer(pMnode, pConsumer); + return -1; + } + SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup); pConsumerNew->updateType = CONSUMER_UPDATE__LOST; diff --git a/source/dnode/mnode/impl/src/mndShow.c b/source/dnode/mnode/impl/src/mndShow.c index 6b70825ed46bdfffb703d3c508971a22ec145b82..6e569a04ccdf3a5654fda65aebf9c8806c21049c 100644 --- a/source/dnode/mnode/impl/src/mndShow.c +++ b/source/dnode/mnode/impl/src/mndShow.c @@ -266,7 +266,7 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { } size = sizeof(SRetrieveMetaTableRsp) + sizeof(int32_t) + sizeof(SSysTableSchema) * pShow->pMeta->numOfColumns + - blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock); + blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock->info.numOfCols); SRetrieveMetaTableRsp *pRsp = rpcMallocCont(size); if (pRsp == NULL) { diff --git a/source/dnode/mnode/impl/src/mndSma.c b/source/dnode/mnode/impl/src/mndSma.c index 1d47f8fc7a1da9cfc5cd2e0ec4d90885bb5b0de1..cde36eac5802a7b10d464c4d478a3e5788dad45e 100644 --- a/source/dnode/mnode/impl/src/mndSma.c +++ b/source/dnode/mnode/impl/src/mndSma.c @@ -894,6 +894,15 @@ static int32_t mndGetTableSma(SMnode *pMnode, STableIndexReq *indexReq, STableIn info.sliding = pSma->sliding; info.dstTbUid = pSma->dstTbUid; info.dstVgId = pSma->dstVgId; + + SVgObj* pVg = mndAcquireVgroup(pMnode, pSma->dstVgId); + if (pVg == NULL) { + code = -1; + sdbRelease(pSdb, pSma); + return code; + } + info.epSet = mndGetVgroupEpset(pMnode, pVg); + info.expr = taosMemoryMalloc(pSma->exprLen + 1); if (info.expr == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; diff --git a/source/dnode/vnode/inc/vnode.h b/source/dnode/vnode/inc/vnode.h index 8c7a78a5afb7b7620a1dc2f64ae8cc70b042bff6..01e57d5eafe8a51ac60f7b10033ffa1df38c28f8 100644 --- a/source/dnode/vnode/inc/vnode.h +++ b/source/dnode/vnode/inc/vnode.h @@ -71,6 +71,9 @@ int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader); int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData); int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen); +void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs); +void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs); + // meta typedef struct SMeta SMeta; // todo: remove typedef struct SMetaReader SMetaReader; diff --git a/source/dnode/vnode/src/inc/vnd.h b/source/dnode/vnode/src/inc/vnd.h index 5f4f7e70daf089d22fa4e80978e787d10dc08c09..b0599b82ef3f7a5d62d43190ff4d09ebe428e001 100644 --- a/source/dnode/vnode/src/inc/vnd.h +++ b/source/dnode/vnode/src/inc/vnd.h @@ -84,7 +84,6 @@ int32_t vnodeAsyncCommit(SVnode* pVnode); int32_t vnodeSyncOpen(SVnode* pVnode, char* path); void vnodeSyncStart(SVnode* pVnode); void vnodeSyncClose(SVnode* pVnode); -int32_t vnodeSyncAlter(SVnode* pVnode, SRpcMsg* pMsg); #ifdef __cplusplus } diff --git a/source/dnode/vnode/src/inc/vnodeInt.h b/source/dnode/vnode/src/inc/vnodeInt.h index de2fc03f6977168caa38715da8d2cbfdd99e68ea..5532f202fcbe9be8cbc50de1cb24b9282eed2911 100644 --- a/source/dnode/vnode/src/inc/vnodeInt.h +++ b/source/dnode/vnode/src/inc/vnodeInt.h @@ -220,8 +220,10 @@ struct SVnode { SWal* pWal; STQ* pTq; SSink* pSink; - int64_t sync; tsem_t canCommit; + int64_t sync; + int32_t syncCount; + sem_t syncSem; SQHandle* pQuery; }; diff --git a/source/dnode/vnode/src/meta/metaTable.c b/source/dnode/vnode/src/meta/metaTable.c index 8d4762a656519f6c2c7e14a9580f9b71799b1e12..9a05f9e5a0c1fd7ca4bb1070deb78101faa8666f 100644 --- a/source/dnode/vnode/src/meta/metaTable.c +++ b/source/dnode/vnode/src/meta/metaTable.c @@ -99,7 +99,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const indexMultiTermAdd(terms, term); } } - tIndexJsonPut(pMeta->pTagIvtIdx, terms, tuid); + indexJsonPut(pMeta->pTagIvtIdx, terms, tuid); indexMultiTermDestroy(terms); #endif return 0; diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 181cddee47bab68470147f3d79da4db38bf9cc7d..a1b8d81d58f6abff1305b158bb48049752599b92 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -448,7 +448,7 @@ int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg) { } int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) { - SStreamDispatchRsp* pRsp = pMsg->pCont; + SStreamDispatchRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); int32_t taskId = pRsp->taskId; SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t)); streamProcessDispatchRsp(pTask, &pTq->pVnode->msgCb, pRsp); diff --git a/source/dnode/vnode/src/tq/tqExec.c b/source/dnode/vnode/src/tq/tqExec.c index 31acc60006076a9d4c6d143f69aeaf5172bcd9b5..7c75d88a83d5813ceb5c34eb6f1e0cb779335d5f 100644 --- a/source/dnode/vnode/src/tq/tqExec.c +++ b/source/dnode/vnode/src/tq/tqExec.c @@ -39,6 +39,9 @@ static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataBlkRsp* pRs static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, int32_t workerId, SMqDataBlkRsp* pRsp) { SSchemaWrapper* pSW = tCloneSSchemaWrapper(pExec->pExecReader[workerId]->pSchemaWrapper); + if (pSW == NULL) { + return -1; + } taosArrayPush(pRsp->blockSchema, &pSW); return 0; } diff --git a/source/dnode/vnode/src/vnd/vnodeOpen.c b/source/dnode/vnode/src/vnd/vnodeOpen.c index a85b8306165326bc07f643718e9b67201e668de6..dfc258b42b75bb3b5da4012a4f44ed0c7d337f05 100644 --- a/source/dnode/vnode/src/vnd/vnodeOpen.c +++ b/source/dnode/vnode/src/vnd/vnodeOpen.c @@ -81,7 +81,9 @@ SVnode *vnodeOpen(const char *path, STfs *pTfs, SMsgCb msgCb) { pVnode->state.applied = info.state.committed; pVnode->pTfs = pTfs; pVnode->msgCb = msgCb; + pVnode->syncCount = 0; + tsem_init(&pVnode->syncSem, 0, 0); tsem_init(&(pVnode->canCommit), 0, 1); // open buffer pool @@ -175,6 +177,7 @@ void vnodeClose(SVnode *pVnode) { vnodeCloseBufPool(pVnode); // destroy handle tsem_destroy(&(pVnode->canCommit)); + tsem_destroy(&pVnode->syncSem); taosMemoryFree(pVnode); } } diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index 7a9c9ef393a34da7317daf3f32cfef042288e446..1524d8fe6fc246578c9732a8caceed0d823e6070 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -91,9 +91,6 @@ int32_t vnodePreprocessReq(SVnode *pVnode, SRpcMsg *pMsg) { } } break; - case TDMT_VND_ALTER_REPLICA: { - code = vnodeSyncAlter(pVnode, pMsg); - } break; default: break; } @@ -107,7 +104,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp int32_t len; int32_t ret; - vTrace("vgId:%d, start to process write request %s, version %" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), + vTrace("vgId:%d, start to process write request %s, index:%" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), version); pVnode->state.applied = version; @@ -173,7 +170,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp break; } - vTrace("vgId:%d, process %s request success, version: %" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), version); + vTrace("vgId:%d, process %s request success, index:%" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), version); if (tqPushMsg(pVnode->pTq, pMsg->pCont, pMsg->contLen, pMsg->msgType, version) < 0) { vError("vgId:%d, failed to push msg to TQ since %s", TD_VID(pVnode), tstrerror(terrno)); diff --git a/source/dnode/vnode/src/vnd/vnodeSync.c b/source/dnode/vnode/src/vnd/vnodeSync.c index 8f98e9cec57b810adcbacb989b6179ac7fc9b2ad..203eecd8ab2402ab90ae0ea569124b8e4036aa17 100644 --- a/source/dnode/vnode/src/vnd/vnodeSync.c +++ b/source/dnode/vnode/src/vnd/vnodeSync.c @@ -16,41 +16,41 @@ #define _DEFAULT_SOURCE #include "vnd.h" -static int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg); -static int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg); -static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode); -static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta); -static void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta); -static void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta); -static int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot); +static inline bool vnodeIsMsgBlock(tmsg_t type) { + return (type == TDMT_VND_CREATE_TABLE) || (type == TDMT_VND_ALTER_CONFIRM) || (type == TDMT_VND_ALTER_REPLICA); +} -int32_t vnodeSyncOpen(SVnode *pVnode, char *path) { - SSyncInfo syncInfo = { - .vgId = pVnode->config.vgId, - .isStandBy = pVnode->config.standby, - .syncCfg = pVnode->config.syncCfg, - .pWal = pVnode->pWal, - .msgcb = NULL, - .FpSendMsg = vnodeSyncSendMsg, - .FpEqMsg = vnodeSyncEqMsg, - }; +static inline bool vnodeIsMsgWeak(tmsg_t type) { return false; } - snprintf(syncInfo.path, sizeof(syncInfo.path), "%s%ssync", path, TD_DIRSEP); - syncInfo.pFsm = vnodeSyncMakeFsm(pVnode); +static inline void vnodeAccumBlockMsg(SVnode *pVnode, tmsg_t type) { + if (!vnodeIsMsgBlock(type)) return; - pVnode->sync = syncOpen(&syncInfo); - if (pVnode->sync <= 0) { - vError("vgId:%d, failed to open sync since %s", pVnode->config.vgId, terrstr()); - return -1; - } + int32_t count = atomic_add_fetch_32(&pVnode->syncCount, 1); + vTrace("vgId:%d, accum block, count:%d type:%s", pVnode->config.vgId, count, TMSG_INFO(type)); +} - setPingTimerMS(pVnode->sync, 3000); - setElectTimerMS(pVnode->sync, 500); - setHeartbeatTimerMS(pVnode->sync, 100); - return 0; +static inline void vnodeWaitBlockMsg(SVnode *pVnode) { + int32_t count = atomic_load_32(&pVnode->syncCount); + if (count <= 0) return; + + vTrace("vgId:%d, wait block finish, count:%d", pVnode->config.vgId, count); + tsem_wait(&pVnode->syncSem); +} + +static inline void vnodePostBlockMsg(SVnode *pVnode, tmsg_t type) { + if (!vnodeIsMsgBlock(type)) return; + + int32_t count = atomic_load_32(&pVnode->syncCount); + if (count <= 0) return; + + count = atomic_sub_fetch_32(&pVnode->syncCount, 1); + vTrace("vgId:%d, post block, count:%d type:%s", pVnode->config.vgId, count, TMSG_INFO(type)); + if (count <= 0) { + tsem_post(&pVnode->syncSem); + } } -int32_t vnodeSyncAlter(SVnode *pVnode, SRpcMsg *pMsg) { +static int32_t vnodeProcessSyncReconfigReq(SVnode *pVnode, SRpcMsg *pMsg) { SAlterVnodeReq req = {0}; if (tDeserializeSAlterVnodeReq((char *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead), &req) != 0) { terrno = TSDB_CODE_INVALID_MSG; @@ -66,29 +66,98 @@ int32_t vnodeSyncAlter(SVnode *pVnode, SRpcMsg *pMsg) { vInfo("vgId:%d, replica:%d %s:%u", TD_VID(pVnode), r, pNode->nodeFqdn, pNode->nodePort); } - int32_t code = syncReconfig(pVnode->sync, &cfg); - if (code == TAOS_SYNC_PROPOSE_SUCCESS) { - // todo refactor - SRpcMsg rsp = {.info = pMsg->info, .code = 0}; - tmsgSendRsp(&rsp); - return TSDB_CODE_ACTION_IN_PROGRESS; + return syncReconfig(pVnode->sync, &cfg); +} + +void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { + SVnode *pVnode = pInfo->ahandle; + int32_t vgId = pVnode->config.vgId; + int32_t code = 0; + SRpcMsg *pMsg = NULL; + + for (int32_t m = 0; m < numOfMsgs; m++) { + if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; + vTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle); + + if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) { + code = vnodeProcessSyncReconfigReq(pVnode, pMsg); + } else { + code = vnodePreprocessReq(pVnode, pMsg); + if (code != 0) { + vError("vgId:%d, failed to pre-process msg:%p since %s", vgId, pMsg, terrstr()); + } else { + code = syncPropose(pVnode->sync, pMsg, vnodeIsMsgWeak(pMsg->msgType)); + } + } + + if (code == 0) { + vnodeAccumBlockMsg(pVnode, pMsg->msgType); + if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) { + // todo refactor + SRpcMsg rsp = {.code = code, .info = pMsg->info}; + tmsgSendRsp(&rsp); + } + } else if (code == TAOS_SYNC_PROPOSE_NOT_LEADER) { + SEpSet newEpSet = {0}; + syncGetEpSet(pVnode->sync, &newEpSet); + SEp *pEp = &newEpSet.eps[newEpSet.inUse]; + if (pEp->port == tsServerPort && strcmp(pEp->fqdn, tsLocalFqdn) == 0) { + newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps; + } + + vTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps, + newEpSet.inUse); + for (int32_t i = 0; i < newEpSet.numOfEps; ++i) { + vTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port); + } + + SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info}; + tmsgSendRedirectRsp(&rsp, &newEpSet); + } else { + if (terrno != 0) code = terrno; + vError("vgId:%d, msg:%p failed to propose since %s, code:0x%x", vgId, pMsg, tstrerror(code), code); + SRpcMsg rsp = {.code = code, .info = pMsg->info}; + tmsgSendRsp(&rsp); + } + + vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code); + rpcFreeCont(pMsg->pCont); + taosFreeQitem(pMsg); } - return code; + vnodeWaitBlockMsg(pVnode); } -void vnodeSyncStart(SVnode *pVnode) { - syncSetMsgCb(pVnode->sync, &pVnode->msgCb); - if (pVnode->config.standby) { - syncStartStandBy(pVnode->sync); - } else { - syncStart(pVnode->sync); +void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { + SVnode *pVnode = pInfo->ahandle; + int32_t vgId = pVnode->config.vgId; + int32_t code = 0; + SRpcMsg *pMsg = NULL; + + for (int32_t i = 0; i < numOfMsgs; ++i) { + if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; + vTrace("vgId:%d, msg:%p get from vnode-apply queue, handle:%p", vgId, pMsg, pMsg->info.handle); + + SRpcMsg rsp = {.code = pMsg->code, .info = pMsg->info}; + if (rsp.code == 0) { + if (vnodeProcessWriteReq(pVnode, pMsg, pMsg->conn.applyIndex, &rsp) < 0) { + rsp.code = terrno; + vError("vgId:%d, msg:%p failed to apply since %s", vgId, pMsg, terrstr()); + } + } + + vnodePostBlockMsg(pVnode, pMsg->msgType); + if (rsp.info.handle != NULL) { + tmsgSendRsp(&rsp); + } + + vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code); + rpcFreeCont(pMsg->pCont); + taosFreeQitem(pMsg); } } -void vnodeSyncClose(SVnode *pVnode) { syncStop(pVnode->sync); } - -int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { +static int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { int32_t code = tmsgPutToQueue(msgcb, SYNC_QUEUE, pMsg); if (code != 0) { rpcFreeCont(pMsg->pCont); @@ -97,7 +166,7 @@ int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { return code; } -int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) { +static int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) { int32_t code = tmsgSendReq(pEpSet, pMsg); if (code != 0) { rpcFreeCont(pMsg->pCont); @@ -106,53 +175,43 @@ int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) { return code; } -int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot) { +static int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot) { vnodeGetSnapshot(pFsm->data, pSnapshot); return 0; } -void vnodeSyncReconfig(struct SSyncFSM *pFsm, SSyncCfg newCfg, SReConfigCbMeta cbMeta) { +static void vnodeSyncReconfig(struct SSyncFSM *pFsm, SSyncCfg newCfg, SReConfigCbMeta cbMeta) { SVnode *pVnode = pFsm->data; vInfo("vgId:%d, sync reconfig is confirmed", TD_VID(pVnode)); // todo rpc response here + // build rpc msg + // put into apply queue } -void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { +static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { + SVnode *pVnode = pFsm->data; + SSnapshot snapshot = {0}; SyncIndex beginIndex = SYNC_INDEX_INVALID; + char logBuf[256] = {0}; + if (pFsm->FpGetSnapshot != NULL) { - SSnapshot snapshot = {0}; - pFsm->FpGetSnapshot(pFsm, &snapshot); + (*pFsm->FpGetSnapshot)(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } if (cbMeta.index > beginIndex) { - char logBuf[256] = {0}; snprintf( logBuf, sizeof(logBuf), "==callback== ==CommitCb== execute, pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s, beginIndex :%ld\n", pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), beginIndex); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); - SVnode *pVnode = pFsm->data; - SyncApplyMsg *pSyncApplyMsg = syncApplyMsgBuild2(pMsg, pVnode->config.vgId, &cbMeta); - SRpcMsg applyMsg; - syncApplyMsg2RpcMsg(pSyncApplyMsg, &applyMsg); - syncApplyMsgDestroy(pSyncApplyMsg); - - // recover handle for response - SRpcMsg saveRpcMsg; - int32_t ret = syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &saveRpcMsg); - if (ret == 1 && cbMeta.state == TAOS_SYNC_STATE_LEADER) { - applyMsg.info = saveRpcMsg.info; - } else { - applyMsg.info.handle = NULL; - applyMsg.info.ahandle = NULL; - } - - // put to applyQ - tmsgPutToQueue(&(pVnode->msgCb), APPLY_QUEUE, &applyMsg); - + SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen, .conn.applyIndex = cbMeta.index}; + rpcMsg.pCont = rpcMallocCont(rpcMsg.contLen); + memcpy(rpcMsg.pCont, pMsg->pCont, pMsg->contLen); + syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info); + tmsgPutToQueue(&pVnode->msgCb, APPLY_QUEUE, &rpcMsg); } else { char logBuf[256] = {0}; snprintf(logBuf, sizeof(logBuf), @@ -164,7 +223,7 @@ void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) } } -void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { +static void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { char logBuf[256] = {0}; snprintf(logBuf, sizeof(logBuf), "==callback== ==PreCommitCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n", pFsm, cbMeta.index, @@ -172,14 +231,14 @@ void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMet syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); } -void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { +static void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { char logBuf[256] = {0}; snprintf(logBuf, sizeof(logBuf), "==callback== ==RollBackCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n", pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state)); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); } -SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { +static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { SSyncFSM *pFsm = taosMemoryCalloc(1, sizeof(SSyncFSM)); pFsm->data = pVnode; pFsm->FpCommitCb = vnodeSyncCommitMsg; @@ -188,6 +247,42 @@ SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { pFsm->FpGetSnapshot = vnodeSyncGetSnapshot; pFsm->FpRestoreFinishCb = NULL; pFsm->FpReConfigCb = vnodeSyncReconfig; - return pFsm; -} \ No newline at end of file +} + +int32_t vnodeSyncOpen(SVnode *pVnode, char *path) { + SSyncInfo syncInfo = { + .vgId = pVnode->config.vgId, + .isStandBy = pVnode->config.standby, + .syncCfg = pVnode->config.syncCfg, + .pWal = pVnode->pWal, + .msgcb = NULL, + .FpSendMsg = vnodeSyncSendMsg, + .FpEqMsg = vnodeSyncEqMsg, + }; + + snprintf(syncInfo.path, sizeof(syncInfo.path), "%s%ssync", path, TD_DIRSEP); + syncInfo.pFsm = vnodeSyncMakeFsm(pVnode); + + pVnode->sync = syncOpen(&syncInfo); + if (pVnode->sync <= 0) { + vError("vgId:%d, failed to open sync since %s", pVnode->config.vgId, terrstr()); + return -1; + } + + setPingTimerMS(pVnode->sync, 3000); + setElectTimerMS(pVnode->sync, 500); + setHeartbeatTimerMS(pVnode->sync, 100); + return 0; +} + +void vnodeSyncStart(SVnode *pVnode) { + syncSetMsgCb(pVnode->sync, &pVnode->msgCb); + if (pVnode->config.standby) { + syncStartStandBy(pVnode->sync); + } else { + syncStart(pVnode->sync); + } +} + +void vnodeSyncClose(SVnode *pVnode) { syncStop(pVnode->sync); } diff --git a/source/libs/catalog/inc/catalogInt.h b/source/libs/catalog/inc/catalogInt.h index 260ed1977e30b53bf0f94db41936eed815e4c91f..3de9ba5f0ef190cb0230f944b9f7cc0962ebe99f 100644 --- a/source/libs/catalog/inc/catalogInt.h +++ b/source/libs/catalog/inc/catalogInt.h @@ -173,7 +173,6 @@ typedef struct SCtgJob { SArray* pTasks; int32_t taskDone; SMetaData jobRes; - int32_t rspCode; uint64_t queryId; SCatalog* pCtg; @@ -201,11 +200,12 @@ typedef struct SCtgMsgCtx { typedef struct SCtgTask { CTG_TASK_TYPE type; - int32_t taskId; - SCtgJob *pJob; - void* taskCtx; - SCtgMsgCtx msgCtx; - void* res; + int32_t taskId; + SCtgJob* pJob; + void* taskCtx; + SCtgMsgCtx msgCtx; + int32_t code; + void* res; } SCtgTask; typedef int32_t (*ctgLanchTaskFp)(SCtgTask*); diff --git a/source/libs/catalog/src/ctgAsync.c b/source/libs/catalog/src/ctgAsync.c index f43e5592446a61302352091961b5dd07aaacf71e..bc2c8dad7ae4f5cd6d35001a39f984e65541def7 100644 --- a/source/libs/catalog/src/ctgAsync.c +++ b/source/libs/catalog/src/ctgAsync.c @@ -437,13 +437,14 @@ _return: int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pTableMeta) { - pJob->jobRes.pTableMeta = taosArrayInit(pJob->tbMetaNum, POINTER_BYTES); + pJob->jobRes.pTableMeta = taosArrayInit(pJob->tbMetaNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pTableMeta) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pTableMeta, &pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pTableMeta, &res); return TSDB_CODE_SUCCESS; } @@ -451,14 +452,14 @@ int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { int32_t ctgDumpDbVgRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pDbVgroup) { - pJob->jobRes.pDbVgroup = taosArrayInit(pJob->dbVgNum, POINTER_BYTES); + pJob->jobRes.pDbVgroup = taosArrayInit(pJob->dbVgNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pDbVgroup) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pDbVgroup, &pTask->res); - pTask->res = NULL; + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pDbVgroup, &res); return TSDB_CODE_SUCCESS; } @@ -466,13 +467,14 @@ int32_t ctgDumpDbVgRes(SCtgTask* pTask) { int32_t ctgDumpTbHashRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pTableHash) { - pJob->jobRes.pTableHash = taosArrayInit(pJob->tbHashNum, sizeof(SVgroupInfo)); + pJob->jobRes.pTableHash = taosArrayInit(pJob->tbHashNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pTableHash) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pTableHash, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pTableHash, &res); return TSDB_CODE_SUCCESS; } @@ -480,21 +482,29 @@ int32_t ctgDumpTbHashRes(SCtgTask* pTask) { int32_t ctgDumpIndexRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pIndex) { - pJob->jobRes.pIndex = taosArrayInit(pJob->indexNum, sizeof(SIndexInfo)); + pJob->jobRes.pIndex = taosArrayInit(pJob->indexNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pIndex) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pIndex, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pIndex, &res); return TSDB_CODE_SUCCESS; } int32_t ctgDumpQnodeRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; + if (NULL == pJob->jobRes.pQnodeList) { + pJob->jobRes.pQnodeList = taosArrayInit(1, sizeof(SMetaRes)); + if (NULL == pJob->jobRes.pQnodeList) { + CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); + } + } - TSWAP(pJob->jobRes.pQnodeList, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pQnodeList, &res); return TSDB_CODE_SUCCESS; } @@ -502,13 +512,14 @@ int32_t ctgDumpQnodeRes(SCtgTask* pTask) { int32_t ctgDumpDbCfgRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pDbCfg) { - pJob->jobRes.pDbCfg = taosArrayInit(pJob->dbCfgNum, sizeof(SDbCfgInfo)); + pJob->jobRes.pDbCfg = taosArrayInit(pJob->dbCfgNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pDbCfg) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pDbCfg, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pDbCfg, &res); return TSDB_CODE_SUCCESS; } @@ -516,13 +527,14 @@ int32_t ctgDumpDbCfgRes(SCtgTask* pTask) { int32_t ctgDumpDbInfoRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pDbInfo) { - pJob->jobRes.pDbInfo = taosArrayInit(pJob->dbInfoNum, sizeof(SDbInfo)); + pJob->jobRes.pDbInfo = taosArrayInit(pJob->dbInfoNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pDbInfo) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pDbInfo, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pDbInfo, &res); return TSDB_CODE_SUCCESS; } @@ -530,13 +542,14 @@ int32_t ctgDumpDbInfoRes(SCtgTask* pTask) { int32_t ctgDumpUdfRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pUdfList) { - pJob->jobRes.pUdfList = taosArrayInit(pJob->udfNum, sizeof(SFuncInfo)); + pJob->jobRes.pUdfList = taosArrayInit(pJob->udfNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pUdfList) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pUdfList, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pUdfList, &res); return TSDB_CODE_SUCCESS; } @@ -544,13 +557,14 @@ int32_t ctgDumpUdfRes(SCtgTask* pTask) { int32_t ctgDumpUserRes(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pUser) { - pJob->jobRes.pUser = taosArrayInit(pJob->userNum, sizeof(bool)); + pJob->jobRes.pUser = taosArrayInit(pJob->userNum, sizeof(SMetaRes)); if (NULL == pJob->jobRes.pUser) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } - taosArrayPush(pJob->jobRes.pUser, pTask->res); + SMetaRes res = {.code = pTask->code, .pRes = pTask->res}; + taosArrayPush(pJob->jobRes.pUser, &res); return TSDB_CODE_SUCCESS; } @@ -559,20 +573,13 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) { SCtgJob* pJob = pTask->pJob; int32_t code = 0; - qDebug("QID:%" PRIx64 " task %d end with rsp %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); + qDebug("QID:0x%" PRIx64 " task %d end with rsp %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); - if (rspCode) { - int32_t lastCode = atomic_val_compare_exchange_32(&pJob->rspCode, 0, rspCode); - if (0 == lastCode) { - CTG_ERR_JRET(rspCode); - } - - return TSDB_CODE_SUCCESS; - } + pTask->code = rspCode; int32_t taskDone = atomic_add_fetch_32(&pJob->taskDone, 1); if (taskDone < taosArrayGetSize(pJob->pTasks)) { - qDebug("task done: %d, total: %d", taskDone, (int32_t)taosArrayGetSize(pJob->pTasks)); + qDebug("QID:0x%" PRIx64 " task done: %d, total: %d", pJob->queryId, taskDone, (int32_t)taosArrayGetSize(pJob->pTasks)); return TSDB_CODE_SUCCESS; } diff --git a/source/libs/catalog/src/ctgUtil.c b/source/libs/catalog/src/ctgUtil.c index 962edf9af14728c31fa1e0362be730889f77f6d4..b5160fe8de064e915fb4eceffdeb61a81f3b850f 100644 --- a/source/libs/catalog/src/ctgUtil.c +++ b/source/libs/catalog/src/ctgUtil.c @@ -47,11 +47,13 @@ char *ctgTaskTypeStr(CTG_TASK_TYPE type) { void ctgFreeSMetaData(SMetaData* pData) { taosArrayDestroy(pData->pTableMeta); pData->pTableMeta = NULL; - + +/* for (int32_t i = 0; i < taosArrayGetSize(pData->pDbVgroup); ++i) { SArray** pArray = taosArrayGet(pData->pDbVgroup, i); taosArrayDestroy(*pArray); } +*/ taosArrayDestroy(pData->pDbVgroup); pData->pDbVgroup = NULL; @@ -61,10 +63,12 @@ void ctgFreeSMetaData(SMetaData* pData) { taosArrayDestroy(pData->pUdfList); pData->pUdfList = NULL; +/* for (int32_t i = 0; i < taosArrayGetSize(pData->pDbCfg); ++i) { SDbCfgInfo* pInfo = taosArrayGet(pData->pDbCfg, i); taosArrayDestroy(pInfo->pRetensions); } +*/ taosArrayDestroy(pData->pDbCfg); pData->pDbCfg = NULL; @@ -320,8 +324,12 @@ void ctgFreeTask(SCtgTask* pTask) { break; } case CTG_TASK_GET_DB_CFG: { - taosMemoryFreeClear(pTask->taskCtx); - taosMemoryFreeClear(pTask->res); + taosMemoryFreeClear(pTask->taskCtx); + if (pTask->res) { + SDbCfgInfo* pInfo = (SDbCfgInfo*)pTask->res; + taosArrayDestroy(pInfo->pRetensions); + taosMemoryFreeClear(pTask->res); + } break; } case CTG_TASK_GET_DB_INFO: { diff --git a/source/libs/command/inc/commandInt.h b/source/libs/command/inc/commandInt.h index a3755a174ea87fd8766a29509df5eed229dabdf2..c8412afdc7a88cd057a76a123abb2723b5ac1185 100644 --- a/source/libs/command/inc/commandInt.h +++ b/source/libs/command/inc/commandInt.h @@ -115,9 +115,9 @@ typedef struct SExplainCtx { #define EXPLAIN_ROW_NEW(level, ...) \ do { \ if (isVerboseLine) { \ - tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s", (level) * 2 + 3, ""); \ + tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s", (level) * 3 + 3, ""); \ } else { \ - tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s%s", (level) * 2, "", "-> "); \ + tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s%s", (level) * 3, "", "-> "); \ } \ tlen += snprintf(tbuf + VARSTR_HEADER_SIZE + tlen, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE - tlen, __VA_ARGS__); \ } while (0) diff --git a/source/libs/command/src/command.c b/source/libs/command/src/command.c index 3034b4b02a293dd4cce91814bde367682db6ac8d..f330b7ce162f9c79189a66c0772fffbc96cb8dcc 100644 --- a/source/libs/command/src/command.c +++ b/source/libs/command/src/command.c @@ -28,91 +28,94 @@ static int32_t getSchemaBytes(const SSchema* pSchema) { } } -// todo : to convert data according to SSDatablock -static void buildRspData(const STableMeta* pMeta, char* pData) { - int32_t* payloadLen = (int32_t*) pData; - uint64_t* groupId = (uint64_t*)(pData + sizeof(int32_t)); +static SSDataBlock* buildDescResultDataBlock() { + SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); + pBlock->info.numOfCols = DESCRIBE_RESULT_COLS; + pBlock->info.hasVarCol = true; - int32_t* pColSizes = (int32_t*)(pData + sizeof(int32_t) + sizeof(uint64_t)); - pData = (char*) pColSizes + DESCRIBE_RESULT_COLS * sizeof(int32_t); + pBlock->pDataBlock = taosArrayInit(4, sizeof(SColumnInfoData)); - int32_t numOfRows = TABLE_TOTAL_COL_NUM(pMeta); + SColumnInfoData infoData = {0}; + infoData.info.type = TSDB_DATA_TYPE_VARCHAR; + infoData.info.bytes = DESCRIBE_RESULT_FIELD_LEN; - // Field - int32_t* pOffset = (int32_t*)pData; - pData += numOfRows * sizeof(int32_t); + taosArrayPush(pBlock->pDataBlock, &infoData); + + infoData.info.type = TSDB_DATA_TYPE_VARCHAR; + infoData.info.bytes = DESCRIBE_RESULT_TYPE_LEN; + taosArrayPush(pBlock->pDataBlock, &infoData); + + infoData.info.type = TSDB_DATA_TYPE_INT; + infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes;; + taosArrayPush(pBlock->pDataBlock, &infoData); + + infoData.info.type = TSDB_DATA_TYPE_VARCHAR; + infoData.info.bytes = DESCRIBE_RESULT_NOTE_LEN; + taosArrayPush(pBlock->pDataBlock, &infoData); + + return pBlock; +} + +static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, STableMeta* pMeta) { + blockDataEnsureCapacity(pBlock, numOfRows); + pBlock->info.rows = numOfRows; + + // field + SColumnInfoData* pCol1 = taosArrayGet(pBlock->pDataBlock, 0); + char buf[DESCRIBE_RESULT_FIELD_LEN] = {0}; for (int32_t i = 0; i < numOfRows; ++i) { - STR_TO_VARSTR(pData, pMeta->schema[i].name); - int16_t len = varDataTLen(pData); - pData += len; - *pOffset = pColSizes[0]; - pOffset += 1; - pColSizes[0] += len; + STR_TO_VARSTR(buf, pMeta->schema[i].name); + colDataAppend(pCol1, i, buf, false); } - + // Type - pOffset = (int32_t*)pData; - pData += numOfRows * sizeof(int32_t); + SColumnInfoData* pCol2 = taosArrayGet(pBlock->pDataBlock, 1); for (int32_t i = 0; i < numOfRows; ++i) { - STR_TO_VARSTR(pData, tDataTypes[pMeta->schema[i].type].name); - int16_t len = varDataTLen(pData); - pData += len; - *pOffset = pColSizes[1]; - pOffset += 1; - pColSizes[1] += len; + STR_TO_VARSTR(buf, tDataTypes[pMeta->schema[i].type].name); + colDataAppend(pCol2, i, buf, false); } // Length - pData += BitmapLen(numOfRows); + SColumnInfoData* pCol3 = taosArrayGet(pBlock->pDataBlock, 2); for (int32_t i = 0; i < numOfRows; ++i) { - *(int32_t*)pData = getSchemaBytes(pMeta->schema + i); - pData += sizeof(int32_t); + int32_t bytes = getSchemaBytes(pMeta->schema + i); + colDataAppend(pCol3, i, (const char*)&bytes, false); } - pColSizes[2] = sizeof(int32_t) * numOfRows; // Note - pOffset = (int32_t*)pData; - pData += numOfRows * sizeof(int32_t); + SColumnInfoData* pCol4 = taosArrayGet(pBlock->pDataBlock, 3); for (int32_t i = 0; i < numOfRows; ++i) { - STR_TO_VARSTR(pData, i >= pMeta->tableInfo.numOfColumns ? "TAG" : ""); - int16_t len = varDataTLen(pData); - pData += len; - *pOffset = pColSizes[3]; - pOffset += 1; - pColSizes[3] += len; - } - - for (int32_t i = 0; i < DESCRIBE_RESULT_COLS; ++i) { - pColSizes[i] = htonl(pColSizes[i]); + STR_TO_VARSTR(buf, i >= pMeta->tableInfo.numOfColumns ? "TAG" : ""); + colDataAppend(pCol4, i, buf, false); } - - - *payloadLen = (int32_t)(pData - (char*)payloadLen); -} - -static int32_t calcRspSize(const STableMeta* pMeta) { - int32_t numOfRows = TABLE_TOTAL_COL_NUM(pMeta); - return sizeof(SRetrieveTableRsp) + - (numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_FIELD_LEN) + - (numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_TYPE_LEN) + - (BitmapLen(numOfRows) + numOfRows * sizeof(int32_t)) + - (numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_NOTE_LEN) + - sizeof(int32_t) + sizeof(uint64_t); } static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) { - SDescribeStmt* pDesc = (SDescribeStmt*)pStmt; - *pRsp = taosMemoryCalloc(1, calcRspSize(pDesc->pMeta)); + SDescribeStmt* pDesc = (SDescribeStmt*) pStmt; + int32_t numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta); + + SSDataBlock* pBlock = buildDescResultDataBlock(); + setDescResultIntoDataBlock(pBlock, numOfRows, pDesc->pMeta); + + size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock); + *pRsp = taosMemoryCalloc(1, rspSize); if (NULL == *pRsp) { return TSDB_CODE_OUT_OF_MEMORY; } + (*pRsp)->useconds = 0; (*pRsp)->completed = 1; (*pRsp)->precision = 0; (*pRsp)->compressed = 0; (*pRsp)->compLen = 0; - (*pRsp)->numOfRows = htonl(TABLE_TOTAL_COL_NUM(pDesc->pMeta)); - buildRspData(pDesc->pMeta, (*pRsp)->data); + (*pRsp)->numOfRows = htonl(numOfRows); + (*pRsp)->numOfCols = htonl(DESCRIBE_RESULT_COLS); + + int32_t len = 0; + blockCompressEncode(pBlock, (*pRsp)->data, &len, DESCRIBE_RESULT_COLS, false); + ASSERT(len == rspSize - sizeof(SRetrieveTableRsp)); + + blockDataDestroy(pBlock); return TSDB_CODE_SUCCESS; } diff --git a/source/libs/command/src/explain.c b/source/libs/command/src/explain.c index 347fb326bb111f93f9aa389257177e68ddf2cded..dc9215b52797ad46c5451611b8325833fa6aefdd 100644 --- a/source/libs/command/src/explain.c +++ b/source/libs/command/src/explain.c @@ -13,6 +13,7 @@ * along with this program. If not, see . */ +#include "tdatablock.h" #include "commandInt.h" #include "plannodes.h" #include "query.h" @@ -657,7 +658,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i if (EXPLAIN_MODE_ANALYZE == ctx->mode) { // sort key - EXPLAIN_ROW_NEW(level, "Sort Key: "); + EXPLAIN_ROW_NEW(level + 1, "Sort Key: "); if (pResNode->pExecInfo) { for (int32_t i = 0; i < LIST_LENGTH(pSortNode->pSortKeys); ++i) { SOrderByExprNode *ptn = nodesListGetNode(pSortNode->pSortKeys, i); @@ -669,7 +670,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level)); // sort method - EXPLAIN_ROW_NEW(level, "Sort Method: "); + EXPLAIN_ROW_NEW(level + 1, "Sort Method: "); int32_t nodeNum = taosArrayGetSize(pResNode->pExecInfo); SExplainExecInfo *execInfo = taosArrayGet(pResNode->pExecInfo, 0); @@ -965,9 +966,32 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) { QRY_ERR_RET(TSDB_CODE_QRY_APP_ERROR); } - int32_t colNum = 1; - int32_t rspSize = sizeof(SRetrieveTableRsp) + sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t) * colNum + - sizeof(int32_t) * rowNum + pCtx->dataSize; + SSDataBlock *pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); + SColumnInfoData infoData = {0}; + infoData.info.type = TSDB_DATA_TYPE_VARCHAR; + infoData.info.bytes = TSDB_EXPLAIN_RESULT_ROW_SIZE; + + pBlock->pDataBlock = taosArrayInit(1, sizeof(SColumnInfoData)); + taosArrayPush(pBlock->pDataBlock, &infoData); + + SColumnInfoData* pInfoData = taosArrayGet(pBlock->pDataBlock, 0); + pInfoData->hasNull = false; + colInfoDataEnsureCapacity(pInfoData, 0, rowNum); + + char buf[1024] = {0}; + for (int32_t i = 0; i < rowNum; ++i) { + SQueryExplainRowInfo *row = taosArrayGet(pCtx->rows, i); + varDataCopy(buf, row->buf); + ASSERT(varDataTLen(row->buf) == row->len); + colDataAppend(pInfoData, i, buf, false); + } + + pBlock->info.numOfCols = 1; + pBlock->info.rows = rowNum; + pBlock->info.hasVarCol = true; + + int32_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock); + SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, rspSize); if (NULL == rsp) { qError("malloc SRetrieveTableRsp failed, size:%d", rspSize); @@ -977,34 +1001,13 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) { rsp->completed = 1; rsp->numOfRows = htonl(rowNum); - // payload length - *(int32_t *)rsp->data = - sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t) * colNum + sizeof(int32_t) * rowNum + pCtx->dataSize; + int32_t len = 0; + blockCompressEncode(pBlock, rsp->data, &len, pBlock->info.numOfCols, 0); + ASSERT(len == rspSize - sizeof(SRetrieveTableRsp)); - // group id - *(uint64_t *)(rsp->data + sizeof(int32_t)) = 0; - - // column length - int32_t *colLength = (int32_t *)(rsp->data + sizeof(int32_t) + sizeof(uint64_t)); - - // varchar column offset segment - int32_t *offset = (int32_t *)((char *)colLength + sizeof(int32_t)); - - // varchar data real payload - char *data = (char *)(offset + rowNum); - - char *start = data; - for (int32_t i = 0; i < rowNum; ++i) { - SQueryExplainRowInfo *row = taosArrayGet(pCtx->rows, i); - offset[i] = data - start; - - varDataCopy(data, row->buf); - ASSERT(varDataTLen(row->buf) == row->len); - data += row->len; - } + rsp->compLen = htonl(len); - *colLength = htonl(data - start); - rsp->compLen = htonl(rspSize); + blockDataDestroy(pBlock); *pRsp = rsp; return TSDB_CODE_SUCCESS; diff --git a/source/libs/executor/inc/executorimpl.h b/source/libs/executor/inc/executorimpl.h index 4f02c559b1cc9adee4f54deb352517cd686bac4a..68d71a0c24349b7b85ec9b86c1469ae07989dc56 100644 --- a/source/libs/executor/inc/executorimpl.h +++ b/source/libs/executor/inc/executorimpl.h @@ -688,6 +688,10 @@ typedef struct SSortedMergeOperatorInfo { int32_t numOfResPerPage; char** groupVal; SArray *groupInfo; + + bool hasGroupId; + uint64_t groupId; + STupleHandle* prefetchedTuple; } SSortedMergeOperatorInfo; typedef struct SSortOperatorInfo { @@ -700,6 +704,10 @@ typedef struct SSortOperatorInfo { int64_t startTs; // sort start time uint64_t sortElapsed; // sort elapsed time, time to flush to disk not included. + + STupleHandle *prefetchedTuple; + bool hasGroupId; + uint64_t groupId; } SSortOperatorInfo; typedef struct STagFilterOperatorInfo { @@ -759,7 +767,7 @@ void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode); SColumn extractColumnFromColumnNode(SColumnNode* pColNode); -SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo); +SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo, SSortOperatorInfo* pInfo); SSDataBlock* loadNextDataBlock(void* param); void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowCellInfoOffset); diff --git a/source/libs/executor/src/dataDispatcher.c b/source/libs/executor/src/dataDispatcher.c index eaa366b7e75ea75d9cd32f1f99b71685feb6baa5..b7c7102143d5d9524a692ad127f9e7a7e91808d3 100644 --- a/source/libs/executor/src/dataDispatcher.c +++ b/source/libs/executor/src/dataDispatcher.c @@ -68,10 +68,10 @@ static bool needCompress(const SSDataBlock* pData, int32_t numOfCols) { } // data format: -// +----------------+--------------+----------+--------------------------------------+-------------+-----------+-------------+-----------+ -// |SDataCacheEntry | total length | group id | column#1 length, column#2 length ... | col1 bitmap | col1 data | col2 bitmap | col2 data | .... -// | | (4 bytes) |(8 bytes) | sizeof(int32_t) * numOfCols | actual size | | actual size | | -// +----------------+--------------+----------+--------------------------------------+-------------+-----------+-------------+-----------+ +// +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+ +// |SDataCacheEntry | total length | group id | col1_schema | col2_schema | col3_schema ...| column#1 length, column#2 length ... | col1 bitmap | col1 data | col2 bitmap | col2 data | .... +// | | (4 bytes) |(8 bytes) |(sizeof(int16_t)+sizeof(int32_t))*numOfCols | sizeof(int32_t) * numOfCols | actual size | | actual size | | +// +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+ // The length of bitmap is decided by number of rows of this data block, and the length of each column data is // recorded in the first segment, next to the struct header static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) { diff --git a/source/libs/executor/src/executorimpl.c b/source/libs/executor/src/executorimpl.c index 3d7faa1c8331016fe7e3a209e8b13ca015f461e7..b4579b90b28e4eecb3dde94d6a179f5dd3da2b01 100644 --- a/source/libs/executor/src/executorimpl.c +++ b/source/libs/executor/src/executorimpl.c @@ -2571,37 +2571,7 @@ int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadIn } } - blockDataEnsureCapacity(pBlock, numOfRows); - - int32_t dataLen = *(int32_t*)pStart; - uint64_t groupId = *(uint64_t*)(pStart + sizeof(int32_t)); - pStart += sizeof(int32_t) + sizeof(uint64_t); - - int32_t* colLen = (int32_t*)(pStart); - pStart += sizeof(int32_t) * numOfCols; - - for (int32_t i = 0; i < numOfCols; ++i) { - colLen[i] = htonl(colLen[i]); - ASSERT(colLen[i] >= 0); - - SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i); - if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) { - pColInfoData->varmeta.length = colLen[i]; - pColInfoData->varmeta.allocLen = colLen[i]; - - memcpy(pColInfoData->varmeta.offset, pStart, sizeof(int32_t) * numOfRows); - pStart += sizeof(int32_t) * numOfRows; - - pColInfoData->pData = taosMemoryMalloc(colLen[i]); - } else { - memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows)); - pStart += BitmapLen(numOfRows); - } - - memcpy(pColInfoData->pData, pStart, colLen[i]); - pStart += colLen[i]; - } - + blockCompressDecode(pBlock, numOfCols, numOfRows, pStart); // data from mnode relocateColumnData(pRes, pColList, pBlock->pDataBlock); taosArrayDestroy(pBlock->pDataBlock); @@ -3133,6 +3103,68 @@ static SSDataBlock* doMerge(SOperatorInfo* pOperator) { return (pInfo->binfo.pRes->info.rows > 0) ? pInfo->binfo.pRes : NULL; } +SSDataBlock* getSortedMergeBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo, + SSortedMergeOperatorInfo *pInfo) { + blockDataCleanup(pDataBlock); + + SSDataBlock* p = tsortGetSortedDataBlock(pHandle); + if (p == NULL) { + return NULL; + } + + blockDataEnsureCapacity(p, capacity); + + while (1) { + STupleHandle* pTupleHandle = NULL; + if (pInfo->prefetchedTuple == NULL) { + pTupleHandle = tsortNextTuple(pHandle); + } else { + pTupleHandle = pInfo->prefetchedTuple; + pInfo->groupId = tsortGetGroupId(pTupleHandle); + pInfo->prefetchedTuple = NULL; + } + + if (pTupleHandle == NULL) { + break; + } + + uint64_t tupleGroupId = tsortGetGroupId(pTupleHandle); + if (!pInfo->hasGroupId) { + pInfo->groupId = tupleGroupId; + pInfo->hasGroupId = true; + appendOneRowToDataBlock(p, pTupleHandle); + } else if (pInfo->groupId == tupleGroupId) { + appendOneRowToDataBlock(p, pTupleHandle); + } else { + pInfo->prefetchedTuple = pTupleHandle; + break; + } + + if (p->info.rows >= capacity) { + break; + } + } + + if (p->info.rows > 0) { + int32_t numOfCols = taosArrayGetSize(pColMatchInfo); + for (int32_t i = 0; i < numOfCols; ++i) { + SColMatchInfo* pmInfo = taosArrayGet(pColMatchInfo, i); + ASSERT(pmInfo->matchType == COL_MATCH_FROM_SLOT_ID); + + SColumnInfoData* pSrc = taosArrayGet(p->pDataBlock, pmInfo->srcSlotId); + SColumnInfoData* pDst = taosArrayGet(pDataBlock->pDataBlock, pmInfo->targetSlotId); + colDataAssign(pDst, pSrc, p->info.rows); + } + + pDataBlock->info.rows = p->info.rows; + pDataBlock->info.capacity = p->info.rows; + pDataBlock->info.groupId = pInfo->groupId; + } + + blockDataDestroy(p); + return (pDataBlock->info.rows > 0) ? pDataBlock : NULL; +} + static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) { if (pOperator->status == OP_EXEC_DONE) { return NULL; @@ -3141,7 +3173,7 @@ static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SSortedMergeOperatorInfo* pInfo = pOperator->info; if (pOperator->status == OP_RES_TO_RETURN) { - return getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, NULL); + return getSortedMergeBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, NULL, pInfo); } int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize; diff --git a/source/libs/executor/src/scanoperator.c b/source/libs/executor/src/scanoperator.c index cc76cc29e4afdbf40c9f2293c6066b5cdb63af2b..1e2a26386bbfee90c5285d3c86cac441d5748ff6 100644 --- a/source/libs/executor/src/scanoperator.c +++ b/source/libs/executor/src/scanoperator.c @@ -737,8 +737,8 @@ static bool isStateWindow(SStreamBlockScanInfo* pInfo) { static bool prepareDataScan(SStreamBlockScanInfo* pInfo) { SSDataBlock* pSDB = pInfo->pUpdateRes; STimeWindow win = { - .skey = INT64_MIN, - .ekey = INT64_MAX, + .skey = INT64_MIN, + .ekey = INT64_MAX, }; bool needRead = false; if (!isStateWindow(pInfo) && pInfo->updateResIndex < pSDB->info.rows) { @@ -846,7 +846,7 @@ static SSDataBlock* doStreamBlockScan(SOperatorInfo* pOperator) { size_t total = taosArrayGetSize(pInfo->pBlockLists); if (pInfo->blockType == STREAM_DATA_TYPE_SSDATA_BLOCK) { if (pInfo->validBlockIndex >= total) { - doClearBufferedBlocks(pInfo); + /*doClearBufferedBlocks(pInfo);*/ pOperator->status = OP_EXEC_DONE; return NULL; } diff --git a/source/libs/executor/src/sortoperator.c b/source/libs/executor/src/sortoperator.c index 95f9514b0778d26b0933628d16aa7da891abee23..dcaa95e28a3d438a6ca7638bc6d925b59dd0aeb8 100644 --- a/source/libs/executor/src/sortoperator.c +++ b/source/libs/executor/src/sortoperator.c @@ -42,6 +42,8 @@ SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pR pInfo->pSortInfo = pSortInfo; pInfo->pColMatchInfo = pColMatchColInfo; + pInfo->hasGroupId = false; + pInfo->prefetchedTuple = NULL; pOperator->name = "SortOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT; pOperator->blocking = true; @@ -81,8 +83,8 @@ void appendOneRowToDataBlock(SSDataBlock* pBlock, STupleHandle* pTupleHandle) { pBlock->info.rows += 1; } -SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, - SArray* pColMatchInfo) { +SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo, + SSortOperatorInfo* pInfo) { blockDataCleanup(pDataBlock); SSDataBlock* p = tsortGetSortedDataBlock(pHandle); @@ -93,14 +95,33 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i blockDataEnsureCapacity(p, capacity); while (1) { - STupleHandle* pTupleHandle = tsortNextTuple(pHandle); + STupleHandle* pTupleHandle = NULL; + if (pInfo->prefetchedTuple == NULL) { + pTupleHandle = tsortNextTuple(pHandle); + } else { + pTupleHandle = pInfo->prefetchedTuple; + pInfo->groupId = tsortGetGroupId(pTupleHandle); + pInfo->prefetchedTuple = NULL; + } + if (pTupleHandle == NULL) { break; } - appendOneRowToDataBlock(p, pTupleHandle); + uint64_t tupleGroupId = tsortGetGroupId(pTupleHandle); + if (!pInfo->hasGroupId) { + pInfo->groupId = tupleGroupId; + pInfo->hasGroupId = true; + appendOneRowToDataBlock(p, pTupleHandle); + } else if (pInfo->groupId == tupleGroupId) { + appendOneRowToDataBlock(p, pTupleHandle); + } else { + pInfo->prefetchedTuple = pTupleHandle; + break; + } + if (p->info.rows >= capacity) { - return pDataBlock; + break; } } @@ -117,6 +138,7 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i pDataBlock->info.rows = p->info.rows; pDataBlock->info.capacity = p->info.rows; + pDataBlock->info.groupId = pInfo->groupId; } blockDataDestroy(p); @@ -188,8 +210,8 @@ SSDataBlock* doSort(SOperatorInfo* pOperator) { longjmp(pTaskInfo->env, code); } - SSDataBlock* pBlock = - getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, pInfo->pColMatchInfo); + SSDataBlock* pBlock = getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, + pInfo->pColMatchInfo, pInfo); if (pBlock != NULL) { pOperator->resultInfo.totalRows += pBlock->info.rows; @@ -230,11 +252,11 @@ typedef struct SMultiwaySortMergeOperatorInfo { SArray* pColMatchInfo; // for index map from table scan output SSDataBlock* pInputBlock; - int64_t startTs; // sort start time + int64_t startTs; // sort start time - bool hasGroupId; - uint64_t groupId; - STupleHandle *prefetchedTuple; + bool hasGroupId; + uint64_t groupId; + STupleHandle* prefetchedTuple; } SMultiwaySortMergeOperatorInfo; int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) { @@ -274,7 +296,7 @@ int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) { } SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, - SArray* pColMatchInfo, SMultiwaySortMergeOperatorInfo* pInfo) { + SArray* pColMatchInfo, SMultiwaySortMergeOperatorInfo* pInfo) { blockDataCleanup(pDataBlock); SSDataBlock* p = tsortGetSortedDataBlock(pHandle); @@ -285,12 +307,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData blockDataEnsureCapacity(p, capacity); while (1) { - STupleHandle* pTupleHandle = NULL; if (pInfo->prefetchedTuple == NULL) { pTupleHandle = tsortNextTuple(pHandle); } else { pTupleHandle = pInfo->prefetchedTuple; + pInfo->groupId = tsortGetGroupId(pTupleHandle); pInfo->prefetchedTuple = NULL; } @@ -307,14 +329,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData appendOneRowToDataBlock(p, pTupleHandle); } else { pInfo->prefetchedTuple = pTupleHandle; - pInfo->groupId = tupleGroupId; break; } if (p->info.rows >= capacity) { break; } - } if (p->info.rows > 0) { @@ -330,13 +350,13 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData pDataBlock->info.rows = p->info.rows; pDataBlock->info.capacity = p->info.rows; + pDataBlock->info.groupId = pInfo->groupId; } blockDataDestroy(p); return (pDataBlock->info.rows > 0) ? pDataBlock : NULL; } - SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { if (pOperator->status == OP_EXEC_DONE) { return NULL; @@ -350,12 +370,8 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { longjmp(pTaskInfo->env, code); } - SSDataBlock* pBlock = - getMultiwaySortedBlockData(pInfo->pSortHandle, - pInfo->binfo.pRes, - pOperator->resultInfo.capacity, - pInfo->pColMatchInfo, - pInfo); + SSDataBlock* pBlock = getMultiwaySortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, + pOperator->resultInfo.capacity, pInfo->pColMatchInfo, pInfo); if (pBlock != NULL) { pOperator->resultInfo.totalRows += pBlock->info.rows; @@ -366,7 +382,7 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { } void destroyMultiwaySortMergeOperatorInfo(void* param, int32_t numOfOutput) { - SMultiwaySortMergeOperatorInfo * pInfo = (SMultiwaySortMergeOperatorInfo*)param; + SMultiwaySortMergeOperatorInfo* pInfo = (SMultiwaySortMergeOperatorInfo*)param; pInfo->binfo.pRes = blockDataDestroy(pInfo->binfo.pRes); pInfo->pInputBlock = blockDataDestroy(pInfo->pInputBlock); @@ -386,9 +402,9 @@ int32_t getMultiwaySortMergeExplainExecInfo(SOperatorInfo* pOptr, void** pOptrEx return TSDB_CODE_SUCCESS; } -SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, int32_t numStreams, SSDataBlock* pInputBlock, - SSDataBlock* pResBlock, SArray* pSortInfo, SArray* pColMatchColInfo, - SExecTaskInfo* pTaskInfo) { +SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, int32_t numStreams, + SSDataBlock* pInputBlock, SSDataBlock* pResBlock, SArray* pSortInfo, + SArray* pColMatchColInfo, SExecTaskInfo* pTaskInfo) { SMultiwaySortMergeOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SMultiwaySortMergeOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); int32_t rowSize = pResBlock->info.rowSize; @@ -412,7 +428,8 @@ SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, pInfo->bufPageSize = rowSize < 1024 ? 1024 : rowSize * 2; pInfo->sortBufSize = pInfo->bufPageSize * 16; - + pInfo->hasGroupId = false; + pInfo->prefetchedTuple = NULL; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(doOpenMultiwaySortMergeOperator, doMultiwaySortMerge, NULL, NULL, diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index d5e7d5c515853e74c73386c46cccbf3973cd1b37..b309478556445acdcdda259739cf65b08ab45ea8 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -1227,7 +1227,10 @@ void doClearWindow(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, char* pData, int SET_RES_WINDOW_KEY(pSup->keyBuf, pData, bytes, groupId); SResultRowPosition* p1 = (SResultRowPosition*)taosHashGet(pSup->pResultRowHashTable, pSup->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes)); - ASSERT(p1); + if (!p1) { + // window has been closed + return; + } doClearWindowImpl(p1, pSup->pResultBuf, pBinfo, numOfOutput); } @@ -2202,12 +2205,12 @@ void destroyStreamSessionAggOperatorInfo(void* param, int32_t numOfOutput) { } } -int32_t initBiasicInfo(SOptrBasicInfo* pBasicInfo, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock, - SDiskbasedBuf* pResultBuf) { +int32_t initBiasicInfo(SOptrBasicInfo* pBasicInfo, SExprInfo* pExprInfo, + int32_t numOfCols, SSDataBlock* pResultBlock) { pBasicInfo->pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pBasicInfo->rowCellInfoOffset); pBasicInfo->pRes = pResultBlock; for (int32_t i = 0; i < numOfCols; ++i) { - pBasicInfo->pCtx[i].pBuf = pResultBuf; + pBasicInfo->pCtx[i].pBuf = NULL; } return TSDB_CODE_SUCCESS; } @@ -2237,16 +2240,15 @@ SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SEx initResultSizeInfo(pOperator, 4096); - code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo"); + code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock); if (code != TSDB_CODE_SUCCESS) { goto _error; } - - code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock, pInfo->streamAggSup.pResultBuf); + pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols); + code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo"); if (code != TSDB_CODE_SUCCESS) { goto _error; } - pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols); pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx)); if (pInfo->pDummyCtx == NULL) { @@ -3101,6 +3103,7 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); int32_t tsSlotId = ((SColumnNode*)pStateNode->window.pTspk)->slotId; SColumnNode* pColNode = (SColumnNode*)((STargetNode*)pStateNode->pStateKey)->pExpr; + int32_t code = TSDB_CODE_OUT_OF_MEMORY; SStreamStateAggOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SStreamStateAggOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); @@ -3121,17 +3124,18 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys .winMap = NULL, }; initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); - int32_t code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo"); + + code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock); if (code != TSDB_CODE_SUCCESS) { goto _error; } - code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock, pInfo->streamAggSup.pResultBuf); + pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols); + code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo"); if (code != TSDB_CODE_SUCCESS) { goto _error; } - pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols); pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx)); if (pInfo->pDummyCtx == NULL) { goto _error; diff --git a/source/libs/executor/src/tsort.c b/source/libs/executor/src/tsort.c index 846da4a32a248a8eb4c4e91f0a0a0791df84facb..ea8ded2c302eddc3b5429aec215952c295e1640a 100644 --- a/source/libs/executor/src/tsort.c +++ b/source/libs/executor/src/tsort.c @@ -144,7 +144,9 @@ static int32_t doAddNewExternalMemSource(SDiskbasedBuf *pBuf, SArray* pAllSource (*sourceId) += 1; int32_t rowSize = blockDataGetSerialRowSize(pSource->src.pBlock); - int32_t numOfRows = (getBufPageSize(pBuf) - blockDataGetSerialMetaSize(pBlock))/rowSize; // The value of numOfRows must be greater than 0, which is guaranteed by the previous memory allocation + + // The value of numOfRows must be greater than 0, which is guaranteed by the previous memory allocation + int32_t numOfRows = (getBufPageSize(pBuf) - blockDataGetSerialMetaSize(pBlock->info.numOfCols))/rowSize; ASSERT(numOfRows > 0); return blockDataEnsureCapacity(pSource->src.pBlock, numOfRows); } @@ -359,6 +361,11 @@ int32_t msortComparFn(const void *pLeft, const void *pRight, void *param) { SSDataBlock* pLeftBlock = pLeftSource->src.pBlock; SSDataBlock* pRightBlock = pRightSource->src.pBlock; + // first sort by block groupId + if (pLeftBlock->info.groupId != pRightBlock->info.groupId) { + return pLeftBlock->info.groupId < pRightBlock->info.groupId ? -1 : 1; + } + for(int32_t i = 0; i < pInfo->size; ++i) { SBlockOrderInfo* pOrder = TARRAY_GET_ELEM(pInfo, i); SColumnInfoData* pLeftColInfoData = TARRAY_GET_ELEM(pLeftBlock->pDataBlock, pOrder->slotId); @@ -521,16 +528,24 @@ static int32_t createInitialSources(SSortHandle* pHandle) { if (pHandle->type == SORT_SINGLESOURCE_SORT) { SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0); taosArrayClear(pHandle->pOrderedSource); - + + bool hasGroupId = false; + SSDataBlock* prefetchedDataBlock = NULL; + while (1) { - SSDataBlock* pBlock = pHandle->fetchfp(source->param); + SSDataBlock* pBlock = NULL; + if (prefetchedDataBlock == NULL) { + pBlock = pHandle->fetchfp(source->param); + } else { + pBlock = prefetchedDataBlock; + prefetchedDataBlock = NULL; + } + if (pBlock == NULL) { break; } - if (pHandle->pDataBlock == NULL) { - pHandle->pDataBlock = createOneDataBlock(pBlock, false); - + if (!hasGroupId) { // calculate the buffer pages according to the total available buffers. int32_t rowSize = blockDataGetRowSize(pBlock); if (rowSize * 4 > 4096) { @@ -542,29 +557,36 @@ static int32_t createInitialSources(SSortHandle* pHandle) { // todo!! pHandle->numOfPages = 1024; sortBufSize = pHandle->numOfPages * pHandle->pageSize; - } - // perform the scalar function calculation before apply the sort - if (pHandle->beforeFp != NULL) { - pHandle->beforeFp(pBlock, pHandle->param); + hasGroupId = true; + pHandle->pDataBlock = createOneDataBlock(pBlock, false); } - // todo relocate the columns - int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock); - if (code != 0) { - return code; - } + if (pHandle->pDataBlock->info.groupId == pBlock->info.groupId) { + // perform the scalar function calculation before apply the sort + if (pHandle->beforeFp != NULL) { + pHandle->beforeFp(pBlock, pHandle->param); + } + // todo relocate the columns + int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock); + if (code != 0) { + return code; + } - size_t size = blockDataGetSize(pHandle->pDataBlock); - if (size > sortBufSize) { - // Perform the in-memory sort and then flush data in the buffer into disk. - int64_t p = taosGetTimestampUs(); - blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo); + size_t size = blockDataGetSize(pHandle->pDataBlock); + if (size > sortBufSize) { + // Perform the in-memory sort and then flush data in the buffer into disk. + int64_t p = taosGetTimestampUs(); + blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo); - int64_t el = taosGetTimestampUs() - p; - pHandle->sortElapsed += el; + int64_t el = taosGetTimestampUs() - p; + pHandle->sortElapsed += el; - doAddToBuf(pHandle->pDataBlock, pHandle); + doAddToBuf(pHandle->pDataBlock, pHandle); + } + } else { + prefetchedDataBlock = pBlock; + pHandle->pDataBlock = createOneDataBlock(pBlock, false); } } diff --git a/source/libs/function/inc/builtinsimpl.h b/source/libs/function/inc/builtinsimpl.h index 981b1eec885ea3857f843b75687d2ff2de6e11c4..3a9b94d71ed9385a56aecd15b4b01e0fc3cdb6aa 100644 --- a/source/libs/function/inc/builtinsimpl.h +++ b/source/libs/function/inc/builtinsimpl.h @@ -78,13 +78,14 @@ bool percentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultI int32_t percentileFunction(SqlFunctionCtx *pCtx); int32_t percentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); -int32_t getApercentileMaxSize(); bool getApercentileFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool apercentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); int32_t apercentileFunction(SqlFunctionCtx *pCtx); int32_t apercentileFunctionMerge(SqlFunctionCtx* pCtx); int32_t apercentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); +int32_t apercentileCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx); +int32_t getApercentileMaxSize(); bool getDiffFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool diffFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResInfo); @@ -102,13 +103,13 @@ int32_t topFunction(SqlFunctionCtx *pCtx); int32_t bottomFunction(SqlFunctionCtx *pCtx); int32_t topBotFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); -int32_t getSpreadInfoSize(); bool getSpreadFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool spreadFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); int32_t spreadFunction(SqlFunctionCtx* pCtx); int32_t spreadFunctionMerge(SqlFunctionCtx* pCtx); int32_t spreadFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t spreadPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); +int32_t getSpreadInfoSize(); bool getElapsedFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool elapsedFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); @@ -118,7 +119,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); bool getHistogramFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool histogramFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); int32_t histogramFunction(SqlFunctionCtx* pCtx); +int32_t histogramFunctionMerge(SqlFunctionCtx* pCtx); int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); +int32_t histogramPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); +int32_t getHistogramInfoSize(); bool getHLLFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); int32_t hllFunction(SqlFunctionCtx* pCtx); diff --git a/source/libs/function/src/builtins.c b/source/libs/function/src/builtins.c index 946c37bfd29f539c60b274e0b3ffd98c83947b50..eca4587c03c97805ba4efe6c1afcf41ae8c1ddbe 100644 --- a/source/libs/function/src/builtins.c +++ b/source/libs/function/src/builtins.c @@ -503,6 +503,58 @@ static int32_t translateHistogram(SFunctionNode* pFunc, char* pErrBuf, int32_t l return TSDB_CODE_SUCCESS; } +static int32_t translateHistogramImpl(SFunctionNode* pFunc, char* pErrBuf, int32_t len, bool isPartial) { + int32_t numOfParams = LIST_LENGTH(pFunc->pParameterList); + if (isPartial) { + if (4 != numOfParams) { + return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); + } + + uint8_t colType = ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type; + if (!IS_NUMERIC_TYPE(colType)) { + return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); + } + + // param1 ~ param3 + for (int32_t i = 1; i < numOfParams; ++i) { + SNode* pParamNode = nodesListGetNode(pFunc->pParameterList, i); + if (QUERY_NODE_VALUE != nodeType(pParamNode)) { + return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); + } + + SValueNode* pValue = (SValueNode*)pParamNode; + + pValue->notReserved = true; + } + + if (((SExprNode*)nodesListGetNode(pFunc->pParameterList, 1))->resType.type != TSDB_DATA_TYPE_BINARY || + ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 2))->resType.type != TSDB_DATA_TYPE_BINARY || + ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 3))->resType.type != TSDB_DATA_TYPE_BIGINT) { + return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); + } + + pFunc->node.resType = (SDataType){.bytes = getHistogramInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY}; + } else { + if (1 != numOfParams) { + return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); + } + + if (((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type != TSDB_DATA_TYPE_BINARY) { + return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); + } + + pFunc->node.resType = (SDataType){.bytes = 512, .type = TSDB_DATA_TYPE_BINARY}; + } + return TSDB_CODE_SUCCESS; +} + +static int32_t translateHistogramPartial(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { + return translateHistogramImpl(pFunc, pErrBuf, len, true); +} +static int32_t translateHistogramMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { + return translateHistogramImpl(pFunc, pErrBuf, len, false); +} + static int32_t translateHLL(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { if (1 != LIST_LENGTH(pFunc->pParameterList)) { return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); @@ -1241,7 +1293,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .getEnvFunc = getPercentileFuncEnv, .initFunc = percentileFunctionSetup, .processFunc = percentileFunction, - .finalizeFunc = percentileFinalize + .finalizeFunc = percentileFinalize, + .invertFunc = NULL, + .combineFunc = NULL, }, { .name = "apercentile", @@ -1252,6 +1306,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .initFunc = apercentileFunctionSetup, .processFunc = apercentileFunction, .finalizeFunc = apercentileFinalize, + .combineFunc = apercentileCombine, .pPartialFunc = "_apercentile_partial", .pMergeFunc = "_apercentile_merge" }, @@ -1391,6 +1446,28 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .getEnvFunc = getHistogramFuncEnv, .initFunc = histogramFunctionSetup, .processFunc = histogramFunction, + .finalizeFunc = histogramFinalize, + .pPartialFunc = "_histogram_partial", + .pMergeFunc = "_histogram_merge" + }, + { + .name = "_histogram_partial", + .type = FUNCTION_TYPE_HISTOGRAM_PARTIAL, + .classification = FUNC_MGT_AGG_FUNC, + .translateFunc = translateHistogramPartial, + .getEnvFunc = getHistogramFuncEnv, + .initFunc = histogramFunctionSetup, + .processFunc = histogramFunction, + .finalizeFunc = histogramPartialFinalize + }, + { + .name = "_histogram_merge", + .type = FUNCTION_TYPE_HISTOGRAM_MERGE, + .classification = FUNC_MGT_AGG_FUNC, + .translateFunc = translateHistogramMerge, + .getEnvFunc = getHistogramFuncEnv, + .initFunc = functionSetup, + .processFunc = histogramFunctionMerge, .finalizeFunc = histogramFinalize }, { diff --git a/source/libs/function/src/builtinsimpl.c b/source/libs/function/src/builtinsimpl.c index 34adf11032b98271c753b2862ca25600c9e55dd1..e0302f959802249527931608ce336c3aca755567 100644 --- a/source/libs/function/src/builtinsimpl.c +++ b/source/libs/function/src/builtinsimpl.c @@ -2187,9 +2187,7 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); SAPercentileInfo* pInfo = (SAPercentileInfo*)GET_ROWCELL_INTERBUF(pResInfo); - int32_t bytesHist = (int32_t)(sizeof(SAPercentileInfo) + sizeof(SHistogramInfo) + sizeof(SHistBin) * (MAX_HISTOGRAM_BIN + 1)); - int32_t bytesDigest = (int32_t)(sizeof(SAPercentileInfo) + TDIGEST_SIZE(COMPRESSION)); - int32_t resultBytes = TMAX(bytesHist, bytesDigest); + int32_t resultBytes = getApercentileMaxSize(); char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); if (pInfo->algo == APERCT_ALGO_TDIGEST) { @@ -2217,6 +2215,26 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { return pResInfo->numOfRes; } +int32_t apercentileCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) { + SResultRowEntryInfo* pDResInfo = GET_RES_INFO(pDestCtx); + SAPercentileInfo* pDBuf = GET_ROWCELL_INTERBUF(pDResInfo); + int32_t type = pDestCtx->input.pData[0]->info.type; + + SResultRowEntryInfo* pSResInfo = GET_RES_INFO(pSourceCtx); + SAPercentileInfo* pSBuf = GET_ROWCELL_INTERBUF(pSResInfo); + ASSERT(pDBuf->algo == pSBuf->algo); + if (pDBuf->algo == APERCT_ALGO_TDIGEST) { + tdigestMerge(pDBuf->pTDigest, pSBuf->pTDigest); + } else { + SHistogramInfo* pTmp = tHistogramMerge(pDBuf->pHisto, pSBuf->pHisto, MAX_HISTOGRAM_BIN); + memcpy(pDBuf->pHisto, pTmp, sizeof(SHistogramInfo) + sizeof(SHistBin) * (MAX_HISTOGRAM_BIN + 1)); + pDBuf->pHisto->elems = (SHistBin*) ((char *)pDBuf->pHisto + sizeof(SHistogramInfo)); + tHistogramDestroy(&pTmp); + } + pDResInfo->numOfRes = TMAX(pDResInfo->numOfRes, pSResInfo->numOfRes); + return TSDB_CODE_SUCCESS; +} + bool getFirstLastFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { SColumnNode* pNode = nodesListGetNode(pFunc->pParameterList, 0); pEnv->calcMemSize = pNode->node.resType.bytes + sizeof(int64_t); @@ -3118,6 +3136,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { return functionFinalize(pCtx, pBlock); } +int32_t getHistogramInfoSize() { + return (int32_t)sizeof(SHistoFuncInfo) + HISTOGRAM_MAX_BINS_NUM * sizeof(SHistoFuncBin); +} + bool getHistogramFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) { pEnv->calcMemSize = sizeof(SHistoFuncInfo) + HISTOGRAM_MAX_BINS_NUM * sizeof(SHistoFuncBin); return true; @@ -3328,6 +3350,30 @@ int32_t histogramFunction(SqlFunctionCtx *pCtx) { return TSDB_CODE_SUCCESS; } +int32_t histogramFunctionMerge(SqlFunctionCtx *pCtx) { + SInputColumnInfoData* pInput = &pCtx->input; + SColumnInfoData* pCol = pInput->pData[0]; + ASSERT(pCol->info.type == TSDB_DATA_TYPE_BINARY); + + SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + + int32_t start = pInput->startRowIndex; + char* data = colDataGetData(pCol, start); + SHistoFuncInfo* pInputInfo = (SHistoFuncInfo *)varDataVal(data); + + pInfo->normalized = pInputInfo->normalized; + pInfo->numOfBins = pInputInfo->numOfBins; + pInfo->totalCount += pInputInfo->totalCount; + for (int32_t k = 0; k < pInfo->numOfBins; ++k) { + pInfo->bins[k].lower = pInputInfo->bins[k].lower; + pInfo->bins[k].upper = pInputInfo->bins[k].upper; + pInfo->bins[k].count += pInputInfo->bins[k].count; + } + + SET_VAL(GET_RES_INFO(pCtx), pInfo->numOfBins, pInfo->numOfBins); + return TSDB_CODE_SUCCESS; +} + int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); @@ -3364,6 +3410,24 @@ int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { return pResInfo->numOfRes; } +int32_t histogramPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { + SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); + SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + int32_t resultBytes = getHistogramInfoSize(); + char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); + + memcpy(varDataVal(res), pInfo, resultBytes); + varDataSetLen(res, resultBytes); + + int32_t slotId = pCtx->pExpr->base.resSchema.slotId; + SColumnInfoData* pCol = taosArrayGet(pBlock->pDataBlock, slotId); + + colDataAppend(pCol, pBlock->info.rows, res, false); + + taosMemoryFree(res); + return 1; +} + bool getHLLFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) { pEnv->calcMemSize = sizeof(SHLLInfo); return true; diff --git a/source/libs/index/inc/indexCache.h b/source/libs/index/inc/indexCache.h index 6e68163d74677ad0b7c9df944b73d2ebe602d93a..6c95eb987bc457eab73184fac1ea88bce6dd70f8 100644 --- a/source/libs/index/inc/indexCache.h +++ b/source/libs/index/inc/indexCache.h @@ -70,7 +70,7 @@ void indexCacheBroadcast(void* cache); void indexCacheWait(void* cache); Iterate* indexCacheIteratorCreate(IndexCache* cache); -void indexCacheIteratorDestroy(Iterate* iiter); +void idxCacheIteratorDestroy(Iterate* iiter); int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid); @@ -82,7 +82,7 @@ void indexCacheUnRef(IndexCache* cache); void indexCacheDebug(IndexCache* cache); -void indexCacheDestroyImm(IndexCache* cache); +void idxCacheDestroyImm(IndexCache* cache); #ifdef __cplusplus } #endif diff --git a/source/libs/index/inc/indexComm.h b/source/libs/index/inc/indexComm.h index 4b23ccabb9eb0160c2822482bcedb2f20088dbd9..bccba98116ef9be61d352bea8423384a4c2b60e3 100644 --- a/source/libs/index/inc/indexComm.h +++ b/source/libs/index/inc/indexComm.h @@ -26,9 +26,9 @@ extern "C" { extern char JSON_COLUMN[]; extern char JSON_VALUE_DELIM; -char* indexPackJsonData(SIndexTerm* itm); -char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip); -char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip); +char* idxPackJsonData(SIndexTerm* itm); +char* idxPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip); +char* idxPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip); typedef enum { MATCH, CONTINUE, BREAK } TExeCond; @@ -40,13 +40,14 @@ TExeCond tDoCompare(__compar_fn_t func, int8_t cmpType, void* a, void* b); _cache_range_compare indexGetCompare(RangeType ty); -int32_t indexConvertData(void* src, int8_t type, void** dst); -int32_t indexConvertDataToStr(void* src, int8_t type, void** dst); +int32_t idxConvertData(void* src, int8_t type, void** dst); +int32_t idxConvertDataToStr(void* src, int8_t type, void** dst); -int32_t indexGetDataByteLen(int8_t type); +int32_t idxGetDataByteLen(int8_t type); -char* indexInt2str(int64_t val, char* dst, int radix); +char* idxInt2str(int64_t val, char* dst, int radix); +int idxUidCompare(const void* a, const void* b); #ifdef __cplusplus } #endif diff --git a/source/libs/index/inc/indexFst.h b/source/libs/index/inc/indexFst.h index 0a360c1c72e621615796e8c5f383110e8ec11db6..6fd12c110cc3a00b72aa009123c2299edf372224 100644 --- a/source/libs/index/inc/indexFst.h +++ b/source/libs/index/inc/indexFst.h @@ -29,9 +29,9 @@ extern "C" { #define OUTPUT_PREFIX(a, b) ((a) > (b) ? (b) : (a) -typedef struct Fst Fst; -typedef struct FstNode FstNode; -typedef struct StreamWithState StreamWithState; +typedef struct Fst Fst; +typedef struct FstNode FstNode; +typedef struct FStmSt FStmSt; typedef enum { Included, Excluded, Unbounded } FstBound; @@ -40,12 +40,12 @@ typedef struct FstBoundWithData { FstBound type; } FstBoundWithData; -typedef struct FstStreamBuilder { +typedef struct FStmBuilder { Fst* fst; - AutomationCtx* aut; + FAutoCtx* aut; FstBoundWithData* min; FstBoundWithData* max; -} FstStreamBuilder, FstStreamWithStateBuilder; +} FStmBuilder, FStmStBuilder; typedef struct FstRange { uint64_t start; @@ -267,17 +267,17 @@ typedef struct Fst { Fst* fstCreate(FstSlice* data); void fstDestroy(Fst* fst); -bool fstGet(Fst* fst, FstSlice* b, Output* out); -FstNode* fstGetNode(Fst* fst, CompiledAddr); -FstNode* fstGetRoot(Fst* fst); -FstType fstGetType(Fst* fst); -CompiledAddr fstGetRootAddr(Fst* fst); -Output fstEmptyFinalOutput(Fst* fst, bool* null); -FstStreamBuilder* fstSearch(Fst* fst, AutomationCtx* ctx); +bool fstGet(Fst* fst, FstSlice* b, Output* out); +FstNode* fstGetNode(Fst* fst, CompiledAddr); +FstNode* fstGetRoot(Fst* fst); +FstType fstGetType(Fst* fst); +CompiledAddr fstGetRootAddr(Fst* fst); +Output fstEmptyFinalOutput(Fst* fst, bool* null); +FStmBuilder* fstSearch(Fst* fst, FAutoCtx* ctx); -FstStreamWithStateBuilder* fstSearchWithState(Fst* fst, AutomationCtx* ctx); +FStmStBuilder* fstSearchWithState(Fst* fst, FAutoCtx* ctx); // into stream to expand later -StreamWithState* streamBuilderIntoStream(FstStreamBuilder* sb); +FStmSt* stmBuilderIntoStm(FStmBuilder* sb); bool fstVerify(Fst* fst); @@ -293,41 +293,40 @@ typedef struct StreamState { void streamStateDestroy(void* s); -typedef struct StreamWithState { +typedef struct FStmSt { Fst* fst; - AutomationCtx* aut; + FAutoCtx* aut; SArray* inp; FstOutput emptyOutput; SArray* stack; // FstBoundWithData* endAt; -} StreamWithState; +} FStmSt; -typedef struct StreamWithStateResult { +typedef struct FStmStRslt { FstSlice data; FstOutput out; void* state; -} StreamWithStateResult; +} FStmStRslt; -StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* state); -void swsResultDestroy(StreamWithStateResult* result); +FStmStRslt* swsResultCreate(FstSlice* data, FstOutput fOut, void* state); +void swsResultDestroy(FStmStRslt* result); typedef void* (*StreamCallback)(void*); -StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstBoundWithData* min, - FstBoundWithData* max); +FStmSt* stmStCreate(Fst* fst, FAutoCtx* automation, FstBoundWithData* min, FstBoundWithData* max); -void streamWithStateDestroy(StreamWithState* sws); +void stmStDestroy(FStmSt* sws); -bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min); +bool stmStSeekMin(FStmSt* sws, FstBoundWithData* min); -StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallback callback); +FStmStRslt* stmStNextWith(FStmSt* sws, StreamCallback callback); -FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut); +FStmBuilder* stmBuilderCreate(Fst* fst, FAutoCtx* aut); -void fstStreamBuilderDestroy(FstStreamBuilder* b); +void stmBuilderDestroy(FStmBuilder* b); // set up bound range // refator later: to simple code by marco -void fstStreamBuilderSetRange(FstStreamBuilder* b, FstSlice* val, RangeType type); +void stmBuilderSetRange(FStmBuilder* b, FstSlice* val, RangeType type); #ifdef __cplusplus } diff --git a/source/libs/index/inc/indexFstAutomation.h b/source/libs/index/inc/indexFstAutomation.h index 3a0f74ee76563d3d3ed2cc1f3a067945eaa0fed8..583bbfedae017bd5bd92909cf8c5beae49b79f05 100644 --- a/source/libs/index/inc/indexFstAutomation.h +++ b/source/libs/index/inc/indexFstAutomation.h @@ -22,24 +22,24 @@ extern "C" { #include "indexFstUtil.h" #include "indexInt.h" -typedef struct AutomationCtx AutomationCtx; +typedef struct FAutoCtx FAutoCtx; typedef enum AutomationType { AUTOMATION_ALWAYS, AUTOMATION_PREFIX, AUTMMATION_MATCH } AutomationType; typedef struct StartWith { - AutomationCtx* autoSelf; + FAutoCtx* autoSelf; } StartWith; typedef struct Complement { - AutomationCtx* autoSelf; + FAutoCtx* autoSelf; } Complement; // automation -typedef struct AutomationCtx { +typedef struct FAutoCtx { AutomationType type; void* stdata; char* data; -} AutomationCtx; +} FAutoCtx; typedef enum ValueType { FST_INT, FST_CHAR, FST_ARRAY } ValueType; typedef enum StartWithStateKind { Done, Running } StartWithStateKind; @@ -60,16 +60,16 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv); void startWithStateValueDestroy(void* sv); typedef struct AutomationFunc { - void* (*start)(AutomationCtx* ctx); - bool (*isMatch)(AutomationCtx* ctx, void*); - bool (*canMatch)(AutomationCtx* ctx, void* data); - bool (*willAlwaysMatch)(AutomationCtx* ctx, void* state); - void* (*accept)(AutomationCtx* ctx, void* state, uint8_t byte); - void* (*acceptEof)(AutomationCtx* ct, void* state); + void* (*start)(FAutoCtx* ctx); + bool (*isMatch)(FAutoCtx* ctx, void*); + bool (*canMatch)(FAutoCtx* ctx, void* data); + bool (*willAlwaysMatch)(FAutoCtx* ctx, void* state); + void* (*accept)(FAutoCtx* ctx, void* state, uint8_t byte); + void* (*acceptEof)(FAutoCtx* ct, void* state); } AutomationFunc; -AutomationCtx* automCtxCreate(void* data, AutomationType atype); -void automCtxDestroy(AutomationCtx* ctx); +FAutoCtx* automCtxCreate(void* data, AutomationType atype); +void automCtxDestroy(FAutoCtx* ctx); extern AutomationFunc automFuncs[]; #ifdef __cplusplus diff --git a/source/libs/index/inc/indexInt.h b/source/libs/index/inc/indexInt.h index f556a738dd3ad624c3fc247f1bfd351bf7fffa09..47f7260d3acefcb80d0900c301527b1cf20c1e60 100644 --- a/source/libs/index/inc/indexInt.h +++ b/source/libs/index/inc/indexInt.h @@ -131,7 +131,7 @@ typedef struct TFileCacheKey { char* colName; int32_t nColName; } ICacheKey; -int indexFlushCacheToTFile(SIndex* sIdx, void*, bool quit); +int idxFlushCacheToTFile(SIndex* sIdx, void*, bool quit); int64_t indexAddRef(void* p); int32_t indexRemoveRef(int64_t ref); diff --git a/source/libs/index/src/index.c b/source/libs/index/src/index.c index 058f3c1915ab67c48b179c6c0b32ef56244c24e8..9b8bee5623db4953fe1616aa532fe60fb836dce6 100644 --- a/source/libs/index/src/index.c +++ b/source/libs/index/src/index.c @@ -77,15 +77,15 @@ typedef struct SIdxColInfo { static TdThreadOnce isInit = PTHREAD_ONCE_INIT; // static void indexInit(); -static int indexTermSearch(SIndex* sIdx, SIndexTermQuery* term, SArray** result); +static int idxTermSearch(SIndex* sIdx, SIndexTermQuery* term, SArray** result); -static void indexInterResultsDestroy(SArray* results); -static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out); +static void idxInterRsltDestroy(SArray* results); +static int idxMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out); -static int indexGenTFile(SIndex* index, IndexCache* cache, SArray* batch); +static int idxGenTFile(SIndex* index, IndexCache* cache, SArray* batch); // merge cache and tfile by opera type -static void indexMergeCacheAndTFile(SArray* result, IterateValue* icache, IterateValue* iTfv, SIdxTRslt* helper); +static void idxMergeCacheAndTFile(SArray* result, IterateValue* icache, IterateValue* iTfv, SIdxTRslt* helper); // static int32_t indexSerialTermKey(SIndexTerm* itm, char* buf); // int32_t indexSerialKey(ICacheKey* key, char* buf); @@ -221,11 +221,11 @@ int indexSearch(SIndex* index, SIndexMultiTermQuery* multiQuerys, SArray* result for (size_t i = 0; i < nQuery; i++) { SIndexTermQuery* qterm = taosArrayGet(multiQuerys->query, i); SArray* trslt = NULL; - indexTermSearch(index, qterm, &trslt); + idxTermSearch(index, qterm, &trslt); taosArrayPush(iRslts, (void*)&trslt); } - indexMergeFinalResults(iRslts, opera, result); - indexInterResultsDestroy(iRslts); + idxMergeFinalResults(iRslts, opera, result); + idxInterRsltDestroy(iRslts); return 0; } @@ -289,7 +289,7 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy tm->nColName = nColName; char* buf = NULL; - int32_t len = indexConvertDataToStr((void*)colVal, INDEX_TYPE_GET_TYPE(colType), (void**)&buf); + int32_t len = idxConvertDataToStr((void*)colVal, INDEX_TYPE_GET_TYPE(colType), (void**)&buf); assert(len != -1); tm->colVal = buf; @@ -319,7 +319,7 @@ void indexMultiTermDestroy(SIndexMultiTerm* terms) { taosArrayDestroy(terms); } -static int indexTermSearch(SIndex* sIdx, SIndexTermQuery* query, SArray** result) { +static int idxTermSearch(SIndex* sIdx, SIndexTermQuery* query, SArray** result) { SIndexTerm* term = query->term; const char* colName = term->colName; int32_t nColName = term->nColName; @@ -374,7 +374,7 @@ END: idxTRsltDestroy(tr); return -1; } -static void indexInterResultsDestroy(SArray* results) { +static void idxInterRsltDestroy(SArray* results) { if (results == NULL) { return; } @@ -387,7 +387,7 @@ static void indexInterResultsDestroy(SArray* results) { taosArrayDestroy(results); } -static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out) { +static int idxMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out) { // refactor, merge interResults into fResults by oType for (int i = 0; i < taosArrayGetSize(in); i--) { SArray* t = taosArrayGetP(in, i); @@ -407,7 +407,7 @@ static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* return 0; } -static void indexMayMergeTempToFinalResult(SArray* result, TFileValue* tfv, SIdxTRslt* tr) { +static void idxMayMergeTempToFinalRslt(SArray* result, TFileValue* tfv, SIdxTRslt* tr) { int32_t sz = taosArrayGetSize(result); if (sz > 0) { TFileValue* lv = taosArrayGetP(result, sz - 1); @@ -427,11 +427,11 @@ static void indexMayMergeTempToFinalResult(SArray* result, TFileValue* tfv, SIdx taosArrayPush(result, &tfv); } } -static void indexMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateValue* tv, SIdxTRslt* tr) { +static void idxMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateValue* tv, SIdxTRslt* tr) { char* colVal = (cv != NULL) ? cv->colVal : tv->colVal; TFileValue* tfv = tfileValueCreate(colVal); - indexMayMergeTempToFinalResult(result, tfv, tr); + idxMayMergeTempToFinalRslt(result, tfv, tr); if (cv != NULL) { uint64_t id = *(uint64_t*)taosArrayGet(cv->val, 0); @@ -446,7 +446,7 @@ static void indexMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateVal taosArrayAddAll(tr->total, tv->val); } } -static void indexDestroyFinalResult(SArray* result) { +static void idxDestroyFinalRslt(SArray* result) { int32_t sz = result ? taosArrayGetSize(result) : 0; for (size_t i = 0; i < sz; i++) { TFileValue* tv = taosArrayGetP(result, i); @@ -455,7 +455,7 @@ static void indexDestroyFinalResult(SArray* result) { taosArrayDestroy(result); } -int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { +int idxFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { if (sIdx == NULL) { return -1; } @@ -475,7 +475,7 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { Iterate* cacheIter = indexCacheIteratorCreate(pCache); if (cacheIter == NULL) { indexError("%p immtable is empty, ignore merge opera", pCache); - indexCacheDestroyImm(pCache); + idxCacheDestroyImm(pCache); tfileReaderUnRef(pReader); atomic_store_32(&pCache->merging, 0); if (quit) { @@ -509,26 +509,26 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { comp = 1; } if (comp == 0) { - indexMergeCacheAndTFile(result, cv, tv, tr); + idxMergeCacheAndTFile(result, cv, tv, tr); cn = cacheIter->next(cacheIter); tn = tfileIter->next(tfileIter); } else if (comp < 0) { - indexMergeCacheAndTFile(result, cv, NULL, tr); + idxMergeCacheAndTFile(result, cv, NULL, tr); cn = cacheIter->next(cacheIter); } else { - indexMergeCacheAndTFile(result, NULL, tv, tr); + idxMergeCacheAndTFile(result, NULL, tv, tr); tn = tfileIter->next(tfileIter); } } - indexMayMergeTempToFinalResult(result, NULL, tr); + idxMayMergeTempToFinalRslt(result, NULL, tr); idxTRsltDestroy(tr); - int ret = indexGenTFile(sIdx, pCache, result); - indexDestroyFinalResult(result); + int ret = idxGenTFile(sIdx, pCache, result); + idxDestroyFinalRslt(result); - indexCacheDestroyImm(pCache); + idxCacheDestroyImm(pCache); - indexCacheIteratorDestroy(cacheIter); + idxCacheIteratorDestroy(cacheIter); tfileIteratorDestroy(tfileIter); tfileReaderUnRef(pReader); @@ -578,7 +578,7 @@ static int64_t indexGetAvaialbleVer(SIndex* sIdx, IndexCache* cache) { tfileReaderUnRef(rd); return ver; } -static int indexGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) { +static int idxGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) { int64_t version = indexGetAvaialbleVer(sIdx, cache); indexInfo("file name version: %" PRId64 "", version); uint8_t colType = cache->type; @@ -625,7 +625,7 @@ int32_t indexSerialCacheKey(ICacheKey* key, char* buf) { char* p = buf; char tbuf[65] = {0}; - indexInt2str((int64_t)key->suid, tbuf, 0); + idxInt2str((int64_t)key->suid, tbuf, 0); SERIALIZE_STR_VAR_TO_BUF(buf, tbuf, strlen(tbuf)); SERIALIZE_VAR_TO_BUF(buf, '_', char); diff --git a/source/libs/index/src/indexCache.c b/source/libs/index/src/indexCache.c index 1e80d9c68d3eeb49abe4d2e1d1be9b6b94eff7d4..20cd9c8b4c8bc2c7381c02dcf9454f8bad7ce979 100644 --- a/source/libs/index/src/indexCache.c +++ b/source/libs/index/src/indexCache.c @@ -26,15 +26,15 @@ #define MEM_SIGNAL_QUIT MEM_THRESHOLD * 20 #define MEM_ESTIMATE_RADIO 1.5 -static void indexMemRef(MemTable* tbl); -static void indexMemUnRef(MemTable* tbl); +static void idxMemRef(MemTable* tbl); +static void idxMemUnRef(MemTable* tbl); -static void indexCacheTermDestroy(CacheTerm* ct); -static int32_t indexCacheTermCompare(const void* l, const void* r); -static int32_t indexCacheJsonTermCompare(const void* l, const void* r); -static char* indexCacheTermGet(const void* pData); +static void idxCacheTermDestroy(CacheTerm* ct); +static int32_t idxCacheTermCompare(const void* l, const void* r); +static int32_t idxCacheJsonTermCompare(const void* l, const void* r); +static char* idxCacheTermGet(const void* pData); -static MemTable* indexInternalCacheCreate(int8_t type); +static MemTable* idxInternalCacheCreate(int8_t type); static int32_t cacheSearchTerm(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s); static int32_t cacheSearchPrefix(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s); @@ -69,7 +69,7 @@ static int32_t (*cacheSearch[][QUERY_MAX])(void* cache, SIndexTerm* ct, SIdxTRsl cacheSearchRange_JSON}}; static void doMergeWork(SSchedMsg* msg); -static bool indexCacheIteratorNext(Iterate* itera); +static bool idxCacheIteratorNext(Iterate* itera); static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STermValueType* s) { if (cache == NULL) { @@ -82,7 +82,7 @@ static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STe pCt->colVal = term->colVal; pCt->version = atomic_load_64(&pCache->version); - char* key = indexCacheTermGet(pCt); + char* key = idxCacheTermGet(pCt); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); while (tSkipListIterNext(iter)) { @@ -134,7 +134,7 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt* pCt->colType = term->colType; pCt->version = atomic_load_64(&pCache->version); - char* key = indexCacheTermGet(pCt); + char* key = idxCacheTermGet(pCt); SSkipListIterator* iter = tSkipListCreateIter(mem->mem); while (tSkipListIterNext(iter)) { @@ -188,10 +188,10 @@ static int32_t cacheSearchTerm_JSON(void* cache, SIndexTerm* term, SIdxTRslt* tr char* exBuf = NULL; if (INDEX_TYPE_CONTAIN_EXTERN_TYPE(term->colType, TSDB_DATA_TYPE_JSON)) { - exBuf = indexPackJsonData(term); + exBuf = idxPackJsonData(term); pCt->colVal = exBuf; } - char* key = indexCacheTermGet(pCt); + char* key = idxCacheTermGet(pCt); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); while (tSkipListIterNext(iter)) { @@ -269,20 +269,19 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR int8_t dType = INDEX_TYPE_GET_TYPE(term->colType); int skip = 0; char* exBuf = NULL; - if (type == CONTAINS) { SIndexTerm tm = {.suid = term->suid, .operType = term->operType, .colType = term->colType, .colName = term->colVal, .nColName = term->nColVal}; - exBuf = indexPackJsonDataPrefixNoType(&tm, &skip); + exBuf = idxPackJsonDataPrefixNoType(&tm, &skip); pCt->colVal = exBuf; } else { - exBuf = indexPackJsonDataPrefix(term, &skip); + exBuf = idxPackJsonDataPrefix(term, &skip); pCt->colVal = exBuf; } - char* key = indexCacheTermGet(pCt); + char* key = idxCacheTermGet(pCt); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); while (tSkipListIterNext(iter)) { @@ -341,7 +340,7 @@ IndexCache* indexCacheCreate(SIndex* idx, uint64_t suid, const char* colName, in return NULL; }; - cache->mem = indexInternalCacheCreate(type); + cache->mem = idxInternalCacheCreate(type); cache->mem->pCache = cache; cache->colName = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? tstrdup(JSON_COLUMN) : tstrdup(colName); cache->type = type; @@ -364,7 +363,7 @@ void indexCacheDebug(IndexCache* cache) { taosThreadMutexLock(&cache->mtx); tbl = cache->mem; - indexMemRef(tbl); + idxMemRef(tbl); taosThreadMutexUnlock(&cache->mtx); { @@ -380,13 +379,13 @@ void indexCacheDebug(IndexCache* cache) { } tSkipListDestroyIter(iter); - indexMemUnRef(tbl); + idxMemUnRef(tbl); } { taosThreadMutexLock(&cache->mtx); tbl = cache->imm; - indexMemRef(tbl); + idxMemRef(tbl); taosThreadMutexUnlock(&cache->mtx); if (tbl != NULL) { SSkipList* slt = tbl->mem; @@ -402,7 +401,7 @@ void indexCacheDebug(IndexCache* cache) { tSkipListDestroyIter(iter); } - indexMemUnRef(tbl); + idxMemUnRef(tbl); } } @@ -427,7 +426,7 @@ void indexCacheWait(void* cache) { IndexCache* pCache = cache; taosThreadCondWait(&pCache->finished, &pCache->mtx); } -void indexCacheDestroyImm(IndexCache* cache) { +void idxCacheDestroyImm(IndexCache* cache) { if (cache == NULL) { return; } @@ -440,8 +439,8 @@ void indexCacheDestroyImm(IndexCache* cache) { taosThreadMutexUnlock(&cache->mtx); - indexMemUnRef(tbl); - indexMemUnRef(tbl); + idxMemUnRef(tbl); + idxMemUnRef(tbl); } void indexCacheDestroy(void* cache) { IndexCache* pCache = cache; @@ -449,8 +448,8 @@ void indexCacheDestroy(void* cache) { return; } - indexMemUnRef(pCache->mem); - indexMemUnRef(pCache->imm); + idxMemUnRef(pCache->mem); + idxMemUnRef(pCache->imm); taosMemoryFree(pCache->colName); taosThreadMutexDestroy(&pCache->mtx); @@ -471,20 +470,20 @@ Iterate* indexCacheIteratorCreate(IndexCache* cache) { } taosThreadMutexLock(&cache->mtx); - indexMemRef(cache->imm); + idxMemRef(cache->imm); MemTable* tbl = cache->imm; iiter->val.val = taosArrayInit(1, sizeof(uint64_t)); iiter->val.colVal = NULL; iiter->iter = tbl != NULL ? tSkipListCreateIter(tbl->mem) : NULL; - iiter->next = indexCacheIteratorNext; + iiter->next = idxCacheIteratorNext; iiter->getValue = indexCacheIteratorGetValue; taosThreadMutexUnlock(&cache->mtx); return iiter; } -void indexCacheIteratorDestroy(Iterate* iter) { +void idxCacheIteratorDestroy(Iterate* iter) { if (iter == NULL) { return; } @@ -518,7 +517,7 @@ static void indexCacheMakeRoomForWrite(IndexCache* cache) { indexCacheRef(cache); cache->imm = cache->mem; - cache->mem = indexInternalCacheCreate(cache->type); + cache->mem = idxInternalCacheCreate(cache->type); cache->mem->pCache = cache; cache->occupiedMem = 0; if (quit == false) { @@ -546,7 +545,7 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) { // set up key ct->colType = term->colType; if (hasJson) { - ct->colVal = indexPackJsonData(term); + ct->colVal = idxPackJsonData(term); } else { ct->colVal = (char*)taosMemoryCalloc(1, sizeof(char) * (term->nColVal + 1)); memcpy(ct->colVal, term->colVal, term->nColVal); @@ -562,9 +561,9 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) { pCache->occupiedMem += estimate; indexCacheMakeRoomForWrite(pCache); MemTable* tbl = pCache->mem; - indexMemRef(tbl); + idxMemRef(tbl); tSkipListPut(tbl->mem, (char*)ct); - indexMemUnRef(tbl); + idxMemUnRef(tbl); taosThreadMutexUnlock(&pCache->mtx); @@ -615,8 +614,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe taosThreadMutexLock(&pCache->mtx); mem = pCache->mem; imm = pCache->imm; - indexMemRef(mem); - indexMemRef(imm); + idxMemRef(mem); + idxMemRef(imm); taosThreadMutexUnlock(&pCache->mtx); int ret = (mem && mem->mem) ? indexQueryMem(mem, query, result, s) : 0; @@ -625,8 +624,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe ret = (imm && imm->mem) ? indexQueryMem(imm, query, result, s) : 0; } - indexMemUnRef(mem); - indexMemUnRef(imm); + idxMemUnRef(mem); + idxMemUnRef(imm); indexInfo("cache search, time cost %" PRIu64 "us", taosGetTimestampUs() - st); return ret; @@ -649,14 +648,14 @@ void indexCacheUnRef(IndexCache* cache) { } } -void indexMemRef(MemTable* tbl) { +void idxMemRef(MemTable* tbl) { if (tbl == NULL) { return; } int ref = T_REF_INC(tbl); UNUSED(ref); } -void indexMemUnRef(MemTable* tbl) { +void idxMemUnRef(MemTable* tbl) { if (tbl == NULL) { return; } @@ -668,18 +667,18 @@ void indexMemUnRef(MemTable* tbl) { } } -static void indexCacheTermDestroy(CacheTerm* ct) { +static void idxCacheTermDestroy(CacheTerm* ct) { if (ct == NULL) { return; } taosMemoryFree(ct->colVal); taosMemoryFree(ct); } -static char* indexCacheTermGet(const void* pData) { +static char* idxCacheTermGet(const void* pData) { CacheTerm* p = (CacheTerm*)pData; return (char*)p; } -static int32_t indexCacheTermCompare(const void* l, const void* r) { +static int32_t idxCacheTermCompare(const void* l, const void* r) { CacheTerm* lt = (CacheTerm*)l; CacheTerm* rt = (CacheTerm*)r; // compare colVal @@ -700,7 +699,7 @@ static int indexFindCh(char* a, char c) { } return p - a; } -static int indexCacheJsonTermCompareImpl(char* a, char* b) { +static int idxCacheJsonTermCompareImpl(char* a, char* b) { // int alen = indexFindCh(a, '&'); // int blen = indexFindCh(b, '&'); @@ -720,7 +719,7 @@ static int indexCacheJsonTermCompareImpl(char* a, char* b) { //} return 0; } -static int32_t indexCacheJsonTermCompare(const void* l, const void* r) { +static int32_t idxCacheJsonTermCompare(const void* l, const void* r) { CacheTerm* lt = (CacheTerm*)l; CacheTerm* rt = (CacheTerm*)r; // compare colVal @@ -730,16 +729,15 @@ static int32_t indexCacheJsonTermCompare(const void* l, const void* r) { } return cmp; } -static MemTable* indexInternalCacheCreate(int8_t type) { +static MemTable* idxInternalCacheCreate(int8_t type) { int ttype = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? TSDB_DATA_TYPE_BINARY : TSDB_DATA_TYPE_BINARY; int32_t (*cmpFn)(const void* l, const void* r) = - INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? indexCacheJsonTermCompare : indexCacheTermCompare; + INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? idxCacheJsonTermCompare : idxCacheTermCompare; MemTable* tbl = taosMemoryCalloc(1, sizeof(MemTable)); - indexMemRef(tbl); + idxMemRef(tbl); if (ttype == TSDB_DATA_TYPE_BINARY || ttype == TSDB_DATA_TYPE_NCHAR) { - tbl->mem = - tSkipListCreate(MAX_SKIP_LIST_LEVEL, ttype, MAX_INDEX_KEY_LEN, cmpFn, SL_ALLOW_DUP_KEY, indexCacheTermGet); + tbl->mem = tSkipListCreate(MAX_SKIP_LIST_LEVEL, ttype, MAX_INDEX_KEY_LEN, cmpFn, SL_ALLOW_DUP_KEY, idxCacheTermGet); } return tbl; } @@ -750,9 +748,9 @@ static void doMergeWork(SSchedMsg* msg) { int quit = msg->thandle ? true : false; taosMemoryFree(msg->thandle); - indexFlushCacheToTFile(sidx, pCache, quit); + idxFlushCacheToTFile(sidx, pCache, quit); } -static bool indexCacheIteratorNext(Iterate* itera) { +static bool idxCacheIteratorNext(Iterate* itera) { SSkipListIterator* iter = itera->iter; if (iter == NULL) { return false; diff --git a/source/libs/index/src/indexComm.c b/source/libs/index/src/indexComm.c index 85c6bb46d7a6dd10e8be0bab473ab1732423daa8..be9243b8ddee6a497ed6ea40a498aab13da47c9c 100644 --- a/source/libs/index/src/indexComm.c +++ b/source/libs/index/src/indexComm.c @@ -48,7 +48,7 @@ char JSON_COLUMN[] = "JSON"; char JSON_VALUE_DELIM = '&'; -char* indexInt2str(int64_t val, char* dst, int radix) { +char* idxInt2str(int64_t val, char* dst, int radix) { char buffer[65] = {0}; char* p; int64_t new_val; @@ -207,7 +207,7 @@ static TExeCond (*rangeCompare[])(void* a, void* b, int8_t type) = { _cache_range_compare indexGetCompare(RangeType ty) { return rangeCompare[ty]; } -char* indexPackJsonData(SIndexTerm* itm) { +char* idxPackJsonData(SIndexTerm* itm) { /* * |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| @@ -235,7 +235,7 @@ char* indexPackJsonData(SIndexTerm* itm) { return buf; } -char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) { +char* idxPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) { /* * |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| @@ -262,7 +262,7 @@ char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) { return buf; } -char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) { +char* idxPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) { /* * |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| @@ -283,7 +283,12 @@ char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) { return buf; } -int32_t indexConvertData(void* src, int8_t type, void** dst) { +int idxUidCompare(const void* a, const void* b) { + uint64_t l = *(uint64_t*)a; + uint64_t r = *(uint64_t*)b; + return l - r; +} +int32_t idxConvertData(void* src, int8_t type, void** dst) { int tlen = -1; switch (type) { case TSDB_DATA_TYPE_TIMESTAMP: @@ -368,44 +373,44 @@ int32_t indexConvertData(void* src, int8_t type, void** dst) { // indexMayFillNumbericData(*dst, tlen); return tlen; } -int32_t indexConvertDataToStr(void* src, int8_t type, void** dst) { +int32_t idxConvertDataToStr(void* src, int8_t type, void** dst) { int tlen = tDataTypes[type].bytes; int32_t bufSize = 64; switch (type) { case TSDB_DATA_TYPE_TIMESTAMP: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(int64_t*)src, *dst, -1); + idxInt2str(*(int64_t*)src, *dst, -1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_BOOL: case TSDB_DATA_TYPE_UTINYINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(uint8_t*)src, *dst, 1); + idxInt2str(*(uint8_t*)src, *dst, 1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_TINYINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(int8_t*)src, *dst, 1); + idxInt2str(*(int8_t*)src, *dst, 1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_SMALLINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(int16_t*)src, *dst, -1); + idxInt2str(*(int16_t*)src, *dst, -1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_USMALLINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(uint16_t*)src, *dst, -1); + idxInt2str(*(uint16_t*)src, *dst, -1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_INT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(int32_t*)src, *dst, -1); + idxInt2str(*(int32_t*)src, *dst, -1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_UINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(uint32_t*)src, *dst, 1); + idxInt2str(*(uint32_t*)src, *dst, 1); tlen = strlen(*dst); break; case TSDB_DATA_TYPE_BIGINT: @@ -415,7 +420,7 @@ int32_t indexConvertDataToStr(void* src, int8_t type, void** dst) { break; case TSDB_DATA_TYPE_UBIGINT: *dst = taosMemoryCalloc(1, bufSize + 1); - indexInt2str(*(uint64_t*)src, *dst, 1); + idxInt2str(*(uint64_t*)src, *dst, 1); tlen = strlen(*dst); case TSDB_DATA_TYPE_FLOAT: *dst = taosMemoryCalloc(1, bufSize + 1); diff --git a/source/libs/index/src/indexFilter.c b/source/libs/index/src/indexFilter.c index f3aa30018bf7b6f2c76f85834bdfff1e4a29c94b..e4af4a7a3f2e1e987781ea667d782f7b3f81662d 100644 --- a/source/libs/index/src/indexFilter.c +++ b/source/libs/index/src/indexFilter.c @@ -355,7 +355,7 @@ static int32_t sifDoIndex(SIFParam *left, SIFParam *right, int8_t operType, SIFP SIndexMultiTermQuery *mtm = indexMultiTermQueryCreate(MUST); indexMultiTermQueryAdd(mtm, tm, qtype); - ret = tIndexJsonSearch(arg->ivtIdx, mtm, output->result); + ret = indexJsonSearch(arg->ivtIdx, mtm, output->result); } else { bool reverse; Filter filterFunc = sifGetFilterFunc(qtype, &reverse); @@ -547,6 +547,8 @@ static int32_t sifExecLogic(SLogicConditionNode *node, SIFCtx *ctx, SIFParam *ou } else if (node->condType == LOGIC_COND_TYPE_NOT) { // taosArrayAddAll(output->result, params[m].result); } + taosArraySort(output->result, idxUidCompare); + taosArrayRemoveDuplicate(output->result, idxUidCompare, NULL); } } else { for (int32_t m = 0; m < node->pParameterList->length; m++) { diff --git a/source/libs/index/src/indexFst.c b/source/libs/index/src/indexFst.c index 892716f38708fed46bc755548436f2477d1e91e5..b368c6faf30cfa3a3833d24a7398f9610a033b1e 100644 --- a/source/libs/index/src/indexFst.c +++ b/source/libs/index/src/indexFst.c @@ -1087,19 +1087,19 @@ bool fstGet(Fst* fst, FstSlice* b, Output* out) { *out = tOut; return true; } -FstStreamBuilder* fstSearch(Fst* fst, AutomationCtx* ctx) { +FStmBuilder* fstSearch(Fst* fst, FAutoCtx* ctx) { // refactor later - return fstStreamBuilderCreate(fst, ctx); + return stmBuilderCreate(fst, ctx); } -StreamWithState* streamBuilderIntoStream(FstStreamBuilder* sb) { +FStmSt* stmBuilderIntoStm(FStmBuilder* sb) { if (sb == NULL) { return NULL; } - return streamWithStateCreate(sb->fst, sb->aut, sb->min, sb->max); + return stmStCreate(sb->fst, sb->aut, sb->min, sb->max); } -FstStreamWithStateBuilder* fstSearchWithState(Fst* fst, AutomationCtx* ctx) { +FStmStBuilder* fstSearchWithState(Fst* fst, FAutoCtx* ctx) { // refactor later - return fstStreamBuilderCreate(fst, ctx); + return stmBuilderCreate(fst, ctx); } FstNode* fstGetRoot(Fst* fst) { @@ -1176,9 +1176,8 @@ bool fstBoundWithDataIsIncluded(FstBoundWithData* bound) { return bound->type == void fstBoundDestroy(FstBoundWithData* bound) { taosMemoryFree(bound); } -StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstBoundWithData* min, - FstBoundWithData* max) { - StreamWithState* sws = taosMemoryCalloc(1, sizeof(StreamWithState)); +FStmSt* stmStCreate(Fst* fst, FAutoCtx* automation, FstBoundWithData* min, FstBoundWithData* max) { + FStmSt* sws = taosMemoryCalloc(1, sizeof(FStmSt)); if (sws == NULL) { return NULL; } @@ -1192,11 +1191,11 @@ StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstB sws->stack = (SArray*)taosArrayInit(256, sizeof(StreamState)); sws->endAt = max; - streamWithStateSeekMin(sws, min); + stmStSeekMin(sws, min); return sws; } -void streamWithStateDestroy(StreamWithState* sws) { +void stmStDestroy(FStmSt* sws) { if (sws == NULL) { return; } @@ -1207,8 +1206,8 @@ void streamWithStateDestroy(StreamWithState* sws) { taosMemoryFree(sws); } -bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min) { - AutomationCtx* aut = sws->aut; +bool stmStSeekMin(FStmSt* sws, FstBoundWithData* min) { + FAutoCtx* aut = sws->aut; if (fstBoundWithDataIsEmpty(min)) { if (fstBoundWithDataIsIncluded(min)) { sws->emptyOutput.out = fstEmptyFinalOutput(sws->fst, &(sws->emptyOutput.null)); @@ -1301,9 +1300,9 @@ bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min) { return false; } -StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallback callback) { - AutomationCtx* aut = sws->aut; - FstOutput output = sws->emptyOutput; +FStmStRslt* stmStNextWith(FStmSt* sws, StreamCallback callback) { + FAutoCtx* aut = sws->aut; + FstOutput output = sws->emptyOutput; if (output.null == false) { FstSlice emptySlice = fstSliceCreate(NULL, 0); if (fstBoundWithDataExceededBy(sws->endAt, &emptySlice)) { @@ -1367,8 +1366,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb return NULL; } if (FST_NODE_IS_FINAL(nextNode) && isMatch) { - FstOutput fOutput = {.null = false, .out = out + FST_NODE_FINAL_OUTPUT(nextNode)}; - StreamWithStateResult* result = swsResultCreate(&slice, fOutput, tState); + FstOutput fOutput = {.null = false, .out = out + FST_NODE_FINAL_OUTPUT(nextNode)}; + FStmStRslt* result = swsResultCreate(&slice, fOutput, tState); taosMemoryFreeClear(buf); fstSliceDestroy(&slice); taosArrayDestroy(nodes); @@ -1382,8 +1381,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb return NULL; } -StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* state) { - StreamWithStateResult* result = taosMemoryCalloc(1, sizeof(StreamWithStateResult)); +FStmStRslt* swsResultCreate(FstSlice* data, FstOutput fOut, void* state) { + FStmStRslt* result = taosMemoryCalloc(1, sizeof(FStmStRslt)); if (result == NULL) { return NULL; } @@ -1393,7 +1392,7 @@ StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* sta result->state = state; return result; } -void swsResultDestroy(StreamWithStateResult* result) { +void swsResultDestroy(FStmStRslt* result) { if (NULL == result) { return; } @@ -1411,8 +1410,8 @@ void streamStateDestroy(void* s) { fstNodeDestroy(ss->node); } -FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut) { - FstStreamBuilder* b = taosMemoryCalloc(1, sizeof(FstStreamBuilder)); +FStmBuilder* stmBuilderCreate(Fst* fst, FAutoCtx* aut) { + FStmBuilder* b = taosMemoryCalloc(1, sizeof(FStmBuilder)); if (NULL == b) { return NULL; } @@ -1423,14 +1422,14 @@ FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut) { b->max = fstBoundStateCreate(Unbounded, NULL); return b; } -void fstStreamBuilderDestroy(FstStreamBuilder* b) { +void stmBuilderDestroy(FStmBuilder* b) { fstSliceDestroy(&b->min->data); fstSliceDestroy(&b->max->data); taosMemoryFreeClear(b->min); taosMemoryFreeClear(b->max); taosMemoryFree(b); } -void fstStreamBuilderSetRange(FstStreamBuilder* b, FstSlice* val, RangeType type) { +void stmBuilderSetRange(FStmBuilder* b, FstSlice* val, RangeType type) { if (b == NULL) { return; } diff --git a/source/libs/index/src/indexFstAutomation.c b/source/libs/index/src/indexFstAutomation.c index 0702e3b94e52f2f1a552f2ed3ca337db86e50bd4..385e832763228494af2b1562f915da14b6500a25 100644 --- a/source/libs/index/src/indexFstAutomation.c +++ b/source/libs/index/src/indexFstAutomation.c @@ -71,19 +71,19 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv) { } // iterate fst -static void* alwaysMatchStart(AutomationCtx* ctx) { return NULL; } -static bool alwaysMatchIsMatch(AutomationCtx* ctx, void* state) { return true; } -static bool alwaysMatchCanMatch(AutomationCtx* ctx, void* state) { return true; } -static bool alwaysMatchWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } -static void* alwaysMatchAccpet(AutomationCtx* ctx, void* state, uint8_t byte) { return NULL; } -static void* alwaysMatchAccpetEof(AutomationCtx* ctx, void* state) { return NULL; } +static void* alwaysMatchStart(FAutoCtx* ctx) { return NULL; } +static bool alwaysMatchIsMatch(FAutoCtx* ctx, void* state) { return true; } +static bool alwaysMatchCanMatch(FAutoCtx* ctx, void* state) { return true; } +static bool alwaysMatchWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; } +static void* alwaysMatchAccpet(FAutoCtx* ctx, void* state, uint8_t byte) { return NULL; } +static void* alwaysMatchAccpetEof(FAutoCtx* ctx, void* state) { return NULL; } // prefix query, impl later -static void* prefixStart(AutomationCtx* ctx) { +static void* prefixStart(FAutoCtx* ctx) { StartWithStateValue* data = (StartWithStateValue*)(ctx->stdata); return startWithStateValueDump(data); }; -static bool prefixIsMatch(AutomationCtx* ctx, void* sv) { +static bool prefixIsMatch(FAutoCtx* ctx, void* sv) { StartWithStateValue* ssv = (StartWithStateValue*)sv; if (ssv == NULL) { return false; @@ -94,15 +94,15 @@ static bool prefixIsMatch(AutomationCtx* ctx, void* sv) { return false; } } -static bool prefixCanMatch(AutomationCtx* ctx, void* sv) { +static bool prefixCanMatch(FAutoCtx* ctx, void* sv) { StartWithStateValue* ssv = (StartWithStateValue*)sv; if (ssv == NULL) { return false; } return ssv->val >= 0; } -static bool prefixWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } -static void* prefixAccept(AutomationCtx* ctx, void* state, uint8_t byte) { +static bool prefixWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; } +static void* prefixAccept(FAutoCtx* ctx, void* state, uint8_t byte) { StartWithStateValue* ssv = (StartWithStateValue*)state; if (ssv == NULL || ctx == NULL) { return NULL; @@ -125,18 +125,18 @@ static void* prefixAccept(AutomationCtx* ctx, void* state, uint8_t byte) { } return NULL; } -static void* prefixAcceptEof(AutomationCtx* ctx, void* state) { return NULL; } +static void* prefixAcceptEof(FAutoCtx* ctx, void* state) { return NULL; } // pattern query, impl later -static void* patternStart(AutomationCtx* ctx) { return NULL; } -static bool patternIsMatch(AutomationCtx* ctx, void* data) { return true; } -static bool patternCanMatch(AutomationCtx* ctx, void* data) { return true; } -static bool patternWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } +static void* patternStart(FAutoCtx* ctx) { return NULL; } +static bool patternIsMatch(FAutoCtx* ctx, void* data) { return true; } +static bool patternCanMatch(FAutoCtx* ctx, void* data) { return true; } +static bool patternWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; } -static void* patternAccept(AutomationCtx* ctx, void* state, uint8_t byte) { return NULL; } +static void* patternAccept(FAutoCtx* ctx, void* state, uint8_t byte) { return NULL; } -static void* patternAcceptEof(AutomationCtx* ctx, void* state) { return NULL; } +static void* patternAcceptEof(FAutoCtx* ctx, void* state) { return NULL; } AutomationFunc automFuncs[] = { {alwaysMatchStart, alwaysMatchIsMatch, alwaysMatchCanMatch, alwaysMatchWillAlwaysMatch, alwaysMatchAccpet, @@ -146,8 +146,8 @@ AutomationFunc automFuncs[] = { // add more search type }; -AutomationCtx* automCtxCreate(void* data, AutomationType atype) { - AutomationCtx* ctx = taosMemoryCalloc(1, sizeof(AutomationCtx)); +FAutoCtx* automCtxCreate(void* data, AutomationType atype) { + FAutoCtx* ctx = taosMemoryCalloc(1, sizeof(FAutoCtx)); if (ctx == NULL) { return NULL; } @@ -169,7 +169,7 @@ AutomationCtx* automCtxCreate(void* data, AutomationType atype) { ctx->stdata = (void*)sv; return ctx; } -void automCtxDestroy(AutomationCtx* ctx) { +void automCtxDestroy(FAutoCtx* ctx) { startWithStateValueDestroy(ctx->stdata); taosMemoryFree(ctx->data); taosMemoryFree(ctx); diff --git a/source/libs/index/src/indexJson.c b/source/libs/index/src/indexJson.c index a4f3be382984ce2c4359b79a2bd96b6df204bbe8..88b3d907bb1989679d9ae2e9871a5f955a85145c 100644 --- a/source/libs/index/src/indexJson.c +++ b/source/libs/index/src/indexJson.c @@ -15,11 +15,11 @@ #include "index.h" #include "indexInt.h" -int tIndexJsonOpen(SIndexJsonOpts *opts, const char *path, SIndexJson **index) { +int indexJsonOpen(SIndexJsonOpts *opts, const char *path, SIndexJson **index) { // handle return indexOpen(opts, path, index); } -int tIndexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) { +int indexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) { for (int i = 0; i < taosArrayGetSize(terms); i++) { SIndexJsonTerm *p = taosArrayGetP(terms, i); if (p->colType == TSDB_DATA_TYPE_BOOL) { @@ -36,7 +36,7 @@ int tIndexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) { return indexPut(index, terms, uid); } -int tIndexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *result) { +int indexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *result) { SArray *terms = tq->query; for (int i = 0; i < taosArrayGetSize(terms); i++) { SIndexJsonTerm *p = taosArrayGetP(terms, i); @@ -54,7 +54,7 @@ int tIndexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *re return indexSearch(index, tq, result); } -void tIndexJsonClose(SIndexJson *index) { +void indexJsonClose(SIndexJson *index) { // handle close return indexClose(index); } diff --git a/source/libs/index/src/indexTfile.c b/source/libs/index/src/indexTfile.c index c2747a7e3032767d2f19e0337b70ae7f6b2ffeef..b64db1dde405e5116eb271df3e633726850553d9 100644 --- a/source/libs/index/src/indexTfile.c +++ b/source/libs/index/src/indexTfile.c @@ -26,15 +26,14 @@ const static uint64_t tfileMagicNumber = 0xdb4775248b80fb57ull; typedef struct TFileFstIter { - FstStreamBuilder* fb; - StreamWithState* st; - AutomationCtx* ctx; - TFileReader* rdr; + FStmBuilder* fb; + FStmSt* st; + FAutoCtx* ctx; + TFileReader* rdr; } TFileFstIter; #define TF_TABLE_TATOAL_SIZE(sz) (sizeof(sz) + sz * sizeof(uint64_t)) -static int tfileUidCompare(const void* a, const void* b); static int tfileStrCompare(const void* a, const void* b); static int tfileValueCompare(const void* a, const void* b, const void* param); static void tfileSerialTableIdsToBuf(char* buf, SArray* tableIds); @@ -245,26 +244,21 @@ static int32_t tfSearchTerm(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { } static int32_t tfSearchPrefix(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { - bool hasJson = INDEX_TYPE_CONTAIN_EXTERN_TYPE(tem->colType, TSDB_DATA_TYPE_JSON); char* p = tem->colVal; uint64_t sz = tem->nColVal; - if (hasJson) { - p = indexPackJsonData(tem); - sz = strlen(p); - } SArray* offsets = taosArrayInit(16, sizeof(uint64_t)); - AutomationCtx* ctx = automCtxCreate((void*)p, AUTOMATION_PREFIX); - FstStreamBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + FAutoCtx* ctx = automCtxCreate((void*)p, AUTOMATION_PREFIX); + FStmBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { taosArrayPush(offsets, &(rt->out.out)); swsResultDestroy(rt); } - streamWithStateDestroy(st); - fstStreamBuilderDestroy(sb); + stmStDestroy(st); + stmBuilderDestroy(sb); int32_t ret = 0; for (int i = 0; i < taosArrayGetSize(offsets); i++) { @@ -275,27 +269,14 @@ static int32_t tfSearchPrefix(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { return TSDB_CODE_TDB_FILE_CORRUPTED; } } - if (hasJson) { - taosMemoryFree(p); - } return 0; } static int32_t tfSearchSuffix(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { - bool hasJson = INDEX_TYPE_CONTAIN_EXTERN_TYPE(tem->colType, TSDB_DATA_TYPE_JSON); - int ret = 0; char* p = tem->colVal; uint64_t sz = tem->nColVal; - if (hasJson) { - p = indexPackJsonData(tem); - sz = strlen(p); - } int64_t st = taosGetTimestampUs(); FstSlice key = fstSliceCreate(p, sz); - /*impl later*/ - if (hasJson) { - taosMemoryFree(p); - } fstSliceDestroy(&key); return 0; } @@ -306,7 +287,7 @@ static int32_t tfSearchRegex(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { char* p = tem->colVal; uint64_t sz = tem->nColVal; if (hasJson) { - p = indexPackJsonData(tem); + p = idxPackJsonData(tem); sz = strlen(p); } int64_t st = taosGetTimestampUs(); @@ -328,16 +309,16 @@ static int32_t tfSearchCompareFunc(void* reader, SIndexTerm* tem, SIdxTRslt* tr, SArray* offsets = taosArrayInit(16, sizeof(uint64_t)); - AutomationCtx* ctx = automCtxCreate((void*)p, AUTOMATION_ALWAYS); - FstStreamBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); + FAutoCtx* ctx = automCtxCreate((void*)p, AUTOMATION_ALWAYS); + FStmBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); FstSlice h = fstSliceCreate((uint8_t*)p, skip); - fstStreamBuilderSetRange(sb, &h, type); + stmBuilderSetRange(sb, &h, type); fstSliceDestroy(&h); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { FstSlice* s = &rt->data; char* ch = (char*)fstSliceData(s, NULL); // if (0 != strncmp(ch, tem->colName, tem->nColName)) { @@ -355,8 +336,8 @@ static int32_t tfSearchCompareFunc(void* reader, SIndexTerm* tem, SIdxTRslt* tr, } swsResultDestroy(rt); } - streamWithStateDestroy(st); - fstStreamBuilderDestroy(sb); + stmStDestroy(st); + stmBuilderDestroy(sb); return TSDB_CODE_SUCCESS; } static int32_t tfSearchLessThan(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { @@ -372,37 +353,17 @@ static int32_t tfSearchGreaterEqual(void* reader, SIndexTerm* tem, SIdxTRslt* tr return tfSearchCompareFunc(reader, tem, tr, GE); } static int32_t tfSearchRange(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { - bool hasJson = INDEX_TYPE_CONTAIN_EXTERN_TYPE(tem->colType, TSDB_DATA_TYPE_JSON); int ret = 0; char* p = tem->colVal; uint64_t sz = tem->nColVal; - if (hasJson) { - p = indexPackJsonData(tem); - sz = strlen(p); - } int64_t st = taosGetTimestampUs(); FstSlice key = fstSliceCreate(p, sz); - // uint64_t offset; - // if (fstGet(((TFileReader*)reader)->fst, &key, &offset)) { - // int64_t et = taosGetTimestampUs(); - // int64_t cost = et - st; - // indexInfo("index: %" PRIu64 ", col: %s, colVal: %s, found table info in tindex, time cost: %" PRIu64 "us", - // tem->suid, tem->colName, tem->colVal, cost); - - // ret = tfileReaderLoadTableIds((TFileReader*)reader, offset, tr->total); - // cost = taosGetTimestampUs() - et; - // indexInfo("index: %" PRIu64 ", col: %s, colVal: %s, load all table info, time cost: %" PRIu64 "us", tem->suid, - // tem->colName, tem->colVal, cost); - //} - if (hasJson) { - taosMemoryFree(p); - } fstSliceDestroy(&key); return 0; } static int32_t tfSearchTerm_JSON(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { int ret = 0; - char* p = indexPackJsonData(tem); + char* p = idxPackJsonData(tem); int sz = strlen(p); int64_t st = taosGetTimestampUs(); @@ -422,8 +383,6 @@ static int32_t tfSearchTerm_JSON(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { } fstSliceDestroy(&key); return 0; - // deprecate api - return TSDB_CODE_SUCCESS; } static int32_t tfSearchEqual_JSON(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { return tfSearchCompareFunc_JSON(reader, tem, tr, EQ); @@ -467,21 +426,21 @@ static int32_t tfSearchCompareFunc_JSON(void* reader, SIndexTerm* tem, SIdxTRslt .colType = tem->colType, .colName = tem->colVal, .nColName = tem->nColVal}; - p = indexPackJsonDataPrefixNoType(&tm, &skip); + p = idxPackJsonDataPrefixNoType(&tm, &skip); } else { - p = indexPackJsonDataPrefix(tem, &skip); + p = idxPackJsonDataPrefix(tem, &skip); } _cache_range_compare cmpFn = indexGetCompare(ctype); SArray* offsets = taosArrayInit(16, sizeof(uint64_t)); - AutomationCtx* ctx = automCtxCreate((void*)p, AUTOMATION_PREFIX); - FstStreamBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); + FAutoCtx* ctx = automCtxCreate((void*)p, AUTOMATION_PREFIX); + FStmBuilder* sb = fstSearch(((TFileReader*)reader)->fst, ctx); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { FstSlice* s = &rt->data; int32_t sz = 0; @@ -509,8 +468,8 @@ static int32_t tfSearchCompareFunc_JSON(void* reader, SIndexTerm* tem, SIdxTRslt } swsResultDestroy(rt); } - streamWithStateDestroy(st); - fstStreamBuilderDestroy(sb); + stmStDestroy(st); + stmBuilderDestroy(sb); return TSDB_CODE_SUCCESS; } int tfileReaderSearch(TFileReader* reader, SIndexTermQuery* query, SIdxTRslt* tr) { @@ -595,8 +554,8 @@ int tfileWriterPut(TFileWriter* tw, void* data, bool order) { // ugly code, refactor later for (size_t i = 0; i < sz; i++) { TFileValue* v = taosArrayGetP((SArray*)data, i); - taosArraySort(v->tableId, tfileUidCompare); - taosArrayRemoveDuplicate(v->tableId, tfileUidCompare, NULL); + taosArraySort(v->tableId, idxUidCompare); + taosArrayRemoveDuplicate(v->tableId, idxUidCompare, NULL); int32_t tbsz = taosArrayGetSize(v->tableId); fstOffset += TF_TABLE_TATOAL_SIZE(tbsz); } @@ -721,8 +680,8 @@ static bool tfileIteratorNext(Iterate* iiter) { char* colVal = NULL; uint64_t offset = 0; - TFileFstIter* tIter = iiter->iter; - StreamWithStateResult* rt = streamWithStateNextWith(tIter->st, NULL); + TFileFstIter* tIter = iiter->iter; + FStmStRslt* rt = stmStNextWith(tIter->st, NULL); if (rt == NULL) { return false; } @@ -756,7 +715,7 @@ static TFileFstIter* tfileFstIteratorCreate(TFileReader* reader) { iter->ctx = automCtxCreate(NULL, AUTOMATION_ALWAYS); iter->fb = fstSearch(reader->fst, iter->ctx); - iter->st = streamBuilderIntoStream(iter->fb); + iter->st = stmBuilderIntoStm(iter->fb); iter->rdr = reader; return iter; } @@ -787,8 +746,8 @@ void tfileIteratorDestroy(Iterate* iter) { iterateValueDestroy(iv, true); TFileFstIter* tIter = iter->iter; - streamWithStateDestroy(tIter->st); - fstStreamBuilderDestroy(tIter->fb); + stmStDestroy(tIter->st); + stmBuilderDestroy(tIter->fb); automCtxDestroy(tIter->ctx); taosMemoryFree(tIter); @@ -808,11 +767,6 @@ TFileReader* tfileGetReaderByCol(IndexTFile* tf, uint64_t suid, char* colName) { return rd; } -static int tfileUidCompare(const void* a, const void* b) { - uint64_t l = *(uint64_t*)a; - uint64_t r = *(uint64_t*)b; - return l - r; -} static int tfileStrCompare(const void* a, const void* b) { int ret = strcmp((char*)a, (char*)b); if (ret == 0) { diff --git a/source/libs/index/test/fstTest.cc b/source/libs/index/test/fstTest.cc index a2d7adf1c7fc6e9ea2c1996206280f953939fd65..332b7370df59db3aa21c44a83f1ee06c0250186d 100644 --- a/source/libs/index/test/fstTest.cc +++ b/source/libs/index/test/fstTest.cc @@ -85,11 +85,11 @@ class FstReadMemory { return ok; } // add later - bool Search(AutomationCtx* ctx, std::vector& result) { - FstStreamBuilder* sb = fstSearch(_fst, ctx); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + bool Search(FAutoCtx* ctx, std::vector& result) { + FStmBuilder* sb = fstSearch(_fst, ctx); + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { // result.push_back((uint64_t)(rt->out.out)); FstSlice* s = &rt->data; int32_t sz = 0; @@ -99,27 +99,27 @@ class FstReadMemory { result.push_back(rt->out.out); swsResultDestroy(rt); } - streamWithStateDestroy(st); - fstStreamBuilderDestroy(sb); + stmStDestroy(st); + stmBuilderDestroy(sb); return true; } - bool SearchRange(AutomationCtx* ctx, const std::string& low, RangeType lowType, const std::string& high, + bool SearchRange(FAutoCtx* ctx, const std::string& low, RangeType lowType, const std::string& high, RangeType highType, std::vector& result) { - FstStreamBuilder* sb = fstSearch(_fst, ctx); + FStmBuilder* sb = fstSearch(_fst, ctx); FstSlice l = fstSliceCreate((uint8_t*)low.c_str(), low.size()); FstSlice h = fstSliceCreate((uint8_t*)high.c_str(), high.size()); // range [low, high); - fstStreamBuilderSetRange(sb, &l, lowType); - fstStreamBuilderSetRange(sb, &h, highType); + stmBuilderSetRange(sb, &l, lowType); + stmBuilderSetRange(sb, &h, highType); fstSliceDestroy(&l); fstSliceDestroy(&h); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { // result.push_back((uint64_t)(rt->out.out)); FstSlice* s = &rt->data; int32_t sz = 0; @@ -129,11 +129,11 @@ class FstReadMemory { result.push_back(rt->out.out); swsResultDestroy(rt); } - streamWithStateDestroy(st); - fstStreamBuilderDestroy(sb); + stmStDestroy(st); + stmBuilderDestroy(sb); return true; } - bool SearchWithTimeCostUs(AutomationCtx* ctx, std::vector& result) { + bool SearchWithTimeCostUs(FAutoCtx* ctx, std::vector& result) { int64_t s = taosGetTimestampUs(); bool ok = this->Search(ctx, result); int64_t e = taosGetTimestampUs(); @@ -253,7 +253,7 @@ void checkFstLongTerm() { // prefix search // std::vector result; - // AutomationCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS); + // FAutoCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS); // m->Search(ctx, result); // std::cout << "size: " << result.size() << std::endl; // assert(result.size() == count); @@ -286,7 +286,7 @@ void checkFstCheckIterator1() { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); m->Search(ctx, result); std::cout << "size: " << result.size() << std::endl; // assert(result.size() == count); @@ -321,7 +321,7 @@ void checkFstCheckIterator2() { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); m->Search(ctx, result); std::cout << "size: " << result.size() << std::endl; // assert(result.size() == count); @@ -361,7 +361,7 @@ void checkFstCheckIteratorPrefix() { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_PREFIX); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_PREFIX); m->Search(ctx, result); assert(result.size() == 1); automCtxDestroy(ctx); @@ -370,7 +370,7 @@ void checkFstCheckIteratorPrefix() { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"Hello", AUTOMATION_PREFIX); + FAutoCtx* ctx = automCtxCreate((void*)"Hello", AUTOMATION_PREFIX); m->Search(ctx, result); assert(result.size() == 2); automCtxDestroy(ctx); @@ -378,7 +378,7 @@ void checkFstCheckIteratorPrefix() { { std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"jddd", AUTOMATION_PREFIX); + FAutoCtx* ctx = automCtxCreate((void*)"jddd", AUTOMATION_PREFIX); m->Search(ctx, result); assert(result.size() == 1); automCtxDestroy(ctx); @@ -412,7 +412,7 @@ void checkFstCheckIteratorRange1() { { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GE, "e", LT, result); assert(result.size() == 3); @@ -421,7 +421,7 @@ void checkFstCheckIteratorRange1() { { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GT, "e", LT, result); assert(result.size() == 2); @@ -430,7 +430,7 @@ void checkFstCheckIteratorRange1() { { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GT, "e", LE, result); assert(result.size() == 3); @@ -439,7 +439,7 @@ void checkFstCheckIteratorRange1() { { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GE, "e", LE, result); assert(result.size() == 4); @@ -473,7 +473,7 @@ void checkFstCheckIteratorRange2() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GE, "ed", LT, result); assert(result.size() == 4); @@ -482,7 +482,7 @@ void checkFstCheckIteratorRange2() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "bb", GE, "ed", LT, result); assert(result.size() == 3); @@ -491,7 +491,7 @@ void checkFstCheckIteratorRange2() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GE, "ed", LE, result); assert(result.size() == 5); @@ -501,7 +501,7 @@ void checkFstCheckIteratorRange2() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GT, "ed", LE, result); assert(result.size() == 4); @@ -510,7 +510,7 @@ void checkFstCheckIteratorRange2() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GT, "ed", LT, result); assert(result.size() == 3); @@ -544,7 +544,7 @@ void checkFstCheckIteratorRange3() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "b", GE, "", (RangeType)10, result); assert(result.size() == 5); @@ -553,7 +553,7 @@ void checkFstCheckIteratorRange3() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "", (RangeType)20, "ab", LE, result); assert(result.size() == 1); @@ -563,7 +563,7 @@ void checkFstCheckIteratorRange3() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "", (RangeType)30, "ab", LT, result); assert(result.size() == 0); @@ -572,7 +572,7 @@ void checkFstCheckIteratorRange3() { { // range search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); // [b, e) m->SearchRange(ctx, "ed", GT, "ed", (RangeType)40, result); assert(result.size() == 0); diff --git a/source/libs/index/test/fstUT.cc b/source/libs/index/test/fstUT.cc index 136c4dafecdceba665d4a8657ed2053fa2486675..b9388e62f7dc0f979c930f92d44edcf17c767a87 100644 --- a/source/libs/index/test/fstUT.cc +++ b/source/libs/index/test/fstUT.cc @@ -106,11 +106,11 @@ class FstReadMemory { return ok; } // add later - bool Search(AutomationCtx* ctx, std::vector& result) { - FstStreamBuilder* sb = fstSearch(_fst, ctx); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + bool Search(FAutoCtx* ctx, std::vector& result) { + FStmBuilder* sb = fstSearch(_fst, ctx); + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; + while ((rt = stmStNextWith(st, NULL)) != NULL) { // result.push_back((uint64_t)(rt->out.out)); FstSlice* s = &rt->data; int32_t sz = 0; @@ -122,7 +122,7 @@ class FstReadMemory { std::cout << std::endl; return true; } - bool SearchWithTimeCostUs(AutomationCtx* ctx, std::vector& result) { + bool SearchWithTimeCostUs(FAutoCtx* ctx, std::vector& result) { int64_t s = taosGetTimestampUs(); bool ok = this->Search(ctx, result); int64_t e = taosGetTimestampUs(); @@ -192,7 +192,7 @@ class TFst { } return fr->Get(k, v); } - bool Search(AutomationCtx* ctx, std::vector& result) { + bool Search(FAutoCtx* ctx, std::vector& result) { // add more return fr->Search(ctx, result); } @@ -231,7 +231,7 @@ TEST_F(FstEnv, writeNormal) { assert(val == 0); std::vector rlt; - AutomationCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS); + FAutoCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS); assert(fst->Search(ctx, rlt) == true); } TEST_F(FstEnv, WriteMillonrRecord) {} diff --git a/source/libs/index/test/indexTests.cc b/source/libs/index/test/indexTests.cc index 327a4ae219322a9d4a773001cde7f13397d75e4d..90dea3a377d24f0f166cc731cb07b8ce1d404b3d 100644 --- a/source/libs/index/test/indexTests.cc +++ b/source/libs/index/test/indexTests.cc @@ -106,17 +106,17 @@ class FstReadMemory { return ok; } // add later - bool Search(AutomationCtx* ctx, std::vector& result) { - FstStreamBuilder* sb = fstSearch(_fst, ctx); - StreamWithState* st = streamBuilderIntoStream(sb); - StreamWithStateResult* rt = NULL; + bool Search(FAutoCtx* ctx, std::vector& result) { + FStmBuilder* sb = fstSearch(_fst, ctx); + FStmSt* st = stmBuilderIntoStm(sb); + FStmStRslt* rt = NULL; - while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { + while ((rt = stmStNextWith(st, NULL)) != NULL) { result.push_back((uint64_t)(rt->out.out)); } return true; } - bool SearchWithTimeCostUs(AutomationCtx* ctx, std::vector& result) { + bool SearchWithTimeCostUs(FAutoCtx* ctx, std::vector& result) { int64_t s = taosGetTimestampUs(); bool ok = this->Search(ctx, result); int64_t e = taosGetTimestampUs(); @@ -220,7 +220,7 @@ void checkFstPrefixSearch() { // prefix search std::vector result; - AutomationCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_PREFIX); + FAutoCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_PREFIX); m->Search(ctx, result); assert(result.size() == count); for (int i = 0; i < result.size(); i++) { diff --git a/source/libs/index/test/jsonUT.cc b/source/libs/index/test/jsonUT.cc index d6bc31a8017c36fb9e6a6cbf535b3c00360c9179..134fb53d2b5dcf57f39a6c3ec098592756bcb02d 100644 --- a/source/libs/index/test/jsonUT.cc +++ b/source/libs/index/test/jsonUT.cc @@ -56,11 +56,11 @@ class JsonEnv : public ::testing::Test { initLog(); opts = indexOptsCreate(); - int ret = tIndexJsonOpen(opts, dir.c_str(), &index); + int ret = indexJsonOpen(opts, dir.c_str(), &index); assert(ret == 0); } virtual void TearDown() { - tIndexJsonClose(index); + indexJsonClose(index); indexOptsDestroy(opts); printf("destory\n"); taosMsleep(1000); @@ -75,7 +75,7 @@ static void WriteData(SIndexJson* index, const std::string& colName, int8_t dtyp (const char*)data, dlen); SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, (int64_t)tableId); + indexJsonPut(index, terms, (int64_t)tableId); indexMultiTermDestroy(terms); } @@ -86,7 +86,7 @@ static void delData(SIndexJson* index, const std::string& colName, int8_t dtype, (const char*)data, dlen); SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, (int64_t)tableId); + indexJsonPut(index, terms, (int64_t)tableId); indexMultiTermDestroy(terms); } @@ -99,7 +99,7 @@ static void Search(SIndexJson* index, const std::string& colNam, int8_t dtype, v SArray* res = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, (EIndexQueryType)filterType); - tIndexJsonSearch(index, mq, res); + indexJsonSearch(index, mq, res); indexMultiTermQueryDestroy(mq); *result = res; } @@ -112,7 +112,7 @@ TEST_F(JsonEnv, testWrite) { colVal.c_str(), colVal.size()); SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -125,7 +125,7 @@ TEST_F(JsonEnv, testWrite) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -138,7 +138,7 @@ TEST_F(JsonEnv, testWrite) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -152,7 +152,7 @@ TEST_F(JsonEnv, testWrite) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_TERM); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(100, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -167,7 +167,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -182,7 +182,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -196,7 +196,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -210,7 +210,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_TERM); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(10, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -225,7 +225,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -240,7 +240,7 @@ TEST_F(JsonEnv, testWriteMillonData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(10, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -258,7 +258,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -271,7 +271,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -284,7 +284,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -297,7 +297,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -310,7 +310,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_TERM); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -324,7 +324,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -339,7 +339,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -354,7 +354,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_LESS_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -369,7 +369,7 @@ TEST_F(JsonEnv, testWriteJsonNumberData) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_LESS_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -385,7 +385,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -398,7 +398,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -412,7 +412,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_TERM); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -426,7 +426,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -441,7 +441,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -455,7 +455,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -469,7 +469,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_LESS_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(1000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -483,7 +483,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i); + indexJsonPut(index, terms, i); indexMultiTermDestroy(terms); } } @@ -498,7 +498,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_LESS_THAN); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(0, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } @@ -511,7 +511,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SIndexMultiTerm* terms = indexMultiTermCreate(); indexMultiTermAdd(terms, term); - tIndexJsonPut(index, terms, i + 1000); + indexJsonPut(index, terms, i + 1000); indexMultiTermDestroy(terms); } } @@ -526,7 +526,7 @@ TEST_F(JsonEnv, testWriteJsonTfileAndCache_INT) { SArray* result = taosArrayInit(1, sizeof(uint64_t)); indexMultiTermQueryAdd(mq, q, QUERY_GREATER_EQUAL); - tIndexJsonSearch(index, mq, result); + indexJsonSearch(index, mq, result); EXPECT_EQ(2000, taosArrayGetSize(result)); indexMultiTermQueryDestroy(mq); } diff --git a/source/libs/index/test/utilUT.cc b/source/libs/index/test/utilUT.cc index 4a30160244d82b8c00b3e7b031d6fd492057ec21..6858d313254b661fdc151f095cf9f6f07242c24c 100644 --- a/source/libs/index/test/utilUT.cc +++ b/source/libs/index/test/utilUT.cc @@ -327,13 +327,13 @@ TEST_F(UtilEnv, testFill) { for (int i = 0; i < 1000000; i++) { int64_t val = i; char buf[65] = {0}; - indexInt2str(val, buf, 1); + idxInt2str(val, buf, 1); EXPECT_EQ(val, taosStr2int64(buf)); } for (int i = 0; i < 1000000; i++) { int64_t val = 0 - i; char buf[65] = {0}; - indexInt2str(val, buf, -1); + idxInt2str(val, buf, -1); EXPECT_EQ(val, taosStr2int64(buf)); } } diff --git a/source/libs/parser/src/parAstParser.c b/source/libs/parser/src/parAstParser.c index c79820a950bb675d2a66602b6042855be311ec28..6555ec3a7dd30a108f4dba53a4110be775e73eb2 100644 --- a/source/libs/parser/src/parAstParser.c +++ b/source/libs/parser/src/parAstParser.c @@ -108,9 +108,10 @@ static int32_t collectMetaKeyFromQuery(SCollectMetaKeyCxt* pCxt, SNode* pStmt); static EDealRes collectMetaKeyFromFunction(SCollectMetaKeyFromExprCxt* pCxt, SFunctionNode* pFunc) { if (fmIsBuiltinFunc(pFunc->functionName)) { - return TSDB_CODE_SUCCESS; + return DEAL_RES_CONTINUE; } - return reserveUdfInCache(pFunc->functionName, pCxt->pComCxt->pMetaCache); + pCxt->errCode = reserveUdfInCache(pFunc->functionName, pCxt->pComCxt->pMetaCache); + return TSDB_CODE_SUCCESS == pCxt->errCode ? DEAL_RES_CONTINUE : DEAL_RES_ERROR; } static int32_t collectMetaKeyFromRealTableImpl(SCollectMetaKeyCxt* pCxt, SRealTableNode* pRealTable, @@ -179,6 +180,10 @@ static int32_t collectMetaKeyFromSelect(SCollectMetaKeyCxt* pCxt, SSelectStmt* p return cxt.errCode; } +static int32_t collectMetaKeyFromAlterDatabase(SCollectMetaKeyCxt* pCxt, SAlterDatabaseStmt* pStmt) { + return reserveDbCfgInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pCxt->pMetaCache); +} + static int32_t collectMetaKeyFromCreateTable(SCollectMetaKeyCxt* pCxt, SCreateTableStmt* pStmt) { int32_t code = reserveDbCfgInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pCxt->pMetaCache); if (TSDB_CODE_SUCCESS == code && NULL == pStmt->pTags) { @@ -376,6 +381,8 @@ static int32_t collectMetaKeyFromQuery(SCollectMetaKeyCxt* pCxt, SNode* pStmt) { return collectMetaKeyFromSetOperator(pCxt, (SSetOperator*)pStmt); case QUERY_NODE_SELECT_STMT: return collectMetaKeyFromSelect(pCxt, (SSelectStmt*)pStmt); + case QUERY_NODE_ALTER_DATABASE_STMT: + return collectMetaKeyFromAlterDatabase(pCxt, (SAlterDatabaseStmt*)pStmt); case QUERY_NODE_CREATE_TABLE_STMT: return collectMetaKeyFromCreateTable(pCxt, (SCreateTableStmt*)pStmt); case QUERY_NODE_CREATE_MULTI_TABLE_STMT: diff --git a/source/libs/parser/src/parInsert.c b/source/libs/parser/src/parInsert.c index 8f17d500abd6984cfa896f860d64a50a29fea446..54c8a1821809ffe867addf788027b54123c1a989 100644 --- a/source/libs/parser/src/parInsert.c +++ b/source/libs/parser/src/parInsert.c @@ -1006,7 +1006,7 @@ static int32_t parseTagsClause(SInsertParseContext* pCxt, SSchema* pSchema, uint } SSchema* pTagSchema = &pSchema[pCxt->tags.boundColumns[i]]; - char* tmpTokenBuf = taosMemoryCalloc(1, sToken.n); //todo this can be optimize with parse column + char* tmpTokenBuf = taosMemoryCalloc(1, sToken.n); // todo this can be optimize with parse column code = checkAndTrimValue(&sToken, tmpTokenBuf, &pCxt->msg); if (code != TSDB_CODE_SUCCESS) { taosMemoryFree(tmpTokenBuf); @@ -1018,7 +1018,7 @@ static int32_t parseTagsClause(SInsertParseContext* pCxt, SSchema* pSchema, uint taosMemoryFree(tmpTokenBuf); goto end; } - if(isNullStr(&sToken)) { + if (isNullStr(&sToken)) { code = tTagNew(pTagVals, 1, true, &pTag); } else { code = parseJsontoTagData(sToken.z, pTagVals, &pTag, &pCxt->msg); @@ -1530,10 +1530,13 @@ typedef struct SInsertParseSyntaxCxt { } SInsertParseSyntaxCxt; static int32_t skipParentheses(SInsertParseSyntaxCxt* pCxt) { - SToken sToken; + SToken sToken; + int32_t expectRightParenthesis = 1; while (1) { NEXT_TOKEN(pCxt->pSql, sToken); - if (TK_NK_RP == sToken.type) { + if (TK_NK_LP == sToken.type) { + ++expectRightParenthesis; + } else if (TK_NK_RP == sToken.type && 0 == --expectRightParenthesis) { break; } if (0 == sToken.n) { @@ -2040,6 +2043,10 @@ int32_t qBuildStmtTagFields(void* pBlock, void* boundTags, int32_t* fieldNum, TA return TSDB_CODE_QRY_APP_ERROR; } + if (pDataBlock->pTableMeta->tableType != TSDB_SUPER_TABLE && pDataBlock->pTableMeta->tableType != TSDB_CHILD_TABLE) { + return TSDB_CODE_TSC_STMT_API_ERROR; + } + SSchema* pSchema = getTableTagSchema(pDataBlock->pTableMeta); if (tags->numOfBound <= 0) { *fieldNum = 0; diff --git a/source/libs/parser/src/parTranslater.c b/source/libs/parser/src/parTranslater.c index f77bbb34f329a5bfabdcf21144338c8ff8e09ea4..96a5c09bbff918d26c725aada44591297704f54f 100644 --- a/source/libs/parser/src/parTranslater.c +++ b/source/libs/parser/src/parTranslater.c @@ -103,15 +103,14 @@ static int32_t collectUseTable(const SName* pName, SHashObj* pDbs) { static int32_t getTableMetaImpl(STranslateContext* pCxt, const SName* pName, STableMeta** pMeta) { SParseContext* pParCxt = pCxt->pParseCxt; - int32_t code = TSDB_CODE_SUCCESS; - if (pParCxt->async) { - code = getTableMetaFromCache(pCxt->pMetaCache, pName, pMeta); - } else { - code = collectUseDatabase(pName, pCxt->pDbs); - if (TSDB_CODE_SUCCESS == code) { - code = collectUseTable(pName, pCxt->pTables); - } - if (TSDB_CODE_SUCCESS == code) { + int32_t code = collectUseDatabase(pName, pCxt->pDbs); + if (TSDB_CODE_SUCCESS == code) { + code = collectUseTable(pName, pCxt->pTables); + } + if (TSDB_CODE_SUCCESS == code) { + if (pParCxt->async) { + code = getTableMetaFromCache(pCxt->pMetaCache, pName, pMeta); + } else { code = catalogGetTableMeta(pParCxt->pCatalog, pParCxt->pTransporter, &pParCxt->mgmtEpSet, pName, pMeta); } } @@ -150,12 +149,11 @@ static int32_t getDBVgInfoImpl(STranslateContext* pCxt, const SName* pName, SArr SParseContext* pParCxt = pCxt->pParseCxt; char fullDbName[TSDB_DB_FNAME_LEN]; tNameGetFullDbName(pName, fullDbName); - int32_t code = TSDB_CODE_SUCCESS; - if (pParCxt->async) { - code = getDbVgInfoFromCache(pCxt->pMetaCache, fullDbName, pVgInfo); - } else { - code = collectUseDatabaseImpl(fullDbName, pCxt->pDbs); - if (TSDB_CODE_SUCCESS == code) { + int32_t code = collectUseDatabaseImpl(fullDbName, pCxt->pDbs); + if (TSDB_CODE_SUCCESS == code) { + if (pParCxt->async) { + code = getDbVgInfoFromCache(pCxt->pMetaCache, fullDbName, pVgInfo); + } else { code = catalogGetDBVgInfo(pParCxt->pCatalog, pParCxt->pTransporter, &pParCxt->mgmtEpSet, fullDbName, pVgInfo); } } @@ -175,15 +173,14 @@ static int32_t getDBVgInfo(STranslateContext* pCxt, const char* pDbName, SArray* static int32_t getTableHashVgroupImpl(STranslateContext* pCxt, const SName* pName, SVgroupInfo* pInfo) { SParseContext* pParCxt = pCxt->pParseCxt; - int32_t code = TSDB_CODE_SUCCESS; - if (pParCxt->async) { - code = getTableVgroupFromCache(pCxt->pMetaCache, pName, pInfo); - } else { - code = collectUseDatabase(pName, pCxt->pDbs); - if (TSDB_CODE_SUCCESS == code) { - code = collectUseTable(pName, pCxt->pTables); - } - if (TSDB_CODE_SUCCESS == code) { + int32_t code = collectUseDatabase(pName, pCxt->pDbs); + if (TSDB_CODE_SUCCESS == code) { + code = collectUseTable(pName, pCxt->pTables); + } + if (TSDB_CODE_SUCCESS == code) { + if (pParCxt->async) { + code = getTableVgroupFromCache(pCxt->pMetaCache, pName, pInfo); + } else { code = catalogGetTableHashVgroup(pParCxt->pCatalog, pParCxt->pTransporter, &pParCxt->mgmtEpSet, pName, pInfo); } } @@ -203,12 +200,11 @@ static int32_t getTableHashVgroup(STranslateContext* pCxt, const char* pDbName, static int32_t getDBVgVersion(STranslateContext* pCxt, const char* pDbFName, int32_t* pVersion, int64_t* pDbId, int32_t* pTableNum) { SParseContext* pParCxt = pCxt->pParseCxt; - int32_t code = TSDB_CODE_SUCCESS; - if (pParCxt->async) { - code = getDbVgVersionFromCache(pCxt->pMetaCache, pDbFName, pVersion, pDbId, pTableNum); - } else { - code = collectUseDatabaseImpl(pDbFName, pCxt->pDbs); - if (TSDB_CODE_SUCCESS == code) { + int32_t code = collectUseDatabaseImpl(pDbFName, pCxt->pDbs); + if (TSDB_CODE_SUCCESS == code) { + if (pParCxt->async) { + code = getDbVgVersionFromCache(pCxt->pMetaCache, pDbFName, pVersion, pDbId, pTableNum); + } else { code = catalogGetDBVgVersion(pParCxt->pCatalog, pDbFName, pVersion, pDbId, pTableNum); } } @@ -224,12 +220,11 @@ static int32_t getDBCfg(STranslateContext* pCxt, const char* pDbName, SDbCfgInfo tNameSetDbName(&name, pCxt->pParseCxt->acctId, pDbName, strlen(pDbName)); char dbFname[TSDB_DB_FNAME_LEN] = {0}; tNameGetFullDbName(&name, dbFname); - int32_t code = TSDB_CODE_SUCCESS; - if (pParCxt->async) { - code = getDbCfgFromCache(pCxt->pMetaCache, dbFname, pInfo); - } else { - code = collectUseDatabaseImpl(dbFname, pCxt->pDbs); - if (TSDB_CODE_SUCCESS == code) { + int32_t code = collectUseDatabaseImpl(dbFname, pCxt->pDbs); + if (TSDB_CODE_SUCCESS == code) { + if (pParCxt->async) { + code = getDbCfgFromCache(pCxt->pMetaCache, dbFname, pInfo); + } else { code = catalogGetDBCfg(pParCxt->pCatalog, pParCxt->pTransporter, &pParCxt->mgmtEpSet, dbFname, pInfo); } } @@ -817,6 +812,10 @@ static EDealRes translateArithmeticOperator(STranslateContext* pCxt, SOperatorNo return DEAL_RES_CONTINUE; } +static bool dataTypeEqual(const SDataType* l, const SDataType* r) { + return (l->type == r->type && l->bytes == r->bytes && l->precision == r->precision && l->scale == r->scale); +} + static EDealRes translateComparisonOperator(STranslateContext* pCxt, SOperatorNode* pOp) { SDataType ldt = ((SExprNode*)(pOp->pLeft))->resType; SDataType rdt = ((SExprNode*)(pOp->pRight))->resType; @@ -824,7 +823,24 @@ static EDealRes translateComparisonOperator(STranslateContext* pCxt, SOperatorNo return generateDealNodeErrMsg(pCxt, TSDB_CODE_PAR_WRONG_VALUE_TYPE, ((SExprNode*)(pOp->pRight))->aliasName); } if (OP_TYPE_IN == pOp->opType || OP_TYPE_NOT_IN == pOp->opType) { - ((SExprNode*)pOp->pRight)->resType = ((SExprNode*)pOp->pLeft)->resType; + SNodeListNode* pRight = (SNodeListNode*)pOp->pRight; + bool first = true; + SDataType targetDt = {0}; + SNode* pNode = NULL; + FOREACH(pNode, pRight->pNodeList) { + SDataType dt = ((SExprNode*)pNode)->resType; + if (first) { + targetDt = dt; + if (targetDt.type != TSDB_DATA_TYPE_NULL) { + first = false; + } + } else if (dt.type != targetDt.type && dt.type != TSDB_DATA_TYPE_NULL) { + return generateDealNodeErrMsg(pCxt, TSDB_CODE_PAR_WRONG_VALUE_TYPE, ((SExprNode*)pNode)->aliasName); + } else if (dt.bytes > targetDt.bytes) { + targetDt.bytes = dt.bytes; + } + } + pRight->dataType = targetDt; } if (nodesIsRegularOp(pOp)) { if (!IS_VAR_DATA_TYPE(((SExprNode*)(pOp->pLeft))->resType.type)) { @@ -2015,10 +2031,6 @@ static SNode* createSetOperProject(const char* pTableAlias, SNode* pNode) { return (SNode*)pCol; } -static bool dataTypeEqual(const SDataType* l, const SDataType* r) { - return (l->type == r->type && l->bytes == r->bytes && l->precision == r->precision && l->scale == r->scale); -} - static int32_t createCastFunc(STranslateContext* pCxt, SNode* pExpr, SDataType dt, SNode** pCast) { SFunctionNode* pFunc = nodesMakeNode(QUERY_NODE_FUNCTION); if (NULL == pFunc) { diff --git a/source/libs/parser/src/parUtil.c b/source/libs/parser/src/parUtil.c index 147ff933fbf864768f89031afc9e56593eef0746..29443876a64143503be5802648e27f722178dd04 100644 --- a/source/libs/parser/src/parUtil.c +++ b/source/libs/parser/src/parUtil.c @@ -484,20 +484,6 @@ static int32_t buildDbReq(SHashObj* pDbsHash, SArray** pDbs) { return TSDB_CODE_SUCCESS; } -static int32_t buildTableMetaReq(SHashObj* pTableMetaHash, SArray** pTableMeta) { - return buildTableReq(pTableMetaHash, pTableMeta); -} - -static int32_t buildDbVgroupReq(SHashObj* pDbVgroupHash, SArray** pDbVgroup) { - return buildDbReq(pDbVgroupHash, pDbVgroup); -} - -static int32_t buildTableVgroupReq(SHashObj* pTableVgroupHash, SArray** pTableVgroup) { - return buildTableReq(pTableVgroupHash, pTableVgroup); -} - -static int32_t buildDbCfgReq(SHashObj* pDbCfgHash, SArray** pDbCfg) { return buildDbReq(pDbCfgHash, pDbCfg); } - static int32_t buildUserAuthReq(SHashObj* pUserAuthHash, SArray** pUserAuth) { if (NULL != pUserAuthHash) { *pUserAuth = taosArrayInit(taosHashGetSize(pUserAuthHash), sizeof(SUserAuthInfo)); @@ -537,15 +523,18 @@ static int32_t buildUdfReq(SHashObj* pUdfHash, SArray** pUdf) { } int32_t buildCatalogReq(const SParseMetaCache* pMetaCache, SCatalogReq* pCatalogReq) { - int32_t code = buildTableMetaReq(pMetaCache->pTableMeta, &pCatalogReq->pTableMeta); + int32_t code = buildTableReq(pMetaCache->pTableMeta, &pCatalogReq->pTableMeta); + if (TSDB_CODE_SUCCESS == code) { + code = buildDbReq(pMetaCache->pDbVgroup, &pCatalogReq->pDbVgroup); + } if (TSDB_CODE_SUCCESS == code) { - code = buildDbVgroupReq(pMetaCache->pDbVgroup, &pCatalogReq->pDbVgroup); + code = buildTableReq(pMetaCache->pTableVgroup, &pCatalogReq->pTableHash); } if (TSDB_CODE_SUCCESS == code) { - code = buildTableVgroupReq(pMetaCache->pTableVgroup, &pCatalogReq->pTableHash); + code = buildDbReq(pMetaCache->pDbCfg, &pCatalogReq->pDbCfg); } if (TSDB_CODE_SUCCESS == code) { - code = buildDbCfgReq(pMetaCache->pDbCfg, &pCatalogReq->pDbCfg); + code = buildDbReq(pMetaCache->pDbInfo, &pCatalogReq->pDbInfo); } if (TSDB_CODE_SUCCESS == code) { code = buildUserAuthReq(pMetaCache->pUserAuth, &pCatalogReq->pUser); @@ -556,51 +545,39 @@ int32_t buildCatalogReq(const SParseMetaCache* pMetaCache, SCatalogReq* pCatalog return code; } -static int32_t putTableMetaToCache(const SArray* pTableMetaReq, const SArray* pTableMetaData, SHashObj* pTableMeta) { - int32_t ntables = taosArrayGetSize(pTableMetaReq); - for (int32_t i = 0; i < ntables; ++i) { - char fullName[TSDB_TABLE_FNAME_LEN]; - tNameExtractFullName(taosArrayGet(pTableMetaReq, i), fullName); - if (TSDB_CODE_SUCCESS != - taosHashPut(pTableMeta, fullName, strlen(fullName), taosArrayGet(pTableMetaData, i), POINTER_BYTES)) { - return TSDB_CODE_OUT_OF_MEMORY; - } - } - return TSDB_CODE_SUCCESS; +static int32_t putMetaDataToHash(const char* pKey, int32_t len, const SArray* pData, int32_t index, SHashObj* pHash) { + SMetaRes* pRes = taosArrayGet(pData, index); + return taosHashPut(pHash, pKey, len, &pRes, POINTER_BYTES); } -static int32_t putDbVgroupToCache(const SArray* pDbVgroupReq, const SArray* pDbVgroupData, SHashObj* pDbVgroup) { - int32_t nvgs = taosArrayGetSize(pDbVgroupReq); - for (int32_t i = 0; i < nvgs; ++i) { - char* pDbFName = taosArrayGet(pDbVgroupReq, i); - if (TSDB_CODE_SUCCESS != - taosHashPut(pDbVgroup, pDbFName, strlen(pDbFName), taosArrayGet(pDbVgroupData, i), POINTER_BYTES)) { - return TSDB_CODE_OUT_OF_MEMORY; - } +static int32_t getMetaDataFromHash(const char* pKey, int32_t len, SHashObj* pHash, void** pOutput) { + SMetaRes** pRes = taosHashGet(pHash, pKey, len); + if (NULL == pRes || NULL == *pRes) { + return TSDB_CODE_PAR_INTERNAL_ERROR; } - return TSDB_CODE_SUCCESS; + if (TSDB_CODE_SUCCESS == (*pRes)->code) { + *pOutput = (*pRes)->pRes; + } + return (*pRes)->code; } -static int32_t putTableVgroupToCache(const SArray* pTableVgroupReq, const SArray* pTableVgroupData, - SHashObj* pTableVgroup) { - int32_t ntables = taosArrayGetSize(pTableVgroupReq); +static int32_t putTableDataToCache(const SArray* pTableReq, const SArray* pTableData, SHashObj* pTable) { + int32_t ntables = taosArrayGetSize(pTableReq); for (int32_t i = 0; i < ntables; ++i) { char fullName[TSDB_TABLE_FNAME_LEN]; - tNameExtractFullName(taosArrayGet(pTableVgroupReq, i), fullName); - SVgroupInfo* pInfo = taosArrayGet(pTableVgroupData, i); - if (TSDB_CODE_SUCCESS != taosHashPut(pTableVgroup, fullName, strlen(fullName), &pInfo, POINTER_BYTES)) { + tNameExtractFullName(taosArrayGet(pTableReq, i), fullName); + if (TSDB_CODE_SUCCESS != putMetaDataToHash(fullName, strlen(fullName), pTableData, i, pTable)) { return TSDB_CODE_OUT_OF_MEMORY; } } return TSDB_CODE_SUCCESS; } -static int32_t putDbCfgToCache(const SArray* pDbCfgReq, const SArray* pDbCfgData, SHashObj* pDbCfg) { - int32_t nvgs = taosArrayGetSize(pDbCfgReq); +static int32_t putDbDataToCache(const SArray* pDbReq, const SArray* pDbData, SHashObj* pDb) { + int32_t nvgs = taosArrayGetSize(pDbReq); for (int32_t i = 0; i < nvgs; ++i) { - char* pDbFName = taosArrayGet(pDbCfgReq, i); - SDbCfgInfo* pInfo = taosArrayGet(pDbCfgData, i); - if (TSDB_CODE_SUCCESS != taosHashPut(pDbCfg, pDbFName, strlen(pDbFName), &pInfo, POINTER_BYTES)) { + char* pDbFName = taosArrayGet(pDbReq, i); + if (TSDB_CODE_SUCCESS != putMetaDataToHash(pDbFName, strlen(pDbFName), pDbData, i, pDb)) { return TSDB_CODE_OUT_OF_MEMORY; } } @@ -613,7 +590,7 @@ static int32_t putUserAuthToCache(const SArray* pUserAuthReq, const SArray* pUse SUserAuthInfo* pUser = taosArrayGet(pUserAuthReq, i); char key[USER_AUTH_KEY_MAX_LEN] = {0}; int32_t len = userAuthToStringExt(pUser->user, pUser->dbFName, pUser->type, key); - if (TSDB_CODE_SUCCESS != taosHashPut(pUserAuth, key, len, taosArrayGet(pUserAuthData, i), sizeof(bool))) { + if (TSDB_CODE_SUCCESS != putMetaDataToHash(key, len, pUserAuthData, i, pUserAuth)) { return TSDB_CODE_OUT_OF_MEMORY; } } @@ -623,9 +600,8 @@ static int32_t putUserAuthToCache(const SArray* pUserAuthReq, const SArray* pUse static int32_t putUdfToCache(const SArray* pUdfReq, const SArray* pUdfData, SHashObj* pUdf) { int32_t num = taosArrayGetSize(pUdfReq); for (int32_t i = 0; i < num; ++i) { - char* pFunc = taosArrayGet(pUdfReq, i); - SFuncInfo* pInfo = taosArrayGet(pUdfData, i); - if (TSDB_CODE_SUCCESS != taosHashPut(pUdf, pFunc, strlen(pFunc), &pInfo, POINTER_BYTES)) { + char* pFunc = taosArrayGet(pUdfReq, i); + if (TSDB_CODE_SUCCESS != putMetaDataToHash(pFunc, strlen(pFunc), pUdfData, i, pUdf)) { return TSDB_CODE_OUT_OF_MEMORY; } } @@ -633,15 +609,18 @@ static int32_t putUdfToCache(const SArray* pUdfReq, const SArray* pUdfData, SHas } int32_t putMetaDataToCache(const SCatalogReq* pCatalogReq, const SMetaData* pMetaData, SParseMetaCache* pMetaCache) { - int32_t code = putTableMetaToCache(pCatalogReq->pTableMeta, pMetaData->pTableMeta, pMetaCache->pTableMeta); + int32_t code = putTableDataToCache(pCatalogReq->pTableMeta, pMetaData->pTableMeta, pMetaCache->pTableMeta); + if (TSDB_CODE_SUCCESS == code) { + code = putDbDataToCache(pCatalogReq->pDbVgroup, pMetaData->pDbVgroup, pMetaCache->pDbVgroup); + } if (TSDB_CODE_SUCCESS == code) { - code = putDbVgroupToCache(pCatalogReq->pDbVgroup, pMetaData->pDbVgroup, pMetaCache->pDbVgroup); + code = putTableDataToCache(pCatalogReq->pTableHash, pMetaData->pTableHash, pMetaCache->pTableVgroup); } if (TSDB_CODE_SUCCESS == code) { - code = putTableVgroupToCache(pCatalogReq->pTableHash, pMetaData->pTableHash, pMetaCache->pTableVgroup); + code = putDbDataToCache(pCatalogReq->pDbCfg, pMetaData->pDbCfg, pMetaCache->pDbCfg); } if (TSDB_CODE_SUCCESS == code) { - code = putDbCfgToCache(pCatalogReq->pDbCfg, pMetaData->pDbCfg, pMetaCache->pDbCfg); + code = putDbDataToCache(pCatalogReq->pDbInfo, pMetaData->pDbInfo, pMetaCache->pDbInfo); } if (TSDB_CODE_SUCCESS == code) { code = putUserAuthToCache(pCatalogReq->pUser, pMetaData->pUser, pMetaCache->pUserAuth); @@ -681,16 +660,15 @@ int32_t reserveTableMetaInCacheExt(const SName* pName, SParseMetaCache* pMetaCac int32_t getTableMetaFromCache(SParseMetaCache* pMetaCache, const SName* pName, STableMeta** pMeta) { char fullName[TSDB_TABLE_FNAME_LEN]; tNameExtractFullName(pName, fullName); - STableMeta** pRes = taosHashGet(pMetaCache->pTableMeta, fullName, strlen(fullName)); - if (NULL == pRes || NULL == *pRes) { - parserError("getTableMetaFromCache error: %s", fullName); - return TSDB_CODE_PAR_INTERNAL_ERROR; - } - *pMeta = tableMetaDup(*pRes); - if (NULL == *pMeta) { - return TSDB_CODE_OUT_OF_MEMORY; + STableMeta* pTableMeta = NULL; + int32_t code = getMetaDataFromHash(fullName, strlen(fullName), pMetaCache->pTableMeta, (void**)&pTableMeta); + if (TSDB_CODE_SUCCESS == code) { + *pMeta = tableMetaDup(pTableMeta); + if (NULL == *pMeta) { + code = TSDB_CODE_OUT_OF_MEMORY; + } } - return TSDB_CODE_SUCCESS; + return code; } static int32_t reserveDbReqInCache(int32_t acctId, const char* pDb, SHashObj** pDbs) { @@ -710,19 +688,16 @@ int32_t reserveDbVgInfoInCache(int32_t acctId, const char* pDb, SParseMetaCache* } int32_t getDbVgInfoFromCache(SParseMetaCache* pMetaCache, const char* pDbFName, SArray** pVgInfo) { - SArray** pRes = taosHashGet(pMetaCache->pDbVgroup, pDbFName, strlen(pDbFName)); - if (NULL == pRes) { - parserError("getDbVgInfoFromCache error: %s", pDbFName); - return TSDB_CODE_PAR_INTERNAL_ERROR; - } - // *pRes is null, which is a legal value, indicating that the user DB has not been created - if (NULL != *pRes) { - *pVgInfo = taosArrayDup(*pRes); + SArray* pVgList = NULL; + int32_t code = getMetaDataFromHash(pDbFName, strlen(pDbFName), pMetaCache->pDbVgroup, (void**)&pVgList); + // pVgList is null, which is a legal value, indicating that the user DB has not been created + if (TSDB_CODE_SUCCESS == code && NULL != pVgList) { + *pVgInfo = taosArrayDup(pVgList); if (NULL == *pVgInfo) { - return TSDB_CODE_OUT_OF_MEMORY; + code = TSDB_CODE_OUT_OF_MEMORY; } } - return TSDB_CODE_SUCCESS; + return code; } int32_t reserveTableVgroupInCache(int32_t acctId, const char* pDb, const char* pTable, SParseMetaCache* pMetaCache) { @@ -738,30 +713,28 @@ int32_t reserveTableVgroupInCacheExt(const SName* pName, SParseMetaCache* pMetaC int32_t getTableVgroupFromCache(SParseMetaCache* pMetaCache, const SName* pName, SVgroupInfo* pVgroup) { char fullName[TSDB_TABLE_FNAME_LEN]; tNameExtractFullName(pName, fullName); - SVgroupInfo** pRes = taosHashGet(pMetaCache->pTableVgroup, fullName, strlen(fullName)); - if (NULL == pRes || NULL == *pRes) { - parserError("getTableVgroupFromCache error: %s", fullName); - return TSDB_CODE_PAR_INTERNAL_ERROR; + SVgroupInfo* pVg = NULL; + int32_t code = getMetaDataFromHash(fullName, strlen(fullName), pMetaCache->pTableVgroup, (void**)&pVg); + if (TSDB_CODE_SUCCESS == code) { + memcpy(pVgroup, pVg, sizeof(SVgroupInfo)); } - memcpy(pVgroup, *pRes, sizeof(SVgroupInfo)); - return TSDB_CODE_SUCCESS; + return code; } int32_t reserveDbVgVersionInCache(int32_t acctId, const char* pDb, SParseMetaCache* pMetaCache) { - return reserveDbReqInCache(acctId, pDb, &pMetaCache->pDbCfg); + return reserveDbReqInCache(acctId, pDb, &pMetaCache->pDbInfo); } int32_t getDbVgVersionFromCache(SParseMetaCache* pMetaCache, const char* pDbFName, int32_t* pVersion, int64_t* pDbId, int32_t* pTableNum) { - SDbInfo** pRes = taosHashGet(pMetaCache->pDbCfg, pDbFName, strlen(pDbFName)); - if (NULL == pRes || NULL == *pRes) { - parserError("getDbVgVersionFromCache error: %s", pDbFName); - return TSDB_CODE_PAR_INTERNAL_ERROR; + SDbInfo* pDbInfo = NULL; + int32_t code = getMetaDataFromHash(pDbFName, strlen(pDbFName), pMetaCache->pDbInfo, (void**)&pDbInfo); + if (TSDB_CODE_SUCCESS == code) { + *pVersion = pDbInfo->vgVer; + *pDbId = pDbInfo->dbId; + *pTableNum = pDbInfo->tbNum; } - *pVersion = (*pRes)->vgVer; - *pDbId = (*pRes)->dbId; - *pTableNum = (*pRes)->tbNum; - return TSDB_CODE_SUCCESS; + return code; } int32_t reserveDbCfgInCache(int32_t acctId, const char* pDb, SParseMetaCache* pMetaCache) { @@ -769,13 +742,12 @@ int32_t reserveDbCfgInCache(int32_t acctId, const char* pDb, SParseMetaCache* pM } int32_t getDbCfgFromCache(SParseMetaCache* pMetaCache, const char* pDbFName, SDbCfgInfo* pInfo) { - SDbCfgInfo** pRes = taosHashGet(pMetaCache->pDbCfg, pDbFName, strlen(pDbFName)); - if (NULL == pRes || NULL == *pRes) { - parserError("getDbCfgFromCache error: %s", pDbFName); - return TSDB_CODE_PAR_INTERNAL_ERROR; + SDbCfgInfo* pDbCfg = NULL; + int32_t code = getMetaDataFromHash(pDbFName, strlen(pDbFName), pMetaCache->pDbCfg, (void**)&pDbCfg); + if (TSDB_CODE_SUCCESS == code) { + memcpy(pInfo, pDbCfg, sizeof(SDbCfgInfo)); } - memcpy(pInfo, *pRes, sizeof(SDbCfgInfo)); - return TSDB_CODE_SUCCESS; + return code; } static int32_t reserveUserAuthInCacheImpl(const char* pKey, int32_t len, SParseMetaCache* pMetaCache) { @@ -808,13 +780,12 @@ int32_t getUserAuthFromCache(SParseMetaCache* pMetaCache, const char* pUser, con bool* pPass) { char key[USER_AUTH_KEY_MAX_LEN] = {0}; int32_t len = userAuthToStringExt(pUser, pDbFName, type, key); - bool* pRes = taosHashGet(pMetaCache->pUserAuth, key, len); - if (NULL == pRes) { - parserError("getUserAuthFromCache error: %s, %s, %d", pUser, pDbFName, type); - return TSDB_CODE_PAR_INTERNAL_ERROR; + bool* pRes = NULL; + int32_t code = getMetaDataFromHash(key, len, pMetaCache->pUserAuth, (void**)&pRes); + if (TSDB_CODE_SUCCESS == code) { + *pPass = *pRes; } - *pPass = *pRes; - return TSDB_CODE_SUCCESS; + return code; } int32_t reserveUdfInCache(const char* pFunc, SParseMetaCache* pMetaCache) { @@ -828,11 +799,10 @@ int32_t reserveUdfInCache(const char* pFunc, SParseMetaCache* pMetaCache) { } int32_t getUdfInfoFromCache(SParseMetaCache* pMetaCache, const char* pFunc, SFuncInfo* pInfo) { - SFuncInfo** pRes = taosHashGet(pMetaCache->pUdf, pFunc, strlen(pFunc)); - if (NULL == pRes || NULL == *pRes) { - parserError("getUdfInfoFromCache error: %s", pFunc); - return TSDB_CODE_PAR_INTERNAL_ERROR; + SFuncInfo* pFuncInfo = NULL; + int32_t code = getMetaDataFromHash(pFunc, strlen(pFunc), pMetaCache->pUdf, (void**)&pFuncInfo); + if (TSDB_CODE_SUCCESS == code) { + memcpy(pInfo, pFuncInfo, sizeof(SFuncInfo)); } - memcpy(pInfo, *pRes, sizeof(SFuncInfo)); - return TSDB_CODE_SUCCESS; + return code; } diff --git a/source/libs/parser/test/mockCatalogService.cpp b/source/libs/parser/test/mockCatalogService.cpp index 4834d2d37711d537d09d0e1b12e2bd8dc9697827..0c37c875c0bb7ae01df598b6c748a6b841c41891 100644 --- a/source/libs/parser/test/mockCatalogService.cpp +++ b/source/libs/parser/test/mockCatalogService.cpp @@ -367,49 +367,40 @@ class MockCatalogServiceImpl { } int32_t getAllTableMeta(SArray* pTableMetaReq, SArray** pTableMetaData) const { - int32_t code = TSDB_CODE_SUCCESS; if (NULL != pTableMetaReq) { int32_t ntables = taosArrayGetSize(pTableMetaReq); - *pTableMetaData = taosArrayInit(ntables, POINTER_BYTES); + *pTableMetaData = taosArrayInit(ntables, sizeof(SMetaRes)); for (int32_t i = 0; i < ntables; ++i) { - STableMeta* pMeta = NULL; - code = catalogGetTableMeta((const SName*)taosArrayGet(pTableMetaReq, i), &pMeta); - if (TSDB_CODE_SUCCESS == code) { - taosArrayPush(*pTableMetaData, &pMeta); - } else { - break; - } + SMetaRes res = {0}; + res.code = catalogGetTableMeta((const SName*)taosArrayGet(pTableMetaReq, i), (STableMeta**)&res.pRes); + taosArrayPush(*pTableMetaData, &res); } } - return code; + return TSDB_CODE_SUCCESS; } int32_t getAllTableVgroup(SArray* pTableVgroupReq, SArray** pTableVgroupData) const { - int32_t code = TSDB_CODE_SUCCESS; if (NULL != pTableVgroupReq) { int32_t ntables = taosArrayGetSize(pTableVgroupReq); - *pTableVgroupData = taosArrayInit(ntables, sizeof(SVgroupInfo)); + *pTableVgroupData = taosArrayInit(ntables, sizeof(SMetaRes)); for (int32_t i = 0; i < ntables; ++i) { - SVgroupInfo vgInfo = {0}; - code = catalogGetTableHashVgroup((const SName*)taosArrayGet(pTableVgroupReq, i), &vgInfo); - if (TSDB_CODE_SUCCESS == code) { - taosArrayPush(*pTableVgroupData, &vgInfo); - } else { - break; - } + SMetaRes res = {0}; + res.pRes = taosMemoryCalloc(1, sizeof(SVgroupInfo)); + res.code = catalogGetTableHashVgroup((const SName*)taosArrayGet(pTableVgroupReq, i), (SVgroupInfo*)res.pRes); + taosArrayPush(*pTableVgroupData, &res); } } - return code; + return TSDB_CODE_SUCCESS; } int32_t getAllDbVgroup(SArray* pDbVgroupReq, SArray** pDbVgroupData) const { int32_t code = TSDB_CODE_SUCCESS; if (NULL != pDbVgroupReq) { int32_t ndbs = taosArrayGetSize(pDbVgroupReq); - *pDbVgroupData = taosArrayInit(ndbs, POINTER_BYTES); + *pDbVgroupData = taosArrayInit(ndbs, sizeof(SMetaRes)); for (int32_t i = 0; i < ndbs; ++i) { - int64_t zeroVg = 0; - taosArrayPush(*pDbVgroupData, &zeroVg); + SMetaRes res = {0}; + taosArrayPush(*pDbVgroupData, &res); } } return code; @@ -419,10 +410,11 @@ class MockCatalogServiceImpl { int32_t code = TSDB_CODE_SUCCESS; if (NULL != pDbCfgReq) { int32_t ndbs = taosArrayGetSize(pDbCfgReq); - *pDbCfgData = taosArrayInit(ndbs, sizeof(SDbCfgInfo)); + *pDbCfgData = taosArrayInit(ndbs, sizeof(SMetaRes)); for (int32_t i = 0; i < ndbs; ++i) { - SDbCfgInfo dbCfg = {0}; - taosArrayPush(*pDbCfgData, &dbCfg); + SMetaRes res = {0}; + res.pRes = taosMemoryCalloc(1, sizeof(SDbCfgInfo)); + taosArrayPush(*pDbCfgData, &res); } } return code; @@ -432,10 +424,11 @@ class MockCatalogServiceImpl { int32_t code = TSDB_CODE_SUCCESS; if (NULL != pDbInfoReq) { int32_t ndbs = taosArrayGetSize(pDbInfoReq); - *pDbInfoData = taosArrayInit(ndbs, sizeof(SDbCfgInfo)); + *pDbInfoData = taosArrayInit(ndbs, sizeof(SMetaRes)); for (int32_t i = 0; i < ndbs; ++i) { - SDbInfo dbInfo = {0}; - taosArrayPush(*pDbInfoData, &dbInfo); + SMetaRes res = {0}; + res.pRes = taosMemoryCalloc(1, sizeof(SDbInfo)); + taosArrayPush(*pDbInfoData, &res); } } return code; @@ -445,31 +438,29 @@ class MockCatalogServiceImpl { int32_t code = TSDB_CODE_SUCCESS; if (NULL != pUserAuthReq) { int32_t num = taosArrayGetSize(pUserAuthReq); - *pUserAuthData = taosArrayInit(num, sizeof(bool)); + *pUserAuthData = taosArrayInit(num, sizeof(SMetaRes)); for (int32_t i = 0; i < num; ++i) { - bool pass = true; - taosArrayPush(*pUserAuthData, &pass); + SMetaRes res = {0}; + res.pRes = taosMemoryCalloc(1, sizeof(bool)); + *(bool*)(res.pRes) = true; + taosArrayPush(*pUserAuthData, &res); } } return code; } int32_t getAllUdf(SArray* pUdfReq, SArray** pUdfData) const { - int32_t code = TSDB_CODE_SUCCESS; if (NULL != pUdfReq) { int32_t num = taosArrayGetSize(pUdfReq); - *pUdfData = taosArrayInit(num, sizeof(SFuncInfo)); + *pUdfData = taosArrayInit(num, sizeof(SMetaRes)); for (int32_t i = 0; i < num; ++i) { - SFuncInfo info = {0}; - code = catalogGetUdfInfo((char*)taosArrayGet(pUdfReq, i), &info); - if (TSDB_CODE_SUCCESS == code) { - taosArrayPush(*pUdfData, &info); - } else { - break; - } + SMetaRes res = {0}; + res.pRes = taosMemoryCalloc(1, sizeof(SFuncInfo)); + res.code = catalogGetUdfInfo((char*)taosArrayGet(pUdfReq, i), (SFuncInfo*)res.pRes); + taosArrayPush(*pUdfData, &res); } } - return code; + return TSDB_CODE_SUCCESS; } uint64_t id_; diff --git a/source/libs/parser/test/parInitialATest.cpp b/source/libs/parser/test/parInitialATest.cpp index 6ac793cb0dc33e6ea8911f275492c948af64d2c8..22b244145bc0fe01759381a3a5bf8499d77d026d 100644 --- a/source/libs/parser/test/parInitialATest.cpp +++ b/source/libs/parser/test/parInitialATest.cpp @@ -39,6 +39,8 @@ TEST_F(ParserInitialATest, alterDatabase) { useDb("root", "test"); run("ALTER DATABASE wxy_db CACHELAST 1 FSYNC 200 WAL 1"); + + run("ALTER DATABASE wxy_db KEEP 2400"); } // todo ALTER local diff --git a/source/libs/qcom/src/querymsg.c b/source/libs/qcom/src/querymsg.c index 6de1620de19c9f33e03b67b70616e50e8a982383..10d0f4a44ad0626f9fe0c6510595bcd6ae8dbe86 100644 --- a/source/libs/qcom/src/querymsg.c +++ b/source/libs/qcom/src/querymsg.c @@ -312,10 +312,6 @@ int32_t queryCreateTableMetaFromMsg(STableMetaRsp *msg, bool isStb, STableMeta * pTableMeta->sversion = msg->sversion; pTableMeta->tversion = msg->tversion; - if (isStb) { - qDebug("stable %s meta returned, suid:%" PRIx64, msg->stbName, pTableMeta->suid); - } - pTableMeta->tableInfo.numOfTags = msg->numOfTags; pTableMeta->tableInfo.precision = msg->precision; pTableMeta->tableInfo.numOfColumns = msg->numOfColumns; @@ -326,6 +322,12 @@ int32_t queryCreateTableMetaFromMsg(STableMetaRsp *msg, bool isStb, STableMeta * pTableMeta->tableInfo.rowSize += pTableMeta->schema[i].bytes; } + qDebug("table %s uid %" PRIx64 " meta returned, type %d vgId %d db %s stb %s suid %" PRIx64 " sver %d tver %d" PRIx64 + " tagNum %d colNum %d precision %d rowSize %d", + msg->tbName, pTableMeta->uid, pTableMeta->tableType, pTableMeta->vgId, msg->dbFName, msg->stbName, pTableMeta->suid, + pTableMeta->sversion, pTableMeta->tversion, pTableMeta->tableInfo.numOfTags, pTableMeta->tableInfo.numOfColumns, + pTableMeta->tableInfo.precision, pTableMeta->tableInfo.rowSize); + *pMeta = pTableMeta; return TSDB_CODE_SUCCESS; } diff --git a/source/libs/qworker/src/qwDbg.c b/source/libs/qworker/src/qwDbg.c index 3914541157e86aa67a55c56cd56afb76da2061cf..368c3bb517e18c8b9254f4d90bdd4b8aa5bb0d2d 100644 --- a/source/libs/qworker/src/qwDbg.c +++ b/source/libs/qworker/src/qwDbg.c @@ -9,7 +9,7 @@ #include "tmsg.h" #include "tname.h" -SQWDebug gQWDebug = {.statusEnable = true, .dumpEnable = true}; +SQWDebug gQWDebug = {.statusEnable = true, .dumpEnable = false}; int32_t qwDbgValidateStatus(QW_FPARAMS_DEF, int8_t oriStatus, int8_t newStatus, bool *ignore) { if (!gQWDebug.statusEnable) { @@ -97,7 +97,11 @@ _return: QW_RET(code); } -void qwDbgDumpSchInfo(SQWSchStatus *sch, int32_t i) {} +void qwDbgDumpSchInfo(SQWorker *mgmt, SQWSchStatus *sch, int32_t i) { + QW_LOCK(QW_READ, &sch->tasksLock); + QW_DLOG("the %dth scheduler status, hbBrokenTs:%" PRId64 ",taskNum:%d", i, sch->hbBrokenTs, taosHashGetSize(sch->tasksHash)); + QW_UNLOCK(QW_READ, &sch->tasksLock); +} void qwDbgDumpMgmtInfo(SQWorker *mgmt) { if (!gQWDebug.dumpEnable) { @@ -106,7 +110,7 @@ void qwDbgDumpMgmtInfo(SQWorker *mgmt) { QW_LOCK(QW_READ, &mgmt->schLock); - /*QW_DUMP("total remain schduler num:%d", taosHashGetSize(mgmt->schHash));*/ + QW_DUMP("total remain scheduler num %d", taosHashGetSize(mgmt->schHash)); void *key = NULL; size_t keyLen = 0; @@ -116,14 +120,14 @@ void qwDbgDumpMgmtInfo(SQWorker *mgmt) { void *pIter = taosHashIterate(mgmt->schHash, NULL); while (pIter) { sch = (SQWSchStatus *)pIter; - qwDbgDumpSchInfo(sch, i); + qwDbgDumpSchInfo(mgmt, sch, i); ++i; pIter = taosHashIterate(mgmt->schHash, pIter); } QW_UNLOCK(QW_READ, &mgmt->schLock); - /*QW_DUMP("total remain ctx num:%d", taosHashGetSize(mgmt->ctxHash));*/ + QW_DUMP("total remain ctx num %d", taosHashGetSize(mgmt->ctxHash)); } diff --git a/source/libs/scalar/inc/sclInt.h b/source/libs/scalar/inc/sclInt.h index 1c2e4a358a2c256cf3ed577be568c2e93fe13cbe..4dd16098ff3df97185c4a6f249a12882456d111d 100644 --- a/source/libs/scalar/inc/sclInt.h +++ b/source/libs/scalar/inc/sclInt.h @@ -22,11 +22,18 @@ extern "C" { #include "thash.h" #include "query.h" +typedef struct SOperatorValueType { + int32_t opResType; + int32_t selfType; + int32_t peerType; +} SOperatorValueType; + typedef struct SScalarCtx { int32_t code; SArray *pBlockList; /* element is SSDataBlock* */ SHashObj *pRes; /* element is SScalarParam */ void *param; // additional parameter (meta actually) for acquire value such as tbname/tags values + SOperatorValueType type; } SScalarCtx; @@ -53,7 +60,7 @@ int32_t doConvertDataType(SValueNode* pValueNode, SScalarParam* out); SColumnInfoData* createColumnInfoData(SDataType* pType, int32_t numOfRows); int32_t sclConvertToTsValueNode(int8_t precision, SValueNode* valueNode); -#define GET_PARAM_TYPE(_c) ((_c)->columnData->info.type) +#define GET_PARAM_TYPE(_c) ((_c)->columnData ? (_c)->columnData->info.type : (_c)->hashValueType) #define GET_PARAM_BYTES(_c) ((_c)->columnData->info.bytes) #define GET_PARAM_PRECISON(_c) ((_c)->columnData->info.precision) diff --git a/source/libs/scalar/src/filter.c b/source/libs/scalar/src/filter.c index 12e9ec5aea3146c1804697b97bf156415fce73c7..6704bfdde32357dde928f0a29dda8fd1a1bcc299 100644 --- a/source/libs/scalar/src/filter.c +++ b/source/libs/scalar/src/filter.c @@ -3669,6 +3669,14 @@ EDealRes fltReviseRewriter(SNode** pNode, void* pContext) { stat->scalarMode = true; return DEAL_RES_CONTINUE; } + } else { + SColumnNode *refNode = (SColumnNode *)node->pLeft; + SNodeListNode *listNode = (SNodeListNode *)node->pRight; + int32_t type = vectorGetConvertType(refNode->node.resType.type, listNode->dataType.type); + if (0 != type && type != refNode->node.resType.type) { + stat->scalarMode = true; + return DEAL_RES_CONTINUE; + } } } diff --git a/source/libs/scalar/src/scalar.c b/source/libs/scalar/src/scalar.c index d2436b9948f2cf7bfa15d061cdc9bbfdfefd6f08..66e4af5ded0404b9f5b2ec7e9b8097178613c2c6 100644 --- a/source/libs/scalar/src/scalar.c +++ b/source/libs/scalar/src/scalar.c @@ -208,7 +208,13 @@ int32_t sclInitParam(SNode* node, SScalarParam *param, SScalarCtx *ctx, int32_t SCL_RET(TSDB_CODE_QRY_INVALID_INPUT); } - SCL_ERR_RET(scalarGenerateSetFromList((void **)¶m->pHashFilter, node, nodeList->dataType.type)); + int32_t type = vectorGetConvertType(ctx->type.selfType, ctx->type.peerType); + if (type == 0) { + type = nodeList->dataType.type; + } + + SCL_ERR_RET(scalarGenerateSetFromList((void **)¶m->pHashFilter, node, type)); + param->hashValueType = type; if (taosHashPut(ctx->pRes, &node, POINTER_BYTES, param, sizeof(*param))) { taosHashCleanup(param->pHashFilter); sclError("taosHashPut nodeList failed, size:%d", (int32_t)sizeof(*param)); @@ -334,6 +340,46 @@ _return: SCL_RET(code); } +int32_t sclGetNodeType(SNode *pNode, SScalarCtx *ctx) { + if (NULL == pNode) { + return -1; + } + + switch (nodeType(pNode)) { + case QUERY_NODE_VALUE: { + SValueNode *valueNode = (SValueNode *)pNode; + return valueNode->node.resType.type; + } + case QUERY_NODE_NODE_LIST: { + SNodeListNode *nodeList = (SNodeListNode *)pNode; + return nodeList->dataType.type; + } + case QUERY_NODE_COLUMN: { + SColumnNode *colNode = (SColumnNode *)pNode; + return colNode->node.resType.type; + } + case QUERY_NODE_FUNCTION: + case QUERY_NODE_OPERATOR: + case QUERY_NODE_LOGIC_CONDITION: { + SScalarParam *res = (SScalarParam *)taosHashGet(ctx->pRes, &pNode, POINTER_BYTES); + if (NULL == res) { + sclError("no result for node, type:%d, node:%p", nodeType(pNode), pNode); + return -1; + } + return res->columnData->info.type; + } + } + + return -1; +} + + +void sclSetOperatorValueType(SOperatorNode *node, SScalarCtx *ctx) { + ctx->type.opResType = node->node.resType.type; + ctx->type.selfType = sclGetNodeType(node->pLeft, ctx); + ctx->type.peerType = sclGetNodeType(node->pRight, ctx); +} + int32_t sclInitOperatorParams(SScalarParam **pParams, SOperatorNode *node, SScalarCtx *ctx, int32_t *rowNum) { int32_t code = 0; int32_t paramNum = scalarGetOperatorParamNum(node->opType); @@ -348,8 +394,11 @@ int32_t sclInitOperatorParams(SScalarParam **pParams, SOperatorNode *node, SScal SCL_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } + sclSetOperatorValueType(node, ctx); + SCL_ERR_JRET(sclInitParam(node->pLeft, ¶mList[0], ctx, rowNum)); if (paramNum > 1) { + TSWAP(ctx->type.selfType, ctx->type.peerType); SCL_ERR_JRET(sclInitParam(node->pRight, ¶mList[1], ctx, rowNum)); } diff --git a/source/libs/scalar/src/sclfunc.c b/source/libs/scalar/src/sclfunc.c index 2b15f5031eaf40d0bad6eeb658be2c4d232866df..2369779105f06685be2120907531ee0a32869c84 100644 --- a/source/libs/scalar/src/sclfunc.c +++ b/source/libs/scalar/src/sclfunc.c @@ -901,12 +901,22 @@ int32_t toISO8601Function(SScalarParam *pInput, int32_t inputNum, SScalarParam * if (hasFraction) { int32_t fracLen = (int32_t)strlen(fraction) + 1; - char *tzInfo = strchr(buf, '+'); - if (tzInfo) { + + char *tzInfo; + if (buf[len - 1] == 'z' || buf[len - 1] == 'Z') { + tzInfo = &buf[len - 1]; memmove(tzInfo + fracLen, tzInfo, strlen(tzInfo)); } else { - tzInfo = strchr(buf, '-'); - memmove(tzInfo + fracLen, tzInfo, strlen(tzInfo)); + tzInfo = strchr(buf, '+'); + if (tzInfo) { + memmove(tzInfo + fracLen, tzInfo, strlen(tzInfo)); + } else { + //search '-' backwards + tzInfo = strrchr(buf, '-'); + if (tzInfo) { + memmove(tzInfo + fracLen, tzInfo, strlen(tzInfo)); + } + } } char tmp[32] = {0}; diff --git a/source/libs/scalar/src/sclvector.c b/source/libs/scalar/src/sclvector.c index 7440a6ae5b504360483090b6a1283e3bf0f69115..7d96a89e853665af8a7c43499468ad1c52976d0c 100644 --- a/source/libs/scalar/src/sclvector.c +++ b/source/libs/scalar/src/sclvector.c @@ -627,6 +627,11 @@ int32_t vectorConvertImpl(const SScalarParam* pIn, SScalarParam* pOut) { SColumnInfoData* pInputCol = pIn->columnData; SColumnInfoData* pOutputCol = pOut->columnData; + if (NULL == pInputCol) { + sclError("input column is NULL, hashFilter %p", pIn->pHashFilter); + return TSDB_CODE_APP_ERROR; + } + int16_t inType = pInputCol->info.type; int16_t outType = pOutputCol->info.type; @@ -827,11 +832,26 @@ int32_t vectorGetConvertType(int32_t type1, int32_t type2) { return gConvertTypes[type2][type1]; } -int32_t vectorConvert(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam* pLeftOut, SScalarParam* pRightOut) { - if (pLeft->pHashFilter != NULL || pRight->pHashFilter != NULL) { - return TSDB_CODE_SUCCESS; +int32_t vectorConvertScalarParam(SScalarParam *input, SScalarParam *output, int32_t type) { + int32_t code = 0; + SDataType t = {.type = type, .bytes = tDataTypes[type].bytes}; + output->numOfRows = input->numOfRows; + + output->columnData = createColumnInfoData(&t, input->numOfRows); + if (output->columnData == NULL) { + return TSDB_CODE_OUT_OF_MEMORY; } + code = vectorConvertImpl(input, output); + if (code) { +// taosMemoryFreeClear(paramOut1->data); + return code; + } + + return TSDB_CODE_SUCCESS; +} + +int32_t vectorConvert(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam* pLeftOut, SScalarParam* pRightOut) { int32_t leftType = GET_PARAM_TYPE(pLeft); int32_t rightType = GET_PARAM_TYPE(pRight); if (leftType == rightType) { @@ -860,31 +880,14 @@ int32_t vectorConvert(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam* p } if (type != GET_PARAM_TYPE(param1)) { - SDataType t = {.type = type, .bytes = tDataTypes[type].bytes}; - paramOut1->numOfRows = param1->numOfRows; - - paramOut1->columnData = createColumnInfoData(&t, param1->numOfRows); - if (paramOut1->columnData == NULL) { - return terrno; - } - - code = vectorConvertImpl(param1, paramOut1); + code = vectorConvertScalarParam(param1, paramOut1, type); if (code) { -// taosMemoryFreeClear(paramOut1->data); return code; } } if (type != GET_PARAM_TYPE(param2)) { - SDataType t = {.type = type, .bytes = tDataTypes[type].bytes}; - paramOut2->numOfRows = param2->numOfRows; - - paramOut2->columnData = createColumnInfoData(&t, param2->numOfRows); - if (paramOut2->columnData == NULL) { - return terrno; - } - - code = vectorConvertImpl(param2, paramOut2); + code = vectorConvertScalarParam(param2, paramOut2, type); if (code) { return code; } diff --git a/source/libs/scheduler/src/schJob.c b/source/libs/scheduler/src/schJob.c index d799169ee49be6a951e6599e8e6a58618b2a6ce8..0128b9292c11adcf80c6eac5e6ef5d2379f5d06c 100644 --- a/source/libs/scheduler/src/schJob.c +++ b/source/libs/scheduler/src/schJob.c @@ -545,7 +545,7 @@ int32_t schSetAddrsFromNodeList(SSchJob *pJob, SSchTask *pTask) { SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - SCH_TASK_DLOG("set %dth condidate addr, id %d, fqdn:%s, port:%d", i, naddr->nodeId, SCH_GET_CUR_EP(naddr)->fqdn, SCH_GET_CUR_EP(naddr)->port); + SCH_TASK_DLOG("set %dth candidate addr, id %d, fqdn:%s, port:%d", i, naddr->nodeId, SCH_GET_CUR_EP(naddr)->fqdn, SCH_GET_CUR_EP(naddr)->port); ++addNum; } @@ -897,6 +897,7 @@ int32_t schProcessOnJobFailureImpl(SSchJob *pJob, int32_t status, int32_t errCod if (SCH_EXEC_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_EXEC_CB, 0)) { schNotifyUserQueryRes(pJob); } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_FETCH_CB, 0)) { + atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); schNotifyUserFetchRes(pJob); } } @@ -925,6 +926,7 @@ int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) { } else if (SCH_EXEC_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_EXEC_CB, 0)) { schNotifyUserQueryRes(pJob); } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_FETCH_CB, 0)) { + atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); schNotifyUserFetchRes(pJob); } @@ -945,6 +947,8 @@ void schProcessOnDataFetched(SSchJob *job) { if (job->attr.syncSchedule) { tsem_post(&job->rspSem); } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&job->userCb, SCH_FETCH_CB, 0)) { + atomic_val_compare_exchange_8(&job->userFetch, 1, 0); + schNotifyUserFetchRes(job); } } @@ -1680,9 +1684,9 @@ int32_t schAsyncFetchRows(SSchJob *pJob) { } if (pJob->attr.explainMode == EXPLAIN_MODE_STATIC) { - SCH_ERR_JRET(schNotifyUserFetchRes(pJob)); - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); + + SCH_ERR_JRET(schNotifyUserFetchRes(pJob)); } else { pJob->userCb = SCH_FETCH_CB; diff --git a/source/libs/stream/src/stream.c b/source/libs/stream/src/stream.c index 823ce6d2fe68703b42f10e1036c25428b08939ec..069595390db7833e025f3f9498e169cfb6980581 100644 --- a/source/libs/stream/src/stream.c +++ b/source/libs/stream/src/stream.c @@ -59,7 +59,7 @@ int32_t streamTaskEnqueue(SStreamTask* pTask, SStreamDispatchReq* pReq, SRpcMsg* // rsp by input status void* buf = rpcMallocCont(sizeof(SMsgHead) + sizeof(SStreamDispatchRsp)); - ((SMsgHead*)buf)->vgId = htonl(pReq->sourceVg); + ((SMsgHead*)buf)->vgId = htonl(pReq->upstreamNodeId); SStreamDispatchRsp* pCont = POINTER_SHIFT(buf, sizeof(SMsgHead)); pCont->inputStatus = status; pCont->streamId = pReq->streamId; @@ -78,7 +78,18 @@ int32_t streamProcessDispatchReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDisp // 2.1. idle: exec // 2.2. executing: return // 2.3. closing: keep trying - streamExec(pTask, pMsgCb); + if (pTask->execType != TASK_EXEC__NONE) { + streamExec(pTask, pMsgCb); + } else { + ASSERT(pTask->sinkType != TASK_SINK__NONE); + while (1) { + void* data = streamQueueNextItem(pTask->inputQueue); + if (data == NULL) return 0; + if (streamTaskOutput(pTask, data) < 0) { + ASSERT(0); + } + } + } // 3. handle output // 3.1 check and set status diff --git a/source/libs/stream/src/streamDispatch.c b/source/libs/stream/src/streamDispatch.c index d1e3fa079916b1413ef189b2eb71680cfbbafb0a..16da418677dad58bbfac70add0c625635a30da6f 100644 --- a/source/libs/stream/src/streamDispatch.c +++ b/source/libs/stream/src/streamDispatch.c @@ -22,6 +22,7 @@ int32_t tEncodeStreamDispatchReq(SEncoder* pEncoder, const SStreamDispatchReq* p if (tEncodeI32(pEncoder, pReq->sourceTaskId) < 0) return -1; if (tEncodeI32(pEncoder, pReq->sourceVg) < 0) return -1; if (tEncodeI32(pEncoder, pReq->sourceChildId) < 0) return -1; + if (tEncodeI32(pEncoder, pReq->upstreamNodeId) < 0) return -1; if (tEncodeI32(pEncoder, pReq->blockNum) < 0) return -1; ASSERT(taosArrayGetSize(pReq->data) == pReq->blockNum); ASSERT(taosArrayGetSize(pReq->dataLen) == pReq->blockNum); @@ -42,6 +43,7 @@ int32_t tDecodeStreamDispatchReq(SDecoder* pDecoder, SStreamDispatchReq* pReq) { if (tDecodeI32(pDecoder, &pReq->sourceTaskId) < 0) return -1; if (tDecodeI32(pDecoder, &pReq->sourceVg) < 0) return -1; if (tDecodeI32(pDecoder, &pReq->sourceChildId) < 0) return -1; + if (tDecodeI32(pDecoder, &pReq->upstreamNodeId) < 0) return -1; if (tDecodeI32(pDecoder, &pReq->blockNum) < 0) return -1; ASSERT(pReq->blockNum > 0); pReq->data = taosArrayInit(pReq->blockNum, sizeof(void*)); @@ -94,6 +96,7 @@ int32_t streamBuildDispatchMsg(SStreamTask* pTask, SStreamDataBlock* data, SRpcM .sourceTaskId = pTask->taskId, .sourceVg = data->sourceVg, .sourceChildId = pTask->childId, + .upstreamNodeId = pTask->nodeId, .blockNum = blockNum, }; @@ -184,13 +187,17 @@ int32_t streamDispatch(SStreamTask* pTask, SMsgCb* pMsgCb) { #endif SStreamDataBlock* pBlock = streamQueueNextItem(pTask->outputQueue); - if (pBlock == NULL) return 0; + if (pBlock == NULL) { + atomic_store_8(&pTask->outputStatus, TASK_OUTPUT_STATUS__NORMAL); + return 0; + } ASSERT(pBlock->type == STREAM_DATA_TYPE_SSDATA_BLOCK); SRpcMsg dispatchMsg = {0}; SEpSet* pEpSet = NULL; if (streamBuildDispatchMsg(pTask, pBlock, &dispatchMsg, &pEpSet) < 0) { ASSERT(0); + atomic_store_8(&pTask->outputStatus, TASK_OUTPUT_STATUS__NORMAL); return -1; } diff --git a/source/libs/stream/src/streamExec.c b/source/libs/stream/src/streamExec.c index 72df516e0d63222e6ebab82a086b9116e7c62432..fe1a85774316068d7691a41d7c2a4ff2ae59e20e 100644 --- a/source/libs/stream/src/streamExec.c +++ b/source/libs/stream/src/streamExec.c @@ -65,6 +65,7 @@ static SArray* streamExecForQall(SStreamTask* pTask, SArray* pRes) { } qRes->type = STREAM_INPUT__DATA_BLOCK; qRes->blocks = pRes; + /*qRes->sourceVg = pTask->nodeId;*/ if (streamTaskOutput(pTask, qRes) < 0) { streamQueueProcessFail(pTask->inputQueue); taosArrayDestroy(pRes); diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index d522b829dd8620c856316626c8c62c7e75ba83a0..795d3e3c278faccdf2178397af4bd93e40870331 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -281,7 +281,7 @@ int32_t syncGetRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg) { return ret; } -int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg) { +int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcHandleInfo* pInfo) { SSyncNode* pSyncNode = (SSyncNode*)taosAcquireRef(tsNodeRefId, rid); if (pSyncNode == NULL) { return TAOS_SYNC_STATE_ERROR; @@ -291,7 +291,7 @@ int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg) { SRespStub stub; int32_t ret = syncRespMgrGetAndDel(pSyncNode->pSyncRespMgr, index, &stub); if (ret == 1) { - memcpy(msg, &(stub.rpcMsg), sizeof(SRpcMsg)); + *pInfo = stub.rpcMsg.info; } taosReleaseRef(tsNodeRefId, pSyncNode->rid); diff --git a/source/libs/transport/src/transSvr.c b/source/libs/transport/src/transSvr.c index 50f99128b2fea235aaee1ef1d337149d98efaac6..cfd711e6eb5e4eea58582d18ecf637e27ab0539d 100644 --- a/source/libs/transport/src/transSvr.c +++ b/source/libs/transport/src/transSvr.c @@ -307,6 +307,13 @@ static void uvHandleReq(SSvrConn* pConn) { if (pHead->noResp == 1) { transMsg.info.refId = -1; } + + // set up conn info + SRpcConnInfo* pConnInfo = &(transMsg.info.connInfo); + pConnInfo->clientIp = (uint32_t)(pConn->addr.sin_addr.s_addr); + pConnInfo->clientPort = ntohs(pConn->addr.sin_port); + tstrncpy(pConnInfo->user, pConn->user, sizeof(pConnInfo->user)); + transReleaseExHandle(refMgt, pConn->refId); STrans* pTransInst = pConn->pTransInst; @@ -1153,23 +1160,6 @@ _return2: rpcFreeCont(msg->pCont); } -int transGetConnInfo(void* thandle, STransHandleInfo* pInfo) { - if (thandle == NULL) { - tTrace("invalid handle %p, failed to Get Conn info", thandle); - return -1; - } - SExHandle* ex = thandle; - SSvrConn* pConn = ex->handle; - if (pConn == NULL) { - tTrace("invalid handle %p, failed to Get Conn info", thandle); - return -1; - } - - struct sockaddr_in addr = pConn->addr; - pInfo->clientIp = (uint32_t)(addr.sin_addr.s_addr); - pInfo->clientPort = ntohs(addr.sin_port); - tstrncpy(pInfo->user, pConn->user, sizeof(pInfo->user)); - return 0; -} +int transGetConnInfo(void* thandle, STransHandleInfo* pConnInfo) { return -1; } #endif diff --git a/tests/script/tsim/query/complex_having.sim b/tests/script/tsim/query/complex_having.sim index ef58228f342f7f38460ebced4e9867dced0c9b44..6a4aa6ea28299e81bbf7fdc0cc8d7bedd103fb45 100644 --- a/tests/script/tsim/query/complex_having.sim +++ b/tests/script/tsim/query/complex_having.sim @@ -105,7 +105,7 @@ endi sql select sum(c1) ,count(c1) from ct4 group by c1 having count(c7) < 1 or sum(c1) > 2 ; print ====> sql : select sum(c1) ,count(c1) from ct4 group by c1 having count(c7) < 1 or sum(c1) > 2 ; print ====> rows: $rows -if $rows != 7 then +if $rows != 8 then return -1 endi @@ -214,9 +214,9 @@ print =================== count all rows sql select count(c1) from stb1 print ====> sql : select count(c1) from stb1 print ====> rows: $data00 -if $data00 != 20 then - print expect 20, actual: $data00 - return -1 +if $data00 != 17 then + print expect 17, actual: $data00 + return -1 endi #================================================= @@ -246,7 +246,7 @@ print =================== count all rows sql select count(c1) from stb1 print ====> sql : select count(c1) from stb1 print ====> rows: $data00 -if $data00 != 20 then +if $data00 != 17 then return -1 endi @@ -279,7 +279,7 @@ endi sql select sum(c1) ,count(c1) from ct4 group by c1 having count(c7) < 1 or sum(c1) > 2 ; print ====> sql : select sum(c1) ,count(c1) from ct4 group by c1 having count(c7) < 1 or sum(c1) > 2 ; print ====> rows: $rows -if $rows != 7 then +if $rows != 8 then return -1 endi diff --git a/tests/script/tsim/query/crash_sql.sim b/tests/script/tsim/query/crash_sql.sim index 44671fbb0db9a4c583e874fb0158783cfbd3343e..b2b9239232a457b723337fbe1a84de509fb33b09 100644 --- a/tests/script/tsim/query/crash_sql.sim +++ b/tests/script/tsim/query/crash_sql.sim @@ -25,7 +25,7 @@ sql connect print =============== create database sql create database db sql show databases -if $rows != 2 then +if $rows != 3 then return -1 endi @@ -96,8 +96,6 @@ sql insert into ct4 values ( '2022-05-21 01:01:01.000', NULL, NULL, NULL, NULL, print ================ start query ====================== print ================ SQL used to cause taosd or taos shell crash -sql select sum(c1) ,count(c1) from ct4 group by c1 having sum(c10) between 0 and 1 ; - - +sql_error select sum(c1) ,count(c1) from ct4 group by c1 having sum(c10) between 0 and 1 ; #system sh/exec.sh -n dnode1 -s stop -x SIGINT \ No newline at end of file diff --git a/tests/script/tsim/stable/tag_filter.sim b/tests/script/tsim/stable/tag_filter.sim index c8edfb1ee3862046875c6f432be8602b43120a9a..04b8f21de9f50b29eb288ab6d5c4fcefc68a83d4 100644 --- a/tests/script/tsim/stable/tag_filter.sim +++ b/tests/script/tsim/stable/tag_filter.sim @@ -31,6 +31,7 @@ if $rows != 1 then return -1 endi + sql select * from db.stb where t1 < 1 if $rows != 0 then return -=1 @@ -56,4 +57,21 @@ if $rows != 5 then return -1 endi +sql select * from db.stb where t1 between 1 and 1 +if $rows != 1 then + return -1 +endi + +sql select * from db.stb where t1 between 1 and 6 +if $rows != 6 then + return -1 +endi + +sql select * from db.stb where t1 between 1 and 7 +if $rows != 6 then + return -1 +endi + + + system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/system-test/2-query/query_cols_tags_and_or.py b/tests/system-test/2-query/query_cols_tags_and_or.py index 5dc5a2f1236ea1d73f3a4a7f4775ec5133adc8d1..f8a44f735f27d9b13d88eea35936ad39c463a3ed 100644 --- a/tests/system-test/2-query/query_cols_tags_and_or.py +++ b/tests/system-test/2-query/query_cols_tags_and_or.py @@ -1066,9 +1066,7 @@ class TDTestCase: tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False # in query_sql = f'select {select_elm} from {tb_name} where c5 in (1, 6.6)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 6) if select_elm == "*" else False + tdSql.error(query_sql) # not in query_sql = f'select {select_elm} from {tb_name} where c5 not in (2, 3)' tdSql.query(query_sql) @@ -1076,14 +1074,10 @@ class TDTestCase: tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False # and query_sql = f'select {select_elm} from {tb_name} where c5 > 0 and c5 >= 1 and c5 < 7 and c5 <= 6.6 and c5 != 2 and c5 <> 2 and c5 = 6.6 and c5 is not null and c5 between 2 and 6.6 and c5 not between 1 and 2 and c5 in (2,6.6) and c5 not in (1,2)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 6) if select_elm == "*" else False + tdSql.error(query_sql) # or query_sql = f'select {select_elm} from {tb_name} where c5 > 6 or c5 >= 6.6 or c5 < 1 or c5 <= 0 or c5 != 1.1 or c5 <> 1.1 or c5 = 5 or c5 is null or c5 between 4 and 5 or c5 not between 1 and 3 or c5 in (4,5) or c5 not in (1.1,3)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 6) if select_elm == "*" else False + tdSql.error(query_sql) # and or query_sql = f'select {select_elm} from {tb_name} where c5 > 0 and c5 >= 1 or c5 < 5 and c5 <= 6.6 and c5 != 2 and c5 <> 2 and c5 = 4 or c5 is not null and c5 between 2 and 4 and c5 not between 1 and 2 and c5 in (2,4) and c5 not in (1,2)' tdSql.query(query_sql) @@ -1151,9 +1145,7 @@ class TDTestCase: tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False # in query_sql = f'select {select_elm} from {tb_name} where c6 in (1, 7.7)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 7) if select_elm == "*" else False + tdSql.error(query_sql) # not in query_sql = f'select {select_elm} from {tb_name} where c6 not in (2, 3)' tdSql.query(query_sql) @@ -1161,14 +1153,10 @@ class TDTestCase: tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False # and query_sql = f'select {select_elm} from {tb_name} where c6 > 0 and c6 >= 1 and c6 < 8 and c6 <= 7.7 and c6 != 2 and c6 <> 2 and c6 = 7.7 and c6 is not null and c6 between 2 and 7.7 and c6 not between 1 and 2 and c6 in (2,7.7) and c6 not in (1,2)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 7) if select_elm == "*" else False + tdSql.error(query_sql) # or query_sql = f'select {select_elm} from {tb_name} where c6 > 7 or c6 >= 7.7 or c6 < 1 or c6 <= 0 or c6 != 1.1 or c6 <> 1.1 or c6 = 5 or c6 is null or c6 between 4 and 5 or c6 not between 1 and 3 or c6 in (4,5) or c6 not in (1.1,3)' - tdSql.query(query_sql) - tdSql.checkRows(1) - tdSql.checkEqual(self.queryLastC10(query_sql), 7) if select_elm == "*" else False + tdSql.error(query_sql) # and or query_sql = f'select {select_elm} from {tb_name} where c6 > 0 and c6 >= 1 or c6 < 5 and c6 <= 7.7 and c6 != 2 and c6 <> 2 and c6 = 4 or c6 is not null and c6 between 2 and 4 and c6 not between 1 and 2 and c6 in (2,4) and c6 not in (1,2)' tdSql.query(query_sql) @@ -1410,9 +1398,7 @@ class TDTestCase: tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False # in query_sql = f'select {select_elm} from {tb_name} where c9 in ("binar", false)' - tdSql.query(query_sql) - tdSql.checkRows(2) - tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False + tdSql.error(query_sql) # # not in query_sql = f'select {select_elm} from {tb_name} where c9 not in (true)' tdSql.query(query_sql) @@ -1421,19 +1407,13 @@ class TDTestCase: # # and query_sql = f'select {select_elm} from {tb_name} where c9 = true and c9 != "false" and c9 <> "binary" and c9 is not null and c9 in ("binary", true) and c9 not in ("binary")' - tdSql.query(query_sql) - tdSql.checkRows(9) - tdSql.checkEqual(self.queryLastC10(query_sql), 9) if select_elm == "*" else False + tdSql.error(query_sql) # # or query_sql = f'select {select_elm} from {tb_name} where c9 = true or c9 != "false" or c9 <> "binary" or c9 = "true" or c9 is not null or c9 in ("binary", true) or c9 not in ("binary")' - tdSql.query(query_sql) - tdSql.checkRows(11) - tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False + tdSql.error(query_sql) # # and or query_sql = f'select {select_elm} from {tb_name} where c9 = true and c9 != "false" or c9 <> "binary" or c9 = "true" and c9 is not null or c9 in ("binary", true) or c9 not in ("binary")' - tdSql.query(query_sql) - tdSql.checkRows(11) - tdSql.checkEqual(self.queryLastC10(query_sql), 11) if select_elm == "*" else False + tdSql.error(query_sql) query_sql = f'select c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13 from {tb_name} where c9 > "binary" and c9 >= "binary8" or c9 < "binary9" and c9 <= "binary" and c9 != 2 and c9 <> 2 and c9 = 4 or c9 is not null and c9 between 2 and 4 and c9 not between 1 and 2 and c9 in (2,4) and c9 not in (1,2)' tdSql.query(query_sql) tdSql.checkRows(9) diff --git a/tests/test/c/tmqSim.c b/tests/test/c/tmqSim.c index d0b582f3758c2b37f1190f61cd9cd337ec708abb..4d9e2275f420349dfaa114eb47a9696054cc9740 100644 --- a/tests/test/c/tmqSim.c +++ b/tests/test/c/tmqSim.c @@ -514,14 +514,14 @@ void* consumeThreadFunc(void* param) { err = tmq_unsubscribe(pInfo->tmq); if (err) { pError("tmq_unsubscribe() fail, reason: %s\n", tmq_err2str(err)); - pInfo->consumeMsgCnt = -1; - return NULL; + /*pInfo->consumeMsgCnt = -1;*/ + /*return NULL;*/ } err = tmq_consumer_close(pInfo->tmq); if (err) { pError("tmq_consumer_close() fail, reason: %s\n", tmq_err2str(err)); - exit(-1); + /*exit(-1);*/ } pInfo->tmq = NULL;