Skip to content

TDengine
TDengine

taosdata / TDengine
大约 2 年 前同步成功

通知 1192
Star 22018
Fork 4786
TDengine
TDengine
提交 d5f6bb79 编写于 作者: W wenzhouwww@live.cn
浏览文件
操作

Merge remote-tracking branch 'origin/3.0' into test/abs_tag_compute

上级 9017f558 1946f39a
展开全部 隐藏空白更改
内联 并排
Showing with 2229 addition and 1443 deletion
docs-cn/20-third-party/01-grafana.mdx
浏览文件 @ d5f6bb79
...@@ -12,12 +12,38 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/ ...@@ -12,12 +12,38 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/
- TDengine 集群已经部署并正常运行 - TDengine 集群已经部署并正常运行
- taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter) - taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter)
记录以下信息：
- TDengine 集群 REST API 地址，如：`http://tdengine.local:6041`。
- TDengine 集群认证信息，可使用用户名及密码。
## 安装 Grafana ## 安装 Grafana
目前 TDengine 支持 Grafana 7.0 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：<https://grafana.com/grafana/download>。 目前 TDengine 支持 Grafana 7.5 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：<https://grafana.com/grafana/download>。
## 配置 Grafana ## 配置 Grafana
### 安装方式一：安装脚本
将集群信息设置为环境变量（也可以使用 [`.env`(dotenv) 文件](https://hexdocs.pm/dotenvy/dotenv-file-format.html) ）：
```sh
export TDENGINE_API=http://tdengine.local:6041
# user + password
export TDENGINE_USER=user
export TDENGINE_PASSWORD=password
```
运行安装脚本：
```sh
bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
```
该脚本将自动安装 Grafana 插件并配置数据源。安装完毕后，需要重启 Grafana 服务后生效。
### 安装方式二：手动安装 TDengine 数据源插件
使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。 使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。
```bash ```bash
...@@ -48,11 +74,7 @@ sudo unzip tdengine-datasource-$GF_VERSION.zip -d /var/lib/grafana/plugins/ ...@@ -48,11 +74,7 @@ sudo unzip tdengine-datasource-$GF_VERSION.zip -d /var/lib/grafana/plugins/
GF_INSTALL_PLUGINS=tdengine-datasource GF_INSTALL_PLUGINS=tdengine-datasource
``` ```
## 使用 Grafana 之后，用户可以直接通过 <http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示：
### 配置数据源
用户可以直接通过 <http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示：
![TDengine Database Grafana plugin add data source](./add_datasource1.webp) ![TDengine Database Grafana plugin add data source](./add_datasource1.webp)
...@@ -92,4 +114,11 @@ GF_INSTALL_PLUGINS=tdengine-datasource ...@@ -92,4 +114,11 @@ GF_INSTALL_PLUGINS=tdengine-datasource
### 导入 Dashboard ### 导入 Dashboard
在 2.3.3.0 及以上版本，您可以导入 TDinsight Dashboard (Grafana Dashboard ID： [15167](https://grafana.com/grafana/dashboards/15167)) 作为 TDengine 集群的监控可视化工具。安装和使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。 在数据源配置页面，您可以为该数据源导入 TDinsight 面板，作为 TDengine 集群的监控可视化工具。该 Dashboard 已发布在 Grafana：[Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167)) 。其他安装方式和相关使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。
使用 TDengine 作为数据源的其他面板，可以[在此搜索](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource)。以下是一份不完全列表：
- [15146](https://grafana.com/grafana/dashboards/15146): 监控多个 TDengine 集群
- [15155](https://grafana.com/grafana/dashboards/15155): TDengine 告警示例
- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight
- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf 采集节点信息的数据展示
docs-en/20-third-party/01-grafana.mdx
浏览文件 @ d5f6bb79
...@@ -14,12 +14,40 @@ In order for Grafana to add the TDengine data source successfully, the following ...@@ -14,12 +14,40 @@ In order for Grafana to add the TDengine data source successfully, the following
1. The TDengine cluster is deployed and functioning properly 1. The TDengine cluster is deployed and functioning properly
2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details. 2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
Record these values:
- TDengine REST API url: `http://tdengine.local:6041`.
- TDengine cluster authorization, with user + password.
## Installing Grafana ## Installing Grafana
TDengine currently supports Grafana versions 7.0 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: <https://grafana.com/grafana/download>. TDengine currently supports Grafana versions 7.5 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: <https://grafana.com/grafana/download>.
## Configuring Grafana ## Configuring Grafana
### Option 1: Install with `install.sh`
Set the url and authorization environment variables by `export` or a [`.env`(dotenv) file](https://hexdocs.pm/dotenvy/dotenv-file-format.html):
```sh
export TDENGINE_API=http://tdengine.local:6041
# user + password
export TDENGINE_USER=user
export TDENGINE_PASSWORD=password
```
Run `install.sh`:
```sh
bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
```
With this script, TDengine data source plugin and the Grafana data source will be installed and created automatically with Grafana provisioning configurations.
And then, restart Grafana service and open Grafana in web-browser, usually <http://localhost:3000>.
### Option 2: Install Plugin Manually
Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation. Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation.
```bash ```bash
...@@ -50,11 +78,7 @@ If Grafana is running in a Docker environment, the TDengine plugin can be automa ...@@ -50,11 +78,7 @@ If Grafana is running in a Docker environment, the TDengine plugin can be automa
GF_INSTALL_PLUGINS=tdengine-datasource GF_INSTALL_PLUGINS=tdengine-datasource
``` ```
## Using Grafana Now users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure.
### Configuring Data Sources
Users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure.
![TDengine Database TDinsight plugin add datasource 1](./grafana/add_datasource1.webp) ![TDengine Database TDinsight plugin add datasource 1](./grafana/add_datasource1.webp)
...@@ -94,4 +118,11 @@ Follow the default prompt to query the average system memory usage for the speci ...@@ -94,4 +118,11 @@ Follow the default prompt to query the average system memory usage for the speci
### Importing the Dashboard ### Importing the Dashboard
In version 2.3.3.0 and above, you can import the TDinsight Dashboard (Grafana Dashboard ID: [15168](https://grafana.com/grafana/dashboards/15167)) as a monitoring visualization tool for TDengine clusters. You can find installation and usage instructions in the TDinsight User Manual (/reference/tdinsight/). You can install TDinsight dashboard in data source configuration page (like `http://localhost:3000/datasources/edit/1/dashboards`) as a monitoring visualization tool for TDengine cluster. The dashboard is published in Grafana as [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167). Check the [TDinsight User Manual](/reference/tdinsight/) for the details.
For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list:
- [15146](https://grafana.com/grafana/dashboards/15146): Monitor multiple TDengine clusters.
- [15155](https://grafana.com/grafana/dashboards/15155): TDengine alert demo.
- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight.
- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf node metrics dashboard using TDengine data source.
include/client/taos.h
浏览文件 @ d5f6bb79
...@@ -103,10 +103,10 @@ typedef void (*__taos_async_fn_t)(void *param, TAOS_RES *, int code); ...@@ -103,10 +103,10 @@ typedef void (*__taos_async_fn_t)(void *param, TAOS_RES *, int code);
typedef struct TAOS_MULTI_BIND { typedef struct TAOS_MULTI_BIND {
int buffer_type; int buffer_type;
void * buffer; void *buffer;
uintptr_t buffer_length; uintptr_t buffer_length;
int32_t * length; int32_t *length;
char * is_null; char *is_null;
int num; int num;
} TAOS_MULTI_BIND; } TAOS_MULTI_BIND;
...@@ -130,7 +130,7 @@ DLL_EXPORT void taos_cleanup(void); ...@@ -130,7 +130,7 @@ DLL_EXPORT void taos_cleanup(void);
DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...); DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...);
DLL_EXPORT setConfRet taos_set_config(const char *config); DLL_EXPORT setConfRet taos_set_config(const char *config);
DLL_EXPORT int taos_init(void); DLL_EXPORT int taos_init(void);
DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port); DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port);
DLL_EXPORT TAOS *taos_connect_l(const char *ip, int ipLen, const char *user, int userLen, const char *pass, int passLen, DLL_EXPORT TAOS *taos_connect_l(const char *ip, int ipLen, const char *user, int userLen, const char *pass, int passLen,
const char *db, int dbLen, uint16_t port); const char *db, int dbLen, uint16_t port);
DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port); DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port);
...@@ -147,17 +147,17 @@ DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name ...@@ -147,17 +147,17 @@ DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name
DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert); DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums); DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes); DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes);
DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx); DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx);
DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt);
DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt);
DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt); DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt);
DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt);
DLL_EXPORT char * taos_stmt_errstr(TAOS_STMT *stmt); DLL_EXPORT char *taos_stmt_errstr(TAOS_STMT *stmt);
DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt);
DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt); DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt);
...@@ -179,11 +179,11 @@ DLL_EXPORT bool taos_is_update_query(TAOS_RES *res); ...@@ -179,11 +179,11 @@ DLL_EXPORT bool taos_is_update_query(TAOS_RES *res);
DLL_EXPORT int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows); DLL_EXPORT int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows);
DLL_EXPORT int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows); DLL_EXPORT int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows);
DLL_EXPORT int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData); DLL_EXPORT int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData);
DLL_EXPORT int * taos_get_column_data_offset(TAOS_RES *res, int columnIndex); DLL_EXPORT int *taos_get_column_data_offset(TAOS_RES *res, int columnIndex);
DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql); DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql);
DLL_EXPORT void taos_reset_current_db(TAOS *taos); DLL_EXPORT void taos_reset_current_db(TAOS *taos);
DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res); DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res);
DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res); DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res);
DLL_EXPORT const char *taos_get_server_info(TAOS *taos); DLL_EXPORT const char *taos_get_server_info(TAOS *taos);
...@@ -194,6 +194,7 @@ DLL_EXPORT int taos_errno(TAOS_RES *tres); ...@@ -194,6 +194,7 @@ DLL_EXPORT int taos_errno(TAOS_RES *tres);
DLL_EXPORT void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param); DLL_EXPORT void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param);
DLL_EXPORT void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param); DLL_EXPORT void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param);
DLL_EXPORT void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param);
// Shuduo: temporary enable for app build // Shuduo: temporary enable for app build
#if 1 #if 1
...@@ -204,17 +205,17 @@ DLL_EXPORT TAOS_RES *taos_consume(TAOS_SUB *tsub); ...@@ -204,17 +205,17 @@ DLL_EXPORT TAOS_RES *taos_consume(TAOS_SUB *tsub);
DLL_EXPORT void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress); DLL_EXPORT void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress);
#endif #endif
DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList); DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList);
DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision); DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision);
/* --------------------------TMQ INTERFACE------------------------------- */ /* --------------------------TMQ INTERFACE------------------------------- */
enum tmq_resp_err_t { enum {
TMQ_RESP_ERR__FAIL = -1, TMQ_RESP_ERR__FAIL = -1,
TMQ_RESP_ERR__SUCCESS = 0, TMQ_RESP_ERR__SUCCESS = 0,
}; };
typedef enum tmq_resp_err_t tmq_resp_err_t; typedef int32_t tmq_resp_err_t;
typedef struct tmq_t tmq_t; typedef struct tmq_t tmq_t;
typedef struct tmq_topic_vgroup_t tmq_topic_vgroup_t; typedef struct tmq_topic_vgroup_t tmq_topic_vgroup_t;
...@@ -229,7 +230,7 @@ DLL_EXPORT tmq_list_t *tmq_list_new(); ...@@ -229,7 +230,7 @@ DLL_EXPORT tmq_list_t *tmq_list_new();
DLL_EXPORT int32_t tmq_list_append(tmq_list_t *, const char *); DLL_EXPORT int32_t tmq_list_append(tmq_list_t *, const char *);
DLL_EXPORT void tmq_list_destroy(tmq_list_t *); DLL_EXPORT void tmq_list_destroy(tmq_list_t *);
DLL_EXPORT int32_t tmq_list_get_size(const tmq_list_t *); DLL_EXPORT int32_t tmq_list_get_size(const tmq_list_t *);
DLL_EXPORT char ** tmq_list_to_c_array(const tmq_list_t *); DLL_EXPORT char **tmq_list_to_c_array(const tmq_list_t *);
DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen); DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen);
...@@ -240,7 +241,7 @@ DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t); ...@@ -240,7 +241,7 @@ DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t);
DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list); DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list);
DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq); DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq);
DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics); DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics);
DLL_EXPORT TAOS_RES * tmq_consumer_poll(tmq_t *tmq, int64_t timeout); DLL_EXPORT TAOS_RES *tmq_consumer_poll(tmq_t *tmq, int64_t timeout);
DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq); DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq);
DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets); DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets);
DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param); DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param);
...@@ -260,7 +261,7 @@ enum tmq_conf_res_t { ...@@ -260,7 +261,7 @@ enum tmq_conf_res_t {
typedef enum tmq_conf_res_t tmq_conf_res_t; typedef enum tmq_conf_res_t tmq_conf_res_t;
DLL_EXPORT tmq_conf_t * tmq_conf_new(); DLL_EXPORT tmq_conf_t *tmq_conf_new();
DLL_EXPORT tmq_conf_res_t tmq_conf_set(tmq_conf_t *conf, const char *key, const char *value); DLL_EXPORT tmq_conf_res_t tmq_conf_set(tmq_conf_t *conf, const char *key, const char *value);
DLL_EXPORT void tmq_conf_destroy(tmq_conf_t *conf); DLL_EXPORT void tmq_conf_destroy(tmq_conf_t *conf);
DLL_EXPORT void tmq_conf_set_auto_commit_cb(tmq_conf_t *conf, tmq_commit_cb *cb, void *param); DLL_EXPORT void tmq_conf_set_auto_commit_cb(tmq_conf_t *conf, tmq_commit_cb *cb, void *param);
......
include/common/tdatablock.h
浏览文件 @ d5f6bb79
...@@ -206,7 +206,7 @@ SSDataBlock* blockDataExtractBlock(SSDataBlock* pBlock, int32_t startIndex, int3 ...@@ -206,7 +206,7 @@ SSDataBlock* blockDataExtractBlock(SSDataBlock* pBlock, int32_t startIndex, int3
size_t blockDataGetSize(const SSDataBlock* pBlock); size_t blockDataGetSize(const SSDataBlock* pBlock);
size_t blockDataGetRowSize(SSDataBlock* pBlock); size_t blockDataGetRowSize(SSDataBlock* pBlock);
double blockDataGetSerialRowSize(const SSDataBlock* pBlock); double blockDataGetSerialRowSize(const SSDataBlock* pBlock);
size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock); size_t blockDataGetSerialMetaSize(uint32_t numOfCols);
int32_t blockDataSort(SSDataBlock* pDataBlock, SArray* pOrderInfo); int32_t blockDataSort(SSDataBlock* pDataBlock, SArray* pOrderInfo);
int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullFirst); int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullFirst);
...@@ -238,7 +238,7 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pSchema, bool ...@@ -238,7 +238,7 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pSchema, bool
const char* stbFullName, int32_t vgId); const char* stbFullName, int32_t vgId);
static FORCE_INLINE int32_t blockGetEncodeSize(const SSDataBlock* pBlock) { static FORCE_INLINE int32_t blockGetEncodeSize(const SSDataBlock* pBlock) {
return blockDataGetSerialMetaSize(pBlock) + blockDataGetSize(pBlock); return blockDataGetSerialMetaSize(pBlock->info.numOfCols) + blockDataGetSize(pBlock);
} }
static FORCE_INLINE int32_t blockCompressColData(SColumnInfoData* pColRes, int32_t numOfRows, char* data, static FORCE_INLINE int32_t blockCompressColData(SColumnInfoData* pColRes, int32_t numOfRows, char* data,
......
include/common/tmsg.h
浏览文件 @ d5f6bb79
...@@ -2500,6 +2500,7 @@ typedef struct { ...@@ -2500,6 +2500,7 @@ typedef struct {
int64_t sliding; int64_t sliding;
int64_t dstTbUid; int64_t dstTbUid;
int32_t dstVgId; // for stream int32_t dstVgId; // for stream
SEpSet epSet;
char* expr; char* expr;
} STableIndexInfo; } STableIndexInfo;
......
include/libs/catalog/catalog.h
浏览文件 @ d5f6bb79
...@@ -71,16 +71,21 @@ typedef struct SCatalogReq { ...@@ -71,16 +71,21 @@ typedef struct SCatalogReq {
bool forceUpdate; bool forceUpdate;
} SCatalogReq; } SCatalogReq;
typedef struct SMetaRes {
int32_t code;
void* pRes;
} SMetaRes;
typedef struct SMetaData { typedef struct SMetaData {
SArray *pDbVgroup; // SArray<SArray<SVgroupInfo>*> SArray *pDbVgroup; // pRes = SArray<SVgroupInfo>*
SArray *pDbCfg; // SArray<SDbCfgInfo> SArray *pDbCfg; // pRes = SDbCfgInfo*
SArray *pDbInfo; // SArray<SDbInfo> SArray *pDbInfo; // pRes = SDbInfo*
SArray *pTableMeta; // SArray<STableMeta*> SArray *pTableMeta; // pRes = STableMeta*
SArray *pTableHash; // SArray<SVgroupInfo> SArray *pTableHash; // pRes = SVgroupInfo*
SArray *pUdfList; // SArray<SFuncInfo> SArray *pUdfList; // pRes = SFuncInfo*
SArray *pIndex; // SArray<SIndexInfo> SArray *pIndex; // pRes = SIndexInfo*
SArray *pUser; // SArray<bool> SArray *pUser; // pRes = bool*
SArray *pQnodeList; // SArray<SQueryNodeAddr> SArray *pQnodeList; // pRes = SQueryNodeAddr*
} SMetaData; } SMetaData;
typedef struct SCatalogCfg { typedef struct SCatalogCfg {
......
include/libs/function/function.h
浏览文件 @ d5f6bb79
...@@ -197,6 +197,7 @@ typedef struct SAggFunctionInfo { ...@@ -197,6 +197,7 @@ typedef struct SAggFunctionInfo {
struct SScalarParam { struct SScalarParam {
SColumnInfoData *columnData; SColumnInfoData *columnData;
SHashObj *pHashFilter; SHashObj *pHashFilter;
int32_t hashValueType;
void *param; // other parameter, such as meta handle from vnode, to extract table name/tag value void *param; // other parameter, such as meta handle from vnode, to extract table name/tag value
int32_t numOfRows; int32_t numOfRows;
}; };
......
include/libs/function/functionMgt.h
浏览文件 @ d5f6bb79
...@@ -127,6 +127,8 @@ typedef enum EFunctionType { ...@@ -127,6 +127,8 @@ typedef enum EFunctionType {
FUNCTION_TYPE_APERCENTILE_MERGE, FUNCTION_TYPE_APERCENTILE_MERGE,
FUNCTION_TYPE_SPREAD_PARTIAL, FUNCTION_TYPE_SPREAD_PARTIAL,
FUNCTION_TYPE_SPREAD_MERGE, FUNCTION_TYPE_SPREAD_MERGE,
FUNCTION_TYPE_HISTOGRAM_PARTIAL,
FUNCTION_TYPE_HISTOGRAM_MERGE,
// user defined funcion // user defined funcion
FUNCTION_TYPE_UDF = 10000 FUNCTION_TYPE_UDF = 10000
......
include/libs/index/index.h
浏览文件 @ d5f6bb79
...@@ -136,14 +136,14 @@ int indexRebuild(SIndex* index, SIndexOpts* opt); ...@@ -136,14 +136,14 @@ int indexRebuild(SIndex* index, SIndexOpts* opt);
* @param index (output, index json object) * @param index (output, index json object)
* @return error code * @return error code
*/ */
int tIndexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index); int indexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index);
/* /*
* close index * close index
* @param index (input, index to be closed) * @param index (input, index to be closed)
* @return void * @return void
*/ */
void tIndexJsonClose(SIndexJson* index); void indexJsonClose(SIndexJson* index);
/* /*
* insert terms into index * insert terms into index
...@@ -152,7 +152,7 @@ void tIndexJsonClose(SIndexJson* index); ...@@ -152,7 +152,7 @@ void tIndexJsonClose(SIndexJson* index);
* @param uid (input, uid of terms) * @param uid (input, uid of terms)
* @return error code * @return error code
*/ */
int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid); int indexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid);
/* /*
* search index * search index
* @param index (input, index object) * @param index (input, index object)
...@@ -161,7 +161,7 @@ int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid); ...@@ -161,7 +161,7 @@ int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid);
* @return error code * @return error code
*/ */
int tIndexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result); int indexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result);
/* /*
* @param * @param
* @param * @param
......
include/libs/qcom/query.h
浏览文件 @ d5f6bb79
...@@ -210,8 +210,8 @@ extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t ...@@ -210,8 +210,8 @@ extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t
#define NEED_CLIENT_RM_TBLMETA_ERROR(_code) \ #define NEED_CLIENT_RM_TBLMETA_ERROR(_code) \
((_code) == TSDB_CODE_PAR_TABLE_NOT_EXIST || (_code) == TSDB_CODE_VND_TB_NOT_EXIST || \ ((_code) == TSDB_CODE_PAR_TABLE_NOT_EXIST || (_code) == TSDB_CODE_VND_TB_NOT_EXIST || \
(_code) == TSDB_CODE_PAR_INVALID_COLUMNS_NUM || (_code) == TSDB_CODE_PAR_INVALID_COLUMN || \ (_code) == TSDB_CODE_PAR_INVALID_COLUMNS_NUM || (_code) == TSDB_CODE_PAR_INVALID_COLUMN || \
(_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code == TSDB_CODE_PAR_VALUE_TOO_LONG) || \ (_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code) == TSDB_CODE_PAR_VALUE_TOO_LONG || \
(_code == TSDB_CODE_PAR_INVALID_DROP_COL)) (_code) == TSDB_CODE_PAR_INVALID_DROP_COL || ((_code) == TSDB_CODE_TDB_INVALID_TABLE_ID))
#define NEED_CLIENT_REFRESH_VG_ERROR(_code) \ #define NEED_CLIENT_REFRESH_VG_ERROR(_code) \
((_code) == TSDB_CODE_VND_HASH_MISMATCH || (_code) == TSDB_CODE_VND_INVALID_VGROUP_ID) ((_code) == TSDB_CODE_VND_HASH_MISMATCH || (_code) == TSDB_CODE_VND_INVALID_VGROUP_ID)
#define NEED_CLIENT_REFRESH_TBLMETA_ERROR(_code) ((_code) == TSDB_CODE_TDB_TABLE_RECREATED) #define NEED_CLIENT_REFRESH_TBLMETA_ERROR(_code) ((_code) == TSDB_CODE_TDB_TABLE_RECREATED)
......
include/libs/stream/tstream.h
浏览文件 @ d5f6bb79
...@@ -339,6 +339,7 @@ typedef struct { ...@@ -339,6 +339,7 @@ typedef struct {
int32_t sourceTaskId; int32_t sourceTaskId;
int32_t sourceVg; int32_t sourceVg;
int32_t sourceChildId; int32_t sourceChildId;
int32_t upstreamNodeId;
#if 0 #if 0
int64_t sourceVer; int64_t sourceVer;
#endif #endif
......
include/libs/sync/syncTools.h
浏览文件 @ d5f6bb79
...@@ -33,7 +33,7 @@ SSyncNode* syncNodeAcquire(int64_t rid); ...@@ -33,7 +33,7 @@ SSyncNode* syncNodeAcquire(int64_t rid);
void syncNodeRelease(SSyncNode* pNode); void syncNodeRelease(SSyncNode* pNode);
int32_t syncGetRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg); int32_t syncGetRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg);
int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg); int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcHandleInfo* pInfo);
void syncSetMsgCb(int64_t rid, const SMsgCb* msgcb); void syncSetMsgCb(int64_t rid, const SMsgCb* msgcb);
char* sync2SimpleStr(int64_t rid); char* sync2SimpleStr(int64_t rid);
......
include/libs/transport/trpc.h
浏览文件 @ d5f6bb79
...@@ -33,7 +33,10 @@ extern int32_t tsRpcHeadSize; ...@@ -33,7 +33,10 @@ extern int32_t tsRpcHeadSize;
typedef struct { typedef struct {
uint32_t clientIp; uint32_t clientIp;
uint16_t clientPort; uint16_t clientPort;
char user[TSDB_USER_LEN]; union {
char user[TSDB_USER_LEN];
int64_t applyIndex;
};
} SRpcConnInfo; } SRpcConnInfo;
typedef struct SRpcHandleInfo { typedef struct SRpcHandleInfo {
...@@ -43,6 +46,7 @@ typedef struct SRpcHandleInfo { ...@@ -43,6 +46,7 @@ typedef struct SRpcHandleInfo {
int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp); int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp);
int32_t persistHandle; // persist handle or not int32_t persistHandle; // persist handle or not
SRpcConnInfo connInfo;
// app info // app info
void *ahandle; // app handle set by client void *ahandle; // app handle set by client
void *wrapper; // wrapper handle void *wrapper; // wrapper handle
......
include/util/taoserror.h
浏览文件 @ d5f6bb79
...@@ -339,9 +339,9 @@ int32_t* taosGetErrno(); ...@@ -339,9 +339,9 @@ int32_t* taosGetErrno();
#define TSDB_CODE_TDB_TAG_VER_OUT_OF_DATE TAOS_DEF_ERROR_CODE(0, 0x060D) #define TSDB_CODE_TDB_TAG_VER_OUT_OF_DATE TAOS_DEF_ERROR_CODE(0, 0x060D)
#define TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x060E) #define TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x060E)
#define TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP TAOS_DEF_ERROR_CODE(0, 0x060F) #define TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP TAOS_DEF_ERROR_CODE(0, 0x060F)
#define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0600) #define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0610)
#define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0601) #define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0611)
#define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0602) #define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0612)
#define TSDB_CODE_TDB_FILE_ALREADY_EXISTS TAOS_DEF_ERROR_CODE(0, 0x0613) #define TSDB_CODE_TDB_FILE_ALREADY_EXISTS TAOS_DEF_ERROR_CODE(0, 0x0613)
#define TSDB_CODE_TDB_TABLE_RECONFIGURE TAOS_DEF_ERROR_CODE(0, 0x0614) #define TSDB_CODE_TDB_TABLE_RECONFIGURE TAOS_DEF_ERROR_CODE(0, 0x0614)
#define TSDB_CODE_TDB_IVD_CREATE_TABLE_INFO TAOS_DEF_ERROR_CODE(0, 0x0615) #define TSDB_CODE_TDB_IVD_CREATE_TABLE_INFO TAOS_DEF_ERROR_CODE(0, 0x0615)
......
include/util/tdef.h
浏览文件 @ d5f6bb79
...@@ -356,8 +356,8 @@ typedef enum ELogicConditionType { ...@@ -356,8 +356,8 @@ typedef enum ELogicConditionType {
#define TSDB_DEFAULT_EXPLAIN_VERBOSE false #define TSDB_DEFAULT_EXPLAIN_VERBOSE false
#define TSDB_EXPLAIN_RESULT_ROW_SIZE 1024 #define TSDB_EXPLAIN_RESULT_ROW_SIZE 512
#define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY PLAN" #define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY_PLAN"
#define TSDB_MAX_FIELD_LEN 16384 #define TSDB_MAX_FIELD_LEN 16384
#define TSDB_MAX_BINARY_LEN (TSDB_MAX_FIELD_LEN - TSDB_KEYSIZE) // keep 16384 #define TSDB_MAX_BINARY_LEN (TSDB_MAX_FIELD_LEN - TSDB_KEYSIZE) // keep 16384
......
source/client/inc/clientInt.h
浏览文件 @ d5f6bb79
...@@ -45,7 +45,7 @@ extern "C" { ...@@ -45,7 +45,7 @@ extern "C" {
#define ERROR_MSG_BUF_DEFAULT_SIZE 512 #define ERROR_MSG_BUF_DEFAULT_SIZE 512
#define HEARTBEAT_INTERVAL 1500 // ms #define HEARTBEAT_INTERVAL 1500 // ms
#define SYNC_ON_TOP_OF_ASYNC 0 #define SYNC_ON_TOP_OF_ASYNC 1
enum { enum {
RES_TYPE__QUERY = 1, RES_TYPE__QUERY = 1,
...@@ -144,7 +144,7 @@ typedef struct STscObj { ...@@ -144,7 +144,7 @@ typedef struct STscObj {
int32_t numOfReqs; // number of sqlObj bound to this connection int32_t numOfReqs; // number of sqlObj bound to this connection
SAppInstInfo* pAppInfo; SAppInstInfo* pAppInfo;
SHashObj* pRequests; SHashObj* pRequests;
int8_t schemalessType; int8_t schemalessType; // todo remove it, this attribute should be move to request
} STscObj; } STscObj;
typedef struct SResultColumn { typedef struct SResultColumn {
...@@ -171,6 +171,7 @@ typedef struct SReqResultInfo { ...@@ -171,6 +171,7 @@ typedef struct SReqResultInfo {
uint32_t current; uint32_t current;
bool completed; bool completed;
int32_t precision; int32_t precision;
bool convertUcs4;
int32_t payloadLen; int32_t payloadLen;
} SReqResultInfo; } SReqResultInfo;
...@@ -222,7 +223,7 @@ typedef struct SSyncQueryParam { ...@@ -222,7 +223,7 @@ typedef struct SSyncQueryParam {
SRequestObj* pRequest; SRequestObj* pRequest;
} SSyncQueryParam; } SSyncQueryParam;
void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4); void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4);
void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4); void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4);
void doSetOneRowPtr(SReqResultInfo* pResultInfo); void doSetOneRowPtr(SReqResultInfo* pResultInfo);
......
source/client/src/clientEnv.c
浏览文件 @ d5f6bb79
...@@ -191,6 +191,8 @@ void *createRequest(STscObj *pObj, int32_t type) { ...@@ -191,6 +191,8 @@ void *createRequest(STscObj *pObj, int32_t type) {
pRequest->requestId = generateRequestId(); pRequest->requestId = generateRequestId();
pRequest->metric.start = taosGetTimestampUs(); pRequest->metric.start = taosGetTimestampUs();
pRequest->body.resInfo.convertUcs4 = true; // convert ucs4 by default
pRequest->type = type; pRequest->type = type;
pRequest->pTscObj = pObj; pRequest->pTscObj = pObj;
pRequest->msgBuf = taosMemoryCalloc(1, ERROR_MSG_BUF_DEFAULT_SIZE); pRequest->msgBuf = taosMemoryCalloc(1, ERROR_MSG_BUF_DEFAULT_SIZE);
......
source/client/src/clientImpl.c
浏览文件 @ d5f6bb79
...@@ -580,7 +580,8 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) { ...@@ -580,7 +580,8 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) {
STscObj* pTscObj = pRequest->pTscObj; STscObj* pTscObj = pRequest->pTscObj;
if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code)) { if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code)) {
tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code),
pRequest->retry, pRequest->requestId);
pRequest->prevCode = code; pRequest->prevCode = code;
doAsyncQuery(pRequest, true); doAsyncQuery(pRequest, true);
return; return;
...@@ -592,6 +593,7 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) { ...@@ -592,6 +593,7 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) {
pRequest->code = code; pRequest->code = code;
} }
tscDebug("schedulerExecCb request type %s", TMSG_INFO(pRequest->type));
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) { if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) {
removeMeta(pTscObj, pRequest->tableList); removeMeta(pTscObj, pRequest->tableList);
} }
...@@ -695,6 +697,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { ...@@ -695,6 +697,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
schedulerAsyncExecJob(pAppInfo->pTransporter, pNodeList, pRequest->body.pDag, &pRequest->body.queryJob, schedulerAsyncExecJob(pAppInfo->pTransporter, pNodeList, pRequest->body.pDag, &pRequest->body.queryJob,
pRequest->sqlstr, pRequest->metric.start, schedulerExecCb, pRequest); pRequest->sqlstr, pRequest->metric.start, schedulerExecCb, pRequest);
} else {
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
} }
//todo not to be released here //todo not to be released here
...@@ -702,7 +706,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { ...@@ -702,7 +706,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) {
break; break;
} }
case QUERY_EXEC_MODE_EMPTY_RESULT: case QUERY_EXEC_MODE_EMPTY_RESULT:
pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT; pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT;
pRequest->body.queryFp(pRequest->body.param, pRequest, 0);
break; break;
default: default:
break; break;
...@@ -1109,7 +1114,7 @@ static void syncFetchFn(void* param, TAOS_RES* res, int32_t numOfRows) { ...@@ -1109,7 +1114,7 @@ static void syncFetchFn(void* param, TAOS_RES* res, int32_t numOfRows) {
tsem_post(&pParam->sem); tsem_post(&pParam->sem);
} }
void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) { void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) {
assert(pRequest != NULL); assert(pRequest != NULL);
SReqResultInfo* pResultInfo = &pRequest->body.resInfo; SReqResultInfo* pResultInfo = &pRequest->body.resInfo;
...@@ -1121,6 +1126,10 @@ void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUc ...@@ -1121,6 +1126,10 @@ void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUc
} }
SSyncQueryParam* pParam = pRequest->body.param; SSyncQueryParam* pParam = pRequest->body.param;
// convert ucs4 to native multi-bytes string
pResultInfo->convertUcs4 = convertUcs4;
taos_fetch_rows_a(pRequest, syncFetchFn, pParam); taos_fetch_rows_a(pRequest, syncFetchFn, pParam);
tsem_wait(&pParam->sem); tsem_wait(&pParam->sem);
} }
...@@ -1339,6 +1348,17 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 ...@@ -1339,6 +1348,17 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32
uint64_t groupId = *(uint64_t*)p; uint64_t groupId = *(uint64_t*)p;
p += sizeof(uint64_t); p += sizeof(uint64_t);
// check fields
for(int32_t i = 0; i < numOfCols; ++i) {
int16_t type = *(int16_t*) p;
p += sizeof(int16_t);
int32_t bytes = *(int32_t*) p;
p += sizeof(int32_t);
// ASSERT(type == pFields[i].type && bytes == pFields[i].bytes);
}
int32_t* colLength = (int32_t*)p; int32_t* colLength = (int32_t*)p;
p += sizeof(int32_t) * numOfCols; p += sizeof(int32_t) * numOfCols;
......
source/client/src/clientMain.c
浏览文件 @ d5f6bb79
...@@ -219,14 +219,13 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { ...@@ -219,14 +219,13 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) {
if (TD_RES_QUERY(res)) { if (TD_RES_QUERY(res)) {
SRequestObj *pRequest = (SRequestObj *)res; SRequestObj *pRequest = (SRequestObj *)res;
#if SYNC_ON_TOP_OF_ASYNC
return doAsyncFetchRows(pRequest, true, true);
#else
if (pRequest->type == TSDB_SQL_RETRIEVE_EMPTY_RESULT || pRequest->type == TSDB_SQL_INSERT || if (pRequest->type == TSDB_SQL_RETRIEVE_EMPTY_RESULT || pRequest->type == TSDB_SQL_INSERT ||
pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0) { pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0) {
return NULL; return NULL;
} }
#if SYNC_ON_TOP_OF_ASYNC
return doAsyncFetchRow(pRequest, true, true);
#else
return doFetchRows(pRequest, true, true); return doFetchRows(pRequest, true, true);
#endif #endif
...@@ -489,6 +488,7 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) { ...@@ -489,6 +488,7 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) {
if (res == NULL) { if (res == NULL) {
return 0; return 0;
} }
if (TD_RES_QUERY(res)) { if (TD_RES_QUERY(res)) {
SRequestObj *pRequest = (SRequestObj *)res; SRequestObj *pRequest = (SRequestObj *)res;
...@@ -500,7 +500,11 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) { ...@@ -500,7 +500,11 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) {
return 0; return 0;
} }
doFetchRows(pRequest, false, true); #if SYNC_ON_TOP_OF_ASYNC
doAsyncFetchRows(pRequest, false, true);
#else
doFetchRows(pRequest, true, true);
#endif
// TODO refactor // TODO refactor
SReqResultInfo *pResultInfo = &pRequest->body.resInfo; SReqResultInfo *pResultInfo = &pRequest->body.resInfo;
...@@ -548,7 +552,11 @@ int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData) { ...@@ -548,7 +552,11 @@ int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData) {
return 0; return 0;
} }
#if SYNC_ON_TOP_OF_ASYNC
doAsyncFetchRows(pRequest, false, false);
#else
doFetchRows(pRequest, false, false); doFetchRows(pRequest, false, false);
#endif
SReqResultInfo *pResultInfo = &pRequest->body.resInfo; SReqResultInfo *pResultInfo = &pRequest->body.resInfo;
...@@ -625,8 +633,10 @@ void retrieveMetaCallback(SMetaData* pResultMeta, void* param, int32_t code) { ...@@ -625,8 +633,10 @@ void retrieveMetaCallback(SMetaData* pResultMeta, void* param, int32_t code) {
taosMemoryFree(pWrapper); taosMemoryFree(pWrapper);
launchAsyncQuery(pRequest, pQuery); launchAsyncQuery(pRequest, pQuery);
} else { } else {
tscDebug("error happens, code:%d", code);
if (NEED_CLIENT_HANDLE_ERROR(code)) { if (NEED_CLIENT_HANDLE_ERROR(code)) {
tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code),
pRequest->retry, pRequest->requestId);
pRequest->prevCode = code; pRequest->prevCode = code;
doAsyncQuery(pRequest, true); doAsyncQuery(pRequest, true);
return; return;
...@@ -691,6 +701,7 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) { ...@@ -691,6 +701,7 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) {
.pTransporter = pTscObj->pAppInfo->pTransporter, .pTransporter = pTscObj->pAppInfo->pTransporter,
.pStmtCb = NULL, .pStmtCb = NULL,
.pUser = pTscObj->user, .pUser = pTscObj->user,
.schemalessType = pTscObj->schemalessType,
.isSuperUser = (0 == strcmp(pTscObj->user, TSDB_DEFAULT_USER)), .isSuperUser = (0 == strcmp(pTscObj->user, TSDB_DEFAULT_USER)),
.async = true,}; .async = true,};
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
...@@ -699,13 +710,14 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) { ...@@ -699,13 +710,14 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) {
void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) { void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) {
SParseContext* pCxt = NULL; SParseContext* pCxt = NULL;
STscObj *pTscObj = pRequest->pTscObj; STscObj *pTscObj = pRequest->pTscObj;
int32_t code = 0;
if (pRequest->retry++ > REQUEST_TOTAL_EXEC_TIMES) { if (pRequest->retry++ > REQUEST_TOTAL_EXEC_TIMES) {
pRequest->code = pRequest->prevCode; code = pRequest->prevCode;
goto _error; goto _error;
} }
int32_t code = createParseContext(pRequest, &pCxt); code = createParseContext(pRequest, &pCxt);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _error; goto _error;
} }
...@@ -742,7 +754,7 @@ void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) { ...@@ -742,7 +754,7 @@ void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) {
} }
_error: _error:
tscError("0x%"PRIx64" error happens, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); tscError("0x%"PRIx64" error happens, code:%d - %s, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code), pRequest->requestId);
terrno = code; terrno = code;
pRequest->code = code; pRequest->code = code;
pRequest->body.queryFp(pRequest->body.param, pRequest, code); pRequest->body.queryFp(pRequest->body.param, pRequest, code);
...@@ -763,11 +775,11 @@ static void fetchCallback(void* pResult, void* param, int32_t code) { ...@@ -763,11 +775,11 @@ static void fetchCallback(void* pResult, void* param, int32_t code) {
} }
if (pRequest->code != TSDB_CODE_SUCCESS) { if (pRequest->code != TSDB_CODE_SUCCESS) {
pRequest->code = code;
pRequest->body.fetchFp(pRequest->body.param, pRequest, 0); pRequest->body.fetchFp(pRequest->body.param, pRequest, 0);
return;
} }
pRequest->code = setQueryResultFromRsp(&pRequest->body.resInfo, (SRetrieveTableRsp*)pResultInfo->pData, true, false); pRequest->code = setQueryResultFromRsp(pResultInfo, (SRetrieveTableRsp*)pResultInfo->pData, pResultInfo->convertUcs4, false);
if (pRequest->code != TSDB_CODE_SUCCESS) { if (pRequest->code != TSDB_CODE_SUCCESS) {
pResultInfo->numOfRows = 0; pResultInfo->numOfRows = 0;
pRequest->code = code; pRequest->code = code;
...@@ -807,6 +819,13 @@ void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) { ...@@ -807,6 +819,13 @@ void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
schedulerAsyncFetchRows(pRequest->body.queryJob, fetchCallback, pRequest); schedulerAsyncFetchRows(pRequest->body.queryJob, fetchCallback, pRequest);
} }
void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param) {
ASSERT(res != NULL && fp != NULL);
SRequestObj *pRequest = res;
pRequest->body.resInfo.convertUcs4 = false;
taos_fetch_rows_a(res, fp, param);
}
TAOS_SUB *taos_subscribe(TAOS *taos, int restart, const char *topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp, TAOS_SUB *taos_subscribe(TAOS *taos, int restart, const char *topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp,
void *param, int interval) { void *param, int interval) {
// TODO // TODO
......
source/client/src/tmq.c
浏览文件 @ d5f6bb79
...@@ -323,7 +323,7 @@ static int32_t tmqMakeTopicVgKey(char* dst, const char* topicName, int32_t vg) { ...@@ -323,7 +323,7 @@ static int32_t tmqMakeTopicVgKey(char* dst, const char* topicName, int32_t vg) {
int32_t tmqCommitCb(void* param, const SDataBuf* pMsg, int32_t code) { int32_t tmqCommitCb(void* param, const SDataBuf* pMsg, int32_t code) {
SMqCommitCbParam* pParam = (SMqCommitCbParam*)param; SMqCommitCbParam* pParam = (SMqCommitCbParam*)param;
pParam->rspErr = code == 0 ? TMQ_RESP_ERR__SUCCESS : TMQ_RESP_ERR__FAIL; pParam->rspErr = code;
if (pParam->async) { if (pParam->async) {
if (pParam->automatic && pParam->tmq->commitCb) { if (pParam->automatic && pParam->tmq->commitCb) {
pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets, pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets,
...@@ -432,12 +432,13 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_ ...@@ -432,12 +432,13 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_
code = pParam->rspErr; code = pParam->rspErr;
tsem_destroy(&pParam->rspSem); tsem_destroy(&pParam->rspSem);
taosMemoryFree(pParam); taosMemoryFree(pParam);
} else {
code = 0;
} }
// avoid double free if msg is sent // avoid double free if msg is sent
buf = NULL; buf = NULL;
code = 0;
END: END:
if (buf) taosMemoryFree(buf); if (buf) taosMemoryFree(buf);
/*if (pParam) taosMemoryFree(pParam);*/ /*if (pParam) taosMemoryFree(pParam);*/
...@@ -445,9 +446,9 @@ END: ...@@ -445,9 +446,9 @@ END:
if (code != 0 && async) { if (code != 0 && async) {
if (automatic) { if (automatic) {
tmq->commitCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam); tmq->commitCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam);
} else { } else {
userCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, userParam); userCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, userParam);
} }
} }
...@@ -493,7 +494,6 @@ int32_t tmqHandleAllDelayedTask(tmq_t* tmq) { ...@@ -493,7 +494,6 @@ int32_t tmqHandleAllDelayedTask(tmq_t* tmq) {
tmqAskEp(tmq, true); tmqAskEp(tmq, true);
taosTmrReset(tmqAssignDelayedHbTask, 1000, tmq, tmqMgmt.timer, &tmq->hbTimer); taosTmrReset(tmqAssignDelayedHbTask, 1000, tmq, tmqMgmt.timer, &tmq->hbTimer);
} else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) { } else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) {
/*tmq_commit(tmq, NULL, true);*/
tmqCommitInner(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam); tmqCommitInner(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam);
taosTmrReset(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, tmq, tmqMgmt.timer, &tmq->commitTimer); taosTmrReset(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, tmq, tmqMgmt.timer, &tmq->commitTimer);
} else if (*pTaskType == TMQ_DELAYED_TASK__REPORT) { } else if (*pTaskType == TMQ_DELAYED_TASK__REPORT) {
...@@ -666,94 +666,6 @@ FAIL: ...@@ -666,94 +666,6 @@ FAIL:
tmq_resp_err_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) { tmq_resp_err_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) {
return tmqCommitInner(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam); return tmqCommitInner(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam);
#if 0
// TODO: add read write lock
SRequestObj* pRequest = NULL;
tmq_resp_err_t resp = TMQ_RESP_ERR__SUCCESS;
// build msg
// send to mnode
SMqCMCommitOffsetReq req;
SArray* pOffsets = NULL;
if (offsets == NULL) {
pOffsets = taosArrayInit(0, sizeof(SMqOffset));
for (int i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) {
SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i);
for (int j = 0; j < taosArrayGetSize(pTopic->vgs); j++) {
SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j);
SMqOffset offset;
strcpy(offset.topicName, pTopic->topicName);
strcpy(offset.cgroup, tmq->groupId);
offset.vgId = pVg->vgId;
offset.offset = pVg->currentOffset;
taosArrayPush(pOffsets, &offset);
}
}
req.num = pOffsets->size;
req.offsets = pOffsets->pData;
} else {
req.num = taosArrayGetSize(&offsets->container);
req.offsets = (SMqOffset*)offsets->container.pData;
}
SEncoder encoder;
tEncoderInit(&encoder, NULL, 0);
tEncodeSMqCMCommitOffsetReq(&encoder, &req);
int32_t tlen = encoder.pos;
void* buf = taosMemoryMalloc(tlen);
if (buf == NULL) {
tEncoderClear(&encoder);
return -1;
}
tEncoderClear(&encoder);
tEncoderInit(&encoder, buf, tlen);
tEncodeSMqCMCommitOffsetReq(&encoder, &req);
tEncoderClear(&encoder);
pRequest = createRequest(tmq->pTscObj, NULL, NULL, TDMT_MND_MQ_COMMIT_OFFSET);
if (pRequest == NULL) {
tscError("failed to malloc request");
}
SMqCommitCbParam* pParam = taosMemoryCalloc(1, sizeof(SMqCommitCbParam));
if (pParam == NULL) {
return -1;
}
pParam->tmq = tmq;
tsem_init(&pParam->rspSem, 0, 0);
pParam->async = async;
pParam->offsets = pOffsets;
pRequest->body.requestMsg = (SDataBuf){
.pData = buf,
.len = tlen,
.handle = NULL,
};
SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest);
sendInfo->requestObjRefId = 0;
sendInfo->param = pParam;
sendInfo->fp = tmqCommitCb;
SEpSet epSet = getEpSet_s(&tmq->pTscObj->pAppInfo->mgmtEp);
int64_t transporterId = 0;
asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
if (!async) {
tsem_wait(&pParam->rspSem);
resp = pParam->rspErr;
tsem_destroy(&pParam->rspSem);
taosMemoryFree(pParam);
if (pOffsets) {
taosArrayDestroy(pOffsets);
}
}
return resp;
#endif
} }
tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) {
...@@ -858,93 +770,6 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para ...@@ -858,93 +770,6 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para
conf->commitCbUserParam = param; conf->commitCbUserParam = param;
} }
#if 0
TAOS_RES* tmq_create_stream(TAOS* taos, const char* streamName, const char* tbName, const char* sql) {
STscObj* pTscObj = (STscObj*)taos;
SRequestObj* pRequest = NULL;
SQuery* pQueryNode = NULL;
char* astStr = NULL;
int32_t sqlLen;
terrno = TSDB_CODE_SUCCESS;
if (taos == NULL || streamName == NULL || sql == NULL) {
tscError("invalid parameters for creating stream, connObj:%p, stream name:%s, sql:%s", taos, streamName, sql);
terrno = TSDB_CODE_TSC_INVALID_INPUT;
goto _return;
}
sqlLen = strlen(sql);
if (strlen(tbName) >= TSDB_TABLE_NAME_LEN) {
tscError("output tb name too long, max length:%d", TSDB_TABLE_NAME_LEN - 1);
terrno = TSDB_CODE_TSC_INVALID_INPUT;
goto _return;
}
if (sqlLen > TSDB_MAX_ALLOWED_SQL_LEN) {
tscError("sql string exceeds max length:%d", TSDB_MAX_ALLOWED_SQL_LEN);
terrno = TSDB_CODE_TSC_EXCEED_SQL_LIMIT;
goto _return;
}
tscDebug("start to create stream: %s", streamName);
int32_t code = 0;
CHECK_CODE_GOTO(buildRequest(pTscObj, sql, sqlLen, &pRequest), _return);
CHECK_CODE_GOTO(parseSql(pRequest, false, &pQueryNode, NULL), _return);
CHECK_CODE_GOTO(nodesNodeToString(pQueryNode->pRoot, false, &astStr, NULL), _return);
/*printf("%s\n", pStr);*/
SName name = {.acctId = pTscObj->acctId, .type = TSDB_TABLE_NAME_T};
strcpy(name.dbname, pRequest->pDb);
strcpy(name.tname, streamName);
SCMCreateStreamReq req = {
.igExists = 1,
.ast = astStr,
.sql = (char*)sql,
};
tNameExtractFullName(&name, req.name);
strcpy(req.targetStbFullName, tbName);
int tlen = tSerializeSCMCreateStreamReq(NULL, 0, &req);
void* buf = taosMemoryMalloc(tlen);
if (buf == NULL) {
goto _return;
}
tSerializeSCMCreateStreamReq(buf, tlen, &req);
pRequest->body.requestMsg = (SDataBuf){
.pData = buf,
.len = tlen,
.handle = NULL,
};
pRequest->type = TDMT_MND_CREATE_STREAM;
SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest);
SEpSet epSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
int64_t transporterId = 0;
asyncSendMsgToServer(pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
tsem_wait(&pRequest->body.rspSem);
_return:
taosMemoryFreeClear(astStr);
qDestroyQuery(pQueryNode);
/*if (sendInfo != NULL) {*/
/*destroySendMsgInfo(sendInfo);*/
/*}*/
if (pRequest != NULL && terrno != TSDB_CODE_SUCCESS) {
pRequest->code = terrno;
}
return pRequest;
}
#endif
#if 0 #if 0
int32_t tmqGetSkipLogNum(tmq_message_t* tmq_message) { int32_t tmqGetSkipLogNum(tmq_message_t* tmq_message) {
if (tmq_message == NULL) return 0; if (tmq_message == NULL) return 0;
...@@ -1474,16 +1299,16 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) { ...@@ -1474,16 +1299,16 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) {
tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) {
if (tmq->status == TMQ_CONSUMER_STATUS__READY) { if (tmq->status == TMQ_CONSUMER_STATUS__READY) {
tmq_resp_err_t rsp = tmq_commit_sync(tmq, NULL); tmq_resp_err_t rsp = tmq_commit_sync(tmq, NULL);
if (rsp == TMQ_RESP_ERR__FAIL) { if (rsp != TMQ_RESP_ERR__SUCCESS) {
return TMQ_RESP_ERR__FAIL; return rsp;
} }
tmq_list_t* lst = tmq_list_new(); tmq_list_t* lst = tmq_list_new();
rsp = tmq_subscribe(tmq, lst); rsp = tmq_subscribe(tmq, lst);
tmq_list_destroy(lst); tmq_list_destroy(lst);
if (rsp == TMQ_RESP_ERR__FAIL) { if (rsp != TMQ_RESP_ERR__SUCCESS) {
return TMQ_RESP_ERR__FAIL; return rsp;
} }
} }
// TODO: free resources // TODO: free resources
...@@ -1493,8 +1318,11 @@ tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { ...@@ -1493,8 +1318,11 @@ tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) {
const char* tmq_err2str(tmq_resp_err_t err) { const char* tmq_err2str(tmq_resp_err_t err) {
if (err == TMQ_RESP_ERR__SUCCESS) { if (err == TMQ_RESP_ERR__SUCCESS) {
return "success"; return "success";
} else if (err == TMQ_RESP_ERR__FAIL) {
return "fail";
} else {
return tstrerror(err);
} }
return "fail";
} }
const char* tmq_get_topic_name(TAOS_RES* res) { const char* tmq_get_topic_name(TAOS_RES* res) {
...@@ -1536,10 +1364,11 @@ const char* tmq_get_table_name(TAOS_RES* res) { ...@@ -1536,10 +1364,11 @@ const char* tmq_get_table_name(TAOS_RES* res) {
} }
return NULL; return NULL;
} }
DLL_EXPORT void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) {
void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) {
tmqCommitInner(tmq, offsets, 0, 1, cb, param); tmqCommitInner(tmq, offsets, 0, 1, cb, param);
} }
DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) { tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) {
return tmqCommitInner(tmq, offsets, 0, 0, NULL, NULL); return tmqCommitInner(tmq, offsets, 0, 0, NULL, NULL);
} }
source/client/test/clientTests.cpp
浏览文件 @ d5f6bb79
...@@ -780,31 +780,28 @@ TEST(testCase, async_api_test) { ...@@ -780,31 +780,28 @@ TEST(testCase, async_api_test) {
taos_query(pConn, "use test"); taos_query(pConn, "use test");
TAOS_RES* pRes = taos_query(pConn, "select * from t1"); TAOS_RES* pRes = taos_query(pConn, "desc abc1.tu");
taos_query(pConn, "alter table t1 add column b int");
pRes = taos_query(pConn, "insert into t1 values(now, 1, 2)");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
printf("failed, reason:%s\n", taos_errstr(pRes)); printf("failed, reason:%s\n", taos_errstr(pRes));
} }
// int32_t n = 0; int32_t n = 0;
// TAOS_ROW pRow = NULL; TAOS_ROW pRow = NULL;
// TAOS_FIELD* pFields = taos_fetch_fields(pRes); TAOS_FIELD* pFields = taos_fetch_fields(pRes);
// int32_t numOfFields = taos_num_fields(pRes); int32_t numOfFields = taos_num_fields(pRes);
//
// char str[512] = {0}; char str[512] = {0};
// while ((pRow = taos_fetch_row(pRes)) != NULL) { while ((pRow = taos_fetch_row(pRes)) != NULL) {
// int32_t* length = taos_fetch_lengths(pRes); int32_t* length = taos_fetch_lengths(pRes);
// for(int32_t i = 0; i < numOfFields; ++i) { for(int32_t i = 0; i < numOfFields; ++i) {
// printf("(%d):%d " , i, length[i]); printf("(%d):%d " , i, length[i]);
// } }
// printf("\n"); printf("\n");
//
// int32_t code = taos_print_row(str, pRow, pFields, numOfFields); int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
// printf("%s\n", str); printf("%s\n", str);
// memset(str, 0, sizeof(str)); memset(str, 0, sizeof(str));
// } }
taos_query_a(pConn, "alter table test.m1 comment 'abcde' ", queryCallback, pConn); taos_query_a(pConn, "alter table test.m1 comment 'abcde' ", queryCallback, pConn);
getchar(); getchar();
......
source/common/src/tdatablock.c
浏览文件 @ d5f6bb79
...@@ -682,9 +682,10 @@ size_t blockDataGetRowSize(SSDataBlock* pBlock) { ...@@ -682,9 +682,10 @@ size_t blockDataGetRowSize(SSDataBlock* pBlock) {
* @param pBlock * @param pBlock
* @return * @return
*/ */
size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock) { size_t blockDataGetSerialMetaSize(uint32_t numOfCols) {
// | total rows/total length | block group id | each column length | // | total rows/total length | block group id | column schema | each column length |
return sizeof(int32_t) + sizeof(uint64_t) + pBlock->info.numOfCols * sizeof(int32_t); return sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t)) +
numOfCols * sizeof(int32_t);
} }
double blockDataGetSerialRowSize(const SSDataBlock* pBlock) { double blockDataGetSerialRowSize(const SSDataBlock* pBlock) {
...@@ -1217,6 +1218,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) { ...@@ -1217,6 +1218,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) {
pBlock->info.numOfCols = numOfCols; pBlock->info.numOfCols = numOfCols;
pBlock->info.hasVarCol = pDataBlock->info.hasVarCol; pBlock->info.hasVarCol = pDataBlock->info.hasVarCol;
pBlock->info.rowSize = pDataBlock->info.rowSize; pBlock->info.rowSize = pDataBlock->info.rowSize;
pBlock->info.groupId = pDataBlock->info.groupId;
for (int32_t i = 0; i < numOfCols; ++i) { for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData colInfo = {0}; SColumnInfoData colInfo = {0};
...@@ -1246,7 +1248,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) { ...@@ -1246,7 +1248,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) {
} }
size_t blockDataGetCapacityInRow(const SSDataBlock* pBlock, size_t pageSize) { size_t blockDataGetCapacityInRow(const SSDataBlock* pBlock, size_t pageSize) {
int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock); int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock->info.numOfCols);
int32_t rowSize = pBlock->info.rowSize; int32_t rowSize = pBlock->info.rowSize;
...@@ -1885,34 +1887,44 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo ...@@ -1885,34 +1887,44 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo
void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols, void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols,
int8_t needCompress) { int8_t needCompress) {
// todo extract method
int32_t* actualLen = (int32_t*)data; int32_t* actualLen = (int32_t*)data;
data += sizeof(int32_t); data += sizeof(int32_t);
uint64_t* groupId = (uint64_t*)data; uint64_t* groupId = (uint64_t*)data;
data += sizeof(uint64_t); data += sizeof(uint64_t);
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
*((int16_t*)data) = pColInfoData->info.type;
data += sizeof(int16_t);
*((int32_t*)data) = pColInfoData->info.bytes;
data += sizeof(int32_t);
}
int32_t* colSizes = (int32_t*)data; int32_t* colSizes = (int32_t*)data;
data += numOfCols * sizeof(int32_t); data += numOfCols * sizeof(int32_t);
*dataLen = (numOfCols * sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t)); *dataLen = blockDataGetSerialMetaSize(numOfCols);
int32_t numOfRows = pBlock->info.rows; int32_t numOfRows = pBlock->info.rows;
for (int32_t col = 0; col < numOfCols; ++col) { for (int32_t col = 0; col < numOfCols; ++col) {
SColumnInfoData* pColRes = (SColumnInfoData*)taosArrayGet(pBlock->pDataBlock, col); SColumnInfoData* pColRes = (SColumnInfoData*)taosArrayGet(pBlock->pDataBlock, col);
// copy the null bitmap // copy the null bitmap
size_t metaSize = 0;
if (IS_VAR_DATA_TYPE(pColRes->info.type)) { if (IS_VAR_DATA_TYPE(pColRes->info.type)) {
size_t metaSize = numOfRows * sizeof(int32_t); metaSize = numOfRows * sizeof(int32_t);
memcpy(data, pColRes->varmeta.offset, metaSize); memcpy(data, pColRes->varmeta.offset, metaSize);
data += metaSize;
(*dataLen) += metaSize;
} else { } else {
int32_t len = BitmapLen(numOfRows); metaSize = BitmapLen(numOfRows);
memcpy(data, pColRes->nullbitmap, len); memcpy(data, pColRes->nullbitmap, metaSize);
data += len;
(*dataLen) += len;
} }
data += metaSize;
(*dataLen) += metaSize;
if (needCompress) { if (needCompress) {
colSizes[col] = blockCompressColData(pColRes, numOfRows, data, needCompress); colSizes[col] = blockCompressColData(pColRes, numOfRows, data, needCompress);
data += colSizes[col]; data += colSizes[col];
...@@ -1933,6 +1945,8 @@ void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen ...@@ -1933,6 +1945,8 @@ void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen
const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData) { const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData) {
blockDataEnsureCapacity(pBlock, numOfRows); blockDataEnsureCapacity(pBlock, numOfRows);
pBlock->info.rows = numOfRows;
const char* pStart = pData; const char* pStart = pData;
int32_t dataLen = *(int32_t*)pStart; int32_t dataLen = *(int32_t*)pStart;
...@@ -1941,6 +1955,29 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t ...@@ -1941,6 +1955,29 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t
pBlock->info.groupId = *(uint64_t*)pStart; pBlock->info.groupId = *(uint64_t*)pStart;
pStart += sizeof(uint64_t); pStart += sizeof(uint64_t);
if (pBlock->pDataBlock == NULL) {
pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData));
taosArraySetSize(pBlock->pDataBlock, numOfCols);
}
pBlock->info.numOfCols = taosArrayGetSize(pBlock->pDataBlock);
ASSERT(pBlock->pDataBlock->size >= numOfCols);
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
pColInfoData->info.type = *(int16_t*)pStart;
pStart += sizeof(int16_t);
pColInfoData->info.bytes = *(int32_t*)pStart;
pStart += sizeof(int32_t);
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
pBlock->info.hasVarCol = true;
}
}
blockDataEnsureCapacity(pBlock, numOfRows);
int32_t* colLen = (int32_t*)pStart; int32_t* colLen = (int32_t*)pStart;
pStart += sizeof(int32_t) * numOfCols; pStart += sizeof(int32_t) * numOfCols;
...@@ -1961,11 +1998,17 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t ...@@ -1961,11 +1998,17 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t
pColInfoData->pData = taosMemoryMalloc(colLen[i]); pColInfoData->pData = taosMemoryMalloc(colLen[i]);
} }
} else { } else {
if (pColInfoData->nullbitmap == NULL) {
pColInfoData->nullbitmap = taosMemoryCalloc(1, BitmapLen(numOfRows));
}
memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows)); memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows));
pStart += BitmapLen(numOfRows); pStart += BitmapLen(numOfRows);
} }
if (colLen[i] > 0) { if (colLen[i] > 0) {
if (pColInfoData->pData == NULL) {
pColInfoData->pData = taosMemoryCalloc(1, colLen[i]);
}
memcpy(pColInfoData->pData, pStart, colLen[i]); memcpy(pColInfoData->pData, pStart, colLen[i]);
} }
......
source/common/src/tdataformat.c
浏览文件 @ d5f6bb79
...@@ -208,7 +208,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -208,7 +208,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
if (iColumn == 0) { if (iColumn == 0) {
ASSERT(pColVal->cid == pTColumn->colId); ASSERT(pColVal->cid == pTColumn->colId);
ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP); ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP);
ASSERT(pTColumn->colId == 0); ASSERT(pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID);
iColVal++; iColVal++;
} else { } else {
...@@ -244,7 +244,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -244,7 +244,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
} }
} }
ASSERT(flags); // ASSERT(flags); // only 1 column(ts)
// decide // decide
uint32_t nData = 0; uint32_t nData = 0;
...@@ -268,7 +268,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -268,7 +268,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
nDataT = BIT2_SIZE(pTSchema->numOfCols - 1) + pTSchema->flen + ntv; nDataT = BIT2_SIZE(pTSchema->numOfCols - 1) + pTSchema->flen + ntv;
break; break;
default: default:
ASSERT(0); break; // only ts column
// ASSERT(0);
} }
uint8_t tflags = 0; uint8_t tflags = 0;
...@@ -283,7 +284,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -283,7 +284,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
tflags |= TSROW_KV_BIG; tflags |= TSROW_KV_BIG;
} }
if (nDataT < nDataK) { if (nDataT <= nDataK) {
nData = nDataT; nData = nDataT;
} else { } else {
nData = nDataK; nData = nDataK;
...@@ -352,7 +353,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -352,7 +353,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
ntv = 0; ntv = 0;
iColVal = 1; iColVal = 1;
if (flags & 0xf0 == 0) { if ((flags & 0xf0) == 0) {
switch (flags & 0xf) { switch (flags & 0xf) {
case TSROW_HAS_VAL: case TSROW_HAS_VAL:
pf = (*ppRow)->pData; pf = (*ppRow)->pData;
...@@ -373,7 +374,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -373,7 +374,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
ptv = pf + pTSchema->flen; ptv = pf + pTSchema->flen;
break; break;
default: default:
ASSERT(0); // ASSERT(0);
break;
} }
} else { } else {
pTSKVRow = (STSKVRow *)(*ppRow)->pData; pTSKVRow = (STSKVRow *)(*ppRow)->pData;
...@@ -417,13 +419,13 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -417,13 +419,13 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
} }
_set_none: _set_none:
if (flags & 0xf0 == 0) { if ((flags & 0xf0) == 0) {
setBitMap(pb, 0, iColumn - 1, flags); setBitMap(pb, 0, iColumn - 1, flags);
} }
continue; continue;
_set_null: _set_null:
if (flags & 0xf0 == 0) { if ((flags & 0xf0) == 0) {
setBitMap(pb, 1, iColumn - 1, flags); setBitMap(pb, 1, iColumn - 1, flags);
} else { } else {
SET_IDX(pidx, pTSKVRow->nCols, nkv, flags); SET_IDX(pidx, pTSKVRow->nCols, nkv, flags);
...@@ -433,7 +435,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S ...@@ -433,7 +435,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
continue; continue;
_set_value: _set_value:
if (flags & 0xf0 == 0) { if ((flags & 0xf0) == 0) {
setBitMap(pb, 2, iColumn - 1, flags); setBitMap(pb, 2, iColumn - 1, flags);
if (IS_VAR_DATA_TYPE(pTColumn->type)) { if (IS_VAR_DATA_TYPE(pTColumn->type)) {
...@@ -489,13 +491,13 @@ void tTSRowFree(STSRow2 *pRow) { ...@@ -489,13 +491,13 @@ void tTSRowFree(STSRow2 *pRow) {
} }
void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) { void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) {
uint8_t isTuple = (pRow->flags & 0xf0 == 0) ? 1 : 0; uint8_t isTuple = ((pRow->flags & 0xf0) == 0) ? 1 : 0;
STColumn *pTColumn = &pTSchema->columns[iCol]; STColumn *pTColumn = &pTSchema->columns[iCol];
uint8_t flags = pRow->flags & (uint8_t)0xf; uint8_t flags = pRow->flags & (uint8_t)0xf;
SValue value; SValue value;
ASSERT(iCol < pTSchema->numOfCols); ASSERT(iCol < pTSchema->numOfCols);
ASSERT(flags); // ASSERT(flags); // only 1 ts column
ASSERT(pRow->sver == pTSchema->version); ASSERT(pRow->sver == pTSchema->version);
if (iCol == 0) { if (iCol == 0) {
...@@ -505,7 +507,7 @@ void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal ...@@ -505,7 +507,7 @@ void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal
if (flags == TSROW_HAS_NONE) { if (flags == TSROW_HAS_NONE) {
goto _return_none; goto _return_none;
} else if (flags == TSROW_HAS_NONE) { } else if (flags == TSROW_HAS_NULL) {
goto _return_null; goto _return_null;
} }
......
source/common/src/tmsg.c
浏览文件 @ d5f6bb79
...@@ -2427,6 +2427,7 @@ int32_t tSerializeSTableIndexInfo(SEncoder *pEncoder, STableIndexInfo* pInfo) { ...@@ -2427,6 +2427,7 @@ int32_t tSerializeSTableIndexInfo(SEncoder *pEncoder, STableIndexInfo* pInfo) {
if (tEncodeI64(pEncoder, pInfo->sliding) < 0) return -1; if (tEncodeI64(pEncoder, pInfo->sliding) < 0) return -1;
if (tEncodeI64(pEncoder, pInfo->dstTbUid) < 0) return -1; if (tEncodeI64(pEncoder, pInfo->dstTbUid) < 0) return -1;
if (tEncodeI32(pEncoder, pInfo->dstVgId) < 0) return -1; if (tEncodeI32(pEncoder, pInfo->dstVgId) < 0) return -1;
if (tEncodeSEpSet(pEncoder, &pInfo->epSet) < 0) return -1;
if (tEncodeCStr(pEncoder, pInfo->expr) < 0) return -1; if (tEncodeCStr(pEncoder, pInfo->expr) < 0) return -1;
return 0; return 0;
} }
...@@ -2459,6 +2460,7 @@ int32_t tDeserializeSTableIndexInfo(SDecoder *pDecoder, STableIndexInfo *pInfo) ...@@ -2459,6 +2460,7 @@ int32_t tDeserializeSTableIndexInfo(SDecoder *pDecoder, STableIndexInfo *pInfo)
if (tDecodeI64(pDecoder, &pInfo->sliding) < 0) return -1; if (tDecodeI64(pDecoder, &pInfo->sliding) < 0) return -1;
if (tDecodeI64(pDecoder, &pInfo->dstTbUid) < 0) return -1; if (tDecodeI64(pDecoder, &pInfo->dstTbUid) < 0) return -1;
if (tDecodeI32(pDecoder, &pInfo->dstVgId) < 0) return -1; if (tDecodeI32(pDecoder, &pInfo->dstVgId) < 0) return -1;
if (tDecodeSEpSet(pDecoder, &pInfo->epSet) < 0) return -1;
if (tDecodeCStrAlloc(pDecoder, &pInfo->expr) < 0) return -1; if (tDecodeCStrAlloc(pDecoder, &pInfo->expr) < 0) return -1;
return 0; return 0;
......
source/common/test/CMakeLists.txt
浏览文件 @ d5f6bb79
...@@ -5,7 +5,12 @@ MESSAGE(STATUS "build parser unit test") ...@@ -5,7 +5,12 @@ MESSAGE(STATUS "build parser unit test")
SET(CMAKE_CXX_STANDARD 11) SET(CMAKE_CXX_STANDARD 11)
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST) AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
ADD_EXECUTABLE(commonTest ${SOURCE_LIST}) ADD_EXECUTABLE(commonTest "")
TARGET_SOURCES(
commonTest
PRIVATE
"commonTests.cpp"
)
TARGET_LINK_LIBRARIES( TARGET_LINK_LIBRARIES(
commonTest commonTest
PUBLIC os util common gtest PUBLIC os util common gtest
...@@ -24,7 +29,12 @@ target_sources( ...@@ -24,7 +29,12 @@ target_sources(
PRIVATE PRIVATE
"dataformatTest.cpp" "dataformatTest.cpp"
) )
target_link_libraries(dataformatTest gtest gtest_main util) target_link_libraries(dataformatTest gtest gtest_main util common)
target_include_directories(
dataformatTest
PUBLIC "${TD_SOURCE_DIR}/include/common"
PUBLIC "${TD_SOURCE_DIR}/include/util"
)
# tmsg test # tmsg test
# add_executable(tmsgTest "") # add_executable(tmsgTest "")
......
source/common/test/dataformatTest.cpp
浏览文件 @ d5f6bb79
#include "gtest/gtest.h" /*
\ No newline at end of file * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <gtest/gtest.h>
#include <taoserror.h>
#include <tdataformat.h>
#include <tglobal.h>
#include <tmsg.h>
#include <iostream>
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wwrite-strings"
#pragma GCC diagnostic ignored "-Wunused-function"
#pragma GCC diagnostic ignored "-Wunused-variable"
#pragma GCC diagnostic ignored "-Wsign-compare"
#define NONE_CSTR "no"
#define NULL_CSTR "nu"
#define NONE_LEN 2
#define NULL_LEN 2
const static int16_t MAX_COLS = 14;
int main(int argc, char **argv) {
testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool
STSchema *genSTSchema(int16_t nCols) {
EXPECT_LE(nCols, MAX_COLS);
SSchema *pSchema = (SSchema *)taosMemoryCalloc(nCols, sizeof(SSchema));
EXPECT_NE(pSchema, nullptr);
for (int16_t i = 0; i < nCols; ++i) {
pSchema[i].colId = PRIMARYKEY_TIMESTAMP_COL_ID + i;
char colName[TSDB_COL_NAME_LEN] = {0};
snprintf(colName, TSDB_COL_NAME_LEN, "c%" PRIi16, i);
strncpy(pSchema[i].name, colName, TSDB_COL_NAME_LEN);
switch (i) {
case 0: {
pSchema[0].type = TSDB_DATA_TYPE_TIMESTAMP;
pSchema[0].bytes = TYPE_BYTES[pSchema[0].type];
} break;
case 1: {
pSchema[1].type = TSDB_DATA_TYPE_INT;
pSchema[1].bytes = TYPE_BYTES[pSchema[1].type];
;
} break;
case 2: {
pSchema[2].type = TSDB_DATA_TYPE_BIGINT;
pSchema[2].bytes = TYPE_BYTES[pSchema[2].type];
} break;
case 3: {
pSchema[3].type = TSDB_DATA_TYPE_FLOAT;
pSchema[3].bytes = TYPE_BYTES[pSchema[3].type];
} break;
case 4: {
pSchema[4].type = TSDB_DATA_TYPE_DOUBLE;
pSchema[4].bytes = TYPE_BYTES[pSchema[4].type];
} break;
case 5: {
pSchema[5].type = TSDB_DATA_TYPE_BINARY;
pSchema[5].bytes = 12;
} break;
case 6: {
pSchema[6].type = TSDB_DATA_TYPE_NCHAR;
pSchema[6].bytes = 42;
} break;
case 7: {
pSchema[7].type = TSDB_DATA_TYPE_TINYINT;
pSchema[7].bytes = TYPE_BYTES[pSchema[7].type];
} break;
case 8: {
pSchema[8].type = TSDB_DATA_TYPE_SMALLINT;
pSchema[8].bytes = TYPE_BYTES[pSchema[8].type];
} break;
case 9: {
pSchema[9].type = TSDB_DATA_TYPE_BOOL;
pSchema[9].bytes = TYPE_BYTES[pSchema[9].type];
} break;
case 10: {
pSchema[10].type = TSDB_DATA_TYPE_UTINYINT;
pSchema[10].bytes = TYPE_BYTES[pSchema[10].type];
} break;
case 11: {
pSchema[11].type = TSDB_DATA_TYPE_USMALLINT;
pSchema[11].bytes = TYPE_BYTES[pSchema[11].type];
} break;
case 12: {
pSchema[12].type = TSDB_DATA_TYPE_UINT;
pSchema[12].bytes = TYPE_BYTES[pSchema[12].type];
} break;
case 13: {
pSchema[13].type = TSDB_DATA_TYPE_UBIGINT;
pSchema[13].bytes = TYPE_BYTES[pSchema[13].type];
} break;
default:
ASSERT(0);
break;
}
}
STSchema *pResult = NULL;
pResult = tdGetSTSChemaFromSSChema(&pSchema, nCols);
taosMemoryFree(pSchema);
return pResult;
}
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool
static int32_t genTestData(const char **data, int16_t nCols, SArray **pArray) {
if (!(*pArray)) {
*pArray = taosArrayInit(nCols, sizeof(SColVal));
if (!(*pArray)) return -1;
}
for (int16_t i = 0; i < nCols; ++i) {
SColVal colVal = {0};
colVal.cid = PRIMARYKEY_TIMESTAMP_COL_ID + i;
if (strncasecmp(data[i], NONE_CSTR, NONE_LEN) == 0) {
colVal.isNone = 1;
taosArrayPush(*pArray, &colVal);
continue;
} else if (strncasecmp(data[i], NULL_CSTR, NULL_LEN) == 0) {
colVal.isNull = 1;
taosArrayPush(*pArray, &colVal);
continue;
}
switch (i) {
case 0:
sscanf(data[i], "%" PRIi64, &colVal.value.ts);
break;
case 1: {
sscanf(data[i], "%" PRIi32, &colVal.value.i32);
} break;
case 2:
sscanf(data[i], "%" PRIi64, &colVal.value.i64);
break;
case 3:
sscanf(data[i], "%f", &colVal.value.f);
break;
case 4:
sscanf(data[i], "%lf", &colVal.value.d);
break;
case 5: {
int16_t dataLen = strlen(data[i]) + 1;
colVal.value.nData = dataLen < 10 ? dataLen : 10;
colVal.value.pData = (uint8_t *)data[i];
} break;
case 6: {
int16_t dataLen = strlen(data[i]) + 1;
colVal.value.nData = dataLen < 40 ? dataLen : 40;
colVal.value.pData = (uint8_t *)data[i]; // just for test, not real nchar
} break;
case 7:
case 9: {
int32_t d8;
sscanf(data[i], "%" PRId32, &d8);
colVal.value.i8 = (int8_t)d8;
} break;
case 8: {
int32_t d16;
sscanf(data[i], "%" PRId32, &d16);
colVal.value.i16 = (int16_t)d16;
} break;
case 10: {
uint32_t u8;
sscanf(data[i], "%" PRId32, &u8);
colVal.value.u8 = (uint8_t)u8;
} break;
case 11: {
uint32_t u16;
sscanf(data[i], "%" PRId32, &u16);
colVal.value.u16 = (uint16_t)u16;
} break;
case 12: {
sscanf(data[i], "%" PRIu32, &colVal.value.u32);
} break;
case 13: {
sscanf(data[i], "%" PRIu64, &colVal.value.u64);
} break;
default:
ASSERT(0);
}
taosArrayPush(*pArray, &colVal);
}
return 0;
}
int32_t debugPrintSColVal(SColVal *cv, int8_t type) {
if (cv->isNone) {
printf("None ");
return 0;
}
if (cv->isNull) {
printf("Null ");
return 0;
}
switch (type) {
case TSDB_DATA_TYPE_BOOL:
printf("%s ", cv->value.i8 == 0 ? "false" : "true");
break;
case TSDB_DATA_TYPE_TINYINT:
printf("%" PRIi8 " ", cv->value.i8);
break;
case TSDB_DATA_TYPE_SMALLINT:
printf("%" PRIi16 " ", cv->value.i16);
break;
case TSDB_DATA_TYPE_INT:
printf("%" PRIi32 " ", cv->value.i32);
break;
case TSDB_DATA_TYPE_BIGINT:
printf("%" PRIi64 " ", cv->value.i64);
break;
case TSDB_DATA_TYPE_FLOAT:
printf("%f ", cv->value.f);
break;
case TSDB_DATA_TYPE_DOUBLE:
printf("%lf ", cv->value.d);
break;
case TSDB_DATA_TYPE_VARCHAR: {
char tv[15] = {0};
snprintf(tv, 15, "%s", cv->value.pData);
printf("%s ", tv);
} break;
case TSDB_DATA_TYPE_TIMESTAMP:
printf("%" PRIi64 " ", cv->value.i64);
break;
case TSDB_DATA_TYPE_NCHAR: {
char tv[15] = {0};
snprintf(tv, 15, "%s", cv->value.pData);
printf("%s ", tv);
} break;
case TSDB_DATA_TYPE_UTINYINT:
printf("%" PRIu8 " ", cv->value.u8);
break;
case TSDB_DATA_TYPE_USMALLINT:
printf("%" PRIu16 " ", cv->value.u16);
break;
case TSDB_DATA_TYPE_UINT:
printf("%" PRIu32 " ", cv->value.u32);
break;
case TSDB_DATA_TYPE_UBIGINT:
printf("%" PRIu64 " ", cv->value.u64);
break;
case TSDB_DATA_TYPE_JSON:
printf("JSON ");
break;
case TSDB_DATA_TYPE_VARBINARY:
printf("VARBIN ");
break;
case TSDB_DATA_TYPE_DECIMAL:
printf("DECIMAL ");
break;
case TSDB_DATA_TYPE_BLOB:
printf("BLOB ");
break;
case TSDB_DATA_TYPE_MEDIUMBLOB:
printf("MedBLOB ");
break;
// case TSDB_DATA_TYPE_BINARY:
// printf("BINARY ");
// break;
case TSDB_DATA_TYPE_MAX:
printf("UNDEF ");
break;
default:
printf("UNDEF ");
break;
}
return 0;
}
void debugPrintTSRow(STSRow2 *row, STSchema *pTSchema, const char *tags, int32_t ln) {
printf("%s:%d %s:v%d:%d ", tags, ln, (row->flags & 0xf0) ? "KV" : "TP", row->sver, row->nData);
for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) {
SColVal cv = {0};
tTSRowGet(row, pTSchema, i, &cv);
debugPrintSColVal(&cv, pTSchema->columns[i].type);
}
printf("\n");
fflush(stdout);
}
static int32_t checkSColVal(const char *rawVal, SColVal *cv, int8_t type) {
ASSERT(rawVal);
if (cv->isNone) {
EXPECT_STRCASEEQ(rawVal, NONE_CSTR);
return 0;
}
if (cv->isNull) {
EXPECT_STRCASEEQ(rawVal, NULL_CSTR);
return 0;
}
SValue rawSVal = {0};
switch (type) {
case TSDB_DATA_TYPE_BOOL:
case TSDB_DATA_TYPE_TINYINT: {
int32_t d8;
sscanf(rawVal, "%" PRId32, &d8);
EXPECT_EQ(cv->value.i8, (int8_t)d8);
} break;
case TSDB_DATA_TYPE_SMALLINT: {
int32_t d16;
sscanf(rawVal, "%" PRId32, &d16);
EXPECT_EQ(cv->value.i16, (int16_t)d16);
} break;
case TSDB_DATA_TYPE_INT: {
sscanf(rawVal, "%" PRId32, &rawSVal.i32);
EXPECT_EQ(cv->value.i32, rawSVal.i32);
} break;
case TSDB_DATA_TYPE_BIGINT: {
sscanf(rawVal, "%" PRIi64, &rawSVal.i64);
EXPECT_EQ(cv->value.i64, rawSVal.i64);
} break;
case TSDB_DATA_TYPE_FLOAT: {
sscanf(rawVal, "%f", &rawSVal.f);
EXPECT_FLOAT_EQ(cv->value.f, rawSVal.f);
} break;
case TSDB_DATA_TYPE_DOUBLE: {
sscanf(rawVal, "%lf", &rawSVal.d);
EXPECT_DOUBLE_EQ(cv->value.d, rawSVal.d);
} break;
case TSDB_DATA_TYPE_VARCHAR: {
EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData);
} break;
case TSDB_DATA_TYPE_TIMESTAMP: {
sscanf(rawVal, "%" PRIi64, &rawSVal.ts);
EXPECT_DOUBLE_EQ(cv->value.ts, rawSVal.ts);
} break;
case TSDB_DATA_TYPE_NCHAR: {
EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData); // informal nchar comparsion
} break;
case TSDB_DATA_TYPE_UTINYINT: {
uint32_t u8;
sscanf(rawVal, "%" PRIu32, &u8);
EXPECT_EQ(cv->value.u8, (uint8_t)u8);
} break;
case TSDB_DATA_TYPE_USMALLINT: {
uint32_t u16;
sscanf(rawVal, "%" PRIu32, &u16);
EXPECT_EQ(cv->value.u16, (uint16_t)u16);
} break;
case TSDB_DATA_TYPE_UINT: {
sscanf(rawVal, "%" PRIu32, &rawSVal.u32);
EXPECT_EQ(cv->value.u32, rawSVal.u32);
} break;
case TSDB_DATA_TYPE_UBIGINT: {
sscanf(rawVal, "%" PRIu64, &rawSVal.u64);
EXPECT_EQ(cv->value.u64, rawSVal.u64);
} break;
case TSDB_DATA_TYPE_JSON:
printf("JSON ");
break;
case TSDB_DATA_TYPE_VARBINARY:
printf("VARBIN ");
break;
case TSDB_DATA_TYPE_DECIMAL:
printf("DECIMAL ");
break;
case TSDB_DATA_TYPE_BLOB:
printf("BLOB ");
break;
case TSDB_DATA_TYPE_MEDIUMBLOB:
printf("MedBLOB ");
break;
// case TSDB_DATA_TYPE_BINARY:
// printf("BINARY ");
// break;
case TSDB_DATA_TYPE_MAX:
printf("UNDEF ");
break;
default:
printf("UNDEF ");
break;
}
return 0;
}
static void checkTSRow(const char **data, STSRow2 *row, STSchema *pTSchema) {
for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) {
SColVal cv = {0};
tTSRowGet(row, pTSchema, i, &cv);
checkSColVal(data[i], &cv, pTSchema->columns[i].type);
}
}
TEST(testCase, AllNormTest) {
int16_t nCols = 1;
STSRow2 *row = nullptr;
SArray *pArray = taosArrayInit(nCols, sizeof(SColVal));
EXPECT_NE(pArray, nullptr);
STSchema *pTSchema = genSTSchema(nCols);
EXPECT_NE(pTSchema, nullptr);
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint,
// c9 bool
char *data[10] = {"1653694220000", "10", "20", "10.1", "10.1", "binary10", "nchar10", "10", "10", "1"};
genTestData((const char **)&data, nCols, &pArray);
tTSRowNew(NULL, pArray, pTSchema, &row);
debugPrintTSRow(row, pTSchema, __func__, __LINE__);
checkTSRow((const char **)&data, row, pTSchema);
taosArrayDestroy(pArray);
taosMemoryFree(pTSchema);
}
#if 1
TEST(testCase, NoneTest) {
const static int nCols = 14;
const static int nRows = 20;
STSRow2 *row = nullptr;
SArray *pArray = taosArrayInit(nCols, sizeof(SColVal));
EXPECT_NE(pArray, nullptr);
STSchema *pTSchema = genSTSchema(nCols);
EXPECT_NE(pTSchema, nullptr);
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint,
// c9 bool c10 tinyint unsigned, c11 smallint unsigned, c12 int unsigned, c13 bigint unsigned
const char *data[nRows][nCols] = {
{"1653694220000", "no", "20", "10.1", "10.1", "binary10", "no", "10", "10", "nu", "10", "20", "30", "40"},
{"1653694220001", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no"},
{"1653694220002", "no", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "nu"},
{"1653694220003", "nu", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "no"},
{"1653694220004", "no", "20", "no", "no", "no", "nchar10", "no", "no", "no", "no", "no", "no", "no"},
{"1653694220005", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"},
{"1653694220006", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"},
{"1653694220007", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"},
{"1653694220008", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"},
{"1653694220009", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"},
{"1653694220010", "-1", "-1", "-1", "-1", "binary10", "nu", "-1", "0", "0", "0", "0", "0", "0"},
{"1653694220011", "-2147483648", "nu", "nu", "nu", "biy10", "nu", "nu", "32767", "no", "nu", "nu", "nu", "100"},
{"1653694220012", "2147483647", "nu", "nu", "nu", "ary10", "nu", "nu", "-32768", "no", "nu", "nu", "nu", "100"},
{"1653694220013", "no", "-9223372036854775818", "nu", "nu", "b1", "nu", "nu", "10", "no", "nu", "nu", "nu", "nu"},
{"1653694220014", "no", "nu", "nu", "nu", "b0", "nu", "nu", "10", "no", "nu", "nu", "nu", "9223372036854775808"},
{"1653694220015", "no", "nu", "nu", "nu", "binary30", "char4", "nu", "10", "no", "nu", "nu", "nu",
"18446744073709551615"},
{"1653694220016", "2147483647", "nu", "nu", "nu", "bin50", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"},
{"1653694220017", "2147483646", "0", "0", "0", "binary10", "0", "0", "0", "0", "255", "0", "0", "0"},
{"1653694220018", "no", "-9223372036854775808", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu",
"4294967295", "100"},
{"1653694220019", "no", "9223372036854775807", "nu", "nu", "bin10", "nu", "nu", "10", "no", "254", "nu", "nu",
"no"}};
for (int r = 0; r < nRows; ++r) {
genTestData((const char **)&data[r], nCols, &pArray);
tTSRowNew(NULL, pArray, pTSchema, &row);
debugPrintTSRow(row, pTSchema, __func__, __LINE__); // debug print
checkTSRow((const char **)&data[r], row, pTSchema); // check
taosMemoryFreeClear(row);
taosArrayClear(pArray);
}
taosArrayDestroy(pArray);
taosMemoryFree(pTSchema);
}
#endif
\ No newline at end of file
source/dnode/mgmt/mgmt_vnode/inc/vmInt.h
浏览文件 @ d5f6bb79
...@@ -34,6 +34,7 @@ typedef struct SVnodeMgmt { ...@@ -34,6 +34,7 @@ typedef struct SVnodeMgmt {
SQWorkerPool fetchPool; SQWorkerPool fetchPool;
SWWorkerPool syncPool; SWWorkerPool syncPool;
SWWorkerPool writePool; SWWorkerPool writePool;
SWWorkerPool applyPool;
SWWorkerPool mergePool; SWWorkerPool mergePool;
SSingleWorker mgmtWorker; SSingleWorker mgmtWorker;
SSingleWorker monitorWorker; SSingleWorker monitorWorker;
...@@ -52,10 +53,9 @@ typedef struct { ...@@ -52,10 +53,9 @@ typedef struct {
typedef struct { typedef struct {
int32_t vgId; int32_t vgId;
int32_t refCount;
int32_t vgVersion; int32_t vgVersion;
int32_t refCount;
int8_t dropped; int8_t dropped;
int8_t accessState;
char *path; char *path;
SVnode *pImpl; SVnode *pImpl;
STaosQueue *pWriteQ; STaosQueue *pWriteQ;
......
source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
浏览文件 @ d5f6bb79
...@@ -323,7 +323,6 @@ SArray *vmGetMsgHandles() { ...@@ -323,7 +323,6 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_MON_VM_INFO, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_MON_VM_INFO, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MON_VM_LOAD, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_MON_VM_LOAD, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER;
// Requests handled by VNODE
if (dmSetMgmtHandle(pArray, TDMT_VND_SUBMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_SUBMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_CONTINUE, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_CONTINUE, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
...@@ -353,7 +352,9 @@ SArray *vmGetMsgHandles() { ...@@ -353,7 +352,9 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
......
source/dnode/mgmt/mgmt_vnode/src/vmInt.c
浏览文件 @ d5f6bb79
...@@ -49,10 +49,9 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) { ...@@ -49,10 +49,9 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) {
} }
pVnode->vgId = pCfg->vgId; pVnode->vgId = pCfg->vgId;
pVnode->refCount = 0;
pVnode->vgVersion = pCfg->vgVersion; pVnode->vgVersion = pCfg->vgVersion;
pVnode->refCount = 0;
pVnode->dropped = 0; pVnode->dropped = 0;
pVnode->accessState = TSDB_VN_ALL_ACCCESS;
pVnode->path = tstrdup(pCfg->path); pVnode->path = tstrdup(pCfg->path);
pVnode->pImpl = pImpl; pVnode->pImpl = pImpl;
...@@ -96,7 +95,7 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { ...@@ -96,7 +95,7 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
dDebug("vgId:%d, vnode is closed", pVnode->vgId); dDebug("vgId:%d, vnode is closed", pVnode->vgId);
if (pVnode->dropped) { if (pVnode->dropped) {
dDebug("vgId:%d, vnode is destroyed for dropped:%d", pVnode->vgId, pVnode->dropped); dInfo("vgId:%d, vnode is destroyed, dropped:%d", pVnode->vgId, pVnode->dropped);
snprintf(path, TSDB_FILENAME_LEN, "vnode%svnode%d", TD_DIRSEP, pVnode->vgId); snprintf(path, TSDB_FILENAME_LEN, "vnode%svnode%d", TD_DIRSEP, pVnode->vgId);
vnodeDestroy(path, pMgmt->pTfs); vnodeDestroy(path, pMgmt->pTfs);
} }
......
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
浏览文件 @ d5f6bb79
...@@ -16,10 +16,8 @@ ...@@ -16,10 +16,8 @@
#define _DEFAULT_SOURCE #define _DEFAULT_SOURCE
#include "vmInt.h" #include "vmInt.h"
#include "sync.h"
#include "syncTools.h"
static inline void vmSendRsp(SRpcMsg *pMsg, int32_t code) { static inline void vmSendRsp(SRpcMsg *pMsg, int32_t code) {
if (pMsg->info.handle == NULL) return;
SRpcMsg rsp = { SRpcMsg rsp = {
.code = code, .code = code,
.pCont = pMsg->info.rsp, .pCont = pMsg->info.rsp,
...@@ -53,9 +51,9 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { ...@@ -53,9 +51,9 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
} }
if (IsReq(pMsg)) { if (IsReq(pMsg)) {
if (code != 0 && terrno != 0) { if (code != 0) {
dError("msg:%p failed to process since %s", pMsg, terrstr()); if (terrno != 0) code = terrno;
code = terrno; dError("msg:%p, failed to process since %s", pMsg, terrstr());
} }
vmSendRsp(pMsg, code); vmSendRsp(pMsg, code);
} }
...@@ -97,110 +95,6 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { ...@@ -97,110 +95,6 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
taosFreeQitem(pMsg); taosFreeQitem(pMsg);
} }
static void vmProcessWriteQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
int32_t code = 0;
SRpcMsg *pMsg = NULL;
SVnodeObj *pVnode = pInfo->ahandle;
int64_t sync = vnodeGetSyncHandle(pVnode->pImpl);
SArray *pArray = taosArrayInit(numOfMsgs, sizeof(SRpcMsg **));
for (int32_t m = 0; m < numOfMsgs; m++) {
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
dTrace("vgId:%d, msg:%p get from vnode-write queue", pVnode->vgId, pMsg);
if (taosArrayPush(pArray, &pMsg) == NULL) {
dError("vgId:%d, failed to push msg:%p to vnode-write array", pVnode->vgId, pMsg);
vmSendRsp(pMsg, TSDB_CODE_OUT_OF_MEMORY);
}
}
for (int32_t m = 0; m < taosArrayGetSize(pArray); m++) {
pMsg = *(SRpcMsg **)taosArrayGet(pArray, m);
code = vnodePreprocessReq(pVnode->pImpl, pMsg);
if (code == TSDB_CODE_ACTION_IN_PROGRESS) {
dTrace("vgId:%d, msg:%p in progress and no rsp", pVnode->vgId, pMsg);
continue;
}
if (pMsg->msgType != TDMT_VND_ALTER_REPLICA) {
code = syncPropose(sync, pMsg, false);
}
if (code == TAOS_SYNC_PROPOSE_SUCCESS) {
dTrace("vgId:%d, msg:%p is proposed and no rsp", pVnode->vgId, pMsg);
continue;
} else if (code == TAOS_SYNC_PROPOSE_NOT_LEADER) {
SEpSet newEpSet = {0};
syncGetEpSet(sync, &newEpSet);
SEp *pEp = &newEpSet.eps[newEpSet.inUse];
if (pEp->port == tsServerPort && strcmp(pEp->fqdn, tsLocalFqdn) == 0) {
newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps;
}
dTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", pVnode->vgId, pMsg,
newEpSet.numOfEps, newEpSet.inUse);
for (int32_t i = 0; i < newEpSet.numOfEps; ++i) {
dTrace("vgId:%d, msg:%p ep:%s:%u", pVnode->vgId, pMsg, newEpSet.eps[i].fqdn, newEpSet.eps[i].port);
}
SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info};
tmsgSendRedirectRsp(&rsp, &newEpSet);
} else {
dError("vgId:%d, msg:%p failed to propose write since %s, code:0x%x", pVnode->vgId, pMsg, tstrerror(code), code);
vmSendRsp(pMsg, code);
}
}
for (int32_t i = 0; i < numOfMsgs; i++) {
pMsg = *(SRpcMsg **)taosArrayGet(pArray, i);
dTrace("vgId:%d, msg:%p is freed", pVnode->vgId, pMsg);
rpcFreeCont(pMsg->pCont);
taosFreeQitem(pMsg);
}
taosArrayDestroy(pArray);
}
static void vmProcessApplyQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
SVnodeObj *pVnode = pInfo->ahandle;
SRpcMsg *pMsg = NULL;
for (int32_t i = 0; i < numOfMsgs; ++i) {
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
dTrace("vgId:%d, msg:%p get from vnode-apply queue", pVnode->vgId, pMsg);
// init response rpc msg
SRpcMsg rsp = {0};
// get original rpc msg
assert(pMsg->msgType == TDMT_SYNC_APPLY_MSG);
SyncApplyMsg *pSyncApplyMsg = syncApplyMsgFromRpcMsg2(pMsg);
syncApplyMsgLog2("==vmProcessApplyQueue==", pSyncApplyMsg);
SRpcMsg originalRpcMsg;
syncApplyMsg2OriginalRpcMsg(pSyncApplyMsg, &originalRpcMsg);
// apply data into tsdb
if (vnodeProcessWriteReq(pVnode->pImpl, &originalRpcMsg, pSyncApplyMsg->fsmMeta.index, &rsp) < 0) {
rsp.code = terrno;
dError("vgId:%d, msg:%p failed to apply since %s", pVnode->vgId, pMsg, terrstr());
}
syncApplyMsgDestroy(pSyncApplyMsg);
rpcFreeCont(originalRpcMsg.pCont);
// if leader, send response
if (pMsg->info.handle != NULL) {
rsp.info = pMsg->info;
tmsgSendRsp(&rsp);
}
dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, rsp.code);
rpcFreeCont(pMsg->pCont);
taosFreeQitem(pMsg);
}
}
static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
SVnodeObj *pVnode = pInfo->ahandle; SVnodeObj *pVnode = pInfo->ahandle;
SRpcMsg *pMsg = NULL; SRpcMsg *pMsg = NULL;
...@@ -211,11 +105,9 @@ static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOf ...@@ -211,11 +105,9 @@ static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOf
int32_t code = vnodeProcessSyncReq(pVnode->pImpl, pMsg, NULL); int32_t code = vnodeProcessSyncReq(pVnode->pImpl, pMsg, NULL);
if (code != 0) { if (code != 0) {
if (terrno != 0) code = terrno;
dError("vgId:%d, msg:%p failed to sync since %s", pVnode->vgId, pMsg, terrstr()); dError("vgId:%d, msg:%p failed to sync since %s", pVnode->vgId, pMsg, terrstr());
if (pMsg->info.handle != NULL) { vmSendRsp(pMsg, code);
if (terrno != 0) code = terrno;
vmSendRsp(pMsg, code);
}
} }
dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, code); dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, code);
...@@ -234,8 +126,8 @@ static void vmProcessMergeQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numO ...@@ -234,8 +126,8 @@ static void vmProcessMergeQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numO
int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo); int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo);
if (code != 0) { if (code != 0) {
dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr());
if (terrno != 0) code = terrno; if (terrno != 0) code = terrno;
dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr());
vmSendRsp(pMsg, code); vmSendRsp(pMsg, code);
} }
...@@ -254,7 +146,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp ...@@ -254,7 +146,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId); SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
if (pVnode == NULL) { if (pVnode == NULL) {
dError("vgId:%d, failed to put msg:%p into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(), dError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(),
TMSG_INFO(pMsg->msgType)); TMSG_INFO(pMsg->msgType));
return terrno != 0 ? terrno : -1; return terrno != 0 ? terrno : -1;
} }
...@@ -322,7 +214,7 @@ int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { ...@@ -322,7 +214,7 @@ int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
if (pMsg == NULL) return -1; if (pMsg == NULL) return -1;
SMsgHead *pHead = pRpc->pCont; SMsgHead *pHead = pRpc->pCont;
dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pMsg->msgType)); dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pRpc->msgType));
pHead->contLen = htonl(pHead->contLen); pHead->contLen = htonl(pHead->contLen);
pHead->vgId = htonl(pHead->vgId); pHead->vgId = htonl(pHead->vgId);
...@@ -362,9 +254,9 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) { ...@@ -362,9 +254,9 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
} }
int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessWriteQueue); pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode->pImpl, (FItems)vnodeProposeMsg);
pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue); pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue);
pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessApplyQueue); pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyMsg);
pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue); pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
pVnode->pFetchQ = tQWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItem)vmProcessFetchQueue); pVnode->pFetchQ = tQWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItem)vmProcessFetchQueue);
pVnode->pMergeQ = tWWorkerAllocQueue(&pMgmt->mergePool, pVnode, (FItems)vmProcessMergeQueue); pVnode->pMergeQ = tWWorkerAllocQueue(&pMgmt->mergePool, pVnode, (FItems)vmProcessMergeQueue);
...@@ -381,8 +273,8 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { ...@@ -381,8 +273,8 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ); tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ);
tWWorkerFreeQueue(&pMgmt->applyPool, pVnode->pApplyQ);
tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ); tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ);
tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pApplyQ);
tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ); tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
tQWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ); tQWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ);
tWWorkerFreeQueue(&pMgmt->mergePool, pVnode->pMergeQ); tWWorkerFreeQueue(&pMgmt->mergePool, pVnode->pMergeQ);
...@@ -413,6 +305,11 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) { ...@@ -413,6 +305,11 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
pWPool->max = tsNumOfVnodeWriteThreads; pWPool->max = tsNumOfVnodeWriteThreads;
if (tWWorkerInit(pWPool) != 0) return -1; if (tWWorkerInit(pWPool) != 0) return -1;
SWWorkerPool *pAPool = &pMgmt->applyPool;
pAPool->name = "vnode-apply";
pAPool->max = tsNumOfVnodeWriteThreads;
if (tWWorkerInit(pAPool) != 0) return -1;
SWWorkerPool *pSPool = &pMgmt->syncPool; SWWorkerPool *pSPool = &pMgmt->syncPool;
pSPool->name = "vnode-sync"; pSPool->name = "vnode-sync";
pSPool->max = tsNumOfVnodeSyncThreads; pSPool->max = tsNumOfVnodeSyncThreads;
...@@ -449,6 +346,7 @@ void vmStopWorker(SVnodeMgmt *pMgmt) { ...@@ -449,6 +346,7 @@ void vmStopWorker(SVnodeMgmt *pMgmt) {
tSingleWorkerCleanup(&pMgmt->monitorWorker); tSingleWorkerCleanup(&pMgmt->monitorWorker);
tSingleWorkerCleanup(&pMgmt->mgmtWorker); tSingleWorkerCleanup(&pMgmt->mgmtWorker);
tWWorkerCleanup(&pMgmt->writePool); tWWorkerCleanup(&pMgmt->writePool);
tWWorkerCleanup(&pMgmt->applyPool);
tWWorkerCleanup(&pMgmt->syncPool); tWWorkerCleanup(&pMgmt->syncPool);
tQWorkerCleanup(&pMgmt->queryPool); tQWorkerCleanup(&pMgmt->queryPool);
tQWorkerCleanup(&pMgmt->fetchPool); tQWorkerCleanup(&pMgmt->fetchPool);
......
source/dnode/mgmt/node_mgmt/src/dmTransport.c
浏览文件 @ d5f6bb79
...@@ -22,17 +22,17 @@ static void dmSendRsp(SRpcMsg *pMsg); ...@@ -22,17 +22,17 @@ static void dmSendRsp(SRpcMsg *pMsg);
static void dmBuildMnodeRedirectRsp(SDnode *pDnode, SRpcMsg *pMsg); static void dmBuildMnodeRedirectRsp(SDnode *pDnode, SRpcMsg *pMsg);
static inline int32_t dmBuildNodeMsg(SRpcMsg *pMsg, SRpcMsg *pRpc) { static inline int32_t dmBuildNodeMsg(SRpcMsg *pMsg, SRpcMsg *pRpc) {
SRpcConnInfo connInfo = {0}; SRpcConnInfo *pConnInfo = &(pRpc->info.connInfo);
if (IsReq(pRpc) && rpcGetConnInfo(pRpc->info.handle, &connInfo) != 0) { // if (IsReq(pRpc)) {
terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; // terrno = TSDB_CODE_MND_NO_USER_FROM_CONN;
dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle); // dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle);
return -1; // return -1;
} //}
memcpy(pMsg, pRpc, sizeof(SRpcMsg)); memcpy(pMsg, pRpc, sizeof(SRpcMsg));
memcpy(pMsg->conn.user, connInfo.user, TSDB_USER_LEN); memcpy(pMsg->conn.user, pConnInfo->user, TSDB_USER_LEN);
pMsg->conn.clientIp = connInfo.clientIp; pMsg->conn.clientIp = pConnInfo->clientIp;
pMsg->conn.clientPort = connInfo.clientPort; pMsg->conn.clientPort = pConnInfo->clientPort;
return 0; return 0;
} }
...@@ -49,9 +49,9 @@ int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) { ...@@ -49,9 +49,9 @@ int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) {
} }
static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) { static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
SDnodeTrans *pTrans = &pDnode->trans; SDnodeTrans * pTrans = &pDnode->trans;
int32_t code = -1; int32_t code = -1;
SRpcMsg *pMsg = NULL; SRpcMsg * pMsg = NULL;
SMgmtWrapper *pWrapper = NULL; SMgmtWrapper *pWrapper = NULL;
SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)]; SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)];
...@@ -167,11 +167,11 @@ int32_t dmInitMsgHandle(SDnode *pDnode) { ...@@ -167,11 +167,11 @@ int32_t dmInitMsgHandle(SDnode *pDnode) {
for (EDndNodeType ntype = DNODE; ntype < NODE_END; ++ntype) { for (EDndNodeType ntype = DNODE; ntype < NODE_END; ++ntype) {
SMgmtWrapper *pWrapper = &pDnode->wrappers[ntype]; SMgmtWrapper *pWrapper = &pDnode->wrappers[ntype];
SArray *pArray = (*pWrapper->func.getHandlesFp)(); SArray * pArray = (*pWrapper->func.getHandlesFp)();
if (pArray == NULL) return -1; if (pArray == NULL) return -1;
for (int32_t i = 0; i < taosArrayGetSize(pArray); ++i) { for (int32_t i = 0; i < taosArrayGetSize(pArray); ++i) {
SMgmtHandle *pMgmt = taosArrayGet(pArray, i); SMgmtHandle * pMgmt = taosArrayGet(pArray, i);
SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pMgmt->msgType)]; SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pMgmt->msgType)];
if (pMgmt->needCheckVgId) { if (pMgmt->needCheckVgId) {
pHandle->needCheckVgId = pMgmt->needCheckVgId; pHandle->needCheckVgId = pMgmt->needCheckVgId;
......
source/dnode/mnode/impl/src/mndConsumer.c
浏览文件 @ d5f6bb79
...@@ -92,6 +92,15 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) { ...@@ -92,6 +92,15 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) {
SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId); SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId);
ASSERT(pConsumer); ASSERT(pConsumer);
mInfo("receive consumer lost msg, consumer id %ld, status %s", pLostMsg->consumerId,
mndConsumerStatusName(pConsumer->status));
if (pConsumer->status != MQ_CONSUMER_STATUS__READY) {
mndReleaseConsumer(pMnode, pConsumer);
return -1;
}
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup); SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup);
pConsumerNew->updateType = CONSUMER_UPDATE__LOST; pConsumerNew->updateType = CONSUMER_UPDATE__LOST;
......
source/dnode/mnode/impl/src/mndShow.c
浏览文件 @ d5f6bb79
...@@ -266,7 +266,7 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { ...@@ -266,7 +266,7 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) {
} }
size = sizeof(SRetrieveMetaTableRsp) + sizeof(int32_t) + sizeof(SSysTableSchema) * pShow->pMeta->numOfColumns + size = sizeof(SRetrieveMetaTableRsp) + sizeof(int32_t) + sizeof(SSysTableSchema) * pShow->pMeta->numOfColumns +
blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock); blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock->info.numOfCols);
SRetrieveMetaTableRsp *pRsp = rpcMallocCont(size); SRetrieveMetaTableRsp *pRsp = rpcMallocCont(size);
if (pRsp == NULL) { if (pRsp == NULL) {
......
source/dnode/mnode/impl/src/mndSma.c
浏览文件 @ d5f6bb79
...@@ -894,6 +894,15 @@ static int32_t mndGetTableSma(SMnode *pMnode, STableIndexReq *indexReq, STableIn ...@@ -894,6 +894,15 @@ static int32_t mndGetTableSma(SMnode *pMnode, STableIndexReq *indexReq, STableIn
info.sliding = pSma->sliding; info.sliding = pSma->sliding;
info.dstTbUid = pSma->dstTbUid; info.dstTbUid = pSma->dstTbUid;
info.dstVgId = pSma->dstVgId; info.dstVgId = pSma->dstVgId;
SVgObj* pVg = mndAcquireVgroup(pMnode, pSma->dstVgId);
if (pVg == NULL) {
code = -1;
sdbRelease(pSdb, pSma);
return code;
}
info.epSet = mndGetVgroupEpset(pMnode, pVg);
info.expr = taosMemoryMalloc(pSma->exprLen + 1); info.expr = taosMemoryMalloc(pSma->exprLen + 1);
if (info.expr == NULL) { if (info.expr == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
......
source/dnode/vnode/inc/vnode.h
浏览文件 @ d5f6bb79
...@@ -71,6 +71,9 @@ int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader); ...@@ -71,6 +71,9 @@ int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader);
int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData); int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData);
int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen); int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen);
void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs);
void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs);
// meta // meta
typedef struct SMeta SMeta; // todo: remove typedef struct SMeta SMeta; // todo: remove
typedef struct SMetaReader SMetaReader; typedef struct SMetaReader SMetaReader;
......
source/dnode/vnode/src/inc/vnd.h
浏览文件 @ d5f6bb79
...@@ -84,7 +84,6 @@ int32_t vnodeAsyncCommit(SVnode* pVnode); ...@@ -84,7 +84,6 @@ int32_t vnodeAsyncCommit(SVnode* pVnode);
int32_t vnodeSyncOpen(SVnode* pVnode, char* path); int32_t vnodeSyncOpen(SVnode* pVnode, char* path);
void vnodeSyncStart(SVnode* pVnode); void vnodeSyncStart(SVnode* pVnode);
void vnodeSyncClose(SVnode* pVnode); void vnodeSyncClose(SVnode* pVnode);
int32_t vnodeSyncAlter(SVnode* pVnode, SRpcMsg* pMsg);
#ifdef __cplusplus #ifdef __cplusplus
} }
......
source/dnode/vnode/src/inc/vnodeInt.h
浏览文件 @ d5f6bb79
...@@ -220,8 +220,10 @@ struct SVnode { ...@@ -220,8 +220,10 @@ struct SVnode {
SWal* pWal; SWal* pWal;
STQ* pTq; STQ* pTq;
SSink* pSink; SSink* pSink;
int64_t sync;
tsem_t canCommit; tsem_t canCommit;
int64_t sync;
int32_t syncCount;
sem_t syncSem;
SQHandle* pQuery; SQHandle* pQuery;
}; };
......
source/dnode/vnode/src/meta/metaTable.c
浏览文件 @ d5f6bb79
...@@ -99,7 +99,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const ...@@ -99,7 +99,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
indexMultiTermAdd(terms, term); indexMultiTermAdd(terms, term);
} }
} }
tIndexJsonPut(pMeta->pTagIvtIdx, terms, tuid); indexJsonPut(pMeta->pTagIvtIdx, terms, tuid);
indexMultiTermDestroy(terms); indexMultiTermDestroy(terms);
#endif #endif
return 0; return 0;
......
source/dnode/vnode/src/tq/tq.c
浏览文件 @ d5f6bb79
...@@ -448,7 +448,7 @@ int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg) { ...@@ -448,7 +448,7 @@ int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg) {
} }
int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) { int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) {
SStreamDispatchRsp* pRsp = pMsg->pCont; SStreamDispatchRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
int32_t taskId = pRsp->taskId; int32_t taskId = pRsp->taskId;
SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t)); SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t));
streamProcessDispatchRsp(pTask, &pTq->pVnode->msgCb, pRsp); streamProcessDispatchRsp(pTask, &pTq->pVnode->msgCb, pRsp);
......
source/dnode/vnode/src/tq/tqExec.c
浏览文件 @ d5f6bb79
...@@ -39,6 +39,9 @@ static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataBlkRsp* pRs ...@@ -39,6 +39,9 @@ static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataBlkRsp* pRs
static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, int32_t workerId, SMqDataBlkRsp* pRsp) { static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, int32_t workerId, SMqDataBlkRsp* pRsp) {
SSchemaWrapper* pSW = tCloneSSchemaWrapper(pExec->pExecReader[workerId]->pSchemaWrapper); SSchemaWrapper* pSW = tCloneSSchemaWrapper(pExec->pExecReader[workerId]->pSchemaWrapper);
if (pSW == NULL) {
return -1;
}
taosArrayPush(pRsp->blockSchema, &pSW); taosArrayPush(pRsp->blockSchema, &pSW);
return 0; return 0;
} }
......
source/dnode/vnode/src/vnd/vnodeOpen.c
浏览文件 @ d5f6bb79
...@@ -81,7 +81,9 @@ SVnode *vnodeOpen(const char *path, STfs *pTfs, SMsgCb msgCb) { ...@@ -81,7 +81,9 @@ SVnode *vnodeOpen(const char *path, STfs *pTfs, SMsgCb msgCb) {
pVnode->state.applied = info.state.committed; pVnode->state.applied = info.state.committed;
pVnode->pTfs = pTfs; pVnode->pTfs = pTfs;
pVnode->msgCb = msgCb; pVnode->msgCb = msgCb;
pVnode->syncCount = 0;
tsem_init(&pVnode->syncSem, 0, 0);
tsem_init(&(pVnode->canCommit), 0, 1); tsem_init(&(pVnode->canCommit), 0, 1);
// open buffer pool // open buffer pool
...@@ -175,6 +177,7 @@ void vnodeClose(SVnode *pVnode) { ...@@ -175,6 +177,7 @@ void vnodeClose(SVnode *pVnode) {
vnodeCloseBufPool(pVnode); vnodeCloseBufPool(pVnode);
// destroy handle // destroy handle
tsem_destroy(&(pVnode->canCommit)); tsem_destroy(&(pVnode->canCommit));
tsem_destroy(&pVnode->syncSem);
taosMemoryFree(pVnode); taosMemoryFree(pVnode);
} }
} }
......
source/dnode/vnode/src/vnd/vnodeSvr.c
浏览文件 @ d5f6bb79
...@@ -91,9 +91,6 @@ int32_t vnodePreprocessReq(SVnode *pVnode, SRpcMsg *pMsg) { ...@@ -91,9 +91,6 @@ int32_t vnodePreprocessReq(SVnode *pVnode, SRpcMsg *pMsg) {
} }
} break; } break;
case TDMT_VND_ALTER_REPLICA: {
code = vnodeSyncAlter(pVnode, pMsg);
} break;
default: default:
break; break;
} }
...@@ -107,7 +104,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp ...@@ -107,7 +104,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
int32_t len; int32_t len;
int32_t ret; int32_t ret;
vTrace("vgId:%d, start to process write request %s, version %" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), vTrace("vgId:%d, start to process write request %s, index:%" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType),
version); version);
pVnode->state.applied = version; pVnode->state.applied = version;
...@@ -173,7 +170,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp ...@@ -173,7 +170,7 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
break; break;
} }
vTrace("vgId:%d, process %s request success, version: %" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), version); vTrace("vgId:%d, process %s request success, index:%" PRId64, TD_VID(pVnode), TMSG_INFO(pMsg->msgType), version);
if (tqPushMsg(pVnode->pTq, pMsg->pCont, pMsg->contLen, pMsg->msgType, version) < 0) { if (tqPushMsg(pVnode->pTq, pMsg->pCont, pMsg->contLen, pMsg->msgType, version) < 0) {
vError("vgId:%d, failed to push msg to TQ since %s", TD_VID(pVnode), tstrerror(terrno)); vError("vgId:%d, failed to push msg to TQ since %s", TD_VID(pVnode), tstrerror(terrno));
......
source/dnode/vnode/src/vnd/vnodeSync.c
浏览文件 @ d5f6bb79
...@@ -16,41 +16,41 @@ ...@@ -16,41 +16,41 @@
#define _DEFAULT_SOURCE #define _DEFAULT_SOURCE
#include "vnd.h" #include "vnd.h"
static int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg); static inline bool vnodeIsMsgBlock(tmsg_t type) {
static int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg); return (type == TDMT_VND_CREATE_TABLE) || (type == TDMT_VND_ALTER_CONFIRM) || (type == TDMT_VND_ALTER_REPLICA);
static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode); }
static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta);
static void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta);
static void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta);
static int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot);
int32_t vnodeSyncOpen(SVnode *pVnode, char *path) { static inline bool vnodeIsMsgWeak(tmsg_t type) { return false; }
SSyncInfo syncInfo = {
.vgId = pVnode->config.vgId,
.isStandBy = pVnode->config.standby,
.syncCfg = pVnode->config.syncCfg,
.pWal = pVnode->pWal,
.msgcb = NULL,
.FpSendMsg = vnodeSyncSendMsg,
.FpEqMsg = vnodeSyncEqMsg,
};
snprintf(syncInfo.path, sizeof(syncInfo.path), "%s%ssync", path, TD_DIRSEP); static inline void vnodeAccumBlockMsg(SVnode *pVnode, tmsg_t type) {
syncInfo.pFsm = vnodeSyncMakeFsm(pVnode); if (!vnodeIsMsgBlock(type)) return;
pVnode->sync = syncOpen(&syncInfo); int32_t count = atomic_add_fetch_32(&pVnode->syncCount, 1);
if (pVnode->sync <= 0) { vTrace("vgId:%d, accum block, count:%d type:%s", pVnode->config.vgId, count, TMSG_INFO(type));
vError("vgId:%d, failed to open sync since %s", pVnode->config.vgId, terrstr()); }
return -1;
}
setPingTimerMS(pVnode->sync, 3000); static inline void vnodeWaitBlockMsg(SVnode *pVnode) {
setElectTimerMS(pVnode->sync, 500); int32_t count = atomic_load_32(&pVnode->syncCount);
setHeartbeatTimerMS(pVnode->sync, 100); if (count <= 0) return;
return 0;
vTrace("vgId:%d, wait block finish, count:%d", pVnode->config.vgId, count);
tsem_wait(&pVnode->syncSem);
}
static inline void vnodePostBlockMsg(SVnode *pVnode, tmsg_t type) {
if (!vnodeIsMsgBlock(type)) return;
int32_t count = atomic_load_32(&pVnode->syncCount);
if (count <= 0) return;
count = atomic_sub_fetch_32(&pVnode->syncCount, 1);
vTrace("vgId:%d, post block, count:%d type:%s", pVnode->config.vgId, count, TMSG_INFO(type));
if (count <= 0) {
tsem_post(&pVnode->syncSem);
}
} }
int32_t vnodeSyncAlter(SVnode *pVnode, SRpcMsg *pMsg) { static int32_t vnodeProcessSyncReconfigReq(SVnode *pVnode, SRpcMsg *pMsg) {
SAlterVnodeReq req = {0}; SAlterVnodeReq req = {0};
if (tDeserializeSAlterVnodeReq((char *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead), &req) != 0) { if (tDeserializeSAlterVnodeReq((char *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead), &req) != 0) {
terrno = TSDB_CODE_INVALID_MSG; terrno = TSDB_CODE_INVALID_MSG;
...@@ -66,29 +66,98 @@ int32_t vnodeSyncAlter(SVnode *pVnode, SRpcMsg *pMsg) { ...@@ -66,29 +66,98 @@ int32_t vnodeSyncAlter(SVnode *pVnode, SRpcMsg *pMsg) {
vInfo("vgId:%d, replica:%d %s:%u", TD_VID(pVnode), r, pNode->nodeFqdn, pNode->nodePort); vInfo("vgId:%d, replica:%d %s:%u", TD_VID(pVnode), r, pNode->nodeFqdn, pNode->nodePort);
} }
int32_t code = syncReconfig(pVnode->sync, &cfg); return syncReconfig(pVnode->sync, &cfg);
if (code == TAOS_SYNC_PROPOSE_SUCCESS) { }
// todo refactor
SRpcMsg rsp = {.info = pMsg->info, .code = 0}; void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
tmsgSendRsp(&rsp); SVnode *pVnode = pInfo->ahandle;
return TSDB_CODE_ACTION_IN_PROGRESS; int32_t vgId = pVnode->config.vgId;
int32_t code = 0;
SRpcMsg *pMsg = NULL;
for (int32_t m = 0; m < numOfMsgs; m++) {
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
vTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle);
if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) {
code = vnodeProcessSyncReconfigReq(pVnode, pMsg);
} else {
code = vnodePreprocessReq(pVnode, pMsg);
if (code != 0) {
vError("vgId:%d, failed to pre-process msg:%p since %s", vgId, pMsg, terrstr());
} else {
code = syncPropose(pVnode->sync, pMsg, vnodeIsMsgWeak(pMsg->msgType));
}
}
if (code == 0) {
vnodeAccumBlockMsg(pVnode, pMsg->msgType);
if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) {
// todo refactor
SRpcMsg rsp = {.code = code, .info = pMsg->info};
tmsgSendRsp(&rsp);
}
} else if (code == TAOS_SYNC_PROPOSE_NOT_LEADER) {
SEpSet newEpSet = {0};
syncGetEpSet(pVnode->sync, &newEpSet);
SEp *pEp = &newEpSet.eps[newEpSet.inUse];
if (pEp->port == tsServerPort && strcmp(pEp->fqdn, tsLocalFqdn) == 0) {
newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps;
}
vTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps,
newEpSet.inUse);
for (int32_t i = 0; i < newEpSet.numOfEps; ++i) {
vTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port);
}
SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info};
tmsgSendRedirectRsp(&rsp, &newEpSet);
} else {
if (terrno != 0) code = terrno;
vError("vgId:%d, msg:%p failed to propose since %s, code:0x%x", vgId, pMsg, tstrerror(code), code);
SRpcMsg rsp = {.code = code, .info = pMsg->info};
tmsgSendRsp(&rsp);
}
vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code);
rpcFreeCont(pMsg->pCont);
taosFreeQitem(pMsg);
} }
return code; vnodeWaitBlockMsg(pVnode);
} }
void vnodeSyncStart(SVnode *pVnode) { void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
syncSetMsgCb(pVnode->sync, &pVnode->msgCb); SVnode *pVnode = pInfo->ahandle;
if (pVnode->config.standby) { int32_t vgId = pVnode->config.vgId;
syncStartStandBy(pVnode->sync); int32_t code = 0;
} else { SRpcMsg *pMsg = NULL;
syncStart(pVnode->sync);
for (int32_t i = 0; i < numOfMsgs; ++i) {
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
vTrace("vgId:%d, msg:%p get from vnode-apply queue, handle:%p", vgId, pMsg, pMsg->info.handle);
SRpcMsg rsp = {.code = pMsg->code, .info = pMsg->info};
if (rsp.code == 0) {
if (vnodeProcessWriteReq(pVnode, pMsg, pMsg->conn.applyIndex, &rsp) < 0) {
rsp.code = terrno;
vError("vgId:%d, msg:%p failed to apply since %s", vgId, pMsg, terrstr());
}
}
vnodePostBlockMsg(pVnode, pMsg->msgType);
if (rsp.info.handle != NULL) {
tmsgSendRsp(&rsp);
}
vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code);
rpcFreeCont(pMsg->pCont);
taosFreeQitem(pMsg);
} }
} }
void vnodeSyncClose(SVnode *pVnode) { syncStop(pVnode->sync); } static int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) {
int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) {
int32_t code = tmsgPutToQueue(msgcb, SYNC_QUEUE, pMsg); int32_t code = tmsgPutToQueue(msgcb, SYNC_QUEUE, pMsg);
if (code != 0) { if (code != 0) {
rpcFreeCont(pMsg->pCont); rpcFreeCont(pMsg->pCont);
...@@ -97,7 +166,7 @@ int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { ...@@ -97,7 +166,7 @@ int32_t vnodeSyncEqMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) {
return code; return code;
} }
int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) { static int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) {
int32_t code = tmsgSendReq(pEpSet, pMsg); int32_t code = tmsgSendReq(pEpSet, pMsg);
if (code != 0) { if (code != 0) {
rpcFreeCont(pMsg->pCont); rpcFreeCont(pMsg->pCont);
...@@ -106,53 +175,43 @@ int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) { ...@@ -106,53 +175,43 @@ int32_t vnodeSyncSendMsg(const SEpSet *pEpSet, SRpcMsg *pMsg) {
return code; return code;
} }
int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot) { static int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot) {
vnodeGetSnapshot(pFsm->data, pSnapshot); vnodeGetSnapshot(pFsm->data, pSnapshot);
return 0; return 0;
} }
void vnodeSyncReconfig(struct SSyncFSM *pFsm, SSyncCfg newCfg, SReConfigCbMeta cbMeta) { static void vnodeSyncReconfig(struct SSyncFSM *pFsm, SSyncCfg newCfg, SReConfigCbMeta cbMeta) {
SVnode *pVnode = pFsm->data; SVnode *pVnode = pFsm->data;
vInfo("vgId:%d, sync reconfig is confirmed", TD_VID(pVnode)); vInfo("vgId:%d, sync reconfig is confirmed", TD_VID(pVnode));
// todo rpc response here // todo rpc response here
// build rpc msg
// put into apply queue
} }
void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
SVnode *pVnode = pFsm->data;
SSnapshot snapshot = {0};
SyncIndex beginIndex = SYNC_INDEX_INVALID; SyncIndex beginIndex = SYNC_INDEX_INVALID;
char logBuf[256] = {0};
if (pFsm->FpGetSnapshot != NULL) { if (pFsm->FpGetSnapshot != NULL) {
SSnapshot snapshot = {0}; (*pFsm->FpGetSnapshot)(pFsm, &snapshot);
pFsm->FpGetSnapshot(pFsm, &snapshot);
beginIndex = snapshot.lastApplyIndex; beginIndex = snapshot.lastApplyIndex;
} }
if (cbMeta.index > beginIndex) { if (cbMeta.index > beginIndex) {
char logBuf[256] = {0};
snprintf( snprintf(
logBuf, sizeof(logBuf), logBuf, sizeof(logBuf),
"==callback== ==CommitCb== execute, pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s, beginIndex :%ld\n", "==callback== ==CommitCb== execute, pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s, beginIndex :%ld\n",
pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), beginIndex); pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), beginIndex);
syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg);
SVnode *pVnode = pFsm->data; SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen, .conn.applyIndex = cbMeta.index};
SyncApplyMsg *pSyncApplyMsg = syncApplyMsgBuild2(pMsg, pVnode->config.vgId, &cbMeta); rpcMsg.pCont = rpcMallocCont(rpcMsg.contLen);
SRpcMsg applyMsg; memcpy(rpcMsg.pCont, pMsg->pCont, pMsg->contLen);
syncApplyMsg2RpcMsg(pSyncApplyMsg, &applyMsg); syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info);
syncApplyMsgDestroy(pSyncApplyMsg); tmsgPutToQueue(&pVnode->msgCb, APPLY_QUEUE, &rpcMsg);
// recover handle for response
SRpcMsg saveRpcMsg;
int32_t ret = syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &saveRpcMsg);
if (ret == 1 && cbMeta.state == TAOS_SYNC_STATE_LEADER) {
applyMsg.info = saveRpcMsg.info;
} else {
applyMsg.info.handle = NULL;
applyMsg.info.ahandle = NULL;
}
// put to applyQ
tmsgPutToQueue(&(pVnode->msgCb), APPLY_QUEUE, &applyMsg);
} else { } else {
char logBuf[256] = {0}; char logBuf[256] = {0};
snprintf(logBuf, sizeof(logBuf), snprintf(logBuf, sizeof(logBuf),
...@@ -164,7 +223,7 @@ void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) ...@@ -164,7 +223,7 @@ void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta)
} }
} }
void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { static void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
char logBuf[256] = {0}; char logBuf[256] = {0};
snprintf(logBuf, sizeof(logBuf), snprintf(logBuf, sizeof(logBuf),
"==callback== ==PreCommitCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n", pFsm, cbMeta.index, "==callback== ==PreCommitCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n", pFsm, cbMeta.index,
...@@ -172,14 +231,14 @@ void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMet ...@@ -172,14 +231,14 @@ void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMet
syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg);
} }
void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { static void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
char logBuf[256] = {0}; char logBuf[256] = {0};
snprintf(logBuf, sizeof(logBuf), "==callback== ==RollBackCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n", snprintf(logBuf, sizeof(logBuf), "==callback== ==RollBackCb== pFsm:%p, index:%ld, isWeak:%d, code:%d, state:%d %s \n",
pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state)); pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state));
syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg);
} }
SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) {
SSyncFSM *pFsm = taosMemoryCalloc(1, sizeof(SSyncFSM)); SSyncFSM *pFsm = taosMemoryCalloc(1, sizeof(SSyncFSM));
pFsm->data = pVnode; pFsm->data = pVnode;
pFsm->FpCommitCb = vnodeSyncCommitMsg; pFsm->FpCommitCb = vnodeSyncCommitMsg;
...@@ -188,6 +247,42 @@ SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { ...@@ -188,6 +247,42 @@ SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) {
pFsm->FpGetSnapshot = vnodeSyncGetSnapshot; pFsm->FpGetSnapshot = vnodeSyncGetSnapshot;
pFsm->FpRestoreFinishCb = NULL; pFsm->FpRestoreFinishCb = NULL;
pFsm->FpReConfigCb = vnodeSyncReconfig; pFsm->FpReConfigCb = vnodeSyncReconfig;
return pFsm; return pFsm;
} }
\ No newline at end of file
int32_t vnodeSyncOpen(SVnode *pVnode, char *path) {
SSyncInfo syncInfo = {
.vgId = pVnode->config.vgId,
.isStandBy = pVnode->config.standby,
.syncCfg = pVnode->config.syncCfg,
.pWal = pVnode->pWal,
.msgcb = NULL,
.FpSendMsg = vnodeSyncSendMsg,
.FpEqMsg = vnodeSyncEqMsg,
};
snprintf(syncInfo.path, sizeof(syncInfo.path), "%s%ssync", path, TD_DIRSEP);
syncInfo.pFsm = vnodeSyncMakeFsm(pVnode);
pVnode->sync = syncOpen(&syncInfo);
if (pVnode->sync <= 0) {
vError("vgId:%d, failed to open sync since %s", pVnode->config.vgId, terrstr());
return -1;
}
setPingTimerMS(pVnode->sync, 3000);
setElectTimerMS(pVnode->sync, 500);
setHeartbeatTimerMS(pVnode->sync, 100);
return 0;
}
void vnodeSyncStart(SVnode *pVnode) {
syncSetMsgCb(pVnode->sync, &pVnode->msgCb);
if (pVnode->config.standby) {
syncStartStandBy(pVnode->sync);
} else {
syncStart(pVnode->sync);
}
}
void vnodeSyncClose(SVnode *pVnode) { syncStop(pVnode->sync); }
source/libs/catalog/inc/catalogInt.h
浏览文件 @ d5f6bb79
...@@ -173,7 +173,6 @@ typedef struct SCtgJob { ...@@ -173,7 +173,6 @@ typedef struct SCtgJob {
SArray* pTasks; SArray* pTasks;
int32_t taskDone; int32_t taskDone;
SMetaData jobRes; SMetaData jobRes;
int32_t rspCode;
uint64_t queryId; uint64_t queryId;
SCatalog* pCtg; SCatalog* pCtg;
...@@ -201,11 +200,12 @@ typedef struct SCtgMsgCtx { ...@@ -201,11 +200,12 @@ typedef struct SCtgMsgCtx {
typedef struct SCtgTask { typedef struct SCtgTask {
CTG_TASK_TYPE type; CTG_TASK_TYPE type;
int32_t taskId; int32_t taskId;
SCtgJob *pJob; SCtgJob* pJob;
void* taskCtx; void* taskCtx;
SCtgMsgCtx msgCtx; SCtgMsgCtx msgCtx;
void* res; int32_t code;
void* res;
} SCtgTask; } SCtgTask;
typedef int32_t (*ctgLanchTaskFp)(SCtgTask*); typedef int32_t (*ctgLanchTaskFp)(SCtgTask*);
......
source/libs/catalog/src/ctgAsync.c
浏览文件 @ d5f6bb79
...@@ -437,13 +437,14 @@ _return: ...@@ -437,13 +437,14 @@ _return:
int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { int32_t ctgDumpTbMetaRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pTableMeta) { if (NULL == pJob->jobRes.pTableMeta) {
pJob->jobRes.pTableMeta = taosArrayInit(pJob->tbMetaNum, POINTER_BYTES); pJob->jobRes.pTableMeta = taosArrayInit(pJob->tbMetaNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pTableMeta) { if (NULL == pJob->jobRes.pTableMeta) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pTableMeta, &pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pTableMeta, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -451,14 +452,14 @@ int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { ...@@ -451,14 +452,14 @@ int32_t ctgDumpTbMetaRes(SCtgTask* pTask) {
int32_t ctgDumpDbVgRes(SCtgTask* pTask) { int32_t ctgDumpDbVgRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pDbVgroup) { if (NULL == pJob->jobRes.pDbVgroup) {
pJob->jobRes.pDbVgroup = taosArrayInit(pJob->dbVgNum, POINTER_BYTES); pJob->jobRes.pDbVgroup = taosArrayInit(pJob->dbVgNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pDbVgroup) { if (NULL == pJob->jobRes.pDbVgroup) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pDbVgroup, &pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
pTask->res = NULL; taosArrayPush(pJob->jobRes.pDbVgroup, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -466,13 +467,14 @@ int32_t ctgDumpDbVgRes(SCtgTask* pTask) { ...@@ -466,13 +467,14 @@ int32_t ctgDumpDbVgRes(SCtgTask* pTask) {
int32_t ctgDumpTbHashRes(SCtgTask* pTask) { int32_t ctgDumpTbHashRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pTableHash) { if (NULL == pJob->jobRes.pTableHash) {
pJob->jobRes.pTableHash = taosArrayInit(pJob->tbHashNum, sizeof(SVgroupInfo)); pJob->jobRes.pTableHash = taosArrayInit(pJob->tbHashNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pTableHash) { if (NULL == pJob->jobRes.pTableHash) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pTableHash, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pTableHash, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -480,21 +482,29 @@ int32_t ctgDumpTbHashRes(SCtgTask* pTask) { ...@@ -480,21 +482,29 @@ int32_t ctgDumpTbHashRes(SCtgTask* pTask) {
int32_t ctgDumpIndexRes(SCtgTask* pTask) { int32_t ctgDumpIndexRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pIndex) { if (NULL == pJob->jobRes.pIndex) {
pJob->jobRes.pIndex = taosArrayInit(pJob->indexNum, sizeof(SIndexInfo)); pJob->jobRes.pIndex = taosArrayInit(pJob->indexNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pIndex) { if (NULL == pJob->jobRes.pIndex) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pIndex, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pIndex, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t ctgDumpQnodeRes(SCtgTask* pTask) { int32_t ctgDumpQnodeRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pQnodeList) {
pJob->jobRes.pQnodeList = taosArrayInit(1, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pQnodeList) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
}
}
TSWAP(pJob->jobRes.pQnodeList, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pQnodeList, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -502,13 +512,14 @@ int32_t ctgDumpQnodeRes(SCtgTask* pTask) { ...@@ -502,13 +512,14 @@ int32_t ctgDumpQnodeRes(SCtgTask* pTask) {
int32_t ctgDumpDbCfgRes(SCtgTask* pTask) { int32_t ctgDumpDbCfgRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pDbCfg) { if (NULL == pJob->jobRes.pDbCfg) {
pJob->jobRes.pDbCfg = taosArrayInit(pJob->dbCfgNum, sizeof(SDbCfgInfo)); pJob->jobRes.pDbCfg = taosArrayInit(pJob->dbCfgNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pDbCfg) { if (NULL == pJob->jobRes.pDbCfg) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pDbCfg, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pDbCfg, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -516,13 +527,14 @@ int32_t ctgDumpDbCfgRes(SCtgTask* pTask) { ...@@ -516,13 +527,14 @@ int32_t ctgDumpDbCfgRes(SCtgTask* pTask) {
int32_t ctgDumpDbInfoRes(SCtgTask* pTask) { int32_t ctgDumpDbInfoRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pDbInfo) { if (NULL == pJob->jobRes.pDbInfo) {
pJob->jobRes.pDbInfo = taosArrayInit(pJob->dbInfoNum, sizeof(SDbInfo)); pJob->jobRes.pDbInfo = taosArrayInit(pJob->dbInfoNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pDbInfo) { if (NULL == pJob->jobRes.pDbInfo) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pDbInfo, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pDbInfo, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -530,13 +542,14 @@ int32_t ctgDumpDbInfoRes(SCtgTask* pTask) { ...@@ -530,13 +542,14 @@ int32_t ctgDumpDbInfoRes(SCtgTask* pTask) {
int32_t ctgDumpUdfRes(SCtgTask* pTask) { int32_t ctgDumpUdfRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pUdfList) { if (NULL == pJob->jobRes.pUdfList) {
pJob->jobRes.pUdfList = taosArrayInit(pJob->udfNum, sizeof(SFuncInfo)); pJob->jobRes.pUdfList = taosArrayInit(pJob->udfNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pUdfList) { if (NULL == pJob->jobRes.pUdfList) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pUdfList, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pUdfList, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -544,13 +557,14 @@ int32_t ctgDumpUdfRes(SCtgTask* pTask) { ...@@ -544,13 +557,14 @@ int32_t ctgDumpUdfRes(SCtgTask* pTask) {
int32_t ctgDumpUserRes(SCtgTask* pTask) { int32_t ctgDumpUserRes(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
if (NULL == pJob->jobRes.pUser) { if (NULL == pJob->jobRes.pUser) {
pJob->jobRes.pUser = taosArrayInit(pJob->userNum, sizeof(bool)); pJob->jobRes.pUser = taosArrayInit(pJob->userNum, sizeof(SMetaRes));
if (NULL == pJob->jobRes.pUser) { if (NULL == pJob->jobRes.pUser) {
CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
taosArrayPush(pJob->jobRes.pUser, pTask->res); SMetaRes res = {.code = pTask->code, .pRes = pTask->res};
taosArrayPush(pJob->jobRes.pUser, &res);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -559,20 +573,13 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) { ...@@ -559,20 +573,13 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) {
SCtgJob* pJob = pTask->pJob; SCtgJob* pJob = pTask->pJob;
int32_t code = 0; int32_t code = 0;
qDebug("QID:%" PRIx64 " task %d end with rsp %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); qDebug("QID:0x%" PRIx64 " task %d end with rsp %s", pJob->queryId, pTask->taskId, tstrerror(rspCode));
if (rspCode) { pTask->code = rspCode;
int32_t lastCode = atomic_val_compare_exchange_32(&pJob->rspCode, 0, rspCode);
if (0 == lastCode) {
CTG_ERR_JRET(rspCode);
}
return TSDB_CODE_SUCCESS;
}
int32_t taskDone = atomic_add_fetch_32(&pJob->taskDone, 1); int32_t taskDone = atomic_add_fetch_32(&pJob->taskDone, 1);
if (taskDone < taosArrayGetSize(pJob->pTasks)) { if (taskDone < taosArrayGetSize(pJob->pTasks)) {
qDebug("task done: %d, total: %d", taskDone, (int32_t)taosArrayGetSize(pJob->pTasks)); qDebug("QID:0x%" PRIx64 " task done: %d, total: %d", pJob->queryId, taskDone, (int32_t)taosArrayGetSize(pJob->pTasks));
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
......
source/libs/catalog/src/ctgUtil.c
浏览文件 @ d5f6bb79
...@@ -47,11 +47,13 @@ char *ctgTaskTypeStr(CTG_TASK_TYPE type) { ...@@ -47,11 +47,13 @@ char *ctgTaskTypeStr(CTG_TASK_TYPE type) {
void ctgFreeSMetaData(SMetaData* pData) { void ctgFreeSMetaData(SMetaData* pData) {
taosArrayDestroy(pData->pTableMeta); taosArrayDestroy(pData->pTableMeta);
pData->pTableMeta = NULL; pData->pTableMeta = NULL;
/*
for (int32_t i = 0; i < taosArrayGetSize(pData->pDbVgroup); ++i) { for (int32_t i = 0; i < taosArrayGetSize(pData->pDbVgroup); ++i) {
SArray** pArray = taosArrayGet(pData->pDbVgroup, i); SArray** pArray = taosArrayGet(pData->pDbVgroup, i);
taosArrayDestroy(*pArray); taosArrayDestroy(*pArray);
} }
*/
taosArrayDestroy(pData->pDbVgroup); taosArrayDestroy(pData->pDbVgroup);
pData->pDbVgroup = NULL; pData->pDbVgroup = NULL;
...@@ -61,10 +63,12 @@ void ctgFreeSMetaData(SMetaData* pData) { ...@@ -61,10 +63,12 @@ void ctgFreeSMetaData(SMetaData* pData) {
taosArrayDestroy(pData->pUdfList); taosArrayDestroy(pData->pUdfList);
pData->pUdfList = NULL; pData->pUdfList = NULL;
/*
for (int32_t i = 0; i < taosArrayGetSize(pData->pDbCfg); ++i) { for (int32_t i = 0; i < taosArrayGetSize(pData->pDbCfg); ++i) {
SDbCfgInfo* pInfo = taosArrayGet(pData->pDbCfg, i); SDbCfgInfo* pInfo = taosArrayGet(pData->pDbCfg, i);
taosArrayDestroy(pInfo->pRetensions); taosArrayDestroy(pInfo->pRetensions);
} }
*/
taosArrayDestroy(pData->pDbCfg); taosArrayDestroy(pData->pDbCfg);
pData->pDbCfg = NULL; pData->pDbCfg = NULL;
...@@ -320,8 +324,12 @@ void ctgFreeTask(SCtgTask* pTask) { ...@@ -320,8 +324,12 @@ void ctgFreeTask(SCtgTask* pTask) {
break; break;
} }
case CTG_TASK_GET_DB_CFG: { case CTG_TASK_GET_DB_CFG: {
taosMemoryFreeClear(pTask->taskCtx); taosMemoryFreeClear(pTask->taskCtx);
taosMemoryFreeClear(pTask->res); if (pTask->res) {
SDbCfgInfo* pInfo = (SDbCfgInfo*)pTask->res;
taosArrayDestroy(pInfo->pRetensions);
taosMemoryFreeClear(pTask->res);
}
break; break;
} }
case CTG_TASK_GET_DB_INFO: { case CTG_TASK_GET_DB_INFO: {
......
source/libs/command/inc/commandInt.h
浏览文件 @ d5f6bb79
...@@ -115,9 +115,9 @@ typedef struct SExplainCtx { ...@@ -115,9 +115,9 @@ typedef struct SExplainCtx {
#define EXPLAIN_ROW_NEW(level, ...) \ #define EXPLAIN_ROW_NEW(level, ...) \
do { \ do { \
if (isVerboseLine) { \ if (isVerboseLine) { \
tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s", (level) * 2 + 3, ""); \ tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s", (level) * 3 + 3, ""); \
} else { \ } else { \
tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s%s", (level) * 2, "", "-> "); \ tlen = snprintf(tbuf + VARSTR_HEADER_SIZE, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE, "%*s%s", (level) * 3, "", "-> "); \
} \ } \
tlen += snprintf(tbuf + VARSTR_HEADER_SIZE + tlen, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE - tlen, __VA_ARGS__); \ tlen += snprintf(tbuf + VARSTR_HEADER_SIZE + tlen, TSDB_EXPLAIN_RESULT_ROW_SIZE - VARSTR_HEADER_SIZE - tlen, __VA_ARGS__); \
} while (0) } while (0)
......
source/libs/command/src/command.c
浏览文件 @ d5f6bb79
...@@ -28,91 +28,94 @@ static int32_t getSchemaBytes(const SSchema* pSchema) { ...@@ -28,91 +28,94 @@ static int32_t getSchemaBytes(const SSchema* pSchema) {
} }
} }
// todo : to convert data according to SSDatablock static SSDataBlock* buildDescResultDataBlock() {
static void buildRspData(const STableMeta* pMeta, char* pData) { SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
int32_t* payloadLen = (int32_t*) pData; pBlock->info.numOfCols = DESCRIBE_RESULT_COLS;
uint64_t* groupId = (uint64_t*)(pData + sizeof(int32_t)); pBlock->info.hasVarCol = true;
int32_t* pColSizes = (int32_t*)(pData + sizeof(int32_t) + sizeof(uint64_t)); pBlock->pDataBlock = taosArrayInit(4, sizeof(SColumnInfoData));
pData = (char*) pColSizes + DESCRIBE_RESULT_COLS * sizeof(int32_t);
int32_t numOfRows = TABLE_TOTAL_COL_NUM(pMeta); SColumnInfoData infoData = {0};
infoData.info.type = TSDB_DATA_TYPE_VARCHAR;
infoData.info.bytes = DESCRIBE_RESULT_FIELD_LEN;
// Field taosArrayPush(pBlock->pDataBlock, &infoData);
int32_t* pOffset = (int32_t*)pData;
pData += numOfRows * sizeof(int32_t); infoData.info.type = TSDB_DATA_TYPE_VARCHAR;
infoData.info.bytes = DESCRIBE_RESULT_TYPE_LEN;
taosArrayPush(pBlock->pDataBlock, &infoData);
infoData.info.type = TSDB_DATA_TYPE_INT;
infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes;;
taosArrayPush(pBlock->pDataBlock, &infoData);
infoData.info.type = TSDB_DATA_TYPE_VARCHAR;
infoData.info.bytes = DESCRIBE_RESULT_NOTE_LEN;
taosArrayPush(pBlock->pDataBlock, &infoData);
return pBlock;
}
static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, STableMeta* pMeta) {
blockDataEnsureCapacity(pBlock, numOfRows);
pBlock->info.rows = numOfRows;
// field
SColumnInfoData* pCol1 = taosArrayGet(pBlock->pDataBlock, 0);
char buf[DESCRIBE_RESULT_FIELD_LEN] = {0};
for (int32_t i = 0; i < numOfRows; ++i) { for (int32_t i = 0; i < numOfRows; ++i) {
STR_TO_VARSTR(pData, pMeta->schema[i].name); STR_TO_VARSTR(buf, pMeta->schema[i].name);
int16_t len = varDataTLen(pData); colDataAppend(pCol1, i, buf, false);
pData += len;
*pOffset = pColSizes[0];
pOffset += 1;
pColSizes[0] += len;
} }
// Type // Type
pOffset = (int32_t*)pData; SColumnInfoData* pCol2 = taosArrayGet(pBlock->pDataBlock, 1);
pData += numOfRows * sizeof(int32_t);
for (int32_t i = 0; i < numOfRows; ++i) { for (int32_t i = 0; i < numOfRows; ++i) {
STR_TO_VARSTR(pData, tDataTypes[pMeta->schema[i].type].name); STR_TO_VARSTR(buf, tDataTypes[pMeta->schema[i].type].name);
int16_t len = varDataTLen(pData); colDataAppend(pCol2, i, buf, false);
pData += len;
*pOffset = pColSizes[1];
pOffset += 1;
pColSizes[1] += len;
} }
// Length // Length
pData += BitmapLen(numOfRows); SColumnInfoData* pCol3 = taosArrayGet(pBlock->pDataBlock, 2);
for (int32_t i = 0; i < numOfRows; ++i) { for (int32_t i = 0; i < numOfRows; ++i) {
*(int32_t*)pData = getSchemaBytes(pMeta->schema + i); int32_t bytes = getSchemaBytes(pMeta->schema + i);
pData += sizeof(int32_t); colDataAppend(pCol3, i, (const char*)&bytes, false);
} }
pColSizes[2] = sizeof(int32_t) * numOfRows;
// Note // Note
pOffset = (int32_t*)pData; SColumnInfoData* pCol4 = taosArrayGet(pBlock->pDataBlock, 3);
pData += numOfRows * sizeof(int32_t);
for (int32_t i = 0; i < numOfRows; ++i) { for (int32_t i = 0; i < numOfRows; ++i) {
STR_TO_VARSTR(pData, i >= pMeta->tableInfo.numOfColumns ? "TAG" : ""); STR_TO_VARSTR(buf, i >= pMeta->tableInfo.numOfColumns ? "TAG" : "");
int16_t len = varDataTLen(pData); colDataAppend(pCol4, i, buf, false);
pData += len;
*pOffset = pColSizes[3];
pOffset += 1;
pColSizes[3] += len;
}
for (int32_t i = 0; i < DESCRIBE_RESULT_COLS; ++i) {
pColSizes[i] = htonl(pColSizes[i]);
} }
*payloadLen = (int32_t)(pData - (char*)payloadLen);
}
static int32_t calcRspSize(const STableMeta* pMeta) {
int32_t numOfRows = TABLE_TOTAL_COL_NUM(pMeta);
return sizeof(SRetrieveTableRsp) +
(numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_FIELD_LEN) +
(numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_TYPE_LEN) +
(BitmapLen(numOfRows) + numOfRows * sizeof(int32_t)) +
(numOfRows * sizeof(int32_t) + numOfRows * DESCRIBE_RESULT_NOTE_LEN) +
sizeof(int32_t) + sizeof(uint64_t);
} }
static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) { static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) {
SDescribeStmt* pDesc = (SDescribeStmt*)pStmt; SDescribeStmt* pDesc = (SDescribeStmt*) pStmt;
*pRsp = taosMemoryCalloc(1, calcRspSize(pDesc->pMeta)); int32_t numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta);
SSDataBlock* pBlock = buildDescResultDataBlock();
setDescResultIntoDataBlock(pBlock, numOfRows, pDesc->pMeta);
size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock);
*pRsp = taosMemoryCalloc(1, rspSize);
if (NULL == *pRsp) { if (NULL == *pRsp) {
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
(*pRsp)->useconds = 0; (*pRsp)->useconds = 0;
(*pRsp)->completed = 1; (*pRsp)->completed = 1;
(*pRsp)->precision = 0; (*pRsp)->precision = 0;
(*pRsp)->compressed = 0; (*pRsp)->compressed = 0;
(*pRsp)->compLen = 0; (*pRsp)->compLen = 0;
(*pRsp)->numOfRows = htonl(TABLE_TOTAL_COL_NUM(pDesc->pMeta)); (*pRsp)->numOfRows = htonl(numOfRows);
buildRspData(pDesc->pMeta, (*pRsp)->data); (*pRsp)->numOfCols = htonl(DESCRIBE_RESULT_COLS);
int32_t len = 0;
blockCompressEncode(pBlock, (*pRsp)->data, &len, DESCRIBE_RESULT_COLS, false);
ASSERT(len == rspSize - sizeof(SRetrieveTableRsp));
blockDataDestroy(pBlock);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
......
source/libs/command/src/explain.c
浏览文件 @ d5f6bb79
...@@ -13,6 +13,7 @@ ...@@ -13,6 +13,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>. * along with this program. If not, see <http://www.gnu.org/licenses/>.
*/ */
#include "tdatablock.h"
#include "commandInt.h" #include "commandInt.h"
#include "plannodes.h" #include "plannodes.h"
#include "query.h" #include "query.h"
...@@ -657,7 +658,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i ...@@ -657,7 +658,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i
if (EXPLAIN_MODE_ANALYZE == ctx->mode) { if (EXPLAIN_MODE_ANALYZE == ctx->mode) {
// sort key // sort key
EXPLAIN_ROW_NEW(level, "Sort Key: "); EXPLAIN_ROW_NEW(level + 1, "Sort Key: ");
if (pResNode->pExecInfo) { if (pResNode->pExecInfo) {
for (int32_t i = 0; i < LIST_LENGTH(pSortNode->pSortKeys); ++i) { for (int32_t i = 0; i < LIST_LENGTH(pSortNode->pSortKeys); ++i) {
SOrderByExprNode *ptn = nodesListGetNode(pSortNode->pSortKeys, i); SOrderByExprNode *ptn = nodesListGetNode(pSortNode->pSortKeys, i);
...@@ -669,7 +670,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i ...@@ -669,7 +670,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i
QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level)); QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level));
// sort method // sort method
EXPLAIN_ROW_NEW(level, "Sort Method: "); EXPLAIN_ROW_NEW(level + 1, "Sort Method: ");
int32_t nodeNum = taosArrayGetSize(pResNode->pExecInfo); int32_t nodeNum = taosArrayGetSize(pResNode->pExecInfo);
SExplainExecInfo *execInfo = taosArrayGet(pResNode->pExecInfo, 0); SExplainExecInfo *execInfo = taosArrayGet(pResNode->pExecInfo, 0);
...@@ -965,9 +966,32 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) { ...@@ -965,9 +966,32 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) {
QRY_ERR_RET(TSDB_CODE_QRY_APP_ERROR); QRY_ERR_RET(TSDB_CODE_QRY_APP_ERROR);
} }
int32_t colNum = 1; SSDataBlock *pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
int32_t rspSize = sizeof(SRetrieveTableRsp) + sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t) * colNum + SColumnInfoData infoData = {0};
sizeof(int32_t) * rowNum + pCtx->dataSize; infoData.info.type = TSDB_DATA_TYPE_VARCHAR;
infoData.info.bytes = TSDB_EXPLAIN_RESULT_ROW_SIZE;
pBlock->pDataBlock = taosArrayInit(1, sizeof(SColumnInfoData));
taosArrayPush(pBlock->pDataBlock, &infoData);
SColumnInfoData* pInfoData = taosArrayGet(pBlock->pDataBlock, 0);
pInfoData->hasNull = false;
colInfoDataEnsureCapacity(pInfoData, 0, rowNum);
char buf[1024] = {0};
for (int32_t i = 0; i < rowNum; ++i) {
SQueryExplainRowInfo *row = taosArrayGet(pCtx->rows, i);
varDataCopy(buf, row->buf);
ASSERT(varDataTLen(row->buf) == row->len);
colDataAppend(pInfoData, i, buf, false);
}
pBlock->info.numOfCols = 1;
pBlock->info.rows = rowNum;
pBlock->info.hasVarCol = true;
int32_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock);
SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, rspSize); SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, rspSize);
if (NULL == rsp) { if (NULL == rsp) {
qError("malloc SRetrieveTableRsp failed, size:%d", rspSize); qError("malloc SRetrieveTableRsp failed, size:%d", rspSize);
...@@ -977,34 +1001,13 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) { ...@@ -977,34 +1001,13 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) {
rsp->completed = 1; rsp->completed = 1;
rsp->numOfRows = htonl(rowNum); rsp->numOfRows = htonl(rowNum);
// payload length int32_t len = 0;
*(int32_t *)rsp->data = blockCompressEncode(pBlock, rsp->data, &len, pBlock->info.numOfCols, 0);
sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t) * colNum + sizeof(int32_t) * rowNum + pCtx->dataSize; ASSERT(len == rspSize - sizeof(SRetrieveTableRsp));
// group id rsp->compLen = htonl(len);
*(uint64_t *)(rsp->data + sizeof(int32_t)) = 0;
// column length
int32_t *colLength = (int32_t *)(rsp->data + sizeof(int32_t) + sizeof(uint64_t));
// varchar column offset segment
int32_t *offset = (int32_t *)((char *)colLength + sizeof(int32_t));
// varchar data real payload
char *data = (char *)(offset + rowNum);
char *start = data;
for (int32_t i = 0; i < rowNum; ++i) {
SQueryExplainRowInfo *row = taosArrayGet(pCtx->rows, i);
offset[i] = data - start;
varDataCopy(data, row->buf);
ASSERT(varDataTLen(row->buf) == row->len);
data += row->len;
}
*colLength = htonl(data - start); blockDataDestroy(pBlock);
rsp->compLen = htonl(rspSize);
*pRsp = rsp; *pRsp = rsp;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
......
source/libs/executor/inc/executorimpl.h
浏览文件 @ d5f6bb79
...@@ -688,6 +688,10 @@ typedef struct SSortedMergeOperatorInfo { ...@@ -688,6 +688,10 @@ typedef struct SSortedMergeOperatorInfo {
int32_t numOfResPerPage; int32_t numOfResPerPage;
char** groupVal; char** groupVal;
SArray *groupInfo; SArray *groupInfo;
bool hasGroupId;
uint64_t groupId;
STupleHandle* prefetchedTuple;
} SSortedMergeOperatorInfo; } SSortedMergeOperatorInfo;
typedef struct SSortOperatorInfo { typedef struct SSortOperatorInfo {
...@@ -700,6 +704,10 @@ typedef struct SSortOperatorInfo { ...@@ -700,6 +704,10 @@ typedef struct SSortOperatorInfo {
int64_t startTs; // sort start time int64_t startTs; // sort start time
uint64_t sortElapsed; // sort elapsed time, time to flush to disk not included. uint64_t sortElapsed; // sort elapsed time, time to flush to disk not included.
STupleHandle *prefetchedTuple;
bool hasGroupId;
uint64_t groupId;
} SSortOperatorInfo; } SSortOperatorInfo;
typedef struct STagFilterOperatorInfo { typedef struct STagFilterOperatorInfo {
...@@ -759,7 +767,7 @@ void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData ...@@ -759,7 +767,7 @@ void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData
SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode); SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode);
SColumn extractColumnFromColumnNode(SColumnNode* pColNode); SColumn extractColumnFromColumnNode(SColumnNode* pColNode);
SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo); SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo, SSortOperatorInfo* pInfo);
SSDataBlock* loadNextDataBlock(void* param); SSDataBlock* loadNextDataBlock(void* param);
void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowCellInfoOffset); void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowCellInfoOffset);
......
source/libs/executor/src/dataDispatcher.c
浏览文件 @ d5f6bb79
...@@ -68,10 +68,10 @@ static bool needCompress(const SSDataBlock* pData, int32_t numOfCols) { ...@@ -68,10 +68,10 @@ static bool needCompress(const SSDataBlock* pData, int32_t numOfCols) {
} }
// data format: // data format:
// +----------------+--------------+----------+--------------------------------------+-------------+-----------+-------------+-----------+ // +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+
// |SDataCacheEntry | total length | group id | column#1 length, column#2 length ... | col1 bitmap | col1 data | col2 bitmap | col2 data | .... // |SDataCacheEntry | total length | group id | col1_schema | col2_schema | col3_schema ...| column#1 length, column#2 length ... | col1 bitmap | col1 data | col2 bitmap | col2 data | ....
// | | (4 bytes) |(8 bytes) | sizeof(int32_t) * numOfCols | actual size | | actual size | | // | | (4 bytes) |(8 bytes) |(sizeof(int16_t)+sizeof(int32_t))*numOfCols | sizeof(int32_t) * numOfCols | actual size | | actual size | |
// +----------------+--------------+----------+--------------------------------------+-------------+-----------+-------------+-----------+ // +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+
// The length of bitmap is decided by number of rows of this data block, and the length of each column data is // The length of bitmap is decided by number of rows of this data block, and the length of each column data is
// recorded in the first segment, next to the struct header // recorded in the first segment, next to the struct header
static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) { static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) {
......
source/libs/executor/src/executorimpl.c
浏览文件 @ d5f6bb79
...@@ -2571,37 +2571,7 @@ int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadIn ...@@ -2571,37 +2571,7 @@ int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadIn
} }
} }
blockDataEnsureCapacity(pBlock, numOfRows); blockCompressDecode(pBlock, numOfCols, numOfRows, pStart);
int32_t dataLen = *(int32_t*)pStart;
uint64_t groupId = *(uint64_t*)(pStart + sizeof(int32_t));
pStart += sizeof(int32_t) + sizeof(uint64_t);
int32_t* colLen = (int32_t*)(pStart);
pStart += sizeof(int32_t) * numOfCols;
for (int32_t i = 0; i < numOfCols; ++i) {
colLen[i] = htonl(colLen[i]);
ASSERT(colLen[i] >= 0);
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
pColInfoData->varmeta.length = colLen[i];
pColInfoData->varmeta.allocLen = colLen[i];
memcpy(pColInfoData->varmeta.offset, pStart, sizeof(int32_t) * numOfRows);
pStart += sizeof(int32_t) * numOfRows;
pColInfoData->pData = taosMemoryMalloc(colLen[i]);
} else {
memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows));
pStart += BitmapLen(numOfRows);
}
memcpy(pColInfoData->pData, pStart, colLen[i]);
pStart += colLen[i];
}
// data from mnode // data from mnode
relocateColumnData(pRes, pColList, pBlock->pDataBlock); relocateColumnData(pRes, pColList, pBlock->pDataBlock);
taosArrayDestroy(pBlock->pDataBlock); taosArrayDestroy(pBlock->pDataBlock);
...@@ -3133,6 +3103,68 @@ static SSDataBlock* doMerge(SOperatorInfo* pOperator) { ...@@ -3133,6 +3103,68 @@ static SSDataBlock* doMerge(SOperatorInfo* pOperator) {
return (pInfo->binfo.pRes->info.rows > 0) ? pInfo->binfo.pRes : NULL; return (pInfo->binfo.pRes->info.rows > 0) ? pInfo->binfo.pRes : NULL;
} }
SSDataBlock* getSortedMergeBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo,
SSortedMergeOperatorInfo *pInfo) {
blockDataCleanup(pDataBlock);
SSDataBlock* p = tsortGetSortedDataBlock(pHandle);
if (p == NULL) {
return NULL;
}
blockDataEnsureCapacity(p, capacity);
while (1) {
STupleHandle* pTupleHandle = NULL;
if (pInfo->prefetchedTuple == NULL) {
pTupleHandle = tsortNextTuple(pHandle);
} else {
pTupleHandle = pInfo->prefetchedTuple;
pInfo->groupId = tsortGetGroupId(pTupleHandle);
pInfo->prefetchedTuple = NULL;
}
if (pTupleHandle == NULL) {
break;
}
uint64_t tupleGroupId = tsortGetGroupId(pTupleHandle);
if (!pInfo->hasGroupId) {
pInfo->groupId = tupleGroupId;
pInfo->hasGroupId = true;
appendOneRowToDataBlock(p, pTupleHandle);
} else if (pInfo->groupId == tupleGroupId) {
appendOneRowToDataBlock(p, pTupleHandle);
} else {
pInfo->prefetchedTuple = pTupleHandle;
break;
}
if (p->info.rows >= capacity) {
break;
}
}
if (p->info.rows > 0) {
int32_t numOfCols = taosArrayGetSize(pColMatchInfo);
for (int32_t i = 0; i < numOfCols; ++i) {
SColMatchInfo* pmInfo = taosArrayGet(pColMatchInfo, i);
ASSERT(pmInfo->matchType == COL_MATCH_FROM_SLOT_ID);
SColumnInfoData* pSrc = taosArrayGet(p->pDataBlock, pmInfo->srcSlotId);
SColumnInfoData* pDst = taosArrayGet(pDataBlock->pDataBlock, pmInfo->targetSlotId);
colDataAssign(pDst, pSrc, p->info.rows);
}
pDataBlock->info.rows = p->info.rows;
pDataBlock->info.capacity = p->info.rows;
pDataBlock->info.groupId = pInfo->groupId;
}
blockDataDestroy(p);
return (pDataBlock->info.rows > 0) ? pDataBlock : NULL;
}
static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) { static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) {
if (pOperator->status == OP_EXEC_DONE) { if (pOperator->status == OP_EXEC_DONE) {
return NULL; return NULL;
...@@ -3141,7 +3173,7 @@ static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) { ...@@ -3141,7 +3173,7 @@ static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SSortedMergeOperatorInfo* pInfo = pOperator->info; SSortedMergeOperatorInfo* pInfo = pOperator->info;
if (pOperator->status == OP_RES_TO_RETURN) { if (pOperator->status == OP_RES_TO_RETURN) {
return getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, NULL); return getSortedMergeBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, NULL, pInfo);
} }
int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize; int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize;
......
source/libs/executor/src/scanoperator.c
浏览文件 @ d5f6bb79
...@@ -737,8 +737,8 @@ static bool isStateWindow(SStreamBlockScanInfo* pInfo) { ...@@ -737,8 +737,8 @@ static bool isStateWindow(SStreamBlockScanInfo* pInfo) {
static bool prepareDataScan(SStreamBlockScanInfo* pInfo) { static bool prepareDataScan(SStreamBlockScanInfo* pInfo) {
SSDataBlock* pSDB = pInfo->pUpdateRes; SSDataBlock* pSDB = pInfo->pUpdateRes;
STimeWindow win = { STimeWindow win = {
.skey = INT64_MIN, .skey = INT64_MIN,
.ekey = INT64_MAX, .ekey = INT64_MAX,
}; };
bool needRead = false; bool needRead = false;
if (!isStateWindow(pInfo) && pInfo->updateResIndex < pSDB->info.rows) { if (!isStateWindow(pInfo) && pInfo->updateResIndex < pSDB->info.rows) {
...@@ -846,7 +846,7 @@ static SSDataBlock* doStreamBlockScan(SOperatorInfo* pOperator) { ...@@ -846,7 +846,7 @@ static SSDataBlock* doStreamBlockScan(SOperatorInfo* pOperator) {
size_t total = taosArrayGetSize(pInfo->pBlockLists); size_t total = taosArrayGetSize(pInfo->pBlockLists);
if (pInfo->blockType == STREAM_DATA_TYPE_SSDATA_BLOCK) { if (pInfo->blockType == STREAM_DATA_TYPE_SSDATA_BLOCK) {
if (pInfo->validBlockIndex >= total) { if (pInfo->validBlockIndex >= total) {
doClearBufferedBlocks(pInfo); /*doClearBufferedBlocks(pInfo);*/
pOperator->status = OP_EXEC_DONE; pOperator->status = OP_EXEC_DONE;
return NULL; return NULL;
} }
......
source/libs/executor/src/sortoperator.c
浏览文件 @ d5f6bb79
...@@ -42,6 +42,8 @@ SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pR ...@@ -42,6 +42,8 @@ SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pR
pInfo->pSortInfo = pSortInfo; pInfo->pSortInfo = pSortInfo;
pInfo->pColMatchInfo = pColMatchColInfo; pInfo->pColMatchInfo = pColMatchColInfo;
pInfo->hasGroupId = false;
pInfo->prefetchedTuple = NULL;
pOperator->name = "SortOperator"; pOperator->name = "SortOperator";
pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT;
pOperator->blocking = true; pOperator->blocking = true;
...@@ -81,8 +83,8 @@ void appendOneRowToDataBlock(SSDataBlock* pBlock, STupleHandle* pTupleHandle) { ...@@ -81,8 +83,8 @@ void appendOneRowToDataBlock(SSDataBlock* pBlock, STupleHandle* pTupleHandle) {
pBlock->info.rows += 1; pBlock->info.rows += 1;
} }
SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo,
SArray* pColMatchInfo) { SSortOperatorInfo* pInfo) {
blockDataCleanup(pDataBlock); blockDataCleanup(pDataBlock);
SSDataBlock* p = tsortGetSortedDataBlock(pHandle); SSDataBlock* p = tsortGetSortedDataBlock(pHandle);
...@@ -93,14 +95,33 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i ...@@ -93,14 +95,33 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i
blockDataEnsureCapacity(p, capacity); blockDataEnsureCapacity(p, capacity);
while (1) { while (1) {
STupleHandle* pTupleHandle = tsortNextTuple(pHandle); STupleHandle* pTupleHandle = NULL;
if (pInfo->prefetchedTuple == NULL) {
pTupleHandle = tsortNextTuple(pHandle);
} else {
pTupleHandle = pInfo->prefetchedTuple;
pInfo->groupId = tsortGetGroupId(pTupleHandle);
pInfo->prefetchedTuple = NULL;
}
if (pTupleHandle == NULL) { if (pTupleHandle == NULL) {
break; break;
} }
appendOneRowToDataBlock(p, pTupleHandle); uint64_t tupleGroupId = tsortGetGroupId(pTupleHandle);
if (!pInfo->hasGroupId) {
pInfo->groupId = tupleGroupId;
pInfo->hasGroupId = true;
appendOneRowToDataBlock(p, pTupleHandle);
} else if (pInfo->groupId == tupleGroupId) {
appendOneRowToDataBlock(p, pTupleHandle);
} else {
pInfo->prefetchedTuple = pTupleHandle;
break;
}
if (p->info.rows >= capacity) { if (p->info.rows >= capacity) {
return pDataBlock; break;
} }
} }
...@@ -117,6 +138,7 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i ...@@ -117,6 +138,7 @@ SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, i
pDataBlock->info.rows = p->info.rows; pDataBlock->info.rows = p->info.rows;
pDataBlock->info.capacity = p->info.rows; pDataBlock->info.capacity = p->info.rows;
pDataBlock->info.groupId = pInfo->groupId;
} }
blockDataDestroy(p); blockDataDestroy(p);
...@@ -188,8 +210,8 @@ SSDataBlock* doSort(SOperatorInfo* pOperator) { ...@@ -188,8 +210,8 @@ SSDataBlock* doSort(SOperatorInfo* pOperator) {
longjmp(pTaskInfo->env, code); longjmp(pTaskInfo->env, code);
} }
SSDataBlock* pBlock = SSDataBlock* pBlock = getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity,
getSortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity, pInfo->pColMatchInfo); pInfo->pColMatchInfo, pInfo);
if (pBlock != NULL) { if (pBlock != NULL) {
pOperator->resultInfo.totalRows += pBlock->info.rows; pOperator->resultInfo.totalRows += pBlock->info.rows;
...@@ -230,11 +252,11 @@ typedef struct SMultiwaySortMergeOperatorInfo { ...@@ -230,11 +252,11 @@ typedef struct SMultiwaySortMergeOperatorInfo {
SArray* pColMatchInfo; // for index map from table scan output SArray* pColMatchInfo; // for index map from table scan output
SSDataBlock* pInputBlock; SSDataBlock* pInputBlock;
int64_t startTs; // sort start time int64_t startTs; // sort start time
bool hasGroupId; bool hasGroupId;
uint64_t groupId; uint64_t groupId;
STupleHandle *prefetchedTuple; STupleHandle* prefetchedTuple;
} SMultiwaySortMergeOperatorInfo; } SMultiwaySortMergeOperatorInfo;
int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) { int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) {
...@@ -274,7 +296,7 @@ int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) { ...@@ -274,7 +296,7 @@ int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) {
} }
SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity,
SArray* pColMatchInfo, SMultiwaySortMergeOperatorInfo* pInfo) { SArray* pColMatchInfo, SMultiwaySortMergeOperatorInfo* pInfo) {
blockDataCleanup(pDataBlock); blockDataCleanup(pDataBlock);
SSDataBlock* p = tsortGetSortedDataBlock(pHandle); SSDataBlock* p = tsortGetSortedDataBlock(pHandle);
...@@ -285,12 +307,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData ...@@ -285,12 +307,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData
blockDataEnsureCapacity(p, capacity); blockDataEnsureCapacity(p, capacity);
while (1) { while (1) {
STupleHandle* pTupleHandle = NULL; STupleHandle* pTupleHandle = NULL;
if (pInfo->prefetchedTuple == NULL) { if (pInfo->prefetchedTuple == NULL) {
pTupleHandle = tsortNextTuple(pHandle); pTupleHandle = tsortNextTuple(pHandle);
} else { } else {
pTupleHandle = pInfo->prefetchedTuple; pTupleHandle = pInfo->prefetchedTuple;
pInfo->groupId = tsortGetGroupId(pTupleHandle);
pInfo->prefetchedTuple = NULL; pInfo->prefetchedTuple = NULL;
} }
...@@ -307,14 +329,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData ...@@ -307,14 +329,12 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData
appendOneRowToDataBlock(p, pTupleHandle); appendOneRowToDataBlock(p, pTupleHandle);
} else { } else {
pInfo->prefetchedTuple = pTupleHandle; pInfo->prefetchedTuple = pTupleHandle;
pInfo->groupId = tupleGroupId;
break; break;
} }
if (p->info.rows >= capacity) { if (p->info.rows >= capacity) {
break; break;
} }
} }
if (p->info.rows > 0) { if (p->info.rows > 0) {
...@@ -330,13 +350,13 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData ...@@ -330,13 +350,13 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData
pDataBlock->info.rows = p->info.rows; pDataBlock->info.rows = p->info.rows;
pDataBlock->info.capacity = p->info.rows; pDataBlock->info.capacity = p->info.rows;
pDataBlock->info.groupId = pInfo->groupId;
} }
blockDataDestroy(p); blockDataDestroy(p);
return (pDataBlock->info.rows > 0) ? pDataBlock : NULL; return (pDataBlock->info.rows > 0) ? pDataBlock : NULL;
} }
SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) {
if (pOperator->status == OP_EXEC_DONE) { if (pOperator->status == OP_EXEC_DONE) {
return NULL; return NULL;
...@@ -350,12 +370,8 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { ...@@ -350,12 +370,8 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) {
longjmp(pTaskInfo->env, code); longjmp(pTaskInfo->env, code);
} }
SSDataBlock* pBlock = SSDataBlock* pBlock = getMultiwaySortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes,
getMultiwaySortedBlockData(pInfo->pSortHandle, pOperator->resultInfo.capacity, pInfo->pColMatchInfo, pInfo);
pInfo->binfo.pRes,
pOperator->resultInfo.capacity,
pInfo->pColMatchInfo,
pInfo);
if (pBlock != NULL) { if (pBlock != NULL) {
pOperator->resultInfo.totalRows += pBlock->info.rows; pOperator->resultInfo.totalRows += pBlock->info.rows;
...@@ -366,7 +382,7 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) { ...@@ -366,7 +382,7 @@ SSDataBlock* doMultiwaySortMerge(SOperatorInfo* pOperator) {
} }
void destroyMultiwaySortMergeOperatorInfo(void* param, int32_t numOfOutput) { void destroyMultiwaySortMergeOperatorInfo(void* param, int32_t numOfOutput) {
SMultiwaySortMergeOperatorInfo * pInfo = (SMultiwaySortMergeOperatorInfo*)param; SMultiwaySortMergeOperatorInfo* pInfo = (SMultiwaySortMergeOperatorInfo*)param;
pInfo->binfo.pRes = blockDataDestroy(pInfo->binfo.pRes); pInfo->binfo.pRes = blockDataDestroy(pInfo->binfo.pRes);
pInfo->pInputBlock = blockDataDestroy(pInfo->pInputBlock); pInfo->pInputBlock = blockDataDestroy(pInfo->pInputBlock);
...@@ -386,9 +402,9 @@ int32_t getMultiwaySortMergeExplainExecInfo(SOperatorInfo* pOptr, void** pOptrEx ...@@ -386,9 +402,9 @@ int32_t getMultiwaySortMergeExplainExecInfo(SOperatorInfo* pOptr, void** pOptrEx
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, int32_t numStreams, SSDataBlock* pInputBlock, SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, int32_t numStreams,
SSDataBlock* pResBlock, SArray* pSortInfo, SArray* pColMatchColInfo, SSDataBlock* pInputBlock, SSDataBlock* pResBlock, SArray* pSortInfo,
SExecTaskInfo* pTaskInfo) { SArray* pColMatchColInfo, SExecTaskInfo* pTaskInfo) {
SMultiwaySortMergeOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SMultiwaySortMergeOperatorInfo)); SMultiwaySortMergeOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SMultiwaySortMergeOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
int32_t rowSize = pResBlock->info.rowSize; int32_t rowSize = pResBlock->info.rowSize;
...@@ -412,7 +428,8 @@ SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, ...@@ -412,7 +428,8 @@ SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams,
pInfo->bufPageSize = rowSize < 1024 ? 1024 : rowSize * 2; pInfo->bufPageSize = rowSize < 1024 ? 1024 : rowSize * 2;
pInfo->sortBufSize = pInfo->bufPageSize * 16; pInfo->sortBufSize = pInfo->bufPageSize * 16;
pInfo->hasGroupId = false;
pInfo->prefetchedTuple = NULL;
pOperator->pTaskInfo = pTaskInfo; pOperator->pTaskInfo = pTaskInfo;
pOperator->fpSet = pOperator->fpSet =
createOperatorFpSet(doOpenMultiwaySortMergeOperator, doMultiwaySortMerge, NULL, NULL, createOperatorFpSet(doOpenMultiwaySortMergeOperator, doMultiwaySortMerge, NULL, NULL,
......
source/libs/executor/src/timewindowoperator.c
浏览文件 @ d5f6bb79
...@@ -1227,7 +1227,10 @@ void doClearWindow(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, char* pData, int ...@@ -1227,7 +1227,10 @@ void doClearWindow(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, char* pData, int
SET_RES_WINDOW_KEY(pSup->keyBuf, pData, bytes, groupId); SET_RES_WINDOW_KEY(pSup->keyBuf, pData, bytes, groupId);
SResultRowPosition* p1 = SResultRowPosition* p1 =
(SResultRowPosition*)taosHashGet(pSup->pResultRowHashTable, pSup->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes)); (SResultRowPosition*)taosHashGet(pSup->pResultRowHashTable, pSup->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes));
ASSERT(p1); if (!p1) {
// window has been closed
return;
}
doClearWindowImpl(p1, pSup->pResultBuf, pBinfo, numOfOutput); doClearWindowImpl(p1, pSup->pResultBuf, pBinfo, numOfOutput);
} }
...@@ -2202,12 +2205,12 @@ void destroyStreamSessionAggOperatorInfo(void* param, int32_t numOfOutput) { ...@@ -2202,12 +2205,12 @@ void destroyStreamSessionAggOperatorInfo(void* param, int32_t numOfOutput) {
} }
} }
int32_t initBiasicInfo(SOptrBasicInfo* pBasicInfo, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock, int32_t initBiasicInfo(SOptrBasicInfo* pBasicInfo, SExprInfo* pExprInfo,
SDiskbasedBuf* pResultBuf) { int32_t numOfCols, SSDataBlock* pResultBlock) {
pBasicInfo->pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pBasicInfo->rowCellInfoOffset); pBasicInfo->pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pBasicInfo->rowCellInfoOffset);
pBasicInfo->pRes = pResultBlock; pBasicInfo->pRes = pResultBlock;
for (int32_t i = 0; i < numOfCols; ++i) { for (int32_t i = 0; i < numOfCols; ++i) {
pBasicInfo->pCtx[i].pBuf = pResultBuf; pBasicInfo->pCtx[i].pBuf = NULL;
} }
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
...@@ -2237,16 +2240,15 @@ SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SEx ...@@ -2237,16 +2240,15 @@ SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SEx
initResultSizeInfo(pOperator, 4096); initResultSizeInfo(pOperator, 4096);
code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo"); code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _error; goto _error;
} }
pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols);
code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock, pInfo->streamAggSup.pResultBuf); code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo");
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _error; goto _error;
} }
pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols);
pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx)); pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx));
if (pInfo->pDummyCtx == NULL) { if (pInfo->pDummyCtx == NULL) {
...@@ -3101,6 +3103,7 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys ...@@ -3101,6 +3103,7 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys
SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc);
int32_t tsSlotId = ((SColumnNode*)pStateNode->window.pTspk)->slotId; int32_t tsSlotId = ((SColumnNode*)pStateNode->window.pTspk)->slotId;
SColumnNode* pColNode = (SColumnNode*)((STargetNode*)pStateNode->pStateKey)->pExpr; SColumnNode* pColNode = (SColumnNode*)((STargetNode*)pStateNode->pStateKey)->pExpr;
int32_t code = TSDB_CODE_OUT_OF_MEMORY;
SStreamStateAggOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SStreamStateAggOperatorInfo)); SStreamStateAggOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SStreamStateAggOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
...@@ -3121,17 +3124,18 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys ...@@ -3121,17 +3124,18 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys
.winMap = NULL, .winMap = NULL,
}; };
initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window);
int32_t code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo");
code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _error; goto _error;
} }
code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock, pInfo->streamAggSup.pResultBuf); pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols);
code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo");
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _error; goto _error;
} }
pInfo->streamAggSup.resultRowSize = getResultRowSize(pInfo->binfo.pCtx, numOfCols);
pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx)); pInfo->pDummyCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfCols, sizeof(SqlFunctionCtx));
if (pInfo->pDummyCtx == NULL) { if (pInfo->pDummyCtx == NULL) {
goto _error; goto _error;
......
source/libs/executor/src/tsort.c
浏览文件 @ d5f6bb79
...@@ -144,7 +144,9 @@ static int32_t doAddNewExternalMemSource(SDiskbasedBuf *pBuf, SArray* pAllSource ...@@ -144,7 +144,9 @@ static int32_t doAddNewExternalMemSource(SDiskbasedBuf *pBuf, SArray* pAllSource
(*sourceId) += 1; (*sourceId) += 1;
int32_t rowSize = blockDataGetSerialRowSize(pSource->src.pBlock); int32_t rowSize = blockDataGetSerialRowSize(pSource->src.pBlock);
int32_t numOfRows = (getBufPageSize(pBuf) - blockDataGetSerialMetaSize(pBlock))/rowSize; // The value of numOfRows must be greater than 0, which is guaranteed by the previous memory allocation
// The value of numOfRows must be greater than 0, which is guaranteed by the previous memory allocation
int32_t numOfRows = (getBufPageSize(pBuf) - blockDataGetSerialMetaSize(pBlock->info.numOfCols))/rowSize;
ASSERT(numOfRows > 0); ASSERT(numOfRows > 0);
return blockDataEnsureCapacity(pSource->src.pBlock, numOfRows); return blockDataEnsureCapacity(pSource->src.pBlock, numOfRows);
} }
...@@ -359,6 +361,11 @@ int32_t msortComparFn(const void *pLeft, const void *pRight, void *param) { ...@@ -359,6 +361,11 @@ int32_t msortComparFn(const void *pLeft, const void *pRight, void *param) {
SSDataBlock* pLeftBlock = pLeftSource->src.pBlock; SSDataBlock* pLeftBlock = pLeftSource->src.pBlock;
SSDataBlock* pRightBlock = pRightSource->src.pBlock; SSDataBlock* pRightBlock = pRightSource->src.pBlock;
// first sort by block groupId
if (pLeftBlock->info.groupId != pRightBlock->info.groupId) {
return pLeftBlock->info.groupId < pRightBlock->info.groupId ? -1 : 1;
}
for(int32_t i = 0; i < pInfo->size; ++i) { for(int32_t i = 0; i < pInfo->size; ++i) {
SBlockOrderInfo* pOrder = TARRAY_GET_ELEM(pInfo, i); SBlockOrderInfo* pOrder = TARRAY_GET_ELEM(pInfo, i);
SColumnInfoData* pLeftColInfoData = TARRAY_GET_ELEM(pLeftBlock->pDataBlock, pOrder->slotId); SColumnInfoData* pLeftColInfoData = TARRAY_GET_ELEM(pLeftBlock->pDataBlock, pOrder->slotId);
...@@ -521,16 +528,24 @@ static int32_t createInitialSources(SSortHandle* pHandle) { ...@@ -521,16 +528,24 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
if (pHandle->type == SORT_SINGLESOURCE_SORT) { if (pHandle->type == SORT_SINGLESOURCE_SORT) {
SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0); SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0);
taosArrayClear(pHandle->pOrderedSource); taosArrayClear(pHandle->pOrderedSource);
bool hasGroupId = false;
SSDataBlock* prefetchedDataBlock = NULL;
while (1) { while (1) {
SSDataBlock* pBlock = pHandle->fetchfp(source->param); SSDataBlock* pBlock = NULL;
if (prefetchedDataBlock == NULL) {
pBlock = pHandle->fetchfp(source->param);
} else {
pBlock = prefetchedDataBlock;
prefetchedDataBlock = NULL;
}
if (pBlock == NULL) { if (pBlock == NULL) {
break; break;
} }
if (pHandle->pDataBlock == NULL) { if (!hasGroupId) {
pHandle->pDataBlock = createOneDataBlock(pBlock, false);
// calculate the buffer pages according to the total available buffers. // calculate the buffer pages according to the total available buffers.
int32_t rowSize = blockDataGetRowSize(pBlock); int32_t rowSize = blockDataGetRowSize(pBlock);
if (rowSize * 4 > 4096) { if (rowSize * 4 > 4096) {
...@@ -542,29 +557,36 @@ static int32_t createInitialSources(SSortHandle* pHandle) { ...@@ -542,29 +557,36 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
// todo!! // todo!!
pHandle->numOfPages = 1024; pHandle->numOfPages = 1024;
sortBufSize = pHandle->numOfPages * pHandle->pageSize; sortBufSize = pHandle->numOfPages * pHandle->pageSize;
}
// perform the scalar function calculation before apply the sort hasGroupId = true;
if (pHandle->beforeFp != NULL) { pHandle->pDataBlock = createOneDataBlock(pBlock, false);
pHandle->beforeFp(pBlock, pHandle->param);
} }
// todo relocate the columns if (pHandle->pDataBlock->info.groupId == pBlock->info.groupId) {
int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock); // perform the scalar function calculation before apply the sort
if (code != 0) { if (pHandle->beforeFp != NULL) {
return code; pHandle->beforeFp(pBlock, pHandle->param);
} }
// todo relocate the columns
int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock);
if (code != 0) {
return code;
}
size_t size = blockDataGetSize(pHandle->pDataBlock); size_t size = blockDataGetSize(pHandle->pDataBlock);
if (size > sortBufSize) { if (size > sortBufSize) {
// Perform the in-memory sort and then flush data in the buffer into disk. // Perform the in-memory sort and then flush data in the buffer into disk.
int64_t p = taosGetTimestampUs(); int64_t p = taosGetTimestampUs();
blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo); blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo);
int64_t el = taosGetTimestampUs() - p; int64_t el = taosGetTimestampUs() - p;
pHandle->sortElapsed += el; pHandle->sortElapsed += el;
doAddToBuf(pHandle->pDataBlock, pHandle); doAddToBuf(pHandle->pDataBlock, pHandle);
}
} else {
prefetchedDataBlock = pBlock;
pHandle->pDataBlock = createOneDataBlock(pBlock, false);
} }
} }
......
source/libs/function/inc/builtinsimpl.h
浏览文件 @ d5f6bb79
...@@ -78,13 +78,14 @@ bool percentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultI ...@@ -78,13 +78,14 @@ bool percentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultI
int32_t percentileFunction(SqlFunctionCtx *pCtx); int32_t percentileFunction(SqlFunctionCtx *pCtx);
int32_t percentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t percentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t getApercentileMaxSize();
bool getApercentileFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getApercentileFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
bool apercentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); bool apercentileFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo);
int32_t apercentileFunction(SqlFunctionCtx *pCtx); int32_t apercentileFunction(SqlFunctionCtx *pCtx);
int32_t apercentileFunctionMerge(SqlFunctionCtx* pCtx); int32_t apercentileFunctionMerge(SqlFunctionCtx* pCtx);
int32_t apercentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t apercentileFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t apercentileCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx);
int32_t getApercentileMaxSize();
bool getDiffFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getDiffFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
bool diffFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResInfo); bool diffFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResInfo);
...@@ -102,13 +103,13 @@ int32_t topFunction(SqlFunctionCtx *pCtx); ...@@ -102,13 +103,13 @@ int32_t topFunction(SqlFunctionCtx *pCtx);
int32_t bottomFunction(SqlFunctionCtx *pCtx); int32_t bottomFunction(SqlFunctionCtx *pCtx);
int32_t topBotFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t topBotFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t getSpreadInfoSize();
bool getSpreadFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getSpreadFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
bool spreadFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); bool spreadFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo);
int32_t spreadFunction(SqlFunctionCtx* pCtx); int32_t spreadFunction(SqlFunctionCtx* pCtx);
int32_t spreadFunctionMerge(SqlFunctionCtx* pCtx); int32_t spreadFunctionMerge(SqlFunctionCtx* pCtx);
int32_t spreadFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t spreadFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t spreadPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t spreadPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t getSpreadInfoSize();
bool getElapsedFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getElapsedFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
bool elapsedFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); bool elapsedFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo);
...@@ -118,7 +119,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); ...@@ -118,7 +119,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
bool getHistogramFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getHistogramFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
bool histogramFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); bool histogramFunctionSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo);
int32_t histogramFunction(SqlFunctionCtx* pCtx); int32_t histogramFunction(SqlFunctionCtx* pCtx);
int32_t histogramFunctionMerge(SqlFunctionCtx* pCtx);
int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t histogramPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t getHistogramInfoSize();
bool getHLLFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getHLLFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
int32_t hllFunction(SqlFunctionCtx* pCtx); int32_t hllFunction(SqlFunctionCtx* pCtx);
......
source/libs/function/src/builtins.c
浏览文件 @ d5f6bb79
...@@ -503,6 +503,58 @@ static int32_t translateHistogram(SFunctionNode* pFunc, char* pErrBuf, int32_t l ...@@ -503,6 +503,58 @@ static int32_t translateHistogram(SFunctionNode* pFunc, char* pErrBuf, int32_t l
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateHistogramImpl(SFunctionNode* pFunc, char* pErrBuf, int32_t len, bool isPartial) {
int32_t numOfParams = LIST_LENGTH(pFunc->pParameterList);
if (isPartial) {
if (4 != numOfParams) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t colType = ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type;
if (!IS_NUMERIC_TYPE(colType)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
// param1 ~ param3
for (int32_t i = 1; i < numOfParams; ++i) {
SNode* pParamNode = nodesListGetNode(pFunc->pParameterList, i);
if (QUERY_NODE_VALUE != nodeType(pParamNode)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
SValueNode* pValue = (SValueNode*)pParamNode;
pValue->notReserved = true;
}
if (((SExprNode*)nodesListGetNode(pFunc->pParameterList, 1))->resType.type != TSDB_DATA_TYPE_BINARY ||
((SExprNode*)nodesListGetNode(pFunc->pParameterList, 2))->resType.type != TSDB_DATA_TYPE_BINARY ||
((SExprNode*)nodesListGetNode(pFunc->pParameterList, 3))->resType.type != TSDB_DATA_TYPE_BIGINT) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = getHistogramInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY};
} else {
if (1 != numOfParams) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
if (((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type != TSDB_DATA_TYPE_BINARY) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = 512, .type = TSDB_DATA_TYPE_BINARY};
}
return TSDB_CODE_SUCCESS;
}
static int32_t translateHistogramPartial(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
return translateHistogramImpl(pFunc, pErrBuf, len, true);
}
static int32_t translateHistogramMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
return translateHistogramImpl(pFunc, pErrBuf, len, false);
}
static int32_t translateHLL(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateHLL(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
...@@ -1241,7 +1293,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { ...@@ -1241,7 +1293,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.getEnvFunc = getPercentileFuncEnv, .getEnvFunc = getPercentileFuncEnv,
.initFunc = percentileFunctionSetup, .initFunc = percentileFunctionSetup,
.processFunc = percentileFunction, .processFunc = percentileFunction,
.finalizeFunc = percentileFinalize .finalizeFunc = percentileFinalize,
.invertFunc = NULL,
.combineFunc = NULL,
}, },
{ {
.name = "apercentile", .name = "apercentile",
...@@ -1252,6 +1306,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { ...@@ -1252,6 +1306,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.initFunc = apercentileFunctionSetup, .initFunc = apercentileFunctionSetup,
.processFunc = apercentileFunction, .processFunc = apercentileFunction,
.finalizeFunc = apercentileFinalize, .finalizeFunc = apercentileFinalize,
.combineFunc = apercentileCombine,
.pPartialFunc = "_apercentile_partial", .pPartialFunc = "_apercentile_partial",
.pMergeFunc = "_apercentile_merge" .pMergeFunc = "_apercentile_merge"
}, },
...@@ -1391,6 +1446,28 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { ...@@ -1391,6 +1446,28 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.getEnvFunc = getHistogramFuncEnv, .getEnvFunc = getHistogramFuncEnv,
.initFunc = histogramFunctionSetup, .initFunc = histogramFunctionSetup,
.processFunc = histogramFunction, .processFunc = histogramFunction,
.finalizeFunc = histogramFinalize,
.pPartialFunc = "_histogram_partial",
.pMergeFunc = "_histogram_merge"
},
{
.name = "_histogram_partial",
.type = FUNCTION_TYPE_HISTOGRAM_PARTIAL,
.classification = FUNC_MGT_AGG_FUNC,
.translateFunc = translateHistogramPartial,
.getEnvFunc = getHistogramFuncEnv,
.initFunc = histogramFunctionSetup,
.processFunc = histogramFunction,
.finalizeFunc = histogramPartialFinalize
},
{
.name = "_histogram_merge",
.type = FUNCTION_TYPE_HISTOGRAM_MERGE,
.classification = FUNC_MGT_AGG_FUNC,
.translateFunc = translateHistogramMerge,
.getEnvFunc = getHistogramFuncEnv,
.initFunc = functionSetup,
.processFunc = histogramFunctionMerge,
.finalizeFunc = histogramFinalize .finalizeFunc = histogramFinalize
}, },
{ {
......
source/libs/function/src/builtinsimpl.c
浏览文件 @ d5f6bb79
...@@ -2187,9 +2187,7 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { ...@@ -2187,9 +2187,7 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx);
SAPercentileInfo* pInfo = (SAPercentileInfo*)GET_ROWCELL_INTERBUF(pResInfo); SAPercentileInfo* pInfo = (SAPercentileInfo*)GET_ROWCELL_INTERBUF(pResInfo);
int32_t bytesHist = (int32_t)(sizeof(SAPercentileInfo) + sizeof(SHistogramInfo) + sizeof(SHistBin) * (MAX_HISTOGRAM_BIN + 1)); int32_t resultBytes = getApercentileMaxSize();
int32_t bytesDigest = (int32_t)(sizeof(SAPercentileInfo) + TDIGEST_SIZE(COMPRESSION));
int32_t resultBytes = TMAX(bytesHist, bytesDigest);
char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char));
if (pInfo->algo == APERCT_ALGO_TDIGEST) { if (pInfo->algo == APERCT_ALGO_TDIGEST) {
...@@ -2217,6 +2215,26 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { ...@@ -2217,6 +2215,26 @@ int32_t apercentilePartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
return pResInfo->numOfRes; return pResInfo->numOfRes;
} }
int32_t apercentileCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) {
SResultRowEntryInfo* pDResInfo = GET_RES_INFO(pDestCtx);
SAPercentileInfo* pDBuf = GET_ROWCELL_INTERBUF(pDResInfo);
int32_t type = pDestCtx->input.pData[0]->info.type;
SResultRowEntryInfo* pSResInfo = GET_RES_INFO(pSourceCtx);
SAPercentileInfo* pSBuf = GET_ROWCELL_INTERBUF(pSResInfo);
ASSERT(pDBuf->algo == pSBuf->algo);
if (pDBuf->algo == APERCT_ALGO_TDIGEST) {
tdigestMerge(pDBuf->pTDigest, pSBuf->pTDigest);
} else {
SHistogramInfo* pTmp = tHistogramMerge(pDBuf->pHisto, pSBuf->pHisto, MAX_HISTOGRAM_BIN);
memcpy(pDBuf->pHisto, pTmp, sizeof(SHistogramInfo) + sizeof(SHistBin) * (MAX_HISTOGRAM_BIN + 1));
pDBuf->pHisto->elems = (SHistBin*) ((char *)pDBuf->pHisto + sizeof(SHistogramInfo));
tHistogramDestroy(&pTmp);
}
pDResInfo->numOfRes = TMAX(pDResInfo->numOfRes, pSResInfo->numOfRes);
return TSDB_CODE_SUCCESS;
}
bool getFirstLastFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { bool getFirstLastFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) {
SColumnNode* pNode = nodesListGetNode(pFunc->pParameterList, 0); SColumnNode* pNode = nodesListGetNode(pFunc->pParameterList, 0);
pEnv->calcMemSize = pNode->node.resType.bytes + sizeof(int64_t); pEnv->calcMemSize = pNode->node.resType.bytes + sizeof(int64_t);
...@@ -3118,6 +3136,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { ...@@ -3118,6 +3136,10 @@ int32_t elapsedFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
return functionFinalize(pCtx, pBlock); return functionFinalize(pCtx, pBlock);
} }
int32_t getHistogramInfoSize() {
return (int32_t)sizeof(SHistoFuncInfo) + HISTOGRAM_MAX_BINS_NUM * sizeof(SHistoFuncBin);
}
bool getHistogramFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) { bool getHistogramFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) {
pEnv->calcMemSize = sizeof(SHistoFuncInfo) + HISTOGRAM_MAX_BINS_NUM * sizeof(SHistoFuncBin); pEnv->calcMemSize = sizeof(SHistoFuncInfo) + HISTOGRAM_MAX_BINS_NUM * sizeof(SHistoFuncBin);
return true; return true;
...@@ -3328,6 +3350,30 @@ int32_t histogramFunction(SqlFunctionCtx *pCtx) { ...@@ -3328,6 +3350,30 @@ int32_t histogramFunction(SqlFunctionCtx *pCtx) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t histogramFunctionMerge(SqlFunctionCtx *pCtx) {
SInputColumnInfoData* pInput = &pCtx->input;
SColumnInfoData* pCol = pInput->pData[0];
ASSERT(pCol->info.type == TSDB_DATA_TYPE_BINARY);
SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
int32_t start = pInput->startRowIndex;
char* data = colDataGetData(pCol, start);
SHistoFuncInfo* pInputInfo = (SHistoFuncInfo *)varDataVal(data);
pInfo->normalized = pInputInfo->normalized;
pInfo->numOfBins = pInputInfo->numOfBins;
pInfo->totalCount += pInputInfo->totalCount;
for (int32_t k = 0; k < pInfo->numOfBins; ++k) {
pInfo->bins[k].lower = pInputInfo->bins[k].lower;
pInfo->bins[k].upper = pInputInfo->bins[k].upper;
pInfo->bins[k].count += pInputInfo->bins[k].count;
}
SET_VAL(GET_RES_INFO(pCtx), pInfo->numOfBins, pInfo->numOfBins);
return TSDB_CODE_SUCCESS;
}
int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx);
SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
...@@ -3364,6 +3410,24 @@ int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { ...@@ -3364,6 +3410,24 @@ int32_t histogramFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
return pResInfo->numOfRes; return pResInfo->numOfRes;
} }
int32_t histogramPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx);
SHistoFuncInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
int32_t resultBytes = getHistogramInfoSize();
char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char));
memcpy(varDataVal(res), pInfo, resultBytes);
varDataSetLen(res, resultBytes);
int32_t slotId = pCtx->pExpr->base.resSchema.slotId;
SColumnInfoData* pCol = taosArrayGet(pBlock->pDataBlock, slotId);
colDataAppend(pCol, pBlock->info.rows, res, false);
taosMemoryFree(res);
return 1;
}
bool getHLLFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) { bool getHLLFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) {
pEnv->calcMemSize = sizeof(SHLLInfo); pEnv->calcMemSize = sizeof(SHLLInfo);
return true; return true;
......
source/libs/index/inc/indexCache.h
浏览文件 @ d5f6bb79
...@@ -70,7 +70,7 @@ void indexCacheBroadcast(void* cache); ...@@ -70,7 +70,7 @@ void indexCacheBroadcast(void* cache);
void indexCacheWait(void* cache); void indexCacheWait(void* cache);
Iterate* indexCacheIteratorCreate(IndexCache* cache); Iterate* indexCacheIteratorCreate(IndexCache* cache);
void indexCacheIteratorDestroy(Iterate* iiter); void idxCacheIteratorDestroy(Iterate* iiter);
int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid); int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid);
...@@ -82,7 +82,7 @@ void indexCacheUnRef(IndexCache* cache); ...@@ -82,7 +82,7 @@ void indexCacheUnRef(IndexCache* cache);
void indexCacheDebug(IndexCache* cache); void indexCacheDebug(IndexCache* cache);
void indexCacheDestroyImm(IndexCache* cache); void idxCacheDestroyImm(IndexCache* cache);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif
......
source/libs/index/inc/indexComm.h
浏览文件 @ d5f6bb79
...@@ -26,9 +26,9 @@ extern "C" { ...@@ -26,9 +26,9 @@ extern "C" {
extern char JSON_COLUMN[]; extern char JSON_COLUMN[];
extern char JSON_VALUE_DELIM; extern char JSON_VALUE_DELIM;
char* indexPackJsonData(SIndexTerm* itm); char* idxPackJsonData(SIndexTerm* itm);
char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip); char* idxPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip);
char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip); char* idxPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip);
typedef enum { MATCH, CONTINUE, BREAK } TExeCond; typedef enum { MATCH, CONTINUE, BREAK } TExeCond;
...@@ -40,13 +40,14 @@ TExeCond tDoCompare(__compar_fn_t func, int8_t cmpType, void* a, void* b); ...@@ -40,13 +40,14 @@ TExeCond tDoCompare(__compar_fn_t func, int8_t cmpType, void* a, void* b);
_cache_range_compare indexGetCompare(RangeType ty); _cache_range_compare indexGetCompare(RangeType ty);
int32_t indexConvertData(void* src, int8_t type, void** dst); int32_t idxConvertData(void* src, int8_t type, void** dst);
int32_t indexConvertDataToStr(void* src, int8_t type, void** dst); int32_t idxConvertDataToStr(void* src, int8_t type, void** dst);
int32_t indexGetDataByteLen(int8_t type); int32_t idxGetDataByteLen(int8_t type);
char* indexInt2str(int64_t val, char* dst, int radix); char* idxInt2str(int64_t val, char* dst, int radix);
int idxUidCompare(const void* a, const void* b);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif
......
source/libs/index/inc/indexFst.h
浏览文件 @ d5f6bb79
...@@ -29,9 +29,9 @@ extern "C" { ...@@ -29,9 +29,9 @@ extern "C" {
#define OUTPUT_PREFIX(a, b) ((a) > (b) ? (b) : (a) #define OUTPUT_PREFIX(a, b) ((a) > (b) ? (b) : (a)
typedef struct Fst Fst; typedef struct Fst Fst;
typedef struct FstNode FstNode; typedef struct FstNode FstNode;
typedef struct StreamWithState StreamWithState; typedef struct FStmSt FStmSt;
typedef enum { Included, Excluded, Unbounded } FstBound; typedef enum { Included, Excluded, Unbounded } FstBound;
...@@ -40,12 +40,12 @@ typedef struct FstBoundWithData { ...@@ -40,12 +40,12 @@ typedef struct FstBoundWithData {
FstBound type; FstBound type;
} FstBoundWithData; } FstBoundWithData;
typedef struct FstStreamBuilder { typedef struct FStmBuilder {
Fst* fst; Fst* fst;
AutomationCtx* aut; FAutoCtx* aut;
FstBoundWithData* min; FstBoundWithData* min;
FstBoundWithData* max; FstBoundWithData* max;
} FstStreamBuilder, FstStreamWithStateBuilder; } FStmBuilder, FStmStBuilder;
typedef struct FstRange { typedef struct FstRange {
uint64_t start; uint64_t start;
...@@ -267,17 +267,17 @@ typedef struct Fst { ...@@ -267,17 +267,17 @@ typedef struct Fst {
Fst* fstCreate(FstSlice* data); Fst* fstCreate(FstSlice* data);
void fstDestroy(Fst* fst); void fstDestroy(Fst* fst);
bool fstGet(Fst* fst, FstSlice* b, Output* out); bool fstGet(Fst* fst, FstSlice* b, Output* out);
FstNode* fstGetNode(Fst* fst, CompiledAddr); FstNode* fstGetNode(Fst* fst, CompiledAddr);
FstNode* fstGetRoot(Fst* fst); FstNode* fstGetRoot(Fst* fst);
FstType fstGetType(Fst* fst); FstType fstGetType(Fst* fst);
CompiledAddr fstGetRootAddr(Fst* fst); CompiledAddr fstGetRootAddr(Fst* fst);
Output fstEmptyFinalOutput(Fst* fst, bool* null); Output fstEmptyFinalOutput(Fst* fst, bool* null);
FstStreamBuilder* fstSearch(Fst* fst, AutomationCtx* ctx); FStmBuilder* fstSearch(Fst* fst, FAutoCtx* ctx);
FstStreamWithStateBuilder* fstSearchWithState(Fst* fst, AutomationCtx* ctx); FStmStBuilder* fstSearchWithState(Fst* fst, FAutoCtx* ctx);
// into stream to expand later // into stream to expand later
StreamWithState* streamBuilderIntoStream(FstStreamBuilder* sb); FStmSt* stmBuilderIntoStm(FStmBuilder* sb);
bool fstVerify(Fst* fst); bool fstVerify(Fst* fst);
...@@ -293,41 +293,40 @@ typedef struct StreamState { ...@@ -293,41 +293,40 @@ typedef struct StreamState {
void streamStateDestroy(void* s); void streamStateDestroy(void* s);
typedef struct StreamWithState { typedef struct FStmSt {
Fst* fst; Fst* fst;
AutomationCtx* aut; FAutoCtx* aut;
SArray* inp; SArray* inp;
FstOutput emptyOutput; FstOutput emptyOutput;
SArray* stack; // <StreamState> SArray* stack; // <StreamState>
FstBoundWithData* endAt; FstBoundWithData* endAt;
} StreamWithState; } FStmSt;
typedef struct StreamWithStateResult { typedef struct FStmStRslt {
FstSlice data; FstSlice data;
FstOutput out; FstOutput out;
void* state; void* state;
} StreamWithStateResult; } FStmStRslt;
StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* state); FStmStRslt* swsResultCreate(FstSlice* data, FstOutput fOut, void* state);
void swsResultDestroy(StreamWithStateResult* result); void swsResultDestroy(FStmStRslt* result);
typedef void* (*StreamCallback)(void*); typedef void* (*StreamCallback)(void*);
StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstBoundWithData* min, FStmSt* stmStCreate(Fst* fst, FAutoCtx* automation, FstBoundWithData* min, FstBoundWithData* max);
FstBoundWithData* max);
void streamWithStateDestroy(StreamWithState* sws); void stmStDestroy(FStmSt* sws);
bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min); bool stmStSeekMin(FStmSt* sws, FstBoundWithData* min);
StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallback callback); FStmStRslt* stmStNextWith(FStmSt* sws, StreamCallback callback);
FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut); FStmBuilder* stmBuilderCreate(Fst* fst, FAutoCtx* aut);
void fstStreamBuilderDestroy(FstStreamBuilder* b); void stmBuilderDestroy(FStmBuilder* b);
// set up bound range // set up bound range
// refator later: to simple code by marco // refator later: to simple code by marco
void fstStreamBuilderSetRange(FstStreamBuilder* b, FstSlice* val, RangeType type); void stmBuilderSetRange(FStmBuilder* b, FstSlice* val, RangeType type);
#ifdef __cplusplus #ifdef __cplusplus
} }
......
source/libs/index/inc/indexFstAutomation.h
浏览文件 @ d5f6bb79
...@@ -22,24 +22,24 @@ extern "C" { ...@@ -22,24 +22,24 @@ extern "C" {
#include "indexFstUtil.h" #include "indexFstUtil.h"
#include "indexInt.h" #include "indexInt.h"
typedef struct AutomationCtx AutomationCtx; typedef struct FAutoCtx FAutoCtx;
typedef enum AutomationType { AUTOMATION_ALWAYS, AUTOMATION_PREFIX, AUTMMATION_MATCH } AutomationType; typedef enum AutomationType { AUTOMATION_ALWAYS, AUTOMATION_PREFIX, AUTMMATION_MATCH } AutomationType;
typedef struct StartWith { typedef struct StartWith {
AutomationCtx* autoSelf; FAutoCtx* autoSelf;
} StartWith; } StartWith;
typedef struct Complement { typedef struct Complement {
AutomationCtx* autoSelf; FAutoCtx* autoSelf;
} Complement; } Complement;
// automation // automation
typedef struct AutomationCtx { typedef struct FAutoCtx {
AutomationType type; AutomationType type;
void* stdata; void* stdata;
char* data; char* data;
} AutomationCtx; } FAutoCtx;
typedef enum ValueType { FST_INT, FST_CHAR, FST_ARRAY } ValueType; typedef enum ValueType { FST_INT, FST_CHAR, FST_ARRAY } ValueType;
typedef enum StartWithStateKind { Done, Running } StartWithStateKind; typedef enum StartWithStateKind { Done, Running } StartWithStateKind;
...@@ -60,16 +60,16 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv); ...@@ -60,16 +60,16 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv);
void startWithStateValueDestroy(void* sv); void startWithStateValueDestroy(void* sv);
typedef struct AutomationFunc { typedef struct AutomationFunc {
void* (*start)(AutomationCtx* ctx); void* (*start)(FAutoCtx* ctx);
bool (*isMatch)(AutomationCtx* ctx, void*); bool (*isMatch)(FAutoCtx* ctx, void*);
bool (*canMatch)(AutomationCtx* ctx, void* data); bool (*canMatch)(FAutoCtx* ctx, void* data);
bool (*willAlwaysMatch)(AutomationCtx* ctx, void* state); bool (*willAlwaysMatch)(FAutoCtx* ctx, void* state);
void* (*accept)(AutomationCtx* ctx, void* state, uint8_t byte); void* (*accept)(FAutoCtx* ctx, void* state, uint8_t byte);
void* (*acceptEof)(AutomationCtx* ct, void* state); void* (*acceptEof)(FAutoCtx* ct, void* state);
} AutomationFunc; } AutomationFunc;
AutomationCtx* automCtxCreate(void* data, AutomationType atype); FAutoCtx* automCtxCreate(void* data, AutomationType atype);
void automCtxDestroy(AutomationCtx* ctx); void automCtxDestroy(FAutoCtx* ctx);
extern AutomationFunc automFuncs[]; extern AutomationFunc automFuncs[];
#ifdef __cplusplus #ifdef __cplusplus
......
source/libs/index/inc/indexInt.h
浏览文件 @ d5f6bb79
...@@ -131,7 +131,7 @@ typedef struct TFileCacheKey { ...@@ -131,7 +131,7 @@ typedef struct TFileCacheKey {
char* colName; char* colName;
int32_t nColName; int32_t nColName;
} ICacheKey; } ICacheKey;
int indexFlushCacheToTFile(SIndex* sIdx, void*, bool quit); int idxFlushCacheToTFile(SIndex* sIdx, void*, bool quit);
int64_t indexAddRef(void* p); int64_t indexAddRef(void* p);
int32_t indexRemoveRef(int64_t ref); int32_t indexRemoveRef(int64_t ref);
......
source/libs/index/src/index.c
浏览文件 @ d5f6bb79
...@@ -77,15 +77,15 @@ typedef struct SIdxColInfo { ...@@ -77,15 +77,15 @@ typedef struct SIdxColInfo {
static TdThreadOnce isInit = PTHREAD_ONCE_INIT; static TdThreadOnce isInit = PTHREAD_ONCE_INIT;
// static void indexInit(); // static void indexInit();
static int indexTermSearch(SIndex* sIdx, SIndexTermQuery* term, SArray** result); static int idxTermSearch(SIndex* sIdx, SIndexTermQuery* term, SArray** result);
static void indexInterResultsDestroy(SArray* results); static void idxInterRsltDestroy(SArray* results);
static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out); static int idxMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out);
static int indexGenTFile(SIndex* index, IndexCache* cache, SArray* batch); static int idxGenTFile(SIndex* index, IndexCache* cache, SArray* batch);
// merge cache and tfile by opera type // merge cache and tfile by opera type
static void indexMergeCacheAndTFile(SArray* result, IterateValue* icache, IterateValue* iTfv, SIdxTRslt* helper); static void idxMergeCacheAndTFile(SArray* result, IterateValue* icache, IterateValue* iTfv, SIdxTRslt* helper);
// static int32_t indexSerialTermKey(SIndexTerm* itm, char* buf); // static int32_t indexSerialTermKey(SIndexTerm* itm, char* buf);
// int32_t indexSerialKey(ICacheKey* key, char* buf); // int32_t indexSerialKey(ICacheKey* key, char* buf);
...@@ -221,11 +221,11 @@ int indexSearch(SIndex* index, SIndexMultiTermQuery* multiQuerys, SArray* result ...@@ -221,11 +221,11 @@ int indexSearch(SIndex* index, SIndexMultiTermQuery* multiQuerys, SArray* result
for (size_t i = 0; i < nQuery; i++) { for (size_t i = 0; i < nQuery; i++) {
SIndexTermQuery* qterm = taosArrayGet(multiQuerys->query, i); SIndexTermQuery* qterm = taosArrayGet(multiQuerys->query, i);
SArray* trslt = NULL; SArray* trslt = NULL;
indexTermSearch(index, qterm, &trslt); idxTermSearch(index, qterm, &trslt);
taosArrayPush(iRslts, (void*)&trslt); taosArrayPush(iRslts, (void*)&trslt);
} }
indexMergeFinalResults(iRslts, opera, result); idxMergeFinalResults(iRslts, opera, result);
indexInterResultsDestroy(iRslts); idxInterRsltDestroy(iRslts);
return 0; return 0;
} }
...@@ -289,7 +289,7 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy ...@@ -289,7 +289,7 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy
tm->nColName = nColName; tm->nColName = nColName;
char* buf = NULL; char* buf = NULL;
int32_t len = indexConvertDataToStr((void*)colVal, INDEX_TYPE_GET_TYPE(colType), (void**)&buf); int32_t len = idxConvertDataToStr((void*)colVal, INDEX_TYPE_GET_TYPE(colType), (void**)&buf);
assert(len != -1); assert(len != -1);
tm->colVal = buf; tm->colVal = buf;
...@@ -319,7 +319,7 @@ void indexMultiTermDestroy(SIndexMultiTerm* terms) { ...@@ -319,7 +319,7 @@ void indexMultiTermDestroy(SIndexMultiTerm* terms) {
taosArrayDestroy(terms); taosArrayDestroy(terms);
} }
static int indexTermSearch(SIndex* sIdx, SIndexTermQuery* query, SArray** result) { static int idxTermSearch(SIndex* sIdx, SIndexTermQuery* query, SArray** result) {
SIndexTerm* term = query->term; SIndexTerm* term = query->term;
const char* colName = term->colName; const char* colName = term->colName;
int32_t nColName = term->nColName; int32_t nColName = term->nColName;
...@@ -374,7 +374,7 @@ END: ...@@ -374,7 +374,7 @@ END:
idxTRsltDestroy(tr); idxTRsltDestroy(tr);
return -1; return -1;
} }
static void indexInterResultsDestroy(SArray* results) { static void idxInterRsltDestroy(SArray* results) {
if (results == NULL) { if (results == NULL) {
return; return;
} }
...@@ -387,7 +387,7 @@ static void indexInterResultsDestroy(SArray* results) { ...@@ -387,7 +387,7 @@ static void indexInterResultsDestroy(SArray* results) {
taosArrayDestroy(results); taosArrayDestroy(results);
} }
static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out) { static int idxMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* out) {
// refactor, merge interResults into fResults by oType // refactor, merge interResults into fResults by oType
for (int i = 0; i < taosArrayGetSize(in); i--) { for (int i = 0; i < taosArrayGetSize(in); i--) {
SArray* t = taosArrayGetP(in, i); SArray* t = taosArrayGetP(in, i);
...@@ -407,7 +407,7 @@ static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray* ...@@ -407,7 +407,7 @@ static int indexMergeFinalResults(SArray* in, EIndexOperatorType oType, SArray*
return 0; return 0;
} }
static void indexMayMergeTempToFinalResult(SArray* result, TFileValue* tfv, SIdxTRslt* tr) { static void idxMayMergeTempToFinalRslt(SArray* result, TFileValue* tfv, SIdxTRslt* tr) {
int32_t sz = taosArrayGetSize(result); int32_t sz = taosArrayGetSize(result);
if (sz > 0) { if (sz > 0) {
TFileValue* lv = taosArrayGetP(result, sz - 1); TFileValue* lv = taosArrayGetP(result, sz - 1);
...@@ -427,11 +427,11 @@ static void indexMayMergeTempToFinalResult(SArray* result, TFileValue* tfv, SIdx ...@@ -427,11 +427,11 @@ static void indexMayMergeTempToFinalResult(SArray* result, TFileValue* tfv, SIdx
taosArrayPush(result, &tfv); taosArrayPush(result, &tfv);
} }
} }
static void indexMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateValue* tv, SIdxTRslt* tr) { static void idxMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateValue* tv, SIdxTRslt* tr) {
char* colVal = (cv != NULL) ? cv->colVal : tv->colVal; char* colVal = (cv != NULL) ? cv->colVal : tv->colVal;
TFileValue* tfv = tfileValueCreate(colVal); TFileValue* tfv = tfileValueCreate(colVal);
indexMayMergeTempToFinalResult(result, tfv, tr); idxMayMergeTempToFinalRslt(result, tfv, tr);
if (cv != NULL) { if (cv != NULL) {
uint64_t id = *(uint64_t*)taosArrayGet(cv->val, 0); uint64_t id = *(uint64_t*)taosArrayGet(cv->val, 0);
...@@ -446,7 +446,7 @@ static void indexMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateVal ...@@ -446,7 +446,7 @@ static void indexMergeCacheAndTFile(SArray* result, IterateValue* cv, IterateVal
taosArrayAddAll(tr->total, tv->val); taosArrayAddAll(tr->total, tv->val);
} }
} }
static void indexDestroyFinalResult(SArray* result) { static void idxDestroyFinalRslt(SArray* result) {
int32_t sz = result ? taosArrayGetSize(result) : 0; int32_t sz = result ? taosArrayGetSize(result) : 0;
for (size_t i = 0; i < sz; i++) { for (size_t i = 0; i < sz; i++) {
TFileValue* tv = taosArrayGetP(result, i); TFileValue* tv = taosArrayGetP(result, i);
...@@ -455,7 +455,7 @@ static void indexDestroyFinalResult(SArray* result) { ...@@ -455,7 +455,7 @@ static void indexDestroyFinalResult(SArray* result) {
taosArrayDestroy(result); taosArrayDestroy(result);
} }
int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { int idxFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) {
if (sIdx == NULL) { if (sIdx == NULL) {
return -1; return -1;
} }
...@@ -475,7 +475,7 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { ...@@ -475,7 +475,7 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) {
Iterate* cacheIter = indexCacheIteratorCreate(pCache); Iterate* cacheIter = indexCacheIteratorCreate(pCache);
if (cacheIter == NULL) { if (cacheIter == NULL) {
indexError("%p immtable is empty, ignore merge opera", pCache); indexError("%p immtable is empty, ignore merge opera", pCache);
indexCacheDestroyImm(pCache); idxCacheDestroyImm(pCache);
tfileReaderUnRef(pReader); tfileReaderUnRef(pReader);
atomic_store_32(&pCache->merging, 0); atomic_store_32(&pCache->merging, 0);
if (quit) { if (quit) {
...@@ -509,26 +509,26 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) { ...@@ -509,26 +509,26 @@ int indexFlushCacheToTFile(SIndex* sIdx, void* cache, bool quit) {
comp = 1; comp = 1;
} }
if (comp == 0) { if (comp == 0) {
indexMergeCacheAndTFile(result, cv, tv, tr); idxMergeCacheAndTFile(result, cv, tv, tr);
cn = cacheIter->next(cacheIter); cn = cacheIter->next(cacheIter);
tn = tfileIter->next(tfileIter); tn = tfileIter->next(tfileIter);
} else if (comp < 0) { } else if (comp < 0) {
indexMergeCacheAndTFile(result, cv, NULL, tr); idxMergeCacheAndTFile(result, cv, NULL, tr);
cn = cacheIter->next(cacheIter); cn = cacheIter->next(cacheIter);
} else { } else {
indexMergeCacheAndTFile(result, NULL, tv, tr); idxMergeCacheAndTFile(result, NULL, tv, tr);
tn = tfileIter->next(tfileIter); tn = tfileIter->next(tfileIter);
} }
} }
indexMayMergeTempToFinalResult(result, NULL, tr); idxMayMergeTempToFinalRslt(result, NULL, tr);
idxTRsltDestroy(tr); idxTRsltDestroy(tr);
int ret = indexGenTFile(sIdx, pCache, result); int ret = idxGenTFile(sIdx, pCache, result);
indexDestroyFinalResult(result); idxDestroyFinalRslt(result);
indexCacheDestroyImm(pCache); idxCacheDestroyImm(pCache);
indexCacheIteratorDestroy(cacheIter); idxCacheIteratorDestroy(cacheIter);
tfileIteratorDestroy(tfileIter); tfileIteratorDestroy(tfileIter);
tfileReaderUnRef(pReader); tfileReaderUnRef(pReader);
...@@ -578,7 +578,7 @@ static int64_t indexGetAvaialbleVer(SIndex* sIdx, IndexCache* cache) { ...@@ -578,7 +578,7 @@ static int64_t indexGetAvaialbleVer(SIndex* sIdx, IndexCache* cache) {
tfileReaderUnRef(rd); tfileReaderUnRef(rd);
return ver; return ver;
} }
static int indexGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) { static int idxGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) {
int64_t version = indexGetAvaialbleVer(sIdx, cache); int64_t version = indexGetAvaialbleVer(sIdx, cache);
indexInfo("file name version: %" PRId64 "", version); indexInfo("file name version: %" PRId64 "", version);
uint8_t colType = cache->type; uint8_t colType = cache->type;
...@@ -625,7 +625,7 @@ int32_t indexSerialCacheKey(ICacheKey* key, char* buf) { ...@@ -625,7 +625,7 @@ int32_t indexSerialCacheKey(ICacheKey* key, char* buf) {
char* p = buf; char* p = buf;
char tbuf[65] = {0}; char tbuf[65] = {0};
indexInt2str((int64_t)key->suid, tbuf, 0); idxInt2str((int64_t)key->suid, tbuf, 0);
SERIALIZE_STR_VAR_TO_BUF(buf, tbuf, strlen(tbuf)); SERIALIZE_STR_VAR_TO_BUF(buf, tbuf, strlen(tbuf));
SERIALIZE_VAR_TO_BUF(buf, '_', char); SERIALIZE_VAR_TO_BUF(buf, '_', char);
......
source/libs/index/src/indexCache.c
浏览文件 @ d5f6bb79
...@@ -26,15 +26,15 @@ ...@@ -26,15 +26,15 @@
#define MEM_SIGNAL_QUIT MEM_THRESHOLD * 20 #define MEM_SIGNAL_QUIT MEM_THRESHOLD * 20
#define MEM_ESTIMATE_RADIO 1.5 #define MEM_ESTIMATE_RADIO 1.5
static void indexMemRef(MemTable* tbl); static void idxMemRef(MemTable* tbl);
static void indexMemUnRef(MemTable* tbl); static void idxMemUnRef(MemTable* tbl);
static void indexCacheTermDestroy(CacheTerm* ct); static void idxCacheTermDestroy(CacheTerm* ct);
static int32_t indexCacheTermCompare(const void* l, const void* r); static int32_t idxCacheTermCompare(const void* l, const void* r);
static int32_t indexCacheJsonTermCompare(const void* l, const void* r); static int32_t idxCacheJsonTermCompare(const void* l, const void* r);
static char* indexCacheTermGet(const void* pData); static char* idxCacheTermGet(const void* pData);
static MemTable* indexInternalCacheCreate(int8_t type); static MemTable* idxInternalCacheCreate(int8_t type);
static int32_t cacheSearchTerm(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s); static int32_t cacheSearchTerm(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s);
static int32_t cacheSearchPrefix(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s); static int32_t cacheSearchPrefix(void* cache, SIndexTerm* ct, SIdxTRslt* tr, STermValueType* s);
...@@ -69,7 +69,7 @@ static int32_t (*cacheSearch[][QUERY_MAX])(void* cache, SIndexTerm* ct, SIdxTRsl ...@@ -69,7 +69,7 @@ static int32_t (*cacheSearch[][QUERY_MAX])(void* cache, SIndexTerm* ct, SIdxTRsl
cacheSearchRange_JSON}}; cacheSearchRange_JSON}};
static void doMergeWork(SSchedMsg* msg); static void doMergeWork(SSchedMsg* msg);
static bool indexCacheIteratorNext(Iterate* itera); static bool idxCacheIteratorNext(Iterate* itera);
static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STermValueType* s) { static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STermValueType* s) {
if (cache == NULL) { if (cache == NULL) {
...@@ -82,7 +82,7 @@ static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STe ...@@ -82,7 +82,7 @@ static int32_t cacheSearchTerm(void* cache, SIndexTerm* term, SIdxTRslt* tr, STe
pCt->colVal = term->colVal; pCt->colVal = term->colVal;
pCt->version = atomic_load_64(&pCache->version); pCt->version = atomic_load_64(&pCache->version);
char* key = indexCacheTermGet(pCt); char* key = idxCacheTermGet(pCt);
SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC);
while (tSkipListIterNext(iter)) { while (tSkipListIterNext(iter)) {
...@@ -134,7 +134,7 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt* ...@@ -134,7 +134,7 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt*
pCt->colType = term->colType; pCt->colType = term->colType;
pCt->version = atomic_load_64(&pCache->version); pCt->version = atomic_load_64(&pCache->version);
char* key = indexCacheTermGet(pCt); char* key = idxCacheTermGet(pCt);
SSkipListIterator* iter = tSkipListCreateIter(mem->mem); SSkipListIterator* iter = tSkipListCreateIter(mem->mem);
while (tSkipListIterNext(iter)) { while (tSkipListIterNext(iter)) {
...@@ -188,10 +188,10 @@ static int32_t cacheSearchTerm_JSON(void* cache, SIndexTerm* term, SIdxTRslt* tr ...@@ -188,10 +188,10 @@ static int32_t cacheSearchTerm_JSON(void* cache, SIndexTerm* term, SIdxTRslt* tr
char* exBuf = NULL; char* exBuf = NULL;
if (INDEX_TYPE_CONTAIN_EXTERN_TYPE(term->colType, TSDB_DATA_TYPE_JSON)) { if (INDEX_TYPE_CONTAIN_EXTERN_TYPE(term->colType, TSDB_DATA_TYPE_JSON)) {
exBuf = indexPackJsonData(term); exBuf = idxPackJsonData(term);
pCt->colVal = exBuf; pCt->colVal = exBuf;
} }
char* key = indexCacheTermGet(pCt); char* key = idxCacheTermGet(pCt);
SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC);
while (tSkipListIterNext(iter)) { while (tSkipListIterNext(iter)) {
...@@ -269,20 +269,19 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR ...@@ -269,20 +269,19 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR
int8_t dType = INDEX_TYPE_GET_TYPE(term->colType); int8_t dType = INDEX_TYPE_GET_TYPE(term->colType);
int skip = 0; int skip = 0;
char* exBuf = NULL; char* exBuf = NULL;
if (type == CONTAINS) { if (type == CONTAINS) {
SIndexTerm tm = {.suid = term->suid, SIndexTerm tm = {.suid = term->suid,
.operType = term->operType, .operType = term->operType,
.colType = term->colType, .colType = term->colType,
.colName = term->colVal, .colName = term->colVal,
.nColName = term->nColVal}; .nColName = term->nColVal};
exBuf = indexPackJsonDataPrefixNoType(&tm, &skip); exBuf = idxPackJsonDataPrefixNoType(&tm, &skip);
pCt->colVal = exBuf; pCt->colVal = exBuf;
} else { } else {
exBuf = indexPackJsonDataPrefix(term, &skip); exBuf = idxPackJsonDataPrefix(term, &skip);
pCt->colVal = exBuf; pCt->colVal = exBuf;
} }
char* key = indexCacheTermGet(pCt); char* key = idxCacheTermGet(pCt);
SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC); SSkipListIterator* iter = tSkipListCreateIterFromVal(mem->mem, key, TSDB_DATA_TYPE_BINARY, TSDB_ORDER_ASC);
while (tSkipListIterNext(iter)) { while (tSkipListIterNext(iter)) {
...@@ -341,7 +340,7 @@ IndexCache* indexCacheCreate(SIndex* idx, uint64_t suid, const char* colName, in ...@@ -341,7 +340,7 @@ IndexCache* indexCacheCreate(SIndex* idx, uint64_t suid, const char* colName, in
return NULL; return NULL;
}; };
cache->mem = indexInternalCacheCreate(type); cache->mem = idxInternalCacheCreate(type);
cache->mem->pCache = cache; cache->mem->pCache = cache;
cache->colName = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? tstrdup(JSON_COLUMN) : tstrdup(colName); cache->colName = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? tstrdup(JSON_COLUMN) : tstrdup(colName);
cache->type = type; cache->type = type;
...@@ -364,7 +363,7 @@ void indexCacheDebug(IndexCache* cache) { ...@@ -364,7 +363,7 @@ void indexCacheDebug(IndexCache* cache) {
taosThreadMutexLock(&cache->mtx); taosThreadMutexLock(&cache->mtx);
tbl = cache->mem; tbl = cache->mem;
indexMemRef(tbl); idxMemRef(tbl);
taosThreadMutexUnlock(&cache->mtx); taosThreadMutexUnlock(&cache->mtx);
{ {
...@@ -380,13 +379,13 @@ void indexCacheDebug(IndexCache* cache) { ...@@ -380,13 +379,13 @@ void indexCacheDebug(IndexCache* cache) {
} }
tSkipListDestroyIter(iter); tSkipListDestroyIter(iter);
indexMemUnRef(tbl); idxMemUnRef(tbl);
} }
{ {
taosThreadMutexLock(&cache->mtx); taosThreadMutexLock(&cache->mtx);
tbl = cache->imm; tbl = cache->imm;
indexMemRef(tbl); idxMemRef(tbl);
taosThreadMutexUnlock(&cache->mtx); taosThreadMutexUnlock(&cache->mtx);
if (tbl != NULL) { if (tbl != NULL) {
SSkipList* slt = tbl->mem; SSkipList* slt = tbl->mem;
...@@ -402,7 +401,7 @@ void indexCacheDebug(IndexCache* cache) { ...@@ -402,7 +401,7 @@ void indexCacheDebug(IndexCache* cache) {
tSkipListDestroyIter(iter); tSkipListDestroyIter(iter);
} }
indexMemUnRef(tbl); idxMemUnRef(tbl);
} }
} }
...@@ -427,7 +426,7 @@ void indexCacheWait(void* cache) { ...@@ -427,7 +426,7 @@ void indexCacheWait(void* cache) {
IndexCache* pCache = cache; IndexCache* pCache = cache;
taosThreadCondWait(&pCache->finished, &pCache->mtx); taosThreadCondWait(&pCache->finished, &pCache->mtx);
} }
void indexCacheDestroyImm(IndexCache* cache) { void idxCacheDestroyImm(IndexCache* cache) {
if (cache == NULL) { if (cache == NULL) {
return; return;
} }
...@@ -440,8 +439,8 @@ void indexCacheDestroyImm(IndexCache* cache) { ...@@ -440,8 +439,8 @@ void indexCacheDestroyImm(IndexCache* cache) {
taosThreadMutexUnlock(&cache->mtx); taosThreadMutexUnlock(&cache->mtx);
indexMemUnRef(tbl); idxMemUnRef(tbl);
indexMemUnRef(tbl); idxMemUnRef(tbl);
} }
void indexCacheDestroy(void* cache) { void indexCacheDestroy(void* cache) {
IndexCache* pCache = cache; IndexCache* pCache = cache;
...@@ -449,8 +448,8 @@ void indexCacheDestroy(void* cache) { ...@@ -449,8 +448,8 @@ void indexCacheDestroy(void* cache) {
return; return;
} }
indexMemUnRef(pCache->mem); idxMemUnRef(pCache->mem);
indexMemUnRef(pCache->imm); idxMemUnRef(pCache->imm);
taosMemoryFree(pCache->colName); taosMemoryFree(pCache->colName);
taosThreadMutexDestroy(&pCache->mtx); taosThreadMutexDestroy(&pCache->mtx);
...@@ -471,20 +470,20 @@ Iterate* indexCacheIteratorCreate(IndexCache* cache) { ...@@ -471,20 +470,20 @@ Iterate* indexCacheIteratorCreate(IndexCache* cache) {
} }
taosThreadMutexLock(&cache->mtx); taosThreadMutexLock(&cache->mtx);
indexMemRef(cache->imm); idxMemRef(cache->imm);
MemTable* tbl = cache->imm; MemTable* tbl = cache->imm;
iiter->val.val = taosArrayInit(1, sizeof(uint64_t)); iiter->val.val = taosArrayInit(1, sizeof(uint64_t));
iiter->val.colVal = NULL; iiter->val.colVal = NULL;
iiter->iter = tbl != NULL ? tSkipListCreateIter(tbl->mem) : NULL; iiter->iter = tbl != NULL ? tSkipListCreateIter(tbl->mem) : NULL;
iiter->next = indexCacheIteratorNext; iiter->next = idxCacheIteratorNext;
iiter->getValue = indexCacheIteratorGetValue; iiter->getValue = indexCacheIteratorGetValue;
taosThreadMutexUnlock(&cache->mtx); taosThreadMutexUnlock(&cache->mtx);
return iiter; return iiter;
} }
void indexCacheIteratorDestroy(Iterate* iter) { void idxCacheIteratorDestroy(Iterate* iter) {
if (iter == NULL) { if (iter == NULL) {
return; return;
} }
...@@ -518,7 +517,7 @@ static void indexCacheMakeRoomForWrite(IndexCache* cache) { ...@@ -518,7 +517,7 @@ static void indexCacheMakeRoomForWrite(IndexCache* cache) {
indexCacheRef(cache); indexCacheRef(cache);
cache->imm = cache->mem; cache->imm = cache->mem;
cache->mem = indexInternalCacheCreate(cache->type); cache->mem = idxInternalCacheCreate(cache->type);
cache->mem->pCache = cache; cache->mem->pCache = cache;
cache->occupiedMem = 0; cache->occupiedMem = 0;
if (quit == false) { if (quit == false) {
...@@ -546,7 +545,7 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) { ...@@ -546,7 +545,7 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) {
// set up key // set up key
ct->colType = term->colType; ct->colType = term->colType;
if (hasJson) { if (hasJson) {
ct->colVal = indexPackJsonData(term); ct->colVal = idxPackJsonData(term);
} else { } else {
ct->colVal = (char*)taosMemoryCalloc(1, sizeof(char) * (term->nColVal + 1)); ct->colVal = (char*)taosMemoryCalloc(1, sizeof(char) * (term->nColVal + 1));
memcpy(ct->colVal, term->colVal, term->nColVal); memcpy(ct->colVal, term->colVal, term->nColVal);
...@@ -562,9 +561,9 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) { ...@@ -562,9 +561,9 @@ int indexCachePut(void* cache, SIndexTerm* term, uint64_t uid) {
pCache->occupiedMem += estimate; pCache->occupiedMem += estimate;
indexCacheMakeRoomForWrite(pCache); indexCacheMakeRoomForWrite(pCache);
MemTable* tbl = pCache->mem; MemTable* tbl = pCache->mem;
indexMemRef(tbl); idxMemRef(tbl);
tSkipListPut(tbl->mem, (char*)ct); tSkipListPut(tbl->mem, (char*)ct);
indexMemUnRef(tbl); idxMemUnRef(tbl);
taosThreadMutexUnlock(&pCache->mtx); taosThreadMutexUnlock(&pCache->mtx);
...@@ -615,8 +614,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe ...@@ -615,8 +614,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe
taosThreadMutexLock(&pCache->mtx); taosThreadMutexLock(&pCache->mtx);
mem = pCache->mem; mem = pCache->mem;
imm = pCache->imm; imm = pCache->imm;
indexMemRef(mem); idxMemRef(mem);
indexMemRef(imm); idxMemRef(imm);
taosThreadMutexUnlock(&pCache->mtx); taosThreadMutexUnlock(&pCache->mtx);
int ret = (mem && mem->mem) ? indexQueryMem(mem, query, result, s) : 0; int ret = (mem && mem->mem) ? indexQueryMem(mem, query, result, s) : 0;
...@@ -625,8 +624,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe ...@@ -625,8 +624,8 @@ int indexCacheSearch(void* cache, SIndexTermQuery* query, SIdxTRslt* result, STe
ret = (imm && imm->mem) ? indexQueryMem(imm, query, result, s) : 0; ret = (imm && imm->mem) ? indexQueryMem(imm, query, result, s) : 0;
} }
indexMemUnRef(mem); idxMemUnRef(mem);
indexMemUnRef(imm); idxMemUnRef(imm);
indexInfo("cache search, time cost %" PRIu64 "us", taosGetTimestampUs() - st); indexInfo("cache search, time cost %" PRIu64 "us", taosGetTimestampUs() - st);
return ret; return ret;
...@@ -649,14 +648,14 @@ void indexCacheUnRef(IndexCache* cache) { ...@@ -649,14 +648,14 @@ void indexCacheUnRef(IndexCache* cache) {
} }
} }
void indexMemRef(MemTable* tbl) { void idxMemRef(MemTable* tbl) {
if (tbl == NULL) { if (tbl == NULL) {
return; return;
} }
int ref = T_REF_INC(tbl); int ref = T_REF_INC(tbl);
UNUSED(ref); UNUSED(ref);
} }
void indexMemUnRef(MemTable* tbl) { void idxMemUnRef(MemTable* tbl) {
if (tbl == NULL) { if (tbl == NULL) {
return; return;
} }
...@@ -668,18 +667,18 @@ void indexMemUnRef(MemTable* tbl) { ...@@ -668,18 +667,18 @@ void indexMemUnRef(MemTable* tbl) {
} }
} }
static void indexCacheTermDestroy(CacheTerm* ct) { static void idxCacheTermDestroy(CacheTerm* ct) {
if (ct == NULL) { if (ct == NULL) {
return; return;
} }
taosMemoryFree(ct->colVal); taosMemoryFree(ct->colVal);
taosMemoryFree(ct); taosMemoryFree(ct);
} }
static char* indexCacheTermGet(const void* pData) { static char* idxCacheTermGet(const void* pData) {
CacheTerm* p = (CacheTerm*)pData; CacheTerm* p = (CacheTerm*)pData;
return (char*)p; return (char*)p;
} }
static int32_t indexCacheTermCompare(const void* l, const void* r) { static int32_t idxCacheTermCompare(const void* l, const void* r) {
CacheTerm* lt = (CacheTerm*)l; CacheTerm* lt = (CacheTerm*)l;
CacheTerm* rt = (CacheTerm*)r; CacheTerm* rt = (CacheTerm*)r;
// compare colVal // compare colVal
...@@ -700,7 +699,7 @@ static int indexFindCh(char* a, char c) { ...@@ -700,7 +699,7 @@ static int indexFindCh(char* a, char c) {
} }
return p - a; return p - a;
} }
static int indexCacheJsonTermCompareImpl(char* a, char* b) { static int idxCacheJsonTermCompareImpl(char* a, char* b) {
// int alen = indexFindCh(a, '&'); // int alen = indexFindCh(a, '&');
// int blen = indexFindCh(b, '&'); // int blen = indexFindCh(b, '&');
...@@ -720,7 +719,7 @@ static int indexCacheJsonTermCompareImpl(char* a, char* b) { ...@@ -720,7 +719,7 @@ static int indexCacheJsonTermCompareImpl(char* a, char* b) {
//} //}
return 0; return 0;
} }
static int32_t indexCacheJsonTermCompare(const void* l, const void* r) { static int32_t idxCacheJsonTermCompare(const void* l, const void* r) {
CacheTerm* lt = (CacheTerm*)l; CacheTerm* lt = (CacheTerm*)l;
CacheTerm* rt = (CacheTerm*)r; CacheTerm* rt = (CacheTerm*)r;
// compare colVal // compare colVal
...@@ -730,16 +729,15 @@ static int32_t indexCacheJsonTermCompare(const void* l, const void* r) { ...@@ -730,16 +729,15 @@ static int32_t indexCacheJsonTermCompare(const void* l, const void* r) {
} }
return cmp; return cmp;
} }
static MemTable* indexInternalCacheCreate(int8_t type) { static MemTable* idxInternalCacheCreate(int8_t type) {
int ttype = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? TSDB_DATA_TYPE_BINARY : TSDB_DATA_TYPE_BINARY; int ttype = INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? TSDB_DATA_TYPE_BINARY : TSDB_DATA_TYPE_BINARY;
int32_t (*cmpFn)(const void* l, const void* r) = int32_t (*cmpFn)(const void* l, const void* r) =
INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? indexCacheJsonTermCompare : indexCacheTermCompare; INDEX_TYPE_CONTAIN_EXTERN_TYPE(type, TSDB_DATA_TYPE_JSON) ? idxCacheJsonTermCompare : idxCacheTermCompare;
MemTable* tbl = taosMemoryCalloc(1, sizeof(MemTable)); MemTable* tbl = taosMemoryCalloc(1, sizeof(MemTable));
indexMemRef(tbl); idxMemRef(tbl);
if (ttype == TSDB_DATA_TYPE_BINARY || ttype == TSDB_DATA_TYPE_NCHAR) { if (ttype == TSDB_DATA_TYPE_BINARY || ttype == TSDB_DATA_TYPE_NCHAR) {
tbl->mem = tbl->mem = tSkipListCreate(MAX_SKIP_LIST_LEVEL, ttype, MAX_INDEX_KEY_LEN, cmpFn, SL_ALLOW_DUP_KEY, idxCacheTermGet);
tSkipListCreate(MAX_SKIP_LIST_LEVEL, ttype, MAX_INDEX_KEY_LEN, cmpFn, SL_ALLOW_DUP_KEY, indexCacheTermGet);
} }
return tbl; return tbl;
} }
...@@ -750,9 +748,9 @@ static void doMergeWork(SSchedMsg* msg) { ...@@ -750,9 +748,9 @@ static void doMergeWork(SSchedMsg* msg) {
int quit = msg->thandle ? true : false; int quit = msg->thandle ? true : false;
taosMemoryFree(msg->thandle); taosMemoryFree(msg->thandle);
indexFlushCacheToTFile(sidx, pCache, quit); idxFlushCacheToTFile(sidx, pCache, quit);
} }
static bool indexCacheIteratorNext(Iterate* itera) { static bool idxCacheIteratorNext(Iterate* itera) {
SSkipListIterator* iter = itera->iter; SSkipListIterator* iter = itera->iter;
if (iter == NULL) { if (iter == NULL) {
return false; return false;
......
source/libs/index/src/indexComm.c
浏览文件 @ d5f6bb79
...@@ -48,7 +48,7 @@ ...@@ -48,7 +48,7 @@
char JSON_COLUMN[] = "JSON"; char JSON_COLUMN[] = "JSON";
char JSON_VALUE_DELIM = '&'; char JSON_VALUE_DELIM = '&';
char* indexInt2str(int64_t val, char* dst, int radix) { char* idxInt2str(int64_t val, char* dst, int radix) {
char buffer[65] = {0}; char buffer[65] = {0};
char* p; char* p;
int64_t new_val; int64_t new_val;
...@@ -207,7 +207,7 @@ static TExeCond (*rangeCompare[])(void* a, void* b, int8_t type) = { ...@@ -207,7 +207,7 @@ static TExeCond (*rangeCompare[])(void* a, void* b, int8_t type) = {
_cache_range_compare indexGetCompare(RangeType ty) { return rangeCompare[ty]; } _cache_range_compare indexGetCompare(RangeType ty) { return rangeCompare[ty]; }
char* indexPackJsonData(SIndexTerm* itm) { char* idxPackJsonData(SIndexTerm* itm) {
/* /*
* |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----colname---->|<-----dataType---->|<--------colVal---------->|
* |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->|
...@@ -235,7 +235,7 @@ char* indexPackJsonData(SIndexTerm* itm) { ...@@ -235,7 +235,7 @@ char* indexPackJsonData(SIndexTerm* itm) {
return buf; return buf;
} }
char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) { char* idxPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) {
/* /*
* |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----colname---->|<-----dataType---->|<--------colVal---------->|
* |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->|
...@@ -262,7 +262,7 @@ char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) { ...@@ -262,7 +262,7 @@ char* indexPackJsonDataPrefix(SIndexTerm* itm, int32_t* skip) {
return buf; return buf;
} }
char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) { char* idxPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) {
/* /*
* |<-----colname---->|<-----dataType---->|<--------colVal---------->| * |<-----colname---->|<-----dataType---->|<--------colVal---------->|
* |<-----string----->|<-----uint8_t----->|<----depend on dataType-->| * |<-----string----->|<-----uint8_t----->|<----depend on dataType-->|
...@@ -283,7 +283,12 @@ char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) { ...@@ -283,7 +283,12 @@ char* indexPackJsonDataPrefixNoType(SIndexTerm* itm, int32_t* skip) {
return buf; return buf;
} }
int32_t indexConvertData(void* src, int8_t type, void** dst) { int idxUidCompare(const void* a, const void* b) {
uint64_t l = *(uint64_t*)a;
uint64_t r = *(uint64_t*)b;
return l - r;
}
int32_t idxConvertData(void* src, int8_t type, void** dst) {
int tlen = -1; int tlen = -1;
switch (type) { switch (type) {
case TSDB_DATA_TYPE_TIMESTAMP: case TSDB_DATA_TYPE_TIMESTAMP:
...@@ -368,44 +373,44 @@ int32_t indexConvertData(void* src, int8_t type, void** dst) { ...@@ -368,44 +373,44 @@ int32_t indexConvertData(void* src, int8_t type, void** dst) {
// indexMayFillNumbericData(*dst, tlen); // indexMayFillNumbericData(*dst, tlen);
return tlen; return tlen;
} }
int32_t indexConvertDataToStr(void* src, int8_t type, void** dst) { int32_t idxConvertDataToStr(void* src, int8_t type, void** dst) {
int tlen = tDataTypes[type].bytes; int tlen = tDataTypes[type].bytes;
int32_t bufSize = 64; int32_t bufSize = 64;
switch (type) { switch (type) {
case TSDB_DATA_TYPE_TIMESTAMP: case TSDB_DATA_TYPE_TIMESTAMP:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(int64_t*)src, *dst, -1); idxInt2str(*(int64_t*)src, *dst, -1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_BOOL: case TSDB_DATA_TYPE_BOOL:
case TSDB_DATA_TYPE_UTINYINT: case TSDB_DATA_TYPE_UTINYINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(uint8_t*)src, *dst, 1); idxInt2str(*(uint8_t*)src, *dst, 1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_TINYINT: case TSDB_DATA_TYPE_TINYINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(int8_t*)src, *dst, 1); idxInt2str(*(int8_t*)src, *dst, 1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_SMALLINT: case TSDB_DATA_TYPE_SMALLINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(int16_t*)src, *dst, -1); idxInt2str(*(int16_t*)src, *dst, -1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_USMALLINT: case TSDB_DATA_TYPE_USMALLINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(uint16_t*)src, *dst, -1); idxInt2str(*(uint16_t*)src, *dst, -1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_INT: case TSDB_DATA_TYPE_INT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(int32_t*)src, *dst, -1); idxInt2str(*(int32_t*)src, *dst, -1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_UINT: case TSDB_DATA_TYPE_UINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(uint32_t*)src, *dst, 1); idxInt2str(*(uint32_t*)src, *dst, 1);
tlen = strlen(*dst); tlen = strlen(*dst);
break; break;
case TSDB_DATA_TYPE_BIGINT: case TSDB_DATA_TYPE_BIGINT:
...@@ -415,7 +420,7 @@ int32_t indexConvertDataToStr(void* src, int8_t type, void** dst) { ...@@ -415,7 +420,7 @@ int32_t indexConvertDataToStr(void* src, int8_t type, void** dst) {
break; break;
case TSDB_DATA_TYPE_UBIGINT: case TSDB_DATA_TYPE_UBIGINT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
indexInt2str(*(uint64_t*)src, *dst, 1); idxInt2str(*(uint64_t*)src, *dst, 1);
tlen = strlen(*dst); tlen = strlen(*dst);
case TSDB_DATA_TYPE_FLOAT: case TSDB_DATA_TYPE_FLOAT:
*dst = taosMemoryCalloc(1, bufSize + 1); *dst = taosMemoryCalloc(1, bufSize + 1);
......
source/libs/index/src/indexFilter.c
浏览文件 @ d5f6bb79
...@@ -355,7 +355,7 @@ static int32_t sifDoIndex(SIFParam *left, SIFParam *right, int8_t operType, SIFP ...@@ -355,7 +355,7 @@ static int32_t sifDoIndex(SIFParam *left, SIFParam *right, int8_t operType, SIFP
SIndexMultiTermQuery *mtm = indexMultiTermQueryCreate(MUST); SIndexMultiTermQuery *mtm = indexMultiTermQueryCreate(MUST);
indexMultiTermQueryAdd(mtm, tm, qtype); indexMultiTermQueryAdd(mtm, tm, qtype);
ret = tIndexJsonSearch(arg->ivtIdx, mtm, output->result); ret = indexJsonSearch(arg->ivtIdx, mtm, output->result);
} else { } else {
bool reverse; bool reverse;
Filter filterFunc = sifGetFilterFunc(qtype, &reverse); Filter filterFunc = sifGetFilterFunc(qtype, &reverse);
...@@ -547,6 +547,8 @@ static int32_t sifExecLogic(SLogicConditionNode *node, SIFCtx *ctx, SIFParam *ou ...@@ -547,6 +547,8 @@ static int32_t sifExecLogic(SLogicConditionNode *node, SIFCtx *ctx, SIFParam *ou
} else if (node->condType == LOGIC_COND_TYPE_NOT) { } else if (node->condType == LOGIC_COND_TYPE_NOT) {
// taosArrayAddAll(output->result, params[m].result); // taosArrayAddAll(output->result, params[m].result);
} }
taosArraySort(output->result, idxUidCompare);
taosArrayRemoveDuplicate(output->result, idxUidCompare, NULL);
} }
} else { } else {
for (int32_t m = 0; m < node->pParameterList->length; m++) { for (int32_t m = 0; m < node->pParameterList->length; m++) {
......
source/libs/index/src/indexFst.c
浏览文件 @ d5f6bb79
...@@ -1087,19 +1087,19 @@ bool fstGet(Fst* fst, FstSlice* b, Output* out) { ...@@ -1087,19 +1087,19 @@ bool fstGet(Fst* fst, FstSlice* b, Output* out) {
*out = tOut; *out = tOut;
return true; return true;
} }
FstStreamBuilder* fstSearch(Fst* fst, AutomationCtx* ctx) { FStmBuilder* fstSearch(Fst* fst, FAutoCtx* ctx) {
// refactor later // refactor later
return fstStreamBuilderCreate(fst, ctx); return stmBuilderCreate(fst, ctx);
} }
StreamWithState* streamBuilderIntoStream(FstStreamBuilder* sb) { FStmSt* stmBuilderIntoStm(FStmBuilder* sb) {
if (sb == NULL) { if (sb == NULL) {
return NULL; return NULL;
} }
return streamWithStateCreate(sb->fst, sb->aut, sb->min, sb->max); return stmStCreate(sb->fst, sb->aut, sb->min, sb->max);
} }
FstStreamWithStateBuilder* fstSearchWithState(Fst* fst, AutomationCtx* ctx) { FStmStBuilder* fstSearchWithState(Fst* fst, FAutoCtx* ctx) {
// refactor later // refactor later
return fstStreamBuilderCreate(fst, ctx); return stmBuilderCreate(fst, ctx);
} }
FstNode* fstGetRoot(Fst* fst) { FstNode* fstGetRoot(Fst* fst) {
...@@ -1176,9 +1176,8 @@ bool fstBoundWithDataIsIncluded(FstBoundWithData* bound) { return bound->type == ...@@ -1176,9 +1176,8 @@ bool fstBoundWithDataIsIncluded(FstBoundWithData* bound) { return bound->type ==
void fstBoundDestroy(FstBoundWithData* bound) { taosMemoryFree(bound); } void fstBoundDestroy(FstBoundWithData* bound) { taosMemoryFree(bound); }
StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstBoundWithData* min, FStmSt* stmStCreate(Fst* fst, FAutoCtx* automation, FstBoundWithData* min, FstBoundWithData* max) {
FstBoundWithData* max) { FStmSt* sws = taosMemoryCalloc(1, sizeof(FStmSt));
StreamWithState* sws = taosMemoryCalloc(1, sizeof(StreamWithState));
if (sws == NULL) { if (sws == NULL) {
return NULL; return NULL;
} }
...@@ -1192,11 +1191,11 @@ StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstB ...@@ -1192,11 +1191,11 @@ StreamWithState* streamWithStateCreate(Fst* fst, AutomationCtx* automation, FstB
sws->stack = (SArray*)taosArrayInit(256, sizeof(StreamState)); sws->stack = (SArray*)taosArrayInit(256, sizeof(StreamState));
sws->endAt = max; sws->endAt = max;
streamWithStateSeekMin(sws, min); stmStSeekMin(sws, min);
return sws; return sws;
} }
void streamWithStateDestroy(StreamWithState* sws) { void stmStDestroy(FStmSt* sws) {
if (sws == NULL) { if (sws == NULL) {
return; return;
} }
...@@ -1207,8 +1206,8 @@ void streamWithStateDestroy(StreamWithState* sws) { ...@@ -1207,8 +1206,8 @@ void streamWithStateDestroy(StreamWithState* sws) {
taosMemoryFree(sws); taosMemoryFree(sws);
} }
bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min) { bool stmStSeekMin(FStmSt* sws, FstBoundWithData* min) {
AutomationCtx* aut = sws->aut; FAutoCtx* aut = sws->aut;
if (fstBoundWithDataIsEmpty(min)) { if (fstBoundWithDataIsEmpty(min)) {
if (fstBoundWithDataIsIncluded(min)) { if (fstBoundWithDataIsIncluded(min)) {
sws->emptyOutput.out = fstEmptyFinalOutput(sws->fst, &(sws->emptyOutput.null)); sws->emptyOutput.out = fstEmptyFinalOutput(sws->fst, &(sws->emptyOutput.null));
...@@ -1301,9 +1300,9 @@ bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min) { ...@@ -1301,9 +1300,9 @@ bool streamWithStateSeekMin(StreamWithState* sws, FstBoundWithData* min) {
return false; return false;
} }
StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallback callback) { FStmStRslt* stmStNextWith(FStmSt* sws, StreamCallback callback) {
AutomationCtx* aut = sws->aut; FAutoCtx* aut = sws->aut;
FstOutput output = sws->emptyOutput; FstOutput output = sws->emptyOutput;
if (output.null == false) { if (output.null == false) {
FstSlice emptySlice = fstSliceCreate(NULL, 0); FstSlice emptySlice = fstSliceCreate(NULL, 0);
if (fstBoundWithDataExceededBy(sws->endAt, &emptySlice)) { if (fstBoundWithDataExceededBy(sws->endAt, &emptySlice)) {
...@@ -1367,8 +1366,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb ...@@ -1367,8 +1366,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb
return NULL; return NULL;
} }
if (FST_NODE_IS_FINAL(nextNode) && isMatch) { if (FST_NODE_IS_FINAL(nextNode) && isMatch) {
FstOutput fOutput = {.null = false, .out = out + FST_NODE_FINAL_OUTPUT(nextNode)}; FstOutput fOutput = {.null = false, .out = out + FST_NODE_FINAL_OUTPUT(nextNode)};
StreamWithStateResult* result = swsResultCreate(&slice, fOutput, tState); FStmStRslt* result = swsResultCreate(&slice, fOutput, tState);
taosMemoryFreeClear(buf); taosMemoryFreeClear(buf);
fstSliceDestroy(&slice); fstSliceDestroy(&slice);
taosArrayDestroy(nodes); taosArrayDestroy(nodes);
...@@ -1382,8 +1381,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb ...@@ -1382,8 +1381,8 @@ StreamWithStateResult* streamWithStateNextWith(StreamWithState* sws, StreamCallb
return NULL; return NULL;
} }
StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* state) { FStmStRslt* swsResultCreate(FstSlice* data, FstOutput fOut, void* state) {
StreamWithStateResult* result = taosMemoryCalloc(1, sizeof(StreamWithStateResult)); FStmStRslt* result = taosMemoryCalloc(1, sizeof(FStmStRslt));
if (result == NULL) { if (result == NULL) {
return NULL; return NULL;
} }
...@@ -1393,7 +1392,7 @@ StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* sta ...@@ -1393,7 +1392,7 @@ StreamWithStateResult* swsResultCreate(FstSlice* data, FstOutput fOut, void* sta
result->state = state; result->state = state;
return result; return result;
} }
void swsResultDestroy(StreamWithStateResult* result) { void swsResultDestroy(FStmStRslt* result) {
if (NULL == result) { if (NULL == result) {
return; return;
} }
...@@ -1411,8 +1410,8 @@ void streamStateDestroy(void* s) { ...@@ -1411,8 +1410,8 @@ void streamStateDestroy(void* s) {
fstNodeDestroy(ss->node); fstNodeDestroy(ss->node);
} }
FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut) { FStmBuilder* stmBuilderCreate(Fst* fst, FAutoCtx* aut) {
FstStreamBuilder* b = taosMemoryCalloc(1, sizeof(FstStreamBuilder)); FStmBuilder* b = taosMemoryCalloc(1, sizeof(FStmBuilder));
if (NULL == b) { if (NULL == b) {
return NULL; return NULL;
} }
...@@ -1423,14 +1422,14 @@ FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut) { ...@@ -1423,14 +1422,14 @@ FstStreamBuilder* fstStreamBuilderCreate(Fst* fst, AutomationCtx* aut) {
b->max = fstBoundStateCreate(Unbounded, NULL); b->max = fstBoundStateCreate(Unbounded, NULL);
return b; return b;
} }
void fstStreamBuilderDestroy(FstStreamBuilder* b) { void stmBuilderDestroy(FStmBuilder* b) {
fstSliceDestroy(&b->min->data); fstSliceDestroy(&b->min->data);
fstSliceDestroy(&b->max->data); fstSliceDestroy(&b->max->data);
taosMemoryFreeClear(b->min); taosMemoryFreeClear(b->min);
taosMemoryFreeClear(b->max); taosMemoryFreeClear(b->max);
taosMemoryFree(b); taosMemoryFree(b);
} }
void fstStreamBuilderSetRange(FstStreamBuilder* b, FstSlice* val, RangeType type) { void stmBuilderSetRange(FStmBuilder* b, FstSlice* val, RangeType type) {
if (b == NULL) { if (b == NULL) {
return; return;
} }
......
source/libs/index/src/indexFstAutomation.c
浏览文件 @ d5f6bb79
...@@ -71,19 +71,19 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv) { ...@@ -71,19 +71,19 @@ StartWithStateValue* startWithStateValueDump(StartWithStateValue* sv) {
} }
// iterate fst // iterate fst
static void* alwaysMatchStart(AutomationCtx* ctx) { return NULL; } static void* alwaysMatchStart(FAutoCtx* ctx) { return NULL; }
static bool alwaysMatchIsMatch(AutomationCtx* ctx, void* state) { return true; } static bool alwaysMatchIsMatch(FAutoCtx* ctx, void* state) { return true; }
static bool alwaysMatchCanMatch(AutomationCtx* ctx, void* state) { return true; } static bool alwaysMatchCanMatch(FAutoCtx* ctx, void* state) { return true; }
static bool alwaysMatchWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } static bool alwaysMatchWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; }
static void* alwaysMatchAccpet(AutomationCtx* ctx, void* state, uint8_t byte) { return NULL; } static void* alwaysMatchAccpet(FAutoCtx* ctx, void* state, uint8_t byte) { return NULL; }
static void* alwaysMatchAccpetEof(AutomationCtx* ctx, void* state) { return NULL; } static void* alwaysMatchAccpetEof(FAutoCtx* ctx, void* state) { return NULL; }
// prefix query, impl later // prefix query, impl later
static void* prefixStart(AutomationCtx* ctx) { static void* prefixStart(FAutoCtx* ctx) {
StartWithStateValue* data = (StartWithStateValue*)(ctx->stdata); StartWithStateValue* data = (StartWithStateValue*)(ctx->stdata);
return startWithStateValueDump(data); return startWithStateValueDump(data);
}; };
static bool prefixIsMatch(AutomationCtx* ctx, void* sv) { static bool prefixIsMatch(FAutoCtx* ctx, void* sv) {
StartWithStateValue* ssv = (StartWithStateValue*)sv; StartWithStateValue* ssv = (StartWithStateValue*)sv;
if (ssv == NULL) { if (ssv == NULL) {
return false; return false;
...@@ -94,15 +94,15 @@ static bool prefixIsMatch(AutomationCtx* ctx, void* sv) { ...@@ -94,15 +94,15 @@ static bool prefixIsMatch(AutomationCtx* ctx, void* sv) {
return false; return false;
} }
} }
static bool prefixCanMatch(AutomationCtx* ctx, void* sv) { static bool prefixCanMatch(FAutoCtx* ctx, void* sv) {
StartWithStateValue* ssv = (StartWithStateValue*)sv; StartWithStateValue* ssv = (StartWithStateValue*)sv;
if (ssv == NULL) { if (ssv == NULL) {
return false; return false;
} }
return ssv->val >= 0; return ssv->val >= 0;
} }
static bool prefixWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } static bool prefixWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; }
static void* prefixAccept(AutomationCtx* ctx, void* state, uint8_t byte) { static void* prefixAccept(FAutoCtx* ctx, void* state, uint8_t byte) {
StartWithStateValue* ssv = (StartWithStateValue*)state; StartWithStateValue* ssv = (StartWithStateValue*)state;
if (ssv == NULL || ctx == NULL) { if (ssv == NULL || ctx == NULL) {
return NULL; return NULL;
...@@ -125,18 +125,18 @@ static void* prefixAccept(AutomationCtx* ctx, void* state, uint8_t byte) { ...@@ -125,18 +125,18 @@ static void* prefixAccept(AutomationCtx* ctx, void* state, uint8_t byte) {
} }
return NULL; return NULL;
} }
static void* prefixAcceptEof(AutomationCtx* ctx, void* state) { return NULL; } static void* prefixAcceptEof(FAutoCtx* ctx, void* state) { return NULL; }
// pattern query, impl later // pattern query, impl later
static void* patternStart(AutomationCtx* ctx) { return NULL; } static void* patternStart(FAutoCtx* ctx) { return NULL; }
static bool patternIsMatch(AutomationCtx* ctx, void* data) { return true; } static bool patternIsMatch(FAutoCtx* ctx, void* data) { return true; }
static bool patternCanMatch(AutomationCtx* ctx, void* data) { return true; } static bool patternCanMatch(FAutoCtx* ctx, void* data) { return true; }
static bool patternWillAlwaysMatch(AutomationCtx* ctx, void* state) { return true; } static bool patternWillAlwaysMatch(FAutoCtx* ctx, void* state) { return true; }
static void* patternAccept(AutomationCtx* ctx, void* state, uint8_t byte) { return NULL; } static void* patternAccept(FAutoCtx* ctx, void* state, uint8_t byte) { return NULL; }
static void* patternAcceptEof(AutomationCtx* ctx, void* state) { return NULL; } static void* patternAcceptEof(FAutoCtx* ctx, void* state) { return NULL; }
AutomationFunc automFuncs[] = { AutomationFunc automFuncs[] = {
{alwaysMatchStart, alwaysMatchIsMatch, alwaysMatchCanMatch, alwaysMatchWillAlwaysMatch, alwaysMatchAccpet, {alwaysMatchStart, alwaysMatchIsMatch, alwaysMatchCanMatch, alwaysMatchWillAlwaysMatch, alwaysMatchAccpet,
...@@ -146,8 +146,8 @@ AutomationFunc automFuncs[] = { ...@@ -146,8 +146,8 @@ AutomationFunc automFuncs[] = {
// add more search type // add more search type
}; };
AutomationCtx* automCtxCreate(void* data, AutomationType atype) { FAutoCtx* automCtxCreate(void* data, AutomationType atype) {
AutomationCtx* ctx = taosMemoryCalloc(1, sizeof(AutomationCtx)); FAutoCtx* ctx = taosMemoryCalloc(1, sizeof(FAutoCtx));
if (ctx == NULL) { if (ctx == NULL) {
return NULL; return NULL;
} }
...@@ -169,7 +169,7 @@ AutomationCtx* automCtxCreate(void* data, AutomationType atype) { ...@@ -169,7 +169,7 @@ AutomationCtx* automCtxCreate(void* data, AutomationType atype) {
ctx->stdata = (void*)sv; ctx->stdata = (void*)sv;
return ctx; return ctx;
} }
void automCtxDestroy(AutomationCtx* ctx) { void automCtxDestroy(FAutoCtx* ctx) {
startWithStateValueDestroy(ctx->stdata); startWithStateValueDestroy(ctx->stdata);
taosMemoryFree(ctx->data); taosMemoryFree(ctx->data);
taosMemoryFree(ctx); taosMemoryFree(ctx);
......
source/libs/index/src/indexJson.c
浏览文件 @ d5f6bb79
...@@ -15,11 +15,11 @@ ...@@ -15,11 +15,11 @@
#include "index.h" #include "index.h"
#include "indexInt.h" #include "indexInt.h"
int tIndexJsonOpen(SIndexJsonOpts *opts, const char *path, SIndexJson **index) { int indexJsonOpen(SIndexJsonOpts *opts, const char *path, SIndexJson **index) {
// handle // handle
return indexOpen(opts, path, index); return indexOpen(opts, path, index);
} }
int tIndexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) { int indexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) {
for (int i = 0; i < taosArrayGetSize(terms); i++) { for (int i = 0; i < taosArrayGetSize(terms); i++) {
SIndexJsonTerm *p = taosArrayGetP(terms, i); SIndexJsonTerm *p = taosArrayGetP(terms, i);
if (p->colType == TSDB_DATA_TYPE_BOOL) { if (p->colType == TSDB_DATA_TYPE_BOOL) {
...@@ -36,7 +36,7 @@ int tIndexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) { ...@@ -36,7 +36,7 @@ int tIndexJsonPut(SIndexJson *index, SIndexJsonMultiTerm *terms, uint64_t uid) {
return indexPut(index, terms, uid); return indexPut(index, terms, uid);
} }
int tIndexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *result) { int indexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *result) {
SArray *terms = tq->query; SArray *terms = tq->query;
for (int i = 0; i < taosArrayGetSize(terms); i++) { for (int i = 0; i < taosArrayGetSize(terms); i++) {
SIndexJsonTerm *p = taosArrayGetP(terms, i); SIndexJsonTerm *p = taosArrayGetP(terms, i);
...@@ -54,7 +54,7 @@ int tIndexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *re ...@@ -54,7 +54,7 @@ int tIndexJsonSearch(SIndexJson *index, SIndexJsonMultiTermQuery *tq, SArray *re
return indexSearch(index, tq, result); return indexSearch(index, tq, result);
} }
void tIndexJsonClose(SIndexJson *index) { void indexJsonClose(SIndexJson *index) {
// handle close // handle close
return indexClose(index); return indexClose(index);
} }
source/libs/index/src/indexTfile.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/index/test/fstTest.cc
浏览文件 @ d5f6bb79
...@@ -85,11 +85,11 @@ class FstReadMemory { ...@@ -85,11 +85,11 @@ class FstReadMemory {
return ok; return ok;
} }
// add later // add later
bool Search(AutomationCtx* ctx, std::vector<uint64_t>& result) { bool Search(FAutoCtx* ctx, std::vector<uint64_t>& result) {
FstStreamBuilder* sb = fstSearch(_fst, ctx); FStmBuilder* sb = fstSearch(_fst, ctx);
StreamWithState* st = streamBuilderIntoStream(sb); FStmSt* st = stmBuilderIntoStm(sb);
StreamWithStateResult* rt = NULL; FStmStRslt* rt = NULL;
while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { while ((rt = stmStNextWith(st, NULL)) != NULL) {
// result.push_back((uint64_t)(rt->out.out)); // result.push_back((uint64_t)(rt->out.out));
FstSlice* s = &rt->data; FstSlice* s = &rt->data;
int32_t sz = 0; int32_t sz = 0;
...@@ -99,27 +99,27 @@ class FstReadMemory { ...@@ -99,27 +99,27 @@ class FstReadMemory {
result.push_back(rt->out.out); result.push_back(rt->out.out);
swsResultDestroy(rt); swsResultDestroy(rt);
} }
streamWithStateDestroy(st); stmStDestroy(st);
fstStreamBuilderDestroy(sb); stmBuilderDestroy(sb);
return true; return true;
} }
bool SearchRange(AutomationCtx* ctx, const std::string& low, RangeType lowType, const std::string& high, bool SearchRange(FAutoCtx* ctx, const std::string& low, RangeType lowType, const std::string& high,
RangeType highType, std::vector<uint64_t>& result) { RangeType highType, std::vector<uint64_t>& result) {
FstStreamBuilder* sb = fstSearch(_fst, ctx); FStmBuilder* sb = fstSearch(_fst, ctx);
FstSlice l = fstSliceCreate((uint8_t*)low.c_str(), low.size()); FstSlice l = fstSliceCreate((uint8_t*)low.c_str(), low.size());
FstSlice h = fstSliceCreate((uint8_t*)high.c_str(), high.size()); FstSlice h = fstSliceCreate((uint8_t*)high.c_str(), high.size());
// range [low, high); // range [low, high);
fstStreamBuilderSetRange(sb, &l, lowType); stmBuilderSetRange(sb, &l, lowType);
fstStreamBuilderSetRange(sb, &h, highType); stmBuilderSetRange(sb, &h, highType);
fstSliceDestroy(&l); fstSliceDestroy(&l);
fstSliceDestroy(&h); fstSliceDestroy(&h);
StreamWithState* st = streamBuilderIntoStream(sb); FStmSt* st = stmBuilderIntoStm(sb);
StreamWithStateResult* rt = NULL; FStmStRslt* rt = NULL;
while ((rt = streamWithStateNextWith(st, NULL)) != NULL) { while ((rt = stmStNextWith(st, NULL)) != NULL) {
// result.push_back((uint64_t)(rt->out.out)); // result.push_back((uint64_t)(rt->out.out));
FstSlice* s = &rt->data; FstSlice* s = &rt->data;
int32_t sz = 0; int32_t sz = 0;
...@@ -129,11 +129,11 @@ class FstReadMemory { ...@@ -129,11 +129,11 @@ class FstReadMemory {
result.push_back(rt->out.out); result.push_back(rt->out.out);
swsResultDestroy(rt); swsResultDestroy(rt);
} }
streamWithStateDestroy(st); stmStDestroy(st);
fstStreamBuilderDestroy(sb); stmBuilderDestroy(sb);
return true; return true;
} }
bool SearchWithTimeCostUs(AutomationCtx* ctx, std::vector<uint64_t>& result) { bool SearchWithTimeCostUs(FAutoCtx* ctx, std::vector<uint64_t>& result) {
int64_t s = taosGetTimestampUs(); int64_t s = taosGetTimestampUs();
bool ok = this->Search(ctx, result); bool ok = this->Search(ctx, result);
int64_t e = taosGetTimestampUs(); int64_t e = taosGetTimestampUs();
...@@ -253,7 +253,7 @@ void checkFstLongTerm() { ...@@ -253,7 +253,7 @@ void checkFstLongTerm() {
// prefix search // prefix search
// std::vector<uint64_t> result; // std::vector<uint64_t> result;
// AutomationCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS); // FAutoCtx* ctx = automCtxCreate((void*)"ab", AUTOMATION_ALWAYS);
// m->Search(ctx, result); // m->Search(ctx, result);
// std::cout << "size: " << result.size() << std::endl; // std::cout << "size: " << result.size() << std::endl;
// assert(result.size() == count); // assert(result.size() == count);
...@@ -286,7 +286,7 @@ void checkFstCheckIterator1() { ...@@ -286,7 +286,7 @@ void checkFstCheckIterator1() {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS);
m->Search(ctx, result); m->Search(ctx, result);
std::cout << "size: " << result.size() << std::endl; std::cout << "size: " << result.size() << std::endl;
// assert(result.size() == count); // assert(result.size() == count);
...@@ -321,7 +321,7 @@ void checkFstCheckIterator2() { ...@@ -321,7 +321,7 @@ void checkFstCheckIterator2() {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"He", AUTOMATION_ALWAYS);
m->Search(ctx, result); m->Search(ctx, result);
std::cout << "size: " << result.size() << std::endl; std::cout << "size: " << result.size() << std::endl;
// assert(result.size() == count); // assert(result.size() == count);
...@@ -361,7 +361,7 @@ void checkFstCheckIteratorPrefix() { ...@@ -361,7 +361,7 @@ void checkFstCheckIteratorPrefix() {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_PREFIX); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_PREFIX);
m->Search(ctx, result); m->Search(ctx, result);
assert(result.size() == 1); assert(result.size() == 1);
automCtxDestroy(ctx); automCtxDestroy(ctx);
...@@ -370,7 +370,7 @@ void checkFstCheckIteratorPrefix() { ...@@ -370,7 +370,7 @@ void checkFstCheckIteratorPrefix() {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"Hello", AUTOMATION_PREFIX); FAutoCtx* ctx = automCtxCreate((void*)"Hello", AUTOMATION_PREFIX);
m->Search(ctx, result); m->Search(ctx, result);
assert(result.size() == 2); assert(result.size() == 2);
automCtxDestroy(ctx); automCtxDestroy(ctx);
...@@ -378,7 +378,7 @@ void checkFstCheckIteratorPrefix() { ...@@ -378,7 +378,7 @@ void checkFstCheckIteratorPrefix() {
{ {
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"jddd", AUTOMATION_PREFIX); FAutoCtx* ctx = automCtxCreate((void*)"jddd", AUTOMATION_PREFIX);
m->Search(ctx, result); m->Search(ctx, result);
assert(result.size() == 1); assert(result.size() == 1);
automCtxDestroy(ctx); automCtxDestroy(ctx);
...@@ -412,7 +412,7 @@ void checkFstCheckIteratorRange1() { ...@@ -412,7 +412,7 @@ void checkFstCheckIteratorRange1() {
{ {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GE, "e", LT, result); m->SearchRange(ctx, "b", GE, "e", LT, result);
assert(result.size() == 3); assert(result.size() == 3);
...@@ -421,7 +421,7 @@ void checkFstCheckIteratorRange1() { ...@@ -421,7 +421,7 @@ void checkFstCheckIteratorRange1() {
{ {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GT, "e", LT, result); m->SearchRange(ctx, "b", GT, "e", LT, result);
assert(result.size() == 2); assert(result.size() == 2);
...@@ -430,7 +430,7 @@ void checkFstCheckIteratorRange1() { ...@@ -430,7 +430,7 @@ void checkFstCheckIteratorRange1() {
{ {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GT, "e", LE, result); m->SearchRange(ctx, "b", GT, "e", LE, result);
assert(result.size() == 3); assert(result.size() == 3);
...@@ -439,7 +439,7 @@ void checkFstCheckIteratorRange1() { ...@@ -439,7 +439,7 @@ void checkFstCheckIteratorRange1() {
{ {
// prefix search // prefix search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GE, "e", LE, result); m->SearchRange(ctx, "b", GE, "e", LE, result);
assert(result.size() == 4); assert(result.size() == 4);
...@@ -473,7 +473,7 @@ void checkFstCheckIteratorRange2() { ...@@ -473,7 +473,7 @@ void checkFstCheckIteratorRange2() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GE, "ed", LT, result); m->SearchRange(ctx, "b", GE, "ed", LT, result);
assert(result.size() == 4); assert(result.size() == 4);
...@@ -482,7 +482,7 @@ void checkFstCheckIteratorRange2() { ...@@ -482,7 +482,7 @@ void checkFstCheckIteratorRange2() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "bb", GE, "ed", LT, result); m->SearchRange(ctx, "bb", GE, "ed", LT, result);
assert(result.size() == 3); assert(result.size() == 3);
...@@ -491,7 +491,7 @@ void checkFstCheckIteratorRange2() { ...@@ -491,7 +491,7 @@ void checkFstCheckIteratorRange2() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GE, "ed", LE, result); m->SearchRange(ctx, "b", GE, "ed", LE, result);
assert(result.size() == 5); assert(result.size() == 5);
...@@ -501,7 +501,7 @@ void checkFstCheckIteratorRange2() { ...@@ -501,7 +501,7 @@ void checkFstCheckIteratorRange2() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GT, "ed", LE, result); m->SearchRange(ctx, "b", GT, "ed", LE, result);
assert(result.size() == 4); assert(result.size() == 4);
...@@ -510,7 +510,7 @@ void checkFstCheckIteratorRange2() { ...@@ -510,7 +510,7 @@ void checkFstCheckIteratorRange2() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GT, "ed", LT, result); m->SearchRange(ctx, "b", GT, "ed", LT, result);
assert(result.size() == 3); assert(result.size() == 3);
...@@ -544,7 +544,7 @@ void checkFstCheckIteratorRange3() { ...@@ -544,7 +544,7 @@ void checkFstCheckIteratorRange3() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "b", GE, "", (RangeType)10, result); m->SearchRange(ctx, "b", GE, "", (RangeType)10, result);
assert(result.size() == 5); assert(result.size() == 5);
...@@ -553,7 +553,7 @@ void checkFstCheckIteratorRange3() { ...@@ -553,7 +553,7 @@ void checkFstCheckIteratorRange3() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "", (RangeType)20, "ab", LE, result); m->SearchRange(ctx, "", (RangeType)20, "ab", LE, result);
assert(result.size() == 1); assert(result.size() == 1);
...@@ -563,7 +563,7 @@ void checkFstCheckIteratorRange3() { ...@@ -563,7 +563,7 @@ void checkFstCheckIteratorRange3() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "", (RangeType)30, "ab", LT, result); m->SearchRange(ctx, "", (RangeType)30, "ab", LT, result);
assert(result.size() == 0); assert(result.size() == 0);
...@@ -572,7 +572,7 @@ void checkFstCheckIteratorRange3() { ...@@ -572,7 +572,7 @@ void checkFstCheckIteratorRange3() {
{ {
// range search // range search
std::vector<uint64_t> result; std::vector<uint64_t> result;
AutomationCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS); FAutoCtx* ctx = automCtxCreate((void*)"he", AUTOMATION_ALWAYS);
// [b, e) // [b, e)
m->SearchRange(ctx, "ed", GT, "ed", (RangeType)40, result); m->SearchRange(ctx, "ed", GT, "ed", (RangeType)40, result);
assert(result.size() == 0); assert(result.size() == 0);
......
source/libs/index/test/fstUT.cc
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/index/test/jsonUT.cc
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/index/test/utilUT.cc
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/parser/src/parInsert.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/parser/src/parUtil.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/qcom/src/querymsg.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/qworker/src/qwDbg.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scalar/inc/sclInt.h
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scalar/src/filter.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scalar/src/scalar.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scalar/src/sclfunc.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scalar/src/sclvector.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/scheduler/src/schJob.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/stream/src/stream.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/stream/src/streamExec.c
浏览文件 @ d5f6bb79
此差异已折叠。
source/libs/sync/src/syncMain.c
浏览文件 @ d5f6bb79
此差异已折叠。
tests/test/c/tmqSim.c
浏览文件 @ d5f6bb79
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或