diff --git a/docs/en/14-reference/03-connector/rust.mdx b/docs/en/14-reference/03-connector/rust.mdx index a5cbaeac8077cda42690d9cc232062a685a51f41..56ca586c7e8ada6e4422596906e01887d4726fd0 100644 --- a/docs/en/14-reference/03-connector/rust.mdx +++ b/docs/en/14-reference/03-connector/rust.mdx @@ -250,7 +250,7 @@ The [Taos] structure is the connection manager in [libtaos] and provides two mai Column information is stored using [ColumnMeta]. - ``rust + ```rust let cols = &q.column_meta; for col in cols { println!("name: {}, type: {:?} , bytes: {}", col.name, col.type_, col.bytes); diff --git a/docs/en/14-reference/05-taosbenchmark.md b/docs/en/14-reference/05-taosbenchmark.md index 7cf1f95eb116b5f87b3bc1e05b647b9b0da3c544..5d47cc06e8034d8c49669d71c3f98c1f587acb33 100644 --- a/docs/en/14-reference/05-taosbenchmark.md +++ b/docs/en/14-reference/05-taosbenchmark.md @@ -7,7 +7,7 @@ description: "taosBenchmark (once called taosdemo ) is a tool for testing the pe ## Introduction -taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can flexibly control the number and type of databases, supertables, tag columns, number and type of data columns, and sub-tables, and types of databases, super tables, the number and types of data columns, the number of sub-tables, the amount of data per sub-table, the time interval for inserting data, the number of working threads, whether and how to insert disordered data, and so on. The installer provides taosdemo as a soft link to taosBenchmark for compatibility and for the convenience of past users. +taosBenchmark (formerly taosdemo ) is a tool for testing the performance of TDengine products. taosBenchmark can test the performance of TDengine's insert, query, and subscription functions and simulate large amounts of data generated by many devices. taosBenchmark can be configured to generate user defined databases, supertables, subtables, and the time series data to populate these for performance benchmarking. taosBenchmark is highly configurable and some of the configurations include the time interval for inserting data, the number of working threads and the capability to insert disordered data. The installer provides taosdemo as a soft link to taosBenchmark for compatibility with past users. ## Installation @@ -21,9 +21,13 @@ There are two ways to install taosBenchmark: ### Configuration and running methods -TaosBenchmark needs to be executed on the terminal of the operating system, it supports two configuration methods: [Command-line arguments](#Command-line arguments in detailed) and [JSON configuration file](#Configuration file arguments in detailed). These two methods are mutually exclusive. Users can use `-f ` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters. +TaosBenchmark needs to be executed on the terminal of the operating system, it supports two configuration methods: [Command-line arguments](#command-line-arguments-in-detail) and [JSON configuration file](#configuration-file-parameters-in-detail). These two methods are mutually exclusive. Users can use `-f ` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters. +<<<<<<< HEAD taosBenchmark supports complete performance testing of TDengine. taosBenchmark supports the TDengine functions in three categories: write, query, and subscribe. These three functions are mutually exclusive, and users can select only one of them each time taosBenchmark runs. It is important to note that the type of functionality to be tested is not configurable when using the command-line configuration method, which can only test writing performance. To test the query and subscription performance of the TDengine, you must use the configuration file method and specify the function type to test via the parameter `filetype` in the configuration file. +======= +taosBenchmark supports the complete performance testing of TDengine by providing functionally to write, query, and subscribe. These three functions are mutually exclusive, users can only select one of them each time taosBenchmark runs. The query and subscribe functionalities are only configurable using a json configuration file by specifying the parameter `filetype`, while write can be performed through both the command-line and a configuration file. +>>>>>>> 108548b4d6 (docs: typo) **Make sure that the TDengine cluster is running correctly before running taosBenchmark. ** @@ -57,9 +61,8 @@ Use the following command-line to run taosBenchmark and control its behavior via taosBenchmark -f ``` -**Here are a few examples of configuration files:** - -#### Example of inserting a scenario JSON configuration file +#### Configuration file examples +##### Example of inserting a scenario JSON configuration file
insert.json @@ -70,7 +73,7 @@ taosBenchmark -f
-#### Query Scenario JSON Profile Example +##### Query Scenario JSON Profile Example
query.json @@ -81,7 +84,7 @@ taosBenchmark -f
-#### Subscription JSON configuration example +##### Subscription JSON configuration example
subscribe.json @@ -172,7 +175,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\) Switch parameter specifying whether to use escape characters in the super table and sub-table names. By default is not used. - **-C/--chinese** : - Switch specifying whether to use Unicode Chinese characters in nchar and binary. By default is not used. + specify whether to use Unicode Chinese characters in nchar and binary, the default is no. - **-N/--normal-table** : This parameter indicates that taosBenchmark will create only normal tables instead of super tables. The default value is false. It can be used if the insert mode is taosc, stmt, and rest. @@ -373,7 +376,7 @@ The configuration parameters for querying the sub-tables or the normal tables ar - **sqls**. - **sql**: the SQL command to be executed. - - **result**: the file to save the query result. If it is unspecified, taosBenchark will not save the result. + - **result**: the file to save the query result. If it is unspecified, taosBenchmark will not save the result. #### Configuration parameters of query super table diff --git a/docs/en/14-reference/06-taosdump.md b/docs/en/14-reference/06-taosdump.md index 5403e40925f633ce62795cc6037fc8c8f7aad07a..96e68d0edbf9a2f6880cc557580d3dfb20def947 100644 --- a/docs/en/14-reference/06-taosdump.md +++ b/docs/en/14-reference/06-taosdump.md @@ -36,6 +36,8 @@ There are two ways to install taosdump: :::tip - taosdump versions after 1.4.1 provide the `-I` argument for parsing Avro file schema and data. If users specify `-s` then only taosdump will parse schema. - Backups after taosdump 1.4.2 use the batch count specified by the `-B` parameter. The default value is 16384. If, in some environments, low network speed or disk performance causes "Error actual dump ... batch ...", then try changing the `-B` parameter to a smaller value. +- The export of taosdump does not support resuming from an interruption. Therefore, if the taosdump process terminates unexpectedly, delete all related files that have been exported or generated. +- The import of taosdump supports resuming from an interruption, but when the process resumes, you will receive some "table already exists" messages, which could be ignored. ::: diff --git a/docs/en/20-third-party/01-grafana.mdx b/docs/en/20-third-party/01-grafana.mdx index 9076f163bea60e87b5d1aaed79d966572463d56d..5dbeb31a231464e48b4f977420f03f0ede81e78e 100644 --- a/docs/en/20-third-party/01-grafana.mdx +++ b/docs/en/20-third-party/01-grafana.mdx @@ -31,38 +31,41 @@ TDengine currently supports Grafana versions 7.5 and above. Users can go to the ### Install Grafana Plugin and Configure Data Source - + -Set the url and authorization environment variables by `export` or a [`.env`(dotenv) file](https://hexdocs.pm/dotenvy/dotenv-file-format.html): +Under Grafana 8, plugin catalog allows you to [browse and manage plugins within Grafana](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog) (but for Grafana 7.x, use **With Script** or **Install & Configure Manually**). Find the page at **Configurations > Plugins**, search **TDengine** and click it to install. -```sh -export TDENGINE_API=http://tdengine.local:6041 -# user + password -export TDENGINE_USER=user -export TDENGINE_PASSWORD=password - -# Other useful variables -# - If to install TDengine data source, default is true -export TDENGINE_DS_ENABLED=false -# - Data source name to be created, default is TDengine -export TDENGINE_DS_NAME=TDengine -# - Data source organization id, default is 1 -export GF_ORG_ID=1 -# - Data source is editable in admin ui or not, default is 0 (false) -export TDENGINE_EDITABLE=1 -``` +![Search tdengine in grafana plugins](./grafana/grafana-plugin-search-tdengine.png) + +Installation may cost some minutes, then you can **Create a TDengine data source**: + +![Install and configure Grafana data source](./grafana/grafana-install-and-config.png) + +Then you can add a TDengine data source by filling up the configuration options. + +![TDengine Database Grafana plugin add data source](./grafana/grafana-data-source.png) + +You can create dashboards with TDengine now. + + + -Run `install.sh`: +On a server with Grafana installed, run `install.sh` with TDengine url and username/passwords will install TDengine data source plugin and add a data source named TDengine. This is the recommended way for Grafana 7.x or [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) users. ```sh -bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" +bash -c "$(curl -fsSL \ + https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" -- \ + -a http://localhost:6041 \ + -u root \ + -p taosdata ``` -With this script, TDengine data source plugin and the Grafana data source will be installed and created automatically with Grafana provisioning configurations. Save the script and type `./install.sh --help` for the full usage of the script. +Restart Grafana service and open Grafana in web-browser, usually . -And then, restart Grafana service and open Grafana in web-browser, usually . +Save the script and type `./install.sh --help` for the full usage of the script. + Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation. @@ -115,6 +118,73 @@ Click `Save & Test` to test. You should see a success message if the test worked ![TDengine Database TDinsight plugin add database 4](./grafana/add_datasource4.webp) + + + +Please refer to [Install plugins in the Docker container](https://grafana.com/docs/grafana/next/setup-grafana/installation/docker/#install-plugins-in-the-docker-container). This will install `tdengine-datasource` plugin when Grafana container starts: + +```bash +docker run -d \ + -p 3000:3000 \ + --name=grafana \ + -e "GF_INSTALL_PLUGINS=tdengine-datasource" \ + grafana/grafana +``` + +You can setup a zero-configuration stack for TDengine + Grafana by [docker-compose](https://docs.docker.com/compose/) and [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) file： + +1. Save the provisioning configuration file to `tdengine.yml`. + + ```yml + apiVersion: 1 + datasources: + - name: TDengine + type: tdengine-datasource + orgId: 1 + url: "$TDENGINE_API" + isDefault: true + secureJsonData: + url: "$TDENGINE_URL" + basicAuth: "$TDENGINE_BASIC_AUTH" + token: "$TDENGINE_CLOUD_TOKEN" + version: 1 + editable: true + ``` + +2. Write `docker-compose.yml` with [TDengine](https://hub.docker.com/r/tdengine/tdengine) and [Grafana](https://hub.docker.com/r/grafana/grafana) image. + + ```yml + version: "3.7" + + services: + tdengine: + image: tdengine/tdengine:2.6.0.2 + environment: + TAOS_FQDN: tdengine + volumes: + - tdengine-data:/var/lib/taos/ + grafana: + image: grafana/grafana:8.5.6 + volumes: + - ./tdengine.yml/:/etc/grafana/provisioning/tdengine.yml + - grafana-data:/var/lib/grafana + environment: + # install tdengine plugin at start + GF_INSTALL_PLUGINS: "tdengine-datasource" + TDENGINE_URL: "http://tdengine:6041" + #printf "$TDENGINE_USER:$TDENGINE_PASSWORD" | base64 + TDENGINE_BASIC_AUTH: "cm9vdDp0YmFzZTEyNQ==" + ports: + - 3000:3000 + volumes: + grafana-data: + tdengine-data: + ``` + +3. Start TDengine and Grafana services: `docker-compose up -d`. + +Open Grafana , and you can add dashboard with TDengine now. + diff --git a/docs/en/20-third-party/grafana/grafana-data-source.png b/docs/en/20-third-party/grafana/grafana-data-source.png new file mode 100644 index 0000000000000000000000000000000000000000..989ffcca0bf5baae8798b0695e259aca35f0442a Binary files /dev/null and b/docs/en/20-third-party/grafana/grafana-data-source.png differ diff --git a/docs/en/20-third-party/grafana/grafana-install-and-config.png b/docs/en/20-third-party/grafana/grafana-install-and-config.png new file mode 100644 index 0000000000000000000000000000000000000000..b918da8b2d62e694fe1797e09cf8f23f103bc97e Binary files /dev/null and b/docs/en/20-third-party/grafana/grafana-install-and-config.png differ diff --git a/docs/en/20-third-party/grafana/grafana-plugin-search-tdengine.png b/docs/en/20-third-party/grafana/grafana-plugin-search-tdengine.png new file mode 100644 index 0000000000000000000000000000000000000000..cf3b66977b64f7dcd617f06024a66066cd62810e Binary files /dev/null and b/docs/en/20-third-party/grafana/grafana-plugin-search-tdengine.png differ diff --git a/docs/zh/14-reference/06-taosdump.md b/docs/zh/14-reference/06-taosdump.md index 3a9f2e9acd215be102991a1d91fba285ef6315bb..95ee20bfbae3f0f57c51b8fcd3bd9b35b4764f3d 100644 --- a/docs/zh/14-reference/06-taosdump.md +++ b/docs/zh/14-reference/06-taosdump.md @@ -39,6 +39,8 @@ taosdump 有两种安装方式: :::tip - taosdump 1.4.1 之后的版本提供 `-I` 参数，用于解析 avro 文件 schema 和数据，如果指定 `-s` 参数将只解析 schema。 - taosdump 1.4.2 之后的备份使用 `-B` 参数指定的批次数，默认值为 16384，如果在某些环境下由于网络速度或磁盘性能不足导致 "Error actual dump .. batch .." 可以通过 `-B` 参数调整为更小的值进行尝试。 +- taosdump 的导出不支持中断恢复，所以当进程意外终止后，正确的处理方式是删除当前已导出或生成的所有相关文件。 +- taosdump 的导入支持中断恢复，但是当进程重新启动时，会收到一些“表已经存在”的提示，可以忽视。 ::: diff --git a/docs/zh/20-third-party/01-grafana.mdx b/docs/zh/20-third-party/01-grafana.mdx index 09c0d786cf5ac3fc9a6c5e6a72087be0b658454d..93090ffd38c3ce66488826c486584dd305dbc20c 100644 --- a/docs/zh/20-third-party/01-grafana.mdx +++ b/docs/zh/20-third-party/01-grafana.mdx @@ -29,39 +29,41 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/ ### 安装 Grafana Plugin 并配置数据源 - + -将集群信息设置为环境变量；也可以使用 `.env` 文件，请参考 [dotenv](https://hexdocs.pm/dotenvy/dotenv-file-format.html)： +使用 Grafana 最新版本（8.5+），您可以在 Grafana 中[浏览和管理插件](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog)（对于 7.x 版本，请使用 **安装脚本** 或 **手动安装并配置** 方式）。在 Grafana 管理界面中的 **Configurations > Plugins** 页面直接搜索并按照提示安装 TDengine。 -```sh -export TDENGINE_API=http://tdengine.local:6041 -# user + password -export TDENGINE_USER=user -export TDENGINE_PASSWORD=password - -# 其他环境变量： -# - 是否安装数据源，默认为 true，表示安装 -export TDENGINE_DS_ENABLED=false -# - 数据源名称，默认为 TDengine -export TDENGINE_DS_NAME=TDengine -# - 数据源所属组织 ID，默认为 1 -export GF_ORG_ID=1 -# - 数据源是否可通过管理面板编辑，默认为 0，表示不可编辑 -export TDENGINE_EDITABLE=1 -``` +![Search tdengine in grafana plugins](grafana-plugin-search-tdengine.png) + +如图示即安装完毕，按照指示 **Create a TDengine data source** 添加数据源。 + +![Install and configure Grafana data source](grafana-install-and-config.png) + +输入 TDengine 相关配置，完成数据源配置。 + +![TDengine Database Grafana plugin add data source](./grafana-data-source.png) + +配置完毕，现在可以使用 TDengine 创建 Dashboard 了。 + + + -运行安装脚本： +对于使用 Grafana 7.x 版本或使用 [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) 配置的用户，可以在 Grafana 服务器上使用安装脚本自动安装插件即添加数据源 Provisioning 配置文件。 ```sh -bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" +bash -c "$(curl -fsSL \ + https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)" -- \ + -a http://localhost:6041 \ + -u root \ + -p taosdata ``` -该脚本将自动安装 Grafana 插件并配置数据源。安装完毕后，需要重启 Grafana 服务后生效。 +安装完毕后，需要重启 Grafana 服务后方可生效。 保存该脚本并执行 `./install.sh --help` 可查看详细帮助文档。 - + 使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。 @@ -113,6 +115,73 @@ GF_INSTALL_PLUGINS=tdengine-datasource ![TDengine Database Grafana plugin add data source](./add_datasource4.webp) + + + +参考 [Grafana 容器化安装说明](https://grafana.com/docs/grafana/next/setup-grafana/installation/docker/#install-plugins-in-the-docker-container)。使用如下命令启动一个容器，并自动安装 TDengine 插件： + +```bash +docker run -d \ + -p 3000:3000 \ + --name=grafana \ + -e "GF_INSTALL_PLUGINS=tdengine-datasource" \ + grafana/grafana +``` + +使用 docker-compose，配置 Grafana Provisioning 自动化配置，体验 TDengine + Grafana 组合的零配置启动： + +1. 保存该文件为 `tdengine.yml`。 + + ```yml + apiVersion: 1 + datasources: + - name: TDengine + type: tdengine-datasource + orgId: 1 + url: "$TDENGINE_API" + isDefault: true + secureJsonData: + url: "$TDENGINE_URL" + basicAuth: "$TDENGINE_BASIC_AUTH" + token: "$TDENGINE_CLOUD_TOKEN" + version: 1 + editable: true + ``` + +2. 保存该文件为 `docker-compose.yml`。 + + ```yml + version: "3.7" + + services: + tdengine: + image: tdengine/tdengine:2.6.0.2 + environment: + TAOS_FQDN: tdengine + volumes: + - tdengine-data:/var/lib/taos/ + grafana: + image: grafana/grafana:8.5.6 + volumes: + - ./tdengine.yml/:/etc/grafana/provisioning/tdengine.yml + - grafana-data:/var/lib/grafana + environment: + # install tdengine plugin at start + GF_INSTALL_PLUGINS: "tdengine-datasource" + TDENGINE_URL: "http://tdengine:6041" + #printf "$TDENGINE_USER:$TDENGINE_PASSWORD" | base64 + TDENGINE_BASIC_AUTH: "cm9vdDp0YmFzZTEyNQ==" + ports: + - 3000:3000 + volumes: + grafana-data: + tdengine-data: + ``` + +3. 使用 docker-compose 命令启动 TDengine + Grafana ：`docker-compose up -d`。 + +打开 Grafana ，现在可以添加 Dashboard 了。 + diff --git a/docs/zh/20-third-party/grafana-data-source.png b/docs/zh/20-third-party/grafana-data-source.png new file mode 100644 index 0000000000000000000000000000000000000000..989ffcca0bf5baae8798b0695e259aca35f0442a Binary files /dev/null and b/docs/zh/20-third-party/grafana-data-source.png differ diff --git a/docs/zh/20-third-party/grafana-install-and-config.png b/docs/zh/20-third-party/grafana-install-and-config.png new file mode 100644 index 0000000000000000000000000000000000000000..b918da8b2d62e694fe1797e09cf8f23f103bc97e Binary files /dev/null and b/docs/zh/20-third-party/grafana-install-and-config.png differ diff --git a/docs/zh/20-third-party/grafana-plugin-search-tdengine.png b/docs/zh/20-third-party/grafana-plugin-search-tdengine.png new file mode 100644 index 0000000000000000000000000000000000000000..cf3b66977b64f7dcd617f06024a66066cd62810e Binary files /dev/null and b/docs/zh/20-third-party/grafana-plugin-search-tdengine.png differ diff --git a/examples/c/stream_demo.c b/examples/c/stream_demo.c index ab59fa5e474cb8e6dde123b18c8cc265570aeeaa..6d341c61c76c85df04bf9cc97bd6666c18c83fe8 100644 --- a/examples/c/stream_demo.c +++ b/examples/c/stream_demo.c @@ -32,6 +32,13 @@ int32_t init_env() { } taos_free_result(pRes); + pRes = taos_query(pConn, "create database if not exists abc2 vgroups 20"); + if (taos_errno(pRes) != 0) { + printf("error in create db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { printf("error in use db, reason:%s\n", taos_errstr(pRes)); @@ -83,7 +90,7 @@ int32_t create_stream() { /*pRes = tmq_create_stream(pConn, "stream1", "out1", sql);*/ pRes = taos_query( pConn, - "create stream stream1 trigger max_delay 10s into outstb as select _wstartts, sum(k) from st1 interval(10m)"); + "create stream stream1 trigger at_once into abc1.outstb as select _wstartts, sum(k) from st1 interval(10m) "); if (taos_errno(pRes) != 0) { printf("failed to create stream stream1, reason:%s\n", taos_errstr(pRes)); return -1; @@ -100,11 +107,4 @@ int main(int argc, char* argv[]) { code = init_env(); } create_stream(); -#if 0 - tmq_t* tmq = build_consumer(); - tmq_list_t* topic_list = build_topic_list(); - /*perf_loop(tmq, topic_list);*/ - /*basic_consume_loop(tmq, topic_list);*/ - sync_consume_loop(tmq, topic_list); -#endif } diff --git a/examples/c/tmq.c b/examples/c/tmq.c index 2e8aa21da7a2bdd83e4a995beccb99ac40228a48..7870d7d9a188030806fc113fbadcb2f76172ef66 100644 --- a/examples/c/tmq.c +++ b/examples/c/tmq.c @@ -47,7 +47,7 @@ int32_t init_env() { return -1; } - TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 2"); + TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 1"); if (taos_errno(pRes) != 0) { printf("error in create db, reason:%s\n", taos_errstr(pRes)); return -1; @@ -146,8 +146,8 @@ int32_t create_topic() { return 0; } -void tmq_commit_cb_print(tmq_t* tmq, tmq_resp_err_t resp, tmq_topic_vgroup_list_t* offsets, void* param) { - printf("commit %d tmq %p offsets %p param %p\n", resp, tmq, offsets, param); +void tmq_commit_cb_print(tmq_t* tmq, int32_t code, void* param) { + printf("commit %d tmq %p param %p\n", code, tmq, param); } tmq_t* build_consumer() { @@ -167,7 +167,7 @@ tmq_t* build_consumer() { tmq_conf_set(conf, "td.connect.user", "root"); tmq_conf_set(conf, "td.connect.pass", "taosdata"); tmq_conf_set(conf, "msg.with.table.name", "true"); - tmq_conf_set(conf, "enable.auto.commit", "false"); + tmq_conf_set(conf, "enable.auto.commit", "true"); tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL); tmq_t* tmq = tmq_consumer_new(conf, NULL, 0); assert(tmq); @@ -183,10 +183,10 @@ tmq_list_t* build_topic_list() { } void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) { - tmq_resp_err_t err; + int32_t code; - if ((err = tmq_subscribe(tmq, topics))) { - fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err)); + if ((code = tmq_subscribe(tmq, topics))) { + fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(code)); printf("subscribe err\n"); return; } @@ -201,12 +201,13 @@ void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) { taos_free_result(tmqmessage); /*} else {*/ /*break;*/ + /*tmq_commit_sync(tmq, NULL);*/ } } - err = tmq_consumer_close(tmq); - if (err) - fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(err)); + code = tmq_consumer_close(tmq); + if (code) + fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(code)); else fprintf(stderr, "%% Consumer closed\n"); } @@ -214,11 +215,11 @@ void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) { void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) { static const int MIN_COMMIT_COUNT = 1; - int msg_count = 0; - tmq_resp_err_t err; + int msg_count = 0; + int32_t code; - if ((err = tmq_subscribe(tmq, topics))) { - fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err)); + if ((code = tmq_subscribe(tmq, topics))) { + fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(code)); return; } @@ -239,14 +240,14 @@ void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) { msg_process(tmqmessage); taos_free_result(tmqmessage); - /*tmq_commit_async(tmq, NULL, tmq_commit_cb_print, NULL);*/ + /*tmq_commit_sync(tmq, NULL);*/ /*if ((++msg_count % MIN_COMMIT_COUNT) == 0) tmq_commit(tmq, NULL, 0);*/ } } - err = tmq_consumer_close(tmq); - if (err) - fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(err)); + code = tmq_consumer_close(tmq); + if (code) + fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(code)); else fprintf(stderr, "%% Consumer closed\n"); } diff --git a/include/client/taos.h b/include/client/taos.h index deb4276b54356df5a125aba16b245410114921da..61538e392a0daa14b3f2def6ed74b0ee51d8d7ef 100644 --- a/include/client/taos.h +++ b/include/client/taos.h @@ -126,47 +126,47 @@ typedef struct setConfRet { char retMsg[RET_MSG_LENGTH]; } setConfRet; -DLL_EXPORT void taos_cleanup(void); -DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...); -DLL_EXPORT setConfRet taos_set_config(const char *config); -DLL_EXPORT int taos_init(void); -DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port); -DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port); -DLL_EXPORT void taos_close(TAOS *taos); - -const char *taos_data_type(int type); - -DLL_EXPORT TAOS_STMT *taos_stmt_init(TAOS *taos); -DLL_EXPORT int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length); -DLL_EXPORT int taos_stmt_set_tbname_tags(TAOS_STMT *stmt, const char *name, TAOS_MULTI_BIND *tags); -DLL_EXPORT int taos_stmt_set_tbname(TAOS_STMT *stmt, const char *name); -DLL_EXPORT int taos_stmt_set_tags(TAOS_STMT *stmt, TAOS_MULTI_BIND *tags); -DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name); -DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); -DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); - -DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert); -DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums); -DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes); -DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); -DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); -DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx); -DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt); -DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt); -DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt); -DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt); -DLL_EXPORT char *taos_stmt_errstr(TAOS_STMT *stmt); -DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt); -DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt); - -DLL_EXPORT TAOS_RES *taos_query(TAOS *taos, const char *sql); - -DLL_EXPORT TAOS_ROW taos_fetch_row(TAOS_RES *res); -DLL_EXPORT int taos_result_precision(TAOS_RES *res); // get the time precision of result -DLL_EXPORT void taos_free_result(TAOS_RES *res); -DLL_EXPORT int taos_field_count(TAOS_RES *res); -DLL_EXPORT int taos_num_fields(TAOS_RES *res); -DLL_EXPORT int taos_affected_rows(TAOS_RES *res); +DLL_EXPORT void taos_cleanup(void); +DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...); +DLL_EXPORT setConfRet taos_set_config(const char *config); +DLL_EXPORT int taos_init(void); +DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port); +DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port); +DLL_EXPORT void taos_close(TAOS *taos); + +const char *taos_data_type(int type); + +DLL_EXPORT TAOS_STMT *taos_stmt_init(TAOS *taos); +DLL_EXPORT int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length); +DLL_EXPORT int taos_stmt_set_tbname_tags(TAOS_STMT *stmt, const char *name, TAOS_MULTI_BIND *tags); +DLL_EXPORT int taos_stmt_set_tbname(TAOS_STMT *stmt, const char *name); +DLL_EXPORT int taos_stmt_set_tags(TAOS_STMT *stmt, TAOS_MULTI_BIND *tags); +DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name); +DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); +DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields); + +DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert); +DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums); +DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes); +DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); +DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind); +DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx); +DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt); +DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt); +DLL_EXPORT char *taos_stmt_errstr(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt); +DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt); + +DLL_EXPORT TAOS_RES *taos_query(TAOS *taos, const char *sql); + +DLL_EXPORT TAOS_ROW taos_fetch_row(TAOS_RES *res); +DLL_EXPORT int taos_result_precision(TAOS_RES *res); // get the time precision of result +DLL_EXPORT void taos_free_result(TAOS_RES *res); +DLL_EXPORT int taos_field_count(TAOS_RES *res); +DLL_EXPORT int taos_num_fields(TAOS_RES *res); +DLL_EXPORT int taos_affected_rows(TAOS_RES *res); DLL_EXPORT TAOS_FIELD *taos_fetch_fields(TAOS_RES *res); DLL_EXPORT int taos_select_db(TAOS *taos, const char *db); @@ -181,8 +181,8 @@ DLL_EXPORT int *taos_get_column_data_offset(TAOS_RES *res, int columnInde DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql); DLL_EXPORT void taos_reset_current_db(TAOS *taos); -DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res); -DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res); +DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res); +DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res); DLL_EXPORT const char *taos_get_server_info(TAOS *taos); DLL_EXPORT const char *taos_get_client_info(); @@ -192,8 +192,8 @@ DLL_EXPORT int taos_errno(TAOS_RES *tres); DLL_EXPORT void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param); DLL_EXPORT void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param); -DLL_EXPORT void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param); -DLL_EXPORT const void *taos_get_raw_block(TAOS_RES* res); +DLL_EXPORT void taos_fetch_raw_block_a(TAOS_RES *res, __taos_async_fn_t fp, void *param); +DLL_EXPORT const void *taos_get_raw_block(TAOS_RES *res); // Shuduo: temporary enable for app build #if 1 @@ -209,21 +209,11 @@ DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLi /* --------------------------TMQ INTERFACE------------------------------- */ -enum { - TMQ_RESP_ERR__FAIL = -1, - TMQ_RESP_ERR__SUCCESS = 0, -}; - -typedef int32_t tmq_resp_err_t; - -typedef struct tmq_t tmq_t; -typedef struct tmq_topic_vgroup_t tmq_topic_vgroup_t; -typedef struct tmq_topic_vgroup_list_t tmq_topic_vgroup_list_t; - +typedef struct tmq_t tmq_t; typedef struct tmq_conf_t tmq_conf_t; typedef struct tmq_list_t tmq_list_t; -typedef void(tmq_commit_cb(tmq_t *, tmq_resp_err_t, tmq_topic_vgroup_list_t *, void *param)); +typedef void(tmq_commit_cb(tmq_t *, int32_t code, void *param)); DLL_EXPORT tmq_list_t *tmq_list_new(); DLL_EXPORT int32_t tmq_list_append(tmq_list_t *, const char *); @@ -233,25 +223,17 @@ DLL_EXPORT char **tmq_list_to_c_array(const tmq_list_t *); DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen); -DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t); +DLL_EXPORT const char *tmq_err2str(int32_t code); /* ------------------------TMQ CONSUMER INTERFACE------------------------ */ -DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list); -DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq); -DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics); - -// timeout: -1 means infinitely waiting +DLL_EXPORT int32_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list); +DLL_EXPORT int32_t tmq_unsubscribe(tmq_t *tmq); +DLL_EXPORT int32_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics); DLL_EXPORT TAOS_RES *tmq_consumer_poll(tmq_t *tmq, int64_t timeout); - -DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq); -DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets); -DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param); - -#if 0 -DLL_EXPORT tmq_resp_err_t tmq_commit(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, int32_t async); -DLL_EXPORT tmq_resp_err_t tmq_seek(tmq_t *tmq, const tmq_topic_vgroup_t *offset); -#endif +DLL_EXPORT int32_t tmq_consumer_close(tmq_t *tmq); +DLL_EXPORT int32_t tmq_commit_sync(tmq_t *tmq, const TAOS_RES *msg); +DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const TAOS_RES *msg, tmq_commit_cb *cb, void *param); /* ----------------------TMQ CONFIGURATION INTERFACE---------------------- */ @@ -275,11 +257,6 @@ DLL_EXPORT const char *tmq_get_db_name(TAOS_RES *res); DLL_EXPORT int32_t tmq_get_vgroup_id(TAOS_RES *res); DLL_EXPORT const char *tmq_get_table_name(TAOS_RES *res); -#if 0 -DLL_EXPORT int64_t tmq_get_request_offset(tmq_message_t *message); -DLL_EXPORT int64_t tmq_get_response_offset(tmq_message_t *message); -#endif - /* ------------------------------ TMQ END -------------------------------- */ #if 1 // Shuduo: temporary enable for app build diff --git a/include/common/tcommon.h b/include/common/tcommon.h index 2e646f4769607947102be1dbae767410a167a6e8..a05287761ef5dbf8d28c075bbc6cb9cac9910651 100644 --- a/include/common/tcommon.h +++ b/include/common/tcommon.h @@ -35,6 +35,7 @@ enum { TMQ_MSG_TYPE__DUMMY = 0, TMQ_MSG_TYPE__POLL_RSP, TMQ_MSG_TYPE__EP_RSP, + TMQ_MSG_TYPE__END_RSP, }; typedef enum EStreamType { diff --git a/include/common/tglobal.h b/include/common/tglobal.h index 1b44b6d7ea226c79e8adde0cbc076bb593be8958..8c03d3ff42b5b0b28f8f1958acfcd7dc8520d464 100644 --- a/include/common/tglobal.h +++ b/include/common/tglobal.h @@ -61,6 +61,7 @@ extern int32_t tsNumOfRpcThreads; extern int32_t tsNumOfCommitThreads; extern int32_t tsNumOfTaskQueueThreads; extern int32_t tsNumOfMnodeQueryThreads; +extern int32_t tsNumOfMnodeFetchThreads; extern int32_t tsNumOfMnodeReadThreads; extern int32_t tsNumOfVnodeQueryThreads; extern int32_t tsNumOfVnodeFetchThreads; diff --git a/include/common/tmsg.h b/include/common/tmsg.h index ab0203adf869c57730099a4e51a089ecca279d50..fd9ff3eb6535008c88037d76bbf5ce4aff6c0a43 100644 --- a/include/common/tmsg.h +++ b/include/common/tmsg.h @@ -106,6 +106,7 @@ typedef enum _mgmt_table { TSDB_MGMT_TABLE_CONNS, TSDB_MGMT_TABLE_QUERIES, TSDB_MGMT_TABLE_VNODES, + TSDB_MGMT_TABLE_APPS, TSDB_MGMT_TABLE_MAX, } EShowType; @@ -428,8 +429,10 @@ STSchema* tdGetSTSChemaFromSSChema(SSchema** pSchema, int32_t nCols); typedef struct { char name[TSDB_TABLE_FNAME_LEN]; int8_t igExists; - float xFilesFactor; - int32_t delay; + int64_t delay1; + int64_t delay2; + int64_t watermark1; + int64_t watermark2; int32_t ttl; int32_t numOfColumns; int32_t numOfTags; @@ -1314,8 +1317,6 @@ int32_t tSerializeSSetStandbyReq(void* buf, int32_t bufLen, SSetStandbyReq* pReq int32_t tDeserializeSSetStandbyReq(void* buf, int32_t bufLen, SSetStandbyReq* pReq); typedef struct { - int32_t connId; // todo remove - int32_t queryId; // todo remove char queryStrId[TSDB_QUERY_ID_LEN]; } SKillQueryReq; @@ -1323,7 +1324,7 @@ int32_t tSerializeSKillQueryReq(void* buf, int32_t bufLen, SKillQueryReq* pReq); int32_t tDeserializeSKillQueryReq(void* buf, int32_t bufLen, SKillQueryReq* pReq); typedef struct { - int32_t connId; + uint32_t connId; } SKillConnReq; int32_t tSerializeSKillConnReq(void* buf, int32_t bufLen, SKillConnReq* pReq); @@ -1775,12 +1776,10 @@ typedef struct { } SDDropTopicReq; typedef struct { - float xFilesFactor; - int32_t delay; - int32_t qmsg1Len; - int32_t qmsg2Len; - char* qmsg1; // pAst1:qmsg1:SRetention1 => trigger aggr task1 - char* qmsg2; // pAst2:qmsg2:SRetention2 => trigger aggr task2 + int64_t maxdelay[2]; + int64_t watermark[2]; + int32_t qmsgLen[2]; + char* qmsg[2]; // pAst:qmsg:SRetention => trigger aggr task1/2 } SRSmaParam; int32_t tEncodeSRSmaParam(SEncoder* pCoder, const SRSmaParam* pRSmaParam); @@ -1816,6 +1815,8 @@ typedef struct SVCreateTbReq { tb_uid_t uid; int64_t ctime; int32_t ttl; + int32_t commentLen; + char* comment; int8_t type; union { struct { @@ -1833,6 +1834,7 @@ int tDecodeSVCreateTbReq(SDecoder* pCoder, SVCreateTbReq* pReq); static FORCE_INLINE void tdDestroySVCreateTbReq(SVCreateTbReq* req) { taosMemoryFreeClear(req->name); + taosMemoryFreeClear(req->comment); if (req->type == TSDB_CHILD_TABLE) { taosMemoryFreeClear(req->ctb.pTag); } else if (req->type == TSDB_NORMAL_TABLE) { @@ -1929,7 +1931,7 @@ typedef struct { // TSDB_ALTER_TABLE_UPDATE_OPTIONS int8_t updateTTL; int32_t newTTL; - int8_t updateComment; + int32_t newCommentLen; char* newComment; } SVAlterTbReq; @@ -2010,7 +2012,6 @@ typedef struct { int64_t useconds; int64_t stime; // timestamp precision ms int64_t reqRid; - int32_t pid; bool stableQuery; char fqdn[TSDB_FQDN_LEN]; int32_t subPlanNum; @@ -2019,8 +2020,6 @@ typedef struct { typedef struct { uint32_t connId; - int32_t pid; - char app[TSDB_APP_NAME_LEN]; SArray* queryDesc; // SArray } SQueryHbReqBasic; @@ -2035,9 +2034,31 @@ typedef struct { SArray* pQnodeList; } SQueryHbRspBasic; +typedef struct SAppClusterSummary { + uint64_t numOfInsertsReq; + uint64_t numOfInsertRows; + uint64_t insertElapsedTime; + uint64_t insertBytes; // submit to tsdb since launched. + + uint64_t fetchBytes; + uint64_t queryElapsedTime; + uint64_t numOfSlowQueries; + uint64_t totalRequests; + uint64_t currentRequests; // the number of SRequestObj +} SAppClusterSummary; + +typedef struct { + int64_t appId; + int32_t pid; + char name[TSDB_APP_NAME_LEN]; + int64_t startTime; + SAppClusterSummary summary; +} SAppHbReq; + typedef struct { SClientHbKey connKey; int64_t clusterId; + SAppHbReq app; SQueryHbReqBasic* query; SHashObj* info; // hash } SClientHbReq; diff --git a/include/common/tmsgcb.h b/include/common/tmsgcb.h index e99377f9b4b27871506d1739520060b8caa51417..b56f7552666dd23650f9911dc23e03b9ae2fa707 100644 --- a/include/common/tmsgcb.h +++ b/include/common/tmsgcb.h @@ -34,7 +34,6 @@ typedef enum { WRITE_QUEUE, APPLY_QUEUE, SYNC_QUEUE, - MERGE_QUEUE, QUEUE_MAX, } EQueueType; diff --git a/include/common/tmsgdef.h b/include/common/tmsgdef.h index f89f548fc5b97bca0fe6cfb0164debda72627c77..eeac619105c1ff9bc909833c02d6d4ba2e41352e 100644 --- a/include/common/tmsgdef.h +++ b/include/common/tmsgdef.h @@ -198,7 +198,7 @@ enum { TD_DEF_MSG_TYPE(TDMT_VND_ALTER_CONFIRM, "alter-confirm", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_VND_ALTER_HASHRANGE, "alter-hashrange", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_VND_COMPACT, "compact", NULL, NULL) - + TD_DEF_MSG_TYPE(TDMT_VND_DROP_TTL_TABLE, "drop-ttl-stb", NULL, NULL) TD_NEW_MSG_SEG(TDMT_QND_MSG) //shared by snode and vnode @@ -236,6 +236,7 @@ enum { TD_DEF_MSG_TYPE(TDMT_SYNC_COMMON_RESPONSE, "sync-common-response", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_SYNC_APPLY_MSG, "sync-apply-msg", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_SYNC_CONFIG_CHANGE, "sync-config-change", NULL, NULL) + TD_DEF_MSG_TYPE(TDMT_SYNC_CONFIG_CHANGE_FINISH, "sync-config-change-finish", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_SYNC_SNAPSHOT_SEND, "sync-snapshot-send", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_SYNC_SNAPSHOT_RSP, "sync-snapshot-rsp", NULL, NULL) TD_DEF_MSG_TYPE(TDMT_SYNC_LEADER_TRANSFER, "sync-leader-transfer", NULL, NULL) diff --git a/include/common/ttokendef.h b/include/common/ttokendef.h index 061c734484b22bb5cb0eb3ea349fcb4766e3e3ae..35b4a5ce7de5301941e9013e55138d65d964616f 100644 --- a/include/common/ttokendef.h +++ b/include/common/ttokendef.h @@ -127,92 +127,92 @@ #define TK_BLOB 109 #define TK_VARBINARY 110 #define TK_DECIMAL 111 -#define TK_FILE_FACTOR 112 -#define TK_NK_FLOAT 113 +#define TK_MAX_DELAY 112 +#define TK_WATERMARK 113 #define TK_ROLLUP 114 #define TK_TTL 115 #define TK_SMA 116 -#define TK_SHOW 117 -#define TK_DATABASES 118 -#define TK_TABLES 119 -#define TK_STABLES 120 -#define TK_MNODES 121 -#define TK_MODULES 122 -#define TK_QNODES 123 -#define TK_FUNCTIONS 124 -#define TK_INDEXES 125 -#define TK_ACCOUNTS 126 -#define TK_APPS 127 -#define TK_CONNECTIONS 128 -#define TK_LICENCE 129 -#define TK_GRANTS 130 -#define TK_QUERIES 131 -#define TK_SCORES 132 -#define TK_TOPICS 133 -#define TK_VARIABLES 134 -#define TK_BNODES 135 -#define TK_SNODES 136 -#define TK_CLUSTER 137 -#define TK_TRANSACTIONS 138 -#define TK_LIKE 139 -#define TK_INDEX 140 -#define TK_FULLTEXT 141 -#define TK_FUNCTION 142 -#define TK_INTERVAL 143 -#define TK_TOPIC 144 -#define TK_AS 145 -#define TK_CONSUMER 146 -#define TK_GROUP 147 -#define TK_DESC 148 -#define TK_DESCRIBE 149 -#define TK_RESET 150 -#define TK_QUERY 151 -#define TK_CACHE 152 -#define TK_EXPLAIN 153 -#define TK_ANALYZE 154 -#define TK_VERBOSE 155 -#define TK_NK_BOOL 156 -#define TK_RATIO 157 -#define TK_COMPACT 158 -#define TK_VNODES 159 -#define TK_IN 160 -#define TK_OUTPUTTYPE 161 -#define TK_AGGREGATE 162 -#define TK_BUFSIZE 163 -#define TK_STREAM 164 -#define TK_INTO 165 -#define TK_TRIGGER 166 -#define TK_AT_ONCE 167 -#define TK_WINDOW_CLOSE 168 -#define TK_MAX_DELAY 169 -#define TK_WATERMARK 170 -#define TK_KILL 171 -#define TK_CONNECTION 172 -#define TK_TRANSACTION 173 -#define TK_BALANCE 174 -#define TK_VGROUP 175 -#define TK_MERGE 176 -#define TK_REDISTRIBUTE 177 -#define TK_SPLIT 178 -#define TK_SYNCDB 179 -#define TK_DELETE 180 -#define TK_NULL 181 -#define TK_NK_QUESTION 182 -#define TK_NK_ARROW 183 -#define TK_ROWTS 184 -#define TK_TBNAME 185 -#define TK_QSTARTTS 186 -#define TK_QENDTS 187 -#define TK_WSTARTTS 188 -#define TK_WENDTS 189 -#define TK_WDURATION 190 -#define TK_CAST 191 -#define TK_NOW 192 -#define TK_TODAY 193 -#define TK_TIMEZONE 194 -#define TK_COUNT 195 -#define TK_FIRST 196 -#define TK_LAST 197 +#define TK_FIRST 117 +#define TK_LAST 118 +#define TK_SHOW 119 +#define TK_DATABASES 120 +#define TK_TABLES 121 +#define TK_STABLES 122 +#define TK_MNODES 123 +#define TK_MODULES 124 +#define TK_QNODES 125 +#define TK_FUNCTIONS 126 +#define TK_INDEXES 127 +#define TK_ACCOUNTS 128 +#define TK_APPS 129 +#define TK_CONNECTIONS 130 +#define TK_LICENCE 131 +#define TK_GRANTS 132 +#define TK_QUERIES 133 +#define TK_SCORES 134 +#define TK_TOPICS 135 +#define TK_VARIABLES 136 +#define TK_BNODES 137 +#define TK_SNODES 138 +#define TK_CLUSTER 139 +#define TK_TRANSACTIONS 140 +#define TK_DISTRIBUTED 141 +#define TK_LIKE 142 +#define TK_INDEX 143 +#define TK_FULLTEXT 144 +#define TK_FUNCTION 145 +#define TK_INTERVAL 146 +#define TK_TOPIC 147 +#define TK_AS 148 +#define TK_CONSUMER 149 +#define TK_GROUP 150 +#define TK_DESC 151 +#define TK_DESCRIBE 152 +#define TK_RESET 153 +#define TK_QUERY 154 +#define TK_CACHE 155 +#define TK_EXPLAIN 156 +#define TK_ANALYZE 157 +#define TK_VERBOSE 158 +#define TK_NK_BOOL 159 +#define TK_RATIO 160 +#define TK_NK_FLOAT 161 +#define TK_COMPACT 162 +#define TK_VNODES 163 +#define TK_IN 164 +#define TK_OUTPUTTYPE 165 +#define TK_AGGREGATE 166 +#define TK_BUFSIZE 167 +#define TK_STREAM 168 +#define TK_INTO 169 +#define TK_TRIGGER 170 +#define TK_AT_ONCE 171 +#define TK_WINDOW_CLOSE 172 +#define TK_KILL 173 +#define TK_CONNECTION 174 +#define TK_TRANSACTION 175 +#define TK_BALANCE 176 +#define TK_VGROUP 177 +#define TK_MERGE 178 +#define TK_REDISTRIBUTE 179 +#define TK_SPLIT 180 +#define TK_SYNCDB 181 +#define TK_DELETE 182 +#define TK_NULL 183 +#define TK_NK_QUESTION 184 +#define TK_NK_ARROW 185 +#define TK_ROWTS 186 +#define TK_TBNAME 187 +#define TK_QSTARTTS 188 +#define TK_QENDTS 189 +#define TK_WSTARTTS 190 +#define TK_WENDTS 191 +#define TK_WDURATION 192 +#define TK_CAST 193 +#define TK_NOW 194 +#define TK_TODAY 195 +#define TK_TIMEZONE 196 +#define TK_COUNT 197 #define TK_LAST_ROW 198 #define TK_BETWEEN 199 #define TK_IS 200 @@ -241,20 +241,22 @@ #define TK_LINEAR 223 #define TK_NEXT 224 #define TK_HAVING 225 -#define TK_ORDER 226 -#define TK_SLIMIT 227 -#define TK_SOFFSET 228 -#define TK_LIMIT 229 -#define TK_OFFSET 230 -#define TK_ASC 231 -#define TK_NULLS 232 -#define TK_ID 233 -#define TK_NK_BITNOT 234 -#define TK_INSERT 235 -#define TK_VALUES 236 -#define TK_IMPORT 237 -#define TK_NK_SEMI 238 -#define TK_FILE 239 +#define TK_RANGE 226 +#define TK_EVERY 227 +#define TK_ORDER 228 +#define TK_SLIMIT 229 +#define TK_SOFFSET 230 +#define TK_LIMIT 231 +#define TK_OFFSET 232 +#define TK_ASC 233 +#define TK_NULLS 234 +#define TK_ID 235 +#define TK_NK_BITNOT 236 +#define TK_INSERT 237 +#define TK_VALUES 238 +#define TK_IMPORT 239 +#define TK_NK_SEMI 240 +#define TK_FILE 241 #define TK_NK_SPACE 300 #define TK_NK_COMMENT 301 diff --git a/include/dnode/mnode/mnode.h b/include/dnode/mnode/mnode.h index 21d8405f582a7f3e1e47cb137a34b3fae71cc8f3..f1cfa58f62f8b80dc7f3725fcba6e0d3395e3d39 100644 --- a/include/dnode/mnode/mnode.h +++ b/include/dnode/mnode/mnode.h @@ -96,6 +96,7 @@ int32_t mndGetLoad(SMnode *pMnode, SMnodeLoad *pLoad); int32_t mndProcessRpcMsg(SRpcMsg *pMsg); int32_t mndProcessSyncMsg(SRpcMsg *pMsg); int32_t mndPreProcessMsg(SRpcMsg *pMsg); +void mndAbortPreprocessMsg(SRpcMsg *pMsg); /** * @brief Generate machine code diff --git a/include/dnode/snode/snode.h b/include/dnode/snode/snode.h index 611bff49f116c3a10363c607474ebb90eb275186..3d0ef2e052ef20978156b76b1dd7d552391b4a8b 100644 --- a/include/dnode/snode/snode.h +++ b/include/dnode/snode/snode.h @@ -16,8 +16,8 @@ #ifndef _TD_SNODE_H_ #define _TD_SNODE_H_ -#include "tmsgcb.h" #include "tmsg.h" +#include "tmsgcb.h" #include "trpc.h" #ifdef __cplusplus @@ -68,8 +68,8 @@ int32_t sndGetLoad(SSnode *pSnode, SSnodeLoad *pLoad); * @param pMsg The request message * @param pRsp The response message */ -void sndProcessUMsg(SSnode *pSnode, SRpcMsg *pMsg); -void sndProcessSMsg(SSnode *pSnode, SRpcMsg *pMsg); +int32_t sndProcessUMsg(SSnode *pSnode, SRpcMsg *pMsg); +int32_t sndProcessSMsg(SSnode *pSnode, SRpcMsg *pMsg); #ifdef __cplusplus } diff --git a/include/libs/catalog/catalog.h b/include/libs/catalog/catalog.h index f3600a509e8be0946b5845634b13c9e6a4a9ed98..a7ac767883a74548755b110c4ab6261075d3a443 100644 --- a/include/libs/catalog/catalog.h +++ b/include/libs/catalog/catalog.h @@ -87,7 +87,7 @@ typedef struct SMetaData { SArray* pUdfList; // pRes = SFuncInfo* SArray* pIndex; // pRes = SIndexInfo* SArray* pUser; // pRes = bool* - SArray* pQnodeList; // pRes = SQueryNodeAddr* + SArray* pQnodeList; // pRes = SArray* } SMetaData; typedef struct SCatalogCfg { @@ -292,6 +292,8 @@ int32_t catalogUpdateVgEpSet(SCatalog* pCtg, const char* dbFName, int32_t vgId, int32_t ctgdLaunchAsyncCall(SCatalog* pCtg, SRequestConnInfo* pConn, uint64_t reqId, bool forceUpdate); +int32_t catalogClearCache(void); + /** * Destroy catalog and relase all resources */ diff --git a/include/libs/executor/executor.h b/include/libs/executor/executor.h index fdedb947d7a9eabaaa6983cb8c0e54b1eb6a8027..083f6ae1b07d4db9e7f4d339b63b8b89c0a8becc 100644 --- a/include/libs/executor/executor.h +++ b/include/libs/executor/executor.h @@ -159,11 +159,14 @@ int64_t qGetQueriedTableUid(qTaskInfo_t tinfo); */ int32_t qGetQualifiedTableIdList(void* pTableList, const char* tagCond, int32_t tagCondLen, SArray* pTableIdList); - void qProcessFetchRsp(void* parent, struct SRpcMsg* pMsg, struct SEpSet* pEpSet); int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t* resNum, SExplainExecInfo** pRes); +int32_t qSerializeTaskStatus(qTaskInfo_t tinfo, char** pOutput, int32_t* len); + +int32_t qDeserializeTaskStatus(qTaskInfo_t tinfo, const char* pInput, int32_t len); + #ifdef __cplusplus } #endif diff --git a/include/libs/function/function.h b/include/libs/function/function.h index 54762a0f17ba45929414adf99f1226d25cac1242..0d8ae289cc4def348556adc5549e948cff7fcda6 100644 --- a/include/libs/function/function.h +++ b/include/libs/function/function.h @@ -158,10 +158,10 @@ typedef struct tExprNode { int32_t nodeType; union { struct {// function node - char functionName[FUNCTIONS_NAME_MAX_LENGTH]; // todo refactor - int32_t functionId; - int32_t num; - struct SFunctionNode *pFunctNode; + char functionName[FUNCTIONS_NAME_MAX_LENGTH]; // todo refactor + int32_t functionId; + int32_t num; + struct SFunctionNode *pFunctNode; } _function; struct { diff --git a/include/libs/function/functionMgt.h b/include/libs/function/functionMgt.h index dcc0a2ba4b5e90762d982532525d30a4ccdeb434..cbaff29cb2de5ab4e93d39caa1cd89810fbdddde 100644 --- a/include/libs/function/functionMgt.h +++ b/include/libs/function/functionMgt.h @@ -121,7 +121,8 @@ typedef enum EFunctionType { // internal function FUNCTION_TYPE_SELECT_VALUE, - FUNCTION_TYPE_BLOCK_DIST, // block distribution aggregate function + FUNCTION_TYPE_BLOCK_DIST, // block distribution aggregate function + FUNCTION_TYPE_BLOCK_DIST_INFO, // block distribution pseudo column function // distributed splitting functions FUNCTION_TYPE_APERCENTILE_PARTIAL = 4000, @@ -185,7 +186,10 @@ bool fmIsUserDefinedFunc(int32_t funcId); bool fmIsDistExecFunc(int32_t funcId); bool fmIsForbidFillFunc(int32_t funcId); bool fmIsForbidStreamFunc(int32_t funcId); +bool fmIsForbidWindowFunc(int32_t funcId); +bool fmIsForbidGroupByFunc(int32_t funcId); bool fmIsIntervalInterpoFunc(int32_t funcId); +bool fmIsInterpFunc(int32_t funcId); int32_t fmGetDistMethod(const SFunctionNode* pFunc, SFunctionNode** pPartialFunc, SFunctionNode** pMergeFunc); diff --git a/include/libs/nodes/cmdnodes.h b/include/libs/nodes/cmdnodes.h index 1282836425e220f67e904089b107bb8853f2f890..13a8dbd7e3e4dc86602aeaff3cb2ee6f0924be9a 100644 --- a/include/libs/nodes/cmdnodes.h +++ b/include/libs/nodes/cmdnodes.h @@ -87,9 +87,14 @@ typedef struct SAlterDatabaseStmt { typedef struct STableOptions { ENodeType type; + bool commentNull; char comment[TSDB_TB_COMMENT_LEN]; - double filesFactor; - int32_t delay; + SNodeList* pMaxDelay; + int64_t maxDelay1; + int64_t maxDelay2; + SNodeList* pWatermark; + int64_t watermark1; + int64_t watermark2; SNodeList* pRollupFuncs; int32_t ttl; SNodeList* pSma; @@ -122,6 +127,7 @@ typedef struct SCreateSubTableClause { bool ignoreExists; SNodeList* pSpecificTags; SNodeList* pValsOfTags; + STableOptions* pOptions; } SCreateSubTableClause; typedef struct SCreateMultiTableStmt { @@ -199,16 +205,30 @@ typedef struct SAlterDnodeStmt { } SAlterDnodeStmt; typedef struct SShowStmt { - ENodeType type; - SNode* pDbName; // SValueNode - SNode* pTbNamePattern; // SValueNode + ENodeType type; + SNode* pDbName; // SValueNode + SNode* pTbName; // SValueNode + EOperatorType tableCondType; } SShowStmt; -typedef struct SShowCreatStmt { +typedef struct SShowCreateDatabaseStmt { + ENodeType type; + char dbName[TSDB_DB_NAME_LEN]; + void* pCfg; // SDbCfgInfo +} SShowCreateDatabaseStmt; + +typedef struct SShowCreateTableStmt { + ENodeType type; + char dbName[TSDB_DB_NAME_LEN]; + char tableName[TSDB_TABLE_NAME_LEN]; + STableMeta* pMeta; +} SShowCreateTableStmt; + +typedef struct SShowTableDistributedStmt { ENodeType type; char dbName[TSDB_DB_NAME_LEN]; char tableName[TSDB_TABLE_NAME_LEN]; -} SShowCreatStmt; +} SShowTableDistributedStmt; typedef enum EIndexType { INDEX_TYPE_SMA = 1, INDEX_TYPE_FULLTEXT } EIndexType; diff --git a/include/libs/nodes/nodes.h b/include/libs/nodes/nodes.h index 3620f454083eba87cfcf92aaf17eecd3e2242c33..975321cd5be179f411694430109a8da933addc67 100644 --- a/include/libs/nodes/nodes.h +++ b/include/libs/nodes/nodes.h @@ -102,6 +102,7 @@ typedef enum ENodeType { QUERY_NODE_EXPLAIN_OPTIONS, QUERY_NODE_STREAM_OPTIONS, QUERY_NODE_LEFT_VALUE, + QUERY_NODE_COLUMN_REF, // Statement nodes are used in parser and planner module. QUERY_NODE_SET_OPERATOR, @@ -184,6 +185,7 @@ typedef enum ENodeType { QUERY_NODE_SHOW_CREATE_TABLE_STMT, QUERY_NODE_SHOW_CREATE_STABLE_STMT, QUERY_NODE_SHOW_TRANSACTIONS_STMT, + QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT, QUERY_NODE_KILL_CONNECTION_STMT, QUERY_NODE_KILL_QUERY_STMT, QUERY_NODE_KILL_TRANSACTION_STMT, @@ -203,6 +205,7 @@ typedef enum ENodeType { QUERY_NODE_LOGIC_PLAN_SORT, QUERY_NODE_LOGIC_PLAN_PARTITION, QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC, + QUERY_NODE_LOGIC_PLAN_INTERP_FUNC, QUERY_NODE_LOGIC_SUBPLAN, QUERY_NODE_LOGIC_PLAN, @@ -213,6 +216,7 @@ typedef enum ENodeType { QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN, QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN, QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN, + QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN, QUERY_NODE_PHYSICAL_PLAN_PROJECT, QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN, QUERY_NODE_PHYSICAL_PLAN_HASH_AGG, @@ -227,11 +231,13 @@ typedef enum ENodeType { QUERY_NODE_PHYSICAL_PLAN_FILL, QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION, QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION, + QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION, QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION, QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE, QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE, QUERY_NODE_PHYSICAL_PLAN_PARTITION, QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC, + QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC, QUERY_NODE_PHYSICAL_PLAN_DISPATCH, QUERY_NODE_PHYSICAL_PLAN_INSERT, QUERY_NODE_PHYSICAL_PLAN_DELETE, diff --git a/include/libs/nodes/plannodes.h b/include/libs/nodes/plannodes.h index 749d58b224bc261b64c7ae8fae07ec150dfcf657..b07e8f39d5bcae86ebd32208613f98c7e2bff497 100644 --- a/include/libs/nodes/plannodes.h +++ b/include/libs/nodes/plannodes.h @@ -39,7 +39,8 @@ typedef enum EScanType { SCAN_TYPE_TABLE, SCAN_TYPE_SYSTEM_TABLE, SCAN_TYPE_STREAM, - SCAN_TYPE_TABLE_MERGE + SCAN_TYPE_TABLE_MERGE, + SCAN_TYPE_BLOCK_INFO } EScanType; typedef struct SScanLogicNode { @@ -97,9 +98,19 @@ typedef struct SProjectLogicNode { typedef struct SIndefRowsFuncLogicNode { SLogicNode node; - SNodeList* pVectorFuncs; + SNodeList* pFuncs; } SIndefRowsFuncLogicNode; +typedef struct SInterpFuncLogicNode { + SLogicNode node; + SNodeList* pFuncs; + STimeWindow timeRange; + int64_t interval; + EFillMode fillMode; + SNode* pFillValues; // SNodeListNode + SNode* pTimeSeries; // SColumnNode +} SInterpFuncLogicNode; + typedef enum EModifyTableType { MODIFY_TABLE_TYPE_INSERT = 1, MODIFY_TABLE_TYPE_DELETE } EModifyTableType; typedef struct SVnodeModifyLogicNode { @@ -130,30 +141,35 @@ typedef struct SMergeLogicNode { typedef enum EWindowType { WINDOW_TYPE_INTERVAL = 1, WINDOW_TYPE_SESSION, WINDOW_TYPE_STATE } EWindowType; -typedef enum EIntervalAlgorithm { +typedef enum EWindowAlgorithm { INTERVAL_ALGO_HASH = 1, INTERVAL_ALGO_MERGE, INTERVAL_ALGO_STREAM_FINAL, INTERVAL_ALGO_STREAM_SEMI, INTERVAL_ALGO_STREAM_SINGLE, -} EIntervalAlgorithm; + SESSION_ALGO_STREAM_SEMI, + SESSION_ALGO_STREAM_FINAL, + SESSION_ALGO_STREAM_SINGLE, + SESSION_ALGO_MERGE, +} EWindowAlgorithm; typedef struct SWindowLogicNode { - SLogicNode node; - EWindowType winType; - SNodeList* pFuncs; - int64_t interval; - int64_t offset; - int64_t sliding; - int8_t intervalUnit; - int8_t slidingUnit; - int64_t sessionGap; - SNode* pTspk; - SNode* pStateExpr; - int8_t triggerType; - int64_t watermark; - double filesFactor; - EIntervalAlgorithm intervalAlgo; + SLogicNode node; + EWindowType winType; + SNodeList* pFuncs; + int64_t interval; + int64_t offset; + int64_t sliding; + int8_t intervalUnit; + int8_t slidingUnit; + int64_t sessionGap; + SNode* pTspk; + SNode* pTsEnd; + SNode* pStateExpr; + int8_t triggerType; + int64_t watermark; + double filesFactor; + EWindowAlgorithm windowAlgo; } SWindowLogicNode; typedef struct SFillLogicNode { @@ -243,6 +259,7 @@ typedef struct SScanPhysiNode { } SScanPhysiNode; typedef SScanPhysiNode STagScanPhysiNode; +typedef SScanPhysiNode SBlockDistScanPhysiNode; typedef struct SSystemTableScanPhysiNode { SScanPhysiNode scan; @@ -286,9 +303,20 @@ typedef struct SProjectPhysiNode { typedef struct SIndefRowsFuncPhysiNode { SPhysiNode node; SNodeList* pExprs; - SNodeList* pVectorFuncs; + SNodeList* pFuncs; } SIndefRowsFuncPhysiNode; +typedef struct SInterpFuncPhysiNode { + SPhysiNode node; + SNodeList* pExprs; + SNodeList* pFuncs; + STimeWindow timeRange; + int64_t interval; + EFillMode fillMode; + SNode* pFillValues; // SNodeListNode + SNode* pTimeSeries; // SColumnNode +} SInterpFuncPhysiNode; + typedef struct SJoinPhysiNode { SPhysiNode node; EJoinType joinType; @@ -331,7 +359,8 @@ typedef struct SWinodwPhysiNode { SPhysiNode node; SNodeList* pExprs; // these are expression list of parameter expression of function SNodeList* pFuncs; - SNode* pTspk; // timestamp primary key + SNode* pTspk; // timestamp primary key + SNode* pTsEnd; // window end timestamp int8_t triggerType; int64_t watermark; double filesFactor; @@ -371,6 +400,8 @@ typedef struct SSessionWinodwPhysiNode { } SSessionWinodwPhysiNode; typedef SSessionWinodwPhysiNode SStreamSessionWinodwPhysiNode; +typedef SSessionWinodwPhysiNode SStreamSemiSessionWinodwPhysiNode; +typedef SSessionWinodwPhysiNode SStreamFinalSessionWinodwPhysiNode; typedef struct SStateWinodwPhysiNode { SWinodwPhysiNode window; diff --git a/include/libs/nodes/querynodes.h b/include/libs/nodes/querynodes.h index 989dcc94929674988ae4c5240e99b2b762912e83..73e487f073ac44d5ddeee0086da203f329b3238d 100644 --- a/include/libs/nodes/querynodes.h +++ b/include/libs/nodes/querynodes.h @@ -65,11 +65,16 @@ typedef struct SColumnNode { char tableName[TSDB_TABLE_NAME_LEN]; char tableAlias[TSDB_TABLE_NAME_LEN]; char colName[TSDB_COL_NAME_LEN]; - SNode* pProjectRef; - int16_t dataBlockId; - int16_t slotId; + // SNode* pProjectRef; + int16_t dataBlockId; + int16_t slotId; } SColumnNode; +typedef struct SColumnRefNode { + ENodeType type; + char colName[TSDB_COL_NAME_LEN]; +} SColumnRefNode; + typedef struct STargetNode { ENodeType type; int16_t dataBlockId; @@ -235,6 +240,9 @@ typedef struct SSelectStmt { SNode* pWindow; SNodeList* pGroupByList; // SGroupingSetNode SNode* pHaving; + SNode* pRange; + SNode* pEvery; + SNode* pFill; SNodeList* pOrderByList; // SOrderByExprNode SLimitNode* pLimit; SLimitNode* pSlimit; @@ -247,6 +255,9 @@ typedef struct SSelectStmt { bool hasIndefiniteRowsFunc; bool hasSelectFunc; bool hasSelectValFunc; + bool hasUniqueFunc; + bool hasTailFunc; + bool hasInterpFunc; } SSelectStmt; typedef enum ESetOperatorType { SET_OP_TYPE_UNION_ALL = 1, SET_OP_TYPE_UNION } ESetOperatorType; @@ -374,6 +385,9 @@ bool nodesIsComparisonOp(const SOperatorNode* pOp); bool nodesIsJsonOp(const SOperatorNode* pOp); bool nodesIsRegularOp(const SOperatorNode* pOp); +bool nodesExprHasColumn(SNode* pNode); +bool nodesExprsHasColumn(SNodeList* pList); + void* nodesGetValueFromNode(SValueNode* pNode); int32_t nodesSetValueNodeValue(SValueNode* pNode, void* value); char* nodesGetStrValueFromNode(SValueNode* pNode); diff --git a/include/libs/planner/planner.h b/include/libs/planner/planner.h index 8ed95b60101472b347b10b285fcacaa3123cce57..c4f71e57a8174c62cf331e4afec35604786282a0 100644 --- a/include/libs/planner/planner.h +++ b/include/libs/planner/planner.h @@ -36,7 +36,6 @@ typedef struct SPlanContext { int64_t watermark; char* pMsg; int32_t msgLen; - // double filesFactor; } SPlanContext; // Create the physical plan for the query, according to the AST. diff --git a/include/libs/qcom/query.h b/include/libs/qcom/query.h index 89eb21556afaa4af30ba0d7636b3c9402d9342a1..d562d07d776052d9e6a797da9adfe507e05ccfd3 100644 --- a/include/libs/qcom/query.h +++ b/include/libs/qcom/query.h @@ -33,8 +33,6 @@ typedef enum { JOB_TASK_STATUS_PARTIAL_SUCCEED, JOB_TASK_STATUS_SUCCEED, JOB_TASK_STATUS_FAILED, - JOB_TASK_STATUS_CANCELLING, - JOB_TASK_STATUS_CANCELLED, JOB_TASK_STATUS_DROPPING, } EJobTaskType; @@ -49,6 +47,10 @@ typedef enum { TARGET_TYPE_OTHER, } ETargetType; +#define QUERY_POLICY_VNODE 1 +#define QUERY_POLICY_HYBRID 2 +#define QUERY_POLICY_QNODE 3 + typedef struct STableComInfo { uint8_t numOfTags; // the number of tags in schema uint8_t precision; // the number of precision diff --git a/include/libs/qworker/qworker.h b/include/libs/qworker/qworker.h index f3f147955a03cb99f1ea806441eddd63ba8f96fe..3b8a37f4208386eb06ff4f3153fab271e3ddb011 100644 --- a/include/libs/qworker/qworker.h +++ b/include/libs/qworker/qworker.h @@ -64,6 +64,8 @@ typedef struct { int32_t qWorkerInit(int8_t nodeType, int32_t nodeId, SQWorkerCfg *cfg, void **qWorkerMgmt, const SMsgCb *pMsgCb); +int32_t qWorkerAbortPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg); + int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg); int32_t qWorkerProcessQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts); diff --git a/include/libs/scheduler/scheduler.h b/include/libs/scheduler/scheduler.h index 8087a6d33cd14e818355651b28b940ede4a0ef37..ecb21335b9cf9edb566c1861042fcaffd6da13c4 100644 --- a/include/libs/scheduler/scheduler.h +++ b/include/libs/scheduler/scheduler.h @@ -73,13 +73,14 @@ typedef void (*schedulerExecCallback)(SQueryResult* pResult, void* param, int32_ typedef void (*schedulerFetchCallback)(void* pResult, void* param, int32_t code); typedef struct SSchedulerReq { - SRequestConnInfo *pConn; - SArray *pNodeList; - SQueryPlan *pDag; - const char *sql; - int64_t startTs; + bool *reqKilled; + SRequestConnInfo *pConn; + SArray *pNodeList; + SQueryPlan *pDag; + const char *sql; + int64_t startTs; schedulerExecCallback fp; - void* cbParam; + void* cbParam; } SSchedulerReq; @@ -127,7 +128,7 @@ void schedulerStopQueryHb(void *pTrans); * Free the query job * @param pJob */ -void schedulerFreeJob(int64_t job); +void schedulerFreeJob(int64_t job, int32_t errCode); void schedulerDestroy(void); diff --git a/include/libs/stream/tstream.h b/include/libs/stream/tstream.h index 2c9d66a828e658cd286d2e5bf4edcc28d7ea37a9..937ac2b408ed0e720d24ae648f920352c7eb5712 100644 --- a/include/libs/stream/tstream.h +++ b/include/libs/stream/tstream.h @@ -152,7 +152,6 @@ void* streamDataBlockDecode(const void* buf, SStreamDataBlock* pInput); typedef struct { char* qmsg; // followings are not applicable to encoder and decoder - void* inputHandle; void* executor; } STaskExec; @@ -240,12 +239,13 @@ struct SStreamTask { int8_t inputType; int8_t status; - int8_t sourceType; int8_t execType; int8_t sinkType; int8_t dispatchType; int16_t dispatchMsgType; + int8_t dataScan; + // node info int32_t childId; int32_t nodeId; @@ -399,15 +399,13 @@ typedef struct { int32_t tDecodeStreamDispatchReq(SDecoder* pDecoder, SStreamDispatchReq* pReq); -int32_t streamLaunchByWrite(SStreamTask* pTask, int32_t vgId, SMsgCb* pMsgCb); +int32_t streamLaunchByWrite(SStreamTask* pTask, int32_t vgId); int32_t streamSetupTrigger(SStreamTask* pTask); -int32_t streamTaskRun(SStreamTask* pTask); - -int32_t streamTaskProcessRunReq(SStreamTask* pTask, SMsgCb* pMsgCb); -int32_t streamProcessDispatchReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDispatchReq* pReq, SRpcMsg* pMsg); -int32_t streamProcessDispatchRsp(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDispatchRsp* pRsp); -int32_t streamProcessRecoverReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamTaskRecoverReq* pReq, SRpcMsg* pMsg); +int32_t streamProcessRunReq(SStreamTask* pTask); +int32_t streamProcessDispatchReq(SStreamTask* pTask, SStreamDispatchReq* pReq, SRpcMsg* pMsg); +int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp); +int32_t streamProcessRecoverReq(SStreamTask* pTask, SStreamTaskRecoverReq* pReq, SRpcMsg* pMsg); int32_t streamProcessRecoverRsp(SStreamTask* pTask, SStreamTaskRecoverRsp* pRsp); #ifdef __cplusplus diff --git a/include/libs/sync/sync.h b/include/libs/sync/sync.h index fc9d419b1f901a48edfae1aef8acc039bbadb57b..e963f25616cc92ebb64a75c9de6daecca50c8116 100644 --- a/include/libs/sync/sync.h +++ b/include/libs/sync/sync.h @@ -24,6 +24,8 @@ extern "C" { #include "tdef.h" #include "tmsgcb.h" +extern bool gRaftDetailLog; + #define SYNC_INDEX_BEGIN 0 #define SYNC_INDEX_INVALID -1 @@ -61,26 +63,35 @@ typedef struct SSyncCfg { } SSyncCfg; typedef struct SFsmCbMeta { - SyncIndex index; - bool isWeak; int32_t code; - ESyncState state; - uint64_t seqNum; + SyncIndex index; SyncTerm term; + uint64_t seqNum; + SyncIndex lastConfigIndex; + ESyncState state; SyncTerm currentTerm; + bool isWeak; uint64_t flag; } SFsmCbMeta; typedef struct SReConfigCbMeta { - int32_t code; - SyncIndex index; - SyncTerm term; - SyncTerm currentTerm; + int32_t code; + SyncIndex index; + SyncTerm term; + uint64_t seqNum; + SyncIndex lastConfigIndex; + ESyncState state; + SyncTerm currentTerm; + bool isWeak; + uint64_t flag; + + // config info SSyncCfg oldCfg; SSyncCfg newCfg; - bool isDrop; - uint64_t flag; - uint64_t seqNum; + SyncIndex newCfgIndex; + SyncTerm newCfgTerm; + uint64_t newCfgSeqNum; + } SReConfigCbMeta; typedef struct SSnapshot { @@ -105,7 +116,8 @@ typedef struct SSyncFSM { void (*FpReConfigCb)(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SReConfigCbMeta cbMeta); void (*FpLeaderTransferCb)(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SFsmCbMeta cbMeta); - int32_t (*FpGetSnapshot)(struct SSyncFSM* pFsm, SSnapshot* pSnapshot); + int32_t (*FpGetSnapshot)(struct SSyncFSM* pFsm, SSnapshot* pSnapshot, void* pReaderParam, void** ppReader); + int32_t (*FpGetSnapshotInfo)(struct SSyncFSM* pFsm, SSnapshot* pSnapshot); int32_t (*FpSnapshotStartRead)(struct SSyncFSM* pFsm, void** ppReader); int32_t (*FpSnapshotStopRead)(struct SSyncFSM* pFsm, void* pReader); @@ -185,13 +197,12 @@ ESyncState syncGetMyRole(int64_t rid); bool syncIsReady(int64_t rid); const char* syncGetMyRoleStr(int64_t rid); SyncTerm syncGetMyTerm(int64_t rid); +SyncGroupId syncGetVgId(int64_t rid); void syncGetEpSet(int64_t rid, SEpSet* pEpSet); -int32_t syncGetVgId(int64_t rid); int32_t syncPropose(int64_t rid, const SRpcMsg* pMsg, bool isWeak); bool syncEnvIsStart(); const char* syncStr(ESyncState state); bool syncIsRestoreFinish(int64_t rid); -int32_t syncGetSnapshotMeta(int64_t rid, struct SSnapshotMeta* sMeta); int32_t syncReconfig(int64_t rid, const SSyncCfg* pNewCfg); diff --git a/include/libs/sync/syncTools.h b/include/libs/sync/syncTools.h index fbf124bf4748e716b28cd4e449f7cfdc5f975a84..37b465e56ea18420cdb4880099c4b2f8bdf4c322 100644 --- a/include/libs/sync/syncTools.h +++ b/include/libs/sync/syncTools.h @@ -489,6 +489,40 @@ void syncLeaderTransferPrint2(char* s, const SyncLeaderTransfer* pMsg); void syncLeaderTransferLog(const SyncLeaderTransfer* pMsg); void syncLeaderTransferLog2(char* s, const SyncLeaderTransfer* pMsg); + +// --------------------------------------------- +typedef struct SyncReconfigFinish { + uint32_t bytes; + int32_t vgId; + uint32_t msgType; + SSyncCfg oldCfg; + SSyncCfg newCfg; + SyncIndex newCfgIndex; + SyncTerm newCfgTerm; + uint64_t newCfgSeqNum; + +} SyncReconfigFinish; + +SyncReconfigFinish* syncReconfigFinishBuild(int32_t vgId); +void syncReconfigFinishDestroy(SyncReconfigFinish* pMsg); +void syncReconfigFinishSerialize(const SyncReconfigFinish* pMsg, char* buf, uint32_t bufLen); +void syncReconfigFinishDeserialize(const char* buf, uint32_t len, SyncReconfigFinish* pMsg); +char* syncReconfigFinishSerialize2(const SyncReconfigFinish* pMsg, uint32_t* len); +SyncReconfigFinish* syncReconfigFinishDeserialize2(const char* buf, uint32_t len); +void syncReconfigFinish2RpcMsg(const SyncReconfigFinish* pMsg, SRpcMsg* pRpcMsg); +void syncReconfigFinishFromRpcMsg(const SRpcMsg* pRpcMsg, SyncReconfigFinish* pMsg); +SyncReconfigFinish* syncReconfigFinishFromRpcMsg2(const SRpcMsg* pRpcMsg); +cJSON* syncReconfigFinish2Json(const SyncReconfigFinish* pMsg); +char* syncReconfigFinish2Str(const SyncReconfigFinish* pMsg); + +// for debug ---------------------- +void syncReconfigFinishPrint(const SyncReconfigFinish* pMsg); +void syncReconfigFinishPrint2(char* s, const SyncReconfigFinish* pMsg); +void syncReconfigFinishLog(const SyncReconfigFinish* pMsg); +void syncReconfigFinishLog2(char* s, const SyncReconfigFinish* pMsg); + + + // on message ---------------------- int32_t syncNodeOnPingCb(SSyncNode* ths, SyncPing* pMsg); int32_t syncNodeOnPingReplyCb(SSyncNode* ths, SyncPingReply* pMsg); diff --git a/include/libs/transport/trpc.h b/include/libs/transport/trpc.h index fd57eef83ae19c00de8415cafff7764ef34ecdc4..77de5f47562897815e6cd8633c09345dee82794a 100644 --- a/include/libs/transport/trpc.h +++ b/include/libs/transport/trpc.h @@ -23,6 +23,7 @@ extern "C" { #include #include "taosdef.h" #include "tmsg.h" +#include "ttrace.h" #define TAOS_CONN_SERVER 0 #define TAOS_CONN_CLIENT 1 @@ -41,12 +42,13 @@ typedef struct { typedef struct SRpcHandleInfo { // rpc info - void * handle; // rpc handle returned to app - int64_t refId; // refid, used by server - int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp); - int32_t persistHandle; // persist handle or not + void * handle; // rpc handle returned to app + int64_t refId; // refid, used by server + int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp); + int32_t persistHandle; // persist handle or not + STraceId traceId; + // int64_t traceId; - SRpcConnInfo connInfo; // app info void *ahandle; // app handle set by client void *wrapper; // wrapper handle @@ -55,6 +57,9 @@ typedef struct SRpcHandleInfo { // resp info void * rsp; int32_t rspLen; + + // conn info + SRpcConnInfo conn; } SRpcHandleInfo; typedef struct SRpcMsg { @@ -63,7 +68,6 @@ typedef struct SRpcMsg { int32_t contLen; int32_t code; SRpcHandleInfo info; - SRpcConnInfo conn; } SRpcMsg; typedef void (*RpcCfp)(void *parent, SRpcMsg *, SEpSet *rf); diff --git a/include/os/osTime.h b/include/os/osTime.h index b9e407cbcf0f664236f87dc01e7a1279e2a39150..949c15ed0d7c7a4e4c317edaace21f29a9eaab36 100644 --- a/include/os/osTime.h +++ b/include/os/osTime.h @@ -23,22 +23,22 @@ extern "C" { // If the error is in a third-party library, place this header file under the third-party library header file. // When you want to use this feature, you should find or add the same function in the following section. #ifndef ALLOW_FORBID_FUNC - #define strptime STRPTIME_FUNC_TAOS_FORBID - #define gettimeofday GETTIMEOFDAY_FUNC_TAOS_FORBID - #define localtime LOCALTIME_FUNC_TAOS_FORBID - #define localtime_s LOCALTIMES_FUNC_TAOS_FORBID - #define localtime_r LOCALTIMER_FUNC_TAOS_FORBID - #define time TIME_FUNC_TAOS_FORBID - #define mktime MKTIME_FUNC_TAOS_FORBID +#define strptime STRPTIME_FUNC_TAOS_FORBID +#define gettimeofday GETTIMEOFDAY_FUNC_TAOS_FORBID +#define localtime LOCALTIME_FUNC_TAOS_FORBID +#define localtime_s LOCALTIMES_FUNC_TAOS_FORBID +#define localtime_r LOCALTIMER_FUNC_TAOS_FORBID +#define time TIME_FUNC_TAOS_FORBID +#define mktime MKTIME_FUNC_TAOS_FORBID #endif #ifdef WINDOWS - #define CLOCK_REALTIME 0 +#define CLOCK_REALTIME 0 - #define MILLISECOND_PER_SECOND (1000i64) +#define MILLISECOND_PER_SECOND (1000i64) #else - #define MILLISECOND_PER_SECOND ((int64_t)1000L) +#define MILLISECOND_PER_SECOND ((int64_t)1000L) #endif #define MILLISECOND_PER_MINUTE (MILLISECOND_PER_SECOND * 60) @@ -82,13 +82,13 @@ static FORCE_INLINE int64_t taosGetTimestampNs() { return (int64_t)systemTime.tv_sec * 1000000000L + (int64_t)systemTime.tv_nsec; } -char *taosStrpTime(const char *buf, const char *fmt, struct tm *tm); +char * taosStrpTime(const char *buf, const char *fmt, struct tm *tm); struct tm *taosLocalTime(const time_t *timep, struct tm *result); -time_t taosTime(time_t *t); -time_t taosMktime(struct tm *timep); +time_t taosTime(time_t *t); +time_t taosMktime(struct tm *timep); #ifdef __cplusplus } #endif -#endif /*_TD_OS_TIME_H_*/ +#endif /*_TD_OS_TIME_H_*/ diff --git a/include/util/taoserror.h b/include/util/taoserror.h index fcbe5cd8782a77d27afc01644967caece454d52c..f4d8c2a663f1f4214434af7e38facd767399e6a1 100644 --- a/include/util/taoserror.h +++ b/include/util/taoserror.h @@ -127,6 +127,8 @@ int32_t* taosGetErrno(); #define TSDB_CODE_TSC_STMT_API_ERROR TAOS_DEF_ERROR_CODE(0, 0X0225) #define TSDB_CODE_TSC_STMT_TBNAME_ERROR TAOS_DEF_ERROR_CODE(0, 0X0226) #define TSDB_CODE_TSC_STMT_CLAUSE_ERROR TAOS_DEF_ERROR_CODE(0, 0X0227) +#define TSDB_CODE_TSC_QUERY_KILLED TAOS_DEF_ERROR_CODE(0, 0X0228) +#define TSDB_CODE_TSC_NO_EXEC_NODE TAOS_DEF_ERROR_CODE(0, 0X0229) // mnode-common #define TSDB_CODE_MND_APP_ERROR TAOS_DEF_ERROR_CODE(0, 0x0300) @@ -414,6 +416,9 @@ int32_t* taosGetErrno(); #define TSDB_CODE_SYN_NOT_LEADER TAOS_DEF_ERROR_CODE(0, 0x090C) #define TSDB_CODE_SYN_ONE_REPLICA TAOS_DEF_ERROR_CODE(0, 0x090D) #define TSDB_CODE_SYN_NOT_IN_NEW_CONFIG TAOS_DEF_ERROR_CODE(0, 0x090E) +#define TSDB_CODE_SYN_NEW_CONFIG_ERROR TAOS_DEF_ERROR_CODE(0, 0x090F) +#define TSDB_CODE_SYN_RECONFIG_NOT_READY TAOS_DEF_ERROR_CODE(0, 0x0910) +#define TSDB_CODE_SYN_PROPOSE_NOT_READY TAOS_DEF_ERROR_CODE(0, 0x0911) #define TSDB_CODE_SYN_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x09FF) // tq @@ -429,7 +434,8 @@ int32_t* taosGetErrno(); #define TSDB_CODE_TQ_META_KEY_NOT_IN_TXN TAOS_DEF_ERROR_CODE(0, 0x0A09) #define TSDB_CODE_TQ_META_KEY_DUP_IN_TXN TAOS_DEF_ERROR_CODE(0, 0x0A0A) #define TSDB_CODE_TQ_GROUP_NOT_SET TAOS_DEF_ERROR_CODE(0, 0x0A0B) -#define TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND TAOS_DEF_ERROR_CODE(0, 0x0A0B) +#define TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND TAOS_DEF_ERROR_CODE(0, 0x0A0C) +#define TSDB_CODE_TQ_NO_COMMITTED_OFFSET TAOS_DEF_ERROR_CODE(0, 0x0A0D) // wal #define TSDB_CODE_WAL_APP_ERROR TAOS_DEF_ERROR_CODE(0, 0x1000) @@ -439,108 +445,6 @@ int32_t* taosGetErrno(); #define TSDB_CODE_WAL_OUT_OF_MEMORY TAOS_DEF_ERROR_CODE(0, 0x1004) #define TSDB_CODE_WAL_LOG_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x1005) -// http -#define TSDB_CODE_HTTP_SERVER_OFFLINE TAOS_DEF_ERROR_CODE(0, 0x1100) //"http server is not online" -#define TSDB_CODE_HTTP_UNSUPPORT_URL TAOS_DEF_ERROR_CODE(0, 0x1101) //"url is not support" -#define TSDB_CODE_HTTP_INVALID_URL TAOS_DEF_ERROR_CODE(0, 0x1102) //invalid url format" -#define TSDB_CODE_HTTP_NO_ENOUGH_MEMORY TAOS_DEF_ERROR_CODE(0, 0x1103) //"no enough memory" -#define TSDB_CODE_HTTP_REQUSET_TOO_BIG TAOS_DEF_ERROR_CODE(0, 0x1104) //"request size is too big" -#define TSDB_CODE_HTTP_NO_AUTH_INFO TAOS_DEF_ERROR_CODE(0, 0x1105) //"no auth info input" -#define TSDB_CODE_HTTP_NO_MSG_INPUT TAOS_DEF_ERROR_CODE(0, 0x1106) //"request is empty" -#define TSDB_CODE_HTTP_NO_SQL_INPUT TAOS_DEF_ERROR_CODE(0, 0x1107) //"no sql input" -#define TSDB_CODE_HTTP_NO_EXEC_USEDB TAOS_DEF_ERROR_CODE(0, 0x1108) //"no need to execute use db cmd" -#define TSDB_CODE_HTTP_SESSION_FULL TAOS_DEF_ERROR_CODE(0, 0x1109) //"session list was full" -#define TSDB_CODE_HTTP_GEN_TAOSD_TOKEN_ERR TAOS_DEF_ERROR_CODE(0, 0x110A) //"generate taosd token error" -#define TSDB_CODE_HTTP_INVALID_MULTI_REQUEST TAOS_DEF_ERROR_CODE(0, 0x110B) //"size of multi request is 0" -#define TSDB_CODE_HTTP_CREATE_GZIP_FAILED TAOS_DEF_ERROR_CODE(0, 0x110C) //"failed to create gzip" -#define TSDB_CODE_HTTP_FINISH_GZIP_FAILED TAOS_DEF_ERROR_CODE(0, 0x110D) //"failed to finish gzip" -#define TSDB_CODE_HTTP_LOGIN_FAILED TAOS_DEF_ERROR_CODE(0, 0x110E) //"failed to login" - -#define TSDB_CODE_HTTP_INVALID_VERSION TAOS_DEF_ERROR_CODE(0, 0x1120) //"invalid http version" -#define TSDB_CODE_HTTP_INVALID_CONTENT_LENGTH TAOS_DEF_ERROR_CODE(0, 0x1121) //"invalid content length" -#define TSDB_CODE_HTTP_INVALID_AUTH_TYPE TAOS_DEF_ERROR_CODE(0, 0x1122) //"invalid type of Authorization" -#define TSDB_CODE_HTTP_INVALID_AUTH_FORMAT TAOS_DEF_ERROR_CODE(0, 0x1123) //"invalid format of Authorization" -#define TSDB_CODE_HTTP_INVALID_BASIC_AUTH TAOS_DEF_ERROR_CODE(0, 0x1124) //"invalid basic Authorization" -#define TSDB_CODE_HTTP_INVALID_TAOSD_AUTH TAOS_DEF_ERROR_CODE(0, 0x1125) //"invalid taosd Authorization" -#define TSDB_CODE_HTTP_PARSE_METHOD_FAILED TAOS_DEF_ERROR_CODE(0, 0x1126) //"failed to parse method" -#define TSDB_CODE_HTTP_PARSE_TARGET_FAILED TAOS_DEF_ERROR_CODE(0, 0x1127) //"failed to parse target" -#define TSDB_CODE_HTTP_PARSE_VERSION_FAILED TAOS_DEF_ERROR_CODE(0, 0x1128) //"failed to parse http version" -#define TSDB_CODE_HTTP_PARSE_SP_FAILED TAOS_DEF_ERROR_CODE(0, 0x1129) //"failed to parse sp" -#define TSDB_CODE_HTTP_PARSE_STATUS_FAILED TAOS_DEF_ERROR_CODE(0, 0x112A) //"failed to parse status" -#define TSDB_CODE_HTTP_PARSE_PHRASE_FAILED TAOS_DEF_ERROR_CODE(0, 0x112B) //"failed to parse phrase" -#define TSDB_CODE_HTTP_PARSE_CRLF_FAILED TAOS_DEF_ERROR_CODE(0, 0x112C) //"failed to parse crlf" -#define TSDB_CODE_HTTP_PARSE_HEADER_FAILED TAOS_DEF_ERROR_CODE(0, 0x112D) //"failed to parse header" -#define TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED TAOS_DEF_ERROR_CODE(0, 0x112E) //"failed to parse header key" -#define TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED TAOS_DEF_ERROR_CODE(0, 0x112F) //"failed to parse header val" -#define TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED TAOS_DEF_ERROR_CODE(0, 0x1130) //"failed to parse chunk size" -#define TSDB_CODE_HTTP_PARSE_CHUNK_FAILED TAOS_DEF_ERROR_CODE(0, 0x1131) //"failed to parse chunk" -#define TSDB_CODE_HTTP_PARSE_END_FAILED TAOS_DEF_ERROR_CODE(0, 0x1132) //"failed to parse end section" -#define TSDB_CODE_HTTP_PARSE_INVALID_STATE TAOS_DEF_ERROR_CODE(0, 0x1134) //"invalid parse state" -#define TSDB_CODE_HTTP_PARSE_ERROR_STATE TAOS_DEF_ERROR_CODE(0, 0x1135) //"failed to parse error section" - -#define TSDB_CODE_HTTP_GC_QUERY_NULL TAOS_DEF_ERROR_CODE(0, 0x1150) //"query size is 0" -#define TSDB_CODE_HTTP_GC_QUERY_SIZE TAOS_DEF_ERROR_CODE(0, 0x1151) //"query size can not more than 100" -#define TSDB_CODE_HTTP_GC_REQ_PARSE_ERROR TAOS_DEF_ERROR_CODE(0, 0x1152) //"parse grafana json error" - -#define TSDB_CODE_HTTP_TG_DB_NOT_INPUT TAOS_DEF_ERROR_CODE(0, 0x1160) //"database name can not be null" -#define TSDB_CODE_HTTP_TG_DB_TOO_LONG TAOS_DEF_ERROR_CODE(0, 0x1161) //"database name too long" -#define TSDB_CODE_HTTP_TG_INVALID_JSON TAOS_DEF_ERROR_CODE(0, 0x1162) //"invalid telegraf json fromat" -#define TSDB_CODE_HTTP_TG_METRICS_NULL TAOS_DEF_ERROR_CODE(0, 0x1163) //"metrics size is 0" -#define TSDB_CODE_HTTP_TG_METRICS_SIZE TAOS_DEF_ERROR_CODE(0, 0x1164) //"metrics size can not more than 1K" -#define TSDB_CODE_HTTP_TG_METRIC_NULL TAOS_DEF_ERROR_CODE(0, 0x1165) //"metric name not find" -#define TSDB_CODE_HTTP_TG_METRIC_TYPE TAOS_DEF_ERROR_CODE(0, 0x1166) //"metric name type should be string" -#define TSDB_CODE_HTTP_TG_METRIC_NAME_NULL TAOS_DEF_ERROR_CODE(0, 0x1167) //"metric name length is 0" -#define TSDB_CODE_HTTP_TG_METRIC_NAME_LONG TAOS_DEF_ERROR_CODE(0, 0x1168) //"metric name length too long" -#define TSDB_CODE_HTTP_TG_TIMESTAMP_NULL TAOS_DEF_ERROR_CODE(0, 0x1169) //"timestamp not find" -#define TSDB_CODE_HTTP_TG_TIMESTAMP_TYPE TAOS_DEF_ERROR_CODE(0, 0x116A) //"timestamp type should be integer" -#define TSDB_CODE_HTTP_TG_TIMESTAMP_VAL_NULL TAOS_DEF_ERROR_CODE(0, 0x116B) //"timestamp value smaller than 0" -#define TSDB_CODE_HTTP_TG_TAGS_NULL TAOS_DEF_ERROR_CODE(0, 0x116C) //"tags not find" -#define TSDB_CODE_HTTP_TG_TAGS_SIZE_0 TAOS_DEF_ERROR_CODE(0, 0x116D) //"tags size is 0" -#define TSDB_CODE_HTTP_TG_TAGS_SIZE_LONG TAOS_DEF_ERROR_CODE(0, 0x116E) //"tags size too long" -#define TSDB_CODE_HTTP_TG_TAG_NULL TAOS_DEF_ERROR_CODE(0, 0x116F) //"tag is null" -#define TSDB_CODE_HTTP_TG_TAG_NAME_NULL TAOS_DEF_ERROR_CODE(0, 0x1170) //"tag name is null" -#define TSDB_CODE_HTTP_TG_TAG_NAME_SIZE TAOS_DEF_ERROR_CODE(0, 0x1171) //"tag name length too long" -#define TSDB_CODE_HTTP_TG_TAG_VALUE_TYPE TAOS_DEF_ERROR_CODE(0, 0x1172) //"tag value type should be number or string" -#define TSDB_CODE_HTTP_TG_TAG_VALUE_NULL TAOS_DEF_ERROR_CODE(0, 0x1173) //"tag value is null" -#define TSDB_CODE_HTTP_TG_TABLE_NULL TAOS_DEF_ERROR_CODE(0, 0x1174) //"table is null" -#define TSDB_CODE_HTTP_TG_TABLE_SIZE TAOS_DEF_ERROR_CODE(0, 0x1175) //"table name length too long" -#define TSDB_CODE_HTTP_TG_FIELDS_NULL TAOS_DEF_ERROR_CODE(0, 0x1176) //"fields not find" -#define TSDB_CODE_HTTP_TG_FIELDS_SIZE_0 TAOS_DEF_ERROR_CODE(0, 0x1177) //"fields size is 0" -#define TSDB_CODE_HTTP_TG_FIELDS_SIZE_LONG TAOS_DEF_ERROR_CODE(0, 0x1178) //"fields size too long" -#define TSDB_CODE_HTTP_TG_FIELD_NULL TAOS_DEF_ERROR_CODE(0, 0x1179) //"field is null" -#define TSDB_CODE_HTTP_TG_FIELD_NAME_NULL TAOS_DEF_ERROR_CODE(0, 0x117A) //"field name is null" -#define TSDB_CODE_HTTP_TG_FIELD_NAME_SIZE TAOS_DEF_ERROR_CODE(0, 0x117B) //"field name length too long" -#define TSDB_CODE_HTTP_TG_FIELD_VALUE_TYPE TAOS_DEF_ERROR_CODE(0, 0x117C) //"field value type should be number or string" -#define TSDB_CODE_HTTP_TG_FIELD_VALUE_NULL TAOS_DEF_ERROR_CODE(0, 0x117D) //"field value is null" -#define TSDB_CODE_HTTP_TG_HOST_NOT_STRING TAOS_DEF_ERROR_CODE(0, 0x117E) //"host type should be string" -#define TSDB_CODE_HTTP_TG_STABLE_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x117F) //"stable not exist" - -#define TSDB_CODE_HTTP_OP_DB_NOT_INPUT TAOS_DEF_ERROR_CODE(0, 0x1190) //"database name can not be null" -#define TSDB_CODE_HTTP_OP_DB_TOO_LONG TAOS_DEF_ERROR_CODE(0, 0x1191) //"database name too long" -#define TSDB_CODE_HTTP_OP_INVALID_JSON TAOS_DEF_ERROR_CODE(0, 0x1192) //"invalid opentsdb json fromat" -#define TSDB_CODE_HTTP_OP_METRICS_NULL TAOS_DEF_ERROR_CODE(0, 0x1193) //"metrics size is 0" -#define TSDB_CODE_HTTP_OP_METRICS_SIZE TAOS_DEF_ERROR_CODE(0, 0x1194) //"metrics size can not more than 10K" -#define TSDB_CODE_HTTP_OP_METRIC_NULL TAOS_DEF_ERROR_CODE(0, 0x1195) //"metric name not find" -#define TSDB_CODE_HTTP_OP_METRIC_TYPE TAOS_DEF_ERROR_CODE(0, 0x1196) //"metric name type should be string" -#define TSDB_CODE_HTTP_OP_METRIC_NAME_NULL TAOS_DEF_ERROR_CODE(0, 0x1197) //"metric name length is 0" -#define TSDB_CODE_HTTP_OP_METRIC_NAME_LONG TAOS_DEF_ERROR_CODE(0, 0x1198) //"metric name length can not more than 22" -#define TSDB_CODE_HTTP_OP_TIMESTAMP_NULL TAOS_DEF_ERROR_CODE(0, 0x1199) //"timestamp not find" -#define TSDB_CODE_HTTP_OP_TIMESTAMP_TYPE TAOS_DEF_ERROR_CODE(0, 0x119A) //"timestamp type should be integer" -#define TSDB_CODE_HTTP_OP_TIMESTAMP_VAL_NULL TAOS_DEF_ERROR_CODE(0, 0x119B) //"timestamp value smaller than 0" -#define TSDB_CODE_HTTP_OP_TAGS_NULL TAOS_DEF_ERROR_CODE(0, 0x119C) //"tags not find" -#define TSDB_CODE_HTTP_OP_TAGS_SIZE_0 TAOS_DEF_ERROR_CODE(0, 0x119D) //"tags size is 0" -#define TSDB_CODE_HTTP_OP_TAGS_SIZE_LONG TAOS_DEF_ERROR_CODE(0, 0x119E) //"tags size too long" -#define TSDB_CODE_HTTP_OP_TAG_NULL TAOS_DEF_ERROR_CODE(0, 0x119F) //"tag is null" -#define TSDB_CODE_HTTP_OP_TAG_NAME_NULL TAOS_DEF_ERROR_CODE(0, 0x11A0) //"tag name is null" -#define TSDB_CODE_HTTP_OP_TAG_NAME_SIZE TAOS_DEF_ERROR_CODE(0, 0x11A1) //"tag name length too long" -#define TSDB_CODE_HTTP_OP_TAG_VALUE_TYPE TAOS_DEF_ERROR_CODE(0, 0x11A2) //"tag value type should be boolean number or string" -#define TSDB_CODE_HTTP_OP_TAG_VALUE_NULL TAOS_DEF_ERROR_CODE(0, 0x11A3) //"tag value is null" -#define TSDB_CODE_HTTP_OP_TAG_VALUE_TOO_LONG TAOS_DEF_ERROR_CODE(0, 0x11A4) //"tag value can not more than 64" -#define TSDB_CODE_HTTP_OP_VALUE_NULL TAOS_DEF_ERROR_CODE(0, 0x11A5) //"value not find" -#define TSDB_CODE_HTTP_OP_VALUE_TYPE TAOS_DEF_ERROR_CODE(0, 0x11A6) //"value type should be boolean number or string" - -#define TSDB_CODE_HTTP_REQUEST_JSON_ERROR TAOS_DEF_ERROR_CODE(0, 0x1F00) //"http request json error" - // tfs #define TSDB_CODE_FS_APP_ERROR TAOS_DEF_ERROR_CODE(0, 0x2200) #define TSDB_CODE_FS_INVLD_CFG TAOS_DEF_ERROR_CODE(0, 0x2201) @@ -570,6 +474,7 @@ int32_t* taosGetErrno(); #define TSDB_CODE_SCH_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x2502) #define TSDB_CODE_SCH_IGNORE_ERROR TAOS_DEF_ERROR_CODE(0, 0x2503) #define TSDB_CODE_SCH_TIMEOUT_ERROR TAOS_DEF_ERROR_CODE(0, 0x2504) +#define TSDB_CODE_SCH_JOB_IS_DROPPING TAOS_DEF_ERROR_CODE(0, 0x2505) #define TSDB_CODE_QW_MSG_ERROR TAOS_DEF_ERROR_CODE(0, 0x2550) //parser @@ -658,6 +563,11 @@ int32_t* taosGetErrno(); #define TSDB_CODE_PAR_INVALID_REDISTRIBUTE_VG TAOS_DEF_ERROR_CODE(0, 0x2656) #define TSDB_CODE_PAR_FILL_NOT_ALLOWED_FUNC TAOS_DEF_ERROR_CODE(0, 0x2657) #define TSDB_CODE_PAR_INVALID_WINDOW_PC TAOS_DEF_ERROR_CODE(0, 0x2658) +#define TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC TAOS_DEF_ERROR_CODE(0, 0x2659) +#define TSDB_CODE_PAR_STREAM_NOT_ALLOWED_FUNC TAOS_DEF_ERROR_CODE(0, 0x265A) +#define TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC TAOS_DEF_ERROR_CODE(0, 0x265B) +#define TSDB_CODE_PAR_INVALID_TABLE_OPTION TAOS_DEF_ERROR_CODE(0, 0x265C) +#define TSDB_CODE_PAR_INVALID_INTERP_CLAUSE TAOS_DEF_ERROR_CODE(0, 0x265D) //planner #define TSDB_CODE_PLAN_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x2700) @@ -706,6 +616,8 @@ int32_t* taosGetErrno(); //index #define TSDB_CODE_INDEX_REBUILDING TAOS_DEF_ERROR_CODE(0, 0x3200) +//tmq +#define TSDB_CODE_TMQ_INVALID_MSG TAOS_DEF_ERROR_CODE(0, 0x4000) #ifdef __cplusplus } diff --git a/include/util/tdef.h b/include/util/tdef.h index a48e823bade5eeb09d1324470981428f7fca437b..c5a8b95a08cc37e4d665e01f5de01eca97b16087 100644 --- a/include/util/tdef.h +++ b/include/util/tdef.h @@ -344,14 +344,13 @@ typedef enum ELogicConditionType { #define TSDB_DB_SCHEMALESS_OFF 0 #define TSDB_DEFAULT_DB_SCHEMALESS TSDB_DB_SCHEMALESS_OFF -// #define TSDB_MIN_ROLLUP_DELAY 1 -// #define TSDB_MAX_ROLLUP_DELAY 10 -// #define TSDB_DEFAULT_ROLLUP_DELAY 1 -#define TSDB_MIN_ROLLUP_FILE_FACTOR 0 -#define TSDB_MAX_ROLLUP_FILE_FACTOR 10 -#define TSDB_DEFAULT_ROLLUP_FILE_FACTOR 0.1 -#define TSDB_MIN_TABLE_TTL 0 -#define TSDB_DEFAULT_TABLE_TTL 0 +#define TSDB_MIN_ROLLUP_MAX_DELAY 1 // unit millisecond +#define TSDB_MAX_ROLLUP_MAX_DELAY (15 * 60 * 1000) +#define TSDB_MIN_ROLLUP_WATERMARK 0 // unit millisecond +#define TSDB_MAX_ROLLUP_WATERMARK (15 * 60 * 1000) +#define TSDB_DEFAULT_ROLLUP_WATERMARK 5000 +#define TSDB_MIN_TABLE_TTL 0 +#define TSDB_DEFAULT_TABLE_TTL 0 #define TSDB_MIN_EXPLAIN_RATIO 0 #define TSDB_MAX_EXPLAIN_RATIO 1 diff --git a/include/util/ttrace.h b/include/util/ttrace.h new file mode 100644 index 0000000000000000000000000000000000000000..206cbbf28d9a785873333e3d32d7a67a0115d603 --- /dev/null +++ b/include/util/ttrace.h @@ -0,0 +1,57 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +#ifndef _TD_TRACE_H_ +#define _TD_TRACE_H_ + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +#pragma(push, 1) + +typedef struct STraceId { + int64_t rootId; + int64_t msgId; +} STraceId; + +#pragma(pop) + +#define TRACE_SET_ROOTID(traceId, root) \ + do { \ + (traceId)->rootId = root; \ + } while (0); + +#define TRACE_GET_ROOTID(traceId) (traceId)->rootId + +#define TRACE_SET_MSGID(traceId, mId) \ + do { \ + (traceId)->msgId = mId; \ + } while (0) + +#define TRACE_GET_MSGID(traceId) (traceId)->msgId + +#define TRACE_TO_STR(traceId, buf) \ + do { \ + sprintf(buf, "0x%" PRIx64 ":0x%" PRIx64 "", traceId->rootId, traceId->msgId); \ + } while (0) + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/source/client/inc/clientInt.h b/source/client/inc/clientInt.h index 69a24e20cd7e4a6d20efc9c12f1766f06af050ae..c39b492620c4c9bb0922438c1e895b8b4661326d 100644 --- a/source/client/inc/clientInt.h +++ b/source/client/inc/clientInt.h @@ -22,6 +22,7 @@ extern "C" { #include "parser.h" #include "planner.h" +#include "catalog.h" #include "query.h" #include "taos.h" #include "tcommon.h" @@ -76,10 +77,12 @@ typedef int32_t (*FHbReqHandle)(SClientHbKey* connKey, void* param, SClientHbReq typedef struct { int8_t inited; + int64_t appId; // ctl int8_t threadStop; TdThread thread; TdThreadMutex lock; // used when app init and cleanup + SHashObj *appSummary; SArray* appHbMgrs; // SArray one for each cluster FHbReqHandle reqHandle[CONN_TYPE__MAX]; FHbRspHandle rspHandle[CONN_TYPE__MAX]; @@ -92,33 +95,20 @@ typedef struct SQueryExecMetric { int64_t rsp; // receive response from server, us } SQueryExecMetric; -typedef struct SInstanceSummary { - uint64_t numOfInsertsReq; - uint64_t numOfInsertRows; - uint64_t insertElapsedTime; - uint64_t insertBytes; // submit to tsdb since launched. - - uint64_t fetchBytes; - uint64_t queryElapsedTime; - uint64_t numOfSlowQueries; - uint64_t totalRequests; - uint64_t currentRequests; // the number of SRequestObj -} SInstanceSummary; - typedef struct SHeartBeatInfo { void* pTimer; // timer, used to send request msg to mnode } SHeartBeatInfo; struct SAppInstInfo { - int64_t numOfConns; - SCorEpSet mgmtEp; - TdThreadMutex qnodeMutex; - SArray* pQnodeList; - SInstanceSummary summary; - SList* pConnList; // STscObj linked list - uint64_t clusterId; - void* pTransporter; - SAppHbMgr* pAppHbMgr; + int64_t numOfConns; + SCorEpSet mgmtEp; + TdThreadMutex qnodeMutex; + SArray* pQnodeList; + SAppClusterSummary summary; + SList* pConnList; // STscObj linked list + uint64_t clusterId; + void* pTransporter; + SAppHbMgr* pAppHbMgr; }; typedef struct SAppInfo { @@ -139,7 +129,7 @@ typedef struct STscObj { int8_t connType; int32_t acctId; uint32_t connId; - uint64_t id; // ref ID returned by taosAddRef + TAOS *id; // ref ID returned by taosAddRef TdThreadMutex mutex; // used to protect the operation on db int32_t numOfReqs; // number of sqlObj bound to this connection SAppInstInfo* pAppInfo; @@ -173,6 +163,7 @@ typedef struct SReqResultInfo { int32_t precision; bool convertUcs4; int32_t payloadLen; + char* convertJson; } SReqResultInfo; typedef struct SRequestSendRecvBody { @@ -215,6 +206,7 @@ typedef struct SRequestObj { SRequestSendRecvBody body; bool stableQuery; + bool killed; uint32_t prevCode; //previous error code: todo refactor, add update flag for catalog uint32_t retry; } SRequestObj; @@ -252,6 +244,7 @@ static FORCE_INLINE SReqResultInfo* tmqGetNextResInfo(TAOS_RES* res, bool conver taosMemoryFreeClear(msg->resInfo.pCol); taosMemoryFreeClear(msg->resInfo.length); taosMemoryFreeClear(msg->resInfo.convertBuf); + taosMemoryFreeClear(msg->resInfo.convertJson); } setQueryResultFromRsp(&msg->resInfo, pRetrieve, convertUcs4, false); return &msg->resInfo; @@ -302,7 +295,7 @@ SRequestObj* launchQuery(STscObj* pTscObj, const char* sql, int sqlLen); int32_t parseSql(SRequestObj* pRequest, bool topicQuery, SQuery** pQuery, SStmtCallback* pStmtCb); -int32_t getPlan(SRequestObj* pRequest, SQuery* pQuery, SQueryPlan** pPlan, SArray** pNodeList); +int32_t getPlan(SRequestObj* pRequest, SQuery* pQuery, SQueryPlan** pPlan, SArray* pNodeList); int32_t buildRequest(STscObj* pTscObj, const char* sql, int sqlLen, SRequestObj** pRequest); @@ -327,12 +320,13 @@ void hbMgrInitMqHbRspHandle(); SRequestObj* launchQueryImpl(SRequestObj* pRequest, SQuery* pQuery, bool keepQuery, void** res); int32_t scheduleQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNodeList); -void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery); +void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery, SMetaData *pResultMeta); int32_t refreshMeta(STscObj* pTscObj, SRequestObj* pRequest); int32_t updateQnodeList(SAppInstInfo* pInfo, SArray* pNodeList); void doAsyncQuery(SRequestObj* pRequest, bool forceUpdateMeta); int32_t removeMeta(STscObj* pTscObj, SArray* tbList);// todo move to clientImpl.c and become a static function int32_t handleAlterTbExecRes(void* res, struct SCatalog* pCatalog);// todo move to xxx +bool qnodeRequired(SRequestObj* pRequest); #ifdef __cplusplus } diff --git a/source/client/src/clientEnv.c b/source/client/src/clientEnv.c index bb60624145f47545769b3a606a25db93268ac1ce..9f04e89694f77153b747892872d0cdb0c173ca6f 100644 --- a/source/client/src/clientEnv.c +++ b/source/client/src/clientEnv.c @@ -38,7 +38,7 @@ static TdThreadOnce tscinit = PTHREAD_ONCE_INIT; volatile int32_t tscInitRes = 0; static void registerRequest(SRequestObj *pRequest) { - STscObj *pTscObj = acquireTscObj(pRequest->pTscObj->id); + STscObj *pTscObj = acquireTscObj(*(int64_t*)pRequest->pTscObj->id); assert(pTscObj != NULL); @@ -48,13 +48,13 @@ static void registerRequest(SRequestObj *pRequest) { int32_t num = atomic_add_fetch_32(&pTscObj->numOfReqs, 1); if (pTscObj->pAppInfo) { - SInstanceSummary *pSummary = &pTscObj->pAppInfo->summary; + SAppClusterSummary *pSummary = &pTscObj->pAppInfo->summary; int32_t total = atomic_add_fetch_64((int64_t *)&pSummary->totalRequests, 1); int32_t currentInst = atomic_add_fetch_64((int64_t *)&pSummary->currentRequests, 1); tscDebug("0x%" PRIx64 " new Request from connObj:0x%" PRIx64 ", current:%d, app current:%d, total:%d, reqId:0x%" PRIx64, - pRequest->self, pRequest->pTscObj->id, num, currentInst, total, pRequest->requestId); + pRequest->self, *(int64_t*)pRequest->pTscObj->id, num, currentInst, total, pRequest->requestId); } } @@ -62,7 +62,7 @@ static void deregisterRequest(SRequestObj *pRequest) { assert(pRequest != NULL); STscObj *pTscObj = pRequest->pTscObj; - SInstanceSummary *pActivity = &pTscObj->pAppInfo->summary; + SAppClusterSummary *pActivity = &pTscObj->pAppInfo->summary; int32_t currentInst = atomic_sub_fetch_64((int64_t *)&pActivity->currentRequests, 1); int32_t num = atomic_sub_fetch_32(&pTscObj->numOfReqs, 1); @@ -70,8 +70,8 @@ static void deregisterRequest(SRequestObj *pRequest) { int64_t duration = taosGetTimestampUs() - pRequest->metric.start; tscDebug("0x%" PRIx64 " free Request from connObj: 0x%" PRIx64 ", reqId:0x%" PRIx64 " elapsed:%" PRIu64 " ms, current:%d, app current:%d", - pRequest->self, pTscObj->id, pRequest->requestId, duration / 1000, num, currentInst); - releaseTscObj(pTscObj->id); + pRequest->self, *(int64_t*)pTscObj->id, pRequest->requestId, duration / 1000, num, currentInst); + releaseTscObj(*(int64_t*)pTscObj->id); } // todo close the transporter properly @@ -80,7 +80,7 @@ void closeTransporter(STscObj *pTscObj) { return; } - tscDebug("free transporter:%p in connObj: 0x%" PRIx64, pTscObj->pAppInfo->pTransporter, pTscObj->id); + tscDebug("free transporter:%p in connObj: 0x%" PRIx64, pTscObj->pAppInfo->pTransporter, *(int64_t*)pTscObj->id); rpcClose(pTscObj->pAppInfo->pTransporter); } @@ -128,7 +128,7 @@ void closeAllRequests(SHashObj *pRequests) { void destroyTscObj(void *pObj) { STscObj *pTscObj = pObj; - SClientHbKey connKey = {.tscRid = pTscObj->id, .connType = pTscObj->connType}; + SClientHbKey connKey = {.tscRid = *(int64_t*)pTscObj->id, .connType = pTscObj->connType}; hbDeregisterConn(pTscObj->pAppInfo->pAppHbMgr, connKey); int64_t connNum = atomic_sub_fetch_64(&pTscObj->pAppInfo->numOfConns, 1); closeAllRequests(pTscObj->pRequests); @@ -137,7 +137,7 @@ void destroyTscObj(void *pObj) { // TODO //closeTransporter(pTscObj); } - tscDebug("connObj 0x%" PRIx64 " destroyed, totalConn:%" PRId64, pTscObj->id, pTscObj->pAppInfo->numOfConns); + tscDebug("connObj 0x%" PRIx64 " destroyed, totalConn:%" PRId64, *(int64_t*)pTscObj->id, pTscObj->pAppInfo->numOfConns); taosThreadMutexDestroy(&pTscObj->mutex); taosMemoryFreeClear(pTscObj); } @@ -166,10 +166,11 @@ void *createTscObj(const char *user, const char *auth, const char *db, int32_t c } taosThreadMutexInit(&pObj->mutex, NULL); - pObj->id = taosAddRef(clientConnRefPool, pObj); + pObj->id = taosMemoryMalloc(sizeof(int64_t)); + *(int64_t*)pObj->id = taosAddRef(clientConnRefPool, pObj); pObj->schemalessType = 1; - tscDebug("connObj created, 0x%" PRIx64, pObj->id); + tscDebug("connObj created, 0x%" PRIx64, *(int64_t*)pObj->id); return pObj; } @@ -211,6 +212,7 @@ void doFreeReqResultInfo(SReqResultInfo *pResInfo) { taosMemoryFreeClear(pResInfo->pCol); taosMemoryFreeClear(pResInfo->fields); taosMemoryFreeClear(pResInfo->userFields); + taosMemoryFreeClear(pResInfo->convertJson); if (pResInfo->convertBuf != NULL) { for (int32_t i = 0; i < pResInfo->numOfCols; ++i) { @@ -229,7 +231,7 @@ static void doDestroyRequest(void *p) { taosHashRemove(pRequest->pTscObj->pRequests, &pRequest->self, sizeof(pRequest->self)); if (pRequest->body.queryJob != 0) { - schedulerFreeJob(pRequest->body.queryJob); + schedulerFreeJob(pRequest->body.queryJob, 0); } taosMemoryFreeClear(pRequest->msgBuf); diff --git a/source/client/src/clientHb.c b/source/client/src/clientHb.c index e0b8f322bfdc446da978376348718017edb86bc9..c6d0d6a860e747a7240b4bab4f9153a07f5ae8c6 100644 --- a/source/client/src/clientHb.c +++ b/source/client/src/clientHb.c @@ -164,6 +164,7 @@ static int32_t hbQueryHbRspHandle(SAppHbMgr *pAppHbMgr, SClientHbRsp *pRsp) { pTscObj->connId = pRsp->query->connId; if (pRsp->query->killRid) { + tscDebug("request rid %" PRIx64 " need to be killed now", pRsp->query->killRid); SRequestObj *pRequest = acquireRequest(pRsp->query->killRid); if (NULL == pRequest) { tscDebug("request 0x%" PRIx64 " not exist to kill", pRsp->query->killRid); @@ -304,7 +305,7 @@ int32_t hbBuildQueryDesc(SQueryHbReqBasic *hbBasic, STscObj *pObj) { while (pIter != NULL) { int64_t *rid = pIter; SRequestObj *pRequest = acquireRequest(*rid); - if (NULL == pRequest) { + if (NULL == pRequest || pRequest->killed) { pIter = taosHashIterate(pObj->pRequests, pIter); continue; } @@ -314,7 +315,6 @@ int32_t hbBuildQueryDesc(SQueryHbReqBasic *hbBasic, STscObj *pObj) { desc.queryId = pRequest->requestId; desc.useconds = now - pRequest->metric.start; desc.reqRid = pRequest->self; - desc.pid = hbBasic->pid; desc.stableQuery = pRequest->stableQuery; taosGetFqdn(desc.fqdn); desc.subPlanNum = pRequest->body.pDag ? pRequest->body.pDag->numOfSubplans : 0; @@ -360,8 +360,6 @@ int32_t hbGetQueryBasicInfo(SClientHbKey *connKey, SClientHbReq *req) { } hbBasic->connId = pTscObj->connId; - hbBasic->pid = taosGetPId(); - taosGetAppName(hbBasic->app, NULL); int32_t numOfQueries = pTscObj->pRequests ? taosHashGetSize(pTscObj->pRequests) : 0; if (numOfQueries <= 0) { @@ -507,6 +505,21 @@ int32_t hbGetExpiredStbInfo(SClientHbKey *connKey, struct SCatalog *pCatalog, SC return TSDB_CODE_SUCCESS; } +int32_t hbGetAppInfo(int64_t clusterId, SClientHbReq *req) { + SAppHbReq* pApp = taosHashGet(clientHbMgr.appSummary, &clusterId, sizeof(clusterId)); + if (NULL != pApp) { + memcpy(&req->app, pApp, sizeof(*pApp)); + } else { + memset(&req->app.summary, 0, sizeof(req->app.summary)); + req->app.pid = taosGetPId(); + req->app.appId = clientHbMgr.appId; + taosGetAppName(req->app.name, NULL); + } + + return TSDB_CODE_SUCCESS; +} + + int32_t hbQueryHbReqHandle(SClientHbKey *connKey, void *param, SClientHbReq *req) { int64_t *clusterId = (int64_t *)param; struct SCatalog *pCatalog = NULL; @@ -517,6 +530,8 @@ int32_t hbQueryHbReqHandle(SClientHbKey *connKey, void *param, SClientHbReq *req return code; } + hbGetAppInfo(*clusterId, req); + hbGetQueryBasicInfo(connKey, req); code = hbGetExpiredUserInfo(connKey, pCatalog, req); @@ -589,6 +604,50 @@ void hbThreadFuncUnexpectedStopped(void) { atomic_store_8(&clientHbMgr.threadStop, 2); } +void hbMergeSummary(SAppClusterSummary* dst, SAppClusterSummary* src) { + dst->numOfInsertsReq += src->numOfInsertsReq; + dst->numOfInsertRows += src->numOfInsertRows; + dst->insertElapsedTime += src->insertElapsedTime; + dst->insertBytes += src->insertBytes; + dst->fetchBytes += src->fetchBytes; + dst->queryElapsedTime += src->queryElapsedTime; + dst->numOfSlowQueries += src->numOfSlowQueries; + dst->totalRequests += src->totalRequests; + dst->currentRequests += src->currentRequests; +} + +int32_t hbGatherAppInfo(void) { + SAppHbReq req = {0}; + int sz = taosArrayGetSize(clientHbMgr.appHbMgrs); + if (sz > 0) { + req.pid = taosGetPId(); + req.appId = clientHbMgr.appId; + taosGetAppName(req.name, NULL); + } + + taosHashClear(clientHbMgr.appSummary); + + for (int32_t i = 0; i < sz; ++i) { + SAppHbMgr *pAppHbMgr = taosArrayGetP(clientHbMgr.appHbMgrs, i); + uint64_t clusterId = pAppHbMgr->pAppInstInfo->clusterId; + SAppHbReq* pApp = taosHashGet(clientHbMgr.appSummary, &clusterId, sizeof(clusterId)); + if (NULL == pApp) { + memcpy(&req.summary, &pAppHbMgr->pAppInstInfo->summary, sizeof(req.summary)); + req.startTime = pAppHbMgr->startTime; + taosHashPut(clientHbMgr.appSummary, &clusterId, sizeof(clusterId), &req, sizeof(req)); + } else { + if (pAppHbMgr->startTime < pApp->startTime) { + pApp->startTime = pAppHbMgr->startTime; + } + + hbMergeSummary(&pApp->summary, &pAppHbMgr->pAppInstInfo->summary); + } + } + + return TSDB_CODE_SUCCESS; +} + + static void *hbThreadFunc(void *param) { setThreadName("hb"); #ifdef WINDOWS @@ -605,6 +664,10 @@ static void *hbThreadFunc(void *param) { taosThreadMutexLock(&clientHbMgr.lock); int sz = taosArrayGetSize(clientHbMgr.appHbMgrs); + if (sz > 0) { + hbGatherAppInfo(); + } + for (int i = 0; i < sz; i++) { SAppHbMgr *pAppHbMgr = taosArrayGetP(clientHbMgr.appHbMgrs, i); @@ -748,6 +811,10 @@ int hbMgrInit() { int8_t old = atomic_val_compare_exchange_8(&clientHbMgr.inited, 0, 1); if (old == 1) return 0; + clientHbMgr.appId = tGenIdPI64(); + tscDebug("app %" PRIx64 " initialized", clientHbMgr.appId); + + clientHbMgr.appSummary = taosHashInit(10, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK); clientHbMgr.appHbMgrs = taosArrayInit(0, sizeof(void *)); taosThreadMutexInit(&clientHbMgr.lock, NULL); diff --git a/source/client/src/clientImpl.c b/source/client/src/clientImpl.c index 10c216147839a36541144c65310eccf8c2d458c6..8920922006d517df10e536ad407e8f83909350c4 100644 --- a/source/client/src/clientImpl.c +++ b/source/client/src/clientImpl.c @@ -263,6 +263,7 @@ void asyncExecLocalCmd(SRequestObj* pRequest, SQuery* pQuery) { int32_t asyncExecDdlQuery(SRequestObj* pRequest, SQuery* pQuery) { // drop table if exists not_exists_table if (NULL == pQuery->pCmdMsg) { + pRequest->body.queryFp(pRequest->body.param, pRequest, 0); return TSDB_CODE_SUCCESS; } @@ -306,6 +307,21 @@ int32_t updateQnodeList(SAppInstInfo* pInfo, SArray* pNodeList) { return TSDB_CODE_SUCCESS; } +bool qnodeRequired(SRequestObj* pRequest) { + if (QUERY_POLICY_VNODE == tsQueryPolicy) { + return false; + } + + SAppInstInfo* pInfo = pRequest->pTscObj->pAppInfo; + bool required = false; + + taosThreadMutexLock(&pInfo->qnodeMutex); + required = (NULL == pInfo->pQnodeList); + taosThreadMutexUnlock(&pInfo->qnodeMutex); + + return required; +} + int32_t getQnodeList(SRequestObj* pRequest, SArray** pNodeList) { SAppInstInfo* pInfo = pRequest->pTscObj->pAppInfo; int32_t code = 0; @@ -336,7 +352,7 @@ int32_t getQnodeList(SRequestObj* pRequest, SArray** pNodeList) { return code; } -int32_t getPlan(SRequestObj* pRequest, SQuery* pQuery, SQueryPlan** pPlan, SArray** pNodeList) { +int32_t getPlan(SRequestObj* pRequest, SQuery* pQuery, SQueryPlan** pPlan, SArray* pNodeList) { pRequest->type = pQuery->msgType; SAppInstInfo* pAppInfo = getAppInfo(pRequest); @@ -348,12 +364,7 @@ int32_t getPlan(SRequestObj* pRequest, SQuery* pQuery, SQueryPlan** pPlan, SArra .pMsg = pRequest->msgBuf, .msgLen = ERROR_MSG_BUF_DEFAULT_SIZE}; - int32_t code = getQnodeList(pRequest, pNodeList); - if (TSDB_CODE_SUCCESS == code) { - code = qCreateQueryPlan(&cxt, pPlan, *pNodeList); - } - - return code; + return qCreateQueryPlan(&cxt, pPlan, pNodeList); } void setResSchemaInfo(SReqResultInfo* pResInfo, const SSchema* pSchema, int32_t numOfCols) { @@ -396,6 +407,195 @@ void setResPrecision(SReqResultInfo* pResInfo, int32_t precision) { pResInfo->precision = precision; } +int32_t buildVnodePolicyNodeList(SRequestObj* pRequest, SArray** pNodeList, SArray* pMnodeList, SArray* pDbVgList) { + SArray* nodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); + + int32_t dbNum = taosArrayGetSize(pDbVgList); + for (int32_t i = 0; i < dbNum; ++i) { + SArray* pVg = taosArrayGetP(pDbVgList, i); + int32_t vgNum = taosArrayGetSize(pVg); + if (vgNum <= 0) { + continue; + } + + for (int32_t j = 0; j < vgNum; ++j) { + SVgroupInfo* pInfo = taosArrayGet(pVg, j); + SQueryNodeLoad load = {0}; + load.addr.nodeId = pInfo->vgId; + load.addr.epSet = pInfo->epSet; + + taosArrayPush(nodeList, &load); + } + } + + int32_t vnodeNum = taosArrayGetSize(nodeList); + if (vnodeNum > 0) { + tscDebug("0x%" PRIx64 " vnode policy, use vnode list, num:%d", pRequest->requestId, vnodeNum); + goto _return; + } + + int32_t mnodeNum = taosArrayGetSize(pMnodeList); + if (mnodeNum <= 0) { + tscDebug("0x%" PRIx64 " vnode policy, empty node list", pRequest->requestId); + goto _return; + } + + void* pData = taosArrayGet(pMnodeList, 0); + taosArrayAddBatch(nodeList, pData, mnodeNum); + + tscDebug("0x%" PRIx64 " vnode policy, use mnode list, num:%d", pRequest->requestId, mnodeNum); + +_return: + + *pNodeList = nodeList; + + return TSDB_CODE_SUCCESS; +} + +int32_t buildQnodePolicyNodeList(SRequestObj* pRequest, SArray** pNodeList, SArray* pMnodeList, SArray* pQnodeList) { + SArray* nodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); + + int32_t qNodeNum = taosArrayGetSize(pQnodeList); + if (qNodeNum > 0) { + void* pData = taosArrayGet(pQnodeList, 0); + taosArrayAddBatch(nodeList, pData, qNodeNum); + tscDebug("0x%" PRIx64 " qnode policy, use qnode list, num:%d", pRequest->requestId, qNodeNum); + goto _return; + } + + int32_t mnodeNum = taosArrayGetSize(pMnodeList); + if (mnodeNum <= 0) { + tscDebug("0x%" PRIx64 " qnode policy, empty node list", pRequest->requestId); + goto _return; + } + + void* pData = taosArrayGet(pMnodeList, 0); + taosArrayAddBatch(nodeList, pData, mnodeNum); + + tscDebug("0x%" PRIx64 " qnode policy, use mnode list, num:%d", pRequest->requestId, mnodeNum); + +_return: + + *pNodeList = nodeList; + + return TSDB_CODE_SUCCESS; +} + + +int32_t buildAsyncExecNodeList(SRequestObj* pRequest, SArray** pNodeList, SArray* pMnodeList, SMetaData *pResultMeta) { + SArray* pDbVgList = NULL; + SArray* pQnodeList = NULL; + int32_t code = 0; + + switch (tsQueryPolicy) { + case QUERY_POLICY_VNODE: { + if (pResultMeta) { + pDbVgList = taosArrayInit(4, POINTER_BYTES); + + int32_t dbNum = taosArrayGetSize(pResultMeta->pDbVgroup); + for (int32_t i = 0; i < dbNum; ++i) { + SMetaRes* pRes = taosArrayGet(pResultMeta->pDbVgroup, i); + if (pRes->code || NULL == pRes->pRes) { + continue; + } + + taosArrayPush(pDbVgList, &pRes->pRes); + } + } + + code = buildVnodePolicyNodeList(pRequest, pNodeList, pMnodeList, pDbVgList); + break; + } + case QUERY_POLICY_HYBRID: + case QUERY_POLICY_QNODE: { + if (pResultMeta && taosArrayGetSize(pResultMeta->pQnodeList) > 0) { + SMetaRes* pRes = taosArrayGet(pResultMeta->pQnodeList, 0); + if (pRes->code) { + pQnodeList = NULL; + } else { + pQnodeList = taosArrayDup((SArray*)pRes->pRes); + } + } else { + SAppInstInfo* pInst = pRequest->pTscObj->pAppInfo; + taosThreadMutexLock(&pInst->qnodeMutex); + if (pInst->pQnodeList) { + pQnodeList = taosArrayDup(pInst->pQnodeList); + } + taosThreadMutexUnlock(&pInst->qnodeMutex); + } + + code = buildQnodePolicyNodeList(pRequest, pNodeList, pMnodeList, pQnodeList); + break; + } + default: + tscError("unknown query policy: %d", tsQueryPolicy); + return TSDB_CODE_TSC_APP_ERROR; + } + + taosArrayDestroy(pDbVgList); + taosArrayDestroy(pQnodeList); + + return code; +} + +int32_t buildSyncExecNodeList(SRequestObj* pRequest, SArray** pNodeList, SArray* pMnodeList) { + SArray* pDbVgList = NULL; + SArray* pQnodeList = NULL; + int32_t code = 0; + + switch (tsQueryPolicy) { + case QUERY_POLICY_VNODE: { + int32_t dbNum = taosArrayGetSize(pRequest->dbList); + if (dbNum > 0) { + SCatalog* pCtg = NULL; + SAppInstInfo* pInst = pRequest->pTscObj->pAppInfo; + code = catalogGetHandle(pInst->clusterId, &pCtg); + if (code != TSDB_CODE_SUCCESS) { + goto _return; + } + + pDbVgList = taosArrayInit(dbNum, POINTER_BYTES); + SArray* pVgList = NULL; + for (int32_t i = 0; i < dbNum; ++i) { + char* dbFName = taosArrayGet(pRequest->dbList, i); + SRequestConnInfo conn = {.pTrans = pInst->pTransporter, + .requestId = pRequest->requestId, + .requestObjRefId = pRequest->self, + .mgmtEps = getEpSet_s(&pInst->mgmtEp)}; + + code = catalogGetDBVgInfo(pCtg, &conn, dbFName, &pVgList); + if (code) { + goto _return; + } + + taosArrayPush(pDbVgList, &pVgList); + } + } + + code = buildVnodePolicyNodeList(pRequest, pNodeList, pMnodeList, pDbVgList); + break; + } + case QUERY_POLICY_HYBRID: + case QUERY_POLICY_QNODE: { + getQnodeList(pRequest, &pQnodeList); + + code = buildQnodePolicyNodeList(pRequest, pNodeList, pMnodeList, pQnodeList); + break; + } + default: + tscError("unknown query policy: %d", tsQueryPolicy); + return TSDB_CODE_TSC_APP_ERROR; + } + +_return: + + taosArrayDestroy(pDbVgList); + taosArrayDestroy(pQnodeList); + + return code; +} + + int32_t scheduleAsyncQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNodeList) { tsem_init(&schdRspSem, 0, 0); @@ -418,7 +618,7 @@ int32_t scheduleAsyncQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNod while (true) { if (code != TSDB_CODE_SUCCESS) { if (pRequest->body.queryJob != 0) { - schedulerFreeJob(pRequest->body.queryJob); + schedulerFreeJob(pRequest->body.queryJob, 0); } pRequest->code = code; @@ -439,7 +639,7 @@ int32_t scheduleAsyncQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNod pRequest->body.resInfo.numOfRows = res.numOfRows; if (pRequest->body.queryJob != 0) { - schedulerFreeJob(pRequest->body.queryJob); + schedulerFreeJob(pRequest->body.queryJob, 0); } } @@ -461,14 +661,15 @@ int32_t scheduleQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNodeList .sql = pRequest->sqlstr, .startTs = pRequest->metric.start, .fp = NULL, - .cbParam = NULL}; + .cbParam = NULL, + .reqKilled = &pRequest->killed}; int32_t code = schedulerExecJob(&req, &pRequest->body.queryJob, &res); pRequest->body.resInfo.execRes = res.res; if (code != TSDB_CODE_SUCCESS) { if (pRequest->body.queryJob != 0) { - schedulerFreeJob(pRequest->body.queryJob); + schedulerFreeJob(pRequest->body.queryJob, 0); } pRequest->code = code; @@ -481,7 +682,7 @@ int32_t scheduleQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNodeList pRequest->body.resInfo.numOfRows = res.numOfRows; if (pRequest->body.queryJob != 0) { - schedulerFreeJob(pRequest->body.queryJob); + schedulerFreeJob(pRequest->body.queryJob, 0); } } @@ -608,6 +809,19 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) { SRequestObj* pRequest = (SRequestObj*)param; pRequest->code = code; + if (TDMT_VND_SUBMIT == pRequest->type || TDMT_VND_DELETE == pRequest->type || + TDMT_VND_CREATE_TABLE == pRequest->type) { + pRequest->body.resInfo.numOfRows = pResult->numOfRows; + + if (pRequest->body.queryJob != 0) { + schedulerFreeJob(pRequest->body.queryJob, 0); + pRequest->body.queryJob = 0; + } + } + + tscDebug("0x%" PRIx64 " enter scheduler exec cb, code:%d - %s, reqId:0x%" PRIx64, + pRequest->self, code, tstrerror(code), pRequest->requestId); + STscObj* pTscObj = pRequest->pTscObj; if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code)) { tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%" PRIx64, @@ -643,12 +857,16 @@ SRequestObj* launchQueryImpl(SRequestObj* pRequest, SQuery* pQuery, bool keepQue code = execDdlQuery(pRequest, pQuery); break; case QUERY_EXEC_MODE_SCHEDULE: { - SArray* pNodeList = NULL; - code = getPlan(pRequest, pQuery, &pRequest->body.pDag, &pNodeList); + SArray* pMnodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); + code = getPlan(pRequest, pQuery, &pRequest->body.pDag, pMnodeList); if (TSDB_CODE_SUCCESS == code) { + SArray* pNodeList = NULL; + buildSyncExecNodeList(pRequest, &pNodeList, pMnodeList); + code = scheduleQuery(pRequest, pRequest->body.pDag, pNodeList); + taosArrayDestroy(pNodeList); } - taosArrayDestroy(pNodeList); + taosArrayDestroy(pMnodeList); break; } case QUERY_EXEC_MODE_EMPTY_RESULT: @@ -697,7 +915,7 @@ SRequestObj* launchQuery(STscObj* pTscObj, const char* sql, int sqlLen) { return launchQueryImpl(pRequest, pQuery, false, NULL); } -void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { +void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery, SMetaData *pResultMeta) { int32_t code = 0; switch (pQuery->execMode) { @@ -708,7 +926,7 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { code = asyncExecDdlQuery(pRequest, pQuery); break; case QUERY_EXEC_MODE_SCHEDULE: { - SArray* pNodeList = taosArrayInit(4, sizeof(struct SQueryNodeAddr)); + SArray* pMnodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); pRequest->type = pQuery->msgType; @@ -721,15 +939,16 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { .msgLen = ERROR_MSG_BUF_DEFAULT_SIZE}; SAppInstInfo* pAppInfo = getAppInfo(pRequest); - if (TSDB_CODE_SUCCESS == code) { - code = qCreateQueryPlan(&cxt, &pRequest->body.pDag, pNodeList); - if (code) { - tscError("0x%" PRIx64 " failed to create query plan, code:%s 0x%" PRIx64, pRequest->self, tstrerror(code), - pRequest->requestId); - } + code = qCreateQueryPlan(&cxt, &pRequest->body.pDag, pMnodeList); + if (code) { + tscError("0x%" PRIx64 " failed to create query plan, code:%s 0x%" PRIx64, pRequest->self, tstrerror(code), + pRequest->requestId); } if (TSDB_CODE_SUCCESS == code) { + SArray* pNodeList = NULL; + buildAsyncExecNodeList(pRequest, &pNodeList, pMnodeList, pResultMeta); + SRequestConnInfo conn = { .pTrans = pAppInfo->pTransporter, .requestId = pRequest->requestId, .requestObjRefId = pRequest->self}; SSchedulerReq req = {.pConn = &conn, @@ -738,8 +957,10 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { .sql = pRequest->sqlstr, .startTs = pRequest->metric.start, .fp = schedulerExecCb, - .cbParam = pRequest}; + .cbParam = pRequest, + .reqKilled = &pRequest->killed}; code = schedulerAsyncExecJob(&req, &pRequest->body.queryJob); + taosArrayDestroy(pNodeList); } else { tscError("0x%" PRIx64 " failed to create query plan, code:%s 0x%" PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); @@ -747,7 +968,7 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) { } // todo not to be released here - taosArrayDestroy(pNodeList); + taosArrayDestroy(pMnodeList); break; } case QUERY_EXEC_MODE_EMPTY_RESULT: @@ -922,7 +1143,7 @@ STscObj* taosConnectImpl(const char* user, const char* auth, const char* db, __t taos_close_internal(pTscObj); pTscObj = NULL; } else { - tscDebug("0x%" PRIx64 " connection is opening, connId:%u, dnodeConn:%p, reqId:0x%" PRIx64, pTscObj->id, + tscDebug("0x%" PRIx64 " connection is opening, connId:%u, dnodeConn:%p, reqId:0x%" PRIx64, *(int64_t*)pTscObj->id, pTscObj->connId, pTscObj->pAppInfo->pTransporter, pRequest->requestId); destroyRequest(pRequest); } @@ -1085,10 +1306,10 @@ TAOS* taos_connect_auth(const char* ip, const char* user, const char* auth, cons STscObj* pObj = taos_connect_internal(ip, user, NULL, auth, db, port, CONN_TYPE__QUERY); if (pObj) { - return (TAOS*)pObj->id; + return pObj->id; } - return (TAOS*)0; + return NULL; } TAOS* taos_connect_l(const char* ip, int ipLen, const char* user, int userLen, const char* pass, int passLen, @@ -1327,70 +1548,174 @@ static int32_t doConvertUCS4(SReqResultInfo* pResultInfo, int32_t numOfRows, int pResultInfo->pCol[i].pData = pResultInfo->convertBuf[i]; pResultInfo->row[i] = pResultInfo->pCol[i].pData; - } else if (type == TSDB_DATA_TYPE_JSON && colLength[i] > 0) { - char* p = taosMemoryRealloc(pResultInfo->convertBuf[i], colLength[i]); - if (p == NULL) { - return TSDB_CODE_OUT_OF_MEMORY; - } + } + } + + return TSDB_CODE_SUCCESS; +} + +static int32_t estimateJsonLen(SReqResultInfo* pResultInfo, int32_t numOfCols, int32_t numOfRows){ + char* p = (char*)pResultInfo->pData; + + int32_t len = sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t)); + int32_t* colLength = (int32_t*)(p + len); + len += sizeof(int32_t) * numOfCols; + + char* pStart = p + len; + for (int32_t i = 0; i < numOfCols; ++i) { + int32_t colLen = htonl(colLength[i]); + + if (pResultInfo->fields[i].type == TSDB_DATA_TYPE_JSON) { + int32_t* offset = (int32_t*)pStart; + int32_t lenTmp = numOfRows * sizeof(int32_t); + len += lenTmp; + pStart += lenTmp; - pResultInfo->convertBuf[i] = p; - int32_t len = 0; - SResultColumn* pCol = &pResultInfo->pCol[i]; for (int32_t j = 0; j < numOfRows; ++j) { - if (pCol->offset[j] != -1) { - char* pStart = pCol->offset[j] + pCol->pData; + if (offset[j] == -1) { + continue; + } + char* data = offset[j] + pStart; + + int32_t jsonInnerType = *data; + char* jsonInnerData = data + CHAR_BYTES; + if (jsonInnerType == TSDB_DATA_TYPE_NULL) { + len += (VARSTR_HEADER_SIZE + strlen(TSDB_DATA_NULL_STR_L)); + } else if (jsonInnerType & TD_TAG_JSON) { + len += (VARSTR_HEADER_SIZE + ((const STag*)(data))->len); + } else if (jsonInnerType == TSDB_DATA_TYPE_NCHAR) { // value -> "value" + len += varDataTLen(jsonInnerData) + CHAR_BYTES * 2; + } else if (jsonInnerType == TSDB_DATA_TYPE_DOUBLE) { + len += (VARSTR_HEADER_SIZE + 32); + } else if (jsonInnerType == TSDB_DATA_TYPE_BOOL) { + len += (VARSTR_HEADER_SIZE + 5); + } else { + ASSERT(0); + } - int32_t jsonInnerType = *pStart; - char* jsonInnerData = pStart + CHAR_BYTES; - char dst[TSDB_MAX_JSON_TAG_LEN] = {0}; - if (jsonInnerType == TSDB_DATA_TYPE_NULL) { - sprintf(varDataVal(dst), "%s", TSDB_DATA_NULL_STR_L); - varDataSetLen(dst, strlen(varDataVal(dst))); - } else if (jsonInnerType == TD_TAG_JSON) { - char* jsonString = parseTagDatatoJson(pStart); - STR_TO_VARSTR(dst, jsonString); - taosMemoryFree(jsonString); - } else if (jsonInnerType == TSDB_DATA_TYPE_NCHAR) { // value -> "value" - *(char*)varDataVal(dst) = '\"'; - int32_t length = taosUcs4ToMbs((TdUcs4*)varDataVal(jsonInnerData), varDataLen(jsonInnerData), - varDataVal(dst) + CHAR_BYTES); - if (length <= 0) { - tscError("charset:%s to %s. convert failed.", DEFAULT_UNICODE_ENCODEC, tsCharset); - length = 0; - } - varDataSetLen(dst, length + CHAR_BYTES * 2); - *(char*)POINTER_SHIFT(varDataVal(dst), length + CHAR_BYTES) = '\"'; - } else if (jsonInnerType == TSDB_DATA_TYPE_DOUBLE) { - double jsonVd = *(double*)(jsonInnerData); - sprintf(varDataVal(dst), "%.9lf", jsonVd); - varDataSetLen(dst, strlen(varDataVal(dst))); - } else if (jsonInnerType == TSDB_DATA_TYPE_BOOL) { - sprintf(varDataVal(dst), "%s", (*((char*)jsonInnerData) == 1) ? "true" : "false"); - varDataSetLen(dst, strlen(varDataVal(dst))); - } else { - ASSERT(0); - } + } + } else if (IS_VAR_DATA_TYPE(pResultInfo->fields[i].type)) { + int32_t lenTmp = numOfRows * sizeof(int32_t); + len += (lenTmp + colLen); + pStart += lenTmp; + } else { + int32_t lenTmp = BitmapLen(pResultInfo->numOfRows); + len += (lenTmp + colLen); + pStart += lenTmp; + } + pStart += colLen; + } + return len; +} - if (len + varDataTLen(dst) > colLength[i]) { - p = taosMemoryRealloc(pResultInfo->convertBuf[i], len + varDataTLen(dst)); - if (p == NULL) { - return TSDB_CODE_OUT_OF_MEMORY; - } +static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int32_t numOfRows) { + bool needConvert = false; + for (int32_t i = 0; i < numOfCols; ++i) { + if (pResultInfo->fields[i].type == TSDB_DATA_TYPE_JSON) { + needConvert = true; + break; + } + } + if(!needConvert) return TSDB_CODE_SUCCESS; - pResultInfo->convertBuf[i] = p; + char* p = (char*)pResultInfo->pData; + int32_t dataLen = estimateJsonLen(pResultInfo, numOfCols, numOfRows); + + pResultInfo->convertJson = taosMemoryCalloc(1, dataLen); + if(pResultInfo->convertJson == NULL) return TSDB_CODE_OUT_OF_MEMORY; + char* p1 = pResultInfo->convertJson; + + int32_t len = sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t)); + memcpy(p1, p, len); + + p += len; + p1 += len; + + len = sizeof(int32_t) * numOfCols; + int32_t* colLength = (int32_t*)p; + int32_t* colLength1 = (int32_t*)p1; + memcpy(p1, p, len); + p += len; + p1 += len; + + char* pStart = p; + char* pStart1 = p1; + for (int32_t i = 0; i < numOfCols; ++i) { + int32_t colLen = htonl(colLength[i]); + int32_t colLen1 = htonl(colLength1[i]); + ASSERT(colLen < dataLen); + + if (pResultInfo->fields[i].type == TSDB_DATA_TYPE_JSON) { + int32_t* offset = (int32_t*)pStart; + int32_t* offset1 = (int32_t*)pStart1; + len = numOfRows * sizeof(int32_t); + memcpy(pStart1, pStart, len); + pStart += len; + pStart1 += len; + + len = 0; + for (int32_t j = 0; j < numOfRows; ++j) { + if (offset[j] == -1) { + continue; + } + char* data = offset[j] + pStart; + + int32_t jsonInnerType = *data; + char* jsonInnerData = data + CHAR_BYTES; + char dst[TSDB_MAX_JSON_TAG_LEN] = {0}; + if (jsonInnerType == TSDB_DATA_TYPE_NULL) { + sprintf(varDataVal(dst), "%s", TSDB_DATA_NULL_STR_L); + varDataSetLen(dst, strlen(varDataVal(dst))); + } else if (jsonInnerType & TD_TAG_JSON) { + char* jsonString = parseTagDatatoJson(data); + STR_TO_VARSTR(dst, jsonString); + taosMemoryFree(jsonString); + } else if (jsonInnerType == TSDB_DATA_TYPE_NCHAR) { // value -> "value" + *(char*)varDataVal(dst) = '\"'; + int32_t length = taosUcs4ToMbs((TdUcs4*)varDataVal(jsonInnerData), varDataLen(jsonInnerData), + varDataVal(dst) + CHAR_BYTES); + if (length <= 0) { + tscError("charset:%s to %s. convert failed.", DEFAULT_UNICODE_ENCODEC, tsCharset); + length = 0; } - p = pResultInfo->convertBuf[i] + len; - memcpy(p, dst, varDataTLen(dst)); - pCol->offset[j] = len; - len += varDataTLen(dst); + varDataSetLen(dst, length + CHAR_BYTES * 2); + *(char*)POINTER_SHIFT(varDataVal(dst), length + CHAR_BYTES) = '\"'; + } else if (jsonInnerType == TSDB_DATA_TYPE_DOUBLE) { + double jsonVd = *(double*)(jsonInnerData); + sprintf(varDataVal(dst), "%.9lf", jsonVd); + varDataSetLen(dst, strlen(varDataVal(dst))); + } else if (jsonInnerType == TSDB_DATA_TYPE_BOOL) { + sprintf(varDataVal(dst), "%s", (*((char*)jsonInnerData) == 1) ? "true" : "false"); + varDataSetLen(dst, strlen(varDataVal(dst))); + } else { + ASSERT(0); } + + offset1[j]= len; + memcpy(pStart1 + len, dst, varDataTLen(dst)); + len += varDataTLen(dst); } + colLen1 = len; + colLength1[i] = htonl(len); + } else if (IS_VAR_DATA_TYPE(pResultInfo->fields[i].type)) { + len = numOfRows * sizeof(int32_t); + memcpy(pStart1, pStart, len); + pStart += len; + pStart1 += len; + memcpy(pStart1, pStart, colLen); + } else { + len = BitmapLen(pResultInfo->numOfRows); + memcpy(pStart1, pStart, len); + pStart += len; + pStart1 += len; + memcpy(pStart1, pStart, colLen); - pResultInfo->pCol[i].pData = pResultInfo->convertBuf[i]; - pResultInfo->row[i] = pResultInfo->pCol[i].pData; } + pStart += colLen; + pStart1 += colLen1; } + pResultInfo->pData = pResultInfo->convertJson; return TSDB_CODE_SUCCESS; } @@ -1405,6 +1730,10 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 if (code != TSDB_CODE_SUCCESS) { return code; } + code = doConvertJson(pResultInfo, numOfCols, numOfRows); + if (code != TSDB_CODE_SUCCESS) { + return code; + } char* p = (char*)pResultInfo->pData; @@ -1448,8 +1777,7 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 pStart += colLength[i]; } - // convert UCS4-LE encoded character to native multi-bytes character in current data block. - if (convertUcs4) { + if(convertUcs4){ code = doConvertUCS4(pResultInfo, numOfRows, numOfCols, colLength); } diff --git a/source/client/src/clientMain.c b/source/client/src/clientMain.c index 13a5acb472dfce238f31909d37beb7ca69e59b12..c5a237f5eda0a457cfff68120f546cea7b8b430f 100644 --- a/source/client/src/clientMain.c +++ b/source/client/src/clientMain.c @@ -99,10 +99,10 @@ TAOS *taos_connect(const char *ip, const char *user, const char *pass, const cha STscObj* pObj = taos_connect_internal(ip, user, pass, NULL, db, port, CONN_TYPE__QUERY); if (pObj) { - return (TAOS*)pObj->id; + return pObj->id; } - return (TAOS*)0; + return NULL; } void taos_close_internal(void *taos) { @@ -111,19 +111,24 @@ void taos_close_internal(void *taos) { } STscObj *pTscObj = (STscObj *)taos; - tscDebug("0x%" PRIx64 " try to close connection, numOfReq:%d", pTscObj->id, pTscObj->numOfReqs); + tscDebug("0x%" PRIx64 " try to close connection, numOfReq:%d", *(int64_t*)pTscObj->id, pTscObj->numOfReqs); - taosRemoveRef(clientConnRefPool, pTscObj->id); + taosRemoveRef(clientConnRefPool, *(int64_t*)pTscObj->id); } void taos_close(TAOS *taos) { - STscObj* pObj = acquireTscObj((int64_t)taos); + if (taos == NULL) { + return; + } + + STscObj* pObj = acquireTscObj(*(int64_t*)taos); if (NULL == pObj) { return; } taos_close_internal(pObj); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); + taosMemoryFree(taos); } @@ -206,7 +211,12 @@ static void syncQueryFn(void *param, void *res, int32_t code) { } TAOS_RES *taos_query(TAOS *taos, const char *sql) { - STscObj* pTscObj = acquireTscObj((int64_t)taos); + if (NULL == taos) { + terrno = TSDB_CODE_TSC_DISCONNECTED; + return NULL; + } + + STscObj* pTscObj = acquireTscObj(*(int64_t*)taos); if (pTscObj == NULL || sql == NULL) { terrno = TSDB_CODE_TSC_DISCONNECTED; return NULL; @@ -219,13 +229,13 @@ TAOS_RES *taos_query(TAOS *taos, const char *sql) { taos_query_a(taos, sql, syncQueryFn, param); tsem_wait(¶m->sem); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return param->pRequest; #else size_t sqlLen = strlen(sql); if (sqlLen > (size_t)TSDB_MAX_ALLOWED_SQL_LEN) { - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); tscError("sql string exceeds max length:%d", TSDB_MAX_ALLOWED_SQL_LEN); terrno = TSDB_CODE_TSC_EXCEED_SQL_LIMIT; return NULL; @@ -233,7 +243,7 @@ TAOS_RES *taos_query(TAOS *taos, const char *sql) { TAOS_RES* pRes = execQuery(pTscObj, sql, sqlLen); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return pRes; #endif @@ -246,13 +256,14 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { if (TD_RES_QUERY(res)) { SRequestObj *pRequest = (SRequestObj *)res; -#if SYNC_ON_TOP_OF_ASYNC - return doAsyncFetchRows(pRequest, true, true); -#else if (pRequest->type == TSDB_SQL_RETRIEVE_EMPTY_RESULT || pRequest->type == TSDB_SQL_INSERT || - pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0) { + pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0 || pRequest->killed) { return NULL; } + +#if SYNC_ON_TOP_OF_ASYNC + return doAsyncFetchRows(pRequest, true, true); +#else return doFetchRows(pRequest, true, true); #endif @@ -452,15 +463,15 @@ int taos_result_precision(TAOS_RES *res) { } int taos_select_db(TAOS *taos, const char *db) { - STscObj* pObj = acquireTscObj((int64_t)taos); + STscObj* pObj = acquireTscObj(*(int64_t*)taos); if (pObj == NULL) { - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); terrno = TSDB_CODE_TSC_DISCONNECTED; return TSDB_CODE_TSC_DISCONNECTED; } if (db == NULL || strlen(db) == 0) { - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); terrno = TSDB_CODE_TSC_INVALID_INPUT; return terrno; } @@ -472,7 +483,7 @@ int taos_select_db(TAOS *taos, const char *db) { int32_t code = taos_errno(pRequest); taos_free_result(pRequest); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return code; } @@ -482,14 +493,20 @@ void taos_stop_query(TAOS_RES *res) { } SRequestObj *pRequest = (SRequestObj *)res; + pRequest->killed = true; + int32_t numOfFields = taos_num_fields(pRequest); - // It is not a query, no need to stop. if (numOfFields == 0) { + tscDebug("request %" PRIx64 " no need to be killed since not query", pRequest->requestId); return; } - schedulerFreeJob(pRequest->body.queryJob); + if (pRequest->body.queryJob) { + schedulerFreeJob(pRequest->body.queryJob, TSDB_CODE_TSC_QUERY_KILLED); + } + + tscDebug("request %" PRIx64 " killed", pRequest->requestId); } bool taos_is_null(TAOS_RES *res, int32_t row, int32_t col) { @@ -619,7 +636,7 @@ int *taos_get_column_data_offset(TAOS_RES *res, int columnIndex) { int taos_validate_sql(TAOS *taos, const char *sql) { return true; } void taos_reset_current_db(TAOS *taos) { - STscObj* pTscObj = acquireTscObj((int64_t)taos); + STscObj* pTscObj = acquireTscObj(*(int64_t*)taos); if (pTscObj == NULL) { terrno = TSDB_CODE_TSC_DISCONNECTED; return; @@ -627,17 +644,17 @@ void taos_reset_current_db(TAOS *taos) { resetConnectDB(pTscObj); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); } const char *taos_get_server_info(TAOS *taos) { - STscObj* pTscObj = acquireTscObj((int64_t)taos); + STscObj* pTscObj = acquireTscObj(*(int64_t*)taos); if (pTscObj == NULL) { terrno = TSDB_CODE_TSC_DISCONNECTED; return NULL; } - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return pTscObj->ver; } @@ -664,6 +681,8 @@ static void destorySqlParseWrapper(SqlParseWrapper *pWrapper) { } void retrieveMetaCallback(SMetaData *pResultMeta, void *param, int32_t code) { + tscDebug("enter meta callback, code %s", tstrerror(code)); + SqlParseWrapper *pWrapper = (SqlParseWrapper *)param; SQuery *pQuery = pWrapper->pQuery; SRequestObj *pRequest = pWrapper->pRequest; @@ -683,7 +702,7 @@ void retrieveMetaCallback(SMetaData *pResultMeta, void *param, int32_t code) { TSWAP(pRequest->tableList, (pQuery)->pTableList); destorySqlParseWrapper(pWrapper); - launchAsyncQuery(pRequest, pQuery); + launchAsyncQuery(pRequest, pQuery, pResultMeta); } else { destorySqlParseWrapper(pWrapper); tscDebug("error happens, code:%d", code); @@ -704,11 +723,11 @@ void retrieveMetaCallback(SMetaData *pResultMeta, void *param, int32_t code) { } void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param) { - STscObj* pTscObj = acquireTscObj((int64_t)taos); + STscObj* pTscObj = acquireTscObj(*(int64_t*)taos); if (pTscObj == NULL || sql == NULL || NULL == fp) { terrno = TSDB_CODE_INVALID_PARA; if (pTscObj) { - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); } else { terrno = TSDB_CODE_TSC_DISCONNECTED; } @@ -789,7 +808,7 @@ void doAsyncQuery(SRequestObj *pRequest, bool updateMetaForce) { SQuery *pQuery = NULL; - SCatalogReq catalogReq = {.forceUpdate = updateMetaForce}; + SCatalogReq catalogReq = {.forceUpdate = updateMetaForce, .qNodeRequired = qnodeRequired(pRequest)}; code = qParseSqlSyntax(pCxt, &pQuery, &catalogReq); if (code != TSDB_CODE_SUCCESS) { goto _error; @@ -830,6 +849,9 @@ static void fetchCallback(void *pResult, void *param, int32_t code) { SReqResultInfo *pResultInfo = &pRequest->body.resInfo; + tscDebug("0x%" PRIx64 " enter scheduler fetch cb, code:%d - %s, reqId:0x%" PRIx64, + pRequest->self, code, tstrerror(code), pRequest->requestId); + pResultInfo->pData = pResult; pResultInfo->numOfRows = 0; @@ -926,7 +948,7 @@ int taos_load_table_info(TAOS *taos, const char *tableNameList) { } TAOS_STMT *taos_stmt_init(TAOS *taos) { - STscObj* pObj = acquireTscObj((int64_t)taos); + STscObj* pObj = acquireTscObj(*(int64_t*)taos); if (NULL == pObj) { tscError("invalid parameter for %s", __FUNCTION__); terrno = TSDB_CODE_TSC_DISCONNECTED; @@ -935,7 +957,7 @@ TAOS_STMT *taos_stmt_init(TAOS *taos) { TAOS_STMT* pStmt = stmtInit(pObj); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return pStmt; } diff --git a/source/client/src/clientMsgHandler.c b/source/client/src/clientMsgHandler.c index bf627ec26b7bd6527be8feac03c33bfe64751039..db8aebb32293de4e20c95b96062283d4187e867e 100644 --- a/source/client/src/clientMsgHandler.c +++ b/source/client/src/clientMsgHandler.c @@ -77,7 +77,7 @@ int32_t processConnectRsp(void* param, const SDataBuf* pMsg, int32_t code) { for (int32_t i = 0; i < connectRsp.epSet.numOfEps; ++i) { tscDebug("0x%" PRIx64 " epSet.fqdn[%d]:%s port:%d, connObj:0x%" PRIx64, pRequest->requestId, i, - connectRsp.epSet.eps[i].fqdn, connectRsp.epSet.eps[i].port, pTscObj->id); + connectRsp.epSet.eps[i].fqdn, connectRsp.epSet.eps[i].port, *(int64_t*)pTscObj->id); } pTscObj->connId = connectRsp.connId; @@ -90,7 +90,7 @@ int32_t processConnectRsp(void* param, const SDataBuf* pMsg, int32_t code) { pTscObj->connType = connectRsp.connType; - hbRegisterConn(pTscObj->pAppInfo->pAppHbMgr, pTscObj->id, connectRsp.clusterId, connectRsp.connType); + hbRegisterConn(pTscObj->pAppInfo->pAppHbMgr, *(int64_t*)pTscObj->id, connectRsp.clusterId, connectRsp.connType); // pRequest->body.resInfo.pRspMsg = pMsg->pData; tscDebug("0x%" PRIx64 " clusterId:%" PRId64 ", totalConn:%" PRId64, pRequest->requestId, connectRsp.clusterId, diff --git a/source/client/src/clientSml.c b/source/client/src/clientSml.c index 25018e0d15a2064b02d6b10dc44e31bc5056d7c6..6fa4e6bffb35c11c4ae04f435dc34a67baddbb24 100644 --- a/source/client/src/clientSml.c +++ b/source/client/src/clientSml.c @@ -309,7 +309,7 @@ static int32_t smlApplySchemaAction(SSmlHandle *info, SSchemaAction *action) { case SCHEMA_ACTION_ADD_COLUMN: { int n = sprintf(result, "alter stable `%s` add column ", action->alterSTable.sTableName); smlBuildColumnDescription(action->alterSTable.field, result + n, capacity - n, &outBytes); - TAOS_RES *res = taos_query((TAOS*)info->taos->id, result); // TODO async doAsyncQuery + TAOS_RES *res = taos_query(info->taos->id, result); // TODO async doAsyncQuery code = taos_errno(res); const char *errStr = taos_errstr(res); if (code != TSDB_CODE_SUCCESS) { @@ -323,7 +323,7 @@ static int32_t smlApplySchemaAction(SSmlHandle *info, SSchemaAction *action) { case SCHEMA_ACTION_ADD_TAG: { int n = sprintf(result, "alter stable `%s` add tag ", action->alterSTable.sTableName); smlBuildColumnDescription(action->alterSTable.field, result + n, capacity - n, &outBytes); - TAOS_RES *res = taos_query((TAOS*)info->taos->id, result); // TODO async doAsyncQuery + TAOS_RES *res = taos_query(info->taos->id, result); // TODO async doAsyncQuery code = taos_errno(res); const char *errStr = taos_errstr(res); if (code != TSDB_CODE_SUCCESS) { @@ -337,7 +337,7 @@ static int32_t smlApplySchemaAction(SSmlHandle *info, SSchemaAction *action) { case SCHEMA_ACTION_CHANGE_COLUMN_SIZE: { int n = sprintf(result, "alter stable `%s` modify column ", action->alterSTable.sTableName); smlBuildColumnDescription(action->alterSTable.field, result + n, capacity - n, &outBytes); - TAOS_RES *res = taos_query((TAOS*)info->taos->id, result); // TODO async doAsyncQuery + TAOS_RES *res = taos_query(info->taos->id, result); // TODO async doAsyncQuery code = taos_errno(res); if (code != TSDB_CODE_SUCCESS) { uError("SML:0x%" PRIx64 " apply schema action. error : %s", info->id, taos_errstr(res)); @@ -350,7 +350,7 @@ static int32_t smlApplySchemaAction(SSmlHandle *info, SSchemaAction *action) { case SCHEMA_ACTION_CHANGE_TAG_SIZE: { int n = sprintf(result, "alter stable `%s` modify tag ", action->alterSTable.sTableName); smlBuildColumnDescription(action->alterSTable.field, result + n, capacity - n, &outBytes); - TAOS_RES *res = taos_query((TAOS*)info->taos->id, result); // TODO async doAsyncQuery + TAOS_RES *res = taos_query(info->taos->id, result); // TODO async doAsyncQuery code = taos_errno(res); if (code != TSDB_CODE_SUCCESS) { uError("SML:0x%" PRIx64 " apply schema action. error : %s", info->id, taos_errstr(res)); @@ -405,7 +405,7 @@ static int32_t smlApplySchemaAction(SSmlHandle *info, SSchemaAction *action) { pos--; ++freeBytes; outBytes = snprintf(pos, freeBytes, ")"); - TAOS_RES *res = taos_query((TAOS*)info->taos->id, result); + TAOS_RES *res = taos_query(info->taos->id, result); code = taos_errno(res); if (code != TSDB_CODE_SUCCESS) { uError("SML:0x%" PRIx64 " apply schema action. error : %s", info->id, taos_errstr(res)); @@ -2274,7 +2274,7 @@ static int32_t smlInsertData(SSmlHandle *info) { // info->affectedRows = taos_affected_rows(info->pRequest); // return info->pRequest->code; - launchAsyncQuery(info->pRequest, info->pQuery); + launchAsyncQuery(info->pRequest, info->pQuery, NULL); return TSDB_CODE_SUCCESS; } @@ -2434,7 +2434,7 @@ static void smlInsertCallback(void *param, void *res, int32_t code) { */ TAOS_RES* taos_schemaless_insert(TAOS* taos, char* lines[], int numLines, int protocol, int precision) { - STscObj* pTscObj = acquireTscObj((int64_t)taos); + STscObj* pTscObj = acquireTscObj(*(int64_t*)taos); if (NULL == pTscObj) { terrno = TSDB_CODE_TSC_DISCONNECTED; uError("SML:taos_schemaless_insert invalid taos"); @@ -2443,7 +2443,7 @@ TAOS_RES* taos_schemaless_insert(TAOS* taos, char* lines[], int numLines, int pr SRequestObj* request = (SRequestObj*)createRequest(pTscObj, TSDB_SQL_INSERT); if(!request){ - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); uError("SML:taos_schemaless_insert error request is null"); return NULL; } @@ -2531,6 +2531,6 @@ end: // ((STscObj *)taos)->schemalessType = 0; pTscObj->schemalessType = 1; uDebug("resultend:%s", request->msgBuf); - releaseTscObj((int64_t)taos); + releaseTscObj(*(int64_t*)taos); return (TAOS_RES*)request; } diff --git a/source/client/src/tmq.c b/source/client/src/tmq.c index dd900e6045294c8d6d6eedee733a682758a24cb9..638b4f1ea559fed1d76009f1f1141aa1c9e36e07 100644 --- a/source/client/src/tmq.c +++ b/source/client/src/tmq.c @@ -48,14 +48,6 @@ struct tmq_list_t { SArray container; }; -struct tmq_topic_vgroup_t { - SMqOffset offset; -}; - -struct tmq_topic_vgroup_list_t { - SArray container; // SArray -}; - struct tmq_conf_t { char clientId[256]; char groupId[TSDB_CGROUP_LEN]; @@ -161,9 +153,9 @@ typedef struct { } SMqPollRspWrapper; typedef struct { - tmq_t* tmq; - tsem_t rspSem; - tmq_resp_err_t rspErr; + tmq_t* tmq; + tsem_t rspSem; + int32_t rspErr; } SMqSubscribeCbParam; typedef struct { @@ -189,7 +181,7 @@ typedef struct { int8_t freeOffsets; tmq_commit_cb* userCb; tsem_t rspSem; - tmq_resp_err_t rspErr; + int32_t rspErr; SArray* offsets; void* userParam; } SMqCommitCbParam; @@ -201,12 +193,12 @@ typedef struct { int8_t freeOffsets; int32_t waitingRspNum; int32_t totalRspNum; - tmq_resp_err_t rspErr; + int32_t rspErr; tmq_commit_cb* userCb; - SArray* successfulOffsets; - SArray* failedOffsets; - void* userParam; - tsem_t rspSem; + /*SArray* successfulOffsets;*/ + /*SArray* failedOffsets;*/ + void* userParam; + tsem_t rspSem; } SMqCommitCbParamSet; typedef struct { @@ -347,10 +339,9 @@ int32_t tmqCommitCb(void* param, const SDataBuf* pMsg, int32_t code) { pParam->rspErr = code; if (pParam->async) { if (pParam->automatic && pParam->tmq->commitCb) { - pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets, - pParam->tmq->commitCbUserParam); + pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, pParam->tmq->commitCbUserParam); } else if (!pParam->automatic && pParam->userCb) { - pParam->userCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets, pParam->userParam); + pParam->userCb(pParam->tmq, pParam->rspErr, pParam->userParam); } if (pParam->freeOffsets) { @@ -368,11 +359,16 @@ int32_t tmqCommitCb2(void* param, const SDataBuf* pBuf, int32_t code) { SMqCommitCbParam2* pParam = (SMqCommitCbParam2*)param; SMqCommitCbParamSet* pParamSet = (SMqCommitCbParamSet*)pParam->params; // push into array +#if 0 if (code == 0) { taosArrayPush(pParamSet->failedOffsets, &pParam->pOffset); } else { taosArrayPush(pParamSet->successfulOffsets, &pParam->pOffset); } +#endif + + /*tscDebug("receive offset commit cb of %s on vg %d, offset is %ld", pParam->pOffset->subKey, pParam->->vgId, + * pOffset->version);*/ // count down waiting rsp int32_t waitingRspNum = atomic_sub_fetch_32(&pParamSet->waitingRspNum, 1); @@ -383,23 +379,149 @@ int32_t tmqCommitCb2(void* param, const SDataBuf* pBuf, int32_t code) { if (pParamSet->async) { // call async cb func if (pParamSet->automatic && pParamSet->tmq->commitCb) { - pParamSet->tmq->commitCb(pParamSet->tmq, pParamSet->rspErr, NULL, pParamSet->tmq->commitCbUserParam); + pParamSet->tmq->commitCb(pParamSet->tmq, pParamSet->rspErr, pParamSet->tmq->commitCbUserParam); } else if (!pParamSet->automatic && pParamSet->userCb) { // sem post - pParamSet->userCb(pParamSet->tmq, pParamSet->rspErr, NULL, pParamSet->userParam); + pParamSet->userCb(pParamSet->tmq, pParamSet->rspErr, pParamSet->userParam); } + } else { + tsem_post(&pParamSet->rspSem); } +#if 0 taosArrayDestroyP(pParamSet->successfulOffsets, taosMemoryFree); taosArrayDestroyP(pParamSet->failedOffsets, taosMemoryFree); +#endif } return 0; } -int32_t tmqCommitInner2(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_t automatic, int8_t async, - tmq_commit_cb* userCb, void* userParam) { +int32_t tmqCommitInner2(tmq_t* tmq, const TAOS_RES* msg, int8_t automatic, int8_t async, tmq_commit_cb* userCb, + void* userParam) { int32_t code = -1; + if (msg != NULL) { + SMqRspObj* pRspObj = (SMqRspObj*)msg; + if (!TD_RES_TMQ(pRspObj)) { + return TSDB_CODE_TMQ_INVALID_MSG; + } + + SMqCommitCbParamSet* pParamSet = taosMemoryCalloc(1, sizeof(SMqCommitCbParamSet)); + if (pParamSet == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return -1; + } + pParamSet->tmq = tmq; + pParamSet->automatic = automatic; + pParamSet->async = async; + pParamSet->freeOffsets = 1; + pParamSet->userCb = userCb; + pParamSet->userParam = userParam; + tsem_init(&pParamSet->rspSem, 0, 0); + + for (int32_t i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { + SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); + if (strcmp(pTopic->topicName, pRspObj->topic) == 0) { + for (int32_t j = 0; j < taosArrayGetSize(pTopic->vgs); j++) { + SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); + if (pVg->vgId == pRspObj->vgId) { + if (pVg->currentOffset < 0 || pVg->committedOffset == pVg->currentOffset) { + tscDebug("consumer %ld skip commit for topic %s vg %d, current offset is %ld, committed offset is %ld", + tmq->consumerId, pTopic->topicName, pVg->vgId, pVg->currentOffset, pVg->committedOffset); + + return 0; + } + + STqOffset* pOffset = taosMemoryCalloc(1, sizeof(STqOffset)); + if (pOffset == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return -1; + } + pOffset->type = TMQ_OFFSET__LOG; + pOffset->version = pVg->currentOffset; + + int32_t groupLen = strlen(tmq->groupId); + memcpy(pOffset->subKey, tmq->groupId, groupLen); + pOffset->subKey[groupLen] = TMQ_SEPARATOR; + strcpy(pOffset->subKey + groupLen + 1, pTopic->topicName); + + int32_t len; + int32_t code; + tEncodeSize(tEncodeSTqOffset, pOffset, len, code); + if (code < 0) { + ASSERT(0); + } + void* buf = taosMemoryCalloc(1, sizeof(SMsgHead) + len); + ((SMsgHead*)buf)->vgId = htonl(pVg->vgId); + + void* abuf = POINTER_SHIFT(buf, sizeof(SMsgHead)); + + SEncoder encoder; + tEncoderInit(&encoder, abuf, len); + tEncodeSTqOffset(&encoder, pOffset); + + // build param + SMqCommitCbParam2* pParam = taosMemoryCalloc(1, sizeof(SMqCommitCbParam2)); + pParam->params = pParamSet; + pParam->pOffset = pOffset; + + // build send info + SMsgSendInfo* pMsgSendInfo = taosMemoryCalloc(1, sizeof(SMsgSendInfo)); + if (pMsgSendInfo == NULL) { + // TODO + continue; + } + pMsgSendInfo->msgInfo = (SDataBuf){ + .pData = buf, + .len = sizeof(SMsgHead) + len, + .handle = NULL, + }; + + tscDebug("consumer %ld commit offset of %s on vg %d, offset is %ld", tmq->consumerId, pOffset->subKey, + pVg->vgId, pOffset->version); + + // TODO: put into cb + pVg->committedOffset = pVg->currentOffset; + + pMsgSendInfo->requestId = generateRequestId(); + pMsgSendInfo->requestObjRefId = 0; + pMsgSendInfo->param = pParam; + pMsgSendInfo->fp = tmqCommitCb2; + pMsgSendInfo->msgType = TDMT_VND_MQ_COMMIT_OFFSET; + // send msg + + int64_t transporterId = 0; + asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &pVg->epSet, &transporterId, pMsgSendInfo); + pParamSet->waitingRspNum++; + pParamSet->totalRspNum++; + } + } + } + } + if (pParamSet->totalRspNum == 0) { + tsem_destroy(&pParamSet->rspSem); + taosMemoryFree(pParamSet); + return 0; + } + + if (!async) { + tsem_wait(&pParamSet->rspSem); + code = pParamSet->rspErr; + tsem_destroy(&pParamSet->rspSem); + } else { + code = 0; + } + + if (code != 0 && async) { + if (automatic) { + tmq->commitCb(tmq, code, tmq->commitCbUserParam); + } else { + userCb(tmq, code, userParam); + } + } + return 0; + } + SMqCommitCbParamSet* pParamSet = taosMemoryCalloc(1, sizeof(SMqCommitCbParamSet)); if (pParamSet == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; @@ -415,17 +537,35 @@ int32_t tmqCommitInner2(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8 for (int32_t i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); + + tscDebug("consumer %ld begin commit for topic %s, vgNum %d", tmq->consumerId, pTopic->topicName, + (int32_t)taosArrayGetSize(pTopic->vgs)); + for (int32_t j = 0; j < taosArrayGetSize(pTopic->vgs); j++) { - SMqClientVg* pVg = taosArrayGet(pTopic->vgs, i); - STqOffset* pOffset = taosMemoryCalloc(1, sizeof(STqOffset)); + SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); + + tscDebug("consumer %ld begin commit for topic %s, vgId %d", tmq->consumerId, pTopic->topicName, pVg->vgId); + + /*if (pVg->currentOffset < 0) {*/ + if (pVg->currentOffset < 0 || pVg->committedOffset == pVg->currentOffset) { + tscDebug("consumer %ld skip commit for topic %s vg %d, current offset is %ld, committed offset is %ld", + tmq->consumerId, pTopic->topicName, pVg->vgId, pVg->currentOffset, pVg->committedOffset); + + continue; + } + STqOffset* pOffset = taosMemoryCalloc(1, sizeof(STqOffset)); if (pOffset == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; return -1; } - int32_t tlen = strlen(tmq->groupId); - memcpy(pOffset->subKey, tmq->groupId, tlen); - pOffset->subKey[tlen] = TMQ_SEPARATOR; - strcpy(pOffset->subKey + tlen + 1, pTopic->topicName); + pOffset->type = TMQ_OFFSET__LOG; + pOffset->version = pVg->currentOffset; + + int32_t groupLen = strlen(tmq->groupId); + memcpy(pOffset->subKey, tmq->groupId, groupLen); + pOffset->subKey[groupLen] = TMQ_SEPARATOR; + strcpy(pOffset->subKey + groupLen + 1, pTopic->topicName); + int32_t len; int32_t code; tEncodeSize(tEncodeSTqOffset, pOffset, len, code); @@ -454,25 +594,36 @@ int32_t tmqCommitInner2(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8 } pMsgSendInfo->msgInfo = (SDataBuf){ .pData = buf, - .len = len, + .len = sizeof(SMsgHead) + len, .handle = NULL, }; + tscDebug("consumer %ld commit offset of %s on vg %d, offset is %ld", tmq->consumerId, pOffset->subKey, pVg->vgId, + pOffset->version); + + // TODO: put into cb + pVg->committedOffset = pVg->currentOffset; + pMsgSendInfo->requestId = generateRequestId(); pMsgSendInfo->requestObjRefId = 0; pMsgSendInfo->param = pParam; pMsgSendInfo->fp = tmqCommitCb2; - pMsgSendInfo->msgType = TDMT_MND_MQ_COMMIT_OFFSET; + pMsgSendInfo->msgType = TDMT_VND_MQ_COMMIT_OFFSET; // send msg - SEpSet epSet = getEpSet_s(&tmq->pTscObj->pAppInfo->mgmtEp); int64_t transporterId = 0; - asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, pMsgSendInfo); + asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &pVg->epSet, &transporterId, pMsgSendInfo); pParamSet->waitingRspNum++; pParamSet->totalRspNum++; } } + if (pParamSet->totalRspNum == 0) { + tsem_destroy(&pParamSet->rspSem); + taosMemoryFree(pParamSet); + return 0; + } + if (!async) { tsem_wait(&pParamSet->rspSem); code = pParamSet->rspErr; @@ -483,21 +634,24 @@ int32_t tmqCommitInner2(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8 if (code != 0 && async) { if (automatic) { - tmq->commitCb(tmq, code, NULL, tmq->commitCbUserParam); + tmq->commitCb(tmq, code, tmq->commitCbUserParam); } else { - userCb(tmq, code, NULL, userParam); + userCb(tmq, code, userParam); } } +#if 0 if (!async) { taosArrayDestroyP(pParamSet->successfulOffsets, taosMemoryFree); taosArrayDestroyP(pParamSet->failedOffsets, taosMemoryFree); } +#endif return 0; } -int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_t automatic, int8_t async, +#if 0 +int32_t tmqCommitInner(tmq_t* tmq, const TAOS_RES* msg, int8_t automatic, int8_t async, tmq_commit_cb* userCb, void* userParam) { SMqCMCommitOffsetReq req; SArray* pOffsets = NULL; @@ -507,7 +661,7 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_ int8_t freeOffsets; int32_t code = -1; - if (offsets == NULL) { + if (msg == NULL) { freeOffsets = 1; pOffsets = taosArrayInit(0, sizeof(SMqOffset)); for (int32_t i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { @@ -524,7 +678,7 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_ } } else { freeOffsets = 0; - pOffsets = (SArray*)&offsets->container; + pOffsets = (SArray*)&msg->container; } req.num = (int32_t)pOffsets->size; @@ -611,6 +765,7 @@ END: } return code; } +#endif void tmqAssignDelayedHbTask(void* param, void* tmrId) { tmq_t* tmq = (tmq_t*)param; @@ -648,7 +803,7 @@ int32_t tmqHandleAllDelayedTask(tmq_t* tmq) { tmqAskEp(tmq, true); taosTmrReset(tmqAssignDelayedHbTask, 1000, tmq, tmqMgmt.timer, &tmq->hbTimer); } else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) { - tmqCommitInner(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam); + tmqCommitInner2(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam); taosTmrReset(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, tmq, tmqMgmt.timer, &tmq->commitTimer); } else if (*pTaskType == TMQ_DELAYED_TASK__REPORT) { } else { @@ -689,7 +844,7 @@ int32_t tmqSubscribeCb(void* param, const SDataBuf* pMsg, int32_t code) { return 0; } -tmq_resp_err_t tmq_subscription(tmq_t* tmq, tmq_list_t** topics) { +int32_t tmq_subscription(tmq_t* tmq, tmq_list_t** topics) { if (*topics == NULL) { *topics = tmq_list_new(); } @@ -697,12 +852,12 @@ tmq_resp_err_t tmq_subscription(tmq_t* tmq, tmq_list_t** topics) { SMqClientTopic* topic = taosArrayGet(tmq->clientTopics, i); tmq_list_append(*topics, strchr(topic->topicName, '.') + 1); } - return TMQ_RESP_ERR__SUCCESS; + return 0; } -tmq_resp_err_t tmq_unsubscribe(tmq_t* tmq) { - tmq_list_t* lst = tmq_list_new(); - tmq_resp_err_t rsp = tmq_subscribe(tmq, lst); +int32_t tmq_unsubscribe(tmq_t* tmq) { + tmq_list_t* lst = tmq_list_new(); + int32_t rsp = tmq_subscribe(tmq, lst); tmq_list_destroy(lst); return rsp; } @@ -818,11 +973,13 @@ FAIL: return NULL; } -tmq_resp_err_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) { - return tmqCommitInner(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam); +#if 0 +int32_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) { + return tmqCommitInner2(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam); } +#endif -tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { +int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { const SArray* container = &topic_list->container; int32_t sz = taosArrayGetSize(container); void* buf = NULL; @@ -862,7 +1019,7 @@ tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { if (sendInfo == NULL) goto FAIL; SMqSubscribeCbParam param = { - .rspErr = TMQ_RESP_ERR__SUCCESS, + .rspErr = 0, .tmq = tmq, }; @@ -943,6 +1100,19 @@ int32_t tmqPollCb(void* param, const SDataBuf* pMsg, int32_t code) { if (code != 0) { tscWarn("msg discard from vg %d, epoch %d, code:%x", vgId, epoch, code); if (pMsg->pData) taosMemoryFree(pMsg->pData); + if (code == TSDB_CODE_TQ_NO_COMMITTED_OFFSET) { + SMqPollRspWrapper* pRspWrapper = taosAllocateQitem(sizeof(SMqPollRspWrapper), DEF_QITEM); + if (pRspWrapper == NULL) { + taosMemoryFree(pMsg->pData); + tscWarn("msg discard from vg %d, epoch %d since out of memory", vgId, epoch); + goto CREATE_MSG_FAIL; + } + pRspWrapper->tmqRspType = TMQ_MSG_TYPE__END_RSP; + /*pRspWrapper->vgHandle = pVg;*/ + /*pRspWrapper->topicHandle = pTopic;*/ + taosWriteQitem(tmq->mqueue, pRspWrapper); + tsem_post(&tmq->rspSem); + } goto CREATE_MSG_FAIL; } @@ -1017,14 +1187,13 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { if (pTopicCur->vgs) { int32_t vgNumCur = taosArrayGetSize(pTopicCur->vgs); tscDebug("consumer %ld new vg num: %d", tmq->consumerId, vgNumCur); - if (vgNumCur == 0) break; for (int32_t j = 0; j < vgNumCur; j++) { SMqClientVg* pVgCur = taosArrayGet(pTopicCur->vgs, j); sprintf(vgKey, "%s:%d", pTopicCur->topicName, pVgCur->vgId); - tscDebug("consumer %ld epoch %d vg %d build %s", tmq->consumerId, epoch, pVgCur->vgId, vgKey); + tscDebug("consumer %ld epoch %d vg %d vgKey is %s, offset is %ld", tmq->consumerId, epoch, pVgCur->vgId, vgKey, + pVgCur->currentOffset); taosHashPut(pHash, vgKey, strlen(vgKey), &pVgCur->currentOffset, sizeof(int64_t)); } - break; } } @@ -1032,7 +1201,6 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { SMqClientTopic topic = {0}; SMqSubTopicEp* pTopicEp = taosArrayGet(pRsp->topics, i); topic.schema = pTopicEp->schema; - taosHashClear(pHash); topic.topicName = strdup(pTopicEp->topic); tstrncpy(topic.db, pTopicEp->db, TSDB_DB_FNAME_LEN); @@ -1049,7 +1217,8 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { offset = *pOffset; } - tscDebug("consumer %ld(epoch %d) offset of vg %d updated to %ld", tmq->consumerId, epoch, pVgEp->vgId, offset); + tscDebug("consumer %ld(epoch %d) offset of vg %d updated to %ld, vgKey is %s", tmq->consumerId, epoch, + pVgEp->vgId, offset, vgKey); SMqClientVg clientVg = { .pollCnt = 0, .currentOffset = offset, @@ -1076,6 +1245,7 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { return set; } +#if 1 bool tmqUpdateEp(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { /*printf("call update ep %d\n", epoch);*/ bool set = false; @@ -1160,6 +1330,7 @@ bool tmqUpdateEp(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) { atomic_store_32(&tmq->epoch, epoch); return set; } +#endif int32_t tmqAskEpCb(void* param, const SDataBuf* pMsg, int32_t code) { SMqAskEpCbParam* pParam = (SMqAskEpCbParam*)param; @@ -1186,7 +1357,7 @@ int32_t tmqAskEpCb(void* param, const SDataBuf* pMsg, int32_t code) { tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &rsp); /*printf("rsp epoch %ld sz %ld\n", rsp.epoch, rsp.topics->size);*/ /*printf("tmq epoch %ld sz %ld\n", tmq->epoch, tmq->clientTopics->size);*/ - tmqUpdateEp(tmq, head->epoch, &rsp); + tmqUpdateEp2(tmq, head->epoch, &rsp); tDeleteSMqAskEpRsp(&rsp); } else { SMqAskEpRspWrapper* pWrapper = taosAllocateQitem(sizeof(SMqAskEpRspWrapper), DEF_QITEM); @@ -1283,7 +1454,8 @@ int32_t tmqAskEp(tmq_t* tmq, bool async) { return code; } -tmq_resp_err_t tmq_seek(tmq_t* tmq, const tmq_topic_vgroup_t* offset) { +#if 0 +int32_t tmq_seek(tmq_t* tmq, const tmq_topic_vgroup_t* offset) { const SMqOffset* pOffset = &offset->offset; if (strcmp(pOffset->cgroup, tmq->groupId) != 0) { return TMQ_RESP_ERR__FAIL; @@ -1305,16 +1477,17 @@ tmq_resp_err_t tmq_seek(tmq_t* tmq, const tmq_topic_vgroup_t* offset) { } return TMQ_RESP_ERR__FAIL; } +#endif SMqPollReq* tmqBuildConsumeReqImpl(tmq_t* tmq, int64_t timeout, SMqClientTopic* pTopic, SMqClientVg* pVg) { int64_t reqOffset; if (pVg->currentOffset >= 0) { reqOffset = pVg->currentOffset; } else { - if (tmq->resetOffsetCfg == TMQ_CONF__RESET_OFFSET__NONE) { - tscError("unable to poll since no committed offset but reset offset is set to none"); - return NULL; - } + /*if (tmq->resetOffsetCfg == TMQ_CONF__RESET_OFFSET__NONE) {*/ + /*tscError("unable to poll since no committed offset but reset offset is set to none");*/ + /*return NULL;*/ + /*}*/ reqOffset = tmq->resetOffsetCfg; } @@ -1326,10 +1499,10 @@ SMqPollReq* tmqBuildConsumeReqImpl(tmq_t* tmq, int64_t timeout, SMqClientTopic* /*strcpy(pReq->topic, pTopic->topicName);*/ /*strcpy(pReq->cgroup, tmq->groupId);*/ - int32_t tlen = strlen(tmq->groupId); - memcpy(pReq->subKey, tmq->groupId, tlen); - pReq->subKey[tlen] = TMQ_SEPARATOR; - strcpy(pReq->subKey + tlen + 1, pTopic->topicName); + int32_t groupLen = strlen(tmq->groupId); + memcpy(pReq->subKey, tmq->groupId, groupLen); + pReq->subKey[groupLen] = TMQ_SEPARATOR; + strcpy(pReq->subKey + groupLen + 1, pTopic->topicName); pReq->withTbName = tmq->withTbName; pReq->timeout = timeout; @@ -1440,7 +1613,7 @@ int32_t tmqHandleNoPollRsp(tmq_t* tmq, SMqRspWrapper* rspWrapper, bool* pReset) if (rspWrapper->epoch > atomic_load_32(&tmq->epoch)) { SMqAskEpRspWrapper* pEpRspWrapper = (SMqAskEpRspWrapper*)rspWrapper; SMqAskEpRsp* rspMsg = &pEpRspWrapper->msg; - tmqUpdateEp(tmq, rspWrapper->epoch, rspMsg); + tmqUpdateEp2(tmq, rspWrapper->epoch, rspMsg); /*tmqClearUnhandleMsg(tmq);*/ *pReset = true; } else { @@ -1462,7 +1635,11 @@ SMqRspObj* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout, bool pollIfReset) { if (rspWrapper == NULL) return NULL; } - if (rspWrapper->tmqRspType == TMQ_MSG_TYPE__POLL_RSP) { + if (rspWrapper->tmqRspType == TMQ_MSG_TYPE__END_RSP) { + taosFreeQitem(rspWrapper); + terrno = TSDB_CODE_TQ_NO_COMMITTED_OFFSET; + return NULL; + } else if (rspWrapper->tmqRspType == TMQ_MSG_TYPE__POLL_RSP) { SMqPollRspWrapper* pollRspWrapper = (SMqPollRspWrapper*)rspWrapper; /*atomic_sub_fetch_32(&tmq->readyRequest, 1);*/ int32_t consumerEpoch = atomic_load_32(&tmq->epoch); @@ -1523,6 +1700,8 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) { rspObj = tmqHandleAllRsp(tmq, timeout, false); if (rspObj) { return (TAOS_RES*)rspObj; + } else if (terrno == TSDB_CODE_TQ_NO_COMMITTED_OFFSET) { + return NULL; } if (timeout != -1) { int64_t endTime = taosGetTimestampMs(); @@ -1539,10 +1718,10 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) { } } -tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { +int32_t tmq_consumer_close(tmq_t* tmq) { if (tmq->status == TMQ_CONSUMER_STATUS__READY) { - tmq_resp_err_t rsp = tmq_commit_sync(tmq, NULL); - if (rsp != TMQ_RESP_ERR__SUCCESS) { + int32_t rsp = tmq_commit_sync(tmq, NULL); + if (rsp != 0) { return rsp; } @@ -1550,18 +1729,18 @@ tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) { rsp = tmq_subscribe(tmq, lst); tmq_list_destroy(lst); - if (rsp != TMQ_RESP_ERR__SUCCESS) { + if (rsp != 0) { return rsp; } } // TODO: free resources - return TMQ_RESP_ERR__SUCCESS; + return 0; } -const char* tmq_err2str(tmq_resp_err_t err) { - if (err == TMQ_RESP_ERR__SUCCESS) { +const char* tmq_err2str(int32_t err) { + if (err == 0) { return "success"; - } else if (err == TMQ_RESP_ERR__FAIL) { + } else if (err == -1) { return "fail"; } else { return tstrerror(err); @@ -1607,10 +1786,8 @@ const char* tmq_get_table_name(TAOS_RES* res) { return NULL; } -void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) { - tmqCommitInner(tmq, offsets, 0, 1, cb, param); +void tmq_commit_async(tmq_t* tmq, const TAOS_RES* msg, tmq_commit_cb* cb, void* param) { + tmqCommitInner2(tmq, msg, 0, 1, cb, param); } -tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) { - return tmqCommitInner(tmq, offsets, 0, 0, NULL, NULL); -} +int32_t tmq_commit_sync(tmq_t* tmq, const TAOS_RES* msg) { return tmqCommitInner2(tmq, msg, 0, 0, NULL, NULL); } diff --git a/source/common/src/systable.c b/source/common/src/systable.c index bcc248d122172861daaf0bad5165340b2a22082d..e7b6342150a4b48242eac408c021ad9cc2bfa170 100644 --- a/source/common/src/systable.c +++ b/source/common/src/systable.c @@ -77,7 +77,7 @@ static const SSysDbTableSchema userDBSchema[] = { {.name = "ntables", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT}, {.name = "replica", .bytes = 1, .type = TSDB_DATA_TYPE_TINYINT}, {.name = "strict", .bytes = 9 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, - {.name = "duration", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, + {.name = "duration", .bytes = 10 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, {.name = "keep", .bytes = 32 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, {.name = "buffer", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "pagesize", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, @@ -124,7 +124,7 @@ static const SSysDbTableSchema userStbsSchema[] = { {.name = "columns", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "tags", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "last_update", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP}, - {.name = "table_comment", .bytes = 1024 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "table_comment", .bytes = TSDB_TB_COMMENT_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, }; static const SSysDbTableSchema streamSchema[] = { @@ -148,7 +148,7 @@ static const SSysDbTableSchema userTblsSchema[] = { {.name = "uid", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT}, {.name = "vgroup_id", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "ttl", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, - {.name = "table_comment", .bytes = 512 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "table_comment", .bytes = TSDB_TB_COMMENT_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, {.name = "type", .bytes = 20 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, }; @@ -220,7 +220,8 @@ static const SSysDbTableSchema transSchema[] = { {.name = "id", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "create_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP}, {.name = "stage", .bytes = TSDB_TRANS_STAGE_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, - {.name = "db", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "db1", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "db2", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR}, {.name = "failed_times", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, {.name = "last_exec_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP}, {.name = "last_action_info", @@ -302,7 +303,7 @@ static const SSysDbTableSchema offsetSchema[] = { }; static const SSysDbTableSchema querySchema[] = { - {.name = "query_id", .bytes = 26 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "query_id", .bytes = TSDB_QUERY_ID_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, {.name = "req_id", .bytes = 8, .type = TSDB_DATA_TYPE_UBIGINT}, {.name = "connId", .bytes = 4, .type = TSDB_DATA_TYPE_UINT}, {.name = "app", .bytes = TSDB_APP_NAME_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, @@ -317,6 +318,24 @@ static const SSysDbTableSchema querySchema[] = { {.name = "sql", .bytes = TSDB_SHOW_SQL_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, }; +static const SSysDbTableSchema appSchema[] = { + {.name = "app_id", .bytes = 8, .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "ip", .bytes = TSDB_IPv4ADDR_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "pid", .bytes = 4, .type = TSDB_DATA_TYPE_INT}, + {.name = "name", .bytes = TSDB_APP_NAME_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR}, + {.name = "start_time", .bytes = 8 , .type = TSDB_DATA_TYPE_TIMESTAMP}, + {.name = "insert_req", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "insert_row", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "insert_time", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "insert_bytes", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "fetch_bytes", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "query_time", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "show_query", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "total_req", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "current_req", .bytes = 8 , .type = TSDB_DATA_TYPE_UBIGINT}, + {.name = "last_access", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP}, +}; + static const SSysTableMeta perfsMeta[] = { {TSDB_PERFS_TABLE_CONNECTIONS, connectionsSchema, tListLen(connectionsSchema)}, {TSDB_PERFS_TABLE_QUERIES, querySchema, tListLen(querySchema)}, @@ -327,6 +346,7 @@ static const SSysTableMeta perfsMeta[] = { {TSDB_PERFS_TABLE_TRANS, transSchema, tListLen(transSchema)}, {TSDB_PERFS_TABLE_SMAS, smaSchema, tListLen(smaSchema)}, {TSDB_PERFS_TABLE_STREAMS, streamSchema, tListLen(streamSchema)}, + {TSDB_PERFS_TABLE_APPS, appSchema, tListLen(appSchema)} }; void getInfosDbMeta(const SSysTableMeta** pInfosTableMeta, size_t* size) { diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index 712b4fcf42a63da1aaf6ce14200efab8864177e4..3c3d3e953dc567f7f9edb83d56d6ff48dca9edf6 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -1713,6 +1713,7 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks } char* buildCtbNameByGroupId(const char* stbName, uint64_t groupId) { + ASSERT(stbName[0] != 0); SArray* tags = taosArrayInit(0, sizeof(void*)); SSmlKv* pTag = taosMemoryCalloc(1, sizeof(SSmlKv)); pTag->key = "group_id"; diff --git a/source/common/src/tglobal.c b/source/common/src/tglobal.c index fbb4f784254b87c7d3975325e24da210cefd09ed..269c92a670ffed95c2cd2b5d1f76bf97d5615f1b 100644 --- a/source/common/src/tglobal.c +++ b/source/common/src/tglobal.c @@ -51,15 +51,16 @@ int32_t tsNumOfShmThreads = 1; int32_t tsNumOfRpcThreads = 1; int32_t tsNumOfCommitThreads = 2; int32_t tsNumOfTaskQueueThreads = 1; -int32_t tsNumOfMnodeQueryThreads = 1; +int32_t tsNumOfMnodeQueryThreads = 2; +int32_t tsNumOfMnodeFetchThreads = 1; int32_t tsNumOfMnodeReadThreads = 1; int32_t tsNumOfVnodeQueryThreads = 2; -int32_t tsNumOfVnodeFetchThreads = 2; +int32_t tsNumOfVnodeFetchThreads = 1; int32_t tsNumOfVnodeWriteThreads = 2; int32_t tsNumOfVnodeSyncThreads = 2; int32_t tsNumOfVnodeMergeThreads = 2; int32_t tsNumOfQnodeQueryThreads = 2; -int32_t tsNumOfQnodeFetchThreads = 2; +int32_t tsNumOfQnodeFetchThreads = 1; int32_t tsNumOfSnodeSharedThreads = 2; int32_t tsNumOfSnodeUniqueThreads = 2; @@ -160,7 +161,7 @@ int32_t tsDiskCfgNum = 0; SDiskCfg tsDiskCfg[TFS_MAX_DISKS] = {0}; // stream scheduler -bool tsStreamSchedV = true; +bool tsSchedStreamToSnode = true; /* * minimum scale for whole system, millisecond by default @@ -184,7 +185,7 @@ char tsCompressor[32] = "ZSTD_COMPRESSOR"; // ZSTD_COMPRESSOR or GZIP_COMPR bool tsStartUdfd = true; // internal -int32_t tsTransPullupInterval = 6; +int32_t tsTransPullupInterval = 2; int32_t tsMqRebalanceInterval = 2; void taosAddDataDir(int32_t index, char *v1, int32_t level, int32_t primary) { @@ -417,8 +418,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) { tsNumOfVnodeQueryThreads = TMAX(tsNumOfVnodeQueryThreads, 1); if (cfgAddInt32(pCfg, "numOfVnodeQueryThreads", tsNumOfVnodeQueryThreads, 1, 1024, 0) != 0) return -1; - tsNumOfVnodeFetchThreads = tsNumOfCores / 2; - tsNumOfVnodeFetchThreads = TRANGE(tsNumOfVnodeFetchThreads, 2, 4); + tsNumOfVnodeFetchThreads = TRANGE(tsNumOfVnodeFetchThreads, 1, 1); if (cfgAddInt32(pCfg, "numOfVnodeFetchThreads", tsNumOfVnodeFetchThreads, 1, 1024, 0) != 0) return -1; tsNumOfVnodeWriteThreads = tsNumOfCores; @@ -437,8 +437,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) { tsNumOfQnodeQueryThreads = TMAX(tsNumOfQnodeQueryThreads, 1); if (cfgAddInt32(pCfg, "numOfQnodeQueryThreads", tsNumOfQnodeQueryThreads, 1, 1024, 0) != 0) return -1; - tsNumOfQnodeFetchThreads = tsNumOfCores / 2; - tsNumOfQnodeFetchThreads = TRANGE(tsNumOfQnodeFetchThreads, 2, 4); + tsNumOfQnodeFetchThreads = TRANGE(tsNumOfQnodeFetchThreads, 1, 1); if (cfgAddInt32(pCfg, "numOfQnodeFetchThreads", tsNumOfQnodeFetchThreads, 1, 1024, 0) != 0) return -1; tsNumOfSnodeSharedThreads = tsNumOfCores / 4; diff --git a/source/common/src/tmsg.c b/source/common/src/tmsg.c index 4749c86e4a53b6ba1048c4dd9e8d93bfca625e5c..d078d22cdf36a2ab365c757217e28f750ec137b3 100644 --- a/source/common/src/tmsg.c +++ b/source/common/src/tmsg.c @@ -191,13 +191,25 @@ static int32_t tSerializeSClientHbReq(SEncoder *pEncoder, const SClientHbReq *pR if (tEncodeSClientHbKey(pEncoder, &pReq->connKey) < 0) return -1; if (pReq->connKey.connType == CONN_TYPE__QUERY) { + if (tEncodeI64(pEncoder, pReq->app.appId) < 0) return -1; + if (tEncodeI32(pEncoder, pReq->app.pid) < 0) return -1; + if (tEncodeCStr(pEncoder, pReq->app.name) < 0) return -1; + if (tEncodeI64(pEncoder, pReq->app.startTime) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.numOfInsertsReq) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.numOfInsertRows) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.insertElapsedTime) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.insertBytes) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.fetchBytes) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.queryElapsedTime) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.numOfSlowQueries) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.totalRequests) < 0) return -1; + if (tEncodeU64(pEncoder, pReq->app.summary.currentRequests) < 0) return -1; + int32_t queryNum = 0; if (pReq->query) { queryNum = 1; if (tEncodeI32(pEncoder, queryNum) < 0) return -1; if (tEncodeU32(pEncoder, pReq->query->connId) < 0) return -1; - if (tEncodeI32(pEncoder, pReq->query->pid) < 0) return -1; - if (tEncodeCStr(pEncoder, pReq->query->app) < 0) return -1; int32_t num = taosArrayGetSize(pReq->query->queryDesc); if (tEncodeI32(pEncoder, num) < 0) return -1; @@ -209,7 +221,6 @@ static int32_t tSerializeSClientHbReq(SEncoder *pEncoder, const SClientHbReq *pR if (tEncodeI64(pEncoder, desc->useconds) < 0) return -1; if (tEncodeI64(pEncoder, desc->stime) < 0) return -1; if (tEncodeI64(pEncoder, desc->reqRid) < 0) return -1; - if (tEncodeI32(pEncoder, desc->pid) < 0) return -1; if (tEncodeI8(pEncoder, desc->stableQuery) < 0) return -1; if (tEncodeCStr(pEncoder, desc->fqdn) < 0) return -1; if (tEncodeI32(pEncoder, desc->subPlanNum) < 0) return -1; @@ -243,14 +254,26 @@ static int32_t tDeserializeSClientHbReq(SDecoder *pDecoder, SClientHbReq *pReq) if (tDecodeSClientHbKey(pDecoder, &pReq->connKey) < 0) return -1; if (pReq->connKey.connType == CONN_TYPE__QUERY) { + if (tDecodeI64(pDecoder, &pReq->app.appId) < 0) return -1; + if (tDecodeI32(pDecoder, &pReq->app.pid) < 0) return -1; + if (tDecodeCStrTo(pDecoder, pReq->app.name) < 0) return -1; + if (tDecodeI64(pDecoder, &pReq->app.startTime) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.numOfInsertsReq) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.numOfInsertRows) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.insertElapsedTime) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.insertBytes) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.fetchBytes) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.queryElapsedTime) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.numOfSlowQueries) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.totalRequests) < 0) return -1; + if (tDecodeU64(pDecoder, &pReq->app.summary.currentRequests) < 0) return -1; + int32_t queryNum = 0; if (tDecodeI32(pDecoder, &queryNum) < 0) return -1; if (queryNum) { pReq->query = taosMemoryCalloc(1, sizeof(*pReq->query)); if (NULL == pReq->query) return -1; if (tDecodeU32(pDecoder, &pReq->query->connId) < 0) return -1; - if (tDecodeI32(pDecoder, &pReq->query->pid) < 0) return -1; - if (tDecodeCStrTo(pDecoder, pReq->query->app) < 0) return -1; int32_t num = 0; if (tDecodeI32(pDecoder, &num) < 0) return -1; @@ -265,8 +288,7 @@ static int32_t tDeserializeSClientHbReq(SDecoder *pDecoder, SClientHbReq *pReq) if (tDecodeI64(pDecoder, &desc.useconds) < 0) return -1; if (tDecodeI64(pDecoder, &desc.stime) < 0) return -1; if (tDecodeI64(pDecoder, &desc.reqRid) < 0) return -1; - if (tDecodeI32(pDecoder, &desc.pid) < 0) return -1; - if (tDecodeI8(pDecoder, (int8_t*)&desc.stableQuery) < 0) return -1; + if (tDecodeI8(pDecoder, (int8_t *)&desc.stableQuery) < 0) return -1; if (tDecodeCStrTo(pDecoder, desc.fqdn) < 0) return -1; if (tDecodeI32(pDecoder, &desc.subPlanNum) < 0) return -1; @@ -474,8 +496,10 @@ int32_t tSerializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pReq if (tStartEncode(&encoder) < 0) return -1; if (tEncodeCStr(&encoder, pReq->name) < 0) return -1; if (tEncodeI8(&encoder, pReq->igExists) < 0) return -1; - if (tEncodeFloat(&encoder, pReq->xFilesFactor) < 0) return -1; - if (tEncodeI32(&encoder, pReq->delay) < 0) return -1; + if (tEncodeI64(&encoder, pReq->delay1) < 0) return -1; + if (tEncodeI64(&encoder, pReq->delay2) < 0) return -1; + if (tEncodeI64(&encoder, pReq->watermark1) < 0) return -1; + if (tEncodeI64(&encoder, pReq->watermark2) < 0) return -1; if (tEncodeI32(&encoder, pReq->ttl) < 0) return -1; if (tEncodeI32(&encoder, pReq->numOfColumns) < 0) return -1; if (tEncodeI32(&encoder, pReq->numOfTags) < 0) return -1; @@ -500,7 +524,7 @@ int32_t tSerializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pReq } if (pReq->commentLen > 0) { - if (tEncodeBinary(&encoder, pReq->comment, pReq->commentLen) < 0) return -1; + if (tEncodeCStr(&encoder, pReq->comment) < 0) return -1; } if (pReq->ast1Len > 0) { if (tEncodeBinary(&encoder, pReq->pAst1, pReq->ast1Len) < 0) return -1; @@ -522,8 +546,10 @@ int32_t tDeserializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pR if (tStartDecode(&decoder) < 0) return -1; if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1; if (tDecodeI8(&decoder, &pReq->igExists) < 0) return -1; - if (tDecodeFloat(&decoder, &pReq->xFilesFactor) < 0) return -1; - if (tDecodeI32(&decoder, &pReq->delay) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->delay1) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->delay2) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->watermark1) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->watermark2) < 0) return -1; if (tDecodeI32(&decoder, &pReq->ttl) < 0) return -1; if (tDecodeI32(&decoder, &pReq->numOfColumns) < 0) return -1; if (tDecodeI32(&decoder, &pReq->numOfTags) < 0) return -1; @@ -563,7 +589,7 @@ int32_t tDeserializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pR } if (pReq->commentLen > 0) { - pReq->comment = taosMemoryMalloc(pReq->commentLen); + pReq->comment = taosMemoryMalloc(pReq->commentLen + 1); if (pReq->comment == NULL) return -1; if (tDecodeCStrTo(&decoder, pReq->comment) < 0) return -1; } @@ -681,7 +707,7 @@ int32_t tDeserializeSMAlterStbReq(void *buf, int32_t bufLen, SMAlterStbReq *pReq if (tDecodeI32(&decoder, &pReq->ttl) < 0) return -1; if (tDecodeI32(&decoder, &pReq->commentLen) < 0) return -1; if (pReq->commentLen > 0) { - pReq->comment = taosMemoryMalloc(pReq->commentLen); + pReq->comment = taosMemoryMalloc(pReq->commentLen + 1); if (pReq->comment == NULL) return -1; if (tDecodeCStrTo(&decoder, pReq->comment) < 0) return -1; } @@ -3410,7 +3436,7 @@ int32_t tSerializeSKillConnReq(void *buf, int32_t bufLen, SKillConnReq *pReq) { tEncoderInit(&encoder, buf, bufLen); if (tStartEncode(&encoder) < 0) return -1; - if (tEncodeI32(&encoder, pReq->connId) < 0) return -1; + if (tEncodeU32(&encoder, pReq->connId) < 0) return -1; tEndEncode(&encoder); int32_t tlen = encoder.pos; @@ -3423,7 +3449,7 @@ int32_t tDeserializeSKillConnReq(void *buf, int32_t bufLen, SKillConnReq *pReq) tDecoderInit(&decoder, buf, bufLen); if (tStartDecode(&decoder) < 0) return -1; - if (tDecodeI32(&decoder, &pReq->connId) < 0) return -1; + if (tDecodeU32(&decoder, &pReq->connId) < 0) return -1; tEndDecode(&decoder); tDecoderClear(&decoder); @@ -4247,39 +4273,34 @@ void tFreeSCMCreateStreamReq(SCMCreateStreamReq *pReq) { } int32_t tEncodeSRSmaParam(SEncoder *pCoder, const SRSmaParam *pRSmaParam) { - if (tEncodeFloat(pCoder, pRSmaParam->xFilesFactor) < 0) return -1; - if (tEncodeI32v(pCoder, pRSmaParam->delay) < 0) return -1; - if (tEncodeI32v(pCoder, pRSmaParam->qmsg1Len) < 0) return -1; - if (tEncodeI32v(pCoder, pRSmaParam->qmsg2Len) < 0) return -1; - if (pRSmaParam->qmsg1Len > 0) { - if (tEncodeBinary(pCoder, pRSmaParam->qmsg1, (uint64_t)pRSmaParam->qmsg1Len) < 0) // qmsg1Len contains len of '\0' - return -1; - } - if (pRSmaParam->qmsg2Len > 0) { - if (tEncodeBinary(pCoder, pRSmaParam->qmsg2, (uint64_t)pRSmaParam->qmsg2Len) < 0) // qmsg2Len contains len of '\0' - return -1; + for (int32_t i = 0; i < 2; ++i) { + if (tEncodeI64v(pCoder, pRSmaParam->maxdelay[i]) < 0) return -1; + if (tEncodeI64v(pCoder, pRSmaParam->watermark[i]) < 0) return -1; + if (tEncodeI32v(pCoder, pRSmaParam->qmsgLen[i]) < 0) return -1; + if (pRSmaParam->qmsgLen[i] > 0) { + if (tEncodeBinary(pCoder, pRSmaParam->qmsg[i], (uint64_t)pRSmaParam->qmsgLen[i]) < + 0) // qmsgLen contains len of '\0' + return -1; + } } return 0; } int32_t tDecodeSRSmaParam(SDecoder *pCoder, SRSmaParam *pRSmaParam) { - if (tDecodeFloat(pCoder, &pRSmaParam->xFilesFactor) < 0) return -1; - if (tDecodeI32v(pCoder, &pRSmaParam->delay) < 0) return -1; - if (tDecodeI32v(pCoder, &pRSmaParam->qmsg1Len) < 0) return -1; - if (tDecodeI32v(pCoder, &pRSmaParam->qmsg2Len) < 0) return -1; - if (pRSmaParam->qmsg1Len > 0) { - uint64_t len; - if (tDecodeBinaryAlloc(pCoder, (void **)&pRSmaParam->qmsg1, &len) < 0) return -1; // qmsg1Len contains len of '\0' - } else { - pRSmaParam->qmsg1 = NULL; - } - if (pRSmaParam->qmsg2Len > 0) { - uint64_t len; - if (tDecodeBinaryAlloc(pCoder, (void **)&pRSmaParam->qmsg2, &len) < 0) return -1; // qmsg2Len contains len of '\0' - } else { - pRSmaParam->qmsg2 = NULL; + for (int32_t i = 0; i < 2; ++i) { + if (tDecodeI64v(pCoder, &pRSmaParam->maxdelay[i]) < 0) return -1; + if (tDecodeI64v(pCoder, &pRSmaParam->watermark[i]) < 0) return -1; + if (tDecodeI32v(pCoder, &pRSmaParam->qmsgLen[i]) < 0) return -1; + if (pRSmaParam->qmsgLen[i] > 0) { + uint64_t len; + if (tDecodeBinaryAlloc(pCoder, (void **)&pRSmaParam->qmsg[i], &len) < 0) + return -1; // qmsgLen contains len of '\0' + } else { + pRSmaParam->qmsg[i] = NULL; + } } + return 0; } @@ -4348,6 +4369,10 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) { if (tEncodeI64(pCoder, pReq->ctime) < 0) return -1; if (tEncodeI32(pCoder, pReq->ttl) < 0) return -1; if (tEncodeI8(pCoder, pReq->type) < 0) return -1; + if (tEncodeI32(pCoder, pReq->commentLen) < 0) return -1; + if (pReq->commentLen > 0) { + if (tEncodeCStr(pCoder, pReq->comment) < 0) return -1; + } if (pReq->type == TSDB_CHILD_TABLE) { if (tEncodeI64(pCoder, pReq->ctb.suid) < 0) return -1; @@ -4371,6 +4396,12 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) { if (tDecodeI64(pCoder, &pReq->ctime) < 0) return -1; if (tDecodeI32(pCoder, &pReq->ttl) < 0) return -1; if (tDecodeI8(pCoder, &pReq->type) < 0) return -1; + if (tDecodeI32(pCoder, &pReq->commentLen) < 0) return -1; + if (pReq->commentLen > 0) { + pReq->comment = taosMemoryMalloc(pReq->commentLen + 1); + if (pReq->comment == NULL) return -1; + if (tDecodeCStrTo(pCoder, pReq->comment) < 0) return -1; + } if (pReq->type == TSDB_CHILD_TABLE) { if (tDecodeI64(pCoder, &pReq->ctb.suid) < 0) return -1; @@ -4724,8 +4755,8 @@ int32_t tEncodeSVAlterTbReq(SEncoder *pEncoder, const SVAlterTbReq *pReq) { if (pReq->updateTTL) { if (tEncodeI32v(pEncoder, pReq->newTTL) < 0) return -1; } - if (tEncodeI8(pEncoder, pReq->updateComment) < 0) return -1; - if (pReq->updateComment) { + if (tEncodeI32v(pEncoder, pReq->newCommentLen) < 0) return -1; + if (pReq->newCommentLen > 0) { if (tEncodeCStr(pEncoder, pReq->newComment) < 0) return -1; } break; @@ -4772,8 +4803,8 @@ int32_t tDecodeSVAlterTbReq(SDecoder *pDecoder, SVAlterTbReq *pReq) { if (pReq->updateTTL) { if (tDecodeI32v(pDecoder, &pReq->newTTL) < 0) return -1; } - if (tDecodeI8(pDecoder, &pReq->updateComment) < 0) return -1; - if (pReq->updateComment) { + if (tDecodeI32v(pDecoder, &pReq->newCommentLen) < 0) return -1; + if (pReq->newCommentLen > 0) { if (tDecodeCStr(pDecoder, &pReq->newComment) < 0) return -1; } break; diff --git a/source/dnode/mgmt/mgmt_dnode/src/dmWorker.c b/source/dnode/mgmt/mgmt_dnode/src/dmWorker.c index bd78fe46effa60c8286cbd901b299f2506de6533..9ffa0e606a6e597e247afa24356794b421e30e5f 100644 --- a/source/dnode/mgmt/mgmt_dnode/src/dmWorker.c +++ b/source/dnode/mgmt/mgmt_dnode/src/dmWorker.c @@ -101,7 +101,8 @@ void dmStopMonitorThread(SDnodeMgmt *pMgmt) { static void dmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { SDnodeMgmt *pMgmt = pInfo->ahandle; int32_t code = -1; - dTrace("msg:%p, will be processed in dnode queue, type:%s", pMsg, TMSG_INFO(pMsg->msgType)); + STraceId * trace = &pMsg->info.traceId; + dGTrace("msg:%p, will be processed in dnode queue, type:%s", pMsg, TMSG_INFO(pMsg->msgType)); switch (pMsg->msgType) { case TDMT_DND_CONFIG_DNODE: diff --git a/source/dnode/mgmt/mgmt_mnode/inc/mmInt.h b/source/dnode/mgmt/mgmt_mnode/inc/mmInt.h index c5c3d76f1e6518fecbb98696f3b53d0ed566bf53..a4c37dd334984ff5e834463ec802b937ff557460 100644 --- a/source/dnode/mgmt/mgmt_mnode/inc/mmInt.h +++ b/source/dnode/mgmt/mgmt_mnode/inc/mmInt.h @@ -30,6 +30,7 @@ typedef struct SMnodeMgmt { const char *path; const char *name; SSingleWorker queryWorker; + SSingleWorker fetchWorker; SSingleWorker readWorker; SSingleWorker writeWorker; SSingleWorker syncWorker; @@ -57,6 +58,7 @@ int32_t mmPutMsgToWriteQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t mmPutMsgToSyncQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t mmPutMsgToReadQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t mmPutMsgToQueryQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); +int32_t mmPutMsgToFetchQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t mmPutMsgToMonitorQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc); diff --git a/source/dnode/mgmt/mgmt_mnode/src/mmHandle.c b/source/dnode/mgmt/mgmt_mnode/src/mmHandle.c index d895b73fb0425955df6f6b1565d9a6369af6fa40..a845ae7b39d2582e52a6a01cfb222d848d3abc25 100644 --- a/source/dnode/mgmt/mgmt_mnode/src/mmHandle.c +++ b/source/dnode/mgmt/mgmt_mnode/src/mmHandle.c @@ -204,8 +204,8 @@ SArray *mmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_CONTINUE, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_HEARTBEAT, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_FETCH, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_HEARTBEAT, mmPutMsgToFetchQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_FETCH, mmPutMsgToFetchQueue, 1) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_STB_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_STB_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_STB_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; @@ -213,7 +213,7 @@ SArray *mmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_SMA_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_CHANGE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_DELETE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TASK, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TASK, mmPutMsgToFetchQueue, 1) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_DROP_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER; diff --git a/source/dnode/mgmt/mgmt_mnode/src/mmInt.c b/source/dnode/mgmt/mgmt_mnode/src/mmInt.c index b7124dfaa51a247f0af296e48bed300a38e20ba3..012e61d23964300df42b79fd52ed48c3af6ee4ce 100644 --- a/source/dnode/mgmt/mgmt_mnode/src/mmInt.c +++ b/source/dnode/mgmt/mgmt_mnode/src/mmInt.c @@ -148,6 +148,10 @@ static int32_t mmStart(SMnodeMgmt *pMgmt) { static void mmStop(SMnodeMgmt *pMgmt) { dDebug("mnode-mgmt start to stop"); + taosThreadRwlockWrlock(&pMgmt->lock); + pMgmt->stopped = 1; + taosThreadRwlockUnlock(&pMgmt->lock); + mndStop(pMgmt->pMnode); } diff --git a/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c b/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c index 7cd7da1aa93825f5e94484a8def66821511c4a0c..60c42d31f5c79363d6bde886e843e4df182f066d 100644 --- a/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c +++ b/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c @@ -48,7 +48,8 @@ static inline void mmSendRsp(SRpcMsg *pMsg, int32_t code) { static void mmProcessRpcMsg(SQueueInfo *pInfo, SRpcMsg *pMsg) { SMnodeMgmt *pMgmt = pInfo->ahandle; int32_t code = -1; - dTrace("msg:%p, get from mnode queue", pMsg); + STraceId * trace = &pMsg->info.traceId; + dGTrace("msg:%p, get from mnode queue", pMsg); switch (pMsg->msgType) { case TDMT_MON_MM_INFO: @@ -122,6 +123,13 @@ int32_t mmPutMsgToQueryQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) { return mmPutMsgToWorker(pMgmt, &pMgmt->queryWorker, pMsg); } +int32_t mmPutMsgToFetchQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) { + pMsg->info.node = pMgmt->pMnode; + + return mmPutMsgToWorker(pMgmt, &pMgmt->fetchWorker, pMsg); +} + + int32_t mmPutMsgToMonitorQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) { return mmPutMsgToWorker(pMgmt, &pMgmt->monitorWorker, pMsg); } @@ -135,6 +143,9 @@ int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { case QUERY_QUEUE: pWorker = &pMgmt->queryWorker; break; + case FETCH_QUEUE: + pWorker = &pMgmt->fetchWorker; + break; case READ_QUEUE: pWorker = &pMgmt->readWorker; break; @@ -167,6 +178,18 @@ int32_t mmStartWorker(SMnodeMgmt *pMgmt) { return -1; } + SSingleWorkerCfg fCfg = { + .min = tsNumOfMnodeFetchThreads, + .max = tsNumOfMnodeFetchThreads, + .name = "mnode-fetch", + .fp = (FItem)mmProcessRpcMsg, + .param = pMgmt, + }; + if (tSingleWorkerInit(&pMgmt->fetchWorker, &fCfg) != 0) { + dError("failed to start mnode-fetch worker since %s", terrstr()); + return -1; + } + SSingleWorkerCfg rCfg = { .min = tsNumOfMnodeReadThreads, .max = tsNumOfMnodeReadThreads, @@ -220,13 +243,11 @@ int32_t mmStartWorker(SMnodeMgmt *pMgmt) { } void mmStopWorker(SMnodeMgmt *pMgmt) { - taosThreadRwlockWrlock(&pMgmt->lock); - pMgmt->stopped = 1; - taosThreadRwlockUnlock(&pMgmt->lock); while (pMgmt->refCount > 0) taosMsleep(10); tSingleWorkerCleanup(&pMgmt->monitorWorker); tSingleWorkerCleanup(&pMgmt->queryWorker); + tSingleWorkerCleanup(&pMgmt->fetchWorker); tSingleWorkerCleanup(&pMgmt->readWorker); tSingleWorkerCleanup(&pMgmt->writeWorker); tSingleWorkerCleanup(&pMgmt->syncWorker); diff --git a/source/dnode/mgmt/mgmt_snode/src/smHandle.c b/source/dnode/mgmt/mgmt_snode/src/smHandle.c index 66ab627e3245678fe1b04ab9c8d2b8876022361a..81576e153e35c490860c4b0b9272dcaa2d798418 100644 --- a/source/dnode/mgmt/mgmt_snode/src/smHandle.c +++ b/source/dnode/mgmt/mgmt_snode/src/smHandle.c @@ -95,9 +95,12 @@ SArray *smGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_MON_SM_INFO, smPutNodeMsgToMonitorQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_DROP, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, smPutNodeMsgToSharedQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, smPutNodeMsgToSharedQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH_RSP, smPutNodeMsgToSharedQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, smPutNodeMsgToSharedQueue, 1) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER_RSP, smPutNodeMsgToSharedQueue, 1) == NULL) goto _OVER; code = 0; _OVER: diff --git a/source/dnode/mgmt/mgmt_snode/src/smWorker.c b/source/dnode/mgmt/mgmt_snode/src/smWorker.c index 34a205232e69f057d879188fe4ed98df5b378e3a..19c1b9b5c7afd9d1726b364ee69d94619f471d2d 100644 --- a/source/dnode/mgmt/mgmt_snode/src/smWorker.c +++ b/source/dnode/mgmt/mgmt_snode/src/smWorker.c @@ -55,7 +55,10 @@ static void smProcessUniqueQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t num taosGetQitem(qall, (void **)&pMsg); dTrace("msg:%p, get from snode-unique queue", pMsg); - sndProcessUMsg(pMgmt->pSnode, pMsg); + if (sndProcessUMsg(pMgmt->pSnode, pMsg) < 0) { + ASSERT(0); + } + smSendRsp(pMsg, 0); dTrace("msg:%p, is freed", pMsg); rpcFreeCont(pMsg->pCont); @@ -67,7 +70,10 @@ static void smProcessSharedQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { SSnodeMgmt *pMgmt = pInfo->ahandle; dTrace("msg:%p, get from snode-shared queue", pMsg); - sndProcessSMsg(pMgmt->pSnode, pMsg); + if (sndProcessSMsg(pMgmt->pSnode, pMsg) < 0) { + smSendRsp(pMsg, terrno); + ASSERT(0); + } dTrace("msg:%p, is freed", pMsg); rpcFreeCont(pMsg->pCont); diff --git a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h index 7e5379b0f8661cd25eed2bd8b0ed6db5e6d5a99b..6f00767eb0301cff76b891e6873abf7e2ef755ea 100644 --- a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h +++ b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h @@ -35,7 +35,6 @@ typedef struct SVnodeMgmt { SWWorkerPool syncPool; SWWorkerPool writePool; SWWorkerPool applyPool; - SWWorkerPool mergePool; SSingleWorker mgmtWorker; SSingleWorker monitorWorker; SHashObj *hash; @@ -63,7 +62,6 @@ typedef struct { STaosQueue *pApplyQ; STaosQueue *pQueryQ; STaosQueue *pFetchQ; - STaosQueue *pMergeQ; } SVnodeObj; typedef struct { diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c index 9946185da6d9a65642dbf725a9456dfb03acbacc..c3ed6d781cc8238ee9c901dc53466ec68da74095 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c @@ -228,7 +228,7 @@ int32_t vmProcessCreateVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { vmReleaseVnode(pMgmt, pVnode); terrno = TSDB_CODE_NODE_ALREADY_DEPLOYED; code = terrno; - goto _OVER; + return 0; } snprintf(path, TSDB_FILENAME_LEN, "vnode%svnode%d", TD_DIRSEP, vnodeCfg.vgId); @@ -334,6 +334,7 @@ SArray *vmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_VND_CANCEL_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TTL_TABLE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_TABLE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c index 9e4e7713f2b1f9e63129f02279ed3742b1b55937..3f053639aab740119721c5b67419648f95a50d73 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c @@ -86,7 +86,6 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { while (!taosQueueEmpty(pVnode->pApplyQ)) taosMsleep(10); while (!taosQueueEmpty(pVnode->pQueryQ)) taosMsleep(10); while (!taosQueueEmpty(pVnode->pFetchQ)) taosMsleep(10); - while (!taosQueueEmpty(pVnode->pMergeQ)) taosMsleep(10); vmFreeQueue(pMgmt, pVnode); vnodeClose(pVnode->pImpl); diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c index 88831384d4e719f2a64ac321671888e4507a20bc..71bbc8ddd43aaf6583bcf4758ca018fe798108d4 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c @@ -31,7 +31,8 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { SVnodeMgmt *pMgmt = pInfo->ahandle; int32_t code = -1; - dTrace("msg:%p, get from vnode-mgmt queue", pMsg); + STraceId *trace = &pMsg->info.traceId; + dGTrace("msg:%p, get from vnode-mgmt queue", pMsg); switch (pMsg->msgType) { case TDMT_MON_VM_INFO: code = vmProcessGetMonitorInfoReq(pMgmt, pMsg); @@ -169,10 +170,6 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp dTrace("vgId:%d, msg:%p put into vnode-sync queue", pVnode->vgId, pMsg); taosWriteQitem(pVnode->pSyncQ, pMsg); break; - case MERGE_QUEUE: - dTrace("vgId:%d, msg:%p put into vnode-merge queue", pVnode->vgId, pMsg); - taosWriteQitem(pVnode->pMergeQ, pMsg); - break; case APPLY_QUEUE: dTrace("vgId:%d, msg:%p put into vnode-apply queue", pVnode->vgId, pMsg); taosWriteQitem(pVnode->pApplyQ, pMsg); @@ -195,8 +192,6 @@ int32_t vmPutMsgToQueryQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsg int32_t vmPutMsgToFetchQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, FETCH_QUEUE); } -int32_t vmPutMsgToMergeQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, MERGE_QUEUE); } - int32_t vmPutMsgToMgmtQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { dTrace("msg:%p, put into vnode-mgmt queue", pMsg); taosWriteQitem(pMgmt->mgmtWorker.queue, pMsg); @@ -242,9 +237,6 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) { case FETCH_QUEUE: size = taosQueueItemSize(pVnode->pFetchQ); break; - case MERGE_QUEUE: - size = taosQueueItemSize(pVnode->pMergeQ); - break; default: break; } @@ -259,10 +251,9 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyMsg); pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue); pVnode->pFetchQ = tQWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItem)vmProcessFetchQueue); - pVnode->pMergeQ = tWWorkerAllocQueue(&pMgmt->mergePool, pVnode, (FItems)vmProcessMergeQueue); if (pVnode->pWriteQ == NULL || pVnode->pSyncQ == NULL || pVnode->pApplyQ == NULL || pVnode->pQueryQ == NULL || - pVnode->pFetchQ == NULL || pVnode->pMergeQ == NULL) { + pVnode->pFetchQ == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; return -1; } @@ -277,13 +268,11 @@ void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ); tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ); tQWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ); - tWWorkerFreeQueue(&pMgmt->mergePool, pVnode->pMergeQ); pVnode->pWriteQ = NULL; pVnode->pSyncQ = NULL; pVnode->pApplyQ = NULL; pVnode->pQueryQ = NULL; pVnode->pFetchQ = NULL; - pVnode->pMergeQ = NULL; dDebug("vgId:%d, queue is freed", pVnode->vgId); } @@ -315,11 +304,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) { pSPool->max = tsNumOfVnodeSyncThreads; if (tWWorkerInit(pSPool) != 0) return -1; - SWWorkerPool *pMPool = &pMgmt->mergePool; - pMPool->name = "vnode-merge"; - pMPool->max = tsNumOfVnodeMergeThreads; - if (tWWorkerInit(pMPool) != 0) return -1; - SSingleWorkerCfg mgmtCfg = { .min = 1, .max = 1, @@ -350,6 +334,5 @@ void vmStopWorker(SVnodeMgmt *pMgmt) { tWWorkerCleanup(&pMgmt->syncPool); tQWorkerCleanup(&pMgmt->queryPool); tQWorkerCleanup(&pMgmt->fetchPool); - tWWorkerCleanup(&pMgmt->mergePool); dDebug("vnode workers are closed"); } diff --git a/source/dnode/mgmt/node_mgmt/src/dmTransport.c b/source/dnode/mgmt/node_mgmt/src/dmTransport.c index b66e5593702707a9506226b1cbc7d458b0d0c1f1..660f512fc520c5657d12c0db11aae58085defc9f 100644 --- a/source/dnode/mgmt/node_mgmt/src/dmTransport.c +++ b/source/dnode/mgmt/node_mgmt/src/dmTransport.c @@ -21,21 +21,6 @@ static void dmSendRedirectRsp(SRpcMsg *pMsg, const SEpSet *pNewEpSet); static void dmSendRsp(SRpcMsg *pMsg); static void dmBuildMnodeRedirectRsp(SDnode *pDnode, SRpcMsg *pMsg); -static inline int32_t dmBuildNodeMsg(SRpcMsg *pMsg, SRpcMsg *pRpc) { - SRpcConnInfo *pConnInfo = &(pRpc->info.connInfo); - // if (IsReq(pRpc)) { - // terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; - // dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle); - // return -1; - //} - - memcpy(pMsg, pRpc, sizeof(SRpcMsg)); - memcpy(pMsg->conn.user, pConnInfo->user, TSDB_USER_LEN); - pMsg->conn.clientIp = pConnInfo->clientIp; - pMsg->conn.clientPort = pConnInfo->clientPort; - return 0; -} - int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) { NodeMsgFp msgFp = pWrapper->msgFps[TMSG_INDEX(pMsg->msgType)]; if (msgFp == NULL) { @@ -55,8 +40,9 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) { SMgmtWrapper *pWrapper = NULL; SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)]; - dTrace("msg:%s is received, handle:%p len:%d code:0x%x app:%p refId:%" PRId64, TMSG_INFO(pRpc->msgType), - pRpc->info.handle, pRpc->contLen, pRpc->code, pRpc->info.ahandle, pRpc->info.refId); + STraceId *trace = &pRpc->info.traceId; + dGTrace("msg:%s is received, handle:%p len:%d code:0x%x app:%p refId:%" PRId64, TMSG_INFO(pRpc->msgType), + pRpc->info.handle, pRpc->contLen, pRpc->code, pRpc->info.ahandle, pRpc->info.refId); if (pRpc->msgType == TDMT_DND_NET_TEST) { dmProcessNetTestReq(pDnode, pRpc); @@ -116,14 +102,10 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) { } pMsg = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM); - if (pMsg == NULL) { - goto _OVER; - } - dTrace("msg:%p, is created, type:%s", pMsg, TMSG_INFO(pRpc->msgType)); + if (pMsg == NULL) goto _OVER; + memcpy(pMsg, pRpc, sizeof(SRpcMsg)); - if (dmBuildNodeMsg(pMsg, pRpc) != 0) { - goto _OVER; - } + dTrace("msg:%p, is created, type:%s handle:%p", pMsg, TMSG_INFO(pRpc->msgType), pMsg->info.handle); if (InParentProc(pWrapper)) { code = dmPutToProcCQueue(&pWrapper->proc, pMsg, DND_FUNC_REQ); diff --git a/source/dnode/mgmt/node_util/inc/dmUtil.h b/source/dnode/mgmt/node_util/inc/dmUtil.h index c142a6cfd892413f1a69e2e7ce1d41524b1dbb27..7897f62f62ec9dd99b27568e6c7f4a74ac6463ca 100644 --- a/source/dnode/mgmt/node_util/inc/dmUtil.h +++ b/source/dnode/mgmt/node_util/inc/dmUtil.h @@ -34,13 +34,13 @@ #include "dnode.h" #include "mnode.h" -#include "qnode.h" #include "monitor.h" +#include "qnode.h" #include "sync.h" #include "wal.h" #include "libs/function/function.h" - +// clang-format off #ifdef __cplusplus extern "C" { #endif @@ -51,6 +51,7 @@ extern "C" { #define dInfo(...) { if (dDebugFlag & DEBUG_INFO) { taosPrintLog("DND ", DEBUG_INFO, 255, __VA_ARGS__); }} #define dDebug(...) { if (dDebugFlag & DEBUG_DEBUG) { taosPrintLog("DND ", DEBUG_DEBUG, dDebugFlag, __VA_ARGS__); }} #define dTrace(...) { if (dDebugFlag & DEBUG_TRACE) { taosPrintLog("DND ", DEBUG_TRACE, dDebugFlag, __VA_ARGS__); }} +#define dGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dTrace(param ",GTID: %s", __VA_ARGS__, buf);} while(0) typedef enum { DNODE = 0, @@ -184,3 +185,4 @@ void dmSetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet); #endif #endif /*_TD_DM_INT_H_*/ +// clang-format on diff --git a/source/dnode/mnode/impl/inc/mndDef.h b/source/dnode/mnode/impl/inc/mndDef.h index 4daeeaa9bf52dedbfc7c57d8a024aefca86e1eec..987b01b96a202ca4e485fe45bf6684e839fc9e37 100644 --- a/source/dnode/mnode/impl/inc/mndDef.h +++ b/source/dnode/mnode/impl/inc/mndDef.h @@ -124,7 +124,8 @@ typedef struct { int32_t lastErrorNo; tmsg_t lastMsgType; SEpSet lastEpset; - char dbname[TSDB_DB_FNAME_LEN]; + char dbname1[TSDB_DB_FNAME_LEN]; + char dbname2[TSDB_DB_FNAME_LEN]; int32_t startFunc; int32_t stopFunc; int32_t paramLen; @@ -340,8 +341,8 @@ typedef struct { int32_t colVer; int32_t smaVer; int32_t nextColId; - float xFilesFactor; - int32_t delay; + int64_t watermark[2]; + int64_t maxdelay[2]; int32_t ttl; int32_t numOfColumns; int32_t numOfTags; diff --git a/source/dnode/mnode/impl/inc/mndInt.h b/source/dnode/mnode/impl/inc/mndInt.h index cc9bc5b634602153d5875c25b22e234544b697a3..c810a0cbc75474154c9f19b5872656bb7fc2dfbe 100644 --- a/source/dnode/mnode/impl/inc/mndInt.h +++ b/source/dnode/mnode/impl/inc/mndInt.h @@ -40,6 +40,8 @@ extern "C" { #define mInfo(...) { if (mDebugFlag & DEBUG_INFO) { taosPrintLog("MND ", DEBUG_INFO, 255, __VA_ARGS__); }} #define mDebug(...) { if (mDebugFlag & DEBUG_DEBUG) { taosPrintLog("MND ", DEBUG_DEBUG, mDebugFlag, __VA_ARGS__); }} #define mTrace(...) { if (mDebugFlag & DEBUG_TRACE) { taosPrintLog("MND ", DEBUG_TRACE, mDebugFlag, __VA_ARGS__); }} +#define mGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); mTrace(param ", GTID: %s", __VA_ARGS__, buf);} while(0) + // clang-format on #define SYSTABLE_SCH_TABLE_NAME_LEN ((TSDB_TABLE_NAME_LEN - 1) + VARSTR_HEADER_SIZE) @@ -54,7 +56,7 @@ typedef void (*ShowFreeIterFp)(SMnode *pMnode, void *pIter); typedef struct SQWorker SQHandle; typedef struct { - const char *name; + const char * name; MndInitFp initFp; MndCleanupFp cleanupFp; } SMnodeStep; @@ -63,11 +65,12 @@ typedef struct { int64_t showId; ShowRetrieveFp retrieveFps[TSDB_MGMT_TABLE_MAX]; ShowFreeIterFp freeIterFps[TSDB_MGMT_TABLE_MAX]; - SCacheObj *cache; + SCacheObj * cache; } SShowMgmt; typedef struct { - SCacheObj *cache; + SCacheObj *connCache; + SCacheObj *appCache; } SProfileMgmt; typedef struct { @@ -99,14 +102,14 @@ typedef struct SMnode { bool stopped; bool restored; bool deploy; - char *path; + char * path; int64_t checkTime; - SSdb *pSdb; - SArray *pSteps; - SQHandle *pQuery; - SHashObj *infosMeta; - SHashObj *perfsMeta; - SWal *pWal; + SSdb * pSdb; + SArray * pSteps; + SQHandle * pQuery; + SHashObj * infosMeta; + SHashObj * perfsMeta; + SWal * pWal; SShowMgmt showMgmt; SProfileMgmt profileMgmt; STelemMgmt telemMgmt; diff --git a/source/dnode/mnode/impl/inc/mndScheduler.h b/source/dnode/mnode/impl/inc/mndScheduler.h index 8e816d2dd673e836f7921425e4f72da535a9c0dd..15d2c6cd5e213955f81e6203b63e2ba26125e44e 100644 --- a/source/dnode/mnode/impl/inc/mndScheduler.h +++ b/source/dnode/mnode/impl/inc/mndScheduler.h @@ -30,7 +30,7 @@ int32_t mndSchedInitSubEp(SMnode* pMnode, const SMqTopicObj* pTopic, SMqSubscrib int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream); int32_t mndConvertRsmaTask(char** pDst, int32_t* pDstLen, const char* ast, int64_t uid, int8_t triggerType, - int64_t watermark, double filesFactor); + int64_t watermark); int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream); diff --git a/source/dnode/mnode/impl/inc/mndTrans.h b/source/dnode/mnode/impl/inc/mndTrans.h index 0175e29a770d1d8521154ccb0778494529993b21..bc2d5c82b1410af66f758408b442912d02b2fd79 100644 --- a/source/dnode/mnode/impl/inc/mndTrans.h +++ b/source/dnode/mnode/impl/inc/mndTrans.h @@ -68,7 +68,7 @@ int32_t mndTransAppendRedoAction(STrans *pTrans, STransAction *pAction); int32_t mndTransAppendUndoAction(STrans *pTrans, STransAction *pAction); void mndTransSetRpcRsp(STrans *pTrans, void *pCont, int32_t contLen); void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *param, int32_t paramLen); -void mndTransSetDbName(STrans *pTrans, const char *dbname); +void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2); void mndTransSetSerial(STrans *pTrans); int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans); diff --git a/source/dnode/mnode/impl/src/mndAuth.c b/source/dnode/mnode/impl/src/mndAuth.c index f036fc48f7b26a6dd71296b3f2bf7a765d911d99..d47fb9dfb4bfa18b337672ae01adedbe743615f8 100644 --- a/source/dnode/mnode/impl/src/mndAuth.c +++ b/source/dnode/mnode/impl/src/mndAuth.c @@ -56,7 +56,7 @@ static int32_t mndProcessAuthReq(SRpcMsg *pReq) { memcpy(authRsp.user, authReq.user, TSDB_USER_LEN); int32_t code = mndRetriveAuth(pReq->info.node, &authRsp); - mTrace("user:%s, auth req received, spi:%d encrypt:%d ruser:%s", pReq->conn.user, authRsp.spi, authRsp.encrypt, + mTrace("user:%s, auth req received, spi:%d encrypt:%d ruser:%s", pReq->info.conn.user, authRsp.spi, authRsp.encrypt, authRsp.user); int32_t contLen = tSerializeSAuthReq(NULL, 0, &authRsp); diff --git a/source/dnode/mnode/impl/src/mndBnode.c b/source/dnode/mnode/impl/src/mndBnode.c index aa908b983dd7530974b9f24b80a24c6387eb9f18..0de40ca67192d9394b44d13b9041b308ab7b3208 100644 --- a/source/dnode/mnode/impl/src/mndBnode.c +++ b/source/dnode/mnode/impl/src/mndBnode.c @@ -292,7 +292,7 @@ static int32_t mndProcessCreateBnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_BNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_BNODE) != 0) { goto _OVER; } @@ -394,7 +394,7 @@ static int32_t mndProcessDropBnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_BNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_BNODE) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndDb.c b/source/dnode/mnode/impl/src/mndDb.c index c12c6d5b4c015dbd6c3d8a78fc2d24002ad90ac7..345464399eef3b75153e9cc6d6634ade49225b00 100644 --- a/source/dnode/mnode/impl/src/mndDb.c +++ b/source/dnode/mnode/impl/src/mndDb.c @@ -477,7 +477,7 @@ static int32_t mndCreateDb(SMnode *pMnode, SRpcMsg *pReq, SCreateDbReq *pCreate, mDebug("trans:%d, used to create db:%s", pTrans->id, pCreate->db); - mndTransSetDbName(pTrans, dbObj.name); + mndTransSetDbName(pTrans, dbObj.name, NULL); if (mndSetCreateDbRedoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER; if (mndSetCreateDbUndoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER; if (mndSetCreateDbCommitLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER; @@ -521,12 +521,12 @@ static int32_t mndProcessCreateDbReq(SRpcMsg *pReq) { goto _OVER; } - pUser = mndAcquireUser(pMnode, pReq->conn.user); + pUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pUser == NULL) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_DB, NULL) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_DB, NULL) != 0) { goto _OVER; } @@ -668,7 +668,7 @@ static int32_t mndAlterDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pOld, SDbObj *p mDebug("trans:%d, used to alter db:%s", pTrans->id, pOld->name); int32_t code = -1; - mndTransSetDbName(pTrans, pOld->name); + mndTransSetDbName(pTrans, pOld->name, NULL); if (mndSetAlterDbRedoLogs(pMnode, pTrans, pOld, pNew) != 0) goto _OVER; if (mndSetAlterDbCommitLogs(pMnode, pTrans, pOld, pNew) != 0) goto _OVER; if (mndSetAlterDbRedoActions(pMnode, pTrans, pOld, pNew) != 0) goto _OVER; @@ -700,7 +700,7 @@ static int32_t mndProcessAlterDbReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_ALTER_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_ALTER_DB, pDb) != 0) { goto _OVER; } @@ -921,7 +921,7 @@ static int32_t mndDropDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb) { if (pTrans == NULL) goto _OVER; mDebug("trans:%d, used to drop db:%s", pTrans->id, pDb->name); - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); if (mndSetDropDbRedoLogs(pMnode, pTrans, pDb) != 0) goto _OVER; if (mndSetDropDbCommitLogs(pMnode, pTrans, pDb) != 0) goto _OVER; @@ -980,7 +980,7 @@ static int32_t mndProcessDropDbReq(SRpcMsg *pReq) { } } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_DROP_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_DB, pDb) != 0) { goto _OVER; } @@ -1127,7 +1127,7 @@ static int32_t mndProcessUseDbReq(SRpcMsg *pReq) { mError("db:%s, failed to process use db req since %s", usedbReq.db, terrstr()); } else { - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_USE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_USE_DB, pDb) != 0) { goto _OVER; } @@ -1252,7 +1252,7 @@ static int32_t mndProcessCompactDbReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_COMPACT_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_COMPACT_DB, pDb) != 0) { goto _OVER; } @@ -1391,11 +1391,13 @@ static void dumpDbInfoData(SSDataBlock *pBlock, SDbObj *pDb, SShowObj *pShow, in pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, rows, (const char *)strict, false); - pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); - colDataAppend(pColInfo, rows, (const char *)&pDb->cfg.daysPerFile, false); - char tmp[128] = {0}; int32_t len = 0; + len = sprintf(&tmp[VARSTR_HEADER_SIZE], "%dm", pDb->cfg.daysPerFile); + varDataSetLen(tmp, len); + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, rows, (const char *)tmp, false); + if (pDb->cfg.daysToKeep0 > pDb->cfg.daysToKeep1 || pDb->cfg.daysToKeep0 > pDb->cfg.daysToKeep2) { len = sprintf(&tmp[VARSTR_HEADER_SIZE], "%dm,%dm,%dm", pDb->cfg.daysToKeep1, pDb->cfg.daysToKeep2, pDb->cfg.daysToKeep0); diff --git a/source/dnode/mnode/impl/src/mndDnode.c b/source/dnode/mnode/impl/src/mndDnode.c index 6d37143fc9f5bb5b871705c4c57ce180d75642d3..58c3570c3631ec0ceee76163d4afd50e047245ad 100644 --- a/source/dnode/mnode/impl/src/mndDnode.c +++ b/source/dnode/mnode/impl/src/mndDnode.c @@ -522,7 +522,7 @@ static int32_t mndProcessCreateDnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_DNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_DNODE) != 0) { goto _OVER; } @@ -623,7 +623,7 @@ static int32_t mndProcessDropDnodeReq(SRpcMsg *pReq) { } } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_MNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_MNODE) != 0) { goto _OVER; } @@ -761,11 +761,12 @@ static int32_t mndRetrieveDnodes(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, (const char *)&pDnode->createdTime, false); - char b[tListLen(offlineReason) + VARSTR_HEADER_SIZE] = {0}; + char *b = taosMemoryCalloc(VARSTR_HEADER_SIZE + strlen(offlineReason[pDnode->offlineReason]) + 1, 1); STR_TO_VARSTR(b, online ? "" : offlineReason[pDnode->offlineReason]); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, b, false); + taosMemoryFreeClear(b); numOfRows++; sdbRelease(pSdb, pDnode); diff --git a/source/dnode/mnode/impl/src/mndFunc.c b/source/dnode/mnode/impl/src/mndFunc.c index dfdc0a3c1abe282a75aa4b3e3523ec35daf792cc..832f1b8e68b0370d1c8ebaff890956d7f3566b0e 100644 --- a/source/dnode/mnode/impl/src/mndFunc.c +++ b/source/dnode/mnode/impl/src/mndFunc.c @@ -318,7 +318,7 @@ static int32_t mndProcessCreateFuncReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_FUNC) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_FUNC) != 0) { goto _OVER; } @@ -365,7 +365,7 @@ static int32_t mndProcessDropFuncReq(SRpcMsg *pReq) { } } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_FUNC) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_FUNC) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndMain.c b/source/dnode/mnode/impl/src/mndMain.c index f277a5169d81007f0ad1d2fb60d05193ed24d7bb..b50781e8a585da2be402c6994371978cd00392dc 100644 --- a/source/dnode/mnode/impl/src/mndMain.c +++ b/source/dnode/mnode/impl/src/mndMain.c @@ -77,12 +77,50 @@ static void mndPullupTelem(SMnode *pMnode) { tmsgPutToQueue(&pMnode->msgCb, READ_QUEUE, &rpcMsg); } +static void mndPushTtlTime(SMnode *pMnode) { + SSdb *pSdb = pMnode->pSdb; + SVgObj *pVgroup = NULL; + void *pIter = NULL; + + while (1) { + pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup); + if (pIter == NULL) break; + + int32_t contLen = sizeof(SMsgHead) + sizeof(int32_t); + SMsgHead *pHead = rpcMallocCont(contLen); + if (pHead == NULL) { + mError("ttl time malloc err. contLen:%d", contLen); + sdbRelease(pSdb, pVgroup); + continue; + } + pHead->contLen = htonl(contLen); + pHead->vgId = htonl(pVgroup->vgId); + + int32_t t = taosGetTimestampSec(); + *(int32_t *)(POINTER_SHIFT(pHead, sizeof(SMsgHead))) = htonl(t); + + SRpcMsg rpcMsg = {.msgType = TDMT_VND_DROP_TTL_TABLE, .pCont = pHead, .contLen = contLen}; + + SEpSet epSet = mndGetVgroupEpset(pMnode, pVgroup); + int32_t code = tmsgSendReq(&epSet, &rpcMsg); + if (code != 0) { + mError("ttl time seed err. code:%d", code); + } + mError("ttl time seed succ. time:%d", t); + sdbRelease(pSdb, pVgroup); + } +} + static void *mndThreadFp(void *param) { SMnode *pMnode = param; int64_t lastTime = 0; setThreadName("mnode-timer"); while (1) { + if (lastTime % (864000) == 0) { // sleep 1 day for ttl + mndPushTtlTime(pMnode); + } + lastTime++; taosMsleep(100); if (mndGetStop(pMnode)) break; @@ -395,15 +433,19 @@ int32_t mndProcessSyncMsg(SRpcMsg *pMsg) { return -1; } - char logBuf[512] = {0}; - char *syncNodeStr = sync2SimpleStr(pMgmt->sync); - snprintf(logBuf, sizeof(logBuf), "==mndProcessSyncMsg== msgType:%d, syncNode: %s", pMsg->msgType, syncNodeStr); - static int64_t mndTick = 0; - if (++mndTick % 10 == 1) { - mTrace("sync trace msg:%s, %s", TMSG_INFO(pMsg->msgType), syncNodeStr); - } - syncRpcMsgLog2(logBuf, pMsg); - taosMemoryFree(syncNodeStr); + do { + char *syncNodeStr = sync2SimpleStr(pMgmt->sync); + static int64_t mndTick = 0; + if (++mndTick % 10 == 1) { + mTrace("vgId:%d, sync heartbeat msg:%s, %s", syncGetVgId(pMgmt->sync), TMSG_INFO(pMsg->msgType), syncNodeStr); + } + if (gRaftDetailLog) { + char logBuf[512] = {0}; + snprintf(logBuf, sizeof(logBuf), "==mndProcessSyncMsg== msgType:%d, syncNode: %s", pMsg->msgType, syncNodeStr); + syncRpcMsgLog2(logBuf, pMsg); + } + taosMemoryFree(syncNodeStr); + } while (0); // ToDo: ugly! use function pointer if (syncNodeSnapshotEnable(pSyncNode)) { @@ -505,12 +547,14 @@ int32_t mndProcessSyncMsg(SRpcMsg *pMsg) { } static int32_t mndCheckMnodeState(SRpcMsg *pMsg) { + if (!IsReq(pMsg)) return 0; if (mndAcquireRpcRef(pMsg->info.node) == 0) return 0; if (IsReq(pMsg) && pMsg->msgType != TDMT_MND_MQ_TIMER && pMsg->msgType != TDMT_MND_TELEM_TIMER && pMsg->msgType != TDMT_MND_TRANS_TIMER) { - mError("msg:%p, failed to check mnode state since %s, app:%p type:%s", pMsg, terrstr(), pMsg->info.ahandle, - TMSG_INFO(pMsg->msgType)); + mError("msg:%p, failed to check mnode state since %s, type:%s", pMsg, terrstr(), TMSG_INFO(pMsg->msgType)); + + mndAbortPreprocessMsg(pMsg); SEpSet epSet = {0}; mndGetMnodeEpSet(pMsg->info.node, &epSet); @@ -533,7 +577,8 @@ static int32_t mndCheckMsgContent(SRpcMsg *pMsg) { if (!IsReq(pMsg)) return 0; if (pMsg->contLen != 0 && pMsg->pCont != NULL) return 0; - mError("msg:%p, failed to check msg content, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType)); + mError("msg:%p, failed to check msg, cont:%p contLen:%d, app:%p type:%s", pMsg, pMsg->pCont, pMsg->contLen, + pMsg->info.ahandle, TMSG_INFO(pMsg->msgType)); terrno = TSDB_CODE_INVALID_MSG_LEN; return -1; } @@ -550,14 +595,15 @@ int32_t mndProcessRpcMsg(SRpcMsg *pMsg) { if (mndCheckMsgContent(pMsg) != 0) return -1; if (mndCheckMnodeState(pMsg) != 0) return -1; - mTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType)); + STraceId *trace = &pMsg->info.traceId; + mGTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType)); int32_t code = (*fp)(pMsg); mndReleaseRpcRef(pMnode); if (code == TSDB_CODE_ACTION_IN_PROGRESS) { mTrace("msg:%p, won't response immediately since in progress", pMsg); } else if (code == 0) { - mTrace("msg:%p, successfully processed and response", pMsg); + mTrace("msg:%p, successfully processed", pMsg); } else { mError("msg:%p, failed to process since %s, app:%p type:%s", pMsg, terrstr(), pMsg->info.ahandle, TMSG_INFO(pMsg->msgType)); diff --git a/source/dnode/mnode/impl/src/mndMnode.c b/source/dnode/mnode/impl/src/mndMnode.c index e82c479cdd416ea87fdfc10cf506a3f33e2aec72..b40cd713e5b1bc002c9503744055998ef4e9b405 100644 --- a/source/dnode/mnode/impl/src/mndMnode.c +++ b/source/dnode/mnode/impl/src/mndMnode.c @@ -414,7 +414,7 @@ static int32_t mndProcessCreateMnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_MNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_MNODE) != 0) { goto _OVER; } @@ -621,7 +621,7 @@ static int32_t mndProcessDropMnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_MNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_MNODE) != 0) { goto _OVER; } @@ -742,7 +742,7 @@ static int32_t mndProcessAlterMnodeReq(SRpcMsg *pReq) { pMgmt->errCode = 0; pMgmt->transId = -1; tsem_wait(&pMgmt->syncSem); - mInfo("alter mnode sync result:%s", tstrerror(pMgmt->errCode)); + mInfo("alter mnode sync result:0x%x %s", pMgmt->errCode, tstrerror(pMgmt->errCode)); terrno = pMgmt->errCode; return pMgmt->errCode; } diff --git a/source/dnode/mnode/impl/src/mndProfile.c b/source/dnode/mnode/impl/src/mndProfile.c index 20d94e341c6155560f189ae28efb8e10547c8385..fd80679316ad4df7dbcac44e636bb29f42c1800d 100644 --- a/source/dnode/mnode/impl/src/mndProfile.c +++ b/source/dnode/mnode/impl/src/mndProfile.c @@ -43,6 +43,16 @@ typedef struct { SArray *pQueries; // SArray } SConnObj; +typedef struct { + int64_t appId; + uint32_t ip; + int32_t pid; + char name[TSDB_APP_NAME_LEN]; + int64_t startTime; + SAppClusterSummary summary; + int64_t lastAccessTimeMs; +} SAppObj; + static SConnObj *mndCreateConn(SMnode *pMnode, const char *user, int8_t connType, uint32_t ip, uint16_t port, int32_t pid, const char *app, int64_t startTime); static void mndFreeConn(SConnObj *pConn); @@ -57,14 +67,24 @@ static int32_t mndProcessKillConnReq(SRpcMsg *pReq); static int32_t mndRetrieveConns(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows); static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows); static void mndCancelGetNextQuery(SMnode *pMnode, void *pIter); +static void mndFreeApp(SAppObj *pApp); +static int32_t mndRetrieveApps(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows); +static void mndCancelGetNextApp(SMnode *pMnode, void *pIter); int32_t mndInitProfile(SMnode *pMnode) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; // in ms - int32_t connCheckTime = tsShellActivityTimer * 2 * 1000; - pMgmt->cache = taosCacheInit(TSDB_DATA_TYPE_INT, connCheckTime, true, (__cache_free_fn_t)mndFreeConn, "conn"); - if (pMgmt->cache == NULL) { + int32_t checkTime = tsShellActivityTimer * 2 * 1000; + pMgmt->connCache = taosCacheInit(TSDB_DATA_TYPE_UINT, checkTime, true, (__cache_free_fn_t)mndFreeConn, "conn"); + if (pMgmt->connCache == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + mError("failed to alloc profile cache since %s", terrstr()); + return -1; + } + + pMgmt->appCache = taosCacheInit(TSDB_DATA_TYPE_BIGINT, checkTime, true, (__cache_free_fn_t)mndFreeApp, "app"); + if (pMgmt->appCache == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; mError("failed to alloc profile cache since %s", terrstr()); return -1; @@ -79,15 +99,22 @@ int32_t mndInitProfile(SMnode *pMnode) { mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_CONNS, mndCancelGetNextConn); mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_QUERIES, mndRetrieveQueries); mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_QUERIES, mndCancelGetNextQuery); + mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_APPS, mndRetrieveApps); + mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_APPS, mndCancelGetNextApp); return 0; } void mndCleanupProfile(SMnode *pMnode) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; - if (pMgmt->cache != NULL) { - taosCacheCleanup(pMgmt->cache); - pMgmt->cache = NULL; + if (pMgmt->connCache != NULL) { + taosCacheCleanup(pMgmt->connCache); + pMgmt->connCache = NULL; + } + + if (pMgmt->appCache != NULL) { + taosCacheCleanup(pMgmt->appCache); + pMgmt->appCache = NULL; } } @@ -97,7 +124,7 @@ static SConnObj *mndCreateConn(SMnode *pMnode, const char *user, int8_t connType char connStr[255] = {0}; int32_t len = snprintf(connStr, sizeof(connStr), "%s%d%d%d%s", user, ip, port, pid, app); - int32_t connId = mndGenerateUid(connStr, len); + uint32_t connId = mndGenerateUid(connStr, len); if (startTime == 0) startTime = taosGetTimestampMs(); SConnObj connObj = {.id = connId, @@ -118,7 +145,7 @@ static SConnObj *mndCreateConn(SMnode *pMnode, const char *user, int8_t connType tstrncpy(connObj.app, app, TSDB_APP_NAME_LEN); int32_t keepTime = tsShellActivityTimer * 3; - SConnObj *pConn = taosCachePut(pMgmt->cache, &connId, sizeof(int32_t), &connObj, sizeof(connObj), keepTime * 1000); + SConnObj *pConn = taosCachePut(pMgmt->connCache, &connId, sizeof(uint32_t), &connObj, sizeof(connObj), keepTime * 1000); if (pConn == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; mError("conn:%d, failed to put into cache since %s, user:%s", connId, user, terrstr()); @@ -133,21 +160,20 @@ static void mndFreeConn(SConnObj *pConn) { taosWLockLatch(&pConn->queryLock); taosArrayDestroyEx(pConn->pQueries, tFreeClientHbQueryDesc); taosWUnLockLatch(&pConn->queryLock); - + mTrace("conn:%u, is destroyed, data:%p", pConn->id, pConn); } static SConnObj *mndAcquireConn(SMnode *pMnode, uint32_t connId) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; - SConnObj *pConn = taosCacheAcquireByKey(pMgmt->cache, &connId, sizeof(connId)); + SConnObj *pConn = taosCacheAcquireByKey(pMgmt->connCache, &connId, sizeof(connId)); if (pConn == NULL) { mDebug("conn:%u, already destroyed", connId); return NULL; } - int32_t keepTime = tsShellActivityTimer * 3; - pConn->lastAccessTimeMs = keepTime * 1000 + (uint64_t)taosGetTimestampMs(); + pConn->lastAccessTimeMs = taosGetTimestampMs(); mTrace("conn:%u, acquired from cache, data:%p", pConn->id, pConn); return pConn; @@ -158,7 +184,7 @@ static void mndReleaseConn(SMnode *pMnode, SConnObj *pConn) { mTrace("conn:%u, released from cache, data:%p", pConn->id, pConn); SProfileMgmt *pMgmt = &pMnode->profileMgmt; - taosCacheRelease(pMgmt->cache, (void **)&pConn, false); + taosCacheRelease(pMgmt->connCache, (void **)&pConn, false); } void *mndGetNextConn(SMnode *pMnode, SCacheIter *pIter) { @@ -194,15 +220,15 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) { goto CONN_OVER; } - taosIp2String(pReq->conn.clientIp, ip); + taosIp2String(pReq->info.conn.clientIp, ip); - pUser = mndAcquireUser(pMnode, pReq->conn.user); + pUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pUser == NULL) { - mError("user:%s, failed to login while acquire user since %s", pReq->conn.user, terrstr()); + mError("user:%s, failed to login while acquire user since %s", pReq->info.conn.user, terrstr()); goto CONN_OVER; } if (0 != strncmp(connReq.passwd, pUser->pass, TSDB_PASSWORD_LEN - 1)) { - mError("user:%s, failed to auth while acquire user, input:%s", pReq->conn.user, connReq.passwd); + mError("user:%s, failed to auth while acquire user, input:%s", pReq->info.conn.user, connReq.passwd); code = TSDB_CODE_RPC_AUTH_FAILURE; goto CONN_OVER; } @@ -213,15 +239,16 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) { pDb = mndAcquireDb(pMnode, db); if (pDb == NULL) { terrno = TSDB_CODE_MND_INVALID_DB; - mError("user:%s, failed to login from %s while use db:%s since %s", pReq->conn.user, ip, connReq.db, terrstr()); + mError("user:%s, failed to login from %s while use db:%s since %s", pReq->info.conn.user, ip, connReq.db, + terrstr()); goto CONN_OVER; } } - pConn = mndCreateConn(pMnode, pReq->conn.user, connReq.connType, pReq->conn.clientIp, pReq->conn.clientPort, - connReq.pid, connReq.app, connReq.startTime); + pConn = mndCreateConn(pMnode, pReq->info.conn.user, connReq.connType, pReq->info.conn.clientIp, + pReq->info.conn.clientPort, connReq.pid, connReq.app, connReq.startTime); if (pConn == NULL) { - mError("user:%s, failed to login from %s while create connection since %s", pReq->conn.user, ip, terrstr()); + mError("user:%s, failed to login from %s while create connection since %s", pReq->info.conn.user, ip, terrstr()); goto CONN_OVER; } @@ -246,7 +273,7 @@ static int32_t mndProcessConnectReq(SRpcMsg *pReq) { pReq->info.rspLen = contLen; pReq->info.rsp = pRsp; - mDebug("user:%s, login from %s:%d, conn:%u, app:%s", pReq->conn.user, ip, pConn->port, pConn->id, connReq.app); + mDebug("user:%s, login from %s:%d, conn:%u, app:%s", pReq->info.conn.user, ip, pConn->port, pConn->id, connReq.app); code = 0; @@ -275,6 +302,77 @@ static int32_t mndSaveQueryList(SConnObj *pConn, SQueryHbReqBasic *pBasic) { return TSDB_CODE_SUCCESS; } +static SAppObj *mndCreateApp(SMnode *pMnode, uint32_t clientIp, SAppHbReq* pReq) { + SProfileMgmt *pMgmt = &pMnode->profileMgmt; + + SAppObj app; + app.appId = pReq->appId; + app.ip = clientIp; + app.pid = pReq->pid; + strcpy(app.name, pReq->name); + app.startTime = pReq->startTime; + memcpy(&app.summary, &pReq->summary, sizeof(pReq->summary)); + app.lastAccessTimeMs = taosGetTimestampMs(); + + int32_t keepTime = tsShellActivityTimer * 3; + SAppObj *pApp = taosCachePut(pMgmt->appCache, &pReq->appId, sizeof(pReq->appId), &app, sizeof(app), keepTime * 1000); + if (pApp == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + mError("failed to app %" PRIx64 " into cache since %s", pReq->appId, terrstr()); + return NULL; + } + + mTrace("app %" PRIx64 " is put into cache", pReq->appId); + return pApp; +} + +static void mndFreeApp(SAppObj *pApp) { + mTrace("app %" PRIx64 " is destroyed", pApp->appId); +} + + +static SAppObj *mndAcquireApp(SMnode *pMnode, int64_t appId) { + SProfileMgmt *pMgmt = &pMnode->profileMgmt; + + SAppObj *pApp = taosCacheAcquireByKey(pMgmt->appCache, &appId, sizeof(appId)); + if (pApp == NULL) { + mDebug("app %" PRIx64 " not in cache", appId); + return NULL; + } + + pApp->lastAccessTimeMs = (uint64_t)taosGetTimestampMs(); + + mTrace("app %" PRIx64 " acquired from cache", appId); + return pApp; +} + +static void mndReleaseApp(SMnode *pMnode, SAppObj *pApp) { + if (pApp == NULL) return; + mTrace("release app %" PRIx64 " to cache", pApp->appId); + + SProfileMgmt *pMgmt = &pMnode->profileMgmt; + taosCacheRelease(pMgmt->appCache, (void **)&pApp, false); +} + +void *mndGetNextApp(SMnode *pMnode, SCacheIter *pIter) { + SAppObj *pApp = NULL; + bool hasNext = taosCacheIterNext(pIter); + if (hasNext) { + size_t dataLen = 0; + pApp = taosCacheIterGetData(pIter, &dataLen); + } else { + taosCacheDestroyIter(pIter); + } + + return pApp; +} + +static void mndCancelGetNextApp(SMnode *pMnode, void *pIter) { + if (pIter != NULL) { + taosCacheDestroyIter(pIter); + } +} + static SClientHbRsp *mndMqHbBuildRsp(SMnode *pMnode, SClientHbReq *pReq) { #if 0 SClientHbRsp* pRsp = taosMemoryMalloc(sizeof(SClientHbRsp)); @@ -340,28 +438,51 @@ static SClientHbRsp *mndMqHbBuildRsp(SMnode *pMnode, SClientHbReq *pReq) { return NULL; } +static int32_t mndUpdateAppInfo(SMnode *pMnode, SClientHbReq *pHbReq, SRpcConnInfo *connInfo) { + SAppHbReq* pReq = &pHbReq->app; + SAppObj *pApp = mndAcquireApp(pMnode, pReq->appId); + if (pApp == NULL) { + pApp = mndCreateApp(pMnode, connInfo->clientIp, pReq); + if (pApp == NULL) { + mError("failed to create new app %" PRIx64 " since %s", pReq->appId, terrstr()); + return -1; + } else { + mDebug("a new app %" PRIx64 "created", pReq->appId); + mndReleaseApp(pMnode, pApp); + return TSDB_CODE_SUCCESS; + } + } + + memcpy(&pApp->summary, &pReq->summary, sizeof(pReq->summary)); + + mndReleaseApp(pMnode, pApp); + + return TSDB_CODE_SUCCESS; +} + static int32_t mndProcessQueryHeartBeat(SMnode *pMnode, SRpcMsg *pMsg, SClientHbReq *pHbReq, SClientHbBatchRsp *pBatchRsp) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; SClientHbRsp hbRsp = {.connKey = pHbReq->connKey, .status = 0, .info = NULL, .query = NULL}; + SRpcConnInfo connInfo = pMsg->info.conn; + + mndUpdateAppInfo(pMnode, pHbReq, &connInfo); if (pHbReq->query) { SQueryHbReqBasic *pBasic = pHbReq->query; - SRpcConnInfo connInfo = pMsg->conn; - SConnObj *pConn = mndAcquireConn(pMnode, pBasic->connId); if (pConn == NULL) { pConn = mndCreateConn(pMnode, connInfo.user, CONN_TYPE__QUERY, connInfo.clientIp, connInfo.clientPort, - pBasic->pid, pBasic->app, 0); + pHbReq->app.pid, pHbReq->app.name, 0); if (pConn == NULL) { mError("user:%s, conn:%u is freed and failed to create new since %s", connInfo.user, pBasic->connId, terrstr()); return -1; } else { - mDebug("user:%s, conn:%u is freed and create a new conn:%u", connInfo.user, pBasic->connId, pConn->id); + mDebug("user:%s, conn:%u is freed, will create a new conn:%u", connInfo.user, pBasic->connId, pConn->id); } } - + SQueryHbRspBasic *rspBasic = taosMemoryCalloc(1, sizeof(SQueryHbRspBasic)); if (rspBasic == NULL) { mndReleaseConn(pMnode, pConn); @@ -386,7 +507,7 @@ static int32_t mndProcessQueryHeartBeat(SMnode *pMnode, SRpcMsg *pMsg, SClientHb mndGetMnodeEpSet(pMnode, &rspBasic->epSet); mndCreateQnodeList(pMnode, &rspBasic->pQnodeList, -1); - + mndReleaseConn(pMnode, pConn); hbRsp.query = rspBasic; @@ -500,7 +621,7 @@ static int32_t mndProcessKillQueryReq(SRpcMsg *pReq) { SMnode *pMnode = pReq->info.node; SProfileMgmt *pMgmt = &pMnode->profileMgmt; - SUserObj *pUser = mndAcquireUser(pMnode, pReq->conn.user); + SUserObj *pUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pUser == NULL) return 0; if (!pUser->superUser) { mndReleaseUser(pMnode, pUser); @@ -515,17 +636,28 @@ static int32_t mndProcessKillQueryReq(SRpcMsg *pReq) { return -1; } - mInfo("kill query msg is received, queryId:%d", killReq.queryId); + mInfo("kill query msg is received, queryId:%s", killReq.queryStrId); + int32_t connId = 0; + uint64_t queryId = 0; + char* p = strchr(killReq.queryStrId, ':'); + if (NULL == p) { + mError("invalid query id %s", killReq.queryStrId); + terrno = TSDB_CODE_MND_INVALID_QUERY_ID; + return -1; + } + *p = 0; + connId = taosStr2Int32(killReq.queryStrId, NULL, 16); + queryId = taosStr2UInt64(p + 1, NULL, 16); - SConnObj *pConn = taosCacheAcquireByKey(pMgmt->cache, &killReq.connId, sizeof(int32_t)); + SConnObj *pConn = taosCacheAcquireByKey(pMgmt->connCache, &connId, sizeof(int32_t)); if (pConn == NULL) { - mError("connId:%d, failed to kill queryId:%d, conn not exist", killReq.connId, killReq.queryId); + mError("connId:%x, failed to kill queryId:%" PRIx64 ", conn not exist", connId, queryId); terrno = TSDB_CODE_MND_INVALID_CONN_ID; return -1; } else { - mInfo("connId:%d, queryId:%d is killed by user:%s", killReq.connId, killReq.queryId, pReq->conn.user); - pConn->killId = killReq.queryId; - taosCacheRelease(pMgmt->cache, (void **)&pConn, false); + mInfo("connId:%x, queryId:%" PRIx64 " is killed by user:%s", connId, queryId, pReq->info.conn.user); + pConn->killId = queryId; + taosCacheRelease(pMgmt->connCache, (void **)&pConn, false); return 0; } } @@ -534,7 +666,7 @@ static int32_t mndProcessKillConnReq(SRpcMsg *pReq) { SMnode *pMnode = pReq->info.node; SProfileMgmt *pMgmt = &pMnode->profileMgmt; - SUserObj *pUser = mndAcquireUser(pMnode, pReq->conn.user); + SUserObj *pUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pUser == NULL) return 0; if (!pUser->superUser) { mndReleaseUser(pMnode, pUser); @@ -549,29 +681,29 @@ static int32_t mndProcessKillConnReq(SRpcMsg *pReq) { return -1; } - SConnObj *pConn = taosCacheAcquireByKey(pMgmt->cache, &killReq.connId, sizeof(int32_t)); + SConnObj *pConn = taosCacheAcquireByKey(pMgmt->connCache, &killReq.connId, sizeof(uint32_t)); if (pConn == NULL) { - mError("connId:%d, failed to kill connection, conn not exist", killReq.connId); + mError("connId:%u, failed to kill connection, conn not exist", killReq.connId); terrno = TSDB_CODE_MND_INVALID_CONN_ID; return -1; } else { - mInfo("connId:%d, is killed by user:%s", killReq.connId, pReq->conn.user); + mInfo("connId:%u, is killed by user:%s", killReq.connId, pReq->info.conn.user); pConn->killed = 1; - taosCacheRelease(pMgmt->cache, (void **)&pConn, false); + taosCacheRelease(pMgmt->connCache, (void **)&pConn, false); return TSDB_CODE_SUCCESS; } } static int32_t mndRetrieveConns(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) { - SMnode *pMnode = pReq->info.node; - SSdb *pSdb = pMnode->pSdb; - int32_t numOfRows = 0; - int32_t cols = 0; - SConnObj *pConn = NULL; - + SMnode *pMnode = pReq->info.node; + SSdb *pSdb = pMnode->pSdb; + int32_t numOfRows = 0; + int32_t cols = 0; + SConnObj *pConn = NULL; + if (pShow->pIter == NULL) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; - pShow->pIter = taosCacheCreateIter(pMgmt->cache); + pShow->pIter = taosCacheCreateIter(pMgmt->connCache); } while (numOfRows < rows) { @@ -619,15 +751,15 @@ static int32_t mndRetrieveConns(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBl } static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) { - SMnode *pMnode = pReq->info.node; - SSdb *pSdb = pMnode->pSdb; - int32_t numOfRows = 0; - int32_t cols = 0; - SConnObj *pConn = NULL; - + SMnode *pMnode = pReq->info.node; + SSdb *pSdb = pMnode->pSdb; + int32_t numOfRows = 0; + int32_t cols = 0; + SConnObj *pConn = NULL; + if (pShow->pIter == NULL) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; - pShow->pIter = taosCacheCreateIter(pMgmt->cache); + pShow->pIter = taosCacheCreateIter(pMgmt->connCache); } while (numOfRows < rows) { @@ -645,7 +777,7 @@ static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p int32_t numOfQueries = taosArrayGetSize(pConn->pQueries); for (int32_t i = 0; i < numOfQueries; ++i) { - SQueryDesc* pQuery = taosArrayGet(pConn->pQueries, i); + SQueryDesc *pQuery = taosArrayGet(pConn->pQueries, i); cols = 0; char queryId[26 + VARSTR_HEADER_SIZE] = {0}; @@ -666,7 +798,7 @@ static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p colDataAppend(pColInfo, numOfRows, (const char *)app, false); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); - colDataAppend(pColInfo, numOfRows, (const char *)&pQuery->pid, false); + colDataAppend(pColInfo, numOfRows, (const char *)&pConn->pid, false); char user[TSDB_USER_LEN + VARSTR_HEADER_SIZE] = {0}; STR_TO_VARSTR(user, pConn->user); @@ -691,14 +823,14 @@ static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, (const char *)&pQuery->subPlanNum, false); - char subStatus[TSDB_SHOW_SUBQUERY_LEN + VARSTR_HEADER_SIZE] = {0}; + char subStatus[TSDB_SHOW_SUBQUERY_LEN + VARSTR_HEADER_SIZE] = {0}; int32_t strSize = sizeof(subStatus); int32_t offset = VARSTR_HEADER_SIZE; for (int32_t i = 0; i < pQuery->subPlanNum && offset < strSize; ++i) { if (i) { offset += snprintf(subStatus + offset, strSize - offset - 1, ","); } - SQuerySubDesc* pDesc = taosArrayGet(pQuery->subDesc, i); + SQuerySubDesc *pDesc = taosArrayGet(pQuery->subDesc, i); offset += snprintf(subStatus + offset, strSize - offset - 1, "%" PRIu64 ":%s", pDesc->tid, pDesc->status); } varDataLen(subStatus) = strlen(&subStatus[VARSTR_HEADER_SIZE]); @@ -712,7 +844,7 @@ static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p numOfRows++; } - + taosRUnLockLatch(&pConn->queryLock); } @@ -720,6 +852,86 @@ static int32_t mndRetrieveQueries(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p return numOfRows; } +static int32_t mndRetrieveApps(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) { + SMnode *pMnode = pReq->info.node; + SSdb *pSdb = pMnode->pSdb; + int32_t numOfRows = 0; + int32_t cols = 0; + SAppObj *pApp = NULL; + + if (pShow->pIter == NULL) { + SProfileMgmt *pMgmt = &pMnode->profileMgmt; + pShow->pIter = taosCacheCreateIter(pMgmt->appCache); + } + + while (numOfRows < rows) { + pApp = mndGetNextApp(pMnode, pShow->pIter); + if (pApp == NULL) { + pShow->pIter = NULL; + break; + } + + cols = 0; + + SColumnInfoData *pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->appId, false); + + char ip[TSDB_IPv4ADDR_LEN + 6 + VARSTR_HEADER_SIZE] = {0}; + sprintf(&ip[VARSTR_HEADER_SIZE], "%s", taosIpStr(pApp->ip)); + varDataLen(ip) = strlen(&ip[VARSTR_HEADER_SIZE]); + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)ip, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->pid, false); + + char name[TSDB_APP_NAME_LEN + 6 + VARSTR_HEADER_SIZE] = {0}; + sprintf(&name[VARSTR_HEADER_SIZE], "%s", pApp->name); + varDataLen(name) = strlen(&name[VARSTR_HEADER_SIZE]); + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)name, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->startTime, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.numOfInsertsReq, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.numOfInsertRows, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.insertElapsedTime, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.insertBytes, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.fetchBytes, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.queryElapsedTime, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.numOfSlowQueries, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.totalRequests, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->summary.currentRequests, false); + + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)&pApp->lastAccessTimeMs, false); + + numOfRows++; + } + + pShow->numOfRows += numOfRows; + return numOfRows; +} + + static void mndCancelGetNextQuery(SMnode *pMnode, void *pIter) { if (pIter != NULL) { taosCacheDestroyIter(pIter); @@ -728,5 +940,5 @@ static void mndCancelGetNextQuery(SMnode *pMnode, void *pIter) { int32_t mndGetNumOfConnections(SMnode *pMnode) { SProfileMgmt *pMgmt = &pMnode->profileMgmt; - return taosCacheGetNumOfObj(pMgmt->cache); + return taosCacheGetNumOfObj(pMgmt->connCache); } diff --git a/source/dnode/mnode/impl/src/mndQnode.c b/source/dnode/mnode/impl/src/mndQnode.c index 595287a3af9515fecc0e73345208606925152994..f5625f32d5a73af783dc0172ba949cc6d4532462 100644 --- a/source/dnode/mnode/impl/src/mndQnode.c +++ b/source/dnode/mnode/impl/src/mndQnode.c @@ -294,7 +294,7 @@ static int32_t mndProcessCreateQnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_QNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_QNODE) != 0) { goto _OVER; } @@ -396,7 +396,7 @@ static int32_t mndProcessDropQnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_QNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_QNODE) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndQuery.c b/source/dnode/mnode/impl/src/mndQuery.c index 5374f48e47521f7cdc477ac469ce0bf5fbbf7aef..f32a3129dece8e55d37d68d61e39787e31fb8bdb 100644 --- a/source/dnode/mnode/impl/src/mndQuery.c +++ b/source/dnode/mnode/impl/src/mndQuery.c @@ -25,6 +25,13 @@ int32_t mndPreProcessMsg(SRpcMsg *pMsg) { return qWorkerPreprocessQueryMsg(pMnode->pQuery, pMsg); } +void mndAbortPreprocessMsg(SRpcMsg *pMsg) { + if (TDMT_VND_QUERY != pMsg->msgType) return; + + SMnode *pMnode = pMsg->info.node; + qWorkerAbortPreprocessQueryMsg(pMnode->pQuery, pMsg); +} + int32_t mndProcessQueryMsg(SRpcMsg *pMsg) { int32_t code = -1; SMnode *pMnode = pMsg->info.node; diff --git a/source/dnode/mnode/impl/src/mndScheduler.c b/source/dnode/mnode/impl/src/mndScheduler.c index 6f8fc748c2693365d675bb96176827683698f2fe..645634d7f3cb66ed979811bc1f9083dd9df7d243 100644 --- a/source/dnode/mnode/impl/src/mndScheduler.c +++ b/source/dnode/mnode/impl/src/mndScheduler.c @@ -33,7 +33,7 @@ #include "tname.h" #include "tuuid.h" -extern bool tsStreamSchedV; +extern bool tsSchedStreamToSnode; static int32_t mndAddTaskToTaskSet(SArray* pArray, SStreamTask* pTask) { int32_t childId = taosArrayGetSize(pArray); @@ -43,7 +43,7 @@ static int32_t mndAddTaskToTaskSet(SArray* pArray, SStreamTask* pTask) { } int32_t mndConvertRsmaTask(char** pDst, int32_t* pDstLen, const char* ast, int64_t uid, int8_t triggerType, - int64_t watermark, double filesFactor) { + int64_t watermark) { SNode* pAst = NULL; SQueryPlan* pPlan = NULL; terrno = TSDB_CODE_SUCCESS; @@ -63,9 +63,8 @@ int32_t mndConvertRsmaTask(char** pDst, int32_t* pDstLen, const char* ast, int64 .topicQuery = false, .streamQuery = true, .rSmaQuery = true, - .triggerType = STREAM_TRIGGER_AT_ONCE, + .triggerType = triggerType, .watermark = watermark, - /*.filesFactor = filesFactor,*/ }; if (qCreateQueryPlan(&cxt, &pPlan, NULL) < 0) { @@ -105,7 +104,7 @@ int32_t mndPersistTaskDeployReq(STrans* pTrans, SStreamTask* pTask, const SEpSet int32_t size = encoder.pos; int32_t tlen = sizeof(SMsgHead) + size; tEncoderClear(&encoder); - void* buf = taosMemoryMalloc(tlen); + void* buf = taosMemoryCalloc(1, tlen); if (buf == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; return -1; @@ -157,6 +156,7 @@ int32_t mndAddDispatcherToInnerTask(SMnode* pMnode, STrans* pTrans, SStreamObj* } sdbRelease(pMnode->pSdb, pDb); + memcpy(pTask->shuffleDispatcher.stbFullName, pStream->targetSTbName, TSDB_TABLE_FNAME_LEN); SArray* pVgs = pTask->shuffleDispatcher.dbInfo.pVgroupInfos; int32_t sz = taosArrayGetSize(pVgs); SArray* sinkLv = taosArrayGetP(pStream->tasks, 0); @@ -166,6 +166,7 @@ int32_t mndAddDispatcherToInnerTask(SMnode* pMnode, STrans* pTrans, SStreamObj* for (int32_t j = 0; j < sinkLvSize; j++) { SStreamTask* pLastLevelTask = taosArrayGetP(sinkLv, j); if (pLastLevelTask->nodeId == pVgInfo->vgId) { + ASSERT(pVgInfo->vgId > 0); pVgInfo->taskId = pLastLevelTask->taskId; ASSERT(pVgInfo->taskId != 0); break; @@ -203,9 +204,11 @@ int32_t mndAssignTaskToVg(SMnode* pMnode, STrans* pTrans, SStreamTask* pTask, SS return 0; } -SSnodeObj* mndSchedFetchSnode(SMnode* pMnode) { +SSnodeObj* mndSchedFetchOneSnode(SMnode* pMnode) { SSnodeObj* pObj = NULL; - pObj = sdbFetch(pMnode->pSdb, SDB_SNODE, NULL, (void**)&pObj); + void* pIter = NULL; + // TODO random fetch + pIter = sdbFetch(pMnode->pSdb, SDB_SNODE, pIter, (void**)&pObj); return pObj; } @@ -213,7 +216,7 @@ int32_t mndAssignTaskToSnode(SMnode* pMnode, STrans* pTrans, SStreamTask* pTask, const SSnodeObj* pSnode) { int32_t msgLen; - pTask->nodeId = 0; + pTask->nodeId = SNODE_HANDLE; pTask->epSet = mndAcquireEpFromSnode(pMnode, pSnode); plan->execNode.nodeId = 0; @@ -223,7 +226,7 @@ int32_t mndAssignTaskToSnode(SMnode* pMnode, STrans* pTrans, SStreamTask* pTask, terrno = TSDB_CODE_QRY_INVALID_INPUT; return -1; } - mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_STREAM_TASK_DEPLOY, 0); + mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_STREAM_TASK_DEPLOY, SNODE_HANDLE); return 0; } @@ -268,7 +271,6 @@ int32_t mndAddShuffleSinkTasksToStream(SMnode* pMnode, STrans* pTrans, SStreamOb pTask->epSet = mndGetVgroupEpset(pMnode, pVgroup); // source - pTask->sourceType = TASK_SOURCE__MERGE; pTask->inputType = TASK_INPUT_TYPE__DATA_BLOCK; // exec @@ -314,7 +316,6 @@ int32_t mndAddFixedSinkTaskToStream(SMnode* pMnode, STrans* pTrans, SStreamObj* #endif pTask->epSet = mndGetVgroupEpset(pMnode, &pStream->fixedSinkVg); // source - pTask->sourceType = TASK_SOURCE__MERGE; pTask->inputType = TASK_INPUT_TYPE__DATA_BLOCK; // exec @@ -371,8 +372,8 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) { } if (totLevel > 1) { - SStreamTask* pFinalTask; - // inner plan + SStreamTask* pInnerTask; + // inner level { SArray* taskInnerLevel = taosArrayInit(0, sizeof(void*)); taosArrayPush(pStream->tasks, &taskInnerLevel); @@ -381,31 +382,51 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) { SSubplan* plan = (SSubplan*)nodesListGetNode(inner->pNodeList, 0); ASSERT(plan->subplanType == SUBPLAN_TYPE_MERGE); - pFinalTask = tNewSStreamTask(pStream->uid); - mndAddTaskToTaskSet(taskInnerLevel, pFinalTask); + pInnerTask = tNewSStreamTask(pStream->uid); + mndAddTaskToTaskSet(taskInnerLevel, pInnerTask); // input - pFinalTask->inputType = TASK_INPUT_TYPE__DATA_BLOCK; + pInnerTask->inputType = TASK_INPUT_TYPE__DATA_BLOCK; // trigger - pFinalTask->triggerParam = pStream->triggerParam; + pInnerTask->triggerParam = pStream->triggerParam; // dispatch - if (mndAddDispatcherToInnerTask(pMnode, pTrans, pStream, pFinalTask) < 0) { + if (mndAddDispatcherToInnerTask(pMnode, pTrans, pStream, pInnerTask) < 0) { qDestroyQueryPlan(pPlan); return -1; } // exec - pFinalTask->execType = TASK_EXEC__PIPE; - SVgObj* pVgroup = mndSchedFetchOneVg(pMnode, pStream->sourceDbUid); - if (mndAssignTaskToVg(pMnode, pTrans, pFinalTask, plan, pVgroup) < 0) { - sdbRelease(pSdb, pVgroup); - qDestroyQueryPlan(pPlan); - return -1; + pInnerTask->execType = TASK_EXEC__PIPE; + + if (tsSchedStreamToSnode) { + SSnodeObj* pSnode = mndSchedFetchOneSnode(pMnode); + if (pSnode == NULL) { + SVgObj* pVgroup = mndSchedFetchOneVg(pMnode, pStream->sourceDbUid); + if (mndAssignTaskToVg(pMnode, pTrans, pInnerTask, plan, pVgroup) < 0) { + sdbRelease(pSdb, pVgroup); + qDestroyQueryPlan(pPlan); + return -1; + } + } else { + if (mndAssignTaskToSnode(pMnode, pTrans, pInnerTask, plan, pSnode) < 0) { + ASSERT(0); + sdbRelease(pSdb, pSnode); + qDestroyQueryPlan(pPlan); + return -1; + } + } + } else { + SVgObj* pVgroup = mndSchedFetchOneVg(pMnode, pStream->sourceDbUid); + if (mndAssignTaskToVg(pMnode, pTrans, pInnerTask, plan, pVgroup) < 0) { + sdbRelease(pSdb, pVgroup); + qDestroyQueryPlan(pPlan); + return -1; + } } } - // source plan + // source level SArray* taskSourceLevel = taosArrayInit(0, sizeof(void*)); taosArrayPush(pStream->tasks, &taskSourceLevel); @@ -425,6 +446,8 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) { SStreamTask* pTask = tNewSStreamTask(pStream->uid); mndAddTaskToTaskSet(taskSourceLevel, pTask); + pTask->dataScan = 1; + // input pTask->inputType = TASK_INPUT_TYPE__SUMBIT_BLOCK; @@ -433,9 +456,9 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) { pTask->dispatchMsgType = TDMT_STREAM_TASK_DISPATCH; pTask->dispatchType = TASK_DISPATCH__FIXED; - pTask->fixedEpDispatcher.taskId = pFinalTask->taskId; - pTask->fixedEpDispatcher.nodeId = pFinalTask->nodeId; - pTask->fixedEpDispatcher.epSet = pFinalTask->epSet; + pTask->fixedEpDispatcher.taskId = pInnerTask->taskId; + pTask->fixedEpDispatcher.nodeId = pInnerTask->nodeId; + pTask->fixedEpDispatcher.epSet = pInnerTask->epSet; // exec pTask->execType = TASK_EXEC__PIPE; @@ -468,6 +491,8 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) { SStreamTask* pTask = tNewSStreamTask(pStream->uid); mndAddTaskToTaskSet(taskOneLevel, pTask); + pTask->dataScan = 1; + // input pTask->inputType = TASK_INPUT_TYPE__SUMBIT_BLOCK; diff --git a/source/dnode/mnode/impl/src/mndShow.c b/source/dnode/mnode/impl/src/mndShow.c index d312955202946ca530a4f583484177c9913b6ddf..fed6a5a721088fc39f24712f922ffe13e80e48d7 100644 --- a/source/dnode/mnode/impl/src/mndShow.c +++ b/source/dnode/mnode/impl/src/mndShow.c @@ -104,6 +104,8 @@ static int32_t convertToRetrieveType(char *name, int32_t len) { type = TSDB_MGMT_TABLE_TOPICS; } else if (strncasecmp(name, TSDB_PERFS_TABLE_STREAMS, len) == 0) { type = TSDB_MGMT_TABLE_STREAMS; + } else if (strncasecmp(name, TSDB_PERFS_TABLE_APPS, len) == 0) { + type = TSDB_MGMT_TABLE_APPS; } else { // ASSERT(0); } @@ -229,7 +231,7 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { mDebug("show:0x%" PRIx64 ", start retrieve data, type:%d", pShow->id, pShow->type); - // if (mndCheckShowAuth(pMnode, pReq->conn.user, pShow->type) != 0) return -1; + // if (mndCheckShowAuth(pMnode, pReq->info.conn.user, pShow->type) != 0) return -1; int32_t numOfCols = pShow->pMeta->numOfColumns; SSDataBlock *pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); diff --git a/source/dnode/mnode/impl/src/mndSma.c b/source/dnode/mnode/impl/src/mndSma.c index 3eb5b03efd481e624980d5a0c2462bf4f6716ba3..c44fb03be23c7740759cf31233b4847756d1af08 100644 --- a/source/dnode/mnode/impl/src/mndSma.c +++ b/source/dnode/mnode/impl/src/mndSma.c @@ -530,7 +530,10 @@ static int32_t mndCreateSma(SMnode *pMnode, SRpcMsg *pReq, SMCreateSmaReq *pCrea smaObj.dbUid = pStb->dbUid; smaObj.intervalUnit = pCreate->intervalUnit; smaObj.slidingUnit = pCreate->slidingUnit; +#if 0 smaObj.timezone = pCreate->timezone; +#endif + smaObj.timezone = tsTimezone; // use timezone of server smaObj.interval = pCreate->interval; smaObj.offset = pCreate->offset; smaObj.sliding = pCreate->sliding; @@ -609,7 +612,7 @@ static int32_t mndCreateSma(SMnode *pMnode, SRpcMsg *pReq, SMCreateSmaReq *pCrea int32_t code = -1; STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB, pReq); if (pTrans == NULL) goto _OVER; - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); mndTransSetSerial(pTrans); mDebug("trans:%d, used to create sma:%s", pTrans->id, pCreate->name); @@ -710,7 +713,7 @@ static int32_t mndProcessCreateSmaReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } @@ -852,7 +855,7 @@ static int32_t mndDropSma(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SSmaObj *p if (pTrans == NULL) goto _OVER; mDebug("trans:%d, used to drop sma:%s", pTrans->id, pSma->name); - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); if (mndSetDropSmaRedoLogs(pMnode, pTrans, pSma) != 0) goto _OVER; if (mndSetDropSmaVgroupRedoLogs(pMnode, pTrans, pVgroup) != 0) goto _OVER; @@ -971,7 +974,7 @@ static int32_t mndProcessDropSmaReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndSnode.c b/source/dnode/mnode/impl/src/mndSnode.c index c84dc2f3ddc3d52b43c6a47fef3f8bf5c3f3d97b..0a99f356b194c94597d53b2f35fd011885d7b419 100644 --- a/source/dnode/mnode/impl/src/mndSnode.c +++ b/source/dnode/mnode/impl/src/mndSnode.c @@ -300,7 +300,7 @@ static int32_t mndProcessCreateSnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_SNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_SNODE) != 0) { goto _OVER; } @@ -404,7 +404,7 @@ static int32_t mndProcessDropSnodeReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_SNODE) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_SNODE) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndStb.c b/source/dnode/mnode/impl/src/mndStb.c index b83aa34a3a59d7e1b2e99ad998280215c995a77e..3e50ea8262045795cc0e8b28c7e5aa967713aacb 100644 --- a/source/dnode/mnode/impl/src/mndStb.c +++ b/source/dnode/mnode/impl/src/mndStb.c @@ -89,8 +89,10 @@ SSdbRaw *mndStbActionEncode(SStbObj *pStb) { SDB_SET_INT32(pRaw, dataPos, pStb->tagVer, _OVER) SDB_SET_INT32(pRaw, dataPos, pStb->colVer, _OVER) SDB_SET_INT32(pRaw, dataPos, pStb->nextColId, _OVER) - SDB_SET_INT32(pRaw, dataPos, (int32_t)(pStb->xFilesFactor * 10000), _OVER) - SDB_SET_INT32(pRaw, dataPos, pStb->delay, _OVER) + SDB_SET_INT64(pRaw, dataPos, pStb->maxdelay[0], _OVER) + SDB_SET_INT64(pRaw, dataPos, pStb->maxdelay[1], _OVER) + SDB_SET_INT64(pRaw, dataPos, pStb->watermark[0], _OVER) + SDB_SET_INT64(pRaw, dataPos, pStb->watermark[1], _OVER) SDB_SET_INT32(pRaw, dataPos, pStb->ttl, _OVER) SDB_SET_INT32(pRaw, dataPos, pStb->numOfColumns, _OVER) SDB_SET_INT32(pRaw, dataPos, pStb->numOfTags, _OVER) @@ -117,7 +119,7 @@ SSdbRaw *mndStbActionEncode(SStbObj *pStb) { } if (pStb->commentLen > 0) { - SDB_SET_BINARY(pRaw, dataPos, pStb->comment, pStb->commentLen, _OVER) + SDB_SET_BINARY(pRaw, dataPos, pStb->comment, pStb->commentLen + 1, _OVER) } if (pStb->ast1Len > 0) { SDB_SET_BINARY(pRaw, dataPos, pStb->pAst1, pStb->ast1Len, _OVER) @@ -168,10 +170,10 @@ static SSdbRow *mndStbActionDecode(SSdbRaw *pRaw) { SDB_GET_INT32(pRaw, dataPos, &pStb->tagVer, _OVER) SDB_GET_INT32(pRaw, dataPos, &pStb->colVer, _OVER) SDB_GET_INT32(pRaw, dataPos, &pStb->nextColId, _OVER) - int32_t xFilesFactor = 0; - SDB_GET_INT32(pRaw, dataPos, &xFilesFactor, _OVER) - pStb->xFilesFactor = xFilesFactor / 10000.0f; - SDB_GET_INT32(pRaw, dataPos, &pStb->delay, _OVER) + SDB_GET_INT64(pRaw, dataPos, &pStb->maxdelay[0], _OVER) + SDB_GET_INT64(pRaw, dataPos, &pStb->maxdelay[1], _OVER) + SDB_GET_INT64(pRaw, dataPos, &pStb->watermark[0], _OVER) + SDB_GET_INT64(pRaw, dataPos, &pStb->watermark[1], _OVER) SDB_GET_INT32(pRaw, dataPos, &pStb->ttl, _OVER) SDB_GET_INT32(pRaw, dataPos, &pStb->numOfColumns, _OVER) SDB_GET_INT32(pRaw, dataPos, &pStb->numOfTags, _OVER) @@ -204,9 +206,9 @@ static SSdbRow *mndStbActionDecode(SSdbRaw *pRaw) { } if (pStb->commentLen > 0) { - pStb->comment = taosMemoryCalloc(pStb->commentLen, 1); + pStb->comment = taosMemoryCalloc(pStb->commentLen + 1, 1); if (pStb->comment == NULL) goto _OVER; - SDB_GET_BINARY(pRaw, dataPos, pStb->comment, pStb->commentLen, _OVER) + SDB_GET_BINARY(pRaw, dataPos, pStb->comment, pStb->commentLen + 1, _OVER) } if (pStb->ast1Len > 0) { pStb->pAst1 = taosMemoryCalloc(pStb->ast1Len, 1); @@ -280,8 +282,8 @@ static int32_t mndStbActionUpdate(SSdb *pSdb, SStbObj *pOld, SStbObj *pNew) { } } - if (pOld->commentLen < pNew->commentLen) { - void *comment = taosMemoryMalloc(pNew->commentLen); + if (pOld->commentLen < pNew->commentLen && pNew->commentLen > 0) { + void *comment = taosMemoryMalloc(pNew->commentLen + 1); if (comment != NULL) { taosMemoryFree(pOld->comment); pOld->comment = comment; @@ -291,6 +293,7 @@ static int32_t mndStbActionUpdate(SSdb *pSdb, SStbObj *pOld, SStbObj *pNew) { taosWUnLockLatch(&pOld->lock); } } + pOld->commentLen = pNew->commentLen; if (pOld->ast1Len < pNew->ast1Len) { void *pAst1 = taosMemoryMalloc(pNew->ast1Len); @@ -330,8 +333,8 @@ static int32_t mndStbActionUpdate(SSdb *pSdb, SStbObj *pOld, SStbObj *pNew) { pOld->numOfTags = pNew->numOfTags; memcpy(pOld->pTags, pNew->pTags, pOld->numOfTags * sizeof(SSchema)); } - if (pNew->commentLen != 0) { - memcpy(pOld->comment, pNew->comment, pNew->commentLen); + if (pNew->commentLen > 0) { + memcpy(pOld->comment, pNew->comment, pNew->commentLen + 1); } if (pNew->ast1Len != 0) { memcpy(pOld->pAst1, pNew->pAst1, pNew->ast1Len); @@ -398,18 +401,18 @@ static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pSt req.schemaTag.pSchema = pStb->pTags; if (req.rollup) { - req.pRSmaParam.xFilesFactor = pStb->xFilesFactor; - req.pRSmaParam.delay = pStb->delay; + req.pRSmaParam.maxdelay[0] = pStb->maxdelay[0]; + req.pRSmaParam.maxdelay[1] = pStb->maxdelay[1]; if (pStb->ast1Len > 0) { - if (mndConvertRsmaTask(&req.pRSmaParam.qmsg1, &req.pRSmaParam.qmsg1Len, pStb->pAst1, pStb->uid, - STREAM_TRIGGER_AT_ONCE, 0, req.pRSmaParam.xFilesFactor) != TSDB_CODE_SUCCESS) { - return NULL; + if (mndConvertRsmaTask(&req.pRSmaParam.qmsg[0], &req.pRSmaParam.qmsgLen[0], pStb->pAst1, pStb->uid, + STREAM_TRIGGER_WINDOW_CLOSE, req.pRSmaParam.watermark[0]) < 0) { + goto _err; } } if (pStb->ast2Len > 0) { - if (mndConvertRsmaTask(&req.pRSmaParam.qmsg2, &req.pRSmaParam.qmsg2Len, pStb->pAst2, pStb->uid, - STREAM_TRIGGER_AT_ONCE, 0, req.pRSmaParam.xFilesFactor) != TSDB_CODE_SUCCESS) { - return NULL; + if (mndConvertRsmaTask(&req.pRSmaParam.qmsg[1], &req.pRSmaParam.qmsgLen[1], pStb->pAst2, pStb->uid, + STREAM_TRIGGER_WINDOW_CLOSE, req.pRSmaParam.watermark[1]) < 0) { + goto _err; } } } @@ -417,17 +420,15 @@ static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pSt int32_t ret = 0; tEncodeSize(tEncodeSVCreateStbReq, &req, contLen, ret); if (ret < 0) { - return NULL; + goto _err; } contLen += sizeof(SMsgHead); SMsgHead *pHead = taosMemoryMalloc(contLen); if (pHead == NULL) { - taosMemoryFreeClear(req.pRSmaParam.qmsg1); - taosMemoryFreeClear(req.pRSmaParam.qmsg2); terrno = TSDB_CODE_OUT_OF_MEMORY; - return NULL; + goto _err; } pHead->contLen = htonl(contLen); @@ -437,17 +438,19 @@ static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pSt tEncoderInit(&encoder, pBuf, contLen - sizeof(SMsgHead)); if (tEncodeSVCreateStbReq(&encoder, &req) < 0) { taosMemoryFreeClear(pHead); - taosMemoryFreeClear(req.pRSmaParam.qmsg1); - taosMemoryFreeClear(req.pRSmaParam.qmsg2); tEncoderClear(&encoder); - return NULL; + goto _err; } tEncoderClear(&encoder); *pContLen = contLen; - taosMemoryFreeClear(req.pRSmaParam.qmsg1); - taosMemoryFreeClear(req.pRSmaParam.qmsg2); + taosMemoryFreeClear(req.pRSmaParam.qmsg[0]); + taosMemoryFreeClear(req.pRSmaParam.qmsg[1]); return pHead; +_err: + taosMemoryFreeClear(req.pRSmaParam.qmsg[0]); + taosMemoryFreeClear(req.pRSmaParam.qmsg[1]); + return NULL; } static void *mndBuildVDropStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb, int32_t *pContLen) { @@ -669,19 +672,21 @@ int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreat pDst->tagVer = 1; pDst->colVer = 1; pDst->nextColId = 1; - pDst->xFilesFactor = pCreate->xFilesFactor; - pDst->delay = pCreate->delay; + pDst->maxdelay[0] = pCreate->delay1; + pDst->maxdelay[1] = pCreate->delay2; + pDst->watermark[0] = pCreate->watermark1; + pDst->watermark[1] = pCreate->watermark2; pDst->ttl = pCreate->ttl; pDst->numOfColumns = pCreate->numOfColumns; pDst->numOfTags = pCreate->numOfTags; pDst->commentLen = pCreate->commentLen; if (pDst->commentLen > 0) { - pDst->comment = taosMemoryCalloc(pDst->commentLen, 1); + pDst->comment = taosMemoryCalloc(pDst->commentLen + 1, 1); if (pDst->comment == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; return -1; } - memcpy(pDst->comment, pCreate->comment, pDst->commentLen); + memcpy(pDst->comment, pCreate->comment, pDst->commentLen + 1); } pDst->ast1Len = pCreate->ast1Len; @@ -758,7 +763,7 @@ _OVER: } int32_t mndAddStbToTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb) { - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); if (mndSetCreateStbRedoLogs(pMnode, pTrans, pDb, pStb) != 0) return -1; if (mndSetCreateStbUndoLogs(pMnode, pTrans, pDb, pStb) != 0) return -1; if (mndSetCreateStbCommitLogs(pMnode, pTrans, pDb, pStb) != 0) return -1; @@ -805,7 +810,7 @@ static int32_t mndProcessCreateStbReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } @@ -835,7 +840,7 @@ _OVER: } static int32_t mndCheckAlterStbReq(SMAlterStbReq *pAlter) { - if (pAlter->commentLen != 0 || pAlter->ttl != 0) return 0; + if (pAlter->commentLen >= 0 || pAlter->ttl != 0) return 0; if (pAlter->numOfFields < 1 || pAlter->numOfFields != (int32_t)taosArrayGetSize(pAlter->pFields)) { terrno = TSDB_CODE_MND_INVALID_STB_OPTION; @@ -890,13 +895,16 @@ static int32_t mndUpdateStbCommentAndTTL(const SStbObj *pOld, SStbObj *pNew, cha int32_t ttl) { if (commentLen > 0) { pNew->commentLen = commentLen; - pNew->comment = taosMemoryCalloc(1, commentLen); + pNew->comment = taosMemoryCalloc(1, commentLen + 1); if (pNew->comment == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; return -1; } - memcpy(pNew->comment, pComment, commentLen); + memcpy(pNew->comment, pComment, commentLen + 1); + } else if (commentLen == 0) { + pNew->commentLen = 0; } + if (ttl >= 0) { pNew->ttl = ttl; } @@ -1206,7 +1214,6 @@ static int32_t mndSetAlterStbRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj sdbRelease(pSdb, pVgroup); return -1; } - STransAction action = {0}; action.epSet = mndGetVgroupEpset(pMnode, pVgroup); action.pCont = pReq; @@ -1396,7 +1403,7 @@ static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *p if (pTrans == NULL) goto _OVER; mDebug("trans:%d, used to alter stb:%s", pTrans->id, pAlter->name); - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); if (needRsp) { void *pCont = NULL; @@ -1454,7 +1461,7 @@ static int32_t mndProcessAlterStbReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } @@ -1537,7 +1544,7 @@ static int32_t mndDropStb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *p if (pTrans == NULL) goto _OVER; mDebug("trans:%d, used to drop stb:%s", pTrans->id, pStb->name); - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); if (mndSetDropStbRedoLogs(pMnode, pTrans, pStb) != 0) goto _OVER; if (mndSetDropStbCommitLogs(pMnode, pTrans, pStb) != 0) goto _OVER; @@ -1584,7 +1591,7 @@ static int32_t mndProcessDropStbReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } @@ -1841,17 +1848,17 @@ static int32_t mndRetrieveStb(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, (const char *)&pStb->updateTime, false); // number of tables - char *p = taosMemoryCalloc(1, pStb->commentLen + 1 + VARSTR_HEADER_SIZE); // check malloc failures - if (p != NULL) { - if (pStb->commentLen != 0) { - STR_TO_VARSTR(p, pStb->comment); - } else { - STR_TO_VARSTR(p, ""); - } - - pColInfo = taosArrayGet(pBlock->pDataBlock, cols); - colDataAppend(pColInfo, numOfRows, (const char *)p, false); - taosMemoryFree(p); + pColInfo = taosArrayGet(pBlock->pDataBlock, cols); + if (pStb->commentLen > 0) { + char comment[TSDB_TB_COMMENT_LEN + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, pStb->comment); + colDataAppend(pColInfo, numOfRows, comment, false); + } else if (pStb->commentLen == 0) { + char comment[VARSTR_HEADER_SIZE + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, ""); + colDataAppend(pColInfo, numOfRows, comment, false); + } else { + colDataAppendNULL(pColInfo, numOfRows); } numOfRows++; diff --git a/source/dnode/mnode/impl/src/mndStream.c b/source/dnode/mnode/impl/src/mndStream.c index 3a42c694c849fd1d03350f87f4f2d086d589d7bf..d432256f1517ffb88bdab81e63e9aa3e94b6b07f 100644 --- a/source/dnode/mnode/impl/src/mndStream.c +++ b/source/dnode/mnode/impl/src/mndStream.c @@ -545,7 +545,7 @@ static int32_t mndCreateStream(SMnode *pMnode, SRpcMsg *pReq, SCMCreateStreamReq return -1; } - if (streamObj.targetSTbName[0] && mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->conn.user) < 0) { + if (streamObj.targetSTbName[0] && mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->info.conn.user) < 0) { mError("trans:%d, failed to create stb for stream since %s", pTrans->id, terrstr()); mndTransDrop(pTrans); return -1; @@ -602,7 +602,7 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } #endif @@ -613,9 +613,9 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) { goto _OVER; } - mndTransSetDbName(pTrans, createStreamReq.sourceDB); + mndTransSetDbName(pTrans, createStreamReq.sourceDB, NULL); // TODO - /*mndTransSetDbName(pTrans, streamObj.targetDb);*/ + /*mndTransSetDbName(pTrans, streamObj.targetDb, NULL);*/ mDebug("trans:%d, used to create stream:%s", pTrans->id, createStreamReq.name); // build stream obj from request @@ -627,7 +627,7 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) { } // create stb for stream - if (mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->conn.user) < 0) { + if (mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->info.conn.user) < 0) { mError("trans:%d, failed to create stb for stream %s since %s", pTrans->id, createStreamReq.name, terrstr()); mndTransDrop(pTrans); goto _OVER; @@ -696,7 +696,7 @@ static int32_t mndProcessDropStreamReq(SRpcMsg *pReq) { #if 0 // todo check auth - pUser = mndAcquireUser(pMnode, pReq->conn.user); + pUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pUser == NULL) { goto DROP_STREAM_OVER; } diff --git a/source/dnode/mnode/impl/src/mndSubscribe.c b/source/dnode/mnode/impl/src/mndSubscribe.c index 65a5d22bec17c2a2a949677f0d54b0348f9300d4..d2b7a61e83ab5faa8ef412d1e4c9b638facfcf36 100644 --- a/source/dnode/mnode/impl/src/mndSubscribe.c +++ b/source/dnode/mnode/impl/src/mndSubscribe.c @@ -403,7 +403,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOutputObj *pOutput) { STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pMsg); - mndTransSetDbName(pTrans, pOutput->pSub->dbName); + mndTransSetDbName(pTrans, pOutput->pSub->dbName, NULL); if (pTrans == NULL) return -1; // make txn: diff --git a/source/dnode/mnode/impl/src/mndSync.c b/source/dnode/mnode/impl/src/mndSync.c index 5d28181f00454a6febfad05e9615ea2c14177cce..07f65b2a90a3f70498568091941fd76be6e644f8 100644 --- a/source/dnode/mnode/impl/src/mndSync.c +++ b/source/dnode/mnode/impl/src/mndSync.c @@ -46,19 +46,21 @@ void mndSyncCommitMsg(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbM int32_t transId = sdbGetIdFromRaw(pMnode->pSdb, pRaw); pMgmt->errCode = cbMeta.code; - mDebug("trans:%d, is proposed, saved:%d code:0x%x, index:%" PRId64 " term:%" PRId64 " role:%s raw:%p", transId, - pMgmt->transId, cbMeta.code, cbMeta.index, cbMeta.term, syncStr(cbMeta.state), pRaw); + mDebug("trans:%d, is proposed, saved:%d code:0x%x, apply index:%" PRId64 " term:%" PRIu64 " config:%" PRId64 + " role:%s raw:%p", + transId, pMgmt->transId, cbMeta.code, cbMeta.index, cbMeta.term, cbMeta.lastConfigIndex, syncStr(cbMeta.state), + pRaw); if (pMgmt->errCode == 0) { sdbWriteWithoutFree(pMnode->pSdb, pRaw); - sdbSetApplyIndex(pMnode->pSdb, cbMeta.index); - sdbSetApplyTerm(pMnode->pSdb, cbMeta.term); + sdbSetApplyInfo(pMnode->pSdb, cbMeta.index, cbMeta.term, cbMeta.lastConfigIndex); } if (pMgmt->transId == transId) { if (pMgmt->errCode != 0) { mError("trans:%d, failed to propose since %s", transId, tstrerror(pMgmt->errCode)); } + pMgmt->transId = 0; tsem_post(&pMgmt->syncSem); } else { STrans *pTrans = mndAcquireTrans(pMnode, transId); @@ -66,33 +68,31 @@ void mndSyncCommitMsg(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbM mndTransExecute(pMnode, pTrans); mndReleaseTrans(pMnode, pTrans); } - - if (cbMeta.index - sdbGetApplyIndex(pMnode->pSdb) > 100) { - SSnapshotMeta sMeta = {0}; - if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) { - sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex); - } - sdbWriteFile(pMnode->pSdb); - } +#if 0 + sdbWriteFile(pMnode->pSdb, SDB_WRITE_DELTA); +#endif } } -int32_t mndSyncGetSnapshot(struct SSyncFSM *pFsm, SSnapshot *pSnapshot) { +int32_t mndSyncGetSnapshot(struct SSyncFSM *pFsm, SSnapshot *pSnapshot, void *pReaderParam, void **ppReader) { + // TODO: + + // atomic operation + // step1. sdbGetCommitInfo + // step2. create ppReader with pReaderParam + + return 0; +} + +int32_t mndSyncGetSnapshotInfo(struct SSyncFSM *pFsm, SSnapshot *pSnapshot) { SMnode *pMnode = pFsm->data; - pSnapshot->lastApplyIndex = sdbGetCommitIndex(pMnode->pSdb); - pSnapshot->lastApplyTerm = sdbGetCommitTerm(pMnode->pSdb); - pSnapshot->lastConfigIndex = sdbGetCurConfig(pMnode->pSdb); + sdbGetCommitInfo(pMnode->pSdb, &pSnapshot->lastApplyIndex, &pSnapshot->lastApplyTerm, &pSnapshot->lastConfigIndex); return 0; } void mndRestoreFinish(struct SSyncFSM *pFsm) { SMnode *pMnode = pFsm->data; - SSnapshotMeta sMeta = {0}; - if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) { - sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex); - } - if (!pMnode->deploy) { mInfo("mnode sync restore finished, and will handle outstanding transactions"); mndTransPullup(pMnode); @@ -122,6 +122,7 @@ void mndReConfig(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SReConfigCbMeta cbM if (pMgmt->errCode != 0) { mError("trans:-1, failed to propose sync reconfig since %s", tstrerror(pMgmt->errCode)); } + pMgmt->transId = 0; tsem_post(&pMgmt->syncSem); } } @@ -169,6 +170,7 @@ SSyncFSM *mndSyncMakeFsm(SMnode *pMnode) { pFsm->FpRestoreFinishCb = mndRestoreFinish; pFsm->FpReConfigCb = mndReConfig; pFsm->FpGetSnapshot = mndSyncGetSnapshot; + pFsm->FpGetSnapshotInfo = mndSyncGetSnapshotInfo; pFsm->FpSnapshotStartRead = mndSnapshotStartRead; pFsm->FpSnapshotStopRead = mndSnapshotStopRead; pFsm->FpSnapshotDoRead = mndSnapshotDoRead; @@ -258,13 +260,17 @@ void mndSyncStart(SMnode *pMnode) { mDebug("mnode sync started, id:%" PRId64 " standby:%d", pMgmt->sync, pMgmt->standby); } -void mndSyncStop(SMnode *pMnode) {} +void mndSyncStop(SMnode *pMnode) { + if (pMnode->syncMgmt.transId != 0) { + pMnode->syncMgmt.transId = 0; + tsem_post(&pMnode->syncMgmt.syncSem); + } +} bool mndIsMaster(SMnode *pMnode) { SSyncMgmt *pMgmt = &pMnode->syncMgmt; - ESyncState state = syncGetMyRole(pMgmt->sync); - if (state != TAOS_SYNC_STATE_LEADER) { + if (!syncIsReady(pMgmt->sync)) { terrno = TSDB_CODE_SYN_NOT_LEADER; return false; } diff --git a/source/dnode/mnode/impl/src/mndTopic.c b/source/dnode/mnode/impl/src/mndTopic.c index 4c2730ce9432bbc0827d23da7b9b58df8938580f..9632c04f4c4166d62ffaad64b8af572e3983aeec 100644 --- a/source/dnode/mnode/impl/src/mndTopic.c +++ b/source/dnode/mnode/impl/src/mndTopic.c @@ -474,7 +474,7 @@ static int32_t mndProcessCreateTopicReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckDbAuth(pMnode, pReq->conn.user, MND_OPER_WRITE_DB, pDb) != 0) { + if (mndCheckDbAuth(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) { goto _OVER; } @@ -566,7 +566,7 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) { #endif STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pReq); - mndTransSetDbName(pTrans, pTopic->db); + mndTransSetDbName(pTrans, pTopic->db, NULL); if (pTrans == NULL) { mError("topic:%s, failed to drop since %s", pTopic->name, terrstr()); return -1; diff --git a/source/dnode/mnode/impl/src/mndTrans.c b/source/dnode/mnode/impl/src/mndTrans.c index dafb2fa1d1151eff10d29af0d41a1160218139b9..de6a44d456c992defcdbae39d97b260b471e43a6 100644 --- a/source/dnode/mnode/impl/src/mndTrans.c +++ b/source/dnode/mnode/impl/src/mndTrans.c @@ -22,8 +22,8 @@ #include "mndSync.h" #include "mndUser.h" -#define TRANS_VER_NUMBER 1 -#define TRANS_ARRAY_SIZE 8 +#define TRANS_VER_NUMBER 1 +#define TRANS_ARRAY_SIZE 8 #define TRANS_RESERVE_SIZE 64 static SSdbRaw *mndTransActionEncode(STrans *pTrans); @@ -52,7 +52,7 @@ static bool mndTransPerformCommitActionStage(SMnode *pMnode, STrans *pTrans); static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans); static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans); static bool mndTransPerfromFinishedStage(SMnode *pMnode, STrans *pTrans); -static bool mndCantExecuteTransAction(SMnode *pMnode) { return !pMnode->deploy && !mndIsMaster(pMnode); } +static bool mndCannotExecuteTransAction(SMnode *pMnode) { return !pMnode->deploy && !mndIsMaster(pMnode); } static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans); static int32_t mndProcessTransReq(SRpcMsg *pReq); @@ -122,7 +122,8 @@ static SSdbRaw *mndTransActionEncode(STrans *pTrans) { SDB_SET_INT8(pRaw, dataPos, pTrans->conflict, _OVER) SDB_SET_INT8(pRaw, dataPos, pTrans->exec, _OVER) SDB_SET_INT64(pRaw, dataPos, pTrans->createdTime, _OVER) - SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname, TSDB_DB_FNAME_LEN, _OVER) + SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER) + SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER) SDB_SET_INT32(pRaw, dataPos, pTrans->redoActionPos, _OVER) int32_t redoActionNum = taosArrayGetSize(pTrans->redoActions); @@ -270,7 +271,8 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) { pTrans->conflict = conflict; pTrans->exec = exec; SDB_GET_INT64(pRaw, dataPos, &pTrans->createdTime, _OVER) - SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname, TSDB_DB_FNAME_LEN, _OVER) + SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER) + SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER) SDB_GET_INT32(pRaw, dataPos, &pTrans->redoActionPos, _OVER) SDB_GET_INT32(pRaw, dataPos, &redoActionNum, _OVER) SDB_GET_INT32(pRaw, dataPos, &undoActionNum, _OVER) @@ -521,9 +523,10 @@ static int32_t mndTransActionUpdate(SSdb *pSdb, STrans *pOld, STrans *pNew) { } if (pOld->stage == TRN_STAGE_ROLLBACK) { - pOld->stage = TRN_STAGE_FINISHED; - mTrace("trans:%d, stage from rollback to finished since perform update action", pNew->id); + pOld->stage = TRN_STAGE_REDO_ACTION; + mTrace("trans:%d, stage from rollback to undoAction since perform update action", pNew->id); } + return 0; } @@ -649,7 +652,14 @@ void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void * pTrans->paramLen = paramLen; } -void mndTransSetDbName(STrans *pTrans, const char *dbname) { memcpy(pTrans->dbname, dbname, TSDB_DB_FNAME_LEN); } +void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2) { + if (dbname1 != NULL) { + memcpy(pTrans->dbname1, dbname1, TSDB_DB_FNAME_LEN); + } + if (dbname2 != NULL) { + memcpy(pTrans->dbname2, dbname2, TSDB_DB_FNAME_LEN); + } +} void mndTransSetSerial(STrans *pTrans) { pTrans->exec = TRN_EXEC_SERIAL; } @@ -661,7 +671,7 @@ static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) { } sdbSetRawStatus(pRaw, SDB_STATUS_READY); - mDebug("trans:%d, sync to other mnodes", pTrans->id); + mDebug("trans:%d, sync to other mnodes, stage:%s", pTrans->id, mndTransStr(pTrans->stage)); int32_t code = mndSyncPropose(pMnode, pRaw, pTrans->id); if (code != 0) { mError("trans:%d, failed to sync since %s", pTrans->id, terrstr()); @@ -674,6 +684,12 @@ static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) { return 0; } +static bool mndCheckDbConflict(const char *db, STrans *pTrans) { + if (db[0] == 0) return false; + if (strcmp(db, pTrans->dbname1) == 0 || strcmp(db, pTrans->dbname2) == 0) return true; + return false; +} + static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) { STrans *pTrans = NULL; void *pIter = NULL; @@ -688,14 +704,21 @@ static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) { if (pNew->conflict == TRN_CONFLICT_GLOBAL) conflict = true; if (pNew->conflict == TRN_CONFLICT_DB) { if (pTrans->conflict == TRN_CONFLICT_GLOBAL) conflict = true; - if (pTrans->conflict == TRN_CONFLICT_DB && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true; - if (pTrans->conflict == TRN_CONFLICT_DB_INSIDE && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true; + if (pTrans->conflict == TRN_CONFLICT_DB || pTrans->conflict == TRN_CONFLICT_DB_INSIDE) { + if (mndCheckDbConflict(pNew->dbname1, pTrans)) conflict = true; + if (mndCheckDbConflict(pNew->dbname2, pTrans)) conflict = true; + } } if (pNew->conflict == TRN_CONFLICT_DB_INSIDE) { if (pTrans->conflict == TRN_CONFLICT_GLOBAL) conflict = true; - if (pTrans->conflict == TRN_CONFLICT_DB && strcmp(pNew->dbname, pTrans->dbname) == 0) conflict = true; + if (pTrans->conflict == TRN_CONFLICT_DB) { + if (mndCheckDbConflict(pNew->dbname1, pTrans)) conflict = true; + if (mndCheckDbConflict(pNew->dbname2, pTrans)) conflict = true; + } } - mError("trans:%d, can't execute since conflict with trans:%d, db:%s", pNew->id, pTrans->id, pTrans->dbname); + + mError("trans:%d, can't execute since conflict with trans:%d, db1:%s db2:%s", pNew->id, pTrans->id, pTrans->dbname1, + pTrans->dbname2); sdbRelease(pMnode->pSdb, pTrans); } @@ -704,7 +727,7 @@ static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) { int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans) { if (pTrans->conflict == TRN_CONFLICT_DB || pTrans->conflict == TRN_CONFLICT_DB_INSIDE) { - if (strlen(pTrans->dbname) == 0) { + if (strlen(pTrans->dbname1) == 0 && strlen(pTrans->dbname2) == 0) { terrno = TSDB_CODE_MND_TRANS_CONFLICT; mError("trans:%d, failed to prepare conflict db not set", pTrans->id); return -1; @@ -781,7 +804,7 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) { sendRsp = true; } } else { - if (pTrans->stage == TRN_STAGE_REDO_ACTION && pTrans->failedTimes > 2) { + if (pTrans->stage == TRN_STAGE_REDO_ACTION && pTrans->failedTimes > 3) { if (code == 0) code = TSDB_CODE_MND_TRANS_UNKNOW_ERROR; sendRsp = true; } @@ -873,7 +896,7 @@ static void mndTransResetActions(SMnode *pMnode, STrans *pTrans, SArray *pArray) pAction->rawWritten = 0; pAction->msgSent = 0; pAction->msgReceived = 0; - if (pAction->errCode == TSDB_CODE_RPC_REDIRECT || pAction->errCode == TSDB_CODE_SYN_NOT_IN_NEW_CONFIG || + if (pAction->errCode == TSDB_CODE_RPC_REDIRECT || pAction->errCode == TSDB_CODE_SYN_NEW_CONFIG_ERROR || pAction->errCode == TSDB_CODE_SYN_INTERNAL_ERROR || pAction->errCode == TSDB_CODE_SYN_NOT_LEADER) { pAction->epSet.inUse = (pAction->epSet.inUse + 1) % pAction->epSet.numOfEps; mDebug("trans:%d, %s:%d execute status is reset and set epset inuse:%d", pTrans->id, mndTransStr(pAction->stage), @@ -915,7 +938,7 @@ static int32_t mndTransWriteSingleLog(SMnode *pMnode, STrans *pTrans, STransActi static int32_t mndTransSendSingleMsg(SMnode *pMnode, STrans *pTrans, STransAction *pAction) { if (pAction->msgSent) return 0; - if (mndCantExecuteTransAction(pMnode)) return -1; + if (mndCannotExecuteTransAction(pMnode)) return -1; int64_t signature = pTrans->id; signature = (signature << 32); @@ -1104,6 +1127,7 @@ static int32_t mndTransExecuteRedoActionsSerial(SMnode *pMnode, STrans *pTrans) } if (code == 0) { + pTrans->failedTimes = 0; pTrans->lastAction = action; pTrans->lastMsgType = 0; pTrans->lastErrorNo = 0; @@ -1115,7 +1139,7 @@ static int32_t mndTransExecuteRedoActionsSerial(SMnode *pMnode, STrans *pTrans) pTrans->lastEpset = pAction->epSet; } - if (mndCantExecuteTransAction(pMnode)) break; + if (mndCannotExecuteTransAction(pMnode)) break; if (code == 0) { pTrans->code = 0; @@ -1158,7 +1182,7 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) { code = mndTransExecuteRedoActions(pMnode, pTrans); } - if (mndCantExecuteTransAction(pMnode)) return false; + if (mndCannotExecuteTransAction(pMnode)) return false; if (code == 0) { pTrans->code = 0; @@ -1171,8 +1195,8 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) { } else { pTrans->code = terrno; if (pTrans->policy == TRN_POLICY_ROLLBACK) { - pTrans->stage = TRN_STAGE_UNDO_ACTION; - mError("trans:%d, stage from redoAction to undoAction since %s", pTrans->id, terrstr()); + pTrans->stage = TRN_STAGE_ROLLBACK; + mError("trans:%d, stage from redoAction to rollback since %s", pTrans->id, terrstr()); continueExec = true; } else { pTrans->failedTimes++; @@ -1185,7 +1209,7 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans) { } static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans) { - if (mndCantExecuteTransAction(pMnode)) return false; + if (mndCannotExecuteTransAction(pMnode)) return false; bool continueExec = true; int32_t code = mndTransCommit(pMnode, pTrans); @@ -1197,16 +1221,9 @@ static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans) { continueExec = true; } else { pTrans->code = terrno; - if (pTrans->policy == TRN_POLICY_ROLLBACK) { - pTrans->stage = TRN_STAGE_UNDO_ACTION; - mError("trans:%d, stage from commit to undoAction since %s, failedTimes:%d", pTrans->id, terrstr(), - pTrans->failedTimes); - continueExec = true; - } else { - pTrans->failedTimes++; - mError("trans:%d, stage keep on commit since %s, failedTimes:%d", pTrans->id, terrstr(), pTrans->failedTimes); - continueExec = false; - } + pTrans->failedTimes++; + mError("trans:%d, stage keep on commit since %s, failedTimes:%d", pTrans->id, terrstr(), pTrans->failedTimes); + continueExec = false; } return continueExec; @@ -1235,11 +1252,9 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans) { bool continueExec = true; int32_t code = mndTransExecuteUndoActions(pMnode, pTrans); - if (mndCantExecuteTransAction(pMnode)) return false; - if (code == 0) { - pTrans->stage = TRN_STAGE_ROLLBACK; - mDebug("trans:%d, stage from undoAction to rollback", pTrans->id); + pTrans->stage = TRN_STAGE_FINISHED; + mDebug("trans:%d, stage from undoAction to finished", pTrans->id); continueExec = true; } else if (code == TSDB_CODE_ACTION_IN_PROGRESS) { mDebug("trans:%d, stage keep on undoAction since %s", pTrans->id, tstrerror(code)); @@ -1254,14 +1269,14 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans) { } static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans) { - if (mndCantExecuteTransAction(pMnode)) return false; + if (mndCannotExecuteTransAction(pMnode)) return false; bool continueExec = true; int32_t code = mndTransRollback(pMnode, pTrans); if (code == 0) { - pTrans->stage = TRN_STAGE_FINISHED; - mDebug("trans:%d, stage from rollback to finished", pTrans->id); + pTrans->stage = TRN_STAGE_UNDO_ACTION; + mDebug("trans:%d, stage from rollback to undoAction", pTrans->id); continueExec = true; } else { pTrans->failedTimes++; @@ -1309,12 +1324,12 @@ void mndTransExecute(SMnode *pMnode, STrans *pTrans) { case TRN_STAGE_COMMIT_ACTION: continueExec = mndTransPerformCommitActionStage(pMnode, pTrans); break; - case TRN_STAGE_UNDO_ACTION: - continueExec = mndTransPerformUndoActionStage(pMnode, pTrans); - break; case TRN_STAGE_ROLLBACK: continueExec = mndTransPerformRollbackStage(pMnode, pTrans); break; + case TRN_STAGE_UNDO_ACTION: + continueExec = mndTransPerformUndoActionStage(pMnode, pTrans); + break; case TRN_STAGE_FINISHED: continueExec = mndTransPerfromFinishedStage(pMnode, pTrans); break; @@ -1369,7 +1384,7 @@ static int32_t mndProcessKillTransReq(SRpcMsg *pReq) { mInfo("trans:%d, start to kill", killReq.transId); - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_KILL_TRANS) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_KILL_TRANS) != 0) { goto _OVER; } @@ -1416,11 +1431,7 @@ void mndTransPullup(SMnode *pMnode) { mndReleaseTrans(pMnode, pTrans); } - SSnapshotMeta sMeta = {0}; - if (syncGetSnapshotMeta(pMnode->syncMgmt.sync, &sMeta) == 0) { - sdbSetCurConfig(pMnode->pSdb, sMeta.lastConfigIndex); - } - sdbWriteFile(pMnode->pSdb); + sdbWriteFile(pMnode->pSdb, SDB_WRITE_DELTA); taosArrayDestroy(pArray); } @@ -1449,10 +1460,15 @@ static int32_t mndRetrieveTrans(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBl pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, (const char *)stage, false); - char dbname[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0}; - STR_WITH_MAXSIZE_TO_VARSTR(dbname, mndGetDbStr(pTrans->dbname), pShow->pMeta->pSchemas[cols].bytes); + char dbname1[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0}; + STR_WITH_MAXSIZE_TO_VARSTR(dbname1, mndGetDbStr(pTrans->dbname1), pShow->pMeta->pSchemas[cols].bytes); + pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); + colDataAppend(pColInfo, numOfRows, (const char *)dbname1, false); + + char dbname2[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0}; + STR_WITH_MAXSIZE_TO_VARSTR(dbname2, mndGetDbStr(pTrans->dbname2), pShow->pMeta->pSchemas[cols].bytes); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); - colDataAppend(pColInfo, numOfRows, (const char *)dbname, false); + colDataAppend(pColInfo, numOfRows, (const char *)dbname2, false); pColInfo = taosArrayGet(pBlock->pDataBlock, cols++); colDataAppend(pColInfo, numOfRows, (const char *)&pTrans->failedTimes, false); diff --git a/source/dnode/mnode/impl/src/mndUser.c b/source/dnode/mnode/impl/src/mndUser.c index 9590823106124ecd8595052d31a608adf1226a46..eb0a818a604da23b995c0493b2e96281d5077e53 100644 --- a/source/dnode/mnode/impl/src/mndUser.c +++ b/source/dnode/mnode/impl/src/mndUser.c @@ -354,13 +354,13 @@ static int32_t mndProcessCreateUserReq(SRpcMsg *pReq) { goto _OVER; } - pOperUser = mndAcquireUser(pMnode, pReq->conn.user); + pOperUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pOperUser == NULL) { terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_CREATE_USER) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_CREATE_USER) != 0) { goto _OVER; } @@ -460,7 +460,7 @@ static int32_t mndProcessAlterUserReq(SRpcMsg *pReq) { goto _OVER; } - pOperUser = mndAcquireUser(pMnode, pReq->conn.user); + pOperUser = mndAcquireUser(pMnode, pReq->info.conn.user); if (pOperUser == NULL) { terrno = TSDB_CODE_MND_NO_USER_FROM_CONN; goto _OVER; @@ -643,7 +643,7 @@ static int32_t mndProcessDropUserReq(SRpcMsg *pReq) { goto _OVER; } - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_DROP_USER) != 0) { + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_DROP_USER) != 0) { goto _OVER; } diff --git a/source/dnode/mnode/impl/src/mndVgroup.c b/source/dnode/mnode/impl/src/mndVgroup.c index 755f4ef0b2afd3b63c634b94ef369eb8ed18d1f9..94ddbcd409c8ef4b1ad6bf8d97bda63edb63ea92 100644 --- a/source/dnode/mnode/impl/src/mndVgroup.c +++ b/source/dnode/mnode/impl/src/mndVgroup.c @@ -1058,7 +1058,7 @@ int32_t mndSetMoveVgroupsInfoToTrans(SMnode *pMnode, STrans *pTrans, int32_t del static int32_t mndAddIncVgroupReplicaToTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup, int32_t newDnodeId) { - mDebug("vgId:%d, will add 1 vnode, replica:%d, dnode:%d", pVgroup->vgId, pVgroup->replica, newDnodeId); + mDebug("vgId:%d, will add 1 vnode, replica:%d dnode:%d", pVgroup->vgId, pVgroup->replica, newDnodeId); SVnodeGid *pGid = &pVgroup->vnodeGid[pVgroup->replica]; pVgroup->replica++; @@ -1074,7 +1074,7 @@ static int32_t mndAddIncVgroupReplicaToTrans(SMnode *pMnode, STrans *pTrans, SDb static int32_t mndAddDecVgroupReplicaFromTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup, int32_t delDnodeId) { - mDebug("vgId:%d, will remove 1 vnode, replica:%d, dnode:%d", pVgroup->vgId, pVgroup->replica, delDnodeId); + mDebug("vgId:%d, will remove 1 vnode, replica:%d dnode:%d", pVgroup->vgId, pVgroup->replica, delDnodeId); SVnodeGid *pGid = NULL; SVnodeGid delGid = {0}; @@ -1116,7 +1116,8 @@ static int32_t mndRedistributeVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, memcpy(&newVg, pVgroup, sizeof(SVgObj)); mInfo("vgId:%d, vgroup info before redistribute, replica:%d", newVg.vgId, newVg.replica); for (int32_t i = 0; i < newVg.replica; ++i) { - mInfo("vgId:%d, vnode:%d dnode:%d", newVg.vgId, i, newVg.vnodeGid[i].dnodeId); + mInfo("vgId:%d, vnode:%d dnode:%d role:%s", newVg.vgId, i, newVg.vnodeGid[i].dnodeId, + syncStr(newVg.vnodeGid[i].role)); } if (pNew1 != NULL && pOld1 != NULL) { @@ -1198,9 +1199,9 @@ static int32_t mndProcessRedistributeVgroupMsg(SRpcMsg *pReq) { goto _OVER; } - mInfo("vgId:%d, start to redistribute to dnode %d:%d:%d", req.vgId, req.dnodeId1, req.dnodeId2, req.dnodeId3); + mInfo("vgId:%d, start to redistribute vgroup to dnode %d:%d:%d", req.vgId, req.dnodeId1, req.dnodeId2, req.dnodeId3); - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_REDISTRIBUTE_VGROUP) != 0) goto _OVER; + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_REDISTRIBUTE_VGROUP) != 0) goto _OVER; pVgroup = mndAcquireVgroup(pMnode, req.vgId); if (pVgroup == NULL) goto _OVER; @@ -1500,7 +1501,7 @@ static int32_t mndProcessSplitVgroupMsg(SRpcMsg *pReq) { pDb = mndAcquireDb(pMnode, pVgroup->dbName); if (pDb == NULL) goto _OVER; - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_SPLIT_VGROUP) != 0) goto _OVER; + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_SPLIT_VGROUP) != 0) goto _OVER; code = mndSplitVgroup(pMnode, pReq, pDb, pVgroup); if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS; @@ -1624,7 +1625,7 @@ static int32_t mndProcessBalanceVgroupMsg(SRpcMsg *pReq) { mDebug("start to balance vgroup"); - if (mndCheckOperAuth(pMnode, pReq->conn.user, MND_OPER_BALANCE_VGROUP) != 0) goto _OVER; + if (mndCheckOperAuth(pMnode, pReq->info.conn.user, MND_OPER_BALANCE_VGROUP) != 0) goto _OVER; while (1) { SDnodeObj *pDnode = NULL; diff --git a/source/dnode/mnode/impl/test/profile/profile.cpp b/source/dnode/mnode/impl/test/profile/profile.cpp index 039700ae9e5d8bfeab1eadb39aa83e4b38827445..e784a41d6f08ac98a5a12710a012e85efd43a2fc 100644 --- a/source/dnode/mnode/impl/test/profile/profile.cpp +++ b/source/dnode/mnode/impl/test/profile/profile.cpp @@ -295,8 +295,7 @@ TEST_F(MndTestProfile, 07_KillQueryMsg) { TEST_F(MndTestProfile, 08_KillQueryMsg_InvalidConn) { SKillQueryReq killReq = {0}; - killReq.connId = 2345; - killReq.queryId = 2345; + strcpy(killReq.queryStrId, "2345:2345"); int32_t contLen = tSerializeSKillQueryReq(NULL, 0, &killReq); void* pReq = rpcMallocCont(contLen); diff --git a/source/dnode/mnode/impl/test/sdb/sdbTest.cpp b/source/dnode/mnode/impl/test/sdb/sdbTest.cpp index 43be55dd1de822d098475747a7b5b6452f379058..bc118ee26e5996e28e6ef3c1380a71a7f868dd3f 100644 --- a/source/dnode/mnode/impl/test/sdb/sdbTest.cpp +++ b/source/dnode/mnode/impl/test/sdb/sdbTest.cpp @@ -493,8 +493,11 @@ TEST_F(MndTestSdb, 01_Write_Str) { ASSERT_EQ(sdbGetSize(pSdb, SDB_USER), 2); ASSERT_EQ(sdbGetMaxId(pSdb, SDB_USER), -1); ASSERT_EQ(sdbGetTableVer(pSdb, SDB_USER), 2); - sdbSetApplyIndex(pSdb, -1); - ASSERT_EQ(sdbGetApplyIndex(pSdb), -1); + sdbSetApplyInfo(pSdb, -1, -1, -1); + // int64_t index, config; + // int64_t term; + // sdbGetCommitInfo(pSdb, &index, &term, &config); + // ASSERT_EQ(index, -1); ASSERT_EQ(mnode.insertTimes, 2); ASSERT_EQ(mnode.deleteTimes, 0); @@ -700,11 +703,12 @@ TEST_F(MndTestSdb, 01_Write_Str) { } // write version - sdbSetApplyIndex(pSdb, 0); - sdbSetApplyIndex(pSdb, 1); - ASSERT_EQ(sdbGetApplyIndex(pSdb), 1); - ASSERT_EQ(sdbWriteFile(pSdb), 0); - ASSERT_EQ(sdbWriteFile(pSdb), 0); + sdbSetApplyInfo(pSdb, 0, 0, 0); + sdbSetApplyInfo(pSdb, 1, 0, 0); + // sdbGetApplyInfo(pSdb, &index, &term, &config); + // ASSERT_EQ(index, 1); + ASSERT_EQ(sdbWriteFile(pSdb, 0), 0); + ASSERT_EQ(sdbWriteFile(pSdb, 0), 0); sdbCleanup(pSdb); ASSERT_EQ(mnode.insertTimes, 7); @@ -772,7 +776,11 @@ TEST_F(MndTestSdb, 01_Read_Str) { ASSERT_EQ(sdbGetSize(pSdb, SDB_USER), 2); ASSERT_EQ(sdbGetMaxId(pSdb, SDB_USER), -1); ASSERT_EQ(sdbGetTableVer(pSdb, SDB_USER), 5); - ASSERT_EQ(sdbGetApplyIndex(pSdb), 1); + + int64_t index, config; + int64_t term; + sdbGetCommitInfo(pSdb, &index, &term, &config); + ASSERT_EQ(index, 1); ASSERT_EQ(mnode.insertTimes, 4); ASSERT_EQ(mnode.deleteTimes, 0); diff --git a/source/dnode/mnode/impl/test/trans/trans2.cpp b/source/dnode/mnode/impl/test/trans/trans2.cpp index 022c82c73d66ab39f9cf07aeb34642278018722d..aee8aa27488da6dc6b8b0cbd06a3b34741e66a18 100644 --- a/source/dnode/mnode/impl/test/trans/trans2.cpp +++ b/source/dnode/mnode/impl/test/trans/trans2.cpp @@ -128,7 +128,7 @@ class MndTestTrans2 : public ::testing::Test { mndTransSetCb(pTrans, TRANS_START_FUNC_TEST, TRANS_STOP_FUNC_TEST, param, strlen(param) + 1); if (pDb != NULL) { - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); } int32_t code = mndTransPrepare(pMnode, pTrans); @@ -201,7 +201,7 @@ class MndTestTrans2 : public ::testing::Test { } if (pDb != NULL) { - mndTransSetDbName(pTrans, pDb->name); + mndTransSetDbName(pTrans, pDb->name, NULL); } int32_t code = mndTransPrepare(pMnode, pTrans); diff --git a/source/dnode/mnode/sdb/inc/sdb.h b/source/dnode/mnode/sdb/inc/sdb.h index 8536c451b7d9c33bfc9fc9abfbe499af60718cde..3b1c4000a83957ea5c7360e40857e9fd4aee8aa2 100644 --- a/source/dnode/mnode/sdb/inc/sdb.h +++ b/source/dnode/mnode/sdb/inc/sdb.h @@ -37,6 +37,8 @@ extern "C" { #define mTrace(...) { if (mDebugFlag & DEBUG_TRACE) { taosPrintLog("MND ", DEBUG_TRACE, mDebugFlag, __VA_ARGS__); }} // clang-format on +#define SDB_WRITE_DELTA 20 + #define SDB_GET_VAL(pData, dataPos, val, pos, func, type) \ { \ if (func(pRaw, dataPos, val) != 0) { \ @@ -169,11 +171,12 @@ typedef struct SSdb { SWal *pWal; char *currDir; char *tmpDir; - int64_t lastCommitVer; - int64_t lastCommitTerm; - int64_t curVer; - int64_t curTerm; - int64_t curConfig; + int64_t commitIndex; + int64_t commitTerm; + int64_t commitConfig; + int64_t applyIndex; + int64_t applyTerm; + int64_t applyConfig; int64_t tableVer[SDB_MAX]; int64_t maxId[SDB_MAX]; EKeyType keyTypes[SDB_MAX]; @@ -257,7 +260,7 @@ int32_t sdbReadFile(SSdb *pSdb); * @param pSdb The sdb object. * @return int32_t 0 for success, -1 for failure. */ -int32_t sdbWriteFile(SSdb *pSdb); +int32_t sdbWriteFile(SSdb *pSdb, int32_t delta); /** * @brief Parse and write raw data to sdb, then free the pRaw object @@ -361,14 +364,8 @@ int64_t sdbGetTableVer(SSdb *pSdb, ESdbType type); * @param index The update value of the apply index. * @return int32_t The current index of sdb */ -void sdbSetApplyIndex(SSdb *pSdb, int64_t index); -void sdbSetApplyTerm(SSdb *pSdb, int64_t term); -void sdbSetCurConfig(SSdb *pSdb, int64_t config); -int64_t sdbGetApplyIndex(SSdb *pSdb); -int64_t sdbGetApplyTerm(SSdb *pSdb); -int64_t sdbGetCommitIndex(SSdb *pSdb); -int64_t sdbGetCommitTerm(SSdb *pSdb); -int64_t sdbGetCurConfig(SSdb *pSdb); +void sdbSetApplyInfo(SSdb *pSdb, int64_t index, int64_t term, int64_t config); +void sdbGetCommitInfo(SSdb *pSdb, int64_t *index, int64_t *term, int64_t *config); SSdbRaw *sdbAllocRaw(ESdbType type, int8_t sver, int32_t dataLen); void sdbFreeRaw(SSdbRaw *pRaw); diff --git a/source/dnode/mnode/sdb/src/sdb.c b/source/dnode/mnode/sdb/src/sdb.c index 39e9c7588833c0185129ef0dea51f8bc8d665df4..fbf66da63277895196555a4185fffb41c2dd2cfb 100644 --- a/source/dnode/mnode/sdb/src/sdb.c +++ b/source/dnode/mnode/sdb/src/sdb.c @@ -53,11 +53,12 @@ SSdb *sdbInit(SSdbOpt *pOption) { } pSdb->pWal = pOption->pWal; - pSdb->curVer = -1; - pSdb->curTerm = -1; - pSdb->lastCommitVer = -1; - pSdb->lastCommitTerm = -1; - pSdb->curConfig = -1; + pSdb->applyIndex = -1; + pSdb->applyTerm = -1; + pSdb->applyConfig = -1; + pSdb->commitIndex = -1; + pSdb->commitTerm = -1; + pSdb->commitConfig = -1; pSdb->pMnode = pOption->pMnode; taosThreadMutexInit(&pSdb->filelock, NULL); mDebug("sdb init successfully"); @@ -67,7 +68,7 @@ SSdb *sdbInit(SSdbOpt *pOption) { void sdbCleanup(SSdb *pSdb) { mDebug("start to cleanup sdb"); - sdbWriteFile(pSdb); + sdbWriteFile(pSdb, 0); if (pSdb->currDir != NULL) { taosMemoryFreeClear(pSdb->currDir); @@ -159,23 +160,24 @@ static int32_t sdbCreateDir(SSdb *pSdb) { return 0; } -void sdbSetApplyIndex(SSdb *pSdb, int64_t index) { pSdb->curVer = index; } - -void sdbSetApplyTerm(SSdb *pSdb, int64_t term) { pSdb->curTerm = term; } - -void sdbSetCurConfig(SSdb *pSdb, int64_t config) { - if (pSdb->curConfig != config) { - mDebug("mnode sync config set from %" PRId64 " to %" PRId64, pSdb->curConfig, config); - pSdb->curConfig = config; - } +void sdbSetApplyInfo(SSdb *pSdb, int64_t index, int64_t term, int64_t config) { +#if 1 + mTrace("mnode apply info changed from index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " to index:%" PRId64 + " term:%" PRId64 " config:%" PRId64, + pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, index, term, config); +#endif + pSdb->applyIndex = index; + pSdb->applyTerm = term; + pSdb->applyConfig = config; } -int64_t sdbGetApplyIndex(SSdb *pSdb) { return pSdb->curVer; } - -int64_t sdbGetApplyTerm(SSdb *pSdb) { return pSdb->curTerm; } - -int64_t sdbGetCommitIndex(SSdb *pSdb) { return pSdb->lastCommitVer; } - -int64_t sdbGetCommitTerm(SSdb *pSdb) { return pSdb->lastCommitTerm; } - -int64_t sdbGetCurConfig(SSdb *pSdb) { return pSdb->curConfig; } \ No newline at end of file +void sdbGetCommitInfo(SSdb *pSdb, int64_t *index, int64_t *term, int64_t *config) { + *index = pSdb->commitIndex; + *term = pSdb->commitTerm; + *config = pSdb->commitConfig; +#if 0 + mTrace("mnode current info, apply index:%" PRId64 " term:%" PRId64 " config:%" PRId64 ", commit index:%" PRId64 + " term:%" PRId64 " config:%" PRId64, + pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, *index, *term, *config); +#endif +} diff --git a/source/dnode/mnode/sdb/src/sdbFile.c b/source/dnode/mnode/sdb/src/sdbFile.c index 34f5d6f23d541710cbc475f433d1c8d5ff8b4c3d..0f4e1276c1612ad0f1707057d9e052eb307fe452 100644 --- a/source/dnode/mnode/sdb/src/sdbFile.c +++ b/source/dnode/mnode/sdb/src/sdbFile.c @@ -67,10 +67,12 @@ static void sdbResetData(SSdb *pSdb) { mDebug("sdb:%s is reset", sdbTableName(i)); } - pSdb->curVer = -1; - pSdb->curTerm = -1; - pSdb->lastCommitVer = -1; - pSdb->lastCommitTerm = -1; + pSdb->applyIndex = -1; + pSdb->applyTerm = -1; + pSdb->applyConfig = -1; + pSdb->commitIndex = -1; + pSdb->commitTerm = -1; + pSdb->commitConfig = -1; mDebug("sdb reset successfully"); } @@ -90,7 +92,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) { return -1; } - ret = taosReadFile(pFile, &pSdb->curVer, sizeof(int64_t)); + ret = taosReadFile(pFile, &pSdb->applyIndex, sizeof(int64_t)); if (ret < 0) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; @@ -100,7 +102,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) { return -1; } - ret = taosReadFile(pFile, &pSdb->curTerm, sizeof(int64_t)); + ret = taosReadFile(pFile, &pSdb->applyTerm, sizeof(int64_t)); if (ret < 0) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; @@ -110,7 +112,7 @@ static int32_t sdbReadFileHead(SSdb *pSdb, TdFilePtr pFile) { return -1; } - ret = taosReadFile(pFile, &pSdb->curConfig, sizeof(int64_t)); + ret = taosReadFile(pFile, &pSdb->applyConfig, sizeof(int64_t)); if (ret < 0) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; @@ -173,17 +175,17 @@ static int32_t sdbWriteFileHead(SSdb *pSdb, TdFilePtr pFile) { return -1; } - if (taosWriteFile(pFile, &pSdb->curVer, sizeof(int64_t)) != sizeof(int64_t)) { + if (taosWriteFile(pFile, &pSdb->applyIndex, sizeof(int64_t)) != sizeof(int64_t)) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; } - if (taosWriteFile(pFile, &pSdb->curTerm, sizeof(int64_t)) != sizeof(int64_t)) { + if (taosWriteFile(pFile, &pSdb->applyTerm, sizeof(int64_t)) != sizeof(int64_t)) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; } - if (taosWriteFile(pFile, &pSdb->curConfig, sizeof(int64_t)) != sizeof(int64_t)) { + if (taosWriteFile(pFile, &pSdb->applyConfig, sizeof(int64_t)) != sizeof(int64_t)) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; } @@ -300,11 +302,12 @@ static int32_t sdbReadFileImp(SSdb *pSdb) { } code = 0; - pSdb->lastCommitVer = pSdb->curVer; - pSdb->lastCommitTerm = pSdb->curTerm; + pSdb->commitIndex = pSdb->applyIndex; + pSdb->commitTerm = pSdb->applyTerm; + pSdb->commitConfig = pSdb->applyConfig; memcpy(pSdb->tableVer, tableVer, sizeof(tableVer)); - mDebug("read sdb file:%s successfully, index:%" PRId64 " term:%" PRId64 " config:%" PRId64, file, pSdb->lastCommitVer, - pSdb->lastCommitTerm, pSdb->curConfig); + mDebug("read sdb file:%s successfully, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64, file, + pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig); _OVER: taosCloseFile(&pFile); @@ -336,9 +339,10 @@ static int32_t sdbWriteFileImp(SSdb *pSdb) { char curfile[PATH_MAX] = {0}; snprintf(curfile, sizeof(curfile), "%s%ssdb.data", pSdb->currDir, TD_DIRSEP); - mDebug("start to write sdb file, current ver:%" PRId64 " term:%" PRId64 ", commit ver:%" PRId64 " term:%" PRId64 - " file:%s", - pSdb->curVer, pSdb->curTerm, pSdb->lastCommitVer, pSdb->lastCommitTerm, curfile); + mDebug("start to write sdb file, apply index:%" PRId64 " term:%" PRId64 " config:%" PRId64 ", commit index:%" PRId64 + " term:%" PRId64 " config:%" PRId64 ", file:%s", + pSdb->applyIndex, pSdb->applyTerm, pSdb->applyConfig, pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig, + curfile); TdFilePtr pFile = taosOpenFile(tmpfile, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_TRUNC); if (pFile == NULL) { @@ -430,25 +434,30 @@ static int32_t sdbWriteFileImp(SSdb *pSdb) { if (code != 0) { mError("failed to write sdb file:%s since %s", curfile, tstrerror(code)); } else { - pSdb->lastCommitVer = pSdb->curVer; - pSdb->lastCommitTerm = pSdb->curTerm; - mDebug("write sdb file successfully, index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s", - pSdb->lastCommitVer, pSdb->lastCommitTerm, pSdb->curConfig, curfile); + pSdb->commitIndex = pSdb->applyIndex; + pSdb->commitTerm = pSdb->applyTerm; + pSdb->commitConfig = pSdb->applyConfig; + mDebug("write sdb file successfully, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s", + pSdb->commitIndex, pSdb->commitTerm, pSdb->commitConfig, curfile); } terrno = code; return code; } -int32_t sdbWriteFile(SSdb *pSdb) { +int32_t sdbWriteFile(SSdb *pSdb, int32_t delta) { int32_t code = 0; - if (pSdb->curVer == pSdb->lastCommitVer) { + if (pSdb->applyIndex == pSdb->commitIndex) { + return 0; + } + + if (pSdb->applyIndex - pSdb->commitIndex < delta) { return 0; } taosThreadMutexLock(&pSdb->filelock); if (pSdb->pWal != NULL) { - code = walBeginSnapshot(pSdb->pWal, pSdb->curVer); + code = walBeginSnapshot(pSdb->pWal, pSdb->applyIndex); } if (code == 0) { code = sdbWriteFileImp(pSdb); @@ -470,7 +479,7 @@ int32_t sdbDeploy(SSdb *pSdb) { return -1; } - if (sdbWriteFile(pSdb) != 0) { + if (sdbWriteFile(pSdb, 0) != 0) { return -1; } @@ -522,9 +531,9 @@ int32_t sdbStartRead(SSdb *pSdb, SSdbIter **ppIter) { snprintf(datafile, sizeof(datafile), "%s%ssdb.data", pSdb->currDir, TD_DIRSEP); taosThreadMutexLock(&pSdb->filelock); - int64_t commitIndex = pSdb->lastCommitVer; - int64_t commitTerm = pSdb->lastCommitTerm; - int64_t curConfig = pSdb->curConfig; + int64_t commitIndex = pSdb->commitIndex; + int64_t commitTerm = pSdb->commitTerm; + int64_t commitConfig = pSdb->commitConfig; if (taosCopyFile(datafile, pIter->name) < 0) { taosThreadMutexUnlock(&pSdb->filelock); terrno = TAOS_SYSTEM_ERROR(errno); @@ -543,8 +552,8 @@ int32_t sdbStartRead(SSdb *pSdb, SSdbIter **ppIter) { } *ppIter = pIter; - mInfo("sdbiter:%p, is created to read snapshot, index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s", pIter, - commitIndex, commitTerm, curConfig, pIter->name); + mInfo("sdbiter:%p, is created to read snapshot, commit index:%" PRId64 " term:%" PRId64 " config:%" PRId64 " file:%s", + pIter, commitIndex, commitTerm, commitConfig, pIter->name); return 0; } diff --git a/source/dnode/snode/inc/sndInt.h b/source/dnode/snode/inc/sndInt.h index 2802537dcd31323a703cd6a4d2222bb8fd9d3d68..8916e2a31cd79c3486c00841979e14fcaf25b2df 100644 --- a/source/dnode/snode/inc/sndInt.h +++ b/source/dnode/snode/inc/sndInt.h @@ -56,7 +56,6 @@ SStreamTask* sndMetaGetTask(SStreamMeta* pMeta, int32_t taskId); int32_t sndMetaRemoveTask(SStreamMeta* pMeta, int32_t taskId); int32_t sndDropTaskOfStream(SStreamMeta* pMeta, int64_t streamId); - int32_t sndStopTaskOfStream(SStreamMeta* pMeta, int64_t streamId); int32_t sndResumeTaskOfStream(SStreamMeta* pMeta, int64_t streamId); diff --git a/source/dnode/snode/src/snode.c b/source/dnode/snode/src/snode.c index cbbe071c5fa386e178273303ae7d2b35bf1b7ad2..84a66c680bb9bbd54114a19ed712b24cb8ebe7a9 100644 --- a/source/dnode/snode/src/snode.c +++ b/source/dnode/snode/src/snode.c @@ -76,45 +76,157 @@ int32_t sndMetaRemoveTask(SStreamMeta *pMeta, int32_t taskId) { return taosHashRemove(pMeta->pHash, &taskId, sizeof(int32_t)); } -static int32_t sndProcessTaskExecReq(SSnode *pSnode, SRpcMsg *pMsg) { - /*SStreamExecMsgHead *pHead = pMsg->pCont;*/ - /*int32_t taskId = pHead->streamTaskId;*/ - /*SStreamTask *pTask = sndMetaGetTask(pSnode->pMeta, taskId);*/ - /*if (pTask == NULL) {*/ - /*return -1;*/ - /*}*/ +static int32_t sndProcessTaskDeployReq(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + char *msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); + int32_t msgLen = pMsg->contLen - sizeof(SMsgHead); + + SStreamTask *pTask = taosMemoryCalloc(1, sizeof(SStreamTask)); + if (pTask == NULL) { + return -1; + } + SDecoder decoder; + tDecoderInit(&decoder, (uint8_t *)msg, msgLen); + if (tDecodeSStreamTask(&decoder, pTask) < 0) { + ASSERT(0); + } + tDecoderClear(&decoder); + + pTask->status = TASK_STATUS__IDLE; + + pTask->inputQueue = streamQueueOpen(); + pTask->outputQueue = streamQueueOpen(); + pTask->inputStatus = TASK_INPUT_STATUS__NORMAL; + pTask->outputStatus = TASK_INPUT_STATUS__NORMAL; + + if (pTask->inputQueue == NULL || pTask->outputQueue == NULL) goto FAIL; + + pTask->pMsgCb = &pNode->msgCb; + + ASSERT(pTask->execType != TASK_EXEC__NONE); + + ASSERT(pTask->dataScan == 0); + pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, NULL); + ASSERT(pTask->exec.executor); + + streamSetupTrigger(pTask); + + qInfo("deploy stream: stream id %ld task id %d child id %d on snode", pTask->streamId, pTask->taskId, pTask->childId); + + taosHashPut(pMeta->pHash, &pTask->taskId, sizeof(int32_t), &pTask, sizeof(void *)); + + return 0; + +FAIL: + if (pTask->inputQueue) streamQueueClose(pTask->inputQueue); + if (pTask->outputQueue) streamQueueClose(pTask->outputQueue); + if (pTask) taosMemoryFree(pTask); + return -1; +} + +static int32_t sndProcessTaskRunReq(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + SStreamTaskRunReq *pReq = pMsg->pCont; + int32_t taskId = pReq->taskId; + SStreamTask *pTask = *(SStreamTask **)taosHashGet(pMeta->pHash, &taskId, sizeof(int32_t)); + streamProcessRunReq(pTask); + return 0; +} + +static int32_t sndProcessTaskDispatchReq(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + + char *msgStr = pMsg->pCont; + char *msgBody = POINTER_SHIFT(msgStr, sizeof(SMsgHead)); + int32_t msgLen = pMsg->contLen - sizeof(SMsgHead); + + SStreamDispatchReq req; + SDecoder decoder; + tDecoderInit(&decoder, msgBody, msgLen); + tDecodeStreamDispatchReq(&decoder, &req); + int32_t taskId = req.taskId; + SStreamTask *pTask = *(SStreamTask **)taosHashGet(pMeta->pHash, &taskId, sizeof(int32_t)); + SRpcMsg rsp = { + .info = pMsg->info, + .code = 0, + }; + streamProcessDispatchReq(pTask, &req, &rsp); + return 0; +} + +static int32_t sndProcessTaskRecoverReq(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + + SStreamTaskRecoverReq *pReq = pMsg->pCont; + int32_t taskId = pReq->taskId; + SStreamTask *pTask = *(SStreamTask **)taosHashGet(pMeta->pHash, &taskId, sizeof(int32_t)); + streamProcessRecoverReq(pTask, pReq, pMsg); + return 0; +} + +static int32_t sndProcessTaskDispatchRsp(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + + SStreamDispatchRsp *pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); + int32_t taskId = pRsp->taskId; + SStreamTask *pTask = *(SStreamTask **)taosHashGet(pMeta->pHash, &taskId, sizeof(int32_t)); + streamProcessDispatchRsp(pTask, pRsp); + return 0; +} + +static int32_t sndProcessTaskRecoverRsp(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + + SStreamTaskRecoverRsp *pRsp = pMsg->pCont; + int32_t taskId = pRsp->taskId; + SStreamTask *pTask = *(SStreamTask **)taosHashGet(pMeta->pHash, &taskId, sizeof(int32_t)); + streamProcessRecoverRsp(pTask, pRsp); return 0; } -void sndProcessUMsg(SSnode *pSnode, SRpcMsg *pMsg) { +static int32_t sndProcessTaskDropReq(SSnode *pNode, SRpcMsg *pMsg) { + SStreamMeta *pMeta = pNode->pMeta; + + char *msg = pMsg->pCont; + int32_t msgLen = pMsg->contLen; + SVDropStreamTaskReq *pReq = (SVDropStreamTaskReq *)msg; + int32_t code = taosHashRemove(pMeta->pHash, &pReq->taskId, sizeof(int32_t)); + ASSERT(code == 0); + if (code == 0) { + // sendrsp + } + return code; +} + +int32_t sndProcessUMsg(SSnode *pSnode, SRpcMsg *pMsg) { // stream deploy // stream stop/resume // operator exec - if (pMsg->msgType == TDMT_STREAM_TASK_DEPLOY) { - void *msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); - SStreamTask *pTask = taosMemoryMalloc(sizeof(SStreamTask)); - if (pTask == NULL) { + switch (pMsg->msgType) { + case TDMT_STREAM_TASK_DEPLOY: + return sndProcessTaskDeployReq(pSnode, pMsg); + case TDMT_VND_STREAM_TASK_DROP: + return sndProcessTaskDropReq(pSnode, pMsg); + default: ASSERT(0); - return; - } - SDecoder decoder; - tDecoderInit(&decoder, msg, pMsg->contLen - sizeof(SMsgHead)); - tDecodeSStreamTask(&decoder, pTask); - tDecoderClear(&decoder); - - sndMetaDeployTask(pSnode->pMeta, pTask); - /*} else if (pMsg->msgType == TDMT_SND_TASK_EXEC) {*/ - /*sndProcessTaskExecReq(pSnode, pMsg);*/ - } else { - ASSERT(0); } + return 0; } -void sndProcessSMsg(SSnode *pSnode, SRpcMsg *pMsg) { - // operator exec - /*if (pMsg->msgType == TDMT_SND_TASK_EXEC) {*/ - /*sndProcessTaskExecReq(pSnode, pMsg);*/ - /*} else {*/ - ASSERT(0); - /*}*/ +int32_t sndProcessSMsg(SSnode *pSnode, SRpcMsg *pMsg) { + switch (pMsg->msgType) { + case TDMT_STREAM_TASK_RUN: + return sndProcessTaskRunReq(pSnode, pMsg); + case TDMT_STREAM_TASK_DISPATCH: + return sndProcessTaskDispatchReq(pSnode, pMsg); + case TDMT_STREAM_TASK_RECOVER: + return sndProcessTaskRecoverReq(pSnode, pMsg); + case TDMT_STREAM_TASK_DISPATCH_RSP: + return sndProcessTaskDispatchRsp(pSnode, pMsg); + case TDMT_STREAM_TASK_RECOVER_RSP: + return sndProcessTaskRecoverRsp(pSnode, pMsg); + default: + ASSERT(0); + } + return 0; } diff --git a/source/dnode/vnode/inc/vnode.h b/source/dnode/vnode/inc/vnode.h index 3cfca66a39b19146c4cd3d3e4ec931945985aa38..70d89102e67dd933a6228402f647a212d953f63a 100644 --- a/source/dnode/vnode/inc/vnode.h +++ b/source/dnode/vnode/inc/vnode.h @@ -116,7 +116,7 @@ typedef void *tsdbReaderT; #define BLOCK_LOAD_TABLE_SEQ_ORDER 2 #define BLOCK_LOAD_TABLE_RR_ORDER 3 -tsdbReaderT *tsdbReaderOpen(SVnode *pVnode, SQueryTableDataCond *pCond, STableListInfo *tableInfoGroup, uint64_t qId, +tsdbReaderT tsdbReaderOpen(SVnode *pVnode, SQueryTableDataCond *pCond, STableListInfo *tableInfoGroup, uint64_t qId, uint64_t taskId); tsdbReaderT tsdbQueryCacheLast(SVnode *pVnode, SQueryTableDataCond *pCond, STableListInfo *groupList, uint64_t qId, void *pMemRef); @@ -210,12 +210,16 @@ struct SMetaEntry { struct { int64_t ctime; int32_t ttlDays; + int32_t commentLen; + char *comment; tb_uid_t suid; uint8_t *pTags; } ctbEntry; struct { int64_t ctime; int32_t ttlDays; + int32_t commentLen; + char *comment; int32_t ncid; // next column id SSchemaWrapper schemaRow; } ntbEntry; diff --git a/source/dnode/vnode/src/inc/sma.h b/source/dnode/vnode/src/inc/sma.h index 1e77022d044cb8d8b62d22a8a0eb04b60163c828..902e5e1bcc291fbe4826b41d8702f716fd6b6a58 100644 --- a/source/dnode/vnode/src/inc/sma.h +++ b/source/dnode/vnode/src/inc/sma.h @@ -32,11 +32,12 @@ extern "C" { #define smaTrace(...) do { if (smaDebugFlag & DEBUG_TRACE) { taosPrintLog("SMA ", DEBUG_TRACE, tsdbDebugFlag, __VA_ARGS__); }} while(0) // clang-format on -typedef struct SSmaEnv SSmaEnv; -typedef struct SSmaStat SSmaStat; -typedef struct SSmaStatItem SSmaStatItem; -typedef struct SSmaKey SSmaKey; -typedef struct SRSmaInfo SRSmaInfo; +typedef struct SSmaEnv SSmaEnv; +typedef struct SSmaStat SSmaStat; +typedef struct SSmaStatItem SSmaStatItem; +typedef struct SSmaKey SSmaKey; +typedef struct SRSmaInfo SRSmaInfo; +typedef struct SRSmaInfoItem SRSmaInfoItem; struct SSmaEnv { TdThreadRwlock lock; diff --git a/source/dnode/vnode/src/inc/vnd.h b/source/dnode/vnode/src/inc/vnd.h index b0599b82ef3f7a5d62d43190ff4d09ebe428e001..9339402d4300c9271017110f5b0da4a4c672bf05 100644 --- a/source/dnode/vnode/src/inc/vnd.h +++ b/source/dnode/vnode/src/inc/vnd.h @@ -18,6 +18,7 @@ #include "sync.h" #include "syncTools.h" +#include "ttrace.h" #include "vnodeInt.h" #ifdef __cplusplus @@ -31,6 +32,7 @@ extern "C" { #define vInfo(...) do { if (vDebugFlag & DEBUG_INFO) { taosPrintLog("VND ", DEBUG_INFO, 255, __VA_ARGS__); }} while(0) #define vDebug(...) do { if (vDebugFlag & DEBUG_DEBUG) { taosPrintLog("VND ", DEBUG_DEBUG, vDebugFlag, __VA_ARGS__); }} while(0) #define vTrace(...) do { if (vDebugFlag & DEBUG_TRACE) { taosPrintLog("VND ", DEBUG_TRACE, vDebugFlag, __VA_ARGS__); }} while(0) +#define vGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vTrace(param " GTID: %s", __VA_ARGS__, buf);} while(0)//#define vDye(...) do // clang-format on // vnodeCfg.c @@ -89,4 +91,4 @@ void vnodeSyncClose(SVnode* pVnode); } #endif -#endif /*_TD_VND_H_*/ \ No newline at end of file +#endif /*_TD_VND_H_*/ diff --git a/source/dnode/vnode/src/inc/vnodeInt.h b/source/dnode/vnode/src/inc/vnodeInt.h index 553c6a40abccbcfccc5d3a8b3ad7669162f87539..c0dfebb08f566e6a9e28d233c7dc7c8dd1a991ea 100644 --- a/source/dnode/vnode/src/inc/vnodeInt.h +++ b/source/dnode/vnode/src/inc/vnodeInt.h @@ -87,6 +87,7 @@ int metaAlterSTable(SMeta* pMeta, int64_t version, SVCreateStbReq* p int metaDropSTable(SMeta* pMeta, int64_t verison, SVDropStbReq* pReq); int metaCreateTable(SMeta* pMeta, int64_t version, SVCreateTbReq* pReq); int metaDropTable(SMeta* pMeta, int64_t version, SVDropTbReq* pReq, SArray* tbUids); +int metaTtlDropTable(SMeta* pMeta, int64_t ttl, SArray* tbUids); int metaAlterTable(SMeta* pMeta, int64_t version, SVAlterTbReq* pReq, STableMetaRsp* pMetaRsp); SSchemaWrapper* metaGetTableSchema(SMeta* pMeta, tb_uid_t uid, int32_t sver, bool isinline); STSchema* metaGetTbTSchema(SMeta* pMeta, tb_uid_t uid, int32_t sver); @@ -105,27 +106,28 @@ int32_t metaSnapshotReaderClose(SMetaSnapshotReader* pReader); int32_t metaSnapshotRead(SMetaSnapshotReader* pReader, void** ppData, uint32_t* nData); void* metaGetIdx(SMeta* pMeta); void* metaGetIvtIdx(SMeta* pMeta); +int metaTtlSmaller(SMeta* pMeta, uint64_t time, SArray* uidList); int32_t metaCreateTSma(SMeta* pMeta, int64_t version, SSmaCfg* pCfg); int32_t metaDropTSma(SMeta* pMeta, int64_t indexUid); // tsdb -int tsdbOpen(SVnode* pVnode, STsdb** ppTsdb, const char* dir, STsdbKeepCfg* pKeepCfg); -int tsdbClose(STsdb** pTsdb); -int32_t tsdbBegin(STsdb* pTsdb); -int32_t tsdbCommit(STsdb* pTsdb); -int tsdbScanAndConvertSubmitMsg(STsdb* pTsdb, SSubmitReq* pMsg); -int tsdbInsertData(STsdb* pTsdb, int64_t version, SSubmitReq* pMsg, SSubmitRsp* pRsp); -int32_t tsdbInsertTableData(STsdb* pTsdb, int64_t version, SSubmitMsgIter* pMsgIter, SSubmitBlk* pBlock, - SSubmitBlkRsp* pRsp); -int32_t tsdbDeleteTableData(STsdb* pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey); -tsdbReaderT* tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, - uint64_t taskId); -tsdbReaderT tsdbQueryCacheLastT(STsdb* tsdb, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, - void* pMemRef); -int32_t tsdbSnapshotReaderOpen(STsdb* pTsdb, STsdbSnapshotReader** ppReader, int64_t sver, int64_t ever); -int32_t tsdbSnapshotReaderClose(STsdbSnapshotReader* pReader); -int32_t tsdbSnapshotRead(STsdbSnapshotReader* pReader, void** ppData, uint32_t* nData); +int tsdbOpen(SVnode* pVnode, STsdb** ppTsdb, const char* dir, STsdbKeepCfg* pKeepCfg); +int tsdbClose(STsdb** pTsdb); +int32_t tsdbBegin(STsdb* pTsdb); +int32_t tsdbCommit(STsdb* pTsdb); +int tsdbScanAndConvertSubmitMsg(STsdb* pTsdb, SSubmitReq* pMsg); +int tsdbInsertData(STsdb* pTsdb, int64_t version, SSubmitReq* pMsg, SSubmitRsp* pRsp); +int32_t tsdbInsertTableData(STsdb* pTsdb, int64_t version, SSubmitMsgIter* pMsgIter, SSubmitBlk* pBlock, + SSubmitBlkRsp* pRsp); +int32_t tsdbDeleteTableData(STsdb* pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey); +tsdbReaderT tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, + uint64_t taskId); +tsdbReaderT tsdbQueryCacheLastT(STsdb* tsdb, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, + void* pMemRef); +int32_t tsdbSnapshotReaderOpen(STsdb* pTsdb, STsdbSnapshotReader** ppReader, int64_t sver, int64_t ever); +int32_t tsdbSnapshotReaderClose(STsdbSnapshotReader* pReader); +int32_t tsdbSnapshotRead(STsdbSnapshotReader* pReader, void** ppData, uint32_t* nData); // tq int tqInit(); @@ -139,7 +141,7 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen); int32_t tqProcessVgDeleteReq(STQ* pTq, char* msg, int32_t msgLen); int32_t tqProcessOffsetCommitReq(STQ* pTq, char* msg, int32_t msgLen); int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId); -int32_t tqProcessTaskDeploy(STQ* pTq, char* msg, int32_t msgLen); +int32_t tqProcessTaskDeployReq(STQ* pTq, char* msg, int32_t msgLen); int32_t tqProcessTaskDropReq(STQ* pTq, char* msg, int32_t msgLen); int32_t tqProcessStreamTrigger(STQ* pTq, SSubmitReq* data); int32_t tqProcessTaskRunReq(STQ* pTq, SRpcMsg* pMsg); @@ -260,7 +262,7 @@ struct SSma { #define SMA_CFG(s) (&(s)->pVnode->config) #define SMA_TSDB_CFG(s) (&(s)->pVnode->config.tsdbCfg) -#define SMA_RETENTION(s) ((SRetention *)&(s)->pVnode->config.tsdbCfg.retentions) +#define SMA_RETENTION(s) ((SRetention*)&(s)->pVnode->config.tsdbCfg.retentions) #define SMA_LOCKED(s) ((s)->locked) #define SMA_META(s) ((s)->pVnode->pMeta) #define SMA_VID(s) TD_VID((s)->pVnode) diff --git a/source/dnode/vnode/src/meta/metaEntry.c b/source/dnode/vnode/src/meta/metaEntry.c index 15ef38719f9f61309ad470c1c157101cec7a9bfb..acf5b0b61367a07740a9282330bc3881625dfff7 100644 --- a/source/dnode/vnode/src/meta/metaEntry.c +++ b/source/dnode/vnode/src/meta/metaEntry.c @@ -29,12 +29,20 @@ int metaEncodeEntry(SEncoder *pCoder, const SMetaEntry *pME) { } else if (pME->type == TSDB_CHILD_TABLE) { if (tEncodeI64(pCoder, pME->ctbEntry.ctime) < 0) return -1; if (tEncodeI32(pCoder, pME->ctbEntry.ttlDays) < 0) return -1; + if (tEncodeI32(pCoder, pME->ctbEntry.commentLen) < 0) return -1; + if (pME->ctbEntry.commentLen > 0){ + if (tEncodeCStr(pCoder, pME->ctbEntry.comment) < 0) return -1; + } if (tEncodeI64(pCoder, pME->ctbEntry.suid) < 0) return -1; debugCheckTags((STag*)pME->ctbEntry.pTags); // TODO: remove after debug if (tEncodeTag(pCoder, (const STag *)pME->ctbEntry.pTags) < 0) return -1; } else if (pME->type == TSDB_NORMAL_TABLE) { if (tEncodeI64(pCoder, pME->ntbEntry.ctime) < 0) return -1; if (tEncodeI32(pCoder, pME->ntbEntry.ttlDays) < 0) return -1; + if (tEncodeI32(pCoder, pME->ntbEntry.commentLen) < 0) return -1; + if (pME->ntbEntry.commentLen > 0){ + if (tEncodeCStr(pCoder, pME->ntbEntry.comment) < 0) return -1; + } if (tEncodeI32v(pCoder, pME->ntbEntry.ncid) < 0) return -1; if (tEncodeSSchemaWrapper(pCoder, &pME->ntbEntry.schemaRow) < 0) return -1; } else if (pME->type == TSDB_TSMA_TABLE) { @@ -61,12 +69,21 @@ int metaDecodeEntry(SDecoder *pCoder, SMetaEntry *pME) { } else if (pME->type == TSDB_CHILD_TABLE) { if (tDecodeI64(pCoder, &pME->ctbEntry.ctime) < 0) return -1; if (tDecodeI32(pCoder, &pME->ctbEntry.ttlDays) < 0) return -1; + if (tDecodeI32(pCoder, &pME->ctbEntry.commentLen) < 0) return -1; + if (pME->ctbEntry.commentLen > 0){ + if (tDecodeCStr(pCoder, &pME->ctbEntry.comment) < 0) + return -1; + } if (tDecodeI64(pCoder, &pME->ctbEntry.suid) < 0) return -1; if (tDecodeTag(pCoder, (STag **)&pME->ctbEntry.pTags) < 0) return -1; // (TODO) debugCheckTags((STag*)pME->ctbEntry.pTags); // TODO: remove after debug } else if (pME->type == TSDB_NORMAL_TABLE) { if (tDecodeI64(pCoder, &pME->ntbEntry.ctime) < 0) return -1; if (tDecodeI32(pCoder, &pME->ntbEntry.ttlDays) < 0) return -1; + if (tDecodeI32(pCoder, &pME->ntbEntry.commentLen) < 0) return -1; + if (pME->ntbEntry.commentLen > 0){ + if (tDecodeCStr(pCoder, &pME->ntbEntry.comment) < 0) return -1; + } if (tDecodeI32v(pCoder, &pME->ntbEntry.ncid) < 0) return -1; if (tDecodeSSchemaWrapperEx(pCoder, &pME->ntbEntry.schemaRow) < 0) return -1; } else if (pME->type == TSDB_TSMA_TABLE) { diff --git a/source/dnode/vnode/src/meta/metaQuery.c b/source/dnode/vnode/src/meta/metaQuery.c index a6339125c42e873a62b231ab3932736023f2204f..a5ca90e55fcba3e20c250e5ffa58d4e71cc5ca07 100644 --- a/source/dnode/vnode/src/meta/metaQuery.c +++ b/source/dnode/vnode/src/meta/metaQuery.c @@ -95,7 +95,7 @@ tb_uid_t metaGetTableEntryUidByName(SMeta *pMeta, const char *name) { metaULock(pMeta); - return 0; + return uid; } int metaReadNext(SMetaReader *pReader) { @@ -218,6 +218,40 @@ _err: return NULL; } +int metaTtlSmaller(SMeta *pMeta, uint64_t ttl, SArray *uidList){ + metaRLock(pMeta); + TBC * pCur; + int ret = tdbTbcOpen(pMeta->pTtlIdx, &pCur, NULL); + if (ret < 0) { + metaULock(pMeta); + return ret; + } + + STtlIdxKey ttlKey = {0}; + ttlKey.dtime = ttl; + ttlKey.uid = INT64_MAX; + int c = 0; + tdbTbcMoveTo(pCur, &ttlKey, sizeof(ttlKey), &c); + if (c < 0) { + tdbTbcMoveToPrev(pCur); + } + + void *pKey = NULL; + int kLen = 0; + while(1){ + ret = tdbTbcPrev(pCur, &pKey, &kLen, NULL, NULL); + if (ret < 0) { + break; + } + ttlKey = *(STtlIdxKey*)pKey; + taosArrayPush(uidList, &ttlKey.uid); + } + tdbTbcClose(pCur); + + tdbFree(pKey); + return 0; +} + struct SMCtbCursor { SMeta * pMeta; TBC * pCur; @@ -628,7 +662,7 @@ int32_t metaFilteTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) { void * tagData = NULL; if (param->val == NULL) { - metaError("vgId:%d failed to filter NULL data", TD_VID(pMeta->pVnode)); + metaError("vgId:%d, failed to filter NULL data", TD_VID(pMeta->pVnode)); return -1; } else { if (IS_VAR_DATA_TYPE(param->type)) { diff --git a/source/dnode/vnode/src/meta/metaTable.c b/source/dnode/vnode/src/meta/metaTable.c index 9a05f9e5a0c1fd7ca4bb1070deb78101faa8666f..bf5d5912f93df374010043c10ef0cdd48368573c 100644 --- a/source/dnode/vnode/src/meta/metaTable.c +++ b/source/dnode/vnode/src/meta/metaTable.c @@ -320,11 +320,15 @@ int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq) { if (me.type == TSDB_CHILD_TABLE) { me.ctbEntry.ctime = pReq->ctime; me.ctbEntry.ttlDays = pReq->ttl; + me.ctbEntry.commentLen = pReq->commentLen; + me.ctbEntry.comment = pReq->comment; me.ctbEntry.suid = pReq->ctb.suid; me.ctbEntry.pTags = pReq->ctb.pTag; } else { me.ntbEntry.ctime = pReq->ctime; me.ntbEntry.ttlDays = pReq->ttl; + me.ntbEntry.commentLen = pReq->commentLen; + me.ntbEntry.comment = pReq->comment; me.ntbEntry.schemaRow = pReq->ntb.schemaRow; me.ntbEntry.ncid = me.ntbEntry.schemaRow.pSchema[me.ntbEntry.schemaRow.nCols - 1].colId + 1; } @@ -359,7 +363,7 @@ int metaDropTable(SMeta *pMeta, int64_t version, SVDropTbReq *pReq, SArray *tbUi metaDropTableByUid(pMeta, uid, &type); metaULock(pMeta); - if (type == TSDB_CHILD_TABLE && tbUids) { + if ((type == TSDB_CHILD_TABLE || type == TSDB_NORMAL_TABLE) && tbUids) { taosArrayPush(tbUids, &uid); } @@ -367,16 +371,57 @@ int metaDropTable(SMeta *pMeta, int64_t version, SVDropTbReq *pReq, SArray *tbUi return 0; } +int metaTtlDropTable(SMeta *pMeta, int64_t ttl, SArray *tbUids) { + metaWLock(pMeta); + int ret = metaTtlSmaller(pMeta, ttl, tbUids); + if(ret != 0){ + return ret; + } + for (int i = 0; i < taosArrayGetSize(tbUids); ++i) { + tb_uid_t *uid = (tb_uid_t *)taosArrayGet(tbUids, i); + metaDropTableByUid(pMeta, *uid, NULL); + } + metaULock(pMeta); + return 0; +} + +static void metaBuildTtlIdxKey(STtlIdxKey *ttlKey, const SMetaEntry *pME){ + int32_t ttlDays; + int64_t ctime; + if (pME->type == TSDB_CHILD_TABLE) { + ctime = pME->ctbEntry.ctime; + ttlDays = pME->ctbEntry.ttlDays; + } else if (pME->type == TSDB_NORMAL_TABLE) { + ctime = pME->ntbEntry.ctime; + ttlDays = pME->ntbEntry.ttlDays; + } else { + ASSERT(0); + } + + if (ttlDays <= 0) return; + + ttlKey->dtime = ctime / 1000 + ttlDays * 24 * 60 * 60; +// ttlKey->dtime = ctime / 1000 + ttlDays; + ttlKey->uid = pME->uid; +} + +static int metaDeleteTtlIdx(SMeta *pMeta, const SMetaEntry *pME) { + STtlIdxKey ttlKey = {0}; + metaBuildTtlIdxKey(&ttlKey, pME); + if(ttlKey.dtime == 0) return 0; + return tdbTbDelete(pMeta->pTtlIdx, &ttlKey, sizeof(ttlKey), &pMeta->txn); +} + + static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) { void * pData = NULL; int nData = 0; int rc = 0; - int64_t version; SMetaEntry e = {0}; SDecoder dc = {0}; rc = tdbTbGet(pMeta->pUidIdx, &uid, sizeof(uid), &pData, &nData); - version = *(int64_t *)pData; + int64_t version = *(int64_t *)pData; tdbTbGet(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), &pData, &nData); @@ -388,15 +433,17 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) { tdbTbDelete(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), &pMeta->txn); tdbTbDelete(pMeta->pNameIdx, e.name, strlen(e.name) + 1, &pMeta->txn); tdbTbDelete(pMeta->pUidIdx, &uid, sizeof(uid), &pMeta->txn); + if(e.type != TSDB_SUPER_TABLE) metaDeleteTtlIdx(pMeta, &e); + if (e.type == TSDB_CHILD_TABLE) { tdbTbDelete(pMeta->pCtbIdx, &(SCtbIdxKey){.suid = e.ctbEntry.suid, .uid = uid}, sizeof(SCtbIdxKey), &pMeta->txn); } else if (e.type == TSDB_NORMAL_TABLE) { // drop schema.db (todo) - // drop ttl.idx (todo) } else if (e.type == TSDB_SUPER_TABLE) { // drop schema.db (todo) } + metaError("ttl drop table:%s", e.name); tDecoderClear(&dc); tdbFree(pData); @@ -545,16 +592,20 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl metaUpdateMetaRsp(uid, pAlterTbReq->tbName, pSchema, pMetaRsp); + if (entry.pBuf) taosMemoryFree(entry.pBuf); if (pNewSchema) taosMemoryFree(pNewSchema); - tDecoderClear(&dc); tdbTbcClose(pTbDbc); tdbTbcClose(pUidIdxc); + tDecoderClear(&dc); + return 0; _err: - tDecoderClear(&dc); + if (entry.pBuf) taosMemoryFree(entry.pBuf); tdbTbcClose(pTbDbc); tdbTbcClose(pUidIdxc); + tDecoderClear(&dc); + return -1; } @@ -706,8 +757,87 @@ _err: } static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) { - // TODO - ASSERT(0); + void * pVal = NULL; + int nVal = 0; + const void * pData = NULL; + int nData = 0; + int ret = 0; + tb_uid_t uid; + int64_t oversion; + SMetaEntry entry = {0}; + int c = 0; + + // search name index + ret = tdbTbGet(pMeta->pNameIdx, pAlterTbReq->tbName, strlen(pAlterTbReq->tbName) + 1, &pVal, &nVal); + if (ret < 0) { + terrno = TSDB_CODE_VND_TABLE_NOT_EXIST; + return -1; + } + + uid = *(tb_uid_t *)pVal; + tdbFree(pVal); + pVal = NULL; + + // search uid index + TBC *pUidIdxc = NULL; + + tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, &pMeta->txn); + tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); + ASSERT(c == 0); + + tdbTbcGet(pUidIdxc, NULL, NULL, &pData, &nData); + oversion = *(int64_t *)pData; + + // search table.db + TBC *pTbDbc = NULL; + + tdbTbcOpen(pMeta->pTbDb, &pTbDbc, &pMeta->txn); + tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); + ASSERT(c == 0); + tdbTbcGet(pTbDbc, NULL, NULL, &pData, &nData); + + // get table entry + SDecoder dc = {0}; + entry.pBuf = taosMemoryMalloc(nData); + memcpy(entry.pBuf, pData, nData); + tDecoderInit(&dc, entry.pBuf, nData); + ret = metaDecodeEntry(&dc, &entry); + ASSERT(ret == 0); + + entry.version = version; + metaWLock(pMeta); + // build SMetaEntry + if (entry.type == TSDB_CHILD_TABLE) { + if(pAlterTbReq->updateTTL) { + metaDeleteTtlIdx(pMeta, &entry); + entry.ctbEntry.ttlDays = pAlterTbReq->newTTL; + metaUpdateTtlIdx(pMeta, &entry); + } + if(pAlterTbReq->newCommentLen >= 0) { + entry.ctbEntry.commentLen = pAlterTbReq->newCommentLen; + entry.ctbEntry.comment = pAlterTbReq->newComment; + } + } else { + if(pAlterTbReq->updateTTL) { + metaDeleteTtlIdx(pMeta, &entry); + entry.ntbEntry.ttlDays = pAlterTbReq->newTTL; + metaUpdateTtlIdx(pMeta, &entry); + } + if(pAlterTbReq->newCommentLen >= 0) { + entry.ntbEntry.commentLen = pAlterTbReq->newCommentLen; + entry.ntbEntry.comment = pAlterTbReq->newComment; + } + } + + // save to table db + metaSaveToTbDb(pMeta, &entry); + tdbTbcUpsert(pUidIdxc, &entry.uid, sizeof(tb_uid_t), &version, sizeof(version), 0); + metaULock(pMeta); + + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tDecoderClear(&dc); + if (entry.pBuf) taosMemoryFree(entry.pBuf); return 0; } @@ -786,25 +916,9 @@ static int metaUpdateNameIdx(SMeta *pMeta, const SMetaEntry *pME) { } static int metaUpdateTtlIdx(SMeta *pMeta, const SMetaEntry *pME) { - int32_t ttlDays; - int64_t ctime; - STtlIdxKey ttlKey; - - if (pME->type == TSDB_CHILD_TABLE) { - ctime = pME->ctbEntry.ctime; - ttlDays = pME->ctbEntry.ttlDays; - } else if (pME->type == TSDB_NORMAL_TABLE) { - ctime = pME->ntbEntry.ctime; - ttlDays = pME->ntbEntry.ttlDays; - } else { - ASSERT(0); - } - - if (ttlDays <= 0) return 0; - - ttlKey.dtime = ctime + ttlDays * 24 * 60 * 60; - ttlKey.uid = pME->uid; - + STtlIdxKey ttlKey = {0}; + metaBuildTtlIdxKey(&ttlKey, pME); + if(ttlKey.dtime == 0) return 0; return tdbTbInsert(pMeta->pTtlIdx, &ttlKey, sizeof(ttlKey), NULL, 0, &pMeta->txn); } diff --git a/source/dnode/vnode/src/sma/sma.c b/source/dnode/vnode/src/sma/sma.c index b5c55a2f83f782879e6019c7da911c4871af83fa..12f93f9400539958455a8d53df14ce6f4891dda2 100644 --- a/source/dnode/vnode/src/sma/sma.c +++ b/source/dnode/vnode/src/sma/sma.c @@ -15,6 +15,8 @@ #include "sma.h" +// functions for external invocation + // TODO: Who is responsible for resource allocate and release? int32_t tdProcessTSmaInsert(SSma* pSma, int64_t indexUid, const char* msg) { int32_t code = TSDB_CODE_SUCCESS; @@ -45,6 +47,9 @@ int32_t smaGetTSmaDays(SVnodeCfg* pCfg, void* pCont, uint32_t contLen, int32_t* return code; } + +// functions for internal invocation + #if 0 /** diff --git a/source/dnode/vnode/src/sma/smaEnv.c b/source/dnode/vnode/src/sma/smaEnv.c index f71c222772925c7d7a3fd583760ccb739e94b681..a80af2b20255fa51d114d9568afadac9318765cf 100644 --- a/source/dnode/vnode/src/sma/smaEnv.c +++ b/source/dnode/vnode/src/sma/smaEnv.c @@ -208,7 +208,6 @@ int32_t tdUnLockSma(SSma *pSma) { int32_t tdCheckAndInitSmaEnv(SSma *pSma, int8_t smaType) { SSmaEnv *pEnv = NULL; - // return if already init switch (smaType) { case TSDB_SMA_TYPE_TIME_RANGE: if ((pEnv = (SSmaEnv *)atomic_load_ptr(&SMA_TSMA_ENV(pSma)))) { @@ -244,3 +243,34 @@ int32_t tdCheckAndInitSmaEnv(SSma *pSma, int8_t smaType) { return TSDB_CODE_SUCCESS; }; + +int32_t smaTimerInit(void **timer, int8_t *initFlag, const char *label) { + int8_t old; + while (1) { + old = atomic_val_compare_exchange_8(initFlag, 0, 2); + if (old != 2) break; + } + + if (old == 0) { + *timer = taosTmrInit(10000, 100, 10000, label); + if (!(*timer)) { + atomic_store_8(initFlag, 0); + return -1; + } + atomic_store_8(initFlag, 1); + } + return 0; +} + +void smaTimerCleanUp(void *timer, int8_t *initFlag) { + int8_t old; + while (1) { + old = atomic_val_compare_exchange_8(initFlag, 1, 2); + if (old != 2) break; + } + + if (old == 1) { + taosTmrCleanUp(timer); + atomic_store_8(initFlag, 0); + } +} diff --git a/source/dnode/vnode/src/sma/smaRollup.c b/source/dnode/vnode/src/sma/smaRollup.c index 0c372dfa70dab94e576456347b5174316db2aa62..1f18f6cb8777452dde659bbdec8a22d9f84e30de 100644 --- a/source/dnode/vnode/src/sma/smaRollup.c +++ b/source/dnode/vnode/src/sma/smaRollup.c @@ -14,14 +14,61 @@ */ #include "sma.h" +#include "tstream.h" static FORCE_INLINE int32_t tdUidStorePut(STbUidStore *pStore, tb_uid_t suid, tb_uid_t *uid); static FORCE_INLINE int32_t tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SArray *tbUids); -static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, qTaskInfo_t *taskInfo, - STSchema *pTSchema, tb_uid_t suid, int8_t level); +static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *rsmaItem, + tb_uid_t suid, int8_t level); + +#define SET_RSMA_INFO_ITEM_PARAMS(__idx, __level) \ + if (param->qmsg[__idx]) { \ + pRSmaInfo->items[__idx].pRsmaInfo = pRSmaInfo; \ + pRSmaInfo->items[__idx].taskInfo = qCreateStreamExecTaskInfo(param->qmsg[0], &handle); \ + if (!pRSmaInfo->items[__idx].taskInfo) { \ + goto _err; \ + } \ + pRSmaInfo->items[__idx].triggerStatus = TASK_TRIGGER_STATUS__IN_ACTIVE; \ + if (param->maxdelay[__idx] < 1) { \ + int64_t msInterval = \ + convertTimeFromPrecisionToUnit(pRetention[__level].freq, pTsdbCfg->precision, TIME_UNIT_MILLISECOND); \ + pRSmaInfo->items[__idx].maxDelay = msInterval; \ + } else { \ + pRSmaInfo->items[__idx].maxDelay = param->maxdelay[__idx]; \ + } \ + if (pRSmaInfo->items[__idx].maxDelay > TSDB_MAX_ROLLUP_MAX_DELAY) { \ + pRSmaInfo->items[__idx].maxDelay = TSDB_MAX_ROLLUP_MAX_DELAY; \ + } \ + pRSmaInfo->items[__idx].level = TSDB_RETENTION_L##__level; \ + pRSmaInfo->items[__idx].tmrHandle = taosTmrInit(10000, 100, 10000, "RSMA"); \ + if (!pRSmaInfo->items[__idx].tmrHandle) { \ + goto _err; \ + } \ + } + +struct SRSmaInfoItem { + SRSmaInfo *pRsmaInfo; + void *taskInfo; // qTaskInfo_t + void *tmrHandle; + tmr_h tmrId; + int8_t level; + int8_t tmrInitFlag; + int8_t triggerStatus; // TASK_TRIGGER_STATUS__IN_ACTIVE/TASK_TRIGGER_STATUS__ACTIVE + int32_t maxDelay; +}; + +typedef struct { + int64_t suid; + SRSmaInfoItem *pItem; + SSma *pSma; + STSchema *pTSchema; +} SRSmaTriggerParam; struct SRSmaInfo { - void *taskInfo[TSDB_RETENTION_L2]; // qTaskInfo_t + STSchema *pTSchema; + SSma *pSma; + int64_t suid; + SRSmaInfoItem items[TSDB_RETENTION_L2]; }; static FORCE_INLINE void tdFreeTaskHandle(qTaskInfo_t *taskHandle) { @@ -33,11 +80,20 @@ static FORCE_INLINE void tdFreeTaskHandle(qTaskInfo_t *taskHandle) { } void *tdFreeRSmaInfo(SRSmaInfo *pInfo) { - for (int32_t i = 0; i < TSDB_RETENTION_MAX; ++i) { - if (pInfo->taskInfo[i]) { - tdFreeTaskHandle(pInfo->taskInfo[i]); + if (pInfo) { + for (int32_t i = 0; i < TSDB_RETENTION_MAX; ++i) { + SRSmaInfoItem *pItem = &pInfo->items[i]; + if (pItem->taskInfo) { + tdFreeTaskHandle(pItem->taskInfo); + } + if (pItem->tmrHandle) { + taosTmrCleanUp(pItem->tmrHandle); + } } + taosMemoryFree(pInfo->pTSchema); + taosMemoryFree(pInfo); } + return NULL; } @@ -69,20 +125,20 @@ static FORCE_INLINE int32_t tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SA return TSDB_CODE_FAILED; } - if (pRSmaInfo->taskInfo[0] && (qUpdateQualifiedTableId(pRSmaInfo->taskInfo[0], tbUids, true) != 0)) { + if (pRSmaInfo->items[0].taskInfo && (qUpdateQualifiedTableId(pRSmaInfo->items[0].taskInfo, tbUids, true) < 0)) { smaError("vgId:%d, update tbUidList failed for uid:%" PRIi64 " since %s", SMA_VID(pSma), *suid, terrstr(terrno)); return TSDB_CODE_FAILED; } else { smaDebug("vgId:%d, update tbUidList succeed for qTaskInfo:%p with suid:%" PRIi64 ", uid:%" PRIi64, SMA_VID(pSma), - pRSmaInfo->taskInfo[0], *suid, *(int64_t *)taosArrayGet(tbUids, 0)); + pRSmaInfo->items[0].taskInfo, *suid, *(int64_t *)taosArrayGet(tbUids, 0)); } - if (pRSmaInfo->taskInfo[1] && (qUpdateQualifiedTableId(pRSmaInfo->taskInfo[1], tbUids, true) != 0)) { + if (pRSmaInfo->items[1].taskInfo && (qUpdateQualifiedTableId(pRSmaInfo->items[1].taskInfo, tbUids, true) < 0)) { smaError("vgId:%d, update tbUidList failed for uid:%" PRIi64 " since %s", SMA_VID(pSma), *suid, terrstr(terrno)); return TSDB_CODE_FAILED; } else { smaDebug("vgId:%d, update tbUidList succeed for qTaskInfo:%p with suid:%" PRIi64 ", uid:%" PRIi64, SMA_VID(pSma), - pRSmaInfo->taskInfo[1], *suid, *(int64_t *)taosArrayGet(tbUids, 0)); + pRSmaInfo->items[1].taskInfo, *suid, *(int64_t *)taosArrayGet(tbUids, 0)); } return TSDB_CODE_SUCCESS; @@ -144,12 +200,12 @@ int32_t tdFetchTbUidList(SSma *pSma, STbUidStore **ppStore, tb_uid_t suid, tb_ui ASSERT(ppStore != NULL); if (!(*ppStore)) { - if (tdUidStoreInit(ppStore) != 0) { + if (tdUidStoreInit(ppStore) < 0) { return TSDB_CODE_FAILED; } } - if (tdUidStorePut(*ppStore, suid, &uid) != 0) { + if (tdUidStorePut(*ppStore, suid, &uid) < 0) { *ppStore = tdUidStoreFree(*ppStore); return TSDB_CODE_FAILED; } @@ -172,11 +228,11 @@ int32_t tdProcessRSmaCreate(SVnode *pVnode, SVCreateStbReq *pReq) { return TSDB_CODE_SUCCESS; } - SMeta *pMeta = pVnode->pMeta; - SMsgCb *pMsgCb = &pVnode->msgCb; + SMeta *pMeta = pVnode->pMeta; + SMsgCb *pMsgCb = &pVnode->msgCb; SRSmaParam *param = &pReq->pRSmaParam; - if ((param->qmsg1Len == 0) && (param->qmsg2Len == 0)) { + if ((param->qmsgLen[0] == 0) && (param->qmsgLen[1] == 0)) { smaWarn("vgId:%d, no qmsg1/qmsg2 for rollup stable %s %" PRIi64, SMA_VID(pSma), pReq->name, pReq->suid); return TSDB_CODE_SUCCESS; } @@ -192,10 +248,12 @@ int32_t tdProcessRSmaCreate(SVnode *pVnode, SVCreateStbReq *pReq) { pRSmaInfo = taosHashGet(SMA_STAT_INFO_HASH(pStat), &pReq->suid, sizeof(tb_uid_t)); if (pRSmaInfo) { + ASSERT(0); // TODO: free original pRSmaInfo is exists abnormally smaWarn("vgId:%d, rsma info already exists for stb: %s, %" PRIi64, SMA_VID(pSma), pReq->name, pReq->suid); return TSDB_CODE_SUCCESS; } + // from write queue: single thead pRSmaInfo = (SRSmaInfo *)taosMemoryCalloc(1, sizeof(SRSmaInfo)); if (!pRSmaInfo) { terrno = TSDB_CODE_OUT_OF_MEMORY; @@ -204,9 +262,8 @@ int32_t tdProcessRSmaCreate(SVnode *pVnode, SVCreateStbReq *pReq) { STqReadHandle *pReadHandle = tqInitSubmitMsgScanner(pMeta); if (!pReadHandle) { - taosMemoryFree(pRSmaInfo); terrno = TSDB_CODE_OUT_OF_MEMORY; - return TSDB_CODE_FAILED; + goto _err; } SReadHandle handle = { @@ -216,32 +273,33 @@ int32_t tdProcessRSmaCreate(SVnode *pVnode, SVCreateStbReq *pReq) { .vnode = pVnode, }; - if (param->qmsg1) { - pRSmaInfo->taskInfo[0] = qCreateStreamExecTaskInfo(param->qmsg1, &handle); - if (!pRSmaInfo->taskInfo[0]) { - taosMemoryFree(pRSmaInfo); - taosMemoryFree(pReadHandle); - return TSDB_CODE_FAILED; - } + STSchema *pTSchema = metaGetTbTSchema(SMA_META(pSma), pReq->suid, -1); + if (!pTSchema) { + terrno = TSDB_CODE_TDB_IVD_TB_SCHEMA_VERSION; + goto _err; } + pRSmaInfo->pTSchema = pTSchema; + pRSmaInfo->pSma = pSma; + pRSmaInfo->suid = pReq->suid; - if (param->qmsg2) { - pRSmaInfo->taskInfo[1] = qCreateStreamExecTaskInfo(param->qmsg2, &handle); - if (!pRSmaInfo->taskInfo[1]) { - taosMemoryFree(pRSmaInfo); - taosMemoryFree(pReadHandle); - return TSDB_CODE_FAILED; - } - } + SRetention *pRetention = SMA_RETENTION(pSma); + STsdbCfg *pTsdbCfg = SMA_TSDB_CFG(pSma); + + SET_RSMA_INFO_ITEM_PARAMS(0, 1); + SET_RSMA_INFO_ITEM_PARAMS(1, 2); if (taosHashPut(SMA_STAT_INFO_HASH(pStat), &pReq->suid, sizeof(tb_uid_t), &pRSmaInfo, sizeof(pRSmaInfo)) != TSDB_CODE_SUCCESS) { - return TSDB_CODE_FAILED; + goto _err; } else { smaDebug("vgId:%d, register rsma info succeed for suid:%" PRIi64, SMA_VID(pSma), pReq->suid); } return TSDB_CODE_SUCCESS; +_err: + tdFreeRSmaInfo(pRSmaInfo); + taosMemoryFree(pReadHandle); + return TSDB_CODE_FAILED; } /** @@ -291,12 +349,12 @@ static int32_t tdUidStorePut(STbUidStore *pStore, tb_uid_t suid, tb_uid_t *uid) terrno = TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_FAILED; } - if (taosHashPut(pStore->uidHash, &suid, sizeof(suid), &pUidArray, sizeof(pUidArray)) != 0) { + if (taosHashPut(pStore->uidHash, &suid, sizeof(suid), &pUidArray, sizeof(pUidArray)) < 0) { return TSDB_CODE_FAILED; } } } else { - if (taosHashPut(pStore->uidHash, &suid, sizeof(suid), NULL, 0) != 0) { + if (taosHashPut(pStore->uidHash, &suid, sizeof(suid), NULL, 0) < 0) { return TSDB_CODE_FAILED; } } @@ -367,22 +425,15 @@ static int32_t tdFetchSubmitReqSuids(SSubmitReq *pMsg, STbUidStore *pStore) { return 0; } -static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, qTaskInfo_t *taskInfo, - STSchema *pTSchema, tb_uid_t suid, int8_t level) { - SArray *pResult = NULL; - - if (!taskInfo) { - smaDebug("vgId:%d, no qTaskInfo to execute rsma %" PRIi8 " task for suid:%" PRIu64, SMA_VID(pSma), level, suid); - return TSDB_CODE_SUCCESS; - } - - smaDebug("vgId:%d, execute rsma %" PRIi8 " task for qTaskInfo:%p suid:%" PRIu64, SMA_VID(pSma), level, taskInfo, suid); +static int32_t tdFetchAndSubmitRSmaResult(SRSmaInfoItem *pItem, int8_t blkType) { + SArray *pResult = NULL; + SRSmaInfo *pRSmaInfo = pItem->pRsmaInfo; + SSma *pSma = pRSmaInfo->pSma; - qSetStreamInput(taskInfo, pMsg, inputType, true); while (1) { SSDataBlock *output = NULL; uint64_t ts; - if (qExecTask(taskInfo, &output, &ts) < 0) { + if (qExecTask(pItem->taskInfo, &output, &ts) < 0) { ASSERT(false); } if (!output) { @@ -402,16 +453,16 @@ static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int3 if (taosArrayGetSize(pResult) > 0) { #if 0 char flag[10] = {0}; - snprintf(flag, 10, "level %" PRIi8, level); + snprintf(flag, 10, "level %" PRIi8, pItem->level); blockDebugShowData(pResult, flag); #endif - STsdb *sinkTsdb = (level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb1 : pSma->pRSmaTsdb2); + STsdb *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb1 : pSma->pRSmaTsdb2); SSubmitReq *pReq = NULL; - if (buildSubmitReqFromDataBlock(&pReq, pResult, pTSchema, SMA_VID(pSma), suid) < 0) { + if (buildSubmitReqFromDataBlock(&pReq, pResult, pRSmaInfo->pTSchema, SMA_VID(pSma), pRSmaInfo->suid) < 0) { taosArrayDestroy(pResult); return TSDB_CODE_FAILED; } - + if (pReq && tdProcessSubmitReq(sinkTsdb, INT64_MAX, pReq) < 0) { taosArrayDestroy(pResult); taosMemoryFreeClear(pReq); @@ -420,10 +471,63 @@ static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int3 taosMemoryFreeClear(pReq); } else { - smaDebug("vgId:%d, no rsma % " PRIi8 " data generated since %s", SMA_VID(pSma), level, tstrerror(terrno)); + smaDebug("vgId:%d, no rsma % " PRIi8 " data generated since %s", SMA_VID(pSma), pItem->level, tstrerror(terrno)); + } + + if (blkType == STREAM_DATA_TYPE_SUBMIT_BLOCK) { + atomic_store_8(&pItem->triggerStatus, TASK_TRIGGER_STATUS__ACTIVE); } taosArrayDestroy(pResult); + return 0; +} + +/** + * @brief trigger to get rsma result + * + * @param param + * @param tmrId + */ +static void rsmaTriggerByTimer(void *param, void *tmrId) { + // SRSmaTriggerParam *pParam = (SRSmaTriggerParam *)param; + // SRSmaInfoItem *pItem = pParam->pItem; + SRSmaInfoItem *pItem = param; + + if (atomic_load_8(&pItem->triggerStatus) == TASK_TRIGGER_STATUS__ACTIVE) { + smaTrace("level %" PRIi8 " status is active for tb suid:%" PRIi64, pItem->level, pItem->pRsmaInfo->suid); + SSDataBlock dataBlock = {.info.type = STREAM_GET_ALL}; + + atomic_store_8(&pItem->triggerStatus, TASK_TRIGGER_STATUS__IN_ACTIVE); + qSetStreamInput(pItem->taskInfo, &dataBlock, STREAM_DATA_TYPE_SSDATA_BLOCK, false); + + tdFetchAndSubmitRSmaResult(pItem, STREAM_DATA_TYPE_SSDATA_BLOCK); + } else { + smaTrace("level %" PRIi8 " status is inactive for tb suid:%" PRIi64, pItem->level, pItem->pRsmaInfo->suid); + } + + // taosTmrReset(rsmaTriggerByTimer, pItem->maxDelay, pItem, pItem->tmrHandle, &pItem->tmrId); +} + +static FORCE_INLINE int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *pItem, + tb_uid_t suid, int8_t level) { + if (!pItem || !pItem->taskInfo) { + smaDebug("vgId:%d, no qTaskInfo to execute rsma %" PRIi8 " task for suid:%" PRIu64, SMA_VID(pSma), level, suid); + return TSDB_CODE_SUCCESS; + } + + smaDebug("vgId:%d, execute rsma %" PRIi8 " task for qTaskInfo:%p suid:%" PRIu64, SMA_VID(pSma), level, + pItem->taskInfo, suid); + + // inputType = STREAM_DATA_TYPE_SUBMIT_BLOCK(1) + if (qSetStreamInput(pItem->taskInfo, pMsg, inputType, true) < 0) { + smaError("vgId:%d, rsma % " PRIi8 " qSetStreamInput failed since %s", SMA_VID(pSma), level, tstrerror(terrno)); + return TSDB_CODE_FAILED; + } + + // SRSmaTriggerParam triggerParam = {.suid = suid, .pItem = pItem, .pSma = pSma, .pTSchema = pTSchema}; + tdFetchAndSubmitRSmaResult(pItem, STREAM_DATA_TYPE_SUBMIT_BLOCK); + atomic_store_8(&pItem->triggerStatus, TASK_TRIGGER_STATUS__ACTIVE); + taosTmrReset(rsmaTriggerByTimer, pItem->maxDelay, pItem, pItem->tmrHandle, &pItem->tmrId); return TSDB_CODE_SUCCESS; } @@ -441,24 +545,18 @@ static int32_t tdExecuteRSma(SSma *pSma, const void *pMsg, int32_t inputType, tb pRSmaInfo = taosHashGet(SMA_STAT_INFO_HASH(pStat), &suid, sizeof(tb_uid_t)); if (!pRSmaInfo || !(pRSmaInfo = *(SRSmaInfo **)pRSmaInfo)) { - smaDebug("vgId:%d, no rsma info for suid:%" PRIu64, SMA_VID(pSma), suid); + smaDebug("vgId:%d, return as no rsma info for suid:%" PRIu64, SMA_VID(pSma), suid); return TSDB_CODE_SUCCESS; } - if (!pRSmaInfo->taskInfo[0]) { - smaDebug("vgId:%d, no rsma qTaskInfo for suid:%" PRIu64, SMA_VID(pSma), suid); + + if (!pRSmaInfo->items[0].taskInfo) { + smaDebug("vgId:%d, return as no rsma qTaskInfo for suid:%" PRIu64, SMA_VID(pSma), suid); return TSDB_CODE_SUCCESS; } if (inputType == STREAM_DATA_TYPE_SUBMIT_BLOCK) { - // TODO: cache STSchema - STSchema *pTSchema = metaGetTbTSchema(SMA_META(pSma), suid, -1); - if (!pTSchema) { - terrno = TSDB_CODE_TDB_IVD_TB_SCHEMA_VERSION; - return TSDB_CODE_FAILED; - } - tdExecuteRSmaImpl(pSma, pMsg, inputType, pRSmaInfo->taskInfo[0], pTSchema, suid, TSDB_RETENTION_L1); - tdExecuteRSmaImpl(pSma, pMsg, inputType, pRSmaInfo->taskInfo[1], pTSchema, suid, TSDB_RETENTION_L2); - taosMemoryFree(pTSchema); + tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[0], suid, TSDB_RETENTION_L1); + tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[1], suid, TSDB_RETENTION_L2); } return TSDB_CODE_SUCCESS; diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index c9d3267adb8414009bfd427f1d2c335070fe5815..ece4b7e2a45685bf1106d10e5fe00350d7183c55 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -124,17 +124,20 @@ int32_t tqProcessOffsetCommitReq(STQ* pTq, char* msg, int32_t msgLen) { tDecoderClear(&decoder); if (offset.type == TMQ_OFFSET__SNAPSHOT) { - tqDebug("receive offset commit msg to %s, offset(type:snapshot) uid: %ld, ts: %ld", offset.subKey, offset.uid, - offset.ts); + tqDebug("receive offset commit msg to %s on vg %d, offset(type:snapshot) uid: %ld, ts: %ld", offset.subKey, + TD_VID(pTq->pVnode), offset.uid, offset.ts); } else if (offset.type == TMQ_OFFSET__LOG) { - tqDebug("receive offset commit msg to %s, offset(type:log) version: %ld", offset.subKey, offset.version); + tqDebug("receive offset commit msg to %s on vg %d, offset(type:log) version: %ld", offset.subKey, + TD_VID(pTq->pVnode), offset.version); } else { ASSERT(0); } - - if (tqOffsetWrite(pTq->pOffsetStore, &offset) < 0) { - ASSERT(0); - return -1; + STqOffset* pOffset = tqOffsetRead(pTq->pOffsetStore, offset.subKey); + if (pOffset == NULL || pOffset->version < offset.version) { + if (tqOffsetWrite(pTq->pOffsetStore, &offset) < 0) { + ASSERT(0); + return -1; + } } return 0; @@ -149,28 +152,45 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) { int32_t code = 0; // get offset to fetch message - if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__EARLIEAST) { - fetchOffset = walGetFirstVer(pTq->pWal); - } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__LATEST) { - fetchOffset = walGetCommittedVer(pTq->pWal); - } else { + if (pReq->currentOffset >= 0) { fetchOffset = pReq->currentOffset + 1; + } else { + STqOffset* pOffset = tqOffsetRead(pTq->pOffsetStore, pReq->subKey); + if (pOffset != NULL) { + ASSERT(pOffset->type == TMQ_OFFSET__LOG); + tqDebug("consumer %ld, restore offset of %s on vg %d, offset(type:log) version: %ld", consumerId, pReq->subKey, + TD_VID(pTq->pVnode), pOffset->version); + fetchOffset = pOffset->version + 1; + } else { + if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__EARLIEAST) { + fetchOffset = walGetFirstVer(pTq->pWal); + } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__LATEST) { + fetchOffset = walGetCommittedVer(pTq->pWal); + } else if (pReq->currentOffset == TMQ_CONF__RESET_OFFSET__NONE) { + tqError("tmq poll: no offset committed for consumer %ld in vg %d, subkey %s", consumerId, TD_VID(pTq->pVnode), + pReq->subKey); + terrno = TSDB_CODE_TQ_NO_COMMITTED_OFFSET; + return -1; + } + tqDebug("consumer %ld, restore offset of %s on vg %d failed, config is %ld, set to %ld", consumerId, pReq->subKey, + TD_VID(pTq->pVnode), pReq->currentOffset, fetchOffset); + } } - tqDebug("tmq poll: consumer %ld (epoch %d) recv poll req in vg %d, req %ld %ld", consumerId, pReq->epoch, - TD_VID(pTq->pVnode), pReq->currentOffset, fetchOffset); + tqDebug("tmq poll: consumer %ld (epoch %d) recv poll req in vg %d, req offset %ld fetch offset %ld", consumerId, + pReq->epoch, TD_VID(pTq->pVnode), pReq->currentOffset, fetchOffset); STqHandle* pHandle = taosHashGet(pTq->handles, pReq->subKey, strlen(pReq->subKey)); /*ASSERT(pHandle);*/ if (pHandle == NULL) { - tqError("tmq poll: no consumer handle for consumer %ld in vg %d, subkey %s", consumerId, pTq->pVnode->config.vgId, + tqError("tmq poll: no consumer handle for consumer %ld in vg %d, subkey %s", consumerId, TD_VID(pTq->pVnode), pReq->subKey); return -1; } if (pHandle->consumerId != consumerId) { tqError("tmq poll: consumer handle mismatch for consumer %ld in vg %d, subkey %s, handle consumer id %ld", - consumerId, pTq->pVnode->config.vgId, pReq->subKey, pHandle->consumerId); + consumerId, TD_VID(pTq->pVnode), pReq->subKey, pHandle->consumerId); return -1; } @@ -284,7 +304,6 @@ int32_t tqProcessVgDeleteReq(STQ* pTq, char* msg, int32_t msgLen) { return 0; } -// TODO: persist meta into tdb int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) { SMqRebVgReq req = {0}; tDecodeSMqRebVgReq(msg, &req); @@ -314,7 +333,6 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) { SReadHandle handle = { .reader = pHandle->execHandle.pExecReader[i], .meta = pTq->pVnode->pMeta, - .pMsgCb = &pTq->pVnode->msgCb, }; pHandle->execHandle.execCol.task[i] = qCreateStreamExecTaskInfo(pHandle->execHandle.execCol.qmsg, &handle); ASSERT(pHandle->execHandle.execCol.task[i]); @@ -326,10 +344,10 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) { pHandle->execHandle.execTb.suid = req.suid; SArray* tbUidList = taosArrayInit(0, sizeof(int64_t)); tsdbGetCtbIdList(pTq->pVnode->pMeta, req.suid, tbUidList); - tqDebug("vg %d, tq try get suid: %ld", pTq->pVnode->config.vgId, req.suid); + tqDebug("vg %d, tq try get suid: %ld", TD_VID(pTq->pVnode), req.suid); for (int32_t i = 0; i < taosArrayGetSize(tbUidList); i++) { int64_t tbUid = *(int64_t*)taosArrayGet(tbUidList, i); - tqDebug("vg %d, idx %d, uid: %ld", pTq->pVnode->config.vgId, i, tbUid); + tqDebug("vg %d, idx %d, uid: %ld", TD_VID(pTq->pVnode), i, tbUid); } for (int32_t i = 0; i < 5; i++) { tqReadHandleSetTbUidList(pHandle->execHandle.pExecReader[i], tbUidList); @@ -354,7 +372,7 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) { return 0; } -int32_t tqProcessTaskDeploy(STQ* pTq, char* msg, int32_t msgLen) { +int32_t tqProcessTaskDeployReq(STQ* pTq, char* msg, int32_t msgLen) { SStreamTask* pTask = taosMemoryCalloc(1, sizeof(SStreamTask)); if (pTask == NULL) { return -1; @@ -380,16 +398,20 @@ int32_t tqProcessTaskDeploy(STQ* pTq, char* msg, int32_t msgLen) { // exec if (pTask->execType != TASK_EXEC__NONE) { // expand runners - STqReadHandle* pStreamReader = tqInitSubmitMsgScanner(pTq->pVnode->pMeta); - SReadHandle handle = { - .reader = pStreamReader, - .meta = pTq->pVnode->pMeta, - .pMsgCb = &pTq->pVnode->msgCb, - .vnode = pTq->pVnode, - }; - pTask->exec.inputHandle = pStreamReader; - pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &handle); - ASSERT(pTask->exec.executor); + if (pTask->dataScan) { + STqReadHandle* pStreamReader = tqInitSubmitMsgScanner(pTq->pVnode->pMeta); + SReadHandle handle = { + .reader = pStreamReader, + .meta = pTq->pVnode->pMeta, + .vnode = pTq->pVnode, + }; + /*pTask->exec.inputHandle = pStreamReader;*/ + pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &handle); + ASSERT(pTask->exec.executor); + } else { + pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, NULL); + ASSERT(pTask->exec.executor); + } } // sink @@ -411,7 +433,7 @@ int32_t tqProcessTaskDeploy(STQ* pTq, char* msg, int32_t msgLen) { streamSetupTrigger(pTask); - tqInfo("deploy stream task id %d child id %d on vg %d", pTask->taskId, pTask->childId, pTq->pVnode->config.vgId); + tqInfo("deploy stream task id %d child id %d on vg %d", pTask->taskId, pTask->childId, TD_VID(pTq->pVnode)); taosHashPut(pTq->pStreamTasks, &pTask->taskId, sizeof(int32_t), &pTask, sizeof(void*)); @@ -444,7 +466,7 @@ int32_t tqProcessStreamTrigger(STQ* pTq, SSubmitReq* pReq) { continue; } - if (streamLaunchByWrite(pTask, pTq->pVnode->config.vgId, &pTq->pVnode->msgCb) < 0) { + if (streamLaunchByWrite(pTask, TD_VID(pTq->pVnode)) < 0) { continue; } } else { @@ -465,7 +487,7 @@ int32_t tqProcessTaskRunReq(STQ* pTq, SRpcMsg* pMsg) { SStreamTaskRunReq* pReq = pMsg->pCont; int32_t taskId = pReq->taskId; SStreamTask* pTask = *(SStreamTask**)taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t)); - streamTaskProcessRunReq(pTask, &pTq->pVnode->msgCb); + streamProcessRunReq(pTask); return 0; } @@ -483,7 +505,7 @@ int32_t tqProcessTaskDispatchReq(STQ* pTq, SRpcMsg* pMsg) { .info = pMsg->info, .code = 0, }; - streamProcessDispatchReq(pTask, &pTq->pVnode->msgCb, &req, &rsp); + streamProcessDispatchReq(pTask, &req, &rsp); return 0; } @@ -491,7 +513,7 @@ int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg) { SStreamTaskRecoverReq* pReq = pMsg->pCont; int32_t taskId = pReq->taskId; SStreamTask* pTask = *(SStreamTask**)taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t)); - streamProcessRecoverReq(pTask, &pTq->pVnode->msgCb, pReq, pMsg); + streamProcessRecoverReq(pTask, pReq, pMsg); return 0; } @@ -499,7 +521,7 @@ int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) { SStreamDispatchRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); int32_t taskId = pRsp->taskId; SStreamTask* pTask = *(SStreamTask**)taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t)); - streamProcessDispatchRsp(pTask, &pTq->pVnode->msgCb, pRsp); + streamProcessDispatchRsp(pTask, pRsp); return 0; } @@ -514,9 +536,9 @@ int32_t tqProcessTaskRecoverRsp(STQ* pTq, SRpcMsg* pMsg) { int32_t tqProcessTaskDropReq(STQ* pTq, char* msg, int32_t msgLen) { SVDropStreamTaskReq* pReq = (SVDropStreamTaskReq*)msg; int32_t code = taosHashRemove(pTq->pStreamTasks, &pReq->taskId, sizeof(int32_t)); + ASSERT(code == 0); if (code == 0) { // sendrsp } - ASSERT(code == 0); return code; } diff --git a/source/dnode/vnode/src/tq/tqCommit.c b/source/dnode/vnode/src/tq/tqCommit.c index 7b116bff2e942bf1a461458ea443548e708756eb..639da22b1c16f39a953ae0b7885344a7ca95768e 100644 --- a/source/dnode/vnode/src/tq/tqCommit.c +++ b/source/dnode/vnode/src/tq/tqCommit.c @@ -15,7 +15,4 @@ #include "tq.h" -int tqCommit(STQ* pTq) { - // do nothing - return 0; -} +int tqCommit(STQ* pTq) { return tqOffsetSnapshot(pTq->pOffsetStore); } diff --git a/source/dnode/vnode/src/tq/tqOffset.c b/source/dnode/vnode/src/tq/tqOffset.c index 8d6cb280659949489a301ad68ff7740ccba5bbc4..e5475d7c300531b7b3e0d8cce369fa9eb0415930 100644 --- a/source/dnode/vnode/src/tq/tqOffset.c +++ b/source/dnode/vnode/src/tq/tqOffset.c @@ -92,6 +92,8 @@ STqOffset* tqOffsetRead(STqOffsetStore* pStore, const char* subscribeKey) { } int32_t tqOffsetWrite(STqOffsetStore* pStore, const STqOffset* pOffset) { + ASSERT(pOffset->type == TMQ_OFFSET__LOG); + ASSERT(pOffset->version >= 0); return taosHashPut(pStore->pHash, pOffset->subKey, strlen(pOffset->subKey), pOffset, sizeof(STqOffset)); } @@ -129,7 +131,7 @@ int32_t tqOffsetSnapshot(STqOffsetStore* pStore) { tEncodeSTqOffset(&encoder, pOffset); // write file int64_t writeLen; - if ((writeLen = taosWriteFile(pFile, buf, totLen)) != bodyLen) { + if ((writeLen = taosWriteFile(pFile, buf, totLen)) != totLen) { ASSERT(0); tqError("write offset incomplete, len %d, write len %ld", bodyLen, writeLen); taosHashCancelIterate(pStore->pHash, pIter); diff --git a/source/dnode/vnode/src/tsdb/tsdbCommit.c b/source/dnode/vnode/src/tsdb/tsdbCommit.c index fe89321ae9606387c0869e4d8f63328d0002e140..e98fd8ae1f6dc6cd0d6f47a2563055f7e452870a 100644 --- a/source/dnode/vnode/src/tsdb/tsdbCommit.c +++ b/source/dnode/vnode/src/tsdb/tsdbCommit.c @@ -111,7 +111,7 @@ int32_t tsdbBegin(STsdb *pTsdb) { int32_t tsdbCommit(STsdb *pTsdb) { if (!pTsdb) return 0; - + int32_t code = 0; SCommitH commith = {0}; SDFileSet *pSet = NULL; @@ -495,7 +495,9 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) { break; } - if (pIter && pIter->pTable && (!pIdx || (pIter->pTable->suid <= pIdx->suid || pIter->pTable->uid <= pIdx->uid))) { + if (pIter && pIter->pTable && + (!pIdx || ((pIter->pTable->suid < pIdx->suid) || + ((pIter->pTable->suid == pIdx->suid) && (pIter->pTable->uid <= pIdx->uid))))) { if (tsdbCommitToTable(pCommith, mIter) < 0) { tsdbCloseCommitFile(pCommith, true); // revert the file change @@ -503,7 +505,7 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) { return -1; } - if (pIdx && (pIter->pTable->uid == pIdx->uid)) { + if (pIdx && ((pIter->pTable->uid == pIdx->uid) && (pIter->pTable->suid == pIdx->suid))) { ++fIter; } ++mIter; @@ -518,6 +520,8 @@ static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) { return -1; } ++fIter; + } else { + ASSERT(0); } } diff --git a/source/dnode/vnode/src/tsdb/tsdbRead.c b/source/dnode/vnode/src/tsdb/tsdbRead.c index e7a40eeeb94f8b81984f0878d52c247be9c9c322..ce73246e51adcb3906b16a56084d44cf2fb0bed4 100644 --- a/source/dnode/vnode/src/tsdb/tsdbRead.c +++ b/source/dnode/vnode/src/tsdb/tsdbRead.c @@ -500,7 +500,7 @@ static int32_t setCurrentSchema(SVnode* pVnode, STsdbReadHandle* pTsdbReadHandle return TSDB_CODE_SUCCESS; } -tsdbReaderT* tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, +tsdbReaderT tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId, uint64_t taskId) { STsdbReadHandle* pTsdbReadHandle = tsdbQueryTablesImpl(pVnode, pCond, qId, taskId); if (pTsdbReadHandle == NULL) { @@ -508,7 +508,7 @@ tsdbReaderT* tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, STableLi } if (emptyQueryTimewindow(pTsdbReadHandle)) { - return (tsdbReaderT*)pTsdbReadHandle; + return (tsdbReaderT)pTsdbReadHandle; } // todo apply the lastkey of table check to avoid to load header file diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index da1a126a765289fa8520df33519ecd57d3f7c1d9..f3595ebfb0972b0d6c1ad9e03fd3902d200fb41d 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -26,6 +26,7 @@ static int32_t vnodeProcessCreateTSmaReq(SVnode *pVnode, int64_t version, void * static int32_t vnodeProcessAlterConfirmReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp); static int32_t vnodeProcessAlterHasnRangeReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp); static int32_t vnodeProcessWriteMsg(SVnode *pVnode, int64_t version, SRpcMsg *pMsg, SRpcMsg *pRsp); +static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp); int32_t vnodePreprocessReq(SVnode *pVnode, SRpcMsg *pMsg) { int32_t code = 0; @@ -134,6 +135,9 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp case TDMT_VND_DROP_TABLE: if (vnodeProcessDropTbReq(pVnode, version, pReq, len, pRsp) < 0) goto _err; break; + case TDMT_VND_DROP_TTL_TABLE: + if (vnodeProcessDropTtlTbReq(pVnode, version, pReq, len, pRsp) < 0) goto _err; + break; case TDMT_VND_CREATE_SMA: { if (vnodeProcessCreateTSmaReq(pVnode, version, pReq, len, pRsp) < 0) goto _err; } break; @@ -163,8 +167,8 @@ int32_t vnodeProcessWriteReq(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp } break; case TDMT_STREAM_TASK_DEPLOY: { - if (tqProcessTaskDeploy(pVnode->pTq, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)), - pMsg->contLen - sizeof(SMsgHead)) < 0) { + if (tqProcessTaskDeployReq(pVnode->pTq, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)), + pMsg->contLen - sizeof(SMsgHead)) < 0) { goto _err; } } break; @@ -300,6 +304,25 @@ void vnodeUpdateMetaRsp(SVnode *pVnode, STableMetaRsp *pMetaRsp) { pMetaRsp->precision = pVnode->config.tsdbCfg.precision; } +static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) { + SArray *tbUids = taosArrayInit(8, sizeof(int64_t)); + if (tbUids == NULL) return TSDB_CODE_OUT_OF_MEMORY; + + int32_t t = ntohl(*(int32_t *)pReq); + vError("rec ttl time:%d", t); + int32_t ret = metaTtlDropTable(pVnode->pMeta, t, tbUids); + if (ret != 0) { + goto end; + } + if (taosArrayGetSize(tbUids) > 0) { + tqUpdateTbUidList(pVnode->pTq, tbUids, false); + } + +end: + taosArrayDestroy(tbUids); + return ret; +} + static int32_t vnodeProcessCreateStbReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) { SVCreateStbReq req = {0}; SDecoder coder; @@ -322,7 +345,10 @@ static int32_t vnodeProcessCreateStbReq(SVnode *pVnode, int64_t version, void *p goto _err; } - tdProcessRSmaCreate(pVnode, &req); + if (tdProcessRSmaCreate(pVnode, &req) < 0) { + pRsp->code = terrno; + goto _err; + } tDecoderClear(&coder); return 0; @@ -774,9 +800,6 @@ static int32_t vnodeProcessCreateTSmaReq(SVnode *pVnode, int64_t version, void * goto _err; } - // record current timezone of server side - req.timezoneInt = tsTimezone; - if (tdProcessTSmaCreate(pVnode->pSma, version, (const char *)&req) < 0) { if (pRsp) pRsp->code = terrno; goto _err; diff --git a/source/dnode/vnode/src/vnd/vnodeSync.c b/source/dnode/vnode/src/vnd/vnodeSync.c index 8662c8369bb6d0361257c793697bbdc7afdc9835..f1c43512cec80c744f9a89f983ecc3c985e83d15 100644 --- a/source/dnode/vnode/src/vnd/vnodeSync.c +++ b/source/dnode/vnode/src/vnd/vnodeSync.c @@ -61,6 +61,7 @@ static int32_t vnodeSetStandBy(SVnode *pVnode) { return -1; } + vInfo("vgId:%d, start to transfer leader", TD_VID(pVnode)); if (syncLeaderTransfer(pVnode->sync) != 0) { vError("vgId:%d, failed to transfer leader since:%s", TD_VID(pVnode), terrstr()); return -1; @@ -83,8 +84,8 @@ static int32_t vnodeProcessAlterReplicaReq(SVnode *pVnode, SRpcMsg *pMsg) { terrno = TSDB_CODE_INVALID_MSG; return TSDB_CODE_INVALID_MSG; } - - vInfo("vgId:%d, start to alter vnode replica to %d, handle:%p", TD_VID(pVnode), req.replica, pMsg->info.handle); + STraceId *trace = &pMsg->info.traceId; + vGTrace("vgId:%d, start to alter vnode replica to %d, handle:%p", TD_VID(pVnode), req.replica, pMsg->info.handle); SSyncCfg cfg = {.replicaNum = req.replica, .myIndex = req.selfIndex}; for (int32_t r = 0; r < req.replica; ++r) { SNodeInfo *pNode = &cfg.nodeInfo[r]; @@ -125,7 +126,8 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { for (int32_t m = 0; m < numOfMsgs; m++) { if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; - vTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle); + STraceId *trace = &pMsg->info.traceId; + vGTrace("vgId:%d, msg:%p get from vnode-write queue handle:%p", vgId, pMsg, pMsg->info.handle); if (pMsg->msgType == TDMT_VND_ALTER_REPLICA) { code = vnodeProcessAlterReplicaReq(pVnode, pMsg); @@ -148,10 +150,10 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps; } - vTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps, - newEpSet.inUse); + vGTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", vgId, pMsg, newEpSet.numOfEps, + newEpSet.inUse); for (int32_t i = 0; i < newEpSet.numOfEps; ++i) { - vTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port); + vGTrace("vgId:%d, msg:%p redirect:%d ep:%s:%u", vgId, pMsg, i, newEpSet.eps[i].fqdn, newEpSet.eps[i].port); } SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info}; @@ -163,7 +165,7 @@ void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { tmsgSendRsp(&rsp); } - vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code); + vGTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, code); rpcFreeCont(pMsg->pCont); taosFreeQitem(pMsg); } @@ -179,12 +181,13 @@ void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { for (int32_t i = 0; i < numOfMsgs; ++i) { if (taosGetQitem(qall, (void **)&pMsg) == 0) continue; - vTrace("vgId:%d, msg:%p get from vnode-apply queue, type:%s handle:%p", vgId, pMsg, TMSG_INFO(pMsg->msgType), - pMsg->info.handle); + STraceId *trace = &pMsg->info.traceId; + vGTrace("vgId:%d, msg:%p get from vnode-apply queue, type:%s handle:%p", vgId, pMsg, TMSG_INFO(pMsg->msgType), + pMsg->info.handle); SRpcMsg rsp = {.code = pMsg->code, .info = pMsg->info}; if (rsp.code == 0) { - if (vnodeProcessWriteReq(pVnode, pMsg, pMsg->conn.applyIndex, &rsp) < 0) { + if (vnodeProcessWriteReq(pVnode, pMsg, pMsg->info.conn.applyIndex, &rsp) < 0) { rsp.code = terrno; vError("vgId:%d, msg:%p failed to apply since %s", vgId, pMsg, terrstr()); } @@ -195,7 +198,7 @@ void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { tmsgSendRsp(&rsp); } - vTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code); + vGTrace("vgId:%d, msg:%p is freed, code:0x%x", vgId, pMsg, rsp.code); rpcFreeCont(pMsg->pCont); taosFreeQitem(pMsg); } @@ -208,20 +211,23 @@ int32_t vnodeProcessSyncReq(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) { SSyncNode *pSyncNode = syncNodeAcquire(pVnode->sync); assert(pSyncNode != NULL); - ESyncState state = syncGetMyRole(pVnode->sync); - SyncTerm currentTerm = syncGetMyTerm(pVnode->sync); - SMsgHead *pHead = pMsg->pCont; + STraceId *trace = &pMsg->info.traceId; - char logBuf[512] = {0}; - char *syncNodeStr = sync2SimpleStr(pVnode->sync); - snprintf(logBuf, sizeof(logBuf), "==vnodeProcessSyncReq== msgType:%d, syncNode: %s", pMsg->msgType, syncNodeStr); - static int64_t vndTick = 0; - if (++vndTick % 10 == 1) { - vTrace("sync trace msg:%s, %s", TMSG_INFO(pMsg->msgType), syncNodeStr); - } - syncRpcMsgLog2(logBuf, pMsg); - taosMemoryFree(syncNodeStr); + do { + char *syncNodeStr = sync2SimpleStr(pVnode->sync); + static int64_t vndTick = 0; + if (++vndTick % 10 == 1) { + vGTrace("vgId:%d, sync heartbeat msg:%s, %s", syncGetVgId(pVnode->sync), TMSG_INFO(pMsg->msgType), syncNodeStr); + } + if (gRaftDetailLog) { + char logBuf[512] = {0}; + snprintf(logBuf, sizeof(logBuf), "==vnodeProcessSyncReq== msgType:%d, syncNode: %s", pMsg->msgType, + syncNodeStr); + syncRpcMsgLog2(logBuf, pMsg); + } + taosMemoryFree(syncNodeStr); + } while (0); SRpcMsg *pRpcMsg = pMsg; @@ -297,7 +303,7 @@ int32_t vnodeProcessSyncReq(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) { ret = -1; } - if (ret != 0) { + if (ret != 0 && terrno == 0) { terrno = TSDB_CODE_SYN_INTERNAL_ERROR; } return ret; @@ -329,11 +335,13 @@ static int32_t vnodeSyncGetSnapshot(SSyncFSM *pFsm, SSnapshot *pSnapshot) { static void vnodeSyncReconfig(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SReConfigCbMeta cbMeta) { SVnode *pVnode = pFsm->data; - SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen, .conn.applyIndex = cbMeta.index}; + SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen}; syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info); + rpcMsg.info.conn.applyIndex = cbMeta.index; - vInfo("vgId:%d, alter vnode replica is confirmed, type:%s contLen:%d seq:%" PRIu64 " handle:%p", TD_VID(pVnode), - TMSG_INFO(pMsg->msgType), pMsg->contLen, cbMeta.seqNum, rpcMsg.info.handle); + STraceId *trace = (STraceId *)&pMsg->info.traceId; + vGTrace("vgId:%d, alter vnode replica is confirmed, type:%s contLen:%d seq:%" PRIu64 " handle:%p", TD_VID(pVnode), + TMSG_INFO(pMsg->msgType), pMsg->contLen, cbMeta.seqNum, rpcMsg.info.handle); if (rpcMsg.info.handle != NULL) { tmsgSendRsp(&rpcMsg); } @@ -347,8 +355,8 @@ static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta c SyncIndex beginIndex = SYNC_INDEX_INVALID; char logBuf[256] = {0}; - if (pFsm->FpGetSnapshot != NULL) { - (*pFsm->FpGetSnapshot)(pFsm, &snapshot); + if (pFsm->FpGetSnapshotInfo != NULL) { + (*pFsm->FpGetSnapshotInfo)(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } @@ -359,10 +367,11 @@ static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta c pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), beginIndex); syncRpcMsgLog2(logBuf, (SRpcMsg *)pMsg); - SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen, .conn.applyIndex = cbMeta.index}; + SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen}; rpcMsg.pCont = rpcMallocCont(rpcMsg.contLen); memcpy(rpcMsg.pCont, pMsg->pCont, pMsg->contLen); syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info); + rpcMsg.info.conn.applyIndex = cbMeta.index; tmsgPutToQueue(&pVnode->msgCb, APPLY_QUEUE, &rpcMsg); } else { @@ -409,7 +418,7 @@ static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) { pFsm->FpCommitCb = vnodeSyncCommitMsg; pFsm->FpPreCommitCb = vnodeSyncPreCommitMsg; pFsm->FpRollBackCb = vnodeSyncRollBackMsg; - pFsm->FpGetSnapshot = vnodeSyncGetSnapshot; + pFsm->FpGetSnapshotInfo = vnodeSyncGetSnapshot; pFsm->FpRestoreFinishCb = NULL; pFsm->FpReConfigCb = vnodeSyncReconfig; pFsm->FpSnapshotStartRead = vnodeSnapshotStartRead; diff --git a/source/libs/catalog/inc/catalogInt.h b/source/libs/catalog/inc/catalogInt.h index df4b9b6ea6c9b30b83b0bb1c25ab9d86a82fa487..7c90c3538cc3428d6c6456c8c67f9acb4818e82c 100644 --- a/source/libs/catalog/inc/catalogInt.h +++ b/source/libs/catalog/inc/catalogInt.h @@ -59,6 +59,7 @@ enum { CTG_OP_UPDATE_VG_EPSET, CTG_OP_UPDATE_TB_INDEX, CTG_OP_DROP_TB_INDEX, + CTG_OP_CLEAR_CACHE, CTG_OP_MAX }; @@ -328,6 +329,10 @@ typedef struct SCtgDropTbIndexMsg { char tbName[TSDB_TABLE_NAME_LEN]; } SCtgDropTbIndexMsg; +typedef struct SCtgClearCacheMsg { + SCatalog* pCtg; +} SCtgClearCacheMsg; + typedef struct SCtgUpdateEpsetMsg { SCatalog* pCtg; char dbFName[TSDB_DB_FNAME_LEN]; @@ -471,8 +476,8 @@ typedef struct SCtgOperation { #define CTG_API_LEAVE(c) do { int32_t __code = c; CTG_UNLOCK(CTG_READ, &gCtgMgmt.lock); CTG_API_DEBUG("CTG API leave %s", __FUNCTION__); CTG_RET(__code); } while (0) #define CTG_API_ENTER() do { CTG_API_DEBUG("CTG API enter %s", __FUNCTION__); CTG_LOCK(CTG_READ, &gCtgMgmt.lock); if (atomic_load_8((int8_t*)&gCtgMgmt.exit)) { CTG_API_LEAVE(TSDB_CODE_CTG_OUT_OF_SERVICE); } } while (0) -void ctgdShowTableMeta(SCatalog* pCtg, const char *tbName, STableMeta* p); -void ctgdShowClusterCache(SCatalog* pCtg); +void ctgdShowTableMeta(SCatalog* pCtg, const char *tbName, STableMeta* p); +void ctgdShowClusterCache(SCatalog* pCtg); int32_t ctgdShowCacheInfo(void); int32_t ctgRemoveTbMetaFromCache(SCatalog* pCtg, SName* pTableName, bool syncReq); @@ -487,8 +492,8 @@ int32_t ctgOpDropTbMeta(SCtgCacheOperation *action); int32_t ctgOpUpdateUser(SCtgCacheOperation *action); int32_t ctgOpUpdateEpset(SCtgCacheOperation *operation); int32_t ctgAcquireVgInfoFromCache(SCatalog* pCtg, const char *dbFName, SCtgDBCache **pCache); -void ctgReleaseDBCache(SCatalog *pCtg, SCtgDBCache *dbCache); -void ctgRUnlockVgInfo(SCtgDBCache *dbCache); +void ctgReleaseDBCache(SCatalog *pCtg, SCtgDBCache *dbCache); +void ctgRUnlockVgInfo(SCtgDBCache *dbCache); int32_t ctgTbMetaExistInCache(SCatalog* pCtg, char *dbFName, char* tbName, int32_t *exist); int32_t ctgReadTbMetaFromCache(SCatalog* pCtg, SCtgTbMetaCtx* ctx, STableMeta** pTableMeta); int32_t ctgReadTbVerFromCache(SCatalog *pCtg, SName *pTableName, int32_t *sver, int32_t *tver, int32_t *tbType, uint64_t *suid, char *stbName); @@ -502,17 +507,20 @@ int32_t ctgUpdateTbMetaEnqueue(SCatalog* pCtg, STableMetaOutput *output, bool sy int32_t ctgUpdateUserEnqueue(SCatalog* pCtg, SGetUserAuthRsp *pAuth, bool syncReq); int32_t ctgUpdateVgEpsetEnqueue(SCatalog* pCtg, char *dbFName, int32_t vgId, SEpSet* pEpSet); int32_t ctgUpdateTbIndexEnqueue(SCatalog* pCtg, STableIndex **pIndex, bool syncOp); +int32_t ctgClearCacheEnqueue(SCatalog* pCtg, bool syncOp); int32_t ctgMetaRentInit(SCtgRentMgmt *mgmt, uint32_t rentSec, int8_t type); int32_t ctgMetaRentAdd(SCtgRentMgmt *mgmt, void *meta, int64_t id, int32_t size); int32_t ctgMetaRentGet(SCtgRentMgmt *mgmt, void **res, uint32_t *num, int32_t size); int32_t ctgUpdateTbMetaToCache(SCatalog* pCtg, STableMetaOutput* pOut, bool syncReq); int32_t ctgStartUpdateThread(); int32_t ctgRelaunchGetTbMetaTask(SCtgTask *pTask); -void ctgReleaseVgInfoToCache(SCatalog* pCtg, SCtgDBCache *dbCache); +void ctgReleaseVgInfoToCache(SCatalog* pCtg, SCtgDBCache *dbCache); int32_t ctgReadTbIndexFromCache(SCatalog* pCtg, SName* pTableName, SArray** pRes); int32_t ctgDropTbIndexEnqueue(SCatalog* pCtg, SName* pName, bool syncOp); int32_t ctgOpDropTbIndex(SCtgCacheOperation *operation); int32_t ctgOpUpdateTbIndex(SCtgCacheOperation *operation); +int32_t ctgOpClearCache(SCtgCacheOperation *operation); + @@ -535,22 +543,22 @@ int32_t ctgMakeAsyncRes(SCtgJob *pJob); int32_t ctgCloneVgInfo(SDBVgInfo *src, SDBVgInfo **dst); int32_t ctgCloneMetaOutput(STableMetaOutput *output, STableMetaOutput **pOutput); int32_t ctgGenerateVgList(SCatalog *pCtg, SHashObj *vgHash, SArray** pList); -void ctgFreeJob(void* job); -void ctgFreeHandle(SCatalog* pCtg); -void ctgFreeVgInfo(SDBVgInfo *vgInfo); +void ctgFreeJob(void* job); +void ctgFreeHandle(SCatalog* pCtg); +void ctgFreeVgInfo(SDBVgInfo *vgInfo); int32_t ctgGetVgInfoFromHashValue(SCatalog *pCtg, SDBVgInfo *dbInfo, const SName *pTableName, SVgroupInfo *pVgroup); -void ctgResetTbMetaTask(SCtgTask* pTask); -void ctgFreeDbCache(SCtgDBCache *dbCache); +void ctgResetTbMetaTask(SCtgTask* pTask); +void ctgFreeDbCache(SCtgDBCache *dbCache); int32_t ctgStbVersionSortCompare(const void* key1, const void* key2); int32_t ctgDbVgVersionSortCompare(const void* key1, const void* key2); int32_t ctgStbVersionSearchCompare(const void* key1, const void* key2); int32_t ctgDbVgVersionSearchCompare(const void* key1, const void* key2); -void ctgFreeSTableMetaOutput(STableMetaOutput* pOutput); +void ctgFreeSTableMetaOutput(STableMetaOutput* pOutput); int32_t ctgUpdateMsgCtx(SCtgMsgCtx* pCtx, int32_t reqType, void* out, char* target); -char *ctgTaskTypeStr(CTG_TASK_TYPE type); +char * ctgTaskTypeStr(CTG_TASK_TYPE type); int32_t ctgUpdateSendTargetInfo(SMsgSendInfo *pMsgSendInfo, int32_t msgType, SCtgTask* pTask); int32_t ctgCloneTableIndex(SArray* pIndex, SArray** pRes); -void ctgFreeSTableIndex(void *info); +void ctgFreeSTableIndex(void *info); extern SCatalogMgmt gCtgMgmt; diff --git a/source/libs/catalog/src/catalog.c b/source/libs/catalog/src/catalog.c index a3576f8738a13715430b4629fe68194511106dbc..bb02895569c180ea442992f60818e541a842eb8d 100644 --- a/source/libs/catalog/src/catalog.c +++ b/source/libs/catalog/src/catalog.c @@ -105,7 +105,7 @@ int32_t ctgRefreshDBVgInfo(SCatalog* pCtg, SRequestConnInfo *pConn, const char* code = ctgGetDBVgInfoFromMnode(pCtg, pConn, &input, &DbOut, NULL); if (code) { if (CTG_DB_NOT_EXIST(code) && (NULL != dbCache)) { - ctgDebug("db no longer exist, dbFName:%s, dbId:%" PRIx64, input.db, input.dbId); + ctgDebug("db no longer exist, dbFName:%s, dbId:0x%" PRIx64, input.db, input.dbId); ctgDropDbCacheEnqueue(pCtg, input.db, input.dbId); } @@ -571,7 +571,7 @@ int32_t catalogGetHandle(uint64_t clusterId, SCatalog** catalogHandle) { } if (NULL == gCtgMgmt.pCluster) { - qError("catalog cluster cache are not ready, clusterId:%" PRIx64, clusterId); + qError("catalog cluster cache are not ready, clusterId:0x%" PRIx64, clusterId); CTG_ERR_RET(TSDB_CODE_CTG_NOT_READY); } @@ -583,7 +583,7 @@ int32_t catalogGetHandle(uint64_t clusterId, SCatalog** catalogHandle) { if (ctg && (*ctg)) { *catalogHandle = *ctg; - qDebug("got catalog handle from cache, clusterId:%" PRIx64 ", CTG:%p", clusterId, *ctg); + qDebug("got catalog handle from cache, clusterId:0x%" PRIx64 ", CTG:%p", clusterId, *ctg); return TSDB_CODE_SUCCESS; } @@ -612,11 +612,11 @@ int32_t catalogGetHandle(uint64_t clusterId, SCatalog** catalogHandle) { continue; } - qError("taosHashPut CTG to cache failed, clusterId:%" PRIx64, clusterId); + qError("taosHashPut CTG to cache failed, clusterId:0x%" PRIx64, clusterId); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - qDebug("add CTG to cache, clusterId:%" PRIx64 ", CTG:%p", clusterId, clusterCtg); + qDebug("add CTG to cache, clusterId:0x%" PRIx64 ", CTG:%p", clusterId, clusterCtg); break; } @@ -640,7 +640,7 @@ void catalogFreeHandle(SCatalog* pCtg) { } if (taosHashRemove(gCtgMgmt.pCluster, &pCtg->clusterId, sizeof(pCtg->clusterId))) { - ctgWarn("taosHashRemove from cluster failed, may already be freed, clusterId:%" PRIx64, pCtg->clusterId); + ctgWarn("taosHashRemove from cluster failed, may already be freed, clusterId:0x%" PRIx64, pCtg->clusterId); return; } @@ -650,7 +650,7 @@ void catalogFreeHandle(SCatalog* pCtg) { ctgFreeHandle(pCtg); - ctgInfo("handle freed, culsterId:%" PRIx64, clusterId); + ctgInfo("handle freed, culsterId:0x%" PRIx64, clusterId); } int32_t catalogGetDBVgVersion(SCatalog* pCtg, const char* dbFName, int32_t* version, int64_t* dbId, int32_t* tableNum) { @@ -1247,6 +1247,23 @@ int32_t catalogUpdateUserAuthInfo(SCatalog* pCtg, SGetUserAuthRsp* pAuth) { CTG_API_LEAVE(ctgUpdateUserEnqueue(pCtg, pAuth, false)); } +int32_t catalogClearCache(void) { + CTG_API_ENTER(); + + qInfo("start to clear catalog cache"); + + if (NULL == gCtgMgmt.pCluster || atomic_load_8((int8_t*)&gCtgMgmt.exit)) { + CTG_API_LEAVE(TSDB_CODE_SUCCESS); + } + + int32_t code = ctgClearCacheEnqueue(NULL, true); + + qInfo("clear catalog cache end, code: %s", tstrerror(code)); + + CTG_API_LEAVE(code); +} + + void catalogDestroy(void) { qInfo("start to destroy catalog"); diff --git a/source/libs/catalog/src/ctgAsync.c b/source/libs/catalog/src/ctgAsync.c index 8dabd56934677917e11f0ccf2c97c7490087f7d7..6adadf50455950e5d49bf0a5a9e178673c6fcf6b 100644 --- a/source/libs/catalog/src/ctgAsync.c +++ b/source/libs/catalog/src/ctgAsync.c @@ -622,7 +622,7 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) { SCtgJob* pJob = pTask->pJob; int32_t code = 0; - qDebug("QID:0x%" PRIx64 " task %d end with rsp %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); + qDebug("QID:0x%" PRIx64 " task %d end with res %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); pTask->code = rspCode; @@ -1276,7 +1276,7 @@ int32_t ctgLaunchJob(SCtgJob *pJob) { for (int32_t i = 0; i < taskNum; ++i) { SCtgTask *pTask = taosArrayGet(pJob->pTasks, i); - qDebug("QID:0x%" PRIx64 " start to launch task %d", pJob->queryId, pTask->taskId); + qDebug("QID:0x%" PRIx64 " ctg start to launch task %d", pJob->queryId, pTask->taskId); CTG_ERR_RET((*gCtgAsyncFps[pTask->type].launchFp)(pTask)); } diff --git a/source/libs/catalog/src/ctgCache.c b/source/libs/catalog/src/ctgCache.c index 0948c01270996d11ba7da78c8d86327341d41093..1de5ee3d7d9ea6c3ce24b6890e12cfc0cb9c75d8 100644 --- a/source/libs/catalog/src/ctgCache.c +++ b/source/libs/catalog/src/ctgCache.c @@ -69,6 +69,11 @@ SCtgOperation gCtgCacheOperation[CTG_OP_MAX] = { CTG_OP_DROP_TB_INDEX, "drop tbIndex", ctgOpDropTbIndex + }, + { + CTG_OP_CLEAR_CACHE, + "clear cache", + ctgOpClearCache } }; @@ -81,7 +86,7 @@ int32_t ctgRLockVgInfo(SCatalog *pCtg, SCtgDBCache *dbCache, bool *inCache) { if (dbCache->deleted) { CTG_UNLOCK(CTG_READ, &dbCache->vgCache.vgLock); - ctgDebug("db is dropping, dbId:%"PRIx64, dbCache->dbId); + ctgDebug("db is dropping, dbId:0x%"PRIx64, dbCache->dbId); *inCache = false; return TSDB_CODE_SUCCESS; @@ -92,7 +97,7 @@ int32_t ctgRLockVgInfo(SCatalog *pCtg, SCtgDBCache *dbCache, bool *inCache) { CTG_UNLOCK(CTG_READ, &dbCache->vgCache.vgLock); *inCache = false; - ctgDebug("db vgInfo is empty, dbId:%"PRIx64, dbCache->dbId); + ctgDebug("db vgInfo is empty, dbId:0x%"PRIx64, dbCache->dbId); return TSDB_CODE_SUCCESS; } @@ -105,7 +110,7 @@ int32_t ctgWLockVgInfo(SCatalog *pCtg, SCtgDBCache *dbCache) { CTG_LOCK(CTG_WRITE, &dbCache->vgCache.vgLock); if (dbCache->deleted) { - ctgDebug("db is dropping, dbId:%"PRIx64, dbCache->dbId); + ctgDebug("db is dropping, dbId:0x%"PRIx64, dbCache->dbId); CTG_UNLOCK(CTG_WRITE, &dbCache->vgCache.vgLock); CTG_ERR_RET(TSDB_CODE_CTG_DB_DROPPED); } @@ -280,27 +285,27 @@ int32_t ctgAcquireStbMetaFromCache(SCatalog* pCtg, char *dbFName, uint64_t suid, int32_t sz = 0; char* stName = taosHashAcquire(dbCache->stbCache, &suid, sizeof(suid)); if (NULL == stName) { - ctgDebug("stb %" PRIx64 " not in cache, dbFName:%s", suid, dbFName); + ctgDebug("stb 0x%" PRIx64 " not in cache, dbFName:%s", suid, dbFName); goto _return; } pCache = taosHashAcquire(dbCache->tbCache, stName, strlen(stName)); if (NULL == pCache) { - ctgDebug("stb %" PRIx64 " name %s not in cache, dbFName:%s", suid, stName, dbFName); + ctgDebug("stb 0x%" PRIx64 " name %s not in cache, dbFName:%s", suid, stName, dbFName); taosHashRelease(dbCache->stbCache, stName); goto _return; } CTG_LOCK(CTG_READ, &pCache->metaLock); if (NULL == pCache->pMeta) { - ctgDebug("stb %" PRIx64 " meta not in cache, dbFName:%s", suid, dbFName); + ctgDebug("stb 0x%" PRIx64 " meta not in cache, dbFName:%s", suid, dbFName); goto _return; } *pDb = dbCache; *pTb = pCache; - ctgDebug("stb %" PRIx64 " meta got in cache, dbFName:%s", suid, dbFName); + ctgDebug("stb 0x%" PRIx64 " meta got in cache, dbFName:%s", suid, dbFName); CTG_CACHE_STAT_INC(tbMetaHitNum, 1); @@ -434,14 +439,14 @@ int32_t ctgReadTbMetaFromCache(SCatalog* pCtg, SCtgTbMetaCtx* ctx, STableMeta** if (NULL == tbCache) { ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache); taosMemoryFreeClear(*pTableMeta); - ctgDebug("stb %" PRIx64 " meta not in cache", ctx->tbInfo.suid); + ctgDebug("stb 0x%" PRIx64 " meta not in cache", ctx->tbInfo.suid); return TSDB_CODE_SUCCESS; } STableMeta* stbMeta = tbCache->pMeta; if (stbMeta->suid != ctx->tbInfo.suid) { ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache); - ctgError("stb suid %" PRIx64 " in stbCache mis-match, expected suid:%"PRIx64 , stbMeta->suid, ctx->tbInfo.suid); + ctgError("stb suid 0x%" PRIx64 " in stbCache mis-match, expected suid 0x%"PRIx64 , stbMeta->suid, ctx->tbInfo.suid); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -492,7 +497,7 @@ int32_t ctgReadTbVerFromCache(SCatalog *pCtg, SName *pTableName, int32_t *sver, *sver = tbMeta->sversion; *tver = tbMeta->tversion; - ctgDebug("Got tb %s ver from cache, dbFName:%s, tbType:%d, sver:%d, tver:%d, suid:%" PRIx64, + ctgDebug("Got tb %s ver from cache, dbFName:%s, tbType:%d, sver:%d, tver:%d, suid:0x%" PRIx64, pTableName->tname, dbFName, *tbType, *sver, *tver, *suid); ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache); @@ -507,14 +512,14 @@ int32_t ctgReadTbVerFromCache(SCatalog *pCtg, SName *pTableName, int32_t *sver, ctgAcquireStbMetaFromCache(pCtg, dbFName, *suid, &dbCache, &tbCache); if (NULL == tbCache) { ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache); - ctgDebug("stb %" PRIx64 " meta not in cache", *suid); + ctgDebug("stb 0x%" PRIx64 " meta not in cache", *suid); return TSDB_CODE_SUCCESS; } STableMeta* stbMeta = tbCache->pMeta; if (stbMeta->suid != *suid) { ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache); - ctgError("stb suid %" PRIx64 " in stbCache mis-match, expected suid:%" PRIx64 , stbMeta->suid, *suid); + ctgError("stb suid 0x%" PRIx64 " in stbCache mis-match, expected suid:0x%" PRIx64 , stbMeta->suid, *suid); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -990,6 +995,33 @@ _return: } +int32_t ctgClearCacheEnqueue(SCatalog* pCtg, bool syncOp) { + int32_t code = 0; + SCtgCacheOperation *op = taosMemoryCalloc(1, sizeof(SCtgCacheOperation)); + op->opId = CTG_OP_CLEAR_CACHE; + op->syncOp = syncOp; + + SCtgClearCacheMsg *msg = taosMemoryMalloc(sizeof(SCtgClearCacheMsg)); + if (NULL == msg) { + ctgError("malloc %d failed", (int32_t)sizeof(SCtgClearCacheMsg)); + CTG_ERR_RET(TSDB_CODE_CTG_MEM_ERROR); + } + + msg->pCtg = pCtg; + op->data = msg; + + CTG_ERR_JRET(ctgEnqueue(pCtg, op)); + + return TSDB_CODE_SUCCESS; + +_return: + + taosMemoryFreeClear(msg); + + CTG_RET(code); +} + + int32_t ctgMetaRentInit(SCtgRentMgmt *mgmt, uint32_t rentSec, int8_t type) { mgmt->slotRIdx = 0; mgmt->slotNum = rentSec / CTG_RENT_SLOT_SECOND; @@ -1019,19 +1051,19 @@ int32_t ctgMetaRentAdd(SCtgRentMgmt *mgmt, void *meta, int64_t id, int32_t size) if (NULL == slot->meta) { slot->meta = taosArrayInit(CTG_DEFAULT_RENT_SLOT_SIZE, size); if (NULL == slot->meta) { - qError("taosArrayInit %d failed, id:%"PRIx64", slot idx:%d, type:%d", CTG_DEFAULT_RENT_SLOT_SIZE, id, widx, mgmt->type); + qError("taosArrayInit %d failed, id:0x%"PRIx64", slot idx:%d, type:%d", CTG_DEFAULT_RENT_SLOT_SIZE, id, widx, mgmt->type); CTG_ERR_JRET(TSDB_CODE_CTG_MEM_ERROR); } } if (NULL == taosArrayPush(slot->meta, meta)) { - qError("taosArrayPush meta to rent failed, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qError("taosArrayPush meta to rent failed, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); CTG_ERR_JRET(TSDB_CODE_CTG_MEM_ERROR); } slot->needSort = true; - qDebug("add meta to rent, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qDebug("add meta to rent, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); _return: @@ -1047,7 +1079,7 @@ int32_t ctgMetaRentUpdate(SCtgRentMgmt *mgmt, void *meta, int64_t id, int32_t si CTG_LOCK(CTG_WRITE, &slot->lock); if (NULL == slot->meta) { - qError("empty meta slot, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qError("empty meta slot, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1060,20 +1092,20 @@ int32_t ctgMetaRentUpdate(SCtgRentMgmt *mgmt, void *meta, int64_t id, int32_t si void *orig = taosArraySearch(slot->meta, &id, searchCompare, TD_EQ); if (NULL == orig) { - qDebug("meta not found in slot, id:%"PRIx64", slot idx:%d, type:%d, size:%d", id, widx, mgmt->type, (int32_t)taosArrayGetSize(slot->meta)); + qDebug("meta not found in slot, id:0x%"PRIx64", slot idx:%d, type:%d, size:%d", id, widx, mgmt->type, (int32_t)taosArrayGetSize(slot->meta)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } memcpy(orig, meta, size); - qDebug("meta in rent updated, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qDebug("meta in rent updated, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); _return: CTG_UNLOCK(CTG_WRITE, &slot->lock); if (code) { - qDebug("meta in rent update failed, will try to add it, code:%x, id:%"PRIx64", slot idx:%d, type:%d", code, id, widx, mgmt->type); + qDebug("meta in rent update failed, will try to add it, code:%x, id:0x%"PRIx64", slot idx:%d, type:%d", code, id, widx, mgmt->type); CTG_RET(ctgMetaRentAdd(mgmt, meta, id, size)); } @@ -1088,7 +1120,7 @@ int32_t ctgMetaRentRemove(SCtgRentMgmt *mgmt, int64_t id, __compar_fn_t sortComp CTG_LOCK(CTG_WRITE, &slot->lock); if (NULL == slot->meta) { - qError("empty meta slot, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qError("empty meta slot, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1100,13 +1132,13 @@ int32_t ctgMetaRentRemove(SCtgRentMgmt *mgmt, int64_t id, __compar_fn_t sortComp int32_t idx = taosArraySearchIdx(slot->meta, &id, searchCompare, TD_EQ); if (idx < 0) { - qError("meta not found in slot, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qError("meta not found in slot, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } taosArrayRemove(slot->meta, idx); - qDebug("meta in rent removed, id:%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); + qDebug("meta in rent removed, id:0x%"PRIx64", slot idx:%d, type:%d", id, widx, mgmt->type); _return: @@ -1219,11 +1251,11 @@ int32_t ctgAddNewDBCache(SCatalog *pCtg, const char *dbFName, uint64_t dbId) { SDbVgVersion vgVersion = {.dbId = newDBCache.dbId, .vgVersion = -1}; strncpy(vgVersion.dbFName, dbFName, sizeof(vgVersion.dbFName)); - ctgDebug("db added to cache, dbFName:%s, dbId:%"PRIx64, dbFName, dbId); + ctgDebug("db added to cache, dbFName:%s, dbId:0x%"PRIx64, dbFName, dbId); CTG_ERR_RET(ctgMetaRentAdd(&pCtg->dbRent, &vgVersion, dbId, sizeof(SDbVgVersion))); - ctgDebug("db added to rent, dbFName:%s, vgVersion:%d, dbId:%"PRIx64, dbFName, vgVersion.vgVersion, dbId); + ctgDebug("db added to rent, dbFName:%s, vgVersion:%d, dbId:0x%"PRIx64, dbFName, vgVersion.vgVersion, dbId); return TSDB_CODE_SUCCESS; @@ -1246,7 +1278,7 @@ void ctgRemoveStbRent(SCatalog* pCtg, SCtgDBCache *dbCache) { suid = taosHashGetKey(pIter, NULL); if (TSDB_CODE_SUCCESS == ctgMetaRentRemove(&pCtg->stbRent, *suid, ctgStbVersionSortCompare, ctgStbVersionSearchCompare)) { - ctgDebug("stb removed from rent, suid:%"PRIx64, *suid); + ctgDebug("stb removed from rent, suid:0x%"PRIx64, *suid); } pIter = taosHashIterate(dbCache->stbCache, pIter); @@ -1257,7 +1289,7 @@ void ctgRemoveStbRent(SCatalog* pCtg, SCtgDBCache *dbCache) { int32_t ctgRemoveDBFromCache(SCatalog* pCtg, SCtgDBCache *dbCache, const char* dbFName) { uint64_t dbId = dbCache->dbId; - ctgInfo("start to remove db from cache, dbFName:%s, dbId:%"PRIx64, dbFName, dbCache->dbId); + ctgInfo("start to remove db from cache, dbFName:%s, dbId:0x%"PRIx64, dbFName, dbCache->dbId); CTG_LOCK(CTG_WRITE, &dbCache->dbLock); @@ -1268,7 +1300,7 @@ int32_t ctgRemoveDBFromCache(SCatalog* pCtg, SCtgDBCache *dbCache, const char* d CTG_UNLOCK(CTG_WRITE, &dbCache->dbLock); CTG_ERR_RET(ctgMetaRentRemove(&pCtg->dbRent, dbId, ctgDbVgVersionSortCompare, ctgDbVgVersionSearchCompare)); - ctgDebug("db removed from rent, dbFName:%s, dbId:%"PRIx64, dbFName, dbId); + ctgDebug("db removed from rent, dbFName:%s, dbId:0x%"PRIx64, dbFName, dbId); if (taosHashRemove(pCtg->dbCache, dbFName, strlen(dbFName))) { ctgInfo("taosHashRemove from dbCache failed, may be removed, dbFName:%s", dbFName); @@ -1276,7 +1308,7 @@ int32_t ctgRemoveDBFromCache(SCatalog* pCtg, SCtgDBCache *dbCache, const char* d } CTG_CACHE_STAT_DEC(dbNum, 1); - ctgInfo("db removed from cache, dbFName:%s, dbId:%"PRIx64, dbFName, dbId); + ctgInfo("db removed from cache, dbFName:%s, dbId:0x%"PRIx64, dbFName, dbId); return TSDB_CODE_SUCCESS; } @@ -1339,7 +1371,7 @@ int32_t ctgUpdateRentStbVersion(SCatalog *pCtg, char* dbFName, char* tbName, uin CTG_ERR_RET(ctgMetaRentUpdate(&pCtg->stbRent, &metaRent, metaRent.suid, sizeof(SSTableVersion), ctgStbVersionSortCompare, ctgStbVersionSearchCompare)); - ctgDebug("db %s,%" PRIx64 " stb %s,%" PRIx64 " sver %d tver %d smaVer %d updated to stbRent", + ctgDebug("db %s,0x%" PRIx64 " stb %s,0x%" PRIx64 " sver %d tver %d smaVer %d updated to stbRent", dbFName, dbId, tbName, suid, metaRent.sversion, metaRent.tversion, metaRent.smaVer); return TSDB_CODE_SUCCESS; @@ -1349,7 +1381,7 @@ int32_t ctgUpdateRentStbVersion(SCatalog *pCtg, char* dbFName, char* tbName, uin int32_t ctgWriteTbMetaToCache(SCatalog *pCtg, SCtgDBCache *dbCache, char *dbFName, uint64_t dbId, char *tbName, STableMeta *meta, int32_t metaSize) { if (NULL == dbCache->tbCache || NULL == dbCache->stbCache) { taosMemoryFree(meta); - ctgError("db is dropping, dbId:%"PRIx64, dbCache->dbId); + ctgError("db is dropping, dbId:0x%"PRIx64, dbCache->dbId); CTG_ERR_RET(TSDB_CODE_CTG_DB_DROPPED); } @@ -1370,10 +1402,10 @@ int32_t ctgWriteTbMetaToCache(SCatalog *pCtg, SCtgDBCache *dbCache, char *dbFNam if (origType == TSDB_SUPER_TABLE) { if (taosHashRemove(dbCache->stbCache, &orig->suid, sizeof(orig->suid))) { - ctgError("stb not exist in stbCache, dbFName:%s, stb:%s, suid:%"PRIx64, dbFName, tbName, orig->suid); + ctgError("stb not exist in stbCache, dbFName:%s, stb:%s, suid:0x%"PRIx64, dbFName, tbName, orig->suid); } else { CTG_CACHE_STAT_DEC(stblNum, 1); - ctgDebug("stb removed from stbCache, dbFName:%s, stb:%s, suid:%"PRIx64, dbFName, tbName, orig->suid); + ctgDebug("stb removed from stbCache, dbFName:%s, stb:%s, suid:0x%"PRIx64, dbFName, tbName, orig->suid); } origSuid = orig->suid; @@ -1407,13 +1439,13 @@ int32_t ctgWriteTbMetaToCache(SCatalog *pCtg, SCtgDBCache *dbCache, char *dbFNam } if (origSuid != meta->suid && taosHashPut(dbCache->stbCache, &meta->suid, sizeof(meta->suid), tbName, strlen(tbName) + 1) != 0) { - ctgError("taosHashPut to stable cache failed, suid:%"PRIx64, meta->suid); + ctgError("taosHashPut to stable cache failed, suid:0x%"PRIx64, meta->suid); CTG_ERR_RET(TSDB_CODE_CTG_MEM_ERROR); } CTG_CACHE_STAT_INC(stblNum, 1); - ctgDebug("stb %" PRIx64 " updated to cache, dbFName:%s, tbName:%s, tbType:%d", meta->suid, dbFName, tbName, meta->tableType); + ctgDebug("stb 0x%" PRIx64 " updated to cache, dbFName:%s, tbName:%s, tbType:%d", meta->suid, dbFName, tbName, meta->tableType); CTG_ERR_RET(ctgUpdateRentStbVersion(pCtg, dbFName, tbName, dbId, meta->suid, pCache)); @@ -1424,7 +1456,7 @@ int32_t ctgWriteTbIndexToCache(SCatalog *pCtg, SCtgDBCache *dbCache, char* dbFNa if (NULL == dbCache->tbCache) { ctgFreeSTableIndex(*index); taosMemoryFreeClear(*index); - ctgError("db is dropping, dbId:%"PRIx64, dbCache->dbId); + ctgError("db is dropping, dbId:0x%"PRIx64, dbCache->dbId); CTG_ERR_RET(TSDB_CODE_CTG_DB_DROPPED); } @@ -1510,7 +1542,7 @@ int32_t ctgOpUpdateVgroup(SCtgCacheOperation *operation) { SCtgDBCache *dbCache = NULL; CTG_ERR_RET(ctgGetAddDBCache(msg->pCtg, dbFName, msg->dbId, &dbCache)); if (NULL == dbCache) { - ctgInfo("conflict db update, ignore this update, dbFName:%s, dbId:%"PRIx64, dbFName, msg->dbId); + ctgInfo("conflict db update, ignore this update, dbFName:%s, dbId:0x%"PRIx64, dbFName, msg->dbId); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1540,7 +1572,7 @@ int32_t ctgOpUpdateVgroup(SCtgCacheOperation *operation) { vgCache->vgInfo = dbInfo; msg->dbInfo = NULL; - ctgDebug("db vgInfo updated, dbFName:%s, vgVer:%d, dbId:%"PRIx64, dbFName, vgVersion.vgVersion, vgVersion.dbId); + ctgDebug("db vgInfo updated, dbFName:%s, vgVer:%d, dbId:0x%"PRIx64, dbFName, vgVersion.vgVersion, vgVersion.dbId); ctgWUnlockVgInfo(dbCache); @@ -1569,7 +1601,7 @@ int32_t ctgOpDropDbCache(SCtgCacheOperation *operation) { } if (dbCache->dbId != msg->dbId) { - ctgInfo("dbId already updated, dbFName:%s, dbId:%"PRIx64 ", targetId:%"PRIx64, msg->dbFName, dbCache->dbId, msg->dbId); + ctgInfo("dbId already updated, dbFName:%s, dbId:0x%"PRIx64 ", targetId:0x%"PRIx64, msg->dbFName, dbCache->dbId, msg->dbId); goto _return; } @@ -1629,7 +1661,7 @@ int32_t ctgOpUpdateTbMeta(SCtgCacheOperation *operation) { CTG_ERR_JRET(ctgGetAddDBCache(pCtg, pMeta->dbFName, pMeta->dbId, &dbCache)); if (NULL == dbCache) { - ctgInfo("conflict db update, ignore this update, dbFName:%s, dbId:%" PRIx64, pMeta->dbFName, pMeta->dbId); + ctgInfo("conflict db update, ignore this update, dbFName:%s, dbId:0x%" PRIx64, pMeta->dbFName, pMeta->dbId); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1673,27 +1705,28 @@ int32_t ctgOpDropStbMeta(SCtgCacheOperation *operation) { } if (msg->dbId && (dbCache->dbId != msg->dbId)) { - ctgDebug("dbId already modified, dbFName:%s, current:%"PRIx64", dbId:%"PRIx64", stb:%s, suid:%"PRIx64, msg->dbFName, dbCache->dbId, msg->dbId, msg->stbName, msg->suid); + ctgDebug("dbId already modified, dbFName:%s, current:0x%"PRIx64", dbId:0x%"PRIx64", stb:%s, suid:0x%"PRIx64, + msg->dbFName, dbCache->dbId, msg->dbId, msg->stbName, msg->suid); return TSDB_CODE_SUCCESS; } if (taosHashRemove(dbCache->stbCache, &msg->suid, sizeof(msg->suid))) { - ctgDebug("stb not exist in stbCache, may be removed, dbFName:%s, stb:%s, suid:%"PRIx64, msg->dbFName, msg->stbName, msg->suid); + ctgDebug("stb not exist in stbCache, may be removed, dbFName:%s, stb:%s, suid:0x%"PRIx64, msg->dbFName, msg->stbName, msg->suid); } else { CTG_CACHE_STAT_DEC(stblNum, 1); } if (taosHashRemove(dbCache->tbCache, msg->stbName, strlen(msg->stbName))) { - ctgError("stb not exist in cache, dbFName:%s, stb:%s, suid:%"PRIx64, msg->dbFName, msg->stbName, msg->suid); + ctgError("stb not exist in cache, dbFName:%s, stb:%s, suid:0x%"PRIx64, msg->dbFName, msg->stbName, msg->suid); } else { CTG_CACHE_STAT_DEC(tblNum, 1); } - ctgInfo("stb removed from cache, dbFName:%s, stbName:%s, suid:%"PRIx64, msg->dbFName, msg->stbName, msg->suid); + ctgInfo("stb removed from cache, dbFName:%s, stbName:%s, suid:0x%"PRIx64, msg->dbFName, msg->stbName, msg->suid); CTG_ERR_JRET(ctgMetaRentRemove(&msg->pCtg->stbRent, msg->suid, ctgStbVersionSortCompare, ctgStbVersionSearchCompare)); - ctgDebug("stb removed from rent, dbFName:%s, stbName:%s, suid:%"PRIx64, msg->dbFName, msg->stbName, msg->suid); + ctgDebug("stb removed from rent, dbFName:%s, stbName:%s, suid:0x%"PRIx64, msg->dbFName, msg->stbName, msg->suid); _return: @@ -1714,7 +1747,7 @@ int32_t ctgOpDropTbMeta(SCtgCacheOperation *operation) { } if (dbCache->dbId != msg->dbId) { - ctgDebug("dbId %" PRIx64 " not match with curId %"PRIx64", dbFName:%s, tbName:%s", msg->dbId, dbCache->dbId, msg->dbFName, msg->tbName); + ctgDebug("dbId 0x%" PRIx64 " not match with curId 0x%"PRIx64", dbFName:%s, tbName:%s", msg->dbId, dbCache->dbId, msg->dbFName, msg->tbName); return TSDB_CODE_SUCCESS; } @@ -1898,6 +1931,37 @@ _return: } +int32_t ctgOpClearCache(SCtgCacheOperation *operation) { + int32_t code = 0; + SCtgClearCacheMsg *msg = operation->data; + SCatalog* pCtg = msg->pCtg; + + if (pCtg) { + catalogFreeHandle(pCtg); + goto _return; + } + + void* pIter = taosHashIterate(gCtgMgmt.pCluster, NULL); + while (pIter) { + pCtg = *(SCatalog**)pIter; + + if (pCtg) { + catalogFreeHandle(pCtg); + } + + pIter = taosHashIterate(gCtgMgmt.pCluster, pIter); + } + + taosHashClear(gCtgMgmt.pCluster); + +_return: + + taosMemoryFreeClear(msg); + + CTG_RET(code); +} + + void ctgUpdateThreadUnexpectedStopped(void) { if (!atomic_load_8((int8_t*)&gCtgMgmt.exit) && CTG_IS_LOCKED(&gCtgMgmt.lock) > 0) CTG_UNLOCK(CTG_READ, &gCtgMgmt.lock); } @@ -1971,11 +2035,10 @@ void* ctgUpdateThreadFunc(void* param) { CTG_RT_STAT_INC(qDoneNum, 1); + ctgdShowCacheInfo(); ctgdShowClusterCache(pCtg); } - if (CTG_IS_LOCKED(&gCtgMgmt.lock)) CTG_UNLOCK(CTG_READ, &gCtgMgmt.lock); - qInfo("catalog update thread stopped"); return NULL; diff --git a/source/libs/catalog/src/ctgDbg.c b/source/libs/catalog/src/ctgDbg.c index ff93bedb213607c3592b057c12beb2ed2f733d54..7f2b919f175adfa79a5bc3916813c8b6619e1b3f 100644 --- a/source/libs/catalog/src/ctgDbg.c +++ b/source/libs/catalog/src/ctgDbg.c @@ -19,7 +19,7 @@ #include "catalogInt.h" extern SCatalogMgmt gCtgMgmt; -SCtgDebug gCTGDebug = {.lockEnable = true, .apiEnable = true}; +SCtgDebug gCTGDebug = {.cacheEnable = true}; void ctgdUserCallback(SMetaData* pResult, void* param, int32_t code) { ASSERT(*(int32_t*)param == 1); @@ -40,9 +40,9 @@ void ctgdUserCallback(SMetaData* pResult, void* param, int32_t code) { STableComInfo *c = &p->tableInfo; if (TSDB_CHILD_TABLE == p->tableType) { - qDebug("table meta: type:%d, vgId:%d, uid:%" PRIx64 ",suid:%" PRIx64, p->tableType, p->vgId, p->uid, p->suid); + qDebug("table meta: type:%d, vgId:%d, uid:0x%" PRIx64 ",suid:0x%" PRIx64, p->tableType, p->vgId, p->uid, p->suid); } else { - qDebug("table meta: type:%d, vgId:%d, uid:%" PRIx64 ",suid:%" PRIx64 ",sv:%d, tv:%d, tagNum:%d, precision:%d, colNum:%d, rowSize:%d", + qDebug("table meta: type:%d, vgId:%d, uid:0x%" PRIx64 ",suid:0x%" PRIx64 ",sv:%d, tv:%d, tagNum:%d, precision:%d, colNum:%d, rowSize:%d", p->tableType, p->vgId, p->uid, p->suid, p->sversion, p->tversion, c->numOfTags, c->precision, c->numOfColumns, c->rowSize); } @@ -75,7 +75,7 @@ void ctgdUserCallback(SMetaData* pResult, void* param, int32_t code) { num = taosArrayGetSize(pResult->pDbInfo); for (int32_t i = 0; i < num; ++i) { SDbInfo *pDb = taosArrayGet(pResult->pDbInfo, i); - qDebug("db %d dbInfo: vgVer:%d, tbNum:%d, dbId:%" PRIx64, i, pDb->vgVer, pDb->tbNum, pDb->dbId); + qDebug("db %d dbInfo: vgVer:%d, tbNum:%d, dbId:0x%" PRIx64, i, pDb->vgVer, pDb->tbNum, pDb->dbId); } } else { qDebug("empty db info"); @@ -333,10 +333,10 @@ void ctgdShowTableMeta(SCatalog* pCtg, const char *tbName, STableMeta* p) { STableComInfo *c = &p->tableInfo; if (TSDB_CHILD_TABLE == p->tableType) { - ctgDebug("table [%s] meta: type:%d, vgId:%d, uid:%" PRIx64 ",suid:%" PRIx64, tbName, p->tableType, p->vgId, p->uid, p->suid); + ctgDebug("table [%s] meta: type:%d, vgId:%d, uid:0x%" PRIx64 ",suid:0x%" PRIx64, tbName, p->tableType, p->vgId, p->uid, p->suid); return; } else { - ctgDebug("table [%s] meta: type:%d, vgId:%d, uid:%" PRIx64 ",suid:%" PRIx64 ",sv:%d, tv:%d, tagNum:%d, precision:%d, colNum:%d, rowSize:%d", + ctgDebug("table [%s] meta: type:%d, vgId:%d, uid:0x%" PRIx64 ",suid:0x%" PRIx64 ",sv:%d, tv:%d, tagNum:%d, precision:%d, colNum:%d, rowSize:%d", tbName, p->tableType, p->vgId, p->uid, p->suid, p->sversion, p->tversion, c->numOfTags, c->precision, c->numOfColumns, c->rowSize); } @@ -377,7 +377,7 @@ void ctgdShowDBCache(SCatalog* pCtg, SHashObj *dbHash) { } } - ctgDebug("[%d] db [%.*s][%"PRIx64"] %s: metaNum:%d, stbNum:%d, vgVersion:%d, hashMethod:%d, vgNum:%d", + ctgDebug("[%d] db [%.*s][0x%"PRIx64"] %s: metaNum:%d, stbNum:%d, vgVersion:%d, hashMethod:%d, vgNum:%d", i, (int32_t)len, dbFName, dbCache->dbId, dbCache->deleted?"deleted":"", metaNum, stbNum, vgVersion, hashMethod, vgNum); pIter = taosHashIterate(dbHash, pIter); @@ -392,13 +392,13 @@ void ctgdShowClusterCache(SCatalog* pCtg) { return; } - ctgDebug("## cluster %"PRIx64" %p cache Info BEGIN ##", pCtg->clusterId, pCtg); + ctgDebug("## cluster 0x%"PRIx64" %p cache Info BEGIN ##", pCtg->clusterId, pCtg); ctgDebug("db:%d meta:%d stb:%d dbRent:%d stbRent:%d", ctgdGetClusterCacheNum(pCtg, CTG_DBG_DB_NUM), ctgdGetClusterCacheNum(pCtg, CTG_DBG_META_NUM), ctgdGetClusterCacheNum(pCtg, CTG_DBG_STB_NUM), ctgdGetClusterCacheNum(pCtg, CTG_DBG_DB_RENT_NUM), ctgdGetClusterCacheNum(pCtg, CTG_DBG_STB_RENT_NUM)); ctgdShowDBCache(pCtg, pCtg->dbCache); - ctgDebug("## cluster %"PRIx64" %p cache Info END ##", pCtg->clusterId, pCtg); + ctgDebug("## cluster 0x%"PRIx64" %p cache Info END ##", pCtg->clusterId, pCtg); } int32_t ctgdShowCacheInfo(void) { @@ -407,6 +407,8 @@ int32_t ctgdShowCacheInfo(void) { } CTG_API_ENTER(); + + qDebug("# total catalog cluster number %d #", taosHashGetSize(gCtgMgmt.pCluster)); SCatalog *pCtg = NULL; void *pIter = taosHashIterate(gCtgMgmt.pCluster, NULL); diff --git a/source/libs/catalog/src/ctgRemote.c b/source/libs/catalog/src/ctgRemote.c index 178025704f7e91045e8970844e0d5590baa5a052..fa1a26283286d01cc4574c4babbb28ad5fb36996 100644 --- a/source/libs/catalog/src/ctgRemote.c +++ b/source/libs/catalog/src/ctgRemote.c @@ -186,13 +186,13 @@ int32_t ctgHandleMsgCallback(void *param, const SDataBuf *pMsg, int32_t rspCode) SCtgJob* pJob = taosAcquireRef(gCtgMgmt.jobPool, cbParam->refId); if (NULL == pJob) { - qDebug("job refId %" PRIx64 " already dropped", cbParam->refId); + qDebug("ctg job refId 0x%" PRIx64 " already dropped", cbParam->refId); goto _return; } SCtgTask *pTask = taosArrayGet(pJob->pTasks, cbParam->taskId); - qDebug("QID:0x%" PRIx64 " task %d start to handle rsp %s", pJob->queryId, pTask->taskId, TMSG_INFO(cbParam->reqType + 1)); + qDebug("QID:0x%" PRIx64 " ctg task %d start to handle rsp %s", pJob->queryId, pTask->taskId, TMSG_INFO(cbParam->reqType + 1)); CTG_ERR_JRET((*gCtgAsyncFps[pTask->type].handleRspFp)(pTask, cbParam->reqType, pMsg, rspCode)); @@ -263,7 +263,7 @@ int32_t ctgAsyncSendMsg(SCatalog* pCtg, SRequestConnInfo *pConn, SCtgTask* pTask CTG_ERR_JRET(code); } - ctgDebug("req msg sent, reqId:0x%" PRIx64 ", msg type:%d, %s", pTask->pJob->queryId, msgType, TMSG_INFO(msgType)); + ctgDebug("ctg req msg sent, reqId:0x%" PRIx64 ", msg type:%d, %s", pTask->pJob->queryId, msgType, TMSG_INFO(msgType)); return TSDB_CODE_SUCCESS; _return: diff --git a/source/libs/catalog/src/ctgUtil.c b/source/libs/catalog/src/ctgUtil.c index e97c34dc268b61b402304aeb51689a99926484ec..476eb371b057e27a2420dd6b35582b58db8aca72 100644 --- a/source/libs/catalog/src/ctgUtil.c +++ b/source/libs/catalog/src/ctgUtil.c @@ -434,7 +434,7 @@ void ctgFreeJob(void* job) { taosMemoryFree(job); - qDebug("QID:%" PRIx64 ", job %" PRIx64 " freed", qid, rid); + qDebug("QID:0x%" PRIx64 ", ctg job 0x%" PRIx64 " freed", qid, rid); } int32_t ctgUpdateMsgCtx(SCtgMsgCtx* pCtx, int32_t reqType, void* out, char* target) { diff --git a/source/libs/command/src/command.c b/source/libs/command/src/command.c index f330b7ce162f9c79189a66c0772fffbc96cb8dcc..c6a0e52a42f6238c7ee3b2cd3af714408debafa2 100644 --- a/source/libs/command/src/command.c +++ b/source/libs/command/src/command.c @@ -14,6 +14,7 @@ */ #include "command.h" +#include "catalog.h" #include "tdatablock.h" static int32_t getSchemaBytes(const SSchema* pSchema) { @@ -46,7 +47,8 @@ static SSDataBlock* buildDescResultDataBlock() { taosArrayPush(pBlock->pDataBlock, &infoData); infoData.info.type = TSDB_DATA_TYPE_INT; - infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes;; + infoData.info.bytes = tDataTypes[TSDB_DATA_TYPE_INT].bytes; + taosArrayPush(pBlock->pDataBlock, &infoData); infoData.info.type = TSDB_DATA_TYPE_VARCHAR; @@ -62,7 +64,7 @@ static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, S // field SColumnInfoData* pCol1 = taosArrayGet(pBlock->pDataBlock, 0); - char buf[DESCRIBE_RESULT_FIELD_LEN] = {0}; + char buf[DESCRIBE_RESULT_FIELD_LEN] = {0}; for (int32_t i = 0; i < numOfRows; ++i) { STR_TO_VARSTR(buf, pMeta->schema[i].name); colDataAppend(pCol1, i, buf, false); @@ -91,8 +93,8 @@ static void setDescResultIntoDataBlock(SSDataBlock* pBlock, int32_t numOfRows, S } static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) { - SDescribeStmt* pDesc = (SDescribeStmt*) pStmt; - int32_t numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta); + SDescribeStmt* pDesc = (SDescribeStmt*)pStmt; + int32_t numOfRows = TABLE_TOTAL_COL_NUM(pDesc->pMeta); SSDataBlock* pBlock = buildDescResultDataBlock(); setDescResultIntoDataBlock(pBlock, numOfRows, pDesc->pMeta); @@ -119,10 +121,15 @@ static int32_t execDescribe(SNode* pStmt, SRetrieveTableRsp** pRsp) { return TSDB_CODE_SUCCESS; } -static int32_t execResetQueryCache() { - // todo - return TSDB_CODE_SUCCESS; -} +static int32_t execResetQueryCache() { return catalogClearCache(); } + +static int32_t execShowCreateDatabase(SShowCreateDatabaseStmt* pStmt) { return TSDB_CODE_FAILED; } + +static int32_t execShowCreateTable(SShowCreateTableStmt* pStmt) { return TSDB_CODE_FAILED; } + +static int32_t execShowCreateSTable(SShowCreateTableStmt* pStmt) { return TSDB_CODE_FAILED; } + +static int32_t execAlterLocal(SAlterLocalStmt* pStmt) { return TSDB_CODE_FAILED; } int32_t qExecCommand(SNode* pStmt, SRetrieveTableRsp** pRsp) { switch (nodeType(pStmt)) { @@ -130,6 +137,14 @@ int32_t qExecCommand(SNode* pStmt, SRetrieveTableRsp** pRsp) { return execDescribe(pStmt, pRsp); case QUERY_NODE_RESET_QUERY_CACHE_STMT: return execResetQueryCache(); + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + return execShowCreateDatabase((SShowCreateDatabaseStmt*)pStmt); + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + return execShowCreateTable((SShowCreateTableStmt*)pStmt); + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: + return execShowCreateSTable((SShowCreateTableStmt*)pStmt); + case QUERY_NODE_ALTER_LOCAL_STMT: + return execAlterLocal((SAlterLocalStmt*)pStmt); default: break; } diff --git a/source/libs/command/src/explain.c b/source/libs/command/src/explain.c index 79861dfa05549a2f5faaa903b7d01bedfe73cd43..ae0f669d656a458a310339e461ad2e55918ee211 100644 --- a/source/libs/command/src/explain.c +++ b/source/libs/command/src/explain.c @@ -194,6 +194,11 @@ int32_t qExplainGenerateResChildren(SPhysiNode *pNode, SExplainGroup *group, SNo pPhysiChildren = fillPhysiNode->node.pChildren; break; } + case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN: { + STableMergeScanPhysiNode *mergePhysiNode = (STableMergeScanPhysiNode *)pNode; + pPhysiChildren = mergePhysiNode->scan.node.pChildren; + break; + } default: qError("not supported physical node type %d", pNode->type); QRY_ERR_RET(TSDB_CODE_QRY_APP_ERROR); @@ -398,6 +403,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i break; } case QUERY_NODE_PHYSICAL_PLAN_TABLE_SEQ_SCAN: + case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN: case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN: { STableScanPhysiNode *pTblScanNode = (STableScanPhysiNode *)pNode; EXPLAIN_ROW_NEW(level, EXPLAIN_TBL_SCAN_FORMAT, pTblScanNode->scan.tableName.tname); @@ -406,6 +412,7 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i QRY_ERR_RET(qExplainBufAppendExecInfo(pResNode->pExecInfo, tbuf, &tlen)); EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); } + EXPLAIN_ROW_APPEND(EXPLAIN_COLUMNS_FORMAT, pTblScanNode->scan.pScanCols->length); EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pTblScanNode->scan.node.pOutputDataBlockDesc->totalRowSize); @@ -420,27 +427,57 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i EXPLAIN_ROW_NEW(level + 1, "I/O: "); int32_t nodeNum = taosArrayGetSize(pResNode->pExecInfo); - for (int32_t i = 0; i < nodeNum; ++i) { + struct STableScanAnalyzeInfo info = {0}; + + int32_t maxIndex = 0; + int32_t totalRows = 0; + for(int32_t i = 0; i < nodeNum; ++i) { SExplainExecInfo *execInfo = taosArrayGet(pResNode->pExecInfo, i); STableScanAnalyzeInfo *pScanInfo = (STableScanAnalyzeInfo *)execInfo->verboseInfo; - EXPLAIN_ROW_APPEND("total_blocks=%d", pScanInfo->totalBlocks); - EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + info.totalBlocks += pScanInfo->totalBlocks; + info.loadBlocks += pScanInfo->loadBlocks; + info.totalRows += pScanInfo->totalRows; + info.skipBlocks += pScanInfo->skipBlocks; + info.filterTime += pScanInfo->filterTime; + info.loadBlockStatis += pScanInfo->loadBlockStatis; + info.totalCheckedRows += pScanInfo->totalCheckedRows; + info.filterOutBlocks += pScanInfo->filterOutBlocks; + + if (pScanInfo->totalRows > totalRows) { + totalRows = pScanInfo->totalRows; + maxIndex = i; + } + } - EXPLAIN_ROW_APPEND("load_blocks=%d", pScanInfo->loadBlocks); - EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + EXPLAIN_ROW_APPEND("total_blocks=%.1f", ((double)info.totalBlocks) / nodeNum); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); - EXPLAIN_ROW_APPEND("load_block_SMAs=%d", pScanInfo->loadBlockStatis); - EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + EXPLAIN_ROW_APPEND("load_blocks=%.1f", ((double)info.loadBlocks) / nodeNum); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); - EXPLAIN_ROW_APPEND("total_rows=%" PRIu64, pScanInfo->totalRows); - EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + EXPLAIN_ROW_APPEND("load_block_SMAs=%.1f", ((double)info.loadBlockStatis) / nodeNum); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); - EXPLAIN_ROW_APPEND("check_rows=%" PRIu64, pScanInfo->totalCheckedRows); - EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); - } + EXPLAIN_ROW_APPEND("total_rows=%.1f", ((double)info.totalRows) / nodeNum); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + EXPLAIN_ROW_APPEND("check_rows=%.1f", ((double)info.totalCheckedRows) / nodeNum); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); EXPLAIN_ROW_END(); + + QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1)); + + //Rows out: Avg 4166.7 rows x 24 workers. Max 4187 rows (seg7) with 0.220 ms to first row, 1.738 ms to end, start offset by 1.470 ms. + SExplainExecInfo *execInfo = taosArrayGet(pResNode->pExecInfo, maxIndex); + STableScanAnalyzeInfo *p1 = (STableScanAnalyzeInfo *)execInfo->verboseInfo; + + EXPLAIN_ROW_NEW(level + 1, " "); + EXPLAIN_ROW_APPEND("max_row_task=%d, total_rows:%" PRId64 ", ep:%s (cost=%.3f..%.3f)", maxIndex, p1->totalRows, "tbd", + execInfo->startupCost, execInfo->totalCost); + EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); + EXPLAIN_ROW_END(); + QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1)); } @@ -625,8 +662,8 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i QRY_ERR_RET(qExplainBufAppendExecInfo(pResNode->pExecInfo, tbuf, &tlen)); EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); } - if (pIndefNode->pVectorFuncs) { - EXPLAIN_ROW_APPEND(EXPLAIN_FUNCTIONS_FORMAT, pIndefNode->pVectorFuncs->length); + if (pIndefNode->pFuncs) { + EXPLAIN_ROW_APPEND(EXPLAIN_FUNCTIONS_FORMAT, pIndefNode->pFuncs->length); EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT); } EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pIndefNode->node.pOutputDataBlockDesc->totalRowSize); diff --git a/source/libs/executor/inc/executil.h b/source/libs/executor/inc/executil.h index a70cff555297ec3ee3b3e497dc38d65c1d5e460d..1117be5db0fd0002b8a951f1ededd185b1c3f30a 100644 --- a/source/libs/executor/inc/executil.h +++ b/source/libs/executor/inc/executil.h @@ -15,7 +15,9 @@ #ifndef TDENGINE_QUERYUTIL_H #define TDENGINE_QUERYUTIL_H -#include +#include "function.h" +#include "nodes.h" +#include "plannodes.h" #include "tbuffer.h" #include "tcommon.h" #include "tpagedbuf.h" @@ -77,7 +79,7 @@ typedef struct SResultRowInfo { struct SqlFunctionCtx; size_t getResultRowSize(struct SqlFunctionCtx* pCtx, int32_t numOfOutput); -int32_t initResultRowInfo(SResultRowInfo* pResultRowInfo, int32_t size); +void initResultRowInfo(SResultRowInfo* pResultRowInfo); void cleanupResultRowInfo(SResultRowInfo* pResultRowInfo); void closeAllResultRows(SResultRowInfo* pResultRowInfo); @@ -86,7 +88,7 @@ void initResultRow(SResultRow *pResultRow); void closeResultRow(SResultRow* pResultRow); bool isResultRowClosed(SResultRow* pResultRow); -struct SResultRowEntryInfo* getResultCell(const SResultRow* pRow, int32_t index, const int32_t* offset); +struct SResultRowEntryInfo* getResultEntryInfo(const SResultRow* pRow, int32_t index, const int32_t* offset); static FORCE_INLINE SResultRow *getResultRowByPos(SDiskbasedBuf* pBuf, SResultRowPosition* pos) { SFilePage* bufPage = (SFilePage*) getBufPage(pBuf, pos->pageId); @@ -98,9 +100,27 @@ void initGroupedResultInfo(SGroupResInfo* pGroupResInfo, SHashObj* pHashmap, void initMultiResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayList); void cleanupGroupResInfo(SGroupResInfo* pGroupResInfo); -bool hashRemainDataInGroupInfo(SGroupResInfo* pGroupResInfo); +bool hasDataInGroupInfo(SGroupResInfo* pGroupResInfo); -bool incNextGroup(SGroupResInfo* pGroupResInfo); int32_t getNumOfTotalRes(SGroupResInfo* pGroupResInfo); +SSDataBlock* createResDataBlock(SDataBlockDescNode* pNode); + +int32_t getTableList(void* metaHandle, SScanPhysiNode* pScanNode, STableListInfo* pListInfo, SNode* pTagCond); +SArray* createSortInfo(SNodeList* pNodeList); +SArray* extractPartitionColInfo(SNodeList* pNodeList); +SArray* extractColMatchInfo(SNodeList* pNodeList, SDataBlockDescNode* pOutputNodeList, int32_t* numOfOutputCols, int32_t type); + +SExprInfo* createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, int32_t* numOfExprs); + +SqlFunctionCtx* createSqlFunctionCtx(SExprInfo* pExprInfo, int32_t numOfOutput, int32_t** rowEntryInfoOffset); +void relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray* pCols, bool outputEveryColumn); +void initExecTimeWindowInfo(SColumnInfoData* pColData, STimeWindow* pQueryWindow); + +SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode); +SColumn extractColumnFromColumnNode(SColumnNode* pColNode); + +int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode); +void cleanupQueryTableDataCond(SQueryTableDataCond* pCond); + #endif // TDENGINE_QUERYUTIL_H diff --git a/source/libs/executor/inc/executorimpl.h b/source/libs/executor/inc/executorimpl.h index 034e2893df17a64fb1514d3b56eae6c305a45dee..9e6613d3c297d67e5f19f6af37608d6ba9148cc0 100644 --- a/source/libs/executor/inc/executorimpl.h +++ b/source/libs/executor/inc/executorimpl.h @@ -49,8 +49,6 @@ typedef int32_t (*__block_search_fn_t)(char* data, int32_t num, int64_t key, int #define Q_STATUS_EQUAL(p, s) (((p) & (s)) != 0u) #define QUERY_IS_ASC_QUERY(q) (GET_FORWARD_DIRECTION_FACTOR((q)->order.order) == QUERY_ASC_FORWARD_STEP) -//#define GET_TABLEGROUP(q, _index) ((SArray*)taosArrayGetP((q)->tableqinfoGroupInfo.pGroupList, (_index))) - #define NEEDTO_COMPRESS_QUERY(size) ((size) > tsCompressColData ? 1 : 0) enum { @@ -64,11 +62,6 @@ enum { TASK_COMPLETED = 0x2u, }; -typedef struct SResultRowCell { - uint64_t groupId; - SResultRowPosition pos; -} SResultRowCell; - /** * If the number of generated results is greater than this value, * query query will be halt and return results to client immediate. @@ -83,7 +76,6 @@ typedef struct SResultInfo { // TODO refactor typedef struct STableQueryInfo { TSKEY lastKey; // last check ts, todo remove it later SResultRowPosition pos; // current active time window -// SVariant tag; } STableQueryInfo; typedef struct SLimit { @@ -123,41 +115,7 @@ typedef struct SOperatorCostInfo { double totalCost; } SOperatorCostInfo; -// The basic query information extracted from the SQueryInfo tree to support the -// execution of query in a data node. -typedef struct STaskAttr { - SLimit limit; - SLimit slimit; - bool stableQuery; // super table query or not - bool topBotQuery; // TODO used bitwise flag - bool groupbyColumn; // denote if this is a groupby normal column query - bool timeWindowInterpo; // if the time window start/end required interpolation - bool tsCompQuery; // is tscomp query - bool diffQuery; // is diff query - bool pointInterpQuery; // point interpolation query - int32_t havingNum; // having expr number - int16_t numOfCols; - int16_t numOfTags; - STimeWindow window; - SInterval interval; - int16_t precision; - int16_t numOfOutput; - int16_t fillType; - int32_t resultRowSize; - int32_t tagLen; // tag value length of current query - - SExprInfo* pExpr1; - SColumnInfo* tagColList; - int32_t numOfFilterCols; - int64_t* fillVal; - void* tsdb; -// STableListInfo tableGroupInfo; // table list - int32_t vgId; -} STaskAttr; - struct SOperatorInfo; -//struct SAggSupporter; -//struct SOptrBasicInfo; typedef int32_t (*__optr_encode_fn_t)(struct SOperatorInfo* pOperator, char** result, int32_t* length); typedef int32_t (*__optr_decode_fn_t)(struct SOperatorInfo* pOperator, char* result); @@ -195,31 +153,6 @@ typedef struct SExecTaskInfo { struct SOperatorInfo* pRoot; } SExecTaskInfo; -typedef struct STaskRuntimeEnv { - STaskAttr* pQueryAttr; - uint32_t status; // query status - uint8_t scanFlag; // denotes reversed scan of data or not - SDiskbasedBuf* pResultBuf; // query result buffer based on blocked-wised disk file - SHashObj* pResultRowHashTable; // quick locate the window object for each result - SHashObj* pResultRowListSet; // used to check if current ResultRowInfo has ResultRow object or not - SArray* pResultRowArrayList; // The array list that contains the Result rows - char* keyBuf; // window key buffer - // The window result objects pool, all the resultRow Objects are allocated and managed by this object. - char** prevRow; - STSBuf* pTsBuf; // timestamp filter list - STSCursor cur; - - char* tagVal; // tag value of current data block -// STableGroupInfo tableqinfoGroupInfo; // this is a table list - struct SOperatorInfo* proot; - SGroupResInfo groupResInfo; - int64_t currentOffset; // dynamic offset value - - STableQueryInfo* current; - SResultInfo resultInfo; - struct SUdfInfo* pUdfInfo; -} STaskRuntimeEnv; - enum { OP_NOT_OPENED = 0x0, OP_OPENED = 0x1, @@ -238,14 +171,20 @@ typedef struct SOperatorFpSet { __optr_explain_fn_t getExplainFn; } SOperatorFpSet; +typedef struct SExprSupp { + SExprInfo* pExprInfo; + int32_t numOfExprs; // the number of scalar expression in group operator + SqlFunctionCtx* pCtx; + int32_t* rowEntryInfoOffset; // offset value for each row result cell info +} SExprSupp; + typedef struct SOperatorInfo { uint8_t operatorType; bool blocking; // block operator or not uint8_t status; // denote if current operator is completed - int32_t numOfExprs; // number of columns of the current operator results - char* name; // name, used to show the query execution plan + char* name; // name, for debug purpose void* info; // extension attribution - SExprInfo* pExpr; + SExprSupp exprSupp; SExecTaskInfo* pTaskInfo; SOperatorCostInfo cost; SResultInfo resultInfo; @@ -260,6 +199,9 @@ typedef enum { EX_SOURCE_DATA_EXHAUSTED = 0x3, } EX_SOURCE_STATUS; +#define COL_MATCH_FROM_COL_ID 0x1 +#define COL_MATCH_FROM_SLOT_ID 0x2 + typedef struct SSourceDataInfo { int32_t index; SRetrieveTableRsp* pRsp; @@ -287,9 +229,6 @@ typedef struct SExchangeInfo { uint64_t self; } SExchangeInfo; -#define COL_MATCH_FROM_COL_ID 0x1 -#define COL_MATCH_FROM_SLOT_ID 0x2 - typedef struct SColMatchInfo { int32_t srcSlotId; // source slot id int32_t colId; @@ -299,8 +238,8 @@ typedef struct SColMatchInfo { } SColMatchInfo; typedef struct SScanInfo { - int32_t numOfAsc; - int32_t numOfDesc; + int32_t numOfAsc; + int32_t numOfDesc; } SScanInfo; typedef struct SSampleExecInfo { @@ -320,17 +259,13 @@ typedef struct STableScanInfo { SNode* pFilterNode; // filter info, which is push down by optimizer SqlFunctionCtx* pCtx; // which belongs to the direct upstream operator operator query context SResultRowInfo* pResultRowInfo; - int32_t* rowCellInfoOffset; + int32_t* rowEntryInfoOffset; SExprInfo* pExpr; SSDataBlock* pResBlock; SArray* pColMatchInfo; int32_t numOfOutput; - SExprInfo* pPseudoExpr; - int32_t numOfPseudoExpr; - SqlFunctionCtx* pPseudoCtx; -// int32_t* rowCellInfoOffset; - + SExprSupp pseudoSup; SQueryTableDataCond cond; int32_t scanFlag; // table scan flag to denote if it is a repeat/reverse/main scan int32_t dataBlockLoadFlag; @@ -434,16 +369,16 @@ typedef struct SSysTableScanInfo { typedef struct SBlockDistInfo { SSDataBlock* pResBlock; void* pHandle; + SReadHandle readHandle; + uint64_t uid; // table uid } SBlockDistInfo; +// todo remove this typedef struct SOptrBasicInfo { SResultRowInfo resultRowInfo; - int32_t* rowCellInfoOffset; // offset value for each row result cell info - SqlFunctionCtx* pCtx; SSDataBlock* pRes; } SOptrBasicInfo; -// TODO move the resultrowsiz together with SOptrBasicInfo:rowCellInfoOffset typedef struct SAggSupporter { SHashObj* pResultRowHashTable; // quick locate the window object for each result char* keyBuf; // window key buffer @@ -490,6 +425,7 @@ typedef struct SStreamFinalIntervalOperatorInfo { SArray* pChildren; SSDataBlock* pUpdateRes; SPhysiNode* pPhyNode; // create new child + bool isFinal; } SStreamFinalIntervalOperatorInfo; typedef struct SAggOperatorInfo { @@ -500,10 +436,7 @@ typedef struct SAggOperatorInfo { STableQueryInfo *current; uint64_t groupId; SGroupResInfo groupResInfo; - SExprInfo *pScalarExprInfo; - int32_t numOfScalarExpr; // the number of scalar expression before the aggregate function can be applied - SqlFunctionCtx *pScalarCtx; // scalar function requried sql function struct. - int32_t *rowCellInfoOffset; // offset value for each row result cell info + SExprSupp scalarExprSup; } SAggOperatorInfo; typedef struct SProjectOperatorInfo { @@ -528,11 +461,7 @@ typedef struct SIndefOperatorInfo { SOptrBasicInfo binfo; SAggSupporter aggSup; SArray* pPseudoColInfo; - - SExprInfo* pScalarExpr; - int32_t numOfScalarExpr; - SqlFunctionCtx* pScalarCtx; - int32_t* rowCellInfoOffset; + SExprSupp scalarSup; } SIndefOperatorInfo; typedef struct SFillOperatorInfo { @@ -544,13 +473,6 @@ typedef struct SFillOperatorInfo { bool multigroupResult; } SFillOperatorInfo; -typedef struct SScalarSupp { - SExprInfo* pScalarExprInfo; - int32_t numOfScalarExpr; // the number of scalar expression in group operator - SqlFunctionCtx* pScalarFuncCtx; - int32_t* rowCellInfoOffset; // offset value for each row result cell info -} SScalarSupp; - typedef struct SGroupbyOperatorInfo { // SOptrBasicInfo should be first, SAggSupporter should be second for stream encode SOptrBasicInfo binfo; @@ -563,7 +485,7 @@ typedef struct SGroupbyOperatorInfo { char* keyBuf; // group by keys for hash int32_t groupKeyLen; // total group by column width SGroupResInfo groupResInfo; - SScalarSupp scalarSup; + SExprSupp scalarSup; } SGroupbyOperatorInfo; typedef struct SDataGroupInfo { @@ -584,10 +506,11 @@ typedef struct SPartitionOperatorInfo { SDiskbasedBuf* pBuf; // query result buffer based on blocked-wised disk file int32_t rowCapacity; // maximum number of rows for each buffer page int32_t* columnOffset; // start position for each column data - void* pGroupIter; // group iterator + SArray* sortedGroupArray; // SDataGroupInfo sorted by group id + int32_t groupIndex; // group index int32_t pageIndex; // page index of current group SSDataBlock* pUpdateRes; - SScalarSupp scalarSupp; + SExprSupp scalarSup; } SPartitionOperatorInfo; typedef struct SWindowRowsSup { @@ -628,14 +551,18 @@ typedef struct SStreamSessionAggOperatorInfo { SGroupResInfo groupResInfo; int64_t gap; // session window gap int32_t primaryTsIndex; // primary timestamp slot id + int32_t endTsIndex; // window end timestamp slot id int32_t order; // current SSDataBlock scan order STimeWindowAggSupp twAggSup; SSDataBlock* pWinBlock; // window result SqlFunctionCtx* pDummyCtx; // for combine - SSDataBlock* pDelRes; + SSDataBlock* pDelRes; // delete result + SSDataBlock* pUpdateRes; // update window SHashObj* pStDeleted; void* pDelIterator; SArray* pChildren; // cache for children's result; final stream operator + SPhysiNode* pPhyNode; // create new child + bool isFinal; } SStreamSessionAggOperatorInfo; typedef struct STimeSliceOperatorInfo { @@ -740,49 +667,39 @@ SOperatorFpSet createOperatorFpSet(__optr_open_fn_t openFn, __optr_fn_t nextFn, int32_t operatorDummyOpenFn(SOperatorInfo* pOperator); void operatorDummyCloseFn(void* param, int32_t numOfCols); int32_t appendDownstream(SOperatorInfo* p, SOperatorInfo** pDownstream, int32_t num); -int32_t initAggInfo(SOptrBasicInfo* pBasicInfo, SAggSupporter* pAggSup, SExprInfo* pExprInfo, int32_t numOfCols, - SSDataBlock* pResultBlock, size_t keyBufSize, const char* pkey); + +void initBasicInfo(SOptrBasicInfo* pInfo, SSDataBlock* pBlock); +void cleanupBasicInfo(SOptrBasicInfo* pInfo); +void initExprSupp(SExprSupp* pSup, SExprInfo* pExprInfo, int32_t numOfExpr); +void cleanupExprSup(SExprSupp* pSup); +int32_t initAggInfo(SExprSupp *pSup, SAggSupporter* pAggSup, SExprInfo* pExprInfo, int32_t numOfCols, size_t keyBufSize, + const char* pkey); void initResultSizeInfo(SOperatorInfo* pOperator, int32_t numOfRows); void doBuildResultDatablock(SOperatorInfo* pOperator, SOptrBasicInfo* pbInfo, SGroupResInfo* pGroupResInfo, SDiskbasedBuf* pBuf); void doApplyFunctions(SExecTaskInfo* taskInfo, SqlFunctionCtx* pCtx, STimeWindow* pWin, SColumnInfoData* pTimeWindowData, int32_t offset, int32_t forwardStep, TSKEY* tsCol, int32_t numOfTotal, int32_t numOfOutput, int32_t order); -int32_t setGroupResultOutputBuf(SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t type, int16_t bytes, - int32_t groupId, SDiskbasedBuf* pBuf, SExecTaskInfo* pTaskInfo, SAggSupporter* pAggSup); -void doDestroyBasicInfo(SOptrBasicInfo* pInfo, int32_t numOfOutput); -int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadInfo, int32_t numOfRows, char* pData, + +int32_t extractDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadInfo, int32_t numOfRows, char* pData, int32_t compLen, int32_t numOfOutput, int64_t startTs, uint64_t* total, SArray* pColList); void getAlignQueryTimeWindow(SInterval* pInterval, int32_t precision, int64_t key, STimeWindow* win); int32_t getTableScanInfo(SOperatorInfo* pOperator, int32_t *order, int32_t* scanFlag); int32_t getBufferPgSize(int32_t rowSize, uint32_t* defaultPgsz, uint32_t* defaultBufsz); -SArray* extractPartitionColInfo(SNodeList* pNodeList); - void doSetOperatorCompleted(SOperatorInfo* pOperator); void doFilter(const SNode* pFilterNode, SSDataBlock* pBlock); -SqlFunctionCtx* createSqlFunctionCtx(SExprInfo* pExprInfo, int32_t numOfOutput, int32_t** rowCellInfoOffset); -void relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray* pCols); -void initExecTimeWindowInfo(SColumnInfoData* pColData, STimeWindow* pQueryWindow); + void cleanupAggSup(SAggSupporter* pAggSup); void destroyBasicOperatorInfo(void* param, int32_t numOfOutput); void appendOneRowToDataBlock(SSDataBlock* pBlock, STupleHandle* pTupleHandle); void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData* pColInfoData, int32_t functionId); -SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode); -SColumn extractColumnFromColumnNode(SColumnNode* pColNode); - -SSDataBlock* getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity, SArray* pColMatchInfo, SSortOperatorInfo* pInfo); -SSDataBlock* loadNextDataBlock(void* param); -void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowCellInfoOffset); +void cleanupExecSupp(SExprSupp* pSupp); -SArray* extractColMatchInfo(SNodeList* pNodeList, SDataBlockDescNode* pOutputNodeList, int32_t* numOfOutputCols, - SExecTaskInfo* pTaskInfo, int32_t type); +SSDataBlock* loadNextDataBlock(void* param); -SExprInfo* createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, int32_t* numOfExprs); -SSDataBlock* createResDataBlock(SDataBlockDescNode* pNode); -int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode); -void clearupQueryTableDataCond(SQueryTableDataCond* pCond); +void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowEntryInfoOffset); SResultRow* doSetResultOutBufByKey(SDiskbasedBuf* pResultBuf, SResultRowInfo* pResultRowInfo, char* pData, int16_t bytes, bool masterscan, uint64_t groupId, @@ -799,9 +716,9 @@ SOperatorInfo* createAggregateOperatorInfo(SOperatorInfo* downstream, SExprInfo* int32_t numOfScalarExpr, SExecTaskInfo* pTaskInfo); SOperatorInfo* createIndefinitOutputOperatorInfo(SOperatorInfo* downstream, SPhysiNode *pNode, SExecTaskInfo* pTaskInfo); -SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t num, SSDataBlock* pResBlock, SLimit* pLimit, SLimit* pSlimit, SNode* pCondition, SExecTaskInfo* pTaskInfo); -SOperatorInfo *createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pResBlock, SArray* pSortInfo, SExprInfo* pExprInfo, int32_t numOfCols, - SArray* pIndexMap, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SProjectPhysiNode* pProjPhyNode, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortPhyNode, SExecTaskInfo* pTaskInfo); + SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, int32_t numStreams, SSDataBlock* pInputBlock, SSDataBlock* pResBlock, SArray* pSortInfo, SArray* pColMatchColInfo, SExecTaskInfo* pTaskInfo); @@ -826,15 +743,15 @@ SOperatorInfo* createSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock, SArray* pGroupColList, SNode* pCondition, SExprInfo* pScalarExprInfo, int32_t numOfScalarExpr, SExecTaskInfo* pTaskInfo); -SOperatorInfo* createDataBlockInfoScanOperator(void* dataReader, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createDataBlockInfoScanOperator(void* dataReader, SReadHandle* readHandle, uint64_t uid, SBlockDistScanPhysiNode* pBlockScanNode, + SExecTaskInfo* pTaskInfo); SOperatorInfo* createStreamScanOperatorInfo(void* pDataReader, SReadHandle* pHandle, STableScanPhysiNode* pTableScanNode, SExecTaskInfo* pTaskInfo, STimeWindowAggSupp* pTwSup); +SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SFillPhysiNode* pPhyFillNode, bool multigroupResult, + SExecTaskInfo* pTaskInfo); -SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExpr, int32_t numOfCols, - SInterval* pInterval, STimeWindow* pWindow, SSDataBlock* pResBlock, int32_t fillType, SNodeListNode* fillVal, - bool multigroupResult, SExecTaskInfo* pTaskInfo); SOperatorInfo* createStatewindowOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExpr, int32_t numOfCols, SSDataBlock* pResBlock, STimeWindowAggSupp *pTwAggSupp, int32_t tsSlotId, SColumn* pStateKeyCol, SExecTaskInfo* pTaskInfo); @@ -843,10 +760,13 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition SOperatorInfo* createTimeSliceOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock, const SNodeListNode* pValNode, SExecTaskInfo* pTaskInfo); -SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResBlock, SNode* pOnCondition, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SJoinPhysiNode* pJoinNode, + SExecTaskInfo* pTaskInfo); -SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, - SSDataBlock* pResBlock, int64_t gap, int32_t tsSlotId, STimeWindowAggSupp* pTwAggSupp, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, + SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo); +SOperatorInfo* createStreamFinalSessionAggOperatorInfo(SOperatorInfo* downstream, + SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, int32_t numOfChild); SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo); @@ -862,8 +782,8 @@ void setInputDataBlock(SOperatorInfo* pOperator, SqlFunctionCtx* pCtx, SSDataBlo bool isTaskKilled(SExecTaskInfo* pTaskInfo); int32_t checkForQueryBuf(size_t numOfTables); -void setTaskKilled(SExecTaskInfo* pTaskInfo); -void queryCostStatis(SExecTaskInfo* pTaskInfo); +void setTaskKilled(SExecTaskInfo* pTaskInfo); +void queryCostStatis(SExecTaskInfo* pTaskInfo); void doDestroyTask(SExecTaskInfo* pTaskInfo); int32_t getMaximumIdleDurationSec(); @@ -882,7 +802,7 @@ int32_t encodeOperator(SOperatorInfo* ops, char** data, int32_t *length); * length: the length of data * return: result code, 0 means success */ -int32_t decodeOperator(SOperatorInfo* ops, char* data, int32_t length); +int32_t decodeOperator(SOperatorInfo* ops, const char* data, int32_t length); void setTaskStatus(SExecTaskInfo* pTaskInfo, int8_t status); int32_t createExecTaskInfoImpl(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SReadHandle* pHandle, uint64_t taskId, @@ -899,12 +819,13 @@ STimeWindow getActiveTimeWindow(SDiskbasedBuf* pBuf, SResultRowInfo* pResultRowI int32_t getNumOfRowsInTimeWindow(SDataBlockInfo* pDataBlockInfo, TSKEY* pPrimaryColumn, int32_t startPos, TSKEY ekey, __block_search_fn_t searchFn, STableQueryInfo* item, int32_t order); int32_t binarySearchForKey(char* pValue, int num, TSKEY key, int order); -int32_t initStreamAggSupporter(SStreamAggSupporter* pSup, const char* pKey, - SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t size); +int32_t initStreamAggSupporter(SStreamAggSupporter* pSup, const char* pKey, SqlFunctionCtx* pCtx, int32_t numOfOutput, + int32_t size); SResultRow* getNewResultRow(SDiskbasedBuf* pResultBuf, int64_t tableGroupId, int32_t interBufSize); -SResultWindowInfo* getSessionTimeWindow(SStreamAggSupporter* pAggSup, TSKEY ts, uint64_t groupId, int64_t gap, int32_t* pIndex); -int32_t updateSessionWindowInfo(SResultWindowInfo* pWinInfo, TSKEY* pTs, int32_t rows, - int32_t start, int64_t gap, SHashObj* pStDeleted); +SResultWindowInfo* getSessionTimeWindow(SStreamAggSupporter* pAggSup, TSKEY startTs, + TSKEY endTs, uint64_t groupId, int64_t gap, int32_t* pIndex); +int32_t updateSessionWindowInfo(SResultWindowInfo* pWinInfo, TSKEY* pStartTs, + TSKEY* pEndTs, int32_t rows, int32_t start, int64_t gap, SHashObj* pStDeleted); bool functionNeedToExecute(SqlFunctionCtx* pCtx); int32_t compareTimeWindow(const void* p1, const void* p2, const void* param); @@ -912,8 +833,6 @@ int32_t finalizeResultRowIntoResultDataBlock(SDiskbasedBuf* pBuf, SResultRowPosi SqlFunctionCtx* pCtx, SExprInfo* pExprInfo, int32_t numOfExprs, const int32_t* rowCellOffset, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo); -int32_t getTableList(void* metaHandle, int32_t tableType, uint64_t tableUid, STableListInfo* pListInfo, - SNode* pTagCond); int32_t createMultipleDataReaders(STableScanPhysiNode* pTableScanNode, SReadHandle* pHandle, STableListInfo* pTableListInfo, SArray* arrayReader, uint64_t queryId, uint64_t taskId, SNode* pTagCond); diff --git a/source/libs/executor/inc/tsort.h b/source/libs/executor/inc/tsort.h index 86ee841cc24d91ff124293e64a14a78196e53840..363f379ee471e9d9d604784f1fcd24880b4ae3e7 100644 --- a/source/libs/executor/inc/tsort.h +++ b/source/libs/executor/inc/tsort.h @@ -63,7 +63,7 @@ typedef int32_t (*_sort_merge_compar_fn_t)(const void* p1, const void* p2, void* * @param type * @return */ -SSortHandle* tsortCreateSortHandle(SArray* pOrderInfo, SArray* pIndexMap, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr); +SSortHandle* tsortCreateSortHandle(SArray* pOrderInfo, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr); /** * diff --git a/source/libs/executor/src/dataDispatcher.c b/source/libs/executor/src/dataDispatcher.c index b7c7102143d5d9524a692ad127f9e7a7e91808d3..802f9ea5b5f6885dbd16674d368c855f037c78f2 100644 --- a/source/libs/executor/src/dataDispatcher.c +++ b/source/libs/executor/src/dataDispatcher.c @@ -75,12 +75,18 @@ static bool needCompress(const SSDataBlock* pData, int32_t numOfCols) { // The length of bitmap is decided by number of rows of this data block, and the length of each column data is // recorded in the first segment, next to the struct header static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) { - int32_t numOfCols = LIST_LENGTH(pHandle->pSchema->pSlots); - + int32_t numOfCols = 0; + SNode* pNode; + FOREACH(pNode, pHandle->pSchema->pSlots) { + SSlotDescNode* pSlotDesc = (SSlotDescNode*)pNode; + if (pSlotDesc->output) { + ++numOfCols; + } + } SDataCacheEntry* pEntry = (SDataCacheEntry*)pBuf->pData; pEntry->compressed = (int8_t)needCompress(pInput->pData, numOfCols); pEntry->numOfRows = pInput->pData->info.rows; - pEntry->numOfCols = pInput->pData->info.numOfCols; + pEntry->numOfCols = numOfCols; pEntry->dataLen = 0; pBuf->useSize = sizeof(SDataCacheEntry); diff --git a/source/libs/executor/src/executil.c b/source/libs/executor/src/executil.c index 01ed30c1896c11c7f9d5291d289d97c0af3ffaff..aadab4f22ad6272bc868984f803b9b2dd1c99a71 100644 --- a/source/libs/executor/src/executil.c +++ b/source/libs/executor/src/executil.c @@ -14,24 +14,20 @@ */ #include "os.h" -#include "tmsg.h" +#include "index.h" +#include "function.h" +#include "functionMgt.h" +#include "tdatablock.h" #include "thash.h" +#include "tmsg.h" #include "executil.h" #include "executorimpl.h" #include "tcompression.h" -#include "tlosertree.h" -typedef struct SCompSupporter { - STableQueryInfo **pTableQueryInfo; - int32_t *rowIndex; - int32_t order; -} SCompSupporter; - -int32_t initResultRowInfo(SResultRowInfo *pResultRowInfo, int32_t size) { +void initResultRowInfo(SResultRowInfo *pResultRowInfo) { pResultRowInfo->size = 0; pResultRowInfo->cur.pageId = -1; - return TSDB_CODE_SUCCESS; } void cleanupResultRowInfo(SResultRowInfo *pResultRowInfo) { @@ -46,21 +42,6 @@ void cleanupResultRowInfo(SResultRowInfo *pResultRowInfo) { } } -void resetResultRowInfo(STaskRuntimeEnv *pRuntimeEnv, SResultRowInfo *pResultRowInfo) { - for (int32_t i = 0; i < pResultRowInfo->size; ++i) { -// SResultRow *pWindowRes = pResultRowInfo->pResult[i]; -// clearResultRow(pRuntimeEnv, pWindowRes); - - int32_t groupIndex = 0; - int64_t uid = 0; - - SET_RES_WINDOW_KEY(pRuntimeEnv->keyBuf, &groupIndex, sizeof(groupIndex), uid); - taosHashRemove(pRuntimeEnv->pResultRowHashTable, (const char *)pRuntimeEnv->keyBuf, GET_RES_WINDOW_KEY_LEN(sizeof(groupIndex))); - } - - pResultRowInfo->size = 0; -} - void closeAllResultRows(SResultRowInfo *pResultRowInfo) { // do nothing } @@ -74,7 +55,7 @@ void closeResultRow(SResultRow* pResultRow) { } // TODO refactor: use macro -SResultRowEntryInfo* getResultCell(const SResultRow* pRow, int32_t index, const int32_t* offset) { +SResultRowEntryInfo* getResultEntryInfo(const SResultRow* pRow, int32_t index, const int32_t* offset) { assert(index >= 0 && offset != NULL); return (SResultRowEntryInfo*)((char*) pRow->pEntryInfo + offset[index]); } @@ -160,7 +141,7 @@ void initMultiResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayL ASSERT(pGroupResInfo->index <= getNumOfTotalRes(pGroupResInfo)); } -bool hashRemainDataInGroupInfo(SGroupResInfo* pGroupResInfo) { +bool hasDataInGroupInfo(SGroupResInfo* pGroupResInfo) { if (pGroupResInfo->pRows == NULL) { return false; } @@ -177,86 +158,540 @@ int32_t getNumOfTotalRes(SGroupResInfo* pGroupResInfo) { return (int32_t) taosArrayGetSize(pGroupResInfo->pRows); } -static int32_t tableResultComparFn(const void *pLeft, const void *pRight, void *param) { - int32_t left = *(int32_t *)pLeft; - int32_t right = *(int32_t *)pRight; +SArray* createSortInfo(SNodeList* pNodeList) { + size_t numOfCols = LIST_LENGTH(pNodeList); + SArray* pList = taosArrayInit(numOfCols, sizeof(SBlockOrderInfo)); + if (pList == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return pList; + } + + for (int32_t i = 0; i < numOfCols; ++i) { + SOrderByExprNode* pSortKey = (SOrderByExprNode*)nodesListGetNode(pNodeList, i); + SBlockOrderInfo bi = {0}; + bi.order = (pSortKey->order == ORDER_ASC) ? TSDB_ORDER_ASC : TSDB_ORDER_DESC; + bi.nullFirst = (pSortKey->nullOrder == NULL_ORDER_FIRST); - SCompSupporter * supporter = (SCompSupporter *)param; + SColumnNode* pColNode = (SColumnNode*)pSortKey->pExpr; + bi.slotId = pColNode->slotId; + taosArrayPush(pList, &bi); + } + + return pList; +} - int32_t leftPos = supporter->rowIndex[left]; - int32_t rightPos = supporter->rowIndex[right]; +SSDataBlock* createResDataBlock(SDataBlockDescNode* pNode) { + int32_t numOfCols = LIST_LENGTH(pNode->pSlots); - /* left source is exhausted */ - if (leftPos == -1) { - return 1; + SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); + pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData)); + + pBlock->info.blockId = pNode->dataBlockId; + pBlock->info.rowSize = pNode->totalRowSize; // todo ?? + pBlock->info.type = STREAM_INVALID; + + for (int32_t i = 0; i < numOfCols; ++i) { + SColumnInfoData idata = {{0}}; + SSlotDescNode* pDescNode = (SSlotDescNode*)nodesListGetNode(pNode->pSlots, i); +// if (!pDescNode->output) { // todo disable it temporarily +// continue; +// } + + idata.info.type = pDescNode->dataType.type; + idata.info.bytes = pDescNode->dataType.bytes; + idata.info.scale = pDescNode->dataType.scale; + idata.info.slotId = pDescNode->slotId; + idata.info.precision = pDescNode->dataType.precision; + + if (IS_VAR_DATA_TYPE(idata.info.type)) { + pBlock->info.hasVarCol = true; + } + + taosArrayPush(pBlock->pDataBlock, &idata); } - /* right source is exhausted*/ - if (rightPos == -1) { - return -1; + pBlock->info.numOfCols = taosArrayGetSize(pBlock->pDataBlock); + return pBlock; +} + +int32_t getTableList(void* metaHandle, SScanPhysiNode* pScanNode, STableListInfo* pListInfo, SNode* pTagCond) { + int32_t code = TSDB_CODE_SUCCESS; + pListInfo->pTableList = taosArrayInit(8, sizeof(STableKeyInfo)); + + uint64_t tableUid = pScanNode->uid; + + if (pScanNode->tableType == TSDB_SUPER_TABLE) { + if (pTagCond) { + SIndexMetaArg metaArg = { + .metaEx = metaHandle, .idx = tsdbGetIdx(metaHandle), .ivtIdx = tsdbGetIvtIdx(metaHandle), .suid = tableUid}; + + SArray* res = taosArrayInit(8, sizeof(uint64_t)); + code = doFilterTag(pTagCond, &metaArg, res); + if (code == TSDB_CODE_INDEX_REBUILDING) { // todo + // doFilter(); + } else if (code != TSDB_CODE_SUCCESS) { + qError("failed to get tableIds, reason: %s, suid: %" PRIu64 "", tstrerror(code), tableUid); + taosArrayDestroy(res); + terrno = code; + return code; + } else { + qDebug("success to get tableIds, size: %d, suid: %" PRIu64 "", (int)taosArrayGetSize(res), tableUid); + } + + for (int i = 0; i < taosArrayGetSize(res); i++) { + STableKeyInfo info = {.lastKey = TSKEY_INITIAL_VAL, .uid = *(uint64_t*)taosArrayGet(res, i)}; + taosArrayPush(pListInfo->pTableList, &info); + } + taosArrayDestroy(res); + } else { + code = tsdbGetAllTableList(metaHandle, tableUid, pListInfo->pTableList); + } + } else { // Create one table group. + STableKeyInfo info = {.lastKey = 0, .uid = tableUid}; + taosArrayPush(pListInfo->pTableList, &info); } - ASSERT(0); - STableQueryInfo** pList = supporter->pTableQueryInfo; -// SResultRow* pWindowRes1 = pList[left]->resInfo.pResult[leftPos]; -// SResultRow * pWindowRes1 = getResultRow(&(pList[left]->resInfo), leftPos); -// TSKEY leftTimestamp = pWindowRes1->win.skey; + return code; +} -// SResultRowInfo *pWindowResInfo2 = &(pList[right]->resInfo); -// SResultRow * pWindowRes2 = getResultRow(pWindowResInfo2, rightPos); -// SResultRow* pWindowRes2 = pList[right]->resInfo.pResult[rightPos]; -// TSKEY rightTimestamp = pWindowRes2->win.skey; +SArray* extractPartitionColInfo(SNodeList* pNodeList) { + if(!pNodeList) { + return NULL; + } -// if (leftTimestamp == rightTimestamp) { - return 0; -// } + size_t numOfCols = LIST_LENGTH(pNodeList); + SArray* pList = taosArrayInit(numOfCols, sizeof(SColumn)); + if (pList == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return NULL; + } + + for (int32_t i = 0; i < numOfCols; ++i) { + SColumnNode* pColNode = (SColumnNode*)nodesListGetNode(pNodeList, i); + + // todo extract method + SColumn c = {0}; + c.slotId = pColNode->slotId; + c.colId = pColNode->colId; + c.type = pColNode->node.resType.type; + c.bytes = pColNode->node.resType.bytes; + c.precision = pColNode->node.resType.precision; + c.scale = pColNode->node.resType.scale; + + taosArrayPush(pList, &c); + } -// if (supporter->order == TSDB_ORDER_ASC) { -// return (leftTimestamp > rightTimestamp)? 1:-1; -// } else { -// return (leftTimestamp < rightTimestamp)? 1:-1; -// } + return pList; } -int32_t tsAscOrder(const void* p1, const void* p2) { - SResultRowCell* pc1 = (SResultRowCell*) p1; - SResultRowCell* pc2 = (SResultRowCell*) p2; - if (pc1->groupId == pc2->groupId) { - ASSERT(0); -// if (pc1->pRow->win.skey == pc2->pRow->win.skey) { -// return 0; -// } else { -// return (pc1->pRow->win.skey < pc2->pRow->win.skey)? -1:1; -// } - } else { - return (pc1->groupId < pc2->groupId)? -1:1; +SArray* extractColMatchInfo(SNodeList* pNodeList, SDataBlockDescNode* pOutputNodeList, int32_t* numOfOutputCols, + int32_t type) { + size_t numOfCols = LIST_LENGTH(pNodeList); + SArray* pList = taosArrayInit(numOfCols, sizeof(SColMatchInfo)); + if (pList == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return NULL; } + + for (int32_t i = 0; i < numOfCols; ++i) { + STargetNode* pNode = (STargetNode*)nodesListGetNode(pNodeList, i); + SColumnNode* pColNode = (SColumnNode*)pNode->pExpr; + + SColMatchInfo c = {0}; + c.output = true; + c.colId = pColNode->colId; + c.srcSlotId = pColNode->slotId; + c.matchType = type; + c.targetSlotId = pNode->slotId; + taosArrayPush(pList, &c); + } + + *numOfOutputCols = 0; + int32_t num = LIST_LENGTH(pOutputNodeList->pSlots); + for (int32_t i = 0; i < num; ++i) { + SSlotDescNode* pNode = (SSlotDescNode*)nodesListGetNode(pOutputNodeList->pSlots, i); + + // todo: add reserve flag check + // it is a column reserved for the arithmetic expression calculation + if (pNode->slotId >= numOfCols) { + (*numOfOutputCols) += 1; + continue; + } + + SColMatchInfo* info = NULL; + for (int32_t j = 0; j < taosArrayGetSize(pList); ++j) { + info = taosArrayGet(pList, j); + if (info->targetSlotId == pNode->slotId) { + break; + } + } + + if (pNode->output) { + (*numOfOutputCols) += 1; + } else { + info->output = false; + } + } + + return pList; } -int32_t tsDescOrder(const void* p1, const void* p2) { - SResultRowCell* pc1 = (SResultRowCell*) p1; - SResultRowCell* pc2 = (SResultRowCell*) p2; +static SResSchema createResSchema(int32_t type, int32_t bytes, int32_t slotId, int32_t scale, int32_t precision, + const char* name) { + SResSchema s = {0}; + s.scale = scale; + s.type = type; + s.bytes = bytes; + s.slotId = slotId; + s.precision = precision; + strncpy(s.name, name, tListLen(s.name)); + + return s; +} - if (pc1->groupId == pc2->groupId) { - ASSERT(0); -// if (pc1->pRow->win.skey == pc2->pRow->win.skey) { -// return 0; -// } else { -// return (pc1->pRow->win.skey < pc2->pRow->win.skey)? 1:-1; -// } +static SColumn* createColumn(int32_t blockId, int32_t slotId, int32_t colId, SDataType* pType) { + SColumn* pCol = taosMemoryCalloc(1, sizeof(SColumn)); + if (pCol == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + return NULL; + } + + pCol->slotId = slotId; + pCol->colId = colId; + pCol->bytes = pType->bytes; + pCol->type = pType->type; + pCol->scale = pType->scale; + pCol->precision = pType->precision; + pCol->dataBlockId = blockId; + + return pCol; +} + +SExprInfo* createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, int32_t* numOfExprs) { + int32_t numOfFuncs = LIST_LENGTH(pNodeList); + int32_t numOfGroupKeys = 0; + if (pGroupKeys != NULL) { + numOfGroupKeys = LIST_LENGTH(pGroupKeys); + } + + *numOfExprs = numOfFuncs + numOfGroupKeys; + SExprInfo* pExprs = taosMemoryCalloc(*numOfExprs, sizeof(SExprInfo)); + + for (int32_t i = 0; i < (*numOfExprs); ++i) { + STargetNode* pTargetNode = NULL; + if (i < numOfFuncs) { + pTargetNode = (STargetNode*)nodesListGetNode(pNodeList, i); + } else { + pTargetNode = (STargetNode*)nodesListGetNode(pGroupKeys, i - numOfFuncs); + } + + SExprInfo* pExp = &pExprs[i]; + + pExp->pExpr = taosMemoryCalloc(1, sizeof(tExprNode)); + pExp->pExpr->_function.num = 1; + pExp->pExpr->_function.functionId = -1; + + int32_t type = nodeType(pTargetNode->pExpr); + // it is a project query, or group by column + if (type == QUERY_NODE_COLUMN) { + pExp->pExpr->nodeType = QUERY_NODE_COLUMN; + SColumnNode* pColNode = (SColumnNode*)pTargetNode->pExpr; + + pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); + pExp->base.numOfParams = 1; + + SDataType* pType = &pColNode->node.resType; + pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, + pType->precision, pColNode->colName); + pExp->base.pParam[0].pCol = createColumn(pColNode->dataBlockId, pColNode->slotId, pColNode->colId, pType); + pExp->base.pParam[0].type = FUNC_PARAM_TYPE_COLUMN; + } else if (type == QUERY_NODE_VALUE) { + pExp->pExpr->nodeType = QUERY_NODE_VALUE; + SValueNode* pValNode = (SValueNode*)pTargetNode->pExpr; + + pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); + pExp->base.numOfParams = 1; + + SDataType* pType = &pValNode->node.resType; + pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, + pType->precision, pValNode->node.aliasName); + pExp->base.pParam[0].type = FUNC_PARAM_TYPE_VALUE; + nodesValueNodeToVariant(pValNode, &pExp->base.pParam[0].param); + } else if (type == QUERY_NODE_FUNCTION) { + pExp->pExpr->nodeType = QUERY_NODE_FUNCTION; + SFunctionNode* pFuncNode = (SFunctionNode*)pTargetNode->pExpr; + + SDataType* pType = &pFuncNode->node.resType; + pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, + pType->precision, pFuncNode->node.aliasName); + + pExp->pExpr->_function.functionId = pFuncNode->funcId; + pExp->pExpr->_function.pFunctNode = pFuncNode; + + strncpy(pExp->pExpr->_function.functionName, pFuncNode->functionName, + tListLen(pExp->pExpr->_function.functionName)); +#if 1 + // todo refactor: add the parameter for tbname function + if (strcmp(pExp->pExpr->_function.functionName, "tbname") == 0) { + pFuncNode->pParameterList = nodesMakeList(); + ASSERT(LIST_LENGTH(pFuncNode->pParameterList) == 0); + SValueNode* res = (SValueNode*)nodesMakeNode(QUERY_NODE_VALUE); + if (NULL == res) { // todo handle error + } else { + res->node.resType = (SDataType){.bytes = sizeof(int64_t), .type = TSDB_DATA_TYPE_BIGINT}; + nodesListAppend(pFuncNode->pParameterList, (SNode*)res); + } + } +#endif + + int32_t numOfParam = LIST_LENGTH(pFuncNode->pParameterList); + + pExp->base.pParam = taosMemoryCalloc(numOfParam, sizeof(SFunctParam)); + pExp->base.numOfParams = numOfParam; + + for (int32_t j = 0; j < numOfParam; ++j) { + SNode* p1 = nodesListGetNode(pFuncNode->pParameterList, j); + if (p1->type == QUERY_NODE_COLUMN) { + SColumnNode* pcn = (SColumnNode*)p1; + + pExp->base.pParam[j].type = FUNC_PARAM_TYPE_COLUMN; + pExp->base.pParam[j].pCol = createColumn(pcn->dataBlockId, pcn->slotId, pcn->colId, &pcn->node.resType); + } else if (p1->type == QUERY_NODE_VALUE) { + SValueNode* pvn = (SValueNode*)p1; + pExp->base.pParam[j].type = FUNC_PARAM_TYPE_VALUE; + nodesValueNodeToVariant(pvn, &pExp->base.pParam[j].param); + } + } + } else if (type == QUERY_NODE_OPERATOR) { + pExp->pExpr->nodeType = QUERY_NODE_OPERATOR; + SOperatorNode* pNode = (SOperatorNode*)pTargetNode->pExpr; + + pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); + pExp->base.numOfParams = 1; + + SDataType* pType = &pNode->node.resType; + pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, + pType->precision, pNode->node.aliasName); + pExp->pExpr->_optrRoot.pRootNode = pTargetNode->pExpr; + } else { + ASSERT(0); + } + } + + return pExprs; +} + +// set the output buffer for the selectivity + tag query +static int32_t setSelectValueColumnInfo(SqlFunctionCtx* pCtx, int32_t numOfOutput) { + int32_t num = 0; + + SqlFunctionCtx* p = NULL; + SqlFunctionCtx** pValCtx = taosMemoryCalloc(numOfOutput, POINTER_BYTES); + if (pValCtx == NULL) { + return TSDB_CODE_QRY_OUT_OF_MEMORY; + } + + for (int32_t i = 0; i < numOfOutput; ++i) { + if (strcmp(pCtx[i].pExpr->pExpr->_function.functionName, "_select_value") == 0) { + pValCtx[num++] = &pCtx[i]; + } else if (fmIsSelectFunc(pCtx[i].functionId)) { + p = &pCtx[i]; + } + } + + if (p != NULL) { + p->subsidiaries.pCtx = pValCtx; + p->subsidiaries.num = num; } else { - return (pc1->groupId < pc2->groupId)? -1:1; + taosMemoryFreeClear(pValCtx); + } + + return TSDB_CODE_SUCCESS; +} + +SqlFunctionCtx* createSqlFunctionCtx(SExprInfo* pExprInfo, int32_t numOfOutput, int32_t** rowEntryInfoOffset) { + SqlFunctionCtx* pFuncCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfOutput, sizeof(SqlFunctionCtx)); + if (pFuncCtx == NULL) { + return NULL; + } + + *rowEntryInfoOffset = taosMemoryCalloc(numOfOutput, sizeof(int32_t)); + if (*rowEntryInfoOffset == 0) { + taosMemoryFreeClear(pFuncCtx); + return NULL; + } + + for (int32_t i = 0; i < numOfOutput; ++i) { + SExprInfo* pExpr = &pExprInfo[i]; + + SExprBasicInfo* pFunct = &pExpr->base; + SqlFunctionCtx* pCtx = &pFuncCtx[i]; + + pCtx->functionId = -1; + pCtx->curBufPage = -1; + pCtx->pExpr = pExpr; + + if (pExpr->pExpr->nodeType == QUERY_NODE_FUNCTION) { + SFuncExecEnv env = {0}; + pCtx->functionId = pExpr->pExpr->_function.pFunctNode->funcId; + + if (fmIsAggFunc(pCtx->functionId) || fmIsIndefiniteRowsFunc(pCtx->functionId)) { + bool isUdaf = fmIsUserDefinedFunc(pCtx->functionId); + if (!isUdaf) { + fmGetFuncExecFuncs(pCtx->functionId, &pCtx->fpSet); + } else { + char* udfName = pExpr->pExpr->_function.pFunctNode->functionName; + strncpy(pCtx->udfName, udfName, strlen(udfName)); + fmGetUdafExecFuncs(pCtx->functionId, &pCtx->fpSet); + } + pCtx->fpSet.getEnv(pExpr->pExpr->_function.pFunctNode, &env); + } else { + fmGetScalarFuncExecFuncs(pCtx->functionId, &pCtx->sfp); + if (pCtx->sfp.getEnv != NULL) { + pCtx->sfp.getEnv(pExpr->pExpr->_function.pFunctNode, &env); + } + } + pCtx->resDataInfo.interBufSize = env.calcMemSize; + } else if (pExpr->pExpr->nodeType == QUERY_NODE_COLUMN || pExpr->pExpr->nodeType == QUERY_NODE_OPERATOR || + pExpr->pExpr->nodeType == QUERY_NODE_VALUE) { + // for simple column, the result buffer needs to hold at least one element. + pCtx->resDataInfo.interBufSize = pFunct->resSchema.bytes; + } + + pCtx->input.numOfInputCols = pFunct->numOfParams; + pCtx->input.pData = taosMemoryCalloc(pFunct->numOfParams, POINTER_BYTES); + pCtx->input.pColumnDataAgg = taosMemoryCalloc(pFunct->numOfParams, POINTER_BYTES); + + pCtx->pTsOutput = NULL; + pCtx->resDataInfo.bytes = pFunct->resSchema.bytes; + pCtx->resDataInfo.type = pFunct->resSchema.type; + pCtx->order = TSDB_ORDER_ASC; + pCtx->start.key = INT64_MIN; + pCtx->end.key = INT64_MIN; + pCtx->numOfParams = pExpr->base.numOfParams; + pCtx->increase = false; + + pCtx->param = pFunct->pParam; + } + + for (int32_t i = 1; i < numOfOutput; ++i) { + (*rowEntryInfoOffset)[i] = + (int32_t)((*rowEntryInfoOffset)[i - 1] + sizeof(SResultRowEntryInfo) + pFuncCtx[i - 1].resDataInfo.interBufSize); } + + setSelectValueColumnInfo(pFuncCtx, numOfOutput); + return pFuncCtx; } -void orderTheResultRows(STaskRuntimeEnv* pRuntimeEnv) { - __compar_fn_t fn = NULL; -// if (pRuntimeEnv->pQueryAttr->order.order == TSDB_ORDER_ASC) { -// fn = tsAscOrder; -// } else { -// fn = tsDescOrder; -// } +// NOTE: sources columns are more than the destination SSDatablock columns. +// doFilter in table scan needs every column even its output is false +void relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray* pCols, bool outputEveryColumn) { + size_t numOfSrcCols = taosArrayGetSize(pCols); + + int32_t i = 0, j = 0; + while (i < numOfSrcCols && j < taosArrayGetSize(pColMatchInfo)) { + SColumnInfoData* p = taosArrayGet(pCols, i); + SColMatchInfo* pmInfo = taosArrayGet(pColMatchInfo, j); + if (!outputEveryColumn && !pmInfo->output) { + j++; + continue; + } - taosArraySort(pRuntimeEnv->pResultRowArrayList, fn); + if (p->info.colId == pmInfo->colId) { + SColumnInfoData* pDst = taosArrayGet(pBlock->pDataBlock, pmInfo->targetSlotId); + colDataAssign(pDst, p, pBlock->info.rows); + i++; + j++; + } else if (p->info.colId < pmInfo->colId) { + i++; + } else { + ASSERT(0); + } + } } + +SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode) { + SInterval interval = { + .interval = pTableScanNode->interval, + .sliding = pTableScanNode->sliding, + .intervalUnit = pTableScanNode->intervalUnit, + .slidingUnit = pTableScanNode->slidingUnit, + .offset = pTableScanNode->offset, + }; + + return interval; +} + +SColumn extractColumnFromColumnNode(SColumnNode* pColNode) { + SColumn c = {0}; + c.slotId = pColNode->slotId; + c.colId = pColNode->colId; + c.type = pColNode->node.resType.type; + c.bytes = pColNode->node.resType.bytes; + c.scale = pColNode->node.resType.scale; + c.precision = pColNode->node.resType.precision; + return c; +} + +int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode) { + pCond->loadExternalRows = false; + + pCond->order = pTableScanNode->scanSeq[0] > 0 ? TSDB_ORDER_ASC : TSDB_ORDER_DESC; + pCond->numOfCols = LIST_LENGTH(pTableScanNode->scan.pScanCols); + pCond->colList = taosMemoryCalloc(pCond->numOfCols, sizeof(SColumnInfo)); + if (pCond->colList == NULL) { + terrno = TSDB_CODE_QRY_OUT_OF_MEMORY; + return terrno; + } + + // pCond->twindow = pTableScanNode->scanRange; + // TODO: get it from stable scan node + pCond->numOfTWindows = 1; + pCond->twindows = taosMemoryCalloc(pCond->numOfTWindows, sizeof(STimeWindow)); + pCond->twindows[0] = pTableScanNode->scanRange; + pCond->suid = pTableScanNode->scan.suid; + +#if 1 + // todo work around a problem, remove it later + for (int32_t i = 0; i < pCond->numOfTWindows; ++i) { + if ((pCond->order == TSDB_ORDER_ASC && pCond->twindows[i].skey > pCond->twindows[i].ekey) || + (pCond->order == TSDB_ORDER_DESC && pCond->twindows[i].skey < pCond->twindows[i].ekey)) { + TSWAP(pCond->twindows[i].skey, pCond->twindows[i].ekey); + } + } +#endif + + for (int32_t i = 0; i < pCond->numOfTWindows; ++i) { + if ((pCond->order == TSDB_ORDER_ASC && pCond->twindows[i].skey > pCond->twindows[i].ekey) || + (pCond->order == TSDB_ORDER_DESC && pCond->twindows[i].skey < pCond->twindows[i].ekey)) { + TSWAP(pCond->twindows[i].skey, pCond->twindows[i].ekey); + } + } + taosqsort(pCond->twindows, pCond->numOfTWindows, sizeof(STimeWindow), pCond, compareTimeWindow); + + pCond->type = BLOCK_LOAD_OFFSET_SEQ_ORDER; + // pCond->type = pTableScanNode->scanFlag; + + int32_t j = 0; + for (int32_t i = 0; i < pCond->numOfCols; ++i) { + STargetNode* pNode = (STargetNode*)nodesListGetNode(pTableScanNode->scan.pScanCols, i); + SColumnNode* pColNode = (SColumnNode*)pNode->pExpr; + if (pColNode->colType == COLUMN_TYPE_TAG) { + continue; + } + + pCond->colList[j].type = pColNode->node.resType.type; + pCond->colList[j].bytes = pColNode->node.resType.bytes; + pCond->colList[j].colId = pColNode->colId; + j += 1; + } + + pCond->numOfCols = j; + return TSDB_CODE_SUCCESS; +} + +void cleanupQueryTableDataCond(SQueryTableDataCond* pCond) { + taosMemoryFree(pCond->twindows); + taosMemoryFree(pCond->colList); +} \ No newline at end of file diff --git a/source/libs/executor/src/executor.c b/source/libs/executor/src/executor.c index b1d076e8f57e1970fc42e0834032d374e8ed149e..cf72dfd7966deec976b90d615d8bb90329f69cc3 100644 --- a/source/libs/executor/src/executor.c +++ b/source/libs/executor/src/executor.c @@ -41,12 +41,18 @@ static int32_t doSetStreamBlock(SOperatorInfo* pOperator, void* input, size_t nu pInfo->assignBlockUid = assignUid; // the block type can not be changed in the streamscan operators +#if 0 if (pInfo->blockType == 0) { pInfo->blockType = type; } else if (pInfo->blockType != type) { ASSERT(0); return TSDB_CODE_QRY_APP_ERROR; } +#endif + // rollup sma, the same qTaskInfo is used to insert data by SubmitReq and fetch result by SSDataBlock + if (pInfo->blockType != type) { + pInfo->blockType = type; + } if (type == STREAM_DATA_TYPE_SUBMIT_BLOCK) { if (tqReadHandleSetMsg(pInfo->streamBlockReader, input, 0) < 0) { @@ -99,7 +105,7 @@ int32_t qSetMultiStreamInput(qTaskInfo_t tinfo, const void* pBlocks, size_t numO } qTaskInfo_t qCreateStreamExecTaskInfo(void* msg, void* streamReadHandle) { - if (msg == NULL || streamReadHandle == NULL) { + if (msg == NULL) { return NULL; } diff --git a/source/libs/executor/src/executorMain.c b/source/libs/executor/src/executorMain.c index 00158d702439518f60841ec6e3c0a8fd38315b70..a9e1e03178b0a28780ecfdd696d44e244d2648a1 100644 --- a/source/libs/executor/src/executorMain.c +++ b/source/libs/executor/src/executorMain.c @@ -13,10 +13,10 @@ * along with this program. If not, see . */ -#include "os.h" -#include "tref.h" #include "dataSinkMgt.h" +#include "os.h" #include "tmsg.h" +#include "tref.h" #include "tudf.h" #include "executor.h" @@ -24,15 +24,13 @@ #include "query.h" static TdThreadOnce initPoolOnce = PTHREAD_ONCE_INIT; -int32_t exchangeObjRefPool = -1; +int32_t exchangeObjRefPool = -1; -static void initRefPool() { - exchangeObjRefPool = taosOpenRef(1024, doDestroyExchangeOperatorInfo); -} +static void initRefPool() { exchangeObjRefPool = taosOpenRef(1024, doDestroyExchangeOperatorInfo); } int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId, SSubplan* pSubplan, qTaskInfo_t* pTaskInfo, DataSinkHandle* handle, const char* sql, EOPTR_EXEC_MODEL model) { - assert(readHandle != NULL && pSubplan != NULL); + assert(pSubplan != NULL); SExecTaskInfo** pTask = (SExecTaskInfo**)pTaskInfo; taosThreadOnce(&initPoolOnce, initRefPool); @@ -47,57 +45,57 @@ int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId, if (code != TSDB_CODE_SUCCESS) { goto _error; } - + if (handle) { void* pSinkParam = NULL; code = createDataSinkParam(pSubplan->pDataSink, &pSinkParam, pTaskInfo); if (code != TSDB_CODE_SUCCESS) { goto _error; } - + code = dsCreateDataSinker(pSubplan->pDataSink, handle, pSinkParam); } - _error: +_error: // if failed to add ref for all tables in this query, abort current query return code; } #ifdef TEST_IMPL // wait moment -int waitMoment(SQInfo* pQInfo){ - if(pQInfo->sql) { - int ms = 0; +int waitMoment(SQInfo* pQInfo) { + if (pQInfo->sql) { + int ms = 0; char* pcnt = strstr(pQInfo->sql, " count(*)"); - if(pcnt) return 0; - + if (pcnt) return 0; + char* pos = strstr(pQInfo->sql, " t_"); - if(pos){ + if (pos) { pos += 3; ms = atoi(pos); - while(*pos >= '0' && *pos <= '9'){ - pos ++; + while (*pos >= '0' && *pos <= '9') { + pos++; } char unit_char = *pos; - if(unit_char == 'h'){ - ms *= 3600*1000; - } else if(unit_char == 'm'){ - ms *= 60*1000; - } else if(unit_char == 's'){ + if (unit_char == 'h') { + ms *= 3600 * 1000; + } else if (unit_char == 'm') { + ms *= 60 * 1000; + } else if (unit_char == 's') { ms *= 1000; } } - if(ms == 0) return 0; + if (ms == 0) return 0; printf("test wait sleep %dms. sql=%s ...\n", ms, pQInfo->sql); - - if(ms < 1000) { + + if (ms < 1000) { taosMsleep(ms); } else { int used_ms = 0; - while(used_ms < ms) { + while (used_ms < ms) { taosMsleep(1000); used_ms += 1000; - if(isTaskKilled(pQInfo)){ + if (isTaskKilled(pQInfo)) { printf("test check query is canceled, sleep break.%s\n", pQInfo->sql); break; } @@ -108,15 +106,14 @@ int waitMoment(SQInfo* pQInfo){ } #endif -int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t *useconds) { +int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t* useconds) { SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo; int64_t threadId = taosGetSelfPthreadId(); *pRes = NULL; int64_t curOwner = 0; if ((curOwner = atomic_val_compare_exchange_64(&pTaskInfo->owner, 0, threadId)) != 0) { - qError("%s-%p execTask is now executed by thread:%p", GET_TASKID(pTaskInfo), pTaskInfo, - (void*)curOwner); + qError("%s-%p execTask is now executed by thread:%p", GET_TASKID(pTaskInfo), pTaskInfo, (void*)curOwner); pTaskInfo->code = TSDB_CODE_QRY_IN_EXEC; return pTaskInfo->code; } @@ -152,18 +149,18 @@ int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t *useconds) { cleanUpUdfs(); - int32_t current = (*pRes != NULL)? (*pRes)->info.rows:0; + int32_t current = (*pRes != NULL) ? (*pRes)->info.rows : 0; uint64_t total = pTaskInfo->pRoot->resultInfo.totalRows; qDebug("%s task suspended, %d rows returned, total:%" PRId64 " rows, in sinkNode:%d, elapsed:%.2f ms", - GET_TASKID(pTaskInfo), current, total, 0, el/1000.0); + GET_TASKID(pTaskInfo), current, total, 0, el / 1000.0); atomic_store_64(&pTaskInfo->owner, 0); return pTaskInfo->code; } int32_t qKillTask(qTaskInfo_t qinfo) { - SExecTaskInfo *pTaskInfo = (SExecTaskInfo *)qinfo; + SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)qinfo; if (pTaskInfo == NULL) { return TSDB_CODE_QRY_INVALID_QHANDLE; @@ -182,7 +179,7 @@ int32_t qKillTask(qTaskInfo_t qinfo) { } int32_t qAsyncKillTask(qTaskInfo_t qinfo) { - SExecTaskInfo *pTaskInfo = (SExecTaskInfo *)qinfo; + SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)qinfo; if (pTaskInfo == NULL) { return TSDB_CODE_QRY_INVALID_QHANDLE; @@ -195,7 +192,7 @@ int32_t qAsyncKillTask(qTaskInfo_t qinfo) { } int32_t qIsTaskCompleted(qTaskInfo_t qinfo) { - SExecTaskInfo *pTaskInfo = (SExecTaskInfo *)qinfo; + SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)qinfo; if (pTaskInfo == NULL) { return TSDB_CODE_QRY_INVALID_QHANDLE; @@ -205,18 +202,37 @@ int32_t qIsTaskCompleted(qTaskInfo_t qinfo) { } void qDestroyTask(qTaskInfo_t qTaskHandle) { - SExecTaskInfo* pTaskInfo = (SExecTaskInfo*) qTaskHandle; - qDebug("%s execTask completed, numOfRows:%"PRId64, GET_TASKID(pTaskInfo), pTaskInfo->pRoot->resultInfo.totalRows); + SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)qTaskHandle; + qDebug("%s execTask completed, numOfRows:%" PRId64, GET_TASKID(pTaskInfo), pTaskInfo->pRoot->resultInfo.totalRows); - queryCostStatis(pTaskInfo); // print the query cost summary + queryCostStatis(pTaskInfo); // print the query cost summary doDestroyTask(pTaskInfo); } -int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t *resNum, SExplainExecInfo **pRes) { - SExecTaskInfo *pTaskInfo = (SExecTaskInfo *)tinfo; - int32_t capacity = 0; +int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t* resNum, SExplainExecInfo** pRes) { + SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo; + int32_t capacity = 0; + + return getOperatorExplainExecInfo(pTaskInfo->pRoot, pRes, &capacity, resNum); +} + +int32_t qSerializeTaskStatus(qTaskInfo_t tinfo, char** pOutput, int32_t* len) { + SExecTaskInfo* pTaskInfo = (struct SExecTaskInfo*)tinfo; + if (pTaskInfo->pRoot == NULL) { + return TSDB_CODE_INVALID_PARA; + } + + return encodeOperator(pTaskInfo->pRoot, pOutput, len); +} + +int32_t qDeserializeTaskStatus(qTaskInfo_t tinfo, const char* pInput, int32_t len) { + SExecTaskInfo* pTaskInfo = (struct SExecTaskInfo*) tinfo; + + if (pTaskInfo == NULL || pInput == NULL || len == 0) { + return TSDB_CODE_INVALID_PARA; + } - return getOperatorExplainExecInfo(pTaskInfo->pRoot, pRes, &capacity, resNum); + return decodeOperator(pTaskInfo->pRoot, pInput, len); } diff --git a/source/libs/executor/src/executorimpl.c b/source/libs/executor/src/executorimpl.c index 40b019eb5d7ea8ed38be08d2da372d97700d00b1..81ca0f4f8c459283c9d1562f522daec0e756e82d 100644 --- a/source/libs/executor/src/executorimpl.c +++ b/source/libs/executor/src/executorimpl.c @@ -90,8 +90,6 @@ static void doSetTagValueToResultBuf(char* output, const char* val, int16_t type static void setBlockStatisInfo(SqlFunctionCtx* pCtx, SExprInfo* pExpr, SSDataBlock* pSDataBlock); -static void destroyTableQueryInfoImpl(STableQueryInfo* pTableQueryInfo); - static void releaseQueryBuf(size_t numOfTables); static void destroySFillOperatorInfo(void* param, int32_t numOfOutput); @@ -143,71 +141,10 @@ static int32_t doCopyToSDataBlock(SExecTaskInfo* taskInfo, SSDataBlock* pBlock, SqlFunctionCtx* pCtx, int32_t numOfExprs); static void initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size); -static void setResultBufSize(STaskAttr* pQueryAttr, SResultInfo* pResultInfo); -static void doSetTableGroupOutputBuf(SAggOperatorInfo* pAggInfo, int32_t numOfOutput, uint64_t groupId, - SExecTaskInfo* pTaskInfo); - -SArray* getOrderCheckColumns(STaskAttr* pQuery); - -typedef struct SRowCompSupporter { - STaskRuntimeEnv* pRuntimeEnv; - int16_t dataOffset; - __compar_fn_t comFunc; -} SRowCompSupporter; - -static int compareRowData(const void* a, const void* b, const void* userData) { - const SResultRow* pRow1 = (const SResultRow*)a; - const SResultRow* pRow2 = (const SResultRow*)b; - - SRowCompSupporter* supporter = (SRowCompSupporter*)userData; - STaskRuntimeEnv* pRuntimeEnv = supporter->pRuntimeEnv; - - SFilePage* page1 = getBufPage(pRuntimeEnv->pResultBuf, pRow1->pageId); - SFilePage* page2 = getBufPage(pRuntimeEnv->pResultBuf, pRow2->pageId); - - int16_t offset = supporter->dataOffset; - return 0; - // char* in1 = getPosInResultPage(pRuntimeEnv->pQueryAttr, page1, pRow1->offset, offset); - // char* in2 = getPosInResultPage(pRuntimeEnv->pQueryAttr, page2, pRow2->offset, offset); - - // return (in1 != NULL && in2 != NULL) ? supporter->comFunc(in1, in2) : 0; -} +static void doSetTableGroupOutputBuf(SOperatorInfo* pOperator, SAggOperatorInfo* pAggInfo, int32_t numOfOutput, + uint64_t groupId); // setup the output buffer for each operator -SSDataBlock* createResDataBlock(SDataBlockDescNode* pNode) { - int32_t numOfCols = LIST_LENGTH(pNode->pSlots); - - SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); - pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData)); - - pBlock->info.blockId = pNode->dataBlockId; - pBlock->info.rowSize = pNode->totalRowSize; // todo ?? - pBlock->info.type = STREAM_INVALID; - - for (int32_t i = 0; i < numOfCols; ++i) { - SColumnInfoData idata = {{0}}; - SSlotDescNode* pDescNode = (SSlotDescNode*)nodesListGetNode(pNode->pSlots, i); - // if (!pDescNode->output) { // todo disable it temporarily - // continue; - // } - - idata.info.type = pDescNode->dataType.type; - idata.info.bytes = pDescNode->dataType.bytes; - idata.info.scale = pDescNode->dataType.scale; - idata.info.slotId = pDescNode->slotId; - idata.info.precision = pDescNode->dataType.precision; - - if (IS_VAR_DATA_TYPE(idata.info.type)) { - pBlock->info.hasVarCol = true; - } - - taosArrayPush(pBlock->pDataBlock, &idata); - } - - pBlock->info.numOfCols = taosArrayGetSize(pBlock->pDataBlock); - return pBlock; -} - static bool hasNull(SColumn* pColumn, SColumnDataAgg* pStatis) { if (TSDB_COL_IS_TAG(pColumn->flag) || TSDB_COL_IS_UD_COL(pColumn->flag) || pColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID) { @@ -221,6 +158,7 @@ static bool hasNull(SColumn* pColumn, SColumnDataAgg* pStatis) { return true; } +#if 0 static bool chkResultRowFromKey(STaskRuntimeEnv* pRuntimeEnv, SResultRowInfo* pResultRowInfo, char* pData, int16_t bytes, bool masterscan, uint64_t uid) { bool existed = false; @@ -257,6 +195,7 @@ static bool chkResultRowFromKey(STaskRuntimeEnv* pRuntimeEnv, SResultRowInfo* pR return p1 != NULL; } +#endif SResultRow* getNewResultRow(SDiskbasedBuf* pResultBuf, int64_t tableGroupId, int32_t interBufSize) { SFilePage* pData = NULL; @@ -365,36 +304,6 @@ SResultRow* doSetResultOutBufByKey(SDiskbasedBuf* pResultBuf, SResultRowInfo* pR return pResult; } -// get the correct time window according to the handled timestamp -static STimeWindow getCurrentActiveTimeWindow(SResultRowInfo* pResultRowInfo, int64_t ts, STaskAttr* pQueryAttr) { - STimeWindow w = {0}; -#if 0 - if (pResultRowInfo->curPos == -1) { // the first window, from the previous stored value - // getInitialStartTimeWindow(pQueryAttr, ts, &w); - - if (pQueryAttr->interval.intervalUnit == 'n' || pQueryAttr->interval.intervalUnit == 'y') { - w.ekey = - taosTimeAdd(w.skey, pQueryAttr->interval.interval, pQueryAttr->interval.intervalUnit, pQueryAttr->precision) - - 1; - } else { - w.ekey = w.skey + pQueryAttr->interval.interval - 1; - } - } else { - w = pRow->win; - } - - /* - * query border check, skey should not be bounded by the query time range, since the value skey will - * be used as the time window index value. So we only change ekey of time window accordingly. - */ - if (w.ekey > pQueryAttr->window.ekey && QUERY_IS_ASC_QUERY(pQueryAttr)) { - w.ekey = pQueryAttr->window.ekey; - } -#endif - - return w; -} - // a new buffer page for each table. Needs to opt this design static int32_t addNewWindowResultBuf(SResultRow* pWindowRes, SDiskbasedBuf* pResultBuf, int32_t tid, uint32_t size) { if (pWindowRes->pageId != -1) { @@ -442,13 +351,6 @@ static int32_t addNewWindowResultBuf(SResultRow* pWindowRes, SDiskbasedBuf* pRes return 0; } -static bool chkWindowOutputBufByKey(STaskRuntimeEnv* pRuntimeEnv, SResultRowInfo* pResultRowInfo, STimeWindow* win, - bool masterscan, SResultRow** pResult, int64_t groupId, SqlFunctionCtx* pCtx, - int32_t numOfOutput, int32_t* rowCellInfoOffset) { - assert(win->skey <= win->ekey); - return chkResultRowFromKey(pRuntimeEnv, pResultRowInfo, (char*)&win->skey, TSDB_KEYSIZE, masterscan, groupId); -} - // query_range_start, query_range_end, window_duration, window_start, window_end void initExecTimeWindowInfo(SColumnInfoData* pColData, STimeWindow* pQueryWindow) { pColData->info.type = TSDB_DATA_TYPE_TIMESTAMP; @@ -517,33 +419,15 @@ void doApplyFunctions(SExecTaskInfo* taskInfo, SqlFunctionCtx* pCtx, STimeWindow } } -static FORCE_INLINE TSKEY reviseWindowEkey(STaskAttr* pQueryAttr, STimeWindow* pWindow) { - TSKEY ekey = -1; - int32_t order = TSDB_ORDER_ASC; - if (order == TSDB_ORDER_ASC) { - ekey = pWindow->ekey; - if (ekey > pQueryAttr->window.ekey) { - ekey = pQueryAttr->window.ekey; - } - } else { - ekey = pWindow->skey; - if (ekey < pQueryAttr->window.ekey) { - ekey = pQueryAttr->window.ekey; - } - } - - return ekey; -} - static int32_t doSetInputDataBlock(SOperatorInfo* pOperator, SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32_t order, int32_t scanFlag, bool createDummyCol); static void doSetInputDataBlockInfo(SOperatorInfo* pOperator, SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32_t order) { - for (int32_t i = 0; i < pOperator->numOfExprs; ++i) { + for (int32_t i = 0; i < pOperator->exprSupp.numOfExprs; ++i) { pCtx[i].order = order; pCtx[i].input.numOfRows = pBlock->info.rows; - setBlockStatisInfo(&pCtx[i], &pOperator->pExpr[i], pBlock); + setBlockStatisInfo(&pCtx[i], &pOperator->exprSupp.pExprInfo[i], pBlock); } } @@ -602,7 +486,7 @@ static int32_t doSetInputDataBlock(SOperatorInfo* pOperator, SqlFunctionCtx* pCt int32_t scanFlag, bool createDummyCol) { int32_t code = TSDB_CODE_SUCCESS; - for (int32_t i = 0; i < pOperator->numOfExprs; ++i) { + for (int32_t i = 0; i < pOperator->exprSupp.numOfExprs; ++i) { pCtx[i].order = order; pCtx[i].input.numOfRows = pBlock->info.rows; @@ -613,7 +497,7 @@ static int32_t doSetInputDataBlock(SOperatorInfo* pOperator, SqlFunctionCtx* pCt pInput->uid = pBlock->info.uid; pInput->colDataAggIsSet = false; - SExprInfo* pOneExpr = &pOperator->pExpr[i]; + SExprInfo* pOneExpr = &pOperator->exprSupp.pExprInfo[i]; for (int32_t j = 0; j < pOneExpr->base.numOfParams; ++j) { SFunctParam* pFuncParam = &pOneExpr->base.pParam[j]; if (pFuncParam->type == FUNC_PARAM_TYPE_COLUMN) { @@ -649,7 +533,7 @@ static int32_t doSetInputDataBlock(SOperatorInfo* pOperator, SqlFunctionCtx* pCt } static int32_t doAggregateImpl(SOperatorInfo* pOperator, TSKEY startTs, SqlFunctionCtx* pCtx) { - for (int32_t k = 0; k < pOperator->numOfExprs; ++k) { + for (int32_t k = 0; k < pOperator->exprSupp.numOfExprs; ++k) { if (functionNeedToExecute(&pCtx[k])) { // todo add a dummy funtion to avoid process check if (pCtx[k].fpSet.process == NULL) { @@ -802,20 +686,6 @@ static void setResultRowKey(SResultRow* pResultRow, char* pData, int16_t type) { } } -int32_t setGroupResultOutputBuf(SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t type, int16_t bytes, - int32_t groupId, SDiskbasedBuf* pBuf, SExecTaskInfo* pTaskInfo, - SAggSupporter* pAggSup) { - SResultRowInfo* pResultRowInfo = &binfo->resultRowInfo; - SqlFunctionCtx* pCtx = binfo->pCtx; - - SResultRow* pResultRow = - doSetResultOutBufByKey(pBuf, pResultRowInfo, (char*)pData, bytes, true, groupId, pTaskInfo, false, pAggSup); - assert(pResultRow != NULL); - - setResultRowInitCtx(pResultRow, pCtx, numOfCols, binfo->rowCellInfoOffset); - return TSDB_CODE_SUCCESS; -} - bool functionNeedToExecute(SqlFunctionCtx* pCtx) { struct SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); @@ -927,137 +797,6 @@ void setBlockStatisInfo(SqlFunctionCtx* pCtx, SExprInfo* pExprInfo, SSDataBlock* // } } -// set the output buffer for the selectivity + tag query -static int32_t setSelectValueColumnInfo(SqlFunctionCtx* pCtx, int32_t numOfOutput) { - int32_t num = 0; - - SqlFunctionCtx* p = NULL; - SqlFunctionCtx** pValCtx = taosMemoryCalloc(numOfOutput, POINTER_BYTES); - if (pValCtx == NULL) { - return TSDB_CODE_QRY_OUT_OF_MEMORY; - } - - for (int32_t i = 0; i < numOfOutput; ++i) { - if (strcmp(pCtx[i].pExpr->pExpr->_function.functionName, "_select_value") == 0) { - pValCtx[num++] = &pCtx[i]; - } else if (fmIsSelectFunc(pCtx[i].functionId)) { - p = &pCtx[i]; - } - // if (functionId == FUNCTION_TAG_DUMMY || functionId == FUNCTION_TS_DUMMY) { - // tagLen += pCtx[i].resDataInfo.bytes; - // pTagCtx[num++] = &pCtx[i]; - // } else if (functionId == FUNCTION_TS || functionId == FUNCTION_TAG) { - // // tag function may be the group by tag column - // // ts may be the required primary timestamp column - // continue; - // } else { - // // the column may be the normal column, group by normal_column, the functionId is FUNCTION_PRJ - // } - } - - if (p != NULL) { - p->subsidiaries.pCtx = pValCtx; - p->subsidiaries.num = num; - } else { - taosMemoryFreeClear(pValCtx); - } - - return TSDB_CODE_SUCCESS; -} - -SqlFunctionCtx* createSqlFunctionCtx(SExprInfo* pExprInfo, int32_t numOfOutput, int32_t** rowCellInfoOffset) { - SqlFunctionCtx* pFuncCtx = (SqlFunctionCtx*)taosMemoryCalloc(numOfOutput, sizeof(SqlFunctionCtx)); - if (pFuncCtx == NULL) { - return NULL; - } - - *rowCellInfoOffset = taosMemoryCalloc(numOfOutput, sizeof(int32_t)); - if (*rowCellInfoOffset == 0) { - taosMemoryFreeClear(pFuncCtx); - return NULL; - } - - for (int32_t i = 0; i < numOfOutput; ++i) { - SExprInfo* pExpr = &pExprInfo[i]; - - SExprBasicInfo* pFunct = &pExpr->base; - SqlFunctionCtx* pCtx = &pFuncCtx[i]; - - pCtx->functionId = -1; - pCtx->curBufPage = -1; - pCtx->pExpr = pExpr; - - if (pExpr->pExpr->nodeType == QUERY_NODE_FUNCTION) { - SFuncExecEnv env = {0}; - pCtx->functionId = pExpr->pExpr->_function.pFunctNode->funcId; - - if (fmIsAggFunc(pCtx->functionId) || fmIsIndefiniteRowsFunc(pCtx->functionId)) { - bool isUdaf = fmIsUserDefinedFunc(pCtx->functionId); - if (!isUdaf) { - fmGetFuncExecFuncs(pCtx->functionId, &pCtx->fpSet); - } else { - char* udfName = pExpr->pExpr->_function.pFunctNode->functionName; - strncpy(pCtx->udfName, udfName, strlen(udfName)); - fmGetUdafExecFuncs(pCtx->functionId, &pCtx->fpSet); - } - pCtx->fpSet.getEnv(pExpr->pExpr->_function.pFunctNode, &env); - } else { - fmGetScalarFuncExecFuncs(pCtx->functionId, &pCtx->sfp); - if (pCtx->sfp.getEnv != NULL) { - pCtx->sfp.getEnv(pExpr->pExpr->_function.pFunctNode, &env); - } - } - pCtx->resDataInfo.interBufSize = env.calcMemSize; - } else if (pExpr->pExpr->nodeType == QUERY_NODE_COLUMN || pExpr->pExpr->nodeType == QUERY_NODE_OPERATOR || - pExpr->pExpr->nodeType == QUERY_NODE_VALUE) { - // for simple column, the result buffer needs to hold at least one element. - pCtx->resDataInfo.interBufSize = pFunct->resSchema.bytes; - } - - pCtx->input.numOfInputCols = pFunct->numOfParams; - pCtx->input.pData = taosMemoryCalloc(pFunct->numOfParams, POINTER_BYTES); - pCtx->input.pColumnDataAgg = taosMemoryCalloc(pFunct->numOfParams, POINTER_BYTES); - - pCtx->pTsOutput = NULL; - pCtx->resDataInfo.bytes = pFunct->resSchema.bytes; - pCtx->resDataInfo.type = pFunct->resSchema.type; - pCtx->order = TSDB_ORDER_ASC; - pCtx->start.key = INT64_MIN; - pCtx->end.key = INT64_MIN; - pCtx->numOfParams = pExpr->base.numOfParams; - pCtx->increase = false; - - pCtx->param = pFunct->pParam; - } - - for (int32_t i = 1; i < numOfOutput; ++i) { - (*rowCellInfoOffset)[i] = - (int32_t)((*rowCellInfoOffset)[i - 1] + sizeof(SResultRowEntryInfo) + pFuncCtx[i - 1].resDataInfo.interBufSize); - } - - setSelectValueColumnInfo(pFuncCtx, numOfOutput); - return pFuncCtx; -} - -static void* destroySqlFunctionCtx(SqlFunctionCtx* pCtx, int32_t numOfOutput) { - if (pCtx == NULL) { - return NULL; - } - - for (int32_t i = 0; i < numOfOutput; ++i) { - for (int32_t j = 0; j < pCtx[i].numOfParams; ++j) { - taosVariantDestroy(&pCtx[i].param[j].param); - } - - taosMemoryFreeClear(pCtx[i].subsidiaries.pCtx); - taosMemoryFree(pCtx[i].input.pData); - taosMemoryFree(pCtx[i].input.pColumnDataAgg); - } - - taosMemoryFreeClear(pCtx); - return NULL; -} - bool isTaskKilled(SExecTaskInfo* pTaskInfo) { // query has been executed more than tsShellActivityTimer, and the retrieve has not arrived // abort current query execution. @@ -1075,36 +814,6 @@ bool isTaskKilled(SExecTaskInfo* pTaskInfo) { void setTaskKilled(SExecTaskInfo* pTaskInfo) { pTaskInfo->code = TSDB_CODE_TSC_QUERY_CANCELLED; } -static bool isCachedLastQuery(STaskAttr* pQueryAttr) { - for (int32_t i = 0; i < pQueryAttr->numOfOutput; ++i) { - int32_t functionId = getExprFunctionId(&pQueryAttr->pExpr1[i]); - // if (functionId == FUNCTION_LAST || functionId == FUNCTION_LAST_DST) { - // continue; - // } - - return false; - } - - int32_t order = TSDB_ORDER_ASC; - if (order != TSDB_ORDER_DESC || !TSWINDOW_IS_EQUAL(pQueryAttr->window, TSWINDOW_DESC_INITIALIZER)) { - return false; - } - - if (pQueryAttr->groupbyColumn) { - return false; - } - - if (pQueryAttr->interval.interval > 0) { - return false; - } - - if (pQueryAttr->numOfFilterCols > 0 || pQueryAttr->havingNum > 0) { - return false; - } - - return true; -} - ///////////////////////////////////////////////////////////////////////////////////////////// // todo refactor : return window void getAlignQueryTimeWindow(SInterval* pInterval, int32_t precision, int64_t key, STimeWindow* win) { @@ -1120,7 +829,9 @@ void getAlignQueryTimeWindow(SInterval* pInterval, int32_t precision, int64_t ke } } +#if 0 static int32_t updateBlockLoadStatus(STaskAttr* pQuery, int32_t status) { + bool hasFirstLastFunc = false; bool hasOtherFunc = false; @@ -1130,7 +841,7 @@ static int32_t updateBlockLoadStatus(STaskAttr* pQuery, int32_t status) { for (int32_t i = 0; i < pQuery->numOfOutput; ++i) { int32_t functionId = getExprFunctionId(&pQuery->pExpr1[i]); -#if 0 + if (functionId == FUNCTION_TS || functionId == FUNCTION_TS_DUMMY || functionId == FUNCTION_TAG || functionId == FUNCTION_TAG_DUMMY) { continue; @@ -1141,7 +852,7 @@ static int32_t updateBlockLoadStatus(STaskAttr* pQuery, int32_t status) { } else { hasOtherFunc = true; } -#endif + } if (hasFirstLastFunc && status == BLK_DATA_NOT_LOAD) { @@ -1155,6 +866,8 @@ static int32_t updateBlockLoadStatus(STaskAttr* pQuery, int32_t status) { return status; } +#endif + // static void updateDataCheckOrder(SQInfo *pQInfo, SQueryTableReq* pQueryMsg, bool stableQuery) { // STaskAttr* pQueryAttr = pQInfo->runtimeEnv.pQueryAttr; // @@ -1266,7 +979,7 @@ static int32_t updateBlockLoadStatus(STaskAttr* pQuery, int32_t status) { // // return filterRangeExecute(pQueryAttr->pFilters, pDataStatis, pQueryAttr->numOfCols, numOfRows); // } - +#if 0 static bool overlapWithTimeWindow(STaskAttr* pQueryAttr, SDataBlockInfo* pBlockInfo) { STimeWindow w = {0}; @@ -1315,55 +1028,7 @@ static bool overlapWithTimeWindow(STaskAttr* pQueryAttr, SDataBlockInfo* pBlockI return false; } - -void doCompactSDataBlock(SSDataBlock* pBlock, int32_t numOfRows, int8_t* p) { - int32_t len = 0; - int32_t start = 0; - for (int32_t j = 0; j < numOfRows; ++j) { - if (p[j] == 1) { - len++; - } else { - if (len > 0) { - int32_t cstart = j - len; - for (int32_t i = 0; i < pBlock->info.numOfCols; ++i) { - SColumnInfoData* pColumnInfoData = taosArrayGet(pBlock->pDataBlock, i); - - int16_t bytes = pColumnInfoData->info.bytes; - memmove(((char*)pColumnInfoData->pData) + start * bytes, pColumnInfoData->pData + cstart * bytes, - len * bytes); - } - - start += len; - len = 0; - } - } - } - - if (len > 0) { - int32_t cstart = numOfRows - len; - for (int32_t i = 0; i < pBlock->info.numOfCols; ++i) { - SColumnInfoData* pColumnInfoData = taosArrayGet(pBlock->pDataBlock, i); - - int16_t bytes = pColumnInfoData->info.bytes; - memmove(pColumnInfoData->pData + start * bytes, pColumnInfoData->pData + cstart * bytes, len * bytes); - } - - start += len; - len = 0; - } - - pBlock->info.rows = start; - pBlock->pBlockAgg = NULL; // clean the block statistics info - - if (start > 0) { - SColumnInfoData* pColumnInfoData = taosArrayGet(pBlock->pDataBlock, 0); - if (pColumnInfoData->info.type == TSDB_DATA_TYPE_TIMESTAMP && - pColumnInfoData->info.colId == PRIMARYKEY_TIMESTAMP_COL_ID) { - pBlock->info.window.skey = *(int64_t*)pColumnInfoData->pData; - pBlock->info.window.ekey = *(int64_t*)(pColumnInfoData->pData + TSDB_KEYSIZE * (start - 1)); - } - } -} +#endif static uint32_t doFilterByBlockTimeWindow(STableScanInfo* pTableScanInfo, SSDataBlock* pBlock) { SqlFunctionCtx* pCtx = pTableScanInfo->pCtx; @@ -1427,17 +1092,17 @@ int32_t loadDataBlockOnDemand(SExecTaskInfo* pTaskInfo, STableScanInfo* pTableSc if (pQueryAttr->pointInterpQuery) { needFilter = chkWindowOutputBufByKey(pRuntimeEnv, pTableScanInfo->pResultRowInfo, &win, masterScan, &pResult, groupId, pTableScanInfo->pCtx, pTableScanInfo->numOfOutput, - pTableScanInfo->rowCellInfoOffset); + pTableScanInfo->rowEntryInfoOffset); } else { if (setResultOutputBufByKey(pRuntimeEnv, pTableScanInfo->pResultRowInfo, pBlock->info.uid, &win, masterScan, &pResult, groupId, pTableScanInfo->pCtx, pTableScanInfo->numOfOutput, - pTableScanInfo->rowCellInfoOffset) != TSDB_CODE_SUCCESS) { + pTableScanInfo->rowEntryInfoOffset) != TSDB_CODE_SUCCESS) { longjmp(pRuntimeEnv->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } } } else if (pQueryAttr->stableQuery && (!pQueryAttr->tsCompQuery) && (!pQueryAttr->diffQuery)) { // stable aggregate, not interval aggregate or normal column aggregate doSetTableGroupOutputBuf(pRuntimeEnv, pTableScanInfo->pResultRowInfo, pTableScanInfo->pCtx, - pTableScanInfo->rowCellInfoOffset, pTableScanInfo->numOfOutput, + pTableScanInfo->rowEntryInfoOffset, pTableScanInfo->numOfOutput, pRuntimeEnv->current->groupIndex); } @@ -1482,7 +1147,7 @@ int32_t loadDataBlockOnDemand(SExecTaskInfo* pTaskInfo, STableScanInfo* pTableSc STimeWindow win = getActiveTimeWindow(pTableScanInfo->pResultRowInfo, k, pQueryAttr); if (setResultOutputBufByKey(pRuntimeEnv, pTableScanInfo->pResultRowInfo, pBlock->info.uid, &win, masterScan, &pResult, groupId, pTableScanInfo->pCtx, pTableScanInfo->numOfOutput, - pTableScanInfo->rowCellInfoOffset) != TSDB_CODE_SUCCESS) { + pTableScanInfo->rowEntryInfoOffset) != TSDB_CODE_SUCCESS) { longjmp(pRuntimeEnv->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } } @@ -1565,14 +1230,14 @@ void initResultRow(SResultRow* pResultRow) { * offset[0] offset[1] offset[2] */ // TODO refactor: some function move away -void setFunctionResultOutput(SOptrBasicInfo* pInfo, SAggSupporter* pSup, int32_t stage, int32_t numOfExprs, - SExecTaskInfo* pTaskInfo) { - SqlFunctionCtx* pCtx = pInfo->pCtx; - SSDataBlock* pDataBlock = pInfo->pRes; - int32_t* rowCellInfoOffset = pInfo->rowCellInfoOffset; +void setFunctionResultOutput(SOperatorInfo* pOperator, SOptrBasicInfo* pInfo, SAggSupporter* pSup, int32_t stage, + int32_t numOfExprs) { + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; + int32_t* rowEntryInfoOffset = pOperator->exprSupp.rowEntryInfoOffset; SResultRowInfo* pResultRowInfo = &pInfo->resultRowInfo; - initResultRowInfo(pResultRowInfo, 16); + initResultRowInfo(pResultRowInfo); int64_t tid = 0; int64_t groupId = 0; @@ -1580,7 +1245,7 @@ void setFunctionResultOutput(SOptrBasicInfo* pInfo, SAggSupporter* pSup, int32_t pTaskInfo, false, pSup); for (int32_t i = 0; i < numOfExprs; ++i) { - struct SResultRowEntryInfo* pEntry = getResultCell(pRow, i, rowCellInfoOffset); + struct SResultRowEntryInfo* pEntry = getResultEntryInfo(pRow, i, rowEntryInfoOffset); cleanupResultRowEntry(pEntry); pCtx[i].resultInfo = pEntry; @@ -1590,42 +1255,6 @@ void setFunctionResultOutput(SOptrBasicInfo* pInfo, SAggSupporter* pSup, int32_t initCtxOutputBuffer(pCtx, numOfExprs); } -void updateOutputBuf(SOptrBasicInfo* pBInfo, int32_t* bufCapacity, int32_t numOfInputRows) { - SSDataBlock* pDataBlock = pBInfo->pRes; - - int32_t newSize = pDataBlock->info.rows + numOfInputRows + 5; // extra output buffer - if ((*bufCapacity) < newSize) { - for (int32_t i = 0; i < pDataBlock->info.numOfCols; ++i) { - SColumnInfoData* pColInfo = taosArrayGet(pDataBlock->pDataBlock, i); - - char* p = taosMemoryRealloc(pColInfo->pData, newSize * pColInfo->info.bytes); - if (p != NULL) { - pColInfo->pData = p; - - // it starts from the tail of the previously generated results. - pBInfo->pCtx[i].pOutput = pColInfo->pData; - (*bufCapacity) = newSize; - } else { - // longjmp - } - } - } - - for (int32_t i = 0; i < pDataBlock->info.numOfCols; ++i) { - SColumnInfoData* pColInfo = taosArrayGet(pDataBlock->pDataBlock, i); - pBInfo->pCtx[i].pOutput = pColInfo->pData + pColInfo->info.bytes * pDataBlock->info.rows; - - // set the correct pointer after the memory buffer reallocated. - int32_t functionId = pBInfo->pCtx[i].functionId; -#if 0 - if (functionId == FUNCTION_TOP || functionId == FUNCTION_BOTTOM || functionId == FUNCTION_DIFF || - functionId == FUNCTION_DERIVATIVE) { - // if (i > 0) pBInfo->pCtx[i].pTsOutput = pBInfo->pCtx[i - 1].pOutput; - } -#endif - } -} - void initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size) { for (int32_t j = 0; j < size; ++j) { struct SResultRowEntryInfo* pResInfo = GET_RES_INFO(&pCtx[j]); @@ -1657,9 +1286,9 @@ void destroyTableQueryInfoImpl(STableQueryInfo* pTableQueryInfo) { // cleanupResultRowInfo(&pTableQueryInfo->resInfo); } -void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowCellInfoOffset) { +void setResultRowInitCtx(SResultRow* pResult, SqlFunctionCtx* pCtx, int32_t numOfOutput, int32_t* rowEntryInfoOffset) { for (int32_t i = 0; i < numOfOutput; ++i) { - pCtx[i].resultInfo = getResultCell(pResult, i, rowCellInfoOffset); + pCtx[i].resultInfo = getResultEntryInfo(pResult, i, rowEntryInfoOffset); struct SResultRowEntryInfo* pResInfo = pCtx[i].resultInfo; if (isRowEntryCompleted(pResInfo) && isRowEntryInitialized(pResInfo)) { @@ -1752,14 +1381,13 @@ void extractQualifiedTupleByFilterResult(SSDataBlock* pBlock, const int8_t* rowR } } -void doSetTableGroupOutputBuf(SAggOperatorInfo* pAggInfo, int32_t numOfOutput, uint64_t groupId, - SExecTaskInfo* pTaskInfo) { +void doSetTableGroupOutputBuf(SOperatorInfo* pOperator, SAggOperatorInfo* pAggInfo, int32_t numOfOutput, + uint64_t groupId) { // for simple group by query without interval, all the tables belong to one group result. - int64_t uid = 0; - + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SResultRowInfo* pResultRowInfo = &pAggInfo->binfo.resultRowInfo; - SqlFunctionCtx* pCtx = pAggInfo->binfo.pCtx; - int32_t* rowCellInfoOffset = pAggInfo->binfo.rowCellInfoOffset; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; + int32_t* rowEntryInfoOffset = pOperator->exprSupp.rowEntryInfoOffset; SResultRow* pResultRow = doSetResultOutBufByKey(pAggInfo->aggSup.pResultBuf, pResultRowInfo, (char*)&groupId, sizeof(groupId), true, groupId, pTaskInfo, false, &pAggInfo->aggSup); @@ -1777,15 +1405,15 @@ void doSetTableGroupOutputBuf(SAggOperatorInfo* pAggInfo, int32_t numOfOutput, u } } - setResultRowInitCtx(pResultRow, pCtx, numOfOutput, rowCellInfoOffset); + setResultRowInitCtx(pResultRow, pCtx, numOfOutput, rowEntryInfoOffset); } -void setExecutionContext(int32_t numOfOutput, uint64_t groupId, SExecTaskInfo* pTaskInfo, SAggOperatorInfo* pAggInfo) { +void setExecutionContext(SOperatorInfo* pOperator, int32_t numOfOutput, uint64_t groupId, SAggOperatorInfo* pAggInfo) { if (pAggInfo->groupId != INT32_MIN && pAggInfo->groupId == groupId) { return; } - doSetTableGroupOutputBuf(pAggInfo, numOfOutput, groupId, pTaskInfo); + doSetTableGroupOutputBuf(pOperator, pAggInfo, numOfOutput, groupId); // record the current active group id pAggInfo->groupId = groupId; @@ -1793,7 +1421,7 @@ void setExecutionContext(int32_t numOfOutput, uint64_t groupId, SExecTaskInfo* p static void doUpdateNumOfRows(SResultRow* pRow, int32_t numOfExprs, const int32_t* rowCellOffset) { for (int32_t j = 0; j < numOfExprs; ++j) { - struct SResultRowEntryInfo* pResInfo = getResultCell(pRow, j, rowCellOffset); + struct SResultRowEntryInfo* pResInfo = getResultEntryInfo(pRow, j, rowCellOffset); if (!isRowEntryInitialized(pResInfo)) { continue; } @@ -1829,7 +1457,7 @@ int32_t finalizeResultRowIntoResultDataBlock(SDiskbasedBuf* pBuf, SResultRowPosi for (int32_t j = 0; j < numOfExprs; ++j) { int32_t slotId = pExprInfo[j].base.resSchema.slotId; - pCtx[j].resultInfo = getResultCell(pRow, j, rowCellOffset); + pCtx[j].resultInfo = getResultEntryInfo(pRow, j, rowCellOffset); if (pCtx[j].fpSet.finalize) { int32_t code = pCtx[j].fpSet.finalize(&pCtx[j], pBlock); if (TAOS_FAILED(code)) { @@ -1894,7 +1522,7 @@ int32_t doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock, SExprI for (int32_t j = 0; j < numOfExprs; ++j) { int32_t slotId = pExprInfo[j].base.resSchema.slotId; - pCtx[j].resultInfo = getResultCell(pRow, j, rowCellOffset); + pCtx[j].resultInfo = getResultEntryInfo(pRow, j, rowCellOffset); if (pCtx[j].fpSet.finalize) { int32_t code = pCtx[j].fpSet.finalize(&pCtx[j], pBlock); if (TAOS_FAILED(code)) { @@ -1937,16 +1565,16 @@ int32_t doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock, SExprI void doBuildResultDatablock(SOperatorInfo* pOperator, SOptrBasicInfo* pbInfo, SGroupResInfo* pGroupResInfo, SDiskbasedBuf* pBuf) { - SExprInfo* pExprInfo = pOperator->pExpr; - int32_t numOfExprs = pOperator->numOfExprs; + SExprInfo* pExprInfo = pOperator->exprSupp.pExprInfo; + int32_t numOfExprs = pOperator->exprSupp.numOfExprs; SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; - int32_t* rowCellOffset = pbInfo->rowCellInfoOffset; + int32_t* rowCellOffset = pOperator->exprSupp.rowEntryInfoOffset; SSDataBlock* pBlock = pbInfo->pRes; - SqlFunctionCtx* pCtx = pbInfo->pCtx; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; blockDataCleanup(pBlock); - if (!hashRemainDataInGroupInfo(pGroupResInfo)) { + if (!hasDataInGroupInfo(pGroupResInfo)) { return; } @@ -1956,7 +1584,7 @@ void doBuildResultDatablock(SOperatorInfo* pOperator, SOptrBasicInfo* pbInfo, SG } static void updateNumOfRowsInResultRows(SqlFunctionCtx* pCtx, int32_t numOfOutput, SResultRowInfo* pResultRowInfo, - int32_t* rowCellInfoOffset) { + int32_t* rowEntryInfoOffset) { // update the number of result for each, only update the number of rows for the corresponding window result. // if (QUERY_IS_INTERVAL_QUERY(pQueryAttr)) { // return; @@ -1971,7 +1599,7 @@ static void updateNumOfRowsInResultRows(SqlFunctionCtx* pCtx, int32_t numOfOutpu continue; } - SResultRowEntryInfo* pCell = getResultCell(pResult, j, rowCellInfoOffset); + SResultRowEntryInfo* pCell = getResultEntryInfo(pResult, j, rowEntryInfoOffset); pResult->numOfRows = (uint16_t)(TMAX(pResult->numOfRows, pCell->numOfRes)); } } @@ -2413,35 +2041,9 @@ static int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInf return TSDB_CODE_SUCCESS; } -// NOTE: sources columns are more than the destination SSDatablock columns. -void relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray* pCols) { - size_t numOfSrcCols = taosArrayGetSize(pCols); - - int32_t i = 0, j = 0; - while (i < numOfSrcCols && j < taosArrayGetSize(pColMatchInfo)) { - SColumnInfoData* p = taosArrayGet(pCols, i); - SColMatchInfo* pmInfo = taosArrayGet(pColMatchInfo, j); - if (!pmInfo->output) { - j++; - continue; - } - - if (p->info.colId == pmInfo->colId) { - SColumnInfoData* pDst = taosArrayGet(pBlock->pDataBlock, pmInfo->targetSlotId); - colDataAssign(pDst, p, pBlock->info.rows); - i++; - j++; - } else if (p->info.colId < pmInfo->colId) { - i++; - } else { - ASSERT(0); - } - } -} - -int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadInfo, int32_t numOfRows, char* pData, - int32_t compLen, int32_t numOfOutput, int64_t startTs, uint64_t* total, - SArray* pColList) { +int32_t extractDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadInfo, int32_t numOfRows, char* pData, + int32_t compLen, int32_t numOfOutput, int64_t startTs, uint64_t* total, + SArray* pColList) { if (pColList == NULL) { // data from other sources blockCompressDecode(pRes, numOfOutput, numOfRows, pData); pRes->info.rows = numOfRows; @@ -2484,7 +2086,7 @@ int32_t setDataBlockFromFetchRsp(SSDataBlock* pRes, SLoadRemoteDataInfo* pLoadIn // data from mnode pRes->info.rows = numOfRows; - relocateColumnData(pRes, pColList, pBlock->pDataBlock); + relocateColumnData(pRes, pColList, pBlock->pDataBlock, false); taosArrayDestroy(pBlock->pDataBlock); taosMemoryFree(pBlock); // blockDataDestroy(pBlock); @@ -2565,8 +2167,9 @@ static SSDataBlock* concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SEx } SRetrieveTableRsp* pTableRsp = pDataInfo->pRsp; - code = setDataBlockFromFetchRsp(pExchangeInfo->pResult, pLoadInfo, pTableRsp->numOfRows, pTableRsp->data, - pTableRsp->compLen, pTableRsp->numOfCols, startTs, &pDataInfo->totalRows, NULL); + code = + extractDataBlockFromFetchRsp(pExchangeInfo->pResult, pLoadInfo, pTableRsp->numOfRows, pTableRsp->data, + pTableRsp->compLen, pTableRsp->numOfCols, startTs, &pDataInfo->totalRows, NULL); if (code != 0) { taosMemoryFreeClear(pDataInfo->pRsp); goto _error; @@ -2680,8 +2283,8 @@ static SSDataBlock* seqLoadRemoteData(SOperatorInfo* pOperator) { SSDataBlock* pRes = pExchangeInfo->pResult; SRetrieveTableRsp* pTableRsp = pDataInfo->pRsp; int32_t code = - setDataBlockFromFetchRsp(pExchangeInfo->pResult, pLoadInfo, pTableRsp->numOfRows, pTableRsp->data, - pTableRsp->compLen, pTableRsp->numOfCols, startTs, &pDataInfo->totalRows, NULL); + extractDataBlockFromFetchRsp(pExchangeInfo->pResult, pLoadInfo, pTableRsp->numOfRows, pTableRsp->data, + pTableRsp->compLen, pTableRsp->numOfCols, startTs, &pDataInfo->totalRows, NULL); if (pRsp->completed == 1) { qDebug("%s fetch msg rsp from vgId:%d, taskId:0x%" PRIx64 " numOfRows:%d, rowsOfSource:%" PRIu64 @@ -2817,7 +2420,7 @@ SOperatorInfo* createExchangeOperatorInfo(void* pTransporter, SExchangePhysiNode pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = pInfo->pResult->info.numOfCols; + pOperator->exprSupp.numOfExprs = pInfo->pResult->info.numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(prepareLoadRemoteData, doLoadRemoteData, NULL, NULL, @@ -2938,7 +2541,7 @@ static bool saveCurrentTuple(char** rowColData, SArray* pColumnList, SSDataBlock static void doMergeImpl(SOperatorInfo* pOperator, int32_t numOfExpr, SSDataBlock* pBlock) { SSortedMergeOperatorInfo* pInfo = pOperator->info; - SqlFunctionCtx* pCtx = pInfo->binfo.pCtx; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; for (int32_t i = 0; i < pBlock->info.numOfCols; ++i) { // pCtx[i].size = 1; } @@ -2953,8 +2556,8 @@ static void doMergeImpl(SOperatorInfo* pOperator, int32_t numOfExpr, SSDataBlock doMergeResultImpl(pInfo, pCtx, numOfExpr, i); } else { doFinalizeResultImpl(pCtx, numOfExpr); - int32_t numOfRows = getNumOfResult(pInfo->binfo.pCtx, pOperator->numOfExprs, NULL); - // setTagValueForMultipleRows(pCtx, pOperator->numOfExprs, numOfRows); + int32_t numOfRows = getNumOfResult(pOperator->exprSupp.pCtx, pOperator->exprSupp.numOfExprs, NULL); + // setTagValueForMultipleRows(pCtx, pOperator->exprSupp.numOfExprs, numOfRows); // TODO check for available buffer; @@ -3001,16 +2604,16 @@ static SSDataBlock* doMerge(SOperatorInfo* pOperator) { break; } - setInputDataBlock(pOperator, pInfo->binfo.pCtx, pDataBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); + setInputDataBlock(pOperator, pOperator->exprSupp.pCtx, pDataBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); // updateOutputBuf(&pInfo->binfo, &pAggInfo->bufCapacity, pBlock->info.rows * pAggInfo->resultRowFactor, // pOperator->pRuntimeEnv, true); - doMergeImpl(pOperator, pOperator->numOfExprs, pDataBlock); + doMergeImpl(pOperator, pOperator->exprSupp.numOfExprs, pDataBlock); // flush to tuple store, and after all data have been handled, return to upstream node or sink node } - doFinalizeResultImpl(pInfo->binfo.pCtx, pOperator->numOfExprs); - int32_t numOfRows = getNumOfResult(pInfo->binfo.pCtx, pOperator->numOfExprs, NULL); - // setTagValueForMultipleRows(pCtx, pOperator->numOfExprs, numOfRows); + doFinalizeResultImpl(pOperator->exprSupp.pCtx, pOperator->exprSupp.numOfExprs); + int32_t numOfRows = getNumOfResult(pOperator->exprSupp.pCtx, pOperator->exprSupp.numOfExprs, NULL); + // setTagValueForMultipleRows(pCtx, pOperator->exprSupp.numOfExprs, numOfRows); // TODO check for available buffer; @@ -3073,8 +2676,8 @@ static SSDataBlock* doSortedMerge(SOperatorInfo* pOperator) { } int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize; - pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, NULL, SORT_MULTISOURCE_MERGE, pInfo->bufPageSize, - numOfBufPage, pInfo->binfo.pRes, "GET_TASKID(pTaskInfo)"); + pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_MULTISOURCE_MERGE, pInfo->bufPageSize, numOfBufPage, + pInfo->binfo.pRes, "GET_TASKID(pTaskInfo)"); tsortSetFetchRawDataFp(pInfo->pSortHandle, loadNextDataBlock, NULL, NULL); @@ -3151,20 +2754,20 @@ SOperatorInfo* createSortedMergeOperatorInfo(SOperatorInfo** downstream, int32_t goto _error; } - pInfo->binfo.pCtx = createSqlFunctionCtx(pExprInfo, num, &pInfo->binfo.rowCellInfoOffset); - initResultRowInfo(&pInfo->binfo.resultRowInfo, (int32_t)1); + pOperator->exprSupp.pCtx = createSqlFunctionCtx(pExprInfo, num, &pOperator->exprSupp.rowEntryInfoOffset); + initResultRowInfo(&pInfo->binfo.resultRowInfo); - if (pInfo->binfo.pCtx == NULL || pInfo->binfo.pRes == NULL) { + if (pOperator->exprSupp.pCtx == NULL || pInfo->binfo.pRes == NULL) { goto _error; } size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; - int32_t code = doInitAggInfoSup(&pInfo->aggSup, pInfo->binfo.pCtx, num, keyBufSize, pTaskInfo->id.str); + int32_t code = doInitAggInfoSup(&pInfo->aggSup, pOperator->exprSupp.pCtx, num, keyBufSize, pTaskInfo->id.str); if (code != TSDB_CODE_SUCCESS) { goto _error; } - setFunctionResultOutput(&pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, num, pTaskInfo); + setFunctionResultOutput(pOperator, &pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, num); code = initGroupCol(pExprInfo, num, pGroupInfo, pInfo); if (code != TSDB_CODE_SUCCESS) { goto _error; @@ -3183,8 +2786,8 @@ SOperatorInfo* createSortedMergeOperatorInfo(SOperatorInfo** downstream, int32_t pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = num; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.numOfExprs = num; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; @@ -3212,7 +2815,8 @@ int32_t getTableScanInfo(SOperatorInfo* pOperator, int32_t* order, int32_t* scan // todo add more information about exchange operation int32_t type = pOperator->operatorType; if (type == QUERY_NODE_PHYSICAL_PLAN_EXCHANGE || type == QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN || - type == QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN || type == QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN) { + type == QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN || type == QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN || + type == QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN) { *order = TSDB_ORDER_ASC; *scanFlag = MAIN_SCAN; return TSDB_CODE_SUCCESS; @@ -3239,7 +2843,7 @@ static int32_t doOpenAggregateOptr(SOperatorInfo* pOperator) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SAggOperatorInfo* pAggInfo = pOperator->info; - SOptrBasicInfo* pInfo = &pAggInfo->binfo; + SExprSupp* pSup = &pOperator->exprSupp; SOperatorInfo* downstream = pOperator->pDownstream[0]; int64_t st = taosGetTimestampUs(); @@ -3259,18 +2863,18 @@ static int32_t doOpenAggregateOptr(SOperatorInfo* pOperator) { } // there is an scalar expression that needs to be calculated before apply the group aggregation. - if (pAggInfo->pScalarExprInfo != NULL) { - code = projectApplyFunctions(pAggInfo->pScalarExprInfo, pBlock, pBlock, pAggInfo->pScalarCtx, - pAggInfo->numOfScalarExpr, NULL); + if (pAggInfo->scalarExprSup.pExprInfo != NULL) { + SExprSupp* pSup1 = &pAggInfo->scalarExprSup; + code = projectApplyFunctions(pSup1->pExprInfo, pBlock, pBlock, pSup1->pCtx, pSup1->numOfExprs, NULL); if (code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, code); } } // the pDataBlock are always the same one, no need to call this again - setExecutionContext(pOperator->numOfExprs, pBlock->info.groupId, pTaskInfo, pAggInfo); - setInputDataBlock(pOperator, pInfo->pCtx, pBlock, order, scanFlag, true); - code = doAggregateImpl(pOperator, 0, pInfo->pCtx); + setExecutionContext(pOperator, pOperator->exprSupp.numOfExprs, pBlock->info.groupId, pAggInfo); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, scanFlag, true); + code = doAggregateImpl(pOperator, 0, pSup->pCtx); if (code != 0) { longjmp(pTaskInfo->env, code); } @@ -3316,7 +2920,7 @@ static SSDataBlock* getAggregateResult(SOperatorInfo* pOperator) { blockDataEnsureCapacity(pInfo->pRes, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, pInfo, &pAggInfo->groupResInfo, pAggInfo->aggSup.pResultBuf); - if (pInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pAggInfo->groupResInfo)) { + if (pInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pAggInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } @@ -3529,6 +3133,7 @@ static SSDataBlock* doProjectOperation(SOperatorInfo* pOperator) { SProjectOperatorInfo* pProjectInfo = pOperator->info; SOptrBasicInfo* pInfo = &pProjectInfo->binfo; + SExprSupp* pSup = &pOperator->exprSupp; SSDataBlock* pRes = pInfo->pRes; blockDataCleanup(pRes); @@ -3546,10 +3151,10 @@ static SSDataBlock* doProjectOperation(SOperatorInfo* pOperator) { setInputDataBlock(pOperator, pInfo->pCtx, pBlock, TSDB_ORDER_ASC); blockDataEnsureCapacity(pInfo->pRes, pBlock->info.rows); - projectApplyFunctions(pOperator->pExpr, pInfo->pRes, pBlock, pInfo->pCtx, pOperator->numOfExprs); + projectApplyFunctions(pOperator->exprSupp.pExprInfo, pInfo->pRes, pBlock, pInfo->pCtx, pOperator->exprSupp.numOfExprs); if (pRes->info.rows >= pProjectInfo->binfo.capacity * 0.8) { - copyTsColoum(pRes, pInfo->pCtx, pOperator->numOfExprs); - resetResultRowEntryResult(pInfo->pCtx, pOperator->numOfExprs); + copyTsColoum(pRes, pInfo->pCtx, pOperator->exprSupp.numOfExprs); + resetResultRowEntryResult(pInfo->pCtx, pOperator->exprSupp.numOfExprs); return pRes; } } @@ -3581,10 +3186,10 @@ static SSDataBlock* doProjectOperation(SOperatorInfo* pOperator) { doFilter(pProjectInfo->pFilterNode, pBlock); - setInputDataBlock(pOperator, pInfo->pCtx, pBlock, order, scanFlag, false); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, scanFlag, false); blockDataEnsureCapacity(pInfo->pRes, pInfo->pRes->info.rows + pBlock->info.rows); - code = projectApplyFunctions(pOperator->pExpr, pInfo->pRes, pBlock, pInfo->pCtx, pOperator->numOfExprs, + code = projectApplyFunctions(pSup->pExprInfo, pInfo->pRes, pBlock, pSup->pCtx, pSup->numOfExprs, pProjectInfo->pPseudoColInfo); if (code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, code); @@ -3735,7 +3340,7 @@ static void destroyOperatorInfo(SOperatorInfo* pOperator) { } if (pOperator->fpSet.closeFn != NULL) { - pOperator->fpSet.closeFn(pOperator->info, pOperator->numOfExprs); + pOperator->fpSet.closeFn(pOperator->info, pOperator->exprSupp.numOfExprs); } if (pOperator->pDownstream != NULL) { @@ -3747,11 +3352,11 @@ static void destroyOperatorInfo(SOperatorInfo* pOperator) { pOperator->numOfDownstream = 0; } - if (pOperator->pExpr != NULL) { - destroyExprInfo(pOperator->pExpr, pOperator->numOfExprs); + if (pOperator->exprSupp.pExprInfo != NULL) { + destroyExprInfo(pOperator->exprSupp.pExprInfo, pOperator->exprSupp.numOfExprs); } - taosMemoryFreeClear(pOperator->pExpr); + taosMemoryFreeClear(pOperator->exprSupp.pExprInfo); taosMemoryFreeClear(pOperator->info); taosMemoryFreeClear(pOperator); } @@ -3801,15 +3406,12 @@ void cleanupAggSup(SAggSupporter* pAggSup) { destroyDiskbasedBuf(pAggSup->pResultBuf); } -int32_t initAggInfo(SOptrBasicInfo* pBasicInfo, SAggSupporter* pAggSup, SExprInfo* pExprInfo, int32_t numOfCols, - SSDataBlock* pResultBlock, size_t keyBufSize, const char* pkey) { - pBasicInfo->pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pBasicInfo->rowCellInfoOffset); - pBasicInfo->pRes = pResultBlock; - - doInitAggInfoSup(pAggSup, pBasicInfo->pCtx, numOfCols, keyBufSize, pkey); - +int32_t initAggInfo(SExprSupp* pSup, SAggSupporter* pAggSup, SExprInfo* pExprInfo, int32_t numOfCols, size_t keyBufSize, + const char* pkey) { + initExprSupp(pSup, pExprInfo, numOfCols); + doInitAggInfoSup(pAggSup, pSup->pCtx, numOfCols, keyBufSize, pkey); for (int32_t i = 0; i < numOfCols; ++i) { - pBasicInfo->pCtx[i].pBuf = pAggSup->pResultBuf; + pSup->pCtx[i].pBuf = pAggSup->pResultBuf; } return TSDB_CODE_SUCCESS; @@ -3824,6 +3426,19 @@ void initResultSizeInfo(SOperatorInfo* pOperator, int32_t numOfRows) { } } +void initBasicInfo(SOptrBasicInfo* pInfo, SSDataBlock* pBlock) { + pInfo->pRes = pBlock; + initResultRowInfo(&pInfo->resultRowInfo); +} + +void initExprSupp(SExprSupp* pSup, SExprInfo* pExprInfo, int32_t numOfExpr) { + pSup->pExprInfo = pExprInfo; + pSup->numOfExprs = numOfExpr; + if (pSup->pExprInfo != NULL) { + pSup->pCtx = createSqlFunctionCtx(pExprInfo, numOfExpr, &pSup->rowEntryInfoOffset); + } +} + SOperatorInfo* createAggregateOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock, SExprInfo* pScalarExprInfo, int32_t numOfScalarExpr, SExecTaskInfo* pTaskInfo) { @@ -3837,29 +3452,20 @@ SOperatorInfo* createAggregateOperatorInfo(SOperatorInfo* downstream, SExprInfo* size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, numOfRows); - int32_t code = - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResultBlock, keyBufSize, pTaskInfo->id.str); + int32_t code = initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); if (code != TSDB_CODE_SUCCESS) { goto _error; } - int32_t numOfGroup = 10; // todo replaced with true value - pInfo->groupId = INT32_MIN; - initResultRowInfo(&pInfo->binfo.resultRowInfo, numOfGroup); - - pInfo->pScalarExprInfo = pScalarExprInfo; - pInfo->numOfScalarExpr = numOfScalarExpr; - if (pInfo->pScalarExprInfo != NULL) { - pInfo->pScalarCtx = createSqlFunctionCtx(pScalarExprInfo, numOfScalarExpr, &pInfo->rowCellInfoOffset); - } + initBasicInfo(&pInfo->binfo, pResultBlock); + initExprSupp(&pInfo->scalarExprSup, pScalarExprInfo, numOfScalarExpr); + pInfo->groupId = INT32_MIN; pOperator->name = "TableAggregate"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_HASH_AGG; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(doOpenAggregateOptr, getAggregateResult, NULL, NULL, destroyAggOperatorInfo, @@ -3879,24 +3485,39 @@ _error: return NULL; } -void doDestroyBasicInfo(SOptrBasicInfo* pInfo, int32_t numOfOutput) { - assert(pInfo != NULL); +static void* destroySqlFunctionCtx(SqlFunctionCtx* pCtx, int32_t numOfOutput) { + if (pCtx == NULL) { + return NULL; + } + + for (int32_t i = 0; i < numOfOutput; ++i) { + for (int32_t j = 0; j < pCtx[i].numOfParams; ++j) { + taosVariantDestroy(&pCtx[i].param[j].param); + } + + taosMemoryFreeClear(pCtx[i].subsidiaries.pCtx); + taosMemoryFree(pCtx[i].input.pData); + taosMemoryFree(pCtx[i].input.pColumnDataAgg); + } - destroySqlFunctionCtx(pInfo->pCtx, numOfOutput); - taosMemoryFreeClear(pInfo->rowCellInfoOffset); + taosMemoryFreeClear(pCtx); + return NULL; +} +void cleanupBasicInfo(SOptrBasicInfo* pInfo) { + assert(pInfo != NULL); cleanupResultRowInfo(&pInfo->resultRowInfo); pInfo->pRes = blockDataDestroy(pInfo->pRes); } void destroyBasicOperatorInfo(void* param, int32_t numOfOutput) { SOptrBasicInfo* pInfo = (SOptrBasicInfo*)param; - doDestroyBasicInfo(pInfo, numOfOutput); + cleanupBasicInfo(pInfo); } void destroyAggOperatorInfo(void* param, int32_t numOfOutput) { SAggOperatorInfo* pInfo = (SAggOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); } void destroySFillOperatorInfo(void* param, int32_t numOfOutput) { @@ -3911,22 +3532,25 @@ static void destroyProjectOperatorInfo(void* param, int32_t numOfOutput) { return; } SProjectOperatorInfo* pInfo = (SProjectOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); cleanupAggSup(&pInfo->aggSup); taosArrayDestroy(pInfo->pPseudoColInfo); } +void cleanupExecSupp(SExprSupp* pSupp) { + destroySqlFunctionCtx(pSupp->pCtx, pSupp->numOfExprs); + destroyExprInfo(pSupp->pExprInfo, pSupp->numOfExprs); + + taosMemoryFree(pSupp->rowEntryInfoOffset); +} + static void destroyIndefinitOperatorInfo(void* param, int32_t numOfOutput) { SIndefOperatorInfo* pInfo = (SIndefOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); taosArrayDestroy(pInfo->pPseudoColInfo); cleanupAggSup(&pInfo->aggSup); - - destroySqlFunctionCtx(pInfo->pScalarCtx, numOfOutput); - destroyExprInfo(pInfo->pScalarExpr, pInfo->numOfScalarExpr); - - taosMemoryFree(pInfo->rowCellInfoOffset); + cleanupExecSupp(&pInfo->scalarSup); } void destroyExchangeOperatorInfo(void* param, int32_t numOfOutput) { @@ -3957,8 +3581,7 @@ static SArray* setRowTsColumnOutputInfo(SqlFunctionCtx* pCtx, int32_t numOfCols) return pList; } -SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t num, - SSDataBlock* pResBlock, SLimit* pLimit, SLimit* pSlimit, SNode* pCondition, +SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SProjectPhysiNode* pProjPhyNode, SExecTaskInfo* pTaskInfo) { SProjectOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SProjectOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); @@ -3966,14 +3589,20 @@ SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SExprInfo* p goto _error; } - pInfo->limit = *pLimit; - pInfo->slimit = *pSlimit; - pInfo->curOffset = pLimit->offset; - pInfo->curSOffset = pSlimit->offset; + int32_t numOfCols = 0; + SExprInfo* pExprInfo = createExprInfo(pProjPhyNode->pProjections, NULL, &numOfCols); + + SSDataBlock* pResBlock = createResDataBlock(pProjPhyNode->node.pOutputDataBlockDesc); + SLimit limit = {.limit = pProjPhyNode->limit, .offset = pProjPhyNode->offset}; + SLimit slimit = {.limit = pProjPhyNode->slimit, .offset = pProjPhyNode->soffset}; + + pInfo->limit = limit; + pInfo->slimit = slimit; + pInfo->curOffset = limit.offset; + pInfo->curSOffset = slimit.offset; pInfo->binfo.pRes = pResBlock; - pInfo->pFilterNode = pCondition; + pInfo->pFilterNode = pProjPhyNode->node.pConditions; - int32_t numOfCols = num; int32_t numOfRows = 4096; size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; @@ -3984,17 +3613,16 @@ SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SExprInfo* p } initResultSizeInfo(pOperator, numOfRows); - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); - setFunctionResultOutput(&pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, numOfCols, pTaskInfo); + initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); + setFunctionResultOutput(pOperator, &pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, numOfCols); - pInfo->pPseudoColInfo = setRowTsColumnOutputInfo(pInfo->binfo.pCtx, numOfCols); + pInfo->pPseudoColInfo = setRowTsColumnOutputInfo(pOperator->exprSupp.pCtx, numOfCols); pOperator->name = "ProjectOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_PROJECT; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = num; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doProjectOperation, NULL, NULL, @@ -4015,6 +3643,7 @@ _error: static SSDataBlock* doApplyIndefinitFunction(SOperatorInfo* pOperator) { SIndefOperatorInfo* pIndefInfo = pOperator->info; SOptrBasicInfo* pInfo = &pIndefInfo->binfo; + SExprSupp* pSup = &pOperator->exprSupp; SSDataBlock* pRes = pInfo->pRes; blockDataCleanup(pRes); @@ -4049,19 +3678,20 @@ static SSDataBlock* doApplyIndefinitFunction(SOperatorInfo* pOperator) { } // there is an scalar expression that needs to be calculated before apply the group aggregation. - if (pIndefInfo->pScalarExpr != NULL) { - code = projectApplyFunctions(pIndefInfo->pScalarExpr, pBlock, pBlock, pIndefInfo->pScalarCtx, - pIndefInfo->numOfScalarExpr, pIndefInfo->pPseudoColInfo); + SExprSupp* pScalarSup = &pIndefInfo->scalarSup; + if (pScalarSup->pExprInfo != NULL) { + code = projectApplyFunctions(pScalarSup->pExprInfo, pBlock, pBlock, pScalarSup->pCtx, pScalarSup->numOfExprs, + pIndefInfo->pPseudoColInfo); if (code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, code); } } - setInputDataBlock(pOperator, pInfo->pCtx, pBlock, order, scanFlag, false); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, scanFlag, false); blockDataEnsureCapacity(pInfo->pRes, pInfo->pRes->info.rows + pBlock->info.rows); - code = projectApplyFunctions(pOperator->pExpr, pInfo->pRes, pBlock, pInfo->pCtx, pOperator->numOfExprs, - pIndefInfo->pPseudoColInfo); + code = projectApplyFunctions(pOperator->exprSupp.pExprInfo, pInfo->pRes, pBlock, pSup->pCtx, + pOperator->exprSupp.numOfExprs, pIndefInfo->pPseudoColInfo); if (code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, code); } @@ -4085,15 +3715,17 @@ SOperatorInfo* createIndefinitOutputOperatorInfo(SOperatorInfo* downstream, SPhy goto _error; } + SExprSupp* pSup = &pOperator->exprSupp; + SIndefRowsFuncPhysiNode* pPhyNode = (SIndefRowsFuncPhysiNode*)pNode; int32_t numOfExpr = 0; - SExprInfo* pExprInfo = createExprInfo(pPhyNode->pVectorFuncs, NULL, &numOfExpr); + SExprInfo* pExprInfo = createExprInfo(pPhyNode->pFuncs, NULL, &numOfExpr); - int32_t numOfScalarExpr = 0; if (pPhyNode->pExprs != NULL) { - pInfo->pScalarExpr = createExprInfo(pPhyNode->pExprs, NULL, &numOfScalarExpr); - pInfo->pScalarCtx = createSqlFunctionCtx(pInfo->pScalarExpr, numOfScalarExpr, &pInfo->rowCellInfoOffset); + SExprSupp* pSup1 = &pInfo->scalarSup; + pSup1->pExprInfo = createExprInfo(pPhyNode->pExprs, NULL, &pSup1->numOfExprs); + pSup1->pCtx = createSqlFunctionCtx(pSup1->pExprInfo, pSup1->numOfExprs, &pSup1->rowEntryInfoOffset); } SSDataBlock* pResBlock = createResDataBlock(pPhyNode->node.pOutputDataBlockDesc); @@ -4109,20 +3741,21 @@ SOperatorInfo* createIndefinitOutputOperatorInfo(SOperatorInfo* downstream, SPhy } initResultSizeInfo(pOperator, numOfRows); - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfExpr, pResBlock, keyBufSize, pTaskInfo->id.str); - setFunctionResultOutput(&pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, numOfExpr, pTaskInfo); + initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfExpr, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); + + setFunctionResultOutput(pOperator, &pInfo->binfo, &pInfo->aggSup, MAIN_SCAN, numOfExpr); pInfo->binfo.pRes = pResBlock; - pInfo->numOfScalarExpr = numOfScalarExpr; - pInfo->pPseudoColInfo = setRowTsColumnOutputInfo(pInfo->binfo.pCtx, numOfExpr); + pInfo->pPseudoColInfo = setRowTsColumnOutputInfo(pSup->pCtx, numOfExpr); pOperator->name = "IndefinitOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_PROJECT; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfExpr; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfExpr; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doApplyIndefinitFunction, NULL, NULL, @@ -4162,18 +3795,9 @@ static int32_t initFillInfo(SFillOperatorInfo* pInfo, SExprInfo* pExpr, int32_t } } -SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExpr, int32_t numOfCols, - SInterval* pInterval, STimeWindow* pWindow, SSDataBlock* pResBlock, - int32_t fillType, SNodeListNode* pValueNode, bool multigroupResult, - SExecTaskInfo* pTaskInfo) { - SFillOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SFillOperatorInfo)); - SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); - - pInfo->pRes = pResBlock; - pInfo->multigroupResult = multigroupResult; - +static int32_t convertFillType(int32_t mode) { int32_t type = TSDB_FILL_NONE; - switch (fillType) { + switch (mode) { case FILL_MODE_PREV: type = TSDB_FILL_PREV; break; @@ -4196,26 +3820,46 @@ SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExp type = TSDB_FILL_NONE; } + return type; +} + +SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SFillPhysiNode* pPhyFillNode, bool multigroupResult, + SExecTaskInfo* pTaskInfo) { + SFillOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SFillOperatorInfo)); + SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); + if (pInfo == NULL || pOperator == NULL) { + goto _error; + } + + int32_t num = 0; + SSDataBlock* pResBlock = createResDataBlock(pPhyFillNode->node.pOutputDataBlockDesc); + SExprInfo* pExprInfo = createExprInfo(pPhyFillNode->pTargets, NULL, &num); + SInterval* pInterval = &((SIntervalAggOperatorInfo*)downstream->info)->interval; + int32_t type = convertFillType(pPhyFillNode->mode); + SResultInfo* pResultInfo = &pOperator->resultInfo; initResultSizeInfo(pOperator, 4096); - int32_t code = initFillInfo(pInfo, pExpr, numOfCols, pValueNode, *pWindow, pResultInfo->capacity, pTaskInfo->id.str, - pInterval, type); + int32_t code = initFillInfo(pInfo, pExprInfo, num, (SNodeListNode*)pPhyFillNode->pValues, pPhyFillNode->timeRange, + pResultInfo->capacity, pTaskInfo->id.str, pInterval, type); if (code != TSDB_CODE_SUCCESS) { goto _error; } + pInfo->pRes = pResBlock; + pInfo->multigroupResult = multigroupResult; pOperator->name = "FillOperator"; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_FILL; - pOperator->pExpr = pExpr; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = num; pOperator->info = pInfo; + pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doFill, NULL, NULL, destroySFillOperatorInfo, NULL, NULL, NULL); - pOperator->pTaskInfo = pTaskInfo; + code = appendDownstream(pOperator, &downstream, 1); return pOperator; @@ -4225,151 +3869,6 @@ _error: return NULL; } -static SResSchema createResSchema(int32_t type, int32_t bytes, int32_t slotId, int32_t scale, int32_t precision, - const char* name) { - SResSchema s = {0}; - s.scale = scale; - s.type = type; - s.bytes = bytes; - s.slotId = slotId; - s.precision = precision; - strncpy(s.name, name, tListLen(s.name)); - - return s; -} - -static SColumn* createColumn(int32_t blockId, int32_t slotId, int32_t colId, SDataType* pType) { - SColumn* pCol = taosMemoryCalloc(1, sizeof(SColumn)); - if (pCol == NULL) { - terrno = TSDB_CODE_OUT_OF_MEMORY; - return NULL; - } - - pCol->slotId = slotId; - pCol->colId = colId; - pCol->bytes = pType->bytes; - pCol->type = pType->type; - pCol->scale = pType->scale; - pCol->precision = pType->precision; - pCol->dataBlockId = blockId; - - return pCol; -} - -SExprInfo* createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, int32_t* numOfExprs) { - int32_t numOfFuncs = LIST_LENGTH(pNodeList); - int32_t numOfGroupKeys = 0; - if (pGroupKeys != NULL) { - numOfGroupKeys = LIST_LENGTH(pGroupKeys); - } - - *numOfExprs = numOfFuncs + numOfGroupKeys; - SExprInfo* pExprs = taosMemoryCalloc(*numOfExprs, sizeof(SExprInfo)); - - for (int32_t i = 0; i < (*numOfExprs); ++i) { - STargetNode* pTargetNode = NULL; - if (i < numOfFuncs) { - pTargetNode = (STargetNode*)nodesListGetNode(pNodeList, i); - } else { - pTargetNode = (STargetNode*)nodesListGetNode(pGroupKeys, i - numOfFuncs); - } - - SExprInfo* pExp = &pExprs[i]; - - pExp->pExpr = taosMemoryCalloc(1, sizeof(tExprNode)); - pExp->pExpr->_function.num = 1; - pExp->pExpr->_function.functionId = -1; - - int32_t type = nodeType(pTargetNode->pExpr); - // it is a project query, or group by column - if (type == QUERY_NODE_COLUMN) { - pExp->pExpr->nodeType = QUERY_NODE_COLUMN; - SColumnNode* pColNode = (SColumnNode*)pTargetNode->pExpr; - - pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); - pExp->base.numOfParams = 1; - - SDataType* pType = &pColNode->node.resType; - pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, - pType->precision, pColNode->colName); - pExp->base.pParam[0].pCol = createColumn(pColNode->dataBlockId, pColNode->slotId, pColNode->colId, pType); - pExp->base.pParam[0].type = FUNC_PARAM_TYPE_COLUMN; - } else if (type == QUERY_NODE_VALUE) { - pExp->pExpr->nodeType = QUERY_NODE_VALUE; - SValueNode* pValNode = (SValueNode*)pTargetNode->pExpr; - - pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); - pExp->base.numOfParams = 1; - - SDataType* pType = &pValNode->node.resType; - pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, - pType->precision, pValNode->node.aliasName); - pExp->base.pParam[0].type = FUNC_PARAM_TYPE_VALUE; - nodesValueNodeToVariant(pValNode, &pExp->base.pParam[0].param); - } else if (type == QUERY_NODE_FUNCTION) { - pExp->pExpr->nodeType = QUERY_NODE_FUNCTION; - SFunctionNode* pFuncNode = (SFunctionNode*)pTargetNode->pExpr; - - SDataType* pType = &pFuncNode->node.resType; - pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, - pType->precision, pFuncNode->node.aliasName); - - pExp->pExpr->_function.functionId = pFuncNode->funcId; - pExp->pExpr->_function.pFunctNode = pFuncNode; - - strncpy(pExp->pExpr->_function.functionName, pFuncNode->functionName, - tListLen(pExp->pExpr->_function.functionName)); -#if 1 - // todo refactor: add the parameter for tbname function - if (strcmp(pExp->pExpr->_function.functionName, "tbname") == 0) { - pFuncNode->pParameterList = nodesMakeList(); - ASSERT(LIST_LENGTH(pFuncNode->pParameterList) == 0); - SValueNode* res = (SValueNode*)nodesMakeNode(QUERY_NODE_VALUE); - if (NULL == res) { // todo handle error - } else { - res->node.resType = (SDataType){.bytes = sizeof(int64_t), .type = TSDB_DATA_TYPE_BIGINT}; - nodesListAppend(pFuncNode->pParameterList, (SNode*)res); - } - } -#endif - - int32_t numOfParam = LIST_LENGTH(pFuncNode->pParameterList); - - pExp->base.pParam = taosMemoryCalloc(numOfParam, sizeof(SFunctParam)); - pExp->base.numOfParams = numOfParam; - - for (int32_t j = 0; j < numOfParam; ++j) { - SNode* p1 = nodesListGetNode(pFuncNode->pParameterList, j); - if (p1->type == QUERY_NODE_COLUMN) { - SColumnNode* pcn = (SColumnNode*)p1; - - pExp->base.pParam[j].type = FUNC_PARAM_TYPE_COLUMN; - pExp->base.pParam[j].pCol = createColumn(pcn->dataBlockId, pcn->slotId, pcn->colId, &pcn->node.resType); - } else if (p1->type == QUERY_NODE_VALUE) { - SValueNode* pvn = (SValueNode*)p1; - pExp->base.pParam[j].type = FUNC_PARAM_TYPE_VALUE; - nodesValueNodeToVariant(pvn, &pExp->base.pParam[j].param); - } - } - } else if (type == QUERY_NODE_OPERATOR) { - pExp->pExpr->nodeType = QUERY_NODE_OPERATOR; - SOperatorNode* pNode = (SOperatorNode*)pTargetNode->pExpr; - - pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam)); - pExp->base.numOfParams = 1; - - SDataType* pType = &pNode->node.resType; - pExp->base.resSchema = createResSchema(pType->type, pType->bytes, pTargetNode->slotId, pType->scale, - pType->precision, pNode->node.aliasName); - pExp->pExpr->_optrRoot.pRootNode = pTargetNode->pExpr; - } else { - ASSERT(0); - } - } - - return pExprs; -} - static SExecTaskInfo* createExecTaskInfo(uint64_t queryId, uint64_t taskId, EOPTR_EXEC_MODEL model, char* dbFName) { SExecTaskInfo* pTaskInfo = taosMemoryCalloc(1, sizeof(SExecTaskInfo)); setTaskStatus(pTaskInfo, TASK_NOT_COMPLETED); @@ -4392,8 +3891,6 @@ static tsdbReaderT doCreateDataReader(STableScanPhysiNode* pTableScanNode, SRead static SArray* extractColumnInfo(SNodeList* pNodeList); -static SArray* createSortInfo(SNodeList* pNodeList); - int32_t extractTableSchemaVersion(SReadHandle* pHandle, uint64_t uid, SExecTaskInfo* pTaskInfo) { SMetaReader mr = {0}; metaReaderInit(&mr, pHandle->meta, 0); @@ -4555,12 +4052,21 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo SScanPhysiNode* pScanPhyNode = (SScanPhysiNode*)pPhyNode; // simple child table. STableScanPhysiNode* pTableScanNode = (STableScanPhysiNode*)pPhyNode; STimeWindowAggSupp twSup = { - .waterMark = pTableScanNode->watermark, .calTrigger = pTableScanNode->triggerType, .maxTs = INT64_MIN}; + .waterMark = pTableScanNode->watermark, + .calTrigger = pTableScanNode->triggerType, + .maxTs = INT64_MIN, + }; tsdbReaderT pDataReader = NULL; - if (pHandle->vnode) { - pDataReader = doCreateDataReader(pTableScanNode, pHandle, pTableListInfo, (uint64_t)queryId, taskId, pTagCond); - } else { - getTableList(pHandle->meta, pScanPhyNode->tableType, pScanPhyNode->uid, pTableListInfo, pTagCond); + + if (pHandle) { + if (pHandle->vnode) { + // for stram + pDataReader = + doCreateDataReader(pTableScanNode, pHandle, pTableListInfo, (uint64_t)queryId, taskId, pTagCond); + } else { + // for tq + getTableList(pHandle->meta, pScanPhyNode, pTableListInfo, pTagCond); + } } if (pDataReader == NULL && terrno != 0) { @@ -4587,13 +4093,53 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo } else if (QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN == type) { STagScanPhysiNode* pScanPhyNode = (STagScanPhysiNode*)pPhyNode; - int32_t code = getTableList(pHandle->meta, pScanPhyNode->tableType, pScanPhyNode->uid, pTableListInfo, - pScanPhyNode->node.pConditions); + int32_t code = getTableList(pHandle->meta, pScanPhyNode, pTableListInfo, pScanPhyNode->node.pConditions); if (code != TSDB_CODE_SUCCESS) { + pTaskInfo->code = terrno; return NULL; } return createTagScanOperatorInfo(pHandle, pScanPhyNode, pTableListInfo, pTaskInfo); + } else if (QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN == type) { + SBlockDistScanPhysiNode* pBlockNode = (SBlockDistScanPhysiNode*) pPhyNode; + pTableListInfo->pTableList = taosArrayInit(4, sizeof(STableKeyInfo)); + + if (pBlockNode->tableType == TSDB_SUPER_TABLE) { + int32_t code = tsdbGetAllTableList(pHandle->meta, pBlockNode->uid, pTableListInfo->pTableList); + if (code != TSDB_CODE_SUCCESS) { + pTaskInfo->code = terrno; + return NULL; + } + } else { // Create one table group. + STableKeyInfo info = {.lastKey = 0, .uid = pBlockNode->uid}; + taosArrayPush(pTableListInfo->pTableList, &info); + } + + SQueryTableDataCond cond = {0}; + + { + cond.order = TSDB_ORDER_ASC; + cond.numOfCols = 1; + cond.colList = taosMemoryCalloc(1, sizeof(SColumnInfo)); + if (cond.colList == NULL) { + terrno = TSDB_CODE_QRY_OUT_OF_MEMORY; + return NULL; + } + + cond.colList->colId = 1; + cond.colList->type = TSDB_DATA_TYPE_TIMESTAMP; + cond.colList->bytes = sizeof(TSKEY); + + cond.numOfTWindows = 1; + cond.twindows = taosMemoryCalloc(1, sizeof(STimeWindow)); + cond.twindows[0] = (STimeWindow){.skey = INT64_MIN, .ekey = INT64_MAX}; + cond.suid = pBlockNode->suid; + cond.type = BLOCK_LOAD_OFFSET_SEQ_ORDER; + } + tsdbReaderT* pReader = tsdbReaderOpen(pHandle->vnode, &cond, pTableListInfo, queryId, taskId); + cleanupQueryTableDataCond(&cond); + + return createDataBlockInfoScanOperator(pReader, pHandle, cond.suid, pBlockNode, pTaskInfo); } else { ASSERT(0); } @@ -4613,14 +4159,7 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo SOperatorInfo* pOptr = NULL; if (QUERY_NODE_PHYSICAL_PLAN_PROJECT == type) { - SProjectPhysiNode* pProjPhyNode = (SProjectPhysiNode*)pPhyNode; - SExprInfo* pExprInfo = createExprInfo(pProjPhyNode->pProjections, NULL, &num); - - SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); - SLimit limit = {.limit = pProjPhyNode->limit, .offset = pProjPhyNode->offset}; - SLimit slimit = {.limit = pProjPhyNode->slimit, .offset = pProjPhyNode->soffset}; - pOptr = createProjectOperatorInfo(ops[0], pExprInfo, num, pResBlock, &limit, &slimit, - pProjPhyNode->node.pConditions, pTaskInfo); + pOptr = createProjectOperatorInfo(ops[0], (SProjectPhysiNode*)pPhyNode, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_HASH_AGG == type) { SAggPhysiNode* pAggNode = (SAggPhysiNode*)pPhyNode; SExprInfo* pExprInfo = createExprInfo(pAggNode->pAggFuncs, pAggNode->pGroupKeys, &num); @@ -4687,21 +4226,7 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo int32_t children = 1; pOptr = createStreamFinalIntervalOperatorInfo(ops[0], pPhyNode, pTaskInfo, children); } else if (QUERY_NODE_PHYSICAL_PLAN_SORT == type) { - SSortPhysiNode* pSortPhyNode = (SSortPhysiNode*)pPhyNode; - - SDataBlockDescNode* pDescNode = pPhyNode->pOutputDataBlockDesc; - - SSDataBlock* pResBlock = createResDataBlock(pDescNode); - SArray* info = createSortInfo(pSortPhyNode->pSortKeys); - - int32_t numOfCols = 0; - SExprInfo* pExprInfo = createExprInfo(pSortPhyNode->pExprs, NULL, &numOfCols); - - int32_t numOfOutputCols = 0; - SArray* pColList = - extractColMatchInfo(pSortPhyNode->pTargets, pDescNode, &numOfOutputCols, pTaskInfo, COL_MATCH_FROM_SLOT_ID); - - pOptr = createSortOperatorInfo(ops[0], pResBlock, info, pExprInfo, numOfCols, pColList, pTaskInfo); + pOptr = createSortOperatorInfo(ops[0], (SSortPhysiNode*)pPhyNode, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_MERGE == type) { SMergePhysiNode* pMergePhyNode = (SMergePhysiNode*)pPhyNode; @@ -4711,7 +4236,7 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo SArray* sortInfo = createSortInfo(pMergePhyNode->pMergeKeys); int32_t numOfOutputCols = 0; SArray* pColList = - extractColMatchInfo(pMergePhyNode->pTargets, pDescNode, &numOfOutputCols, pTaskInfo, COL_MATCH_FROM_SLOT_ID); + extractColMatchInfo(pMergePhyNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID); SPhysiNode* pChildNode = (SPhysiNode*)nodesListGetNode(pPhyNode->pChildren, 0); SSDataBlock* pInputDataBlock = createResDataBlock(pChildNode->pOutputDataBlockDesc); pOptr = createMultiwaySortMergeOperatorInfo(ops, size, pInputDataBlock, pResBlock, sortInfo, pColList, pTaskInfo); @@ -4728,18 +4253,13 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo pOptr = createSessionAggOperatorInfo(ops[0], pExprInfo, num, pResBlock, pSessionNode->gap, tsSlotId, &as, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION == type) { - SSessionWinodwPhysiNode* pSessionNode = (SSessionWinodwPhysiNode*)pPhyNode; - - STimeWindowAggSupp as = {.waterMark = pSessionNode->window.watermark, - .calTrigger = pSessionNode->window.triggerType}; - - SExprInfo* pExprInfo = createExprInfo(pSessionNode->window.pFuncs, NULL, &num); - SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); - int32_t tsSlotId = ((SColumnNode*)pSessionNode->window.pTspk)->slotId; - - pOptr = createStreamSessionAggOperatorInfo(ops[0], pExprInfo, num, pResBlock, pSessionNode->gap, tsSlotId, &as, - pTaskInfo); - + pOptr = createStreamSessionAggOperatorInfo(ops[0], pPhyNode, pTaskInfo); + } else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION == type) { + int32_t children = 0; + pOptr = createStreamFinalSessionAggOperatorInfo(ops[0], pPhyNode, pTaskInfo, children); + } else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION == type) { + int32_t children = 1; + pOptr = createStreamFinalSessionAggOperatorInfo(ops[0], pPhyNode, pTaskInfo, children); } else if (QUERY_NODE_PHYSICAL_PLAN_PARTITION == type) { pOptr = createPartitionOperatorInfo(ops[0], (SPartitionPhysiNode*)pPhyNode, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE == type) { @@ -4757,19 +4277,9 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo } else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE == type) { pOptr = createStreamStateAggOperatorInfo(ops[0], pPhyNode, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN == type) { - SJoinPhysiNode* pJoinNode = (SJoinPhysiNode*)pPhyNode; - SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); - - SExprInfo* pExprInfo = createExprInfo(pJoinNode->pTargets, NULL, &num); - pOptr = createMergeJoinOperatorInfo(ops, size, pExprInfo, num, pResBlock, pJoinNode->pOnConditions, pTaskInfo); + pOptr = createMergeJoinOperatorInfo(ops, size, (SJoinPhysiNode*)pPhyNode, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_FILL == type) { - SFillPhysiNode* pFillNode = (SFillPhysiNode*)pPhyNode; - SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); - SExprInfo* pExprInfo = createExprInfo(pFillNode->pTargets, NULL, &num); - - SInterval* pInterval = &((SIntervalAggOperatorInfo*)ops[0]->info)->interval; - pOptr = createFillOperatorInfo(ops[0], pExprInfo, num, pInterval, &pFillNode->timeRange, pResBlock, pFillNode->mode, - (SNodeListNode*)pFillNode->pValues, false, pTaskInfo); + pOptr = createFillOperatorInfo(ops[0], (SFillPhysiNode*)pPhyNode, false, pTaskInfo); } else if (QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC == type) { pOptr = createIndefinitOutputOperatorInfo(ops[0], pPhyNode, pTaskInfo); } else { @@ -4792,79 +4302,6 @@ int32_t compareTimeWindow(const void* p1, const void* p2, const void* param) { return 0; } -int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode) { - pCond->loadExternalRows = false; - - pCond->order = pTableScanNode->scanSeq[0] > 0 ? TSDB_ORDER_ASC : TSDB_ORDER_DESC; - pCond->numOfCols = LIST_LENGTH(pTableScanNode->scan.pScanCols); - pCond->colList = taosMemoryCalloc(pCond->numOfCols, sizeof(SColumnInfo)); - if (pCond->colList == NULL) { - terrno = TSDB_CODE_QRY_OUT_OF_MEMORY; - return terrno; - } - - // pCond->twindow = pTableScanNode->scanRange; - // TODO: get it from stable scan node - pCond->numOfTWindows = 1; - pCond->twindows = taosMemoryCalloc(pCond->numOfTWindows, sizeof(STimeWindow)); - pCond->twindows[0] = pTableScanNode->scanRange; - pCond->suid = pTableScanNode->scan.suid; - -#if 1 - // todo work around a problem, remove it later - for (int32_t i = 0; i < pCond->numOfTWindows; ++i) { - if ((pCond->order == TSDB_ORDER_ASC && pCond->twindows[i].skey > pCond->twindows[i].ekey) || - (pCond->order == TSDB_ORDER_DESC && pCond->twindows[i].skey < pCond->twindows[i].ekey)) { - TSWAP(pCond->twindows[i].skey, pCond->twindows[i].ekey); - } - } -#endif - - for (int32_t i = 0; i < pCond->numOfTWindows; ++i) { - if ((pCond->order == TSDB_ORDER_ASC && pCond->twindows[i].skey > pCond->twindows[i].ekey) || - (pCond->order == TSDB_ORDER_DESC && pCond->twindows[i].skey < pCond->twindows[i].ekey)) { - TSWAP(pCond->twindows[i].skey, pCond->twindows[i].ekey); - } - } - taosqsort(pCond->twindows, pCond->numOfTWindows, sizeof(STimeWindow), pCond, compareTimeWindow); - - pCond->type = BLOCK_LOAD_OFFSET_SEQ_ORDER; - // pCond->type = pTableScanNode->scanFlag; - - int32_t j = 0; - for (int32_t i = 0; i < pCond->numOfCols; ++i) { - STargetNode* pNode = (STargetNode*)nodesListGetNode(pTableScanNode->scan.pScanCols, i); - SColumnNode* pColNode = (SColumnNode*)pNode->pExpr; - if (pColNode->colType == COLUMN_TYPE_TAG) { - continue; - } - - pCond->colList[j].type = pColNode->node.resType.type; - pCond->colList[j].bytes = pColNode->node.resType.bytes; - pCond->colList[j].colId = pColNode->colId; - j += 1; - } - - pCond->numOfCols = j; - return TSDB_CODE_SUCCESS; -} - -void clearupQueryTableDataCond(SQueryTableDataCond* pCond) { - taosMemoryFree(pCond->twindows); - taosMemoryFree(pCond->colList); -} - -SColumn extractColumnFromColumnNode(SColumnNode* pColNode) { - SColumn c = {0}; - c.slotId = pColNode->slotId; - c.colId = pColNode->colId; - c.type = pColNode->node.resType.type; - c.bytes = pColNode->node.resType.bytes; - c.scale = pColNode->node.resType.scale; - c.precision = pColNode->node.resType.precision; - return c; -} - SArray* extractColumnInfo(SNodeList* pNodeList) { size_t numOfCols = LIST_LENGTH(pNodeList); SArray* pList = taosArrayInit(numOfCols, sizeof(SColumn)); @@ -4898,146 +4335,9 @@ SArray* extractColumnInfo(SNodeList* pNodeList) { return pList; } -SArray* extractPartitionColInfo(SNodeList* pNodeList) { - if(!pNodeList) { - return NULL; - } - - size_t numOfCols = LIST_LENGTH(pNodeList); - SArray* pList = taosArrayInit(numOfCols, sizeof(SColumn)); - if (pList == NULL) { - terrno = TSDB_CODE_OUT_OF_MEMORY; - return NULL; - } - - for (int32_t i = 0; i < numOfCols; ++i) { - SColumnNode* pColNode = (SColumnNode*)nodesListGetNode(pNodeList, i); - - // todo extract method - SColumn c = {0}; - c.slotId = pColNode->slotId; - c.colId = pColNode->colId; - c.type = pColNode->node.resType.type; - c.bytes = pColNode->node.resType.bytes; - c.precision = pColNode->node.resType.precision; - c.scale = pColNode->node.resType.scale; - - taosArrayPush(pList, &c); - } - - return pList; -} - -SArray* createSortInfo(SNodeList* pNodeList) { - size_t numOfCols = LIST_LENGTH(pNodeList); - SArray* pList = taosArrayInit(numOfCols, sizeof(SBlockOrderInfo)); - if (pList == NULL) { - terrno = TSDB_CODE_OUT_OF_MEMORY; - return pList; - } - - for (int32_t i = 0; i < numOfCols; ++i) { - SOrderByExprNode* pSortKey = (SOrderByExprNode*)nodesListGetNode(pNodeList, i); - SBlockOrderInfo bi = {0}; - bi.order = (pSortKey->order == ORDER_ASC) ? TSDB_ORDER_ASC : TSDB_ORDER_DESC; - bi.nullFirst = (pSortKey->nullOrder == NULL_ORDER_FIRST); - - SColumnNode* pColNode = (SColumnNode*)pSortKey->pExpr; - bi.slotId = pColNode->slotId; - taosArrayPush(pList, &bi); - } - - return pList; -} - -SArray* extractColMatchInfo(SNodeList* pNodeList, SDataBlockDescNode* pOutputNodeList, int32_t* numOfOutputCols, - SExecTaskInfo* pTaskInfo, int32_t type) { - size_t numOfCols = LIST_LENGTH(pNodeList); - SArray* pList = taosArrayInit(numOfCols, sizeof(SColMatchInfo)); - if (pList == NULL) { - terrno = TSDB_CODE_OUT_OF_MEMORY; - return NULL; - } - - for (int32_t i = 0; i < numOfCols; ++i) { - STargetNode* pNode = (STargetNode*)nodesListGetNode(pNodeList, i); - SColumnNode* pColNode = (SColumnNode*)pNode->pExpr; - - SColMatchInfo c = {0}; - c.output = true; - c.colId = pColNode->colId; - c.srcSlotId = pColNode->slotId; - c.matchType = type; - c.targetSlotId = pNode->slotId; - taosArrayPush(pList, &c); - } - - *numOfOutputCols = 0; - int32_t num = LIST_LENGTH(pOutputNodeList->pSlots); - for (int32_t i = 0; i < num; ++i) { - SSlotDescNode* pNode = (SSlotDescNode*)nodesListGetNode(pOutputNodeList->pSlots, i); - - // todo: add reserve flag check - // it is a column reserved for the arithmetic expression calculation - if (pNode->slotId >= numOfCols) { - (*numOfOutputCols) += 1; - continue; - } - - SColMatchInfo* info = taosArrayGet(pList, pNode->slotId); - if (pNode->output) { - (*numOfOutputCols) += 1; - } else { - info->output = false; - } - } - - return pList; -} - -int32_t getTableList(void* metaHandle, int32_t tableType, uint64_t tableUid, STableListInfo* pListInfo, - SNode* pTagCond) { - int32_t code = TSDB_CODE_SUCCESS; - pListInfo->pTableList = taosArrayInit(8, sizeof(STableKeyInfo)); - - if (tableType == TSDB_SUPER_TABLE) { - if (pTagCond) { - SIndexMetaArg metaArg = { - .metaEx = metaHandle, .idx = tsdbGetIdx(metaHandle), .ivtIdx = tsdbGetIvtIdx(metaHandle), .suid = tableUid}; - - SArray* res = taosArrayInit(8, sizeof(uint64_t)); - code = doFilterTag(pTagCond, &metaArg, res); - if (code == TSDB_CODE_INDEX_REBUILDING) { // todo - // doFilter(); - } else if (code != TSDB_CODE_SUCCESS) { - qError("failed to get tableIds, reason: %s, suid: %" PRIu64 "", tstrerror(code), tableUid); - taosArrayDestroy(res); - terrno = code; - return code; - } else { - qDebug("sucess to get tableIds, size: %d, suid: %" PRIu64 "", (int)taosArrayGetSize(res), tableUid); - } - - for (int i = 0; i < taosArrayGetSize(res); i++) { - STableKeyInfo info = {.lastKey = TSKEY_INITIAL_VAL, .uid = *(uint64_t*)taosArrayGet(res, i)}; - taosArrayPush(pListInfo->pTableList, &info); - } - taosArrayDestroy(res); - } else { - code = tsdbGetAllTableList(metaHandle, tableUid, pListInfo->pTableList); - } - } else { // Create one table group. - STableKeyInfo info = {.lastKey = 0, .uid = tableUid}; - taosArrayPush(pListInfo->pTableList, &info); - } - - return code; -} - tsdbReaderT doCreateDataReader(STableScanPhysiNode* pTableScanNode, SReadHandle* pHandle, STableListInfo* pTableListInfo, uint64_t queryId, uint64_t taskId, SNode* pTagCond) { - int32_t code = - getTableList(pHandle->meta, pTableScanNode->scan.tableType, pTableScanNode->scan.uid, pTableListInfo, pTagCond); + int32_t code = getTableList(pHandle->meta, &pTableScanNode->scan, pTableListInfo, pTagCond); if (code != TSDB_CODE_SUCCESS) { goto _error; } @@ -5055,7 +4355,7 @@ tsdbReaderT doCreateDataReader(STableScanPhysiNode* pTableScanNode, SReadHandle* } tsdbReaderT* pReader = tsdbReaderOpen(pHandle->vnode, &cond, pTableListInfo, queryId, taskId); - clearupQueryTableDataCond(&cond); + cleanupQueryTableDataCond(&cond); return pReader; @@ -5116,15 +4416,17 @@ int32_t encodeOperator(SOperatorInfo* ops, char** result, int32_t* length) { return TDB_CODE_SUCCESS; } -int32_t decodeOperator(SOperatorInfo* ops, char* result, int32_t length) { +int32_t decodeOperator(SOperatorInfo* ops, const char* result, int32_t length) { int32_t code = TDB_CODE_SUCCESS; if (ops->fpSet.decodeResultRow) { if (result == NULL) { return TSDB_CODE_TSC_INVALID_INPUT; } + ASSERT(length == *(int32_t*)result); - char* data = result + sizeof(int32_t); - code = ops->fpSet.decodeResultRow(ops, data); + + const char* data = result + sizeof(int32_t); + code = ops->fpSet.decodeResultRow(ops, (char*)data); if (code != TDB_CODE_SUCCESS) { return code; } diff --git a/source/libs/executor/src/groupoperator.c b/source/libs/executor/src/groupoperator.c index 4fc25688c467effa96da8035a5225be07ce39198..482326a30eb24b1af1180bc4ba233d6d7e49cbe4 100644 --- a/source/libs/executor/src/groupoperator.c +++ b/source/libs/executor/src/groupoperator.c @@ -26,15 +26,18 @@ #include "ttypes.h" #include "executorInt.h" +static void* getCurrentDataGroupInfo(const SPartitionOperatorInfo* pInfo, SDataGroupInfo** pGroupInfo, int32_t len); static int32_t* setupColumnOffset(const SSDataBlock* pBlock, int32_t rowCapacity); -static void* getCurrentDataGroupInfo(const SPartitionOperatorInfo* pInfo, SDataGroupInfo** pGroupInfo, int32_t len); +static int32_t setGroupResultOutputBuf(SOperatorInfo* pOperator, SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t bytes, + int32_t groupId, SDiskbasedBuf* pBuf, SAggSupporter* pAggSup); static void destroyGroupOperatorInfo(void* param, int32_t numOfOutput) { SGroupbyOperatorInfo* pInfo = (SGroupbyOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); taosMemoryFreeClear(pInfo->keyBuf); taosArrayDestroy(pInfo->pGroupCols); taosArrayDestroy(pInfo->pGroupColVals); + cleanupExecSupp(&pInfo->scalarSup); } static int32_t initGroupOptrInfo(SArray** pGroupColVals, int32_t* keyLen, char** keyBuf, const SArray* pGroupColList) { @@ -214,7 +217,7 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SGroupbyOperatorInfo* pInfo = pOperator->info; - SqlFunctionCtx* pCtx = pInfo->binfo.pCtx; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; int32_t numOfGroupCols = taosArrayGetSize(pInfo->pGroupCols); // if (type == TSDB_DATA_TYPE_FLOAT || type == TSDB_DATA_TYPE_DOUBLE) { // qError("QInfo:0x%"PRIx64" group by not supported on double/float columns, abort", GET_TASKID(pRuntimeEnv)); @@ -248,16 +251,16 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) { } len = buildGroupKeys(pInfo->keyBuf, pInfo->pGroupColVals); - int32_t ret = setGroupResultOutputBuf(&(pInfo->binfo), pOperator->numOfExprs, pInfo->keyBuf, TSDB_DATA_TYPE_VARCHAR, len, 0, pInfo->aggSup.pResultBuf, pTaskInfo, &pInfo->aggSup); + int32_t ret = setGroupResultOutputBuf(pOperator, &(pInfo->binfo), pOperator->exprSupp.numOfExprs, pInfo->keyBuf, len, 0, pInfo->aggSup.pResultBuf, &pInfo->aggSup); if (ret != TSDB_CODE_SUCCESS) { // null data, too many state code longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } int32_t rowIndex = j - num; - doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->numOfExprs, TSDB_ORDER_ASC); + doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->exprSupp.numOfExprs, TSDB_ORDER_ASC); // assign the group keys or user input constant values if required - doAssignGroupKeys(pCtx, pOperator->numOfExprs, pBlock->info.rows, rowIndex); + doAssignGroupKeys(pCtx, pOperator->exprSupp.numOfExprs, pBlock->info.rows, rowIndex); recordNewGroupKeys(pInfo->pGroupCols, pInfo->pGroupColVals, pBlock, j); num = 1; } @@ -265,15 +268,15 @@ static void doHashGroupbyAgg(SOperatorInfo* pOperator, SSDataBlock* pBlock) { if (num > 0) { len = buildGroupKeys(pInfo->keyBuf, pInfo->pGroupColVals); int32_t ret = - setGroupResultOutputBuf(&(pInfo->binfo), pOperator->numOfExprs, pInfo->keyBuf, TSDB_DATA_TYPE_VARCHAR, len, - 0, pInfo->aggSup.pResultBuf, pTaskInfo, &pInfo->aggSup); + setGroupResultOutputBuf(pOperator, &(pInfo->binfo), pOperator->exprSupp.numOfExprs, pInfo->keyBuf, len, + 0, pInfo->aggSup.pResultBuf, &pInfo->aggSup); if (ret != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } int32_t rowIndex = pBlock->info.rows - num; - doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->numOfExprs, TSDB_ORDER_ASC); - doAssignGroupKeys(pCtx, pOperator->numOfExprs, pBlock->info.rows, rowIndex); + doApplyFunctions(pTaskInfo, pCtx, &w, NULL, rowIndex, num, NULL, pBlock->info.rows, pOperator->exprSupp.numOfExprs, TSDB_ORDER_ASC); + doAssignGroupKeys(pCtx, pOperator->exprSupp.numOfExprs, pBlock->info.rows, rowIndex); } } @@ -291,7 +294,7 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) { doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); size_t rows = pRes->info.rows; - if (rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } @@ -317,11 +320,11 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) { } // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pInfo->binfo.pCtx, pBlock, order, scanFlag, true); + setInputDataBlock(pOperator, pOperator->exprSupp.pCtx, pBlock, order, scanFlag, true); // there is an scalar expression that needs to be calculated right before apply the group aggregation. - if (pInfo->scalarSup.pScalarExprInfo != NULL) { - pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pScalarExprInfo, pBlock, pBlock, pInfo->scalarSup.pScalarFuncCtx, pInfo->scalarSup.numOfScalarExpr, NULL); + if (pInfo->scalarSup.pExprInfo != NULL) { + pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pExprInfo, pBlock, pBlock, pInfo->scalarSup.pCtx, pInfo->scalarSup.numOfExprs, NULL); if (pTaskInfo->code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, pTaskInfo->code); } @@ -355,7 +358,7 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) { doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); doFilter(pInfo->pCondition, pRes); - bool hasRemain = hashRemainDataInGroupInfo(&pInfo->groupResInfo); + bool hasRemain = hasDataInGroupInfo(&pInfo->groupResInfo); if (!hasRemain) { doSetOperatorCompleted(pOperator); break; @@ -384,9 +387,9 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SExprInfo* pEx pInfo->pGroupCols = pGroupColList; pInfo->pCondition = pCondition; - pInfo->scalarSup.pScalarExprInfo = pScalarExprInfo; - pInfo->scalarSup.numOfScalarExpr = numOfScalarExpr; - pInfo->scalarSup.pScalarFuncCtx = createSqlFunctionCtx(pScalarExprInfo, numOfScalarExpr, &pInfo->scalarSup.rowCellInfoOffset); + pInfo->scalarSup.pExprInfo = pScalarExprInfo; + pInfo->scalarSup.numOfExprs = numOfScalarExpr; + pInfo->scalarSup.pCtx = createSqlFunctionCtx(pScalarExprInfo, numOfScalarExpr, &pInfo->scalarSup.rowEntryInfoOffset); int32_t code = initGroupOptrInfo(&pInfo->pGroupColVals, &pInfo->groupKeyLen, &pInfo->keyBuf, pGroupColList); if (code != TSDB_CODE_SUCCESS) { @@ -394,15 +397,16 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SExprInfo* pEx } initResultSizeInfo(pOperator, 4096); - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResultBlock, pInfo->groupKeyLen, pTaskInfo->id.str); - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, pInfo->groupKeyLen, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResultBlock); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pOperator->name = "GroupbyAggOperator"; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; // pOperator->operatorType = OP_Groupby; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->pTaskInfo = pTaskInfo; @@ -439,9 +443,9 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) { // group id - size_t numOfCols = pOperator->numOfExprs; + size_t numOfCols = pOperator->exprSupp.numOfExprs; for(int32_t i = 0; i < numOfCols; ++i) { - SExprInfo* pExpr = &pOperator->pExpr[i]; + SExprInfo* pExpr = &pOperator->exprSupp.pExprInfo[i]; int32_t slotId = pExpr->base.pParam[0].pCol->slotId; SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, slotId); @@ -582,24 +586,30 @@ static void clearPartitionOperator(SPartitionOperatorInfo* pInfo) { while( (ite = taosHashIterate(pInfo->pGroupSet, ite)) != NULL ) { taosArrayDestroy( ((SDataGroupInfo *)ite)->pPageList); } - taosHashClear(pInfo->pGroupSet); + taosArrayClear(pInfo->sortedGroupArray); clearDiskbasedBuf(pInfo->pBuf); } +static int compareDataGroupInfo(const void* group1, const void* group2) { + const SDataGroupInfo* pGroupInfo1 = group1; + const SDataGroupInfo* pGroupInfo2 = group2; + return pGroupInfo1->groupId - pGroupInfo2->groupId; +} + static SSDataBlock* buildPartitionResult(SOperatorInfo* pOperator) { SPartitionOperatorInfo* pInfo = pOperator->info; - SDataGroupInfo* pGroupInfo = pInfo->pGroupIter; - if (pInfo->pGroupIter == NULL || pInfo->pageIndex >= taosArrayGetSize(pGroupInfo->pPageList)) { + SDataGroupInfo* pGroupInfo = (pInfo->groupIndex != -1) ? taosArrayGet(pInfo->sortedGroupArray, pInfo->groupIndex) : NULL; + if (pInfo->groupIndex == -1 || pInfo->pageIndex >= taosArrayGetSize(pGroupInfo->pPageList)) { // try next group data - pInfo->pGroupIter = taosHashIterate(pInfo->pGroupSet, pInfo->pGroupIter); - if (pInfo->pGroupIter == NULL) { + ++pInfo->groupIndex; + if (pInfo->groupIndex >= taosArrayGetSize(pInfo->sortedGroupArray)) { doSetOperatorCompleted(pOperator); clearPartitionOperator(pInfo); return NULL; } - pGroupInfo = pInfo->pGroupIter; + pGroupInfo = taosArrayGet(pInfo->sortedGroupArray, pInfo->groupIndex); pInfo->pageIndex = 0; } @@ -643,8 +653,8 @@ static SSDataBlock* hashPartition(SOperatorInfo* pOperator) { } // there is an scalar expression that needs to be calculated right before apply the group aggregation. - if (pInfo->scalarSupp.pScalarExprInfo != NULL) { - pTaskInfo->code = projectApplyFunctions(pInfo->scalarSupp.pScalarExprInfo, pBlock, pBlock, pInfo->scalarSupp.pScalarFuncCtx, pInfo->scalarSupp.numOfScalarExpr, NULL); + if (pInfo->scalarSup.pExprInfo != NULL) { + pTaskInfo->code = projectApplyFunctions(pInfo->scalarSup.pExprInfo, pBlock, pBlock, pInfo->scalarSup.pCtx, pInfo->scalarSup.numOfExprs, NULL); if (pTaskInfo->code != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, pTaskInfo->code); } @@ -653,6 +663,20 @@ static SSDataBlock* hashPartition(SOperatorInfo* pOperator) { doHashPartition(pOperator, pBlock); } + SArray* groupArray = taosArrayInit(taosHashGetSize(pInfo->pGroupSet), sizeof(SDataGroupInfo)); + void* pGroupIter = NULL; + pGroupIter = taosHashIterate(pInfo->pGroupSet, NULL); + while (pGroupIter != NULL) { + SDataGroupInfo* pGroupInfo = pGroupIter; + taosArrayPush(groupArray, pGroupInfo); + pGroupIter = taosHashIterate(pInfo->pGroupSet, pGroupIter); + } + + taosArraySort(groupArray, compareDataGroupInfo); + pInfo->sortedGroupArray = groupArray; + pInfo->groupIndex = -1; + taosHashClear(pInfo->pGroupSet); + pOperator->cost.openCost = (taosGetTimestampUs() - st) / 1000.0; pOperator->status = OP_RES_TO_RETURN; @@ -662,16 +686,21 @@ static SSDataBlock* hashPartition(SOperatorInfo* pOperator) { static void destroyPartitionOperatorInfo(void* param, int32_t numOfOutput) { SPartitionOperatorInfo* pInfo = (SPartitionOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); taosArrayDestroy(pInfo->pGroupCols); + for(int i = 0; i < taosArrayGetSize(pInfo->pGroupColVals); i++){ SGroupKeys key = *(SGroupKeys*)taosArrayGet(pInfo->pGroupColVals, i); taosMemoryFree(key.pData); } + taosArrayDestroy(pInfo->pGroupColVals); taosMemoryFree(pInfo->keyBuf); + taosArrayDestroy(pInfo->sortedGroupArray); taosHashCleanup(pInfo->pGroupSet); taosMemoryFree(pInfo->columnOffset); + + cleanupExecSupp(&pInfo->scalarSup); } SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartitionPhysiNode* pPartNode, SExecTaskInfo* pTaskInfo) { @@ -689,10 +718,10 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition pInfo->pGroupCols = extractPartitionColInfo(pPartNode->pPartitionKeys); if (pPartNode->pExprs != NULL) { - pInfo->scalarSupp.numOfScalarExpr = 0; - pInfo->scalarSupp.pScalarExprInfo = createExprInfo(pPartNode->pExprs, NULL, &pInfo->scalarSupp.numOfScalarExpr); - pInfo->scalarSupp.pScalarFuncCtx = createSqlFunctionCtx( - pInfo->scalarSupp.pScalarExprInfo, pInfo->scalarSupp.numOfScalarExpr, &pInfo->scalarSupp.rowCellInfoOffset); + pInfo->scalarSup.numOfExprs = 0; + pInfo->scalarSup.pExprInfo = createExprInfo(pPartNode->pExprs, NULL, &pInfo->scalarSup.numOfExprs); + pInfo->scalarSup.pCtx = createSqlFunctionCtx( + pInfo->scalarSup.pExprInfo, pInfo->scalarSup.numOfExprs, &pInfo->scalarSup.rowEntryInfoOffset); } _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY); @@ -722,8 +751,8 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition pOperator->status = OP_NOT_OPENED; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_PARTITION; pInfo->binfo.pRes = pResBlock; - pOperator->numOfExprs = numOfCols; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->info = pInfo; pOperator->pTaskInfo = pTaskInfo; @@ -738,4 +767,18 @@ SOperatorInfo* createPartitionOperatorInfo(SOperatorInfo* downstream, SPartition taosMemoryFreeClear(pInfo); taosMemoryFreeClear(pOperator); return NULL; +} + +int32_t setGroupResultOutputBuf(SOperatorInfo* pOperator, SOptrBasicInfo* binfo, int32_t numOfCols, char* pData, int16_t bytes, + int32_t groupId, SDiskbasedBuf* pBuf, SAggSupporter* pAggSup) { + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SResultRowInfo* pResultRowInfo = &binfo->resultRowInfo; + SqlFunctionCtx* pCtx = pOperator->exprSupp.pCtx; + + SResultRow* pResultRow = + doSetResultOutBufByKey(pBuf, pResultRowInfo, (char*)pData, bytes, true, groupId, pTaskInfo, false, pAggSup); + assert(pResultRow != NULL); + + setResultRowInitCtx(pResultRow, pCtx, numOfCols, pOperator->exprSupp.rowEntryInfoOffset); + return TSDB_CODE_SUCCESS; } \ No newline at end of file diff --git a/source/libs/executor/src/joinoperator.c b/source/libs/executor/src/joinoperator.c index 7c8ab244a1d6cfc9a5e358ab1fbc174d7ea1ce06..6fbda778087b345c7aeef19e068f582451eab006 100644 --- a/source/libs/executor/src/joinoperator.c +++ b/source/libs/executor/src/joinoperator.c @@ -28,27 +28,32 @@ static SSDataBlock* doMergeJoin(struct SOperatorInfo* pOperator); static void destroyMergeJoinOperator(void* param, int32_t numOfOutput); static void extractTimeCondition(SJoinOperatorInfo* Info, SLogicConditionNode* pLogicConditionNode); -SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SExprInfo* pExprInfo, - int32_t numOfCols, SSDataBlock* pResBlock, SNode* pOnCondition, - SExecTaskInfo* pTaskInfo) { +SOperatorInfo* createMergeJoinOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SJoinPhysiNode* pJoinNode, + SExecTaskInfo* pTaskInfo) { SJoinOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SJoinOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); if (pOperator == NULL || pInfo == NULL) { goto _error; } + SSDataBlock* pResBlock = createResDataBlock(pJoinNode->node.pOutputDataBlockDesc); + + int32_t numOfCols = 0; + SExprInfo* pExprInfo = createExprInfo(pJoinNode->pTargets, NULL, &numOfCols); + initResultSizeInfo(pOperator, 4096); - pInfo->pRes = pResBlock; - pOperator->name = "MergeJoinOperator"; + pInfo->pRes = pResBlock; + pOperator->name = "MergeJoinOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN; - pOperator->blocking = false; - pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; - pOperator->info = pInfo; - pOperator->pTaskInfo = pTaskInfo; - + pOperator->blocking = false; + pOperator->status = OP_NOT_OPENED; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; + pOperator->info = pInfo; + pOperator->pTaskInfo = pTaskInfo; + + SNode* pOnCondition = pJoinNode->pOnConditions; if (nodeType(pOnCondition) == QUERY_NODE_OPERATOR) { SOperatorNode* pNode = (SOperatorNode*)pOnCondition; setJoinColumnInfo(&pInfo->leftCol, (SColumnNode*)pNode->pLeft); @@ -127,10 +132,10 @@ SSDataBlock* doMergeJoin(struct SOperatorInfo* pOperator) { // only the timestamp match support for ordinary table ASSERT(pLeftCol->info.type == TSDB_DATA_TYPE_TIMESTAMP); if (*(int64_t*)pLeftVal == *(int64_t*)pRightVal) { - for (int32_t i = 0; i < pOperator->numOfExprs; ++i) { + for (int32_t i = 0; i < pOperator->exprSupp.numOfExprs; ++i) { SColumnInfoData* pDst = taosArrayGet(pRes->pDataBlock, i); - SExprInfo* pExprInfo = &pOperator->pExpr[i]; + SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[i]; int32_t blockId = pExprInfo->base.pParam[0].pCol->dataBlockId; int32_t slotId = pExprInfo->base.pParam[0].pCol->slotId; diff --git a/source/libs/executor/src/scanoperator.c b/source/libs/executor/src/scanoperator.c index a8aad8fe688fc4b363674a1dd35c58ecddfe98c6..e4f940d991789d7341f558de70e0ed8785fb82a2 100644 --- a/source/libs/executor/src/scanoperator.c +++ b/source/libs/executor/src/scanoperator.c @@ -13,6 +13,7 @@ * along with this program. If not, see . */ +#include #include "filter.h" #include "function.h" #include "functionMgt.h" @@ -258,12 +259,12 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanInfo* pTableSca return terrno; } - relocateColumnData(pBlock, pTableScanInfo->pColMatchInfo, pCols); + relocateColumnData(pBlock, pTableScanInfo->pColMatchInfo, pCols, true); // currently only the tbname pseudo column - if (pTableScanInfo->numOfPseudoExpr > 0) { - addTagPseudoColumnData(&pTableScanInfo->readHandle, pTableScanInfo->pPseudoExpr, pTableScanInfo->numOfPseudoExpr, - pBlock); + if (pTableScanInfo->pseudoSup.numOfExprs > 0) { + SExprSupp* pSup = &pTableScanInfo->pseudoSup; + addTagPseudoColumnData(&pTableScanInfo->readHandle, pSup->pExprInfo, pSup->numOfExprs, pBlock); } int64_t st = taosGetTimestampMs(); @@ -367,13 +368,14 @@ void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData* p static SSDataBlock* doTableScanImpl(SOperatorInfo* pOperator) { STableScanInfo* pTableScanInfo = pOperator->info; + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SSDataBlock* pBlock = pTableScanInfo->pResBlock; int64_t st = taosGetTimestampUs(); while (tsdbNextDataBlock(pTableScanInfo->dataReader)) { - if (isTaskKilled(pOperator->pTaskInfo)) { - longjmp(pOperator->pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED); + if (isTaskKilled(pTaskInfo)) { + longjmp(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED); } // process this data block based on the probabilities @@ -396,7 +398,7 @@ static SSDataBlock* doTableScanImpl(SOperatorInfo* pOperator) { continue; } - uint64_t* groupId = taosHashGet(pOperator->pTaskInfo->tableqinfoList.map, &pBlock->info.uid, sizeof(int64_t)); + uint64_t* groupId = taosHashGet(pTaskInfo->tableqinfoList.map, &pBlock->info.uid, sizeof(int64_t)); if (groupId) { pBlock->info.groupId = *groupId; } @@ -496,18 +498,6 @@ static SSDataBlock* doTableScan(SOperatorInfo* pOperator) { return NULL; } -SInterval extractIntervalInfo(const STableScanPhysiNode* pTableScanNode) { - SInterval interval = { - .interval = pTableScanNode->interval, - .sliding = pTableScanNode->sliding, - .intervalUnit = pTableScanNode->intervalUnit, - .slidingUnit = pTableScanNode->slidingUnit, - .offset = pTableScanNode->offset, - }; - - return interval; -} - static int32_t getTableScannerExecInfo(struct SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len) { SFileBlockLoadRecorder* pRecorder = taosMemoryCalloc(1, sizeof(SFileBlockLoadRecorder)); STableScanInfo* pTableScanInfo = pOptr->info; @@ -520,7 +510,7 @@ static int32_t getTableScannerExecInfo(struct SOperatorInfo* pOptr, void** pOptr static void destroyTableScanOperatorInfo(void* param, int32_t numOfOutput) { STableScanInfo* pTableScanInfo = (STableScanInfo*)param; blockDataDestroy(pTableScanInfo->pResBlock); - clearupQueryTableDataCond(&pTableScanInfo->cond); + cleanupQueryTableDataCond(&pTableScanInfo->cond); tsdbCleanupReadHandle(pTableScanInfo->dataReader); @@ -537,11 +527,12 @@ SOperatorInfo* createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode, goto _error; } + // taosSsleep(20); + SDataBlockDescNode* pDescNode = pTableScanNode->scan.node.pOutputDataBlockDesc; int32_t numOfCols = 0; - SArray* pColList = - extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, pTaskInfo, COL_MATCH_FROM_COL_ID); + SArray* pColList = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID); int32_t code = initQueryTableDataCond(&pInfo->cond, pTableScanNode); if (code != TSDB_CODE_SUCCESS) { @@ -549,8 +540,9 @@ SOperatorInfo* createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode, } if (pTableScanNode->scan.pScanPseudoCols != NULL) { - pInfo->pPseudoExpr = createExprInfo(pTableScanNode->scan.pScanPseudoCols, NULL, &pInfo->numOfPseudoExpr); - pInfo->pPseudoCtx = createSqlFunctionCtx(pInfo->pPseudoExpr, pInfo->numOfPseudoExpr, &pInfo->rowCellInfoOffset); + SExprSupp* pSup = &pInfo->pseudoSup; + pSup->pExprInfo = createExprInfo(pTableScanNode->scan.pScanPseudoCols, NULL, &pSup->numOfExprs); + pSup->pCtx = createSqlFunctionCtx(pSup->pExprInfo, pSup->numOfExprs, &pSup->rowEntryInfoOffset); } pInfo->scanInfo = (SScanInfo){.numOfAsc = pTableScanNode->scanSeq[0], .numOfDesc = pTableScanNode->scanSeq[1]}; @@ -574,7 +566,7 @@ SOperatorInfo* createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode, pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doTableScan, NULL, NULL, destroyTableScanOperatorInfo, @@ -599,44 +591,76 @@ SOperatorInfo* createTableSeqScanOperatorInfo(void* pReadHandle, SExecTaskInfo* pInfo->dataReader = pReadHandle; // pInfo->prevGroupId = -1; - pOperator->name = "TableSeqScanOperator"; + pOperator->name = "TableSeqScanOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_TABLE_SEQ_SCAN; - pOperator->blocking = false; - pOperator->status = OP_NOT_OPENED; - pOperator->info = pInfo; - pOperator->pTaskInfo = pTaskInfo; + pOperator->blocking = false; + pOperator->status = OP_NOT_OPENED; + pOperator->info = pInfo; + pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doTableScanImpl, NULL, NULL, NULL, NULL, NULL, NULL); return pOperator; } +static int32_t doGetTableRowSize(void* pMeta, uint64_t uid) { + int32_t rowLen = 0; + + SMetaReader mr = {0}; + metaReaderInit(&mr, pMeta, 0); + metaGetTableEntryByUid(&mr, uid); + if (mr.me.type == TSDB_SUPER_TABLE) { + int32_t numOfCols = mr.me.stbEntry.schemaRow.nCols; + for(int32_t i = 0; i < numOfCols; ++i) { + rowLen += mr.me.stbEntry.schemaRow.pSchema[i].bytes; + } + } else if (mr.me.type == TSDB_CHILD_TABLE) { + uint64_t suid = mr.me.ctbEntry.suid; + metaGetTableEntryByUid(&mr, suid); + int32_t numOfCols = mr.me.stbEntry.schemaRow.nCols; + + for(int32_t i = 0; i < numOfCols; ++i) { + rowLen += mr.me.stbEntry.schemaRow.pSchema[i].bytes; + } + } else if (mr.me.type == TSDB_NORMAL_TABLE) { + int32_t numOfCols = mr.me.ntbEntry.schemaRow.nCols; + for(int32_t i = 0; i < numOfCols; ++i) { + rowLen += mr.me.ntbEntry.schemaRow.pSchema[i].bytes; + } + } + + metaReaderClear(&mr); + return rowLen; +} + static SSDataBlock* doBlockInfoScan(SOperatorInfo* pOperator) { if (pOperator->status == OP_EXEC_DONE) { return NULL; } - STableScanInfo* pTableScanInfo = pOperator->info; + SBlockDistInfo* pBlockScanInfo = pOperator->info; - STableBlockDistInfo blockDistInfo = {0}; - blockDistInfo.maxRows = INT_MIN; - blockDistInfo.minRows = INT_MAX; + STableBlockDistInfo blockDistInfo = {.minRows = INT_MAX, .maxRows = INT_MIN}; + blockDistInfo.rowSize = doGetTableRowSize(pBlockScanInfo->readHandle.meta, pBlockScanInfo->uid); - tsdbGetFileBlocksDistInfo(pTableScanInfo->dataReader, &blockDistInfo); - blockDistInfo.numOfInmemRows = (int32_t)tsdbGetNumOfRowsInMemTable(pTableScanInfo->dataReader); + tsdbGetFileBlocksDistInfo(pBlockScanInfo->pHandle, &blockDistInfo); + blockDistInfo.numOfInmemRows = (int32_t)tsdbGetNumOfRowsInMemTable(pBlockScanInfo->pHandle); - SSDataBlock* pBlock = pTableScanInfo->pResBlock; - pBlock->info.rows = 1; + SSDataBlock* pBlock = pBlockScanInfo->pResBlock; - SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, 0); + int32_t slotId = pOperator->exprSupp.pExprInfo->base.resSchema.slotId; + SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, slotId); int32_t len = tSerializeBlockDistInfo(NULL, 0, &blockDistInfo); char* p = taosMemoryCalloc(1, len + VARSTR_HEADER_SIZE); tSerializeBlockDistInfo(varDataVal(p), len, &blockDistInfo); varDataSetLen(p, len); + blockDataEnsureCapacity(pBlock, 1); colDataAppend(pColInfo, 0, p, false); taosMemoryFree(p); + pBlock->info.rows = 1; + pOperator->status = OP_EXEC_DONE; return pBlock; } @@ -646,7 +670,8 @@ static void destroyBlockDistScanOperatorInfo(void* param, int32_t numOfOutput) { blockDataDestroy(pDistInfo->pResBlock); } -SOperatorInfo* createDataBlockInfoScanOperator(void* dataReader, SExecTaskInfo* pTaskInfo) { +SOperatorInfo* createDataBlockInfoScanOperator(void* dataReader, SReadHandle* readHandle, uint64_t uid, + SBlockDistScanPhysiNode* pBlockScanNode, SExecTaskInfo* pTaskInfo) { SBlockDistInfo* pInfo = taosMemoryCalloc(1, sizeof(SBlockDistInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); if (pInfo == NULL || pOperator == NULL) { @@ -654,21 +679,20 @@ SOperatorInfo* createDataBlockInfoScanOperator(void* dataReader, SExecTaskInfo* goto _error; } - pInfo->pHandle = dataReader; - - pInfo->pResBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); - - SColumnInfoData infoData = {0}; - infoData.info.type = TSDB_DATA_TYPE_VARCHAR; - infoData.info.bytes = 1024; - - taosArrayPush(pInfo->pResBlock->pDataBlock, &infoData); + pInfo->pHandle = dataReader; + pInfo->readHandle = *readHandle; + pInfo->uid = uid; + pInfo->pResBlock = createResDataBlock(pBlockScanNode->node.pOutputDataBlockDesc); - pOperator->name = "DataBlockInfoScanOperator"; - // pOperator->operatorType = OP_TableBlockInfoScan; - pOperator->blocking = false; - pOperator->status = OP_NOT_OPENED; - pOperator->info = pInfo; + int32_t numOfCols = 0; + SExprInfo* pExprInfo = createExprInfo(pBlockScanNode->pScanPseudoCols, NULL, &numOfCols); + initExprSupp(&pOperator->exprSupp, pExprInfo, numOfCols); + + pOperator->name = "DataBlockDistScanOperator"; + pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN; + pOperator->blocking = false; + pOperator->status = OP_NOT_OPENED; + pOperator->info = pInfo; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doBlockInfoScan, NULL, NULL, @@ -717,10 +741,10 @@ static bool prepareDataScan(SStreamBlockScanInfo* pInfo) { int64_t gap = pInfo->sessionSup.gap; int32_t winIndex = 0; SResultWindowInfo* pCurWin = - getSessionTimeWindow(pAggSup, tsCols[pInfo->updateResIndex], pSDB->info.groupId, gap, &winIndex); + getSessionTimeWindow(pAggSup, tsCols[pInfo->updateResIndex], INT64_MIN, pSDB->info.groupId, gap, &winIndex); win = pCurWin->win; pInfo->updateResIndex += - updateSessionWindowInfo(pCurWin, tsCols, pSDB->info.rows, pInfo->updateResIndex, gap, NULL); + updateSessionWindowInfo(pCurWin, tsCols, NULL, pSDB->info.rows, pInfo->updateResIndex, gap, NULL); } else { win = getActiveTimeWindow(NULL, &dumyInfo, tsCols[pInfo->updateResIndex], &pInfo->interval, pInfo->interval.precision, NULL); @@ -798,23 +822,23 @@ static SSDataBlock* doDataScan(SStreamBlockScanInfo* pInfo) { if (!pResult) { return NULL; } - + if (pResult->info.groupId == pInfo->groupId) { return pResult; } } -/* Todo(liuyao) for partition by column - SSDataBlock* pBlock = createOneDataBlock(pResult, true); - blockDataCleanup(pResult); - for (int32_t i = 0; i < pBlock->info.rows; i++) { - uint64_t id = getGroupId(pInfo->pOperatorDumy, pBlock, i); - if (id == pInfo->groupId) { - copyOneRow(pResult, pBlock, i); + /* Todo(liuyao) for partition by column + SSDataBlock* pBlock = createOneDataBlock(pResult, true); + blockDataCleanup(pResult); + for (int32_t i = 0; i < pBlock->info.rows; i++) { + uint64_t id = getGroupId(pInfo->pOperatorDumy, pBlock, i); + if (id == pInfo->groupId) { + copyOneRow(pResult, pBlock, i); + } } - } - return pResult; -*/ + return pResult; + */ } static void setUpdateData(SStreamBlockScanInfo* pInfo, SSDataBlock* pBlock, SSDataBlock* pUpdateBlock) { @@ -829,7 +853,7 @@ static void setUpdateData(SStreamBlockScanInfo* pInfo, SSDataBlock* pBlock, SSDa int32_t rowId = *(int32_t*)taosArrayGet(pInfo->tsArray, pInfo->tsArrayIndex); pInfo->groupId = getGroupId(pInfo->pOperatorDumy, pBlock, rowId); int32_t i = 0; - for ( ; i < size; i++) { + for (; i < size; i++) { rowId = *(int32_t*)taosArrayGet(pInfo->tsArray, i + pInfo->tsArrayIndex); uint64_t id = getGroupId(pInfo->pOperatorDumy, pBlock, rowId); if (pInfo->groupId != id) { @@ -1059,13 +1083,9 @@ SOperatorInfo* createStreamScanOperatorInfo(void* pDataReader, SReadHandle* pHan SScanPhysiNode* pScanPhyNode = &pTableScanNode->scan; SDataBlockDescNode* pDescNode = pScanPhyNode->node.pOutputDataBlockDesc; - SOperatorInfo* pTableScanDummy = createTableScanOperatorInfo(pTableScanNode, pDataReader, pHandle, pTaskInfo); - - STableScanInfo* pSTInfo = (STableScanInfo*)pTableScanDummy->info; int32_t numOfCols = 0; - pInfo->pColMatchInfo = - extractColMatchInfo(pScanPhyNode->pScanCols, pDescNode, &numOfCols, pTaskInfo, COL_MATCH_FROM_COL_ID); + pInfo->pColMatchInfo = extractColMatchInfo(pScanPhyNode->pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID); int32_t numOfOutput = taosArrayGetSize(pInfo->pColMatchInfo); SArray* pColIds = taosArrayInit(numOfOutput, sizeof(int16_t)); @@ -1079,16 +1099,6 @@ SOperatorInfo* createStreamScanOperatorInfo(void* pDataReader, SReadHandle* pHan } } - // set the extract column id to streamHandle - tqReadHandleSetColIdList((STqReadHandle*)pHandle->reader, pColIds); - SArray* tableIdList = extractTableIdList(&pTaskInfo->tableqinfoList); - int32_t code = tqReadHandleSetTbUidList(pHandle->reader, tableIdList); - if (code != 0) { - taosArrayDestroy(tableIdList); - goto _error; - } - taosArrayDestroy(tableIdList); - pInfo->pBlockLists = taosArrayInit(4, POINTER_BYTES); if (pInfo->pBlockLists == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; @@ -1100,10 +1110,31 @@ SOperatorInfo* createStreamScanOperatorInfo(void* pDataReader, SReadHandle* pHan goto _error; } - if (pSTInfo->interval.interval > 0 && pDataReader) { - pInfo->pUpdateInfo = updateInfoInitP(&pSTInfo->interval, pTwSup->waterMark); - } else { - pInfo->pUpdateInfo = NULL; + if (pHandle) { + SOperatorInfo* pTableScanDummy = createTableScanOperatorInfo(pTableScanNode, pDataReader, pHandle, pTaskInfo); + STableScanInfo* pSTInfo = (STableScanInfo*)pTableScanDummy->info; + if (pSTInfo->interval.interval > 0) { + pInfo->pUpdateInfo = updateInfoInitP(&pSTInfo->interval, pTwSup->waterMark); + } else { + pInfo->pUpdateInfo = NULL; + } + pInfo->pOperatorDumy = pTableScanDummy; + pInfo->interval = pSTInfo->interval; + + pInfo->readHandle = *pHandle; + ASSERT(pHandle->reader); + pInfo->streamBlockReader = pHandle->reader; + pInfo->tableUid = pScanPhyNode->uid; + + // set the extract column id to streamHandle + tqReadHandleSetColIdList((STqReadHandle*)pHandle->reader, pColIds); + SArray* tableIdList = extractTableIdList(&pTaskInfo->tableqinfoList); + int32_t code = tqReadHandleSetTbUidList(pHandle->reader, tableIdList); + if (code != 0) { + taosArrayDestroy(tableIdList); + goto _error; + } + taosArrayDestroy(tableIdList); } // create the pseduo columns info @@ -1111,25 +1142,20 @@ SOperatorInfo* createStreamScanOperatorInfo(void* pDataReader, SReadHandle* pHan pInfo->pPseudoExpr = createExprInfo(pTableScanNode->scan.pScanPseudoCols, NULL, &pInfo->numOfPseudoExpr); } - pInfo->readHandle = *pHandle; - pInfo->tableUid = pScanPhyNode->uid; - pInfo->streamBlockReader = pHandle->reader; pInfo->pRes = createResDataBlock(pDescNode); pInfo->pUpdateRes = createResDataBlock(pDescNode); pInfo->pCondition = pScanPhyNode->node.pConditions; pInfo->pDataReader = pDataReader; pInfo->scanMode = STREAM_SCAN_FROM_READERHANDLE; - pInfo->pOperatorDumy = pTableScanDummy; - pInfo->interval = pSTInfo->interval; pInfo->sessionSup = (SessionWindowSupporter){.pStreamAggSup = NULL, .gap = -1}; pInfo->groupId = 0; - + pOperator->name = "StreamBlockScanOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = pInfo->pRes->info.numOfCols; + pOperator->exprSupp.numOfExprs = pInfo->pRes->info.numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = @@ -1379,12 +1405,6 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { pColInfoData = taosArrayGet(p->pDataBlock, 6); colDataAppend(pColInfoData, numOfRows, (char*)&vgId, false); - // table comment - // todo: set the correct comment - pColInfoData = taosArrayGet(p->pDataBlock, 8); - colDataAppendNULL(pColInfoData, numOfRows); - - char str[256] = {0}; int32_t tableType = pInfo->pCur->mr.me.type; if (tableType == TSDB_CHILD_TABLE) { // create time @@ -1401,11 +1421,25 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { colDataAppend(pColInfoData, numOfRows, (char*)&mr.me.stbEntry.schemaRow.nCols, false); // super table name - STR_TO_VARSTR(str, mr.me.name); + STR_TO_VARSTR(n, mr.me.name); pColInfoData = taosArrayGet(p->pDataBlock, 4); - colDataAppend(pColInfoData, numOfRows, str, false); + colDataAppend(pColInfoData, numOfRows, n, false); metaReaderClear(&mr); + // table comment + pColInfoData = taosArrayGet(p->pDataBlock, 8); + if (pInfo->pCur->mr.me.ctbEntry.commentLen > 0) { + char comment[TSDB_TB_COMMENT_LEN + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, pInfo->pCur->mr.me.ctbEntry.comment); + colDataAppend(pColInfoData, numOfRows, comment, false); + } else if (pInfo->pCur->mr.me.ctbEntry.commentLen == 0) { + char comment[VARSTR_HEADER_SIZE + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, ""); + colDataAppend(pColInfoData, numOfRows, comment, false); + } else { + colDataAppendNULL(pColInfoData, numOfRows); + } + // uid pColInfoData = taosArrayGet(p->pDataBlock, 5); colDataAppend(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.uid, false); @@ -1414,7 +1448,7 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { pColInfoData = taosArrayGet(p->pDataBlock, 7); colDataAppend(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.ctbEntry.ttlDays, false); - STR_TO_VARSTR(str, "CHILD_TABLE"); + STR_TO_VARSTR(n, "CHILD_TABLE"); } else if (tableType == TSDB_NORMAL_TABLE) { // create time pColInfoData = taosArrayGet(p->pDataBlock, 2); @@ -1428,6 +1462,20 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { pColInfoData = taosArrayGet(p->pDataBlock, 4); colDataAppendNULL(pColInfoData, numOfRows); + // table comment + pColInfoData = taosArrayGet(p->pDataBlock, 8); + if (pInfo->pCur->mr.me.ntbEntry.commentLen > 0) { + char comment[TSDB_TB_COMMENT_LEN + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, pInfo->pCur->mr.me.ntbEntry.comment); + colDataAppend(pColInfoData, numOfRows, comment, false); + } else if (pInfo->pCur->mr.me.ntbEntry.commentLen == 0) { + char comment[VARSTR_HEADER_SIZE + VARSTR_HEADER_SIZE] = {0}; + STR_TO_VARSTR(comment, ""); + colDataAppend(pColInfoData, numOfRows, comment, false); + } else { + colDataAppendNULL(pColInfoData, numOfRows); + } + // uid pColInfoData = taosArrayGet(p->pDataBlock, 5); colDataAppend(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.uid, false); @@ -1436,11 +1484,11 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { pColInfoData = taosArrayGet(p->pDataBlock, 7); colDataAppend(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.ntbEntry.ttlDays, false); - STR_TO_VARSTR(str, "NORMAL_TABLE"); + STR_TO_VARSTR(n, "NORMAL_TABLE"); } pColInfoData = taosArrayGet(p->pDataBlock, 9); - colDataAppend(pColInfoData, numOfRows, str, false); + colDataAppend(pColInfoData, numOfRows, n, false); if (++numOfRows >= pOperator->resultInfo.capacity) { break; @@ -1457,7 +1505,7 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { p->info.rows = numOfRows; pInfo->pRes->info.rows = numOfRows; - relocateColumnData(pInfo->pRes, pInfo->scanCols, p->pDataBlock); + relocateColumnData(pInfo->pRes, pInfo->scanCols, p->pDataBlock, false); doFilterResult(pInfo); blockDataDestroy(p); @@ -1523,8 +1571,8 @@ static SSDataBlock* doSysTableScan(SOperatorInfo* pOperator) { } } - setDataBlockFromFetchRsp(pInfo->pRes, &pInfo->loadInfo, pRsp->numOfRows, pRsp->data, pRsp->compLen, - pOperator->numOfExprs, startTs, NULL, pInfo->scanCols); + extractDataBlockFromFetchRsp(pInfo->pRes, &pInfo->loadInfo, pRsp->numOfRows, pRsp->data, pRsp->compLen, + pOperator->exprSupp.numOfExprs, startTs, NULL, pInfo->scanCols); // todo log the filter info doFilterResult(pInfo); @@ -1549,7 +1597,7 @@ int32_t buildSysDbTableInfo(const SSysTableScanInfo* pInfo, int32_t capacity) { getPerfDbMeta(&pSysDbTableMeta, &size); p->info.rows = buildDbTableInfoBlock(p, pSysDbTableMeta, size, TSDB_PERFORMANCE_SCHEMA_DB); - relocateColumnData(pInfo->pRes, pInfo->scanCols, p->pDataBlock); + relocateColumnData(pInfo->pRes, pInfo->scanCols, p->pDataBlock, false); pInfo->pRes->info.rows = p->info.rows; blockDataDestroy(p); @@ -1612,7 +1660,7 @@ SOperatorInfo* createSysTableScanOperatorInfo(void* readHandle, SSystemTableScan SSDataBlock* pResBlock = createResDataBlock(pDescNode); int32_t num = 0; - SArray* colList = extractColMatchInfo(pScanNode->pScanCols, pDescNode, &num, pTaskInfo, COL_MATCH_FROM_COL_ID); + SArray* colList = extractColMatchInfo(pScanNode->pScanCols, pDescNode, &num, COL_MATCH_FROM_COL_ID); pInfo->accountId = pScanPhyNode->accountId; pInfo->showRewrite = pScanPhyNode->showRewrite; @@ -1639,7 +1687,7 @@ SOperatorInfo* createSysTableScanOperatorInfo(void* readHandle, SSystemTableScan pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = pResBlock->info.numOfCols; + pOperator->exprSupp.numOfExprs = pResBlock->info.numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = @@ -1670,11 +1718,11 @@ static SSDataBlock* doTagScan(SOperatorInfo* pOperator) { int32_t count = 0; SArray* pa = GET_TABLEGROUP(pRuntimeEnv, 0); - int32_t functionId = getExprFunctionId(&pOperator->pExpr[0]); + int32_t functionId = getExprFunctionId(&pOperator->exprSupp.pExprInfo[0]); if (functionId == FUNCTION_TID_TAG) { // return the tags & table Id assert(pQueryAttr->numOfOutput == 1); - SExprInfo* pExprInfo = &pOperator->pExpr[0]; + SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[0]; int32_t rsize = pExprInfo->base.resSchema.bytes; count = 0; @@ -1737,7 +1785,7 @@ static SSDataBlock* doTagScan(SOperatorInfo* pOperator) { #endif STagScanInfo* pInfo = pOperator->info; - SExprInfo* pExprInfo = &pOperator->pExpr[0]; + SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[0]; SSDataBlock* pRes = pInfo->pRes; int32_t size = taosArrayGetSize(pInfo->pTableList->pTableList); @@ -1755,7 +1803,7 @@ static SSDataBlock* doTagScan(SOperatorInfo* pOperator) { STableKeyInfo* item = taosArrayGet(pInfo->pTableList->pTableList, pInfo->curPos); metaGetTableEntryByUid(&mr, item->uid); - for (int32_t j = 0; j < pOperator->numOfExprs; ++j) { + for (int32_t j = 0; j < pOperator->exprSupp.numOfExprs; ++j) { SColumnInfoData* pDst = taosArrayGet(pRes->pDataBlock, pExprInfo[j].base.resSchema.slotId); // refactor later @@ -1818,27 +1866,28 @@ SOperatorInfo* createTagScanOperatorInfo(SReadHandle* pReadHandle, STagScanPhysi SDataBlockDescNode* pDescNode = pPhyNode->node.pOutputDataBlockDesc; + int32_t num = 0; int32_t numOfExprs = 0; SExprInfo* pExprInfo = createExprInfo(pPhyNode->pScanPseudoCols, NULL, &numOfExprs); + SArray* colList = extractColMatchInfo(pPhyNode->pScanPseudoCols, pDescNode, &num, COL_MATCH_FROM_COL_ID); - int32_t num = 0; - SArray* colList = extractColMatchInfo(pPhyNode->pScanPseudoCols, pDescNode, &num, pTaskInfo, COL_MATCH_FROM_COL_ID); - pInfo->pTableList = pTableListInfo; - pInfo->pColMatchInfo = colList; - pInfo->pRes = createResDataBlock(pDescNode); - ; - pInfo->readHandle = *pReadHandle; - pInfo->curPos = 0; - pInfo->pFilterNode = pPhyNode->node.pConditions; + initExprSupp(&pOperator->exprSupp, pExprInfo, numOfExprs); + + pInfo->pTableList = pTableListInfo; + pInfo->pColMatchInfo = colList; + pInfo->pRes = createResDataBlock(pDescNode); + pInfo->readHandle = *pReadHandle; + pInfo->curPos = 0; + pInfo->pFilterNode = pPhyNode->node.pConditions; + pOperator->name = "TagScanOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN; - pOperator->blocking = false; - pOperator->status = OP_NOT_OPENED; - pOperator->info = pInfo; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfExprs; - pOperator->pTaskInfo = pTaskInfo; + + pOperator->blocking = false; + pOperator->status = OP_NOT_OPENED; + pOperator->info = pInfo; + pOperator->pTaskInfo = pTaskInfo; initResultSizeInfo(pOperator, 4096); blockDataEnsureCapacity(pInfo->pRes, pOperator->resultInfo.capacity); @@ -1882,7 +1931,7 @@ typedef struct STableMergeScanInfo { SNode* pFilterNode; // filter info, which is push down by optimizer SqlFunctionCtx* pCtx; // which belongs to the direct upstream operator operator query context SResultRowInfo* pResultRowInfo; - int32_t* rowCellInfoOffset; + int32_t* rowEntryInfoOffset; SExprInfo* pExpr; SSDataBlock* pResBlock; SArray* pColMatchInfo; @@ -1891,7 +1940,7 @@ typedef struct STableMergeScanInfo { SExprInfo* pPseudoExpr; int32_t numOfPseudoExpr; SqlFunctionCtx* pPseudoCtx; - // int32_t* rowCellInfoOffset; + // int32_t* rowEntryInfoOffset; SQueryTableDataCond cond; int32_t scanFlag; // table scan flag to denote if it is a repeat/reverse/main scan @@ -1907,8 +1956,7 @@ typedef struct STableMergeScanInfo { int32_t createMultipleDataReaders(STableScanPhysiNode* pTableScanNode, SReadHandle* pHandle, STableListInfo* pTableListInfo, SArray* arrayReader, uint64_t queryId, uint64_t taskId, SNode* pTagCond) { - int32_t code = - getTableList(pHandle->meta, pTableScanNode->scan.tableType, pTableScanNode->scan.uid, pTableListInfo, pTagCond); + int32_t code = getTableList(pHandle->meta, &pTableScanNode->scan, pTableListInfo, pTagCond); if (code != TSDB_CODE_SUCCESS) { goto _error; } @@ -1935,7 +1983,7 @@ int32_t createMultipleDataReaders(STableScanPhysiNode* pTableScanNode, SReadHand taosArrayDestroy(subListInfo->pTableList); taosMemoryFree(subListInfo); } - clearupQueryTableDataCond(&cond); + cleanupQueryTableDataCond(&cond); return 0; @@ -1945,7 +1993,7 @@ _error: static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeScanInfo* pTableScanInfo, int32_t readerIdx, SSDataBlock* pBlock, uint32_t* status) { - SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; STableMergeScanInfo* pInfo = pOperator->info; SFileBlockLoadRecorder* pCost = &pTableScanInfo->readRecorder; @@ -2031,7 +2079,7 @@ static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeSc return terrno; } - relocateColumnData(pBlock, pTableScanInfo->pColMatchInfo, pCols); + relocateColumnData(pBlock, pTableScanInfo->pColMatchInfo, pCols, true); // currently only the tbname pseudo column if (pTableScanInfo->numOfPseudoExpr > 0) { @@ -2132,9 +2180,8 @@ int32_t doOpenTableMergeScanOperator(SOperatorInfo* pOperator) { int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize; - pInfo->pSortHandle = - tsortCreateSortHandle(pInfo->pSortInfo, pInfo->pColMatchInfo, SORT_MULTISOURCE_MERGE, pInfo->bufPageSize, - numOfBufPage, pInfo->pSortInputBlock, pTaskInfo->id.str); + pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_MULTISOURCE_MERGE, pInfo->bufPageSize, numOfBufPage, + pInfo->pSortInputBlock, pTaskInfo->id.str); tsortSetFetchRawDataFp(pInfo->pSortHandle, getTableDataBlock, NULL, NULL); @@ -2170,25 +2217,17 @@ SSDataBlock* getSortedTableMergeScanBlockData(SSortHandle* pHandle, int32_t capa blockDataEnsureCapacity(p, capacity); while (1) { - STupleHandle* pTupleHandle = NULL; - if (pInfo->prefetchedTuple == NULL) { - pTupleHandle = tsortNextTuple(pHandle); - } else { - pTupleHandle = pInfo->prefetchedTuple; - } - + STupleHandle* pTupleHandle = tsortNextTuple(pHandle); if (pTupleHandle == NULL) { break; } appendOneRowToDataBlock(p, pTupleHandle); - if (p->info.rows >= capacity) { break; } } - qDebug("%s get sorted row blocks, rows:%d", GET_TASKID(pTaskInfo), p->info.rows); return (p->info.rows > 0) ? p : NULL; } @@ -2206,8 +2245,7 @@ SSDataBlock* doTableMergeScan(SOperatorInfo* pOperator) { longjmp(pTaskInfo->env, code); } - SSDataBlock* pBlock = - getSortedTableMergeScanBlockData(pInfo->pSortHandle, pOperator->resultInfo.capacity, pOperator); + SSDataBlock* pBlock = getSortedTableMergeScanBlockData(pInfo->pSortHandle, pOperator->resultInfo.capacity, pOperator); if (pBlock != NULL) { pOperator->resultInfo.totalRows += pBlock->info.rows; @@ -2219,7 +2257,7 @@ SSDataBlock* doTableMergeScan(SOperatorInfo* pOperator) { void destroyTableMergeScanOperatorInfo(void* param, int32_t numOfOutput) { STableMergeScanInfo* pTableScanInfo = (STableMergeScanInfo*)param; - clearupQueryTableDataCond(&pTableScanInfo->cond); + cleanupQueryTableDataCond(&pTableScanInfo->cond); for (int32_t i = 0; i < taosArrayGetSize(pTableScanInfo->dataReaders); ++i) { tsdbReaderT* reader = taosArrayGetP(pTableScanInfo->dataReaders, i); @@ -2240,20 +2278,20 @@ void destroyTableMergeScanOperatorInfo(void* param, int32_t numOfOutput) { typedef struct STableMergeScanExecInfo { SFileBlockLoadRecorder blockRecorder; - SSortExecInfo sortExecInfo; + SSortExecInfo sortExecInfo; } STableMergeScanExecInfo; int32_t getTableMergeScanExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len) { ASSERT(pOptr != NULL); // TODO: merge these two info into one struct STableMergeScanExecInfo* execInfo = taosMemoryCalloc(1, sizeof(STableMergeScanExecInfo)); - STableMergeScanInfo* pInfo = pOptr->info; + STableMergeScanInfo* pInfo = pOptr->info; execInfo->blockRecorder = pInfo->readRecorder; execInfo->sortExecInfo = tsortGetSortExecInfo(pInfo->pSortHandle); *pOptrExplain = execInfo; *len = sizeof(STableMergeScanExecInfo); - + return TSDB_CODE_SUCCESS; } @@ -2261,7 +2299,6 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN SReadHandle* readHandle, SExecTaskInfo* pTaskInfo) { STableMergeScanInfo* pInfo = taosMemoryCalloc(1, sizeof(STableMergeScanInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); - if (pInfo == NULL || pOperator == NULL) { goto _error; } @@ -2269,8 +2306,7 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN SDataBlockDescNode* pDescNode = pTableScanNode->scan.node.pOutputDataBlockDesc; int32_t numOfCols = 0; - SArray* pColList = - extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, pTaskInfo, COL_MATCH_FROM_COL_ID); + SArray* pColList = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID); int32_t code = initQueryTableDataCond(&pInfo->cond, pTableScanNode); if (code != TSDB_CODE_SUCCESS) { @@ -2279,7 +2315,7 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN if (pTableScanNode->scan.pScanPseudoCols != NULL) { pInfo->pPseudoExpr = createExprInfo(pTableScanNode->scan.pScanPseudoCols, NULL, &pInfo->numOfPseudoExpr); - pInfo->pPseudoCtx = createSqlFunctionCtx(pInfo->pPseudoExpr, pInfo->numOfPseudoExpr, &pInfo->rowCellInfoOffset); + pInfo->pPseudoCtx = createSqlFunctionCtx(pInfo->pPseudoExpr, pInfo->numOfPseudoExpr, &pInfo->rowEntryInfoOffset); } pInfo->scanInfo = (SScanInfo){.numOfAsc = pTableScanNode->scanSeq[0], .numOfDesc = pTableScanNode->scanSeq[1]}; @@ -2288,7 +2324,6 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN pInfo->interval = extractIntervalInfo(pTableScanNode); pInfo->sample.sampleRatio = pTableScanNode->ratio; pInfo->sample.seed = taosGetTimestampSec(); - pInfo->dataBlockLoadFlag = pTableScanNode->dataRequired; pInfo->pFilterNode = pTableScanNode->scan.node.pConditions; pInfo->dataReaders = dataReaders; @@ -2307,21 +2342,26 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN taosArrayPush(pInfo->sortSourceParams, param); taosMemoryFree(param); } + pInfo->pSortInfo = generateSortByTsInfo(pInfo->cond.order); pInfo->pSortInputBlock = createOneDataBlock(pInfo->pResBlock, false); + int32_t rowSize = pInfo->pResBlock->info.rowSize; - int32_t blockMetaSize = (int32_t)blockDataGetSerialMetaSize(pInfo->pResBlock->info.numOfCols); - pInfo->bufPageSize = (rowSize * 2 + blockMetaSize) < 1024 ? 1024 : (rowSize * 2 + blockMetaSize); - pInfo->sortBufSize = pInfo->bufPageSize * 16; + pInfo->bufPageSize = getProperSortPageSize(rowSize); + + // todo the total available buffer should be determined by total capacity of buffer of this task. + // the additional one is reserved for merge result + pInfo->sortBufSize = pInfo->bufPageSize * (taosArrayGetSize(dataReaders) + 1); pInfo->hasGroupId = false; pInfo->prefetchedTuple = NULL; pOperator->name = "TableMergeScanOperator"; + // TODO : change it pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->pTaskInfo = pTaskInfo; initResultSizeInfo(pOperator, 1024); diff --git a/source/libs/executor/src/sortoperator.c b/source/libs/executor/src/sortoperator.c index 81899b68cd51a4c11a5f12728145b07090e56839..9d13276e6d68d2aaf380d0ac3111c2214c6f0e75 100644 --- a/source/libs/executor/src/sortoperator.c +++ b/source/libs/executor/src/sortoperator.c @@ -22,41 +22,52 @@ static int32_t getExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain static void destroyOrderOperatorInfo(void* param, int32_t numOfOutput); -SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSDataBlock* pResBlock, SArray* pSortInfo, - SExprInfo* pExprInfo, int32_t numOfCols, SArray* pColMatchColInfo, - SExecTaskInfo* pTaskInfo) { +SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortPhyNode, SExecTaskInfo* pTaskInfo) { SSortOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SSortOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); - int32_t rowSize = pResBlock->info.rowSize; - - if (pInfo == NULL || pOperator == NULL || rowSize > 100 * 1024 * 1024) { + if (pInfo == NULL || pOperator == NULL/* || rowSize > 100 * 1024 * 1024*/) { goto _error; } - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; - pInfo->binfo.pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pInfo->binfo.rowCellInfoOffset); + SDataBlockDescNode* pDescNode = pSortPhyNode->node.pOutputDataBlockDesc; + + int32_t numOfCols = 0; + SSDataBlock* pResBlock = createResDataBlock(pDescNode); + SExprInfo* pExprInfo = createExprInfo(pSortPhyNode->pExprs, NULL, &numOfCols); + + int32_t numOfOutputCols = 0; + SArray* pColMatchColInfo = + extractColMatchInfo(pSortPhyNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID); + + pOperator->exprSupp.pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pOperator->exprSupp.rowEntryInfoOffset); pInfo->binfo.pRes = pResBlock; initResultSizeInfo(pOperator, 1024); - pInfo->pSortInfo = pSortInfo; + pInfo->pSortInfo = createSortInfo(pSortPhyNode->pSortKeys);; pInfo->pColMatchInfo = pColMatchColInfo; pOperator->name = "SortOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; pOperator->info = pInfo; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; + pOperator->pTaskInfo = pTaskInfo; // lazy evaluation for the following parameter since the input datablock is not known till now. - // pInfo->bufPageSize = rowSize < 1024 ? 1024 * 2 : rowSize * 2; // there are headers, so pageSize = rowSize + - // header pInfo->sortBufSize = pInfo->bufPageSize * 16; // TODO dynamic set the available sort buffer + // pInfo->bufPageSize = rowSize < 1024 ? 1024 * 2 : rowSize * 2; + // there are headers, so pageSize = rowSize + header pInfo->sortBufSize = pInfo->bufPageSize * 16; + // TODO dynamic set the available sort buffer - pOperator->pTaskInfo = pTaskInfo; pOperator->fpSet = createOperatorFpSet(doOpenSortOperator, doSort, NULL, NULL, destroyOrderOperatorInfo, NULL, NULL, getExplainExecInfo); int32_t code = appendDownstream(pOperator, &downstream, 1); + if (code != TSDB_CODE_SUCCESS) { + goto _error; + } + return pOperator; _error: @@ -134,9 +145,9 @@ SSDataBlock* loadNextDataBlock(void* param) { void applyScalarFunction(SSDataBlock* pBlock, void* param) { SOperatorInfo* pOperator = param; SSortOperatorInfo* pSort = pOperator->info; - if (pOperator->pExpr != NULL) { + if (pOperator->exprSupp.pExprInfo != NULL) { int32_t code = - projectApplyFunctions(pOperator->pExpr, pBlock, pBlock, pSort->binfo.pCtx, pOperator->numOfExprs, NULL); + projectApplyFunctions(pOperator->exprSupp.pExprInfo, pBlock, pBlock, pOperator->exprSupp.pCtx, pOperator->exprSupp.numOfExprs, NULL); if (code != TSDB_CODE_SUCCESS) { longjmp(pOperator->pTaskInfo->env, code); } @@ -154,7 +165,7 @@ int32_t doOpenSortOperator(SOperatorInfo* pOperator) { pInfo->startTs = taosGetTimestampUs(); // pInfo->binfo.pRes is not equalled to the input datablock. - pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, pInfo->pColMatchInfo, SORT_SINGLESOURCE_SORT, -1, -1, + pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_SINGLESOURCE_SORT, -1, -1, NULL, pTaskInfo->id.str); tsortSetFetchRawDataFp(pInfo->pSortHandle, loadNextDataBlock, applyScalarFunction, pOperator); @@ -248,7 +259,7 @@ int32_t doOpenMultiwaySortMergeOperator(SOperatorInfo* pOperator) { int32_t numOfBufPage = pInfo->sortBufSize / pInfo->bufPageSize; - pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, pInfo->pColMatchInfo, SORT_MULTISOURCE_MERGE, + pInfo->pSortHandle = tsortCreateSortHandle(pInfo->pSortInfo, SORT_MULTISOURCE_MERGE, pInfo->bufPageSize, numOfBufPage, pInfo->pInputBlock, pTaskInfo->id.str); tsortSetFetchRawDataFp(pInfo->pSortHandle, loadNextDataBlock, NULL, NULL); @@ -392,10 +403,8 @@ SOperatorInfo* createMultiwaySortMergeOperatorInfo(SOperatorInfo** downStreams, pInfo->bufPageSize = getProperSortPageSize(rowSize); - uint32_t numOfSources = taosArrayGetSize(pSortInfo); - numOfSources = TMAX(4, numOfSources); - - pInfo->sortBufSize = numOfSources * pInfo->bufPageSize; + // one additional is reserved for merged result. + pInfo->sortBufSize = pInfo->bufPageSize * (numStreams + 1); pOperator->fpSet = createOperatorFpSet(doOpenMultiwaySortMergeOperator, doMultiwaySortMerge, NULL, NULL, diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index d7ae82352235295c327f4437192732fb8f3c39f1..c556e94c74b8f3795cca197fd26544579aceeaad 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -24,6 +24,8 @@ typedef enum SResultTsInterpType { RESULT_ROW_END_INTERP = 2, } SResultTsInterpType; +#define IS_FINAL_OP(op) ((op)->isFinal) + static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator); static int64_t* extractTsCol(SSDataBlock* pBlock, const SIntervalAggOperatorInfo* pInfo); @@ -103,7 +105,7 @@ STimeWindow getActiveTimeWindow(SDiskbasedBuf* pBuf, SResultRowInfo* pResultRowI static int32_t setTimeWindowOutputBuf(SResultRowInfo* pResultRowInfo, STimeWindow* win, bool masterscan, SResultRow** pResult, int64_t tableGroupId, SqlFunctionCtx* pCtx, - int32_t numOfOutput, int32_t* rowCellInfoOffset, SAggSupporter* pAggSup, + int32_t numOfOutput, int32_t* rowEntryInfoOffset, SAggSupporter* pAggSup, SExecTaskInfo* pTaskInfo) { assert(win->skey <= win->ekey); SResultRow* pResultRow = doSetResultOutBufByKey(pAggSup->pResultBuf, pResultRowInfo, (char*)&win->skey, TSDB_KEYSIZE, @@ -118,7 +120,7 @@ static int32_t setTimeWindowOutputBuf(SResultRowInfo* pResultRowInfo, STimeWindo pResultRow->win = (*win); *pResult = pResultRow; - setResultRowInitCtx(pResultRow, pCtx, numOfOutput, rowCellInfoOffset); + setResultRowInitCtx(pResultRow, pCtx, numOfOutput, rowEntryInfoOffset); return TSDB_CODE_SUCCESS; } @@ -322,12 +324,12 @@ static void getNextTimeWindow(SInterval* pInterval, int32_t precision, int32_t o tw->ekey -= 1; } -void doTimeWindowInterpolation(SIntervalAggOperatorInfo* pInfo, int32_t numOfExprs, SArray* pDataBlock, TSKEY prevTs, - int32_t prevRowIndex, TSKEY curTs, int32_t curRowIndex, TSKEY windowKey, int32_t type) { - SqlFunctionCtx* pCtx = pInfo->binfo.pCtx; +void doTimeWindowInterpolation(SArray* pPrevValues, SArray* pDataBlock, TSKEY prevTs, int32_t prevRowIndex, + TSKEY curTs, int32_t curRowIndex, TSKEY windowKey, int32_t type, SExprSupp* pSup) { + SqlFunctionCtx* pCtx = pSup->pCtx; int32_t index = 1; - for (int32_t k = 0; k < numOfExprs; ++k) { + for (int32_t k = 0; k < pSup->numOfExprs; ++k) { if (!fmIsIntervalInterpoFunc(pCtx[k].functionId)) { pCtx[k].start.key = INT64_MIN; continue; @@ -340,7 +342,7 @@ void doTimeWindowInterpolation(SIntervalAggOperatorInfo* pInfo, int32_t numOfExp double v1 = 0, v2 = 0, v = 0; if (prevRowIndex == -1) { - SGroupKeys* p = taosArrayGet(pInfo->pPrevValues, index); + SGroupKeys* p = taosArrayGet(pPrevValues, index); GET_TYPED_DATA(v1, double, pColInfo->info.type, p->pData); } else { GET_TYPED_DATA(v1, double, pColInfo->info.type, colDataGetData(pColInfo, prevRowIndex)); @@ -403,8 +405,8 @@ static void setNotInterpoWindowKey(SqlFunctionCtx* pCtx, int32_t numOfOutput, in } } -static bool setTimeWindowInterpolationStartTs(SIntervalAggOperatorInfo* pInfo, SqlFunctionCtx* pCtx, int32_t numOfExprs, - int32_t pos, SSDataBlock* pBlock, const TSKEY* tsCols, STimeWindow* win) { +static bool setTimeWindowInterpolationStartTs(SIntervalAggOperatorInfo* pInfo, int32_t pos, SSDataBlock* pBlock, const TSKEY* tsCols, + STimeWindow* win, SExprSupp* pSup) { bool ascQuery = (pInfo->order == TSDB_ORDER_ASC); TSKEY curTs = tsCols[pos]; @@ -416,23 +418,23 @@ static bool setTimeWindowInterpolationStartTs(SIntervalAggOperatorInfo* pInfo, S // start exactly from this point, no need to do interpolation TSKEY key = ascQuery ? win->skey : win->ekey; if (key == curTs) { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_START_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_START_INTERP); return true; } // it is the first time window, no need to do interpolation if (pTsKey->isNull && pos == 0) { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_START_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_START_INTERP); } else { TSKEY prevTs = ((pos == 0) ? lastTs : tsCols[pos - 1]); - doTimeWindowInterpolation(pInfo, numOfExprs, pBlock->pDataBlock, prevTs, pos - 1, curTs, pos, key, - RESULT_ROW_START_INTERP); + doTimeWindowInterpolation(pInfo->pPrevValues, pBlock->pDataBlock, prevTs, pos - 1, curTs, pos, key, + RESULT_ROW_START_INTERP, pSup); } return true; } -static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, SqlFunctionCtx* pCtx, int32_t numOfExprs, +static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, SExprSupp* pSup, int32_t endRowIndex, SArray* pDataBlock, const TSKEY* tsCols, TSKEY blockEkey, STimeWindow* win) { int32_t order = pInfo->order; @@ -442,13 +444,13 @@ static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, Sql // not ended in current data block, do not invoke interpolation if ((key > blockEkey && (order == TSDB_ORDER_ASC)) || (key < blockEkey && (order == TSDB_ORDER_DESC))) { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_END_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP); return false; } // there is actual end point of current time window, no interpolation needs if (key == actualEndKey) { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_END_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP); return true; } @@ -456,8 +458,8 @@ static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, Sql assert(nextRowIndex >= 0); TSKEY nextKey = tsCols[nextRowIndex]; - doTimeWindowInterpolation(pInfo, numOfExprs, pDataBlock, actualEndKey, endRowIndex, nextKey, nextRowIndex, key, - RESULT_ROW_END_INTERP); + doTimeWindowInterpolation(pInfo->pPrevValues, pDataBlock, actualEndKey, endRowIndex, nextKey, nextRowIndex, key, + RESULT_ROW_END_INTERP, pSup); return true; } @@ -546,9 +548,8 @@ static void setResultRowInterpo(SResultRow* pResult, SResultTsInterpType type) { } } -static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataBlock* pBlock, int32_t numOfExprs, - SqlFunctionCtx* pCtx, SResultRow* pResult, STimeWindow* win, int32_t startPos, - int32_t forwardRows) { +static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataBlock* pBlock, SResultRow* pResult, STimeWindow* win, int32_t startPos, + int32_t forwardRows, SExprSupp* pSup) { if (!pInfo->timeWindowInterpo) { return; } @@ -564,12 +565,12 @@ static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataB TSKEY* tsCols = (TSKEY*)(pColInfo->pData); bool done = isResultRowInterpolated(pResult, RESULT_ROW_START_INTERP); if (!done) { // it is not interpolated, now start to generated the interpolated value - bool interp = setTimeWindowInterpolationStartTs(pInfo, pCtx, numOfExprs, startPos, pBlock, tsCols, win); + bool interp = setTimeWindowInterpolationStartTs(pInfo, startPos, pBlock, tsCols, win, pSup); if (interp) { setResultRowInterpo(pResult, RESULT_ROW_START_INTERP); } } else { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_START_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_START_INTERP); } // point interpolation does not require the end key time window interpolation. @@ -583,13 +584,12 @@ static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataB int32_t endRowIndex = startPos + forwardRows - 1; TSKEY endKey = (pInfo->order == TSDB_ORDER_ASC) ? pBlock->info.window.ekey : pBlock->info.window.skey; - bool interp = - setTimeWindowInterpolationEndTs(pInfo, pCtx, numOfExprs, endRowIndex, pBlock->pDataBlock, tsCols, endKey, win); + bool interp = setTimeWindowInterpolationEndTs(pInfo, pSup, endRowIndex, pBlock->pDataBlock, tsCols, endKey, win); if (interp) { setResultRowInterpo(pResult, RESULT_ROW_END_INTERP); } } else { - setNotInterpoWindowKey(pCtx, numOfExprs, RESULT_ROW_END_INTERP); + setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP); } } @@ -628,9 +628,10 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num SExecTaskInfo* pTaskInfo = pOperatorInfo->pTaskInfo; SIntervalAggOperatorInfo* pInfo = (SIntervalAggOperatorInfo*)pOperatorInfo->info; + SExprSupp* pSup = &pOperatorInfo->exprSupp; int32_t startPos = 0; - int32_t numOfOutput = pOperatorInfo->numOfExprs; + int32_t numOfOutput = pSup->numOfExprs; uint64_t groupId = pBlock->info.groupId; SResultRow* pResult = NULL; @@ -655,8 +656,8 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num STimeWindow w = pr->win; int32_t ret = - setTimeWindowOutputBuf(pResultRowInfo, &w, (scanFlag == MAIN_SCAN), &pResult, groupId, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(pResultRowInfo, &w, (scanFlag == MAIN_SCAN), &pResult, groupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } @@ -665,13 +666,13 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num SGroupKeys* pTsKey = taosArrayGet(pInfo->pPrevValues, 0); int64_t prevTs = *(int64_t*)pTsKey->pData; - doTimeWindowInterpolation(pInfo, numOfOutput, pBlock->pDataBlock, prevTs, -1, tsCols[startPos], startPos, w.ekey, - RESULT_ROW_END_INTERP); + doTimeWindowInterpolation(pInfo->pPrevValues, pBlock->pDataBlock, prevTs, -1, tsCols[startPos], startPos, w.ekey, + RESULT_ROW_END_INTERP, pSup); setResultRowInterpo(pResult, RESULT_ROW_END_INTERP); - setNotInterpoWindowKey(pInfo->binfo.pCtx, numOfExprs, RESULT_ROW_START_INTERP); + setNotInterpoWindowKey(pSup->pCtx, numOfExprs, RESULT_ROW_START_INTERP); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &w, &pInfo->twAggSup.timeWindowData, startPos, 0, tsCols, + doApplyFunctions(pTaskInfo, pSup->pCtx, &w, &pInfo->twAggSup.timeWindowData, startPos, 0, tsCols, pBlock->info.rows, numOfExprs, pInfo->order); if (isResultRowInterpolated(pResult, RESULT_ROW_END_INTERP)) { @@ -683,6 +684,13 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num } } +void printDataBlock(SSDataBlock* pBlock, const char* flag) { + SArray* blocks = taosArrayInit(1, sizeof(SSDataBlock)); + taosArrayPush(blocks, pBlock); + blockDebugShowData(blocks, flag); + taosArrayDestroy(blocks); +} + typedef int64_t (*__get_value_fn_t)(void* data, int32_t index); int32_t binarySearch(void* keyList, int num, TSKEY key, int order, __get_value_fn_t getValuefn) { @@ -798,9 +806,10 @@ static void hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul SIntervalAggOperatorInfo* pInfo = (SIntervalAggOperatorInfo*)pOperatorInfo->info; SExecTaskInfo* pTaskInfo = pOperatorInfo->pTaskInfo; + SExprSupp* pSup = &pOperatorInfo->exprSupp; int32_t startPos = 0; - int32_t numOfOutput = pOperatorInfo->numOfExprs; + int32_t numOfOutput = pSup->numOfExprs; int64_t* tsCols = extractTsCol(pBlock, pInfo); uint64_t tableGroupId = pBlock->info.groupId; bool ascScan = (pInfo->order == TSDB_ORDER_ASC); @@ -811,8 +820,8 @@ static void hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul pInfo->interval.precision, &pInfo->win); int32_t ret = - setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } @@ -835,18 +844,18 @@ static void hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul // restore current time window ret = - setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } // window start key interpolation - doWindowBorderInterpolation(pInfo, pBlock, numOfOutput, pInfo->binfo.pCtx, pResult, &win, startPos, forwardRows); + doWindowBorderInterpolation(pInfo, pBlock, pResult, &win, startPos, forwardRows, pSup); } updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &win, true); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &win, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, tsCols, + doApplyFunctions(pTaskInfo, pSup->pCtx, &win, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, tsCols, pBlock->info.rows, numOfOutput, pInfo->order); doCloseWindow(pResultRowInfo, pInfo, pResult); @@ -861,7 +870,7 @@ static void hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul // null data, failed to allocate more memory buffer int32_t code = setTimeWindowOutputBuf(pResultRowInfo, &nextWin, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, - pInfo->binfo.pCtx, numOfOutput, pInfo->binfo.rowCellInfoOffset, + pSup->pCtx, numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (code != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); @@ -878,11 +887,10 @@ static void hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul getNumOfRowsInTimeWindow(&pBlock->info, tsCols, startPos, ekey, binarySearchForKey, NULL, pInfo->order); // window start(end) key interpolation - doWindowBorderInterpolation(pInfo, pBlock, numOfOutput, pInfo->binfo.pCtx, pResult, &nextWin, startPos, - forwardRows); + doWindowBorderInterpolation(pInfo, pBlock, pResult, &nextWin, startPos, forwardRows, pSup); updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &nextWin, true); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &nextWin, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, + doApplyFunctions(pTaskInfo, pSup->pCtx, &nextWin, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, tsCols, pBlock->info.rows, numOfOutput, pInfo->order); doCloseWindow(pResultRowInfo, pInfo, pResult); } @@ -937,6 +945,7 @@ static int32_t doOpenIntervalAgg(SOperatorInfo* pOperator) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SIntervalAggOperatorInfo* pInfo = pOperator->info; + SExprSupp* pSup = &pOperator->exprSupp; int32_t scanFlag = MAIN_SCAN; @@ -952,7 +961,7 @@ static int32_t doOpenIntervalAgg(SOperatorInfo* pOperator) { getTableScanInfo(pOperator, &pInfo->order, &scanFlag); // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pInfo->binfo.pCtx, pBlock, pInfo->order, scanFlag, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, pInfo->order, scanFlag, true); hashIntervalAgg(pOperator, &pInfo->binfo.resultRowInfo, pBlock, scanFlag, NULL); #if 0 // test for encode/decode result info @@ -993,12 +1002,13 @@ static bool compareVal(const char* v, const SStateKeys* pKey) { static void doStateWindowAggImpl(SOperatorInfo* pOperator, SStateWindowOperatorInfo* pInfo, SSDataBlock* pBlock) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SExprSupp* pSup = &pOperator->exprSupp; SColumnInfoData* pStateColInfoData = taosArrayGet(pBlock->pDataBlock, pInfo->stateCol.slotId); int64_t gid = pBlock->info.groupId; bool masterScan = true; - int32_t numOfOutput = pOperator->numOfExprs; + int32_t numOfOutput = pOperator->exprSupp.numOfExprs; int16_t bytes = pStateColInfoData->info.bytes; SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, pInfo->tsSlotId); @@ -1041,14 +1051,14 @@ static void doStateWindowAggImpl(SOperatorInfo* pOperator, SStateWindowOperatorI pRowSup->win.ekey = pRowSup->win.skey; int32_t ret = - setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &window, masterScan, &pResult, gid, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &window, masterScan, &pResult, gid, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { // null data, too many state code longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &window, false); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &window, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, + doApplyFunctions(pTaskInfo, pSup->pCtx, &window, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, pRowSup->numOfRows, NULL, pBlock->info.rows, numOfOutput, TSDB_ORDER_ASC); // here we start a new session window @@ -1067,14 +1077,14 @@ static void doStateWindowAggImpl(SOperatorInfo* pOperator, SStateWindowOperatorI SResultRow* pResult = NULL; pRowSup->win.ekey = tsList[pBlock->info.rows - 1]; int32_t ret = - setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &pRowSup->win, masterScan, &pResult, gid, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &pRowSup->win, masterScan, &pResult, gid, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { // null data, too many state code longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pRowSup->win, false); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &pRowSup->win, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, + doApplyFunctions(pTaskInfo, pSup->pCtx, &pRowSup->win, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, pRowSup->numOfRows, NULL, pBlock->info.rows, numOfOutput, TSDB_ORDER_ASC); } @@ -1086,11 +1096,13 @@ static SSDataBlock* doStateWindowAgg(SOperatorInfo* pOperator) { SStateWindowOperatorInfo* pInfo = pOperator->info; SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SExprSupp* pSup = &pOperator->exprSupp; + SOptrBasicInfo* pBInfo = &pInfo->binfo; if (pOperator->status == OP_RES_TO_RETURN) { doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); return NULL; } @@ -1108,7 +1120,7 @@ static SSDataBlock* doStateWindowAgg(SOperatorInfo* pOperator) { break; } - setInputDataBlock(pOperator, pBInfo->pCtx, pBlock, order, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, MAIN_SCAN, true); blockDataUpdateTsWindow(pBlock, pInfo->tsSlotId); doStateWindowAggImpl(pOperator, pInfo, pBlock); @@ -1122,7 +1134,7 @@ static SSDataBlock* doStateWindowAgg(SOperatorInfo* pOperator) { initGroupedResultInfo(&pInfo->groupResInfo, pInfo->aggSup.pResultRowHashTable, TSDB_ORDER_ASC); blockDataEnsureCapacity(pBInfo->pRes, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } @@ -1153,7 +1165,7 @@ static SSDataBlock* doBuildIntervalResult(SOperatorInfo* pOperator) { blockDataEnsureCapacity(pBlock, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pBlock->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBlock->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } @@ -1166,7 +1178,7 @@ static SSDataBlock* doBuildIntervalResult(SOperatorInfo* pOperator) { // todo merged with the build group result. static void finalizeUpdatedResult(int32_t numOfOutput, SDiskbasedBuf* pBuf, SArray* pUpdateList, - int32_t* rowCellInfoOffset) { + int32_t* rowEntryInfoOffset) { size_t num = taosArrayGetSize(pUpdateList); for (int32_t i = 0; i < num; ++i) { @@ -1176,7 +1188,7 @@ static void finalizeUpdatedResult(int32_t numOfOutput, SDiskbasedBuf* pBuf, SArr SResultRow* pRow = (SResultRow*)((char*)bufPage + pPos->pos.offset); for (int32_t j = 0; j < numOfOutput; ++j) { - SResultRowEntryInfo* pEntry = getResultCell(pRow, j, rowCellInfoOffset); + SResultRowEntryInfo* pEntry = getResultEntryInfo(pRow, j, rowEntryInfoOffset); if (pRow->numOfRows < pEntry->numOfRes) { pRow->numOfRows = pEntry->numOfRes; } @@ -1195,11 +1207,11 @@ static void setInverFunction(SqlFunctionCtx* pCtx, int32_t num, EStreamType type } } -void doClearWindowImpl(SResultRowPosition* p1, SDiskbasedBuf* pResultBuf, SOptrBasicInfo* pBinfo, int32_t numOfOutput) { +void doClearWindowImpl(SResultRowPosition* p1, SDiskbasedBuf* pResultBuf, SExprSupp *pSup, int32_t numOfOutput) { SResultRow* pResult = getResultRowByPos(pResultBuf, p1); - SqlFunctionCtx* pCtx = pBinfo->pCtx; + SqlFunctionCtx* pCtx = pSup->pCtx; for (int32_t i = 0; i < numOfOutput; ++i) { - pCtx[i].resultInfo = getResultCell(pResult, i, pBinfo->rowCellInfoOffset); + pCtx[i].resultInfo = getResultEntryInfo(pResult, i, pSup->rowEntryInfoOffset); struct SResultRowEntryInfo* pResInfo = pCtx[i].resultInfo; if (fmIsWindowPseudoColumnFunc(pCtx[i].functionId)) { continue; @@ -1211,19 +1223,19 @@ void doClearWindowImpl(SResultRowPosition* p1, SDiskbasedBuf* pResultBuf, SOptrB } } -void doClearWindow(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, char* pData, int16_t bytes, uint64_t groupId, +void doClearWindow(SAggSupporter* pAggSup, SExprSupp *pSup, char* pData, int16_t bytes, uint64_t groupId, int32_t numOfOutput) { - SET_RES_WINDOW_KEY(pSup->keyBuf, pData, bytes, groupId); + SET_RES_WINDOW_KEY(pAggSup->keyBuf, pData, bytes, groupId); SResultRowPosition* p1 = - (SResultRowPosition*)taosHashGet(pSup->pResultRowHashTable, pSup->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes)); + (SResultRowPosition*)taosHashGet(pAggSup->pResultRowHashTable, pAggSup->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes)); if (!p1) { // window has been closed return; } - doClearWindowImpl(p1, pSup->pResultBuf, pBinfo, numOfOutput); + doClearWindowImpl(p1, pAggSup->pResultBuf, pSup, numOfOutput); } -static void doClearWindows(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, SInterval* pInterval, int32_t tsIndex, +static void doClearWindows(SAggSupporter* pAggSup, SExprSupp* pSup1, SInterval* pInterval, int32_t tsIndex, int32_t numOfOutput, SSDataBlock* pBlock, SArray* pUpWins) { SColumnInfoData* pColDataInfo = taosArrayGet(pBlock->pDataBlock, tsIndex); TSKEY* tsCols = (TSKEY*)pColDataInfo->pData; @@ -1233,7 +1245,7 @@ static void doClearWindows(SAggSupporter* pSup, SOptrBasicInfo* pBinfo, SInterva dumyInfo.cur.pageId = -1; STimeWindow win = getActiveTimeWindow(NULL, &dumyInfo, tsCols[i], pInterval, pInterval->precision, NULL); step = getNumOfRowsInTimeWindow(&pBlock->info, tsCols, i, win.ekey, binarySearchForKey, NULL, TSDB_ORDER_ASC); - doClearWindow(pSup, pBinfo, (char*)&win.skey, sizeof(TKEY), pBlock->info.groupId, numOfOutput); + doClearWindow(pAggSup, pSup1, (char*)&win.skey, sizeof(TKEY), pBlock->info.groupId, numOfOutput); if (pUpWins) { taosArrayPush(pUpWins, &win); } @@ -1294,6 +1306,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; pInfo->order = TSDB_ORDER_ASC; + SExprSupp* pSup = &pOperator->exprSupp; if (pOperator->status == OP_EXEC_DONE) { return NULL; @@ -1301,7 +1314,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { if (pOperator->status == OP_RES_TO_RETURN) { doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pInfo->binfo.pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pInfo->binfo.pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { pOperator->status = OP_EXEC_DONE; } return pInfo->binfo.pRes->info.rows == 0 ? NULL : pInfo->binfo.pRes; @@ -1317,7 +1330,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { } if (pBlock->info.type == STREAM_REPROCESS) { - doClearWindows(&pInfo->aggSup, &pInfo->binfo, &pInfo->interval, 0, pOperator->numOfExprs, pBlock, NULL); + doClearWindows(&pInfo->aggSup, &pOperator->exprSupp, &pInfo->interval, 0, pOperator->exprSupp.numOfExprs, pBlock, NULL); qDebug("%s clear existed time window results for updates checked", GET_TASKID(pTaskInfo)); continue; } else if (pBlock->info.type == STREAM_GET_ALL) { @@ -1328,9 +1341,9 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { // The timewindow that overlaps the timestamps of the input pBlock need to be recalculated and return to the // caller. Note that all the time window are not close till now. // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pInfo->binfo.pCtx, pBlock, pInfo->order, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, pInfo->order, MAIN_SCAN, true); if (pInfo->invertible) { - setInverFunction(pInfo->binfo.pCtx, pOperator->numOfExprs, pBlock->info.type); + setInverFunction(pSup->pCtx, pOperator->exprSupp.numOfExprs, pBlock->info.type); } pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, pBlock->info.window.ekey); @@ -1338,7 +1351,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { } closeIntervalWindow(pInfo->aggSup.pResultRowHashTable, &pInfo->twAggSup, &pInfo->interval, pUpdated); - finalizeUpdatedResult(pOperator->numOfExprs, pInfo->aggSup.pResultBuf, pUpdated, pInfo->binfo.rowCellInfoOffset); + finalizeUpdatedResult(pOperator->exprSupp.numOfExprs, pInfo->aggSup.pResultBuf, pUpdated, pSup->rowEntryInfoOffset); initMultiResInfoFromArrayList(&pInfo->groupResInfo, pUpdated); blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); @@ -1350,19 +1363,19 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { static void destroyStateWindowOperatorInfo(void* param, int32_t numOfOutput) { SStateWindowOperatorInfo* pInfo = (SStateWindowOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); taosMemoryFreeClear(pInfo->stateKey.pData); } void destroyIntervalOperatorInfo(void* param, int32_t numOfOutput) { SIntervalAggOperatorInfo* pInfo = (SIntervalAggOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); cleanupAggSup(&pInfo->aggSup); } void destroyStreamFinalIntervalOperatorInfo(void* param, int32_t numOfOutput) { SStreamFinalIntervalOperatorInfo* pInfo = (SStreamFinalIntervalOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); cleanupAggSup(&pInfo->aggSup); if (pInfo->pChildren) { int32_t size = taosArrayGetSize(pInfo->pChildren); @@ -1452,23 +1465,25 @@ SOperatorInfo* createIntervalOperatorInfo(SOperatorInfo* downstream, SExprInfo* pInfo->primaryTsIndex = primaryTsSlotId; + SExprSupp* pSup = &pOperator->exprSupp; + size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, 4096); - int32_t code = - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); - + int32_t code = initAggInfo(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); + if (isStream) { ASSERT(numOfCols > 0); - increaseTs(pInfo->binfo.pCtx); + increaseTs(pSup->pCtx); } initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pInfo->win); - pInfo->invertible = allInvertible(pInfo->binfo.pCtx, numOfCols); + pInfo->invertible = allInvertible(pSup->pCtx, numOfCols); pInfo->invertible = false; // Todo(liuyao): Dependent TSDB API - pInfo->timeWindowInterpo = timeWindowinterpNeeded(pInfo->binfo.pCtx, numOfCols, pInfo); + pInfo->timeWindowInterpo = timeWindowinterpNeeded(pSup->pCtx, numOfCols, pInfo); if (pInfo->timeWindowInterpo) { pInfo->binfo.resultRowInfo.openWindow = tdListNew(sizeof(SResultRowPosition)); if (pInfo->binfo.resultRowInfo.openWindow == NULL) { @@ -1476,15 +1491,15 @@ SOperatorInfo* createIntervalOperatorInfo(SOperatorInfo* downstream, SExprInfo* } } - initResultRowInfo(&pInfo->binfo.resultRowInfo, (int32_t)1); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pOperator->name = "TimeIntervalAggOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_HASH_INTERVAL; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->fpSet = createOperatorFpSet(doOpenIntervalAgg, doBuildIntervalResult, doStreamIntervalAgg, NULL, @@ -1525,23 +1540,23 @@ SOperatorInfo* createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SExpr size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, numOfRows); - int32_t code = - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); + int32_t code = initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pInfo->win); if (code != TSDB_CODE_SUCCESS) { goto _error; } - initResultRowInfo(&pInfo->binfo.resultRowInfo, (int32_t)1); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pOperator->name = "StreamTimeIntervalAggOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_HASH_INTERVAL; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->fpSet = createOperatorFpSet(doOpenIntervalAgg, doStreamIntervalAgg, doStreamIntervalAgg, NULL, @@ -1565,11 +1580,12 @@ _error: // todo handle multiple tables cases. static void doSessionWindowAggImpl(SOperatorInfo* pOperator, SSessionAggOperatorInfo* pInfo, SSDataBlock* pBlock) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SExprSupp* pSup = &pOperator->exprSupp; SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, pInfo->tsSlotId); bool masterScan = true; - int32_t numOfOutput = pOperator->numOfExprs; + int32_t numOfOutput = pOperator->exprSupp.numOfExprs; int64_t gid = pBlock->info.groupId; int64_t gap = pInfo->gap; @@ -1602,15 +1618,15 @@ static void doSessionWindowAggImpl(SOperatorInfo* pOperator, SSessionAggOperator pRowSup->win.ekey = pRowSup->win.skey; int32_t ret = - setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &window, masterScan, &pResult, gid, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &window, masterScan, &pResult, gid, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { // null data, too many state code longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } // pInfo->numOfRows data belong to the current session window updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &window, false); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &window, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, + doApplyFunctions(pTaskInfo, pSup->pCtx, &window, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, pRowSup->numOfRows, NULL, pBlock->info.rows, numOfOutput, TSDB_ORDER_ASC); // here we start a new session window @@ -1622,14 +1638,14 @@ static void doSessionWindowAggImpl(SOperatorInfo* pOperator, SSessionAggOperator SResultRow* pResult = NULL; pRowSup->win.ekey = tsList[pBlock->info.rows - 1]; int32_t ret = - setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &pRowSup->win, masterScan, &pResult, gid, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, &pRowSup->win, masterScan, &pResult, gid, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS) { // null data, too many state code longjmp(pTaskInfo->env, TSDB_CODE_QRY_APP_ERROR); } updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pRowSup->win, false); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &pRowSup->win, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, + doApplyFunctions(pTaskInfo, pSup->pCtx, &pRowSup->win, &pInfo->twAggSup.timeWindowData, pRowSup->startRowIndex, pRowSup->numOfRows, NULL, pBlock->info.rows, numOfOutput, TSDB_ORDER_ASC); } @@ -1640,10 +1656,11 @@ static SSDataBlock* doSessionWindowAgg(SOperatorInfo* pOperator) { SSessionAggOperatorInfo* pInfo = pOperator->info; SOptrBasicInfo* pBInfo = &pInfo->binfo; + SExprSupp* pSup = &pOperator->exprSupp; if (pOperator->status == OP_RES_TO_RETURN) { doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); return NULL; } @@ -1663,7 +1680,7 @@ static SSDataBlock* doSessionWindowAgg(SOperatorInfo* pOperator) { } // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pBInfo->pCtx, pBlock, order, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, MAIN_SCAN, true); blockDataUpdateTsWindow(pBlock, pInfo->tsSlotId); doSessionWindowAggImpl(pOperator, pInfo, pBlock); @@ -1678,7 +1695,7 @@ static SSDataBlock* doSessionWindowAgg(SOperatorInfo* pOperator) { initGroupedResultInfo(&pInfo->groupResInfo, pInfo->aggSup.pResultRowHashTable, TSDB_ORDER_ASC); blockDataEnsureCapacity(pBInfo->pRes, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } @@ -1711,10 +1728,11 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) { STimeSliceOperatorInfo* pSliceInfo = pOperator->info; SSDataBlock* pResBlock = pSliceInfo->binfo.pRes; + SExprSupp* pSup = &pOperator->exprSupp; // if (pOperator->status == OP_RES_TO_RETURN) { // // doBuildResultDatablock(&pRuntimeEnv->groupResInfo, pRuntimeEnv, pIntervalInfo->pRes); -// if (pResBlock->info.rows == 0 || !hashRemainDataInGroupInfo(&pSliceInfo->groupResInfo)) { +// if (pResBlock->info.rows == 0 || !hasDataInGroupInfo(&pSliceInfo->groupResInfo)) { // doSetOperatorCompleted(pOperator); // } // @@ -1733,15 +1751,15 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) { } // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pSliceInfo->binfo.pCtx, pBlock, order, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, order, MAIN_SCAN, true); SColumnInfoData* pTsCol = taosArrayGet(pBlock->pDataBlock, 0); for(int32_t i = 0; i < pBlock->info.rows; ++i) { int64_t ts = *(int64_t*) colDataGetData(pTsCol, i); if (ts == pSliceInfo->current) { - for(int32_t j = 0; j < pOperator->numOfExprs; ++j) { - SExprInfo* pExprInfo = &pOperator->pExpr[j]; + for(int32_t j = 0; j < pOperator->exprSupp.numOfExprs; ++j) { + SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[j]; int32_t dstSlot = pExprInfo->base.resSchema.slotId; int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId; @@ -1764,8 +1782,8 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) { int64_t nextTs = *(int64_t*) colDataGetData(pTsCol, i + 1); if (nextTs > pSliceInfo->current) { // output the result - for (int32_t j = 0; j < pOperator->numOfExprs; ++j) { - SExprInfo* pExprInfo = &pOperator->pExpr[j]; + for (int32_t j = 0; j < pOperator->exprSupp.numOfExprs; ++j) { + SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[j]; int32_t dstSlot = pExprInfo->base.resSchema.slotId; int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId; @@ -1903,12 +1921,14 @@ SOperatorInfo* createTimeSliceOperatorInfo(SOperatorInfo* downstream, SExprInfo* goto _error; } - int32_t code = initTimesliceInfo(pInfo, pInfo->binfo.pCtx, numOfCols); + SExprSupp* pSup = &pOperator->exprSupp; + + int32_t code = initTimesliceInfo(pInfo, pSup->pCtx, numOfCols); if (code != TSDB_CODE_SUCCESS) { goto _error; } - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pInfo->pFillColInfo = createFillColInfo(pExprInfo, numOfCols, pValNode); pInfo->binfo.pRes = pResultBlock; @@ -1917,8 +1937,8 @@ SOperatorInfo* createTimeSliceOperatorInfo(SOperatorInfo* downstream, SExprInfo* // pOperator->operatorType = OP_AllTimeWindow; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->pTaskInfo = pTaskInfo; @@ -1955,8 +1975,10 @@ SOperatorInfo* createStatewindowOperatorInfo(SOperatorInfo* downstream, SExprInf size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, 4096); - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExpr, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExpr, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); + + initResultRowInfo(&pInfo->binfo.resultRowInfo); pInfo->twAggSup = *pTwAggSup; initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); @@ -1966,8 +1988,8 @@ SOperatorInfo* createStatewindowOperatorInfo(SOperatorInfo* downstream, SExprInf pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExpr; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExpr; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->pTaskInfo = pTaskInfo; pOperator->info = pInfo; @@ -1984,7 +2006,7 @@ _error: void destroySWindowOperatorInfo(void* param, int32_t numOfOutput) { SSessionAggOperatorInfo* pInfo = (SSessionAggOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); } SOperatorInfo* createSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, @@ -1999,14 +2021,15 @@ SOperatorInfo* createSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, 4096); - int32_t code = - initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); + int32_t code = initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); if (code != TSDB_CODE_SUCCESS) { goto _error; } + initBasicInfo(&pInfo->binfo, pResBlock); + pInfo->twAggSup = *pTwAggSupp; - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + initResultRowInfo(&pInfo->binfo.resultRowInfo); initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); pInfo->tsSlotId = tsSlotId; @@ -2018,8 +2041,8 @@ SOperatorInfo* createSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doSessionWindowAgg, NULL, NULL, @@ -2040,8 +2063,6 @@ _error: return NULL; } -bool isFinalInterval(SStreamFinalIntervalOperatorInfo* pInfo) { return pInfo->pChildren != NULL; } - void compactFunctions(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx, int32_t numOfOutput, SExecTaskInfo* pTaskInfo) { for (int32_t k = 0; k < numOfOutput; ++k) { @@ -2060,23 +2081,25 @@ void compactFunctions(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx, int3 } } -static void rebuildIntervalWindow(SStreamFinalIntervalOperatorInfo* pInfo, SArray* pWinArray, int32_t groupId, +static void rebuildIntervalWindow(SStreamFinalIntervalOperatorInfo* pInfo, SExprSupp* pSup, SArray* pWinArray, int32_t groupId, int32_t numOfOutput, SExecTaskInfo* pTaskInfo) { int32_t size = taosArrayGetSize(pWinArray); ASSERT(pInfo->pChildren); for (int32_t i = 0; i < size; i++) { STimeWindow* pParentWin = taosArrayGet(pWinArray, i); SResultRow* pCurResult = NULL; - setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, pParentWin, true, &pCurResult, 0, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(&pInfo->binfo.resultRowInfo, pParentWin, true, &pCurResult, 0, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); int32_t numOfChildren = taosArrayGetSize(pInfo->pChildren); for (int32_t j = 0; j < numOfChildren; j++) { SOperatorInfo* pChildOp = taosArrayGetP(pInfo->pChildren, j); SIntervalAggOperatorInfo* pChInfo = pChildOp->info; + SExprSupp* pChildSup = &pChildOp->exprSupp; + SResultRow* pChResult = NULL; - setTimeWindowOutputBuf(&pChInfo->binfo.resultRowInfo, pParentWin, true, &pChResult, 0, pChInfo->binfo.pCtx, - pChildOp->numOfExprs, pChInfo->binfo.rowCellInfoOffset, &pChInfo->aggSup, pTaskInfo); - compactFunctions(pInfo->binfo.pCtx, pChInfo->binfo.pCtx, numOfOutput, pTaskInfo); + setTimeWindowOutputBuf(&pChInfo->binfo.resultRowInfo, pParentWin, true, &pChResult, 0, pChildSup->pCtx, + pChildSup->numOfExprs, pChildSup->rowEntryInfoOffset, &pChInfo->aggSup, pTaskInfo); + compactFunctions(pSup->pCtx, pChildSup->pCtx, numOfOutput, pTaskInfo); } } } @@ -2093,7 +2116,8 @@ static void doHashInterval(SOperatorInfo* pOperatorInfo, SSDataBlock* pSDataBloc SStreamFinalIntervalOperatorInfo* pInfo = (SStreamFinalIntervalOperatorInfo*)pOperatorInfo->info; SResultRowInfo* pResultRowInfo = &(pInfo->binfo.resultRowInfo); SExecTaskInfo* pTaskInfo = pOperatorInfo->pTaskInfo; - int32_t numOfOutput = pOperatorInfo->numOfExprs; + SExprSupp* pSup = &pOperatorInfo->exprSupp; + int32_t numOfOutput = pSup->numOfExprs; int32_t step = 1; bool ascScan = true; TSKEY* tsCols = NULL; @@ -2112,16 +2136,15 @@ static void doHashInterval(SOperatorInfo* pOperatorInfo, SSDataBlock* pSDataBloc STimeWindow nextWin = getActiveTimeWindow(pInfo->aggSup.pResultBuf, pResultRowInfo, ts, &pInfo->interval, pInfo->interval.precision, NULL); while (1) { - if (isFinalInterval(pInfo) && isCloseWindow(&nextWin, &pInfo->twAggSup) && + if (IS_FINAL_OP(pInfo) && isCloseWindow(&nextWin, &pInfo->twAggSup) && isDeletedWindow(&nextWin, tableGroupId, &pInfo->aggSup)) { SArray* pUpWins = taosArrayInit(8, sizeof(STimeWindow)); taosArrayPush(pUpWins, &nextWin); - rebuildIntervalWindow(pInfo, pUpWins, pInfo->binfo.pRes->info.groupId, - pOperatorInfo->numOfExprs, pOperatorInfo->pTaskInfo); + rebuildIntervalWindow(pInfo, pSup, pUpWins, pInfo->binfo.pRes->info.groupId, pSup->numOfExprs, pOperatorInfo->pTaskInfo); taosArrayDestroy(pUpWins); } - int32_t code = setTimeWindowOutputBuf(pResultRowInfo, &nextWin, true, &pResult, tableGroupId, pInfo->binfo.pCtx, - numOfOutput, pInfo->binfo.rowCellInfoOffset, &pInfo->aggSup, pTaskInfo); + int32_t code = setTimeWindowOutputBuf(pResultRowInfo, &nextWin, true, &pResult, tableGroupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo); if (code != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } @@ -2134,11 +2157,8 @@ static void doHashInterval(SOperatorInfo* pOperatorInfo, SSDataBlock* pSDataBloc if (pInfo->twAggSup.calTrigger == STREAM_TRIGGER_AT_ONCE && pUpdated) { saveResultRow(pResult, tableGroupId, pUpdated); } - // window start(end) key interpolation - // doWindowBorderInterpolation(pInfo, pSDataBlock, numOfOutput, pInfo->binfo.pCtx, pResult, &nextWin, startPos, - // forwardRows); updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &nextWin, true); - doApplyFunctions(pTaskInfo, pInfo->binfo.pCtx, &nextWin, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, + doApplyFunctions(pTaskInfo, pSup->pCtx, &nextWin, &pInfo->twAggSup.timeWindowData, startPos, forwardRows, tsCols, pSDataBlock->info.rows, numOfOutput, TSDB_ORDER_ASC); int32_t prevEndPos = (forwardRows - 1) * step + startPos; ASSERT(pSDataBlock->info.window.skey > 0 && pSDataBlock->info.window.ekey > 0); @@ -2153,7 +2173,7 @@ static void clearStreamIntervalOperator(SStreamFinalIntervalOperatorInfo* pInfo) taosHashClear(pInfo->aggSup.pResultRowHashTable); clearDiskbasedBuf(pInfo->aggSup.pResultBuf); cleanupResultRowInfo(&pInfo->binfo.resultRowInfo); - initResultRowInfo(&pInfo->binfo.resultRowInfo, 1); + initResultRowInfo(&pInfo->binfo.resultRowInfo); } static void clearUpdateDataBlock(SSDataBlock* pBlock) { @@ -2190,14 +2210,16 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { SArray* pUpdated = taosArrayInit(4, POINTER_BYTES); TSKEY maxTs = INT64_MIN; + SExprSupp* pSup = &pOperator->exprSupp; + if (pOperator->status == OP_EXEC_DONE) { return NULL; } else if (pOperator->status == OP_RES_TO_RETURN) { doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); if (pInfo->binfo.pRes->info.rows == 0) { pOperator->status = OP_EXEC_DONE; - if (isFinalInterval(pInfo) || pInfo->pUpdateRes->info.rows == 0) { - if (!isFinalInterval(pInfo)) { + if (IS_FINAL_OP(pInfo) || pInfo->pUpdateRes->info.rows == 0) { + if (!IS_FINAL_OP(pInfo)) { // semi interval operator clear disk buffer clearStreamIntervalOperator(pInfo); } @@ -2219,15 +2241,17 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { if (pBlock->info.type == STREAM_REPROCESS) { SArray* pUpWins = taosArrayInit(8, sizeof(STimeWindow)); - doClearWindows(&pInfo->aggSup, &pInfo->binfo, &pInfo->interval, pInfo->primaryTsIndex, pOperator->numOfExprs, + doClearWindows(&pInfo->aggSup, pSup, &pInfo->interval, pInfo->primaryTsIndex, pOperator->exprSupp.numOfExprs, pBlock, pUpWins); - if (isFinalInterval(pInfo)) { + if (IS_FINAL_OP(pInfo)) { int32_t childIndex = getChildIndex(pBlock); SOperatorInfo* pChildOp = taosArrayGetP(pInfo->pChildren, childIndex); SIntervalAggOperatorInfo* pChildInfo = pChildOp->info; - doClearWindows(&pChildInfo->aggSup, &pChildInfo->binfo, &pChildInfo->interval, pChildInfo->primaryTsIndex, - pChildOp->numOfExprs, pBlock, NULL); - rebuildIntervalWindow(pInfo, pUpWins, pInfo->binfo.pRes->info.groupId, pOperator->numOfExprs, + SExprSupp* pChildSup = &pChildOp->exprSupp; + + doClearWindows(&pChildInfo->aggSup, pChildSup, &pChildInfo->interval, pChildInfo->primaryTsIndex, + pChildSup->numOfExprs, pBlock, NULL); + rebuildIntervalWindow(pInfo, pSup, pUpWins, pInfo->binfo.pRes->info.groupId, pOperator->exprSupp.numOfExprs, pOperator->pTaskInfo); taosArrayDestroy(pUpWins); continue; @@ -2236,14 +2260,14 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { copyUpdateDataBlock(pInfo->pUpdateRes, pBlock, pInfo->primaryTsIndex); taosArrayDestroy(pUpWins); break; - } else if (pBlock->info.type == STREAM_GET_ALL && isFinalInterval(pInfo)) { + } else if (pBlock->info.type == STREAM_GET_ALL && IS_FINAL_OP(pInfo)) { getAllIntervalWindow(pInfo->aggSup.pResultRowHashTable, pUpdated); continue; } - setInputDataBlock(pOperator, pInfo->binfo.pCtx, pBlock, pInfo->order, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, pInfo->order, MAIN_SCAN, true); doHashInterval(pOperator, pBlock, pBlock->info.groupId, pUpdated); - if (isFinalInterval(pInfo)) { + if (IS_FINAL_OP(pInfo)) { int32_t chIndex = getChildIndex(pBlock); int32_t size = taosArrayGetSize(pInfo->pChildren); // if chIndex + 1 - size > 0, add new child @@ -2256,18 +2280,18 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { } SOperatorInfo* pChildOp = taosArrayGetP(pInfo->pChildren, chIndex); SStreamFinalIntervalOperatorInfo* pChInfo = pChildOp->info; - setInputDataBlock(pChildOp, pChInfo->binfo.pCtx, pBlock, pChInfo->order, MAIN_SCAN, true); + setInputDataBlock(pChildOp, pChildOp->exprSupp.pCtx, pBlock, pChInfo->order, MAIN_SCAN, true); doHashInterval(pChildOp, pBlock, pBlock->info.groupId, NULL); } maxTs = TMAX(maxTs, pBlock->info.window.ekey); } pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, maxTs); - if (isFinalInterval(pInfo)) { + if (IS_FINAL_OP(pInfo)) { closeIntervalWindow(pInfo->aggSup.pResultRowHashTable, &pInfo->twAggSup, &pInfo->interval, pUpdated); } - finalizeUpdatedResult(pOperator->numOfExprs, pInfo->aggSup.pResultBuf, pUpdated, pInfo->binfo.rowCellInfoOffset); + finalizeUpdatedResult(pOperator->exprSupp.numOfExprs, pInfo->aggSup.pResultBuf, pUpdated, pSup->rowEntryInfoOffset); initMultiResInfoFromArrayList(&pInfo->groupResInfo, pUpdated); blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); @@ -2292,6 +2316,7 @@ SOperatorInfo* createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, if (pInfo == NULL || pOperator == NULL) { goto _error; } + pInfo->order = TSDB_ORDER_ASC; pInfo->interval = (SInterval){.interval = pIntervalPhyNode->interval, .sliding = pIntervalPhyNode->sliding, @@ -2311,15 +2336,17 @@ SOperatorInfo* createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, int32_t numOfCols = 0; SExprInfo* pExprInfo = createExprInfo(pIntervalPhyNode->window.pFuncs, NULL, &numOfCols); SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); - int32_t code = initAggInfo(&pInfo->binfo, &pInfo->aggSup, pExprInfo, numOfCols, - pResBlock, keyBufSize, pTaskInfo->id.str); + + int32_t code = initAggInfo(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&pInfo->binfo, pResBlock); + ASSERT(numOfCols > 0); - increaseTs(pInfo->binfo.pCtx); + increaseTs(pOperator->exprSupp.pCtx); initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); if (code != TSDB_CODE_SUCCESS) { goto _error; } - initResultRowInfo(&pInfo->binfo.resultRowInfo, (int32_t)1); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pInfo->pChildren = NULL; if (numOfChild > 0) { pInfo->pChildren = taosArrayInit(numOfChild, sizeof(SOperatorInfo)); @@ -2333,7 +2360,7 @@ SOperatorInfo* createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, } } // semi interval operator does not catch result - if (!isFinalInterval(pInfo)) { + if (!IS_FINAL_OP(pInfo)) { pInfo->twAggSup.calTrigger = STREAM_TRIGGER_AT_ONCE; } pInfo->pUpdateRes = createResDataBlock(pPhyNode->pOutputDataBlockDesc); @@ -2341,13 +2368,20 @@ SOperatorInfo* createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, blockDataEnsureCapacity(pInfo->pUpdateRes, 128); pInfo->pPhyNode = (SPhysiNode*)nodesCloneNode((SNode*)pPhyNode); - pOperator->name = "StreamFinalIntervalOperator"; - pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL; + if (pPhyNode->type == QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL) { + pInfo->isFinal = true; + pOperator->name = "StreamFinalIntervalOperator"; + } else { + pInfo->isFinal = false; + pOperator->name = "StreamSemiIntervalOperator"; + } + + pOperator->operatorType = pPhyNode->type; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->fpSet = @@ -2382,7 +2416,7 @@ void destroyStreamAggSupporter(SStreamAggSupporter* pSup) { void destroyStreamSessionAggOperatorInfo(void* param, int32_t numOfOutput) { SStreamSessionAggOperatorInfo* pInfo = (SStreamSessionAggOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); destroyStreamAggSupporter(&pInfo->streamAggSup); cleanupGroupResInfo(&pInfo->groupResInfo); if (pInfo->pChildren != NULL) { @@ -2397,14 +2431,16 @@ void destroyStreamSessionAggOperatorInfo(void* param, int32_t numOfOutput) { } } -int32_t initBiasicInfo(SOptrBasicInfo* pBasicInfo, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock) { - pBasicInfo->pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pBasicInfo->rowCellInfoOffset); - pBasicInfo->pRes = pResultBlock; +int32_t initBasicInfoEx(SOptrBasicInfo* pBasicInfo, SExprSupp* pSup, SExprInfo* pExprInfo, int32_t numOfCols, SSDataBlock* pResultBlock) { + initExprSupp(pSup, pExprInfo, numOfCols); + initBasicInfo(pBasicInfo, pResultBlock); + for (int32_t i = 0; i < numOfCols; ++i) { - pBasicInfo->pCtx[i].pBuf = NULL; + pSup->pCtx[i].pBuf = NULL; } + ASSERT(numOfCols > 0); - increaseTs(pBasicInfo->pCtx); + increaseTs(pSup->pCtx); return TSDB_CODE_SUCCESS; } @@ -2425,24 +2461,28 @@ int32_t initSessionAggSupporter(SStreamAggSupporter* pSup, const char* pKey, Sql return initStreamAggSupporter(pSup, pKey, pCtx, numOfOutput, sizeof(SResultWindowInfo)); } -SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, int32_t numOfCols, - SSDataBlock* pResBlock, int64_t gap, int32_t tsSlotId, - STimeWindowAggSupp* pTwAggSupp, SExecTaskInfo* pTaskInfo) { - int32_t code = TSDB_CODE_OUT_OF_MEMORY; +SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo) { + SSessionWinodwPhysiNode* pSessionNode = (SSessionWinodwPhysiNode*)pPhyNode; + int32_t numOfCols = 0; + SExprInfo* pExprInfo = createExprInfo(pSessionNode->window.pFuncs, NULL, &numOfCols); + SSDataBlock* pResBlock = createResDataBlock(pPhyNode->pOutputDataBlockDesc); + int32_t code = TSDB_CODE_OUT_OF_MEMORY; SStreamSessionAggOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SStreamSessionAggOperatorInfo)); - SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); + SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); if (pInfo == NULL || pOperator == NULL) { goto _error; } initResultSizeInfo(pOperator, 4096); + SExprSupp* pSup = &pOperator->exprSupp; - code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock); + code = initBasicInfoEx(&pInfo->binfo, pSup, pExprInfo, numOfCols, pResBlock); if (code != TSDB_CODE_SUCCESS) { goto _error; } - code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo", pInfo->binfo.pCtx, numOfCols); + + code = initSessionAggSupporter(&pInfo->streamAggSup, "StreamSessionAggOperatorInfo", pSup->pCtx, numOfCols); if (code != TSDB_CODE_SUCCESS) { goto _error; } @@ -2451,14 +2491,21 @@ SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SEx if (pInfo->pDummyCtx == NULL) { goto _error; } - initDummyFunction(pInfo->pDummyCtx, pInfo->binfo.pCtx, numOfCols); + initDummyFunction(pInfo->pDummyCtx, pSup->pCtx, numOfCols); - pInfo->twAggSup = *pTwAggSupp; - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + pInfo->twAggSup = (STimeWindowAggSupp) { + .waterMark = pSessionNode->window.watermark, + .calTrigger = pSessionNode->window.triggerType, + .maxTs = INT64_MIN}; + + initResultRowInfo(&pInfo->binfo.resultRowInfo); initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); - pInfo->primaryTsIndex = tsSlotId; - pInfo->gap = gap; + pInfo->primaryTsIndex = ((SColumnNode*)pSessionNode->window.pTspk)->slotId; + if (pSessionNode->window.pTsEnd) { + pInfo->endTsIndex = ((SColumnNode*)pSessionNode->window.pTsEnd)->slotId; + } + pInfo->gap = pSessionNode->gap; pInfo->binfo.pRes = pResBlock; pInfo->order = TSDB_ORDER_ASC; _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY); @@ -2467,20 +2514,24 @@ SOperatorInfo* createStreamSessionAggOperatorInfo(SOperatorInfo* downstream, SEx pInfo->pDelRes = createOneDataBlock(pResBlock, false); blockDataEnsureCapacity(pInfo->pDelRes, 64); pInfo->pChildren = NULL; + pInfo->isFinal = false; + pInfo->pPhyNode = pPhyNode; pOperator->name = "StreamSessionWindowAggOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = pInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doStreamSessionAgg, NULL, NULL, destroyStreamSessionAggOperatorInfo, aggEncodeResultRow, aggDecodeResultRow, NULL); pOperator->pTaskInfo = pTaskInfo; - initDownStream(downstream, &pInfo->streamAggSup, pInfo->gap, pInfo->twAggSup.waterMark, pOperator->operatorType); - code = appendDownstream(pOperator, &downstream, 1); + if (downstream) { + initDownStream(downstream, &pInfo->streamAggSup, pInfo->gap, pInfo->twAggSup.waterMark, pOperator->operatorType); + code = appendDownstream(pOperator, &downstream, 1); + } return pOperator; _error: @@ -2530,21 +2581,22 @@ SArray* getWinInfos(SStreamAggSupporter* pAggSup, uint64_t groupId) { return pWinInfos; } -SResultWindowInfo* getSessionTimeWindow(SStreamAggSupporter* pAggSup, TSKEY ts, uint64_t groupId, int64_t gap, int32_t* pIndex) { +SResultWindowInfo* getSessionTimeWindow(SStreamAggSupporter* pAggSup, TSKEY startTs, + TSKEY endTs, uint64_t groupId, int64_t gap, int32_t* pIndex) { SArray* pWinInfos = getWinInfos(pAggSup, groupId); pAggSup->pCurWins = pWinInfos; int32_t size = taosArrayGetSize(pWinInfos); if (size == 0) { *pIndex = 0; - return addNewSessionWindow(pWinInfos, ts); + return addNewSessionWindow(pWinInfos, startTs); } // find the first position which is smaller than the key - int32_t index = binarySearch(pWinInfos, size, ts, TSDB_ORDER_DESC, getSessionWindowEndkey); + int32_t index = binarySearch(pWinInfos, size, startTs, TSDB_ORDER_DESC, getSessionWindowEndkey); SResultWindowInfo* pWin = NULL; if (index >= 0) { pWin = taosArrayGet(pWinInfos, index); - if (isInWindow(pWin, ts, gap)) { + if (isInWindow(pWin, startTs, gap)) { *pIndex = index; return pWin; } @@ -2552,40 +2604,46 @@ SResultWindowInfo* getSessionTimeWindow(SStreamAggSupporter* pAggSup, TSKEY ts, if (index + 1 < size) { pWin = taosArrayGet(pWinInfos, index + 1); - if (isInWindow(pWin, ts, gap)) { + if (isInWindow(pWin, startTs, gap)) { *pIndex = index + 1; return pWin; + } else if (endTs != INT64_MIN && isInWindow(pWin, endTs, gap)) { + *pIndex = index; + return pWin; } } if (index == size - 1) { *pIndex = taosArrayGetSize(pWinInfos); - return addNewSessionWindow(pWinInfos, ts); + return addNewSessionWindow(pWinInfos, startTs); } *pIndex = index + 1; - return insertNewSessionWindow(pWinInfos, ts, index + 1); + return insertNewSessionWindow(pWinInfos, startTs, index + 1); } -int32_t updateSessionWindowInfo(SResultWindowInfo* pWinInfo, TSKEY* pTs, int32_t rows, int32_t start, int64_t gap, - SHashObj* pStDeleted) { +int32_t updateSessionWindowInfo(SResultWindowInfo* pWinInfo, TSKEY* pStartTs, + TSKEY* pEndTs, int32_t rows, int32_t start, int64_t gap, SHashObj* pStDeleted) { for (int32_t i = start; i < rows; ++i) { - if (!isInWindow(pWinInfo, pTs[i], gap)) { + if (!isInWindow(pWinInfo, pStartTs[i], gap) && (!pEndTs || !isInWindow(pWinInfo, pEndTs[i], gap)) ) { return i - start; } - if (pWinInfo->win.skey > pTs[i]) { + if (pWinInfo->win.skey > pStartTs[i]) { if (pStDeleted && pWinInfo->isOutput) { taosHashPut(pStDeleted, &pWinInfo->pos, sizeof(SResultRowPosition), &pWinInfo->win.skey, sizeof(TSKEY)); pWinInfo->isOutput = false; } - pWinInfo->win.skey = pTs[i]; + pWinInfo->win.skey = pStartTs[i]; + } + pWinInfo->win.ekey = TMAX(pWinInfo->win.ekey, pStartTs[i]); + if (pEndTs) { + pWinInfo->win.ekey = TMAX(pWinInfo->win.ekey, pEndTs[i]); } - pWinInfo->win.ekey = TMAX(pWinInfo->win.ekey, pTs[i]); } return rows - start; } static int32_t setWindowOutputBuf(SResultWindowInfo* pWinInfo, SResultRow** pResult, SqlFunctionCtx* pCtx, - uint64_t groupId, int32_t numOfOutput, int32_t* rowCellInfoOffset, + uint64_t groupId, int32_t numOfOutput, int32_t* rowEntryInfoOffset, SStreamAggSupporter* pAggSup, SExecTaskInfo* pTaskInfo) { assert(pWinInfo->win.skey <= pWinInfo->win.ekey); // too many time window in query @@ -2614,39 +2672,42 @@ static int32_t setWindowOutputBuf(SResultWindowInfo* pWinInfo, SResultRow** pRes // set time window for current result (*pResult)->win = pWinInfo->win; - setResultRowInitCtx(*pResult, pCtx, numOfOutput, rowCellInfoOffset); + setResultRowInitCtx(*pResult, pCtx, numOfOutput, rowEntryInfoOffset); return TSDB_CODE_SUCCESS; } static int32_t doOneWindowAggImpl(int32_t tsColId, SOptrBasicInfo* pBinfo, SStreamAggSupporter* pAggSup, SColumnInfoData* pTimeWindowData, SSDataBlock* pSDataBlock, SResultWindowInfo* pCurWin, SResultRow** pResult, int32_t startIndex, int32_t winRows, - int32_t numOutput, SExecTaskInfo* pTaskInfo) { + int32_t numOutput, SOperatorInfo* pOperator) { + SExprSupp* pSup = &pOperator->exprSupp; + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SColumnInfoData* pColDataInfo = taosArrayGet(pSDataBlock->pDataBlock, tsColId); TSKEY* tsCols = (int64_t*)pColDataInfo->pData; - int32_t code = setWindowOutputBuf(pCurWin, pResult, pBinfo->pCtx, pSDataBlock->info.groupId, numOutput, - pBinfo->rowCellInfoOffset, pAggSup, pTaskInfo); + int32_t code = setWindowOutputBuf(pCurWin, pResult, pSup->pCtx, pSDataBlock->info.groupId, numOutput, + pSup->rowEntryInfoOffset, pAggSup, pTaskInfo); if (code != TSDB_CODE_SUCCESS || (*pResult) == NULL) { return TSDB_CODE_QRY_OUT_OF_MEMORY; } - updateTimeWindowInfo(pTimeWindowData, &pCurWin->win, true); - doApplyFunctions(pTaskInfo, pBinfo->pCtx, &pCurWin->win, pTimeWindowData, startIndex, winRows, tsCols, + updateTimeWindowInfo(pTimeWindowData, &pCurWin->win, false); + doApplyFunctions(pTaskInfo, pSup->pCtx, &pCurWin->win, pTimeWindowData, startIndex, winRows, tsCols, pSDataBlock->info.rows, numOutput, TSDB_ORDER_ASC); return TSDB_CODE_SUCCESS; } static int32_t doOneWindowAgg(SStreamSessionAggOperatorInfo* pInfo, SSDataBlock* pSDataBlock, SResultWindowInfo* pCurWin, SResultRow** pResult, int32_t startIndex, int32_t winRows, - int32_t numOutput, SExecTaskInfo* pTaskInfo) { + int32_t numOutput, SOperatorInfo * pOperator) { return doOneWindowAggImpl(pInfo->primaryTsIndex, &pInfo->binfo, &pInfo->streamAggSup, &pInfo->twAggSup.timeWindowData, - pSDataBlock, pCurWin, pResult, startIndex, winRows, numOutput, pTaskInfo); + pSDataBlock, pCurWin, pResult, startIndex, winRows, numOutput, pOperator); } static int32_t doOneStateWindowAgg(SStreamStateAggOperatorInfo* pInfo, SSDataBlock* pSDataBlock, SResultWindowInfo* pCurWin, SResultRow** pResult, int32_t startIndex, - int32_t winRows, int32_t numOutput, SExecTaskInfo* pTaskInfo) { + int32_t winRows, int32_t numOutput, SOperatorInfo * pOperator) { return doOneWindowAggImpl(pInfo->primaryTsIndex, &pInfo->binfo, &pInfo->streamAggSup, &pInfo->twAggSup.timeWindowData, - pSDataBlock, pCurWin, pResult, startIndex, winRows, numOutput, pTaskInfo); + pSDataBlock, pCurWin, pResult, startIndex, winRows, numOutput, pOperator); } int32_t getNumCompactWindow(SArray* pWinInfos, int32_t startIndex, int64_t gap) { @@ -2664,10 +2725,13 @@ int32_t getNumCompactWindow(SArray* pWinInfos, int32_t startIndex, int64_t gap) } void compactTimeWindow(SStreamSessionAggOperatorInfo* pInfo, int32_t startIndex, int32_t num, uint64_t groupId, - int32_t numOfOutput, SExecTaskInfo* pTaskInfo, SHashObj* pStUpdated, SHashObj* pStDeleted) { + int32_t numOfOutput, SHashObj* pStUpdated, SHashObj* pStDeleted, SOperatorInfo* pOperator) { + SExprSupp* pSup = &pOperator->exprSupp; + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + SResultWindowInfo* pCurWin = taosArrayGet(pInfo->streamAggSup.pCurWins, startIndex); SResultRow* pCurResult = NULL; - setWindowOutputBuf(pCurWin, &pCurResult, pInfo->binfo.pCtx, groupId, numOfOutput, pInfo->binfo.rowCellInfoOffset, + setWindowOutputBuf(pCurWin, &pCurResult, pSup->pCtx, groupId, numOfOutput, pSup->rowEntryInfoOffset, &pInfo->streamAggSup, pTaskInfo); num += startIndex + 1; ASSERT(num <= taosArrayGetSize(pInfo->streamAggSup.pCurWins)); @@ -2675,10 +2739,10 @@ void compactTimeWindow(SStreamSessionAggOperatorInfo* pInfo, int32_t startIndex, for (int32_t i = startIndex + 1; i < num; i++) { SResultWindowInfo* pWinInfo = taosArrayGet(pInfo->streamAggSup.pCurWins, i); SResultRow* pWinResult = NULL; - setWindowOutputBuf(pWinInfo, &pWinResult, pInfo->pDummyCtx, groupId, numOfOutput, pInfo->binfo.rowCellInfoOffset, + setWindowOutputBuf(pWinInfo, &pWinResult, pInfo->pDummyCtx, groupId, numOfOutput, pSup->rowEntryInfoOffset, &pInfo->streamAggSup, pTaskInfo); pCurWin->win.ekey = TMAX(pCurWin->win.ekey, pWinInfo->win.ekey); - compactFunctions(pInfo->binfo.pCtx, pInfo->pDummyCtx, numOfOutput, pTaskInfo); + compactFunctions(pSup->pCtx, pInfo->pDummyCtx, numOfOutput, pTaskInfo); taosHashRemove(pStUpdated, &pWinInfo->pos, sizeof(SResultRowPosition)); if (pWinInfo->isOutput) { taosHashPut(pStDeleted, &pWinInfo->pos, sizeof(SResultRowPosition), &pWinInfo->win.skey, sizeof(TSKEY)); @@ -2693,25 +2757,33 @@ typedef struct SWinRes { uint64_t groupId; } SWinRes; -static void doStreamSessionAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBlock, SHashObj* pStUpdated, - SHashObj* pStDeleted) { +static void doStreamSessionAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBlock, + SHashObj* pStUpdated, SHashObj* pStDeleted, bool hasEndTs) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SStreamSessionAggOperatorInfo* pInfo = pOperator->info; bool masterScan = true; - int32_t numOfOutput = pOperator->numOfExprs; + int32_t numOfOutput = pOperator->exprSupp.numOfExprs; uint64_t groupId = pSDataBlock->info.groupId; int64_t gap = pInfo->gap; int64_t code = TSDB_CODE_SUCCESS; int32_t step = 1; bool ascScan = true; - TSKEY* tsCols = NULL; + TSKEY* startTsCols = NULL; + TSKEY* endTsCols = NULL; SResultRow* pResult = NULL; int32_t winRows = 0; if (pSDataBlock->pDataBlock != NULL) { - SColumnInfoData* pColDataInfo = taosArrayGet(pSDataBlock->pDataBlock, pInfo->primaryTsIndex); - tsCols = (int64_t*)pColDataInfo->pData; + SColumnInfoData* pStartTsCol = taosArrayGet(pSDataBlock->pDataBlock, pInfo->primaryTsIndex); + startTsCols = (int64_t*) pStartTsCol->pData; + SColumnInfoData* pEndTsCol = NULL; + if (hasEndTs) { + pEndTsCol = taosArrayGet(pSDataBlock->pDataBlock, pInfo->endTsIndex); + } else { + pEndTsCol = taosArrayGet(pSDataBlock->pDataBlock, pInfo->primaryTsIndex); + } + endTsCols = (int64_t*) pEndTsCol->pData; } else { return; } @@ -2719,22 +2791,21 @@ static void doStreamSessionAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSData SStreamAggSupporter* pAggSup = &pInfo->streamAggSup; for (int32_t i = 0; i < pSDataBlock->info.rows;) { int32_t winIndex = 0; - SResultWindowInfo* pCurWin = getSessionTimeWindow(pAggSup, tsCols[i], groupId, gap, &winIndex); - winRows = updateSessionWindowInfo(pCurWin, tsCols, pSDataBlock->info.rows, i, pInfo->gap, pStDeleted); - code = doOneWindowAgg(pInfo, pSDataBlock, pCurWin, &pResult, i, winRows, numOfOutput, pTaskInfo); + SResultWindowInfo* pCurWin = getSessionTimeWindow(pAggSup, startTsCols[i], + endTsCols[i], groupId, gap, &winIndex); + winRows = updateSessionWindowInfo(pCurWin, startTsCols, endTsCols, + pSDataBlock->info.rows, i, pInfo->gap, pStDeleted); + code = doOneWindowAgg(pInfo, pSDataBlock, pCurWin, &pResult, i, winRows, numOfOutput, pOperator); if (code != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } - // window start(end) key interpolation - // doWindowBorderInterpolation(pOperatorInfo, pSDataBlock, pInfo->binfo.pCtx, pResult, &nextWin, startPos, - // forwardRows, - // pInfo->order, false); + int32_t winNum = getNumCompactWindow(pAggSup->pCurWins, winIndex, gap); if (winNum > 0) { - compactTimeWindow(pInfo, winIndex, winNum, groupId, numOfOutput, pTaskInfo, pStUpdated, pStDeleted); + compactTimeWindow(pInfo, winIndex, winNum, groupId, numOfOutput, pStUpdated, pStDeleted, pOperator); } pCurWin->isClosed = false; - if (pInfo->twAggSup.calTrigger == STREAM_TRIGGER_AT_ONCE) { + if (pInfo->twAggSup.calTrigger == STREAM_TRIGGER_AT_ONCE && pStUpdated) { SWinRes value = {.ts = pCurWin->win.skey, .groupId = groupId}; code = taosHashPut(pStUpdated, &pCurWin->pos, sizeof(SResultRowPosition), &value, sizeof(SWinRes)); if (code != TSDB_CODE_SUCCESS) { @@ -2746,17 +2817,17 @@ static void doStreamSessionAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSData } } -static void doClearSessionWindows(SStreamAggSupporter* pAggSup, SOptrBasicInfo* pBinfo, SSDataBlock* pBlock, +static void doClearSessionWindows(SStreamAggSupporter* pAggSup, SExprSupp* pSup, SSDataBlock* pBlock, int32_t tsIndex, int32_t numOfOutput, int64_t gap, SArray* result) { SColumnInfoData* pColDataInfo = taosArrayGet(pBlock->pDataBlock, tsIndex); TSKEY* tsCols = (TSKEY*)pColDataInfo->pData; int32_t step = 0; for (int32_t i = 0; i < pBlock->info.rows; i += step) { int32_t winIndex = 0; - SResultWindowInfo* pCurWin = getSessionTimeWindow(pAggSup, tsCols[i], pBlock->info.groupId, gap, &winIndex); - step = updateSessionWindowInfo(pCurWin, tsCols, pBlock->info.rows, i, gap, NULL); + SResultWindowInfo* pCurWin = getSessionTimeWindow(pAggSup, tsCols[i], INT64_MIN, pBlock->info.groupId, gap, &winIndex); + step = updateSessionWindowInfo(pCurWin, tsCols, NULL, pBlock->info.rows, i, gap, NULL); ASSERT(isInWindow(pCurWin, tsCols[i], gap)); - doClearWindowImpl(&pCurWin->pos, pAggSup->pResultBuf, pBinfo, numOfOutput); + doClearWindowImpl(&pCurWin->pos, pAggSup->pResultBuf, pSup, numOfOutput); if (result) { taosArrayPush(result, pCurWin); } @@ -2802,13 +2873,17 @@ void doBuildDeleteDataBlock(SHashObj* pStDeleted, SSDataBlock* pBlock, void** It } static void rebuildTimeWindow(SStreamSessionAggOperatorInfo* pInfo, SArray* pWinArray, int32_t groupId, - int32_t numOfOutput, SExecTaskInfo* pTaskInfo) { + int32_t numOfOutput, SOperatorInfo* pOperator) { + SExprSupp* pSup = &pOperator->exprSupp; + SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; + int32_t size = taosArrayGetSize(pWinArray); ASSERT(pInfo->pChildren); + for (int32_t i = 0; i < size; i++) { SResultWindowInfo* pParentWin = taosArrayGet(pWinArray, i); SResultRow* pCurResult = NULL; - setWindowOutputBuf(pParentWin, &pCurResult, pInfo->binfo.pCtx, groupId, numOfOutput, pInfo->binfo.rowCellInfoOffset, + setWindowOutputBuf(pParentWin, &pCurResult, pSup->pCtx, groupId, numOfOutput, pSup->rowEntryInfoOffset, &pInfo->streamAggSup, pTaskInfo); int32_t numOfChildren = taosArrayGetSize(pInfo->pChildren); for (int32_t j = 0; j < numOfChildren; j++) { @@ -2817,13 +2892,16 @@ static void rebuildTimeWindow(SStreamSessionAggOperatorInfo* pInfo, SArray* pWin SArray* pChWins = getWinInfos(&pChInfo->streamAggSup, groupId); int32_t chWinSize = taosArrayGetSize(pChWins); int32_t index = binarySearch(pChWins, chWinSize, pParentWin->win.skey, TSDB_ORDER_DESC, getSessionWindowEndkey); - for (int32_t k = index; k > 0 && k < chWinSize; k++) { + if (index < 0) { + index = 0; + } + for (int32_t k = index; k < chWinSize; k++) { SResultWindowInfo* pcw = taosArrayGet(pChWins, k); if (pParentWin->win.skey <= pcw->win.skey && pcw->win.ekey <= pParentWin->win.ekey) { SResultRow* pChResult = NULL; - setWindowOutputBuf(pcw, &pChResult, pChInfo->binfo.pCtx, groupId, numOfOutput, - pChInfo->binfo.rowCellInfoOffset, &pChInfo->streamAggSup, pTaskInfo); - compactFunctions(pInfo->binfo.pCtx, pChInfo->binfo.pCtx, numOfOutput, pTaskInfo); + setWindowOutputBuf(pcw, &pChResult, pChild->exprSupp.pCtx, groupId, numOfOutput, + pChild->exprSupp.rowEntryInfoOffset, &pChInfo->streamAggSup, pTaskInfo); + compactFunctions(pSup->pCtx, pChild->exprSupp.pCtx, numOfOutput, pTaskInfo); continue; } break; @@ -2832,11 +2910,9 @@ static void rebuildTimeWindow(SStreamSessionAggOperatorInfo* pInfo, SArray* pWin } } -bool isFinalSession(SStreamSessionAggOperatorInfo* pInfo) { return pInfo->pChildren != NULL; } - typedef SResultWindowInfo* (*__get_win_info_)(void*); -SResultWindowInfo* getSessionWinInfo(void* pData) { return (SResultWindowInfo*)pData; } -SResultWindowInfo* getStateWinInfo(void* pData) { return &((SStateWindowInfo*)pData)->winInfo; } +SResultWindowInfo* getResWinForSession(void* pData) { return (SResultWindowInfo*)pData; } +SResultWindowInfo* getResWinForState(void* pData) { return &((SStateWindowInfo*)pData)->winInfo; } int32_t closeSessionWindow(SHashObj* pHashMap, STimeWindowAggSupp* pTwSup, SArray* pClosed, __get_win_info_ fn) { @@ -2884,19 +2960,19 @@ int32_t getAllSessionWindow(SHashObj* pHashMap, SArray* pClosed, __get_win_info_ } static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { - if (pOperator->status == OP_EXEC_DONE) { - return NULL; - } - + SExprSupp* pSup = &pOperator->exprSupp; SStreamSessionAggOperatorInfo* pInfo = pOperator->info; SOptrBasicInfo* pBInfo = &pInfo->binfo; - if (pOperator->status == OP_RES_TO_RETURN) { + TSKEY maxTs = INT64_MIN; + if (pOperator->status == OP_EXEC_DONE) { + return NULL; + } else if (pOperator->status == OP_RES_TO_RETURN) { doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); if (pInfo->pDelRes->info.rows > 0) { return pInfo->pDelRes; } doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; @@ -2914,42 +2990,156 @@ static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { if (pBlock->info.type == STREAM_REPROCESS) { SArray* pWins = taosArrayInit(16, sizeof(SResultWindowInfo)); - doClearSessionWindows(&pInfo->streamAggSup, &pInfo->binfo, pBlock, 0, pOperator->numOfExprs, pInfo->gap, pWins); - if (isFinalSession(pInfo)) { - int32_t childIndex = 0; // Todo(liuyao) get child id from SSDataBlock + doClearSessionWindows(&pInfo->streamAggSup, &pOperator->exprSupp, pBlock, 0, pOperator->exprSupp.numOfExprs, pInfo->gap, pWins); + if (IS_FINAL_OP(pInfo)) { + int32_t childIndex = getChildIndex(pBlock); SOperatorInfo* pChildOp = taosArrayGetP(pInfo->pChildren, childIndex); SStreamSessionAggOperatorInfo* pChildInfo = pChildOp->info; - doClearSessionWindows(&pChildInfo->streamAggSup, &pChildInfo->binfo, pBlock, 0, pChildOp->numOfExprs, + doClearSessionWindows(&pChildInfo->streamAggSup, &pChildOp->exprSupp, pBlock, 0, pChildOp->exprSupp.numOfExprs, pChildInfo->gap, NULL); - rebuildTimeWindow(pInfo, pWins, pBlock->info.groupId, pOperator->numOfExprs, pOperator->pTaskInfo); + rebuildTimeWindow(pInfo, pWins, pBlock->info.groupId, pOperator->exprSupp.numOfExprs, pOperator); } taosArrayDestroy(pWins); continue; } else if (pBlock->info.type == STREAM_GET_ALL) { - getAllSessionWindow(pInfo->streamAggSup.pResultRows, pUpdated, getSessionWinInfo); + getAllSessionWindow(pInfo->streamAggSup.pResultRows, pUpdated, getResWinForSession); continue; } // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pBInfo->pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); - if (isFinalSession(pInfo)) { - int32_t childIndex = 0; // Todo(liuyao) get child id from SSDataBlock - SOptrBasicInfo* pChildOp = taosArrayGetP(pInfo->pChildren, childIndex); - doStreamSessionAggImpl(pOperator, pBlock, NULL, NULL); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); + doStreamSessionAggImpl(pOperator, pBlock, pStUpdated, pInfo->pStDeleted, IS_FINAL_OP(pInfo)); + if (IS_FINAL_OP(pInfo)) { + int32_t chIndex = getChildIndex(pBlock); + int32_t size = taosArrayGetSize(pInfo->pChildren); + // if chIndex + 1 - size > 0, add new child + for (int32_t i = 0; i < chIndex + 1 - size; i++) { + SOperatorInfo* pChildOp = createStreamFinalSessionAggOperatorInfo(NULL, pInfo->pPhyNode, pOperator->pTaskInfo, 0); + if (!pChildOp) { + longjmp(pOperator->pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); + } + taosArrayPush(pInfo->pChildren, &pChildOp); + } + SOperatorInfo* pChildOp = taosArrayGetP(pInfo->pChildren, chIndex); + setInputDataBlock(pChildOp, pChildOp->exprSupp.pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); + doStreamSessionAggImpl(pChildOp, pBlock, NULL, NULL, true); } - doStreamSessionAggImpl(pOperator, pBlock, pStUpdated, pInfo->pStDeleted); - pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, pBlock->info.window.ekey); + maxTs = TMAX(maxTs, pBlock->info.window.ekey); } + + pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, maxTs); // restore the value pOperator->status = OP_RES_TO_RETURN; closeSessionWindow(pInfo->streamAggSup.pResultRows, &pInfo->twAggSup, pUpdated, - getSessionWinInfo); + getResWinForSession); + copyUpdateResult(pStUpdated, pUpdated); + taosHashCleanup(pStUpdated); + + finalizeUpdatedResult(pSup->numOfExprs, pInfo->streamAggSup.pResultBuf, pUpdated, + pSup->rowEntryInfoOffset); + initMultiResInfoFromArrayList(&pInfo->groupResInfo, pUpdated); + blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); + doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + if (pInfo->pDelRes->info.rows > 0) { + return pInfo->pDelRes; + } + doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); + return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; +} + +static void clearStreamSessionOperator(SStreamSessionAggOperatorInfo* pInfo) { + void **pIte = NULL; + while ((pIte = taosHashIterate(pInfo->streamAggSup.pResultRows, pIte)) != NULL) { + SArray *pWins = (SArray *) (*pIte); + int32_t size = taosArrayGetSize(pWins); + for (int32_t i = 0; i < size; i++) { + SResultWindowInfo* pWin = (SResultWindowInfo*)taosArrayGet(pWins, i); + pWin->pos.pageId = -1; + pWin->pos.offset = -1; + } + } + clearDiskbasedBuf(pInfo->streamAggSup.pResultBuf); + cleanupResultRowInfo(&pInfo->binfo.resultRowInfo); + initResultRowInfo(&pInfo->binfo.resultRowInfo); +} + +static void removeSessionResults(SHashObj* pHashMap, SArray* pWins) { + int32_t size = taosArrayGetSize(pWins); + for (int32_t i = 0; i < size; i++) { + SResultWindowInfo* pWin = taosArrayGet(pWins, i); + taosHashRemove(pHashMap, &pWin->pos, sizeof(SResultRowPosition)); + } +} + +static SSDataBlock* doStreamSessionSemiAgg(SOperatorInfo* pOperator) { + SStreamSessionAggOperatorInfo* pInfo = pOperator->info; + SOptrBasicInfo* pBInfo = &pInfo->binfo; + TSKEY maxTs = INT64_MIN; + SExprSupp* pSup = &pOperator->exprSupp; + + if (pOperator->status == OP_EXEC_DONE) { + return NULL; + } else if (pOperator->status == OP_RES_TO_RETURN) { + doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + if (pInfo->pDelRes->info.rows > 0) { + return pInfo->pDelRes; + } + doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); + if (pInfo->binfo.pRes->info.rows == 0) { + pOperator->status = OP_EXEC_DONE; + if (pInfo->pUpdateRes->info.rows == 0) { + // semi interval operator clear disk buffer + clearStreamSessionOperator(pInfo); + return NULL; + } + // process the rest of the data + pOperator->status = OP_OPENED; + return pInfo->pUpdateRes; + } + return pInfo->binfo.pRes; + } + + _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY); + SHashObj* pStUpdated = taosHashInit(64, hashFn, true, HASH_NO_LOCK); + SOperatorInfo* downstream = pOperator->pDownstream[0]; + SArray* pUpdated = taosArrayInit(16, POINTER_BYTES); + while (1) { + SSDataBlock* pBlock = downstream->fpSet.getNextFn(downstream); + if (pBlock == NULL) { + clearUpdateDataBlock(pInfo->pUpdateRes); + break; + } + + if (pBlock->info.type == STREAM_REPROCESS) { + SArray* pWins = taosArrayInit(16, sizeof(SResultWindowInfo)); + doClearSessionWindows(&pInfo->streamAggSup, pSup, pBlock, 0, pSup->numOfExprs, pInfo->gap, pWins); + removeSessionResults(pStUpdated, pWins); + taosArrayDestroy(pWins); + copyUpdateDataBlock(pInfo->pUpdateRes, pBlock, pInfo->primaryTsIndex); + break; + } else if (pBlock->info.type == STREAM_GET_ALL) { + getAllSessionWindow(pInfo->streamAggSup.pResultRows, pUpdated, getResWinForSession); + continue; + } + + // the pDataBlock are always the same one, no need to call this again + setInputDataBlock(pOperator, pSup->pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); + doStreamSessionAggImpl(pOperator, pBlock, pStUpdated, pInfo->pStDeleted, false); + maxTs = TMAX(pInfo->twAggSup.maxTs, pBlock->info.window.ekey); + } + + pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, maxTs); + // restore the value + pOperator->status = OP_RES_TO_RETURN; + // semi operator + // closeSessionWindow(pInfo->streamAggSup.pResultRows, &pInfo->twAggSup, pUpdated, + // getResWinForSession); copyUpdateResult(pStUpdated, pUpdated); taosHashCleanup(pStUpdated); - finalizeUpdatedResult(pOperator->numOfExprs, pInfo->streamAggSup.pResultBuf, pUpdated, - pInfo->binfo.rowCellInfoOffset); + finalizeUpdatedResult(pOperator->exprSupp.numOfExprs, pInfo->streamAggSup.pResultBuf, pUpdated, + pSup->rowEntryInfoOffset); initMultiResInfoFromArrayList(&pInfo->groupResInfo, pUpdated); blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); @@ -2957,38 +3147,57 @@ static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { return pInfo->pDelRes; } doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); + if (pInfo->binfo.pRes->info.rows == 0) { + pOperator->status = OP_EXEC_DONE; + if (pInfo->pUpdateRes->info.rows == 0) { + return NULL; + } + // process the rest of the data + pOperator->status = OP_OPENED; + return pInfo->pUpdateRes; + } return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; } -SOperatorInfo* createStreamFinalSessionAggOperatorInfo(SOperatorInfo* downstream, SExprInfo* pExprInfo, - int32_t numOfCols, SSDataBlock* pResBlock, int64_t gap, - int32_t tsSlotId, STimeWindowAggSupp* pTwAggSupp, - SExecTaskInfo* pTaskInfo) { - int32_t code = TSDB_CODE_OUT_OF_MEMORY; - SStreamSessionAggOperatorInfo* pInfo = NULL; - SOperatorInfo* pOperator = createStreamSessionAggOperatorInfo(downstream, pExprInfo, numOfCols, pResBlock, gap, - tsSlotId, pTwAggSupp, pTaskInfo); +SOperatorInfo* createStreamFinalSessionAggOperatorInfo(SOperatorInfo* downstream, + SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, int32_t numOfChild) { + int32_t code = TSDB_CODE_OUT_OF_MEMORY; + SOperatorInfo* pOperator = createStreamSessionAggOperatorInfo(downstream, pPhyNode, pTaskInfo); if (pOperator == NULL) { goto _error; } - pOperator->name = "StreamFinalSessionWindowAggOperator"; - pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION; - int32_t numOfChild = 1; // Todo(liuyao) get it from phy plan - pInfo = pOperator->info; - pInfo->pChildren = taosArrayInit(8, sizeof(void*)); - for (int32_t i = 0; i < numOfChild; i++) { - SOperatorInfo* pChild = - createStreamSessionAggOperatorInfo(NULL, pExprInfo, numOfCols, NULL, gap, tsSlotId, pTwAggSupp, pTaskInfo); - if (pChild == NULL) { - goto _error; + SStreamSessionAggOperatorInfo* pInfo = pOperator->info; + + if (pPhyNode->type == QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION) { + pInfo->isFinal = true; + pOperator->name = "StreamSessionFinalAggOperator"; + } else { + pInfo->isFinal = false; + pInfo->pUpdateRes = createResDataBlock(pPhyNode->pOutputDataBlockDesc); + pInfo->pUpdateRes->info.type = STREAM_REPROCESS; + blockDataEnsureCapacity(pInfo->pUpdateRes, 128); + pOperator->name = "StreamSessionSemiAggOperator"; + pOperator->fpSet = + createOperatorFpSet(operatorDummyOpenFn, doStreamSessionSemiAgg, NULL, NULL, destroyStreamSessionAggOperatorInfo, + aggEncodeResultRow, aggDecodeResultRow, NULL); + } + pOperator->operatorType = pPhyNode->type; + if (numOfChild > 0) { + pInfo->pChildren = taosArrayInit(numOfChild, sizeof(void*)); + for (int32_t i = 0; i < numOfChild; i++) { + SOperatorInfo* pChild = + createStreamFinalSessionAggOperatorInfo(NULL, pPhyNode, pTaskInfo, 0); + if (pChild == NULL) { + goto _error; + } + taosArrayPush(pInfo->pChildren, &pChild); } - taosArrayPush(pInfo->pChildren, &pChild); } return pOperator; _error: if (pInfo != NULL) { - destroyStreamSessionAggOperatorInfo(pInfo, numOfCols); + destroyStreamSessionAggOperatorInfo(pInfo, pOperator->exprSupp.numOfExprs); } taosMemoryFreeClear(pInfo); @@ -2999,7 +3208,7 @@ _error: void destroyStreamStateOperatorInfo(void* param, int32_t numOfOutput) { SStreamStateAggOperatorInfo* pInfo = (SStreamStateAggOperatorInfo*)param; - doDestroyBasicInfo(&pInfo->binfo, numOfOutput); + cleanupBasicInfo(&pInfo->binfo); destroyStreamAggSupporter(&pInfo->streamAggSup); cleanupGroupResInfo(&pInfo->groupResInfo); if (pInfo->pChildren != NULL) { @@ -3207,7 +3416,7 @@ static void doStreamStateAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SStreamStateAggOperatorInfo* pInfo = pOperator->info; bool masterScan = true; - int32_t numOfOutput = pOperator->numOfExprs; + int32_t numOfOutput = pOperator->exprSupp.numOfExprs; int64_t groupId = pSDataBlock->info.groupId; int64_t code = TSDB_CODE_SUCCESS; int32_t step = 1; @@ -3239,7 +3448,7 @@ static void doStreamStateAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl deleteWindow(pAggSup->pCurWins, winIndex); continue; } - code = doOneStateWindowAgg(pInfo, pSDataBlock, &pCurWin->winInfo, &pResult, i, winRows, numOfOutput, pTaskInfo); + code = doOneStateWindowAgg(pInfo, pSDataBlock, &pCurWin->winInfo, &pResult, i, winRows, numOfOutput, pOperator); if (code != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } @@ -3261,6 +3470,7 @@ static SSDataBlock* doStreamStateAgg(SOperatorInfo* pOperator) { return NULL; } + SExprSupp* pSup = &pOperator->exprSupp; SStreamStateAggOperatorInfo* pInfo = pOperator->info; SOptrBasicInfo* pBInfo = &pInfo->binfo; if (pOperator->status == OP_RES_TO_RETURN) { @@ -3269,7 +3479,7 @@ static SSDataBlock* doStreamStateAgg(SOperatorInfo* pOperator) { return pInfo->pDelRes; } doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); - if (pBInfo->pRes->info.rows == 0 || !hashRemainDataInGroupInfo(&pInfo->groupResInfo)) { + if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; @@ -3290,12 +3500,12 @@ static SSDataBlock* doStreamStateAgg(SOperatorInfo* pOperator) { pSeUpdated, pInfo->pSeDeleted); continue; } else if (pBlock->info.type == STREAM_GET_ALL) { - getAllSessionWindow(pInfo->streamAggSup.pResultRows, pUpdated, getStateWinInfo); + getAllSessionWindow(pInfo->streamAggSup.pResultRows, pUpdated, getResWinForState); continue; } // the pDataBlock are always the same one, no need to call this again - setInputDataBlock(pOperator, pBInfo->pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, TSDB_ORDER_ASC, MAIN_SCAN, true); doStreamStateAggImpl(pOperator, pBlock, pSeUpdated, pInfo->pSeDeleted); pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, pBlock->info.window.ekey); } @@ -3303,12 +3513,12 @@ static SSDataBlock* doStreamStateAgg(SOperatorInfo* pOperator) { pOperator->status = OP_RES_TO_RETURN; closeSessionWindow(pInfo->streamAggSup.pResultRows, &pInfo->twAggSup, pUpdated, - getStateWinInfo); + getResWinForState); copyUpdateResult(pSeUpdated, pUpdated); taosHashCleanup(pSeUpdated); - finalizeUpdatedResult(pOperator->numOfExprs, pInfo->streamAggSup.pResultBuf, pUpdated, - pInfo->binfo.rowCellInfoOffset); + finalizeUpdatedResult(pOperator->exprSupp.numOfExprs, pInfo->streamAggSup.pResultBuf, pUpdated, + pSup->rowEntryInfoOffset); initMultiResInfoFromArrayList(&pInfo->groupResInfo, pUpdated); blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); doBuildDeleteDataBlock(pInfo->pSeDeleted, pInfo->pDelRes, &pInfo->pDelIterator); @@ -3337,12 +3547,14 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys goto _error; } + SExprSupp* pSup = &pOperator->exprSupp; + int32_t numOfCols = 0; SExprInfo* pExprInfo = createExprInfo(pStateNode->window.pFuncs, NULL, &numOfCols); pInfo->stateCol = extractColumnFromColumnNode(pColNode); initResultSizeInfo(pOperator, 4096); - initResultRowInfo(&pInfo->binfo.resultRowInfo, 8); + initResultRowInfo(&pInfo->binfo.resultRowInfo); pInfo->twAggSup = (STimeWindowAggSupp){ .waterMark = pStateNode->window.watermark, .calTrigger = pStateNode->window.triggerType, @@ -3350,12 +3562,12 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys }; initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); - code = initBiasicInfo(&pInfo->binfo, pExprInfo, numOfCols, pResBlock); + code = initBasicInfoEx(&pInfo->binfo, pSup, pExprInfo, numOfCols, pResBlock); if (code != TSDB_CODE_SUCCESS) { goto _error; } - code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo", pInfo->binfo.pCtx, numOfCols); + code = initStateAggSupporter(&pInfo->streamAggSup, "StreamStateAggOperatorInfo", pSup->pCtx, numOfCols); if (code != TSDB_CODE_SUCCESS) { goto _error; } @@ -3365,7 +3577,7 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys goto _error; } - initDummyFunction(pInfo->pDummyCtx, pInfo->binfo.pCtx, numOfCols); + initDummyFunction(pInfo->pDummyCtx, pSup->pCtx, numOfCols); pInfo->primaryTsIndex = tsSlotId; pInfo->order = TSDB_ORDER_ASC; _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY); @@ -3379,8 +3591,8 @@ SOperatorInfo* createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhys pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE; pOperator->blocking = true; pOperator->status = OP_NOT_OPENED; - pOperator->numOfExprs = numOfCols; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.numOfExprs = numOfCols; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; pOperator->info = pInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doStreamStateAgg, NULL, NULL, @@ -3418,6 +3630,8 @@ static int32_t outputMergeIntervalResult(SOperatorInfo* pOperatorInfo, uint64_t SMergeIntervalAggOperatorInfo* miaInfo = pOperatorInfo->info; SIntervalAggOperatorInfo* iaInfo = &miaInfo->intervalAggOperatorInfo; SExecTaskInfo* pTaskInfo = pOperatorInfo->pTaskInfo; + + SExprSupp* pSup = &pOperatorInfo->exprSupp; bool ascScan = (iaInfo->order == TSDB_ORDER_ASC); SET_RES_WINDOW_KEY(iaInfo->aggSup.keyBuf, &wstartTs, TSDB_KEYSIZE, tableGroupId); @@ -3425,8 +3639,8 @@ static int32_t outputMergeIntervalResult(SOperatorInfo* pOperatorInfo, uint64_t GET_RES_WINDOW_KEY_LEN(TSDB_KEYSIZE)); ASSERT(p1 != NULL); - finalizeResultRowIntoResultDataBlock(iaInfo->aggSup.pResultBuf, p1, iaInfo->binfo.pCtx, pOperatorInfo->pExpr, - pOperatorInfo->numOfExprs, iaInfo->binfo.rowCellInfoOffset, pResultBlock, + finalizeResultRowIntoResultDataBlock(iaInfo->aggSup.pResultBuf, p1, pSup->pCtx, pSup->pExprInfo, + pSup->numOfExprs, pSup->rowEntryInfoOffset, pResultBlock, pTaskInfo); taosHashRemove(iaInfo->aggSup.pResultRowHashTable, iaInfo->aggSup.keyBuf, GET_RES_WINDOW_KEY_LEN(TSDB_KEYSIZE)); @@ -3439,9 +3653,10 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo* SIntervalAggOperatorInfo* iaInfo = &miaInfo->intervalAggOperatorInfo; SExecTaskInfo* pTaskInfo = pOperatorInfo->pTaskInfo; + SExprSupp* pSup = &pOperatorInfo->exprSupp; int32_t startPos = 0; - int32_t numOfOutput = pOperatorInfo->numOfExprs; + int32_t numOfOutput = pSup->numOfExprs; int64_t* tsCols = extractTsCol(pBlock, iaInfo); uint64_t tableGroupId = pBlock->info.groupId; TSKEY blockStartTs = getStartTsKey(&pBlock->info.window, tsCols); @@ -3453,8 +3668,8 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo* //TODO: remove the hash table usage (groupid + winkey => result row position) int32_t ret = - setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, iaInfo->binfo.pCtx, - numOfOutput, iaInfo->binfo.rowCellInfoOffset, &iaInfo->aggSup, pTaskInfo); + setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &iaInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } @@ -3471,7 +3686,7 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo* continue; } else { updateTimeWindowInfo(&iaInfo->twAggSup.timeWindowData, &currWin, true); - doApplyFunctions(pTaskInfo, iaInfo->binfo.pCtx, &currWin, &iaInfo->twAggSup.timeWindowData, startPos, + doApplyFunctions(pTaskInfo, pSup->pCtx, &currWin, &iaInfo->twAggSup.timeWindowData, startPos, currPos - startPos, tsCols, pBlock->info.rows, numOfOutput, iaInfo->order); outputMergeIntervalResult(pOperatorInfo, tableGroupId, pResultBlock, currTs); @@ -3480,15 +3695,15 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo* currWin.skey = currTs; currWin.ekey = taosTimeAdd(currWin.skey, iaInfo->interval.interval, iaInfo->interval.intervalUnit, iaInfo->interval.precision) - 1; startPos = currPos; - ret = setTimeWindowOutputBuf(pResultRowInfo, &currWin, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, iaInfo->binfo.pCtx, - numOfOutput, iaInfo->binfo.rowCellInfoOffset, &iaInfo->aggSup, pTaskInfo); + ret = setTimeWindowOutputBuf(pResultRowInfo, &currWin, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pSup->pCtx, + numOfOutput, pSup->rowEntryInfoOffset, &iaInfo->aggSup, pTaskInfo); if (ret != TSDB_CODE_SUCCESS || pResult == NULL) { longjmp(pTaskInfo->env, TSDB_CODE_QRY_OUT_OF_MEMORY); } } } updateTimeWindowInfo(&iaInfo->twAggSup.timeWindowData, &currWin, true); - doApplyFunctions(pTaskInfo, iaInfo->binfo.pCtx, &currWin, &iaInfo->twAggSup.timeWindowData, startPos, + doApplyFunctions(pTaskInfo, pSup->pCtx, &currWin, &iaInfo->twAggSup.timeWindowData, startPos, currPos - startPos, tsCols, pBlock->info.rows, numOfOutput, iaInfo->order); outputMergeIntervalResult(pOperatorInfo, tableGroupId, pResultBlock, currTs); @@ -3503,6 +3718,7 @@ static SSDataBlock* doMergeIntervalAgg(SOperatorInfo* pOperator) { return NULL; } + SExprSupp* pSup = &pOperator->exprSupp; SSDataBlock* pRes = iaInfo->binfo.pRes; blockDataCleanup(pRes); blockDataEnsureCapacity(pRes, pOperator->resultInfo.capacity); @@ -3533,7 +3749,7 @@ static SSDataBlock* doMergeIntervalAgg(SOperatorInfo* pOperator) { } getTableScanInfo(pOperator, &iaInfo->order, &scanFlag); - setInputDataBlock(pOperator, iaInfo->binfo.pCtx, pBlock, iaInfo->order, scanFlag, true); + setInputDataBlock(pOperator, pSup->pCtx, pBlock, iaInfo->order, scanFlag, true); doMergeIntervalAggImpl(pOperator, &iaInfo->binfo.resultRowInfo, pBlock, scanFlag, pRes); if (pRes->info.rows >= pOperator->resultInfo.threshold) { @@ -3563,42 +3779,40 @@ SOperatorInfo* createMergeIntervalOperatorInfo(SOperatorInfo* downstream, SExprI } SIntervalAggOperatorInfo* iaInfo = &miaInfo->intervalAggOperatorInfo; + SExprSupp* pSup = &pOperator->exprSupp; - iaInfo->win = pTaskInfo->window; - iaInfo->order = TSDB_ORDER_ASC; - iaInfo->interval = *pInterval; - + iaInfo->win = pTaskInfo->window; + iaInfo->order = TSDB_ORDER_ASC; + iaInfo->interval = *pInterval; iaInfo->execModel = pTaskInfo->execModel; - iaInfo->primaryTsIndex = primaryTsSlotId; size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES; initResultSizeInfo(pOperator, 4096); - int32_t code = - initAggInfo(&iaInfo->binfo, &iaInfo->aggSup, pExprInfo, numOfCols, pResBlock, keyBufSize, pTaskInfo->id.str); + int32_t code = initAggInfo(&pOperator->exprSupp, &iaInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str); + initBasicInfo(&iaInfo->binfo, pResBlock); initExecTimeWindowInfo(&iaInfo->twAggSup.timeWindowData, &iaInfo->win); - iaInfo->timeWindowInterpo = timeWindowinterpNeeded(iaInfo->binfo.pCtx, numOfCols, iaInfo); + iaInfo->timeWindowInterpo = timeWindowinterpNeeded(pSup->pCtx, numOfCols, iaInfo); if (iaInfo->timeWindowInterpo) { iaInfo->binfo.resultRowInfo.openWindow = tdListNew(sizeof(SResultRowPosition)); } - // iaInfo->pTableQueryInfo = initTableQueryInfo(pTableGroupInfo); - if (code != TSDB_CODE_SUCCESS /* || iaInfo->pTableQueryInfo == NULL*/) { + if (code != TSDB_CODE_SUCCESS) { goto _error; } - initResultRowInfo(&iaInfo->binfo.resultRowInfo, (int32_t)1); + initResultRowInfo(&iaInfo->binfo.resultRowInfo); pOperator->name = "TimeMergeIntervalAggOperator"; pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_MERGE_INTERVAL; pOperator->blocking = false; pOperator->status = OP_NOT_OPENED; - pOperator->pExpr = pExprInfo; + pOperator->exprSupp.pExprInfo = pExprInfo; pOperator->pTaskInfo = pTaskInfo; - pOperator->numOfExprs = numOfCols; + pOperator->exprSupp.numOfExprs = numOfCols; pOperator->info = miaInfo; pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doMergeIntervalAgg, NULL, NULL, diff --git a/source/libs/executor/src/tsort.c b/source/libs/executor/src/tsort.c index 1502387360cb5a1722949e243e3bb054ef15d61d..f21cad2dd62156d3acb87fdc811c747bd6b8ddad 100644 --- a/source/libs/executor/src/tsort.c +++ b/source/libs/executor/src/tsort.c @@ -71,7 +71,7 @@ SSDataBlock* tsortGetSortedDataBlock(const SSortHandle* pSortHandle) { * @param type * @return */ -SSortHandle* tsortCreateSortHandle(SArray* pSortInfo, SArray* pIndexMap, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr) { +SSortHandle* tsortCreateSortHandle(SArray* pSortInfo, int32_t type, int32_t pageSize, int32_t numOfPages, SSDataBlock* pBlock, const char* idstr) { SSortHandle* pSortHandle = taosMemoryCalloc(1, sizeof(SSortHandle)); pSortHandle->type = type; @@ -195,6 +195,11 @@ static int32_t doAddToBuf(SSDataBlock* pDataBlock, SSortHandle* pHandle) { return doAddNewExternalMemSource(pHandle->pBuf, pHandle->pOrderedSource, pBlock, &pHandle->sourceId); } +static void setCurrentSourceIsDone(SSortSource* pSource, SSortHandle* pHandle) { + pSource->src.rowIndex = -1; + ++pHandle->numOfCompletedSources; +} + static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int32_t startIndex, int32_t endIndex, SSortHandle* pHandle) { cmpParam->pSources = taosArrayGet(pSources, startIndex); cmpParam->numOfSources = (endIndex - startIndex + 1); @@ -205,8 +210,10 @@ static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int for (int32_t i = 0; i < cmpParam->numOfSources; ++i) { SSortSource* pSource = cmpParam->pSources[i]; + // set current source is done if (taosArrayGetSize(pSource->pageIdList) == 0) { - return TSDB_CODE_SUCCESS; + setCurrentSourceIsDone(pSource, pHandle); + continue; } SPageInfo* pPgInfo = *(SPageInfo**)taosArrayGet(pSource->pageIdList, pSource->pageIndex); @@ -233,10 +240,9 @@ static int32_t sortComparInit(SMsortComparParam* cmpParam, SArray* pSources, int SSortSource* pSource = cmpParam->pSources[i]; pSource->src.pBlock = pHandle->fetchfp(pSource->param); - // set current source id done + // set current source is done if (pSource->src.pBlock == NULL) { - pSource->src.rowIndex = -1; - ++pHandle->numOfCompletedSources; + setCurrentSourceIsDone(pSource, pHandle); } } } diff --git a/source/libs/executor/test/sortTests.cpp b/source/libs/executor/test/sortTests.cpp index c037fae75f258411c3cecc32b4d3032bc48d332f..66ed078bbe305d65906b8ab00ffbb47004cd311f 100644 --- a/source/libs/executor/test/sortTests.cpp +++ b/source/libs/executor/test/sortTests.cpp @@ -209,7 +209,7 @@ TEST(testCase, inMem_sort_Test) { SArray* orderInfo = taosArrayInit(1, sizeof(SBlockOrderInfo)); taosArrayPush(orderInfo, &oi); - SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_SINGLESOURCE_SORT, 1024, 5, NULL, "test_abc"); + SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_SINGLESOURCE_SORT, 1024, 5, NULL, "test_abc"); tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL); _info* pInfo = (_info*) taosMemoryCalloc(1, sizeof(_info)); @@ -298,7 +298,7 @@ TEST(testCase, external_mem_sort_Test) { SArray* orderInfo = taosArrayInit(1, sizeof(SBlockOrderInfo)); taosArrayPush(orderInfo, &oi); - SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_SINGLESOURCE_SORT, 128, 3, NULL, "test_abc"); + SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_SINGLESOURCE_SORT, 128, 3, NULL, "test_abc"); tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL); SSortSource* ps = static_cast(taosMemoryCalloc(1, sizeof(SSortSource))); @@ -365,7 +365,7 @@ TEST(testCase, ordered_merge_sort_Test) { taosArrayPush(pBlock->pDataBlock, &colInfo); } - SSortHandle* phandle = tsortCreateSortHandle(orderInfo, NULL, SORT_MULTISOURCE_MERGE, 1024, 5, pBlock,"test_abc"); + SSortHandle* phandle = tsortCreateSortHandle(orderInfo, SORT_MULTISOURCE_MERGE, 1024, 5, pBlock,"test_abc"); tsortSetFetchRawDataFp(phandle, getSingleColDummyBlock, NULL, NULL); tsortSetComparFp(phandle, docomp); diff --git a/source/libs/function/inc/builtins.h b/source/libs/function/inc/builtins.h index bc91875006b0c45162f52505084c9971b17e5429..256500ff8ce00e16fe0356da8fa1d7ce279e34ab 100644 --- a/source/libs/function/inc/builtins.h +++ b/source/libs/function/inc/builtins.h @@ -24,22 +24,24 @@ extern "C" { typedef int32_t (*FTranslateFunc)(SFunctionNode* pFunc, char* pErrBuf, int32_t len); typedef EFuncDataRequired (*FFuncDataRequired)(SFunctionNode* pFunc, STimeWindow* pTimeWindow); +typedef int32_t (*FCreateMergeFuncParameters)(SNodeList* pRawParameters, SNode* pPartialRes, SNodeList** pParameters); typedef struct SBuiltinFuncDefinition { - const char* name; - EFunctionType type; - uint64_t classification; - FTranslateFunc translateFunc; - FFuncDataRequired dataRequiredFunc; - FExecGetEnv getEnvFunc; - FExecInit initFunc; - FExecProcess processFunc; - FScalarExecProcess sprocessFunc; - FExecFinalize finalizeFunc; - FExecProcess invertFunc; - FExecCombine combineFunc; - const char* pPartialFunc; - const char* pMergeFunc; + const char* name; + EFunctionType type; + uint64_t classification; + FTranslateFunc translateFunc; + FFuncDataRequired dataRequiredFunc; + FExecGetEnv getEnvFunc; + FExecInit initFunc; + FExecProcess processFunc; + FScalarExecProcess sprocessFunc; + FExecFinalize finalizeFunc; + FExecProcess invertFunc; + FExecCombine combineFunc; + const char* pPartialFunc; + const char* pMergeFunc; + FCreateMergeFuncParameters createMergeParaFuc; } SBuiltinFuncDefinition; extern const SBuiltinFuncDefinition funcMgtBuiltins[]; diff --git a/source/libs/function/inc/builtinsimpl.h b/source/libs/function/inc/builtinsimpl.h index c243c1c175ee0e3a909af3db4252127bf9aa404d..e691d562c6b4426015a7a7d10906d6c4b307cb5f 100644 --- a/source/libs/function/inc/builtinsimpl.h +++ b/source/libs/function/inc/builtinsimpl.h @@ -192,6 +192,8 @@ int32_t twaFunction(SqlFunctionCtx *pCtx); int32_t twaFinalize(struct SqlFunctionCtx *pCtx, SSDataBlock* pBlock); bool getSelectivityFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv); + +bool blockDistSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo); int32_t blockDistFunction(SqlFunctionCtx *pCtx); int32_t blockDistFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); diff --git a/source/libs/function/inc/functionMgtInt.h b/source/libs/function/inc/functionMgtInt.h index 1443c26820fe539f12e8692a843150061f587376..44e1e6f7b10285594c6a497365e42862e25e3d97 100644 --- a/source/libs/function/inc/functionMgtInt.h +++ b/source/libs/function/inc/functionMgtInt.h @@ -44,13 +44,13 @@ extern "C" { #define FUNC_MGT_FORBID_FILL_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(15) #define FUNC_MGT_INTERVAL_INTERPO_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(16) #define FUNC_MGT_FORBID_STREAM_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(17) +#define FUNC_MGT_FORBID_WINDOW_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(18) +#define FUNC_MGT_FORBID_GROUP_BY_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(19) #define FUNC_MGT_TEST_MASK(val, mask) (((val) & (mask)) != 0) #define FUNC_UDF_ID_START 5000 -extern const int funcMgtUdfNum; - #ifdef __cplusplus } #endif diff --git a/source/libs/function/src/builtins.c b/source/libs/function/src/builtins.c index b0741908a6ae7e11271a84a784cfe711f624bb7c..cfad00f45898a60dc196edae522c67246d5afb69 100644 --- a/source/libs/function/src/builtins.c +++ b/source/libs/function/src/builtins.c @@ -419,65 +419,14 @@ static int32_t translateTopBot(SFunctionNode* pFunc, char* pErrBuf, int32_t len) return TSDB_CODE_SUCCESS; } -static int32_t translateTopBotImpl(SFunctionNode* pFunc, char* pErrBuf, int32_t len, bool isPartial) { - int32_t numOfParams = LIST_LENGTH(pFunc->pParameterList); - - if (isPartial) { - if (2 != numOfParams) { - return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); - } - - uint8_t para1Type = ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type; - uint8_t para2Type = ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 1))->resType.type; - if (!IS_NUMERIC_TYPE(para1Type) || !IS_INTEGER_TYPE(para2Type)) { - return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); - } - - // param1 - SNode* pParamNode1 = nodesListGetNode(pFunc->pParameterList, 1); - if (nodeType(pParamNode1) != QUERY_NODE_VALUE) { - return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); - } - - SValueNode* pValue = (SValueNode*)pParamNode1; - if (pValue->node.resType.type != TSDB_DATA_TYPE_BIGINT) { - return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); - } - - if (pValue->datum.i < 1 || pValue->datum.i > 100) { - return invaildFuncParaValueErrMsg(pErrBuf, len, pFunc->functionName); - } - - pValue->notReserved = true; - - // set result type - pFunc->node.resType = - (SDataType){.bytes = getTopBotInfoSize(pValue->datum.i) + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY}; - } else { - if (1 != numOfParams) { - return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); - } - - uint8_t para1Type = ((SExprNode*)nodesListGetNode(pFunc->pParameterList, 0))->resType.type; - if (TSDB_DATA_TYPE_BINARY != para1Type) { - return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); - } - - // Do nothing. We can only access output of partial functions as input, - // so original input type cannot be obtained, resType will be set same - // as original function input type after merge function created. +int32_t topBotCreateMergePara(SNodeList* pRawParameters, SNode* pPartialRes, SNodeList** pParameters) { + int32_t code = nodesListMakeAppend(pParameters, pPartialRes); + if (TSDB_CODE_SUCCESS == code) { + code = nodesListStrictAppend(*pParameters, nodesCloneNode(nodesListGetNode(pRawParameters, 1))); } return TSDB_CODE_SUCCESS; } -static int32_t translateTopBotPartial(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { - return translateTopBotImpl(pFunc, pErrBuf, len, true); -} - -static int32_t translateTopBotMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { - return translateTopBotImpl(pFunc, pErrBuf, len, false); -} - static int32_t translateSpread(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { if (1 != LIST_LENGTH(pFunc->pParameterList)) { return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); @@ -1017,7 +966,7 @@ static int32_t translateDerivative(SFunctionNode* pFunc, char* pErrBuf, int32_t return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); } - SNode* pParamNode2 = nodesListGetNode(pFunc->pParameterList, 2); + SNode* pParamNode2 = nodesListGetNode(pFunc->pParameterList, 2); SValueNode* pValue2 = (SValueNode*)pParamNode2; pValue2->notReserved = true; @@ -1086,8 +1035,8 @@ static int32_t translateUnique(SFunctionNode* pFunc, char* pErrBuf, int32_t len) } SNode* pPara = nodesListGetNode(pFunc->pParameterList, 0); - if (QUERY_NODE_COLUMN != nodeType(pPara)) { - return buildFuncErrMsg(pErrBuf, len, TSDB_CODE_FUNC_FUNTION_ERROR, "The parameters of UNIQUE can only be columns"); + if (!nodesExprHasColumn(pPara)) { + return buildFuncErrMsg(pErrBuf, len, TSDB_CODE_FUNC_FUNTION_ERROR, "The parameters of UNIQUE must contain columns"); } pFunc->node.resType = ((SExprNode*)pPara)->resType; @@ -1508,6 +1457,11 @@ static int32_t translateBlockDistFunc(SFunctionNode* pFunc, char* pErrBuf, int32 return TSDB_CODE_SUCCESS; } +static int32_t translateBlockDistInfoFunc(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { + pFunc->node.resType = (SDataType){.bytes = 128, .type = TSDB_DATA_TYPE_VARCHAR}; + return TSDB_CODE_SUCCESS; +} + static bool getBlockDistFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) { pEnv->calcMemSize = sizeof(STableBlockDistInfo); return true; @@ -1722,30 +1676,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .processFunc = topFunction, .finalizeFunc = topBotFinalize, .combineFunc = topCombine, - .pPartialFunc = "_top_partial", - .pMergeFunc = "_top_merge" - }, - { - .name = "_top_partial", - .type = FUNCTION_TYPE_TOP_PARTIAL, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_INDEFINITE_ROWS_FUNC, - .translateFunc = translateTopBotPartial, - .getEnvFunc = getTopBotFuncEnv, - .initFunc = topBotFunctionSetup, - .processFunc = topFunction, - .finalizeFunc = topBotPartialFinalize, - .combineFunc = topCombine, - }, - { - .name = "_top_merge", - .type = FUNCTION_TYPE_TOP_MERGE, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_INDEFINITE_ROWS_FUNC, - .translateFunc = translateTopBotMerge, - .getEnvFunc = getTopBotMergeFuncEnv, - .initFunc = functionSetup, - .processFunc = topFunctionMerge, - .finalizeFunc = topBotMergeFinalize, - .combineFunc = topCombine, + .pPartialFunc = "top", + .pMergeFunc = "top", + .createMergeParaFuc = topBotCreateMergePara }, { .name = "bottom", @@ -1757,30 +1690,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .processFunc = bottomFunction, .finalizeFunc = topBotFinalize, .combineFunc = bottomCombine, - .pPartialFunc = "_bottom_partial", - .pMergeFunc = "_bottom_merge" - }, - { - .name = "_bottom_partial", - .type = FUNCTION_TYPE_BOTTOM_PARTIAL, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_INDEFINITE_ROWS_FUNC, - .translateFunc = translateTopBotPartial, - .getEnvFunc = getTopBotFuncEnv, - .initFunc = topBotFunctionSetup, - .processFunc = bottomFunction, - .finalizeFunc = topBotPartialFinalize, - .combineFunc = bottomCombine, - }, - { - .name = "_bottom_merge", - .type = FUNCTION_TYPE_BOTTOM_MERGE, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_INDEFINITE_ROWS_FUNC, - .translateFunc = translateTopBotMerge, - .getEnvFunc = getTopBotMergeFuncEnv, - .initFunc = functionSetup, - .processFunc = bottomFunctionMerge, - .finalizeFunc = topBotMergeFinalize, - .combineFunc = bottomCombine, + .pPartialFunc = "bottom", + .pMergeFunc = "bottom", + .createMergeParaFuc = topBotCreateMergePara }, { .name = "spread", @@ -1867,7 +1779,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "interp", .type = FUNCTION_TYPE_INTERP, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_INTERVAL_INTERPO_FUNC, + .classification = FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_INTERVAL_INTERPO_FUNC, .translateFunc = translateFirstLast, .getEnvFunc = getSelectivityFuncEnv, .initFunc = functionSetup, @@ -2114,7 +2026,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "tail", .type = FUNCTION_TYPE_TAIL, - .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_FORBID_STREAM_FUNC, + .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_FORBID_STREAM_FUNC | FUNC_MGT_FORBID_WINDOW_FUNC | FUNC_MGT_FORBID_GROUP_BY_FUNC, .translateFunc = translateTail, .getEnvFunc = getTailFuncEnv, .initFunc = tailFunctionSetup, @@ -2124,7 +2036,8 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "unique", .type = FUNCTION_TYPE_UNIQUE, - .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_FORBID_STREAM_FUNC, + .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | + FUNC_MGT_FORBID_STREAM_FUNC | FUNC_MGT_FORBID_WINDOW_FUNC | FUNC_MGT_FORBID_GROUP_BY_FUNC, .translateFunc = translateUnique, .getEnvFunc = getUniqueFuncEnv, .initFunc = uniqueFunctionSetup, @@ -2509,7 +2422,9 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .getEnvFunc = getSelectivityFuncEnv, // todo remove this function later. .initFunc = functionSetup, .processFunc = NULL, - .finalizeFunc = NULL + .finalizeFunc = NULL, + .pPartialFunc = "_select_value", + .pMergeFunc = "_select_value" }, { .name = "_block_dist", @@ -2517,8 +2432,15 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .classification = FUNC_MGT_AGG_FUNC, .translateFunc = translateBlockDistFunc, .getEnvFunc = getBlockDistFuncEnv, + .initFunc = blockDistSetup, .processFunc = blockDistFunction, .finalizeFunc = blockDistFinalize + }, + { + .name = "_block_dist_info", + .type = FUNCTION_TYPE_BLOCK_DIST_INFO, + .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_SCAN_PC_FUNC, + .translateFunc = translateBlockDistInfoFunc, } }; // clang-format on diff --git a/source/libs/function/src/builtinsimpl.c b/source/libs/function/src/builtinsimpl.c index 55900c7c7ecbd5cdfd3449636547a040ea7d6d7d..076ae6146021b33179eb2271086c4b5a613176fd 100644 --- a/source/libs/function/src/builtinsimpl.c +++ b/source/libs/function/src/builtinsimpl.c @@ -67,8 +67,7 @@ typedef struct STopBotResItem { typedef struct STopBotRes { int32_t maxSize; - int16_t type; // store the original input type, used in merge function - int32_t numOfItems; + int16_t type; STopBotResItem* pItems; } STopBotRes; @@ -794,8 +793,8 @@ int32_t avgFunctionMerge(SqlFunctionCtx* pCtx) { SAvgRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); + int32_t start = pInput->startRowIndex; + char* data = colDataGetData(pCol, start); SAvgRes* pInputInfo = (SAvgRes*)varDataVal(data); avgTransferInfo(pInputInfo, pInfo); @@ -908,9 +907,9 @@ int32_t avgFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t avgPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - SAvgRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t resultBytes = getAvgInfoSize(); - char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); + SAvgRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + int32_t resultBytes = getAvgInfoSize(); + char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); memcpy(varDataVal(res), pInfo, resultBytes); varDataSetLen(res, resultBytes); @@ -1418,7 +1417,7 @@ int32_t minmaxFunctionFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { void setNullSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32_t rowIndex) { for (int32_t j = 0; j < pCtx->subsidiaries.num; ++j) { SqlFunctionCtx* pc = pCtx->subsidiaries.pCtx[j]; - int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; + int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; SColumnInfoData* pDstCol = taosArrayGet(pBlock->pDataBlock, dstSlotId); colDataAppendNULL(pDstCol, rowIndex); @@ -1481,11 +1480,13 @@ int32_t minMaxCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx, int3 if (pSBuf->assign && ((((*(double*)&pDBuf->v) < (*(double*)&pSBuf->v)) ^ isMinFunc) || !pDBuf->assign)) { *(double*)&pDBuf->v = *(double*)&pSBuf->v; replaceTupleData(&pDBuf->tuplePos, &pSBuf->tuplePos); + pDBuf->assign = true; } } else { if (pSBuf->assign && (((pDBuf->v < pSBuf->v) ^ isMinFunc) || !pDBuf->assign)) { pDBuf->v = pSBuf->v; replaceTupleData(&pDBuf->tuplePos, &pSBuf->tuplePos); + pDBuf->assign = true; } } pDResInfo->numOfRes = TMAX(pDResInfo->numOfRes, pSResInfo->numOfRes); @@ -1663,8 +1664,8 @@ int32_t stddevFunctionMerge(SqlFunctionCtx* pCtx) { SStddevRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); + int32_t start = pInput->startRowIndex; + char* data = colDataGetData(pCol, start); SStddevRes* pInputInfo = (SStddevRes*)varDataVal(data); stddevTransferInfo(pInputInfo, pInfo); @@ -1745,10 +1746,10 @@ int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { if (IS_INTEGER_TYPE(type)) { avg = pStddevRes->isum / ((double)pStddevRes->count); - pStddevRes->result = sqrt(pStddevRes->quadraticISum / ((double)pStddevRes->count) - avg * avg); + pStddevRes->result = sqrt(fabs(pStddevRes->quadraticISum / ((double)pStddevRes->count) - avg * avg)); } else { avg = pStddevRes->dsum / ((double)pStddevRes->count); - pStddevRes->result = sqrt(pStddevRes->quadraticDSum / ((double)pStddevRes->count) - avg * avg); + pStddevRes->result = sqrt(fabs(pStddevRes->quadraticDSum / ((double)pStddevRes->count) - avg * avg)); } return functionFinalize(pCtx, pBlock); @@ -1756,9 +1757,9 @@ int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t stddevPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - SStddevRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t resultBytes = getStddevInfoSize(); - char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); + SStddevRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + int32_t resultBytes = getStddevInfoSize(); + char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); memcpy(varDataVal(res), pInfo, resultBytes); varDataSetLen(res, resultBytes); @@ -2360,12 +2361,10 @@ int32_t apercentileCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) return TSDB_CODE_SUCCESS; } -int32_t getFirstLastInfoSize(int32_t resBytes) { - return sizeof(SFirstLastRes) + resBytes + sizeof(int64_t); -} +int32_t getFirstLastInfoSize(int32_t resBytes) { return sizeof(SFirstLastRes) + resBytes + sizeof(int64_t); } bool getFirstLastFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { - SColumnNode* pNode = (SColumnNode *)nodesListGetNode(pFunc->pParameterList, 0); + SColumnNode* pNode = (SColumnNode*)nodesListGetNode(pFunc->pParameterList, 0); pEnv->calcMemSize = sizeof(SFirstLastRes) + pNode->node.resType.bytes + sizeof(int64_t); return true; } @@ -2390,14 +2389,14 @@ int32_t firstFunction(SqlFunctionCtx* pCtx) { int32_t numOfElems = 0; SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - SFirstLastRes *pInfo = GET_ROWCELL_INTERBUF(pResInfo); + SFirstLastRes* pInfo = GET_ROWCELL_INTERBUF(pResInfo); SInputColumnInfoData* pInput = &pCtx->input; SColumnInfoData* pInputCol = pInput->pData[0]; - int32_t type = pInputCol->info.type; + int32_t type = pInputCol->info.type; int32_t bytes = pInputCol->info.bytes; - pInfo->bytes = bytes; + pInfo->bytes = bytes; // All null data column, return directly. if (pInput->colDataAggIsSet && (pInput->pColumnDataAgg[0]->numOfNull == pInput->totalRows)) { @@ -2488,14 +2487,14 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { int32_t numOfElems = 0; SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - SFirstLastRes *pInfo = GET_ROWCELL_INTERBUF(pResInfo); + SFirstLastRes* pInfo = GET_ROWCELL_INTERBUF(pResInfo); SInputColumnInfoData* pInput = &pCtx->input; SColumnInfoData* pInputCol = pInput->pData[0]; - int32_t type = pInputCol->info.type; + int32_t type = pInputCol->info.type; int32_t bytes = pInputCol->info.bytes; - pInfo->bytes = bytes; + pInfo->bytes = bytes; // All null data column, return directly. if (pInput->colDataAggIsSet && (pInput->pColumnDataAgg[0]->numOfNull == pInput->totalRows)) { @@ -2563,12 +2562,9 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { } static void firstLastTransferInfo(SFirstLastRes* pInput, SFirstLastRes* pOutput, bool isFirst) { - if (!pInput->hasResult) { - return; - } pOutput->bytes = pInput->bytes; - TSKEY *tsIn = (TSKEY*)(pInput->buf + pInput->bytes); - TSKEY *tsOut = (TSKEY*)(pOutput->buf + pInput->bytes); + TSKEY* tsIn = (TSKEY*)(pInput->buf + pInput->bytes); + TSKEY* tsOut = (TSKEY*)(pOutput->buf + pInput->bytes); if (pOutput->hasResult) { if (isFirst) { if (*tsIn > *tsOut) { @@ -2586,31 +2582,29 @@ static void firstLastTransferInfo(SFirstLastRes* pInput, SFirstLastRes* pOutput, return; } -static int32_t firstLastFunctionMergeImpl(SqlFunctionCtx *pCtx, bool isFirstQuery) { +static int32_t firstLastFunctionMergeImpl(SqlFunctionCtx* pCtx, bool isFirstQuery) { SInputColumnInfoData* pInput = &pCtx->input; - SColumnInfoData* pCol = pInput->pData[0]; + SColumnInfoData* pCol = pInput->pData[0]; ASSERT(pCol->info.type == TSDB_DATA_TYPE_BINARY); SFirstLastRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); - SFirstLastRes* pInputInfo = (SFirstLastRes *)varDataVal(data); + int32_t start = pInput->startRowIndex; + char* data = colDataGetData(pCol, start); + SFirstLastRes* pInputInfo = (SFirstLastRes*)varDataVal(data); firstLastTransferInfo(pInputInfo, pInfo, isFirstQuery); - SET_VAL(GET_RES_INFO(pCtx), 1, 1); + int32_t numOfElems = pInputInfo->hasResult ? 1 : 0; + + SET_VAL(GET_RES_INFO(pCtx), numOfElems, 1); return TSDB_CODE_SUCCESS; } -int32_t firstFunctionMerge(SqlFunctionCtx *pCtx) { - return firstLastFunctionMergeImpl(pCtx, true); -} +int32_t firstFunctionMerge(SqlFunctionCtx* pCtx) { return firstLastFunctionMergeImpl(pCtx, true); } -int32_t lastFunctionMerge(SqlFunctionCtx *pCtx) { - return firstLastFunctionMergeImpl(pCtx, false); -} +int32_t lastFunctionMerge(SqlFunctionCtx* pCtx) { return firstLastFunctionMergeImpl(pCtx, false); } int32_t firstLastFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t slotId = pCtx->pExpr->base.resSchema.slotId; @@ -2627,9 +2621,10 @@ int32_t firstLastFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t firstLastPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); - SFirstLastRes* pRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t resultBytes = getFirstLastInfoSize(pRes->bytes); - char *res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); + SFirstLastRes* pRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + + int32_t resultBytes = getFirstLastInfoSize(pRes->bytes); + char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); memcpy(varDataVal(res), pRes, resultBytes); varDataSetLen(res, resultBytes); @@ -2876,12 +2871,6 @@ bool getTopBotFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { return true; } -bool getTopBotMergeFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { - // intermediate result is binary and length contains VAR header size - pEnv->calcMemSize = pFunc->node.resType.bytes; - return true; -} - bool topBotFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResInfo) { if (!functionSetup(pCtx, pResInfo)) { return false; @@ -2956,50 +2945,6 @@ int32_t bottomFunction(SqlFunctionCtx* pCtx) { return TSDB_CODE_SUCCESS; } -static void topBotTransferInfo(SqlFunctionCtx* pCtx, STopBotRes* pInput, bool isTopQuery) { - for (int32_t i = 0; i < pInput->numOfItems; i++) { - addResult(pCtx, &pInput->pItems[i], pInput->type, isTopQuery); - } -} - -int32_t topFunctionMerge(SqlFunctionCtx* pCtx) { - SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); - SInputColumnInfoData* pInput = &pCtx->input; - SColumnInfoData* pCol = pInput->pData[0]; - ASSERT(pCol->info.type == TSDB_DATA_TYPE_BINARY); - - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); - STopBotRes* pInputInfo = (STopBotRes*)varDataVal(data); - STopBotRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - - pInfo->maxSize = pInputInfo->maxSize; - pInfo->type = pInputInfo->type; - topBotTransferInfo(pCtx, pInputInfo, true); - SET_VAL(GET_RES_INFO(pCtx), pEntryInfo->numOfRes, pEntryInfo->numOfRes); - - return TSDB_CODE_SUCCESS; -} - -int32_t bottomFunctionMerge(SqlFunctionCtx* pCtx) { - SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); - SInputColumnInfoData* pInput = &pCtx->input; - SColumnInfoData* pCol = pInput->pData[0]; - ASSERT(pCol->info.type == TSDB_DATA_TYPE_BINARY); - - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); - STopBotRes* pInputInfo = (STopBotRes*)varDataVal(data); - STopBotRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - - pInfo->maxSize = pInputInfo->maxSize; - pInfo->type = pInputInfo->type; - topBotTransferInfo(pCtx, pInputInfo, false); - SET_VAL(GET_RES_INFO(pCtx), pEntryInfo->numOfRes, pEntryInfo->numOfRes); - - return TSDB_CODE_SUCCESS; -} - static int32_t topBotResComparFn(const void* p1, const void* p2, const void* param) { uint16_t type = *(uint16_t*)param; @@ -3048,8 +2993,6 @@ void doAddIntoResult(SqlFunctionCtx* pCtx, void* pData, int32_t rowIndex, SSData // allocate the buffer and keep the data of this row into the new allocated buffer pEntryInfo->numOfRes++; - // accumulate number of items for each vgroup, this info is needed for merge - pRes->numOfItems++; taosheapsort((void*)pItems, sizeof(STopBotResItem), pEntryInfo->numOfRes, (const void*)&type, topBotResComparFn, !isTopQuery); } else { // replace the minimum value in the result @@ -3148,7 +3091,7 @@ void copyTupleData(SqlFunctionCtx* pCtx, int32_t rowIndex, const SSDataBlock* pS releaseBufPage(pCtx->pBuf, pPage); } -int32_t topBotFinalizeImpl(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, bool isMerge) { +int32_t topBotFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); STopBotRes* pRes = GET_ROWCELL_INTERBUF(pEntryInfo); @@ -3168,39 +3111,13 @@ int32_t topBotFinalizeImpl(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, bool isMer colDataAppend(pCol, currentRow, (const char*)&pItem->v.i, false); } - if (!isMerge) { - setSelectivityValue(pCtx, pBlock, &pRes->pItems[i].tuplePos, currentRow); - } + setSelectivityValue(pCtx, pBlock, &pRes->pItems[i].tuplePos, currentRow); currentRow += 1; } return pEntryInfo->numOfRes; } -int32_t topBotFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { return topBotFinalizeImpl(pCtx, pBlock, false); } - -int32_t topBotMergeFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { - return topBotFinalizeImpl(pCtx, pBlock, true); -} - -int32_t topBotPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { - SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); - STopBotRes* pRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t resultBytes = getTopBotInfoSize(pRes->maxSize); - char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); - - memcpy(varDataVal(res), pRes, resultBytes); - varDataSetLen(res, resultBytes); - - int32_t slotId = pCtx->pExpr->base.resSchema.slotId; - SColumnInfoData* pCol = taosArrayGet(pBlock->pDataBlock, slotId); - - colDataAppend(pCol, pBlock->info.rows, res, false); - - taosMemoryFree(res); - return 1; -} - void addResult(SqlFunctionCtx* pCtx, STopBotResItem* pSourceItem, int16_t type, bool isTopQuery) { SResultRowEntryInfo* pEntryInfo = GET_RES_INFO(pCtx); STopBotRes* pRes = getTopBotOutputInfo(pCtx); @@ -3215,8 +3132,6 @@ void addResult(SqlFunctionCtx* pCtx, STopBotResItem* pSourceItem, int16_t type, pItem->tuplePos.pageId = -1; replaceTupleData(&pItem->tuplePos, &pSourceItem->tuplePos); pEntryInfo->numOfRes++; - // accumulate number of items for each vgroup, this info is needed for merge - pRes->numOfItems++; taosheapsort((void*)pItems, sizeof(STopBotResItem), pEntryInfo->numOfRes, (const void*)&type, topBotResComparFn, !isTopQuery); } else { // replace the minimum value in the result @@ -3369,8 +3284,8 @@ int32_t spreadFunctionMerge(SqlFunctionCtx* pCtx) { SSpreadInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t start = pInput->startRowIndex; - char* data = colDataGetData(pCol, start); + int32_t start = pInput->startRowIndex; + char* data = colDataGetData(pCol, start); SSpreadInfo* pInputInfo = (SSpreadInfo*)varDataVal(data); spreadTransferInfo(pInputInfo, pInfo); @@ -3390,9 +3305,9 @@ int32_t spreadFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t spreadPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - SSpreadInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); - int32_t resultBytes = getSpreadInfoSize(); - char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); + SSpreadInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + int32_t resultBytes = getSpreadInfoSize(); + char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); memcpy(varDataVal(res), pInfo, resultBytes); varDataSetLen(res, resultBytes); @@ -3484,8 +3399,8 @@ int32_t elapsedFunction(SqlFunctionCtx* pCtx) { SColumnInfoData* pCol = pInput->pData[0]; - int32_t start = pInput->startRowIndex; - TSKEY* ptsList = (int64_t*)colDataGetData(pCol, 0); + int32_t start = pInput->startRowIndex; + TSKEY* ptsList = (int64_t*)colDataGetData(pCol, 0); if (pCtx->order == TSDB_ORDER_DESC) { if (pCtx->start.key == INT64_MIN) { pInfo->max = @@ -5008,7 +4923,19 @@ int32_t twaFinalize(struct SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { return functionFinalize(pCtx, pBlock); } +bool blockDistSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResultInfo) { + if (!functionSetup(pCtx, pResultInfo)) { + return false; + } + + STableBlockDistInfo* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); + pInfo->minRows = INT32_MAX; + return true; +} + int32_t blockDistFunction(SqlFunctionCtx* pCtx) { + const int32_t BLOCK_DIST_RESULT_ROWS = 24; + SInputColumnInfoData* pInput = &pCtx->input; SColumnInfoData* pInputCol = pInput->pData[0]; @@ -5026,6 +4953,11 @@ int32_t blockDistFunction(SqlFunctionCtx* pCtx) { pDistInfo->totalRows += p1.totalRows; pDistInfo->numOfFiles += p1.numOfFiles; + pDistInfo->defMinRows = p1.defMinRows; + pDistInfo->defMaxRows = p1.defMaxRows; + pDistInfo->rowSize = p1.rowSize; + pDistInfo->numOfSmallBlocks = p1.numOfSmallBlocks; + if (pDistInfo->minRows > p1.minRows) { pDistInfo->minRows = p1.minRows; } @@ -5037,7 +4969,7 @@ int32_t blockDistFunction(SqlFunctionCtx* pCtx) { pDistInfo->blockRowsHisto[i] += p1.blockRowsHisto[i]; } - pResInfo->numOfRes = 1; + pResInfo->numOfRes = BLOCK_DIST_RESULT_ROWS; // default output rows return TSDB_CODE_SUCCESS; } @@ -5049,7 +4981,7 @@ int32_t tSerializeBlockDistInfo(void* buf, int32_t bufLen, const STableBlockDist if (tEncodeU32(&encoder, pInfo->rowSize) < 0) return -1; if (tEncodeU16(&encoder, pInfo->numOfFiles) < 0) return -1; - if (tEncodeU32(&encoder, pInfo->rowSize) < 0) return -1; + if (tEncodeU32(&encoder, pInfo->numOfBlocks) < 0) return -1; if (tEncodeU32(&encoder, pInfo->numOfTables) < 0) return -1; if (tEncodeU64(&encoder, pInfo->totalSize) < 0) return -1; @@ -5080,7 +5012,7 @@ int32_t tDeserializeBlockDistInfo(void* buf, int32_t bufLen, STableBlockDistInfo if (tDecodeU32(&decoder, &pInfo->rowSize) < 0) return -1; if (tDecodeU16(&decoder, &pInfo->numOfFiles) < 0) return -1; - if (tDecodeU32(&decoder, &pInfo->rowSize) < 0) return -1; + if (tDecodeU32(&decoder, &pInfo->numOfBlocks) < 0) return -1; if (tDecodeU32(&decoder, &pInfo->numOfTables) < 0) return -1; if (tDecodeU64(&decoder, &pInfo->totalSize) < 0) return -1; @@ -5102,32 +5034,29 @@ int32_t tDeserializeBlockDistInfo(void* buf, int32_t bufLen, STableBlockDistInfo int32_t blockDistFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); - char* pData = GET_ROWCELL_INTERBUF(pResInfo); + STableBlockDistInfo* pData = GET_ROWCELL_INTERBUF(pResInfo); SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, 0); int32_t row = 0; - - STableBlockDistInfo info = {0}; - tDeserializeBlockDistInfo(varDataVal(pData), varDataLen(pData), &info); - char st[256] = {0}; + double totalRawSize = pData->totalRows * pData->rowSize; int32_t len = - sprintf(st + VARSTR_HEADER_SIZE, "Blocks=[%d] Size=[%.3fKb] Average_Block_size=[%.3fKb] Compression_Ratio=[%.3f]", - info.numOfBlocks, info.totalSize / 1024.0, info.totalSize / (info.numOfBlocks * 1024.0), - info.totalSize / (info.totalRows * info.rowSize * 1.0)); + sprintf(st + VARSTR_HEADER_SIZE, "Total_Blocks=[%d] Total_Size=[%.2f Kb] Average_size=[%.2f Kb] Compression_Ratio=[%.2f %c]", + pData->numOfBlocks, pData->totalSize / 1024.0, ((double)pData->totalSize) / pData->numOfBlocks, + pData->totalSize * 100 / totalRawSize, '%'); varDataSetLen(st, len); colDataAppend(pColInfo, row++, st, false); - len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Rows=[%ld] MinRows=[%d] MaxRows=[%d] Averge_Rows=[%ld] Inmem_Rows=[%d]", - info.totalRows, info.minRows, info.maxRows, info.totalRows / info.numOfBlocks, info.numOfInmemRows); + len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Rows=[%"PRId64"] Inmem_Rows=[%d] MinRows=[%d] MaxRows=[%d] Average_Rows=[%"PRId64"]", + pData->totalRows, pData->numOfInmemRows, pData->minRows, pData->maxRows, pData->totalRows / pData->numOfBlocks); varDataSetLen(st, len); colDataAppend(pColInfo, row++, st, false); - len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Tables=[%d] Total_Files=[%d] Total_Vgroups=[%d]", info.numOfTables, - info.numOfFiles, 0); + len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Tables=[%d] Total_Files=[%d] Total_Vgroups=[%d]", pData->numOfTables, + pData->numOfFiles, 0); varDataSetLen(st, len); colDataAppend(pColInfo, row++, st, false); @@ -5139,48 +5068,64 @@ int32_t blockDistFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t maxVal = 0; int32_t minVal = INT32_MAX; - for (int32_t i = 0; i < sizeof(info.blockRowsHisto) / sizeof(info.blockRowsHisto[0]); ++i) { - if (maxVal < info.blockRowsHisto[i]) { - maxVal = info.blockRowsHisto[i]; + for (int32_t i = 0; i < tListLen(pData->blockRowsHisto); ++i) { + if (maxVal < pData->blockRowsHisto[i]) { + maxVal = pData->blockRowsHisto[i]; } - if (minVal > info.blockRowsHisto[i]) { - minVal = info.blockRowsHisto[i]; + if (minVal > pData->blockRowsHisto[i]) { + minVal = pData->blockRowsHisto[i]; } } int32_t delta = maxVal - minVal; int32_t step = delta / 50; + if (step == 0) { + step = 1; + } + + int32_t numOfBuckets = sizeof(pData->blockRowsHisto) / sizeof(pData->blockRowsHisto[0]); + int32_t bucketRange = (pData->maxRows - pData->minRows) / numOfBuckets; + + bool singleModel = false; + if (bucketRange == 0) { + singleModel = true; + step = 20; + bucketRange = (pData->defMaxRows - pData->defMinRows) / numOfBuckets; + } - int32_t numOfBuckets = sizeof(info.blockRowsHisto) / sizeof(info.blockRowsHisto[0]); - int32_t bucketRange = (info.maxRows - info.minRows) / numOfBuckets; + for (int32_t i = 0; i < tListLen(pData->blockRowsHisto); ++i) { + len = sprintf(st + VARSTR_HEADER_SIZE, "%04d |", pData->defMinRows + bucketRange * (i + 1)); - for (int32_t i = 0; i < 20; ++i) { - len += sprintf(st + VARSTR_HEADER_SIZE, "%04d |", info.defMinRows + bucketRange * (i + 1)); + int32_t num = 0; + if (singleModel && pData->blockRowsHisto[i] > 0) { + num = 20; + } else { + num = (pData->blockRowsHisto[i] + step - 1) / step; + } - int32_t num = (info.blockRowsHisto[i] + step - 1) / step; for (int32_t j = 0; j < num; ++j) { int32_t x = sprintf(st + VARSTR_HEADER_SIZE + len, "%c", '|'); len += x; } - double v = info.blockRowsHisto[i] * 100.0 / info.numOfBlocks; - len += sprintf(st + VARSTR_HEADER_SIZE + len, " %d (%.3f%c)", info.blockRowsHisto[i], v, '%'); + double v = pData->blockRowsHisto[i] * 100.0 / pData->numOfBlocks; + len += sprintf(st + VARSTR_HEADER_SIZE + len, " %d (%.2f%c)", pData->blockRowsHisto[i], v, '%'); printf("%s\n", st); varDataSetLen(st, len); colDataAppend(pColInfo, row++, st, false); } - return row; + return TSDB_CODE_SUCCESS; } typedef struct SDerivInfo { - double prevValue; // previous value - TSKEY prevTs; // previous timestamp - bool ignoreNegative;// ignore the negative value - int64_t tsWindow; // time window for derivative - bool valueSet; // the value has been set already + double prevValue; // previous value + TSKEY prevTs; // previous timestamp + bool ignoreNegative; // ignore the negative value + int64_t tsWindow; // time window for derivative + bool valueSet; // the value has been set already } SDerivInfo; bool getDerivativeFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv) { @@ -5188,7 +5133,7 @@ bool getDerivativeFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv) { return true; } -bool derivativeFuncSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResInfo) { +bool derivativeFuncSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResInfo) { if (!functionSetup(pCtx, pResInfo)) { return false; // not initialized since it has been initialized } @@ -5196,25 +5141,25 @@ bool derivativeFuncSetup(SqlFunctionCtx *pCtx, SResultRowEntryInfo* pResInfo) { SDerivInfo* pDerivInfo = GET_ROWCELL_INTERBUF(pResInfo); pDerivInfo->ignoreNegative = pCtx->param[2].param.i; - pDerivInfo->prevTs = -1; + pDerivInfo->prevTs = -1; pDerivInfo->tsWindow = pCtx->param[1].param.i; pDerivInfo->valueSet = false; return true; } -int32_t derivativeFunction(SqlFunctionCtx *pCtx) { - SResultRowEntryInfo *pResInfo = GET_RES_INFO(pCtx); - SDerivInfo* pDerivInfo = GET_ROWCELL_INTERBUF(pResInfo); +int32_t derivativeFunction(SqlFunctionCtx* pCtx) { + SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); + SDerivInfo* pDerivInfo = GET_ROWCELL_INTERBUF(pResInfo); SInputColumnInfoData* pInput = &pCtx->input; - SColumnInfoData* pInputCol = pInput->pData[0]; + SColumnInfoData* pInputCol = pInput->pData[0]; - int32_t numOfElems = 0; + int32_t numOfElems = 0; SColumnInfoData* pOutput = (SColumnInfoData*)pCtx->pOutput; SColumnInfoData* pTsOutput = pCtx->pTsOutput; int32_t i = pInput->startRowIndex; - TSKEY* tsList = (int64_t*)pInput->pPTS->pData; + TSKEY* tsList = (int64_t*)pInput->pPTS->pData; double v = 0; @@ -5224,7 +5169,7 @@ int32_t derivativeFunction(SqlFunctionCtx *pCtx) { continue; } - char* d = (char*)pInputCol->pData + pInputCol->info.bytes * i; + char* d = (char*)pInputCol->pData + pInputCol->info.bytes * i; GET_TYPED_DATA(v, double, pInputCol->info.type, d); int32_t pos = pCtx->offset + numOfElems; @@ -5251,7 +5196,7 @@ int32_t derivativeFunction(SqlFunctionCtx *pCtx) { continue; } - char* d = (char*)pInputCol->pData + pInputCol->info.bytes * i; + char* d = (char*)pInputCol->pData + pInputCol->info.bytes * i; GET_TYPED_DATA(v, double, pInputCol->info.type, d); int32_t pos = pCtx->offset + numOfElems; @@ -5277,7 +5222,7 @@ int32_t derivativeFunction(SqlFunctionCtx *pCtx) { return numOfElems; } -int32_t interpFunction(SqlFunctionCtx *pCtx) { +int32_t interpFunction(SqlFunctionCtx* pCtx) { #if 0 int32_t fillType = (int32_t) pCtx->param[2].i64; //bool ascQuery = (pCtx->order == TSDB_ORDER_ASC); @@ -5368,4 +5313,4 @@ int32_t interpFunction(SqlFunctionCtx *pCtx) { #endif return TSDB_CODE_SUCCESS; -} \ No newline at end of file +} diff --git a/source/libs/function/src/functionMgt.c b/source/libs/function/src/functionMgt.c index 0c762513cb6bd6bb6787b4d6cd1d45a5b41a097b..710b01ce59eb04573ed3ec9befe3046f1876df4e 100644 --- a/source/libs/function/src/functionMgt.c +++ b/source/libs/function/src/functionMgt.c @@ -70,11 +70,21 @@ int32_t fmFuncMgtInit() { } int32_t fmGetFuncInfo(SFunctionNode* pFunc, char* pMsg, int32_t msgLen) { - void* pVal = taosHashGet(gFunMgtService.pFuncNameHashTable, pFunc->functionName, strlen(pFunc->functionName)); - if (NULL != pVal) { - pFunc->funcId = *(int32_t*)pVal; - pFunc->funcType = funcMgtBuiltins[pFunc->funcId].type; - return funcMgtBuiltins[pFunc->funcId].translateFunc(pFunc, pMsg, msgLen); + if (NULL != gFunMgtService.pFuncNameHashTable) { + void* pVal = taosHashGet(gFunMgtService.pFuncNameHashTable, pFunc->functionName, strlen(pFunc->functionName)); + if (NULL != pVal) { + pFunc->funcId = *(int32_t*)pVal; + pFunc->funcType = funcMgtBuiltins[pFunc->funcId].type; + return funcMgtBuiltins[pFunc->funcId].translateFunc(pFunc, pMsg, msgLen); + } + return TSDB_CODE_FUNC_NOT_BUILTIN_FUNTION; + } + for (int32_t i = 0; i < funcMgtBuiltinsNum; ++i) { + if (0 == strcmp(funcMgtBuiltins[i].name, pFunc->functionName)) { + pFunc->funcId = i; + pFunc->funcType = funcMgtBuiltins[pFunc->funcId].type; + return funcMgtBuiltins[pFunc->funcId].translateFunc(pFunc, pMsg, msgLen); + } } return TSDB_CODE_FUNC_NOT_BUILTIN_FUNTION; } @@ -165,6 +175,17 @@ bool fmIsIntervalInterpoFunc(int32_t funcId) { return isSpecificClassifyFunc(fun bool fmIsForbidStreamFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_FORBID_STREAM_FUNC); } +bool fmIsForbidWindowFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_FORBID_WINDOW_FUNC); } + +bool fmIsForbidGroupByFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_FORBID_GROUP_BY_FUNC); } + +bool fmIsInterpFunc(int32_t funcId) { + if (funcId < 0 || funcId >= funcMgtBuiltinsNum) { + return false; + } + return FUNCTION_TYPE_INTERP == funcMgtBuiltins[funcId].type; +} + void fmFuncMgtDestroy() { void* m = gFunMgtService.pFuncNameHashTable; if (m != NULL && atomic_val_compare_exchange_ptr((void**)&gFunMgtService.pFuncNameHashTable, m, 0) == m) { @@ -206,7 +227,7 @@ bool fmIsInvertible(int32_t funcId) { return res; } -//function has same input/output type +// function has same input/output type bool fmIsSameInOutType(int32_t funcId) { bool res = false; switch (funcMgtBuiltins[funcId].type) { @@ -229,17 +250,7 @@ bool fmIsSameInOutType(int32_t funcId) { static int32_t getFuncInfo(SFunctionNode* pFunc) { char msg[64] = {0}; - if (NULL != gFunMgtService.pFuncNameHashTable) { - return fmGetFuncInfo(pFunc, msg, sizeof(msg)); - } - for (int32_t i = 0; i < funcMgtBuiltinsNum; ++i) { - if (0 == strcmp(funcMgtBuiltins[i].name, pFunc->functionName)) { - pFunc->funcId = i; - pFunc->funcType = funcMgtBuiltins[pFunc->funcId].type; - return funcMgtBuiltins[pFunc->funcId].translateFunc(pFunc, msg, sizeof(msg)); - } - } - return TSDB_CODE_FUNC_NOT_BUILTIN_FUNTION; + return fmGetFuncInfo(pFunc, msg, sizeof(msg)); } static SFunctionNode* createFunction(const char* pName, SNodeList* pParameterList) { @@ -257,14 +268,14 @@ static SFunctionNode* createFunction(const char* pName, SNodeList* pParameterLis return pFunc; } -static SColumnNode* createColumnByFunc(const SFunctionNode* pFunc) { +static SNode* createColumnByFunc(const SFunctionNode* pFunc) { SColumnNode* pCol = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN); if (NULL == pCol) { return NULL; } strcpy(pCol->colName, pFunc->node.aliasName); pCol->node.resType = pFunc->node.resType; - return pCol; + return (SNode*)pCol; } bool fmIsDistExecFunc(int32_t funcId) { @@ -292,21 +303,47 @@ static int32_t createPartialFunction(const SFunctionNode* pSrcFunc, SFunctionNod return TSDB_CODE_SUCCESS; } +static int32_t createMergeFuncPara(const SFunctionNode* pSrcFunc, const SFunctionNode* pPartialFunc, + SNodeList** pParameterList) { + SNode* pRes = createColumnByFunc(pPartialFunc); + if (NULL != funcMgtBuiltins[pSrcFunc->funcId].createMergeParaFuc) { + return funcMgtBuiltins[pSrcFunc->funcId].createMergeParaFuc(pSrcFunc->pParameterList, pRes, pParameterList); + } else { + return nodesListMakeStrictAppend(pParameterList, pRes); + } +} + static int32_t createMergeFunction(const SFunctionNode* pSrcFunc, const SFunctionNode* pPartialFunc, SFunctionNode** pMergeFunc) { - SNodeList* pParameterList = NULL; - nodesListMakeStrictAppend(&pParameterList, (SNode*)createColumnByFunc(pPartialFunc)); - *pMergeFunc = createFunction(funcMgtBuiltins[pSrcFunc->funcId].pMergeFunc, pParameterList); - if (NULL == *pMergeFunc) { - nodesDestroyList(pParameterList); - return TSDB_CODE_OUT_OF_MEMORY; + SNodeList* pParameterList = NULL; + SFunctionNode* pFunc = NULL; + + int32_t code = createMergeFuncPara(pSrcFunc, pPartialFunc, &pParameterList); + if (TSDB_CODE_SUCCESS == code) { + pFunc = createFunction(funcMgtBuiltins[pSrcFunc->funcId].pMergeFunc, pParameterList); + if (NULL == pFunc) { + code = TSDB_CODE_OUT_OF_MEMORY; + } } - //overwrite function restype set by translate function - if (fmIsSameInOutType(pSrcFunc->funcId)) { - (*pMergeFunc)->node.resType = pSrcFunc->node.resType; + if (TSDB_CODE_SUCCESS == code) { + // overwrite function restype set by translate function + if (fmIsSameInOutType(pSrcFunc->funcId)) { + pFunc->node.resType = pSrcFunc->node.resType; + } + strcpy(pFunc->node.aliasName, pSrcFunc->node.aliasName); } - strcpy((*pMergeFunc)->node.aliasName, pSrcFunc->node.aliasName); - return TSDB_CODE_SUCCESS; + + if (TSDB_CODE_SUCCESS == code) { + *pMergeFunc = pFunc; + } else { + if (NULL != pFunc) { + pFunc->pParameterList = NULL; + nodesDestroyNode((SNode*)pFunc); + } + nodesDestroyList(pParameterList); + } + + return code; } int32_t fmGetDistMethod(const SFunctionNode* pFunc, SFunctionNode** pPartialFunc, SFunctionNode** pMergeFunc) { diff --git a/source/libs/function/src/tudf.c b/source/libs/function/src/tudf.c index 472d67260730ca10522ee0d07fc1d608b132688e..f6ae027e48cc01a3d098d25156de168365abd04a 100644 --- a/source/libs/function/src/tudf.c +++ b/source/libs/function/src/tudf.c @@ -972,6 +972,11 @@ void releaseUdfFuncHandle(char* udfName) { } int32_t cleanUpUdfs() { + int8_t initialized = atomic_load_8(&gUdfdProxy.initialized); + if (!initialized) { + return TSDB_CODE_SUCCESS; + } + uv_mutex_lock(&gUdfdProxy.udfStubsMutex); int32_t i = 0; SArray* udfStubs = taosArrayInit(16, sizeof(SUdfcFuncStub)); diff --git a/source/libs/nodes/src/nodesCloneFuncs.c b/source/libs/nodes/src/nodesCloneFuncs.c index 2e30b01357464acbd0c50eb842526879f3c5c4aa..5eaca60ea5d8c9f9439a7b9139789622ecad7dbf 100644 --- a/source/libs/nodes/src/nodesCloneFuncs.c +++ b/source/libs/nodes/src/nodesCloneFuncs.c @@ -423,11 +423,12 @@ static SNode* logicWindowCopy(const SWindowLogicNode* pSrc, SWindowLogicNode* pD COPY_SCALAR_FIELD(slidingUnit); COPY_SCALAR_FIELD(sessionGap); CLONE_NODE_FIELD(pTspk); + CLONE_NODE_FIELD(pTsEnd); CLONE_NODE_FIELD(pStateExpr); COPY_SCALAR_FIELD(triggerType); COPY_SCALAR_FIELD(watermark); COPY_SCALAR_FIELD(filesFactor); - COPY_SCALAR_FIELD(intervalAlgo); + COPY_SCALAR_FIELD(windowAlgo); return (SNode*)pDst; } @@ -454,7 +455,18 @@ static SNode* logicPartitionCopy(const SPartitionLogicNode* pSrc, SPartitionLogi static SNode* logicIndefRowsFuncCopy(const SIndefRowsFuncLogicNode* pSrc, SIndefRowsFuncLogicNode* pDst) { COPY_BASE_OBJECT_FIELD(node, logicNodeCopy); - CLONE_NODE_LIST_FIELD(pVectorFuncs); + CLONE_NODE_LIST_FIELD(pFuncs); + return (SNode*)pDst; +} + +static SNode* logicInterpFuncCopy(const SInterpFuncLogicNode* pSrc, SInterpFuncLogicNode* pDst) { + COPY_BASE_OBJECT_FIELD(node, logicNodeCopy); + CLONE_NODE_LIST_FIELD(pFuncs); + COPY_OBJECT_FIELD(timeRange, sizeof(STimeWindow)); + COPY_SCALAR_FIELD(interval); + COPY_SCALAR_FIELD(fillMode); + CLONE_NODE_FIELD(pFillValues); + CLONE_NODE_FIELD(pTimeSeries); return (SNode*)pDst; } @@ -522,6 +534,7 @@ static SNode* physiWindowCopy(const SWinodwPhysiNode* pSrc, SWinodwPhysiNode* pD CLONE_NODE_LIST_FIELD(pExprs); CLONE_NODE_LIST_FIELD(pFuncs); CLONE_NODE_FIELD(pTspk); + CLONE_NODE_FIELD(pTsEnd); COPY_SCALAR_FIELD(triggerType); COPY_SCALAR_FIELD(watermark); COPY_SCALAR_FIELD(filesFactor); @@ -538,6 +551,12 @@ static SNode* physiIntervalCopy(const SIntervalPhysiNode* pSrc, SIntervalPhysiNo return (SNode*)pDst; } +static SNode* physiSessionCopy(const SSessionWinodwPhysiNode* pSrc, SSessionWinodwPhysiNode* pDst) { + COPY_BASE_OBJECT_FIELD(window, physiWindowCopy); + COPY_SCALAR_FIELD(gap); + return (SNode*)pDst; +} + static SNode* dataBlockDescCopy(const SDataBlockDescNode* pSrc, SDataBlockDescNode* pDst) { COPY_SCALAR_FIELD(dataBlockId); CLONE_NODE_LIST_FIELD(pSlots); @@ -662,6 +681,8 @@ SNode* nodesCloneNode(const SNode* pNode) { return logicPartitionCopy((const SPartitionLogicNode*)pNode, (SPartitionLogicNode*)pDst); case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return logicIndefRowsFuncCopy((const SIndefRowsFuncLogicNode*)pNode, (SIndefRowsFuncLogicNode*)pDst); + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return logicInterpFuncCopy((const SInterpFuncLogicNode*)pNode, (SInterpFuncLogicNode*)pDst); case QUERY_NODE_LOGIC_SUBPLAN: return logicSubplanCopy((const SLogicSubplan*)pNode, (SLogicSubplan*)pDst); case QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN: @@ -678,6 +699,9 @@ SNode* nodesCloneNode(const SNode* pNode) { case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL: case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_INTERVAL: return physiIntervalCopy((const SIntervalPhysiNode*)pNode, (SIntervalPhysiNode*)pDst); + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: + return physiSessionCopy((const SSessionWinodwPhysiNode*)pNode, (SSessionWinodwPhysiNode*)pDst); default: break; } diff --git a/source/libs/nodes/src/nodesCodeFuncs.c b/source/libs/nodes/src/nodesCodeFuncs.c index fb6a428d3c82c190b76be62b24ddc149ce9b46ee..bde5159087634a2ad06b6b4a1e38ff06f7a32948 100644 --- a/source/libs/nodes/src/nodesCodeFuncs.c +++ b/source/libs/nodes/src/nodesCodeFuncs.c @@ -172,8 +172,6 @@ const char* nodesNodeName(ENodeType type) { return "ShowSubscribesStmt"; case QUERY_NODE_SHOW_SMAS_STMT: return "ShowSmasStmt"; - case QUERY_NODE_SHOW_CONFIGS_STMT: - return "ShowConfigsStmt"; case QUERY_NODE_SHOW_QUERIES_STMT: return "ShowQueriesStmt"; case QUERY_NODE_SHOW_VNODES_STMT: @@ -204,6 +202,8 @@ const char* nodesNodeName(ENodeType type) { return "LogicPartition"; case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return "LogicIndefRowsFunc"; + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return "LogicInterpFunc"; case QUERY_NODE_LOGIC_SUBPLAN: return "LogicSubplan"; case QUERY_NODE_LOGIC_PLAN: @@ -218,6 +218,8 @@ const char* nodesNodeName(ENodeType type) { return "PhysiSreamScan"; case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN: return "PhysiSystemTableScan"; + case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN: + return "PhysiBlockDistScan"; case QUERY_NODE_PHYSICAL_PLAN_PROJECT: return "PhysiProject"; case QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN: @@ -246,6 +248,10 @@ const char* nodesNodeName(ENodeType type) { return "PhysiSessionWindow"; case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION: return "PhysiStreamSessionWindow"; + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: + return "PhysiStreamSemiSessionWindow"; + case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: + return "PhysiStreamFinalSessionWindow"; case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE: return "PhysiStateWindow"; case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE: @@ -254,6 +260,8 @@ const char* nodesNodeName(ENodeType type) { return "PhysiPartition"; case QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC: return "PhysiIndefRowsFunc"; + case QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC: + return "PhysiInterpFunc"; case QUERY_NODE_PHYSICAL_PLAN_DISPATCH: return "PhysiDispatch"; case QUERY_NODE_PHYSICAL_PLAN_INSERT: @@ -931,14 +939,14 @@ static int32_t jsonToLogicPartitionNode(const SJson* pJson, void* pObj) { return code; } -static const char* jkIndefRowsFuncLogicPlanVectorFuncs = "VectorFuncs"; +static const char* jkIndefRowsFuncLogicPlanFuncs = "Funcs"; static int32_t logicIndefRowsFuncNodeToJson(const void* pObj, SJson* pJson) { const SIndefRowsFuncLogicNode* pNode = (const SIndefRowsFuncLogicNode*)pObj; int32_t code = logicPlanNodeToJson(pObj, pJson); if (TSDB_CODE_SUCCESS == code) { - code = nodeListToJson(pJson, jkIndefRowsFuncLogicPlanVectorFuncs, pNode->pVectorFuncs); + code = nodeListToJson(pJson, jkIndefRowsFuncLogicPlanFuncs, pNode->pFuncs); } return code; @@ -949,7 +957,52 @@ static int32_t jsonToLogicIndefRowsFuncNode(const SJson* pJson, void* pObj) { int32_t code = jsonToLogicPlanNode(pJson, pObj); if (TSDB_CODE_SUCCESS == code) { - code = jsonToNodeList(pJson, jkIndefRowsFuncLogicPlanVectorFuncs, &pNode->pVectorFuncs); + code = jsonToNodeList(pJson, jkIndefRowsFuncLogicPlanFuncs, &pNode->pFuncs); + } + + return code; +} + +static const char* jkInterpFuncLogicPlanFuncs = "Funcs"; +static const char* jkInterpFuncLogicPlanStartTime = "StartTime"; +static const char* jkInterpFuncLogicPlanEndTime = "EndTime"; +static const char* jkInterpFuncLogicPlanInterval = "Interval"; + +static int32_t logicInterpFuncNodeToJson(const void* pObj, SJson* pJson) { + const SInterpFuncLogicNode* pNode = (const SInterpFuncLogicNode*)pObj; + + int32_t code = logicPlanNodeToJson(pObj, pJson); + if (TSDB_CODE_SUCCESS == code) { + code = nodeListToJson(pJson, jkInterpFuncLogicPlanFuncs, pNode->pFuncs); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncLogicPlanStartTime, pNode->timeRange.skey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncLogicPlanEndTime, pNode->timeRange.ekey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncLogicPlanInterval, pNode->interval); + } + + return code; +} + +static int32_t jsonToLogicInterpFuncNode(const SJson* pJson, void* pObj) { + SInterpFuncLogicNode* pNode = (SInterpFuncLogicNode*)pObj; + + int32_t code = jsonToLogicPlanNode(pJson, pObj); + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeList(pJson, jkInterpFuncLogicPlanFuncs, &pNode->pFuncs); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncLogicPlanStartTime, &pNode->timeRange.skey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncLogicPlanEndTime, &pNode->timeRange.ekey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncLogicPlanInterval, &pNode->interval); } return code; @@ -1780,6 +1833,7 @@ static int32_t jsonToPhysiSortNode(const SJson* pJson, void* pObj) { static const char* jkWindowPhysiPlanExprs = "Exprs"; static const char* jkWindowPhysiPlanFuncs = "Funcs"; static const char* jkWindowPhysiPlanTsPk = "TsPk"; +static const char* jkWindowPhysiPlanTsEnd = "TsEnd"; static const char* jkWindowPhysiPlanTriggerType = "TriggerType"; static const char* jkWindowPhysiPlanWatermark = "Watermark"; static const char* jkWindowPhysiPlanFilesFactor = "FilesFactor"; @@ -1797,6 +1851,9 @@ static int32_t physiWindowNodeToJson(const void* pObj, SJson* pJson) { if (TSDB_CODE_SUCCESS == code) { code = tjsonAddObject(pJson, jkWindowPhysiPlanTsPk, nodeToJson, pNode->pTspk); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddObject(pJson, jkWindowPhysiPlanTsEnd, nodeToJson, pNode->pTsEnd); + } if (TSDB_CODE_SUCCESS == code) { code = tjsonAddIntegerToObject(pJson, jkWindowPhysiPlanTriggerType, pNode->triggerType); } @@ -1823,6 +1880,9 @@ static int32_t jsonToPhysiWindowNode(const SJson* pJson, void* pObj) { if (TSDB_CODE_SUCCESS == code) { code = jsonToNodeObject(pJson, jkWindowPhysiPlanTsPk, (SNode**)&pNode->pTspk); } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeObject(pJson, jkWindowPhysiPlanTsEnd, (SNode**)&pNode->pTsEnd); + } if (TSDB_CODE_SUCCESS == code) { tjsonGetNumberValue(pJson, jkWindowPhysiPlanTriggerType, pNode->triggerType, code); ; @@ -2038,7 +2098,7 @@ static int32_t jsonToPhysiPartitionNode(const SJson* pJson, void* pObj) { } static const char* jkIndefRowsFuncPhysiPlanExprs = "Exprs"; -static const char* jkIndefRowsFuncPhysiPlanVectorFuncs = "VectorFuncs"; +static const char* jkIndefRowsFuncPhysiPlanFuncs = "Funcs"; static int32_t physiIndefRowsFuncNodeToJson(const void* pObj, SJson* pJson) { const SIndefRowsFuncPhysiNode* pNode = (const SIndefRowsFuncPhysiNode*)pObj; @@ -2048,7 +2108,7 @@ static int32_t physiIndefRowsFuncNodeToJson(const void* pObj, SJson* pJson) { code = nodeListToJson(pJson, jkIndefRowsFuncPhysiPlanExprs, pNode->pExprs); } if (TSDB_CODE_SUCCESS == code) { - code = nodeListToJson(pJson, jkIndefRowsFuncPhysiPlanVectorFuncs, pNode->pVectorFuncs); + code = nodeListToJson(pJson, jkIndefRowsFuncPhysiPlanFuncs, pNode->pFuncs); } return code; @@ -2062,7 +2122,80 @@ static int32_t jsonToPhysiIndefRowsFuncNode(const SJson* pJson, void* pObj) { code = jsonToNodeList(pJson, jkIndefRowsFuncPhysiPlanExprs, &pNode->pExprs); } if (TSDB_CODE_SUCCESS == code) { - code = jsonToNodeList(pJson, jkIndefRowsFuncPhysiPlanVectorFuncs, &pNode->pVectorFuncs); + code = jsonToNodeList(pJson, jkIndefRowsFuncPhysiPlanFuncs, &pNode->pFuncs); + } + + return code; +} + +static const char* jkInterpFuncPhysiPlanExprs = "Exprs"; +static const char* jkInterpFuncPhysiPlanFuncs = "Funcs"; +static const char* jkInterpFuncPhysiPlanStartTime = "StartTime"; +static const char* jkInterpFuncPhysiPlanEndTime = "EndTime"; +static const char* jkInterpFuncPhysiPlanInterval = "Interval"; +static const char* jkInterpFuncPhysiPlanFillMode = "FillMode"; +static const char* jkInterpFuncPhysiPlanFillValues = "FillValues"; +static const char* jkInterpFuncPhysiPlanTimeSeries = "TimeSeries"; + +static int32_t physiInterpFuncNodeToJson(const void* pObj, SJson* pJson) { + const SInterpFuncPhysiNode* pNode = (const SInterpFuncPhysiNode*)pObj; + + int32_t code = physicPlanNodeToJson(pObj, pJson); + if (TSDB_CODE_SUCCESS == code) { + code = nodeListToJson(pJson, jkInterpFuncPhysiPlanExprs, pNode->pExprs); + } + if (TSDB_CODE_SUCCESS == code) { + code = nodeListToJson(pJson, jkInterpFuncPhysiPlanFuncs, pNode->pFuncs); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncPhysiPlanStartTime, pNode->timeRange.skey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncPhysiPlanEndTime, pNode->timeRange.ekey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncPhysiPlanInterval, pNode->interval); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkInterpFuncPhysiPlanFillMode, pNode->fillMode); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddObject(pJson, jkInterpFuncPhysiPlanFillValues, nodeToJson, pNode->pFillValues); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddObject(pJson, jkInterpFuncPhysiPlanTimeSeries, nodeToJson, pNode->pTimeSeries); + } + + return code; +} + +static int32_t jsonToPhysiInterpFuncNode(const SJson* pJson, void* pObj) { + SInterpFuncPhysiNode* pNode = (SInterpFuncPhysiNode*)pObj; + + int32_t code = jsonToPhysicPlanNode(pJson, pObj); + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeList(pJson, jkInterpFuncPhysiPlanExprs, &pNode->pExprs); + } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeList(pJson, jkInterpFuncPhysiPlanFuncs, &pNode->pFuncs); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncPhysiPlanStartTime, &pNode->timeRange.skey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncPhysiPlanEndTime, &pNode->timeRange.ekey); + } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkInterpFuncPhysiPlanInterval, &pNode->interval); + } + if (TSDB_CODE_SUCCESS == code) { + tjsonGetNumberValue(pJson, jkInterpFuncPhysiPlanFillMode, pNode->fillMode, code); + } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeObject(pJson, jkInterpFuncPhysiPlanFillValues, &pNode->pFillValues); + } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeObject(pJson, jkInterpFuncPhysiPlanTimeSeries, &pNode->pTimeSeries); } return code; @@ -3964,11 +4097,14 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) { return logicPartitionNodeToJson(pObj, pJson); case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return logicIndefRowsFuncNodeToJson(pObj, pJson); + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return logicInterpFuncNodeToJson(pObj, pJson); case QUERY_NODE_LOGIC_SUBPLAN: return logicSubplanToJson(pObj, pJson); case QUERY_NODE_LOGIC_PLAN: return logicPlanToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN: + case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN: return physiTagScanNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN: case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN: @@ -3998,6 +4134,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) { return physiFillNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION: case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: return physiSessionWindowNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE: case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE: @@ -4006,6 +4144,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) { return physiPartitionNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC: return physiIndefRowsFuncNodeToJson(pObj, pJson); + case QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC: + return physiInterpFuncNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_DISPATCH: return physiDispatchNodeToJson(pObj, pJson); case QUERY_NODE_PHYSICAL_PLAN_INSERT: @@ -4097,11 +4237,14 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) { return jsonToLogicPartitionNode(pJson, pObj); case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return jsonToLogicIndefRowsFuncNode(pJson, pObj); + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return jsonToLogicInterpFuncNode(pJson, pObj); case QUERY_NODE_LOGIC_SUBPLAN: return jsonToLogicSubplan(pJson, pObj); case QUERY_NODE_LOGIC_PLAN: return jsonToLogicPlan(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN: + case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN: return jsonToPhysiTagScanNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN: case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN: @@ -4131,6 +4274,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) { return jsonToPhysiFillNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION: case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: return jsonToPhysiSessionWindowNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE: case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE: @@ -4139,6 +4284,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) { return jsonToPhysiPartitionNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC: return jsonToPhysiIndefRowsFuncNode(pJson, pObj); + case QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC: + return jsonToPhysiInterpFuncNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_DISPATCH: return jsonToPhysiDispatchNode(pJson, pObj); case QUERY_NODE_PHYSICAL_PLAN_DELETE: diff --git a/source/libs/nodes/src/nodesUtilFuncs.c b/source/libs/nodes/src/nodesUtilFuncs.c index a45ba53ad173b3bbb88881e3d3db8624ea94e51e..a48071ef52397450d3160def9d4f38adbf920d24 100644 --- a/source/libs/nodes/src/nodesUtilFuncs.c +++ b/source/libs/nodes/src/nodesUtilFuncs.c @@ -88,6 +88,8 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SStreamOptions)); case QUERY_NODE_LEFT_VALUE: return makeNode(type, sizeof(SLeftValueNode)); + case QUERY_NODE_COLUMN_REF: + return makeNode(type, sizeof(SColumnDefNode)); case QUERY_NODE_SET_OPERATOR: return makeNode(type, sizeof(SSetOperator)); case QUERY_NODE_SELECT_STMT: @@ -148,6 +150,8 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SDropTopicStmt)); case QUERY_NODE_DROP_CGROUP_STMT: return makeNode(type, sizeof(SDropCGroupStmt)); + case QUERY_NODE_ALTER_LOCAL_STMT: + return makeNode(type, sizeof(SAlterLocalStmt)); case QUERY_NODE_EXPLAIN_STMT: return makeNode(type, sizeof(SExplainStmt)); case QUERY_NODE_DESCRIBE_STMT: @@ -198,18 +202,21 @@ SNode* nodesMakeNode(ENodeType type) { case QUERY_NODE_SHOW_CONSUMERS_STMT: case QUERY_NODE_SHOW_SUBSCRIBES_STMT: case QUERY_NODE_SHOW_SMAS_STMT: - case QUERY_NODE_SHOW_CONFIGS_STMT: case QUERY_NODE_SHOW_CONNECTIONS_STMT: case QUERY_NODE_SHOW_QUERIES_STMT: case QUERY_NODE_SHOW_VNODES_STMT: case QUERY_NODE_SHOW_APPS_STMT: case QUERY_NODE_SHOW_SCORES_STMT: case QUERY_NODE_SHOW_VARIABLE_STMT: + case QUERY_NODE_SHOW_TRANSACTIONS_STMT: + return makeNode(type, sizeof(SShowStmt)); case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + return makeNode(type, sizeof(SShowCreateDatabaseStmt)); case QUERY_NODE_SHOW_CREATE_TABLE_STMT: case QUERY_NODE_SHOW_CREATE_STABLE_STMT: - case QUERY_NODE_SHOW_TRANSACTIONS_STMT: - return makeNode(type, sizeof(SShowStmt)); + return makeNode(type, sizeof(SShowCreateTableStmt)); + case QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT: + return makeNode(type, sizeof(SShowTableDistributedStmt)); case QUERY_NODE_KILL_QUERY_STMT: return makeNode(type, sizeof(SKillQueryStmt)); case QUERY_NODE_KILL_TRANSACTION_STMT: @@ -243,6 +250,8 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SPartitionLogicNode)); case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return makeNode(type, sizeof(SIndefRowsFuncLogicNode)); + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return makeNode(type, sizeof(SInterpFuncLogicNode)); case QUERY_NODE_LOGIC_SUBPLAN: return makeNode(type, sizeof(SLogicSubplan)); case QUERY_NODE_LOGIC_PLAN: @@ -259,6 +268,8 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SStreamScanPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN: return makeNode(type, sizeof(SSystemTableScanPhysiNode)); + case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN: + return makeNode(type, sizeof(SBlockDistScanPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_PROJECT: return makeNode(type, sizeof(SProjectPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN: @@ -287,6 +298,10 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SSessionWinodwPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION: return makeNode(type, sizeof(SStreamSessionWinodwPhysiNode)); + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: + return makeNode(type, sizeof(SStreamSemiSessionWinodwPhysiNode)); + case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: + return makeNode(type, sizeof(SStreamFinalSessionWinodwPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE: return makeNode(type, sizeof(SStateWinodwPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE: @@ -295,6 +310,8 @@ SNode* nodesMakeNode(ENodeType type) { return makeNode(type, sizeof(SPartitionPhysiNode)); case QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC: return makeNode(type, sizeof(SIndefRowsFuncPhysiNode)); + case QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC: + return makeNode(type, sizeof(SInterpFuncLogicNode)); case QUERY_NODE_PHYSICAL_PLAN_DISPATCH: return makeNode(type, sizeof(SDataDispatcherNode)); case QUERY_NODE_PHYSICAL_PLAN_INSERT: @@ -339,6 +356,7 @@ static void destroyWinodwPhysiNode(SWinodwPhysiNode* pNode) { nodesDestroyList(pNode->pExprs); nodesDestroyList(pNode->pFuncs); nodesDestroyNode(pNode->pTspk); + nodesDestroyNode(pNode->pTsEnd); } static void destroyScanPhysiNode(SScanPhysiNode* pNode) { @@ -512,6 +530,7 @@ void nodesDestroyNode(SNode* pNode) { SCreateSubTableClause* pStmt = (SCreateSubTableClause*)pNode; nodesDestroyList(pStmt->pSpecificTags); nodesDestroyList(pStmt->pValsOfTags); + nodesDestroyNode((SNode*)pStmt->pOptions); break; } case QUERY_NODE_CREATE_MULTI_TABLE_STMT: @@ -613,25 +632,29 @@ void nodesDestroyNode(SNode* pNode) { case QUERY_NODE_SHOW_CONSUMERS_STMT: case QUERY_NODE_SHOW_SUBSCRIBES_STMT: case QUERY_NODE_SHOW_SMAS_STMT: - case QUERY_NODE_SHOW_CONFIGS_STMT: case QUERY_NODE_SHOW_CONNECTIONS_STMT: case QUERY_NODE_SHOW_QUERIES_STMT: case QUERY_NODE_SHOW_VNODES_STMT: case QUERY_NODE_SHOW_APPS_STMT: case QUERY_NODE_SHOW_SCORES_STMT: case QUERY_NODE_SHOW_VARIABLE_STMT: - case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: - case QUERY_NODE_SHOW_CREATE_TABLE_STMT: - case QUERY_NODE_SHOW_CREATE_STABLE_STMT: case QUERY_NODE_SHOW_TRANSACTIONS_STMT: { SShowStmt* pStmt = (SShowStmt*)pNode; nodesDestroyNode(pStmt->pDbName); - nodesDestroyNode(pStmt->pTbNamePattern); + nodesDestroyNode(pStmt->pTbName); break; } - case QUERY_NODE_KILL_CONNECTION_STMT: // no pointer field - case QUERY_NODE_KILL_QUERY_STMT: // no pointer field - case QUERY_NODE_KILL_TRANSACTION_STMT: // no pointer field + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + taosMemoryFreeClear(((SShowCreateDatabaseStmt*)pNode)->pCfg); + break; + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: + taosMemoryFreeClear(((SShowCreateTableStmt*)pNode)->pMeta); + break; + case QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT: // no pointer field + case QUERY_NODE_KILL_CONNECTION_STMT: // no pointer field + case QUERY_NODE_KILL_QUERY_STMT: // no pointer field + case QUERY_NODE_KILL_TRANSACTION_STMT: // no pointer field break; case QUERY_NODE_DELETE_STMT: { SDeleteStmt* pStmt = (SDeleteStmt*)pNode; @@ -701,6 +724,7 @@ void nodesDestroyNode(SNode* pNode) { destroyLogicNode((SLogicNode*)pLogicNode); nodesDestroyList(pLogicNode->pFuncs); nodesDestroyNode(pLogicNode->pTspk); + nodesDestroyNode(pLogicNode->pTsEnd); break; } case QUERY_NODE_LOGIC_PLAN_FILL: { @@ -725,7 +749,13 @@ void nodesDestroyNode(SNode* pNode) { case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: { SIndefRowsFuncLogicNode* pLogicNode = (SIndefRowsFuncLogicNode*)pNode; destroyLogicNode((SLogicNode*)pLogicNode); - nodesDestroyList(pLogicNode->pVectorFuncs); + nodesDestroyList(pLogicNode->pFuncs); + break; + } + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: { + SInterpFuncLogicNode* pLogicNode = (SInterpFuncLogicNode*)pNode; + destroyLogicNode((SLogicNode*)pLogicNode); + nodesDestroyList(pLogicNode->pFuncs); break; } case QUERY_NODE_LOGIC_SUBPLAN: { @@ -744,6 +774,7 @@ void nodesDestroyNode(SNode* pNode) { case QUERY_NODE_PHYSICAL_PLAN_TABLE_SEQ_SCAN: case QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN: case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN: + case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN: destroyScanPhysiNode((SScanPhysiNode*)pNode); break; case QUERY_NODE_PHYSICAL_PLAN_PROJECT: { @@ -804,6 +835,7 @@ void nodesDestroyNode(SNode* pNode) { } case QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION: case QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION: + case QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION: case QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION: destroyWinodwPhysiNode((SWinodwPhysiNode*)pNode); break; @@ -826,7 +858,14 @@ void nodesDestroyNode(SNode* pNode) { SIndefRowsFuncPhysiNode* pPhyNode = (SIndefRowsFuncPhysiNode*)pNode; destroyPhysiNode((SPhysiNode*)pPhyNode); nodesDestroyList(pPhyNode->pExprs); - nodesDestroyList(pPhyNode->pVectorFuncs); + nodesDestroyList(pPhyNode->pFuncs); + break; + } + case QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC: { + SInterpFuncPhysiNode* pPhyNode = (SInterpFuncPhysiNode*)pNode; + destroyPhysiNode((SPhysiNode*)pPhyNode); + nodesDestroyList(pPhyNode->pExprs); + nodesDestroyList(pPhyNode->pFuncs); break; } case QUERY_NODE_PHYSICAL_PLAN_DISPATCH: @@ -1463,6 +1502,26 @@ int32_t nodesCollectSpecialNodes(SSelectStmt* pSelect, ESqlClause clause, ENodeT return TSDB_CODE_SUCCESS; } +static EDealRes hasColumn(SNode* pNode, void* pContext) { + if (QUERY_NODE_COLUMN == nodeType(pNode)) { + *(bool*)pContext = true; + return DEAL_RES_END; + } + return DEAL_RES_CONTINUE; +} + +bool nodesExprHasColumn(SNode* pNode) { + bool hasCol = false; + nodesWalkExprPostOrder(pNode, hasColumn, &hasCol); + return hasCol; +} + +bool nodesExprsHasColumn(SNodeList* pList) { + bool hasCol = false; + nodesWalkExprsPostOrder(pList, hasColumn, &hasCol); + return hasCol; +} + char* nodesGetFillModeString(EFillMode mode) { switch (mode) { case FILL_MODE_NONE: diff --git a/source/libs/parser/inc/parAst.h b/source/libs/parser/inc/parAst.h index b8c6ffc8434a863396ded781a3dfe49638b082d3..64fcccc827a93b2571f52ae5d3e5219a29c13043 100644 --- a/source/libs/parser/inc/parAst.h +++ b/source/libs/parser/inc/parAst.h @@ -59,8 +59,8 @@ typedef enum EDatabaseOptionType { typedef enum ETableOptionType { TABLE_OPTION_COMMENT = 1, - TABLE_OPTION_FILE_FACTOR, - TABLE_OPTION_DELAY, + TABLE_OPTION_MAXDELAY, + TABLE_OPTION_WATERMARK, TABLE_OPTION_ROLLUP, TABLE_OPTION_TTL, TABLE_OPTION_SMA @@ -109,6 +109,7 @@ SNode* createIntervalWindowNode(SAstCreateContext* pCxt, SNode* pInterval, SNode SNode* pFill); SNode* createFillNode(SAstCreateContext* pCxt, EFillMode mode, SNode* pValues); SNode* createGroupingSetNode(SAstCreateContext* pCxt, SNode* pNode); +SNode* createInterpTimeRange(SAstCreateContext* pCxt, SNode* pStart, SNode* pEnd); SNode* addWhereClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pWhere); SNode* addPartitionByClause(SAstCreateContext* pCxt, SNode* pStmt, SNodeList* pPartitionByList); @@ -118,6 +119,9 @@ SNode* addHavingClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pHaving); SNode* addOrderByClause(SAstCreateContext* pCxt, SNode* pStmt, SNodeList* pOrderByList); SNode* addSlimitClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pSlimit); SNode* addLimitClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pLimit); +SNode* addRangeClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pRange); +SNode* addEveryClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pEvery); +SNode* addFillClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pFill); SNode* createSelectStmt(SAstCreateContext* pCxt, bool isDistinct, SNodeList* pProjectionList, SNode* pTable); SNode* createSetOperator(SAstCreateContext* pCxt, ESetOperatorType type, SNode* pLeft, SNode* pRight); @@ -151,9 +155,12 @@ SNode* createAlterTableRenameCol(SAstCreateContext* pCxt, SNode* pRealTable, int SToken* pNewColName); SNode* createAlterTableSetTag(SAstCreateContext* pCxt, SNode* pRealTable, SToken* pTagName, SNode* pVal); SNode* createUseDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName); -SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, SNode* pTbNamePattern); -SNode* createShowCreateDatabaseStmt(SAstCreateContext* pCxt, const SToken* pDbName); +SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type); +SNode* createShowStmtWithCond(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, SNode* pTbName, + EOperatorType tableCondType); +SNode* createShowCreateDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName); SNode* createShowCreateTableStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pRealTable); +SNode* createShowTableDistributedStmt(SAstCreateContext* pCxt, SNode* pRealTable); SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const SToken* pPassword); SNode* createAlterUserStmt(SAstCreateContext* pCxt, SToken* pUserName, int8_t alterType, const SToken* pVal); SNode* createDropUserStmt(SAstCreateContext* pCxt, SToken* pUserName); diff --git a/source/libs/parser/inc/sql.y b/source/libs/parser/inc/sql.y index 0d9e6e4a73542a5c948c7b88d446a08bc7546151..4c808fc60981f4c0a482759275503ad0b939465c 100644 --- a/source/libs/parser/inc/sql.y +++ b/source/libs/parser/inc/sql.y @@ -111,7 +111,7 @@ priv_level(A) ::= db_name(B) NK_DOT NK_STAR. /************************************************ create/drop/alter dnode *********************************************/ cmd ::= CREATE DNODE dnode_endpoint(A). { pCxt->pRootNode = createCreateDnodeStmt(pCxt, &A, NULL); } -cmd ::= CREATE DNODE dnode_host_name(A) PORT NK_INTEGER(B). { pCxt->pRootNode = createCreateDnodeStmt(pCxt, &A, &B); } +cmd ::= CREATE DNODE dnode_endpoint(A) PORT NK_INTEGER(B). { pCxt->pRootNode = createCreateDnodeStmt(pCxt, &A, &B); } cmd ::= DROP DNODE NK_INTEGER(A). { pCxt->pRootNode = createDropDnodeStmt(pCxt, &A); } cmd ::= DROP DNODE dnode_endpoint(A). { pCxt->pRootNode = createDropDnodeStmt(pCxt, &A); } cmd ::= ALTER DNODE NK_INTEGER(A) NK_STRING(B). { pCxt->pRootNode = createAlterDnodeStmt(pCxt, &A, &B, NULL); } @@ -122,11 +122,8 @@ cmd ::= ALTER ALL DNODES NK_STRING(A) NK_STRING(B). %type dnode_endpoint { SToken } %destructor dnode_endpoint { } dnode_endpoint(A) ::= NK_STRING(B). { A = B; } - -%type dnode_host_name { SToken } -%destructor dnode_host_name { } -dnode_host_name(A) ::= NK_ID(B). { A = B; } -dnode_host_name(A) ::= NK_IPTOKEN(B). { A = B; } +dnode_endpoint(A) ::= NK_ID(B). { A = B; } +dnode_endpoint(A) ::= NK_IPTOKEN(B). { A = B; } /************************************************ alter local *********************************************************/ cmd ::= ALTER LOCAL NK_STRING(A). { pCxt->pRootNode = createAlterLocalStmt(pCxt, &A, NULL); } @@ -317,8 +314,8 @@ tags_def(A) ::= TAGS NK_LP column_def_list(B) NK_RP. table_options(A) ::= . { A = createDefaultTableOptions(pCxt); } table_options(A) ::= table_options(B) COMMENT NK_STRING(C). { A = setTableOption(pCxt, B, TABLE_OPTION_COMMENT, &C); } -//table_options(A) ::= table_options(B) DELAY NK_INTEGER(C). { A = setTableOption(pCxt, B, TABLE_OPTION_DELAY, &C); } -table_options(A) ::= table_options(B) FILE_FACTOR NK_FLOAT(C). { A = setTableOption(pCxt, B, TABLE_OPTION_FILE_FACTOR, &C); } +table_options(A) ::= table_options(B) MAX_DELAY duration_list(C). { A = setTableOption(pCxt, B, TABLE_OPTION_MAXDELAY, C); } +table_options(A) ::= table_options(B) WATERMARK duration_list(C). { A = setTableOption(pCxt, B, TABLE_OPTION_WATERMARK, C); } table_options(A) ::= table_options(B) ROLLUP NK_LP rollup_func_list(C) NK_RP. { A = setTableOption(pCxt, B, TABLE_OPTION_ROLLUP, C); } table_options(A) ::= table_options(B) TTL NK_INTEGER(C). { A = setTableOption(pCxt, B, TABLE_OPTION_TTL, &C); } table_options(A) ::= table_options(B) SMA NK_LP col_name_list(C) NK_RP. { A = setTableOption(pCxt, B, TABLE_OPTION_SMA, C); } @@ -331,6 +328,11 @@ alter_table_options(A) ::= alter_table_options(B) alter_table_option(C). alter_table_option(A) ::= COMMENT NK_STRING(B). { A.type = TABLE_OPTION_COMMENT; A.val = B; } alter_table_option(A) ::= TTL NK_INTEGER(B). { A.type = TABLE_OPTION_TTL; A.val = B; } +%type duration_list { SNodeList* } +%destructor duration_list { nodesDestroyList($$); } +duration_list(A) ::= duration_literal(B). { A = createNodeList(pCxt, releaseRawExprNode(pCxt, B)); } +duration_list(A) ::= duration_list(B) NK_COMMA duration_literal(C). { A = addNodeToList(pCxt, B, releaseRawExprNode(pCxt, C)); } + %type rollup_func_list { SNodeList* } %destructor rollup_func_list { nodesDestroyList($$); } rollup_func_list(A) ::= rollup_func_name(B). { A = createNodeList(pCxt, B); } @@ -348,34 +350,35 @@ col_name_list(A) ::= col_name_list(B) NK_COMMA col_name(C). col_name(A) ::= column_name(B). { A = createColumnNode(pCxt, NULL, &B); } /************************************************ show ****************************************************************/ -cmd ::= SHOW DNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DNODES_STMT, NULL, NULL); } -cmd ::= SHOW USERS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USERS_STMT, NULL, NULL); } -cmd ::= SHOW DATABASES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT, NULL, NULL); } -cmd ::= SHOW db_name_cond_opt(A) TABLES like_pattern_opt(B). { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TABLES_STMT, A, B); } -cmd ::= SHOW db_name_cond_opt(A) STABLES like_pattern_opt(B). { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STABLES_STMT, A, B); } -cmd ::= SHOW db_name_cond_opt(A) VGROUPS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VGROUPS_STMT, A, NULL); } -cmd ::= SHOW MNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MNODES_STMT, NULL, NULL); } -cmd ::= SHOW MODULES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MODULES_STMT, NULL, NULL); } -cmd ::= SHOW QNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QNODES_STMT, NULL, NULL); } -cmd ::= SHOW FUNCTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_FUNCTIONS_STMT, NULL, NULL); } -cmd ::= SHOW INDEXES FROM table_name_cond(A) from_db_opt(B). { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, A, B); } -cmd ::= SHOW STREAMS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STREAMS_STMT, NULL, NULL); } +cmd ::= SHOW DNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DNODES_STMT); } +cmd ::= SHOW USERS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USERS_STMT); } +cmd ::= SHOW DATABASES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT); } +cmd ::= SHOW db_name_cond_opt(A) TABLES like_pattern_opt(B). { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_TABLES_STMT, A, B, OP_TYPE_LIKE); } +cmd ::= SHOW db_name_cond_opt(A) STABLES like_pattern_opt(B). { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_STABLES_STMT, A, B, OP_TYPE_LIKE); } +cmd ::= SHOW db_name_cond_opt(A) VGROUPS. { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_VGROUPS_STMT, A, NULL, OP_TYPE_LIKE); } +cmd ::= SHOW MNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MNODES_STMT); } +cmd ::= SHOW MODULES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MODULES_STMT); } +cmd ::= SHOW QNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QNODES_STMT); } +cmd ::= SHOW FUNCTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_FUNCTIONS_STMT); } +cmd ::= SHOW INDEXES FROM table_name_cond(A) from_db_opt(B). { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, B, A, OP_TYPE_EQUAL); } +cmd ::= SHOW STREAMS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STREAMS_STMT); } cmd ::= SHOW ACCOUNTS. { pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_EXPRIE_STATEMENT); } -cmd ::= SHOW APPS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_APPS_STMT, NULL, NULL); } -cmd ::= SHOW CONNECTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CONNECTIONS_STMT, NULL, NULL); } -cmd ::= SHOW LICENCE. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT, NULL, NULL); } -cmd ::= SHOW GRANTS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT, NULL, NULL); } +cmd ::= SHOW APPS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_APPS_STMT); } +cmd ::= SHOW CONNECTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CONNECTIONS_STMT); } +cmd ::= SHOW LICENCE. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT); } +cmd ::= SHOW GRANTS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT); } cmd ::= SHOW CREATE DATABASE db_name(A). { pCxt->pRootNode = createShowCreateDatabaseStmt(pCxt, &A); } cmd ::= SHOW CREATE TABLE full_table_name(A). { pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_TABLE_STMT, A); } cmd ::= SHOW CREATE STABLE full_table_name(A). { pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_STABLE_STMT, A); } -cmd ::= SHOW QUERIES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QUERIES_STMT, NULL, NULL); } -cmd ::= SHOW SCORES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SCORES_STMT, NULL, NULL); } -cmd ::= SHOW TOPICS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TOPICS_STMT, NULL, NULL); } -cmd ::= SHOW VARIABLES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VARIABLE_STMT, NULL, NULL); } -cmd ::= SHOW BNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_BNODES_STMT, NULL, NULL); } -cmd ::= SHOW SNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SNODES_STMT, NULL, NULL); } -cmd ::= SHOW CLUSTER. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CLUSTER_STMT, NULL, NULL); } -cmd ::= SHOW TRANSACTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TRANSACTIONS_STMT, NULL, NULL); } +cmd ::= SHOW QUERIES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QUERIES_STMT); } +cmd ::= SHOW SCORES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SCORES_STMT); } +cmd ::= SHOW TOPICS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TOPICS_STMT); } +cmd ::= SHOW VARIABLES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VARIABLE_STMT); } +cmd ::= SHOW BNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_BNODES_STMT); } +cmd ::= SHOW SNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SNODES_STMT); } +cmd ::= SHOW CLUSTER. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CLUSTER_STMT); } +cmd ::= SHOW TRANSACTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TRANSACTIONS_STMT); } +cmd ::= SHOW TABLE DISTRIBUTED full_table_name(A). { pCxt->pRootNode = createShowTableDistributedStmt(pCxt, A); } db_name_cond_opt(A) ::= . { A = createDefaultDatabaseCondValue(pCxt); } db_name_cond_opt(A) ::= db_name(B) NK_DOT. { A = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &B); } @@ -788,7 +791,7 @@ join_type(A) ::= INNER. /************************************************ query_specification *************************************************/ query_specification(A) ::= SELECT set_quantifier_opt(B) select_list(C) from_clause(D) where_clause_opt(E) - partition_by_clause_opt(F) twindow_clause_opt(G) + partition_by_clause_opt(F) range_opt(J) every_opt(K) fill_opt(L) twindow_clause_opt(G) group_by_clause_opt(H) having_clause_opt(I). { A = createSelectStmt(pCxt, B, C, D); A = addWhereClause(pCxt, A, E); @@ -796,6 +799,9 @@ query_specification(A) ::= A = addWindowClauseClause(pCxt, A, G); A = addGroupByClause(pCxt, A, H); A = addHavingClause(pCxt, A, I); + A = addRangeClause(pCxt, A, J); + A = addEveryClause(pCxt, A, K); + A = addFillClause(pCxt, A, L); } %type set_quantifier_opt { bool } @@ -857,14 +863,20 @@ fill_mode(A) ::= NEXT. group_by_clause_opt(A) ::= . { A = NULL; } group_by_clause_opt(A) ::= GROUP BY group_by_list(B). { A = B; } -%type group_by_list { SNodeList* } -%destructor group_by_list { nodesDestroyList($$); } -group_by_list(A) ::= expression(B). { A = createNodeList(pCxt, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, B))); } -group_by_list(A) ::= group_by_list(B) NK_COMMA expression(C). { A = addNodeToList(pCxt, B, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, C))); } +%type group_by_list { SNodeList* } +%destructor group_by_list { nodesDestroyList($$); } +group_by_list(A) ::= expression(B). { A = createNodeList(pCxt, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, B))); } +group_by_list(A) ::= group_by_list(B) NK_COMMA expression(C). { A = addNodeToList(pCxt, B, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, C))); } having_clause_opt(A) ::= . { A = NULL; } having_clause_opt(A) ::= HAVING search_condition(B). { A = B; } +range_opt(A) ::= . { A = NULL; } +range_opt(A) ::= RANGE NK_LP expression(B) NK_COMMA expression(C) NK_RP. { A = createInterpTimeRange(pCxt, releaseRawExprNode(pCxt, B), releaseRawExprNode(pCxt, C)); } + +every_opt(A) ::= . { A = NULL; } +every_opt(A) ::= EVERY NK_LP duration_literal(B) NK_RP. { A = releaseRawExprNode(pCxt, B); } + /************************************************ query_expression ****************************************************/ query_expression(A) ::= query_expression_body(B) diff --git a/source/libs/parser/src/parAstCreater.c b/source/libs/parser/src/parAstCreater.c index 8a7987e4ec0b389619c76b8a9bff95e06b4239f9..ccb0d37da65a6be158dfb0295815151bb3ce36f1 100644 --- a/source/libs/parser/src/parAstCreater.c +++ b/source/libs/parser/src/parAstCreater.c @@ -18,6 +18,7 @@ #include "parAst.h" #include "parUtil.h" +#include "tglobal.h" #include "ttime.h" #define CHECK_OUT_OF_MEM(p) \ @@ -36,6 +37,14 @@ } \ } while (0) +#define COPY_STRING_FORM_ID_TOKEN(buf, pToken) strncpy(buf, (pToken)->z, TMIN((pToken)->n, sizeof(buf) - 1)) +#define COPY_STRING_FORM_STR_TOKEN(buf, pToken) \ + do { \ + if ((pToken)->n > 2) { \ + strncpy(buf, (pToken)->z + 1, TMIN((pToken)->n - 2, sizeof(buf) - 1)); \ + } \ + } while (0) + SToken nil_token = {.type = TK_NK_NIL, .n = 0, .z = NULL}; void initAstCreateContext(SParseContext* pParseCxt, SAstCreateContext* pCxt) { @@ -50,12 +59,6 @@ void initAstCreateContext(SParseContext* pParseCxt, SAstCreateContext* pCxt) { pCxt->errCode = TSDB_CODE_SUCCESS; } -static void copyStringFormStringToken(SToken* pToken, char* pBuf, int32_t len) { - if (pToken->n > 2) { - strncpy(pBuf, pToken->z + 1, TMIN(pToken->n - 2, len - 1)); - } -} - static void trimEscape(SToken* pName) { // todo need to deal with `ioo``ii` -> ioo`ii if (NULL != pName && pName->n > 1 && '`' == pName->z[0]) { @@ -108,50 +111,52 @@ static bool checkPassword(SAstCreateContext* pCxt, const SToken* pPasswordToken, return TSDB_CODE_SUCCESS == pCxt->errCode; } -static bool checkAndSplitEndpoint(SAstCreateContext* pCxt, const SToken* pEp, char* pFqdn, int32_t* pPort) { - if (NULL == pEp) { - pCxt->errCode = TSDB_CODE_PAR_SYNTAX_ERROR; - } else if (pEp->n >= TSDB_FQDN_LEN + 2 + 6) { // format 'fqdn:port' - pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NAME_OR_PASSWD_TOO_LONG); - } else { - char ep[TSDB_FQDN_LEN + 2 + 6]; - strncpy(ep, pEp->z, pEp->n); - strdequote(ep); - strtrim(ep); - char* pColon = strchr(ep, ':'); - if (NULL == pColon) { - pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_ENDPOINT); - } else { - strncpy(pFqdn, ep, pColon - ep); - *pPort = taosStr2Int32(pColon + 1, NULL, 10); - if (*pPort >= UINT16_MAX || *pPort <= 0) { - pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_PORT); - } - } +static int32_t parsePort(SAstCreateContext* pCxt, const char* p, int32_t* pPort) { + *pPort = taosStr2Int32(p, NULL, 10); + if (*pPort >= UINT16_MAX || *pPort <= 0) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_PORT); } - return TSDB_CODE_SUCCESS == pCxt->errCode; + return TSDB_CODE_SUCCESS; } -static bool checkFqdn(SAstCreateContext* pCxt, const SToken* pFqdn) { - if (NULL == pFqdn) { - pCxt->errCode = TSDB_CODE_PAR_SYNTAX_ERROR; - } else { - if (pFqdn->n >= TSDB_FQDN_LEN) { - pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NAME_OR_PASSWD_TOO_LONG); - } +static int32_t parseEndpoint(SAstCreateContext* pCxt, const SToken* pEp, char* pFqdn, int32_t* pPort) { + if (pEp->n >= (NULL == pPort ? (TSDB_FQDN_LEN + 1 + 5) : TSDB_FQDN_LEN)) { // format 'fqdn:port' or 'fqdn' + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NAME_OR_PASSWD_TOO_LONG); } - return TSDB_CODE_SUCCESS == pCxt->errCode; + + char ep[TSDB_FQDN_LEN + 1 + 5] = {0}; + COPY_STRING_FORM_ID_TOKEN(ep, pEp); + strdequote(ep); + strtrim(ep); + if (NULL == pPort) { + strcpy(pFqdn, ep); + return TSDB_CODE_SUCCESS; + } + char* pColon = strchr(ep, ':'); + if (NULL == pColon) { + *pPort = tsServerPort; + strcpy(pFqdn, ep); + return TSDB_CODE_SUCCESS; + } + strncpy(pFqdn, ep, pColon - ep); + return parsePort(pCxt, pColon + 1, pPort); } -static bool checkPort(SAstCreateContext* pCxt, const SToken* pPortToken, int32_t* pPort) { - if (NULL == pPortToken) { +static bool checkAndSplitEndpoint(SAstCreateContext* pCxt, const SToken* pEp, const SToken* pPortToken, char* pFqdn, + int32_t* pPort) { + if (NULL == pEp) { pCxt->errCode = TSDB_CODE_PAR_SYNTAX_ERROR; - } else { - *pPort = taosStr2Int32(pPortToken->z, NULL, 10); - if (*pPort >= UINT16_MAX || *pPort <= 0) { - pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_PORT); - } + return false; } + + if (NULL != pPortToken) { + pCxt->errCode = parsePort(pCxt, pPortToken->z, pPort); + } + + if (TSDB_CODE_SUCCESS == pCxt->errCode) { + pCxt->errCode = parseEndpoint(pCxt, pEp, pFqdn, (NULL != pPortToken ? NULL : pPort)); + } + return TSDB_CODE_SUCCESS == pCxt->errCode; } @@ -274,9 +279,9 @@ SNode* createColumnNode(SAstCreateContext* pCxt, SToken* pTableAlias, SToken* pC SColumnNode* col = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN); CHECK_OUT_OF_MEM(col); if (NULL != pTableAlias) { - strncpy(col->tableAlias, pTableAlias->z, pTableAlias->n); + COPY_STRING_FORM_ID_TOKEN(col->tableAlias, pTableAlias); } - strncpy(col->colName, pColumnName->z, pColumnName->n); + COPY_STRING_FORM_ID_TOKEN(col->colName, pColumnName); return (SNode*)col; } @@ -417,7 +422,7 @@ SNode* createFunctionNode(SAstCreateContext* pCxt, const SToken* pFuncName, SNod } SFunctionNode* func = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION); CHECK_OUT_OF_MEM(func); - strncpy(func->functionName, pFuncName->z, pFuncName->n); + COPY_STRING_FORM_ID_TOKEN(func->functionName, pFuncName); func->pParameterList = pParameterList; return (SNode*)func; } @@ -464,16 +469,16 @@ SNode* createRealTableNode(SAstCreateContext* pCxt, SToken* pDbName, SToken* pTa SRealTableNode* realTable = (SRealTableNode*)nodesMakeNode(QUERY_NODE_REAL_TABLE); CHECK_OUT_OF_MEM(realTable); if (NULL != pDbName) { - strncpy(realTable->table.dbName, pDbName->z, pDbName->n); + COPY_STRING_FORM_ID_TOKEN(realTable->table.dbName, pDbName); } else { strcpy(realTable->table.dbName, pCxt->pQueryCxt->db); } if (NULL != pTableAlias && TK_NK_NIL != pTableAlias->type) { - strncpy(realTable->table.tableAlias, pTableAlias->z, pTableAlias->n); + COPY_STRING_FORM_ID_TOKEN(realTable->table.tableAlias, pTableAlias); } else { - strncpy(realTable->table.tableAlias, pTableName->z, pTableName->n); + COPY_STRING_FORM_ID_TOKEN(realTable->table.tableAlias, pTableName); } - strncpy(realTable->table.tableName, pTableName->z, pTableName->n); + COPY_STRING_FORM_ID_TOKEN(realTable->table.tableName, pTableName); return (SNode*)realTable; } @@ -483,7 +488,7 @@ SNode* createTempTableNode(SAstCreateContext* pCxt, SNode* pSubquery, const STok CHECK_OUT_OF_MEM(tempTable); tempTable->pSubquery = pSubquery; if (NULL != pTableAlias && TK_NK_NIL != pTableAlias->type) { - strncpy(tempTable->table.tableAlias, pTableAlias->z, pTableAlias->n); + COPY_STRING_FORM_ID_TOKEN(tempTable->table.tableAlias, pTableAlias); } else { sprintf(tempTable->table.tableAlias, "%p", tempTable); } @@ -594,6 +599,11 @@ SNode* createGroupingSetNode(SAstCreateContext* pCxt, SNode* pNode) { return (SNode*)groupingSet; } +SNode* createInterpTimeRange(SAstCreateContext* pCxt, SNode* pStart, SNode* pEnd) { + CHECK_PARSER_STATUS(pCxt); + return createBetweenAnd(pCxt, createPrimaryKeyCol(pCxt), pStart, pEnd); +} + SNode* setProjectionAlias(SAstCreateContext* pCxt, SNode* pNode, const SToken* pAlias) { CHECK_PARSER_STATUS(pCxt); int32_t len = TMIN(sizeof(((SExprNode*)pNode)->aliasName) - 1, pAlias->n); @@ -648,6 +658,8 @@ SNode* addOrderByClause(SAstCreateContext* pCxt, SNode* pStmt, SNodeList* pOrder CHECK_PARSER_STATUS(pCxt); if (QUERY_NODE_SELECT_STMT == nodeType(pStmt)) { ((SSelectStmt*)pStmt)->pOrderByList = pOrderByList; + } else { + ((SSetOperator*)pStmt)->pOrderByList = pOrderByList; } return pStmt; } @@ -664,6 +676,35 @@ SNode* addLimitClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pLimit) { CHECK_PARSER_STATUS(pCxt); if (QUERY_NODE_SELECT_STMT == nodeType(pStmt)) { ((SSelectStmt*)pStmt)->pLimit = (SLimitNode*)pLimit; + } else { + ((SSetOperator*)pStmt)->pLimit = pLimit; + } + return pStmt; +} + +SNode* addRangeClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pRange) { + CHECK_PARSER_STATUS(pCxt); + if (QUERY_NODE_SELECT_STMT == nodeType(pStmt)) { + ((SSelectStmt*)pStmt)->pRange = pRange; + } + return pStmt; +} + +SNode* addEveryClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pEvery) { + CHECK_PARSER_STATUS(pCxt); + if (QUERY_NODE_SELECT_STMT == nodeType(pStmt)) { + ((SSelectStmt*)pStmt)->pEvery = pEvery; + } + return pStmt; +} + +SNode* addFillClause(SAstCreateContext* pCxt, SNode* pStmt, SNode* pFill) { + CHECK_PARSER_STATUS(pCxt); + if (QUERY_NODE_SELECT_STMT == nodeType(pStmt) && NULL != pFill) { + SFillNode* pFillClause = (SFillNode*)pFill; + nodesDestroyNode(pFillClause->pWStartTs); + pFillClause->pWStartTs = createPrimaryKeyCol(pCxt); + ((SSelectStmt*)pStmt)->pFill = (SNode*)pFillClause; } return pStmt; } @@ -783,8 +824,7 @@ SNode* setDatabaseOption(SAstCreateContext* pCxt, SNode* pOptions, EDatabaseOpti ((SDatabaseOptions*)pOptions)->pagesize = taosStr2Int32(((SToken*)pVal)->z, NULL, 10); break; case DB_OPTION_PRECISION: - copyStringFormStringToken((SToken*)pVal, ((SDatabaseOptions*)pOptions)->precisionStr, - sizeof(((SDatabaseOptions*)pOptions)->precisionStr)); + COPY_STRING_FORM_STR_TOKEN(((SDatabaseOptions*)pOptions)->precisionStr, (SToken*)pVal); break; case DB_OPTION_REPLICA: ((SDatabaseOptions*)pOptions)->replica = taosStr2Int8(((SToken*)pVal)->z, NULL, 10); @@ -833,7 +873,7 @@ SNode* createCreateDatabaseStmt(SAstCreateContext* pCxt, bool ignoreExists, STok } SCreateDatabaseStmt* pStmt = (SCreateDatabaseStmt*)nodesMakeNode(QUERY_NODE_CREATE_DATABASE_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->dbName, pDbName->z, pDbName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); pStmt->ignoreExists = ignoreExists; pStmt->pOptions = (SDatabaseOptions*)pOptions; return (SNode*)pStmt; @@ -846,7 +886,7 @@ SNode* createDropDatabaseStmt(SAstCreateContext* pCxt, bool ignoreNotExists, STo } SDropDatabaseStmt* pStmt = (SDropDatabaseStmt*)nodesMakeNode(QUERY_NODE_DROP_DATABASE_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->dbName, pDbName->z, pDbName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); pStmt->ignoreNotExists = ignoreNotExists; return (SNode*)pStmt; } @@ -858,7 +898,7 @@ SNode* createAlterDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName, SNode* } SAlterDatabaseStmt* pStmt = (SAlterDatabaseStmt*)nodesMakeNode(QUERY_NODE_ALTER_DATABASE_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->dbName, pDbName->z, pDbName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); pStmt->pOptions = (SDatabaseOptions*)pOptions; return (SNode*)pStmt; } @@ -867,9 +907,12 @@ SNode* createDefaultTableOptions(SAstCreateContext* pCxt) { CHECK_PARSER_STATUS(pCxt); STableOptions* pOptions = (STableOptions*)nodesMakeNode(QUERY_NODE_TABLE_OPTIONS); CHECK_OUT_OF_MEM(pOptions); - // pOptions->delay = TSDB_DEFAULT_ROLLUP_DELAY; - pOptions->filesFactor = TSDB_DEFAULT_ROLLUP_FILE_FACTOR; + pOptions->maxDelay1 = -1; + pOptions->maxDelay2 = -1; + pOptions->watermark1 = TSDB_DEFAULT_ROLLUP_WATERMARK; + pOptions->watermark2 = TSDB_DEFAULT_ROLLUP_WATERMARK; pOptions->ttl = TSDB_DEFAULT_TABLE_TTL; + pOptions->commentNull = true; // mark null return (SNode*)pOptions; } @@ -877,8 +920,8 @@ SNode* createAlterTableOptions(SAstCreateContext* pCxt) { CHECK_PARSER_STATUS(pCxt); STableOptions* pOptions = (STableOptions*)nodesMakeNode(QUERY_NODE_TABLE_OPTIONS); CHECK_OUT_OF_MEM(pOptions); - pOptions->delay = -1; pOptions->ttl = -1; + pOptions->commentNull = true; // mark null return (SNode*)pOptions; } @@ -887,19 +930,28 @@ SNode* setTableOption(SAstCreateContext* pCxt, SNode* pOptions, ETableOptionType switch (type) { case TABLE_OPTION_COMMENT: if (checkComment(pCxt, (SToken*)pVal, true)) { - copyStringFormStringToken((SToken*)pVal, ((STableOptions*)pOptions)->comment, - sizeof(((STableOptions*)pOptions)->comment)); + ((STableOptions*)pOptions)->commentNull = false; + COPY_STRING_FORM_STR_TOKEN(((STableOptions*)pOptions)->comment, (SToken*)pVal); } break; - case TABLE_OPTION_DELAY: - ((STableOptions*)pOptions)->delay = taosStr2Int32(((SToken*)pVal)->z, NULL, 10); + case TABLE_OPTION_MAXDELAY: + ((STableOptions*)pOptions)->pMaxDelay = pVal; + break; + case TABLE_OPTION_WATERMARK: + ((STableOptions*)pOptions)->pWatermark = pVal; break; case TABLE_OPTION_ROLLUP: ((STableOptions*)pOptions)->pRollupFuncs = pVal; break; - case TABLE_OPTION_TTL: - ((STableOptions*)pOptions)->ttl = taosStr2Int32(((SToken*)pVal)->z, NULL, 10); + case TABLE_OPTION_TTL: { + int64_t ttl = taosStr2Int64(((SToken*)pVal)->z, NULL, 10); + if (ttl > INT32_MAX) { + ttl = INT32_MAX; + } + // ttl can not be smaller than 0, because there is a limitation in sql.y (TTL NK_INTEGER) + ((STableOptions*)pOptions)->ttl = ttl; break; + } case TABLE_OPTION_SMA: ((STableOptions*)pOptions)->pSma = pVal; break; @@ -916,7 +968,7 @@ SNode* createColumnDefNode(SAstCreateContext* pCxt, SToken* pColName, SDataType } SColumnDefNode* pCol = (SColumnDefNode*)nodesMakeNode(QUERY_NODE_COLUMN_DEF); CHECK_OUT_OF_MEM(pCol); - strncpy(pCol->colName, pColName->z, pColName->n); + COPY_STRING_FORM_ID_TOKEN(pCol->colName, pColName); pCol->dataType = dataType; if (NULL != pComment) { trimString(pComment->z, pComment->n, pCol->comments, sizeof(pCol->comments)); @@ -962,9 +1014,9 @@ SNode* createCreateSubTableClause(SAstCreateContext* pCxt, bool ignoreExists, SN pStmt->ignoreExists = ignoreExists; pStmt->pSpecificTags = pSpecificTags; pStmt->pValsOfTags = pValsOfTags; + pStmt->pOptions = (STableOptions*)pOptions; nodesDestroyNode(pRealTable); nodesDestroyNode(pUseRealTable); - nodesDestroyNode(pOptions); return (SNode*)pStmt; } @@ -1031,7 +1083,7 @@ SNode* createAlterTableAddModifyCol(SAstCreateContext* pCxt, SNode* pRealTable, SAlterTableStmt* pStmt = (SAlterTableStmt*)nodesMakeNode(QUERY_NODE_ALTER_TABLE_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->alterType = alterType; - strncpy(pStmt->colName, pColName->z, pColName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->colName, pColName); pStmt->dataType = dataType; return createAlterTableStmtFinalize(pRealTable, pStmt); } @@ -1044,7 +1096,7 @@ SNode* createAlterTableDropCol(SAstCreateContext* pCxt, SNode* pRealTable, int8_ SAlterTableStmt* pStmt = (SAlterTableStmt*)nodesMakeNode(QUERY_NODE_ALTER_TABLE_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->alterType = alterType; - strncpy(pStmt->colName, pColName->z, pColName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->colName, pColName); return createAlterTableStmtFinalize(pRealTable, pStmt); } @@ -1057,8 +1109,8 @@ SNode* createAlterTableRenameCol(SAstCreateContext* pCxt, SNode* pRealTable, int SAlterTableStmt* pStmt = (SAlterTableStmt*)nodesMakeNode(QUERY_NODE_ALTER_TABLE_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->alterType = alterType; - strncpy(pStmt->colName, pOldColName->z, pOldColName->n); - strncpy(pStmt->newColName, pNewColName->z, pNewColName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->colName, pOldColName); + COPY_STRING_FORM_ID_TOKEN(pStmt->newColName, pNewColName); return createAlterTableStmtFinalize(pRealTable, pStmt); } @@ -1070,7 +1122,7 @@ SNode* createAlterTableSetTag(SAstCreateContext* pCxt, SNode* pRealTable, SToken SAlterTableStmt* pStmt = (SAlterTableStmt*)nodesMakeNode(QUERY_NODE_ALTER_TABLE_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->alterType = TSDB_ALTER_TABLE_UPDATE_TAG_VAL; - strncpy(pStmt->colName, pTagName->z, pTagName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->colName, pTagName); pStmt->pVal = (SValueNode*)pVal; return createAlterTableStmtFinalize(pRealTable, pStmt); } @@ -1082,7 +1134,7 @@ SNode* createUseDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName) { } SUseDatabaseStmt* pStmt = (SUseDatabaseStmt*)nodesMakeNode(QUERY_NODE_USE_DATABASE_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->dbName, pDbName->z, pDbName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); return (SNode*)pStmt; } @@ -1091,7 +1143,15 @@ static bool needDbShowStmt(ENodeType type) { QUERY_NODE_SHOW_VGROUPS_STMT == type; } -SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, SNode* pTbNamePattern) { +SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type) { + CHECK_PARSER_STATUS(pCxt); + SShowStmt* pStmt = (SShowStmt*)nodesMakeNode(type); + CHECK_OUT_OF_MEM(pStmt); + return (SNode*)pStmt; +} + +SNode* createShowStmtWithCond(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, SNode* pTbName, + EOperatorType tableCondType) { CHECK_PARSER_STATUS(pCxt); if (needDbShowStmt(type) && NULL == pDbName && NULL == pCxt->pQueryCxt->db) { snprintf(pCxt->pQueryCxt->pMsg, pCxt->pQueryCxt->msgLen, "db not specified"); @@ -1101,22 +1161,40 @@ SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, S SShowStmt* pStmt = (SShowStmt*)nodesMakeNode(type); CHECK_OUT_OF_MEM(pStmt); pStmt->pDbName = pDbName; - pStmt->pTbNamePattern = pTbNamePattern; + pStmt->pTbName = pTbName; + pStmt->tableCondType = tableCondType; return (SNode*)pStmt; } -SNode* createShowCreateDatabaseStmt(SAstCreateContext* pCxt, const SToken* pDbName) { +SNode* createShowCreateDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName) { CHECK_PARSER_STATUS(pCxt); - SNode* pStmt = nodesMakeNode(QUERY_NODE_SHOW_CREATE_DATABASE_STMT); + if (!checkDbName(pCxt, pDbName, true)) { + return NULL; + } + SShowCreateDatabaseStmt* pStmt = (SShowCreateDatabaseStmt*)nodesMakeNode(QUERY_NODE_SHOW_CREATE_DATABASE_STMT); CHECK_OUT_OF_MEM(pStmt); - return pStmt; + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); + return (SNode*)pStmt; } SNode* createShowCreateTableStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pRealTable) { CHECK_PARSER_STATUS(pCxt); - SNode* pStmt = nodesMakeNode(type); + SShowCreateTableStmt* pStmt = (SShowCreateTableStmt*)nodesMakeNode(type); CHECK_OUT_OF_MEM(pStmt); - return pStmt; + strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName); + strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName); + nodesDestroyNode(pRealTable); + return (SNode*)pStmt; +} + +SNode* createShowTableDistributedStmt(SAstCreateContext* pCxt, SNode* pRealTable) { + CHECK_PARSER_STATUS(pCxt); + SShowTableDistributedStmt* pStmt = (SShowTableDistributedStmt*)nodesMakeNode(QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT); + CHECK_OUT_OF_MEM(pStmt); + strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName); + strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName); + nodesDestroyNode(pRealTable); + return (SNode*)pStmt; } SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const SToken* pPassword) { @@ -1127,7 +1205,7 @@ SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const ST } SCreateUserStmt* pStmt = (SCreateUserStmt*)nodesMakeNode(QUERY_NODE_CREATE_USER_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->useName, pUserName->z, pUserName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->useName, pUserName); strcpy(pStmt->password, password); return (SNode*)pStmt; } @@ -1139,7 +1217,7 @@ SNode* createAlterUserStmt(SAstCreateContext* pCxt, SToken* pUserName, int8_t al } SAlterUserStmt* pStmt = (SAlterUserStmt*)nodesMakeNode(QUERY_NODE_ALTER_USER_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->useName, pUserName->z, pUserName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->useName, pUserName); if (TSDB_ALTER_USER_PASSWD == alterType) { char password[TSDB_USET_PASSWORD_LEN] = {0}; if (!checkPassword(pCxt, pVal, password)) { @@ -1159,29 +1237,18 @@ SNode* createDropUserStmt(SAstCreateContext* pCxt, SToken* pUserName) { } SDropUserStmt* pStmt = (SDropUserStmt*)nodesMakeNode(QUERY_NODE_DROP_USER_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->useName, pUserName->z, pUserName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->useName, pUserName); return (SNode*)pStmt; } SNode* createCreateDnodeStmt(SAstCreateContext* pCxt, const SToken* pFqdn, const SToken* pPort) { CHECK_PARSER_STATUS(pCxt); - int32_t port = 0; - char fqdn[TSDB_FQDN_LEN] = {0}; - if (NULL == pPort) { - if (!checkAndSplitEndpoint(pCxt, pFqdn, fqdn, &port)) { - return NULL; - } - } else if (!checkFqdn(pCxt, pFqdn) || !checkPort(pCxt, pPort, &port)) { - return NULL; - } SCreateDnodeStmt* pStmt = (SCreateDnodeStmt*)nodesMakeNode(QUERY_NODE_CREATE_DNODE_STMT); CHECK_OUT_OF_MEM(pStmt); - if (NULL == pPort) { - strcpy(pStmt->fqdn, fqdn); - } else { - strncpy(pStmt->fqdn, pFqdn->z, pFqdn->n); + if (!checkAndSplitEndpoint(pCxt, pFqdn, pPort, pStmt->fqdn, &pStmt->port)) { + nodesDestroyNode((SNode*)pStmt); + return NULL; } - pStmt->port = port; return (SNode*)pStmt; } @@ -1192,7 +1259,7 @@ SNode* createDropDnodeStmt(SAstCreateContext* pCxt, const SToken* pDnode) { if (TK_NK_INTEGER == pDnode->type) { pStmt->dnodeId = taosStr2Int32(pDnode->z, NULL, 10); } else { - if (!checkAndSplitEndpoint(pCxt, pDnode, pStmt->fqdn, &pStmt->port)) { + if (!checkAndSplitEndpoint(pCxt, pDnode, NULL, pStmt->fqdn, &pStmt->port)) { nodesDestroyNode((SNode*)pStmt); return NULL; } @@ -1223,8 +1290,8 @@ SNode* createCreateIndexStmt(SAstCreateContext* pCxt, EIndexType type, bool igno CHECK_OUT_OF_MEM(pStmt); pStmt->indexType = type; pStmt->ignoreExists = ignoreExists; - strncpy(pStmt->indexName, pIndexName->z, pIndexName->n); - strncpy(pStmt->tableName, pTableName->z, pTableName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->indexName, pIndexName); + COPY_STRING_FORM_ID_TOKEN(pStmt->tableName, pTableName); pStmt->pCols = pCols; pStmt->pOptions = (SIndexOptions*)pOptions; return (SNode*)pStmt; @@ -1250,8 +1317,8 @@ SNode* createDropIndexStmt(SAstCreateContext* pCxt, bool ignoreNotExists, SToken SDropIndexStmt* pStmt = (SDropIndexStmt*)nodesMakeNode(QUERY_NODE_DROP_INDEX_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->ignoreNotExists = ignoreNotExists; - strncpy(pStmt->indexName, pIndexName->z, pIndexName->n); - strncpy(pStmt->tableName, pTableName->z, pTableName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->indexName, pIndexName); + COPY_STRING_FORM_ID_TOKEN(pStmt->tableName, pTableName); return (SNode*)pStmt; } @@ -1276,14 +1343,14 @@ SNode* createCreateTopicStmt(SAstCreateContext* pCxt, bool ignoreExists, const S CHECK_PARSER_STATUS(pCxt); SCreateTopicStmt* pStmt = (SCreateTopicStmt*)nodesMakeNode(QUERY_NODE_CREATE_TOPIC_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->topicName, pTopicName->z, pTopicName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->topicName, pTopicName); pStmt->ignoreExists = ignoreExists; if (NULL != pRealTable) { strcpy(pStmt->subDbName, ((SRealTableNode*)pRealTable)->table.dbName); strcpy(pStmt->subSTbName, ((SRealTableNode*)pRealTable)->table.tableName); nodesDestroyNode(pRealTable); } else if (NULL != pSubDbName) { - strncpy(pStmt->subDbName, pSubDbName->z, pSubDbName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->subDbName, pSubDbName); } else { pStmt->pQuery = pQuery; } @@ -1294,7 +1361,7 @@ SNode* createDropTopicStmt(SAstCreateContext* pCxt, bool ignoreNotExists, const CHECK_PARSER_STATUS(pCxt); SDropTopicStmt* pStmt = (SDropTopicStmt*)nodesMakeNode(QUERY_NODE_DROP_TOPIC_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->topicName, pTopicName->z, pTopicName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->topicName, pTopicName); pStmt->ignoreNotExists = ignoreNotExists; return (SNode*)pStmt; } @@ -1305,8 +1372,8 @@ SNode* createDropCGroupStmt(SAstCreateContext* pCxt, bool ignoreNotExists, const SDropCGroupStmt* pStmt = (SDropCGroupStmt*)nodesMakeNode(QUERY_NODE_DROP_CGROUP_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->ignoreNotExists = ignoreNotExists; - strncpy(pStmt->topicName, pTopicName->z, pTopicName->n); - strncpy(pStmt->cgroup, pCGroupId->z, pCGroupId->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->topicName, pTopicName); + COPY_STRING_FORM_ID_TOKEN(pStmt->cgroup, pCGroupId); return (SNode*)pStmt; } @@ -1386,9 +1453,9 @@ SNode* createCreateFunctionStmt(SAstCreateContext* pCxt, bool ignoreExists, bool SCreateFunctionStmt* pStmt = (SCreateFunctionStmt*)nodesMakeNode(QUERY_NODE_CREATE_FUNCTION_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->ignoreExists = ignoreExists; - strncpy(pStmt->funcName, pFuncName->z, pFuncName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->funcName, pFuncName); pStmt->isAgg = aggFunc; - strncpy(pStmt->libraryPath, pLibPath->z + 1, pLibPath->n - 2); + COPY_STRING_FORM_STR_TOKEN(pStmt->libraryPath, pLibPath); pStmt->outputDt = dataType; pStmt->bufSize = bufSize; return (SNode*)pStmt; @@ -1399,7 +1466,7 @@ SNode* createDropFunctionStmt(SAstCreateContext* pCxt, bool ignoreNotExists, con SDropFunctionStmt* pStmt = (SDropFunctionStmt*)nodesMakeNode(QUERY_NODE_DROP_FUNCTION_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->ignoreNotExists = ignoreNotExists; - strncpy(pStmt->funcName, pFuncName->z, pFuncName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->funcName, pFuncName); return (SNode*)pStmt; } @@ -1416,7 +1483,7 @@ SNode* createCreateStreamStmt(SAstCreateContext* pCxt, bool ignoreExists, const CHECK_PARSER_STATUS(pCxt); SCreateStreamStmt* pStmt = (SCreateStreamStmt*)nodesMakeNode(QUERY_NODE_CREATE_STREAM_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->streamName, pStreamName->z, pStreamName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->streamName, pStreamName); if (NULL != pRealTable) { strcpy(pStmt->targetDbName, ((SRealTableNode*)pRealTable)->table.dbName); strcpy(pStmt->targetTabName, ((SRealTableNode*)pRealTable)->table.tableName); @@ -1432,7 +1499,7 @@ SNode* createDropStreamStmt(SAstCreateContext* pCxt, bool ignoreNotExists, const CHECK_PARSER_STATUS(pCxt); SDropStreamStmt* pStmt = (SDropStreamStmt*)nodesMakeNode(QUERY_NODE_DROP_STREAM_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->streamName, pStreamName->z, TMIN(pStreamName->n, sizeof(pStmt->streamName) - 1)); + COPY_STRING_FORM_ID_TOKEN(pStmt->streamName, pStreamName); pStmt->ignoreNotExists = ignoreNotExists; return (SNode*)pStmt; } @@ -1449,7 +1516,7 @@ SNode* createKillQueryStmt(SAstCreateContext* pCxt, const SToken* pQueryId) { CHECK_PARSER_STATUS(pCxt); SKillQueryStmt* pStmt = (SKillQueryStmt*)nodesMakeNode(QUERY_NODE_KILL_QUERY_STMT); CHECK_OUT_OF_MEM(pStmt); - strncpy(pStmt->queryId, pQueryId->z, TMIN(pQueryId->n, sizeof(pStmt->queryId) - 1)); + trimString(pQueryId->z, pQueryId->n, pStmt->queryId, sizeof(pStmt->queryId) - 1); return (SNode*)pStmt; } @@ -1501,8 +1568,8 @@ SNode* createGrantStmt(SAstCreateContext* pCxt, int64_t privileges, SToken* pDbN SGrantStmt* pStmt = (SGrantStmt*)nodesMakeNode(QUERY_NODE_GRANT_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->privileges = privileges; - strncpy(pStmt->dbName, pDbName->z, pDbName->n); - strncpy(pStmt->userName, pUserName->z, pUserName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); + COPY_STRING_FORM_ID_TOKEN(pStmt->userName, pUserName); return (SNode*)pStmt; } @@ -1514,8 +1581,8 @@ SNode* createRevokeStmt(SAstCreateContext* pCxt, int64_t privileges, SToken* pDb SRevokeStmt* pStmt = (SRevokeStmt*)nodesMakeNode(QUERY_NODE_REVOKE_STMT); CHECK_OUT_OF_MEM(pStmt); pStmt->privileges = privileges; - strncpy(pStmt->dbName, pDbName->z, pDbName->n); - strncpy(pStmt->userName, pUserName->z, pUserName->n); + COPY_STRING_FORM_ID_TOKEN(pStmt->dbName, pDbName); + COPY_STRING_FORM_ID_TOKEN(pStmt->userName, pUserName); return (SNode*)pStmt; } diff --git a/source/libs/parser/src/parAstParser.c b/source/libs/parser/src/parAstParser.c index 1aa1320a3c3c4ed291792b450148b870606f71ec..60589784511dbe2eab5ab16906f5149e7e2bd5da 100644 --- a/source/libs/parser/src/parAstParser.c +++ b/source/libs/parser/src/parAstParser.c @@ -387,6 +387,29 @@ static int32_t collectMetaKeyFromShowQueries(SCollectMetaKeyCxt* pCxt, SShowStmt pCxt->pMetaCache); } +static int32_t collectMetaKeyFromShowConfigs(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) { + return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_CONFIGS, + pCxt->pMetaCache); +} + +static int32_t collectMetaKeyFromShowVariables(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) { + return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_CONFIGS, + pCxt->pMetaCache); +} + +static int32_t collectMetaKeyFromShowCreateDatabase(SCollectMetaKeyCxt* pCxt, SShowCreateDatabaseStmt* pStmt) { + return reserveDbCfgInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pCxt->pMetaCache); +} + +static int32_t collectMetaKeyFromShowCreateTable(SCollectMetaKeyCxt* pCxt, SShowCreateTableStmt* pStmt) { + return reserveTableMetaInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache); +} + +static int32_t collectMetaKeyFromShowApps(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) { + return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_PERFORMANCE_SCHEMA_DB, TSDB_PERFS_TABLE_APPS, + pCxt->pMetaCache); +} + static int32_t collectMetaKeyFromShowTransactions(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) { return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_PERFORMANCE_SCHEMA_DB, TSDB_PERFS_TABLE_TRANS, pCxt->pMetaCache); @@ -396,6 +419,24 @@ static int32_t collectMetaKeyFromDelete(SCollectMetaKeyCxt* pCxt, SDeleteStmt* p return collectMetaKeyFromRealTableImpl(pCxt, (SRealTableNode*)pStmt->pFromTable, AUTH_TYPE_WRITE); } +static int32_t collectMetaKeyFromShowBlockDist(SCollectMetaKeyCxt* pCxt, SShowTableDistributedStmt* pStmt) { + SName name = {.type = TSDB_TABLE_NAME_T, .acctId = pCxt->pParseCxt->acctId}; + strcpy(name.dbname, pStmt->dbName); + strcpy(name.tname, pStmt->tableName); + int32_t code = catalogRemoveTableMeta(pCxt->pParseCxt->pCatalog, &name); + if (TSDB_CODE_SUCCESS == code) { + code = reserveTableMetaInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache); + } + + if (TSDB_CODE_SUCCESS == code) { + code = reserveTableVgroupInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache); + } + if (TSDB_CODE_SUCCESS == code) { + code = reserveDbVgInfoInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pCxt->pMetaCache); + } + return code; +} + static int32_t collectMetaKeyFromQuery(SCollectMetaKeyCxt* pCxt, SNode* pStmt) { pCxt->pStmt = pStmt; switch (nodeType(pStmt)) { @@ -461,10 +502,21 @@ static int32_t collectMetaKeyFromQuery(SCollectMetaKeyCxt* pCxt, SNode* pStmt) { return collectMetaKeyFromShowConnections(pCxt, (SShowStmt*)pStmt); case QUERY_NODE_SHOW_QUERIES_STMT: return collectMetaKeyFromShowQueries(pCxt, (SShowStmt*)pStmt); + case QUERY_NODE_SHOW_VARIABLE_STMT: + return collectMetaKeyFromShowVariables(pCxt, (SShowStmt*)pStmt); + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + return collectMetaKeyFromShowCreateDatabase(pCxt, (SShowCreateDatabaseStmt*)pStmt); + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: + return collectMetaKeyFromShowCreateTable(pCxt, (SShowCreateTableStmt*)pStmt); + case QUERY_NODE_SHOW_APPS_STMT: + return collectMetaKeyFromShowApps(pCxt, (SShowStmt*)pStmt); case QUERY_NODE_SHOW_TRANSACTIONS_STMT: return collectMetaKeyFromShowTransactions(pCxt, (SShowStmt*)pStmt); case QUERY_NODE_DELETE_STMT: return collectMetaKeyFromDelete(pCxt, (SDeleteStmt*)pStmt); + case QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT: + return collectMetaKeyFromShowBlockDist(pCxt, (SShowTableDistributedStmt*)pStmt); default: break; } diff --git a/source/libs/parser/src/parInsert.c b/source/libs/parser/src/parInsert.c index 626960bf100f8fdce7b8efbbc1ba029790e4719f..c8b78fcc8bf6acfc242a975014ef2d11b38c4d23 100644 --- a/source/libs/parser/src/parInsert.c +++ b/source/libs/parser/src/parInsert.c @@ -803,6 +803,7 @@ static void buildCreateTbReq(SVCreateTbReq* pTbReq, const char* tname, STag* pTa pTbReq->name = strdup(tname); pTbReq->ctb.suid = suid; pTbReq->ctb.pTag = (uint8_t*)pTag; + pTbReq->commentLen = -1; return; } diff --git a/source/libs/parser/src/parTokenizer.c b/source/libs/parser/src/parTokenizer.c index 55ea7171fa41c185453ade03655d0d70c936f72b..21f227b7eac945d5e2ff6dafc5c90fa211df3200 100644 --- a/source/libs/parser/src/parTokenizer.c +++ b/source/libs/parser/src/parTokenizer.c @@ -68,12 +68,12 @@ static SKeyword keywordTable[] = { {"CONTAINS", TK_CONTAINS}, {"DATABASE", TK_DATABASE}, {"DATABASES", TK_DATABASES}, - // {"DAYS", TK_DAYS}, {"DBS", TK_DBS}, {"DELETE", TK_DELETE}, {"DESC", TK_DESC}, {"DESCRIBE", TK_DESCRIBE}, {"DISTINCT", TK_DISTINCT}, + {"DISTRIBUTED", TK_DISTRIBUTED}, {"DNODE", TK_DNODE}, {"DNODES", TK_DNODES}, {"DOUBLE", TK_DOUBLE}, @@ -81,7 +81,7 @@ static SKeyword keywordTable[] = { {"DURATION", TK_DURATION}, {"EXISTS", TK_EXISTS}, {"EXPLAIN", TK_EXPLAIN}, - {"FILE_FACTOR", TK_FILE_FACTOR}, + {"EVERY", TK_EVERY}, {"FILL", TK_FILL}, {"FIRST", TK_FIRST}, {"FLOAT", TK_FLOAT}, @@ -153,6 +153,7 @@ static SKeyword keywordTable[] = { {"QTIME", TK_QTIME}, {"QUERIES", TK_QUERIES}, {"QUERY", TK_QUERY}, + {"RANGE", TK_RANGE}, {"RATIO", TK_RATIO}, {"READ", TK_READ}, {"REDISTRIBUTE", TK_REDISTRIBUTE}, @@ -259,7 +260,6 @@ static SKeyword keywordTable[] = { // {"LP", TK_LP}, // {"RP", TK_RP}, // {"COMMA", TK_COMMA}, - // {"EVERY", TK_EVERY}, // {"VARIABLE", TK_VARIABLE}, // {"UPDATE", TK_UPDATE}, // {"CHANGE", TK_CHANGE}, diff --git a/source/libs/parser/src/parTranslater.c b/source/libs/parser/src/parTranslater.c index 5d26a98dc868d80f3230dfd5595e0af377e55ece..a60dba3a9a46a81a0a20dd98397dc83593e59468 100644 --- a/source/libs/parser/src/parTranslater.c +++ b/source/libs/parser/src/parTranslater.c @@ -36,6 +36,7 @@ typedef struct STranslateContext { int32_t currLevel; ESqlClause currClause; SSelectStmt* pCurrSelectStmt; + SSetOperator* pCurrSetOperator; SCmdMsgInfo* pCmdMsg; SHashObj* pDbs; SHashObj* pTables; @@ -432,7 +433,7 @@ static void setColumnInfoBySchema(const SRealTableNode* pTable, const SSchema* p static void setColumnInfoByExpr(const STableNode* pTable, SExprNode* pExpr, SColumnNode** pColRef) { SColumnNode* pCol = *pColRef; - pCol->pProjectRef = (SNode*)pExpr; + // pCol->pProjectRef = (SNode*)pExpr; if (NULL == pExpr->pAssociation) { pExpr->pAssociation = taosArrayInit(TARRAY_MIN_SIZE, POINTER_BYTES); } @@ -613,17 +614,36 @@ static EDealRes translateColumnWithoutPrefix(STranslateContext* pCxt, SColumnNod return DEAL_RES_CONTINUE; } -static bool translateColumnUseAlias(STranslateContext* pCxt, SColumnNode** pCol) { - SNodeList* pProjectionList = pCxt->pCurrSelectStmt->pProjectionList; +static SNodeList* getProjectListFromCxt(STranslateContext* pCxt) { + if (NULL != pCxt->pCurrSelectStmt) { + return pCxt->pCurrSelectStmt->pProjectionList; + } else if (NULL != pCxt->pCurrSetOperator) { + return pCxt->pCurrSetOperator->pProjectionList; + } else { + return NULL; + } +} + +static EDealRes translateColumnUseAlias(STranslateContext* pCxt, SColumnNode** pCol, bool* pFound) { + SNodeList* pProjectionList = getProjectListFromCxt(pCxt); SNode* pNode; FOREACH(pNode, pProjectionList) { SExprNode* pExpr = (SExprNode*)pNode; if (0 == strcmp((*pCol)->colName, pExpr->aliasName)) { - setColumnInfoByExpr(NULL, pExpr, pCol); - return true; + SColumnRefNode* pColRef = (SColumnRefNode*)nodesMakeNode(QUERY_NODE_COLUMN_REF); + if (NULL == pColRef) { + pCxt->errCode = TSDB_CODE_OUT_OF_MEMORY; + return DEAL_RES_ERROR; + } + strcpy(pColRef->colName, pExpr->aliasName); + nodesDestroyNode(*(SNode**)pCol); + *(SNode**)pCol = (SNode*)pColRef; + *pFound = true; + return DEAL_RES_CONTINUE; } } - return false; + *pFound = false; + return DEAL_RES_CONTINUE; } static EDealRes translateColumn(STranslateContext* pCxt, SColumnNode** pCol) { @@ -638,9 +658,11 @@ static EDealRes translateColumn(STranslateContext* pCxt, SColumnNode** pCol) { } else { bool found = false; if (SQL_CLAUSE_ORDER_BY == pCxt->currClause) { - found = translateColumnUseAlias(pCxt, pCol); + res = translateColumnUseAlias(pCxt, pCol, &found); + } + if (DEAL_RES_ERROR != res && !found) { + res = translateColumnWithoutPrefix(pCxt, pCol); } - res = (found ? DEAL_RES_CONTINUE : translateColumnWithoutPrefix(pCxt, pCol)); } return res; } @@ -1098,11 +1120,44 @@ static int32_t translateWindowPseudoColumnFunc(STranslateContext* pCxt, SFunctio return TSDB_CODE_SUCCESS; } +static int32_t translateForbidWindowFunc(STranslateContext* pCxt, SFunctionNode* pFunc) { + if (!fmIsForbidWindowFunc(pFunc->funcId)) { + return TSDB_CODE_SUCCESS; + } + if (NULL != pCxt->pCurrSelectStmt->pWindow) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC, pFunc->functionName); + } + return TSDB_CODE_SUCCESS; +} + +static int32_t translateForbidStreamFunc(STranslateContext* pCxt, SFunctionNode* pFunc) { + if (!fmIsForbidStreamFunc(pFunc->funcId)) { + return TSDB_CODE_SUCCESS; + } + if (pCxt->createStream) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_STREAM_NOT_ALLOWED_FUNC, pFunc->functionName); + } + return TSDB_CODE_SUCCESS; +} + +static int32_t translateForbidGroupByFunc(STranslateContext* pCxt, SFunctionNode* pFunc) { + if (!fmIsForbidGroupByFunc(pFunc->funcId)) { + return TSDB_CODE_SUCCESS; + } + if (NULL != pCxt->pCurrSelectStmt->pGroupByList) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC, pFunc->functionName); + } + return TSDB_CODE_SUCCESS; +} + static void setFuncClassification(SSelectStmt* pSelect, SFunctionNode* pFunc) { if (NULL != pSelect) { pSelect->hasAggFuncs = pSelect->hasAggFuncs ? true : fmIsAggFunc(pFunc->funcId); pSelect->hasRepeatScanFuncs = pSelect->hasRepeatScanFuncs ? true : fmIsRepeatScanFunc(pFunc->funcId); pSelect->hasIndefiniteRowsFunc = pSelect->hasIndefiniteRowsFunc ? true : fmIsIndefiniteRowsFunc(pFunc->funcId); + pSelect->hasUniqueFunc = pSelect->hasUniqueFunc ? true : (FUNCTION_TYPE_UNIQUE == pFunc->funcType); + pSelect->hasTailFunc = pSelect->hasTailFunc ? true : (FUNCTION_TYPE_TAIL == pFunc->funcType); + pSelect->hasInterpFunc = pSelect->hasInterpFunc ? true : (FUNCTION_TYPE_INTERP == pFunc->funcType); } } @@ -1130,6 +1185,15 @@ static EDealRes translateFunction(STranslateContext* pCxt, SFunctionNode* pFunc) if (TSDB_CODE_SUCCESS == pCxt->errCode) { pCxt->errCode = translateWindowPseudoColumnFunc(pCxt, pFunc); } + if (TSDB_CODE_SUCCESS == pCxt->errCode) { + pCxt->errCode = translateForbidWindowFunc(pCxt, pFunc); + } + if (TSDB_CODE_SUCCESS == pCxt->errCode) { + pCxt->errCode = translateForbidStreamFunc(pCxt, pFunc); + } + if (TSDB_CODE_SUCCESS == pCxt->errCode) { + pCxt->errCode = translateForbidGroupByFunc(pCxt, pFunc); + } if (TSDB_CODE_SUCCESS == pCxt->errCode) { setFuncClassification(pCxt->pCurrSelectStmt, pFunc); } @@ -1387,7 +1451,8 @@ static int32_t addMnodeToVgroupList(const SEpSet* pEpSet, SArray** pVgroupList) } static int32_t setSysTableVgroupList(STranslateContext* pCxt, SName* pName, SRealTableNode* pRealTable) { - if (0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLES)) { + if (0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLES) && + 0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLE_DISTRIBUTED)) { return TSDB_CODE_SUCCESS; } @@ -1468,7 +1533,8 @@ static bool joinTableIsSingleTable(SJoinTableNode* pJoinTable) { static bool isSingleTable(SRealTableNode* pRealTable) { int8_t tableType = pRealTable->pMeta->tableType; if (TSDB_SYSTEM_TABLE == tableType) { - return 0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLES); + return 0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLES) && + 0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_USER_TABLE_DISTRIBUTED); } return (TSDB_CHILD_TABLE == tableType || TSDB_NORMAL_TABLE == tableType); } @@ -1762,11 +1828,11 @@ static int32_t translateOrderByPosition(STranslateContext* pCxt, SNodeList* pPro } else if (0 == pos || pos > LIST_LENGTH(pProjectionList)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_WRONG_NUMBER_OF_SELECT); } else { - SColumnNode* pCol = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN); + SColumnRefNode* pCol = (SColumnRefNode*)nodesMakeNode(QUERY_NODE_COLUMN_REF); if (NULL == pCol) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_OUT_OF_MEMORY); } - setColumnInfoByExpr(NULL, (SExprNode*)nodesListGetNode(pProjectionList, pos - 1), &pCol); + strcpy(pCol->colName, ((SExprNode*)nodesListGetNode(pProjectionList, pos - 1))->aliasName); ((SOrderByExprNode*)pNode)->pExpr = (SNode*)pCol; nodesDestroyNode(pExpr); } @@ -1781,16 +1847,15 @@ static int32_t translateOrderByPosition(STranslateContext* pCxt, SNodeList* pPro static int32_t translateOrderBy(STranslateContext* pCxt, SSelectStmt* pSelect) { bool other; int32_t code = translateOrderByPosition(pCxt, pSelect->pProjectionList, pSelect->pOrderByList, &other); - if (TSDB_CODE_SUCCESS != code) { - return code; - } - if (!other) { - return TSDB_CODE_SUCCESS; - } - pCxt->currClause = SQL_CLAUSE_ORDER_BY; - code = translateExprList(pCxt, pSelect->pOrderByList); if (TSDB_CODE_SUCCESS == code) { - code = checkExprListForGroupBy(pCxt, pSelect->pOrderByList); + if (!other) { + return TSDB_CODE_SUCCESS; + } + pCxt->currClause = SQL_CLAUSE_ORDER_BY; + code = translateExprList(pCxt, pSelect->pOrderByList); + if (TSDB_CODE_SUCCESS == code) { + code = checkExprListForGroupBy(pCxt, pSelect->pOrderByList); + } } return code; } @@ -1867,26 +1932,33 @@ static int32_t getFillTimeRange(STranslateContext* pCxt, SNode* pWhere, STimeWin return code; } -static int32_t checkFill(STranslateContext* pCxt, SIntervalWindowNode* pInterval) { - SFillNode* pFill = (SFillNode*)pInterval->pFill; +static int32_t checkFill(STranslateContext* pCxt, SFillNode* pFill, SValueNode* pInterval) { + if (FILL_MODE_NONE == pFill->mode) { + return TSDB_CODE_SUCCESS; + } + if (TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_INITIALIZER) || TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_DESC_INITIALIZER)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_FILL_TIME_RANGE); } - int64_t timeRange = TABS(pFill->timeRange.skey - pFill->timeRange.ekey); - int64_t intervalRange = 0; - SValueNode* pInter = (SValueNode*)pInterval->pInterval; - if (TIME_IS_VAR_DURATION(pInter->unit)) { + // interp FILL clause + if (NULL == pInterval) { + return TSDB_CODE_SUCCESS; + } + + int64_t timeRange = TABS(pFill->timeRange.skey - pFill->timeRange.ekey); + int64_t intervalRange = 0; + if (TIME_IS_VAR_DURATION(pInterval->unit)) { int64_t f = 1; - if (pInter->unit == 'n') { + if (pInterval->unit == 'n') { f = 30L * MILLISECOND_PER_DAY; - } else if (pInter->unit == 'y') { + } else if (pInterval->unit == 'y') { f = 365L * MILLISECOND_PER_DAY; } - intervalRange = pInter->datum.i * f; + intervalRange = pInterval->datum.i * f; } else { - intervalRange = pInter->datum.i; + intervalRange = pInterval->datum.i; } if ((timeRange == 0) || (timeRange / intervalRange) >= MAX_INTERVAL_TIME_WINDOW) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_FILL_TIME_RANGE); @@ -1902,7 +1974,7 @@ static int32_t translateFill(STranslateContext* pCxt, SNode* pWhere, SIntervalWi int32_t code = getFillTimeRange(pCxt, pWhere, &(((SFillNode*)pInterval->pFill)->timeRange)); if (TSDB_CODE_SUCCESS == code) { - code = checkFill(pCxt, pInterval); + code = checkFill(pCxt, (SFillNode*)pInterval->pFill, (SValueNode*)pInterval->pInterval); } return code; } @@ -2044,6 +2116,64 @@ static int32_t translateWindow(STranslateContext* pCxt, SSelectStmt* pSelect) { return code; } +static int32_t createDefaultFillNode(STranslateContext* pCxt, SNode** pOutput) { + SFillNode* pFill = (SFillNode*)nodesMakeNode(QUERY_NODE_FILL); + if (NULL == pFill) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + pFill->mode = FILL_MODE_NONE; + + SColumnNode* pCol = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN); + if (NULL == pCol) { + nodesDestroyNode((SNode*)pFill); + return TSDB_CODE_OUT_OF_MEMORY; + } + pCol->colId = PRIMARYKEY_TIMESTAMP_COL_ID; + strcpy(pCol->colName, PK_TS_COL_INTERNAL_NAME); + pFill->pWStartTs = (SNode*)pCol; + + *pOutput = (SNode*)pFill; + return TSDB_CODE_SUCCESS; +} + +static int32_t translateInterpFill(STranslateContext* pCxt, SSelectStmt* pSelect) { + int32_t code = TSDB_CODE_SUCCESS; + + if (NULL == pSelect->pFill) { + code = createDefaultFillNode(pCxt, &pSelect->pFill); + } + if (TSDB_CODE_SUCCESS == code) { + code = translateExpr(pCxt, &pSelect->pFill); + } + if (TSDB_CODE_SUCCESS == code) { + code = getFillTimeRange(pCxt, pSelect->pRange, &(((SFillNode*)pSelect->pFill)->timeRange)); + } + if (TSDB_CODE_SUCCESS == code) { + code = checkFill(pCxt, (SFillNode*)pSelect->pFill, (SValueNode*)pSelect->pEvery); + } + + return code; +} + +static int32_t translateInterp(STranslateContext* pCxt, SSelectStmt* pSelect) { + if (!pSelect->hasInterpFunc) { + if (NULL != pSelect->pRange || NULL != pSelect->pEvery || NULL != pSelect->pFill) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_INTERP_CLAUSE); + } + return TSDB_CODE_SUCCESS; + } + + int32_t code = translateExpr(pCxt, &pSelect->pRange); + if (TSDB_CODE_SUCCESS == code) { + code = translateExpr(pCxt, &pSelect->pEvery); + } + if (TSDB_CODE_SUCCESS == code) { + code = translateInterpFill(pCxt, pSelect); + } + return code; +} + static int32_t translatePartitionBy(STranslateContext* pCxt, SNodeList* pPartitionByList) { pCxt->currClause = SQL_CLAUSE_PARTITION_BY; return translateExprList(pCxt, pPartitionByList); @@ -2116,6 +2246,172 @@ static int32_t rewriteTimelineFunc(STranslateContext* pCxt, SSelectStmt* pSelect return pCxt->errCode; } +typedef struct SRwriteUniqueCxt { + STranslateContext* pTranslateCxt; + SNode* pExpr; +} SRwriteUniqueCxt; + +static EDealRes rewriteSeletcValueFunc(STranslateContext* pCxt, SNode** pNode) { + SFunctionNode* pFirst = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION); + if (NULL == pFirst) { + pCxt->errCode = TSDB_CODE_OUT_OF_MEMORY; + return DEAL_RES_ERROR; + } + strcpy(pFirst->functionName, "first"); + TSWAP(pFirst->pParameterList, ((SFunctionNode*)*pNode)->pParameterList); + strcpy(pFirst->node.aliasName, ((SExprNode*)*pNode)->aliasName); + nodesDestroyNode(*pNode); + *pNode = (SNode*)pFirst; + pCxt->errCode = fmGetFuncInfo(pFirst, pCxt->msgBuf.buf, pCxt->msgBuf.len); + pCxt->pCurrSelectStmt->hasAggFuncs = true; + return TSDB_CODE_SUCCESS == pCxt->errCode ? DEAL_RES_IGNORE_CHILD : DEAL_RES_ERROR; +} + +static EDealRes rewriteUniqueFunc(SNode** pNode, void* pContext) { + SRwriteUniqueCxt* pCxt = pContext; + if (QUERY_NODE_FUNCTION == nodeType(*pNode)) { + SFunctionNode* pFunc = (SFunctionNode*)*pNode; + if (FUNCTION_TYPE_UNIQUE == pFunc->funcType) { + SNode* pExpr = nodesListGetNode(pFunc->pParameterList, 0); + NODES_CLEAR_LIST(pFunc->pParameterList); + strcpy(((SExprNode*)pExpr)->aliasName, ((SExprNode*)*pNode)->aliasName); + nodesDestroyNode(*pNode); + *pNode = pExpr; + pCxt->pExpr = pExpr; + return DEAL_RES_IGNORE_CHILD; + } else if (FUNCTION_TYPE_SELECT_VALUE == pFunc->funcType) { + return rewriteSeletcValueFunc(pCxt->pTranslateCxt, pNode); + } + } + return DEAL_RES_CONTINUE; +} + +static SNode* createGroupingSet(SNode* pExpr) { + SGroupingSetNode* pGroupingSet = (SGroupingSetNode*)nodesMakeNode(QUERY_NODE_GROUPING_SET); + if (NULL == pGroupingSet) { + return NULL; + } + pGroupingSet->groupingSetType = GP_TYPE_NORMAL; + if (TSDB_CODE_SUCCESS != nodesListMakeStrictAppend(&pGroupingSet->pParameterList, nodesCloneNode(pExpr))) { + nodesDestroyNode((SNode*)pGroupingSet); + return NULL; + } + return (SNode*)pGroupingSet; +} + +static int32_t rewriteUniqueStmt(STranslateContext* pCxt, SSelectStmt* pSelect) { + if (!pSelect->hasUniqueFunc) { + return TSDB_CODE_SUCCESS; + } + + SRwriteUniqueCxt cxt = {.pTranslateCxt = pCxt, .pExpr = NULL}; + nodesRewriteExprs(pSelect->pProjectionList, rewriteUniqueFunc, &cxt); + if (TSDB_CODE_SUCCESS == cxt.pTranslateCxt->errCode) { + cxt.pTranslateCxt->errCode = nodesListMakeStrictAppend(&pSelect->pGroupByList, createGroupingSet(cxt.pExpr)); + } + pSelect->hasIndefiniteRowsFunc = false; + return cxt.pTranslateCxt->errCode; +} + +typedef struct SRwriteTailCxt { + STranslateContext* pTranslateCxt; + int64_t limit; + int64_t offset; +} SRwriteTailCxt; + +static EDealRes rewriteTailFunc(SNode** pNode, void* pContext) { + SRwriteTailCxt* pCxt = pContext; + if (QUERY_NODE_FUNCTION == nodeType(*pNode)) { + SFunctionNode* pFunc = (SFunctionNode*)*pNode; + if (FUNCTION_TYPE_TAIL == pFunc->funcType) { + pCxt->limit = ((SValueNode*)nodesListGetNode(pFunc->pParameterList, 1))->datum.i; + if (3 == LIST_LENGTH(pFunc->pParameterList)) { + pCxt->offset = ((SValueNode*)nodesListGetNode(pFunc->pParameterList, 2))->datum.i; + } + SNode* pExpr = nodesListGetNode(pFunc->pParameterList, 0); + strcpy(((SExprNode*)pExpr)->aliasName, ((SExprNode*)*pNode)->aliasName); + NODES_CLEAR_LIST(pFunc->pParameterList); + nodesDestroyNode(*pNode); + *pNode = pExpr; + return DEAL_RES_IGNORE_CHILD; + } + } + return DEAL_RES_CONTINUE; +} + +static int32_t createLimieNode(SRwriteTailCxt* pCxt, SLimitNode** pOutput) { + *pOutput = (SLimitNode*)nodesMakeNode(QUERY_NODE_LIMIT); + if (NULL == *pOutput) { + return TSDB_CODE_OUT_OF_MEMORY; + } + (*pOutput)->limit = pCxt->limit; + (*pOutput)->offset = pCxt->offset; + return TSDB_CODE_SUCCESS; +} + +static SNode* createOrderByExpr(STranslateContext* pCxt) { + SOrderByExprNode* pOrder = (SOrderByExprNode*)nodesMakeNode(QUERY_NODE_ORDER_BY_EXPR); + if (NULL == pOrder) { + return NULL; + } + pCxt->errCode = createPrimaryKeyCol(pCxt, &pOrder->pExpr); + if (TSDB_CODE_SUCCESS != pCxt->errCode) { + nodesDestroyNode((SNode*)pOrder); + return NULL; + } + pOrder->order = ORDER_DESC; + pOrder->nullOrder = NULL_ORDER_FIRST; + return (SNode*)pOrder; +} + +static int32_t rewriteTailStmt(STranslateContext* pCxt, SSelectStmt* pSelect) { + if (!pSelect->hasTailFunc) { + return TSDB_CODE_SUCCESS; + } + + SRwriteTailCxt cxt = {.pTranslateCxt = pCxt, .limit = -1, .offset = -1}; + nodesRewriteExprs(pSelect->pProjectionList, rewriteTailFunc, &cxt); + int32_t code = nodesListMakeStrictAppend(&pSelect->pOrderByList, createOrderByExpr(pCxt)); + if (TSDB_CODE_SUCCESS == code) { + code = createLimieNode(&cxt, &pSelect->pLimit); + } + pSelect->hasIndefiniteRowsFunc = false; + return code; +} + +typedef struct SReplaceOrderByAliasCxt { + STranslateContext* pTranslateCxt; + SNodeList* pProjectionList; +} SReplaceOrderByAliasCxt; + +static EDealRes replaceOrderByAliasImpl(SNode** pNode, void* pContext) { + SReplaceOrderByAliasCxt* pCxt = pContext; + if (QUERY_NODE_COLUMN_REF == nodeType(*pNode)) { + SNodeList* pProjectionList = pCxt->pProjectionList; + SNode* pProject = NULL; + FOREACH(pProject, pProjectionList) { + SExprNode* pExpr = (SExprNode*)pProject; + if (0 == strcmp(((SColumnRefNode*)*pNode)->colName, pExpr->aliasName)) { + SNode* pNew = nodesCloneNode(pProject); + if (NULL == pNew) { + pCxt->pTranslateCxt->errCode = TSDB_CODE_OUT_OF_MEMORY; + return DEAL_RES_ERROR; + } + nodesDestroyNode(*pNode); + *pNode = pNew; + return DEAL_RES_CONTINUE; + } + } + } + return DEAL_RES_CONTINUE; +} + +static int32_t replaceOrderByAlias(STranslateContext* pCxt, SNodeList* pProjectionList, SNodeList* pOrderByList) { + SReplaceOrderByAliasCxt cxt = {.pTranslateCxt = pCxt, .pProjectionList = pProjectionList}; + nodesRewriteExprsPostOrder(pOrderByList, replaceOrderByAliasImpl, &cxt); + return pCxt->errCode; +} + static int32_t translateSelect(STranslateContext* pCxt, SSelectStmt* pSelect) { pCxt->pCurrSelectStmt = pSelect; int32_t code = translateFrom(pCxt, pSelect->pFromTable); @@ -2147,9 +2443,21 @@ static int32_t translateSelect(STranslateContext* pCxt, SSelectStmt* pSelect) { if (TSDB_CODE_SUCCESS == code) { code = checkLimit(pCxt, pSelect); } + if (TSDB_CODE_SUCCESS == code) { + code = translateInterp(pCxt, pSelect); + } + if (TSDB_CODE_SUCCESS == code) { + code = rewriteUniqueStmt(pCxt, pSelect); + } + // if (TSDB_CODE_SUCCESS == code) { + // code = rewriteTailStmt(pCxt, pSelect); + // } if (TSDB_CODE_SUCCESS == code) { code = rewriteTimelineFunc(pCxt, pSelect); } + if (TSDB_CODE_SUCCESS == code) { + code = replaceOrderByAlias(pCxt, pSelect->pProjectionList, pSelect->pOrderByList); + } return code; } @@ -2186,7 +2494,7 @@ static int32_t createCastFunc(STranslateContext* pCxt, SNode* pExpr, SDataType d return TSDB_CODE_SUCCESS; } -static int32_t translateSetOperatorImpl(STranslateContext* pCxt, SSetOperator* pSetOperator) { +static int32_t translateSetOperProject(STranslateContext* pCxt, SSetOperator* pSetOperator) { SNodeList* pLeftProjections = getProjectList(pSetOperator->pLeft); SNodeList* pRightProjections = getProjectList(pSetOperator->pRight); if (LIST_LENGTH(pLeftProjections) != LIST_LENGTH(pRightProjections)) { @@ -2221,6 +2529,23 @@ static uint8_t calcSetOperatorPrecision(SSetOperator* pSetOperator) { return calcPrecision(getStmtPrecision(pSetOperator->pLeft), getStmtPrecision(pSetOperator->pRight)); } +static int32_t translateSetOperOrderBy(STranslateContext* pCxt, SSetOperator* pSetOperator) { + bool other; + int32_t code = translateOrderByPosition(pCxt, pSetOperator->pProjectionList, pSetOperator->pOrderByList, &other); + if (TSDB_CODE_SUCCESS == code) { + if (other) { + pCxt->currClause = SQL_CLAUSE_ORDER_BY; + pCxt->pCurrSelectStmt = NULL; + pCxt->pCurrSetOperator = pSetOperator; + code = translateExprList(pCxt, pSetOperator->pOrderByList); + } + } + if (TSDB_CODE_SUCCESS == code) { + code = replaceOrderByAlias(pCxt, pSetOperator->pProjectionList, pSetOperator->pOrderByList); + } + return code; +} + static int32_t translateSetOperator(STranslateContext* pCxt, SSetOperator* pSetOperator) { int32_t code = translateQuery(pCxt, pSetOperator->pLeft); if (TSDB_CODE_SUCCESS == code) { @@ -2231,7 +2556,10 @@ static int32_t translateSetOperator(STranslateContext* pCxt, SSetOperator* pSetO } if (TSDB_CODE_SUCCESS == code) { pSetOperator->precision = calcSetOperatorPrecision(pSetOperator); - code = translateSetOperatorImpl(pCxt, pSetOperator); + code = translateSetOperProject(pCxt, pSetOperator); + } + if (TSDB_CODE_SUCCESS == code) { + code = translateSetOperOrderBy(pCxt, pSetOperator); } return code; } @@ -2366,7 +2694,7 @@ static int32_t checkDbDaysOption(STranslateContext* pCxt, SDatabaseOptions* pOpt if (TIME_UNIT_MINUTE != pOptions->pDaysPerFile->unit && TIME_UNIT_HOUR != pOptions->pDaysPerFile->unit && TIME_UNIT_DAY != pOptions->pDaysPerFile->unit) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_OPTION_UNIT, "daysPerFile", - pOptions->pDaysPerFile->unit); + pOptions->pDaysPerFile->unit, TIME_UNIT_MINUTE, TIME_UNIT_HOUR, TIME_UNIT_DAY); } pOptions->daysPerFile = getBigintFromValueNode(pOptions->pDaysPerFile); } @@ -2668,14 +2996,6 @@ static SColumnDefNode* findColDef(SNodeList* pCols, const SColumnNode* pCol) { return NULL; } -static int32_t checTableFactorOption(STranslateContext* pCxt, float val) { - if (val < TSDB_MIN_ROLLUP_FILE_FACTOR || val > TSDB_MAX_ROLLUP_FILE_FACTOR) { - return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_F_RANGE_OPTION, "file_factor", val, - TSDB_MIN_ROLLUP_FILE_FACTOR, TSDB_MAX_ROLLUP_FILE_FACTOR); - } - return TSDB_CODE_SUCCESS; -} - static int32_t checkTableSmaOption(STranslateContext* pCxt, SCreateTableStmt* pStmt) { if (NULL != pStmt->pOptions->pSma) { SNode* pNode = NULL; @@ -2825,33 +3145,77 @@ static int32_t checkTableSchema(STranslateContext* pCxt, SCreateTableStmt* pStmt return code; } -static int32_t checkSchemalessDb(STranslateContext* pCxt, const char* pDbName) { - // if (0 != pCxt->pParseCxt->schemalessType) { - // return TSDB_CODE_SUCCESS; - // } - // SDbCfgInfo info = {0}; - // int32_t code = getDBCfg(pCxt, pDbName, &info); - // if (TSDB_CODE_SUCCESS == code) { - // code = info.schemaless ? TSDB_CODE_SML_INVALID_DB_CONF : TSDB_CODE_SUCCESS; - // } - // return code; - return TSDB_CODE_SUCCESS; +static int32_t getTableDelayOrWatermarkOption(STranslateContext* pCxt, const char* pName, int64_t minVal, + int64_t maxVal, SValueNode* pVal, int64_t* pMaxDelay) { + int32_t code = (DEAL_RES_ERROR == translateValue(pCxt, pVal) ? pCxt->errCode : TSDB_CODE_SUCCESS); + if (TSDB_CODE_SUCCESS == code && TIME_UNIT_MILLISECOND != pVal->unit && TIME_UNIT_SECOND != pVal->unit && + TIME_UNIT_MINUTE != pVal->unit) { + code = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_OPTION_UNIT, pName, pVal->unit, + TIME_UNIT_MILLISECOND, TIME_UNIT_SECOND, TIME_UNIT_MINUTE); + } + if (TSDB_CODE_SUCCESS == code) { + code = checkRangeOption(pCxt, pName, pVal->datum.i, minVal, maxVal); + } + if (TSDB_CODE_SUCCESS == code) { + *pMaxDelay = pVal->datum.i; + } + return code; +} + +static int32_t getTableMaxDelayOption(STranslateContext* pCxt, SValueNode* pVal, int64_t* pMaxDelay) { + return getTableDelayOrWatermarkOption(pCxt, "maxDelay", TSDB_MIN_ROLLUP_MAX_DELAY, TSDB_MAX_ROLLUP_MAX_DELAY, pVal, + pMaxDelay); +} + +static int32_t checkTableMaxDelayOption(STranslateContext* pCxt, STableOptions* pOptions) { + if (NULL == pOptions->pMaxDelay) { + return TSDB_CODE_SUCCESS; + } + + if (LIST_LENGTH(pOptions->pMaxDelay) > 2) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_TABLE_OPTION, "maxdelay"); + } + + int32_t code = + getTableMaxDelayOption(pCxt, (SValueNode*)nodesListGetNode(pOptions->pMaxDelay, 0), &pOptions->maxDelay1); + if (TSDB_CODE_SUCCESS == code && 2 == LIST_LENGTH(pOptions->pMaxDelay)) { + code = getTableMaxDelayOption(pCxt, (SValueNode*)nodesListGetNode(pOptions->pMaxDelay, 1), &pOptions->maxDelay2); + } + + return code; +} + +static int32_t getTableWatermarkOption(STranslateContext* pCxt, SValueNode* pVal, int64_t* pMaxDelay) { + return getTableDelayOrWatermarkOption(pCxt, "watermark", TSDB_MIN_ROLLUP_WATERMARK, TSDB_MAX_ROLLUP_WATERMARK, pVal, + pMaxDelay); +} + +static int32_t checkTableWatermarkOption(STranslateContext* pCxt, STableOptions* pOptions) { + if (NULL == pOptions->pWatermark) { + return TSDB_CODE_SUCCESS; + } + + if (LIST_LENGTH(pOptions->pWatermark) > 2) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_TABLE_OPTION, "watermark"); + } + + int32_t code = + getTableWatermarkOption(pCxt, (SValueNode*)nodesListGetNode(pOptions->pWatermark, 0), &pOptions->watermark1); + if (TSDB_CODE_SUCCESS == code && 2 == LIST_LENGTH(pOptions->pWatermark)) { + code = getTableWatermarkOption(pCxt, (SValueNode*)nodesListGetNode(pOptions->pWatermark, 1), &pOptions->watermark2); + } + + return code; } static int32_t checkCreateTable(STranslateContext* pCxt, SCreateTableStmt* pStmt) { - int32_t code = checkSchemalessDb(pCxt, pStmt->dbName); + int32_t code = checkTableMaxDelayOption(pCxt, pStmt->pOptions); if (TSDB_CODE_SUCCESS == code) { - code = checTableFactorOption(pCxt, pStmt->pOptions->filesFactor); + code = checkTableWatermarkOption(pCxt, pStmt->pOptions); } - // if (TSDB_CODE_SUCCESS == code) { - // code = checkRangeOption(pCxt, "delay", pStmt->pOptions->delay, TSDB_MIN_ROLLUP_DELAY, TSDB_MAX_ROLLUP_DELAY); - // } if (TSDB_CODE_SUCCESS == code) { code = checkTableRollupOption(pCxt, pStmt->pOptions->pRollupFuncs); } - if (TSDB_CODE_SUCCESS == code) { - code = checkRangeOption(pCxt, "ttl", pStmt->pOptions->ttl, TSDB_MIN_TABLE_TTL, INT32_MAX); - } if (TSDB_CODE_SUCCESS == code) { code = checkTableSmaOption(pCxt, pStmt); } @@ -3088,19 +3452,23 @@ static int32_t buildRollupAst(STranslateContext* pCxt, SCreateTableStmt* pStmt, static int32_t buildCreateStbReq(STranslateContext* pCxt, SCreateTableStmt* pStmt, SMCreateStbReq* pReq) { pReq->igExists = pStmt->ignoreExists; - // pReq->delay = pStmt->pOptions->delay; - pReq->xFilesFactor = pStmt->pOptions->filesFactor; - pReq->ttl = pStmt->pOptions->ttl; + pReq->delay1 = pStmt->pOptions->maxDelay1; + pReq->delay2 = pStmt->pOptions->maxDelay2; + pReq->watermark1 = pStmt->pOptions->watermark1; + pReq->watermark2 = pStmt->pOptions->watermark2; + // pReq->ttl = pStmt->pOptions->ttl; columnDefNodeToField(pStmt->pCols, &pReq->pColumns); columnDefNodeToField(pStmt->pTags, &pReq->pTags); pReq->numOfColumns = LIST_LENGTH(pStmt->pCols); pReq->numOfTags = LIST_LENGTH(pStmt->pTags); - if ('\0' != pStmt->pOptions->comment[0]) { + if (pStmt->pOptions->commentNull == false) { pReq->comment = strdup(pStmt->pOptions->comment); if (NULL == pReq->comment) { return TSDB_CODE_OUT_OF_MEMORY; } - pReq->commentLen = strlen(pStmt->pOptions->comment) + 1; + pReq->commentLen = strlen(pStmt->pOptions->comment); + } else { + pReq->commentLen = -1; } SName tableName; @@ -3152,14 +3520,17 @@ static int32_t buildAlterSuperTableReq(STranslateContext* pCxt, SAlterTableStmt* pAlterReq->alterType = pStmt->alterType; if (TSDB_ALTER_TABLE_UPDATE_OPTIONS == pStmt->alterType) { - pAlterReq->ttl = pStmt->pOptions->ttl; - if ('\0' != pStmt->pOptions->comment[0]) { + // pAlterReq->ttl = pStmt->pOptions->ttl; + if (pStmt->pOptions->commentNull == false) { pAlterReq->comment = strdup(pStmt->pOptions->comment); if (NULL == pAlterReq->comment) { return TSDB_CODE_OUT_OF_MEMORY; } - pAlterReq->commentLen = strlen(pStmt->pOptions->comment) + 1; + pAlterReq->commentLen = strlen(pStmt->pOptions->comment); + } else { + pAlterReq->commentLen = -1; } + return TSDB_CODE_SUCCESS; } @@ -3314,27 +3685,6 @@ static int32_t translateAlterDnode(STranslateContext* pCxt, SAlterDnodeStmt* pSt return buildCmdMsg(pCxt, TDMT_MND_CONFIG_DNODE, (FSerializeFunc)tSerializeSMCfgDnodeReq, &cfgReq); } -static int32_t nodeTypeToShowType(ENodeType nt) { - switch (nt) { - case QUERY_NODE_SHOW_CONNECTIONS_STMT: - return TSDB_MGMT_TABLE_CONNS; - case QUERY_NODE_SHOW_LICENCE_STMT: - return TSDB_MGMT_TABLE_GRANTS; - case QUERY_NODE_SHOW_QUERIES_STMT: - return TSDB_MGMT_TABLE_QUERIES; - case QUERY_NODE_SHOW_VARIABLE_STMT: - return TSDB_MGMT_TABLE_CONFIGS; - default: - break; - } - return 0; -} - -static int32_t translateShow(STranslateContext* pCxt, SShowStmt* pStmt) { - SShowReq showReq = {.type = nodeTypeToShowType(nodeType(pStmt))}; - return buildCmdMsg(pCxt, TDMT_MND_SHOW, (FSerializeFunc)tSerializeSShowReq, &showReq); -} - static int32_t getSmaIndexDstVgId(STranslateContext* pCxt, char* pTableName, int32_t* pVgId) { SVgroupInfo vg = {0}; int32_t code = getTableHashVgroup(pCxt, pCxt->pParseCxt->db, pTableName, &vg); @@ -3633,7 +3983,7 @@ static int32_t translateDescribe(STranslateContext* pCxt, SDescribeStmt* pStmt) static int32_t translateKillConnection(STranslateContext* pCxt, SKillStmt* pStmt) { SKillConnReq killReq = {0}; killReq.connId = pStmt->targetId; - return buildCmdMsg(pCxt, TDMT_MND_KILL_CONN, (FSerializeFunc)tSerializeSKillQueryReq, &killReq); + return buildCmdMsg(pCxt, TDMT_MND_KILL_CONN, (FSerializeFunc)tSerializeSKillConnReq, &killReq); } static int32_t translateKillQuery(STranslateContext* pCxt, SKillQueryStmt* pStmt) { @@ -3880,6 +4230,18 @@ static int32_t translateSplitVgroup(STranslateContext* pCxt, SSplitVgroupStmt* p return buildCmdMsg(pCxt, TDMT_MND_SPLIT_VGROUP, (FSerializeFunc)tSerializeSSplitVgroupReq, &req); } +static int32_t translateShowCreateDatabase(STranslateContext* pCxt, SShowCreateDatabaseStmt* pStmt) { + pStmt->pCfg = taosMemoryCalloc(1, sizeof(SDbCfgInfo)); + if (NULL == pStmt->pCfg) { + return TSDB_CODE_OUT_OF_MEMORY; + } + return getDBCfg(pCxt, pStmt->dbName, (SDbCfgInfo*)pStmt->pCfg); +} + +static int32_t translateShowCreateTable(STranslateContext* pCxt, SShowCreateTableStmt* pStmt) { + return getTableMeta(pCxt, pStmt->dbName, pStmt->tableName, &pStmt->pMeta); +} + static int32_t translateQuery(STranslateContext* pCxt, SNode* pNode) { int32_t code = TSDB_CODE_SUCCESS; switch (nodeType(pNode)) { @@ -3934,12 +4296,6 @@ static int32_t translateQuery(STranslateContext* pCxt, SNode* pNode) { case QUERY_NODE_ALTER_DNODE_STMT: code = translateAlterDnode(pCxt, (SAlterDnodeStmt*)pNode); break; - case QUERY_NODE_SHOW_CONNECTIONS_STMT: - case QUERY_NODE_SHOW_QUERIES_STMT: - case QUERY_NODE_SHOW_TOPICS_STMT: - case QUERY_NODE_SHOW_VARIABLE_STMT: - code = translateShow(pCxt, (SShowStmt*)pNode); - break; case QUERY_NODE_CREATE_INDEX_STMT: code = translateCreateIndex(pCxt, (SCreateIndexStmt*)pNode); break; @@ -4015,6 +4371,13 @@ static int32_t translateQuery(STranslateContext* pCxt, SNode* pNode) { case QUERY_NODE_SPLIT_VGROUP_STMT: code = translateSplitVgroup(pCxt, (SSplitVgroupStmt*)pNode); break; + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + code = translateShowCreateDatabase(pCxt, (SShowCreateDatabaseStmt*)pNode); + break; + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: + code = translateShowCreateTable(pCxt, (SShowCreateTableStmt*)pNode); + break; default: break; } @@ -4097,6 +4460,42 @@ static int32_t extractDescribeResultSchema(int32_t* numOfCols, SSchema** pSchema return TSDB_CODE_SUCCESS; } +static int32_t extractShowCreateDatabaseResultSchema(int32_t* numOfCols, SSchema** pSchema) { + *numOfCols = 2; + *pSchema = taosMemoryCalloc((*numOfCols), sizeof(SSchema)); + if (NULL == (*pSchema)) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + (*pSchema)[0].type = TSDB_DATA_TYPE_BINARY; + (*pSchema)[0].bytes = TSDB_DB_NAME_LEN; + strcpy((*pSchema)[0].name, "Database"); + + (*pSchema)[1].type = TSDB_DATA_TYPE_BINARY; + (*pSchema)[1].bytes = TSDB_MAX_BINARY_LEN; + strcpy((*pSchema)[1].name, "Create Database"); + + return TSDB_CODE_SUCCESS; +} + +static int32_t extractShowCreateTableResultSchema(int32_t* numOfCols, SSchema** pSchema) { + *numOfCols = 2; + *pSchema = taosMemoryCalloc((*numOfCols), sizeof(SSchema)); + if (NULL == (*pSchema)) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + (*pSchema)[0].type = TSDB_DATA_TYPE_BINARY; + (*pSchema)[0].bytes = TSDB_TABLE_NAME_LEN; + strcpy((*pSchema)[0].name, "Table"); + + (*pSchema)[1].type = TSDB_DATA_TYPE_BINARY; + (*pSchema)[1].bytes = TSDB_MAX_BINARY_LEN; + strcpy((*pSchema)[1].name, "Create Table"); + + return TSDB_CODE_SUCCESS; +} + int32_t extractResultSchema(const SNode* pRoot, int32_t* numOfCols, SSchema** pSchema) { if (NULL == pRoot) { return TSDB_CODE_SUCCESS; @@ -4110,6 +4509,11 @@ int32_t extractResultSchema(const SNode* pRoot, int32_t* numOfCols, SSchema** pS return extractExplainResultSchema(numOfCols, pSchema); case QUERY_NODE_DESCRIBE_STMT: return extractDescribeResultSchema(numOfCols, pSchema); + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + return extractShowCreateDatabaseResultSchema(numOfCols, pSchema); + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: + return extractShowCreateTableResultSchema(numOfCols, pSchema); default: break; } @@ -4201,28 +4605,36 @@ static const char* getSysTableName(ENodeType type) { return NULL; } -static int32_t createSelectStmtForShow(ENodeType showType, SSelectStmt** pStmt) { +static int32_t createSimpleSelectStmt(const char* pDb, const char* pTable, SSelectStmt** pStmt) { SSelectStmt* pSelect = (SSelectStmt*)nodesMakeNode(QUERY_NODE_SELECT_STMT); if (NULL == pSelect) { return TSDB_CODE_OUT_OF_MEMORY; } sprintf(pSelect->stmtName, "%p", pSelect); - SRealTableNode* pTable = (SRealTableNode*)nodesMakeNode(QUERY_NODE_REAL_TABLE); - if (NULL == pTable) { + SRealTableNode* pRealTable = (SRealTableNode*)nodesMakeNode(QUERY_NODE_REAL_TABLE); + if (NULL == pRealTable) { nodesDestroyNode((SNode*)pSelect); return TSDB_CODE_OUT_OF_MEMORY; } - strcpy(pTable->table.dbName, getSysDbName(showType)); - strcpy(pTable->table.tableName, getSysTableName(showType)); - strcpy(pTable->table.tableAlias, pTable->table.tableName); - pSelect->pFromTable = (SNode*)pTable; + strcpy(pRealTable->table.dbName, pDb); + strcpy(pRealTable->table.tableName, pTable); + strcpy(pRealTable->table.tableAlias, pTable); + pSelect->pFromTable = (SNode*)pRealTable; *pStmt = pSelect; return TSDB_CODE_SUCCESS; } +static int32_t createSelectStmtForShow(ENodeType showType, SSelectStmt** pStmt) { + return createSimpleSelectStmt(getSysDbName(showType), getSysTableName(showType), pStmt); +} + +static int32_t createSelectStmtForShowTableDist(SShowTableDistributedStmt* pStmt, SSelectStmt** pOutput) { + return createSimpleSelectStmt(pStmt->dbName, pStmt->tableName, pOutput); +} + static int32_t createOperatorNode(EOperatorType opType, const char* pColName, SNode* pRight, SNode** pOp) { if (NULL == pRight) { return TSDB_CODE_SUCCESS; @@ -4276,7 +4688,7 @@ static int32_t createShowCondition(const SShowStmt* pShow, SSelectStmt* pSelect) SNode* pTbCond = NULL; if (TSDB_CODE_SUCCESS != createOperatorNode(OP_TYPE_EQUAL, "db_name", pShow->pDbName, &pDbCond) || TSDB_CODE_SUCCESS != - createOperatorNode(OP_TYPE_LIKE, getTbNameColName(nodeType(pShow)), pShow->pTbNamePattern, &pTbCond)) { + createOperatorNode(pShow->tableCondType, getTbNameColName(nodeType(pShow)), pShow->pTbName, &pTbCond)) { nodesDestroyNode(pDbCond); nodesDestroyNode(pTbCond); return TSDB_CODE_OUT_OF_MEMORY; @@ -4313,6 +4725,46 @@ static int32_t rewriteShow(STranslateContext* pCxt, SQuery* pQuery) { return code; } +static SNode* createBlockDistInfoFunc() { + SFunctionNode* pFunc = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION); + if (NULL == pFunc) { + return NULL; + } + + strcpy(pFunc->functionName, "_block_dist_info"); + strcpy(pFunc->node.aliasName, "_block_dist_info"); + return (SNode*)pFunc; +} + +static SNode* createBlockDistFunc() { + SFunctionNode* pFunc = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION); + if (NULL == pFunc) { + return NULL; + } + + strcpy(pFunc->functionName, "_block_dist"); + strcpy(pFunc->node.aliasName, "_block_dist"); + if (TSDB_CODE_SUCCESS != nodesListMakeStrictAppend(&pFunc->pParameterList, createBlockDistInfoFunc())) { + nodesDestroyNode((SNode*)pFunc); + return NULL; + } + return (SNode*)pFunc; +} + +static int32_t rewriteShowTableDist(STranslateContext* pCxt, SQuery* pQuery) { + SSelectStmt* pStmt = NULL; + int32_t code = createSelectStmtForShowTableDist((SShowTableDistributedStmt*)pQuery->pRoot, &pStmt); + if (TSDB_CODE_SUCCESS == code) { + code = nodesListMakeStrictAppend(&pStmt->pProjectionList, createBlockDistFunc()); + } + if (TSDB_CODE_SUCCESS == code) { + pQuery->showRewrite = true; + nodesDestroyNode(pQuery->pRoot); + pQuery->pRoot = (SNode*)pStmt; + } + return code; +} + typedef struct SVgroupCreateTableBatch { SVCreateTbBatchReq req; SVgroupInfo info; @@ -4334,6 +4786,16 @@ static int32_t buildNormalTableBatchReq(int32_t acctId, const SCreateTableStmt* SVCreateTbReq req = {0}; req.type = TD_NORMAL_TABLE; req.name = strdup(pStmt->tableName); + req.ttl = pStmt->pOptions->ttl; + if (pStmt->pOptions->commentNull == false) { + req.comment = strdup(pStmt->pOptions->comment); + if (NULL == req.comment) { + return TSDB_CODE_OUT_OF_MEMORY; + } + req.commentLen = strlen(pStmt->pOptions->comment); + } else { + req.commentLen = -1; + } req.ntb.schemaRow.nCols = LIST_LENGTH(pStmt->pCols); req.ntb.schemaRow.version = 1; req.ntb.schemaRow.pSchema = taosMemoryCalloc(req.ntb.schemaRow.nCols, sizeof(SSchema)); @@ -4484,6 +4946,13 @@ static void addCreateTbReqIntoVgroup(int32_t acctId, SHashObj* pVgroupHashmap, S struct SVCreateTbReq req = {0}; req.type = TD_CHILD_TABLE; req.name = strdup(pStmt->tableName); + req.ttl = pStmt->pOptions->ttl; + if (pStmt->pOptions->commentNull == false) { + req.comment = strdup(pStmt->pOptions->comment); + req.commentLen = strlen(pStmt->pOptions->comment); + } else { + req.commentLen = -1; + } req.ctb.suid = suid; req.ctb.pTag = (uint8_t*)pTag; if (pStmt->ignoreExists) { @@ -4757,10 +5226,7 @@ static int32_t rewriteCreateMultiTable(STranslateContext* pCxt, SQuery* pQuery) SNode* pNode; FOREACH(pNode, pStmt->pSubTables) { SCreateSubTableClause* pClause = (SCreateSubTableClause*)pNode; - code = checkSchemalessDb(pCxt, pClause->dbName); - if (TSDB_CODE_SUCCESS == code) { - code = rewriteCreateSubTable(pCxt, pClause, pVgroupHashmap); - } + code = rewriteCreateSubTable(pCxt, pClause, pVgroupHashmap); if (TSDB_CODE_SUCCESS != code) { taosHashCleanup(pVgroupHashmap); return code; @@ -5058,18 +5524,19 @@ static int32_t buildUpdateOptionsReq(STranslateContext* pCxt, SAlterTableStmt* p int32_t code = TSDB_CODE_SUCCESS; if (-1 != pStmt->pOptions->ttl) { - code = checkRangeOption(pCxt, "ttl", pStmt->pOptions->ttl, TSDB_MIN_TABLE_TTL, INT32_MAX); - if (TSDB_CODE_SUCCESS == code) { - pReq->updateTTL = true; - pReq->newTTL = pStmt->pOptions->ttl; - } + pReq->updateTTL = true; + pReq->newTTL = pStmt->pOptions->ttl; } - if (TSDB_CODE_SUCCESS == code && '\0' != pStmt->pOptions->comment[0]) { - pReq->updateComment = true; - pReq->newComment = strdup(pStmt->pOptions->comment); - if (NULL == pReq->newComment) { - code = TSDB_CODE_OUT_OF_MEMORY; + if (TSDB_CODE_SUCCESS == code) { + if (pStmt->pOptions->commentNull == false) { + pReq->newComment = strdup(pStmt->pOptions->comment); + if (NULL == pReq->newComment) { + code = TSDB_CODE_OUT_OF_MEMORY; + } + pReq->newCommentLen = strlen(pReq->newComment); + } else { + pReq->newCommentLen = -1; } } @@ -5188,6 +5655,9 @@ static int32_t rewriteAlterTableImpl(STranslateContext* pCxt, SAlterTableStmt* p pStmt->alterType == TSDB_ALTER_TABLE_UPDATE_TAG_BYTES)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_ONLY_ONE_JSON_TAG); } + if (pStmt->alterType == TSDB_ALTER_TABLE_UPDATE_OPTIONS && -1 != pStmt->pOptions->ttl) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_ALTER_TABLE); + } return TSDB_CODE_SUCCESS; } else if (TSDB_CHILD_TABLE != pTableMeta->tableType && TSDB_NORMAL_TABLE != pTableMeta->tableType) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_ALTER_TABLE); @@ -5254,6 +5724,9 @@ static int32_t rewriteQuery(STranslateContext* pCxt, SQuery* pQuery) { case QUERY_NODE_SHOW_APPS_STMT: code = rewriteShow(pCxt, pQuery); break; + case QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT: + code = rewriteShowTableDist(pCxt, pQuery); + break; case QUERY_NODE_CREATE_TABLE_STMT: if (NULL == ((SCreateTableStmt*)pQuery->pRoot)->pTags) { code = rewriteCreateTable(pCxt, pQuery); @@ -5333,10 +5806,14 @@ static int32_t setQuery(STranslateContext* pCxt, SQuery* pQuery) { pQuery->msgType = toMsgType(((SVnodeModifOpStmt*)pQuery->pRoot)->sqlNodeType); break; case QUERY_NODE_DESCRIBE_STMT: + case QUERY_NODE_SHOW_CREATE_DATABASE_STMT: + case QUERY_NODE_SHOW_CREATE_TABLE_STMT: + case QUERY_NODE_SHOW_CREATE_STABLE_STMT: pQuery->execMode = QUERY_EXEC_MODE_LOCAL; pQuery->haveResultSet = true; break; case QUERY_NODE_RESET_QUERY_CACHE_STMT: + case QUERY_NODE_ALTER_LOCAL_STMT: pQuery->execMode = QUERY_EXEC_MODE_LOCAL; break; default: diff --git a/source/libs/parser/src/parUtil.c b/source/libs/parser/src/parUtil.c index 42e887bb9d69c1a20bf877f8d230dfae668bdbbd..d94b430c452a0e131a1ef91f836615dad90bddaa 100644 --- a/source/libs/parser/src/parUtil.c +++ b/source/libs/parser/src/parUtil.c @@ -90,7 +90,7 @@ static char* getSyntaxErrFormat(int32_t errCode) { case TSDB_CODE_PAR_GROUPBY_WINDOW_COEXIST: return "GROUP BY and WINDOW-clause can't be used together"; case TSDB_CODE_PAR_INVALID_OPTION_UNIT: - return "Invalid option %s unit: %c, only m, h, d allowed"; + return "Invalid option %s unit: %c, only %c, %c, %c allowed"; case TSDB_CODE_PAR_INVALID_KEEP_UNIT: return "Invalid option keep unit: %c, only m, h, d allowed"; case TSDB_CODE_PAR_AGG_FUNC_NESTING: @@ -185,9 +185,19 @@ static char* getSyntaxErrFormat(int32_t errCode) { case TSDB_CODE_PAR_INVALID_REDISTRIBUTE_VG: return "The REDISTRIBUTE VGROUP statement only support 1 to 3 dnodes"; case TSDB_CODE_PAR_FILL_NOT_ALLOWED_FUNC: - return "%s function not allowed in fill query"; + return "%s function does not supportted in fill query"; case TSDB_CODE_PAR_INVALID_WINDOW_PC: - return "_WSTARTTS, _WENDTS and _WDURATION can only be used in window queries"; + return "_WSTARTTS, _WENDTS and _WDURATION can only be used in window query"; + case TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC: + return "%s function does not supportted in time window query"; + case TSDB_CODE_PAR_STREAM_NOT_ALLOWED_FUNC: + return "%s function does not supportted in stream query"; + case TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC: + return "%s function does not supportted in group query"; + case TSDB_CODE_PAR_INVALID_TABLE_OPTION: + return "Invalid option %s"; + case TSDB_CODE_PAR_INVALID_INTERP_CLAUSE: + return "Invalid usage of RANGE clause, EVERY clause or FILL clause"; case TSDB_CODE_OUT_OF_MEMORY: return "Out of memory"; default: diff --git a/source/libs/parser/src/sql.c b/source/libs/parser/src/sql.c index e81610c1b459a93540cc70f5145eb762718ee0be..422661219e9d0b4ca8d0bc1d2ef11f480e48633b 100644 --- a/source/libs/parser/src/sql.c +++ b/source/libs/parser/src/sql.c @@ -104,25 +104,25 @@ #endif /************* Begin control #defines *****************************************/ #define YYCODETYPE unsigned short int -#define YYNOCODE 360 +#define YYNOCODE 364 #define YYACTIONTYPE unsigned short int #define ParseTOKENTYPE SToken typedef union { int yyinit; ParseTOKENTYPE yy0; - EFillMode yy54; - int32_t yy100; - bool yy137; - int64_t yy189; - SToken yy209; - ENullOrder yy217; - SDataType yy304; - EOperatorType yy380; - SNodeList* yy424; - EOrder yy578; - SAlterOption yy605; - EJoinType yy612; - SNode* yy632; + SToken yy57; + EOrder yy162; + EJoinType yy204; + SAlterOption yy221; + int32_t yy228; + EFillMode yy270; + EOperatorType yy324; + int64_t yy389; + SNode* yy392; + SDataType yy448; + bool yy481; + ENullOrder yy529; + SNodeList* yy600; } YYMINORTYPE; #ifndef YYSTACKDEPTH #define YYSTACKDEPTH 100 @@ -138,17 +138,17 @@ typedef union { #define ParseCTX_FETCH #define ParseCTX_STORE #define YYFALLBACK 1 -#define YYNSTATE 619 -#define YYNRULE 454 -#define YYNTOKEN 240 -#define YY_MAX_SHIFT 618 -#define YY_MIN_SHIFTREDUCE 906 -#define YY_MAX_SHIFTREDUCE 1359 -#define YY_ERROR_ACTION 1360 -#define YY_ACCEPT_ACTION 1361 -#define YY_NO_ACTION 1362 -#define YY_MIN_REDUCE 1363 -#define YY_MAX_REDUCE 1816 +#define YYNSTATE 636 +#define YYNRULE 464 +#define YYNTOKEN 242 +#define YY_MAX_SHIFT 635 +#define YY_MIN_SHIFTREDUCE 928 +#define YY_MAX_SHIFTREDUCE 1391 +#define YY_ERROR_ACTION 1392 +#define YY_ACCEPT_ACTION 1393 +#define YY_NO_ACTION 1394 +#define YY_MIN_REDUCE 1395 +#define YY_MAX_REDUCE 1858 /************* End control #defines *******************************************/ #define YY_NLOOKAHEAD ((int)(sizeof(yy_lookahead)/sizeof(yy_lookahead[0]))) @@ -215,607 +215,603 @@ typedef union { ** yy_default[] Default action for each state. ** *********** Begin parsing tables **********************************************/ -#define YY_ACTTAB_COUNT (2152) +#define YY_ACTTAB_COUNT (2123) static const YYACTIONTYPE yy_action[] = { - /* 0 */ 28, 231, 1663, 1650, 611, 610, 297, 1650, 358, 1485, - /* 10 */ 314, 1483, 35, 33, 352, 36, 34, 32, 31, 30, - /* 20 */ 306, 24, 1173, 1647, 533, 442, 441, 1647, 79, 1647, - /* 30 */ 1679, 36, 34, 32, 31, 30, 152, 493, 517, 1643, - /* 40 */ 1649, 115, 278, 1643, 1649, 1643, 1649, 1171, 516, 1486, - /* 50 */ 536, 533, 1633, 1494, 536, 1794, 536, 497, 14, 1746, - /* 60 */ 35, 33, 1300, 356, 1179, 1663, 114, 147, 306, 1692, - /* 70 */ 1173, 1791, 82, 1664, 519, 1666, 1667, 515, 532, 536, - /* 80 */ 1494, 1, 1732, 1743, 1794, 1314, 280, 1728, 36, 34, - /* 90 */ 32, 31, 30, 1679, 410, 1171, 1793, 1651, 1794, 520, - /* 100 */ 1791, 517, 309, 615, 112, 1583, 14, 1746, 35, 33, - /* 110 */ 149, 516, 1179, 1172, 1791, 1633, 306, 1647, 1173, 145, - /* 120 */ 1739, 1740, 532, 1744, 36, 34, 32, 31, 30, 2, - /* 130 */ 63, 1742, 1692, 1643, 1649, 84, 1664, 519, 1666, 1667, - /* 140 */ 515, 1364, 536, 1171, 536, 1732, 953, 1794, 952, 1731, - /* 150 */ 1728, 615, 1490, 72, 14, 392, 1174, 393, 1395, 148, - /* 160 */ 1179, 1172, 96, 1791, 532, 95, 94, 93, 92, 91, - /* 170 */ 90, 89, 88, 87, 1487, 954, 56, 2, 132, 201, - /* 180 */ 1375, 1177, 1178, 39, 1224, 1225, 1227, 1228, 1229, 1230, - /* 190 */ 1231, 512, 534, 1239, 1240, 1241, 1242, 1243, 1244, 615, - /* 200 */ 400, 952, 393, 1395, 1174, 55, 1197, 66, 133, 1172, - /* 210 */ 96, 150, 1451, 95, 94, 93, 92, 91, 90, 89, - /* 220 */ 88, 87, 65, 279, 1356, 38, 429, 194, 567, 1177, - /* 230 */ 1178, 1363, 1224, 1225, 1227, 1228, 1229, 1230, 1231, 512, - /* 240 */ 534, 1239, 1240, 1241, 1242, 1243, 1244, 566, 487, 565, - /* 250 */ 564, 563, 1174, 55, 63, 105, 104, 103, 102, 101, - /* 260 */ 100, 99, 98, 97, 936, 35, 33, 110, 36, 34, - /* 270 */ 32, 31, 30, 306, 1746, 1173, 1489, 1177, 1178, 1198, - /* 280 */ 1224, 1225, 1227, 1228, 1229, 1230, 1231, 512, 534, 1239, - /* 290 */ 1240, 1241, 1242, 1243, 1244, 55, 445, 444, 1741, 1538, - /* 300 */ 1171, 443, 940, 941, 111, 440, 296, 1355, 439, 438, - /* 310 */ 437, 1536, 569, 35, 33, 1245, 1679, 1179, 26, 310, - /* 320 */ 1386, 306, 391, 1173, 486, 395, 290, 130, 36, 34, - /* 330 */ 32, 31, 30, 1196, 8, 150, 1496, 1043, 559, 558, - /* 340 */ 557, 1047, 556, 1049, 1050, 555, 1052, 552, 1171, 1058, - /* 350 */ 549, 1060, 1061, 546, 543, 150, 615, 450, 482, 618, - /* 360 */ 485, 35, 33, 585, 583, 1179, 1172, 142, 397, 306, - /* 370 */ 1633, 1173, 458, 248, 1195, 291, 1324, 289, 288, 1532, - /* 380 */ 433, 318, 9, 150, 435, 107, 193, 399, 1574, 1576, - /* 390 */ 395, 607, 603, 599, 595, 247, 1171, 1472, 453, 157, - /* 400 */ 533, 129, 1361, 447, 615, 1425, 434, 572, 192, 1174, - /* 410 */ 1794, 1385, 357, 1179, 1172, 479, 1322, 1323, 1325, 1326, - /* 420 */ 80, 1571, 1792, 242, 61, 150, 1791, 60, 159, 1494, - /* 430 */ 9, 488, 483, 51, 1177, 1178, 50, 1224, 1225, 1227, - /* 440 */ 1228, 1229, 1230, 1231, 512, 534, 1239, 1240, 1241, 1242, - /* 450 */ 1243, 1244, 615, 55, 410, 316, 529, 1174, 54, 319, - /* 460 */ 322, 1633, 1172, 130, 150, 445, 444, 130, 1538, 562, - /* 470 */ 443, 382, 1496, 111, 440, 311, 1496, 439, 438, 437, - /* 480 */ 1536, 474, 1177, 1178, 203, 1224, 1225, 1227, 1228, 1229, - /* 490 */ 1230, 1231, 512, 534, 1239, 1240, 1241, 1242, 1243, 1244, - /* 500 */ 1384, 1794, 1147, 1199, 197, 1174, 1575, 1576, 36, 34, - /* 510 */ 32, 31, 30, 147, 1307, 161, 160, 1791, 35, 33, - /* 520 */ 1197, 1663, 1155, 1156, 195, 1470, 306, 351, 1173, 350, - /* 530 */ 1177, 1178, 493, 1224, 1225, 1227, 1228, 1229, 1230, 1231, - /* 540 */ 512, 534, 1239, 1240, 1241, 1242, 1243, 1244, 253, 1679, - /* 550 */ 1633, 1524, 533, 1171, 1383, 1621, 533, 517, 1382, 1250, - /* 560 */ 277, 114, 1195, 1005, 367, 1197, 1479, 516, 106, 375, - /* 570 */ 1179, 1633, 387, 466, 533, 431, 497, 1422, 1211, 469, - /* 580 */ 1007, 1494, 569, 150, 1381, 1494, 368, 2, 1692, 343, - /* 590 */ 388, 82, 1664, 519, 1666, 1667, 515, 1481, 536, 112, - /* 600 */ 331, 1732, 1477, 1494, 1633, 280, 1728, 198, 1633, 615, - /* 610 */ 345, 341, 533, 495, 144, 1739, 1740, 1794, 1744, 1172, - /* 620 */ 1794, 940, 941, 1538, 106, 1376, 11, 10, 222, 147, - /* 630 */ 317, 436, 147, 1791, 1633, 1536, 1791, 591, 590, 589, - /* 640 */ 321, 1494, 588, 587, 586, 116, 581, 580, 579, 578, - /* 650 */ 577, 576, 575, 574, 123, 570, 32, 31, 30, 1380, - /* 660 */ 386, 1299, 1174, 381, 380, 379, 378, 377, 374, 373, - /* 670 */ 372, 371, 370, 366, 365, 364, 363, 362, 361, 360, - /* 680 */ 359, 499, 1663, 520, 1379, 1226, 1197, 1177, 1178, 1584, - /* 690 */ 1224, 1225, 1227, 1228, 1229, 1230, 1231, 512, 534, 1239, - /* 700 */ 1240, 1241, 1242, 1243, 1244, 1264, 131, 1276, 1378, 1633, - /* 710 */ 1679, 259, 573, 533, 1466, 7, 511, 533, 496, 1471, - /* 720 */ 1200, 533, 459, 257, 53, 409, 1269, 52, 516, 1491, - /* 730 */ 1226, 533, 1633, 1610, 1633, 533, 36, 34, 32, 31, - /* 740 */ 30, 1377, 1494, 244, 162, 1374, 1494, 467, 1663, 1692, - /* 750 */ 1494, 501, 83, 1664, 519, 1666, 1667, 515, 1633, 536, - /* 760 */ 1494, 1373, 1732, 1794, 1494, 25, 299, 1728, 143, 55, - /* 770 */ 36, 34, 32, 31, 30, 147, 1679, 1469, 533, 1791, - /* 780 */ 223, 533, 281, 130, 517, 475, 1759, 435, 533, 1372, - /* 790 */ 530, 1633, 1497, 531, 516, 1633, 185, 1371, 1633, 183, - /* 800 */ 320, 476, 1370, 1369, 1663, 81, 1211, 1494, 1368, 434, - /* 810 */ 1494, 1633, 567, 504, 1262, 1692, 281, 1494, 274, 1664, - /* 820 */ 519, 1666, 1667, 515, 509, 536, 1751, 1295, 493, 561, - /* 830 */ 457, 566, 1679, 565, 564, 563, 59, 58, 355, 1633, - /* 840 */ 514, 156, 1452, 455, 1538, 207, 349, 1633, 1262, 584, - /* 850 */ 516, 1226, 1633, 1633, 1633, 1367, 1537, 114, 1633, 276, - /* 860 */ 1173, 226, 339, 1263, 337, 333, 329, 153, 324, 1298, - /* 870 */ 567, 1692, 1412, 1182, 273, 1664, 519, 1666, 1667, 515, - /* 880 */ 513, 536, 510, 1704, 1268, 1171, 1366, 1407, 1295, 566, - /* 890 */ 346, 565, 564, 563, 446, 112, 120, 1263, 187, 150, - /* 900 */ 189, 186, 1179, 188, 191, 1633, 1663, 190, 46, 448, - /* 910 */ 146, 1739, 1740, 1405, 1744, 11, 10, 1653, 1268, 1358, - /* 920 */ 1359, 480, 1181, 27, 304, 1257, 1258, 1259, 1260, 1261, - /* 930 */ 1265, 1266, 1267, 210, 1679, 451, 1633, 471, 502, 1185, - /* 940 */ 78, 615, 496, 37, 37, 460, 37, 1254, 233, 1321, - /* 950 */ 74, 1172, 516, 217, 1655, 1680, 1633, 27, 304, 1257, - /* 960 */ 1258, 1259, 1260, 1261, 1265, 1266, 1267, 118, 1663, 1396, - /* 970 */ 119, 1533, 120, 1692, 212, 46, 83, 1664, 519, 1666, - /* 980 */ 1667, 515, 977, 536, 1270, 1232, 1732, 1128, 1184, 235, - /* 990 */ 299, 1728, 143, 541, 1174, 428, 1679, 1762, 494, 978, - /* 1000 */ 1663, 225, 505, 119, 517, 228, 120, 230, 525, 3, - /* 1010 */ 1760, 241, 5, 1036, 516, 121, 252, 119, 1633, 1177, - /* 1020 */ 1178, 323, 1195, 326, 330, 286, 287, 1005, 1679, 249, - /* 1030 */ 1139, 369, 1573, 376, 1064, 1692, 517, 158, 83, 1664, - /* 1040 */ 519, 1666, 1667, 515, 1068, 536, 516, 1074, 1732, 384, - /* 1050 */ 1633, 389, 299, 1728, 1807, 383, 1072, 1201, 122, 385, - /* 1060 */ 1663, 390, 1204, 1766, 401, 398, 165, 1692, 402, 167, - /* 1070 */ 83, 1664, 519, 1666, 1667, 515, 1203, 536, 1205, 404, - /* 1080 */ 1732, 403, 170, 406, 299, 1728, 1807, 172, 1679, 407, - /* 1090 */ 1202, 175, 1663, 408, 62, 1789, 517, 411, 178, 430, - /* 1100 */ 432, 1484, 182, 86, 1480, 184, 516, 295, 124, 1179, - /* 1110 */ 1633, 125, 1482, 1478, 1615, 126, 127, 1614, 196, 461, - /* 1120 */ 1679, 199, 250, 202, 465, 462, 1200, 1692, 517, 468, - /* 1130 */ 83, 1664, 519, 1666, 1667, 515, 472, 536, 516, 205, - /* 1140 */ 1732, 481, 1633, 1663, 299, 1728, 1807, 497, 470, 478, - /* 1150 */ 473, 523, 1773, 1772, 298, 1750, 490, 208, 1763, 1692, - /* 1160 */ 211, 6, 264, 1664, 519, 1666, 1667, 515, 1753, 536, - /* 1170 */ 484, 1679, 216, 1663, 477, 113, 1295, 218, 1199, 517, - /* 1180 */ 40, 300, 1747, 506, 503, 18, 527, 1582, 1794, 516, - /* 1190 */ 137, 521, 522, 1633, 219, 1581, 526, 308, 497, 237, - /* 1200 */ 149, 1679, 528, 224, 1791, 251, 1663, 1495, 1713, 517, - /* 1210 */ 1692, 1810, 1790, 264, 1664, 519, 1666, 1667, 515, 516, - /* 1220 */ 536, 239, 71, 1633, 73, 539, 246, 254, 500, 614, - /* 1230 */ 136, 227, 1467, 229, 1679, 507, 1663, 265, 47, 1794, - /* 1240 */ 1692, 256, 517, 84, 1664, 519, 1666, 1667, 515, 258, - /* 1250 */ 536, 147, 516, 1732, 275, 1791, 1633, 508, 1728, 266, - /* 1260 */ 1627, 1626, 57, 1625, 1679, 325, 1622, 327, 328, 332, - /* 1270 */ 1166, 1167, 517, 1692, 154, 1620, 134, 1664, 519, 1666, - /* 1280 */ 1667, 515, 516, 536, 334, 335, 1633, 1663, 336, 1619, - /* 1290 */ 338, 1618, 340, 1617, 342, 1616, 1663, 344, 1600, 155, - /* 1300 */ 1142, 347, 1141, 1692, 348, 1594, 84, 1664, 519, 1666, - /* 1310 */ 1667, 515, 1593, 536, 353, 1679, 1732, 354, 1592, 498, - /* 1320 */ 1808, 1729, 1591, 517, 1679, 1111, 1566, 1565, 1564, 1563, - /* 1330 */ 1562, 1561, 517, 516, 1560, 1559, 1558, 1633, 1557, 1556, - /* 1340 */ 1555, 1554, 516, 1553, 1552, 1551, 1633, 1550, 1549, 117, - /* 1350 */ 1663, 1548, 1547, 1546, 1692, 1545, 1544, 269, 1664, 519, - /* 1360 */ 1666, 1667, 515, 1692, 536, 1543, 134, 1664, 519, 1666, - /* 1370 */ 1667, 515, 1113, 536, 1542, 1541, 1540, 1539, 1679, 1424, - /* 1380 */ 1392, 943, 163, 108, 942, 1391, 517, 1608, 1602, 1590, - /* 1390 */ 1663, 171, 1589, 394, 489, 140, 516, 164, 109, 169, - /* 1400 */ 1633, 396, 1579, 303, 45, 174, 1663, 1473, 1423, 1421, - /* 1410 */ 1809, 1419, 1417, 412, 413, 414, 417, 1692, 1679, 971, - /* 1420 */ 274, 1664, 519, 1666, 1667, 515, 514, 536, 416, 420, - /* 1430 */ 418, 421, 422, 1415, 1679, 426, 516, 424, 1404, 425, - /* 1440 */ 1633, 1403, 517, 1390, 1475, 1077, 181, 1078, 1474, 1413, - /* 1450 */ 1004, 582, 516, 1003, 1663, 584, 1633, 1692, 292, 305, - /* 1460 */ 273, 1664, 519, 1666, 1667, 515, 1408, 536, 1002, 1705, - /* 1470 */ 449, 1001, 998, 1692, 997, 996, 274, 1664, 519, 1666, - /* 1480 */ 1667, 515, 1679, 536, 293, 1406, 1663, 294, 1389, 452, - /* 1490 */ 517, 180, 1388, 454, 456, 85, 1607, 1149, 1601, 463, - /* 1500 */ 516, 1588, 1587, 141, 1633, 1586, 1578, 307, 49, 427, - /* 1510 */ 423, 419, 415, 179, 1679, 67, 1663, 204, 464, 4, - /* 1520 */ 37, 1692, 517, 206, 274, 1664, 519, 1666, 1667, 515, - /* 1530 */ 15, 536, 516, 128, 209, 43, 1633, 1320, 64, 200, - /* 1540 */ 135, 177, 213, 215, 1679, 22, 48, 1653, 214, 1313, - /* 1550 */ 68, 23, 517, 1692, 42, 1292, 260, 1664, 519, 1666, - /* 1560 */ 1667, 515, 516, 536, 221, 1291, 1633, 1663, 138, 1349, - /* 1570 */ 17, 1338, 1344, 1343, 301, 1348, 10, 1347, 302, 19, - /* 1580 */ 1255, 139, 1234, 1692, 151, 29, 268, 1664, 519, 1666, - /* 1590 */ 1667, 515, 12, 536, 1233, 1679, 20, 1219, 176, 16, - /* 1600 */ 168, 518, 173, 517, 405, 41, 13, 1663, 1577, 238, - /* 1610 */ 74, 524, 1652, 516, 232, 21, 234, 1633, 1318, 1189, - /* 1620 */ 236, 240, 166, 69, 70, 243, 1695, 540, 1236, 535, - /* 1630 */ 44, 1071, 1065, 538, 1692, 1679, 315, 270, 1664, 519, - /* 1640 */ 1666, 1667, 515, 517, 536, 542, 544, 1663, 1062, 1059, - /* 1650 */ 545, 547, 548, 516, 550, 1053, 553, 1633, 551, 1057, - /* 1660 */ 1051, 1056, 1663, 554, 1055, 1054, 75, 1042, 76, 560, - /* 1670 */ 1073, 77, 1070, 568, 1692, 1679, 969, 261, 1664, 519, - /* 1680 */ 1666, 1667, 515, 517, 536, 993, 1011, 245, 991, 571, - /* 1690 */ 1679, 986, 1008, 516, 990, 989, 988, 1633, 517, 987, - /* 1700 */ 985, 984, 1006, 981, 980, 979, 976, 975, 516, 974, - /* 1710 */ 1420, 592, 1633, 1663, 1692, 593, 594, 271, 1664, 519, - /* 1720 */ 1666, 1667, 515, 1418, 536, 1663, 597, 596, 598, 1692, - /* 1730 */ 1416, 600, 262, 1664, 519, 1666, 1667, 515, 601, 536, - /* 1740 */ 602, 1679, 604, 605, 606, 1402, 608, 609, 1401, 517, - /* 1750 */ 1414, 1387, 613, 1679, 612, 1175, 255, 616, 617, 516, - /* 1760 */ 1362, 517, 1362, 1633, 1362, 1663, 1362, 1362, 1362, 1362, - /* 1770 */ 1362, 516, 1362, 1362, 1362, 1633, 1663, 1362, 1362, 1362, - /* 1780 */ 1692, 1362, 1362, 272, 1664, 519, 1666, 1667, 515, 1362, - /* 1790 */ 536, 1362, 1692, 1679, 1362, 263, 1664, 519, 1666, 1667, - /* 1800 */ 515, 517, 536, 1362, 1679, 1362, 1362, 1362, 1362, 1362, - /* 1810 */ 1362, 516, 517, 1362, 1362, 1633, 1362, 1362, 1362, 1362, - /* 1820 */ 1362, 1362, 516, 1362, 1362, 1362, 1633, 1663, 1362, 1362, - /* 1830 */ 1362, 1362, 1692, 1362, 1362, 1675, 1664, 519, 1666, 1667, - /* 1840 */ 515, 1362, 536, 1692, 1362, 1362, 1674, 1664, 519, 1666, - /* 1850 */ 1667, 515, 1362, 536, 1362, 1679, 1362, 1663, 1362, 1362, - /* 1860 */ 1362, 1362, 1362, 517, 1362, 1362, 1362, 1362, 1362, 1362, - /* 1870 */ 1362, 1362, 1362, 516, 1362, 1362, 1362, 1633, 1362, 1362, - /* 1880 */ 1362, 1362, 1362, 1362, 1362, 1679, 1362, 1663, 1362, 1362, - /* 1890 */ 1362, 1362, 1362, 517, 1692, 1362, 1362, 1673, 1664, 519, - /* 1900 */ 1666, 1667, 515, 516, 536, 1362, 1362, 1633, 1362, 1362, - /* 1910 */ 1362, 1362, 1362, 1362, 1362, 1679, 1362, 1362, 1362, 1663, - /* 1920 */ 1362, 1362, 1362, 517, 1692, 1362, 1362, 284, 1664, 519, - /* 1930 */ 1666, 1667, 515, 516, 536, 1362, 1362, 1633, 1362, 1362, - /* 1940 */ 1362, 1362, 1362, 313, 312, 1362, 1362, 1679, 1362, 1362, - /* 1950 */ 1362, 1362, 1663, 1187, 1692, 517, 1362, 283, 1664, 519, - /* 1960 */ 1666, 1667, 515, 1362, 536, 516, 1362, 1362, 1362, 1633, - /* 1970 */ 1362, 1362, 1362, 1362, 1362, 1362, 1362, 1362, 1180, 1362, - /* 1980 */ 1679, 1362, 1362, 1362, 1362, 1663, 1692, 1362, 517, 285, - /* 1990 */ 1664, 519, 1666, 1667, 515, 1179, 536, 1362, 516, 1362, - /* 2000 */ 1362, 1362, 1633, 1362, 1362, 1362, 1362, 1362, 1362, 1362, - /* 2010 */ 1362, 1362, 1362, 1679, 1362, 1362, 493, 1362, 1362, 1692, - /* 2020 */ 1362, 517, 282, 1664, 519, 1666, 1667, 515, 1362, 536, - /* 2030 */ 1362, 516, 1362, 1362, 537, 1633, 1362, 1362, 1362, 1362, - /* 2040 */ 1362, 1362, 1362, 1362, 1183, 114, 493, 1362, 1362, 1362, - /* 2050 */ 1362, 1362, 1692, 1362, 1362, 267, 1664, 519, 1666, 1667, - /* 2060 */ 515, 1362, 536, 1362, 497, 1362, 1362, 1362, 1362, 1362, - /* 2070 */ 1362, 1362, 1362, 1362, 1362, 114, 1362, 1362, 1362, 1362, - /* 2080 */ 1362, 1362, 1362, 112, 1362, 1362, 1362, 1188, 1362, 1362, - /* 2090 */ 1362, 1362, 1362, 1362, 497, 1362, 1362, 1362, 220, 1739, - /* 2100 */ 492, 1362, 491, 1362, 1362, 1794, 1362, 1362, 1362, 1362, - /* 2110 */ 1362, 1362, 1191, 112, 1362, 1362, 1362, 149, 1362, 1362, - /* 2120 */ 1362, 1791, 1362, 534, 1239, 1240, 1362, 1362, 220, 1739, - /* 2130 */ 492, 1362, 491, 1362, 1362, 1794, 1362, 1362, 1362, 1362, - /* 2140 */ 1362, 1362, 1362, 1362, 1362, 1362, 1362, 147, 1362, 1362, - /* 2150 */ 1362, 1791, + /* 0 */ 1688, 1836, 1515, 309, 407, 1688, 408, 1427, 326, 1393, + /* 10 */ 137, 1685, 37, 35, 1483, 1835, 1685, 1701, 412, 1833, + /* 20 */ 318, 551, 1201, 551, 1223, 38, 36, 34, 33, 32, + /* 30 */ 325, 324, 109, 415, 109, 408, 1427, 1681, 1687, 446, + /* 40 */ 1215, 451, 1681, 1687, 1717, 30, 240, 1199, 554, 1526, + /* 50 */ 1784, 1526, 535, 554, 22, 551, 550, 1671, 14, 534, + /* 60 */ 37, 35, 1328, 322, 1207, 1208, 157, 335, 318, 1701, + /* 70 */ 1201, 134, 515, 365, 1781, 38, 36, 34, 33, 32, + /* 80 */ 1528, 1, 1207, 1526, 1730, 67, 550, 85, 1702, 537, + /* 90 */ 1704, 1705, 533, 58, 554, 1199, 1717, 1770, 113, 76, + /* 100 */ 371, 292, 1766, 632, 514, 551, 14, 1521, 1836, 1671, + /* 110 */ 1418, 534, 1207, 1836, 1836, 538, 369, 1270, 1271, 958, + /* 120 */ 1519, 555, 152, 465, 550, 1622, 1833, 154, 152, 2, + /* 130 */ 425, 1833, 1833, 1526, 206, 82, 1730, 290, 473, 86, + /* 140 */ 1702, 537, 1704, 1705, 533, 1362, 554, 40, 118, 1770, + /* 150 */ 1671, 632, 198, 311, 1766, 148, 1518, 962, 963, 1202, + /* 160 */ 58, 1200, 70, 328, 468, 1270, 1271, 232, 56, 462, + /* 170 */ 364, 134, 363, 1226, 197, 490, 1797, 1216, 330, 1211, + /* 180 */ 1528, 1571, 1573, 1205, 1206, 41, 1252, 1253, 1255, 1256, + /* 190 */ 1257, 1258, 1259, 530, 552, 1267, 1268, 1269, 1272, 53, + /* 200 */ 1225, 1219, 52, 1572, 1573, 26, 59, 1202, 1836, 1200, + /* 210 */ 505, 155, 552, 1267, 1268, 38, 36, 34, 33, 32, + /* 220 */ 37, 35, 153, 155, 1417, 1701, 1833, 397, 318, 67, + /* 230 */ 1201, 1205, 1206, 302, 1252, 1253, 1255, 1256, 1257, 1258, + /* 240 */ 1259, 530, 552, 1267, 1268, 1269, 1272, 511, 1717, 460, + /* 250 */ 459, 1522, 1717, 484, 458, 1199, 504, 114, 455, 1659, + /* 260 */ 535, 454, 453, 452, 1671, 1671, 14, 534, 37, 35, + /* 270 */ 1107, 1108, 1207, 166, 165, 117, 318, 155, 1201, 1292, + /* 280 */ 515, 1504, 303, 58, 301, 300, 356, 448, 58, 2, + /* 290 */ 155, 450, 1730, 503, 1836, 85, 1702, 537, 1704, 1705, + /* 300 */ 533, 1297, 554, 1199, 344, 1770, 358, 354, 152, 292, + /* 310 */ 1766, 632, 1833, 449, 115, 38, 36, 34, 33, 32, + /* 320 */ 1207, 1836, 136, 1517, 1407, 1270, 1271, 1502, 513, 149, + /* 330 */ 1777, 1778, 1327, 1782, 1685, 152, 27, 8, 425, 1833, + /* 340 */ 1065, 577, 576, 575, 1069, 574, 1071, 1072, 573, 1074, + /* 350 */ 570, 1396, 1080, 567, 1082, 1083, 564, 561, 81, 632, + /* 360 */ 1681, 1687, 38, 36, 34, 33, 32, 1202, 78, 1200, + /* 370 */ 1352, 554, 99, 1270, 1271, 98, 97, 96, 95, 94, + /* 380 */ 93, 92, 91, 90, 588, 1342, 38, 36, 34, 33, + /* 390 */ 32, 1205, 1206, 517, 1252, 1253, 1255, 1256, 1257, 1258, + /* 400 */ 1259, 530, 552, 1267, 1268, 1269, 1272, 497, 1350, 1351, + /* 410 */ 1353, 1354, 975, 155, 974, 1202, 99, 1200, 155, 98, + /* 420 */ 97, 96, 95, 94, 93, 92, 91, 90, 37, 35, + /* 430 */ 1273, 406, 414, 1701, 410, 410, 318, 162, 1201, 1205, + /* 440 */ 1206, 976, 1252, 1253, 1255, 1256, 1257, 1258, 1259, 530, + /* 450 */ 552, 1267, 1268, 1269, 1272, 38, 36, 34, 33, 32, + /* 460 */ 1717, 474, 65, 1199, 1836, 64, 1027, 155, 535, 975, + /* 470 */ 199, 974, 1578, 1671, 1416, 534, 37, 35, 1834, 308, + /* 480 */ 1207, 1454, 1833, 1029, 318, 1784, 1201, 592, 1576, 1498, + /* 490 */ 38, 36, 34, 33, 32, 1415, 444, 9, 976, 500, + /* 500 */ 1730, 1388, 1836, 87, 1702, 537, 1704, 1705, 533, 1780, + /* 510 */ 554, 1199, 200, 1770, 1671, 1226, 152, 1769, 1766, 632, + /* 520 */ 1833, 538, 133, 1578, 147, 1239, 69, 291, 1207, 321, + /* 530 */ 323, 1621, 588, 1270, 1271, 1671, 293, 1565, 1414, 1576, + /* 540 */ 1326, 610, 609, 608, 333, 9, 607, 606, 605, 119, + /* 550 */ 600, 599, 598, 597, 596, 595, 594, 593, 127, 589, + /* 560 */ 1239, 1395, 38, 36, 34, 33, 32, 632, 1290, 506, + /* 570 */ 501, 1183, 1184, 1227, 1335, 1202, 1413, 1200, 1671, 1412, + /* 580 */ 1225, 1270, 1271, 1387, 28, 108, 107, 106, 105, 104, + /* 590 */ 103, 102, 101, 100, 38, 36, 34, 33, 32, 1205, + /* 600 */ 1206, 1411, 1252, 1253, 1255, 1256, 1257, 1258, 1259, 530, + /* 610 */ 552, 1267, 1268, 1269, 1272, 551, 1671, 1578, 134, 1671, + /* 620 */ 1291, 1224, 331, 1202, 329, 1200, 370, 1529, 1278, 1410, + /* 630 */ 134, 457, 456, 1576, 1225, 1207, 37, 35, 1457, 1528, + /* 640 */ 1228, 1671, 1296, 1526, 318, 1503, 1201, 1205, 1206, 527, + /* 650 */ 1252, 1253, 1255, 1256, 1257, 1258, 1259, 530, 552, 1267, + /* 660 */ 1268, 1269, 1272, 11, 10, 34, 33, 32, 1784, 1671, + /* 670 */ 580, 1199, 38, 36, 34, 33, 32, 29, 316, 1285, + /* 680 */ 1286, 1287, 1288, 1289, 1293, 1294, 1295, 584, 1207, 511, + /* 690 */ 1569, 1689, 1779, 604, 602, 1409, 7, 551, 460, 459, + /* 700 */ 586, 231, 1685, 458, 551, 2, 114, 455, 372, 450, + /* 710 */ 454, 453, 452, 1406, 1578, 382, 1225, 117, 293, 125, + /* 720 */ 124, 583, 582, 581, 289, 1526, 1223, 632, 1681, 1687, + /* 730 */ 1577, 449, 1526, 390, 551, 1671, 402, 515, 586, 554, + /* 740 */ 585, 1270, 1271, 1569, 1304, 383, 135, 1254, 1405, 1404, + /* 750 */ 1290, 269, 263, 1671, 403, 1556, 115, 125, 124, 583, + /* 760 */ 582, 581, 1526, 267, 55, 1403, 551, 54, 212, 551, + /* 770 */ 551, 229, 1777, 510, 1282, 509, 551, 424, 1836, 1612, + /* 780 */ 1523, 1648, 522, 1202, 167, 1200, 373, 253, 1671, 1671, + /* 790 */ 164, 591, 154, 519, 1526, 551, 1833, 1526, 1526, 962, + /* 800 */ 963, 1254, 1291, 1511, 1526, 1671, 482, 1205, 1206, 58, + /* 810 */ 1252, 1253, 1255, 1256, 1257, 1258, 1259, 530, 552, 1267, + /* 820 */ 1268, 1269, 1272, 1526, 1296, 1513, 401, 1789, 1323, 396, + /* 830 */ 395, 394, 393, 392, 389, 388, 387, 386, 385, 381, + /* 840 */ 380, 379, 378, 377, 376, 375, 374, 84, 1402, 1509, + /* 850 */ 190, 1701, 1201, 188, 203, 1401, 1400, 1399, 1398, 29, + /* 860 */ 316, 1285, 1286, 1287, 1288, 1289, 1293, 1294, 1295, 1323, + /* 870 */ 603, 192, 194, 196, 191, 193, 195, 1199, 1717, 62, + /* 880 */ 61, 368, 529, 1254, 161, 999, 514, 1501, 1671, 551, + /* 890 */ 362, 1671, 1701, 534, 1207, 1671, 1671, 1671, 1671, 472, + /* 900 */ 548, 288, 1000, 123, 352, 48, 350, 346, 342, 158, + /* 910 */ 337, 359, 470, 216, 11, 10, 1210, 1526, 1730, 1717, + /* 920 */ 1209, 86, 1702, 537, 1704, 1705, 533, 535, 554, 551, + /* 930 */ 1444, 1770, 1671, 632, 534, 311, 1766, 148, 39, 155, + /* 940 */ 549, 1390, 1391, 579, 486, 39, 1349, 1433, 481, 39, + /* 950 */ 1408, 1484, 461, 1439, 218, 1691, 235, 1526, 1798, 1730, + /* 960 */ 551, 1437, 86, 1702, 537, 1704, 1705, 533, 242, 554, + /* 970 */ 1701, 332, 1770, 523, 121, 463, 311, 1766, 1849, 1298, + /* 980 */ 586, 122, 520, 466, 123, 498, 1260, 1804, 1526, 1202, + /* 990 */ 1156, 1200, 1693, 628, 475, 443, 1701, 1717, 223, 125, + /* 1000 */ 124, 583, 582, 581, 1718, 535, 334, 1428, 48, 244, + /* 1010 */ 1671, 559, 534, 1205, 1206, 543, 1800, 122, 123, 110, + /* 1020 */ 122, 1566, 250, 1717, 512, 1058, 3, 234, 237, 239, + /* 1030 */ 1213, 535, 5, 336, 1212, 1223, 1671, 1730, 534, 343, + /* 1040 */ 86, 1702, 537, 1704, 1705, 533, 339, 554, 1701, 262, + /* 1050 */ 1770, 1027, 1086, 298, 311, 1766, 1849, 1701, 1090, 1097, + /* 1060 */ 1095, 126, 299, 1730, 1167, 1827, 86, 1702, 537, 1704, + /* 1070 */ 1705, 533, 259, 554, 384, 1717, 1770, 1614, 163, 399, + /* 1080 */ 311, 1766, 1849, 535, 1717, 391, 398, 400, 1671, 404, + /* 1090 */ 534, 1788, 535, 1229, 405, 413, 1232, 1671, 416, 534, + /* 1100 */ 170, 417, 172, 515, 1231, 418, 1701, 1233, 419, 175, + /* 1110 */ 421, 177, 515, 422, 1230, 1730, 423, 426, 278, 1702, + /* 1120 */ 537, 1704, 1705, 533, 1730, 554, 180, 278, 1702, 537, + /* 1130 */ 1704, 1705, 533, 1717, 554, 66, 183, 445, 447, 1516, + /* 1140 */ 187, 535, 307, 1512, 1836, 89, 1671, 1701, 534, 189, + /* 1150 */ 201, 260, 476, 1836, 128, 129, 1514, 1510, 154, 1653, + /* 1160 */ 130, 131, 1833, 477, 204, 1701, 1652, 152, 483, 207, + /* 1170 */ 1228, 1833, 487, 1730, 1717, 485, 87, 1702, 537, 1704, + /* 1180 */ 1705, 533, 532, 554, 488, 480, 1770, 1671, 210, 534, + /* 1190 */ 526, 1766, 1717, 499, 494, 214, 1811, 496, 541, 1810, + /* 1200 */ 535, 217, 310, 502, 1791, 1671, 6, 534, 1801, 508, + /* 1210 */ 495, 492, 222, 493, 1730, 1323, 1701, 286, 1702, 537, + /* 1220 */ 1704, 1705, 533, 531, 554, 528, 1742, 1227, 116, 227, + /* 1230 */ 42, 524, 1730, 142, 635, 138, 1702, 537, 1704, 1705, + /* 1240 */ 533, 1785, 554, 1717, 312, 226, 224, 521, 258, 18, + /* 1250 */ 1620, 535, 1619, 228, 539, 544, 1671, 1751, 534, 225, + /* 1260 */ 145, 1832, 545, 233, 246, 546, 626, 622, 618, 614, + /* 1270 */ 257, 540, 1527, 320, 248, 261, 75, 1701, 77, 557, + /* 1280 */ 516, 1850, 518, 1730, 525, 1852, 87, 1702, 537, 1704, + /* 1290 */ 1705, 533, 236, 554, 1701, 83, 1770, 1570, 251, 238, + /* 1300 */ 264, 1767, 1499, 631, 1717, 256, 141, 511, 270, 49, + /* 1310 */ 279, 271, 535, 266, 268, 1665, 1664, 1671, 60, 534, + /* 1320 */ 1663, 1717, 338, 1660, 340, 341, 1194, 1195, 159, 535, + /* 1330 */ 491, 547, 345, 1658, 1671, 117, 534, 347, 348, 349, + /* 1340 */ 1657, 351, 1656, 353, 1730, 1701, 355, 287, 1702, 537, + /* 1350 */ 1704, 1705, 533, 1655, 554, 515, 1654, 357, 1638, 489, + /* 1360 */ 360, 1730, 208, 361, 282, 1702, 537, 1704, 1705, 533, + /* 1370 */ 160, 554, 1717, 511, 115, 1170, 1632, 1169, 1631, 366, + /* 1380 */ 535, 1175, 367, 202, 1630, 1671, 1629, 534, 1142, 229, + /* 1390 */ 1777, 510, 1607, 509, 63, 1606, 1836, 1605, 1604, 1701, + /* 1400 */ 1603, 117, 1602, 507, 1601, 1600, 1599, 1598, 1701, 1597, + /* 1410 */ 152, 1596, 1730, 1595, 1833, 138, 1702, 537, 1704, 1705, + /* 1420 */ 533, 1594, 554, 1593, 1592, 1591, 1717, 1590, 1589, 120, + /* 1430 */ 1588, 1587, 1586, 1585, 535, 1717, 1144, 1582, 1581, 1671, + /* 1440 */ 115, 534, 1584, 532, 1583, 1580, 1579, 1456, 1671, 1424, + /* 1450 */ 534, 146, 315, 111, 965, 150, 1777, 1778, 168, 1782, + /* 1460 */ 1423, 1851, 1701, 409, 964, 1646, 1730, 169, 1640, 287, + /* 1470 */ 1702, 537, 1704, 1705, 533, 1730, 554, 511, 286, 1702, + /* 1480 */ 537, 1704, 1705, 533, 1628, 554, 176, 1743, 112, 1717, + /* 1490 */ 1627, 174, 411, 1617, 179, 1505, 1455, 535, 993, 1453, + /* 1500 */ 185, 429, 1671, 427, 534, 117, 428, 1451, 433, 1449, + /* 1510 */ 431, 432, 140, 1701, 437, 317, 435, 436, 442, 438, + /* 1520 */ 434, 430, 184, 1447, 440, 439, 441, 1701, 1436, 1730, + /* 1530 */ 1435, 1422, 287, 1702, 537, 1704, 1705, 533, 1101, 554, + /* 1540 */ 1717, 1507, 47, 186, 115, 1506, 1100, 68, 535, 1026, + /* 1550 */ 182, 1025, 1024, 1671, 1717, 534, 601, 1023, 603, 151, + /* 1560 */ 1777, 1778, 535, 1782, 1445, 1020, 319, 1671, 1019, 534, + /* 1570 */ 304, 1440, 1018, 305, 464, 1438, 306, 1421, 469, 467, + /* 1580 */ 1730, 1420, 1701, 287, 1702, 537, 1704, 1705, 533, 471, + /* 1590 */ 554, 1701, 88, 1645, 1730, 1639, 1177, 272, 1702, 537, + /* 1600 */ 1704, 1705, 533, 1626, 554, 1701, 51, 478, 132, 1717, + /* 1610 */ 181, 1625, 173, 1624, 178, 205, 420, 535, 1717, 1616, + /* 1620 */ 71, 479, 1671, 50, 534, 209, 535, 4, 211, 1364, + /* 1630 */ 39, 1671, 1717, 534, 23, 171, 221, 15, 213, 45, + /* 1640 */ 535, 215, 1348, 139, 219, 1671, 24, 534, 1341, 1730, + /* 1650 */ 220, 1691, 273, 1702, 537, 1704, 1705, 533, 1730, 554, + /* 1660 */ 72, 274, 1702, 537, 1704, 1705, 533, 25, 554, 1320, + /* 1670 */ 44, 230, 1730, 1319, 16, 281, 1702, 537, 1704, 1705, + /* 1680 */ 533, 1701, 554, 43, 143, 17, 1381, 1370, 1376, 1375, + /* 1690 */ 313, 13, 1380, 1379, 314, 10, 1283, 19, 1701, 144, + /* 1700 */ 1262, 31, 1247, 1261, 12, 20, 156, 1701, 1717, 21, + /* 1710 */ 1615, 243, 536, 241, 1346, 245, 535, 73, 247, 542, + /* 1720 */ 74, 1671, 1690, 534, 78, 1717, 249, 252, 1217, 1733, + /* 1730 */ 556, 1264, 553, 535, 1717, 46, 558, 327, 1671, 1087, + /* 1740 */ 534, 560, 535, 1084, 562, 563, 565, 1671, 1730, 534, + /* 1750 */ 1081, 283, 1702, 537, 1704, 1705, 533, 566, 554, 1701, + /* 1760 */ 568, 1075, 571, 569, 1079, 1730, 1073, 572, 275, 1702, + /* 1770 */ 537, 1704, 1705, 533, 1730, 554, 1078, 284, 1702, 537, + /* 1780 */ 1704, 1705, 533, 1701, 554, 1064, 1717, 578, 79, 80, + /* 1790 */ 1077, 1096, 1076, 57, 535, 254, 1092, 991, 587, 1671, + /* 1800 */ 1701, 534, 1015, 1033, 590, 255, 1013, 1012, 1011, 1010, + /* 1810 */ 1717, 1008, 1009, 1007, 1006, 1030, 1028, 1003, 535, 1002, + /* 1820 */ 1001, 1452, 998, 1671, 997, 534, 1730, 1717, 996, 276, + /* 1830 */ 1702, 537, 1704, 1705, 533, 535, 554, 611, 612, 613, + /* 1840 */ 1671, 1450, 534, 615, 616, 617, 1448, 621, 619, 620, + /* 1850 */ 1730, 1701, 1446, 285, 1702, 537, 1704, 1705, 533, 623, + /* 1860 */ 554, 624, 1701, 625, 1434, 627, 1419, 1730, 634, 629, + /* 1870 */ 277, 1702, 537, 1704, 1705, 533, 630, 554, 1717, 1203, + /* 1880 */ 265, 633, 1394, 1394, 1394, 1394, 535, 1394, 1394, 1717, + /* 1890 */ 1394, 1671, 1394, 534, 1394, 1394, 1394, 535, 1394, 1394, + /* 1900 */ 1394, 1394, 1671, 1701, 534, 1394, 1394, 1394, 1394, 1394, + /* 1910 */ 1394, 1394, 1394, 1394, 1701, 1394, 1394, 1394, 1730, 1394, + /* 1920 */ 1394, 1713, 1702, 537, 1704, 1705, 533, 1394, 554, 1730, + /* 1930 */ 1717, 1394, 1712, 1702, 537, 1704, 1705, 533, 535, 554, + /* 1940 */ 1394, 1717, 1394, 1671, 1394, 534, 1394, 1394, 1394, 535, + /* 1950 */ 1394, 1394, 1394, 1394, 1671, 1394, 534, 1394, 1394, 1394, + /* 1960 */ 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1701, 1394, 1394, + /* 1970 */ 1730, 1394, 1394, 1711, 1702, 537, 1704, 1705, 533, 1701, + /* 1980 */ 554, 1730, 1394, 1394, 296, 1702, 537, 1704, 1705, 533, + /* 1990 */ 1394, 554, 1394, 1701, 1717, 1394, 1394, 1394, 1394, 1394, + /* 2000 */ 1394, 1394, 535, 1394, 1394, 1394, 1717, 1671, 1394, 534, + /* 2010 */ 1394, 1394, 1394, 1394, 535, 1394, 1394, 1394, 1394, 1671, + /* 2020 */ 1717, 534, 1394, 1394, 1394, 1394, 1394, 1394, 535, 1394, + /* 2030 */ 1394, 1394, 1394, 1671, 1730, 534, 1394, 295, 1702, 537, + /* 2040 */ 1704, 1705, 533, 1394, 554, 1701, 1730, 1394, 1394, 297, + /* 2050 */ 1702, 537, 1704, 1705, 533, 1394, 554, 1394, 1394, 1394, + /* 2060 */ 1730, 1394, 1394, 294, 1702, 537, 1704, 1705, 533, 1394, + /* 2070 */ 554, 1394, 1717, 1394, 1394, 1394, 1394, 1394, 1394, 1394, + /* 2080 */ 535, 1394, 1394, 1394, 1394, 1671, 1394, 534, 1394, 1394, + /* 2090 */ 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, + /* 2100 */ 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, 1394, + /* 2110 */ 1394, 1394, 1730, 1394, 1394, 280, 1702, 537, 1704, 1705, + /* 2120 */ 533, 1394, 554, }; static const YYCODETYPE yy_lookahead[] = { - /* 0 */ 324, 325, 243, 273, 251, 252, 276, 273, 250, 273, - /* 10 */ 276, 272, 12, 13, 298, 12, 13, 14, 15, 16, - /* 20 */ 20, 2, 22, 293, 250, 257, 258, 293, 253, 293, - /* 30 */ 271, 12, 13, 14, 15, 16, 262, 250, 279, 309, - /* 40 */ 310, 266, 284, 309, 310, 309, 310, 47, 289, 274, - /* 50 */ 320, 250, 293, 279, 320, 339, 320, 298, 58, 311, - /* 60 */ 12, 13, 14, 262, 64, 243, 279, 351, 20, 310, - /* 70 */ 22, 355, 313, 314, 315, 316, 317, 318, 20, 320, - /* 80 */ 279, 81, 323, 335, 339, 82, 327, 328, 12, 13, - /* 90 */ 14, 15, 16, 271, 57, 47, 351, 273, 339, 289, - /* 100 */ 355, 279, 292, 103, 317, 295, 58, 311, 12, 13, - /* 110 */ 351, 289, 64, 113, 355, 293, 20, 293, 22, 332, - /* 120 */ 333, 334, 20, 336, 12, 13, 14, 15, 16, 81, - /* 130 */ 255, 335, 310, 309, 310, 313, 314, 315, 316, 317, - /* 140 */ 318, 0, 320, 47, 320, 323, 20, 339, 22, 327, - /* 150 */ 328, 103, 277, 253, 58, 246, 156, 248, 249, 351, - /* 160 */ 64, 113, 21, 355, 20, 24, 25, 26, 27, 28, - /* 170 */ 29, 30, 31, 32, 274, 49, 4, 81, 242, 55, - /* 180 */ 244, 181, 182, 81, 184, 185, 186, 187, 188, 189, - /* 190 */ 190, 191, 192, 193, 194, 195, 196, 197, 198, 103, - /* 200 */ 246, 22, 248, 249, 156, 81, 20, 83, 256, 113, - /* 210 */ 21, 211, 260, 24, 25, 26, 27, 28, 29, 30, - /* 220 */ 31, 32, 165, 166, 148, 81, 47, 170, 93, 181, - /* 230 */ 182, 0, 184, 185, 186, 187, 188, 189, 190, 191, - /* 240 */ 192, 193, 194, 195, 196, 197, 198, 112, 20, 114, - /* 250 */ 115, 116, 156, 81, 255, 24, 25, 26, 27, 28, - /* 260 */ 29, 30, 31, 32, 4, 12, 13, 268, 12, 13, - /* 270 */ 14, 15, 16, 20, 311, 22, 277, 181, 182, 20, - /* 280 */ 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, - /* 290 */ 194, 195, 196, 197, 198, 81, 60, 61, 335, 271, - /* 300 */ 47, 65, 42, 43, 68, 69, 278, 231, 72, 73, - /* 310 */ 74, 283, 57, 12, 13, 14, 271, 64, 2, 263, - /* 320 */ 243, 20, 247, 22, 279, 250, 35, 271, 12, 13, - /* 330 */ 14, 15, 16, 20, 81, 211, 280, 94, 95, 96, - /* 340 */ 97, 98, 99, 100, 101, 102, 103, 104, 47, 106, - /* 350 */ 107, 108, 109, 110, 111, 211, 103, 4, 143, 19, - /* 360 */ 315, 12, 13, 257, 258, 64, 113, 270, 14, 20, - /* 370 */ 293, 22, 19, 33, 20, 84, 181, 86, 87, 282, - /* 380 */ 89, 281, 81, 211, 93, 45, 33, 247, 288, 289, - /* 390 */ 250, 51, 52, 53, 54, 55, 47, 0, 45, 55, - /* 400 */ 250, 145, 240, 50, 103, 0, 115, 64, 55, 156, - /* 410 */ 339, 243, 262, 64, 113, 220, 221, 222, 223, 224, - /* 420 */ 80, 279, 351, 83, 80, 211, 355, 83, 286, 279, - /* 430 */ 81, 216, 217, 80, 181, 182, 83, 184, 185, 186, - /* 440 */ 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, - /* 450 */ 197, 198, 103, 81, 57, 263, 116, 156, 3, 263, - /* 460 */ 298, 293, 113, 271, 211, 60, 61, 271, 271, 92, - /* 470 */ 65, 75, 280, 68, 69, 278, 280, 72, 73, 74, - /* 480 */ 283, 141, 181, 182, 144, 184, 185, 186, 187, 188, - /* 490 */ 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, - /* 500 */ 243, 339, 162, 20, 164, 156, 288, 289, 12, 13, - /* 510 */ 14, 15, 16, 351, 14, 119, 120, 355, 12, 13, - /* 520 */ 20, 243, 167, 168, 169, 0, 20, 155, 22, 157, - /* 530 */ 181, 182, 250, 184, 185, 186, 187, 188, 189, 190, - /* 540 */ 191, 192, 193, 194, 195, 196, 197, 198, 264, 271, - /* 550 */ 293, 267, 250, 47, 243, 0, 250, 279, 243, 14, - /* 560 */ 18, 279, 20, 47, 262, 20, 272, 289, 262, 27, - /* 570 */ 64, 293, 30, 302, 250, 269, 298, 0, 82, 298, - /* 580 */ 64, 279, 57, 211, 243, 279, 262, 81, 310, 151, - /* 590 */ 48, 313, 314, 315, 316, 317, 318, 272, 320, 317, - /* 600 */ 45, 323, 272, 279, 293, 327, 328, 272, 293, 103, - /* 610 */ 172, 173, 250, 331, 332, 333, 334, 339, 336, 113, - /* 620 */ 339, 42, 43, 271, 262, 244, 1, 2, 145, 351, - /* 630 */ 278, 269, 351, 355, 293, 283, 355, 60, 61, 62, - /* 640 */ 63, 279, 65, 66, 67, 68, 69, 70, 71, 72, - /* 650 */ 73, 74, 75, 76, 77, 78, 14, 15, 16, 243, - /* 660 */ 118, 4, 156, 121, 122, 123, 124, 125, 126, 127, - /* 670 */ 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, - /* 680 */ 138, 226, 243, 289, 243, 185, 20, 181, 182, 295, - /* 690 */ 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, - /* 700 */ 194, 195, 196, 197, 198, 139, 18, 82, 243, 293, - /* 710 */ 271, 23, 259, 250, 261, 37, 272, 250, 279, 0, - /* 720 */ 20, 250, 298, 35, 36, 262, 160, 39, 289, 262, - /* 730 */ 185, 250, 293, 262, 293, 250, 12, 13, 14, 15, - /* 740 */ 16, 243, 279, 262, 56, 243, 279, 262, 243, 310, - /* 750 */ 279, 41, 313, 314, 315, 316, 317, 318, 293, 320, - /* 760 */ 279, 243, 323, 339, 279, 199, 327, 328, 329, 81, - /* 770 */ 12, 13, 14, 15, 16, 351, 271, 0, 250, 355, - /* 780 */ 341, 250, 58, 271, 279, 346, 347, 93, 250, 243, - /* 790 */ 262, 293, 280, 262, 289, 293, 85, 243, 293, 88, - /* 800 */ 262, 296, 243, 243, 243, 117, 82, 279, 243, 115, - /* 810 */ 279, 293, 93, 41, 90, 310, 58, 279, 313, 314, - /* 820 */ 315, 316, 317, 318, 58, 320, 209, 210, 250, 272, - /* 830 */ 21, 112, 271, 114, 115, 116, 148, 149, 150, 293, - /* 840 */ 279, 153, 260, 34, 271, 145, 158, 293, 90, 41, - /* 850 */ 289, 185, 293, 293, 293, 243, 283, 279, 293, 171, - /* 860 */ 22, 358, 174, 139, 176, 177, 178, 179, 180, 212, - /* 870 */ 93, 310, 0, 47, 313, 314, 315, 316, 317, 318, - /* 880 */ 319, 320, 321, 322, 160, 47, 243, 0, 210, 112, - /* 890 */ 82, 114, 115, 116, 22, 317, 41, 139, 85, 211, - /* 900 */ 85, 88, 64, 88, 85, 293, 243, 88, 41, 22, - /* 910 */ 332, 333, 334, 0, 336, 1, 2, 44, 160, 196, - /* 920 */ 197, 349, 47, 199, 200, 201, 202, 203, 204, 205, - /* 930 */ 206, 207, 208, 41, 271, 22, 293, 82, 228, 113, - /* 940 */ 81, 103, 279, 41, 41, 306, 41, 181, 41, 82, - /* 950 */ 91, 113, 289, 343, 81, 271, 293, 199, 200, 201, - /* 960 */ 202, 203, 204, 205, 206, 207, 208, 41, 243, 249, - /* 970 */ 41, 282, 41, 310, 82, 41, 313, 314, 315, 316, - /* 980 */ 317, 318, 47, 320, 82, 82, 323, 82, 113, 82, - /* 990 */ 327, 328, 329, 41, 156, 251, 271, 312, 337, 64, - /* 1000 */ 243, 352, 230, 41, 279, 352, 41, 352, 82, 340, - /* 1010 */ 347, 82, 213, 82, 289, 41, 82, 41, 293, 181, - /* 1020 */ 182, 308, 20, 250, 45, 307, 257, 47, 271, 300, - /* 1030 */ 154, 250, 250, 287, 82, 310, 279, 40, 313, 314, - /* 1040 */ 315, 316, 317, 318, 82, 320, 289, 82, 323, 139, - /* 1050 */ 293, 250, 327, 328, 329, 285, 82, 20, 82, 285, - /* 1060 */ 243, 245, 20, 338, 304, 245, 255, 310, 289, 255, - /* 1070 */ 313, 314, 315, 316, 317, 318, 20, 320, 20, 299, - /* 1080 */ 323, 297, 255, 297, 327, 328, 329, 255, 271, 279, - /* 1090 */ 20, 255, 243, 290, 255, 338, 279, 250, 255, 245, - /* 1100 */ 271, 271, 271, 250, 271, 271, 289, 245, 271, 64, - /* 1110 */ 293, 271, 271, 271, 293, 271, 271, 293, 253, 163, - /* 1120 */ 271, 253, 304, 253, 289, 303, 20, 310, 279, 250, - /* 1130 */ 313, 314, 315, 316, 317, 318, 279, 320, 289, 253, - /* 1140 */ 323, 219, 293, 243, 327, 328, 329, 298, 297, 293, - /* 1150 */ 290, 218, 348, 348, 293, 338, 147, 294, 312, 310, - /* 1160 */ 294, 225, 313, 314, 315, 316, 317, 318, 345, 320, - /* 1170 */ 293, 271, 344, 243, 214, 279, 210, 308, 20, 279, - /* 1180 */ 40, 232, 311, 229, 227, 81, 291, 294, 339, 289, - /* 1190 */ 342, 293, 293, 293, 330, 294, 142, 293, 298, 279, - /* 1200 */ 351, 271, 290, 353, 355, 267, 243, 279, 326, 279, - /* 1210 */ 310, 359, 354, 313, 314, 315, 316, 317, 318, 289, - /* 1220 */ 320, 253, 253, 293, 81, 275, 253, 250, 354, 245, - /* 1230 */ 305, 353, 261, 353, 271, 354, 243, 265, 301, 339, - /* 1240 */ 310, 254, 279, 313, 314, 315, 316, 317, 318, 241, - /* 1250 */ 320, 351, 289, 323, 265, 355, 293, 327, 328, 265, - /* 1260 */ 0, 0, 40, 0, 271, 72, 0, 47, 175, 175, - /* 1270 */ 47, 47, 279, 310, 47, 0, 313, 314, 315, 316, - /* 1280 */ 317, 318, 289, 320, 47, 47, 293, 243, 175, 0, - /* 1290 */ 175, 0, 47, 0, 22, 0, 243, 47, 0, 81, - /* 1300 */ 113, 160, 156, 310, 159, 0, 313, 314, 315, 316, - /* 1310 */ 317, 318, 0, 320, 152, 271, 323, 151, 0, 356, - /* 1320 */ 357, 328, 0, 279, 271, 44, 0, 0, 0, 0, - /* 1330 */ 0, 0, 279, 289, 0, 0, 0, 293, 0, 0, - /* 1340 */ 0, 0, 289, 0, 0, 0, 293, 0, 0, 40, - /* 1350 */ 243, 0, 0, 0, 310, 0, 0, 313, 314, 315, - /* 1360 */ 316, 317, 318, 310, 320, 0, 313, 314, 315, 316, - /* 1370 */ 317, 318, 22, 320, 0, 0, 0, 0, 271, 0, - /* 1380 */ 0, 14, 40, 37, 14, 0, 279, 0, 0, 0, - /* 1390 */ 243, 147, 0, 44, 350, 41, 289, 38, 37, 37, - /* 1400 */ 293, 44, 0, 296, 90, 37, 243, 0, 0, 0, - /* 1410 */ 357, 0, 0, 47, 45, 37, 45, 310, 271, 59, - /* 1420 */ 313, 314, 315, 316, 317, 318, 279, 320, 47, 47, - /* 1430 */ 37, 45, 37, 0, 271, 37, 289, 47, 0, 45, - /* 1440 */ 293, 0, 279, 0, 0, 22, 88, 47, 0, 0, - /* 1450 */ 47, 41, 289, 47, 243, 41, 293, 310, 22, 296, - /* 1460 */ 313, 314, 315, 316, 317, 318, 0, 320, 47, 322, - /* 1470 */ 48, 47, 47, 310, 47, 47, 313, 314, 315, 316, - /* 1480 */ 317, 318, 271, 320, 22, 0, 243, 22, 0, 47, - /* 1490 */ 279, 33, 0, 22, 22, 20, 0, 47, 0, 22, - /* 1500 */ 289, 0, 0, 45, 293, 0, 0, 296, 145, 51, - /* 1510 */ 52, 53, 54, 55, 271, 81, 243, 37, 145, 41, - /* 1520 */ 41, 310, 279, 140, 313, 314, 315, 316, 317, 318, - /* 1530 */ 215, 320, 289, 161, 82, 41, 293, 82, 80, 142, - /* 1540 */ 81, 83, 81, 44, 271, 81, 145, 44, 41, 82, - /* 1550 */ 81, 41, 279, 310, 41, 82, 313, 314, 315, 316, - /* 1560 */ 317, 318, 289, 320, 44, 82, 293, 243, 44, 82, - /* 1570 */ 41, 82, 47, 47, 47, 47, 2, 47, 47, 41, - /* 1580 */ 181, 44, 82, 310, 44, 81, 313, 314, 315, 316, - /* 1590 */ 317, 318, 81, 320, 82, 271, 81, 22, 140, 215, - /* 1600 */ 142, 183, 144, 279, 146, 209, 215, 243, 0, 37, - /* 1610 */ 91, 143, 44, 289, 82, 81, 81, 293, 82, 22, - /* 1620 */ 81, 140, 164, 81, 81, 44, 81, 47, 82, 81, - /* 1630 */ 81, 113, 82, 92, 310, 271, 47, 313, 314, 315, - /* 1640 */ 316, 317, 318, 279, 320, 81, 47, 243, 82, 82, - /* 1650 */ 81, 47, 81, 289, 47, 82, 47, 293, 81, 105, - /* 1660 */ 82, 105, 243, 81, 105, 105, 81, 22, 81, 93, - /* 1670 */ 47, 81, 22, 58, 310, 271, 59, 313, 314, 315, - /* 1680 */ 316, 317, 318, 279, 320, 47, 64, 41, 47, 79, - /* 1690 */ 271, 22, 64, 289, 47, 47, 47, 293, 279, 47, - /* 1700 */ 47, 47, 47, 47, 47, 47, 47, 47, 289, 47, - /* 1710 */ 0, 47, 293, 243, 310, 45, 37, 313, 314, 315, - /* 1720 */ 316, 317, 318, 0, 320, 243, 45, 47, 37, 310, - /* 1730 */ 0, 47, 313, 314, 315, 316, 317, 318, 45, 320, - /* 1740 */ 37, 271, 47, 45, 37, 0, 47, 46, 0, 279, - /* 1750 */ 0, 0, 21, 271, 22, 22, 22, 21, 20, 289, - /* 1760 */ 360, 279, 360, 293, 360, 243, 360, 360, 360, 360, - /* 1770 */ 360, 289, 360, 360, 360, 293, 243, 360, 360, 360, - /* 1780 */ 310, 360, 360, 313, 314, 315, 316, 317, 318, 360, - /* 1790 */ 320, 360, 310, 271, 360, 313, 314, 315, 316, 317, - /* 1800 */ 318, 279, 320, 360, 271, 360, 360, 360, 360, 360, - /* 1810 */ 360, 289, 279, 360, 360, 293, 360, 360, 360, 360, - /* 1820 */ 360, 360, 289, 360, 360, 360, 293, 243, 360, 360, - /* 1830 */ 360, 360, 310, 360, 360, 313, 314, 315, 316, 317, - /* 1840 */ 318, 360, 320, 310, 360, 360, 313, 314, 315, 316, - /* 1850 */ 317, 318, 360, 320, 360, 271, 360, 243, 360, 360, - /* 1860 */ 360, 360, 360, 279, 360, 360, 360, 360, 360, 360, - /* 1870 */ 360, 360, 360, 289, 360, 360, 360, 293, 360, 360, - /* 1880 */ 360, 360, 360, 360, 360, 271, 360, 243, 360, 360, - /* 1890 */ 360, 360, 360, 279, 310, 360, 360, 313, 314, 315, - /* 1900 */ 316, 317, 318, 289, 320, 360, 360, 293, 360, 360, - /* 1910 */ 360, 360, 360, 360, 360, 271, 360, 360, 360, 243, - /* 1920 */ 360, 360, 360, 279, 310, 360, 360, 313, 314, 315, - /* 1930 */ 316, 317, 318, 289, 320, 360, 360, 293, 360, 360, - /* 1940 */ 360, 360, 360, 12, 13, 360, 360, 271, 360, 360, - /* 1950 */ 360, 360, 243, 22, 310, 279, 360, 313, 314, 315, - /* 1960 */ 316, 317, 318, 360, 320, 289, 360, 360, 360, 293, - /* 1970 */ 360, 360, 360, 360, 360, 360, 360, 360, 47, 360, - /* 1980 */ 271, 360, 360, 360, 360, 243, 310, 360, 279, 313, - /* 1990 */ 314, 315, 316, 317, 318, 64, 320, 360, 289, 360, - /* 2000 */ 360, 360, 293, 360, 360, 360, 360, 360, 360, 360, - /* 2010 */ 360, 360, 360, 271, 360, 360, 250, 360, 360, 310, - /* 2020 */ 360, 279, 313, 314, 315, 316, 317, 318, 360, 320, - /* 2030 */ 360, 289, 360, 360, 103, 293, 360, 360, 360, 360, - /* 2040 */ 360, 360, 360, 360, 113, 279, 250, 360, 360, 360, - /* 2050 */ 360, 360, 310, 360, 360, 313, 314, 315, 316, 317, - /* 2060 */ 318, 360, 320, 360, 298, 360, 360, 360, 360, 360, - /* 2070 */ 360, 360, 360, 360, 360, 279, 360, 360, 360, 360, - /* 2080 */ 360, 360, 360, 317, 360, 360, 360, 156, 360, 360, - /* 2090 */ 360, 360, 360, 360, 298, 360, 360, 360, 332, 333, - /* 2100 */ 334, 360, 336, 360, 360, 339, 360, 360, 360, 360, - /* 2110 */ 360, 360, 181, 317, 360, 360, 360, 351, 360, 360, - /* 2120 */ 360, 355, 360, 192, 193, 194, 360, 360, 332, 333, - /* 2130 */ 334, 360, 336, 360, 360, 339, 360, 360, 360, 360, - /* 2140 */ 360, 360, 360, 360, 360, 360, 360, 351, 360, 360, - /* 2150 */ 360, 355, 360, 360, 360, 360, 360, 360, 360, 360, - /* 2160 */ 360, 360, 360, 360, 360, 360, 360, 360, 360, 360, - /* 2170 */ 360, 360, 360, 360, 360, 360, 360, 360, 360, 360, - /* 2180 */ 360, 360, 360, 360, + /* 0 */ 274, 341, 273, 277, 248, 274, 250, 251, 277, 242, + /* 10 */ 257, 285, 12, 13, 261, 355, 285, 245, 14, 359, + /* 20 */ 20, 252, 22, 252, 20, 12, 13, 14, 15, 16, + /* 30 */ 12, 13, 263, 248, 263, 250, 251, 311, 312, 270, + /* 40 */ 22, 270, 311, 312, 272, 326, 327, 47, 322, 280, + /* 50 */ 313, 280, 280, 322, 41, 252, 20, 285, 58, 287, + /* 60 */ 12, 13, 14, 264, 64, 47, 263, 300, 20, 245, + /* 70 */ 22, 272, 300, 300, 337, 12, 13, 14, 15, 16, + /* 80 */ 281, 81, 64, 280, 312, 256, 20, 315, 316, 317, + /* 90 */ 318, 319, 320, 81, 322, 47, 272, 325, 269, 254, + /* 100 */ 252, 329, 330, 103, 280, 252, 58, 278, 341, 285, + /* 110 */ 245, 287, 64, 341, 341, 287, 263, 117, 118, 4, + /* 120 */ 275, 103, 355, 4, 20, 297, 359, 355, 355, 81, + /* 130 */ 57, 359, 359, 280, 55, 254, 312, 289, 19, 315, + /* 140 */ 316, 317, 318, 319, 320, 82, 322, 81, 267, 325, + /* 150 */ 285, 103, 33, 329, 330, 331, 275, 42, 43, 159, + /* 160 */ 81, 161, 83, 264, 45, 117, 118, 343, 3, 50, + /* 170 */ 158, 272, 160, 20, 55, 351, 352, 159, 283, 161, + /* 180 */ 281, 286, 287, 183, 184, 81, 186, 187, 188, 189, + /* 190 */ 190, 191, 192, 193, 194, 195, 196, 197, 198, 80, + /* 200 */ 20, 183, 83, 286, 287, 2, 4, 159, 341, 161, + /* 210 */ 20, 211, 194, 195, 196, 12, 13, 14, 15, 16, + /* 220 */ 12, 13, 355, 211, 245, 245, 359, 75, 20, 256, + /* 230 */ 22, 183, 184, 35, 186, 187, 188, 189, 190, 191, + /* 240 */ 192, 193, 194, 195, 196, 197, 198, 252, 272, 60, + /* 250 */ 61, 278, 272, 300, 65, 47, 280, 68, 69, 0, + /* 260 */ 280, 72, 73, 74, 285, 285, 58, 287, 12, 13, + /* 270 */ 117, 118, 64, 121, 122, 280, 20, 211, 22, 142, + /* 280 */ 300, 0, 84, 81, 86, 87, 154, 89, 81, 81, + /* 290 */ 211, 93, 312, 317, 341, 315, 316, 317, 318, 319, + /* 300 */ 320, 164, 322, 47, 45, 325, 174, 175, 355, 329, + /* 310 */ 330, 103, 359, 115, 319, 12, 13, 14, 15, 16, + /* 320 */ 64, 341, 244, 274, 246, 117, 118, 0, 333, 334, + /* 330 */ 335, 336, 4, 338, 285, 355, 199, 81, 57, 359, + /* 340 */ 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, + /* 350 */ 104, 0, 106, 107, 108, 109, 110, 111, 81, 103, + /* 360 */ 311, 312, 12, 13, 14, 15, 16, 159, 91, 161, + /* 370 */ 183, 322, 21, 117, 118, 24, 25, 26, 27, 28, + /* 380 */ 29, 30, 31, 32, 57, 82, 12, 13, 14, 15, + /* 390 */ 16, 183, 184, 228, 186, 187, 188, 189, 190, 191, + /* 400 */ 192, 193, 194, 195, 196, 197, 198, 220, 221, 222, + /* 410 */ 223, 224, 20, 211, 22, 159, 21, 161, 211, 24, + /* 420 */ 25, 26, 27, 28, 29, 30, 31, 32, 12, 13, + /* 430 */ 14, 249, 249, 245, 252, 252, 20, 55, 22, 183, + /* 440 */ 184, 49, 186, 187, 188, 189, 190, 191, 192, 193, + /* 450 */ 194, 195, 196, 197, 198, 12, 13, 14, 15, 16, + /* 460 */ 272, 300, 80, 47, 341, 83, 47, 211, 280, 20, + /* 470 */ 113, 22, 272, 285, 245, 287, 12, 13, 355, 279, + /* 480 */ 64, 0, 359, 64, 20, 313, 22, 260, 288, 262, + /* 490 */ 12, 13, 14, 15, 16, 245, 47, 81, 49, 146, + /* 500 */ 312, 151, 341, 315, 316, 317, 318, 319, 320, 337, + /* 510 */ 322, 47, 112, 325, 285, 20, 355, 329, 330, 103, + /* 520 */ 359, 287, 148, 272, 271, 82, 169, 170, 64, 295, + /* 530 */ 279, 297, 57, 117, 118, 285, 58, 284, 245, 288, + /* 540 */ 212, 60, 61, 62, 63, 81, 65, 66, 67, 68, + /* 550 */ 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, + /* 560 */ 82, 0, 12, 13, 14, 15, 16, 103, 90, 216, + /* 570 */ 217, 171, 172, 20, 14, 159, 245, 161, 285, 245, + /* 580 */ 20, 117, 118, 233, 2, 24, 25, 26, 27, 28, + /* 590 */ 29, 30, 31, 32, 12, 13, 14, 15, 16, 183, + /* 600 */ 184, 245, 186, 187, 188, 189, 190, 191, 192, 193, + /* 610 */ 194, 195, 196, 197, 198, 252, 285, 272, 272, 285, + /* 620 */ 142, 20, 264, 159, 279, 161, 263, 281, 14, 245, + /* 630 */ 272, 258, 259, 288, 20, 64, 12, 13, 0, 281, + /* 640 */ 20, 285, 164, 280, 20, 0, 22, 183, 184, 58, + /* 650 */ 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, + /* 660 */ 196, 197, 198, 1, 2, 14, 15, 16, 313, 285, + /* 670 */ 92, 47, 12, 13, 14, 15, 16, 199, 200, 201, + /* 680 */ 202, 203, 204, 205, 206, 207, 208, 282, 64, 252, + /* 690 */ 285, 274, 337, 258, 259, 245, 37, 252, 60, 61, + /* 700 */ 93, 148, 285, 65, 252, 81, 68, 69, 263, 93, + /* 710 */ 72, 73, 74, 245, 272, 263, 20, 280, 58, 112, + /* 720 */ 113, 114, 115, 116, 18, 280, 20, 103, 311, 312, + /* 730 */ 288, 115, 280, 27, 252, 285, 30, 300, 93, 322, + /* 740 */ 282, 117, 118, 285, 82, 263, 18, 187, 245, 245, + /* 750 */ 90, 23, 265, 285, 48, 268, 319, 112, 113, 114, + /* 760 */ 115, 116, 280, 35, 36, 245, 252, 39, 148, 252, + /* 770 */ 252, 334, 335, 336, 183, 338, 252, 263, 341, 280, + /* 780 */ 263, 263, 41, 159, 56, 161, 80, 263, 285, 285, + /* 790 */ 291, 64, 355, 41, 280, 252, 359, 280, 280, 42, + /* 800 */ 43, 187, 142, 273, 280, 285, 263, 183, 184, 81, + /* 810 */ 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, + /* 820 */ 196, 197, 198, 280, 164, 273, 120, 209, 210, 123, + /* 830 */ 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, + /* 840 */ 134, 135, 136, 137, 138, 139, 140, 119, 245, 273, + /* 850 */ 85, 245, 22, 88, 273, 245, 245, 245, 245, 199, + /* 860 */ 200, 201, 202, 203, 204, 205, 206, 207, 208, 210, + /* 870 */ 41, 85, 85, 85, 88, 88, 88, 47, 272, 151, + /* 880 */ 152, 153, 273, 187, 156, 47, 280, 0, 285, 252, + /* 890 */ 162, 285, 245, 287, 64, 285, 285, 285, 285, 21, + /* 900 */ 263, 173, 64, 41, 176, 41, 178, 179, 180, 181, + /* 910 */ 182, 82, 34, 41, 1, 2, 47, 280, 312, 272, + /* 920 */ 47, 315, 316, 317, 318, 319, 320, 280, 322, 252, + /* 930 */ 0, 325, 285, 103, 287, 329, 330, 331, 41, 211, + /* 940 */ 263, 117, 118, 273, 82, 41, 82, 0, 304, 41, + /* 950 */ 246, 261, 22, 0, 82, 44, 362, 280, 352, 312, + /* 960 */ 252, 0, 315, 316, 317, 318, 319, 320, 41, 322, + /* 970 */ 245, 263, 325, 232, 41, 22, 329, 330, 331, 82, + /* 980 */ 93, 41, 230, 22, 41, 353, 82, 340, 280, 159, + /* 990 */ 82, 161, 81, 46, 308, 253, 245, 272, 348, 112, + /* 1000 */ 113, 114, 115, 116, 272, 280, 253, 251, 41, 82, + /* 1010 */ 285, 41, 287, 183, 184, 82, 314, 41, 41, 41, + /* 1020 */ 41, 284, 82, 272, 339, 82, 342, 356, 356, 356, + /* 1030 */ 161, 280, 213, 310, 161, 20, 285, 312, 287, 45, + /* 1040 */ 315, 316, 317, 318, 319, 320, 252, 322, 245, 82, + /* 1050 */ 325, 47, 82, 309, 329, 330, 331, 245, 82, 82, + /* 1060 */ 82, 82, 258, 312, 157, 340, 315, 316, 317, 318, + /* 1070 */ 319, 320, 302, 322, 252, 272, 325, 252, 40, 142, + /* 1080 */ 329, 330, 331, 280, 272, 292, 290, 290, 285, 252, + /* 1090 */ 287, 340, 280, 20, 247, 247, 20, 285, 306, 287, + /* 1100 */ 256, 287, 256, 300, 20, 299, 245, 20, 301, 256, + /* 1110 */ 299, 256, 300, 280, 20, 312, 293, 252, 315, 316, + /* 1120 */ 317, 318, 319, 320, 312, 322, 256, 315, 316, 317, + /* 1130 */ 318, 319, 320, 272, 322, 256, 256, 247, 272, 272, + /* 1140 */ 272, 280, 247, 272, 341, 252, 285, 245, 287, 272, + /* 1150 */ 254, 306, 167, 341, 272, 272, 272, 272, 355, 285, + /* 1160 */ 272, 272, 359, 305, 254, 245, 285, 355, 252, 254, + /* 1170 */ 20, 359, 280, 312, 272, 299, 315, 316, 317, 318, + /* 1180 */ 319, 320, 280, 322, 293, 287, 325, 285, 254, 287, + /* 1190 */ 329, 330, 272, 219, 285, 296, 347, 285, 218, 347, + /* 1200 */ 280, 296, 285, 285, 350, 285, 225, 287, 314, 150, + /* 1210 */ 227, 214, 349, 226, 312, 210, 245, 315, 316, 317, + /* 1220 */ 318, 319, 320, 321, 322, 323, 324, 20, 280, 310, + /* 1230 */ 40, 231, 312, 347, 19, 315, 316, 317, 318, 319, + /* 1240 */ 320, 313, 322, 272, 234, 344, 346, 229, 33, 81, + /* 1250 */ 296, 280, 296, 332, 285, 145, 285, 328, 287, 345, + /* 1260 */ 45, 358, 294, 357, 280, 293, 51, 52, 53, 54, + /* 1270 */ 55, 285, 280, 285, 254, 268, 254, 245, 81, 276, + /* 1280 */ 360, 361, 358, 312, 358, 363, 315, 316, 317, 318, + /* 1290 */ 319, 320, 357, 322, 245, 80, 325, 285, 83, 357, + /* 1300 */ 252, 330, 262, 247, 272, 254, 307, 252, 266, 303, + /* 1310 */ 266, 266, 280, 255, 243, 0, 0, 285, 40, 287, + /* 1320 */ 0, 272, 72, 0, 47, 177, 47, 47, 47, 280, + /* 1330 */ 298, 116, 177, 0, 285, 280, 287, 47, 47, 177, + /* 1340 */ 0, 177, 0, 47, 312, 245, 22, 315, 316, 317, + /* 1350 */ 318, 319, 320, 0, 322, 300, 0, 47, 0, 144, + /* 1360 */ 164, 312, 147, 163, 315, 316, 317, 318, 319, 320, + /* 1370 */ 81, 322, 272, 252, 319, 161, 0, 159, 0, 155, + /* 1380 */ 280, 166, 154, 168, 0, 285, 0, 287, 44, 334, + /* 1390 */ 335, 336, 0, 338, 141, 0, 341, 0, 0, 245, + /* 1400 */ 0, 280, 0, 354, 0, 0, 0, 0, 245, 0, + /* 1410 */ 355, 0, 312, 0, 359, 315, 316, 317, 318, 319, + /* 1420 */ 320, 0, 322, 0, 0, 0, 272, 0, 0, 40, + /* 1430 */ 0, 0, 0, 0, 280, 272, 22, 0, 0, 285, + /* 1440 */ 319, 287, 0, 280, 0, 0, 0, 0, 285, 0, + /* 1450 */ 287, 41, 298, 37, 14, 334, 335, 336, 40, 338, + /* 1460 */ 0, 361, 245, 44, 14, 0, 312, 38, 0, 315, + /* 1470 */ 316, 317, 318, 319, 320, 312, 322, 252, 315, 316, + /* 1480 */ 317, 318, 319, 320, 0, 322, 150, 324, 37, 272, + /* 1490 */ 0, 37, 44, 0, 37, 0, 0, 280, 59, 0, + /* 1500 */ 33, 37, 285, 47, 287, 280, 45, 0, 37, 0, + /* 1510 */ 47, 45, 45, 245, 37, 298, 47, 45, 51, 52, + /* 1520 */ 53, 54, 55, 0, 45, 47, 37, 245, 0, 312, + /* 1530 */ 0, 0, 315, 316, 317, 318, 319, 320, 47, 322, + /* 1540 */ 272, 0, 90, 88, 319, 0, 22, 80, 280, 47, + /* 1550 */ 83, 47, 47, 285, 272, 287, 41, 47, 41, 334, + /* 1560 */ 335, 336, 280, 338, 0, 47, 298, 285, 47, 287, + /* 1570 */ 22, 0, 47, 22, 48, 0, 22, 0, 22, 47, + /* 1580 */ 312, 0, 245, 315, 316, 317, 318, 319, 320, 22, + /* 1590 */ 322, 245, 20, 0, 312, 0, 47, 315, 316, 317, + /* 1600 */ 318, 319, 320, 0, 322, 245, 148, 22, 165, 272, + /* 1610 */ 143, 0, 145, 0, 147, 145, 149, 280, 272, 0, + /* 1620 */ 81, 148, 285, 148, 287, 37, 280, 41, 143, 82, + /* 1630 */ 41, 285, 272, 287, 81, 168, 44, 215, 81, 41, + /* 1640 */ 280, 82, 82, 81, 81, 285, 81, 287, 82, 312, + /* 1650 */ 41, 44, 315, 316, 317, 318, 319, 320, 312, 322, + /* 1660 */ 81, 315, 316, 317, 318, 319, 320, 41, 322, 82, + /* 1670 */ 41, 44, 312, 82, 215, 315, 316, 317, 318, 319, + /* 1680 */ 320, 245, 322, 209, 44, 41, 82, 82, 47, 47, + /* 1690 */ 47, 215, 47, 47, 47, 2, 183, 41, 245, 44, + /* 1700 */ 82, 81, 22, 82, 81, 81, 44, 245, 272, 81, + /* 1710 */ 0, 81, 185, 82, 82, 81, 280, 81, 37, 146, + /* 1720 */ 81, 285, 44, 287, 91, 272, 143, 44, 22, 81, + /* 1730 */ 92, 82, 81, 280, 272, 81, 47, 47, 285, 82, + /* 1740 */ 287, 81, 280, 82, 47, 81, 47, 285, 312, 287, + /* 1750 */ 82, 315, 316, 317, 318, 319, 320, 81, 322, 245, + /* 1760 */ 47, 82, 47, 81, 105, 312, 82, 81, 315, 316, + /* 1770 */ 317, 318, 319, 320, 312, 322, 105, 315, 316, 317, + /* 1780 */ 318, 319, 320, 245, 322, 22, 272, 93, 81, 81, + /* 1790 */ 105, 47, 105, 81, 280, 41, 22, 59, 58, 285, + /* 1800 */ 245, 287, 47, 64, 79, 41, 47, 47, 47, 47, + /* 1810 */ 272, 22, 47, 47, 47, 64, 47, 47, 280, 47, + /* 1820 */ 47, 0, 47, 285, 47, 287, 312, 272, 47, 315, + /* 1830 */ 316, 317, 318, 319, 320, 280, 322, 47, 45, 37, + /* 1840 */ 285, 0, 287, 47, 45, 37, 0, 37, 47, 45, + /* 1850 */ 312, 245, 0, 315, 316, 317, 318, 319, 320, 47, + /* 1860 */ 322, 45, 245, 37, 0, 47, 0, 312, 20, 22, + /* 1870 */ 315, 316, 317, 318, 319, 320, 21, 322, 272, 22, + /* 1880 */ 22, 21, 364, 364, 364, 364, 280, 364, 364, 272, + /* 1890 */ 364, 285, 364, 287, 364, 364, 364, 280, 364, 364, + /* 1900 */ 364, 364, 285, 245, 287, 364, 364, 364, 364, 364, + /* 1910 */ 364, 364, 364, 364, 245, 364, 364, 364, 312, 364, + /* 1920 */ 364, 315, 316, 317, 318, 319, 320, 364, 322, 312, + /* 1930 */ 272, 364, 315, 316, 317, 318, 319, 320, 280, 322, + /* 1940 */ 364, 272, 364, 285, 364, 287, 364, 364, 364, 280, + /* 1950 */ 364, 364, 364, 364, 285, 364, 287, 364, 364, 364, + /* 1960 */ 364, 364, 364, 364, 364, 364, 364, 245, 364, 364, + /* 1970 */ 312, 364, 364, 315, 316, 317, 318, 319, 320, 245, + /* 1980 */ 322, 312, 364, 364, 315, 316, 317, 318, 319, 320, + /* 1990 */ 364, 322, 364, 245, 272, 364, 364, 364, 364, 364, + /* 2000 */ 364, 364, 280, 364, 364, 364, 272, 285, 364, 287, + /* 2010 */ 364, 364, 364, 364, 280, 364, 364, 364, 364, 285, + /* 2020 */ 272, 287, 364, 364, 364, 364, 364, 364, 280, 364, + /* 2030 */ 364, 364, 364, 285, 312, 287, 364, 315, 316, 317, + /* 2040 */ 318, 319, 320, 364, 322, 245, 312, 364, 364, 315, + /* 2050 */ 316, 317, 318, 319, 320, 364, 322, 364, 364, 364, + /* 2060 */ 312, 364, 364, 315, 316, 317, 318, 319, 320, 364, + /* 2070 */ 322, 364, 272, 364, 364, 364, 364, 364, 364, 364, + /* 2080 */ 280, 364, 364, 364, 364, 285, 364, 287, 364, 364, + /* 2090 */ 364, 364, 364, 364, 364, 364, 364, 364, 364, 364, + /* 2100 */ 364, 364, 364, 364, 364, 364, 364, 364, 364, 364, + /* 2110 */ 364, 364, 312, 364, 364, 315, 316, 317, 318, 319, + /* 2120 */ 320, 364, 322, 364, }; -#define YY_SHIFT_COUNT (618) +#define YY_SHIFT_COUNT (635) #define YY_SHIFT_MIN (0) -#define YY_SHIFT_MAX (1931) +#define YY_SHIFT_MAX (1866) static const unsigned short int yy_shift_ofst[] = { - /* 0 */ 688, 0, 0, 48, 96, 96, 96, 96, 253, 253, - /* 10 */ 96, 96, 301, 349, 506, 349, 349, 349, 349, 349, - /* 20 */ 349, 349, 349, 349, 349, 349, 349, 349, 349, 349, - /* 30 */ 349, 349, 349, 349, 349, 349, 349, 349, 144, 144, - /* 40 */ 102, 102, 102, 1931, 1931, 1931, 1931, 372, 124, 214, - /* 50 */ 58, 58, 260, 260, 172, 214, 214, 58, 58, 58, - /* 60 */ 58, 58, 58, 58, 37, 58, 58, 186, 228, 259, - /* 70 */ 186, 58, 58, 186, 58, 186, 186, 259, 186, 58, - /* 80 */ 255, 542, 724, 758, 758, 189, 236, 838, 838, 838, - /* 90 */ 838, 838, 838, 838, 838, 838, 838, 838, 838, 838, - /* 100 */ 838, 838, 838, 838, 838, 838, 291, 126, 354, 354, - /* 110 */ 397, 516, 483, 483, 483, 525, 516, 313, 259, 186, - /* 120 */ 186, 259, 377, 343, 243, 243, 243, 243, 243, 243, - /* 130 */ 243, 340, 141, 405, 76, 195, 57, 215, 500, 545, - /* 140 */ 579, 179, 694, 700, 617, 678, 617, 455, 455, 455, - /* 150 */ 657, 666, 799, 1002, 979, 980, 876, 1002, 1002, 997, - /* 160 */ 910, 910, 1002, 1037, 1037, 1042, 37, 259, 37, 1056, - /* 170 */ 1058, 37, 1056, 37, 313, 1070, 37, 37, 1002, 37, - /* 180 */ 1037, 186, 186, 186, 186, 186, 186, 186, 186, 186, - /* 190 */ 186, 186, 1002, 1037, 1045, 1045, 1042, 255, 956, 259, - /* 200 */ 255, 1002, 1056, 255, 313, 1070, 255, 1106, 922, 933, - /* 210 */ 1045, 922, 933, 1045, 1045, 186, 936, 1009, 960, 799, - /* 220 */ 966, 313, 1158, 1140, 954, 957, 949, 954, 957, 954, - /* 230 */ 957, 1104, 933, 1045, 1045, 933, 1045, 1054, 313, 1070, - /* 240 */ 255, 377, 255, 313, 1143, 343, 1002, 255, 1037, 2152, - /* 250 */ 2152, 2152, 2152, 2152, 2152, 2152, 577, 1458, 231, 353, - /* 260 */ 3, 19, 316, 256, 496, 719, 777, 112, 112, 112, - /* 270 */ 112, 112, 112, 112, 112, 135, 438, 344, 396, 355, - /* 280 */ 625, 566, 642, 642, 642, 642, 555, 808, 711, 813, - /* 290 */ 815, 819, 872, 887, 913, 809, 855, 867, 892, 914, - /* 300 */ 723, 710, 772, 902, 766, 903, 873, 905, 907, 926, - /* 310 */ 929, 931, 826, 875, 934, 952, 962, 965, 974, 976, - /* 320 */ 859, 935, 1260, 1261, 1222, 1263, 1193, 1266, 1220, 1093, - /* 330 */ 1223, 1224, 1227, 1094, 1275, 1237, 1238, 1113, 1289, 1115, - /* 340 */ 1291, 1245, 1293, 1272, 1295, 1250, 1298, 1218, 1141, 1145, - /* 350 */ 1187, 1146, 1305, 1312, 1162, 1166, 1318, 1322, 1281, 1326, - /* 360 */ 1327, 1328, 1329, 1330, 1331, 1334, 1335, 1336, 1338, 1339, - /* 370 */ 1340, 1341, 1343, 1344, 1345, 1347, 1348, 1309, 1351, 1352, - /* 380 */ 1353, 1355, 1356, 1365, 1350, 1374, 1375, 1376, 1377, 1379, - /* 390 */ 1380, 1342, 1346, 1354, 1367, 1349, 1370, 1357, 1385, 1359, - /* 400 */ 1361, 1387, 1388, 1389, 1362, 1244, 1392, 1402, 1368, 1407, - /* 410 */ 1360, 1408, 1409, 1366, 1369, 1378, 1411, 1381, 1371, 1393, - /* 420 */ 1412, 1382, 1386, 1395, 1433, 1390, 1394, 1398, 1438, 1441, - /* 430 */ 1443, 1444, 1314, 1358, 1400, 1423, 1448, 1403, 1406, 1421, - /* 440 */ 1424, 1410, 1414, 1425, 1427, 1428, 1449, 1436, 1466, 1462, - /* 450 */ 1422, 1485, 1465, 1442, 1488, 1471, 1492, 1472, 1475, 1496, - /* 460 */ 1363, 1450, 1498, 1372, 1477, 1373, 1397, 1501, 1502, 1505, - /* 470 */ 1401, 1506, 1434, 1480, 1383, 1478, 1479, 1315, 1452, 1494, - /* 480 */ 1455, 1459, 1461, 1464, 1467, 1507, 1499, 1503, 1469, 1510, - /* 490 */ 1384, 1473, 1483, 1520, 1396, 1513, 1524, 1487, 1529, 1391, - /* 500 */ 1489, 1525, 1526, 1527, 1528, 1530, 1531, 1489, 1574, 1399, - /* 510 */ 1538, 1500, 1504, 1512, 1537, 1511, 1515, 1540, 1575, 1418, - /* 520 */ 1534, 1532, 1536, 1535, 1539, 1468, 1542, 1608, 1572, 1481, - /* 530 */ 1543, 1519, 1568, 1581, 1545, 1546, 1548, 1597, 1549, 1541, - /* 540 */ 1550, 1580, 1589, 1564, 1566, 1599, 1569, 1567, 1604, 1571, - /* 550 */ 1573, 1607, 1577, 1578, 1609, 1582, 1554, 1556, 1559, 1560, - /* 560 */ 1645, 1576, 1585, 1587, 1623, 1590, 1518, 1650, 1617, 1615, - /* 570 */ 1638, 1622, 1610, 1646, 1641, 1647, 1648, 1649, 1652, 1669, - /* 580 */ 1653, 1654, 1628, 1410, 1655, 1414, 1656, 1657, 1658, 1659, - /* 590 */ 1660, 1662, 1710, 1664, 1670, 1679, 1723, 1680, 1681, 1691, - /* 600 */ 1730, 1684, 1693, 1703, 1750, 1695, 1698, 1707, 1745, 1699, - /* 610 */ 1701, 1748, 1751, 1732, 1731, 1733, 1734, 1736, 1738, + /* 0 */ 728, 0, 0, 48, 208, 208, 208, 208, 256, 256, + /* 10 */ 208, 208, 416, 464, 624, 464, 464, 464, 464, 464, + /* 20 */ 464, 464, 464, 464, 464, 464, 464, 464, 464, 464, + /* 30 */ 464, 464, 464, 464, 464, 464, 464, 464, 464, 464, + /* 40 */ 66, 66, 104, 104, 104, 18, 18, 18, 18, 12, + /* 50 */ 79, 207, 36, 36, 115, 115, 202, 153, 207, 207, + /* 60 */ 36, 36, 36, 36, 36, 36, 36, 36, 73, 36, + /* 70 */ 36, 180, 190, 495, 180, 36, 36, 180, 36, 180, + /* 80 */ 180, 180, 36, 475, 706, 478, 660, 660, 395, 189, + /* 90 */ 830, 830, 830, 830, 830, 830, 830, 830, 830, 830, + /* 100 */ 830, 830, 830, 830, 830, 830, 830, 830, 830, 198, + /* 110 */ 153, 4, 4, 281, 419, 553, 553, 553, 327, 419, + /* 120 */ 601, 495, 180, 180, 571, 571, 578, 727, 246, 246, + /* 130 */ 246, 246, 246, 246, 246, 1215, 351, 638, 350, 187, + /* 140 */ 449, 357, 353, 560, 614, 392, 757, 616, 620, 618, + /* 150 */ 659, 618, 165, 165, 165, 328, 696, 819, 1015, 994, + /* 160 */ 1004, 907, 1015, 1015, 1038, 937, 937, 1015, 1073, 1073, + /* 170 */ 1076, 73, 495, 73, 1084, 1087, 73, 1084, 73, 601, + /* 180 */ 1094, 73, 73, 1015, 73, 1073, 180, 180, 180, 180, + /* 190 */ 180, 180, 180, 180, 180, 180, 180, 1015, 1073, 571, + /* 200 */ 571, 1076, 475, 985, 495, 475, 1015, 1084, 475, 601, + /* 210 */ 1094, 475, 1150, 571, 974, 980, 571, 974, 980, 571, + /* 220 */ 571, 180, 981, 1059, 974, 983, 987, 997, 819, 1005, + /* 230 */ 601, 1207, 1190, 1000, 1018, 1010, 1000, 1018, 1000, 1018, + /* 240 */ 1168, 980, 571, 571, 980, 571, 1110, 601, 1094, 475, + /* 250 */ 578, 475, 601, 1197, 571, 727, 1015, 475, 1073, 2123, + /* 260 */ 2123, 2123, 2123, 2123, 2123, 2123, 481, 1467, 561, 119, + /* 270 */ 645, 887, 63, 13, 303, 203, 582, 374, 443, 607, + /* 280 */ 550, 550, 550, 550, 550, 550, 550, 550, 132, 382, + /* 290 */ 152, 400, 662, 137, 651, 651, 651, 651, 259, 829, + /* 300 */ 765, 786, 787, 788, 930, 953, 961, 878, 862, 864, + /* 310 */ 872, 913, 824, 752, 741, 897, 591, 904, 911, 908, + /* 320 */ 927, 933, 940, 943, 869, 873, 967, 970, 976, 977, + /* 330 */ 978, 979, 277, 838, 947, 1315, 1316, 1278, 1320, 1250, + /* 340 */ 1323, 1277, 1148, 1279, 1280, 1281, 1155, 1333, 1290, 1291, + /* 350 */ 1162, 1340, 1164, 1342, 1296, 1353, 1324, 1356, 1310, 1358, + /* 360 */ 1289, 1196, 1200, 1214, 1218, 1376, 1378, 1224, 1228, 1384, + /* 370 */ 1386, 1344, 1392, 1253, 1395, 1397, 1398, 1400, 1402, 1404, + /* 380 */ 1405, 1406, 1407, 1409, 1411, 1413, 1421, 1423, 1424, 1425, + /* 390 */ 1427, 1428, 1389, 1430, 1431, 1432, 1433, 1442, 1444, 1414, + /* 400 */ 1437, 1438, 1445, 1446, 1447, 1449, 1418, 1416, 1410, 1440, + /* 410 */ 1419, 1450, 1448, 1460, 1429, 1451, 1465, 1468, 1484, 1454, + /* 420 */ 1336, 1490, 1493, 1457, 1495, 1439, 1496, 1499, 1456, 1461, + /* 430 */ 1464, 1507, 1463, 1466, 1471, 1509, 1469, 1472, 1477, 1523, + /* 440 */ 1478, 1479, 1489, 1528, 1530, 1531, 1541, 1452, 1455, 1491, + /* 450 */ 1524, 1545, 1502, 1504, 1505, 1510, 1515, 1517, 1518, 1521, + /* 460 */ 1525, 1564, 1548, 1571, 1551, 1526, 1575, 1554, 1532, 1577, + /* 470 */ 1556, 1581, 1567, 1572, 1593, 1458, 1549, 1595, 1443, 1585, + /* 480 */ 1473, 1470, 1603, 1611, 1613, 1475, 1619, 1539, 1588, 1485, + /* 490 */ 1586, 1589, 1422, 1553, 1547, 1557, 1559, 1598, 1560, 1562, + /* 500 */ 1563, 1565, 1566, 1609, 1592, 1607, 1579, 1626, 1459, 1587, + /* 510 */ 1591, 1627, 1474, 1629, 1640, 1604, 1644, 1476, 1605, 1641, + /* 520 */ 1642, 1643, 1645, 1646, 1647, 1605, 1693, 1513, 1656, 1618, + /* 530 */ 1620, 1621, 1655, 1623, 1624, 1662, 1680, 1527, 1628, 1631, + /* 540 */ 1632, 1630, 1634, 1573, 1636, 1710, 1681, 1583, 1639, 1633, + /* 550 */ 1678, 1683, 1648, 1649, 1651, 1706, 1654, 1638, 1657, 1689, + /* 560 */ 1690, 1660, 1661, 1697, 1664, 1668, 1699, 1676, 1679, 1713, + /* 570 */ 1682, 1684, 1715, 1686, 1659, 1671, 1685, 1687, 1763, 1694, + /* 580 */ 1707, 1708, 1744, 1712, 1754, 1754, 1774, 1738, 1740, 1755, + /* 590 */ 1739, 1725, 1764, 1759, 1760, 1761, 1762, 1765, 1789, 1766, + /* 600 */ 1767, 1751, 1515, 1769, 1517, 1770, 1772, 1773, 1775, 1777, + /* 610 */ 1781, 1821, 1790, 1793, 1802, 1841, 1796, 1799, 1808, 1846, + /* 620 */ 1801, 1804, 1810, 1852, 1812, 1816, 1826, 1864, 1818, 1866, + /* 630 */ 1847, 1855, 1857, 1858, 1860, 1848, }; -#define YY_REDUCE_COUNT (255) -#define YY_REDUCE_MIN (-324) -#define YY_REDUCE_MAX (1796) +#define YY_REDUCE_COUNT (265) +#define YY_REDUCE_MIN (-340) +#define YY_REDUCE_MAX (1800) static const short yy_reduce_ofst[] = { - /* 0 */ 162, -241, 278, 439, 663, 725, 757, 817, 849, 900, - /* 10 */ -178, 930, 561, 963, 993, 505, 1044, 1053, 1107, 1147, - /* 20 */ 1163, 1211, 1243, 1273, 1324, 1364, 1404, 1419, 1470, 1482, - /* 30 */ 1522, 1533, 1584, 1614, 1644, 1676, 1709, 1742, 1766, 1796, - /* 40 */ 282, -213, 578, -270, -266, -264, -176, -284, 281, 424, - /* 50 */ 306, 362, -91, -46, -255, -192, 71, -226, -199, 150, - /* 60 */ 302, 324, 463, 467, -1, 471, 485, 28, 45, -190, - /* 70 */ 56, 528, 531, 197, 481, 192, 352, 100, 196, 538, - /* 80 */ -225, -242, -324, -324, -324, -64, -48, 77, 168, 257, - /* 90 */ 311, 315, 341, 416, 441, 465, 498, 502, 518, 546, - /* 100 */ 554, 559, 560, 565, 612, 643, 97, -247, 75, 140, - /* 110 */ -125, -232, -252, -204, -37, -100, 106, 142, 394, 512, - /* 120 */ 573, 218, 284, 453, -261, 294, 325, 330, 335, 444, - /* 130 */ 557, 271, 381, 582, 503, 572, 639, 610, 684, 684, - /* 140 */ 720, 744, 689, 685, 661, 661, 661, 649, 653, 655, - /* 150 */ 669, 684, 713, 773, 718, 769, 729, 781, 782, 746, - /* 160 */ 770, 774, 801, 816, 820, 760, 811, 779, 814, 784, - /* 170 */ 780, 827, 786, 832, 810, 803, 836, 839, 847, 843, - /* 180 */ 854, 829, 830, 831, 833, 834, 837, 840, 841, 842, - /* 190 */ 844, 845, 853, 862, 821, 824, 818, 865, 822, 835, - /* 200 */ 868, 879, 851, 870, 857, 860, 886, 846, 804, 863, - /* 210 */ 856, 805, 866, 861, 877, 684, 823, 828, 848, 869, - /* 220 */ 661, 896, 871, 864, 858, 850, 852, 874, 878, 881, - /* 230 */ 880, 882, 893, 898, 899, 901, 904, 895, 920, 912, - /* 240 */ 968, 938, 969, 928, 950, 971, 977, 973, 984, 937, - /* 250 */ 925, 972, 989, 994, 987, 1008, + /* 0 */ -233, -228, -20, -176, 606, 647, 725, 751, 803, 812, + /* 10 */ 188, 861, 902, 920, 971, 1032, 1049, 1100, 1154, 1163, + /* 20 */ 1217, 1268, 1282, 1337, 1346, 1360, 1436, 1453, 1462, 1514, + /* 30 */ 1538, 1555, 1606, 1617, 1658, 1669, 1722, 1734, 1748, 1800, + /* 40 */ 437, 1055, -5, 1121, 1225, -274, -269, 49, 417, -227, + /* 50 */ -47, 161, -231, -229, -244, -215, -340, -105, -133, 123, + /* 60 */ -197, -147, 363, 445, 452, 482, 514, 517, -171, 518, + /* 70 */ 543, 200, -24, 234, -201, 637, 677, 251, 524, -101, + /* 80 */ 345, 358, 708, -119, -152, -281, -281, -281, 78, -247, + /* 90 */ -135, -21, 229, 250, 293, 331, 334, 356, 384, 450, + /* 100 */ 468, 503, 504, 520, 603, 610, 611, 612, 613, 253, + /* 110 */ -83, 182, 183, -27, 373, -263, 172, 355, -155, 435, + /* 120 */ 499, -172, 346, 442, 405, 458, 487, 227, -271, 530, + /* 130 */ 552, 576, 581, 609, 670, 644, 704, 690, 594, 632, + /* 140 */ 742, 686, 650, 732, 732, 753, 756, 737, 702, 685, + /* 150 */ 685, 685, 671, 672, 673, 684, 732, 723, 794, 744, + /* 160 */ 804, 770, 822, 825, 793, 796, 797, 837, 847, 848, + /* 170 */ 792, 844, 814, 846, 806, 807, 853, 811, 855, 833, + /* 180 */ 823, 870, 879, 865, 880, 890, 866, 867, 868, 871, + /* 190 */ 877, 882, 883, 884, 885, 888, 889, 893, 895, 874, + /* 200 */ 881, 845, 896, 858, 898, 910, 916, 876, 915, 892, + /* 210 */ 891, 934, 894, 909, 849, 899, 912, 852, 905, 917, + /* 220 */ 918, 732, 854, 863, 886, 900, 914, 901, 919, 685, + /* 230 */ 948, 928, 921, 903, 906, 922, 924, 935, 926, 942, + /* 240 */ 929, 954, 969, 986, 956, 988, 968, 984, 972, 1020, + /* 250 */ 1007, 1022, 992, 1003, 1012, 1040, 1048, 1051, 1056, 1006, + /* 260 */ 999, 1042, 1044, 1045, 1058, 1071, }; static const YYACTIONTYPE yy_default[] = { - /* 0 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 10 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 20 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 30 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 40 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 50 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 60 */ 1360, 1360, 1360, 1360, 1429, 1360, 1360, 1360, 1360, 1360, - /* 70 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 80 */ 1427, 1567, 1360, 1734, 1360, 1360, 1360, 1360, 1360, 1360, - /* 90 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 100 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 110 */ 1429, 1360, 1745, 1745, 1745, 1427, 1360, 1360, 1360, 1360, - /* 120 */ 1360, 1360, 1523, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 130 */ 1360, 1603, 1360, 1360, 1811, 1360, 1609, 1769, 1360, 1360, - /* 140 */ 1360, 1360, 1476, 1761, 1737, 1751, 1738, 1796, 1796, 1796, - /* 150 */ 1754, 1360, 1765, 1360, 1360, 1360, 1595, 1360, 1360, 1572, - /* 160 */ 1569, 1569, 1360, 1360, 1360, 1360, 1429, 1360, 1429, 1360, - /* 170 */ 1360, 1429, 1360, 1429, 1360, 1360, 1429, 1429, 1360, 1429, - /* 180 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 190 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1427, 1605, 1360, - /* 200 */ 1427, 1360, 1360, 1427, 1360, 1360, 1427, 1360, 1776, 1774, - /* 210 */ 1360, 1776, 1774, 1360, 1360, 1360, 1788, 1784, 1767, 1765, - /* 220 */ 1751, 1360, 1360, 1360, 1802, 1798, 1814, 1802, 1798, 1802, - /* 230 */ 1798, 1360, 1774, 1360, 1360, 1774, 1360, 1580, 1360, 1360, - /* 240 */ 1427, 1360, 1427, 1360, 1492, 1360, 1360, 1427, 1360, 1597, - /* 250 */ 1611, 1526, 1526, 1526, 1430, 1365, 1360, 1360, 1360, 1360, - /* 260 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1678, 1787, 1786, - /* 270 */ 1710, 1709, 1708, 1706, 1677, 1488, 1360, 1360, 1360, 1360, - /* 280 */ 1360, 1360, 1671, 1672, 1670, 1669, 1360, 1360, 1360, 1360, - /* 290 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1735, - /* 300 */ 1360, 1799, 1803, 1360, 1360, 1360, 1654, 1360, 1360, 1360, - /* 310 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 320 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 330 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 340 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 350 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 360 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 370 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 380 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 390 */ 1360, 1360, 1360, 1394, 1360, 1360, 1360, 1360, 1360, 1360, - /* 400 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 410 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 420 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 430 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 440 */ 1360, 1457, 1456, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 450 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 460 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 470 */ 1360, 1360, 1360, 1360, 1360, 1758, 1768, 1360, 1360, 1360, - /* 480 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1654, 1360, 1785, - /* 490 */ 1360, 1744, 1740, 1360, 1360, 1736, 1360, 1360, 1797, 1360, - /* 500 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1730, 1360, - /* 510 */ 1703, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1665, - /* 520 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 530 */ 1360, 1360, 1653, 1360, 1694, 1360, 1360, 1360, 1360, 1360, - /* 540 */ 1360, 1360, 1360, 1520, 1360, 1360, 1360, 1360, 1360, 1360, - /* 550 */ 1360, 1360, 1360, 1360, 1360, 1360, 1505, 1503, 1502, 1501, - /* 560 */ 1360, 1498, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 570 */ 1360, 1360, 1360, 1449, 1360, 1360, 1360, 1360, 1360, 1360, - /* 580 */ 1360, 1360, 1360, 1440, 1360, 1439, 1360, 1360, 1360, 1360, - /* 590 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 600 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, - /* 610 */ 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, 1360, + /* 0 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 10 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 20 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 30 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 40 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 50 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 60 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1461, 1392, + /* 70 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 80 */ 1392, 1392, 1392, 1459, 1608, 1392, 1772, 1392, 1392, 1392, + /* 90 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 100 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 110 */ 1392, 1392, 1392, 1461, 1392, 1783, 1783, 1783, 1459, 1392, + /* 120 */ 1392, 1392, 1392, 1392, 1392, 1392, 1555, 1392, 1392, 1392, + /* 130 */ 1392, 1392, 1392, 1392, 1392, 1641, 1392, 1392, 1853, 1392, + /* 140 */ 1392, 1647, 1807, 1392, 1392, 1392, 1392, 1508, 1799, 1775, + /* 150 */ 1789, 1776, 1838, 1838, 1838, 1792, 1392, 1803, 1392, 1392, + /* 160 */ 1392, 1633, 1392, 1392, 1613, 1610, 1610, 1392, 1392, 1392, + /* 170 */ 1392, 1461, 1392, 1461, 1392, 1392, 1461, 1392, 1461, 1392, + /* 180 */ 1392, 1461, 1461, 1392, 1461, 1392, 1392, 1392, 1392, 1392, + /* 190 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 200 */ 1392, 1392, 1459, 1643, 1392, 1459, 1392, 1392, 1459, 1392, + /* 210 */ 1392, 1459, 1392, 1392, 1814, 1812, 1392, 1814, 1812, 1392, + /* 220 */ 1392, 1392, 1826, 1822, 1814, 1830, 1828, 1805, 1803, 1789, + /* 230 */ 1392, 1392, 1392, 1844, 1840, 1856, 1844, 1840, 1844, 1840, + /* 240 */ 1392, 1812, 1392, 1392, 1812, 1392, 1618, 1392, 1392, 1459, + /* 250 */ 1392, 1459, 1392, 1524, 1392, 1392, 1392, 1459, 1392, 1635, + /* 260 */ 1649, 1558, 1558, 1558, 1462, 1397, 1392, 1392, 1392, 1392, + /* 270 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1520, + /* 280 */ 1716, 1825, 1824, 1748, 1747, 1746, 1744, 1715, 1392, 1392, + /* 290 */ 1392, 1392, 1392, 1392, 1709, 1710, 1708, 1707, 1392, 1392, + /* 300 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 310 */ 1392, 1773, 1392, 1841, 1845, 1392, 1392, 1392, 1692, 1392, + /* 320 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 330 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 340 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 350 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 360 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 370 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 380 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 390 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 400 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1426, 1392, + /* 410 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 420 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 430 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 440 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 450 */ 1392, 1392, 1392, 1392, 1392, 1392, 1489, 1488, 1392, 1392, + /* 460 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 470 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 480 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 490 */ 1796, 1806, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 500 */ 1392, 1392, 1392, 1392, 1392, 1692, 1392, 1823, 1392, 1782, + /* 510 */ 1778, 1392, 1392, 1774, 1392, 1392, 1839, 1392, 1392, 1392, + /* 520 */ 1392, 1392, 1392, 1392, 1392, 1392, 1768, 1392, 1741, 1392, + /* 530 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1703, 1392, 1392, + /* 540 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 550 */ 1691, 1392, 1732, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 560 */ 1392, 1552, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 570 */ 1392, 1392, 1392, 1392, 1537, 1535, 1534, 1533, 1392, 1530, + /* 580 */ 1392, 1392, 1392, 1392, 1561, 1560, 1392, 1392, 1392, 1392, + /* 590 */ 1392, 1392, 1481, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 600 */ 1392, 1392, 1472, 1392, 1471, 1392, 1392, 1392, 1392, 1392, + /* 610 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 620 */ 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, 1392, + /* 630 */ 1392, 1392, 1392, 1392, 1392, 1392, }; /********** End of lemon-generated parsing tables *****************************/ @@ -947,11 +943,13 @@ static const YYCODETYPE yyFallback[] = { 0, /* BLOB => nothing */ 0, /* VARBINARY => nothing */ 0, /* DECIMAL => nothing */ - 0, /* FILE_FACTOR => nothing */ - 0, /* NK_FLOAT => nothing */ + 0, /* MAX_DELAY => nothing */ + 0, /* WATERMARK => nothing */ 0, /* ROLLUP => nothing */ 0, /* TTL => nothing */ 0, /* SMA => nothing */ + 0, /* FIRST => nothing */ + 0, /* LAST => nothing */ 0, /* SHOW => nothing */ 0, /* DATABASES => nothing */ 0, /* TABLES => nothing */ @@ -974,6 +972,7 @@ static const YYCODETYPE yyFallback[] = { 0, /* SNODES => nothing */ 0, /* CLUSTER => nothing */ 0, /* TRANSACTIONS => nothing */ + 0, /* DISTRIBUTED => nothing */ 0, /* LIKE => nothing */ 0, /* INDEX => nothing */ 0, /* FULLTEXT => nothing */ @@ -993,6 +992,7 @@ static const YYCODETYPE yyFallback[] = { 0, /* VERBOSE => nothing */ 0, /* NK_BOOL => nothing */ 0, /* RATIO => nothing */ + 0, /* NK_FLOAT => nothing */ 0, /* COMPACT => nothing */ 0, /* VNODES => nothing */ 0, /* IN => nothing */ @@ -1004,8 +1004,6 @@ static const YYCODETYPE yyFallback[] = { 0, /* TRIGGER => nothing */ 0, /* AT_ONCE => nothing */ 0, /* WINDOW_CLOSE => nothing */ - 0, /* MAX_DELAY => nothing */ - 0, /* WATERMARK => nothing */ 0, /* KILL => nothing */ 0, /* CONNECTION => nothing */ 0, /* TRANSACTION => nothing */ @@ -1031,8 +1029,6 @@ static const YYCODETYPE yyFallback[] = { 0, /* TODAY => nothing */ 0, /* TIMEZONE => nothing */ 0, /* COUNT => nothing */ - 0, /* FIRST => nothing */ - 0, /* LAST => nothing */ 0, /* LAST_ROW => nothing */ 0, /* BETWEEN => nothing */ 0, /* IS => nothing */ @@ -1061,6 +1057,8 @@ static const YYCODETYPE yyFallback[] = { 0, /* LINEAR => nothing */ 0, /* NEXT => nothing */ 0, /* HAVING => nothing */ + 0, /* RANGE => nothing */ + 0, /* EVERY => nothing */ 0, /* ORDER => nothing */ 0, /* SLIMIT => nothing */ 0, /* SOFFSET => nothing */ @@ -1069,12 +1067,12 @@ static const YYCODETYPE yyFallback[] = { 0, /* ASC => nothing */ 0, /* NULLS => nothing */ 0, /* ID => nothing */ - 233, /* NK_BITNOT => ID */ - 233, /* INSERT => ID */ - 233, /* VALUES => ID */ - 233, /* IMPORT => ID */ - 233, /* NK_SEMI => ID */ - 233, /* FILE => ID */ + 235, /* NK_BITNOT => ID */ + 235, /* INSERT => ID */ + 235, /* VALUES => ID */ + 235, /* IMPORT => ID */ + 235, /* NK_SEMI => ID */ + 235, /* FILE => ID */ }; #endif /* YYFALLBACK */ @@ -1274,92 +1272,92 @@ static const char *const yyTokenName[] = { /* 109 */ "BLOB", /* 110 */ "VARBINARY", /* 111 */ "DECIMAL", - /* 112 */ "FILE_FACTOR", - /* 113 */ "NK_FLOAT", + /* 112 */ "MAX_DELAY", + /* 113 */ "WATERMARK", /* 114 */ "ROLLUP", /* 115 */ "TTL", /* 116 */ "SMA", - /* 117 */ "SHOW", - /* 118 */ "DATABASES", - /* 119 */ "TABLES", - /* 120 */ "STABLES", - /* 121 */ "MNODES", - /* 122 */ "MODULES", - /* 123 */ "QNODES", - /* 124 */ "FUNCTIONS", - /* 125 */ "INDEXES", - /* 126 */ "ACCOUNTS", - /* 127 */ "APPS", - /* 128 */ "CONNECTIONS", - /* 129 */ "LICENCE", - /* 130 */ "GRANTS", - /* 131 */ "QUERIES", - /* 132 */ "SCORES", - /* 133 */ "TOPICS", - /* 134 */ "VARIABLES", - /* 135 */ "BNODES", - /* 136 */ "SNODES", - /* 137 */ "CLUSTER", - /* 138 */ "TRANSACTIONS", - /* 139 */ "LIKE", - /* 140 */ "INDEX", - /* 141 */ "FULLTEXT", - /* 142 */ "FUNCTION", - /* 143 */ "INTERVAL", - /* 144 */ "TOPIC", - /* 145 */ "AS", - /* 146 */ "CONSUMER", - /* 147 */ "GROUP", - /* 148 */ "DESC", - /* 149 */ "DESCRIBE", - /* 150 */ "RESET", - /* 151 */ "QUERY", - /* 152 */ "CACHE", - /* 153 */ "EXPLAIN", - /* 154 */ "ANALYZE", - /* 155 */ "VERBOSE", - /* 156 */ "NK_BOOL", - /* 157 */ "RATIO", - /* 158 */ "COMPACT", - /* 159 */ "VNODES", - /* 160 */ "IN", - /* 161 */ "OUTPUTTYPE", - /* 162 */ "AGGREGATE", - /* 163 */ "BUFSIZE", - /* 164 */ "STREAM", - /* 165 */ "INTO", - /* 166 */ "TRIGGER", - /* 167 */ "AT_ONCE", - /* 168 */ "WINDOW_CLOSE", - /* 169 */ "MAX_DELAY", - /* 170 */ "WATERMARK", - /* 171 */ "KILL", - /* 172 */ "CONNECTION", - /* 173 */ "TRANSACTION", - /* 174 */ "BALANCE", - /* 175 */ "VGROUP", - /* 176 */ "MERGE", - /* 177 */ "REDISTRIBUTE", - /* 178 */ "SPLIT", - /* 179 */ "SYNCDB", - /* 180 */ "DELETE", - /* 181 */ "NULL", - /* 182 */ "NK_QUESTION", - /* 183 */ "NK_ARROW", - /* 184 */ "ROWTS", - /* 185 */ "TBNAME", - /* 186 */ "QSTARTTS", - /* 187 */ "QENDTS", - /* 188 */ "WSTARTTS", - /* 189 */ "WENDTS", - /* 190 */ "WDURATION", - /* 191 */ "CAST", - /* 192 */ "NOW", - /* 193 */ "TODAY", - /* 194 */ "TIMEZONE", - /* 195 */ "COUNT", - /* 196 */ "FIRST", - /* 197 */ "LAST", + /* 117 */ "FIRST", + /* 118 */ "LAST", + /* 119 */ "SHOW", + /* 120 */ "DATABASES", + /* 121 */ "TABLES", + /* 122 */ "STABLES", + /* 123 */ "MNODES", + /* 124 */ "MODULES", + /* 125 */ "QNODES", + /* 126 */ "FUNCTIONS", + /* 127 */ "INDEXES", + /* 128 */ "ACCOUNTS", + /* 129 */ "APPS", + /* 130 */ "CONNECTIONS", + /* 131 */ "LICENCE", + /* 132 */ "GRANTS", + /* 133 */ "QUERIES", + /* 134 */ "SCORES", + /* 135 */ "TOPICS", + /* 136 */ "VARIABLES", + /* 137 */ "BNODES", + /* 138 */ "SNODES", + /* 139 */ "CLUSTER", + /* 140 */ "TRANSACTIONS", + /* 141 */ "DISTRIBUTED", + /* 142 */ "LIKE", + /* 143 */ "INDEX", + /* 144 */ "FULLTEXT", + /* 145 */ "FUNCTION", + /* 146 */ "INTERVAL", + /* 147 */ "TOPIC", + /* 148 */ "AS", + /* 149 */ "CONSUMER", + /* 150 */ "GROUP", + /* 151 */ "DESC", + /* 152 */ "DESCRIBE", + /* 153 */ "RESET", + /* 154 */ "QUERY", + /* 155 */ "CACHE", + /* 156 */ "EXPLAIN", + /* 157 */ "ANALYZE", + /* 158 */ "VERBOSE", + /* 159 */ "NK_BOOL", + /* 160 */ "RATIO", + /* 161 */ "NK_FLOAT", + /* 162 */ "COMPACT", + /* 163 */ "VNODES", + /* 164 */ "IN", + /* 165 */ "OUTPUTTYPE", + /* 166 */ "AGGREGATE", + /* 167 */ "BUFSIZE", + /* 168 */ "STREAM", + /* 169 */ "INTO", + /* 170 */ "TRIGGER", + /* 171 */ "AT_ONCE", + /* 172 */ "WINDOW_CLOSE", + /* 173 */ "KILL", + /* 174 */ "CONNECTION", + /* 175 */ "TRANSACTION", + /* 176 */ "BALANCE", + /* 177 */ "VGROUP", + /* 178 */ "MERGE", + /* 179 */ "REDISTRIBUTE", + /* 180 */ "SPLIT", + /* 181 */ "SYNCDB", + /* 182 */ "DELETE", + /* 183 */ "NULL", + /* 184 */ "NK_QUESTION", + /* 185 */ "NK_ARROW", + /* 186 */ "ROWTS", + /* 187 */ "TBNAME", + /* 188 */ "QSTARTTS", + /* 189 */ "QENDTS", + /* 190 */ "WSTARTTS", + /* 191 */ "WENDTS", + /* 192 */ "WDURATION", + /* 193 */ "CAST", + /* 194 */ "NOW", + /* 195 */ "TODAY", + /* 196 */ "TIMEZONE", + /* 197 */ "COUNT", /* 198 */ "LAST_ROW", /* 199 */ "BETWEEN", /* 200 */ "IS", @@ -1388,140 +1386,144 @@ static const char *const yyTokenName[] = { /* 223 */ "LINEAR", /* 224 */ "NEXT", /* 225 */ "HAVING", - /* 226 */ "ORDER", - /* 227 */ "SLIMIT", - /* 228 */ "SOFFSET", - /* 229 */ "LIMIT", - /* 230 */ "OFFSET", - /* 231 */ "ASC", - /* 232 */ "NULLS", - /* 233 */ "ID", - /* 234 */ "NK_BITNOT", - /* 235 */ "INSERT", - /* 236 */ "VALUES", - /* 237 */ "IMPORT", - /* 238 */ "NK_SEMI", - /* 239 */ "FILE", - /* 240 */ "cmd", - /* 241 */ "account_options", - /* 242 */ "alter_account_options", - /* 243 */ "literal", - /* 244 */ "alter_account_option", - /* 245 */ "user_name", - /* 246 */ "privileges", - /* 247 */ "priv_level", - /* 248 */ "priv_type_list", - /* 249 */ "priv_type", - /* 250 */ "db_name", - /* 251 */ "dnode_endpoint", - /* 252 */ "dnode_host_name", - /* 253 */ "not_exists_opt", - /* 254 */ "db_options", - /* 255 */ "exists_opt", - /* 256 */ "alter_db_options", - /* 257 */ "integer_list", - /* 258 */ "variable_list", - /* 259 */ "retention_list", - /* 260 */ "alter_db_option", - /* 261 */ "retention", - /* 262 */ "full_table_name", - /* 263 */ "column_def_list", - /* 264 */ "tags_def_opt", - /* 265 */ "table_options", - /* 266 */ "multi_create_clause", - /* 267 */ "tags_def", - /* 268 */ "multi_drop_clause", - /* 269 */ "alter_table_clause", - /* 270 */ "alter_table_options", - /* 271 */ "column_name", - /* 272 */ "type_name", - /* 273 */ "signed_literal", - /* 274 */ "create_subtable_clause", - /* 275 */ "specific_tags_opt", - /* 276 */ "literal_list", - /* 277 */ "drop_table_clause", - /* 278 */ "col_name_list", - /* 279 */ "table_name", - /* 280 */ "column_def", - /* 281 */ "func_name_list", - /* 282 */ "alter_table_option", - /* 283 */ "col_name", - /* 284 */ "db_name_cond_opt", - /* 285 */ "like_pattern_opt", - /* 286 */ "table_name_cond", - /* 287 */ "from_db_opt", - /* 288 */ "func_name", - /* 289 */ "function_name", - /* 290 */ "index_name", - /* 291 */ "index_options", - /* 292 */ "func_list", - /* 293 */ "duration_literal", - /* 294 */ "sliding_opt", - /* 295 */ "func", - /* 296 */ "expression_list", - /* 297 */ "topic_name", - /* 298 */ "query_expression", - /* 299 */ "cgroup_name", - /* 300 */ "analyze_opt", - /* 301 */ "explain_options", - /* 302 */ "agg_func_opt", - /* 303 */ "bufsize_opt", - /* 304 */ "stream_name", - /* 305 */ "stream_options", - /* 306 */ "into_opt", - /* 307 */ "dnode_list", - /* 308 */ "where_clause_opt", - /* 309 */ "signed", - /* 310 */ "literal_func", - /* 311 */ "table_alias", - /* 312 */ "column_alias", - /* 313 */ "expression", - /* 314 */ "pseudo_column", - /* 315 */ "column_reference", - /* 316 */ "function_expression", - /* 317 */ "subquery", - /* 318 */ "star_func", - /* 319 */ "star_func_para_list", - /* 320 */ "noarg_func", - /* 321 */ "other_para_list", - /* 322 */ "star_func_para", - /* 323 */ "predicate", - /* 324 */ "compare_op", - /* 325 */ "in_op", - /* 326 */ "in_predicate_value", - /* 327 */ "boolean_value_expression", - /* 328 */ "boolean_primary", - /* 329 */ "common_expression", - /* 330 */ "from_clause", - /* 331 */ "table_reference_list", - /* 332 */ "table_reference", - /* 333 */ "table_primary", - /* 334 */ "joined_table", - /* 335 */ "alias_opt", - /* 336 */ "parenthesized_joined_table", - /* 337 */ "join_type", - /* 338 */ "search_condition", - /* 339 */ "query_specification", - /* 340 */ "set_quantifier_opt", - /* 341 */ "select_list", - /* 342 */ "partition_by_clause_opt", - /* 343 */ "twindow_clause_opt", - /* 344 */ "group_by_clause_opt", - /* 345 */ "having_clause_opt", - /* 346 */ "select_sublist", - /* 347 */ "select_item", - /* 348 */ "fill_opt", - /* 349 */ "fill_mode", - /* 350 */ "group_by_list", - /* 351 */ "query_expression_body", - /* 352 */ "order_by_clause_opt", - /* 353 */ "slimit_clause_opt", - /* 354 */ "limit_clause_opt", - /* 355 */ "query_primary", - /* 356 */ "sort_specification_list", - /* 357 */ "sort_specification", - /* 358 */ "ordering_specification_opt", - /* 359 */ "null_ordering_opt", + /* 226 */ "RANGE", + /* 227 */ "EVERY", + /* 228 */ "ORDER", + /* 229 */ "SLIMIT", + /* 230 */ "SOFFSET", + /* 231 */ "LIMIT", + /* 232 */ "OFFSET", + /* 233 */ "ASC", + /* 234 */ "NULLS", + /* 235 */ "ID", + /* 236 */ "NK_BITNOT", + /* 237 */ "INSERT", + /* 238 */ "VALUES", + /* 239 */ "IMPORT", + /* 240 */ "NK_SEMI", + /* 241 */ "FILE", + /* 242 */ "cmd", + /* 243 */ "account_options", + /* 244 */ "alter_account_options", + /* 245 */ "literal", + /* 246 */ "alter_account_option", + /* 247 */ "user_name", + /* 248 */ "privileges", + /* 249 */ "priv_level", + /* 250 */ "priv_type_list", + /* 251 */ "priv_type", + /* 252 */ "db_name", + /* 253 */ "dnode_endpoint", + /* 254 */ "not_exists_opt", + /* 255 */ "db_options", + /* 256 */ "exists_opt", + /* 257 */ "alter_db_options", + /* 258 */ "integer_list", + /* 259 */ "variable_list", + /* 260 */ "retention_list", + /* 261 */ "alter_db_option", + /* 262 */ "retention", + /* 263 */ "full_table_name", + /* 264 */ "column_def_list", + /* 265 */ "tags_def_opt", + /* 266 */ "table_options", + /* 267 */ "multi_create_clause", + /* 268 */ "tags_def", + /* 269 */ "multi_drop_clause", + /* 270 */ "alter_table_clause", + /* 271 */ "alter_table_options", + /* 272 */ "column_name", + /* 273 */ "type_name", + /* 274 */ "signed_literal", + /* 275 */ "create_subtable_clause", + /* 276 */ "specific_tags_opt", + /* 277 */ "literal_list", + /* 278 */ "drop_table_clause", + /* 279 */ "col_name_list", + /* 280 */ "table_name", + /* 281 */ "column_def", + /* 282 */ "duration_list", + /* 283 */ "rollup_func_list", + /* 284 */ "alter_table_option", + /* 285 */ "duration_literal", + /* 286 */ "rollup_func_name", + /* 287 */ "function_name", + /* 288 */ "col_name", + /* 289 */ "db_name_cond_opt", + /* 290 */ "like_pattern_opt", + /* 291 */ "table_name_cond", + /* 292 */ "from_db_opt", + /* 293 */ "index_name", + /* 294 */ "index_options", + /* 295 */ "func_list", + /* 296 */ "sliding_opt", + /* 297 */ "func", + /* 298 */ "expression_list", + /* 299 */ "topic_name", + /* 300 */ "query_expression", + /* 301 */ "cgroup_name", + /* 302 */ "analyze_opt", + /* 303 */ "explain_options", + /* 304 */ "agg_func_opt", + /* 305 */ "bufsize_opt", + /* 306 */ "stream_name", + /* 307 */ "stream_options", + /* 308 */ "into_opt", + /* 309 */ "dnode_list", + /* 310 */ "where_clause_opt", + /* 311 */ "signed", + /* 312 */ "literal_func", + /* 313 */ "table_alias", + /* 314 */ "column_alias", + /* 315 */ "expression", + /* 316 */ "pseudo_column", + /* 317 */ "column_reference", + /* 318 */ "function_expression", + /* 319 */ "subquery", + /* 320 */ "star_func", + /* 321 */ "star_func_para_list", + /* 322 */ "noarg_func", + /* 323 */ "other_para_list", + /* 324 */ "star_func_para", + /* 325 */ "predicate", + /* 326 */ "compare_op", + /* 327 */ "in_op", + /* 328 */ "in_predicate_value", + /* 329 */ "boolean_value_expression", + /* 330 */ "boolean_primary", + /* 331 */ "common_expression", + /* 332 */ "from_clause", + /* 333 */ "table_reference_list", + /* 334 */ "table_reference", + /* 335 */ "table_primary", + /* 336 */ "joined_table", + /* 337 */ "alias_opt", + /* 338 */ "parenthesized_joined_table", + /* 339 */ "join_type", + /* 340 */ "search_condition", + /* 341 */ "query_specification", + /* 342 */ "set_quantifier_opt", + /* 343 */ "select_list", + /* 344 */ "partition_by_clause_opt", + /* 345 */ "range_opt", + /* 346 */ "every_opt", + /* 347 */ "fill_opt", + /* 348 */ "twindow_clause_opt", + /* 349 */ "group_by_clause_opt", + /* 350 */ "having_clause_opt", + /* 351 */ "select_sublist", + /* 352 */ "select_item", + /* 353 */ "fill_mode", + /* 354 */ "group_by_list", + /* 355 */ "query_expression_body", + /* 356 */ "order_by_clause_opt", + /* 357 */ "slimit_clause_opt", + /* 358 */ "limit_clause_opt", + /* 359 */ "query_primary", + /* 360 */ "sort_specification_list", + /* 361 */ "sort_specification", + /* 362 */ "ordering_specification_opt", + /* 363 */ "null_ordering_opt", }; #endif /* defined(YYCOVERAGE) || !defined(NDEBUG) */ @@ -1568,7 +1570,7 @@ static const char *const yyRuleName[] = { /* 36 */ "priv_level ::= NK_STAR NK_DOT NK_STAR", /* 37 */ "priv_level ::= db_name NK_DOT NK_STAR", /* 38 */ "cmd ::= CREATE DNODE dnode_endpoint", - /* 39 */ "cmd ::= CREATE DNODE dnode_host_name PORT NK_INTEGER", + /* 39 */ "cmd ::= CREATE DNODE dnode_endpoint PORT NK_INTEGER", /* 40 */ "cmd ::= DROP DNODE NK_INTEGER", /* 41 */ "cmd ::= DROP DNODE dnode_endpoint", /* 42 */ "cmd ::= ALTER DNODE NK_INTEGER NK_STRING", @@ -1576,8 +1578,8 @@ static const char *const yyRuleName[] = { /* 44 */ "cmd ::= ALTER ALL DNODES NK_STRING", /* 45 */ "cmd ::= ALTER ALL DNODES NK_STRING NK_STRING", /* 46 */ "dnode_endpoint ::= NK_STRING", - /* 47 */ "dnode_host_name ::= NK_ID", - /* 48 */ "dnode_host_name ::= NK_IPTOKEN", + /* 47 */ "dnode_endpoint ::= NK_ID", + /* 48 */ "dnode_endpoint ::= NK_IPTOKEN", /* 49 */ "cmd ::= ALTER LOCAL NK_STRING", /* 50 */ "cmd ::= ALTER LOCAL NK_STRING NK_STRING", /* 51 */ "cmd ::= CREATE QNODE ON DNODE NK_INTEGER", @@ -1694,295 +1696,305 @@ static const char *const yyRuleName[] = { /* 162 */ "tags_def ::= TAGS NK_LP column_def_list NK_RP", /* 163 */ "table_options ::=", /* 164 */ "table_options ::= table_options COMMENT NK_STRING", - /* 165 */ "table_options ::= table_options FILE_FACTOR NK_FLOAT", - /* 166 */ "table_options ::= table_options ROLLUP NK_LP func_name_list NK_RP", - /* 167 */ "table_options ::= table_options TTL NK_INTEGER", - /* 168 */ "table_options ::= table_options SMA NK_LP col_name_list NK_RP", - /* 169 */ "alter_table_options ::= alter_table_option", - /* 170 */ "alter_table_options ::= alter_table_options alter_table_option", - /* 171 */ "alter_table_option ::= COMMENT NK_STRING", - /* 172 */ "alter_table_option ::= TTL NK_INTEGER", - /* 173 */ "col_name_list ::= col_name", - /* 174 */ "col_name_list ::= col_name_list NK_COMMA col_name", - /* 175 */ "col_name ::= column_name", - /* 176 */ "cmd ::= SHOW DNODES", - /* 177 */ "cmd ::= SHOW USERS", - /* 178 */ "cmd ::= SHOW DATABASES", - /* 179 */ "cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt", - /* 180 */ "cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt", - /* 181 */ "cmd ::= SHOW db_name_cond_opt VGROUPS", - /* 182 */ "cmd ::= SHOW MNODES", - /* 183 */ "cmd ::= SHOW MODULES", - /* 184 */ "cmd ::= SHOW QNODES", - /* 185 */ "cmd ::= SHOW FUNCTIONS", - /* 186 */ "cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt", - /* 187 */ "cmd ::= SHOW STREAMS", - /* 188 */ "cmd ::= SHOW ACCOUNTS", - /* 189 */ "cmd ::= SHOW APPS", - /* 190 */ "cmd ::= SHOW CONNECTIONS", - /* 191 */ "cmd ::= SHOW LICENCE", - /* 192 */ "cmd ::= SHOW GRANTS", - /* 193 */ "cmd ::= SHOW CREATE DATABASE db_name", - /* 194 */ "cmd ::= SHOW CREATE TABLE full_table_name", - /* 195 */ "cmd ::= SHOW CREATE STABLE full_table_name", - /* 196 */ "cmd ::= SHOW QUERIES", - /* 197 */ "cmd ::= SHOW SCORES", - /* 198 */ "cmd ::= SHOW TOPICS", - /* 199 */ "cmd ::= SHOW VARIABLES", - /* 200 */ "cmd ::= SHOW BNODES", - /* 201 */ "cmd ::= SHOW SNODES", - /* 202 */ "cmd ::= SHOW CLUSTER", - /* 203 */ "cmd ::= SHOW TRANSACTIONS", - /* 204 */ "db_name_cond_opt ::=", - /* 205 */ "db_name_cond_opt ::= db_name NK_DOT", - /* 206 */ "like_pattern_opt ::=", - /* 207 */ "like_pattern_opt ::= LIKE NK_STRING", - /* 208 */ "table_name_cond ::= table_name", - /* 209 */ "from_db_opt ::=", - /* 210 */ "from_db_opt ::= FROM db_name", - /* 211 */ "func_name_list ::= func_name", - /* 212 */ "func_name_list ::= func_name_list NK_COMMA func_name", - /* 213 */ "func_name ::= function_name", - /* 214 */ "cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options", - /* 215 */ "cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP", - /* 216 */ "cmd ::= DROP INDEX exists_opt index_name ON table_name", - /* 217 */ "index_options ::=", - /* 218 */ "index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt", - /* 219 */ "index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt", - /* 220 */ "func_list ::= func", - /* 221 */ "func_list ::= func_list NK_COMMA func", - /* 222 */ "func ::= function_name NK_LP expression_list NK_RP", - /* 223 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression", - /* 224 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name", - /* 225 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name", - /* 226 */ "cmd ::= DROP TOPIC exists_opt topic_name", - /* 227 */ "cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name", - /* 228 */ "cmd ::= DESC full_table_name", - /* 229 */ "cmd ::= DESCRIBE full_table_name", - /* 230 */ "cmd ::= RESET QUERY CACHE", - /* 231 */ "cmd ::= EXPLAIN analyze_opt explain_options query_expression", - /* 232 */ "analyze_opt ::=", - /* 233 */ "analyze_opt ::= ANALYZE", - /* 234 */ "explain_options ::=", - /* 235 */ "explain_options ::= explain_options VERBOSE NK_BOOL", - /* 236 */ "explain_options ::= explain_options RATIO NK_FLOAT", - /* 237 */ "cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP", - /* 238 */ "cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt", - /* 239 */ "cmd ::= DROP FUNCTION exists_opt function_name", - /* 240 */ "agg_func_opt ::=", - /* 241 */ "agg_func_opt ::= AGGREGATE", - /* 242 */ "bufsize_opt ::=", - /* 243 */ "bufsize_opt ::= BUFSIZE NK_INTEGER", - /* 244 */ "cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression", - /* 245 */ "cmd ::= DROP STREAM exists_opt stream_name", - /* 246 */ "into_opt ::=", - /* 247 */ "into_opt ::= INTO full_table_name", - /* 248 */ "stream_options ::=", - /* 249 */ "stream_options ::= stream_options TRIGGER AT_ONCE", - /* 250 */ "stream_options ::= stream_options TRIGGER WINDOW_CLOSE", - /* 251 */ "stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal", - /* 252 */ "stream_options ::= stream_options WATERMARK duration_literal", - /* 253 */ "cmd ::= KILL CONNECTION NK_INTEGER", - /* 254 */ "cmd ::= KILL QUERY NK_STRING", - /* 255 */ "cmd ::= KILL TRANSACTION NK_INTEGER", - /* 256 */ "cmd ::= BALANCE VGROUP", - /* 257 */ "cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER", - /* 258 */ "cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list", - /* 259 */ "cmd ::= SPLIT VGROUP NK_INTEGER", - /* 260 */ "dnode_list ::= DNODE NK_INTEGER", - /* 261 */ "dnode_list ::= dnode_list DNODE NK_INTEGER", - /* 262 */ "cmd ::= SYNCDB db_name REPLICA", - /* 263 */ "cmd ::= DELETE FROM full_table_name where_clause_opt", - /* 264 */ "cmd ::= query_expression", - /* 265 */ "literal ::= NK_INTEGER", - /* 266 */ "literal ::= NK_FLOAT", - /* 267 */ "literal ::= NK_STRING", - /* 268 */ "literal ::= NK_BOOL", - /* 269 */ "literal ::= TIMESTAMP NK_STRING", - /* 270 */ "literal ::= duration_literal", - /* 271 */ "literal ::= NULL", - /* 272 */ "literal ::= NK_QUESTION", - /* 273 */ "duration_literal ::= NK_VARIABLE", - /* 274 */ "signed ::= NK_INTEGER", - /* 275 */ "signed ::= NK_PLUS NK_INTEGER", - /* 276 */ "signed ::= NK_MINUS NK_INTEGER", - /* 277 */ "signed ::= NK_FLOAT", - /* 278 */ "signed ::= NK_PLUS NK_FLOAT", - /* 279 */ "signed ::= NK_MINUS NK_FLOAT", - /* 280 */ "signed_literal ::= signed", - /* 281 */ "signed_literal ::= NK_STRING", - /* 282 */ "signed_literal ::= NK_BOOL", - /* 283 */ "signed_literal ::= TIMESTAMP NK_STRING", - /* 284 */ "signed_literal ::= duration_literal", - /* 285 */ "signed_literal ::= NULL", - /* 286 */ "signed_literal ::= literal_func", - /* 287 */ "literal_list ::= signed_literal", - /* 288 */ "literal_list ::= literal_list NK_COMMA signed_literal", - /* 289 */ "db_name ::= NK_ID", - /* 290 */ "table_name ::= NK_ID", - /* 291 */ "column_name ::= NK_ID", - /* 292 */ "function_name ::= NK_ID", - /* 293 */ "table_alias ::= NK_ID", - /* 294 */ "column_alias ::= NK_ID", - /* 295 */ "user_name ::= NK_ID", - /* 296 */ "index_name ::= NK_ID", - /* 297 */ "topic_name ::= NK_ID", - /* 298 */ "stream_name ::= NK_ID", - /* 299 */ "cgroup_name ::= NK_ID", - /* 300 */ "expression ::= literal", - /* 301 */ "expression ::= pseudo_column", - /* 302 */ "expression ::= column_reference", - /* 303 */ "expression ::= function_expression", - /* 304 */ "expression ::= subquery", - /* 305 */ "expression ::= NK_LP expression NK_RP", - /* 306 */ "expression ::= NK_PLUS expression", - /* 307 */ "expression ::= NK_MINUS expression", - /* 308 */ "expression ::= expression NK_PLUS expression", - /* 309 */ "expression ::= expression NK_MINUS expression", - /* 310 */ "expression ::= expression NK_STAR expression", - /* 311 */ "expression ::= expression NK_SLASH expression", - /* 312 */ "expression ::= expression NK_REM expression", - /* 313 */ "expression ::= column_reference NK_ARROW NK_STRING", - /* 314 */ "expression_list ::= expression", - /* 315 */ "expression_list ::= expression_list NK_COMMA expression", - /* 316 */ "column_reference ::= column_name", - /* 317 */ "column_reference ::= table_name NK_DOT column_name", - /* 318 */ "pseudo_column ::= ROWTS", - /* 319 */ "pseudo_column ::= TBNAME", - /* 320 */ "pseudo_column ::= table_name NK_DOT TBNAME", - /* 321 */ "pseudo_column ::= QSTARTTS", - /* 322 */ "pseudo_column ::= QENDTS", - /* 323 */ "pseudo_column ::= WSTARTTS", - /* 324 */ "pseudo_column ::= WENDTS", - /* 325 */ "pseudo_column ::= WDURATION", - /* 326 */ "function_expression ::= function_name NK_LP expression_list NK_RP", - /* 327 */ "function_expression ::= star_func NK_LP star_func_para_list NK_RP", - /* 328 */ "function_expression ::= CAST NK_LP expression AS type_name NK_RP", - /* 329 */ "function_expression ::= literal_func", - /* 330 */ "literal_func ::= noarg_func NK_LP NK_RP", - /* 331 */ "literal_func ::= NOW", - /* 332 */ "noarg_func ::= NOW", - /* 333 */ "noarg_func ::= TODAY", - /* 334 */ "noarg_func ::= TIMEZONE", - /* 335 */ "star_func ::= COUNT", - /* 336 */ "star_func ::= FIRST", - /* 337 */ "star_func ::= LAST", - /* 338 */ "star_func ::= LAST_ROW", - /* 339 */ "star_func_para_list ::= NK_STAR", - /* 340 */ "star_func_para_list ::= other_para_list", - /* 341 */ "other_para_list ::= star_func_para", - /* 342 */ "other_para_list ::= other_para_list NK_COMMA star_func_para", - /* 343 */ "star_func_para ::= expression", - /* 344 */ "star_func_para ::= table_name NK_DOT NK_STAR", - /* 345 */ "predicate ::= expression compare_op expression", - /* 346 */ "predicate ::= expression BETWEEN expression AND expression", - /* 347 */ "predicate ::= expression NOT BETWEEN expression AND expression", - /* 348 */ "predicate ::= expression IS NULL", - /* 349 */ "predicate ::= expression IS NOT NULL", - /* 350 */ "predicate ::= expression in_op in_predicate_value", - /* 351 */ "compare_op ::= NK_LT", - /* 352 */ "compare_op ::= NK_GT", - /* 353 */ "compare_op ::= NK_LE", - /* 354 */ "compare_op ::= NK_GE", - /* 355 */ "compare_op ::= NK_NE", - /* 356 */ "compare_op ::= NK_EQ", - /* 357 */ "compare_op ::= LIKE", - /* 358 */ "compare_op ::= NOT LIKE", - /* 359 */ "compare_op ::= MATCH", - /* 360 */ "compare_op ::= NMATCH", - /* 361 */ "compare_op ::= CONTAINS", - /* 362 */ "in_op ::= IN", - /* 363 */ "in_op ::= NOT IN", - /* 364 */ "in_predicate_value ::= NK_LP expression_list NK_RP", - /* 365 */ "boolean_value_expression ::= boolean_primary", - /* 366 */ "boolean_value_expression ::= NOT boolean_primary", - /* 367 */ "boolean_value_expression ::= boolean_value_expression OR boolean_value_expression", - /* 368 */ "boolean_value_expression ::= boolean_value_expression AND boolean_value_expression", - /* 369 */ "boolean_primary ::= predicate", - /* 370 */ "boolean_primary ::= NK_LP boolean_value_expression NK_RP", - /* 371 */ "common_expression ::= expression", - /* 372 */ "common_expression ::= boolean_value_expression", - /* 373 */ "from_clause ::= FROM table_reference_list", - /* 374 */ "table_reference_list ::= table_reference", - /* 375 */ "table_reference_list ::= table_reference_list NK_COMMA table_reference", - /* 376 */ "table_reference ::= table_primary", - /* 377 */ "table_reference ::= joined_table", - /* 378 */ "table_primary ::= table_name alias_opt", - /* 379 */ "table_primary ::= db_name NK_DOT table_name alias_opt", - /* 380 */ "table_primary ::= subquery alias_opt", - /* 381 */ "table_primary ::= parenthesized_joined_table", - /* 382 */ "alias_opt ::=", - /* 383 */ "alias_opt ::= table_alias", - /* 384 */ "alias_opt ::= AS table_alias", - /* 385 */ "parenthesized_joined_table ::= NK_LP joined_table NK_RP", - /* 386 */ "parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP", - /* 387 */ "joined_table ::= table_reference join_type JOIN table_reference ON search_condition", - /* 388 */ "join_type ::=", - /* 389 */ "join_type ::= INNER", - /* 390 */ "query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt twindow_clause_opt group_by_clause_opt having_clause_opt", - /* 391 */ "set_quantifier_opt ::=", - /* 392 */ "set_quantifier_opt ::= DISTINCT", - /* 393 */ "set_quantifier_opt ::= ALL", - /* 394 */ "select_list ::= NK_STAR", - /* 395 */ "select_list ::= select_sublist", - /* 396 */ "select_sublist ::= select_item", - /* 397 */ "select_sublist ::= select_sublist NK_COMMA select_item", - /* 398 */ "select_item ::= common_expression", - /* 399 */ "select_item ::= common_expression column_alias", - /* 400 */ "select_item ::= common_expression AS column_alias", - /* 401 */ "select_item ::= table_name NK_DOT NK_STAR", - /* 402 */ "where_clause_opt ::=", - /* 403 */ "where_clause_opt ::= WHERE search_condition", - /* 404 */ "partition_by_clause_opt ::=", - /* 405 */ "partition_by_clause_opt ::= PARTITION BY expression_list", - /* 406 */ "twindow_clause_opt ::=", - /* 407 */ "twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP", - /* 408 */ "twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP", - /* 409 */ "twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt", - /* 410 */ "twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt", - /* 411 */ "sliding_opt ::=", - /* 412 */ "sliding_opt ::= SLIDING NK_LP duration_literal NK_RP", - /* 413 */ "fill_opt ::=", - /* 414 */ "fill_opt ::= FILL NK_LP fill_mode NK_RP", - /* 415 */ "fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP", - /* 416 */ "fill_mode ::= NONE", - /* 417 */ "fill_mode ::= PREV", - /* 418 */ "fill_mode ::= NULL", - /* 419 */ "fill_mode ::= LINEAR", - /* 420 */ "fill_mode ::= NEXT", - /* 421 */ "group_by_clause_opt ::=", - /* 422 */ "group_by_clause_opt ::= GROUP BY group_by_list", - /* 423 */ "group_by_list ::= expression", - /* 424 */ "group_by_list ::= group_by_list NK_COMMA expression", - /* 425 */ "having_clause_opt ::=", - /* 426 */ "having_clause_opt ::= HAVING search_condition", - /* 427 */ "query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt", - /* 428 */ "query_expression_body ::= query_primary", - /* 429 */ "query_expression_body ::= query_expression_body UNION ALL query_expression_body", - /* 430 */ "query_expression_body ::= query_expression_body UNION query_expression_body", - /* 431 */ "query_primary ::= query_specification", - /* 432 */ "query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP", - /* 433 */ "order_by_clause_opt ::=", - /* 434 */ "order_by_clause_opt ::= ORDER BY sort_specification_list", - /* 435 */ "slimit_clause_opt ::=", - /* 436 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER", - /* 437 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER", - /* 438 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER", - /* 439 */ "limit_clause_opt ::=", - /* 440 */ "limit_clause_opt ::= LIMIT NK_INTEGER", - /* 441 */ "limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER", - /* 442 */ "limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER", - /* 443 */ "subquery ::= NK_LP query_expression NK_RP", - /* 444 */ "search_condition ::= common_expression", - /* 445 */ "sort_specification_list ::= sort_specification", - /* 446 */ "sort_specification_list ::= sort_specification_list NK_COMMA sort_specification", - /* 447 */ "sort_specification ::= expression ordering_specification_opt null_ordering_opt", - /* 448 */ "ordering_specification_opt ::=", - /* 449 */ "ordering_specification_opt ::= ASC", - /* 450 */ "ordering_specification_opt ::= DESC", - /* 451 */ "null_ordering_opt ::=", - /* 452 */ "null_ordering_opt ::= NULLS FIRST", - /* 453 */ "null_ordering_opt ::= NULLS LAST", + /* 165 */ "table_options ::= table_options MAX_DELAY duration_list", + /* 166 */ "table_options ::= table_options WATERMARK duration_list", + /* 167 */ "table_options ::= table_options ROLLUP NK_LP rollup_func_list NK_RP", + /* 168 */ "table_options ::= table_options TTL NK_INTEGER", + /* 169 */ "table_options ::= table_options SMA NK_LP col_name_list NK_RP", + /* 170 */ "alter_table_options ::= alter_table_option", + /* 171 */ "alter_table_options ::= alter_table_options alter_table_option", + /* 172 */ "alter_table_option ::= COMMENT NK_STRING", + /* 173 */ "alter_table_option ::= TTL NK_INTEGER", + /* 174 */ "duration_list ::= duration_literal", + /* 175 */ "duration_list ::= duration_list NK_COMMA duration_literal", + /* 176 */ "rollup_func_list ::= rollup_func_name", + /* 177 */ "rollup_func_list ::= rollup_func_list NK_COMMA rollup_func_name", + /* 178 */ "rollup_func_name ::= function_name", + /* 179 */ "rollup_func_name ::= FIRST", + /* 180 */ "rollup_func_name ::= LAST", + /* 181 */ "col_name_list ::= col_name", + /* 182 */ "col_name_list ::= col_name_list NK_COMMA col_name", + /* 183 */ "col_name ::= column_name", + /* 184 */ "cmd ::= SHOW DNODES", + /* 185 */ "cmd ::= SHOW USERS", + /* 186 */ "cmd ::= SHOW DATABASES", + /* 187 */ "cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt", + /* 188 */ "cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt", + /* 189 */ "cmd ::= SHOW db_name_cond_opt VGROUPS", + /* 190 */ "cmd ::= SHOW MNODES", + /* 191 */ "cmd ::= SHOW MODULES", + /* 192 */ "cmd ::= SHOW QNODES", + /* 193 */ "cmd ::= SHOW FUNCTIONS", + /* 194 */ "cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt", + /* 195 */ "cmd ::= SHOW STREAMS", + /* 196 */ "cmd ::= SHOW ACCOUNTS", + /* 197 */ "cmd ::= SHOW APPS", + /* 198 */ "cmd ::= SHOW CONNECTIONS", + /* 199 */ "cmd ::= SHOW LICENCE", + /* 200 */ "cmd ::= SHOW GRANTS", + /* 201 */ "cmd ::= SHOW CREATE DATABASE db_name", + /* 202 */ "cmd ::= SHOW CREATE TABLE full_table_name", + /* 203 */ "cmd ::= SHOW CREATE STABLE full_table_name", + /* 204 */ "cmd ::= SHOW QUERIES", + /* 205 */ "cmd ::= SHOW SCORES", + /* 206 */ "cmd ::= SHOW TOPICS", + /* 207 */ "cmd ::= SHOW VARIABLES", + /* 208 */ "cmd ::= SHOW BNODES", + /* 209 */ "cmd ::= SHOW SNODES", + /* 210 */ "cmd ::= SHOW CLUSTER", + /* 211 */ "cmd ::= SHOW TRANSACTIONS", + /* 212 */ "cmd ::= SHOW TABLE DISTRIBUTED full_table_name", + /* 213 */ "db_name_cond_opt ::=", + /* 214 */ "db_name_cond_opt ::= db_name NK_DOT", + /* 215 */ "like_pattern_opt ::=", + /* 216 */ "like_pattern_opt ::= LIKE NK_STRING", + /* 217 */ "table_name_cond ::= table_name", + /* 218 */ "from_db_opt ::=", + /* 219 */ "from_db_opt ::= FROM db_name", + /* 220 */ "cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options", + /* 221 */ "cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP", + /* 222 */ "cmd ::= DROP INDEX exists_opt index_name ON table_name", + /* 223 */ "index_options ::=", + /* 224 */ "index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt", + /* 225 */ "index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt", + /* 226 */ "func_list ::= func", + /* 227 */ "func_list ::= func_list NK_COMMA func", + /* 228 */ "func ::= function_name NK_LP expression_list NK_RP", + /* 229 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression", + /* 230 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name", + /* 231 */ "cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name", + /* 232 */ "cmd ::= DROP TOPIC exists_opt topic_name", + /* 233 */ "cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name", + /* 234 */ "cmd ::= DESC full_table_name", + /* 235 */ "cmd ::= DESCRIBE full_table_name", + /* 236 */ "cmd ::= RESET QUERY CACHE", + /* 237 */ "cmd ::= EXPLAIN analyze_opt explain_options query_expression", + /* 238 */ "analyze_opt ::=", + /* 239 */ "analyze_opt ::= ANALYZE", + /* 240 */ "explain_options ::=", + /* 241 */ "explain_options ::= explain_options VERBOSE NK_BOOL", + /* 242 */ "explain_options ::= explain_options RATIO NK_FLOAT", + /* 243 */ "cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP", + /* 244 */ "cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt", + /* 245 */ "cmd ::= DROP FUNCTION exists_opt function_name", + /* 246 */ "agg_func_opt ::=", + /* 247 */ "agg_func_opt ::= AGGREGATE", + /* 248 */ "bufsize_opt ::=", + /* 249 */ "bufsize_opt ::= BUFSIZE NK_INTEGER", + /* 250 */ "cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression", + /* 251 */ "cmd ::= DROP STREAM exists_opt stream_name", + /* 252 */ "into_opt ::=", + /* 253 */ "into_opt ::= INTO full_table_name", + /* 254 */ "stream_options ::=", + /* 255 */ "stream_options ::= stream_options TRIGGER AT_ONCE", + /* 256 */ "stream_options ::= stream_options TRIGGER WINDOW_CLOSE", + /* 257 */ "stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal", + /* 258 */ "stream_options ::= stream_options WATERMARK duration_literal", + /* 259 */ "cmd ::= KILL CONNECTION NK_INTEGER", + /* 260 */ "cmd ::= KILL QUERY NK_STRING", + /* 261 */ "cmd ::= KILL TRANSACTION NK_INTEGER", + /* 262 */ "cmd ::= BALANCE VGROUP", + /* 263 */ "cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER", + /* 264 */ "cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list", + /* 265 */ "cmd ::= SPLIT VGROUP NK_INTEGER", + /* 266 */ "dnode_list ::= DNODE NK_INTEGER", + /* 267 */ "dnode_list ::= dnode_list DNODE NK_INTEGER", + /* 268 */ "cmd ::= SYNCDB db_name REPLICA", + /* 269 */ "cmd ::= DELETE FROM full_table_name where_clause_opt", + /* 270 */ "cmd ::= query_expression", + /* 271 */ "literal ::= NK_INTEGER", + /* 272 */ "literal ::= NK_FLOAT", + /* 273 */ "literal ::= NK_STRING", + /* 274 */ "literal ::= NK_BOOL", + /* 275 */ "literal ::= TIMESTAMP NK_STRING", + /* 276 */ "literal ::= duration_literal", + /* 277 */ "literal ::= NULL", + /* 278 */ "literal ::= NK_QUESTION", + /* 279 */ "duration_literal ::= NK_VARIABLE", + /* 280 */ "signed ::= NK_INTEGER", + /* 281 */ "signed ::= NK_PLUS NK_INTEGER", + /* 282 */ "signed ::= NK_MINUS NK_INTEGER", + /* 283 */ "signed ::= NK_FLOAT", + /* 284 */ "signed ::= NK_PLUS NK_FLOAT", + /* 285 */ "signed ::= NK_MINUS NK_FLOAT", + /* 286 */ "signed_literal ::= signed", + /* 287 */ "signed_literal ::= NK_STRING", + /* 288 */ "signed_literal ::= NK_BOOL", + /* 289 */ "signed_literal ::= TIMESTAMP NK_STRING", + /* 290 */ "signed_literal ::= duration_literal", + /* 291 */ "signed_literal ::= NULL", + /* 292 */ "signed_literal ::= literal_func", + /* 293 */ "literal_list ::= signed_literal", + /* 294 */ "literal_list ::= literal_list NK_COMMA signed_literal", + /* 295 */ "db_name ::= NK_ID", + /* 296 */ "table_name ::= NK_ID", + /* 297 */ "column_name ::= NK_ID", + /* 298 */ "function_name ::= NK_ID", + /* 299 */ "table_alias ::= NK_ID", + /* 300 */ "column_alias ::= NK_ID", + /* 301 */ "user_name ::= NK_ID", + /* 302 */ "index_name ::= NK_ID", + /* 303 */ "topic_name ::= NK_ID", + /* 304 */ "stream_name ::= NK_ID", + /* 305 */ "cgroup_name ::= NK_ID", + /* 306 */ "expression ::= literal", + /* 307 */ "expression ::= pseudo_column", + /* 308 */ "expression ::= column_reference", + /* 309 */ "expression ::= function_expression", + /* 310 */ "expression ::= subquery", + /* 311 */ "expression ::= NK_LP expression NK_RP", + /* 312 */ "expression ::= NK_PLUS expression", + /* 313 */ "expression ::= NK_MINUS expression", + /* 314 */ "expression ::= expression NK_PLUS expression", + /* 315 */ "expression ::= expression NK_MINUS expression", + /* 316 */ "expression ::= expression NK_STAR expression", + /* 317 */ "expression ::= expression NK_SLASH expression", + /* 318 */ "expression ::= expression NK_REM expression", + /* 319 */ "expression ::= column_reference NK_ARROW NK_STRING", + /* 320 */ "expression_list ::= expression", + /* 321 */ "expression_list ::= expression_list NK_COMMA expression", + /* 322 */ "column_reference ::= column_name", + /* 323 */ "column_reference ::= table_name NK_DOT column_name", + /* 324 */ "pseudo_column ::= ROWTS", + /* 325 */ "pseudo_column ::= TBNAME", + /* 326 */ "pseudo_column ::= table_name NK_DOT TBNAME", + /* 327 */ "pseudo_column ::= QSTARTTS", + /* 328 */ "pseudo_column ::= QENDTS", + /* 329 */ "pseudo_column ::= WSTARTTS", + /* 330 */ "pseudo_column ::= WENDTS", + /* 331 */ "pseudo_column ::= WDURATION", + /* 332 */ "function_expression ::= function_name NK_LP expression_list NK_RP", + /* 333 */ "function_expression ::= star_func NK_LP star_func_para_list NK_RP", + /* 334 */ "function_expression ::= CAST NK_LP expression AS type_name NK_RP", + /* 335 */ "function_expression ::= literal_func", + /* 336 */ "literal_func ::= noarg_func NK_LP NK_RP", + /* 337 */ "literal_func ::= NOW", + /* 338 */ "noarg_func ::= NOW", + /* 339 */ "noarg_func ::= TODAY", + /* 340 */ "noarg_func ::= TIMEZONE", + /* 341 */ "star_func ::= COUNT", + /* 342 */ "star_func ::= FIRST", + /* 343 */ "star_func ::= LAST", + /* 344 */ "star_func ::= LAST_ROW", + /* 345 */ "star_func_para_list ::= NK_STAR", + /* 346 */ "star_func_para_list ::= other_para_list", + /* 347 */ "other_para_list ::= star_func_para", + /* 348 */ "other_para_list ::= other_para_list NK_COMMA star_func_para", + /* 349 */ "star_func_para ::= expression", + /* 350 */ "star_func_para ::= table_name NK_DOT NK_STAR", + /* 351 */ "predicate ::= expression compare_op expression", + /* 352 */ "predicate ::= expression BETWEEN expression AND expression", + /* 353 */ "predicate ::= expression NOT BETWEEN expression AND expression", + /* 354 */ "predicate ::= expression IS NULL", + /* 355 */ "predicate ::= expression IS NOT NULL", + /* 356 */ "predicate ::= expression in_op in_predicate_value", + /* 357 */ "compare_op ::= NK_LT", + /* 358 */ "compare_op ::= NK_GT", + /* 359 */ "compare_op ::= NK_LE", + /* 360 */ "compare_op ::= NK_GE", + /* 361 */ "compare_op ::= NK_NE", + /* 362 */ "compare_op ::= NK_EQ", + /* 363 */ "compare_op ::= LIKE", + /* 364 */ "compare_op ::= NOT LIKE", + /* 365 */ "compare_op ::= MATCH", + /* 366 */ "compare_op ::= NMATCH", + /* 367 */ "compare_op ::= CONTAINS", + /* 368 */ "in_op ::= IN", + /* 369 */ "in_op ::= NOT IN", + /* 370 */ "in_predicate_value ::= NK_LP expression_list NK_RP", + /* 371 */ "boolean_value_expression ::= boolean_primary", + /* 372 */ "boolean_value_expression ::= NOT boolean_primary", + /* 373 */ "boolean_value_expression ::= boolean_value_expression OR boolean_value_expression", + /* 374 */ "boolean_value_expression ::= boolean_value_expression AND boolean_value_expression", + /* 375 */ "boolean_primary ::= predicate", + /* 376 */ "boolean_primary ::= NK_LP boolean_value_expression NK_RP", + /* 377 */ "common_expression ::= expression", + /* 378 */ "common_expression ::= boolean_value_expression", + /* 379 */ "from_clause ::= FROM table_reference_list", + /* 380 */ "table_reference_list ::= table_reference", + /* 381 */ "table_reference_list ::= table_reference_list NK_COMMA table_reference", + /* 382 */ "table_reference ::= table_primary", + /* 383 */ "table_reference ::= joined_table", + /* 384 */ "table_primary ::= table_name alias_opt", + /* 385 */ "table_primary ::= db_name NK_DOT table_name alias_opt", + /* 386 */ "table_primary ::= subquery alias_opt", + /* 387 */ "table_primary ::= parenthesized_joined_table", + /* 388 */ "alias_opt ::=", + /* 389 */ "alias_opt ::= table_alias", + /* 390 */ "alias_opt ::= AS table_alias", + /* 391 */ "parenthesized_joined_table ::= NK_LP joined_table NK_RP", + /* 392 */ "parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP", + /* 393 */ "joined_table ::= table_reference join_type JOIN table_reference ON search_condition", + /* 394 */ "join_type ::=", + /* 395 */ "join_type ::= INNER", + /* 396 */ "query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt range_opt every_opt fill_opt twindow_clause_opt group_by_clause_opt having_clause_opt", + /* 397 */ "set_quantifier_opt ::=", + /* 398 */ "set_quantifier_opt ::= DISTINCT", + /* 399 */ "set_quantifier_opt ::= ALL", + /* 400 */ "select_list ::= NK_STAR", + /* 401 */ "select_list ::= select_sublist", + /* 402 */ "select_sublist ::= select_item", + /* 403 */ "select_sublist ::= select_sublist NK_COMMA select_item", + /* 404 */ "select_item ::= common_expression", + /* 405 */ "select_item ::= common_expression column_alias", + /* 406 */ "select_item ::= common_expression AS column_alias", + /* 407 */ "select_item ::= table_name NK_DOT NK_STAR", + /* 408 */ "where_clause_opt ::=", + /* 409 */ "where_clause_opt ::= WHERE search_condition", + /* 410 */ "partition_by_clause_opt ::=", + /* 411 */ "partition_by_clause_opt ::= PARTITION BY expression_list", + /* 412 */ "twindow_clause_opt ::=", + /* 413 */ "twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP", + /* 414 */ "twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP", + /* 415 */ "twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt", + /* 416 */ "twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt", + /* 417 */ "sliding_opt ::=", + /* 418 */ "sliding_opt ::= SLIDING NK_LP duration_literal NK_RP", + /* 419 */ "fill_opt ::=", + /* 420 */ "fill_opt ::= FILL NK_LP fill_mode NK_RP", + /* 421 */ "fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP", + /* 422 */ "fill_mode ::= NONE", + /* 423 */ "fill_mode ::= PREV", + /* 424 */ "fill_mode ::= NULL", + /* 425 */ "fill_mode ::= LINEAR", + /* 426 */ "fill_mode ::= NEXT", + /* 427 */ "group_by_clause_opt ::=", + /* 428 */ "group_by_clause_opt ::= GROUP BY group_by_list", + /* 429 */ "group_by_list ::= expression", + /* 430 */ "group_by_list ::= group_by_list NK_COMMA expression", + /* 431 */ "having_clause_opt ::=", + /* 432 */ "having_clause_opt ::= HAVING search_condition", + /* 433 */ "range_opt ::=", + /* 434 */ "range_opt ::= RANGE NK_LP expression NK_COMMA expression NK_RP", + /* 435 */ "every_opt ::=", + /* 436 */ "every_opt ::= EVERY NK_LP duration_literal NK_RP", + /* 437 */ "query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt", + /* 438 */ "query_expression_body ::= query_primary", + /* 439 */ "query_expression_body ::= query_expression_body UNION ALL query_expression_body", + /* 440 */ "query_expression_body ::= query_expression_body UNION query_expression_body", + /* 441 */ "query_primary ::= query_specification", + /* 442 */ "query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP", + /* 443 */ "order_by_clause_opt ::=", + /* 444 */ "order_by_clause_opt ::= ORDER BY sort_specification_list", + /* 445 */ "slimit_clause_opt ::=", + /* 446 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER", + /* 447 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER", + /* 448 */ "slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER", + /* 449 */ "limit_clause_opt ::=", + /* 450 */ "limit_clause_opt ::= LIMIT NK_INTEGER", + /* 451 */ "limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER", + /* 452 */ "limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER", + /* 453 */ "subquery ::= NK_LP query_expression NK_RP", + /* 454 */ "search_condition ::= common_expression", + /* 455 */ "sort_specification_list ::= sort_specification", + /* 456 */ "sort_specification_list ::= sort_specification_list NK_COMMA sort_specification", + /* 457 */ "sort_specification ::= expression ordering_specification_opt null_ordering_opt", + /* 458 */ "ordering_specification_opt ::=", + /* 459 */ "ordering_specification_opt ::= ASC", + /* 460 */ "ordering_specification_opt ::= DESC", + /* 461 */ "null_ordering_opt ::=", + /* 462 */ "null_ordering_opt ::= NULLS FIRST", + /* 463 */ "null_ordering_opt ::= NULLS LAST", }; #endif /* NDEBUG */ @@ -2109,174 +2121,176 @@ static void yy_destructor( */ /********* Begin destructor definitions ***************************************/ /* Default NON-TERMINAL Destructor */ - case 240: /* cmd */ - case 243: /* literal */ - case 254: /* db_options */ - case 256: /* alter_db_options */ - case 261: /* retention */ - case 262: /* full_table_name */ - case 265: /* table_options */ - case 269: /* alter_table_clause */ - case 270: /* alter_table_options */ - case 273: /* signed_literal */ - case 274: /* create_subtable_clause */ - case 277: /* drop_table_clause */ - case 280: /* column_def */ - case 283: /* col_name */ - case 284: /* db_name_cond_opt */ - case 285: /* like_pattern_opt */ - case 286: /* table_name_cond */ - case 287: /* from_db_opt */ - case 288: /* func_name */ - case 291: /* index_options */ - case 293: /* duration_literal */ - case 294: /* sliding_opt */ - case 295: /* func */ - case 298: /* query_expression */ - case 301: /* explain_options */ - case 305: /* stream_options */ - case 306: /* into_opt */ - case 308: /* where_clause_opt */ - case 309: /* signed */ - case 310: /* literal_func */ - case 313: /* expression */ - case 314: /* pseudo_column */ - case 315: /* column_reference */ - case 316: /* function_expression */ - case 317: /* subquery */ - case 322: /* star_func_para */ - case 323: /* predicate */ - case 326: /* in_predicate_value */ - case 327: /* boolean_value_expression */ - case 328: /* boolean_primary */ - case 329: /* common_expression */ - case 330: /* from_clause */ - case 331: /* table_reference_list */ - case 332: /* table_reference */ - case 333: /* table_primary */ - case 334: /* joined_table */ - case 336: /* parenthesized_joined_table */ - case 338: /* search_condition */ - case 339: /* query_specification */ - case 343: /* twindow_clause_opt */ - case 345: /* having_clause_opt */ - case 347: /* select_item */ - case 348: /* fill_opt */ - case 351: /* query_expression_body */ - case 353: /* slimit_clause_opt */ - case 354: /* limit_clause_opt */ - case 355: /* query_primary */ - case 357: /* sort_specification */ + case 242: /* cmd */ + case 245: /* literal */ + case 255: /* db_options */ + case 257: /* alter_db_options */ + case 262: /* retention */ + case 263: /* full_table_name */ + case 266: /* table_options */ + case 270: /* alter_table_clause */ + case 271: /* alter_table_options */ + case 274: /* signed_literal */ + case 275: /* create_subtable_clause */ + case 278: /* drop_table_clause */ + case 281: /* column_def */ + case 285: /* duration_literal */ + case 286: /* rollup_func_name */ + case 288: /* col_name */ + case 289: /* db_name_cond_opt */ + case 290: /* like_pattern_opt */ + case 291: /* table_name_cond */ + case 292: /* from_db_opt */ + case 294: /* index_options */ + case 296: /* sliding_opt */ + case 297: /* func */ + case 300: /* query_expression */ + case 303: /* explain_options */ + case 307: /* stream_options */ + case 308: /* into_opt */ + case 310: /* where_clause_opt */ + case 311: /* signed */ + case 312: /* literal_func */ + case 315: /* expression */ + case 316: /* pseudo_column */ + case 317: /* column_reference */ + case 318: /* function_expression */ + case 319: /* subquery */ + case 324: /* star_func_para */ + case 325: /* predicate */ + case 328: /* in_predicate_value */ + case 329: /* boolean_value_expression */ + case 330: /* boolean_primary */ + case 331: /* common_expression */ + case 332: /* from_clause */ + case 333: /* table_reference_list */ + case 334: /* table_reference */ + case 335: /* table_primary */ + case 336: /* joined_table */ + case 338: /* parenthesized_joined_table */ + case 340: /* search_condition */ + case 341: /* query_specification */ + case 345: /* range_opt */ + case 346: /* every_opt */ + case 347: /* fill_opt */ + case 348: /* twindow_clause_opt */ + case 350: /* having_clause_opt */ + case 352: /* select_item */ + case 355: /* query_expression_body */ + case 357: /* slimit_clause_opt */ + case 358: /* limit_clause_opt */ + case 359: /* query_primary */ + case 361: /* sort_specification */ { - nodesDestroyNode((yypminor->yy632)); + nodesDestroyNode((yypminor->yy392)); } break; - case 241: /* account_options */ - case 242: /* alter_account_options */ - case 244: /* alter_account_option */ - case 303: /* bufsize_opt */ + case 243: /* account_options */ + case 244: /* alter_account_options */ + case 246: /* alter_account_option */ + case 305: /* bufsize_opt */ { } break; - case 245: /* user_name */ - case 247: /* priv_level */ - case 250: /* db_name */ - case 251: /* dnode_endpoint */ - case 252: /* dnode_host_name */ - case 271: /* column_name */ - case 279: /* table_name */ - case 289: /* function_name */ - case 290: /* index_name */ - case 297: /* topic_name */ - case 299: /* cgroup_name */ - case 304: /* stream_name */ - case 311: /* table_alias */ - case 312: /* column_alias */ - case 318: /* star_func */ - case 320: /* noarg_func */ - case 335: /* alias_opt */ + case 247: /* user_name */ + case 249: /* priv_level */ + case 252: /* db_name */ + case 253: /* dnode_endpoint */ + case 272: /* column_name */ + case 280: /* table_name */ + case 287: /* function_name */ + case 293: /* index_name */ + case 299: /* topic_name */ + case 301: /* cgroup_name */ + case 306: /* stream_name */ + case 313: /* table_alias */ + case 314: /* column_alias */ + case 320: /* star_func */ + case 322: /* noarg_func */ + case 337: /* alias_opt */ { } break; - case 246: /* privileges */ - case 248: /* priv_type_list */ - case 249: /* priv_type */ + case 248: /* privileges */ + case 250: /* priv_type_list */ + case 251: /* priv_type */ { } break; - case 253: /* not_exists_opt */ - case 255: /* exists_opt */ - case 300: /* analyze_opt */ - case 302: /* agg_func_opt */ - case 340: /* set_quantifier_opt */ + case 254: /* not_exists_opt */ + case 256: /* exists_opt */ + case 302: /* analyze_opt */ + case 304: /* agg_func_opt */ + case 342: /* set_quantifier_opt */ { } break; - case 257: /* integer_list */ - case 258: /* variable_list */ - case 259: /* retention_list */ - case 263: /* column_def_list */ - case 264: /* tags_def_opt */ - case 266: /* multi_create_clause */ - case 267: /* tags_def */ - case 268: /* multi_drop_clause */ - case 275: /* specific_tags_opt */ - case 276: /* literal_list */ - case 278: /* col_name_list */ - case 281: /* func_name_list */ - case 292: /* func_list */ - case 296: /* expression_list */ - case 307: /* dnode_list */ - case 319: /* star_func_para_list */ - case 321: /* other_para_list */ - case 341: /* select_list */ - case 342: /* partition_by_clause_opt */ - case 344: /* group_by_clause_opt */ - case 346: /* select_sublist */ - case 350: /* group_by_list */ - case 352: /* order_by_clause_opt */ - case 356: /* sort_specification_list */ + case 258: /* integer_list */ + case 259: /* variable_list */ + case 260: /* retention_list */ + case 264: /* column_def_list */ + case 265: /* tags_def_opt */ + case 267: /* multi_create_clause */ + case 268: /* tags_def */ + case 269: /* multi_drop_clause */ + case 276: /* specific_tags_opt */ + case 277: /* literal_list */ + case 279: /* col_name_list */ + case 282: /* duration_list */ + case 283: /* rollup_func_list */ + case 295: /* func_list */ + case 298: /* expression_list */ + case 309: /* dnode_list */ + case 321: /* star_func_para_list */ + case 323: /* other_para_list */ + case 343: /* select_list */ + case 344: /* partition_by_clause_opt */ + case 349: /* group_by_clause_opt */ + case 351: /* select_sublist */ + case 354: /* group_by_list */ + case 356: /* order_by_clause_opt */ + case 360: /* sort_specification_list */ { - nodesDestroyList((yypminor->yy424)); + nodesDestroyList((yypminor->yy600)); } break; - case 260: /* alter_db_option */ - case 282: /* alter_table_option */ + case 261: /* alter_db_option */ + case 284: /* alter_table_option */ { } break; - case 272: /* type_name */ + case 273: /* type_name */ { } break; - case 324: /* compare_op */ - case 325: /* in_op */ + case 326: /* compare_op */ + case 327: /* in_op */ { } break; - case 337: /* join_type */ + case 339: /* join_type */ { } break; - case 349: /* fill_mode */ + case 353: /* fill_mode */ { } break; - case 358: /* ordering_specification_opt */ + case 362: /* ordering_specification_opt */ { } break; - case 359: /* null_ordering_opt */ + case 363: /* null_ordering_opt */ { } @@ -2575,460 +2589,470 @@ static const struct { YYCODETYPE lhs; /* Symbol on the left-hand side of the rule */ signed char nrhs; /* Negative of the number of RHS symbols in the rule */ } yyRuleInfo[] = { - { 240, -6 }, /* (0) cmd ::= CREATE ACCOUNT NK_ID PASS NK_STRING account_options */ - { 240, -4 }, /* (1) cmd ::= ALTER ACCOUNT NK_ID alter_account_options */ - { 241, 0 }, /* (2) account_options ::= */ - { 241, -3 }, /* (3) account_options ::= account_options PPS literal */ - { 241, -3 }, /* (4) account_options ::= account_options TSERIES literal */ - { 241, -3 }, /* (5) account_options ::= account_options STORAGE literal */ - { 241, -3 }, /* (6) account_options ::= account_options STREAMS literal */ - { 241, -3 }, /* (7) account_options ::= account_options QTIME literal */ - { 241, -3 }, /* (8) account_options ::= account_options DBS literal */ - { 241, -3 }, /* (9) account_options ::= account_options USERS literal */ - { 241, -3 }, /* (10) account_options ::= account_options CONNS literal */ - { 241, -3 }, /* (11) account_options ::= account_options STATE literal */ - { 242, -1 }, /* (12) alter_account_options ::= alter_account_option */ - { 242, -2 }, /* (13) alter_account_options ::= alter_account_options alter_account_option */ - { 244, -2 }, /* (14) alter_account_option ::= PASS literal */ - { 244, -2 }, /* (15) alter_account_option ::= PPS literal */ - { 244, -2 }, /* (16) alter_account_option ::= TSERIES literal */ - { 244, -2 }, /* (17) alter_account_option ::= STORAGE literal */ - { 244, -2 }, /* (18) alter_account_option ::= STREAMS literal */ - { 244, -2 }, /* (19) alter_account_option ::= QTIME literal */ - { 244, -2 }, /* (20) alter_account_option ::= DBS literal */ - { 244, -2 }, /* (21) alter_account_option ::= USERS literal */ - { 244, -2 }, /* (22) alter_account_option ::= CONNS literal */ - { 244, -2 }, /* (23) alter_account_option ::= STATE literal */ - { 240, -5 }, /* (24) cmd ::= CREATE USER user_name PASS NK_STRING */ - { 240, -5 }, /* (25) cmd ::= ALTER USER user_name PASS NK_STRING */ - { 240, -5 }, /* (26) cmd ::= ALTER USER user_name PRIVILEGE NK_STRING */ - { 240, -3 }, /* (27) cmd ::= DROP USER user_name */ - { 240, -6 }, /* (28) cmd ::= GRANT privileges ON priv_level TO user_name */ - { 240, -6 }, /* (29) cmd ::= REVOKE privileges ON priv_level FROM user_name */ - { 246, -1 }, /* (30) privileges ::= ALL */ - { 246, -1 }, /* (31) privileges ::= priv_type_list */ - { 248, -1 }, /* (32) priv_type_list ::= priv_type */ - { 248, -3 }, /* (33) priv_type_list ::= priv_type_list NK_COMMA priv_type */ - { 249, -1 }, /* (34) priv_type ::= READ */ - { 249, -1 }, /* (35) priv_type ::= WRITE */ - { 247, -3 }, /* (36) priv_level ::= NK_STAR NK_DOT NK_STAR */ - { 247, -3 }, /* (37) priv_level ::= db_name NK_DOT NK_STAR */ - { 240, -3 }, /* (38) cmd ::= CREATE DNODE dnode_endpoint */ - { 240, -5 }, /* (39) cmd ::= CREATE DNODE dnode_host_name PORT NK_INTEGER */ - { 240, -3 }, /* (40) cmd ::= DROP DNODE NK_INTEGER */ - { 240, -3 }, /* (41) cmd ::= DROP DNODE dnode_endpoint */ - { 240, -4 }, /* (42) cmd ::= ALTER DNODE NK_INTEGER NK_STRING */ - { 240, -5 }, /* (43) cmd ::= ALTER DNODE NK_INTEGER NK_STRING NK_STRING */ - { 240, -4 }, /* (44) cmd ::= ALTER ALL DNODES NK_STRING */ - { 240, -5 }, /* (45) cmd ::= ALTER ALL DNODES NK_STRING NK_STRING */ - { 251, -1 }, /* (46) dnode_endpoint ::= NK_STRING */ - { 252, -1 }, /* (47) dnode_host_name ::= NK_ID */ - { 252, -1 }, /* (48) dnode_host_name ::= NK_IPTOKEN */ - { 240, -3 }, /* (49) cmd ::= ALTER LOCAL NK_STRING */ - { 240, -4 }, /* (50) cmd ::= ALTER LOCAL NK_STRING NK_STRING */ - { 240, -5 }, /* (51) cmd ::= CREATE QNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (52) cmd ::= DROP QNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (53) cmd ::= CREATE BNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (54) cmd ::= DROP BNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (55) cmd ::= CREATE SNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (56) cmd ::= DROP SNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (57) cmd ::= CREATE MNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (58) cmd ::= DROP MNODE ON DNODE NK_INTEGER */ - { 240, -5 }, /* (59) cmd ::= CREATE DATABASE not_exists_opt db_name db_options */ - { 240, -4 }, /* (60) cmd ::= DROP DATABASE exists_opt db_name */ - { 240, -2 }, /* (61) cmd ::= USE db_name */ - { 240, -4 }, /* (62) cmd ::= ALTER DATABASE db_name alter_db_options */ - { 253, -3 }, /* (63) not_exists_opt ::= IF NOT EXISTS */ - { 253, 0 }, /* (64) not_exists_opt ::= */ - { 255, -2 }, /* (65) exists_opt ::= IF EXISTS */ - { 255, 0 }, /* (66) exists_opt ::= */ - { 254, 0 }, /* (67) db_options ::= */ - { 254, -3 }, /* (68) db_options ::= db_options BUFFER NK_INTEGER */ - { 254, -3 }, /* (69) db_options ::= db_options CACHELAST NK_INTEGER */ - { 254, -3 }, /* (70) db_options ::= db_options COMP NK_INTEGER */ - { 254, -3 }, /* (71) db_options ::= db_options DURATION NK_INTEGER */ - { 254, -3 }, /* (72) db_options ::= db_options DURATION NK_VARIABLE */ - { 254, -3 }, /* (73) db_options ::= db_options FSYNC NK_INTEGER */ - { 254, -3 }, /* (74) db_options ::= db_options MAXROWS NK_INTEGER */ - { 254, -3 }, /* (75) db_options ::= db_options MINROWS NK_INTEGER */ - { 254, -3 }, /* (76) db_options ::= db_options KEEP integer_list */ - { 254, -3 }, /* (77) db_options ::= db_options KEEP variable_list */ - { 254, -3 }, /* (78) db_options ::= db_options PAGES NK_INTEGER */ - { 254, -3 }, /* (79) db_options ::= db_options PAGESIZE NK_INTEGER */ - { 254, -3 }, /* (80) db_options ::= db_options PRECISION NK_STRING */ - { 254, -3 }, /* (81) db_options ::= db_options REPLICA NK_INTEGER */ - { 254, -3 }, /* (82) db_options ::= db_options STRICT NK_INTEGER */ - { 254, -3 }, /* (83) db_options ::= db_options WAL NK_INTEGER */ - { 254, -3 }, /* (84) db_options ::= db_options VGROUPS NK_INTEGER */ - { 254, -3 }, /* (85) db_options ::= db_options SINGLE_STABLE NK_INTEGER */ - { 254, -3 }, /* (86) db_options ::= db_options RETENTIONS retention_list */ - { 254, -3 }, /* (87) db_options ::= db_options SCHEMALESS NK_INTEGER */ - { 256, -1 }, /* (88) alter_db_options ::= alter_db_option */ - { 256, -2 }, /* (89) alter_db_options ::= alter_db_options alter_db_option */ - { 260, -2 }, /* (90) alter_db_option ::= BUFFER NK_INTEGER */ - { 260, -2 }, /* (91) alter_db_option ::= CACHELAST NK_INTEGER */ - { 260, -2 }, /* (92) alter_db_option ::= FSYNC NK_INTEGER */ - { 260, -2 }, /* (93) alter_db_option ::= KEEP integer_list */ - { 260, -2 }, /* (94) alter_db_option ::= KEEP variable_list */ - { 260, -2 }, /* (95) alter_db_option ::= PAGES NK_INTEGER */ - { 260, -2 }, /* (96) alter_db_option ::= REPLICA NK_INTEGER */ - { 260, -2 }, /* (97) alter_db_option ::= STRICT NK_INTEGER */ - { 260, -2 }, /* (98) alter_db_option ::= WAL NK_INTEGER */ - { 257, -1 }, /* (99) integer_list ::= NK_INTEGER */ - { 257, -3 }, /* (100) integer_list ::= integer_list NK_COMMA NK_INTEGER */ - { 258, -1 }, /* (101) variable_list ::= NK_VARIABLE */ - { 258, -3 }, /* (102) variable_list ::= variable_list NK_COMMA NK_VARIABLE */ - { 259, -1 }, /* (103) retention_list ::= retention */ - { 259, -3 }, /* (104) retention_list ::= retention_list NK_COMMA retention */ - { 261, -3 }, /* (105) retention ::= NK_VARIABLE NK_COLON NK_VARIABLE */ - { 240, -9 }, /* (106) cmd ::= CREATE TABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def_opt table_options */ - { 240, -3 }, /* (107) cmd ::= CREATE TABLE multi_create_clause */ - { 240, -9 }, /* (108) cmd ::= CREATE STABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def table_options */ - { 240, -3 }, /* (109) cmd ::= DROP TABLE multi_drop_clause */ - { 240, -4 }, /* (110) cmd ::= DROP STABLE exists_opt full_table_name */ - { 240, -3 }, /* (111) cmd ::= ALTER TABLE alter_table_clause */ - { 240, -3 }, /* (112) cmd ::= ALTER STABLE alter_table_clause */ - { 269, -2 }, /* (113) alter_table_clause ::= full_table_name alter_table_options */ - { 269, -5 }, /* (114) alter_table_clause ::= full_table_name ADD COLUMN column_name type_name */ - { 269, -4 }, /* (115) alter_table_clause ::= full_table_name DROP COLUMN column_name */ - { 269, -5 }, /* (116) alter_table_clause ::= full_table_name MODIFY COLUMN column_name type_name */ - { 269, -5 }, /* (117) alter_table_clause ::= full_table_name RENAME COLUMN column_name column_name */ - { 269, -5 }, /* (118) alter_table_clause ::= full_table_name ADD TAG column_name type_name */ - { 269, -4 }, /* (119) alter_table_clause ::= full_table_name DROP TAG column_name */ - { 269, -5 }, /* (120) alter_table_clause ::= full_table_name MODIFY TAG column_name type_name */ - { 269, -5 }, /* (121) alter_table_clause ::= full_table_name RENAME TAG column_name column_name */ - { 269, -6 }, /* (122) alter_table_clause ::= full_table_name SET TAG column_name NK_EQ signed_literal */ - { 266, -1 }, /* (123) multi_create_clause ::= create_subtable_clause */ - { 266, -2 }, /* (124) multi_create_clause ::= multi_create_clause create_subtable_clause */ - { 274, -10 }, /* (125) create_subtable_clause ::= not_exists_opt full_table_name USING full_table_name specific_tags_opt TAGS NK_LP literal_list NK_RP table_options */ - { 268, -1 }, /* (126) multi_drop_clause ::= drop_table_clause */ - { 268, -2 }, /* (127) multi_drop_clause ::= multi_drop_clause drop_table_clause */ - { 277, -2 }, /* (128) drop_table_clause ::= exists_opt full_table_name */ - { 275, 0 }, /* (129) specific_tags_opt ::= */ - { 275, -3 }, /* (130) specific_tags_opt ::= NK_LP col_name_list NK_RP */ - { 262, -1 }, /* (131) full_table_name ::= table_name */ - { 262, -3 }, /* (132) full_table_name ::= db_name NK_DOT table_name */ - { 263, -1 }, /* (133) column_def_list ::= column_def */ - { 263, -3 }, /* (134) column_def_list ::= column_def_list NK_COMMA column_def */ - { 280, -2 }, /* (135) column_def ::= column_name type_name */ - { 280, -4 }, /* (136) column_def ::= column_name type_name COMMENT NK_STRING */ - { 272, -1 }, /* (137) type_name ::= BOOL */ - { 272, -1 }, /* (138) type_name ::= TINYINT */ - { 272, -1 }, /* (139) type_name ::= SMALLINT */ - { 272, -1 }, /* (140) type_name ::= INT */ - { 272, -1 }, /* (141) type_name ::= INTEGER */ - { 272, -1 }, /* (142) type_name ::= BIGINT */ - { 272, -1 }, /* (143) type_name ::= FLOAT */ - { 272, -1 }, /* (144) type_name ::= DOUBLE */ - { 272, -4 }, /* (145) type_name ::= BINARY NK_LP NK_INTEGER NK_RP */ - { 272, -1 }, /* (146) type_name ::= TIMESTAMP */ - { 272, -4 }, /* (147) type_name ::= NCHAR NK_LP NK_INTEGER NK_RP */ - { 272, -2 }, /* (148) type_name ::= TINYINT UNSIGNED */ - { 272, -2 }, /* (149) type_name ::= SMALLINT UNSIGNED */ - { 272, -2 }, /* (150) type_name ::= INT UNSIGNED */ - { 272, -2 }, /* (151) type_name ::= BIGINT UNSIGNED */ - { 272, -1 }, /* (152) type_name ::= JSON */ - { 272, -4 }, /* (153) type_name ::= VARCHAR NK_LP NK_INTEGER NK_RP */ - { 272, -1 }, /* (154) type_name ::= MEDIUMBLOB */ - { 272, -1 }, /* (155) type_name ::= BLOB */ - { 272, -4 }, /* (156) type_name ::= VARBINARY NK_LP NK_INTEGER NK_RP */ - { 272, -1 }, /* (157) type_name ::= DECIMAL */ - { 272, -4 }, /* (158) type_name ::= DECIMAL NK_LP NK_INTEGER NK_RP */ - { 272, -6 }, /* (159) type_name ::= DECIMAL NK_LP NK_INTEGER NK_COMMA NK_INTEGER NK_RP */ - { 264, 0 }, /* (160) tags_def_opt ::= */ - { 264, -1 }, /* (161) tags_def_opt ::= tags_def */ - { 267, -4 }, /* (162) tags_def ::= TAGS NK_LP column_def_list NK_RP */ - { 265, 0 }, /* (163) table_options ::= */ - { 265, -3 }, /* (164) table_options ::= table_options COMMENT NK_STRING */ - { 265, -3 }, /* (165) table_options ::= table_options FILE_FACTOR NK_FLOAT */ - { 265, -5 }, /* (166) table_options ::= table_options ROLLUP NK_LP func_name_list NK_RP */ - { 265, -3 }, /* (167) table_options ::= table_options TTL NK_INTEGER */ - { 265, -5 }, /* (168) table_options ::= table_options SMA NK_LP col_name_list NK_RP */ - { 270, -1 }, /* (169) alter_table_options ::= alter_table_option */ - { 270, -2 }, /* (170) alter_table_options ::= alter_table_options alter_table_option */ - { 282, -2 }, /* (171) alter_table_option ::= COMMENT NK_STRING */ - { 282, -2 }, /* (172) alter_table_option ::= TTL NK_INTEGER */ - { 278, -1 }, /* (173) col_name_list ::= col_name */ - { 278, -3 }, /* (174) col_name_list ::= col_name_list NK_COMMA col_name */ - { 283, -1 }, /* (175) col_name ::= column_name */ - { 240, -2 }, /* (176) cmd ::= SHOW DNODES */ - { 240, -2 }, /* (177) cmd ::= SHOW USERS */ - { 240, -2 }, /* (178) cmd ::= SHOW DATABASES */ - { 240, -4 }, /* (179) cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt */ - { 240, -4 }, /* (180) cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt */ - { 240, -3 }, /* (181) cmd ::= SHOW db_name_cond_opt VGROUPS */ - { 240, -2 }, /* (182) cmd ::= SHOW MNODES */ - { 240, -2 }, /* (183) cmd ::= SHOW MODULES */ - { 240, -2 }, /* (184) cmd ::= SHOW QNODES */ - { 240, -2 }, /* (185) cmd ::= SHOW FUNCTIONS */ - { 240, -5 }, /* (186) cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt */ - { 240, -2 }, /* (187) cmd ::= SHOW STREAMS */ - { 240, -2 }, /* (188) cmd ::= SHOW ACCOUNTS */ - { 240, -2 }, /* (189) cmd ::= SHOW APPS */ - { 240, -2 }, /* (190) cmd ::= SHOW CONNECTIONS */ - { 240, -2 }, /* (191) cmd ::= SHOW LICENCE */ - { 240, -2 }, /* (192) cmd ::= SHOW GRANTS */ - { 240, -4 }, /* (193) cmd ::= SHOW CREATE DATABASE db_name */ - { 240, -4 }, /* (194) cmd ::= SHOW CREATE TABLE full_table_name */ - { 240, -4 }, /* (195) cmd ::= SHOW CREATE STABLE full_table_name */ - { 240, -2 }, /* (196) cmd ::= SHOW QUERIES */ - { 240, -2 }, /* (197) cmd ::= SHOW SCORES */ - { 240, -2 }, /* (198) cmd ::= SHOW TOPICS */ - { 240, -2 }, /* (199) cmd ::= SHOW VARIABLES */ - { 240, -2 }, /* (200) cmd ::= SHOW BNODES */ - { 240, -2 }, /* (201) cmd ::= SHOW SNODES */ - { 240, -2 }, /* (202) cmd ::= SHOW CLUSTER */ - { 240, -2 }, /* (203) cmd ::= SHOW TRANSACTIONS */ - { 284, 0 }, /* (204) db_name_cond_opt ::= */ - { 284, -2 }, /* (205) db_name_cond_opt ::= db_name NK_DOT */ - { 285, 0 }, /* (206) like_pattern_opt ::= */ - { 285, -2 }, /* (207) like_pattern_opt ::= LIKE NK_STRING */ - { 286, -1 }, /* (208) table_name_cond ::= table_name */ - { 287, 0 }, /* (209) from_db_opt ::= */ - { 287, -2 }, /* (210) from_db_opt ::= FROM db_name */ - { 281, -1 }, /* (211) func_name_list ::= func_name */ - { 281, -3 }, /* (212) func_name_list ::= func_name_list NK_COMMA func_name */ - { 288, -1 }, /* (213) func_name ::= function_name */ - { 240, -8 }, /* (214) cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options */ - { 240, -10 }, /* (215) cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP */ - { 240, -6 }, /* (216) cmd ::= DROP INDEX exists_opt index_name ON table_name */ - { 291, 0 }, /* (217) index_options ::= */ - { 291, -9 }, /* (218) index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt */ - { 291, -11 }, /* (219) index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt */ - { 292, -1 }, /* (220) func_list ::= func */ - { 292, -3 }, /* (221) func_list ::= func_list NK_COMMA func */ - { 295, -4 }, /* (222) func ::= function_name NK_LP expression_list NK_RP */ - { 240, -6 }, /* (223) cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression */ - { 240, -7 }, /* (224) cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name */ - { 240, -7 }, /* (225) cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name */ - { 240, -4 }, /* (226) cmd ::= DROP TOPIC exists_opt topic_name */ - { 240, -7 }, /* (227) cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name */ - { 240, -2 }, /* (228) cmd ::= DESC full_table_name */ - { 240, -2 }, /* (229) cmd ::= DESCRIBE full_table_name */ - { 240, -3 }, /* (230) cmd ::= RESET QUERY CACHE */ - { 240, -4 }, /* (231) cmd ::= EXPLAIN analyze_opt explain_options query_expression */ - { 300, 0 }, /* (232) analyze_opt ::= */ - { 300, -1 }, /* (233) analyze_opt ::= ANALYZE */ - { 301, 0 }, /* (234) explain_options ::= */ - { 301, -3 }, /* (235) explain_options ::= explain_options VERBOSE NK_BOOL */ - { 301, -3 }, /* (236) explain_options ::= explain_options RATIO NK_FLOAT */ - { 240, -6 }, /* (237) cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP */ - { 240, -10 }, /* (238) cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt */ - { 240, -4 }, /* (239) cmd ::= DROP FUNCTION exists_opt function_name */ - { 302, 0 }, /* (240) agg_func_opt ::= */ - { 302, -1 }, /* (241) agg_func_opt ::= AGGREGATE */ - { 303, 0 }, /* (242) bufsize_opt ::= */ - { 303, -2 }, /* (243) bufsize_opt ::= BUFSIZE NK_INTEGER */ - { 240, -8 }, /* (244) cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression */ - { 240, -4 }, /* (245) cmd ::= DROP STREAM exists_opt stream_name */ - { 306, 0 }, /* (246) into_opt ::= */ - { 306, -2 }, /* (247) into_opt ::= INTO full_table_name */ - { 305, 0 }, /* (248) stream_options ::= */ - { 305, -3 }, /* (249) stream_options ::= stream_options TRIGGER AT_ONCE */ - { 305, -3 }, /* (250) stream_options ::= stream_options TRIGGER WINDOW_CLOSE */ - { 305, -4 }, /* (251) stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal */ - { 305, -3 }, /* (252) stream_options ::= stream_options WATERMARK duration_literal */ - { 240, -3 }, /* (253) cmd ::= KILL CONNECTION NK_INTEGER */ - { 240, -3 }, /* (254) cmd ::= KILL QUERY NK_STRING */ - { 240, -3 }, /* (255) cmd ::= KILL TRANSACTION NK_INTEGER */ - { 240, -2 }, /* (256) cmd ::= BALANCE VGROUP */ - { 240, -4 }, /* (257) cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER */ - { 240, -4 }, /* (258) cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list */ - { 240, -3 }, /* (259) cmd ::= SPLIT VGROUP NK_INTEGER */ - { 307, -2 }, /* (260) dnode_list ::= DNODE NK_INTEGER */ - { 307, -3 }, /* (261) dnode_list ::= dnode_list DNODE NK_INTEGER */ - { 240, -3 }, /* (262) cmd ::= SYNCDB db_name REPLICA */ - { 240, -4 }, /* (263) cmd ::= DELETE FROM full_table_name where_clause_opt */ - { 240, -1 }, /* (264) cmd ::= query_expression */ - { 243, -1 }, /* (265) literal ::= NK_INTEGER */ - { 243, -1 }, /* (266) literal ::= NK_FLOAT */ - { 243, -1 }, /* (267) literal ::= NK_STRING */ - { 243, -1 }, /* (268) literal ::= NK_BOOL */ - { 243, -2 }, /* (269) literal ::= TIMESTAMP NK_STRING */ - { 243, -1 }, /* (270) literal ::= duration_literal */ - { 243, -1 }, /* (271) literal ::= NULL */ - { 243, -1 }, /* (272) literal ::= NK_QUESTION */ - { 293, -1 }, /* (273) duration_literal ::= NK_VARIABLE */ - { 309, -1 }, /* (274) signed ::= NK_INTEGER */ - { 309, -2 }, /* (275) signed ::= NK_PLUS NK_INTEGER */ - { 309, -2 }, /* (276) signed ::= NK_MINUS NK_INTEGER */ - { 309, -1 }, /* (277) signed ::= NK_FLOAT */ - { 309, -2 }, /* (278) signed ::= NK_PLUS NK_FLOAT */ - { 309, -2 }, /* (279) signed ::= NK_MINUS NK_FLOAT */ - { 273, -1 }, /* (280) signed_literal ::= signed */ - { 273, -1 }, /* (281) signed_literal ::= NK_STRING */ - { 273, -1 }, /* (282) signed_literal ::= NK_BOOL */ - { 273, -2 }, /* (283) signed_literal ::= TIMESTAMP NK_STRING */ - { 273, -1 }, /* (284) signed_literal ::= duration_literal */ - { 273, -1 }, /* (285) signed_literal ::= NULL */ - { 273, -1 }, /* (286) signed_literal ::= literal_func */ - { 276, -1 }, /* (287) literal_list ::= signed_literal */ - { 276, -3 }, /* (288) literal_list ::= literal_list NK_COMMA signed_literal */ - { 250, -1 }, /* (289) db_name ::= NK_ID */ - { 279, -1 }, /* (290) table_name ::= NK_ID */ - { 271, -1 }, /* (291) column_name ::= NK_ID */ - { 289, -1 }, /* (292) function_name ::= NK_ID */ - { 311, -1 }, /* (293) table_alias ::= NK_ID */ - { 312, -1 }, /* (294) column_alias ::= NK_ID */ - { 245, -1 }, /* (295) user_name ::= NK_ID */ - { 290, -1 }, /* (296) index_name ::= NK_ID */ - { 297, -1 }, /* (297) topic_name ::= NK_ID */ - { 304, -1 }, /* (298) stream_name ::= NK_ID */ - { 299, -1 }, /* (299) cgroup_name ::= NK_ID */ - { 313, -1 }, /* (300) expression ::= literal */ - { 313, -1 }, /* (301) expression ::= pseudo_column */ - { 313, -1 }, /* (302) expression ::= column_reference */ - { 313, -1 }, /* (303) expression ::= function_expression */ - { 313, -1 }, /* (304) expression ::= subquery */ - { 313, -3 }, /* (305) expression ::= NK_LP expression NK_RP */ - { 313, -2 }, /* (306) expression ::= NK_PLUS expression */ - { 313, -2 }, /* (307) expression ::= NK_MINUS expression */ - { 313, -3 }, /* (308) expression ::= expression NK_PLUS expression */ - { 313, -3 }, /* (309) expression ::= expression NK_MINUS expression */ - { 313, -3 }, /* (310) expression ::= expression NK_STAR expression */ - { 313, -3 }, /* (311) expression ::= expression NK_SLASH expression */ - { 313, -3 }, /* (312) expression ::= expression NK_REM expression */ - { 313, -3 }, /* (313) expression ::= column_reference NK_ARROW NK_STRING */ - { 296, -1 }, /* (314) expression_list ::= expression */ - { 296, -3 }, /* (315) expression_list ::= expression_list NK_COMMA expression */ - { 315, -1 }, /* (316) column_reference ::= column_name */ - { 315, -3 }, /* (317) column_reference ::= table_name NK_DOT column_name */ - { 314, -1 }, /* (318) pseudo_column ::= ROWTS */ - { 314, -1 }, /* (319) pseudo_column ::= TBNAME */ - { 314, -3 }, /* (320) pseudo_column ::= table_name NK_DOT TBNAME */ - { 314, -1 }, /* (321) pseudo_column ::= QSTARTTS */ - { 314, -1 }, /* (322) pseudo_column ::= QENDTS */ - { 314, -1 }, /* (323) pseudo_column ::= WSTARTTS */ - { 314, -1 }, /* (324) pseudo_column ::= WENDTS */ - { 314, -1 }, /* (325) pseudo_column ::= WDURATION */ - { 316, -4 }, /* (326) function_expression ::= function_name NK_LP expression_list NK_RP */ - { 316, -4 }, /* (327) function_expression ::= star_func NK_LP star_func_para_list NK_RP */ - { 316, -6 }, /* (328) function_expression ::= CAST NK_LP expression AS type_name NK_RP */ - { 316, -1 }, /* (329) function_expression ::= literal_func */ - { 310, -3 }, /* (330) literal_func ::= noarg_func NK_LP NK_RP */ - { 310, -1 }, /* (331) literal_func ::= NOW */ - { 320, -1 }, /* (332) noarg_func ::= NOW */ - { 320, -1 }, /* (333) noarg_func ::= TODAY */ - { 320, -1 }, /* (334) noarg_func ::= TIMEZONE */ - { 318, -1 }, /* (335) star_func ::= COUNT */ - { 318, -1 }, /* (336) star_func ::= FIRST */ - { 318, -1 }, /* (337) star_func ::= LAST */ - { 318, -1 }, /* (338) star_func ::= LAST_ROW */ - { 319, -1 }, /* (339) star_func_para_list ::= NK_STAR */ - { 319, -1 }, /* (340) star_func_para_list ::= other_para_list */ - { 321, -1 }, /* (341) other_para_list ::= star_func_para */ - { 321, -3 }, /* (342) other_para_list ::= other_para_list NK_COMMA star_func_para */ - { 322, -1 }, /* (343) star_func_para ::= expression */ - { 322, -3 }, /* (344) star_func_para ::= table_name NK_DOT NK_STAR */ - { 323, -3 }, /* (345) predicate ::= expression compare_op expression */ - { 323, -5 }, /* (346) predicate ::= expression BETWEEN expression AND expression */ - { 323, -6 }, /* (347) predicate ::= expression NOT BETWEEN expression AND expression */ - { 323, -3 }, /* (348) predicate ::= expression IS NULL */ - { 323, -4 }, /* (349) predicate ::= expression IS NOT NULL */ - { 323, -3 }, /* (350) predicate ::= expression in_op in_predicate_value */ - { 324, -1 }, /* (351) compare_op ::= NK_LT */ - { 324, -1 }, /* (352) compare_op ::= NK_GT */ - { 324, -1 }, /* (353) compare_op ::= NK_LE */ - { 324, -1 }, /* (354) compare_op ::= NK_GE */ - { 324, -1 }, /* (355) compare_op ::= NK_NE */ - { 324, -1 }, /* (356) compare_op ::= NK_EQ */ - { 324, -1 }, /* (357) compare_op ::= LIKE */ - { 324, -2 }, /* (358) compare_op ::= NOT LIKE */ - { 324, -1 }, /* (359) compare_op ::= MATCH */ - { 324, -1 }, /* (360) compare_op ::= NMATCH */ - { 324, -1 }, /* (361) compare_op ::= CONTAINS */ - { 325, -1 }, /* (362) in_op ::= IN */ - { 325, -2 }, /* (363) in_op ::= NOT IN */ - { 326, -3 }, /* (364) in_predicate_value ::= NK_LP expression_list NK_RP */ - { 327, -1 }, /* (365) boolean_value_expression ::= boolean_primary */ - { 327, -2 }, /* (366) boolean_value_expression ::= NOT boolean_primary */ - { 327, -3 }, /* (367) boolean_value_expression ::= boolean_value_expression OR boolean_value_expression */ - { 327, -3 }, /* (368) boolean_value_expression ::= boolean_value_expression AND boolean_value_expression */ - { 328, -1 }, /* (369) boolean_primary ::= predicate */ - { 328, -3 }, /* (370) boolean_primary ::= NK_LP boolean_value_expression NK_RP */ - { 329, -1 }, /* (371) common_expression ::= expression */ - { 329, -1 }, /* (372) common_expression ::= boolean_value_expression */ - { 330, -2 }, /* (373) from_clause ::= FROM table_reference_list */ - { 331, -1 }, /* (374) table_reference_list ::= table_reference */ - { 331, -3 }, /* (375) table_reference_list ::= table_reference_list NK_COMMA table_reference */ - { 332, -1 }, /* (376) table_reference ::= table_primary */ - { 332, -1 }, /* (377) table_reference ::= joined_table */ - { 333, -2 }, /* (378) table_primary ::= table_name alias_opt */ - { 333, -4 }, /* (379) table_primary ::= db_name NK_DOT table_name alias_opt */ - { 333, -2 }, /* (380) table_primary ::= subquery alias_opt */ - { 333, -1 }, /* (381) table_primary ::= parenthesized_joined_table */ - { 335, 0 }, /* (382) alias_opt ::= */ - { 335, -1 }, /* (383) alias_opt ::= table_alias */ - { 335, -2 }, /* (384) alias_opt ::= AS table_alias */ - { 336, -3 }, /* (385) parenthesized_joined_table ::= NK_LP joined_table NK_RP */ - { 336, -3 }, /* (386) parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP */ - { 334, -6 }, /* (387) joined_table ::= table_reference join_type JOIN table_reference ON search_condition */ - { 337, 0 }, /* (388) join_type ::= */ - { 337, -1 }, /* (389) join_type ::= INNER */ - { 339, -9 }, /* (390) query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt twindow_clause_opt group_by_clause_opt having_clause_opt */ - { 340, 0 }, /* (391) set_quantifier_opt ::= */ - { 340, -1 }, /* (392) set_quantifier_opt ::= DISTINCT */ - { 340, -1 }, /* (393) set_quantifier_opt ::= ALL */ - { 341, -1 }, /* (394) select_list ::= NK_STAR */ - { 341, -1 }, /* (395) select_list ::= select_sublist */ - { 346, -1 }, /* (396) select_sublist ::= select_item */ - { 346, -3 }, /* (397) select_sublist ::= select_sublist NK_COMMA select_item */ - { 347, -1 }, /* (398) select_item ::= common_expression */ - { 347, -2 }, /* (399) select_item ::= common_expression column_alias */ - { 347, -3 }, /* (400) select_item ::= common_expression AS column_alias */ - { 347, -3 }, /* (401) select_item ::= table_name NK_DOT NK_STAR */ - { 308, 0 }, /* (402) where_clause_opt ::= */ - { 308, -2 }, /* (403) where_clause_opt ::= WHERE search_condition */ - { 342, 0 }, /* (404) partition_by_clause_opt ::= */ - { 342, -3 }, /* (405) partition_by_clause_opt ::= PARTITION BY expression_list */ - { 343, 0 }, /* (406) twindow_clause_opt ::= */ - { 343, -6 }, /* (407) twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP */ - { 343, -4 }, /* (408) twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP */ - { 343, -6 }, /* (409) twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt */ - { 343, -8 }, /* (410) twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt */ - { 294, 0 }, /* (411) sliding_opt ::= */ - { 294, -4 }, /* (412) sliding_opt ::= SLIDING NK_LP duration_literal NK_RP */ - { 348, 0 }, /* (413) fill_opt ::= */ - { 348, -4 }, /* (414) fill_opt ::= FILL NK_LP fill_mode NK_RP */ - { 348, -6 }, /* (415) fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP */ - { 349, -1 }, /* (416) fill_mode ::= NONE */ - { 349, -1 }, /* (417) fill_mode ::= PREV */ - { 349, -1 }, /* (418) fill_mode ::= NULL */ - { 349, -1 }, /* (419) fill_mode ::= LINEAR */ - { 349, -1 }, /* (420) fill_mode ::= NEXT */ - { 344, 0 }, /* (421) group_by_clause_opt ::= */ - { 344, -3 }, /* (422) group_by_clause_opt ::= GROUP BY group_by_list */ - { 350, -1 }, /* (423) group_by_list ::= expression */ - { 350, -3 }, /* (424) group_by_list ::= group_by_list NK_COMMA expression */ - { 345, 0 }, /* (425) having_clause_opt ::= */ - { 345, -2 }, /* (426) having_clause_opt ::= HAVING search_condition */ - { 298, -4 }, /* (427) query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt */ - { 351, -1 }, /* (428) query_expression_body ::= query_primary */ - { 351, -4 }, /* (429) query_expression_body ::= query_expression_body UNION ALL query_expression_body */ - { 351, -3 }, /* (430) query_expression_body ::= query_expression_body UNION query_expression_body */ - { 355, -1 }, /* (431) query_primary ::= query_specification */ - { 355, -6 }, /* (432) query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP */ - { 352, 0 }, /* (433) order_by_clause_opt ::= */ - { 352, -3 }, /* (434) order_by_clause_opt ::= ORDER BY sort_specification_list */ - { 353, 0 }, /* (435) slimit_clause_opt ::= */ - { 353, -2 }, /* (436) slimit_clause_opt ::= SLIMIT NK_INTEGER */ - { 353, -4 }, /* (437) slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER */ - { 353, -4 }, /* (438) slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER */ - { 354, 0 }, /* (439) limit_clause_opt ::= */ - { 354, -2 }, /* (440) limit_clause_opt ::= LIMIT NK_INTEGER */ - { 354, -4 }, /* (441) limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER */ - { 354, -4 }, /* (442) limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER */ - { 317, -3 }, /* (443) subquery ::= NK_LP query_expression NK_RP */ - { 338, -1 }, /* (444) search_condition ::= common_expression */ - { 356, -1 }, /* (445) sort_specification_list ::= sort_specification */ - { 356, -3 }, /* (446) sort_specification_list ::= sort_specification_list NK_COMMA sort_specification */ - { 357, -3 }, /* (447) sort_specification ::= expression ordering_specification_opt null_ordering_opt */ - { 358, 0 }, /* (448) ordering_specification_opt ::= */ - { 358, -1 }, /* (449) ordering_specification_opt ::= ASC */ - { 358, -1 }, /* (450) ordering_specification_opt ::= DESC */ - { 359, 0 }, /* (451) null_ordering_opt ::= */ - { 359, -2 }, /* (452) null_ordering_opt ::= NULLS FIRST */ - { 359, -2 }, /* (453) null_ordering_opt ::= NULLS LAST */ + { 242, -6 }, /* (0) cmd ::= CREATE ACCOUNT NK_ID PASS NK_STRING account_options */ + { 242, -4 }, /* (1) cmd ::= ALTER ACCOUNT NK_ID alter_account_options */ + { 243, 0 }, /* (2) account_options ::= */ + { 243, -3 }, /* (3) account_options ::= account_options PPS literal */ + { 243, -3 }, /* (4) account_options ::= account_options TSERIES literal */ + { 243, -3 }, /* (5) account_options ::= account_options STORAGE literal */ + { 243, -3 }, /* (6) account_options ::= account_options STREAMS literal */ + { 243, -3 }, /* (7) account_options ::= account_options QTIME literal */ + { 243, -3 }, /* (8) account_options ::= account_options DBS literal */ + { 243, -3 }, /* (9) account_options ::= account_options USERS literal */ + { 243, -3 }, /* (10) account_options ::= account_options CONNS literal */ + { 243, -3 }, /* (11) account_options ::= account_options STATE literal */ + { 244, -1 }, /* (12) alter_account_options ::= alter_account_option */ + { 244, -2 }, /* (13) alter_account_options ::= alter_account_options alter_account_option */ + { 246, -2 }, /* (14) alter_account_option ::= PASS literal */ + { 246, -2 }, /* (15) alter_account_option ::= PPS literal */ + { 246, -2 }, /* (16) alter_account_option ::= TSERIES literal */ + { 246, -2 }, /* (17) alter_account_option ::= STORAGE literal */ + { 246, -2 }, /* (18) alter_account_option ::= STREAMS literal */ + { 246, -2 }, /* (19) alter_account_option ::= QTIME literal */ + { 246, -2 }, /* (20) alter_account_option ::= DBS literal */ + { 246, -2 }, /* (21) alter_account_option ::= USERS literal */ + { 246, -2 }, /* (22) alter_account_option ::= CONNS literal */ + { 246, -2 }, /* (23) alter_account_option ::= STATE literal */ + { 242, -5 }, /* (24) cmd ::= CREATE USER user_name PASS NK_STRING */ + { 242, -5 }, /* (25) cmd ::= ALTER USER user_name PASS NK_STRING */ + { 242, -5 }, /* (26) cmd ::= ALTER USER user_name PRIVILEGE NK_STRING */ + { 242, -3 }, /* (27) cmd ::= DROP USER user_name */ + { 242, -6 }, /* (28) cmd ::= GRANT privileges ON priv_level TO user_name */ + { 242, -6 }, /* (29) cmd ::= REVOKE privileges ON priv_level FROM user_name */ + { 248, -1 }, /* (30) privileges ::= ALL */ + { 248, -1 }, /* (31) privileges ::= priv_type_list */ + { 250, -1 }, /* (32) priv_type_list ::= priv_type */ + { 250, -3 }, /* (33) priv_type_list ::= priv_type_list NK_COMMA priv_type */ + { 251, -1 }, /* (34) priv_type ::= READ */ + { 251, -1 }, /* (35) priv_type ::= WRITE */ + { 249, -3 }, /* (36) priv_level ::= NK_STAR NK_DOT NK_STAR */ + { 249, -3 }, /* (37) priv_level ::= db_name NK_DOT NK_STAR */ + { 242, -3 }, /* (38) cmd ::= CREATE DNODE dnode_endpoint */ + { 242, -5 }, /* (39) cmd ::= CREATE DNODE dnode_endpoint PORT NK_INTEGER */ + { 242, -3 }, /* (40) cmd ::= DROP DNODE NK_INTEGER */ + { 242, -3 }, /* (41) cmd ::= DROP DNODE dnode_endpoint */ + { 242, -4 }, /* (42) cmd ::= ALTER DNODE NK_INTEGER NK_STRING */ + { 242, -5 }, /* (43) cmd ::= ALTER DNODE NK_INTEGER NK_STRING NK_STRING */ + { 242, -4 }, /* (44) cmd ::= ALTER ALL DNODES NK_STRING */ + { 242, -5 }, /* (45) cmd ::= ALTER ALL DNODES NK_STRING NK_STRING */ + { 253, -1 }, /* (46) dnode_endpoint ::= NK_STRING */ + { 253, -1 }, /* (47) dnode_endpoint ::= NK_ID */ + { 253, -1 }, /* (48) dnode_endpoint ::= NK_IPTOKEN */ + { 242, -3 }, /* (49) cmd ::= ALTER LOCAL NK_STRING */ + { 242, -4 }, /* (50) cmd ::= ALTER LOCAL NK_STRING NK_STRING */ + { 242, -5 }, /* (51) cmd ::= CREATE QNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (52) cmd ::= DROP QNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (53) cmd ::= CREATE BNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (54) cmd ::= DROP BNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (55) cmd ::= CREATE SNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (56) cmd ::= DROP SNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (57) cmd ::= CREATE MNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (58) cmd ::= DROP MNODE ON DNODE NK_INTEGER */ + { 242, -5 }, /* (59) cmd ::= CREATE DATABASE not_exists_opt db_name db_options */ + { 242, -4 }, /* (60) cmd ::= DROP DATABASE exists_opt db_name */ + { 242, -2 }, /* (61) cmd ::= USE db_name */ + { 242, -4 }, /* (62) cmd ::= ALTER DATABASE db_name alter_db_options */ + { 254, -3 }, /* (63) not_exists_opt ::= IF NOT EXISTS */ + { 254, 0 }, /* (64) not_exists_opt ::= */ + { 256, -2 }, /* (65) exists_opt ::= IF EXISTS */ + { 256, 0 }, /* (66) exists_opt ::= */ + { 255, 0 }, /* (67) db_options ::= */ + { 255, -3 }, /* (68) db_options ::= db_options BUFFER NK_INTEGER */ + { 255, -3 }, /* (69) db_options ::= db_options CACHELAST NK_INTEGER */ + { 255, -3 }, /* (70) db_options ::= db_options COMP NK_INTEGER */ + { 255, -3 }, /* (71) db_options ::= db_options DURATION NK_INTEGER */ + { 255, -3 }, /* (72) db_options ::= db_options DURATION NK_VARIABLE */ + { 255, -3 }, /* (73) db_options ::= db_options FSYNC NK_INTEGER */ + { 255, -3 }, /* (74) db_options ::= db_options MAXROWS NK_INTEGER */ + { 255, -3 }, /* (75) db_options ::= db_options MINROWS NK_INTEGER */ + { 255, -3 }, /* (76) db_options ::= db_options KEEP integer_list */ + { 255, -3 }, /* (77) db_options ::= db_options KEEP variable_list */ + { 255, -3 }, /* (78) db_options ::= db_options PAGES NK_INTEGER */ + { 255, -3 }, /* (79) db_options ::= db_options PAGESIZE NK_INTEGER */ + { 255, -3 }, /* (80) db_options ::= db_options PRECISION NK_STRING */ + { 255, -3 }, /* (81) db_options ::= db_options REPLICA NK_INTEGER */ + { 255, -3 }, /* (82) db_options ::= db_options STRICT NK_INTEGER */ + { 255, -3 }, /* (83) db_options ::= db_options WAL NK_INTEGER */ + { 255, -3 }, /* (84) db_options ::= db_options VGROUPS NK_INTEGER */ + { 255, -3 }, /* (85) db_options ::= db_options SINGLE_STABLE NK_INTEGER */ + { 255, -3 }, /* (86) db_options ::= db_options RETENTIONS retention_list */ + { 255, -3 }, /* (87) db_options ::= db_options SCHEMALESS NK_INTEGER */ + { 257, -1 }, /* (88) alter_db_options ::= alter_db_option */ + { 257, -2 }, /* (89) alter_db_options ::= alter_db_options alter_db_option */ + { 261, -2 }, /* (90) alter_db_option ::= BUFFER NK_INTEGER */ + { 261, -2 }, /* (91) alter_db_option ::= CACHELAST NK_INTEGER */ + { 261, -2 }, /* (92) alter_db_option ::= FSYNC NK_INTEGER */ + { 261, -2 }, /* (93) alter_db_option ::= KEEP integer_list */ + { 261, -2 }, /* (94) alter_db_option ::= KEEP variable_list */ + { 261, -2 }, /* (95) alter_db_option ::= PAGES NK_INTEGER */ + { 261, -2 }, /* (96) alter_db_option ::= REPLICA NK_INTEGER */ + { 261, -2 }, /* (97) alter_db_option ::= STRICT NK_INTEGER */ + { 261, -2 }, /* (98) alter_db_option ::= WAL NK_INTEGER */ + { 258, -1 }, /* (99) integer_list ::= NK_INTEGER */ + { 258, -3 }, /* (100) integer_list ::= integer_list NK_COMMA NK_INTEGER */ + { 259, -1 }, /* (101) variable_list ::= NK_VARIABLE */ + { 259, -3 }, /* (102) variable_list ::= variable_list NK_COMMA NK_VARIABLE */ + { 260, -1 }, /* (103) retention_list ::= retention */ + { 260, -3 }, /* (104) retention_list ::= retention_list NK_COMMA retention */ + { 262, -3 }, /* (105) retention ::= NK_VARIABLE NK_COLON NK_VARIABLE */ + { 242, -9 }, /* (106) cmd ::= CREATE TABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def_opt table_options */ + { 242, -3 }, /* (107) cmd ::= CREATE TABLE multi_create_clause */ + { 242, -9 }, /* (108) cmd ::= CREATE STABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def table_options */ + { 242, -3 }, /* (109) cmd ::= DROP TABLE multi_drop_clause */ + { 242, -4 }, /* (110) cmd ::= DROP STABLE exists_opt full_table_name */ + { 242, -3 }, /* (111) cmd ::= ALTER TABLE alter_table_clause */ + { 242, -3 }, /* (112) cmd ::= ALTER STABLE alter_table_clause */ + { 270, -2 }, /* (113) alter_table_clause ::= full_table_name alter_table_options */ + { 270, -5 }, /* (114) alter_table_clause ::= full_table_name ADD COLUMN column_name type_name */ + { 270, -4 }, /* (115) alter_table_clause ::= full_table_name DROP COLUMN column_name */ + { 270, -5 }, /* (116) alter_table_clause ::= full_table_name MODIFY COLUMN column_name type_name */ + { 270, -5 }, /* (117) alter_table_clause ::= full_table_name RENAME COLUMN column_name column_name */ + { 270, -5 }, /* (118) alter_table_clause ::= full_table_name ADD TAG column_name type_name */ + { 270, -4 }, /* (119) alter_table_clause ::= full_table_name DROP TAG column_name */ + { 270, -5 }, /* (120) alter_table_clause ::= full_table_name MODIFY TAG column_name type_name */ + { 270, -5 }, /* (121) alter_table_clause ::= full_table_name RENAME TAG column_name column_name */ + { 270, -6 }, /* (122) alter_table_clause ::= full_table_name SET TAG column_name NK_EQ signed_literal */ + { 267, -1 }, /* (123) multi_create_clause ::= create_subtable_clause */ + { 267, -2 }, /* (124) multi_create_clause ::= multi_create_clause create_subtable_clause */ + { 275, -10 }, /* (125) create_subtable_clause ::= not_exists_opt full_table_name USING full_table_name specific_tags_opt TAGS NK_LP literal_list NK_RP table_options */ + { 269, -1 }, /* (126) multi_drop_clause ::= drop_table_clause */ + { 269, -2 }, /* (127) multi_drop_clause ::= multi_drop_clause drop_table_clause */ + { 278, -2 }, /* (128) drop_table_clause ::= exists_opt full_table_name */ + { 276, 0 }, /* (129) specific_tags_opt ::= */ + { 276, -3 }, /* (130) specific_tags_opt ::= NK_LP col_name_list NK_RP */ + { 263, -1 }, /* (131) full_table_name ::= table_name */ + { 263, -3 }, /* (132) full_table_name ::= db_name NK_DOT table_name */ + { 264, -1 }, /* (133) column_def_list ::= column_def */ + { 264, -3 }, /* (134) column_def_list ::= column_def_list NK_COMMA column_def */ + { 281, -2 }, /* (135) column_def ::= column_name type_name */ + { 281, -4 }, /* (136) column_def ::= column_name type_name COMMENT NK_STRING */ + { 273, -1 }, /* (137) type_name ::= BOOL */ + { 273, -1 }, /* (138) type_name ::= TINYINT */ + { 273, -1 }, /* (139) type_name ::= SMALLINT */ + { 273, -1 }, /* (140) type_name ::= INT */ + { 273, -1 }, /* (141) type_name ::= INTEGER */ + { 273, -1 }, /* (142) type_name ::= BIGINT */ + { 273, -1 }, /* (143) type_name ::= FLOAT */ + { 273, -1 }, /* (144) type_name ::= DOUBLE */ + { 273, -4 }, /* (145) type_name ::= BINARY NK_LP NK_INTEGER NK_RP */ + { 273, -1 }, /* (146) type_name ::= TIMESTAMP */ + { 273, -4 }, /* (147) type_name ::= NCHAR NK_LP NK_INTEGER NK_RP */ + { 273, -2 }, /* (148) type_name ::= TINYINT UNSIGNED */ + { 273, -2 }, /* (149) type_name ::= SMALLINT UNSIGNED */ + { 273, -2 }, /* (150) type_name ::= INT UNSIGNED */ + { 273, -2 }, /* (151) type_name ::= BIGINT UNSIGNED */ + { 273, -1 }, /* (152) type_name ::= JSON */ + { 273, -4 }, /* (153) type_name ::= VARCHAR NK_LP NK_INTEGER NK_RP */ + { 273, -1 }, /* (154) type_name ::= MEDIUMBLOB */ + { 273, -1 }, /* (155) type_name ::= BLOB */ + { 273, -4 }, /* (156) type_name ::= VARBINARY NK_LP NK_INTEGER NK_RP */ + { 273, -1 }, /* (157) type_name ::= DECIMAL */ + { 273, -4 }, /* (158) type_name ::= DECIMAL NK_LP NK_INTEGER NK_RP */ + { 273, -6 }, /* (159) type_name ::= DECIMAL NK_LP NK_INTEGER NK_COMMA NK_INTEGER NK_RP */ + { 265, 0 }, /* (160) tags_def_opt ::= */ + { 265, -1 }, /* (161) tags_def_opt ::= tags_def */ + { 268, -4 }, /* (162) tags_def ::= TAGS NK_LP column_def_list NK_RP */ + { 266, 0 }, /* (163) table_options ::= */ + { 266, -3 }, /* (164) table_options ::= table_options COMMENT NK_STRING */ + { 266, -3 }, /* (165) table_options ::= table_options MAX_DELAY duration_list */ + { 266, -3 }, /* (166) table_options ::= table_options WATERMARK duration_list */ + { 266, -5 }, /* (167) table_options ::= table_options ROLLUP NK_LP rollup_func_list NK_RP */ + { 266, -3 }, /* (168) table_options ::= table_options TTL NK_INTEGER */ + { 266, -5 }, /* (169) table_options ::= table_options SMA NK_LP col_name_list NK_RP */ + { 271, -1 }, /* (170) alter_table_options ::= alter_table_option */ + { 271, -2 }, /* (171) alter_table_options ::= alter_table_options alter_table_option */ + { 284, -2 }, /* (172) alter_table_option ::= COMMENT NK_STRING */ + { 284, -2 }, /* (173) alter_table_option ::= TTL NK_INTEGER */ + { 282, -1 }, /* (174) duration_list ::= duration_literal */ + { 282, -3 }, /* (175) duration_list ::= duration_list NK_COMMA duration_literal */ + { 283, -1 }, /* (176) rollup_func_list ::= rollup_func_name */ + { 283, -3 }, /* (177) rollup_func_list ::= rollup_func_list NK_COMMA rollup_func_name */ + { 286, -1 }, /* (178) rollup_func_name ::= function_name */ + { 286, -1 }, /* (179) rollup_func_name ::= FIRST */ + { 286, -1 }, /* (180) rollup_func_name ::= LAST */ + { 279, -1 }, /* (181) col_name_list ::= col_name */ + { 279, -3 }, /* (182) col_name_list ::= col_name_list NK_COMMA col_name */ + { 288, -1 }, /* (183) col_name ::= column_name */ + { 242, -2 }, /* (184) cmd ::= SHOW DNODES */ + { 242, -2 }, /* (185) cmd ::= SHOW USERS */ + { 242, -2 }, /* (186) cmd ::= SHOW DATABASES */ + { 242, -4 }, /* (187) cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt */ + { 242, -4 }, /* (188) cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt */ + { 242, -3 }, /* (189) cmd ::= SHOW db_name_cond_opt VGROUPS */ + { 242, -2 }, /* (190) cmd ::= SHOW MNODES */ + { 242, -2 }, /* (191) cmd ::= SHOW MODULES */ + { 242, -2 }, /* (192) cmd ::= SHOW QNODES */ + { 242, -2 }, /* (193) cmd ::= SHOW FUNCTIONS */ + { 242, -5 }, /* (194) cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt */ + { 242, -2 }, /* (195) cmd ::= SHOW STREAMS */ + { 242, -2 }, /* (196) cmd ::= SHOW ACCOUNTS */ + { 242, -2 }, /* (197) cmd ::= SHOW APPS */ + { 242, -2 }, /* (198) cmd ::= SHOW CONNECTIONS */ + { 242, -2 }, /* (199) cmd ::= SHOW LICENCE */ + { 242, -2 }, /* (200) cmd ::= SHOW GRANTS */ + { 242, -4 }, /* (201) cmd ::= SHOW CREATE DATABASE db_name */ + { 242, -4 }, /* (202) cmd ::= SHOW CREATE TABLE full_table_name */ + { 242, -4 }, /* (203) cmd ::= SHOW CREATE STABLE full_table_name */ + { 242, -2 }, /* (204) cmd ::= SHOW QUERIES */ + { 242, -2 }, /* (205) cmd ::= SHOW SCORES */ + { 242, -2 }, /* (206) cmd ::= SHOW TOPICS */ + { 242, -2 }, /* (207) cmd ::= SHOW VARIABLES */ + { 242, -2 }, /* (208) cmd ::= SHOW BNODES */ + { 242, -2 }, /* (209) cmd ::= SHOW SNODES */ + { 242, -2 }, /* (210) cmd ::= SHOW CLUSTER */ + { 242, -2 }, /* (211) cmd ::= SHOW TRANSACTIONS */ + { 242, -4 }, /* (212) cmd ::= SHOW TABLE DISTRIBUTED full_table_name */ + { 289, 0 }, /* (213) db_name_cond_opt ::= */ + { 289, -2 }, /* (214) db_name_cond_opt ::= db_name NK_DOT */ + { 290, 0 }, /* (215) like_pattern_opt ::= */ + { 290, -2 }, /* (216) like_pattern_opt ::= LIKE NK_STRING */ + { 291, -1 }, /* (217) table_name_cond ::= table_name */ + { 292, 0 }, /* (218) from_db_opt ::= */ + { 292, -2 }, /* (219) from_db_opt ::= FROM db_name */ + { 242, -8 }, /* (220) cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options */ + { 242, -10 }, /* (221) cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP */ + { 242, -6 }, /* (222) cmd ::= DROP INDEX exists_opt index_name ON table_name */ + { 294, 0 }, /* (223) index_options ::= */ + { 294, -9 }, /* (224) index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt */ + { 294, -11 }, /* (225) index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt */ + { 295, -1 }, /* (226) func_list ::= func */ + { 295, -3 }, /* (227) func_list ::= func_list NK_COMMA func */ + { 297, -4 }, /* (228) func ::= function_name NK_LP expression_list NK_RP */ + { 242, -6 }, /* (229) cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression */ + { 242, -7 }, /* (230) cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name */ + { 242, -7 }, /* (231) cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name */ + { 242, -4 }, /* (232) cmd ::= DROP TOPIC exists_opt topic_name */ + { 242, -7 }, /* (233) cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name */ + { 242, -2 }, /* (234) cmd ::= DESC full_table_name */ + { 242, -2 }, /* (235) cmd ::= DESCRIBE full_table_name */ + { 242, -3 }, /* (236) cmd ::= RESET QUERY CACHE */ + { 242, -4 }, /* (237) cmd ::= EXPLAIN analyze_opt explain_options query_expression */ + { 302, 0 }, /* (238) analyze_opt ::= */ + { 302, -1 }, /* (239) analyze_opt ::= ANALYZE */ + { 303, 0 }, /* (240) explain_options ::= */ + { 303, -3 }, /* (241) explain_options ::= explain_options VERBOSE NK_BOOL */ + { 303, -3 }, /* (242) explain_options ::= explain_options RATIO NK_FLOAT */ + { 242, -6 }, /* (243) cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP */ + { 242, -10 }, /* (244) cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt */ + { 242, -4 }, /* (245) cmd ::= DROP FUNCTION exists_opt function_name */ + { 304, 0 }, /* (246) agg_func_opt ::= */ + { 304, -1 }, /* (247) agg_func_opt ::= AGGREGATE */ + { 305, 0 }, /* (248) bufsize_opt ::= */ + { 305, -2 }, /* (249) bufsize_opt ::= BUFSIZE NK_INTEGER */ + { 242, -8 }, /* (250) cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression */ + { 242, -4 }, /* (251) cmd ::= DROP STREAM exists_opt stream_name */ + { 308, 0 }, /* (252) into_opt ::= */ + { 308, -2 }, /* (253) into_opt ::= INTO full_table_name */ + { 307, 0 }, /* (254) stream_options ::= */ + { 307, -3 }, /* (255) stream_options ::= stream_options TRIGGER AT_ONCE */ + { 307, -3 }, /* (256) stream_options ::= stream_options TRIGGER WINDOW_CLOSE */ + { 307, -4 }, /* (257) stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal */ + { 307, -3 }, /* (258) stream_options ::= stream_options WATERMARK duration_literal */ + { 242, -3 }, /* (259) cmd ::= KILL CONNECTION NK_INTEGER */ + { 242, -3 }, /* (260) cmd ::= KILL QUERY NK_STRING */ + { 242, -3 }, /* (261) cmd ::= KILL TRANSACTION NK_INTEGER */ + { 242, -2 }, /* (262) cmd ::= BALANCE VGROUP */ + { 242, -4 }, /* (263) cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER */ + { 242, -4 }, /* (264) cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list */ + { 242, -3 }, /* (265) cmd ::= SPLIT VGROUP NK_INTEGER */ + { 309, -2 }, /* (266) dnode_list ::= DNODE NK_INTEGER */ + { 309, -3 }, /* (267) dnode_list ::= dnode_list DNODE NK_INTEGER */ + { 242, -3 }, /* (268) cmd ::= SYNCDB db_name REPLICA */ + { 242, -4 }, /* (269) cmd ::= DELETE FROM full_table_name where_clause_opt */ + { 242, -1 }, /* (270) cmd ::= query_expression */ + { 245, -1 }, /* (271) literal ::= NK_INTEGER */ + { 245, -1 }, /* (272) literal ::= NK_FLOAT */ + { 245, -1 }, /* (273) literal ::= NK_STRING */ + { 245, -1 }, /* (274) literal ::= NK_BOOL */ + { 245, -2 }, /* (275) literal ::= TIMESTAMP NK_STRING */ + { 245, -1 }, /* (276) literal ::= duration_literal */ + { 245, -1 }, /* (277) literal ::= NULL */ + { 245, -1 }, /* (278) literal ::= NK_QUESTION */ + { 285, -1 }, /* (279) duration_literal ::= NK_VARIABLE */ + { 311, -1 }, /* (280) signed ::= NK_INTEGER */ + { 311, -2 }, /* (281) signed ::= NK_PLUS NK_INTEGER */ + { 311, -2 }, /* (282) signed ::= NK_MINUS NK_INTEGER */ + { 311, -1 }, /* (283) signed ::= NK_FLOAT */ + { 311, -2 }, /* (284) signed ::= NK_PLUS NK_FLOAT */ + { 311, -2 }, /* (285) signed ::= NK_MINUS NK_FLOAT */ + { 274, -1 }, /* (286) signed_literal ::= signed */ + { 274, -1 }, /* (287) signed_literal ::= NK_STRING */ + { 274, -1 }, /* (288) signed_literal ::= NK_BOOL */ + { 274, -2 }, /* (289) signed_literal ::= TIMESTAMP NK_STRING */ + { 274, -1 }, /* (290) signed_literal ::= duration_literal */ + { 274, -1 }, /* (291) signed_literal ::= NULL */ + { 274, -1 }, /* (292) signed_literal ::= literal_func */ + { 277, -1 }, /* (293) literal_list ::= signed_literal */ + { 277, -3 }, /* (294) literal_list ::= literal_list NK_COMMA signed_literal */ + { 252, -1 }, /* (295) db_name ::= NK_ID */ + { 280, -1 }, /* (296) table_name ::= NK_ID */ + { 272, -1 }, /* (297) column_name ::= NK_ID */ + { 287, -1 }, /* (298) function_name ::= NK_ID */ + { 313, -1 }, /* (299) table_alias ::= NK_ID */ + { 314, -1 }, /* (300) column_alias ::= NK_ID */ + { 247, -1 }, /* (301) user_name ::= NK_ID */ + { 293, -1 }, /* (302) index_name ::= NK_ID */ + { 299, -1 }, /* (303) topic_name ::= NK_ID */ + { 306, -1 }, /* (304) stream_name ::= NK_ID */ + { 301, -1 }, /* (305) cgroup_name ::= NK_ID */ + { 315, -1 }, /* (306) expression ::= literal */ + { 315, -1 }, /* (307) expression ::= pseudo_column */ + { 315, -1 }, /* (308) expression ::= column_reference */ + { 315, -1 }, /* (309) expression ::= function_expression */ + { 315, -1 }, /* (310) expression ::= subquery */ + { 315, -3 }, /* (311) expression ::= NK_LP expression NK_RP */ + { 315, -2 }, /* (312) expression ::= NK_PLUS expression */ + { 315, -2 }, /* (313) expression ::= NK_MINUS expression */ + { 315, -3 }, /* (314) expression ::= expression NK_PLUS expression */ + { 315, -3 }, /* (315) expression ::= expression NK_MINUS expression */ + { 315, -3 }, /* (316) expression ::= expression NK_STAR expression */ + { 315, -3 }, /* (317) expression ::= expression NK_SLASH expression */ + { 315, -3 }, /* (318) expression ::= expression NK_REM expression */ + { 315, -3 }, /* (319) expression ::= column_reference NK_ARROW NK_STRING */ + { 298, -1 }, /* (320) expression_list ::= expression */ + { 298, -3 }, /* (321) expression_list ::= expression_list NK_COMMA expression */ + { 317, -1 }, /* (322) column_reference ::= column_name */ + { 317, -3 }, /* (323) column_reference ::= table_name NK_DOT column_name */ + { 316, -1 }, /* (324) pseudo_column ::= ROWTS */ + { 316, -1 }, /* (325) pseudo_column ::= TBNAME */ + { 316, -3 }, /* (326) pseudo_column ::= table_name NK_DOT TBNAME */ + { 316, -1 }, /* (327) pseudo_column ::= QSTARTTS */ + { 316, -1 }, /* (328) pseudo_column ::= QENDTS */ + { 316, -1 }, /* (329) pseudo_column ::= WSTARTTS */ + { 316, -1 }, /* (330) pseudo_column ::= WENDTS */ + { 316, -1 }, /* (331) pseudo_column ::= WDURATION */ + { 318, -4 }, /* (332) function_expression ::= function_name NK_LP expression_list NK_RP */ + { 318, -4 }, /* (333) function_expression ::= star_func NK_LP star_func_para_list NK_RP */ + { 318, -6 }, /* (334) function_expression ::= CAST NK_LP expression AS type_name NK_RP */ + { 318, -1 }, /* (335) function_expression ::= literal_func */ + { 312, -3 }, /* (336) literal_func ::= noarg_func NK_LP NK_RP */ + { 312, -1 }, /* (337) literal_func ::= NOW */ + { 322, -1 }, /* (338) noarg_func ::= NOW */ + { 322, -1 }, /* (339) noarg_func ::= TODAY */ + { 322, -1 }, /* (340) noarg_func ::= TIMEZONE */ + { 320, -1 }, /* (341) star_func ::= COUNT */ + { 320, -1 }, /* (342) star_func ::= FIRST */ + { 320, -1 }, /* (343) star_func ::= LAST */ + { 320, -1 }, /* (344) star_func ::= LAST_ROW */ + { 321, -1 }, /* (345) star_func_para_list ::= NK_STAR */ + { 321, -1 }, /* (346) star_func_para_list ::= other_para_list */ + { 323, -1 }, /* (347) other_para_list ::= star_func_para */ + { 323, -3 }, /* (348) other_para_list ::= other_para_list NK_COMMA star_func_para */ + { 324, -1 }, /* (349) star_func_para ::= expression */ + { 324, -3 }, /* (350) star_func_para ::= table_name NK_DOT NK_STAR */ + { 325, -3 }, /* (351) predicate ::= expression compare_op expression */ + { 325, -5 }, /* (352) predicate ::= expression BETWEEN expression AND expression */ + { 325, -6 }, /* (353) predicate ::= expression NOT BETWEEN expression AND expression */ + { 325, -3 }, /* (354) predicate ::= expression IS NULL */ + { 325, -4 }, /* (355) predicate ::= expression IS NOT NULL */ + { 325, -3 }, /* (356) predicate ::= expression in_op in_predicate_value */ + { 326, -1 }, /* (357) compare_op ::= NK_LT */ + { 326, -1 }, /* (358) compare_op ::= NK_GT */ + { 326, -1 }, /* (359) compare_op ::= NK_LE */ + { 326, -1 }, /* (360) compare_op ::= NK_GE */ + { 326, -1 }, /* (361) compare_op ::= NK_NE */ + { 326, -1 }, /* (362) compare_op ::= NK_EQ */ + { 326, -1 }, /* (363) compare_op ::= LIKE */ + { 326, -2 }, /* (364) compare_op ::= NOT LIKE */ + { 326, -1 }, /* (365) compare_op ::= MATCH */ + { 326, -1 }, /* (366) compare_op ::= NMATCH */ + { 326, -1 }, /* (367) compare_op ::= CONTAINS */ + { 327, -1 }, /* (368) in_op ::= IN */ + { 327, -2 }, /* (369) in_op ::= NOT IN */ + { 328, -3 }, /* (370) in_predicate_value ::= NK_LP expression_list NK_RP */ + { 329, -1 }, /* (371) boolean_value_expression ::= boolean_primary */ + { 329, -2 }, /* (372) boolean_value_expression ::= NOT boolean_primary */ + { 329, -3 }, /* (373) boolean_value_expression ::= boolean_value_expression OR boolean_value_expression */ + { 329, -3 }, /* (374) boolean_value_expression ::= boolean_value_expression AND boolean_value_expression */ + { 330, -1 }, /* (375) boolean_primary ::= predicate */ + { 330, -3 }, /* (376) boolean_primary ::= NK_LP boolean_value_expression NK_RP */ + { 331, -1 }, /* (377) common_expression ::= expression */ + { 331, -1 }, /* (378) common_expression ::= boolean_value_expression */ + { 332, -2 }, /* (379) from_clause ::= FROM table_reference_list */ + { 333, -1 }, /* (380) table_reference_list ::= table_reference */ + { 333, -3 }, /* (381) table_reference_list ::= table_reference_list NK_COMMA table_reference */ + { 334, -1 }, /* (382) table_reference ::= table_primary */ + { 334, -1 }, /* (383) table_reference ::= joined_table */ + { 335, -2 }, /* (384) table_primary ::= table_name alias_opt */ + { 335, -4 }, /* (385) table_primary ::= db_name NK_DOT table_name alias_opt */ + { 335, -2 }, /* (386) table_primary ::= subquery alias_opt */ + { 335, -1 }, /* (387) table_primary ::= parenthesized_joined_table */ + { 337, 0 }, /* (388) alias_opt ::= */ + { 337, -1 }, /* (389) alias_opt ::= table_alias */ + { 337, -2 }, /* (390) alias_opt ::= AS table_alias */ + { 338, -3 }, /* (391) parenthesized_joined_table ::= NK_LP joined_table NK_RP */ + { 338, -3 }, /* (392) parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP */ + { 336, -6 }, /* (393) joined_table ::= table_reference join_type JOIN table_reference ON search_condition */ + { 339, 0 }, /* (394) join_type ::= */ + { 339, -1 }, /* (395) join_type ::= INNER */ + { 341, -12 }, /* (396) query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt range_opt every_opt fill_opt twindow_clause_opt group_by_clause_opt having_clause_opt */ + { 342, 0 }, /* (397) set_quantifier_opt ::= */ + { 342, -1 }, /* (398) set_quantifier_opt ::= DISTINCT */ + { 342, -1 }, /* (399) set_quantifier_opt ::= ALL */ + { 343, -1 }, /* (400) select_list ::= NK_STAR */ + { 343, -1 }, /* (401) select_list ::= select_sublist */ + { 351, -1 }, /* (402) select_sublist ::= select_item */ + { 351, -3 }, /* (403) select_sublist ::= select_sublist NK_COMMA select_item */ + { 352, -1 }, /* (404) select_item ::= common_expression */ + { 352, -2 }, /* (405) select_item ::= common_expression column_alias */ + { 352, -3 }, /* (406) select_item ::= common_expression AS column_alias */ + { 352, -3 }, /* (407) select_item ::= table_name NK_DOT NK_STAR */ + { 310, 0 }, /* (408) where_clause_opt ::= */ + { 310, -2 }, /* (409) where_clause_opt ::= WHERE search_condition */ + { 344, 0 }, /* (410) partition_by_clause_opt ::= */ + { 344, -3 }, /* (411) partition_by_clause_opt ::= PARTITION BY expression_list */ + { 348, 0 }, /* (412) twindow_clause_opt ::= */ + { 348, -6 }, /* (413) twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP */ + { 348, -4 }, /* (414) twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP */ + { 348, -6 }, /* (415) twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt */ + { 348, -8 }, /* (416) twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt */ + { 296, 0 }, /* (417) sliding_opt ::= */ + { 296, -4 }, /* (418) sliding_opt ::= SLIDING NK_LP duration_literal NK_RP */ + { 347, 0 }, /* (419) fill_opt ::= */ + { 347, -4 }, /* (420) fill_opt ::= FILL NK_LP fill_mode NK_RP */ + { 347, -6 }, /* (421) fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP */ + { 353, -1 }, /* (422) fill_mode ::= NONE */ + { 353, -1 }, /* (423) fill_mode ::= PREV */ + { 353, -1 }, /* (424) fill_mode ::= NULL */ + { 353, -1 }, /* (425) fill_mode ::= LINEAR */ + { 353, -1 }, /* (426) fill_mode ::= NEXT */ + { 349, 0 }, /* (427) group_by_clause_opt ::= */ + { 349, -3 }, /* (428) group_by_clause_opt ::= GROUP BY group_by_list */ + { 354, -1 }, /* (429) group_by_list ::= expression */ + { 354, -3 }, /* (430) group_by_list ::= group_by_list NK_COMMA expression */ + { 350, 0 }, /* (431) having_clause_opt ::= */ + { 350, -2 }, /* (432) having_clause_opt ::= HAVING search_condition */ + { 345, 0 }, /* (433) range_opt ::= */ + { 345, -6 }, /* (434) range_opt ::= RANGE NK_LP expression NK_COMMA expression NK_RP */ + { 346, 0 }, /* (435) every_opt ::= */ + { 346, -4 }, /* (436) every_opt ::= EVERY NK_LP duration_literal NK_RP */ + { 300, -4 }, /* (437) query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt */ + { 355, -1 }, /* (438) query_expression_body ::= query_primary */ + { 355, -4 }, /* (439) query_expression_body ::= query_expression_body UNION ALL query_expression_body */ + { 355, -3 }, /* (440) query_expression_body ::= query_expression_body UNION query_expression_body */ + { 359, -1 }, /* (441) query_primary ::= query_specification */ + { 359, -6 }, /* (442) query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP */ + { 356, 0 }, /* (443) order_by_clause_opt ::= */ + { 356, -3 }, /* (444) order_by_clause_opt ::= ORDER BY sort_specification_list */ + { 357, 0 }, /* (445) slimit_clause_opt ::= */ + { 357, -2 }, /* (446) slimit_clause_opt ::= SLIMIT NK_INTEGER */ + { 357, -4 }, /* (447) slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER */ + { 357, -4 }, /* (448) slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER */ + { 358, 0 }, /* (449) limit_clause_opt ::= */ + { 358, -2 }, /* (450) limit_clause_opt ::= LIMIT NK_INTEGER */ + { 358, -4 }, /* (451) limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER */ + { 358, -4 }, /* (452) limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER */ + { 319, -3 }, /* (453) subquery ::= NK_LP query_expression NK_RP */ + { 340, -1 }, /* (454) search_condition ::= common_expression */ + { 360, -1 }, /* (455) sort_specification_list ::= sort_specification */ + { 360, -3 }, /* (456) sort_specification_list ::= sort_specification_list NK_COMMA sort_specification */ + { 361, -3 }, /* (457) sort_specification ::= expression ordering_specification_opt null_ordering_opt */ + { 362, 0 }, /* (458) ordering_specification_opt ::= */ + { 362, -1 }, /* (459) ordering_specification_opt ::= ASC */ + { 362, -1 }, /* (460) ordering_specification_opt ::= DESC */ + { 363, 0 }, /* (461) null_ordering_opt ::= */ + { 363, -2 }, /* (462) null_ordering_opt ::= NULLS FIRST */ + { 363, -2 }, /* (463) null_ordering_opt ::= NULLS LAST */ }; static void yy_accept(yyParser*); /* Forward Declaration */ @@ -3117,11 +3141,11 @@ static YYACTIONTYPE yy_reduce( YYMINORTYPE yylhsminor; case 0: /* cmd ::= CREATE ACCOUNT NK_ID PASS NK_STRING account_options */ { pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_EXPRIE_STATEMENT); } - yy_destructor(yypParser,241,&yymsp[0].minor); + yy_destructor(yypParser,243,&yymsp[0].minor); break; case 1: /* cmd ::= ALTER ACCOUNT NK_ID alter_account_options */ { pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_EXPRIE_STATEMENT); } - yy_destructor(yypParser,242,&yymsp[0].minor); + yy_destructor(yypParser,244,&yymsp[0].minor); break; case 2: /* account_options ::= */ { } @@ -3135,20 +3159,20 @@ static YYACTIONTYPE yy_reduce( case 9: /* account_options ::= account_options USERS literal */ yytestcase(yyruleno==9); case 10: /* account_options ::= account_options CONNS literal */ yytestcase(yyruleno==10); case 11: /* account_options ::= account_options STATE literal */ yytestcase(yyruleno==11); -{ yy_destructor(yypParser,241,&yymsp[-2].minor); +{ yy_destructor(yypParser,243,&yymsp[-2].minor); { } - yy_destructor(yypParser,243,&yymsp[0].minor); + yy_destructor(yypParser,245,&yymsp[0].minor); } break; case 12: /* alter_account_options ::= alter_account_option */ -{ yy_destructor(yypParser,244,&yymsp[0].minor); +{ yy_destructor(yypParser,246,&yymsp[0].minor); { } } break; case 13: /* alter_account_options ::= alter_account_options alter_account_option */ -{ yy_destructor(yypParser,242,&yymsp[-1].minor); +{ yy_destructor(yypParser,244,&yymsp[-1].minor); { } - yy_destructor(yypParser,244,&yymsp[0].minor); + yy_destructor(yypParser,246,&yymsp[0].minor); } break; case 14: /* alter_account_option ::= PASS literal */ @@ -3162,63 +3186,63 @@ static YYACTIONTYPE yy_reduce( case 22: /* alter_account_option ::= CONNS literal */ yytestcase(yyruleno==22); case 23: /* alter_account_option ::= STATE literal */ yytestcase(yyruleno==23); { } - yy_destructor(yypParser,243,&yymsp[0].minor); + yy_destructor(yypParser,245,&yymsp[0].minor); break; case 24: /* cmd ::= CREATE USER user_name PASS NK_STRING */ -{ pCxt->pRootNode = createCreateUserStmt(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy0); } +{ pCxt->pRootNode = createCreateUserStmt(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy0); } break; case 25: /* cmd ::= ALTER USER user_name PASS NK_STRING */ -{ pCxt->pRootNode = createAlterUserStmt(pCxt, &yymsp[-2].minor.yy209, TSDB_ALTER_USER_PASSWD, &yymsp[0].minor.yy0); } +{ pCxt->pRootNode = createAlterUserStmt(pCxt, &yymsp[-2].minor.yy57, TSDB_ALTER_USER_PASSWD, &yymsp[0].minor.yy0); } break; case 26: /* cmd ::= ALTER USER user_name PRIVILEGE NK_STRING */ -{ pCxt->pRootNode = createAlterUserStmt(pCxt, &yymsp[-2].minor.yy209, TSDB_ALTER_USER_PRIVILEGES, &yymsp[0].minor.yy0); } +{ pCxt->pRootNode = createAlterUserStmt(pCxt, &yymsp[-2].minor.yy57, TSDB_ALTER_USER_PRIVILEGES, &yymsp[0].minor.yy0); } break; case 27: /* cmd ::= DROP USER user_name */ -{ pCxt->pRootNode = createDropUserStmt(pCxt, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createDropUserStmt(pCxt, &yymsp[0].minor.yy57); } break; case 28: /* cmd ::= GRANT privileges ON priv_level TO user_name */ -{ pCxt->pRootNode = createGrantStmt(pCxt, yymsp[-4].minor.yy189, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createGrantStmt(pCxt, yymsp[-4].minor.yy389, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57); } break; case 29: /* cmd ::= REVOKE privileges ON priv_level FROM user_name */ -{ pCxt->pRootNode = createRevokeStmt(pCxt, yymsp[-4].minor.yy189, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createRevokeStmt(pCxt, yymsp[-4].minor.yy389, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57); } break; case 30: /* privileges ::= ALL */ -{ yymsp[0].minor.yy189 = PRIVILEGE_TYPE_ALL; } +{ yymsp[0].minor.yy389 = PRIVILEGE_TYPE_ALL; } break; case 31: /* privileges ::= priv_type_list */ case 32: /* priv_type_list ::= priv_type */ yytestcase(yyruleno==32); -{ yylhsminor.yy189 = yymsp[0].minor.yy189; } - yymsp[0].minor.yy189 = yylhsminor.yy189; +{ yylhsminor.yy389 = yymsp[0].minor.yy389; } + yymsp[0].minor.yy389 = yylhsminor.yy389; break; case 33: /* priv_type_list ::= priv_type_list NK_COMMA priv_type */ -{ yylhsminor.yy189 = yymsp[-2].minor.yy189 | yymsp[0].minor.yy189; } - yymsp[-2].minor.yy189 = yylhsminor.yy189; +{ yylhsminor.yy389 = yymsp[-2].minor.yy389 | yymsp[0].minor.yy389; } + yymsp[-2].minor.yy389 = yylhsminor.yy389; break; case 34: /* priv_type ::= READ */ -{ yymsp[0].minor.yy189 = PRIVILEGE_TYPE_READ; } +{ yymsp[0].minor.yy389 = PRIVILEGE_TYPE_READ; } break; case 35: /* priv_type ::= WRITE */ -{ yymsp[0].minor.yy189 = PRIVILEGE_TYPE_WRITE; } +{ yymsp[0].minor.yy389 = PRIVILEGE_TYPE_WRITE; } break; case 36: /* priv_level ::= NK_STAR NK_DOT NK_STAR */ -{ yylhsminor.yy209 = yymsp[-2].minor.yy0; } - yymsp[-2].minor.yy209 = yylhsminor.yy209; +{ yylhsminor.yy57 = yymsp[-2].minor.yy0; } + yymsp[-2].minor.yy57 = yylhsminor.yy57; break; case 37: /* priv_level ::= db_name NK_DOT NK_STAR */ -{ yylhsminor.yy209 = yymsp[-2].minor.yy209; } - yymsp[-2].minor.yy209 = yylhsminor.yy209; +{ yylhsminor.yy57 = yymsp[-2].minor.yy57; } + yymsp[-2].minor.yy57 = yylhsminor.yy57; break; case 38: /* cmd ::= CREATE DNODE dnode_endpoint */ -{ pCxt->pRootNode = createCreateDnodeStmt(pCxt, &yymsp[0].minor.yy209, NULL); } +{ pCxt->pRootNode = createCreateDnodeStmt(pCxt, &yymsp[0].minor.yy57, NULL); } break; - case 39: /* cmd ::= CREATE DNODE dnode_host_name PORT NK_INTEGER */ -{ pCxt->pRootNode = createCreateDnodeStmt(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy0); } + case 39: /* cmd ::= CREATE DNODE dnode_endpoint PORT NK_INTEGER */ +{ pCxt->pRootNode = createCreateDnodeStmt(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy0); } break; case 40: /* cmd ::= DROP DNODE NK_INTEGER */ { pCxt->pRootNode = createDropDnodeStmt(pCxt, &yymsp[0].minor.yy0); } break; case 41: /* cmd ::= DROP DNODE dnode_endpoint */ -{ pCxt->pRootNode = createDropDnodeStmt(pCxt, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createDropDnodeStmt(pCxt, &yymsp[0].minor.yy57); } break; case 42: /* cmd ::= ALTER DNODE NK_INTEGER NK_STRING */ { pCxt->pRootNode = createAlterDnodeStmt(pCxt, &yymsp[-1].minor.yy0, &yymsp[0].minor.yy0, NULL); } @@ -3233,28 +3257,28 @@ static YYACTIONTYPE yy_reduce( { pCxt->pRootNode = createAlterDnodeStmt(pCxt, NULL, &yymsp[-1].minor.yy0, &yymsp[0].minor.yy0); } break; case 46: /* dnode_endpoint ::= NK_STRING */ - case 47: /* dnode_host_name ::= NK_ID */ yytestcase(yyruleno==47); - case 48: /* dnode_host_name ::= NK_IPTOKEN */ yytestcase(yyruleno==48); - case 289: /* db_name ::= NK_ID */ yytestcase(yyruleno==289); - case 290: /* table_name ::= NK_ID */ yytestcase(yyruleno==290); - case 291: /* column_name ::= NK_ID */ yytestcase(yyruleno==291); - case 292: /* function_name ::= NK_ID */ yytestcase(yyruleno==292); - case 293: /* table_alias ::= NK_ID */ yytestcase(yyruleno==293); - case 294: /* column_alias ::= NK_ID */ yytestcase(yyruleno==294); - case 295: /* user_name ::= NK_ID */ yytestcase(yyruleno==295); - case 296: /* index_name ::= NK_ID */ yytestcase(yyruleno==296); - case 297: /* topic_name ::= NK_ID */ yytestcase(yyruleno==297); - case 298: /* stream_name ::= NK_ID */ yytestcase(yyruleno==298); - case 299: /* cgroup_name ::= NK_ID */ yytestcase(yyruleno==299); - case 332: /* noarg_func ::= NOW */ yytestcase(yyruleno==332); - case 333: /* noarg_func ::= TODAY */ yytestcase(yyruleno==333); - case 334: /* noarg_func ::= TIMEZONE */ yytestcase(yyruleno==334); - case 335: /* star_func ::= COUNT */ yytestcase(yyruleno==335); - case 336: /* star_func ::= FIRST */ yytestcase(yyruleno==336); - case 337: /* star_func ::= LAST */ yytestcase(yyruleno==337); - case 338: /* star_func ::= LAST_ROW */ yytestcase(yyruleno==338); -{ yylhsminor.yy209 = yymsp[0].minor.yy0; } - yymsp[0].minor.yy209 = yylhsminor.yy209; + case 47: /* dnode_endpoint ::= NK_ID */ yytestcase(yyruleno==47); + case 48: /* dnode_endpoint ::= NK_IPTOKEN */ yytestcase(yyruleno==48); + case 295: /* db_name ::= NK_ID */ yytestcase(yyruleno==295); + case 296: /* table_name ::= NK_ID */ yytestcase(yyruleno==296); + case 297: /* column_name ::= NK_ID */ yytestcase(yyruleno==297); + case 298: /* function_name ::= NK_ID */ yytestcase(yyruleno==298); + case 299: /* table_alias ::= NK_ID */ yytestcase(yyruleno==299); + case 300: /* column_alias ::= NK_ID */ yytestcase(yyruleno==300); + case 301: /* user_name ::= NK_ID */ yytestcase(yyruleno==301); + case 302: /* index_name ::= NK_ID */ yytestcase(yyruleno==302); + case 303: /* topic_name ::= NK_ID */ yytestcase(yyruleno==303); + case 304: /* stream_name ::= NK_ID */ yytestcase(yyruleno==304); + case 305: /* cgroup_name ::= NK_ID */ yytestcase(yyruleno==305); + case 338: /* noarg_func ::= NOW */ yytestcase(yyruleno==338); + case 339: /* noarg_func ::= TODAY */ yytestcase(yyruleno==339); + case 340: /* noarg_func ::= TIMEZONE */ yytestcase(yyruleno==340); + case 341: /* star_func ::= COUNT */ yytestcase(yyruleno==341); + case 342: /* star_func ::= FIRST */ yytestcase(yyruleno==342); + case 343: /* star_func ::= LAST */ yytestcase(yyruleno==343); + case 344: /* star_func ::= LAST_ROW */ yytestcase(yyruleno==344); +{ yylhsminor.yy57 = yymsp[0].minor.yy0; } + yymsp[0].minor.yy57 = yylhsminor.yy57; break; case 49: /* cmd ::= ALTER LOCAL NK_STRING */ { pCxt->pRootNode = createAlterLocalStmt(pCxt, &yymsp[0].minor.yy0, NULL); } @@ -3287,1155 +3311,1178 @@ static YYACTIONTYPE yy_reduce( { pCxt->pRootNode = createDropComponentNodeStmt(pCxt, QUERY_NODE_DROP_MNODE_STMT, &yymsp[0].minor.yy0); } break; case 59: /* cmd ::= CREATE DATABASE not_exists_opt db_name db_options */ -{ pCxt->pRootNode = createCreateDatabaseStmt(pCxt, yymsp[-2].minor.yy137, &yymsp[-1].minor.yy209, yymsp[0].minor.yy632); } +{ pCxt->pRootNode = createCreateDatabaseStmt(pCxt, yymsp[-2].minor.yy481, &yymsp[-1].minor.yy57, yymsp[0].minor.yy392); } break; case 60: /* cmd ::= DROP DATABASE exists_opt db_name */ -{ pCxt->pRootNode = createDropDatabaseStmt(pCxt, yymsp[-1].minor.yy137, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createDropDatabaseStmt(pCxt, yymsp[-1].minor.yy481, &yymsp[0].minor.yy57); } break; case 61: /* cmd ::= USE db_name */ -{ pCxt->pRootNode = createUseDatabaseStmt(pCxt, &yymsp[0].minor.yy209); } +{ pCxt->pRootNode = createUseDatabaseStmt(pCxt, &yymsp[0].minor.yy57); } break; case 62: /* cmd ::= ALTER DATABASE db_name alter_db_options */ -{ pCxt->pRootNode = createAlterDatabaseStmt(pCxt, &yymsp[-1].minor.yy209, yymsp[0].minor.yy632); } +{ pCxt->pRootNode = createAlterDatabaseStmt(pCxt, &yymsp[-1].minor.yy57, yymsp[0].minor.yy392); } break; case 63: /* not_exists_opt ::= IF NOT EXISTS */ -{ yymsp[-2].minor.yy137 = true; } +{ yymsp[-2].minor.yy481 = true; } break; case 64: /* not_exists_opt ::= */ case 66: /* exists_opt ::= */ yytestcase(yyruleno==66); - case 232: /* analyze_opt ::= */ yytestcase(yyruleno==232); - case 240: /* agg_func_opt ::= */ yytestcase(yyruleno==240); - case 391: /* set_quantifier_opt ::= */ yytestcase(yyruleno==391); -{ yymsp[1].minor.yy137 = false; } + case 238: /* analyze_opt ::= */ yytestcase(yyruleno==238); + case 246: /* agg_func_opt ::= */ yytestcase(yyruleno==246); + case 397: /* set_quantifier_opt ::= */ yytestcase(yyruleno==397); +{ yymsp[1].minor.yy481 = false; } break; case 65: /* exists_opt ::= IF EXISTS */ -{ yymsp[-1].minor.yy137 = true; } +{ yymsp[-1].minor.yy481 = true; } break; case 67: /* db_options ::= */ -{ yymsp[1].minor.yy632 = createDefaultDatabaseOptions(pCxt); } +{ yymsp[1].minor.yy392 = createDefaultDatabaseOptions(pCxt); } break; case 68: /* db_options ::= db_options BUFFER NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_BUFFER, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_BUFFER, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 69: /* db_options ::= db_options CACHELAST NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_CACHELAST, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_CACHELAST, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 70: /* db_options ::= db_options COMP NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_COMP, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_COMP, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 71: /* db_options ::= db_options DURATION NK_INTEGER */ case 72: /* db_options ::= db_options DURATION NK_VARIABLE */ yytestcase(yyruleno==72); -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_DAYS, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_DAYS, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 73: /* db_options ::= db_options FSYNC NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_FSYNC, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_FSYNC, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 74: /* db_options ::= db_options MAXROWS NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_MAXROWS, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_MAXROWS, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 75: /* db_options ::= db_options MINROWS NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_MINROWS, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_MINROWS, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 76: /* db_options ::= db_options KEEP integer_list */ case 77: /* db_options ::= db_options KEEP variable_list */ yytestcase(yyruleno==77); -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_KEEP, yymsp[0].minor.yy424); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_KEEP, yymsp[0].minor.yy600); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 78: /* db_options ::= db_options PAGES NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_PAGES, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_PAGES, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 79: /* db_options ::= db_options PAGESIZE NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_PAGESIZE, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_PAGESIZE, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 80: /* db_options ::= db_options PRECISION NK_STRING */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_PRECISION, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_PRECISION, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 81: /* db_options ::= db_options REPLICA NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_REPLICA, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_REPLICA, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 82: /* db_options ::= db_options STRICT NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_STRICT, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_STRICT, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 83: /* db_options ::= db_options WAL NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_WAL, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_WAL, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 84: /* db_options ::= db_options VGROUPS NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_VGROUPS, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_VGROUPS, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 85: /* db_options ::= db_options SINGLE_STABLE NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_SINGLE_STABLE, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_SINGLE_STABLE, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 86: /* db_options ::= db_options RETENTIONS retention_list */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_RETENTIONS, yymsp[0].minor.yy424); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_RETENTIONS, yymsp[0].minor.yy600); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 87: /* db_options ::= db_options SCHEMALESS NK_INTEGER */ -{ yylhsminor.yy632 = setDatabaseOption(pCxt, yymsp[-2].minor.yy632, DB_OPTION_SCHEMALESS, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setDatabaseOption(pCxt, yymsp[-2].minor.yy392, DB_OPTION_SCHEMALESS, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 88: /* alter_db_options ::= alter_db_option */ -{ yylhsminor.yy632 = createAlterDatabaseOptions(pCxt); yylhsminor.yy632 = setAlterDatabaseOption(pCxt, yylhsminor.yy632, &yymsp[0].minor.yy605); } - yymsp[0].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterDatabaseOptions(pCxt); yylhsminor.yy392 = setAlterDatabaseOption(pCxt, yylhsminor.yy392, &yymsp[0].minor.yy221); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; case 89: /* alter_db_options ::= alter_db_options alter_db_option */ -{ yylhsminor.yy632 = setAlterDatabaseOption(pCxt, yymsp[-1].minor.yy632, &yymsp[0].minor.yy605); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setAlterDatabaseOption(pCxt, yymsp[-1].minor.yy392, &yymsp[0].minor.yy221); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; case 90: /* alter_db_option ::= BUFFER NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_BUFFER; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_BUFFER; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 91: /* alter_db_option ::= CACHELAST NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_CACHELAST; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_CACHELAST; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 92: /* alter_db_option ::= FSYNC NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_FSYNC; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_FSYNC; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 93: /* alter_db_option ::= KEEP integer_list */ case 94: /* alter_db_option ::= KEEP variable_list */ yytestcase(yyruleno==94); -{ yymsp[-1].minor.yy605.type = DB_OPTION_KEEP; yymsp[-1].minor.yy605.pList = yymsp[0].minor.yy424; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_KEEP; yymsp[-1].minor.yy221.pList = yymsp[0].minor.yy600; } break; case 95: /* alter_db_option ::= PAGES NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_PAGES; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_PAGES; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 96: /* alter_db_option ::= REPLICA NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_REPLICA; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_REPLICA; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 97: /* alter_db_option ::= STRICT NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_STRICT; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_STRICT; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 98: /* alter_db_option ::= WAL NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = DB_OPTION_WAL; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } +{ yymsp[-1].minor.yy221.type = DB_OPTION_WAL; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; case 99: /* integer_list ::= NK_INTEGER */ -{ yylhsminor.yy424 = createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy424 = yylhsminor.yy424; +{ yylhsminor.yy600 = createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; case 100: /* integer_list ::= integer_list NK_COMMA NK_INTEGER */ - case 261: /* dnode_list ::= dnode_list DNODE NK_INTEGER */ yytestcase(yyruleno==261); -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-2].minor.yy424, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } - yymsp[-2].minor.yy424 = yylhsminor.yy424; + case 267: /* dnode_list ::= dnode_list DNODE NK_INTEGER */ yytestcase(yyruleno==267); +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-2].minor.yy600, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } + yymsp[-2].minor.yy600 = yylhsminor.yy600; break; case 101: /* variable_list ::= NK_VARIABLE */ -{ yylhsminor.yy424 = createNodeList(pCxt, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy424 = yylhsminor.yy424; +{ yylhsminor.yy600 = createNodeList(pCxt, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; case 102: /* variable_list ::= variable_list NK_COMMA NK_VARIABLE */ -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-2].minor.yy424, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } - yymsp[-2].minor.yy424 = yylhsminor.yy424; +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-2].minor.yy600, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } + yymsp[-2].minor.yy600 = yylhsminor.yy600; break; case 103: /* retention_list ::= retention */ case 123: /* multi_create_clause ::= create_subtable_clause */ yytestcase(yyruleno==123); case 126: /* multi_drop_clause ::= drop_table_clause */ yytestcase(yyruleno==126); case 133: /* column_def_list ::= column_def */ yytestcase(yyruleno==133); - case 173: /* col_name_list ::= col_name */ yytestcase(yyruleno==173); - case 211: /* func_name_list ::= func_name */ yytestcase(yyruleno==211); - case 220: /* func_list ::= func */ yytestcase(yyruleno==220); - case 287: /* literal_list ::= signed_literal */ yytestcase(yyruleno==287); - case 341: /* other_para_list ::= star_func_para */ yytestcase(yyruleno==341); - case 396: /* select_sublist ::= select_item */ yytestcase(yyruleno==396); - case 445: /* sort_specification_list ::= sort_specification */ yytestcase(yyruleno==445); -{ yylhsminor.yy424 = createNodeList(pCxt, yymsp[0].minor.yy632); } - yymsp[0].minor.yy424 = yylhsminor.yy424; + case 176: /* rollup_func_list ::= rollup_func_name */ yytestcase(yyruleno==176); + case 181: /* col_name_list ::= col_name */ yytestcase(yyruleno==181); + case 226: /* func_list ::= func */ yytestcase(yyruleno==226); + case 293: /* literal_list ::= signed_literal */ yytestcase(yyruleno==293); + case 347: /* other_para_list ::= star_func_para */ yytestcase(yyruleno==347); + case 402: /* select_sublist ::= select_item */ yytestcase(yyruleno==402); + case 455: /* sort_specification_list ::= sort_specification */ yytestcase(yyruleno==455); +{ yylhsminor.yy600 = createNodeList(pCxt, yymsp[0].minor.yy392); } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; case 104: /* retention_list ::= retention_list NK_COMMA retention */ case 134: /* column_def_list ::= column_def_list NK_COMMA column_def */ yytestcase(yyruleno==134); - case 174: /* col_name_list ::= col_name_list NK_COMMA col_name */ yytestcase(yyruleno==174); - case 212: /* func_name_list ::= func_name_list NK_COMMA func_name */ yytestcase(yyruleno==212); - case 221: /* func_list ::= func_list NK_COMMA func */ yytestcase(yyruleno==221); - case 288: /* literal_list ::= literal_list NK_COMMA signed_literal */ yytestcase(yyruleno==288); - case 342: /* other_para_list ::= other_para_list NK_COMMA star_func_para */ yytestcase(yyruleno==342); - case 397: /* select_sublist ::= select_sublist NK_COMMA select_item */ yytestcase(yyruleno==397); - case 446: /* sort_specification_list ::= sort_specification_list NK_COMMA sort_specification */ yytestcase(yyruleno==446); -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-2].minor.yy424, yymsp[0].minor.yy632); } - yymsp[-2].minor.yy424 = yylhsminor.yy424; + case 177: /* rollup_func_list ::= rollup_func_list NK_COMMA rollup_func_name */ yytestcase(yyruleno==177); + case 182: /* col_name_list ::= col_name_list NK_COMMA col_name */ yytestcase(yyruleno==182); + case 227: /* func_list ::= func_list NK_COMMA func */ yytestcase(yyruleno==227); + case 294: /* literal_list ::= literal_list NK_COMMA signed_literal */ yytestcase(yyruleno==294); + case 348: /* other_para_list ::= other_para_list NK_COMMA star_func_para */ yytestcase(yyruleno==348); + case 403: /* select_sublist ::= select_sublist NK_COMMA select_item */ yytestcase(yyruleno==403); + case 456: /* sort_specification_list ::= sort_specification_list NK_COMMA sort_specification */ yytestcase(yyruleno==456); +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-2].minor.yy600, yymsp[0].minor.yy392); } + yymsp[-2].minor.yy600 = yylhsminor.yy600; break; case 105: /* retention ::= NK_VARIABLE NK_COLON NK_VARIABLE */ -{ yylhsminor.yy632 = createNodeListNodeEx(pCxt, createDurationValueNode(pCxt, &yymsp[-2].minor.yy0), createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createNodeListNodeEx(pCxt, createDurationValueNode(pCxt, &yymsp[-2].minor.yy0), createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 106: /* cmd ::= CREATE TABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def_opt table_options */ case 108: /* cmd ::= CREATE STABLE not_exists_opt full_table_name NK_LP column_def_list NK_RP tags_def table_options */ yytestcase(yyruleno==108); -{ pCxt->pRootNode = createCreateTableStmt(pCxt, yymsp[-6].minor.yy137, yymsp[-5].minor.yy632, yymsp[-3].minor.yy424, yymsp[-1].minor.yy424, yymsp[0].minor.yy632); } +{ pCxt->pRootNode = createCreateTableStmt(pCxt, yymsp[-6].minor.yy481, yymsp[-5].minor.yy392, yymsp[-3].minor.yy600, yymsp[-1].minor.yy600, yymsp[0].minor.yy392); } break; case 107: /* cmd ::= CREATE TABLE multi_create_clause */ -{ pCxt->pRootNode = createCreateMultiTableStmt(pCxt, yymsp[0].minor.yy424); } +{ pCxt->pRootNode = createCreateMultiTableStmt(pCxt, yymsp[0].minor.yy600); } break; case 109: /* cmd ::= DROP TABLE multi_drop_clause */ -{ pCxt->pRootNode = createDropTableStmt(pCxt, yymsp[0].minor.yy424); } +{ pCxt->pRootNode = createDropTableStmt(pCxt, yymsp[0].minor.yy600); } break; case 110: /* cmd ::= DROP STABLE exists_opt full_table_name */ -{ pCxt->pRootNode = createDropSuperTableStmt(pCxt, yymsp[-1].minor.yy137, yymsp[0].minor.yy632); } +{ pCxt->pRootNode = createDropSuperTableStmt(pCxt, yymsp[-1].minor.yy481, yymsp[0].minor.yy392); } break; case 111: /* cmd ::= ALTER TABLE alter_table_clause */ case 112: /* cmd ::= ALTER STABLE alter_table_clause */ yytestcase(yyruleno==112); - case 264: /* cmd ::= query_expression */ yytestcase(yyruleno==264); -{ pCxt->pRootNode = yymsp[0].minor.yy632; } + case 270: /* cmd ::= query_expression */ yytestcase(yyruleno==270); +{ pCxt->pRootNode = yymsp[0].minor.yy392; } break; case 113: /* alter_table_clause ::= full_table_name alter_table_options */ -{ yylhsminor.yy632 = createAlterTableModifyOptions(pCxt, yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableModifyOptions(pCxt, yymsp[-1].minor.yy392, yymsp[0].minor.yy392); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; case 114: /* alter_table_clause ::= full_table_name ADD COLUMN column_name type_name */ -{ yylhsminor.yy632 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_ADD_COLUMN, &yymsp[-1].minor.yy209, yymsp[0].minor.yy304); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_ADD_COLUMN, &yymsp[-1].minor.yy57, yymsp[0].minor.yy448); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 115: /* alter_table_clause ::= full_table_name DROP COLUMN column_name */ -{ yylhsminor.yy632 = createAlterTableDropCol(pCxt, yymsp[-3].minor.yy632, TSDB_ALTER_TABLE_DROP_COLUMN, &yymsp[0].minor.yy209); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableDropCol(pCxt, yymsp[-3].minor.yy392, TSDB_ALTER_TABLE_DROP_COLUMN, &yymsp[0].minor.yy57); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; case 116: /* alter_table_clause ::= full_table_name MODIFY COLUMN column_name type_name */ -{ yylhsminor.yy632 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES, &yymsp[-1].minor.yy209, yymsp[0].minor.yy304); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES, &yymsp[-1].minor.yy57, yymsp[0].minor.yy448); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 117: /* alter_table_clause ::= full_table_name RENAME COLUMN column_name column_name */ -{ yylhsminor.yy632 = createAlterTableRenameCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_UPDATE_COLUMN_NAME, &yymsp[-1].minor.yy209, &yymsp[0].minor.yy209); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableRenameCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_UPDATE_COLUMN_NAME, &yymsp[-1].minor.yy57, &yymsp[0].minor.yy57); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 118: /* alter_table_clause ::= full_table_name ADD TAG column_name type_name */ -{ yylhsminor.yy632 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_ADD_TAG, &yymsp[-1].minor.yy209, yymsp[0].minor.yy304); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_ADD_TAG, &yymsp[-1].minor.yy57, yymsp[0].minor.yy448); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 119: /* alter_table_clause ::= full_table_name DROP TAG column_name */ -{ yylhsminor.yy632 = createAlterTableDropCol(pCxt, yymsp[-3].minor.yy632, TSDB_ALTER_TABLE_DROP_TAG, &yymsp[0].minor.yy209); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableDropCol(pCxt, yymsp[-3].minor.yy392, TSDB_ALTER_TABLE_DROP_TAG, &yymsp[0].minor.yy57); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; case 120: /* alter_table_clause ::= full_table_name MODIFY TAG column_name type_name */ -{ yylhsminor.yy632 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_UPDATE_TAG_BYTES, &yymsp[-1].minor.yy209, yymsp[0].minor.yy304); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableAddModifyCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_UPDATE_TAG_BYTES, &yymsp[-1].minor.yy57, yymsp[0].minor.yy448); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 121: /* alter_table_clause ::= full_table_name RENAME TAG column_name column_name */ -{ yylhsminor.yy632 = createAlterTableRenameCol(pCxt, yymsp[-4].minor.yy632, TSDB_ALTER_TABLE_UPDATE_TAG_NAME, &yymsp[-1].minor.yy209, &yymsp[0].minor.yy209); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableRenameCol(pCxt, yymsp[-4].minor.yy392, TSDB_ALTER_TABLE_UPDATE_TAG_NAME, &yymsp[-1].minor.yy57, &yymsp[0].minor.yy57); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; case 122: /* alter_table_clause ::= full_table_name SET TAG column_name NK_EQ signed_literal */ -{ yylhsminor.yy632 = createAlterTableSetTag(pCxt, yymsp[-5].minor.yy632, &yymsp[-2].minor.yy209, yymsp[0].minor.yy632); } - yymsp[-5].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createAlterTableSetTag(pCxt, yymsp[-5].minor.yy392, &yymsp[-2].minor.yy57, yymsp[0].minor.yy392); } + yymsp[-5].minor.yy392 = yylhsminor.yy392; break; case 124: /* multi_create_clause ::= multi_create_clause create_subtable_clause */ case 127: /* multi_drop_clause ::= multi_drop_clause drop_table_clause */ yytestcase(yyruleno==127); -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-1].minor.yy424, yymsp[0].minor.yy632); } - yymsp[-1].minor.yy424 = yylhsminor.yy424; +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-1].minor.yy600, yymsp[0].minor.yy392); } + yymsp[-1].minor.yy600 = yylhsminor.yy600; break; case 125: /* create_subtable_clause ::= not_exists_opt full_table_name USING full_table_name specific_tags_opt TAGS NK_LP literal_list NK_RP table_options */ -{ yylhsminor.yy632 = createCreateSubTableClause(pCxt, yymsp[-9].minor.yy137, yymsp[-8].minor.yy632, yymsp[-6].minor.yy632, yymsp[-5].minor.yy424, yymsp[-2].minor.yy424, yymsp[0].minor.yy632); } - yymsp[-9].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createCreateSubTableClause(pCxt, yymsp[-9].minor.yy481, yymsp[-8].minor.yy392, yymsp[-6].minor.yy392, yymsp[-5].minor.yy600, yymsp[-2].minor.yy600, yymsp[0].minor.yy392); } + yymsp[-9].minor.yy392 = yylhsminor.yy392; break; case 128: /* drop_table_clause ::= exists_opt full_table_name */ -{ yylhsminor.yy632 = createDropTableClause(pCxt, yymsp[-1].minor.yy137, yymsp[0].minor.yy632); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createDropTableClause(pCxt, yymsp[-1].minor.yy481, yymsp[0].minor.yy392); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; case 129: /* specific_tags_opt ::= */ case 160: /* tags_def_opt ::= */ yytestcase(yyruleno==160); - case 404: /* partition_by_clause_opt ::= */ yytestcase(yyruleno==404); - case 421: /* group_by_clause_opt ::= */ yytestcase(yyruleno==421); - case 433: /* order_by_clause_opt ::= */ yytestcase(yyruleno==433); -{ yymsp[1].minor.yy424 = NULL; } + case 410: /* partition_by_clause_opt ::= */ yytestcase(yyruleno==410); + case 427: /* group_by_clause_opt ::= */ yytestcase(yyruleno==427); + case 443: /* order_by_clause_opt ::= */ yytestcase(yyruleno==443); +{ yymsp[1].minor.yy600 = NULL; } break; case 130: /* specific_tags_opt ::= NK_LP col_name_list NK_RP */ -{ yymsp[-2].minor.yy424 = yymsp[-1].minor.yy424; } +{ yymsp[-2].minor.yy600 = yymsp[-1].minor.yy600; } break; case 131: /* full_table_name ::= table_name */ -{ yylhsminor.yy632 = createRealTableNode(pCxt, NULL, &yymsp[0].minor.yy209, NULL); } - yymsp[0].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createRealTableNode(pCxt, NULL, &yymsp[0].minor.yy57, NULL); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; case 132: /* full_table_name ::= db_name NK_DOT table_name */ -{ yylhsminor.yy632 = createRealTableNode(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209, NULL); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createRealTableNode(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57, NULL); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; case 135: /* column_def ::= column_name type_name */ -{ yylhsminor.yy632 = createColumnDefNode(pCxt, &yymsp[-1].minor.yy209, yymsp[0].minor.yy304, NULL); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createColumnDefNode(pCxt, &yymsp[-1].minor.yy57, yymsp[0].minor.yy448, NULL); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; case 136: /* column_def ::= column_name type_name COMMENT NK_STRING */ -{ yylhsminor.yy632 = createColumnDefNode(pCxt, &yymsp[-3].minor.yy209, yymsp[-2].minor.yy304, &yymsp[0].minor.yy0); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = createColumnDefNode(pCxt, &yymsp[-3].minor.yy57, yymsp[-2].minor.yy448, &yymsp[0].minor.yy0); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; case 137: /* type_name ::= BOOL */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_BOOL); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_BOOL); } break; case 138: /* type_name ::= TINYINT */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_TINYINT); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_TINYINT); } break; case 139: /* type_name ::= SMALLINT */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_SMALLINT); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_SMALLINT); } break; case 140: /* type_name ::= INT */ case 141: /* type_name ::= INTEGER */ yytestcase(yyruleno==141); -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_INT); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_INT); } break; case 142: /* type_name ::= BIGINT */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_BIGINT); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_BIGINT); } break; case 143: /* type_name ::= FLOAT */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_FLOAT); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_FLOAT); } break; case 144: /* type_name ::= DOUBLE */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_DOUBLE); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_DOUBLE); } break; case 145: /* type_name ::= BINARY NK_LP NK_INTEGER NK_RP */ -{ yymsp[-3].minor.yy304 = createVarLenDataType(TSDB_DATA_TYPE_BINARY, &yymsp[-1].minor.yy0); } +{ yymsp[-3].minor.yy448 = createVarLenDataType(TSDB_DATA_TYPE_BINARY, &yymsp[-1].minor.yy0); } break; case 146: /* type_name ::= TIMESTAMP */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_TIMESTAMP); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_TIMESTAMP); } break; case 147: /* type_name ::= NCHAR NK_LP NK_INTEGER NK_RP */ -{ yymsp[-3].minor.yy304 = createVarLenDataType(TSDB_DATA_TYPE_NCHAR, &yymsp[-1].minor.yy0); } +{ yymsp[-3].minor.yy448 = createVarLenDataType(TSDB_DATA_TYPE_NCHAR, &yymsp[-1].minor.yy0); } break; case 148: /* type_name ::= TINYINT UNSIGNED */ -{ yymsp[-1].minor.yy304 = createDataType(TSDB_DATA_TYPE_UTINYINT); } +{ yymsp[-1].minor.yy448 = createDataType(TSDB_DATA_TYPE_UTINYINT); } break; case 149: /* type_name ::= SMALLINT UNSIGNED */ -{ yymsp[-1].minor.yy304 = createDataType(TSDB_DATA_TYPE_USMALLINT); } +{ yymsp[-1].minor.yy448 = createDataType(TSDB_DATA_TYPE_USMALLINT); } break; case 150: /* type_name ::= INT UNSIGNED */ -{ yymsp[-1].minor.yy304 = createDataType(TSDB_DATA_TYPE_UINT); } +{ yymsp[-1].minor.yy448 = createDataType(TSDB_DATA_TYPE_UINT); } break; case 151: /* type_name ::= BIGINT UNSIGNED */ -{ yymsp[-1].minor.yy304 = createDataType(TSDB_DATA_TYPE_UBIGINT); } +{ yymsp[-1].minor.yy448 = createDataType(TSDB_DATA_TYPE_UBIGINT); } break; case 152: /* type_name ::= JSON */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_JSON); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_JSON); } break; case 153: /* type_name ::= VARCHAR NK_LP NK_INTEGER NK_RP */ -{ yymsp[-3].minor.yy304 = createVarLenDataType(TSDB_DATA_TYPE_VARCHAR, &yymsp[-1].minor.yy0); } +{ yymsp[-3].minor.yy448 = createVarLenDataType(TSDB_DATA_TYPE_VARCHAR, &yymsp[-1].minor.yy0); } break; case 154: /* type_name ::= MEDIUMBLOB */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_MEDIUMBLOB); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_MEDIUMBLOB); } break; case 155: /* type_name ::= BLOB */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_BLOB); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_BLOB); } break; case 156: /* type_name ::= VARBINARY NK_LP NK_INTEGER NK_RP */ -{ yymsp[-3].minor.yy304 = createVarLenDataType(TSDB_DATA_TYPE_VARBINARY, &yymsp[-1].minor.yy0); } +{ yymsp[-3].minor.yy448 = createVarLenDataType(TSDB_DATA_TYPE_VARBINARY, &yymsp[-1].minor.yy0); } break; case 157: /* type_name ::= DECIMAL */ -{ yymsp[0].minor.yy304 = createDataType(TSDB_DATA_TYPE_DECIMAL); } +{ yymsp[0].minor.yy448 = createDataType(TSDB_DATA_TYPE_DECIMAL); } break; case 158: /* type_name ::= DECIMAL NK_LP NK_INTEGER NK_RP */ -{ yymsp[-3].minor.yy304 = createDataType(TSDB_DATA_TYPE_DECIMAL); } +{ yymsp[-3].minor.yy448 = createDataType(TSDB_DATA_TYPE_DECIMAL); } break; case 159: /* type_name ::= DECIMAL NK_LP NK_INTEGER NK_COMMA NK_INTEGER NK_RP */ -{ yymsp[-5].minor.yy304 = createDataType(TSDB_DATA_TYPE_DECIMAL); } +{ yymsp[-5].minor.yy448 = createDataType(TSDB_DATA_TYPE_DECIMAL); } break; case 161: /* tags_def_opt ::= tags_def */ - case 340: /* star_func_para_list ::= other_para_list */ yytestcase(yyruleno==340); - case 395: /* select_list ::= select_sublist */ yytestcase(yyruleno==395); -{ yylhsminor.yy424 = yymsp[0].minor.yy424; } - yymsp[0].minor.yy424 = yylhsminor.yy424; + case 346: /* star_func_para_list ::= other_para_list */ yytestcase(yyruleno==346); + case 401: /* select_list ::= select_sublist */ yytestcase(yyruleno==401); +{ yylhsminor.yy600 = yymsp[0].minor.yy600; } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; case 162: /* tags_def ::= TAGS NK_LP column_def_list NK_RP */ -{ yymsp[-3].minor.yy424 = yymsp[-1].minor.yy424; } +{ yymsp[-3].minor.yy600 = yymsp[-1].minor.yy600; } break; case 163: /* table_options ::= */ -{ yymsp[1].minor.yy632 = createDefaultTableOptions(pCxt); } +{ yymsp[1].minor.yy392 = createDefaultTableOptions(pCxt); } break; case 164: /* table_options ::= table_options COMMENT NK_STRING */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-2].minor.yy632, TABLE_OPTION_COMMENT, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-2].minor.yy392, TABLE_OPTION_COMMENT, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 165: /* table_options ::= table_options MAX_DELAY duration_list */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-2].minor.yy392, TABLE_OPTION_MAXDELAY, yymsp[0].minor.yy600); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 165: /* table_options ::= table_options FILE_FACTOR NK_FLOAT */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-2].minor.yy632, TABLE_OPTION_FILE_FACTOR, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 166: /* table_options ::= table_options WATERMARK duration_list */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-2].minor.yy392, TABLE_OPTION_WATERMARK, yymsp[0].minor.yy600); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 166: /* table_options ::= table_options ROLLUP NK_LP func_name_list NK_RP */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-4].minor.yy632, TABLE_OPTION_ROLLUP, yymsp[-1].minor.yy424); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; + case 167: /* table_options ::= table_options ROLLUP NK_LP rollup_func_list NK_RP */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-4].minor.yy392, TABLE_OPTION_ROLLUP, yymsp[-1].minor.yy600); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; - case 167: /* table_options ::= table_options TTL NK_INTEGER */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-2].minor.yy632, TABLE_OPTION_TTL, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 168: /* table_options ::= table_options TTL NK_INTEGER */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-2].minor.yy392, TABLE_OPTION_TTL, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 168: /* table_options ::= table_options SMA NK_LP col_name_list NK_RP */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-4].minor.yy632, TABLE_OPTION_SMA, yymsp[-1].minor.yy424); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; + case 169: /* table_options ::= table_options SMA NK_LP col_name_list NK_RP */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-4].minor.yy392, TABLE_OPTION_SMA, yymsp[-1].minor.yy600); } + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; - case 169: /* alter_table_options ::= alter_table_option */ -{ yylhsminor.yy632 = createAlterTableOptions(pCxt); yylhsminor.yy632 = setTableOption(pCxt, yylhsminor.yy632, yymsp[0].minor.yy605.type, &yymsp[0].minor.yy605.val); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 170: /* alter_table_options ::= alter_table_option */ +{ yylhsminor.yy392 = createAlterTableOptions(pCxt); yylhsminor.yy392 = setTableOption(pCxt, yylhsminor.yy392, yymsp[0].minor.yy221.type, &yymsp[0].minor.yy221.val); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 170: /* alter_table_options ::= alter_table_options alter_table_option */ -{ yylhsminor.yy632 = setTableOption(pCxt, yymsp[-1].minor.yy632, yymsp[0].minor.yy605.type, &yymsp[0].minor.yy605.val); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + case 171: /* alter_table_options ::= alter_table_options alter_table_option */ +{ yylhsminor.yy392 = setTableOption(pCxt, yymsp[-1].minor.yy392, yymsp[0].minor.yy221.type, &yymsp[0].minor.yy221.val); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 171: /* alter_table_option ::= COMMENT NK_STRING */ -{ yymsp[-1].minor.yy605.type = TABLE_OPTION_COMMENT; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } + case 172: /* alter_table_option ::= COMMENT NK_STRING */ +{ yymsp[-1].minor.yy221.type = TABLE_OPTION_COMMENT; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; - case 172: /* alter_table_option ::= TTL NK_INTEGER */ -{ yymsp[-1].minor.yy605.type = TABLE_OPTION_TTL; yymsp[-1].minor.yy605.val = yymsp[0].minor.yy0; } + case 173: /* alter_table_option ::= TTL NK_INTEGER */ +{ yymsp[-1].minor.yy221.type = TABLE_OPTION_TTL; yymsp[-1].minor.yy221.val = yymsp[0].minor.yy0; } break; - case 175: /* col_name ::= column_name */ -{ yylhsminor.yy632 = createColumnNode(pCxt, NULL, &yymsp[0].minor.yy209); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 174: /* duration_list ::= duration_literal */ + case 320: /* expression_list ::= expression */ yytestcase(yyruleno==320); +{ yylhsminor.yy600 = createNodeList(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy392)); } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; - case 176: /* cmd ::= SHOW DNODES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DNODES_STMT, NULL, NULL); } + case 175: /* duration_list ::= duration_list NK_COMMA duration_literal */ + case 321: /* expression_list ::= expression_list NK_COMMA expression */ yytestcase(yyruleno==321); +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-2].minor.yy600, releaseRawExprNode(pCxt, yymsp[0].minor.yy392)); } + yymsp[-2].minor.yy600 = yylhsminor.yy600; break; - case 177: /* cmd ::= SHOW USERS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USERS_STMT, NULL, NULL); } + case 178: /* rollup_func_name ::= function_name */ +{ yylhsminor.yy392 = createFunctionNode(pCxt, &yymsp[0].minor.yy57, NULL); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 178: /* cmd ::= SHOW DATABASES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT, NULL, NULL); } + case 179: /* rollup_func_name ::= FIRST */ + case 180: /* rollup_func_name ::= LAST */ yytestcase(yyruleno==180); +{ yylhsminor.yy392 = createFunctionNode(pCxt, &yymsp[0].minor.yy0, NULL); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 179: /* cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TABLES_STMT, yymsp[-2].minor.yy632, yymsp[0].minor.yy632); } + case 183: /* col_name ::= column_name */ +{ yylhsminor.yy392 = createColumnNode(pCxt, NULL, &yymsp[0].minor.yy57); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 180: /* cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STABLES_STMT, yymsp[-2].minor.yy632, yymsp[0].minor.yy632); } + case 184: /* cmd ::= SHOW DNODES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DNODES_STMT); } break; - case 181: /* cmd ::= SHOW db_name_cond_opt VGROUPS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VGROUPS_STMT, yymsp[-1].minor.yy632, NULL); } + case 185: /* cmd ::= SHOW USERS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USERS_STMT); } break; - case 182: /* cmd ::= SHOW MNODES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MNODES_STMT, NULL, NULL); } + case 186: /* cmd ::= SHOW DATABASES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT); } break; - case 183: /* cmd ::= SHOW MODULES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MODULES_STMT, NULL, NULL); } + case 187: /* cmd ::= SHOW db_name_cond_opt TABLES like_pattern_opt */ +{ pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_TABLES_STMT, yymsp[-2].minor.yy392, yymsp[0].minor.yy392, OP_TYPE_LIKE); } break; - case 184: /* cmd ::= SHOW QNODES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QNODES_STMT, NULL, NULL); } + case 188: /* cmd ::= SHOW db_name_cond_opt STABLES like_pattern_opt */ +{ pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_STABLES_STMT, yymsp[-2].minor.yy392, yymsp[0].minor.yy392, OP_TYPE_LIKE); } break; - case 185: /* cmd ::= SHOW FUNCTIONS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_FUNCTIONS_STMT, NULL, NULL); } + case 189: /* cmd ::= SHOW db_name_cond_opt VGROUPS */ +{ pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_VGROUPS_STMT, yymsp[-1].minor.yy392, NULL, OP_TYPE_LIKE); } break; - case 186: /* cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } + case 190: /* cmd ::= SHOW MNODES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MNODES_STMT); } break; - case 187: /* cmd ::= SHOW STREAMS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STREAMS_STMT, NULL, NULL); } + case 191: /* cmd ::= SHOW MODULES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MODULES_STMT); } break; - case 188: /* cmd ::= SHOW ACCOUNTS */ + case 192: /* cmd ::= SHOW QNODES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QNODES_STMT); } + break; + case 193: /* cmd ::= SHOW FUNCTIONS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_FUNCTIONS_STMT); } + break; + case 194: /* cmd ::= SHOW INDEXES FROM table_name_cond from_db_opt */ +{ pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, yymsp[0].minor.yy392, yymsp[-1].minor.yy392, OP_TYPE_EQUAL); } + break; + case 195: /* cmd ::= SHOW STREAMS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_STREAMS_STMT); } + break; + case 196: /* cmd ::= SHOW ACCOUNTS */ { pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_EXPRIE_STATEMENT); } break; - case 189: /* cmd ::= SHOW APPS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_APPS_STMT, NULL, NULL); } + case 197: /* cmd ::= SHOW APPS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_APPS_STMT); } break; - case 190: /* cmd ::= SHOW CONNECTIONS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CONNECTIONS_STMT, NULL, NULL); } + case 198: /* cmd ::= SHOW CONNECTIONS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CONNECTIONS_STMT); } break; - case 191: /* cmd ::= SHOW LICENCE */ - case 192: /* cmd ::= SHOW GRANTS */ yytestcase(yyruleno==192); -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT, NULL, NULL); } + case 199: /* cmd ::= SHOW LICENCE */ + case 200: /* cmd ::= SHOW GRANTS */ yytestcase(yyruleno==200); +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_LICENCE_STMT); } break; - case 193: /* cmd ::= SHOW CREATE DATABASE db_name */ -{ pCxt->pRootNode = createShowCreateDatabaseStmt(pCxt, &yymsp[0].minor.yy209); } + case 201: /* cmd ::= SHOW CREATE DATABASE db_name */ +{ pCxt->pRootNode = createShowCreateDatabaseStmt(pCxt, &yymsp[0].minor.yy57); } break; - case 194: /* cmd ::= SHOW CREATE TABLE full_table_name */ -{ pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_TABLE_STMT, yymsp[0].minor.yy632); } + case 202: /* cmd ::= SHOW CREATE TABLE full_table_name */ +{ pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_TABLE_STMT, yymsp[0].minor.yy392); } break; - case 195: /* cmd ::= SHOW CREATE STABLE full_table_name */ -{ pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_STABLE_STMT, yymsp[0].minor.yy632); } + case 203: /* cmd ::= SHOW CREATE STABLE full_table_name */ +{ pCxt->pRootNode = createShowCreateTableStmt(pCxt, QUERY_NODE_SHOW_CREATE_STABLE_STMT, yymsp[0].minor.yy392); } break; - case 196: /* cmd ::= SHOW QUERIES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QUERIES_STMT, NULL, NULL); } + case 204: /* cmd ::= SHOW QUERIES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QUERIES_STMT); } break; - case 197: /* cmd ::= SHOW SCORES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SCORES_STMT, NULL, NULL); } + case 205: /* cmd ::= SHOW SCORES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SCORES_STMT); } break; - case 198: /* cmd ::= SHOW TOPICS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TOPICS_STMT, NULL, NULL); } + case 206: /* cmd ::= SHOW TOPICS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TOPICS_STMT); } break; - case 199: /* cmd ::= SHOW VARIABLES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VARIABLE_STMT, NULL, NULL); } + case 207: /* cmd ::= SHOW VARIABLES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_VARIABLE_STMT); } break; - case 200: /* cmd ::= SHOW BNODES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_BNODES_STMT, NULL, NULL); } + case 208: /* cmd ::= SHOW BNODES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_BNODES_STMT); } break; - case 201: /* cmd ::= SHOW SNODES */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SNODES_STMT, NULL, NULL); } + case 209: /* cmd ::= SHOW SNODES */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_SNODES_STMT); } break; - case 202: /* cmd ::= SHOW CLUSTER */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CLUSTER_STMT, NULL, NULL); } + case 210: /* cmd ::= SHOW CLUSTER */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_CLUSTER_STMT); } break; - case 203: /* cmd ::= SHOW TRANSACTIONS */ -{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TRANSACTIONS_STMT, NULL, NULL); } + case 211: /* cmd ::= SHOW TRANSACTIONS */ +{ pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_TRANSACTIONS_STMT); } break; - case 204: /* db_name_cond_opt ::= */ - case 209: /* from_db_opt ::= */ yytestcase(yyruleno==209); -{ yymsp[1].minor.yy632 = createDefaultDatabaseCondValue(pCxt); } + case 212: /* cmd ::= SHOW TABLE DISTRIBUTED full_table_name */ +{ pCxt->pRootNode = createShowTableDistributedStmt(pCxt, yymsp[0].minor.yy392); } break; - case 205: /* db_name_cond_opt ::= db_name NK_DOT */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[-1].minor.yy209); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + case 213: /* db_name_cond_opt ::= */ + case 218: /* from_db_opt ::= */ yytestcase(yyruleno==218); +{ yymsp[1].minor.yy392 = createDefaultDatabaseCondValue(pCxt); } break; - case 206: /* like_pattern_opt ::= */ - case 217: /* index_options ::= */ yytestcase(yyruleno==217); - case 246: /* into_opt ::= */ yytestcase(yyruleno==246); - case 402: /* where_clause_opt ::= */ yytestcase(yyruleno==402); - case 406: /* twindow_clause_opt ::= */ yytestcase(yyruleno==406); - case 411: /* sliding_opt ::= */ yytestcase(yyruleno==411); - case 413: /* fill_opt ::= */ yytestcase(yyruleno==413); - case 425: /* having_clause_opt ::= */ yytestcase(yyruleno==425); - case 435: /* slimit_clause_opt ::= */ yytestcase(yyruleno==435); - case 439: /* limit_clause_opt ::= */ yytestcase(yyruleno==439); -{ yymsp[1].minor.yy632 = NULL; } + case 214: /* db_name_cond_opt ::= db_name NK_DOT */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[-1].minor.yy57); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 207: /* like_pattern_opt ::= LIKE NK_STRING */ -{ yymsp[-1].minor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0); } + case 215: /* like_pattern_opt ::= */ + case 223: /* index_options ::= */ yytestcase(yyruleno==223); + case 252: /* into_opt ::= */ yytestcase(yyruleno==252); + case 408: /* where_clause_opt ::= */ yytestcase(yyruleno==408); + case 412: /* twindow_clause_opt ::= */ yytestcase(yyruleno==412); + case 417: /* sliding_opt ::= */ yytestcase(yyruleno==417); + case 419: /* fill_opt ::= */ yytestcase(yyruleno==419); + case 431: /* having_clause_opt ::= */ yytestcase(yyruleno==431); + case 433: /* range_opt ::= */ yytestcase(yyruleno==433); + case 435: /* every_opt ::= */ yytestcase(yyruleno==435); + case 445: /* slimit_clause_opt ::= */ yytestcase(yyruleno==445); + case 449: /* limit_clause_opt ::= */ yytestcase(yyruleno==449); +{ yymsp[1].minor.yy392 = NULL; } break; - case 208: /* table_name_cond ::= table_name */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy209); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 216: /* like_pattern_opt ::= LIKE NK_STRING */ +{ yymsp[-1].minor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0); } break; - case 210: /* from_db_opt ::= FROM db_name */ -{ yymsp[-1].minor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy209); } + case 217: /* table_name_cond ::= table_name */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy57); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 213: /* func_name ::= function_name */ -{ yylhsminor.yy632 = createFunctionNode(pCxt, &yymsp[0].minor.yy209, NULL); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 219: /* from_db_opt ::= FROM db_name */ +{ yymsp[-1].minor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy57); } break; - case 214: /* cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options */ -{ pCxt->pRootNode = createCreateIndexStmt(pCxt, INDEX_TYPE_SMA, yymsp[-4].minor.yy137, &yymsp[-3].minor.yy209, &yymsp[-1].minor.yy209, NULL, yymsp[0].minor.yy632); } + case 220: /* cmd ::= CREATE SMA INDEX not_exists_opt index_name ON table_name index_options */ +{ pCxt->pRootNode = createCreateIndexStmt(pCxt, INDEX_TYPE_SMA, yymsp[-4].minor.yy481, &yymsp[-3].minor.yy57, &yymsp[-1].minor.yy57, NULL, yymsp[0].minor.yy392); } break; - case 215: /* cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP */ -{ pCxt->pRootNode = createCreateIndexStmt(pCxt, INDEX_TYPE_FULLTEXT, yymsp[-6].minor.yy137, &yymsp[-5].minor.yy209, &yymsp[-3].minor.yy209, yymsp[-1].minor.yy424, NULL); } + case 221: /* cmd ::= CREATE FULLTEXT INDEX not_exists_opt index_name ON table_name NK_LP col_name_list NK_RP */ +{ pCxt->pRootNode = createCreateIndexStmt(pCxt, INDEX_TYPE_FULLTEXT, yymsp[-6].minor.yy481, &yymsp[-5].minor.yy57, &yymsp[-3].minor.yy57, yymsp[-1].minor.yy600, NULL); } break; - case 216: /* cmd ::= DROP INDEX exists_opt index_name ON table_name */ -{ pCxt->pRootNode = createDropIndexStmt(pCxt, yymsp[-3].minor.yy137, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209); } + case 222: /* cmd ::= DROP INDEX exists_opt index_name ON table_name */ +{ pCxt->pRootNode = createDropIndexStmt(pCxt, yymsp[-3].minor.yy481, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57); } break; - case 218: /* index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt */ -{ yymsp[-8].minor.yy632 = createIndexOption(pCxt, yymsp[-6].minor.yy424, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), NULL, yymsp[0].minor.yy632); } + case 224: /* index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_RP sliding_opt */ +{ yymsp[-8].minor.yy392 = createIndexOption(pCxt, yymsp[-6].minor.yy600, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), NULL, yymsp[0].minor.yy392); } break; - case 219: /* index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt */ -{ yymsp[-10].minor.yy632 = createIndexOption(pCxt, yymsp[-8].minor.yy424, releaseRawExprNode(pCxt, yymsp[-4].minor.yy632), releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), yymsp[0].minor.yy632); } + case 225: /* index_options ::= FUNCTION NK_LP func_list NK_RP INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt */ +{ yymsp[-10].minor.yy392 = createIndexOption(pCxt, yymsp[-8].minor.yy600, releaseRawExprNode(pCxt, yymsp[-4].minor.yy392), releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), yymsp[0].minor.yy392); } break; - case 222: /* func ::= function_name NK_LP expression_list NK_RP */ -{ yylhsminor.yy632 = createFunctionNode(pCxt, &yymsp[-3].minor.yy209, yymsp[-1].minor.yy424); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + case 228: /* func ::= function_name NK_LP expression_list NK_RP */ +{ yylhsminor.yy392 = createFunctionNode(pCxt, &yymsp[-3].minor.yy57, yymsp[-1].minor.yy600); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 223: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression */ -{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-3].minor.yy137, &yymsp[-2].minor.yy209, yymsp[0].minor.yy632, NULL, NULL); } + case 229: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS query_expression */ +{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-3].minor.yy481, &yymsp[-2].minor.yy57, yymsp[0].minor.yy392, NULL, NULL); } break; - case 224: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name */ -{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-4].minor.yy137, &yymsp[-3].minor.yy209, NULL, &yymsp[0].minor.yy209, NULL); } + case 230: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS DATABASE db_name */ +{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-4].minor.yy481, &yymsp[-3].minor.yy57, NULL, &yymsp[0].minor.yy57, NULL); } break; - case 225: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name */ -{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-4].minor.yy137, &yymsp[-3].minor.yy209, NULL, NULL, yymsp[0].minor.yy632); } + case 231: /* cmd ::= CREATE TOPIC not_exists_opt topic_name AS STABLE full_table_name */ +{ pCxt->pRootNode = createCreateTopicStmt(pCxt, yymsp[-4].minor.yy481, &yymsp[-3].minor.yy57, NULL, NULL, yymsp[0].minor.yy392); } break; - case 226: /* cmd ::= DROP TOPIC exists_opt topic_name */ -{ pCxt->pRootNode = createDropTopicStmt(pCxt, yymsp[-1].minor.yy137, &yymsp[0].minor.yy209); } + case 232: /* cmd ::= DROP TOPIC exists_opt topic_name */ +{ pCxt->pRootNode = createDropTopicStmt(pCxt, yymsp[-1].minor.yy481, &yymsp[0].minor.yy57); } break; - case 227: /* cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name */ -{ pCxt->pRootNode = createDropCGroupStmt(pCxt, yymsp[-3].minor.yy137, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209); } + case 233: /* cmd ::= DROP CONSUMER GROUP exists_opt cgroup_name ON topic_name */ +{ pCxt->pRootNode = createDropCGroupStmt(pCxt, yymsp[-3].minor.yy481, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57); } break; - case 228: /* cmd ::= DESC full_table_name */ - case 229: /* cmd ::= DESCRIBE full_table_name */ yytestcase(yyruleno==229); -{ pCxt->pRootNode = createDescribeStmt(pCxt, yymsp[0].minor.yy632); } + case 234: /* cmd ::= DESC full_table_name */ + case 235: /* cmd ::= DESCRIBE full_table_name */ yytestcase(yyruleno==235); +{ pCxt->pRootNode = createDescribeStmt(pCxt, yymsp[0].minor.yy392); } break; - case 230: /* cmd ::= RESET QUERY CACHE */ + case 236: /* cmd ::= RESET QUERY CACHE */ { pCxt->pRootNode = createResetQueryCacheStmt(pCxt); } break; - case 231: /* cmd ::= EXPLAIN analyze_opt explain_options query_expression */ -{ pCxt->pRootNode = createExplainStmt(pCxt, yymsp[-2].minor.yy137, yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } + case 237: /* cmd ::= EXPLAIN analyze_opt explain_options query_expression */ +{ pCxt->pRootNode = createExplainStmt(pCxt, yymsp[-2].minor.yy481, yymsp[-1].minor.yy392, yymsp[0].minor.yy392); } break; - case 233: /* analyze_opt ::= ANALYZE */ - case 241: /* agg_func_opt ::= AGGREGATE */ yytestcase(yyruleno==241); - case 392: /* set_quantifier_opt ::= DISTINCT */ yytestcase(yyruleno==392); -{ yymsp[0].minor.yy137 = true; } + case 239: /* analyze_opt ::= ANALYZE */ + case 247: /* agg_func_opt ::= AGGREGATE */ yytestcase(yyruleno==247); + case 398: /* set_quantifier_opt ::= DISTINCT */ yytestcase(yyruleno==398); +{ yymsp[0].minor.yy481 = true; } break; - case 234: /* explain_options ::= */ -{ yymsp[1].minor.yy632 = createDefaultExplainOptions(pCxt); } + case 240: /* explain_options ::= */ +{ yymsp[1].minor.yy392 = createDefaultExplainOptions(pCxt); } break; - case 235: /* explain_options ::= explain_options VERBOSE NK_BOOL */ -{ yylhsminor.yy632 = setExplainVerbose(pCxt, yymsp[-2].minor.yy632, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 241: /* explain_options ::= explain_options VERBOSE NK_BOOL */ +{ yylhsminor.yy392 = setExplainVerbose(pCxt, yymsp[-2].minor.yy392, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 236: /* explain_options ::= explain_options RATIO NK_FLOAT */ -{ yylhsminor.yy632 = setExplainRatio(pCxt, yymsp[-2].minor.yy632, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 242: /* explain_options ::= explain_options RATIO NK_FLOAT */ +{ yylhsminor.yy392 = setExplainRatio(pCxt, yymsp[-2].minor.yy392, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 237: /* cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP */ -{ pCxt->pRootNode = createCompactStmt(pCxt, yymsp[-1].minor.yy424); } + case 243: /* cmd ::= COMPACT VNODES IN NK_LP integer_list NK_RP */ +{ pCxt->pRootNode = createCompactStmt(pCxt, yymsp[-1].minor.yy600); } break; - case 238: /* cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt */ -{ pCxt->pRootNode = createCreateFunctionStmt(pCxt, yymsp[-6].minor.yy137, yymsp[-8].minor.yy137, &yymsp[-5].minor.yy209, &yymsp[-3].minor.yy0, yymsp[-1].minor.yy304, yymsp[0].minor.yy100); } + case 244: /* cmd ::= CREATE agg_func_opt FUNCTION not_exists_opt function_name AS NK_STRING OUTPUTTYPE type_name bufsize_opt */ +{ pCxt->pRootNode = createCreateFunctionStmt(pCxt, yymsp[-6].minor.yy481, yymsp[-8].minor.yy481, &yymsp[-5].minor.yy57, &yymsp[-3].minor.yy0, yymsp[-1].minor.yy448, yymsp[0].minor.yy228); } break; - case 239: /* cmd ::= DROP FUNCTION exists_opt function_name */ -{ pCxt->pRootNode = createDropFunctionStmt(pCxt, yymsp[-1].minor.yy137, &yymsp[0].minor.yy209); } + case 245: /* cmd ::= DROP FUNCTION exists_opt function_name */ +{ pCxt->pRootNode = createDropFunctionStmt(pCxt, yymsp[-1].minor.yy481, &yymsp[0].minor.yy57); } break; - case 242: /* bufsize_opt ::= */ -{ yymsp[1].minor.yy100 = 0; } + case 248: /* bufsize_opt ::= */ +{ yymsp[1].minor.yy228 = 0; } break; - case 243: /* bufsize_opt ::= BUFSIZE NK_INTEGER */ -{ yymsp[-1].minor.yy100 = taosStr2Int32(yymsp[0].minor.yy0.z, NULL, 10); } + case 249: /* bufsize_opt ::= BUFSIZE NK_INTEGER */ +{ yymsp[-1].minor.yy228 = taosStr2Int32(yymsp[0].minor.yy0.z, NULL, 10); } break; - case 244: /* cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression */ -{ pCxt->pRootNode = createCreateStreamStmt(pCxt, yymsp[-5].minor.yy137, &yymsp[-4].minor.yy209, yymsp[-2].minor.yy632, yymsp[-3].minor.yy632, yymsp[0].minor.yy632); } + case 250: /* cmd ::= CREATE STREAM not_exists_opt stream_name stream_options into_opt AS query_expression */ +{ pCxt->pRootNode = createCreateStreamStmt(pCxt, yymsp[-5].minor.yy481, &yymsp[-4].minor.yy57, yymsp[-2].minor.yy392, yymsp[-3].minor.yy392, yymsp[0].minor.yy392); } break; - case 245: /* cmd ::= DROP STREAM exists_opt stream_name */ -{ pCxt->pRootNode = createDropStreamStmt(pCxt, yymsp[-1].minor.yy137, &yymsp[0].minor.yy209); } + case 251: /* cmd ::= DROP STREAM exists_opt stream_name */ +{ pCxt->pRootNode = createDropStreamStmt(pCxt, yymsp[-1].minor.yy481, &yymsp[0].minor.yy57); } break; - case 247: /* into_opt ::= INTO full_table_name */ - case 373: /* from_clause ::= FROM table_reference_list */ yytestcase(yyruleno==373); - case 403: /* where_clause_opt ::= WHERE search_condition */ yytestcase(yyruleno==403); - case 426: /* having_clause_opt ::= HAVING search_condition */ yytestcase(yyruleno==426); -{ yymsp[-1].minor.yy632 = yymsp[0].minor.yy632; } + case 253: /* into_opt ::= INTO full_table_name */ + case 379: /* from_clause ::= FROM table_reference_list */ yytestcase(yyruleno==379); + case 409: /* where_clause_opt ::= WHERE search_condition */ yytestcase(yyruleno==409); + case 432: /* having_clause_opt ::= HAVING search_condition */ yytestcase(yyruleno==432); +{ yymsp[-1].minor.yy392 = yymsp[0].minor.yy392; } break; - case 248: /* stream_options ::= */ -{ yymsp[1].minor.yy632 = createStreamOptions(pCxt); } + case 254: /* stream_options ::= */ +{ yymsp[1].minor.yy392 = createStreamOptions(pCxt); } break; - case 249: /* stream_options ::= stream_options TRIGGER AT_ONCE */ -{ ((SStreamOptions*)yymsp[-2].minor.yy632)->triggerType = STREAM_TRIGGER_AT_ONCE; yylhsminor.yy632 = yymsp[-2].minor.yy632; } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 255: /* stream_options ::= stream_options TRIGGER AT_ONCE */ +{ ((SStreamOptions*)yymsp[-2].minor.yy392)->triggerType = STREAM_TRIGGER_AT_ONCE; yylhsminor.yy392 = yymsp[-2].minor.yy392; } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 250: /* stream_options ::= stream_options TRIGGER WINDOW_CLOSE */ -{ ((SStreamOptions*)yymsp[-2].minor.yy632)->triggerType = STREAM_TRIGGER_WINDOW_CLOSE; yylhsminor.yy632 = yymsp[-2].minor.yy632; } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 256: /* stream_options ::= stream_options TRIGGER WINDOW_CLOSE */ +{ ((SStreamOptions*)yymsp[-2].minor.yy392)->triggerType = STREAM_TRIGGER_WINDOW_CLOSE; yylhsminor.yy392 = yymsp[-2].minor.yy392; } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 251: /* stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal */ -{ ((SStreamOptions*)yymsp[-3].minor.yy632)->triggerType = STREAM_TRIGGER_MAX_DELAY; ((SStreamOptions*)yymsp[-3].minor.yy632)->pDelay = releaseRawExprNode(pCxt, yymsp[0].minor.yy632); yylhsminor.yy632 = yymsp[-3].minor.yy632; } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + case 257: /* stream_options ::= stream_options TRIGGER MAX_DELAY duration_literal */ +{ ((SStreamOptions*)yymsp[-3].minor.yy392)->triggerType = STREAM_TRIGGER_MAX_DELAY; ((SStreamOptions*)yymsp[-3].minor.yy392)->pDelay = releaseRawExprNode(pCxt, yymsp[0].minor.yy392); yylhsminor.yy392 = yymsp[-3].minor.yy392; } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 252: /* stream_options ::= stream_options WATERMARK duration_literal */ -{ ((SStreamOptions*)yymsp[-2].minor.yy632)->pWatermark = releaseRawExprNode(pCxt, yymsp[0].minor.yy632); yylhsminor.yy632 = yymsp[-2].minor.yy632; } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 258: /* stream_options ::= stream_options WATERMARK duration_literal */ +{ ((SStreamOptions*)yymsp[-2].minor.yy392)->pWatermark = releaseRawExprNode(pCxt, yymsp[0].minor.yy392); yylhsminor.yy392 = yymsp[-2].minor.yy392; } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 253: /* cmd ::= KILL CONNECTION NK_INTEGER */ + case 259: /* cmd ::= KILL CONNECTION NK_INTEGER */ { pCxt->pRootNode = createKillStmt(pCxt, QUERY_NODE_KILL_CONNECTION_STMT, &yymsp[0].minor.yy0); } break; - case 254: /* cmd ::= KILL QUERY NK_STRING */ + case 260: /* cmd ::= KILL QUERY NK_STRING */ { pCxt->pRootNode = createKillQueryStmt(pCxt, &yymsp[0].minor.yy0); } break; - case 255: /* cmd ::= KILL TRANSACTION NK_INTEGER */ + case 261: /* cmd ::= KILL TRANSACTION NK_INTEGER */ { pCxt->pRootNode = createKillStmt(pCxt, QUERY_NODE_KILL_TRANSACTION_STMT, &yymsp[0].minor.yy0); } break; - case 256: /* cmd ::= BALANCE VGROUP */ + case 262: /* cmd ::= BALANCE VGROUP */ { pCxt->pRootNode = createBalanceVgroupStmt(pCxt); } break; - case 257: /* cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER */ + case 263: /* cmd ::= MERGE VGROUP NK_INTEGER NK_INTEGER */ { pCxt->pRootNode = createMergeVgroupStmt(pCxt, &yymsp[-1].minor.yy0, &yymsp[0].minor.yy0); } break; - case 258: /* cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list */ -{ pCxt->pRootNode = createRedistributeVgroupStmt(pCxt, &yymsp[-1].minor.yy0, yymsp[0].minor.yy424); } + case 264: /* cmd ::= REDISTRIBUTE VGROUP NK_INTEGER dnode_list */ +{ pCxt->pRootNode = createRedistributeVgroupStmt(pCxt, &yymsp[-1].minor.yy0, yymsp[0].minor.yy600); } break; - case 259: /* cmd ::= SPLIT VGROUP NK_INTEGER */ + case 265: /* cmd ::= SPLIT VGROUP NK_INTEGER */ { pCxt->pRootNode = createSplitVgroupStmt(pCxt, &yymsp[0].minor.yy0); } break; - case 260: /* dnode_list ::= DNODE NK_INTEGER */ -{ yymsp[-1].minor.yy424 = createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } - break; - case 262: /* cmd ::= SYNCDB db_name REPLICA */ -{ pCxt->pRootNode = createSyncdbStmt(pCxt, &yymsp[-1].minor.yy209); } - break; - case 263: /* cmd ::= DELETE FROM full_table_name where_clause_opt */ -{ pCxt->pRootNode = createDeleteStmt(pCxt, yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } - break; - case 265: /* literal ::= NK_INTEGER */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 266: /* literal ::= NK_FLOAT */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 267: /* literal ::= NK_STRING */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 268: /* literal ::= NK_BOOL */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BOOL, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 269: /* literal ::= TIMESTAMP NK_STRING */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_TIMESTAMP, &yymsp[0].minor.yy0)); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; - break; - case 270: /* literal ::= duration_literal */ - case 280: /* signed_literal ::= signed */ yytestcase(yyruleno==280); - case 300: /* expression ::= literal */ yytestcase(yyruleno==300); - case 301: /* expression ::= pseudo_column */ yytestcase(yyruleno==301); - case 302: /* expression ::= column_reference */ yytestcase(yyruleno==302); - case 303: /* expression ::= function_expression */ yytestcase(yyruleno==303); - case 304: /* expression ::= subquery */ yytestcase(yyruleno==304); - case 329: /* function_expression ::= literal_func */ yytestcase(yyruleno==329); - case 365: /* boolean_value_expression ::= boolean_primary */ yytestcase(yyruleno==365); - case 369: /* boolean_primary ::= predicate */ yytestcase(yyruleno==369); - case 371: /* common_expression ::= expression */ yytestcase(yyruleno==371); - case 372: /* common_expression ::= boolean_value_expression */ yytestcase(yyruleno==372); - case 374: /* table_reference_list ::= table_reference */ yytestcase(yyruleno==374); - case 376: /* table_reference ::= table_primary */ yytestcase(yyruleno==376); - case 377: /* table_reference ::= joined_table */ yytestcase(yyruleno==377); - case 381: /* table_primary ::= parenthesized_joined_table */ yytestcase(yyruleno==381); - case 428: /* query_expression_body ::= query_primary */ yytestcase(yyruleno==428); - case 431: /* query_primary ::= query_specification */ yytestcase(yyruleno==431); -{ yylhsminor.yy632 = yymsp[0].minor.yy632; } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 271: /* literal ::= NULL */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_NULL, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 272: /* literal ::= NK_QUESTION */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createPlaceholderValueNode(pCxt, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 273: /* duration_literal ::= NK_VARIABLE */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 274: /* signed ::= NK_INTEGER */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 275: /* signed ::= NK_PLUS NK_INTEGER */ -{ yymsp[-1].minor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0); } - break; - case 276: /* signed ::= NK_MINUS NK_INTEGER */ + case 266: /* dnode_list ::= DNODE NK_INTEGER */ +{ yymsp[-1].minor.yy600 = createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } + break; + case 268: /* cmd ::= SYNCDB db_name REPLICA */ +{ pCxt->pRootNode = createSyncdbStmt(pCxt, &yymsp[-1].minor.yy57); } + break; + case 269: /* cmd ::= DELETE FROM full_table_name where_clause_opt */ +{ pCxt->pRootNode = createDeleteStmt(pCxt, yymsp[-1].minor.yy392, yymsp[0].minor.yy392); } + break; + case 271: /* literal ::= NK_INTEGER */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 272: /* literal ::= NK_FLOAT */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 273: /* literal ::= NK_STRING */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 274: /* literal ::= NK_BOOL */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_BOOL, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 275: /* literal ::= TIMESTAMP NK_STRING */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_TIMESTAMP, &yymsp[0].minor.yy0)); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; + break; + case 276: /* literal ::= duration_literal */ + case 286: /* signed_literal ::= signed */ yytestcase(yyruleno==286); + case 306: /* expression ::= literal */ yytestcase(yyruleno==306); + case 307: /* expression ::= pseudo_column */ yytestcase(yyruleno==307); + case 308: /* expression ::= column_reference */ yytestcase(yyruleno==308); + case 309: /* expression ::= function_expression */ yytestcase(yyruleno==309); + case 310: /* expression ::= subquery */ yytestcase(yyruleno==310); + case 335: /* function_expression ::= literal_func */ yytestcase(yyruleno==335); + case 371: /* boolean_value_expression ::= boolean_primary */ yytestcase(yyruleno==371); + case 375: /* boolean_primary ::= predicate */ yytestcase(yyruleno==375); + case 377: /* common_expression ::= expression */ yytestcase(yyruleno==377); + case 378: /* common_expression ::= boolean_value_expression */ yytestcase(yyruleno==378); + case 380: /* table_reference_list ::= table_reference */ yytestcase(yyruleno==380); + case 382: /* table_reference ::= table_primary */ yytestcase(yyruleno==382); + case 383: /* table_reference ::= joined_table */ yytestcase(yyruleno==383); + case 387: /* table_primary ::= parenthesized_joined_table */ yytestcase(yyruleno==387); + case 438: /* query_expression_body ::= query_primary */ yytestcase(yyruleno==438); + case 441: /* query_primary ::= query_specification */ yytestcase(yyruleno==441); +{ yylhsminor.yy392 = yymsp[0].minor.yy392; } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 277: /* literal ::= NULL */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createValueNode(pCxt, TSDB_DATA_TYPE_NULL, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 278: /* literal ::= NK_QUESTION */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createPlaceholderValueNode(pCxt, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 279: /* duration_literal ::= NK_VARIABLE */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createDurationValueNode(pCxt, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 280: /* signed ::= NK_INTEGER */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 281: /* signed ::= NK_PLUS NK_INTEGER */ +{ yymsp[-1].minor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &yymsp[0].minor.yy0); } + break; + case 282: /* signed ::= NK_MINUS NK_INTEGER */ { SToken t = yymsp[-1].minor.yy0; t.n = (yymsp[0].minor.yy0.z + yymsp[0].minor.yy0.n) - yymsp[-1].minor.yy0.z; - yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &t); + yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BIGINT, &t); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 277: /* signed ::= NK_FLOAT */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 283: /* signed ::= NK_FLOAT */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 278: /* signed ::= NK_PLUS NK_FLOAT */ -{ yymsp[-1].minor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0); } + case 284: /* signed ::= NK_PLUS NK_FLOAT */ +{ yymsp[-1].minor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &yymsp[0].minor.yy0); } break; - case 279: /* signed ::= NK_MINUS NK_FLOAT */ + case 285: /* signed ::= NK_MINUS NK_FLOAT */ { SToken t = yymsp[-1].minor.yy0; t.n = (yymsp[0].minor.yy0.z + yymsp[0].minor.yy0.n) - yymsp[-1].minor.yy0.z; - yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &t); + yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_DOUBLE, &t); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 281: /* signed_literal ::= NK_STRING */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 287: /* signed_literal ::= NK_STRING */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 282: /* signed_literal ::= NK_BOOL */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_BOOL, &yymsp[0].minor.yy0); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 288: /* signed_literal ::= NK_BOOL */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_BOOL, &yymsp[0].minor.yy0); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 283: /* signed_literal ::= TIMESTAMP NK_STRING */ -{ yymsp[-1].minor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_TIMESTAMP, &yymsp[0].minor.yy0); } + case 289: /* signed_literal ::= TIMESTAMP NK_STRING */ +{ yymsp[-1].minor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_TIMESTAMP, &yymsp[0].minor.yy0); } break; - case 284: /* signed_literal ::= duration_literal */ - case 286: /* signed_literal ::= literal_func */ yytestcase(yyruleno==286); - case 343: /* star_func_para ::= expression */ yytestcase(yyruleno==343); - case 398: /* select_item ::= common_expression */ yytestcase(yyruleno==398); - case 444: /* search_condition ::= common_expression */ yytestcase(yyruleno==444); -{ yylhsminor.yy632 = releaseRawExprNode(pCxt, yymsp[0].minor.yy632); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 290: /* signed_literal ::= duration_literal */ + case 292: /* signed_literal ::= literal_func */ yytestcase(yyruleno==292); + case 349: /* star_func_para ::= expression */ yytestcase(yyruleno==349); + case 404: /* select_item ::= common_expression */ yytestcase(yyruleno==404); + case 454: /* search_condition ::= common_expression */ yytestcase(yyruleno==454); +{ yylhsminor.yy392 = releaseRawExprNode(pCxt, yymsp[0].minor.yy392); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 285: /* signed_literal ::= NULL */ -{ yylhsminor.yy632 = createValueNode(pCxt, TSDB_DATA_TYPE_NULL, &yymsp[0].minor.yy0); } - yymsp[0].minor.yy632 = yylhsminor.yy632; + case 291: /* signed_literal ::= NULL */ +{ yylhsminor.yy392 = createValueNode(pCxt, TSDB_DATA_TYPE_NULL, &yymsp[0].minor.yy0); } + yymsp[0].minor.yy392 = yylhsminor.yy392; break; - case 305: /* expression ::= NK_LP expression NK_RP */ - case 370: /* boolean_primary ::= NK_LP boolean_value_expression NK_RP */ yytestcase(yyruleno==370); -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, releaseRawExprNode(pCxt, yymsp[-1].minor.yy632)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 311: /* expression ::= NK_LP expression NK_RP */ + case 376: /* boolean_primary ::= NK_LP boolean_value_expression NK_RP */ yytestcase(yyruleno==376); +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, releaseRawExprNode(pCxt, yymsp[-1].minor.yy392)); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 306: /* expression ::= NK_PLUS expression */ + case 312: /* expression ::= NK_PLUS expression */ { - SToken t = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &t, releaseRawExprNode(pCxt, yymsp[0].minor.yy632)); + SToken t = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &t, releaseRawExprNode(pCxt, yymsp[0].minor.yy392)); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 307: /* expression ::= NK_MINUS expression */ + case 313: /* expression ::= NK_MINUS expression */ { - SToken t = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &t, createOperatorNode(pCxt, OP_TYPE_MINUS, releaseRawExprNode(pCxt, yymsp[0].minor.yy632), NULL)); + SToken t = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &t, createOperatorNode(pCxt, OP_TYPE_MINUS, releaseRawExprNode(pCxt, yymsp[0].minor.yy392), NULL)); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 308: /* expression ::= expression NK_PLUS expression */ + case 314: /* expression ::= expression NK_PLUS expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_ADD, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_ADD, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 309: /* expression ::= expression NK_MINUS expression */ + case 315: /* expression ::= expression NK_MINUS expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_SUB, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_SUB, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 310: /* expression ::= expression NK_STAR expression */ + case 316: /* expression ::= expression NK_STAR expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_MULTI, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_MULTI, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 311: /* expression ::= expression NK_SLASH expression */ + case 317: /* expression ::= expression NK_SLASH expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_DIV, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_DIV, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 312: /* expression ::= expression NK_REM expression */ + case 318: /* expression ::= expression NK_REM expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_REM, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, OP_TYPE_REM, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 313: /* expression ::= column_reference NK_ARROW NK_STRING */ + case 319: /* expression ::= column_reference NK_ARROW NK_STRING */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_JSON_GET_VALUE, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_JSON_GET_VALUE, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[0].minor.yy0))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; - break; - case 314: /* expression_list ::= expression */ -{ yylhsminor.yy424 = createNodeList(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy632)); } - yymsp[0].minor.yy424 = yylhsminor.yy424; - break; - case 315: /* expression_list ::= expression_list NK_COMMA expression */ -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-2].minor.yy424, releaseRawExprNode(pCxt, yymsp[0].minor.yy632)); } - yymsp[-2].minor.yy424 = yylhsminor.yy424; - break; - case 316: /* column_reference ::= column_name */ -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy209, createColumnNode(pCxt, NULL, &yymsp[0].minor.yy209)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 317: /* column_reference ::= table_name NK_DOT column_name */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209, createColumnNode(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy209)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; - break; - case 318: /* pseudo_column ::= ROWTS */ - case 319: /* pseudo_column ::= TBNAME */ yytestcase(yyruleno==319); - case 321: /* pseudo_column ::= QSTARTTS */ yytestcase(yyruleno==321); - case 322: /* pseudo_column ::= QENDTS */ yytestcase(yyruleno==322); - case 323: /* pseudo_column ::= WSTARTTS */ yytestcase(yyruleno==323); - case 324: /* pseudo_column ::= WENDTS */ yytestcase(yyruleno==324); - case 325: /* pseudo_column ::= WDURATION */ yytestcase(yyruleno==325); - case 331: /* literal_func ::= NOW */ yytestcase(yyruleno==331); -{ yylhsminor.yy632 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[0].minor.yy0, NULL)); } - yymsp[0].minor.yy632 = yylhsminor.yy632; - break; - case 320: /* pseudo_column ::= table_name NK_DOT TBNAME */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[0].minor.yy0, createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[-2].minor.yy209)))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; - break; - case 326: /* function_expression ::= function_name NK_LP expression_list NK_RP */ - case 327: /* function_expression ::= star_func NK_LP star_func_para_list NK_RP */ yytestcase(yyruleno==327); -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-3].minor.yy209, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[-3].minor.yy209, yymsp[-1].minor.yy424)); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; - break; - case 328: /* function_expression ::= CAST NK_LP expression AS type_name NK_RP */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-5].minor.yy0, &yymsp[0].minor.yy0, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy632), yymsp[-1].minor.yy304)); } - yymsp[-5].minor.yy632 = yylhsminor.yy632; - break; - case 330: /* literal_func ::= noarg_func NK_LP NK_RP */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[-2].minor.yy209, NULL)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; - break; - case 339: /* star_func_para_list ::= NK_STAR */ -{ yylhsminor.yy424 = createNodeList(pCxt, createColumnNode(pCxt, NULL, &yymsp[0].minor.yy0)); } - yymsp[0].minor.yy424 = yylhsminor.yy424; - break; - case 344: /* star_func_para ::= table_name NK_DOT NK_STAR */ - case 401: /* select_item ::= table_name NK_DOT NK_STAR */ yytestcase(yyruleno==401); -{ yylhsminor.yy632 = createColumnNode(pCxt, &yymsp[-2].minor.yy209, &yymsp[0].minor.yy0); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; - break; - case 345: /* predicate ::= expression compare_op expression */ - case 350: /* predicate ::= expression in_op in_predicate_value */ yytestcase(yyruleno==350); + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 322: /* column_reference ::= column_name */ +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy57, createColumnNode(pCxt, NULL, &yymsp[0].minor.yy57)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 323: /* column_reference ::= table_name NK_DOT column_name */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57, createColumnNode(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy57)); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 324: /* pseudo_column ::= ROWTS */ + case 325: /* pseudo_column ::= TBNAME */ yytestcase(yyruleno==325); + case 327: /* pseudo_column ::= QSTARTTS */ yytestcase(yyruleno==327); + case 328: /* pseudo_column ::= QENDTS */ yytestcase(yyruleno==328); + case 329: /* pseudo_column ::= WSTARTTS */ yytestcase(yyruleno==329); + case 330: /* pseudo_column ::= WENDTS */ yytestcase(yyruleno==330); + case 331: /* pseudo_column ::= WDURATION */ yytestcase(yyruleno==331); + case 337: /* literal_func ::= NOW */ yytestcase(yyruleno==337); +{ yylhsminor.yy392 = createRawExprNode(pCxt, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[0].minor.yy0, NULL)); } + yymsp[0].minor.yy392 = yylhsminor.yy392; + break; + case 326: /* pseudo_column ::= table_name NK_DOT TBNAME */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[0].minor.yy0, createNodeList(pCxt, createValueNode(pCxt, TSDB_DATA_TYPE_BINARY, &yymsp[-2].minor.yy57)))); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 332: /* function_expression ::= function_name NK_LP expression_list NK_RP */ + case 333: /* function_expression ::= star_func NK_LP star_func_para_list NK_RP */ yytestcase(yyruleno==333); +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-3].minor.yy57, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[-3].minor.yy57, yymsp[-1].minor.yy600)); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; + break; + case 334: /* function_expression ::= CAST NK_LP expression AS type_name NK_RP */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-5].minor.yy0, &yymsp[0].minor.yy0, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), yymsp[-1].minor.yy448)); } + yymsp[-5].minor.yy392 = yylhsminor.yy392; + break; + case 336: /* literal_func ::= noarg_func NK_LP NK_RP */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy0, createFunctionNode(pCxt, &yymsp[-2].minor.yy57, NULL)); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 345: /* star_func_para_list ::= NK_STAR */ +{ yylhsminor.yy600 = createNodeList(pCxt, createColumnNode(pCxt, NULL, &yymsp[0].minor.yy0)); } + yymsp[0].minor.yy600 = yylhsminor.yy600; + break; + case 350: /* star_func_para ::= table_name NK_DOT NK_STAR */ + case 407: /* select_item ::= table_name NK_DOT NK_STAR */ yytestcase(yyruleno==407); +{ yylhsminor.yy392 = createColumnNode(pCxt, &yymsp[-2].minor.yy57, &yymsp[0].minor.yy0); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; + break; + case 351: /* predicate ::= expression compare_op expression */ + case 356: /* predicate ::= expression in_op in_predicate_value */ yytestcase(yyruleno==356); { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, yymsp[-1].minor.yy380, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createOperatorNode(pCxt, yymsp[-1].minor.yy324, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 346: /* predicate ::= expression BETWEEN expression AND expression */ + case 352: /* predicate ::= expression BETWEEN expression AND expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-4].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createBetweenAnd(pCxt, releaseRawExprNode(pCxt, yymsp[-4].minor.yy632), releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-4].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createBetweenAnd(pCxt, releaseRawExprNode(pCxt, yymsp[-4].minor.yy392), releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-4].minor.yy632 = yylhsminor.yy632; + yymsp[-4].minor.yy392 = yylhsminor.yy392; break; - case 347: /* predicate ::= expression NOT BETWEEN expression AND expression */ + case 353: /* predicate ::= expression NOT BETWEEN expression AND expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-5].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createNotBetweenAnd(pCxt, releaseRawExprNode(pCxt, yymsp[-5].minor.yy632), releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-5].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createNotBetweenAnd(pCxt, releaseRawExprNode(pCxt, yymsp[-5].minor.yy392), releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-5].minor.yy632 = yylhsminor.yy632; + yymsp[-5].minor.yy392 = yylhsminor.yy392; break; - case 348: /* predicate ::= expression IS NULL */ + case 354: /* predicate ::= expression IS NULL */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_IS_NULL, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), NULL)); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_IS_NULL, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), NULL)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 349: /* predicate ::= expression IS NOT NULL */ + case 355: /* predicate ::= expression IS NOT NULL */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-3].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_IS_NOT_NULL, releaseRawExprNode(pCxt, yymsp[-3].minor.yy632), NULL)); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-3].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &yymsp[0].minor.yy0, createOperatorNode(pCxt, OP_TYPE_IS_NOT_NULL, releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), NULL)); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 351: /* compare_op ::= NK_LT */ -{ yymsp[0].minor.yy380 = OP_TYPE_LOWER_THAN; } + case 357: /* compare_op ::= NK_LT */ +{ yymsp[0].minor.yy324 = OP_TYPE_LOWER_THAN; } break; - case 352: /* compare_op ::= NK_GT */ -{ yymsp[0].minor.yy380 = OP_TYPE_GREATER_THAN; } + case 358: /* compare_op ::= NK_GT */ +{ yymsp[0].minor.yy324 = OP_TYPE_GREATER_THAN; } break; - case 353: /* compare_op ::= NK_LE */ -{ yymsp[0].minor.yy380 = OP_TYPE_LOWER_EQUAL; } + case 359: /* compare_op ::= NK_LE */ +{ yymsp[0].minor.yy324 = OP_TYPE_LOWER_EQUAL; } break; - case 354: /* compare_op ::= NK_GE */ -{ yymsp[0].minor.yy380 = OP_TYPE_GREATER_EQUAL; } + case 360: /* compare_op ::= NK_GE */ +{ yymsp[0].minor.yy324 = OP_TYPE_GREATER_EQUAL; } break; - case 355: /* compare_op ::= NK_NE */ -{ yymsp[0].minor.yy380 = OP_TYPE_NOT_EQUAL; } + case 361: /* compare_op ::= NK_NE */ +{ yymsp[0].minor.yy324 = OP_TYPE_NOT_EQUAL; } break; - case 356: /* compare_op ::= NK_EQ */ -{ yymsp[0].minor.yy380 = OP_TYPE_EQUAL; } + case 362: /* compare_op ::= NK_EQ */ +{ yymsp[0].minor.yy324 = OP_TYPE_EQUAL; } break; - case 357: /* compare_op ::= LIKE */ -{ yymsp[0].minor.yy380 = OP_TYPE_LIKE; } + case 363: /* compare_op ::= LIKE */ +{ yymsp[0].minor.yy324 = OP_TYPE_LIKE; } break; - case 358: /* compare_op ::= NOT LIKE */ -{ yymsp[-1].minor.yy380 = OP_TYPE_NOT_LIKE; } + case 364: /* compare_op ::= NOT LIKE */ +{ yymsp[-1].minor.yy324 = OP_TYPE_NOT_LIKE; } break; - case 359: /* compare_op ::= MATCH */ -{ yymsp[0].minor.yy380 = OP_TYPE_MATCH; } + case 365: /* compare_op ::= MATCH */ +{ yymsp[0].minor.yy324 = OP_TYPE_MATCH; } break; - case 360: /* compare_op ::= NMATCH */ -{ yymsp[0].minor.yy380 = OP_TYPE_NMATCH; } + case 366: /* compare_op ::= NMATCH */ +{ yymsp[0].minor.yy324 = OP_TYPE_NMATCH; } break; - case 361: /* compare_op ::= CONTAINS */ -{ yymsp[0].minor.yy380 = OP_TYPE_JSON_CONTAINS; } + case 367: /* compare_op ::= CONTAINS */ +{ yymsp[0].minor.yy324 = OP_TYPE_JSON_CONTAINS; } break; - case 362: /* in_op ::= IN */ -{ yymsp[0].minor.yy380 = OP_TYPE_IN; } + case 368: /* in_op ::= IN */ +{ yymsp[0].minor.yy324 = OP_TYPE_IN; } break; - case 363: /* in_op ::= NOT IN */ -{ yymsp[-1].minor.yy380 = OP_TYPE_NOT_IN; } + case 369: /* in_op ::= NOT IN */ +{ yymsp[-1].minor.yy324 = OP_TYPE_NOT_IN; } break; - case 364: /* in_predicate_value ::= NK_LP expression_list NK_RP */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, createNodeListNode(pCxt, yymsp[-1].minor.yy424)); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 370: /* in_predicate_value ::= NK_LP expression_list NK_RP */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, createNodeListNode(pCxt, yymsp[-1].minor.yy600)); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 366: /* boolean_value_expression ::= NOT boolean_primary */ + case 372: /* boolean_value_expression ::= NOT boolean_primary */ { - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_NOT, releaseRawExprNode(pCxt, yymsp[0].minor.yy632), NULL)); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-1].minor.yy0, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_NOT, releaseRawExprNode(pCxt, yymsp[0].minor.yy392), NULL)); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 367: /* boolean_value_expression ::= boolean_value_expression OR boolean_value_expression */ + case 373: /* boolean_value_expression ::= boolean_value_expression OR boolean_value_expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_OR, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_OR, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 368: /* boolean_value_expression ::= boolean_value_expression AND boolean_value_expression */ + case 374: /* boolean_value_expression ::= boolean_value_expression AND boolean_value_expression */ { - SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy632); - SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy632); - yylhsminor.yy632 = createRawExprNodeExt(pCxt, &s, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_AND, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); + SToken s = getTokenFromRawExprNode(pCxt, yymsp[-2].minor.yy392); + SToken e = getTokenFromRawExprNode(pCxt, yymsp[0].minor.yy392); + yylhsminor.yy392 = createRawExprNodeExt(pCxt, &s, &e, createLogicConditionNode(pCxt, LOGIC_COND_TYPE_AND, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 375: /* table_reference_list ::= table_reference_list NK_COMMA table_reference */ -{ yylhsminor.yy632 = createJoinTableNode(pCxt, JOIN_TYPE_INNER, yymsp[-2].minor.yy632, yymsp[0].minor.yy632, NULL); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 381: /* table_reference_list ::= table_reference_list NK_COMMA table_reference */ +{ yylhsminor.yy392 = createJoinTableNode(pCxt, JOIN_TYPE_INNER, yymsp[-2].minor.yy392, yymsp[0].minor.yy392, NULL); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 378: /* table_primary ::= table_name alias_opt */ -{ yylhsminor.yy632 = createRealTableNode(pCxt, NULL, &yymsp[-1].minor.yy209, &yymsp[0].minor.yy209); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + case 384: /* table_primary ::= table_name alias_opt */ +{ yylhsminor.yy392 = createRealTableNode(pCxt, NULL, &yymsp[-1].minor.yy57, &yymsp[0].minor.yy57); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 379: /* table_primary ::= db_name NK_DOT table_name alias_opt */ -{ yylhsminor.yy632 = createRealTableNode(pCxt, &yymsp[-3].minor.yy209, &yymsp[-1].minor.yy209, &yymsp[0].minor.yy209); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + case 385: /* table_primary ::= db_name NK_DOT table_name alias_opt */ +{ yylhsminor.yy392 = createRealTableNode(pCxt, &yymsp[-3].minor.yy57, &yymsp[-1].minor.yy57, &yymsp[0].minor.yy57); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 380: /* table_primary ::= subquery alias_opt */ -{ yylhsminor.yy632 = createTempTableNode(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy632), &yymsp[0].minor.yy209); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + case 386: /* table_primary ::= subquery alias_opt */ +{ yylhsminor.yy392 = createTempTableNode(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy392), &yymsp[0].minor.yy57); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 382: /* alias_opt ::= */ -{ yymsp[1].minor.yy209 = nil_token; } + case 388: /* alias_opt ::= */ +{ yymsp[1].minor.yy57 = nil_token; } break; - case 383: /* alias_opt ::= table_alias */ -{ yylhsminor.yy209 = yymsp[0].minor.yy209; } - yymsp[0].minor.yy209 = yylhsminor.yy209; + case 389: /* alias_opt ::= table_alias */ +{ yylhsminor.yy57 = yymsp[0].minor.yy57; } + yymsp[0].minor.yy57 = yylhsminor.yy57; break; - case 384: /* alias_opt ::= AS table_alias */ -{ yymsp[-1].minor.yy209 = yymsp[0].minor.yy209; } + case 390: /* alias_opt ::= AS table_alias */ +{ yymsp[-1].minor.yy57 = yymsp[0].minor.yy57; } break; - case 385: /* parenthesized_joined_table ::= NK_LP joined_table NK_RP */ - case 386: /* parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP */ yytestcase(yyruleno==386); -{ yymsp[-2].minor.yy632 = yymsp[-1].minor.yy632; } + case 391: /* parenthesized_joined_table ::= NK_LP joined_table NK_RP */ + case 392: /* parenthesized_joined_table ::= NK_LP parenthesized_joined_table NK_RP */ yytestcase(yyruleno==392); +{ yymsp[-2].minor.yy392 = yymsp[-1].minor.yy392; } break; - case 387: /* joined_table ::= table_reference join_type JOIN table_reference ON search_condition */ -{ yylhsminor.yy632 = createJoinTableNode(pCxt, yymsp[-4].minor.yy612, yymsp[-5].minor.yy632, yymsp[-2].minor.yy632, yymsp[0].minor.yy632); } - yymsp[-5].minor.yy632 = yylhsminor.yy632; + case 393: /* joined_table ::= table_reference join_type JOIN table_reference ON search_condition */ +{ yylhsminor.yy392 = createJoinTableNode(pCxt, yymsp[-4].minor.yy204, yymsp[-5].minor.yy392, yymsp[-2].minor.yy392, yymsp[0].minor.yy392); } + yymsp[-5].minor.yy392 = yylhsminor.yy392; break; - case 388: /* join_type ::= */ -{ yymsp[1].minor.yy612 = JOIN_TYPE_INNER; } + case 394: /* join_type ::= */ +{ yymsp[1].minor.yy204 = JOIN_TYPE_INNER; } break; - case 389: /* join_type ::= INNER */ -{ yymsp[0].minor.yy612 = JOIN_TYPE_INNER; } + case 395: /* join_type ::= INNER */ +{ yymsp[0].minor.yy204 = JOIN_TYPE_INNER; } break; - case 390: /* query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt twindow_clause_opt group_by_clause_opt having_clause_opt */ + case 396: /* query_specification ::= SELECT set_quantifier_opt select_list from_clause where_clause_opt partition_by_clause_opt range_opt every_opt fill_opt twindow_clause_opt group_by_clause_opt having_clause_opt */ { - yymsp[-8].minor.yy632 = createSelectStmt(pCxt, yymsp[-7].minor.yy137, yymsp[-6].minor.yy424, yymsp[-5].minor.yy632); - yymsp[-8].minor.yy632 = addWhereClause(pCxt, yymsp[-8].minor.yy632, yymsp[-4].minor.yy632); - yymsp[-8].minor.yy632 = addPartitionByClause(pCxt, yymsp[-8].minor.yy632, yymsp[-3].minor.yy424); - yymsp[-8].minor.yy632 = addWindowClauseClause(pCxt, yymsp[-8].minor.yy632, yymsp[-2].minor.yy632); - yymsp[-8].minor.yy632 = addGroupByClause(pCxt, yymsp[-8].minor.yy632, yymsp[-1].minor.yy424); - yymsp[-8].minor.yy632 = addHavingClause(pCxt, yymsp[-8].minor.yy632, yymsp[0].minor.yy632); + yymsp[-11].minor.yy392 = createSelectStmt(pCxt, yymsp[-10].minor.yy481, yymsp[-9].minor.yy600, yymsp[-8].minor.yy392); + yymsp[-11].minor.yy392 = addWhereClause(pCxt, yymsp[-11].minor.yy392, yymsp[-7].minor.yy392); + yymsp[-11].minor.yy392 = addPartitionByClause(pCxt, yymsp[-11].minor.yy392, yymsp[-6].minor.yy600); + yymsp[-11].minor.yy392 = addWindowClauseClause(pCxt, yymsp[-11].minor.yy392, yymsp[-2].minor.yy392); + yymsp[-11].minor.yy392 = addGroupByClause(pCxt, yymsp[-11].minor.yy392, yymsp[-1].minor.yy600); + yymsp[-11].minor.yy392 = addHavingClause(pCxt, yymsp[-11].minor.yy392, yymsp[0].minor.yy392); + yymsp[-11].minor.yy392 = addRangeClause(pCxt, yymsp[-11].minor.yy392, yymsp[-5].minor.yy392); + yymsp[-11].minor.yy392 = addEveryClause(pCxt, yymsp[-11].minor.yy392, yymsp[-4].minor.yy392); + yymsp[-11].minor.yy392 = addFillClause(pCxt, yymsp[-11].minor.yy392, yymsp[-3].minor.yy392); } break; - case 393: /* set_quantifier_opt ::= ALL */ -{ yymsp[0].minor.yy137 = false; } + case 399: /* set_quantifier_opt ::= ALL */ +{ yymsp[0].minor.yy481 = false; } + break; + case 400: /* select_list ::= NK_STAR */ +{ yymsp[0].minor.yy600 = NULL; } break; - case 394: /* select_list ::= NK_STAR */ -{ yymsp[0].minor.yy424 = NULL; } + case 405: /* select_item ::= common_expression column_alias */ +{ yylhsminor.yy392 = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy392), &yymsp[0].minor.yy57); } + yymsp[-1].minor.yy392 = yylhsminor.yy392; break; - case 399: /* select_item ::= common_expression column_alias */ -{ yylhsminor.yy632 = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy632), &yymsp[0].minor.yy209); } - yymsp[-1].minor.yy632 = yylhsminor.yy632; + case 406: /* select_item ::= common_expression AS column_alias */ +{ yylhsminor.yy392 = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), &yymsp[0].minor.yy57); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 400: /* select_item ::= common_expression AS column_alias */ -{ yylhsminor.yy632 = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), &yymsp[0].minor.yy209); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 411: /* partition_by_clause_opt ::= PARTITION BY expression_list */ + case 428: /* group_by_clause_opt ::= GROUP BY group_by_list */ yytestcase(yyruleno==428); + case 444: /* order_by_clause_opt ::= ORDER BY sort_specification_list */ yytestcase(yyruleno==444); +{ yymsp[-2].minor.yy600 = yymsp[0].minor.yy600; } break; - case 405: /* partition_by_clause_opt ::= PARTITION BY expression_list */ - case 422: /* group_by_clause_opt ::= GROUP BY group_by_list */ yytestcase(yyruleno==422); - case 434: /* order_by_clause_opt ::= ORDER BY sort_specification_list */ yytestcase(yyruleno==434); -{ yymsp[-2].minor.yy424 = yymsp[0].minor.yy424; } + case 413: /* twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP */ +{ yymsp[-5].minor.yy392 = createSessionWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), releaseRawExprNode(pCxt, yymsp[-1].minor.yy392)); } break; - case 407: /* twindow_clause_opt ::= SESSION NK_LP column_reference NK_COMMA duration_literal NK_RP */ -{ yymsp[-5].minor.yy632 = createSessionWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy632), releaseRawExprNode(pCxt, yymsp[-1].minor.yy632)); } + case 414: /* twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP */ +{ yymsp[-3].minor.yy392 = createStateWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy392)); } break; - case 408: /* twindow_clause_opt ::= STATE_WINDOW NK_LP expression NK_RP */ -{ yymsp[-3].minor.yy632 = createStateWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-1].minor.yy632)); } + case 415: /* twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt */ +{ yymsp[-5].minor.yy392 = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), NULL, yymsp[-1].minor.yy392, yymsp[0].minor.yy392); } break; - case 409: /* twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_RP sliding_opt fill_opt */ -{ yymsp[-5].minor.yy632 = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy632), NULL, yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } + case 416: /* twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt */ +{ yymsp[-7].minor.yy392 = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-5].minor.yy392), releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), yymsp[-1].minor.yy392, yymsp[0].minor.yy392); } break; - case 410: /* twindow_clause_opt ::= INTERVAL NK_LP duration_literal NK_COMMA duration_literal NK_RP sliding_opt fill_opt */ -{ yymsp[-7].minor.yy632 = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, yymsp[-5].minor.yy632), releaseRawExprNode(pCxt, yymsp[-3].minor.yy632), yymsp[-1].minor.yy632, yymsp[0].minor.yy632); } + case 418: /* sliding_opt ::= SLIDING NK_LP duration_literal NK_RP */ + case 436: /* every_opt ::= EVERY NK_LP duration_literal NK_RP */ yytestcase(yyruleno==436); +{ yymsp[-3].minor.yy392 = releaseRawExprNode(pCxt, yymsp[-1].minor.yy392); } break; - case 412: /* sliding_opt ::= SLIDING NK_LP duration_literal NK_RP */ -{ yymsp[-3].minor.yy632 = releaseRawExprNode(pCxt, yymsp[-1].minor.yy632); } + case 420: /* fill_opt ::= FILL NK_LP fill_mode NK_RP */ +{ yymsp[-3].minor.yy392 = createFillNode(pCxt, yymsp[-1].minor.yy270, NULL); } break; - case 414: /* fill_opt ::= FILL NK_LP fill_mode NK_RP */ -{ yymsp[-3].minor.yy632 = createFillNode(pCxt, yymsp[-1].minor.yy54, NULL); } + case 421: /* fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP */ +{ yymsp[-5].minor.yy392 = createFillNode(pCxt, FILL_MODE_VALUE, createNodeListNode(pCxt, yymsp[-1].minor.yy600)); } break; - case 415: /* fill_opt ::= FILL NK_LP VALUE NK_COMMA literal_list NK_RP */ -{ yymsp[-5].minor.yy632 = createFillNode(pCxt, FILL_MODE_VALUE, createNodeListNode(pCxt, yymsp[-1].minor.yy424)); } + case 422: /* fill_mode ::= NONE */ +{ yymsp[0].minor.yy270 = FILL_MODE_NONE; } break; - case 416: /* fill_mode ::= NONE */ -{ yymsp[0].minor.yy54 = FILL_MODE_NONE; } + case 423: /* fill_mode ::= PREV */ +{ yymsp[0].minor.yy270 = FILL_MODE_PREV; } break; - case 417: /* fill_mode ::= PREV */ -{ yymsp[0].minor.yy54 = FILL_MODE_PREV; } + case 424: /* fill_mode ::= NULL */ +{ yymsp[0].minor.yy270 = FILL_MODE_NULL; } break; - case 418: /* fill_mode ::= NULL */ -{ yymsp[0].minor.yy54 = FILL_MODE_NULL; } + case 425: /* fill_mode ::= LINEAR */ +{ yymsp[0].minor.yy270 = FILL_MODE_LINEAR; } break; - case 419: /* fill_mode ::= LINEAR */ -{ yymsp[0].minor.yy54 = FILL_MODE_LINEAR; } + case 426: /* fill_mode ::= NEXT */ +{ yymsp[0].minor.yy270 = FILL_MODE_NEXT; } break; - case 420: /* fill_mode ::= NEXT */ -{ yymsp[0].minor.yy54 = FILL_MODE_NEXT; } + case 429: /* group_by_list ::= expression */ +{ yylhsminor.yy600 = createNodeList(pCxt, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } + yymsp[0].minor.yy600 = yylhsminor.yy600; break; - case 423: /* group_by_list ::= expression */ -{ yylhsminor.yy424 = createNodeList(pCxt, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); } - yymsp[0].minor.yy424 = yylhsminor.yy424; + case 430: /* group_by_list ::= group_by_list NK_COMMA expression */ +{ yylhsminor.yy600 = addNodeToList(pCxt, yymsp[-2].minor.yy600, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy392))); } + yymsp[-2].minor.yy600 = yylhsminor.yy600; break; - case 424: /* group_by_list ::= group_by_list NK_COMMA expression */ -{ yylhsminor.yy424 = addNodeToList(pCxt, yymsp[-2].minor.yy424, createGroupingSetNode(pCxt, releaseRawExprNode(pCxt, yymsp[0].minor.yy632))); } - yymsp[-2].minor.yy424 = yylhsminor.yy424; + case 434: /* range_opt ::= RANGE NK_LP expression NK_COMMA expression NK_RP */ +{ yymsp[-5].minor.yy392 = createInterpTimeRange(pCxt, releaseRawExprNode(pCxt, yymsp[-3].minor.yy392), releaseRawExprNode(pCxt, yymsp[-1].minor.yy392)); } break; - case 427: /* query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt */ + case 437: /* query_expression ::= query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt */ { - yylhsminor.yy632 = addOrderByClause(pCxt, yymsp[-3].minor.yy632, yymsp[-2].minor.yy424); - yylhsminor.yy632 = addSlimitClause(pCxt, yylhsminor.yy632, yymsp[-1].minor.yy632); - yylhsminor.yy632 = addLimitClause(pCxt, yylhsminor.yy632, yymsp[0].minor.yy632); + yylhsminor.yy392 = addOrderByClause(pCxt, yymsp[-3].minor.yy392, yymsp[-2].minor.yy600); + yylhsminor.yy392 = addSlimitClause(pCxt, yylhsminor.yy392, yymsp[-1].minor.yy392); + yylhsminor.yy392 = addLimitClause(pCxt, yylhsminor.yy392, yymsp[0].minor.yy392); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 429: /* query_expression_body ::= query_expression_body UNION ALL query_expression_body */ -{ yylhsminor.yy632 = createSetOperator(pCxt, SET_OP_TYPE_UNION_ALL, yymsp[-3].minor.yy632, yymsp[0].minor.yy632); } - yymsp[-3].minor.yy632 = yylhsminor.yy632; + case 439: /* query_expression_body ::= query_expression_body UNION ALL query_expression_body */ +{ yylhsminor.yy392 = createSetOperator(pCxt, SET_OP_TYPE_UNION_ALL, yymsp[-3].minor.yy392, yymsp[0].minor.yy392); } + yymsp[-3].minor.yy392 = yylhsminor.yy392; break; - case 430: /* query_expression_body ::= query_expression_body UNION query_expression_body */ -{ yylhsminor.yy632 = createSetOperator(pCxt, SET_OP_TYPE_UNION, yymsp[-2].minor.yy632, yymsp[0].minor.yy632); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 440: /* query_expression_body ::= query_expression_body UNION query_expression_body */ +{ yylhsminor.yy392 = createSetOperator(pCxt, SET_OP_TYPE_UNION, yymsp[-2].minor.yy392, yymsp[0].minor.yy392); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 432: /* query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP */ -{ yymsp[-5].minor.yy632 = yymsp[-4].minor.yy632; } - yy_destructor(yypParser,352,&yymsp[-3].minor); - yy_destructor(yypParser,353,&yymsp[-2].minor); - yy_destructor(yypParser,354,&yymsp[-1].minor); + case 442: /* query_primary ::= NK_LP query_expression_body order_by_clause_opt slimit_clause_opt limit_clause_opt NK_RP */ +{ yymsp[-5].minor.yy392 = yymsp[-4].minor.yy392; } + yy_destructor(yypParser,356,&yymsp[-3].minor); + yy_destructor(yypParser,357,&yymsp[-2].minor); + yy_destructor(yypParser,358,&yymsp[-1].minor); break; - case 436: /* slimit_clause_opt ::= SLIMIT NK_INTEGER */ - case 440: /* limit_clause_opt ::= LIMIT NK_INTEGER */ yytestcase(yyruleno==440); -{ yymsp[-1].minor.yy632 = createLimitNode(pCxt, &yymsp[0].minor.yy0, NULL); } + case 446: /* slimit_clause_opt ::= SLIMIT NK_INTEGER */ + case 450: /* limit_clause_opt ::= LIMIT NK_INTEGER */ yytestcase(yyruleno==450); +{ yymsp[-1].minor.yy392 = createLimitNode(pCxt, &yymsp[0].minor.yy0, NULL); } break; - case 437: /* slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER */ - case 441: /* limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER */ yytestcase(yyruleno==441); -{ yymsp[-3].minor.yy632 = createLimitNode(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0); } + case 447: /* slimit_clause_opt ::= SLIMIT NK_INTEGER SOFFSET NK_INTEGER */ + case 451: /* limit_clause_opt ::= LIMIT NK_INTEGER OFFSET NK_INTEGER */ yytestcase(yyruleno==451); +{ yymsp[-3].minor.yy392 = createLimitNode(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0); } break; - case 438: /* slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER */ - case 442: /* limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER */ yytestcase(yyruleno==442); -{ yymsp[-3].minor.yy632 = createLimitNode(pCxt, &yymsp[0].minor.yy0, &yymsp[-2].minor.yy0); } + case 448: /* slimit_clause_opt ::= SLIMIT NK_INTEGER NK_COMMA NK_INTEGER */ + case 452: /* limit_clause_opt ::= LIMIT NK_INTEGER NK_COMMA NK_INTEGER */ yytestcase(yyruleno==452); +{ yymsp[-3].minor.yy392 = createLimitNode(pCxt, &yymsp[0].minor.yy0, &yymsp[-2].minor.yy0); } break; - case 443: /* subquery ::= NK_LP query_expression NK_RP */ -{ yylhsminor.yy632 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, yymsp[-1].minor.yy632); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 453: /* subquery ::= NK_LP query_expression NK_RP */ +{ yylhsminor.yy392 = createRawExprNodeExt(pCxt, &yymsp[-2].minor.yy0, &yymsp[0].minor.yy0, yymsp[-1].minor.yy392); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 447: /* sort_specification ::= expression ordering_specification_opt null_ordering_opt */ -{ yylhsminor.yy632 = createOrderByExprNode(pCxt, releaseRawExprNode(pCxt, yymsp[-2].minor.yy632), yymsp[-1].minor.yy578, yymsp[0].minor.yy217); } - yymsp[-2].minor.yy632 = yylhsminor.yy632; + case 457: /* sort_specification ::= expression ordering_specification_opt null_ordering_opt */ +{ yylhsminor.yy392 = createOrderByExprNode(pCxt, releaseRawExprNode(pCxt, yymsp[-2].minor.yy392), yymsp[-1].minor.yy162, yymsp[0].minor.yy529); } + yymsp[-2].minor.yy392 = yylhsminor.yy392; break; - case 448: /* ordering_specification_opt ::= */ -{ yymsp[1].minor.yy578 = ORDER_ASC; } + case 458: /* ordering_specification_opt ::= */ +{ yymsp[1].minor.yy162 = ORDER_ASC; } break; - case 449: /* ordering_specification_opt ::= ASC */ -{ yymsp[0].minor.yy578 = ORDER_ASC; } + case 459: /* ordering_specification_opt ::= ASC */ +{ yymsp[0].minor.yy162 = ORDER_ASC; } break; - case 450: /* ordering_specification_opt ::= DESC */ -{ yymsp[0].minor.yy578 = ORDER_DESC; } + case 460: /* ordering_specification_opt ::= DESC */ +{ yymsp[0].minor.yy162 = ORDER_DESC; } break; - case 451: /* null_ordering_opt ::= */ -{ yymsp[1].minor.yy217 = NULL_ORDER_DEFAULT; } + case 461: /* null_ordering_opt ::= */ +{ yymsp[1].minor.yy529 = NULL_ORDER_DEFAULT; } break; - case 452: /* null_ordering_opt ::= NULLS FIRST */ -{ yymsp[-1].minor.yy217 = NULL_ORDER_FIRST; } + case 462: /* null_ordering_opt ::= NULLS FIRST */ +{ yymsp[-1].minor.yy529 = NULL_ORDER_FIRST; } break; - case 453: /* null_ordering_opt ::= NULLS LAST */ -{ yymsp[-1].minor.yy217 = NULL_ORDER_LAST; } + case 463: /* null_ordering_opt ::= NULLS LAST */ +{ yymsp[-1].minor.yy529 = NULL_ORDER_LAST; } break; default: break; diff --git a/source/libs/parser/test/mockCatalogService.cpp b/source/libs/parser/test/mockCatalogService.cpp index 68f9e9d36d517a0cbfd34417c8276aadd680f49a..9758da78991adccfffedaa06ba4b0f0d7f58ac2f 100644 --- a/source/libs/parser/test/mockCatalogService.cpp +++ b/source/libs/parser/test/mockCatalogService.cpp @@ -285,7 +285,7 @@ class MockCatalogServiceImpl { } void createSmaIndex(const SMCreateSmaReq* pReq) { - STableIndexInfo info; + STableIndexInfo info = {0}; info.intervalUnit = pReq->intervalUnit; info.slidingUnit = pReq->slidingUnit; info.interval = pReq->interval; diff --git a/source/libs/parser/test/parInitialATest.cpp b/source/libs/parser/test/parInitialATest.cpp index cc23fbbc60ce2ff8b457af05f78721bc9f1c7563..021348dcc7310c819a1745fa244bbe158385d3c0 100644 --- a/source/libs/parser/test/parInitialATest.cpp +++ b/source/libs/parser/test/parInitialATest.cpp @@ -43,7 +43,40 @@ TEST_F(ParserInitialATest, alterDatabase) { run("ALTER DATABASE wxy_db KEEP 2400"); } -// todo ALTER local +TEST_F(ParserInitialATest, alterLocal) { + useDb("root", "test"); + + pair expect; + + auto clearAlterLocal = [&]() { + expect.first.clear(); + expect.second.clear(); + }; + + auto setAlterLocalFunc = [&](const char* pConfig, const char* pValue = nullptr) { + expect.first.assign(pConfig); + if (nullptr != pValue) { + expect.second.assign(pValue); + } + }; + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_ALTER_LOCAL_STMT); + ASSERT_EQ(pQuery->execMode, QUERY_EXEC_MODE_LOCAL); + SAlterLocalStmt* pStmt = (SAlterLocalStmt*)pQuery->pRoot; + ASSERT_EQ(string(pStmt->config), expect.first); + ASSERT_EQ(string(pStmt->value), expect.second); + }); + + setAlterLocalFunc("resetlog"); + run("ALTER LOCAL 'resetlog'"); + clearAlterLocal(); + + setAlterLocalFunc("querypolicy", "2"); + run("ALTER LOCAL 'querypolicy' '2'"); + clearAlterLocal(); +} + // todo ALTER stable /* @@ -88,10 +121,10 @@ TEST_F(ParserInitialATest, alterSTable) { int32_t len = snprintf(expect.name, sizeof(expect.name), "0.test.%s", pTbname); expect.name[len] = '\0'; expect.alterType = alterType; - expect.ttl = ttl; +// expect.ttl = ttl; if (nullptr != pComment) { expect.comment = strdup(pComment); - expect.commentLen = strlen(pComment) + 1; + expect.commentLen = strlen(pComment); } expect.numOfFields = numOfFields; @@ -147,9 +180,9 @@ TEST_F(ParserInitialATest, alterSTable) { tFreeSMAltertbReq(&req); }); - setAlterStbReqFunc("st1", TSDB_ALTER_TABLE_UPDATE_OPTIONS, 0, nullptr, 0, 0, nullptr, nullptr, 10); - run("ALTER TABLE st1 TTL 10"); - clearAlterStbReq(); +// setAlterStbReqFunc("st1", TSDB_ALTER_TABLE_UPDATE_OPTIONS, 0, nullptr, 0, 0, nullptr, nullptr, 10); +// run("ALTER TABLE st1 TTL 10"); +// clearAlterStbReq(); setAlterStbReqFunc("st1", TSDB_ALTER_TABLE_UPDATE_OPTIONS, 0, nullptr, 0, 0, nullptr, "test"); run("ALTER TABLE st1 COMMENT 'test'"); @@ -256,7 +289,7 @@ TEST_F(ParserInitialATest, alterTable) { expect.newTTL = ttl; } if (nullptr != pComment) { - expect.updateComment = true; + expect.newCommentLen = strlen(pComment); expect.newComment = pComment; } }; @@ -295,9 +328,10 @@ TEST_F(ParserInitialATest, alterTable) { ASSERT_EQ(memcmp(req.pTagVal, expect.pTagVal, expect.nTagVal), 0); ASSERT_EQ(req.updateTTL, expect.updateTTL); ASSERT_EQ(req.newTTL, expect.newTTL); - ASSERT_EQ(req.updateComment, expect.updateComment); if (nullptr != expect.newComment) { ASSERT_EQ(std::string(req.newComment), std::string(expect.newComment)); + ASSERT_EQ(req.newCommentLen, strlen(req.newComment)); + ASSERT_EQ(expect.newCommentLen, strlen(expect.newComment)); } tDecoderClear(&coder); diff --git a/source/libs/parser/test/parInitialCTest.cpp b/source/libs/parser/test/parInitialCTest.cpp index 8b2c2aade7ae7df398841279184da53e15608825..d188744adf57385980279d6490f1d0e86db7ddac 100644 --- a/source/libs/parser/test/parInitialCTest.cpp +++ b/source/libs/parser/test/parInitialCTest.cpp @@ -14,7 +14,6 @@ */ #include "parTestUtil.h" -#include "ttime.h" using namespace std; @@ -242,9 +241,47 @@ TEST_F(ParserInitialCTest, createDatabaseSemanticCheck) { TEST_F(ParserInitialCTest, createDnode) { useDb("root", "test"); - run("CREATE DNODE abc1 PORT 7000"); + SCreateDnodeReq expect = {0}; - run("CREATE DNODE 1.1.1.1 PORT 9000"); + auto clearCreateDnodeReq = [&]() { memset(&expect, 0, sizeof(SCreateDnodeReq)); }; + + auto setCreateDnodeReqFunc = [&](const char* pFqdn, int32_t port = tsServerPort) { + strcpy(expect.fqdn, pFqdn); + expect.port = port; + }; + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_CREATE_DNODE_STMT); + SCreateDnodeReq req = {0}; + ASSERT_TRUE(TSDB_CODE_SUCCESS == tDeserializeSCreateDnodeReq(pQuery->pCmdMsg->pMsg, pQuery->pCmdMsg->msgLen, &req)); + + ASSERT_EQ(std::string(req.fqdn), std::string(expect.fqdn)); + ASSERT_EQ(req.port, expect.port); + }); + + setCreateDnodeReqFunc("abc1", 7030); + run("CREATE DNODE 'abc1' PORT 7030"); + clearCreateDnodeReq(); + + setCreateDnodeReqFunc("1.1.1.1", 8030); + run("CREATE DNODE 1.1.1.1 PORT 8030"); + clearCreateDnodeReq(); + + setCreateDnodeReqFunc("host1", 9030); + run("CREATE DNODE host1 PORT 9030"); + clearCreateDnodeReq(); + + setCreateDnodeReqFunc("abc2", 7040); + run("CREATE DNODE 'abc2:7040'"); + clearCreateDnodeReq(); + + setCreateDnodeReqFunc("1.1.1.2"); + run("CREATE DNODE 1.1.1.2"); + clearCreateDnodeReq(); + + setCreateDnodeReqFunc("host2"); + run("CREATE DNODE host2"); + clearCreateDnodeReq(); } // CREATE [AGGREGATE] FUNCTION [IF NOT EXISTS] func_name AS library_path OUTPUTTYPE type_name [BUFSIZE value] @@ -322,17 +359,21 @@ TEST_F(ParserInitialCTest, createStable) { memset(&expect, 0, sizeof(SMCreateStbReq)); }; - auto setCreateStbReqFunc = [&](const char* pTbname, int8_t igExists = 0, - float xFilesFactor = TSDB_DEFAULT_ROLLUP_FILE_FACTOR, + auto setCreateStbReqFunc = [&](const char* pTbname, int8_t igExists = 0, int64_t delay1 = -1, int64_t delay2 = -1, + int64_t watermark1 = TSDB_DEFAULT_ROLLUP_WATERMARK, + int64_t watermark2 = TSDB_DEFAULT_ROLLUP_WATERMARK, int32_t ttl = TSDB_DEFAULT_TABLE_TTL, const char* pComment = nullptr) { int32_t len = snprintf(expect.name, sizeof(expect.name), "0.test.%s", pTbname); expect.name[len] = '\0'; expect.igExists = igExists; - expect.xFilesFactor = xFilesFactor; - expect.ttl = ttl; + expect.delay1 = delay1; + expect.delay2 = delay2; + expect.watermark1 = watermark1; + expect.watermark2 = watermark2; +// expect.ttl = ttl; if (nullptr != pComment) { expect.comment = strdup(pComment); - expect.commentLen = strlen(pComment) + 1; + expect.commentLen = strlen(pComment); } }; @@ -366,12 +407,14 @@ TEST_F(ParserInitialCTest, createStable) { ASSERT_EQ(std::string(req.name), std::string(expect.name)); ASSERT_EQ(req.igExists, expect.igExists); - ASSERT_EQ(req.xFilesFactor, expect.xFilesFactor); - ASSERT_EQ(req.delay, expect.delay); + ASSERT_EQ(req.delay1, expect.delay1); + ASSERT_EQ(req.delay2, expect.delay2); + ASSERT_EQ(req.watermark1, expect.watermark1); + ASSERT_EQ(req.watermark2, expect.watermark2); ASSERT_EQ(req.ttl, expect.ttl); ASSERT_EQ(req.numOfColumns, expect.numOfColumns); ASSERT_EQ(req.numOfTags, expect.numOfTags); - ASSERT_EQ(req.commentLen, expect.commentLen); +// ASSERT_EQ(req.commentLen, expect.commentLen); ASSERT_EQ(req.ast1Len, expect.ast1Len); ASSERT_EQ(req.ast2Len, expect.ast2Len); @@ -418,7 +461,8 @@ TEST_F(ParserInitialCTest, createStable) { run("CREATE STABLE t1(ts TIMESTAMP, c1 INT) TAGS(id INT)"); clearCreateStbReq(); - setCreateStbReqFunc("t1", 1, 0.1, 100, "test create table"); + setCreateStbReqFunc("t1", 1, 100 * MILLISECOND_PER_SECOND, 10 * MILLISECOND_PER_MINUTE, 10, + 1 * MILLISECOND_PER_MINUTE, 100, "test create table"); addFieldToCreateStbReqFunc(true, "ts", TSDB_DATA_TYPE_TIMESTAMP, 0, 0); addFieldToCreateStbReqFunc(true, "c1", TSDB_DATA_TYPE_INT); addFieldToCreateStbReqFunc(true, "c2", TSDB_DATA_TYPE_UINT); @@ -456,15 +500,20 @@ TEST_F(ParserInitialCTest, createStable) { "TAGS (a1 TIMESTAMP, a2 INT, a3 INT UNSIGNED, a4 BIGINT, a5 BIGINT UNSIGNED, a6 FLOAT, a7 DOUBLE, " "a8 BINARY(20), a9 SMALLINT, a10 SMALLINT UNSIGNED COMMENT 'test column comment', a11 TINYINT, " "a12 TINYINT UNSIGNED, a13 BOOL, a14 NCHAR(30), a15 VARCHAR(50)) " - "TTL 100 COMMENT 'test create table' SMA(c1, c2, c3) ROLLUP (MIN) FILE_FACTOR 0.1"); + "TTL 100 COMMENT 'test create table' SMA(c1, c2, c3) ROLLUP (MIN) MAX_DELAY 100s,10m WATERMARK 10a,1m"); clearCreateStbReq(); } TEST_F(ParserInitialCTest, createStableSemanticCheck) { useDb("root", "test"); - run("CREATE STABLE stb2 (ts TIMESTAMP, c1 INT) TAGS (tag1 INT) ROLLUP(CEIL) FILE_FACTOR 0.1", - TSDB_CODE_PAR_INVALID_ROLLUP_OPTION, PARSER_STAGE_TRANSLATE); + run("CREATE STABLE stb2 (ts TIMESTAMP, c1 INT) TAGS (tag1 INT) ROLLUP(CEIL)", TSDB_CODE_PAR_INVALID_ROLLUP_OPTION); + + run("CREATE STABLE stb2 (ts TIMESTAMP, c1 INT) TAGS (tag1 INT) ROLLUP(MAX) MAX_DELAY 0s WATERMARK 1m", + TSDB_CODE_PAR_INVALID_RANGE_OPTION); + + run("CREATE STABLE stb2 (ts TIMESTAMP, c1 INT) TAGS (tag1 INT) ROLLUP(MAX) MAX_DELAY 10s WATERMARK 18m", + TSDB_CODE_PAR_INVALID_RANGE_OPTION); } TEST_F(ParserInitialCTest, createStream) { @@ -477,7 +526,7 @@ TEST_F(ParserInitialCTest, createStream) { memset(&expect, 0, sizeof(SCMCreateStreamReq)); }; - auto setCreateStbReqFunc = + auto setCreateStreamReqFunc = [&](const char* pStream, const char* pSrcDb, const char* pSql, const char* pDstStb = nullptr, int8_t igExists = 0, int8_t triggerType = STREAM_TRIGGER_AT_ONCE, int64_t maxDelay = 0, int64_t watermark = 0) { snprintf(expect.name, sizeof(expect.name), "0.%s", pStream); @@ -509,25 +558,31 @@ TEST_F(ParserInitialCTest, createStream) { tFreeSCMCreateStreamReq(&req); }); - setCreateStbReqFunc("s1", "test", "create stream s1 as select * from t1"); + setCreateStreamReqFunc("s1", "test", "create stream s1 as select * from t1"); run("CREATE STREAM s1 AS SELECT * FROM t1"); clearCreateStreamReq(); - setCreateStbReqFunc("s1", "test", "create stream if not exists s1 as select * from t1", nullptr, 1); + setCreateStreamReqFunc("s1", "test", "create stream if not exists s1 as select * from t1", nullptr, 1); run("CREATE STREAM IF NOT EXISTS s1 AS SELECT * FROM t1"); clearCreateStreamReq(); - setCreateStbReqFunc("s1", "test", "create stream s1 into st1 as select * from t1", "st1"); + setCreateStreamReqFunc("s1", "test", "create stream s1 into st1 as select * from t1", "st1"); run("CREATE STREAM s1 INTO st1 AS SELECT * FROM t1"); clearCreateStreamReq(); - setCreateStbReqFunc("s1", "test", - "create stream if not exists s1 trigger max_delay 20s watermark 10s into st1 as select * from t1", - "st1", 1, STREAM_TRIGGER_MAX_DELAY, 20 * MILLISECOND_PER_SECOND, 10 * MILLISECOND_PER_SECOND); + setCreateStreamReqFunc( + "s1", "test", "create stream if not exists s1 trigger max_delay 20s watermark 10s into st1 as select * from t1", + "st1", 1, STREAM_TRIGGER_MAX_DELAY, 20 * MILLISECOND_PER_SECOND, 10 * MILLISECOND_PER_SECOND); run("CREATE STREAM IF NOT EXISTS s1 TRIGGER MAX_DELAY 20s WATERMARK 10s INTO st1 AS SELECT * FROM t1"); clearCreateStreamReq(); } +TEST_F(ParserInitialCTest, createStreamSemanticCheck) { + useDb("root", "test"); + + run("CREATE STREAM s1 AS SELECT PERCENTILE(c1, 30) FROM t1", TSDB_CODE_PAR_STREAM_NOT_ALLOWED_FUNC); +} + TEST_F(ParserInitialCTest, createTable) { useDb("root", "test"); @@ -546,7 +601,7 @@ TEST_F(ParserInitialCTest, createTable) { "TAGS (a1 TIMESTAMP, a2 INT, a3 INT UNSIGNED, a4 BIGINT, a5 BIGINT UNSIGNED, a6 FLOAT, a7 DOUBLE, a8 BINARY(20), " "a9 SMALLINT, a10 SMALLINT UNSIGNED COMMENT 'test column comment', a11 TINYINT, a12 TINYINT UNSIGNED, a13 BOOL, " "a14 NCHAR(30), a15 VARCHAR(50)) " - "TTL 100 COMMENT 'test create table' SMA(c1, c2, c3) ROLLUP (MIN) FILE_FACTOR 0.1"); + "TTL 100 COMMENT 'test create table' SMA(c1, c2, c3) ROLLUP (MIN)"); run("CREATE TABLE IF NOT EXISTS t1 USING st1 TAGS(1, 'wxy', NOW)"); diff --git a/source/libs/parser/test/parInitialDTest.cpp b/source/libs/parser/test/parInitialDTest.cpp index 8562926789a8f4a8307547d30688ab519d1f52ce..4ecbb7b6d80990c38b37eb1106a5e8d0bf140b28 100644 --- a/source/libs/parser/test/parInitialDTest.cpp +++ b/source/libs/parser/test/parInitialDTest.cpp @@ -64,8 +64,9 @@ TEST_F(ParserInitialDTest, dropConsumerGroup) { SMDropCgroupReq expect = {0}; - auto setDropCgroupReqFunc = [&](const char* pTopicName, const char* pCGroupName, int8_t igNotExists = 0) { - memset(&expect, 0, sizeof(SMDropCgroupReq)); + auto clearDropCgroupReq = [&]() { memset(&expect, 0, sizeof(SMDropCgroupReq)); }; + + auto setDropCgroupReq = [&](const char* pTopicName, const char* pCGroupName, int8_t igNotExists = 0) { snprintf(expect.topic, sizeof(expect.topic), "0.%s", pTopicName); strcpy(expect.cgroup, pCGroupName); expect.igNotExists = igNotExists; @@ -81,15 +82,51 @@ TEST_F(ParserInitialDTest, dropConsumerGroup) { ASSERT_EQ(req.igNotExists, expect.igNotExists); }); - setDropCgroupReqFunc("tp1", "cg1"); + setDropCgroupReq("tp1", "cg1"); run("DROP CONSUMER GROUP cg1 ON tp1"); + clearDropCgroupReq(); - setDropCgroupReqFunc("tp1", "cg1", 1); + setDropCgroupReq("tp1", "cg1", 1); run("DROP CONSUMER GROUP IF EXISTS cg1 ON tp1"); + clearDropCgroupReq(); } // todo DROP database + // todo DROP dnode +TEST_F(ParserInitialDTest, dropDnode) { + useDb("root", "test"); + + SDropDnodeReq expect = {0}; + + auto clearDropDnodeReq = [&]() { memset(&expect, 0, sizeof(SDropDnodeReq)); }; + + auto setDropDnodeReqById = [&](int32_t dnodeId) { expect.dnodeId = dnodeId; }; + + auto setDropDnodeReqByEndpoint = [&](const char* pFqdn, int32_t port) { + strcpy(expect.fqdn, pFqdn); + expect.port = port; + }; + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_DROP_DNODE_STMT); + SDropDnodeReq req = {0}; + ASSERT_TRUE(TSDB_CODE_SUCCESS == tDeserializeSDropDnodeReq(pQuery->pCmdMsg->pMsg, pQuery->pCmdMsg->msgLen, &req)); + + ASSERT_EQ(req.dnodeId, expect.dnodeId); + ASSERT_EQ(std::string(req.fqdn), std::string(expect.fqdn)); + ASSERT_EQ(req.port, expect.port); + }); + + setDropDnodeReqById(1); + run("DROP DNODE 1"); + clearDropDnodeReq(); + + setDropDnodeReqByEndpoint("host1", 7030); + run("DROP DNODE 'host1:7030'"); + clearDropDnodeReq(); +} + // todo DROP function TEST_F(ParserInitialDTest, dropIndex) { diff --git a/source/libs/parser/test/parSelectTest.cpp b/source/libs/parser/test/parSelectTest.cpp index c87520e26206cd4e2c35aacc8fd428849e9b77a2..11e09cd83d89f5ba02df18f5a439786ffe6d83c0 100644 --- a/source/libs/parser/test/parSelectTest.cpp +++ b/source/libs/parser/test/parSelectTest.cpp @@ -161,6 +161,40 @@ TEST_F(ParserSelectTest, useDefinedFunc) { run("SELECT udf2(c1) FROM t1 GROUP BY c2"); } +TEST_F(ParserSelectTest, uniqueFunc) { + useDb("root", "test"); + + run("SELECT UNIQUE(c1) FROM t1"); + + run("SELECT UNIQUE(c2 + 10) FROM t1 WHERE c1 > 10"); + + run("SELECT UNIQUE(c2 + 10), ts, c2 FROM t1 WHERE c1 > 10"); +} + +TEST_F(ParserSelectTest, uniqueFuncSemanticCheck) { + useDb("root", "test"); + + run("SELECT UNIQUE(c1) FROM t1 INTERVAL(10S)", TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC); + + run("SELECT UNIQUE(c1) FROM t1 GROUP BY c2", TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC); +} + +TEST_F(ParserSelectTest, tailFunc) { + useDb("root", "test"); + + run("SELECT TAIL(c1, 10) FROM t1"); + + run("SELECT TAIL(c2 + 10, 10, 80) FROM t1 WHERE c1 > 10"); +} + +TEST_F(ParserSelectTest, tailFuncSemanticCheck) { + useDb("root", "test"); + + run("SELECT TAIL(c1, 10) FROM t1 INTERVAL(10S)", TSDB_CODE_PAR_WINDOW_NOT_ALLOWED_FUNC); + + run("SELECT TAIL(c1, 10) FROM t1 GROUP BY c2", TSDB_CODE_PAR_GROUP_BY_NOT_ALLOWED_FUNC); +} + TEST_F(ParserSelectTest, groupBy) { useDb("root", "test"); @@ -222,6 +256,22 @@ TEST_F(ParserSelectTest, intervalSemanticCheck) { run("SELECT _WSTARTTS, _WENDTS, _WDURATION, sum(c1) FROM t1", TSDB_CODE_PAR_INVALID_WINDOW_PC); } +TEST_F(ParserSelectTest, interp) { + useDb("root", "test"); + + run("SELECT INTERP(c1) FROM t1"); + + run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00')"); + + run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') FILL(LINEAR)"); + + run("SELECT INTERP(c1) FROM t1 EVERY(5s)"); + + run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s)"); + + run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s) FILL(LINEAR)"); +} + TEST_F(ParserSelectTest, subquery) { useDb("root", "test"); @@ -328,6 +378,8 @@ TEST_F(ParserSelectTest, setOperator) { run("(SELECT * FROM t1) UNION ALL (SELECT * FROM t1)"); run("SELECT c1 FROM (SELECT c1 FROM t1 UNION ALL SELECT c1 FROM t1)"); + + run("SELECT c1, c2 FROM t1 UNION ALL SELECT c1 as a, c2 as b FROM t1 ORDER BY c1"); } TEST_F(ParserSelectTest, informationSchema) { diff --git a/source/libs/parser/test/parShowToUse.cpp b/source/libs/parser/test/parShowToUse.cpp index d468ff25e330236da3f37e3c4f4472e4d8eb4910..940b6ea8ac101bff71416d0abd4e20e2d2cd908b 100644 --- a/source/libs/parser/test/parShowToUse.cpp +++ b/source/libs/parser/test/parShowToUse.cpp @@ -24,9 +24,45 @@ class ParserShowToUseTest : public ParserDdlTest {}; // todo SHOW accounts // todo SHOW apps // todo SHOW connections -// todo SHOW create database -// todo SHOW create stable -// todo SHOW create table + +TEST_F(ParserShowToUseTest, showCreateDatabase) { + useDb("root", "test"); + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_SHOW_CREATE_DATABASE_STMT); + ASSERT_EQ(pQuery->execMode, QUERY_EXEC_MODE_LOCAL); + ASSERT_TRUE(pQuery->haveResultSet); + ASSERT_NE(((SShowCreateDatabaseStmt*)pQuery->pRoot)->pCfg, nullptr); + }); + + run("SHOW CREATE DATABASE test"); +} + +TEST_F(ParserShowToUseTest, showCreateSTable) { + useDb("root", "test"); + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_SHOW_CREATE_STABLE_STMT); + ASSERT_EQ(pQuery->execMode, QUERY_EXEC_MODE_LOCAL); + ASSERT_TRUE(pQuery->haveResultSet); + ASSERT_NE(((SShowCreateTableStmt*)pQuery->pRoot)->pMeta, nullptr); + }); + + run("SHOW CREATE STABLE st1"); +} + +TEST_F(ParserShowToUseTest, showCreateTable) { + useDb("root", "test"); + + setCheckDdlFunc([&](const SQuery* pQuery, ParserStage stage) { + ASSERT_EQ(nodeType(pQuery->pRoot), QUERY_NODE_SHOW_CREATE_TABLE_STMT); + ASSERT_EQ(pQuery->execMode, QUERY_EXEC_MODE_LOCAL); + ASSERT_TRUE(pQuery->haveResultSet); + ASSERT_NE(((SShowCreateTableStmt*)pQuery->pRoot)->pMeta, nullptr); + }); + + run("SHOW CREATE TABLE t1"); +} TEST_F(ParserShowToUseTest, showDatabases) { useDb("root", "test"); diff --git a/source/libs/parser/test/parTestUtil.h b/source/libs/parser/test/parTestUtil.h index afdb76344837781c14145eb565187d482b161399..16c3d05b38cd66ba148696fa629728b04ccf4afa 100644 --- a/source/libs/parser/test/parTestUtil.h +++ b/source/libs/parser/test/parTestUtil.h @@ -20,8 +20,11 @@ #define ALLOW_FORBID_FUNC +#include "cmdnodes.h" #include "querynodes.h" #include "taoserror.h" +#include "tglobal.h" +#include "ttime.h" namespace ParserTest { diff --git a/source/libs/planner/inc/planInt.h b/source/libs/planner/inc/planInt.h index e0728c86545137d724bc5bd5c5576ca60dfb9e12..b7fba8323580c05816857f9d607bc3ffdf6c42df 100644 --- a/source/libs/planner/inc/planInt.h +++ b/source/libs/planner/inc/planInt.h @@ -23,10 +23,6 @@ extern "C" { #include "planner.h" #include "taoserror.h" -#define QUERY_POLICY_VNODE 1 -#define QUERY_POLICY_HYBRID 2 -#define QUERY_POLICY_QNODE 3 - #define planFatal(param, ...) qFatal("PLAN: " param, __VA_ARGS__) #define planError(param, ...) qError("PLAN: " param, __VA_ARGS__) #define planWarn(param, ...) qWarn("PLAN: " param, __VA_ARGS__) diff --git a/source/libs/planner/src/planLogicCreater.c b/source/libs/planner/src/planLogicCreater.c index cbc74a2711d5ef4621041dcff86853b86888b55e..c7490faa0c28524830f16515e2eb16278b894706 100644 --- a/source/libs/planner/src/planLogicCreater.c +++ b/source/libs/planner/src/planLogicCreater.c @@ -91,7 +91,7 @@ static EDealRes doNameExpr(SNode* pNode, void* pContext) { return DEAL_RES_CONTINUE; } -static int32_t rewriteExprForSelect(SNodeList* pExprs, SSelectStmt* pSelect, ESqlClause clause) { +static int32_t rewriteExprsForSelect(SNodeList* pExprs, SSelectStmt* pSelect, ESqlClause clause) { nodesWalkExprs(pExprs, doNameExpr, NULL); SRewriteExprCxt cxt = {.errCode = TSDB_CODE_SUCCESS, .pExprs = pExprs}; nodesRewriteSelectStmt(pSelect, clause, doRewriteExpr, &cxt); @@ -156,7 +156,11 @@ static EScanType getScanType(SLogicPlanContext* pCxt, SNodeList* pScanPseudoCols if (NULL == pScanCols) { // select count(*) from t - return NULL == pScanPseudoCols ? SCAN_TYPE_TABLE : SCAN_TYPE_TAG; + return NULL == pScanPseudoCols + ? SCAN_TYPE_TABLE + : ((FUNCTION_TYPE_BLOCK_DIST_INFO == ((SFunctionNode*)nodesListGetNode(pScanPseudoCols, 0))->funcType) + ? SCAN_TYPE_BLOCK_INFO + : SCAN_TYPE_TAG); } if (TSDB_SYSTEM_TABLE == tableType) { @@ -262,7 +266,7 @@ static int32_t createScanLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect // rewrite the expression in subsequent clauses if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pScan->pScanPseudoCols, pSelect, SQL_CLAUSE_FROM); + code = rewriteExprsForSelect(pScan->pScanPseudoCols, pSelect, SQL_CLAUSE_FROM); } pScan->scanType = getScanType(pCxt, pScan->pScanPseudoCols, pScan->pScanCols, pScan->tableType); @@ -421,7 +425,7 @@ static int32_t createAggLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect, // rewrite the expression in subsequent clauses if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pAgg->pGroupKeys, pSelect, SQL_CLAUSE_GROUP_BY); + code = rewriteExprsForSelect(pAgg->pGroupKeys, pSelect, SQL_CLAUSE_GROUP_BY); } if (TSDB_CODE_SUCCESS == code && pSelect->hasAggFuncs) { @@ -430,7 +434,7 @@ static int32_t createAggLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect, // rewrite the expression in subsequent clauses if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pAgg->pAggFuncs, pSelect, SQL_CLAUSE_GROUP_BY); + code = rewriteExprsForSelect(pAgg->pAggFuncs, pSelect, SQL_CLAUSE_GROUP_BY); } if (TSDB_CODE_SUCCESS == code && NULL != pSelect->pHaving) { @@ -469,14 +473,15 @@ static int32_t createIndefRowsFuncLogicNode(SLogicPlanContext* pCxt, SSelectStmt return TSDB_CODE_OUT_OF_MEMORY; } - int32_t code = nodesCollectFuncs(pSelect, SQL_CLAUSE_SELECT, fmIsVectorFunc, &pIdfRowsFunc->pVectorFuncs); + // indefinite rows functions and _select_values functions + int32_t code = nodesCollectFuncs(pSelect, SQL_CLAUSE_SELECT, fmIsVectorFunc, &pIdfRowsFunc->pFuncs); if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pIdfRowsFunc->pVectorFuncs, pSelect, SQL_CLAUSE_SELECT); + code = rewriteExprsForSelect(pIdfRowsFunc->pFuncs, pSelect, SQL_CLAUSE_SELECT); } // set the output if (TSDB_CODE_SUCCESS == code) { - code = createColumnByRewriteExprs(pIdfRowsFunc->pVectorFuncs, &pIdfRowsFunc->node.pTargets); + code = createColumnByRewriteExprs(pIdfRowsFunc->pFuncs, &pIdfRowsFunc->node.pTargets); } if (TSDB_CODE_SUCCESS == code) { @@ -488,6 +493,50 @@ static int32_t createIndefRowsFuncLogicNode(SLogicPlanContext* pCxt, SSelectStmt return code; } +static int32_t createInterpFuncLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect, SLogicNode** pLogicNode) { + if (!pSelect->hasInterpFunc) { + return TSDB_CODE_SUCCESS; + } + + SInterpFuncLogicNode* pInterpFunc = (SInterpFuncLogicNode*)nodesMakeNode(QUERY_NODE_LOGIC_PLAN_INTERP_FUNC); + if (NULL == pInterpFunc) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + int32_t code = nodesCollectFuncs(pSelect, SQL_CLAUSE_SELECT, fmIsInterpFunc, &pInterpFunc->pFuncs); + if (TSDB_CODE_SUCCESS == code) { + code = rewriteExprsForSelect(pInterpFunc->pFuncs, pSelect, SQL_CLAUSE_SELECT); + } + + if (TSDB_CODE_SUCCESS == code && NULL != pSelect->pFill) { + SFillNode* pFill = (SFillNode*)pSelect->pFill; + pInterpFunc->timeRange = pFill->timeRange; + pInterpFunc->fillMode = pFill->mode; + pInterpFunc->pTimeSeries = nodesCloneNode(pFill->pWStartTs); + pInterpFunc->pFillValues = nodesCloneNode(pFill->pValues); + if (NULL == pInterpFunc->pTimeSeries || (NULL != pFill->pValues && NULL == pInterpFunc->pFillValues)) { + code = TSDB_CODE_OUT_OF_MEMORY; + } + } + + if (TSDB_CODE_SUCCESS == code && NULL != pSelect->pEvery) { + pInterpFunc->interval = ((SValueNode*)pSelect->pEvery)->datum.i; + } + + // set the output + if (TSDB_CODE_SUCCESS == code) { + code = createColumnByRewriteExprs(pInterpFunc->pFuncs, &pInterpFunc->node.pTargets); + } + + if (TSDB_CODE_SUCCESS == code) { + *pLogicNode = (SLogicNode*)pInterpFunc; + } else { + nodesDestroyNode((SNode*)pInterpFunc); + } + + return code; +} + static int32_t createWindowLogicNodeFinalize(SLogicPlanContext* pCxt, SSelectStmt* pSelect, SWindowLogicNode* pWindow, SLogicNode** pLogicNode) { int32_t code = nodesCollectFuncs(pSelect, SQL_CLAUSE_WINDOW, fmIsWindowClauseFunc, &pWindow->pFuncs); @@ -502,7 +551,7 @@ static int32_t createWindowLogicNodeFinalize(SLogicPlanContext* pCxt, SSelectStm } if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pWindow->pFuncs, pSelect, SQL_CLAUSE_WINDOW); + code = rewriteExprsForSelect(pWindow->pFuncs, pSelect, SQL_CLAUSE_WINDOW); } if (TSDB_CODE_SUCCESS == code) { @@ -548,12 +597,14 @@ static int32_t createWindowLogicNodeBySession(SLogicPlanContext* pCxt, SSessionW pWindow->winType = WINDOW_TYPE_SESSION; pWindow->sessionGap = ((SValueNode*)pSession->pGap)->datum.i; + pWindow->windowAlgo = pCxt->pPlanCxt->streamQuery ? SESSION_ALGO_STREAM_SINGLE : SESSION_ALGO_MERGE; pWindow->pTspk = nodesCloneNode((SNode*)pSession->pCol); if (NULL == pWindow->pTspk) { nodesDestroyNode((SNode*)pWindow); return TSDB_CODE_OUT_OF_MEMORY; } + pWindow->pTsEnd = nodesCloneNode((SNode*)pSession->pCol); return createWindowLogicNodeFinalize(pCxt, pSelect, pWindow, pLogicNode); } @@ -572,7 +623,7 @@ static int32_t createWindowLogicNodeByInterval(SLogicPlanContext* pCxt, SInterva pWindow->sliding = (NULL != pInterval->pSliding ? ((SValueNode*)pInterval->pSliding)->datum.i : pWindow->interval); pWindow->slidingUnit = (NULL != pInterval->pSliding ? ((SValueNode*)pInterval->pSliding)->unit : pWindow->intervalUnit); - pWindow->intervalAlgo = pCxt->pPlanCxt->streamQuery ? INTERVAL_ALGO_STREAM_SINGLE : INTERVAL_ALGO_HASH; + pWindow->windowAlgo = pCxt->pPlanCxt->streamQuery ? INTERVAL_ALGO_STREAM_SINGLE : INTERVAL_ALGO_HASH; pWindow->pTspk = nodesCloneNode(pInterval->pCol); if (NULL == pWindow->pTspk) { @@ -772,7 +823,7 @@ static int32_t createDistinctLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSe // rewrite the expression in subsequent clauses if (TSDB_CODE_SUCCESS == code) { - code = rewriteExprForSelect(pAgg->pGroupKeys, pSelect, SQL_CLAUSE_DISTINCT); + code = rewriteExprsForSelect(pAgg->pGroupKeys, pSelect, SQL_CLAUSE_DISTINCT); } // set the output @@ -807,6 +858,9 @@ static int32_t createSelectLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSele if (TSDB_CODE_SUCCESS == code) { code = createSelectRootLogicNode(pCxt, pSelect, createIndefRowsFuncLogicNode, &pRoot); } + if (TSDB_CODE_SUCCESS == code) { + code = createSelectRootLogicNode(pCxt, pSelect, createInterpFuncLogicNode, &pRoot); + } if (TSDB_CODE_SUCCESS == code) { code = createSelectRootLogicNode(pCxt, pSelect, createDistinctLogicNode, &pRoot); } diff --git a/source/libs/planner/src/planOptimizer.c b/source/libs/planner/src/planOptimizer.c index 9d7cd0cf27d1674a2555c30dbc3cd9f4d3d61f5e..5ed49f6aaed632245456a0c64e9145becca23965 100644 --- a/source/libs/planner/src/planOptimizer.c +++ b/source/libs/planner/src/planOptimizer.c @@ -731,6 +731,7 @@ static int32_t opkDoOptimized(SOptimizeContext* pCxt, SSortLogicNode* pSort, SNo FOREACH(pNode, pSort->node.pParent->pChildren) { if (nodesEqualNode(pNode, (SNode*)pSort)) { REPLACE_NODE(pDownNode); + ((SLogicNode*)pDownNode)->pParent = pSort->node.pParent; break; } } @@ -1081,13 +1082,89 @@ static int32_t partTagsOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogicSub return code; } +static bool eliminateProjOptMayBeOptimized(SLogicNode* pNode) { + //TODO: enable this optimization after new mechanising that map projection and targets of project node + if (NULL != pNode->pParent) { + return false; + } + + if (QUERY_NODE_LOGIC_PLAN_PROJECT != nodeType(pNode) || 1 != LIST_LENGTH(pNode->pChildren)) { + return false; + } + + SProjectLogicNode* pProjectNode = (SProjectLogicNode*)pNode; + if (-1 != pProjectNode->limit || -1 != pProjectNode->slimit || -1 != pProjectNode->offset || -1 != pProjectNode->soffset) { + return false; + } + + SHashObj* pProjColNameHash = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK); + SNode* pProjection; + FOREACH(pProjection, pProjectNode->pProjections) { + SExprNode* pExprNode = (SExprNode*)pProjection; + if (QUERY_NODE_COLUMN != nodeType(pExprNode)) { + taosHashCleanup(pProjColNameHash); + return false; + } + + char* projColumnName = ((SColumnNode*)pProjection)->colName; + int32_t* pExist = taosHashGet(pProjColNameHash, projColumnName, strlen(projColumnName)); + if (NULL != pExist) { + taosHashCleanup(pProjColNameHash); + return false; + } else { + int32_t exist = 1; + taosHashPut(pProjColNameHash, projColumnName, strlen(projColumnName), &exist, sizeof(exist)); + } + } + taosHashCleanup(pProjColNameHash); + + return true; +} + +static int32_t eliminateProjOptimizeImpl(SOptimizeContext* pCxt, SLogicSubplan* pLogicSubplan, SProjectLogicNode* pProjectNode) { + SLogicNode* pChild = (SLogicNode*)nodesListGetNode(pProjectNode->node.pChildren, 0); + SNodeList* pNewChildTargets = nodesMakeList(); + + SNode* pProjection = NULL; + FOREACH(pProjection, pProjectNode->pProjections) { + SNode* pChildTarget = NULL; + FOREACH(pChildTarget, pChild->pTargets) { + if (strcmp(((SColumnNode*)pProjection)->colName, ((SColumnNode*)pChildTarget)->colName) == 0) { + nodesListAppend(pNewChildTargets, nodesCloneNode(pChildTarget)); + break; + } + } + } + nodesDestroyList(pChild->pTargets); + pChild->pTargets = pNewChildTargets; + + int32_t code = replaceLogicNode(pLogicSubplan, (SLogicNode*)pProjectNode, pChild); + if (TSDB_CODE_SUCCESS == code) { + NODES_CLEAR_LIST(pProjectNode->node.pChildren); + nodesDestroyNode((SNode*)pProjectNode); + } + return code; +} + +static int32_t eliminateProjOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogicSubplan) { + SProjectLogicNode* pProjectNode = + (SProjectLogicNode*)optFindPossibleNode(pLogicSubplan->pNode, eliminateProjOptMayBeOptimized); + + if (NULL == pProjectNode) { + return TSDB_CODE_SUCCESS; + } + + return eliminateProjOptimizeImpl(pCxt, pLogicSubplan, pProjectNode); +} + // clang-format off static const SOptimizeRule optimizeRuleSet[] = { {.pName = "OptimizeScanData", .optimizeFunc = osdOptimize}, {.pName = "ConditionPushDown", .optimizeFunc = cpdOptimize}, {.pName = "OrderByPrimaryKey", .optimizeFunc = opkOptimize}, {.pName = "SmaIndex", .optimizeFunc = smaOptimize}, - {.pName = "PartitionByTags", .optimizeFunc = partTagsOptimize} + {.pName = "PartitionByTags", .optimizeFunc = partTagsOptimize}, + {.pName = "EliminateProject", .optimizeFunc = eliminateProjOptimize} }; // clang-format on diff --git a/source/libs/planner/src/planPhysiCreater.c b/source/libs/planner/src/planPhysiCreater.c index 2974b3ef8cb7517f1de34ca1e81b26bb4df6130c..6af5a15147b4805eec1c374020cdb93c1f67abf8 100644 --- a/source/libs/planner/src/planPhysiCreater.c +++ b/source/libs/planner/src/planPhysiCreater.c @@ -35,7 +35,7 @@ typedef struct SPhysiPlanContext { int32_t errCode; int16_t nextDataBlockId; SArray* pLocationHelper; - SArray* pExecNodeList; + SArray* pExecNodeList; // SArray } SPhysiPlanContext; static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char* pKey) { @@ -450,34 +450,37 @@ static void vgroupInfoToNodeAddr(const SVgroupInfo* vg, SQueryNodeAddr* pNodeAdd pNodeAddr->epSet = vg->epSet; } -static int32_t createTagScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan, SScanLogicNode* pScanLogicNode, - SPhysiNode** pPhyNode) { - STagScanPhysiNode* pTagScan = - (STagScanPhysiNode*)makePhysiNode(pCxt, (SLogicNode*)pScanLogicNode, QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN); - if (NULL == pTagScan) { - return TSDB_CODE_OUT_OF_MEMORY; - } - vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode); - SQueryNodeLoad node = {.addr = pSubplan->execNode, .load = 0}; - taosArrayPush(pCxt->pExecNodeList, &pSubplan->execNode); - return createScanPhysiNodeFinalize(pCxt, pSubplan, pScanLogicNode, (SScanPhysiNode*)pTagScan, pPhyNode); -} - static ENodeType getScanOperatorType(EScanType scanType) { switch (scanType) { + case SCAN_TYPE_TAG: + return QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN; case SCAN_TYPE_TABLE: return QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN; case SCAN_TYPE_STREAM: return QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN; case SCAN_TYPE_TABLE_MERGE: - // return QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN; return QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN; + case SCAN_TYPE_BLOCK_INFO: + return QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN; default: break; } return QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN; } +static int32_t createSimpleScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan, SScanLogicNode* pScanLogicNode, + SPhysiNode** pPhyNode) { + SScanPhysiNode* pScan = + (SScanPhysiNode*)makePhysiNode(pCxt, (SLogicNode*)pScanLogicNode, getScanOperatorType(pScanLogicNode->scanType)); + if (NULL == pScan) { + return TSDB_CODE_OUT_OF_MEMORY; + } + vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode); + SQueryNodeLoad node = {.addr = pSubplan->execNode, .load = 0}; + taosArrayPush(pCxt->pExecNodeList, &node); + return createScanPhysiNodeFinalize(pCxt, pSubplan, pScanLogicNode, pScan, pPhyNode); +} + static int32_t createTableScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan, SScanLogicNode* pScanLogicNode, SPhysiNode** pPhyNode) { STableScanPhysiNode* pTableScan = (STableScanPhysiNode*)makePhysiNode(pCxt, (SLogicNode*)pScanLogicNode, @@ -493,10 +496,6 @@ static int32_t createTableScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubp vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode); pSubplan->execNodeStat.tableNum = pScanLogicNode->pVgroupList->vgroups[0].numOfTable; } - if (pCxt->pExecNodeList) { - SQueryNodeLoad node = {.addr = pSubplan->execNode, .load = 0}; - taosArrayPush(pCxt->pExecNodeList, &node); - } tNameGetFullDbName(&pScanLogicNode->tableName, pSubplan->dbFName); pTableScan->dataRequired = pScanLogicNode->dataRequired; pTableScan->pDynamicScanFuncs = nodesCloneList(pScanLogicNode->pDynamicScanFuncs); @@ -529,14 +528,12 @@ static int32_t createSystemTableScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pScan->showRewrite = pScanLogicNode->showRewrite; pScan->accountId = pCxt->pPlanCxt->acctId; - if (0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_USER_TABLES)) { + if (0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_USER_TABLES) || + 0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_USER_TABLE_DISTRIBUTED)) { vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode); - SQueryNodeLoad node = {.addr = pSubplan->execNode, .load = 0}; - taosArrayPush(pCxt->pExecNodeList, &pSubplan->execNode); - } else { - SQueryNodeLoad node = {.addr = {.nodeId = MNODE_HANDLE, .epSet = pCxt->pPlanCxt->mgmtEpSet}, .load = 0}; - taosArrayPush(pCxt->pExecNodeList, &node); } + SQueryNodeLoad node = {.addr = {.nodeId = MNODE_HANDLE, .epSet = pCxt->pPlanCxt->mgmtEpSet}, .load = 0}; + taosArrayPush(pCxt->pExecNodeList, &node); pScan->mgmtEpSet = pCxt->pPlanCxt->mgmtEpSet; tNameGetFullDbName(&pScanLogicNode->tableName, pSubplan->dbFName); @@ -557,7 +554,8 @@ static int32_t createScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan, SPhysiNode** pPhyNode) { switch (pScanLogicNode->scanType) { case SCAN_TYPE_TAG: - return createTagScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode); + case SCAN_TYPE_BLOCK_INFO: + return createSimpleScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode); case SCAN_TYPE_TABLE: return createTableScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode); case SCAN_TYPE_SYSTEM_TABLE: @@ -820,8 +818,8 @@ static int32_t createIndefRowsFuncPhysiNode(SPhysiPlanContext* pCxt, SNodeList* } SNodeList* pPrecalcExprs = NULL; - SNodeList* pVectorFuncs = NULL; - int32_t code = rewritePrecalcExprs(pCxt, pFuncLogicNode->pVectorFuncs, &pPrecalcExprs, &pVectorFuncs); + SNodeList* pFuncs = NULL; + int32_t code = rewritePrecalcExprs(pCxt, pFuncLogicNode->pFuncs, &pPrecalcExprs, &pFuncs); SDataBlockDescNode* pChildTupe = (((SPhysiNode*)nodesListGetNode(pChildren, 0))->pOutputDataBlockDesc); // push down expression to pOutputDataBlockDesc of child node @@ -832,10 +830,10 @@ static int32_t createIndefRowsFuncPhysiNode(SPhysiPlanContext* pCxt, SNodeList* } } - if (TSDB_CODE_SUCCESS == code && NULL != pVectorFuncs) { - code = setListSlotId(pCxt, pChildTupe->dataBlockId, -1, pVectorFuncs, &pIdfRowsFunc->pVectorFuncs); + if (TSDB_CODE_SUCCESS == code) { + code = setListSlotId(pCxt, pChildTupe->dataBlockId, -1, pFuncs, &pIdfRowsFunc->pFuncs); if (TSDB_CODE_SUCCESS == code) { - code = addDataBlockSlots(pCxt, pIdfRowsFunc->pVectorFuncs, pIdfRowsFunc->node.pOutputDataBlockDesc); + code = addDataBlockSlots(pCxt, pIdfRowsFunc->pFuncs, pIdfRowsFunc->node.pOutputDataBlockDesc); } } @@ -848,6 +846,54 @@ static int32_t createIndefRowsFuncPhysiNode(SPhysiPlanContext* pCxt, SNodeList* return code; } +static int32_t createInterpFuncPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pChildren, + SInterpFuncLogicNode* pFuncLogicNode, SPhysiNode** pPhyNode) { + SInterpFuncPhysiNode* pInterpFunc = + (SInterpFuncPhysiNode*)makePhysiNode(pCxt, (SLogicNode*)pFuncLogicNode, QUERY_NODE_PHYSICAL_PLAN_INTERP_FUNC); + if (NULL == pInterpFunc) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + SNodeList* pPrecalcExprs = NULL; + SNodeList* pFuncs = NULL; + int32_t code = rewritePrecalcExprs(pCxt, pFuncLogicNode->pFuncs, &pPrecalcExprs, &pFuncs); + + SDataBlockDescNode* pChildTupe = (((SPhysiNode*)nodesListGetNode(pChildren, 0))->pOutputDataBlockDesc); + // push down expression to pOutputDataBlockDesc of child node + if (TSDB_CODE_SUCCESS == code && NULL != pPrecalcExprs) { + code = setListSlotId(pCxt, pChildTupe->dataBlockId, -1, pPrecalcExprs, &pInterpFunc->pExprs); + if (TSDB_CODE_SUCCESS == code) { + code = pushdownDataBlockSlots(pCxt, pInterpFunc->pExprs, pChildTupe); + } + } + + if (TSDB_CODE_SUCCESS == code) { + code = setListSlotId(pCxt, pChildTupe->dataBlockId, -1, pFuncs, &pInterpFunc->pFuncs); + if (TSDB_CODE_SUCCESS == code) { + code = addDataBlockSlots(pCxt, pInterpFunc->pFuncs, pInterpFunc->node.pOutputDataBlockDesc); + } + } + + if (TSDB_CODE_SUCCESS == code) { + pInterpFunc->timeRange = pFuncLogicNode->timeRange; + pInterpFunc->interval = pFuncLogicNode->interval; + pInterpFunc->fillMode = pFuncLogicNode->fillMode; + pInterpFunc->pFillValues = nodesCloneNode(pFuncLogicNode->pFillValues); + pInterpFunc->pTimeSeries = nodesCloneNode(pFuncLogicNode->pTimeSeries); + if (NULL == pInterpFunc->pTimeSeries || (NULL != pFuncLogicNode->pFillValues && NULL == pInterpFunc->pFillValues)) { + code = TSDB_CODE_OUT_OF_MEMORY; + } + } + + if (TSDB_CODE_SUCCESS == code) { + *pPhyNode = (SPhysiNode*)pInterpFunc; + } else { + nodesDestroyNode((SNode*)pInterpFunc); + } + + return code; +} + static int32_t createProjectPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pChildren, SProjectLogicNode* pProjectLogicNode, SPhysiNode** pPhyNode) { SProjectPhysiNode* pProject = @@ -956,6 +1002,9 @@ static int32_t createWindowPhysiNodeFinalize(SPhysiPlanContext* pCxt, SNodeList* if (TSDB_CODE_SUCCESS == code) { code = setNodeSlotId(pCxt, pChildTupe->dataBlockId, -1, pWindowLogicNode->pTspk, &pWindow->pTspk); } + if (TSDB_CODE_SUCCESS == code && pWindowLogicNode->pTsEnd) { + code = setNodeSlotId(pCxt, pChildTupe->dataBlockId, -1, pWindowLogicNode->pTsEnd, &pWindow->pTsEnd); + } if (TSDB_CODE_SUCCESS == code && NULL != pFuncs) { code = setListSlotId(pCxt, pChildTupe->dataBlockId, -1, pFuncs, &pWindow->pFuncs); @@ -977,8 +1026,8 @@ static int32_t createWindowPhysiNodeFinalize(SPhysiPlanContext* pCxt, SNodeList* return code; } -static ENodeType getIntervalOperatorType(EIntervalAlgorithm intervalAlgo) { - switch (intervalAlgo) { +static ENodeType getIntervalOperatorType(EWindowAlgorithm windowAlgo) { + switch (windowAlgo) { case INTERVAL_ALGO_HASH: return QUERY_NODE_PHYSICAL_PLAN_HASH_INTERVAL; case INTERVAL_ALGO_MERGE: @@ -989,6 +1038,14 @@ static ENodeType getIntervalOperatorType(EIntervalAlgorithm intervalAlgo) { return QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_INTERVAL; case INTERVAL_ALGO_STREAM_SINGLE: return QUERY_NODE_PHYSICAL_PLAN_STREAM_INTERVAL; + case SESSION_ALGO_STREAM_FINAL: + return QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION; + case SESSION_ALGO_STREAM_SEMI: + return QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_SESSION; + case SESSION_ALGO_STREAM_SINGLE: + return QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION; + case SESSION_ALGO_MERGE: + return QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION; default: break; } @@ -998,7 +1055,7 @@ static ENodeType getIntervalOperatorType(EIntervalAlgorithm intervalAlgo) { static int32_t createIntervalPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pChildren, SWindowLogicNode* pWindowLogicNode, SPhysiNode** pPhyNode) { SIntervalPhysiNode* pInterval = (SIntervalPhysiNode*)makePhysiNode( - pCxt, (SLogicNode*)pWindowLogicNode, getIntervalOperatorType(pWindowLogicNode->intervalAlgo)); + pCxt, (SLogicNode*)pWindowLogicNode, getIntervalOperatorType(pWindowLogicNode->windowAlgo)); if (NULL == pInterval) { return TSDB_CODE_OUT_OF_MEMORY; } @@ -1015,8 +1072,7 @@ static int32_t createIntervalPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pChil static int32_t createSessionWindowPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pChildren, SWindowLogicNode* pWindowLogicNode, SPhysiNode** pPhyNode) { SSessionWinodwPhysiNode* pSession = (SSessionWinodwPhysiNode*)makePhysiNode( - pCxt, (SLogicNode*)pWindowLogicNode, - (pCxt->pPlanCxt->streamQuery ? QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION : QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION)); + pCxt, (SLogicNode*)pWindowLogicNode, getIntervalOperatorType(pWindowLogicNode->windowAlgo)); if (NULL == pSession) { return TSDB_CODE_OUT_OF_MEMORY; } @@ -1283,6 +1339,8 @@ static int32_t doCreatePhysiNode(SPhysiPlanContext* pCxt, SLogicNode* pLogicNode return createFillPhysiNode(pCxt, pChildren, (SFillLogicNode*)pLogicNode, pPhyNode); case QUERY_NODE_LOGIC_PLAN_INDEF_ROWS_FUNC: return createIndefRowsFuncPhysiNode(pCxt, pChildren, (SIndefRowsFuncLogicNode*)pLogicNode, pPhyNode); + case QUERY_NODE_LOGIC_PLAN_INTERP_FUNC: + return createInterpFuncPhysiNode(pCxt, pChildren, (SInterpFuncLogicNode*)pLogicNode, pPhyNode); case QUERY_NODE_LOGIC_PLAN_MERGE: return createMergePhysiNode(pCxt, (SMergeLogicNode*)pLogicNode, pPhyNode); default: @@ -1373,8 +1431,6 @@ static SSubplan* makeSubplan(SPhysiPlanContext* pCxt, SLogicSubplan* pLogicSubpl static int32_t buildInsertSubplan(SPhysiPlanContext* pCxt, SVnodeModifyLogicNode* pModify, SSubplan* pSubplan) { pSubplan->msgType = pModify->msgType; pSubplan->execNode.epSet = pModify->pVgDataBlocks->vg.epSet; - SQueryNodeLoad node = {.addr = pSubplan->execNode, .load = 0}; - taosArrayPush(pCxt->pExecNodeList, &node); return createDataInserter(pCxt, pModify->pVgDataBlocks, &pSubplan->pDataSink); } diff --git a/source/libs/planner/src/planSpliter.c b/source/libs/planner/src/planSpliter.c index 3a9e1d74056a01d182bee21529138ff72a7e0d67..390d665760bcda5d5fd0acfecad27ca3bb82193b 100644 --- a/source/libs/planner/src/planSpliter.c +++ b/source/libs/planner/src/planSpliter.c @@ -176,7 +176,7 @@ static bool stbSplNeedSplit(bool streamQuery, SLogicNode* pNode) { return !stbSplHasGatherExecFunc(((SAggLogicNode*)pNode)->pAggFuncs) && stbSplHasMultiTbScan(streamQuery, pNode); case QUERY_NODE_LOGIC_PLAN_WINDOW: { SWindowLogicNode* pWindow = (SWindowLogicNode*)pNode; - if (WINDOW_TYPE_INTERVAL != pWindow->winType) { + if (WINDOW_TYPE_STATE == pWindow->winType || (!streamQuery && WINDOW_TYPE_SESSION == pWindow->winType) ) { return false; } return !stbSplHasGatherExecFunc(pWindow->pFuncs) && stbSplHasMultiTbScan(streamQuery, pNode); @@ -257,6 +257,34 @@ static int32_t stbSplAppendWStart(SNodeList* pFuncs, int32_t* pIndex) { return code; } +static int32_t stbSplAppendWEnd(SWindowLogicNode* pWin, int32_t* pIndex) { + int32_t index = 0; + SNode* pFunc = NULL; + FOREACH(pFunc, pWin->pFuncs) { + if (FUNCTION_TYPE_WENDTS == ((SFunctionNode*)pFunc)->funcType) { + *pIndex = index; + return TSDB_CODE_SUCCESS; + } + ++index; + } + + SFunctionNode* pWEnd = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION); + if (NULL == pWEnd) { + return TSDB_CODE_OUT_OF_MEMORY; + } + strcpy(pWEnd->functionName, "_wendts"); + snprintf(pWEnd->node.aliasName, sizeof(pWEnd->node.aliasName), "%s.%p", pWEnd->functionName, pWEnd); + int32_t code = fmGetFuncInfo(pWEnd, NULL, 0); + if (TSDB_CODE_SUCCESS == code) { + code = nodesListStrictAppend(pWin->pFuncs, (SNode*)pWEnd); + } + *pIndex = index; + if (TSDB_CODE_SUCCESS == code) { + code = createColumnByRewriteExpr(nodesListGetNode(pWin->pFuncs, index), &pWin->node.pTargets); + } + return code; +} + static int32_t stbSplCreatePartWindowNode(SWindowLogicNode* pMergeWindow, SLogicNode** pPartWindow) { SNodeList* pFunc = pMergeWindow->pFuncs; pMergeWindow->pFuncs = NULL; @@ -376,8 +404,8 @@ static int32_t stbSplSplitIntervalForBatch(SSplitContext* pCxt, SStableSplitInfo SLogicNode* pPartWindow = NULL; int32_t code = stbSplCreatePartWindowNode((SWindowLogicNode*)pInfo->pSplitNode, &pPartWindow); if (TSDB_CODE_SUCCESS == code) { - ((SWindowLogicNode*)pPartWindow)->intervalAlgo = INTERVAL_ALGO_HASH; - ((SWindowLogicNode*)pInfo->pSplitNode)->intervalAlgo = INTERVAL_ALGO_MERGE; + ((SWindowLogicNode*)pPartWindow)->windowAlgo = INTERVAL_ALGO_HASH; + ((SWindowLogicNode*)pInfo->pSplitNode)->windowAlgo = INTERVAL_ALGO_MERGE; SNodeList* pMergeKeys = NULL; code = stbSplCreateMergeKeysByPrimaryKey(((SWindowLogicNode*)pInfo->pSplitNode)->pTspk, &pMergeKeys); if (TSDB_CODE_SUCCESS == code) { @@ -400,8 +428,8 @@ static int32_t stbSplSplitIntervalForStream(SSplitContext* pCxt, SStableSplitInf SLogicNode* pPartWindow = NULL; int32_t code = stbSplCreatePartWindowNode((SWindowLogicNode*)pInfo->pSplitNode, &pPartWindow); if (TSDB_CODE_SUCCESS == code) { - ((SWindowLogicNode*)pPartWindow)->intervalAlgo = INTERVAL_ALGO_STREAM_SEMI; - ((SWindowLogicNode*)pInfo->pSplitNode)->intervalAlgo = INTERVAL_ALGO_STREAM_FINAL; + ((SWindowLogicNode*)pPartWindow)->windowAlgo = INTERVAL_ALGO_STREAM_SEMI; + ((SWindowLogicNode*)pInfo->pSplitNode)->windowAlgo = INTERVAL_ALGO_STREAM_FINAL; code = stbSplCreateExchangeNode(pCxt, pInfo->pSplitNode, pPartWindow); } if (TSDB_CODE_SUCCESS == code) { @@ -421,8 +449,39 @@ static int32_t stbSplSplitInterval(SSplitContext* pCxt, SStableSplitInfo* pInfo) } } +static int32_t stbSplSplitSessionForStream(SSplitContext* pCxt, SStableSplitInfo* pInfo) { + SLogicNode* pPartWindow = NULL; + int32_t code = stbSplCreatePartWindowNode((SWindowLogicNode*)pInfo->pSplitNode, &pPartWindow); + if (TSDB_CODE_SUCCESS == code) { + SWindowLogicNode* pPartWin = (SWindowLogicNode*)pPartWindow; + SWindowLogicNode* pMergeWin = (SWindowLogicNode*)pInfo->pSplitNode; + pPartWin->windowAlgo = SESSION_ALGO_STREAM_SEMI; + pMergeWin->windowAlgo = SESSION_ALGO_STREAM_FINAL; + int32_t index = 0; + int32_t code = stbSplAppendWEnd(pPartWin, &index); + if (TSDB_CODE_SUCCESS == code) { + pMergeWin->pTsEnd = nodesCloneNode(nodesListGetNode(pPartWin->node.pTargets, index)); + if (NULL == pMergeWin->pTsEnd) { + code = TSDB_CODE_OUT_OF_MEMORY; + } + } + code = stbSplCreateExchangeNode(pCxt, pInfo->pSplitNode, pPartWindow); + } + if (TSDB_CODE_SUCCESS == code) { + code = nodesListMakeStrictAppend(&pInfo->pSubplan->pChildren, + (SNode*)splCreateScanSubplan(pCxt, pPartWindow, SPLIT_FLAG_STABLE_SPLIT)); + } + pInfo->pSubplan->subplanType = SUBPLAN_TYPE_MERGE; + ++(pCxt->groupId); + return code; +} + static int32_t stbSplSplitSession(SSplitContext* pCxt, SStableSplitInfo* pInfo) { - return TSDB_CODE_PLAN_INTERNAL_ERROR; + if (pCxt->pPlanCxt->streamQuery) { + return stbSplSplitSessionForStream(pCxt, pInfo); + } else { + return TSDB_CODE_PLAN_INTERNAL_ERROR; + } } static int32_t stbSplSplitWindowNode(SSplitContext* pCxt, SStableSplitInfo* pInfo) { @@ -537,10 +596,12 @@ static int32_t stbSplCreateMergeKeys(SNodeList* pSortKeys, SNodeList* pTargets, SNode* pNode = NULL; FOREACH(pNode, pSortKeys) { SOrderByExprNode* pSortKey = (SOrderByExprNode*)pNode; + SExprNode* pSortExpr = (SExprNode*)pSortKey->pExpr; SNode* pTarget = NULL; bool found = false; FOREACH(pTarget, pTargets) { - if (0 == strcmp(((SExprNode*)pSortKey->pExpr)->aliasName, ((SColumnNode*)pTarget)->colName)) { + if ((QUERY_NODE_COLUMN == nodeType(pSortExpr) && nodesEqualNode((SNode*)pSortExpr, pTarget)) || + (0 == strcmp(pSortExpr->aliasName, ((SColumnNode*)pTarget)->colName))) { code = nodesListMakeStrictAppend(&pMergeKeys, stbSplCreateOrderByExpr(pSortKey, pTarget)); if (TSDB_CODE_SUCCESS != code) { break; @@ -549,7 +610,7 @@ static int32_t stbSplCreateMergeKeys(SNodeList* pSortKeys, SNodeList* pTargets, } } if (TSDB_CODE_SUCCESS == code && !found) { - SNode* pCol = stbSplCreateColumnNode((SExprNode*)pSortKey->pExpr); + SNode* pCol = stbSplCreateColumnNode(pSortExpr); code = nodesListMakeStrictAppend(&pMergeKeys, stbSplCreateOrderByExpr(pSortKey, pCol)); if (TSDB_CODE_SUCCESS == code) { code = nodesListStrictAppend(pTargets, pCol); diff --git a/source/libs/planner/src/planUtil.c b/source/libs/planner/src/planUtil.c index 7f650c7c9a9c67a5919d2b58bfcfb421f7fe454c..77e4e055306f60402216aa7efcb82d8b6bcbaab4 100644 --- a/source/libs/planner/src/planUtil.c +++ b/source/libs/planner/src/planUtil.c @@ -107,6 +107,7 @@ int32_t createColumnByRewriteExpr(SNode* pExpr, SNodeList** pList) { int32_t replaceLogicNode(SLogicSubplan* pSubplan, SLogicNode* pOld, SLogicNode* pNew) { if (NULL == pOld->pParent) { pSubplan->pNode = (SLogicNode*)pNew; + pNew->pParent = NULL; return TSDB_CODE_SUCCESS; } diff --git a/source/libs/planner/src/planner.c b/source/libs/planner/src/planner.c index c4385d576eef29e2975ae419a53e3186704628b5..e75c8375fb766a662b87c8e5c7e43af28a45b097 100644 --- a/source/libs/planner/src/planner.c +++ b/source/libs/planner/src/planner.c @@ -21,7 +21,7 @@ static void dumpQueryPlan(SQueryPlan* pPlan) { char* pStr = NULL; nodesNodeToString((SNode*)pPlan, false, &pStr, NULL); - planDebugL("Query Plan: %s", pStr); + planDebugL("QID:0x%" PRIx64 " Query Plan: %s", pPlan->queryId, pStr); taosMemoryFree(pStr); } diff --git a/source/libs/planner/test/planBasicTest.cpp b/source/libs/planner/test/planBasicTest.cpp index 4b84079f7bd417b538a789ff964675208d9ea0bf..f74c7df355ad74b1db1192caf83c60fca7673f79 100644 --- a/source/libs/planner/test/planBasicTest.cpp +++ b/source/libs/planner/test/planBasicTest.cpp @@ -52,4 +52,34 @@ TEST_F(PlanBasicTest, func) { run("SELECT PERCENTILE(c1, 60) FROM t1"); run("SELECT TOP(c1, 60) FROM t1"); + + run("SELECT TOP(c1, 60) FROM st1"); +} + +TEST_F(PlanBasicTest, uniqueFunc) { + useDb("root", "test"); + + run("SELECT UNIQUE(c1) FROM t1"); + + run("SELECT UNIQUE(c2 + 10) FROM t1 WHERE c1 > 10"); + + run("SELECT UNIQUE(c2 + 10), ts, c2 FROM t1 WHERE c1 > 10"); + + run("SELECT UNIQUE(c1) a FROM t1 ORDER BY a"); +} + +TEST_F(PlanBasicTest, tailFunc) { + useDb("root", "test"); + + run("SELECT TAIL(c1, 10) FROM t1"); + + run("SELECT TAIL(c2 + 10, 10, 80) FROM t1 WHERE c1 > 10"); +} + +TEST_F(PlanBasicTest, interpFunc) { + useDb("root", "test"); + + run("SELECT INTERP(c1) FROM t1"); + + run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s) FILL(LINEAR)"); } diff --git a/source/libs/planner/test/planOptimizeTest.cpp b/source/libs/planner/test/planOptimizeTest.cpp index 84ccea668d55cbcb0edff9920c25faa0ec0b57aa..6b514ef2c327757eab7af92a177e46a757a9f189 100644 --- a/source/libs/planner/test/planOptimizeTest.cpp +++ b/source/libs/planner/test/planOptimizeTest.cpp @@ -52,3 +52,13 @@ TEST_F(PlanOptimizeTest, orderByPrimaryKey) { run("SELECT COUNT(*) FROM t1 INTERVAL(10S) ORDER BY _WSTARTTS DESC"); } + +TEST_F(PlanOptimizeTest, eliminateProjection) { + useDb("root", "test"); + + run("SELECT c1, sum(c3) FROM t1 GROUP BY c1"); + run("SELECT c1 FROM t1"); + run("SELECT * FROM st1"); + run("SELECT c1 FROM st1s3"); + //run("select 1-abs(c1) from (select unique(c1) c1 from st1s3) order by 1 nulls first"); +} diff --git a/source/libs/planner/test/planOrderByTest.cpp b/source/libs/planner/test/planOrderByTest.cpp index 39e93fcff929ddb6a43122c81a47f7f13a7ff4f6..2ca662cf8665946faef361f9b557d4e32403b983 100644 --- a/source/libs/planner/test/planOrderByTest.cpp +++ b/source/libs/planner/test/planOrderByTest.cpp @@ -27,6 +27,10 @@ TEST_F(PlanOrderByTest, basic) { run("SELECT c1 FROM t1 ORDER BY c1"); // ORDER BY key is not in the projection list run("SELECT c1 FROM t1 ORDER BY c2"); + + run("SELECT c1 AS a FROM t1 ORDER BY a"); + + run("SELECT c1 + 10 AS a FROM t1 ORDER BY a"); } TEST_F(PlanOrderByTest, expr) { @@ -41,6 +45,12 @@ TEST_F(PlanOrderByTest, nullsOrder) { run("SELECT * FROM t1 ORDER BY c1 DESC NULLS FIRST"); } +TEST_F(PlanOrderByTest, withGroupBy) { + useDb("root", "test"); + + run("SELECT SUM(c1) AS a FROM t1 GROUP BY c2 ORDER BY a"); +} + TEST_F(PlanOrderByTest, stable) { useDb("root", "test"); @@ -51,4 +61,6 @@ TEST_F(PlanOrderByTest, stable) { run("SELECT c2 FROM st1 ORDER BY c1"); run("SELECT c2 FROM st1 PARTITION BY c2 ORDER BY c1"); + + run("SELECT c1 AS a FROM st1 ORDER BY a"); } diff --git a/source/libs/planner/test/planOtherTest.cpp b/source/libs/planner/test/planOtherTest.cpp index f9feb8d1fe90fd3c985103322b329ff819b8a2d3..0f2540924ca5d9bd59ab259cce2ee5f8d5e80186 100644 --- a/source/libs/planner/test/planOtherTest.cpp +++ b/source/libs/planner/test/planOtherTest.cpp @@ -70,6 +70,10 @@ TEST_F(PlanOtherTest, show) { useDb("root", "test"); run("SHOW DATABASES"); + + run("SHOW TABLE DISTRIBUTED t1"); + + run("SHOW TABLE DISTRIBUTED st1"); } TEST_F(PlanOtherTest, delete) { diff --git a/source/libs/planner/test/planSetOpTest.cpp b/source/libs/planner/test/planSetOpTest.cpp index 5568d3cfbb994b83c3531957a33fc7ae1ff20eb3..bf26f55c02b1c146c0ff8aad723a593956e4945f 100644 --- a/source/libs/planner/test/planSetOpTest.cpp +++ b/source/libs/planner/test/planSetOpTest.cpp @@ -23,9 +23,9 @@ class PlanSetOpTest : public PlannerTestBase {}; TEST_F(PlanSetOpTest, unionAll) { useDb("root", "test"); - // sql 1: single UNION ALL operator + // single UNION ALL operator run("SELECT c1, c2 FROM t1 WHERE c1 > 10 UNION ALL SELECT c1, c2 FROM t1 WHERE c1 > 20"); - // sql 2: multi UNION ALL operator + // multi UNION ALL operator run("SELECT c1, c2 FROM t1 WHERE c1 > 10 " "UNION ALL SELECT c1, c2 FROM t1 WHERE c1 > 20 " "UNION ALL SELECT c1, c2 FROM t1 WHERE c1 > 30"); @@ -46,6 +46,14 @@ TEST_F(PlanSetOpTest, unionAllWithSubquery) { run("SELECT ts FROM (SELECT ts FROM st1) UNION ALL SELECT ts FROM (SELECT ts FROM st1)"); } +TEST_F(PlanSetOpTest, unionAllWithOrderBy) { + useDb("root", "test"); + + run("SELECT c1, c2 FROM t1 WHERE c1 > 10 UNION ALL SELECT c1, c2 FROM t1 WHERE c1 > 20 ORDER BY c1"); + + run("SELECT c1, c2 FROM t1 WHERE c1 > 10 UNION ALL SELECT c1, c2 FROM t1 WHERE c1 > 20 ORDER BY 1"); +} + TEST_F(PlanSetOpTest, union) { useDb("root", "test"); diff --git a/source/libs/qcom/src/queryUtil.c b/source/libs/qcom/src/queryUtil.c index a5a499aaf5bc3b38998528d1550bd9b16c1d7671..2120d24d2612e804a16827125cc7356449dd0d70 100644 --- a/source/libs/qcom/src/queryUtil.c +++ b/source/libs/qcom/src/queryUtil.c @@ -154,7 +154,7 @@ int32_t asyncSendMsgToServerExt(void* pTransporter, SEpSet* epSet, int64_t* pTra .info.persistHandle = persistHandle, .code = 0}; assert(pInfo->fp != NULL); - + TRACE_SET_ROOTID(&rpcMsg.info.traceId, pInfo->requestId); rpcSendRequestWithCtx(pTransporter, epSet, &rpcMsg, pTransporterId, rpcCtx); return TSDB_CODE_SUCCESS; } @@ -177,10 +177,6 @@ char* jobTaskStatusStr(int32_t status) { return "SUCCEED"; case JOB_TASK_STATUS_FAILED: return "FAILED"; - case JOB_TASK_STATUS_CANCELLING: - return "CANCELLING"; - case JOB_TASK_STATUS_CANCELLED: - return "CANCELLED"; case JOB_TASK_STATUS_DROPPING: return "DROPPING"; default: @@ -208,14 +204,14 @@ void destroyQueryExecRes(SQueryExecRes* pRes) { switch (pRes->msgType) { case TDMT_VND_ALTER_TABLE: case TDMT_MND_ALTER_STB: { - tFreeSTableMetaRsp((STableMetaRsp *)pRes->res); + tFreeSTableMetaRsp((STableMetaRsp*)pRes->res); taosMemoryFreeClear(pRes->res); break; } case TDMT_VND_SUBMIT: { tFreeSSubmitRsp((SSubmitRsp*)pRes->res); break; - } + } case TDMT_VND_QUERY: { taosArrayDestroy((SArray*)pRes->res); break; @@ -224,5 +220,3 @@ void destroyQueryExecRes(SQueryExecRes* pRes) { qError("invalid exec result for request type %d", pRes->msgType); } } - - diff --git a/source/libs/qworker/inc/qwMsg.h b/source/libs/qworker/inc/qwMsg.h index ecff861f50d5c31593edeaab28097308a5316b64..8c7c030dce584dca94d843d85876b3cf29538ac2 100644 --- a/source/libs/qworker/inc/qwMsg.h +++ b/source/libs/qworker/inc/qwMsg.h @@ -23,6 +23,7 @@ extern "C" { #include "qwInt.h" #include "dataSinkMgt.h" +int32_t qwAbortPrerocessQuery(QW_FPARAMS_DEF); int32_t qwPrerocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg); int32_t qwProcessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg, int8_t taskType, int8_t explain, const char* sql); int32_t qwProcessCQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg); diff --git a/source/libs/qworker/src/qwDbg.c b/source/libs/qworker/src/qwDbg.c index 368c3bb517e18c8b9254f4d90bdd4b8aa5bb0d2d..add9700a3a68ea8b0b435be54e3df3c4d0d9d7d0 100644 --- a/source/libs/qworker/src/qwDbg.c +++ b/source/libs/qworker/src/qwDbg.c @@ -44,40 +44,30 @@ int32_t qwDbgValidateStatus(QW_FPARAMS_DEF, int8_t oriStatus, int8_t newStatus, break; case JOB_TASK_STATUS_EXECUTING: if (newStatus != JOB_TASK_STATUS_PARTIAL_SUCCEED && newStatus != JOB_TASK_STATUS_SUCCEED && - newStatus != JOB_TASK_STATUS_FAILED && newStatus != JOB_TASK_STATUS_CANCELLING && - newStatus != JOB_TASK_STATUS_CANCELLED && newStatus != JOB_TASK_STATUS_DROPPING) { + newStatus != JOB_TASK_STATUS_FAILED && newStatus != JOB_TASK_STATUS_DROPPING) { QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; case JOB_TASK_STATUS_PARTIAL_SUCCEED: if (newStatus != JOB_TASK_STATUS_EXECUTING && newStatus != JOB_TASK_STATUS_SUCCEED && - newStatus != JOB_TASK_STATUS_CANCELLED && newStatus != JOB_TASK_STATUS_FAILED && - newStatus != JOB_TASK_STATUS_DROPPING) { + newStatus != JOB_TASK_STATUS_FAILED && newStatus != JOB_TASK_STATUS_DROPPING) { QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; case JOB_TASK_STATUS_SUCCEED: - if (newStatus != JOB_TASK_STATUS_CANCELLED && newStatus != JOB_TASK_STATUS_DROPPING && - newStatus != JOB_TASK_STATUS_FAILED) { + if (newStatus != JOB_TASK_STATUS_DROPPING && newStatus != JOB_TASK_STATUS_FAILED) { QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; case JOB_TASK_STATUS_FAILED: - if (newStatus != JOB_TASK_STATUS_CANCELLED && newStatus != JOB_TASK_STATUS_DROPPING) { + if (newStatus != JOB_TASK_STATUS_DROPPING) { QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; - case JOB_TASK_STATUS_CANCELLING: - if (newStatus != JOB_TASK_STATUS_CANCELLED) { - QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); - } - - break; - case JOB_TASK_STATUS_CANCELLED: case JOB_TASK_STATUS_DROPPING: if (newStatus != JOB_TASK_STATUS_FAILED && newStatus != JOB_TASK_STATUS_PARTIAL_SUCCEED) { QW_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); diff --git a/source/libs/qworker/src/qwMsg.c b/source/libs/qworker/src/qwMsg.c index 5635ec8fc61c2de62b7de4a34168fdcf03e40820..82a62b5c5a65c5a8e62e4103b674c81b4fe6b03a 100644 --- a/source/libs/qworker/src/qwMsg.c +++ b/source/libs/qworker/src/qwMsg.c @@ -283,6 +283,26 @@ int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) { return TSDB_CODE_SUCCESS; } +int32_t qWorkerAbortPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) { + if (NULL == qWorkerMgmt || NULL == pMsg) { + QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); + } + + SSubQueryMsg *msg = pMsg->pCont; + SQWorker * mgmt = (SQWorker *)qWorkerMgmt; + + uint64_t sId = msg->sId; + uint64_t qId = msg->queryId; + uint64_t tId = msg->taskId; + int64_t rId = msg->refId; + + QW_SCH_TASK_DLOG("Abort prerocessQuery start, handle:%p", pMsg->info.handle); + qwAbortPrerocessQuery(QW_FPARAMS()); + QW_SCH_TASK_DLOG("Abort prerocessQuery end, handle:%p", pMsg->info.handle); + + return TSDB_CODE_SUCCESS; +} + int32_t qWorkerProcessQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts) { if (NULL == node || NULL == qWorkerMgmt || NULL == pMsg) { QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); diff --git a/source/libs/qworker/src/qworker.c b/source/libs/qworker/src/qworker.c index 57ebb89ed22c86c01b911a1e7a390e2bc485d07a..28181d7e11291a984ae53b0bff2bc46282977548 100644 --- a/source/libs/qworker/src/qworker.c +++ b/source/libs/qworker/src/qworker.c @@ -482,6 +482,13 @@ _return: QW_RET(code); } +int32_t qwAbortPrerocessQuery(QW_FPARAMS_DEF) { + QW_ERR_RET(qwDropTask(QW_FPARAMS())); + + QW_RET(TSDB_CODE_SUCCESS); +} + + int32_t qwPrerocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) { int32_t code = 0; bool queryRsped = false; @@ -526,7 +533,7 @@ int32_t qwProcessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg, int8_t taskType, int8_t ex atomic_store_8(&ctx->taskType, taskType); atomic_store_8(&ctx->explain, explain); - /*QW_TASK_DLOGL("subplan json string, len:%d, %s", qwMsg->msgLen, qwMsg->msg);*/ + QW_TASK_DLOGL("subplan json string, len:%d, %s", qwMsg->msgLen, qwMsg->msg); code = qStringToSubplan(qwMsg->msg, &plan); if (TSDB_CODE_SUCCESS != code) { @@ -614,6 +621,8 @@ int32_t qwProcessCQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) { QW_SET_EVENT_PROCESSED(ctx, QW_EVENT_FETCH); qwBuildAndSendFetchRsp(&qwMsg->connInfo, rsp, dataLen, code); + rsp = NULL; + QW_TASK_DLOG("fetch rsp send, handle:%p, code:%x - %s, dataLen:%d", qwMsg->connInfo.handle, code, tstrerror(code), dataLen); } else { @@ -633,7 +642,7 @@ int32_t qwProcessCQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) { rsp = NULL; qwMsg->connInfo = ctx->dataConnInfo; - qwBuildAndSendFetchRsp(&qwMsg->connInfo, rsp, 0, code); + qwBuildAndSendFetchRsp(&qwMsg->connInfo, NULL, 0, code); QW_TASK_DLOG("fetch rsp send, handle:%p, code:%x - %s, dataLen:%d", qwMsg->connInfo.handle, code, tstrerror(code), 0); } diff --git a/source/libs/scheduler/inc/schedulerInt.h b/source/libs/scheduler/inc/schedulerInt.h index 8875ebdf736662af357084eed98137791687c5b4..6b2570c5b70132032c7df84107aa2b294b79233a 100644 --- a/source/libs/scheduler/inc/schedulerInt.h +++ b/source/libs/scheduler/inc/schedulerInt.h @@ -48,6 +48,12 @@ enum { SCH_FETCH_CB, }; +typedef enum { + SCH_OP_NULL = 0, + SCH_OP_EXEC, + SCH_OP_FETCH, +} SCH_OP_TYPE; + typedef struct SSchTrans { void *pTrans; void *pHandle; @@ -188,11 +194,15 @@ typedef struct SSchTask { typedef struct SSchJobAttr { EExplainMode explainMode; - bool syncSchedule; bool queryJob; bool needFlowCtrl; } SSchJobAttr; +typedef struct { + int32_t op; + bool sync; +} SSchOpStatus; + typedef struct SSchJob { int64_t refId; uint64_t queryId; @@ -200,7 +210,7 @@ typedef struct SSchJob { int32_t levelNum; int32_t taskNum; SRequestConnInfo conn; - SArray *nodeList; // qnode/vnode list, SArray + SArray *nodeList; // qnode/vnode list, SArray SArray *levels; // starting from 0. SArray SNodeList *subPlans; // subplan pointer copied from DAG, no need to free it in scheduler @@ -217,8 +227,8 @@ typedef struct SSchJob { int8_t status; SQueryNodeAddr resNode; tsem_t rspSem; - int8_t userFetch; - int32_t remoteFetch; + SSchOpStatus opStatus; + bool *reqKilled; SSchTask *fetchTask; int32_t errCode; SRWLatch resLock; @@ -227,7 +237,6 @@ typedef struct SSchJob { int32_t resNumOfRows; SSchResInfo userRes; const char *sql; - int32_t userCb; SQueryProfileSummary summary; } SSchJob; @@ -285,6 +294,10 @@ extern SSchedulerMgmt schMgmt; #define SCH_GET_JOB_STATUS(job) atomic_load_8(&(job)->status) #define SCH_GET_JOB_STATUS_STR(job) jobTaskStatusStr(SCH_GET_JOB_STATUS(job)) +#define SCH_JOB_IN_SYNC_OP(job) ((job)->opStatus.op && (job)->opStatus.sync) +#define SCH_JOB_IN_ASYNC_EXEC_OP(job) (((job)->opStatus.op == SCH_OP_EXEC) && (!(job)->opStatus.sync)) +#define SCH_JOB_IN_ASYNC_FETCH_OP(job) (((job)->opStatus.op == SCH_OP_FETCH) && (!(job)->opStatus.sync)) + #define SCH_SET_JOB_NEED_FLOW_CTRL(_job) (_job)->attr.needFlowCtrl = true #define SCH_JOB_NEED_FLOW_CTRL(_job) ((_job)->attr.needFlowCtrl) #define SCH_TASK_NEED_FLOW_CTRL(_job, _task) (SCH_IS_DATA_SRC_QRY_TASK(_task) && SCH_JOB_NEED_FLOW_CTRL(_job) && SCH_IS_LEVEL_UNFINISHED((_task)->level)) @@ -356,7 +369,7 @@ int32_t schMakeBrokenLinkVal(SSchJob *pJob, SSchTask *pTask, SRpcBrokenlinkVal * int32_t schAppendTaskExecNode(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr, int32_t execIdx); int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync); int32_t schExecJobImpl(SSchedulerReq *pReq, int64_t *job, SQueryResult* pRes, bool sync); -int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus); +int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus); int32_t schCancelJob(SSchJob *pJob); int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode); uint64_t schGenTaskId(void); @@ -368,6 +381,8 @@ int32_t schAsyncFetchRows(SSchJob *pJob); int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execIdx); int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId* pEpId, SArray* pStatusList); void schFreeSMsgSendInfo(SMsgSendInfo *msgSendInfo); +char* schGetOpStr(SCH_OP_TYPE type); +int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync); #ifdef __cplusplus diff --git a/source/libs/scheduler/src/schJob.c b/source/libs/scheduler/src/schJob.c index 31536f413dedc73299b4370adf62b92a196a23a2..72809e1f93f7cc827ce41036ba5ed5dcf8734e86 100644 --- a/source/libs/scheduler/src/schJob.c +++ b/source/libs/scheduler/src/schJob.c @@ -21,9 +21,9 @@ #include "tref.h" #include "trpc.h" -FORCE_INLINE SSchJob *schAcquireJob(int64_t refId) { qDebug("acquire jobId:0x%"PRIx64, refId); return (SSchJob *)taosAcquireRef(schMgmt.jobRef, refId); } +FORCE_INLINE SSchJob *schAcquireJob(int64_t refId) { qDebug("sch acquire jobId:0x%"PRIx64, refId); return (SSchJob *)taosAcquireRef(schMgmt.jobRef, refId); } -FORCE_INLINE int32_t schReleaseJob(int64_t refId) { qDebug("release jobId:0x%"PRIx64, refId); return taosReleaseRef(schMgmt.jobRef, refId); } +FORCE_INLINE int32_t schReleaseJob(int64_t refId) { qDebug("sch release jobId:0x%"PRIx64, refId); return taosReleaseRef(schMgmt.jobRef, refId); } int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel) { pTask->plan = pPlan; @@ -47,14 +47,14 @@ int32_t schInitJob(SSchedulerReq *pReq, SSchJob **pSchJob, SQueryResult* pRes, b int64_t refId = -1; SSchJob *pJob = taosMemoryCalloc(1, sizeof(SSchJob)); if (NULL == pJob) { - qError("QID:%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); + qError("QID:0x%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } pJob->attr.explainMode = pReq->pDag->explainInfo.mode; - pJob->attr.syncSchedule = syncSchedule; pJob->conn = *pReq->pConn; pJob->sql = pReq->sql; + pJob->reqKilled = pReq->reqKilled; pJob->userRes.queryRes = pRes; pJob->userRes.execFp = pReq->fp; pJob->userRes.userParam = pReq->cbParam; @@ -108,7 +108,7 @@ int32_t schInitJob(SSchedulerReq *pReq, SSchJob **pSchJob, SQueryResult* pRes, b atomic_add_fetch_32(&schMgmt.jobNum, 1); if (NULL == schAcquireJob(refId)) { - SCH_JOB_ELOG("schAcquireJob job failed, refId:%" PRIx64, refId); + SCH_JOB_ELOG("schAcquireJob job failed, refId:0x%" PRIx64, refId); SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); } @@ -116,7 +116,7 @@ int32_t schInitJob(SSchedulerReq *pReq, SSchJob **pSchJob, SQueryResult* pRes, b SCH_JOB_DLOG("job refId:0x%" PRIx64" created", pJob->refId); - pJob->status = JOB_TASK_STATUS_NOT_START; + schUpdateJobStatus(pJob, JOB_TASK_STATUS_NOT_START); *pSchJob = pJob; @@ -155,18 +155,57 @@ void schFreeTask(SSchJob *pJob, SSchTask *pTask) { } } + +void schUpdateJobErrCode(SSchJob *pJob, int32_t errCode) { + if (TSDB_CODE_SUCCESS == errCode) { + return; + } + + int32_t origCode = atomic_load_32(&pJob->errCode); + if (TSDB_CODE_SUCCESS == origCode) { + if (origCode == atomic_val_compare_exchange_32(&pJob->errCode, origCode, errCode)) { + goto _return; + } + + origCode = atomic_load_32(&pJob->errCode); + } + + if (NEED_CLIENT_HANDLE_ERROR(origCode)) { + return; + } + + if (NEED_CLIENT_HANDLE_ERROR(errCode)) { + atomic_store_32(&pJob->errCode, errCode); + goto _return; + } + + return; + +_return: + + SCH_JOB_DLOG("job errCode updated to %x - %s", errCode, tstrerror(errCode)); +} + + + FORCE_INLINE bool schJobNeedToStop(SSchJob *pJob, int8_t *pStatus) { int8_t status = SCH_GET_JOB_STATUS(pJob); if (pStatus) { *pStatus = status; } - return (status == JOB_TASK_STATUS_FAILED || status == JOB_TASK_STATUS_CANCELLED || - status == JOB_TASK_STATUS_CANCELLING || status == JOB_TASK_STATUS_DROPPING || + if (*pJob->reqKilled) { + schUpdateJobStatus(pJob, JOB_TASK_STATUS_DROPPING); + schUpdateJobErrCode(pJob, TSDB_CODE_TSC_QUERY_KILLED); + + return true; + } + + return (status == JOB_TASK_STATUS_FAILED || status == JOB_TASK_STATUS_DROPPING || status == JOB_TASK_STATUS_SUCCEED); } -int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { +int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { int32_t code = 0; int8_t oriStatus = 0; @@ -175,7 +214,11 @@ int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { oriStatus = SCH_GET_JOB_STATUS(pJob); if (oriStatus == newStatus) { - SCH_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); + if (newStatus == JOB_TASK_STATUS_DROPPING) { + SCH_ERR_JRET(TSDB_CODE_SCH_JOB_IS_DROPPING); + } + + SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR); } switch (oriStatus) { @@ -186,14 +229,13 @@ int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { break; case JOB_TASK_STATUS_NOT_START: - if (newStatus != JOB_TASK_STATUS_EXECUTING) { + if (newStatus != JOB_TASK_STATUS_EXECUTING && newStatus != JOB_TASK_STATUS_DROPPING) { SCH_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; case JOB_TASK_STATUS_EXECUTING: if (newStatus != JOB_TASK_STATUS_PARTIAL_SUCCEED && newStatus != JOB_TASK_STATUS_FAILED && - newStatus != JOB_TASK_STATUS_CANCELLING && newStatus != JOB_TASK_STATUS_CANCELLED && newStatus != JOB_TASK_STATUS_DROPPING) { SCH_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } @@ -208,13 +250,11 @@ int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { break; case JOB_TASK_STATUS_SUCCEED: case JOB_TASK_STATUS_FAILED: - case JOB_TASK_STATUS_CANCELLING: if (newStatus != JOB_TASK_STATUS_DROPPING) { SCH_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); } break; - case JOB_TASK_STATUS_CANCELLED: case JOB_TASK_STATUS_DROPPING: SCH_ERR_JRET(TSDB_CODE_QRY_JOB_FREED); break; @@ -238,8 +278,67 @@ int32_t schChkUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { _return: SCH_JOB_ELOG("invalid job status update, from %s to %s", jobTaskStatusStr(oriStatus), jobTaskStatusStr(newStatus)); - SCH_ERR_RET(code); + SCH_RET(code); +} + + +void schEndOperation(SSchJob *pJob) { + int32_t op = atomic_load_32(&pJob->opStatus.op); + if (SCH_OP_NULL == op) { + SCH_JOB_DLOG("job already not in any operation, status:%s", jobTaskStatusStr(pJob->status)); + return; + } + + atomic_store_32(&pJob->opStatus.op, SCH_OP_NULL); + + SCH_JOB_DLOG("job end %s operation", schGetOpStr(op)); +} + +int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync) { + int32_t code = 0; + int8_t status = 0; + + if (schJobNeedToStop(pJob, &status)) { + SCH_JOB_ELOG("abort op %s cause of job need to stop", schGetOpStr(type)); + SCH_ERR_JRET(pJob->errCode); + } + + if (SCH_OP_NULL != atomic_val_compare_exchange_32(&pJob->opStatus.op, SCH_OP_NULL, type)) { + SCH_JOB_ELOG("job already in %s operation", schGetOpStr(pJob->opStatus.op)); + SCH_ERR_JRET(TSDB_CODE_TSC_APP_ERROR); + } + + SCH_JOB_DLOG("job start %s operation", schGetOpStr(pJob->opStatus.op)); + + pJob->opStatus.sync = sync; + + switch (type) { + case SCH_OP_EXEC: + SCH_ERR_JRET(schUpdateJobStatus(pJob, JOB_TASK_STATUS_EXECUTING)); + break; + case SCH_OP_FETCH: + if (!SCH_JOB_NEED_FETCH(pJob)) { + SCH_JOB_ELOG("no need to fetch data, status:%s", SCH_GET_JOB_STATUS_STR(pJob)); + SCH_ERR_JRET(TSDB_CODE_QRY_APP_ERROR); + } + + if (status != JOB_TASK_STATUS_PARTIAL_SUCCEED) { + SCH_JOB_ELOG("job status error for fetch, status:%s", jobTaskStatusStr(status)); + SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); + } + break; + default: + SCH_JOB_ELOG("unknown operation type %d", type); + SCH_ERR_JRET(TSDB_CODE_TSC_APP_ERROR); + } + return TSDB_CODE_SUCCESS; + +_return: + + schEndOperation(pJob); + + SCH_RET(code); } int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { @@ -278,7 +377,7 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - SCH_TASK_DLOG("children info, the %d child TID %" PRIx64, n, (*childTask)->taskId); + SCH_TASK_DLOG("children info, the %d child TID 0x%" PRIx64, n, (*childTask)->taskId); } if (parentNum > 0) { @@ -312,7 +411,7 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - SCH_TASK_DLOG("parents info, the %d parent TID %" PRIx64, n, (*parentTask)->taskId); + SCH_TASK_DLOG("parents info, the %d parent TID 0x%" PRIx64, n, (*parentTask)->taskId); } SCH_TASK_DLOG("level:%d, parentNum:%d, childNum:%d", i, parentNum, childNum); @@ -515,7 +614,7 @@ int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) { ++pJob->taskNum; } - SCH_JOB_DLOG("level initialized, taskNum:%d", taskNum); + SCH_JOB_DLOG("level %d initialized, taskNum:%d", i, taskNum); } SCH_ERR_JRET(schBuildTaskRalation(pJob, planToTask)); @@ -537,8 +636,9 @@ int32_t schSetAddrsFromNodeList(SSchJob *pJob, SSchTask *pTask) { nodeNum = taosArrayGetSize(pJob->nodeList); for (int32_t i = 0; i < nodeNum && addNum < SCH_MAX_CANDIDATE_EP_NUM; ++i) { - SQueryNodeAddr *naddr = taosArrayGet(pJob->nodeList, i); - + SQueryNodeLoad *nload = taosArrayGet(pJob->nodeList, i); + SQueryNodeAddr *naddr = &nload->addr; + if (NULL == taosArrayPush(pTask->candidateAddrs, naddr)) { SCH_TASK_ELOG("taosArrayPush execNode to candidate addrs failed, addNum:%d, errno:%d", addNum, errno); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); @@ -552,7 +652,7 @@ int32_t schSetAddrsFromNodeList(SSchJob *pJob, SSchTask *pTask) { if (addNum <= 0) { SCH_TASK_ELOG("no available execNode as candidates, nodeNum:%d", nodeNum); - SCH_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); + SCH_ERR_RET(TSDB_CODE_TSC_NO_EXEC_NODE); } return TSDB_CODE_SUCCESS; @@ -577,7 +677,7 @@ int32_t schSetTaskCandidateAddrs(SSchJob *pJob, SSchTask *pTask) { SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); } - SCH_TASK_DLOG("use execNode from plan as candidate addr, numOfEps:%d", pTask->plan->execNode.epSet.numOfEps); + SCH_TASK_DLOG("use execNode in plan as candidate addr, numOfEps:%d", pTask->plan->execNode.epSet.numOfEps); return TSDB_CODE_SUCCESS; } @@ -785,37 +885,6 @@ int32_t schHandleTaskRetry(SSchJob *pJob, SSchTask *pTask) { return TSDB_CODE_SUCCESS; } -void schUpdateJobErrCode(SSchJob *pJob, int32_t errCode) { - if (TSDB_CODE_SUCCESS == errCode) { - return; - } - - int32_t origCode = atomic_load_32(&pJob->errCode); - if (TSDB_CODE_SUCCESS == origCode) { - if (origCode == atomic_val_compare_exchange_32(&pJob->errCode, origCode, errCode)) { - goto _return; - } - - origCode = atomic_load_32(&pJob->errCode); - } - - if (NEED_CLIENT_HANDLE_ERROR(origCode)) { - return; - } - - if (NEED_CLIENT_HANDLE_ERROR(errCode)) { - atomic_store_32(&pJob->errCode, errCode); - goto _return; - } - - return; - -_return: - - SCH_JOB_DLOG("job errCode updated to %x - %s", errCode, tstrerror(errCode)); -} - - int32_t schSetJobQueryRes(SSchJob* pJob, SQueryResult* pRes) { pRes->code = atomic_load_32(&pJob->errCode); pRes->numOfRows = pJob->resNumOfRows; @@ -828,7 +897,7 @@ int32_t schSetJobQueryRes(SSchJob* pJob, SQueryResult* pRes) { int32_t schSetJobFetchRes(SSchJob* pJob, void** pData) { int32_t code = 0; if (pJob->resData && ((SRetrieveTableRsp *)pJob->resData)->completed) { - SCH_ERR_RET(schChkUpdateJobStatus(pJob, JOB_TASK_STATUS_SUCCEED)); + SCH_ERR_RET(schUpdateJobStatus(pJob, JOB_TASK_STATUS_SUCCEED)); } while (true) { @@ -855,15 +924,17 @@ int32_t schSetJobFetchRes(SSchJob* pJob, void** pData) { return TSDB_CODE_SUCCESS; } -int32_t schNotifyUserQueryRes(SSchJob* pJob) { - pJob->userRes.queryRes = taosMemoryCalloc(1, sizeof(*pJob->userRes.queryRes)); - if (pJob->userRes.queryRes) { - schSetJobQueryRes(pJob, pJob->userRes.queryRes); +int32_t schNotifyUserExecRes(SSchJob* pJob) { + SQueryResult* pRes = taosMemoryCalloc(1, sizeof(SQueryResult)); + if (pRes) { + schSetJobQueryRes(pJob, pRes); } - (*pJob->userRes.execFp)(pJob->userRes.queryRes, pJob->userRes.userParam, atomic_load_32(&pJob->errCode)); + schEndOperation(pJob); - pJob->userRes.queryRes = NULL; + SCH_JOB_DLOG("sch start to invoke exec cb, code: %s", tstrerror(pJob->errCode)); + (*pJob->userRes.execFp)(pRes, pJob->userRes.userParam, atomic_load_32(&pJob->errCode)); + SCH_JOB_DLOG("sch end from query cb, code: %s", tstrerror(pJob->errCode)); return TSDB_CODE_SUCCESS; } @@ -871,36 +942,52 @@ int32_t schNotifyUserQueryRes(SSchJob* pJob) { int32_t schNotifyUserFetchRes(SSchJob* pJob) { void* pRes = NULL; - SCH_ERR_RET(schSetJobFetchRes(pJob, &pRes)); + schSetJobFetchRes(pJob, &pRes); + + schEndOperation(pJob); + SCH_JOB_DLOG("sch start to invoke fetch cb, code: %s", tstrerror(pJob->errCode)); (*pJob->userRes.fetchFp)(pRes, pJob->userRes.userParam, atomic_load_32(&pJob->errCode)); + SCH_JOB_DLOG("sch end from fetch cb, code: %s", tstrerror(pJob->errCode)); return TSDB_CODE_SUCCESS; } +void schPostJobRes(SSchJob *pJob, SCH_OP_TYPE op) { + if (SCH_OP_NULL == pJob->opStatus.op) { + SCH_JOB_DLOG("job not in any op, no need to post job res, status:%s", jobTaskStatusStr(pJob->status)); + return; + } + + if (op && pJob->opStatus.op != op) { + SCH_JOB_ELOG("job in op %s mis-match with expected %s", schGetOpStr(pJob->opStatus.op), schGetOpStr(op)); + return; + } + + if (SCH_JOB_IN_SYNC_OP(pJob)) { + tsem_post(&pJob->rspSem); + } else if (SCH_JOB_IN_ASYNC_EXEC_OP(pJob)) { + schNotifyUserExecRes(pJob); + } else if (SCH_JOB_IN_ASYNC_FETCH_OP(pJob)) { + schNotifyUserFetchRes(pJob); + } else { + SCH_JOB_ELOG("job not in any operation, status:%s", jobTaskStatusStr(pJob->status)); + } +} + int32_t schProcessOnJobFailureImpl(SSchJob *pJob, int32_t status, int32_t errCode) { // if already FAILED, no more processing - SCH_ERR_RET(schChkUpdateJobStatus(pJob, status)); + SCH_ERR_RET(schUpdateJobStatus(pJob, status)); schUpdateJobErrCode(pJob, errCode); - - if (atomic_load_8(&pJob->userFetch) || pJob->attr.syncSchedule) { - tsem_post(&pJob->rspSem); - } - + int32_t code = atomic_load_32(&pJob->errCode); - - SCH_JOB_DLOG("job failed with error: %s", tstrerror(code)); - - if (!pJob->attr.syncSchedule) { - if (SCH_EXEC_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_EXEC_CB, 0)) { - schNotifyUserQueryRes(pJob); - } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_FETCH_CB, 0)) { - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); - schNotifyUserFetchRes(pJob); - } + if (code) { + SCH_JOB_DLOG("job failed with error: %s", tstrerror(code)); } + schPostJobRes(pJob, 0); + SCH_RET(code); } @@ -918,20 +1005,9 @@ int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode) { int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) { int32_t code = 0; - SCH_ERR_RET(schChkUpdateJobStatus(pJob, JOB_TASK_STATUS_PARTIAL_SUCCEED)); - - if (pJob->attr.syncSchedule) { - tsem_post(&pJob->rspSem); - } else if (SCH_EXEC_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_EXEC_CB, 0)) { - schNotifyUserQueryRes(pJob); - } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&pJob->userCb, SCH_FETCH_CB, 0)) { - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); - schNotifyUserFetchRes(pJob); - } + SCH_ERR_RET(schUpdateJobStatus(pJob, JOB_TASK_STATUS_PARTIAL_SUCCEED)); - if (atomic_load_8(&pJob->userFetch)) { - SCH_ERR_JRET(schFetchFromRemote(pJob)); - } + schPostJobRes(pJob, SCH_OP_EXEC); return TSDB_CODE_SUCCESS; @@ -940,16 +1016,8 @@ _return: SCH_RET(schProcessOnJobFailure(pJob, code)); } -void schProcessOnDataFetched(SSchJob *job) { - atomic_val_compare_exchange_32(&job->remoteFetch, 1, 0); - - if (job->attr.syncSchedule) { - tsem_post(&job->rspSem); - } else if (SCH_FETCH_CB == atomic_val_compare_exchange_32(&job->userCb, SCH_FETCH_CB, 0)) { - atomic_val_compare_exchange_8(&job->userFetch, 1, 0); - - schNotifyUserFetchRes(job); - } +void schProcessOnDataFetched(SSchJob *pJob) { + schPostJobRes(pJob, SCH_OP_FETCH); } // Note: no more task error processing, handled in function internal @@ -1109,7 +1177,7 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) { SCH_UNLOCK(SCH_WRITE, &parent->lock); if (SCH_TASK_READY_FOR_LAUNCH(readyNum, parent)) { - SCH_TASK_DLOG("all %d children task done, start to launch parent task %" PRIx64, readyNum, parent->taskId); + SCH_TASK_DLOG("all %d children task done, start to launch parent task 0x%" PRIx64, readyNum, parent->taskId); SCH_ERR_RET(schLaunchTask(pJob, parent)); } } @@ -1127,15 +1195,8 @@ _return: int32_t schFetchFromRemote(SSchJob *pJob) { int32_t code = 0; - if (atomic_val_compare_exchange_32(&pJob->remoteFetch, 0, 1) != 0) { - SCH_JOB_ELOG("prior fetching not finished, remoteFetch:%d", atomic_load_32(&pJob->remoteFetch)); - return TSDB_CODE_SUCCESS; - } - void *resData = atomic_load_ptr(&pJob->resData); if (resData) { - atomic_val_compare_exchange_32(&pJob->remoteFetch, 1, 0); - SCH_JOB_DLOG("res already fetched, res:%p", resData); return TSDB_CODE_SUCCESS; } @@ -1146,8 +1207,6 @@ int32_t schFetchFromRemote(SSchJob *pJob) { _return: - atomic_val_compare_exchange_32(&pJob->remoteFetch, 1, 0); - SCH_RET(schProcessOnTaskFailure(pJob, pJob->fetchTask, code)); } @@ -1291,7 +1350,7 @@ int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTas int32_t schGetTaskInJob(SSchJob *pJob, uint64_t taskId, SSchTask **pTask) { schGetTaskFromList(pJob->taskList, taskId, pTask); if (NULL == *pTask) { - SCH_JOB_ELOG("task not found in job task list, taskId:%" PRIx64, taskId); + SCH_JOB_ELOG("task not found in job task list, taskId:0x%" PRIx64, taskId); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } @@ -1382,8 +1441,6 @@ int32_t schLaunchLevelTasks(SSchJob *pJob, SSchLevel *level) { int32_t schLaunchJob(SSchJob *pJob) { SSchLevel *level = taosArrayGet(pJob->levels, pJob->levelIdx); - SCH_ERR_RET(schChkUpdateJobStatus(pJob, JOB_TASK_STATUS_EXECUTING)); - SCH_ERR_RET(schChkJobNeedFlowCtrl(pJob, level)); SCH_ERR_RET(schLaunchLevelTasks(pJob, level)); @@ -1466,9 +1523,9 @@ void schFreeJobImpl(void *job) { taosMemoryFreeClear(pJob->userRes.queryRes); taosMemoryFreeClear(pJob->resData); - taosMemoryFreeClear(pJob); + taosMemoryFree(pJob); - qDebug("QID:0x%" PRIx64 " job freed, refId:%" PRIx64 ", pointer:%p", queryId, refId, pJob); + qDebug("QID:0x%" PRIx64 " sch job freed, refId:0x%" PRIx64 ", pointer:%p", queryId, refId, pJob); int32_t jobNum = atomic_sub_fetch_32(&schMgmt.jobNum, 1); if (jobNum == 0) { @@ -1483,26 +1540,36 @@ int32_t schExecJobImpl(SSchedulerReq *pReq, int64_t *job, SQueryResult* pRes, bo int32_t code = 0; SSchJob *pJob = NULL; - SCH_ERR_RET(schInitJob(pReq, &pJob, pRes, sync)); + SCH_ERR_JRET(schInitJob(pReq, &pJob, pRes, sync)); - qDebug("QID:0x%" PRIx64 " job refId 0x%"PRIx64 " started", pReq->pDag->queryId, pJob->refId); + qDebug("QID:0x%" PRIx64 " sch job refId 0x%"PRIx64 " started", pReq->pDag->queryId, pJob->refId); *job = pJob->refId; - if (!sync) { - pJob->userCb = SCH_EXEC_CB; - } + SCH_ERR_JRET(schBeginOperation(pJob, SCH_OP_EXEC, sync)); - SCH_ERR_JRET(schLaunchJob(pJob)); + code = schLaunchJob(pJob); if (sync) { SCH_JOB_DLOG("will wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); tsem_wait(&pJob->rspSem); + + schEndOperation(pJob); + } else if (code) { + schPostJobRes(pJob, SCH_OP_EXEC); } - SCH_JOB_DLOG("job exec done, job status:%s, jobId:0x%"PRIx64, SCH_GET_JOB_STATUS_STR(pJob), pJob->refId); + SCH_JOB_DLOG("job exec done, job status:%s, jobId:0x%" PRIx64, SCH_GET_JOB_STATUS_STR(pJob), pJob->refId); + + schReleaseJob(pJob->refId); + + SCH_RET(code); _return: + if (!sync) { + pReq->fp(NULL, pReq->cbParam, code); + } + schReleaseJob(pJob->refId); SCH_RET(code); @@ -1536,10 +1603,10 @@ int32_t schAsyncExecJob(SSchedulerReq *pReq, int64_t *pJob) { *pJob = 0; if (EXPLAIN_MODE_STATIC == pReq->pDag->explainInfo.mode) { - SCH_ERR_RET(schExecStaticExplainJob(pReq, pJob, false)); - } else { - SCH_ERR_RET(schExecJobImpl(pReq, pJob, NULL, false)); + SCH_RET(schExecStaticExplainJob(pReq, pJob, false)); } + + SCH_ERR_RET(schExecJobImpl(pReq, pJob, NULL, false)); return code; } @@ -1550,19 +1617,29 @@ int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync) { int32_t code = 0; SSchJob *pJob = taosMemoryCalloc(1, sizeof(SSchJob)); if (NULL == pJob) { - qError("QID:%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); - SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); + qError("QID:0x%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); + code = TSDB_CODE_QRY_OUT_OF_MEMORY; + pReq->fp(NULL, pReq->cbParam, code); + SCH_ERR_RET(code); } pJob->sql = pReq->sql; + pJob->reqKilled = pReq->reqKilled; pJob->attr.queryJob = true; - pJob->attr.syncSchedule = sync; pJob->attr.explainMode = pReq->pDag->explainInfo.mode; pJob->queryId = pReq->pDag->queryId; pJob->subPlans = pReq->pDag->pSubplans; pJob->userRes.execFp = pReq->fp; pJob->userRes.userParam = pReq->cbParam; + schUpdateJobStatus(pJob, JOB_TASK_STATUS_NOT_START); + + code = schBeginOperation(pJob, SCH_OP_EXEC, sync); + if (code) { + pReq->fp(NULL, pReq->cbParam, code); + SCH_ERR_RET(code); + } + SCH_ERR_JRET(qExecStaticExplain(pReq->pDag, (SRetrieveTableRsp **)&pJob->resData)); int64_t refId = taosAddRef(schMgmt.jobRef, pJob); @@ -1572,21 +1649,23 @@ int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync) { } if (NULL == schAcquireJob(refId)) { - SCH_JOB_ELOG("schAcquireJob job failed, refId:%" PRIx64, refId); - SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); + SCH_JOB_ELOG("schAcquireJob job failed, refId:0x%" PRIx64, refId); + SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); } pJob->refId = refId; - SCH_JOB_DLOG("job refId:%" PRIx64, pJob->refId); + SCH_JOB_DLOG("job refId:0x%" PRIx64, pJob->refId); pJob->status = JOB_TASK_STATUS_PARTIAL_SUCCEED; *job = pJob->refId; SCH_JOB_DLOG("job exec done, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); - - if (!pJob->attr.syncSchedule) { - code = schNotifyUserQueryRes(pJob); + + if (!sync) { + schPostJobRes(pJob, SCH_OP_EXEC); + } else { + schEndOperation(pJob); } schReleaseJob(pJob->refId); @@ -1595,56 +1674,29 @@ int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync) { _return: + schEndOperation(pJob); + if (!sync) { + pReq->fp(NULL, pReq->cbParam, code); + } + schFreeJobImpl(pJob); + SCH_RET(code); } int32_t schFetchRows(SSchJob *pJob) { int32_t code = 0; - int8_t status = SCH_GET_JOB_STATUS(pJob); - if (status == JOB_TASK_STATUS_DROPPING) { - SCH_JOB_ELOG("job is dropping, status:%s", jobTaskStatusStr(status)); - SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); - } - - if (!SCH_JOB_NEED_FETCH(pJob)) { - SCH_JOB_ELOG("no need to fetch data, status:%s", SCH_GET_JOB_STATUS_STR(pJob)); - SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR); - } - - if (atomic_val_compare_exchange_8(&pJob->userFetch, 0, 1) != 0) { - SCH_JOB_ELOG("prior fetching not finished, userFetch:%d", atomic_load_8(&pJob->userFetch)); - SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR); - } - - if (JOB_TASK_STATUS_FAILED == status || JOB_TASK_STATUS_DROPPING == status) { - SCH_JOB_ELOG("job failed or dropping, status:%s", jobTaskStatusStr(status)); - SCH_ERR_JRET(atomic_load_32(&pJob->errCode)); - } else if (status == JOB_TASK_STATUS_SUCCEED) { - SCH_JOB_DLOG("job already succeed, status:%s", jobTaskStatusStr(status)); - goto _return; - } else if (status != JOB_TASK_STATUS_PARTIAL_SUCCEED) { - SCH_JOB_ELOG("job status error for fetch, status:%s", jobTaskStatusStr(status)); - SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); - } - if (!(pJob->attr.explainMode == EXPLAIN_MODE_STATIC)) { SCH_ERR_JRET(schFetchFromRemote(pJob)); tsem_wait(&pJob->rspSem); - - status = SCH_GET_JOB_STATUS(pJob); - if (JOB_TASK_STATUS_FAILED == status || JOB_TASK_STATUS_DROPPING == status) { - SCH_JOB_ELOG("job failed or dropping, status:%s", jobTaskStatusStr(status)); - SCH_ERR_JRET(atomic_load_32(&pJob->errCode)); - } } SCH_ERR_JRET(schSetJobFetchRes(pJob, pJob->userRes.fetchRes)); _return: - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); + schEndOperation(pJob); SCH_RET(code); } @@ -1652,50 +1704,14 @@ _return: int32_t schAsyncFetchRows(SSchJob *pJob) { int32_t code = 0; - int8_t status = SCH_GET_JOB_STATUS(pJob); - if (status == JOB_TASK_STATUS_DROPPING) { - SCH_JOB_ELOG("job is dropping, status:%s", jobTaskStatusStr(status)); - SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); - } - - if (!SCH_JOB_NEED_FETCH(pJob)) { - SCH_JOB_ELOG("no need to fetch data, status:%s", SCH_GET_JOB_STATUS_STR(pJob)); - SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR); - } - - if (atomic_val_compare_exchange_8(&pJob->userFetch, 0, 1) != 0) { - SCH_JOB_ELOG("prior fetching not finished, userFetch:%d", atomic_load_8(&pJob->userFetch)); - SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR); - } - - if (JOB_TASK_STATUS_FAILED == status || JOB_TASK_STATUS_DROPPING == status) { - SCH_JOB_ELOG("job failed or dropping, status:%s", jobTaskStatusStr(status)); - SCH_ERR_JRET(atomic_load_32(&pJob->errCode)); - } else if (status == JOB_TASK_STATUS_SUCCEED) { - SCH_JOB_DLOG("job already succeed, status:%s", jobTaskStatusStr(status)); - goto _return; - } else if (status != JOB_TASK_STATUS_PARTIAL_SUCCEED) { - SCH_JOB_ELOG("job status error for fetch, status:%s", jobTaskStatusStr(status)); - SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); - } - if (pJob->attr.explainMode == EXPLAIN_MODE_STATIC) { - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); - - SCH_ERR_JRET(schNotifyUserFetchRes(pJob)); - } else { - pJob->userCb = SCH_FETCH_CB; - - SCH_ERR_JRET(schFetchFromRemote(pJob)); + schPostJobRes(pJob, SCH_OP_FETCH); + return TSDB_CODE_SUCCESS; } + + SCH_ERR_RET(schFetchFromRemote(pJob)); return TSDB_CODE_SUCCESS; - -_return: - - atomic_val_compare_exchange_8(&pJob->userFetch, 1, 0); - - SCH_RET(code); } diff --git a/source/libs/scheduler/src/schRemote.c b/source/libs/scheduler/src/schRemote.c index ddcb53d62f9d342d9bc0ef266163b53db1911108..0bd747785cb162ed6ad10b7c7fc6f5f77ab28d9e 100644 --- a/source/libs/scheduler/src/schRemote.c +++ b/source/libs/scheduler/src/schRemote.c @@ -315,8 +315,6 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t msgType, ch return TSDB_CODE_SUCCESS; } - atomic_val_compare_exchange_32(&pJob->remoteFetch, 1, 0); - SCH_ERR_JRET(schFetchFromRemote(pJob)); taosMemoryFreeClear(msg); @@ -346,7 +344,7 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t msgType, ch } case TDMT_VND_DROP_TASK_RSP: { // SHOULD NEVER REACH HERE - SCH_TASK_ELOG("invalid status to handle drop task rsp, refId:%" PRIx64, pJob->refId); + SCH_TASK_ELOG("invalid status to handle drop task rsp, refId:0x%" PRIx64, pJob->refId); SCH_ERR_JRET(TSDB_CODE_SCH_INTERNAL_ERROR); break; } @@ -376,7 +374,7 @@ int32_t schHandleCallback(void *param, const SDataBuf *pMsg, int32_t msgType, in SSchJob *pJob = schAcquireJob(pParam->refId); if (NULL == pJob) { - qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "taosAcquireRef job failed, may be dropped, refId:%" PRIx64, + qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "taosAcquireRef job failed, may be dropped, refId:0x%" PRIx64, pParam->queryId, pParam->taskId, pParam->refId); SCH_ERR_JRET(TSDB_CODE_QRY_JOB_FREED); } @@ -445,7 +443,7 @@ int32_t schHandleExplainCallback(void *param, const SDataBuf *pMsg, int32_t code int32_t schHandleDropCallback(void *param, const SDataBuf *pMsg, int32_t code) { SSchTaskCallbackParam *pParam = (SSchTaskCallbackParam *)param; - qDebug("QID:%" PRIx64 ",TID:%" PRIx64 " drop task rsp received, code:%x", pParam->queryId, pParam->taskId, code); + qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 " drop task rsp received, code:0x%x", pParam->queryId, pParam->taskId, code); taosMemoryFreeClear(param); return TSDB_CODE_SUCCESS; } diff --git a/source/libs/scheduler/src/schUtil.c b/source/libs/scheduler/src/schUtil.c index 66483187daa52cc28b54fd3a2a1317c705e9d483..73077cbf0f6632e0a7a7b9979e0b6d17baf90fd3 100644 --- a/source/libs/scheduler/src/schUtil.c +++ b/source/libs/scheduler/src/schUtil.c @@ -21,6 +21,18 @@ #include "tref.h" #include "trpc.h" +char* schGetOpStr(SCH_OP_TYPE type) { + switch (type) { + case SCH_OP_NULL: + return "NULL"; + case SCH_OP_EXEC: + return "EXEC"; + case SCH_OP_FETCH: + return "FETCH"; + default: + return "UNKNOWN"; + } +} void schCleanClusterHb(void* pTrans) { SCH_LOCK(SCH_WRITE, &schMgmt.hbLock); @@ -188,7 +200,7 @@ int32_t schUpdateHbConnection(SQueryNodeEpId *epId, SSchTrans *trans) { SCH_UNLOCK(SCH_WRITE, &hb->lock); SCH_UNLOCK(SCH_READ, &schMgmt.hbLock); - qDebug("hb connection updated, sId:%" PRIx64 ", nodeId:%d, fqdn:%s, port:%d, pTrans:%p, pHandle:%p", schMgmt.sId, + qDebug("hb connection updated, sId:0x%" PRIx64 ", nodeId:%d, fqdn:%s, port:%d, pTrans:%p, pHandle:%p", schMgmt.sId, epId->nodeId, epId->ep.fqdn, epId->ep.port, trans->pTrans, trans->pHandle); return TSDB_CODE_SUCCESS; diff --git a/source/libs/scheduler/src/scheduler.c b/source/libs/scheduler/src/scheduler.c index 0eaeeae9cb75edbe5c3b974af4acb61eee6e713d..57a405ffa3ffd524e7203a2935a4135ec3bd5208 100644 --- a/source/libs/scheduler/src/scheduler.c +++ b/source/libs/scheduler/src/scheduler.c @@ -62,12 +62,14 @@ int32_t schedulerInit(SSchedulerCfg *cfg) { SCH_ERR_RET(TSDB_CODE_QRY_SYS_ERROR); } - qInfo("scheduler %" PRIx64 " initizlized, maxJob:%u", schMgmt.sId, schMgmt.cfg.maxJobNum); + qInfo("scheduler 0x%" PRIx64 " initizlized, maxJob:%u", schMgmt.sId, schMgmt.cfg.maxJobNum); return TSDB_CODE_SUCCESS; } int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJob, SQueryResult *pRes) { + qDebug("scheduler sync exec job start"); + if (NULL == pReq || NULL == pJob || NULL == pRes) { SCH_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } @@ -76,21 +78,27 @@ int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJob, SQueryResult *pRes) } int32_t schedulerAsyncExecJob(SSchedulerReq *pReq, int64_t *pJob) { + qDebug("scheduler async exec job start"); + int32_t code = 0; if (NULL == pReq || NULL == pJob) { - code = TSDB_CODE_QRY_INVALID_INPUT; - } else { - code = schAsyncExecJob(pReq, pJob); + SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_INPUT); } + + schAsyncExecJob(pReq, pJob); + +_return: if (code != TSDB_CODE_SUCCESS) { pReq->fp(NULL, pReq->cbParam, code); } - return code; + SCH_RET(code); } int32_t schedulerFetchRows(int64_t job, void **pData) { + qDebug("scheduler sync fetch rows start"); + if (NULL == pData) { SCH_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); } @@ -102,7 +110,8 @@ int32_t schedulerFetchRows(int64_t job, void **pData) { SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); } - pJob->attr.syncSchedule = true; + SCH_ERR_RET(schBeginOperation(pJob, SCH_OP_FETCH, true)); + pJob->userRes.fetchRes = pData; code = schFetchRows(pJob); @@ -112,24 +121,32 @@ int32_t schedulerFetchRows(int64_t job, void **pData) { } void schedulerAsyncFetchRows(int64_t job, schedulerFetchCallback fp, void* param) { + qDebug("scheduler async fetch rows start"); + + int32_t code = 0; if (NULL == fp || NULL == param) { - fp(NULL, param, TSDB_CODE_QRY_INVALID_INPUT); - return; + SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_INPUT); } SSchJob *pJob = schAcquireJob(job); if (NULL == pJob) { - qError("acquire job from jobRef list failed, may be dropped, jobId:0x%" PRIx64, job); - fp(NULL, param, TSDB_CODE_SCH_STATUS_ERROR); - return; + qError("acquire sch job from job list failed, may be dropped, jobId:0x%" PRIx64, job); + SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); } - pJob->attr.syncSchedule = false; + SCH_ERR_JRET(schBeginOperation(pJob, SCH_OP_FETCH, false)); + pJob->userRes.fetchFp = fp; pJob->userRes.userParam = param; - /*code = */schAsyncFetchRows(pJob); + SCH_ERR_JRET(schAsyncFetchRows(pJob)); +_return: + + if (code) { + fp(NULL, param, code); + } + schReleaseJob(job); } @@ -137,12 +154,12 @@ int32_t schedulerGetTasksStatus(int64_t job, SArray *pSub) { int32_t code = 0; SSchJob *pJob = schAcquireJob(job); if (NULL == pJob) { - qDebug("acquire job from jobRef list failed, may not started or dropped, refId:%" PRIx64, job); + qDebug("acquire job from jobRef list failed, may not started or dropped, refId:0x%" PRIx64, job); SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); } if (pJob->status < JOB_TASK_STATUS_NOT_START || pJob->levelNum <= 0 || NULL == pJob->levels) { - qDebug("job not initialized or not executable job, refId:%" PRIx64, job); + qDebug("job not initialized or not executable job, refId:0x%" PRIx64, job); SCH_ERR_JRET(TSDB_CODE_SCH_STATUS_ERROR); } @@ -188,21 +205,23 @@ void schedulerStopQueryHb(void *pTrans) { schCleanClusterHb(pTrans); } -void schedulerFreeJob(int64_t job) { +void schedulerFreeJob(int64_t job, int32_t errCode) { SSchJob *pJob = schAcquireJob(job); if (NULL == pJob) { qError("acquire job from jobRef list failed, may be dropped, jobId:0x%" PRIx64, job); return; } - if (atomic_load_8(&pJob->userFetch) > 0) { - schProcessOnJobDropped(pJob, TSDB_CODE_QRY_JOB_FREED); + int32_t code = schProcessOnJobDropped(pJob, errCode); + if (TSDB_CODE_SCH_JOB_IS_DROPPING == code) { + SCH_JOB_DLOG("sch job is already dropping, refId:0x%" PRIx64, job); + return; } - SCH_JOB_DLOG("start to remove job from jobRef list, refId:%" PRIx64, job); + SCH_JOB_DLOG("start to remove job from jobRef list, refId:0x%" PRIx64, job); if (taosRemoveRef(schMgmt.jobRef, job)) { - SCH_JOB_ELOG("remove job from job list failed, refId:%" PRIx64, job); + SCH_JOB_ELOG("remove job from job list failed, refId:0x%" PRIx64, job); } schReleaseJob(job); diff --git a/source/libs/scheduler/test/schedulerTests.cpp b/source/libs/scheduler/test/schedulerTests.cpp index 9b624ee5cdecac00694dfdbf857fd106ba7c152d..e5cc3cd48188f51ad9431c3e21027292998213ad 100644 --- a/source/libs/scheduler/test/schedulerTests.cpp +++ b/source/libs/scheduler/test/schedulerTests.cpp @@ -457,7 +457,7 @@ void schtFreeQueryJob(int32_t freeThread) { int64_t job = queryJobRefId; if (job && atomic_val_compare_exchange_64(&queryJobRefId, job, 0)) { - schedulerFreeJob(job); + schedulerFreeJob(job, 0); if (freeThread) { if (++freeNum % schtTestPrintNum == 0) { printf("FreeNum:%d\n", freeNum); @@ -724,7 +724,7 @@ TEST(queryTest, normalCase) { schReleaseJob(job); - schedulerFreeJob(job); + schedulerFreeJob(job, 0); schtFreeQueryDag(&dag); @@ -828,7 +828,7 @@ TEST(queryTest, readyFirstCase) { schReleaseJob(job); - schedulerFreeJob(job); + schedulerFreeJob(job, 0); schtFreeQueryDag(&dag); @@ -940,7 +940,7 @@ TEST(queryTest, flowCtrlCase) { schReleaseJob(job); - schedulerFreeJob(job); + schedulerFreeJob(job, 0); schtFreeQueryDag(&dag); @@ -994,7 +994,7 @@ TEST(insertTest, normalCase) { ASSERT_EQ(code, 0); ASSERT_EQ(res.numOfRows, 20); - schedulerFreeJob(insertJobRefId); + schedulerFreeJob(insertJobRefId, 0); schedulerDestroy(); } diff --git a/source/libs/stream/src/stream.c b/source/libs/stream/src/stream.c index 6dcbfad957519fc50242da5d4b8d8d6f9add5ddc..38a1ad14b1b4abf124a3badf905f2b30a15a0f80 100644 --- a/source/libs/stream/src/stream.c +++ b/source/libs/stream/src/stream.c @@ -64,7 +64,7 @@ void streamTriggerByTimer(void* param, void* tmrId) { atomic_store_8(&pTask->triggerStatus, TASK_TRIGGER_STATUS__IN_ACTIVE); streamTaskInput(pTask, (SStreamQueueItem*)trigger); - streamLaunchByWrite(pTask, pTask->nodeId, pTask->pMsgCb); + streamLaunchByWrite(pTask, pTask->nodeId); } taosTmrReset(streamTriggerByTimer, (int32_t)pTask->triggerParam, pTask, streamEnv.timer, &pTask->timer); @@ -81,7 +81,7 @@ int32_t streamSetupTrigger(SStreamTask* pTask) { return 0; } -int32_t streamLaunchByWrite(SStreamTask* pTask, int32_t vgId, SMsgCb* pMsgCb) { +int32_t streamLaunchByWrite(SStreamTask* pTask, int32_t vgId) { int8_t execStatus = atomic_load_8(&pTask->status); if (execStatus == TASK_STATUS__IDLE || execStatus == TASK_STATUS__CLOSING) { SStreamTaskRunReq* pRunReq = rpcMallocCont(sizeof(SStreamTaskRunReq)); @@ -96,7 +96,7 @@ int32_t streamLaunchByWrite(SStreamTask* pTask, int32_t vgId, SMsgCb* pMsgCb) { .pCont = pRunReq, .contLen = sizeof(SStreamTaskRunReq), }; - tmsgPutToQueue(pMsgCb, FETCH_QUEUE, &msg); + tmsgPutToQueue(pTask->pMsgCb, FETCH_QUEUE, &msg); } return 0; } @@ -136,7 +136,9 @@ int32_t streamTaskEnqueue(SStreamTask* pTask, SStreamDispatchReq* pReq, SRpcMsg* return status == TASK_INPUT_STATUS__NORMAL ? 0 : -1; } -int32_t streamProcessDispatchReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDispatchReq* pReq, SRpcMsg* pRsp) { +int32_t streamProcessDispatchReq(SStreamTask* pTask, SStreamDispatchReq* pReq, SRpcMsg* pRsp) { + qInfo("task %d receive dispatch req from node %d task %d", pTask->taskId, pReq->upstreamNodeId, pReq->sourceTaskId); + // 1. handle input streamTaskEnqueue(pTask, pReq, pRsp); @@ -145,7 +147,7 @@ int32_t streamProcessDispatchReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDisp // 2.2. executing: return // 2.3. closing: keep trying if (pTask->execType != TASK_EXEC__NONE) { - streamExec(pTask, pMsgCb); + streamExec(pTask, pTask->pMsgCb); } else { ASSERT(pTask->sinkType != TASK_SINK__NONE); while (1) { @@ -161,34 +163,38 @@ int32_t streamProcessDispatchReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDisp // 3.1 check and set status // 3.2 dispatch / sink if (pTask->dispatchType != TASK_DISPATCH__NONE) { - streamDispatch(pTask, pMsgCb); + streamDispatch(pTask, pTask->pMsgCb); } return 0; } -int32_t streamProcessDispatchRsp(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamDispatchRsp* pRsp) { +int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp) { ASSERT(pRsp->inputStatus == TASK_OUTPUT_STATUS__NORMAL || pRsp->inputStatus == TASK_OUTPUT_STATUS__BLOCKED); + + qInfo("task %d receive dispatch rsp", pTask->taskId); + int8_t old = atomic_exchange_8(&pTask->outputStatus, pRsp->inputStatus); ASSERT(old == TASK_OUTPUT_STATUS__WAIT); if (pRsp->inputStatus == TASK_INPUT_STATUS__BLOCKED) { // TODO: init recover timer + ASSERT(0); return 0; } // continue dispatch - streamDispatch(pTask, pMsgCb); + streamDispatch(pTask, pTask->pMsgCb); return 0; } -int32_t streamTaskProcessRunReq(SStreamTask* pTask, SMsgCb* pMsgCb) { - streamExec(pTask, pMsgCb); +int32_t streamProcessRunReq(SStreamTask* pTask) { + streamExec(pTask, pTask->pMsgCb); if (pTask->dispatchType != TASK_DISPATCH__NONE) { - streamDispatch(pTask, pMsgCb); + streamDispatch(pTask, pTask->pMsgCb); } return 0; } -int32_t streamProcessRecoverReq(SStreamTask* pTask, SMsgCb* pMsgCb, SStreamTaskRecoverReq* pReq, SRpcMsg* pMsg) { +int32_t streamProcessRecoverReq(SStreamTask* pTask, SStreamTaskRecoverReq* pReq, SRpcMsg* pMsg) { // return 0; } diff --git a/source/libs/stream/src/streamDispatch.c b/source/libs/stream/src/streamDispatch.c index 59ec2b5cebcce508cc7b75bcea446ca17f9ddc43..1894f697c01d550fd4877d2dc533d527b4d51eb4 100644 --- a/source/libs/stream/src/streamDispatch.c +++ b/source/libs/stream/src/streamDispatch.c @@ -134,6 +134,7 @@ int32_t streamBuildDispatchMsg(SStreamTask* pTask, SStreamDataBlock* data, SRpcM int32_t sz = taosArrayGetSize(vgInfo); for (int32_t i = 0; i < sz; i++) { SVgroupInfo* pVgInfo = taosArrayGet(vgInfo, i); + ASSERT(pVgInfo->vgId > 0); if (hashValue >= pVgInfo->hashBegin && hashValue <= pVgInfo->hashEnd) { vgId = pVgInfo->vgId; downstreamTaskId = pVgInfo->taskId; @@ -143,7 +144,7 @@ int32_t streamBuildDispatchMsg(SStreamTask* pTask, SStreamDataBlock* data, SRpcM } } - ASSERT(vgId != 0); + ASSERT(vgId > 0 || vgId == SNODE_HANDLE); req.taskId = downstreamTaskId; qInfo("dispatch from task %d (child id %d) to down stream task %d in vnode %d", pTask->taskId, pTask->childId, @@ -198,6 +199,8 @@ int32_t streamDispatch(SStreamTask* pTask, SMsgCb* pMsgCb) { } ASSERT(pBlock->type == STREAM_DATA_TYPE_SSDATA_BLOCK); + qInfo("stream continue dispatching: task %d", pTask->taskId); + SRpcMsg dispatchMsg = {0}; SEpSet* pEpSet = NULL; if (streamBuildDispatchMsg(pTask, pBlock, &dispatchMsg, &pEpSet) < 0) { diff --git a/source/libs/stream/src/streamExec.c b/source/libs/stream/src/streamExec.c index 04428136ae614c109f173436d1f7203365bb080d..5a71fccab886f56ddc5c9de178879a0f1feb145a 100644 --- a/source/libs/stream/src/streamExec.c +++ b/source/libs/stream/src/streamExec.c @@ -107,18 +107,19 @@ int32_t streamExec(SStreamTask* pTask, SMsgCb* pMsgCb) { pRes = streamExecForQall(pTask, pRes); if (pRes == NULL) goto FAIL; - break; + taosArrayDestroy(pRes); + atomic_store_8(&pTask->status, TASK_STATUS__IDLE); + return 0; } else if (execStatus == TASK_STATUS__CLOSING) { continue; } else if (execStatus == TASK_STATUS__EXECUTING) { - break; + ASSERT(taosArrayGetSize(pRes) == 0); + taosArrayDestroy(pRes); + return 0; } else { ASSERT(0); } } - if (pRes) taosArrayDestroy(pRes); - atomic_store_8(&pTask->status, TASK_STATUS__IDLE); - return 0; FAIL: if (pRes) taosArrayDestroy(pRes); atomic_store_8(&pTask->status, TASK_STATUS__IDLE); diff --git a/source/libs/stream/src/streamTask.c b/source/libs/stream/src/streamTask.c index 7a7d9d15ad85c7026381283b265bb72b81f32c75..b5a63d937ae6be238406839921ba8a514568001a 100644 --- a/source/libs/stream/src/streamTask.c +++ b/source/libs/stream/src/streamTask.c @@ -36,11 +36,11 @@ int32_t tEncodeSStreamTask(SEncoder* pEncoder, const SStreamTask* pTask) { if (tEncodeI32(pEncoder, pTask->taskId) < 0) return -1; if (tEncodeI8(pEncoder, pTask->inputType) < 0) return -1; if (tEncodeI8(pEncoder, pTask->status) < 0) return -1; - if (tEncodeI8(pEncoder, pTask->sourceType) < 0) return -1; if (tEncodeI8(pEncoder, pTask->execType) < 0) return -1; if (tEncodeI8(pEncoder, pTask->sinkType) < 0) return -1; if (tEncodeI8(pEncoder, pTask->dispatchType) < 0) return -1; if (tEncodeI16(pEncoder, pTask->dispatchMsgType) < 0) return -1; + if (tEncodeI8(pEncoder, pTask->dataScan) < 0) return -1; if (tEncodeI32(pEncoder, pTask->childId) < 0) return -1; if (tEncodeI32(pEncoder, pTask->nodeId) < 0) return -1; @@ -70,7 +70,7 @@ int32_t tEncodeSStreamTask(SEncoder* pEncoder, const SStreamTask* pTask) { if (tEncodeSEpSet(pEncoder, &pTask->fixedEpDispatcher.epSet) < 0) return -1; } else if (pTask->dispatchType == TASK_DISPATCH__SHUFFLE) { if (tSerializeSUseDbRspImp(pEncoder, &pTask->shuffleDispatcher.dbInfo) < 0) return -1; - /*if (tEncodeI8(pEncoder, pTask->shuffleDispatcher.hashMethod) < 0) return -1;*/ + if (tEncodeCStr(pEncoder, pTask->shuffleDispatcher.stbFullName) < 0) return -1; } if (tEncodeI64(pEncoder, pTask->triggerParam) < 0) return -1; @@ -84,11 +84,11 @@ int32_t tDecodeSStreamTask(SDecoder* pDecoder, SStreamTask* pTask) { if (tDecodeI32(pDecoder, &pTask->taskId) < 0) return -1; if (tDecodeI8(pDecoder, &pTask->inputType) < 0) return -1; if (tDecodeI8(pDecoder, &pTask->status) < 0) return -1; - if (tDecodeI8(pDecoder, &pTask->sourceType) < 0) return -1; if (tDecodeI8(pDecoder, &pTask->execType) < 0) return -1; if (tDecodeI8(pDecoder, &pTask->sinkType) < 0) return -1; if (tDecodeI8(pDecoder, &pTask->dispatchType) < 0) return -1; if (tDecodeI16(pDecoder, &pTask->dispatchMsgType) < 0) return -1; + if (tDecodeI8(pDecoder, &pTask->dataScan) < 0) return -1; if (tDecodeI32(pDecoder, &pTask->childId) < 0) return -1; if (tDecodeI32(pDecoder, &pTask->nodeId) < 0) return -1; @@ -119,8 +119,8 @@ int32_t tDecodeSStreamTask(SDecoder* pDecoder, SStreamTask* pTask) { if (tDecodeI32(pDecoder, &pTask->fixedEpDispatcher.nodeId) < 0) return -1; if (tDecodeSEpSet(pDecoder, &pTask->fixedEpDispatcher.epSet) < 0) return -1; } else if (pTask->dispatchType == TASK_DISPATCH__SHUFFLE) { - /*if (tDecodeI8(pDecoder, &pTask->shuffleDispatcher.hashMethod) < 0) return -1;*/ if (tDeserializeSUseDbRspImp(pDecoder, &pTask->shuffleDispatcher.dbInfo) < 0) return -1; + if (tDecodeCStrTo(pDecoder, pTask->shuffleDispatcher.stbFullName) < 0) return -1; } if (tDecodeI64(pDecoder, &pTask->triggerParam) < 0) return -1; diff --git a/source/libs/sync/inc/syncInt.h b/source/libs/sync/inc/syncInt.h index b00c7cbda1fcc430b4dd7e52b72ac40e739a9158..63db395425c23d2c4ac50fda93b88b974c4787ab 100644 --- a/source/libs/sync/inc/syncInt.h +++ b/source/libs/sync/inc/syncInt.h @@ -159,7 +159,8 @@ typedef struct SSyncNode { SSyncSnapshotSender* senders[TSDB_MAX_REPLICA]; SSyncSnapshotReceiver* pNewNodeReceiver; - // SSnapshotMeta sMeta; + // is config changing + bool changing; } SSyncNode; @@ -171,7 +172,8 @@ void syncNodeClose(SSyncNode* pSyncNode); int32_t syncNodePropose(SSyncNode* pSyncNode, const SRpcMsg* pMsg, bool isWeak); // option -bool syncNodeSnapshotEnable(SSyncNode* pSyncNode); +bool syncNodeSnapshotEnable(SSyncNode* pSyncNode); +SyncIndex syncNodeGetSnapshotConfigIndex(SSyncNode* pSyncNode, SyncIndex snapshotLastApplyIndex); // ping -------------- int32_t syncNodePing(SSyncNode* pSyncNode, const SRaftId* destRaftId, SyncPing* pMsg); @@ -194,9 +196,10 @@ int32_t syncNodeSendMsgById(const SRaftId* destRaftId, SSyncNode* pSyncNode, SRp int32_t syncNodeSendMsgByInfo(const SNodeInfo* nodeInfo, SSyncNode* pSyncNode, SRpcMsg* pMsg); cJSON* syncNode2Json(const SSyncNode* pSyncNode); char* syncNode2Str(const SSyncNode* pSyncNode); +void syncNodeEventLog(const SSyncNode* pSyncNode, char* str); char* syncNode2SimpleStr(const SSyncNode* pSyncNode); bool syncNodeInConfig(SSyncNode* pSyncNode, const SSyncCfg* config); -void syncNodeUpdateConfig(SSyncNode* pSyncNode, SSyncCfg* newConfig, SyncIndex lastConfigChangeIndex, bool* isDrop); +void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* newConfig, SyncIndex lastConfigChangeIndex); SSyncNode* syncNodeAcquire(int64_t rid); void syncNodeRelease(SSyncNode* pNode); @@ -236,9 +239,15 @@ int32_t syncNodeUpdateNewConfigIndex(SSyncNode* ths, SSyncCfg* pNewCfg); bool syncNodeInRaftGroup(SSyncNode* ths, SRaftId* pRaftId); SSyncSnapshotSender* syncNodeGetSnapshotSender(SSyncNode* ths, SRaftId* pDestId); +int32_t syncGetSnapshotMeta(int64_t rid, struct SSnapshotMeta* sMeta); +int32_t syncGetSnapshotMetaByIndex(int64_t rid, SyncIndex snapshotIndex, struct SSnapshotMeta* sMeta); + void syncStartNormal(int64_t rid); void syncStartStandBy(int64_t rid); +bool syncNodeCanChange(SSyncNode* pSyncNode); +bool syncNodeCheckNewConfig(SSyncNode* pSyncNode, const SSyncCfg* pNewCfg); + // for debug -------------- void syncNodePrint(SSyncNode* pObj); void syncNodePrint2(char* s, SSyncNode* pObj); diff --git a/source/libs/sync/inc/syncRaftCfg.h b/source/libs/sync/inc/syncRaftCfg.h index e72e1e7be706523a0aec15b7488e6b61380e4f82..7f45276e9f2bb5891e96b22650fd3eddd37a338b 100644 --- a/source/libs/sync/inc/syncRaftCfg.h +++ b/source/libs/sync/inc/syncRaftCfg.h @@ -29,6 +29,8 @@ extern "C" { #define CONFIG_FILE_LEN 1024 +#define MAX_CONFIG_INDEX_COUNT 512 + typedef struct SRaftCfg { SSyncCfg cfg; TdFilePtr pFile; @@ -36,14 +38,20 @@ typedef struct SRaftCfg { int8_t isStandBy; int8_t snapshotEnable; SyncIndex lastConfigIndex; + + SyncIndex configIndexArr[MAX_CONFIG_INDEX_COUNT]; + int32_t configIndexCount; + } SRaftCfg; SRaftCfg *raftCfgOpen(const char *path); int32_t raftCfgClose(SRaftCfg *pRaftCfg); int32_t raftCfgPersist(SRaftCfg *pRaftCfg); +int32_t raftCfgAddConfigIndex(SRaftCfg *pRaftCfg, SyncIndex configIndex); cJSON * syncCfg2Json(SSyncCfg *pSyncCfg); char * syncCfg2Str(SSyncCfg *pSyncCfg); +char * syncCfg2SimpleStr(SSyncCfg *pSyncCfg); int32_t syncCfgFromJson(const cJSON *pRoot, SSyncCfg *pSyncCfg); int32_t syncCfgFromStr(const char *s, SSyncCfg *pSyncCfg); @@ -65,6 +73,7 @@ void syncCfgPrint(SSyncCfg *pCfg); void syncCfgPrint2(char *s, SSyncCfg *pCfg); void syncCfgLog(SSyncCfg *pCfg); void syncCfgLog2(char *s, SSyncCfg *pCfg); +void syncCfgLog3(char *s, SSyncCfg *pCfg); void raftCfgPrint(SRaftCfg *pCfg); void raftCfgPrint2(char *s, SRaftCfg *pCfg); diff --git a/source/libs/sync/inc/syncSnapshot.h b/source/libs/sync/inc/syncSnapshot.h index a6170a92e3f29b2b420803932fdc9469f46a5f68..069154fb93416c3b90840ed81ea8ac2bebf2d8b3 100644 --- a/source/libs/sync/inc/syncSnapshot.h +++ b/source/libs/sync/inc/syncSnapshot.h @@ -39,8 +39,8 @@ typedef struct SSyncSnapshotSender { bool start; int32_t seq; int32_t ack; - void * pReader; - void * pCurrentBlock; + void *pReader; + void *pCurrentBlock; int32_t blockLen; SSnapshot snapshot; SSyncCfg lastConfig; @@ -59,16 +59,18 @@ void snapshotSenderStart(SSyncSnapshotSender *pSender); void snapshotSenderStop(SSyncSnapshotSender *pSender); int32_t snapshotSend(SSyncSnapshotSender *pSender); int32_t snapshotReSend(SSyncSnapshotSender *pSender); -cJSON * snapshotSender2Json(SSyncSnapshotSender *pSender); -char * snapshotSender2Str(SSyncSnapshotSender *pSender); +cJSON *snapshotSender2Json(SSyncSnapshotSender *pSender); +char *snapshotSender2Str(SSyncSnapshotSender *pSender); +char *snapshotSender2SimpleStr(SSyncSnapshotSender *pSender, char *event); typedef struct SSyncSnapshotReceiver { bool start; - int32_t ack; - void * pWriter; - SyncTerm term; - SyncTerm privateTerm; + int32_t ack; + void *pWriter; + SyncTerm term; + SyncTerm privateTerm; + SSnapshot snapshot; SSyncNode *pSyncNode; SRaftId fromId; @@ -77,11 +79,12 @@ typedef struct SSyncSnapshotReceiver { SSyncSnapshotReceiver *snapshotReceiverCreate(SSyncNode *pSyncNode, SRaftId fromId); void snapshotReceiverDestroy(SSyncSnapshotReceiver *pReceiver); -void snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SRaftId fromId); -bool snapshotReceiverIsStart(SSyncSnapshotReceiver *pReceiver); -void snapshotReceiverStop(SSyncSnapshotReceiver *pReceiver, bool apply); -cJSON * snapshotReceiver2Json(SSyncSnapshotReceiver *pReceiver); -char * snapshotReceiver2Str(SSyncSnapshotReceiver *pReceiver); +void snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SyncSnapshotSend *pBeginMsg); +bool snapshotReceiverIsStart(SSyncSnapshotReceiver *pReceiver); +void snapshotReceiverStop(SSyncSnapshotReceiver *pReceiver, bool apply); +cJSON *snapshotReceiver2Json(SSyncSnapshotReceiver *pReceiver); +char *snapshotReceiver2Str(SSyncSnapshotReceiver *pReceiver); +char *snapshotReceiver2SimpleStr(SSyncSnapshotReceiver *pReceiver, char *event); int32_t syncNodeOnSnapshotSendCb(SSyncNode *ths, SyncSnapshotSend *pMsg); int32_t syncNodeOnSnapshotRspCb(SSyncNode *ths, SyncSnapshotRsp *pMsg); diff --git a/source/libs/sync/src/syncAppendEntries.c b/source/libs/sync/src/syncAppendEntries.c index 89b4761212bd4ea50451fed0821adb862df1dc98..d2726201cceb50c910e0cebfff23322c6747a1ef 100644 --- a/source/libs/sync/src/syncAppendEntries.c +++ b/source/libs/sync/src/syncAppendEntries.c @@ -208,8 +208,9 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { SRpcMsg rpcMsg; syncEntry2OriginalRpc(pRollBackEntry, &rpcMsg); - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pRollBackEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pRollBackEntry->isWeak; cbMeta.code = 0; cbMeta.state = ths->state; @@ -234,8 +235,9 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { if (ths->pFsm != NULL) { // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pAppendEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pAppendEntry->isWeak; cbMeta.code = 2; cbMeta.state = ths->state; @@ -266,8 +268,9 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { if (ths->pFsm != NULL) { // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pAppendEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pAppendEntry->isWeak; cbMeta.code = 3; cbMeta.state = ths->state; @@ -326,358 +329,6 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { return ret; } -#if 0 -int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { - int32_t ret = 0; - - char logBuf[128] = {0}; - snprintf(logBuf, sizeof(logBuf), "==syncNodeOnAppendEntriesCb== term:%lu", ths->pRaftStore->currentTerm); - syncAppendEntriesLog2(logBuf, pMsg); - - if (pMsg->term > ths->pRaftStore->currentTerm) { - syncNodeUpdateTerm(ths, pMsg->term); - } - assert(pMsg->term <= ths->pRaftStore->currentTerm); - - // reset elect timer - if (pMsg->term == ths->pRaftStore->currentTerm) { - ths->leaderCache = pMsg->srcId; - syncNodeResetElectTimer(ths); - } - assert(pMsg->dataLen >= 0); - - SyncTerm localPreLogTerm = 0; - if (pMsg->prevLogIndex >= SYNC_INDEX_BEGIN && pMsg->prevLogIndex <= ths->pLogStore->getLastIndex(ths->pLogStore)) { - SSyncRaftEntry* pEntry = ths->pLogStore->getEntry(ths->pLogStore, pMsg->prevLogIndex); - assert(pEntry != NULL); - localPreLogTerm = pEntry->term; - syncEntryDestory(pEntry); - } - - bool logOK = - (pMsg->prevLogIndex == SYNC_INDEX_INVALID) || - ((pMsg->prevLogIndex >= SYNC_INDEX_BEGIN) && - (pMsg->prevLogIndex <= ths->pLogStore->getLastIndex(ths->pLogStore)) && (pMsg->prevLogTerm == localPreLogTerm)); - - // reject request - if ((pMsg->term < ths->pRaftStore->currentTerm) || - ((pMsg->term == ths->pRaftStore->currentTerm) && (ths->state == TAOS_SYNC_STATE_FOLLOWER) && !logOK)) { - sTrace( - "syncNodeOnAppendEntriesCb --> reject, pMsg->term:%lu, ths->pRaftStore->currentTerm:%lu, ths->state:%d, " - "logOK:%d", - pMsg->term, ths->pRaftStore->currentTerm, ths->state, logOK); - - SyncAppendEntriesReply* pReply = syncAppendEntriesReplyBuild(ths->vgId); - pReply->srcId = ths->myRaftId; - pReply->destId = pMsg->srcId; - pReply->term = ths->pRaftStore->currentTerm; - pReply->success = false; - pReply->matchIndex = SYNC_INDEX_INVALID; - - SRpcMsg rpcMsg; - syncAppendEntriesReply2RpcMsg(pReply, &rpcMsg); - syncNodeSendMsgById(&pReply->destId, ths, &rpcMsg); - syncAppendEntriesReplyDestroy(pReply); - - return ret; - } - - // return to follower state - if (pMsg->term == ths->pRaftStore->currentTerm && ths->state == TAOS_SYNC_STATE_CANDIDATE) { - sTrace( - "syncNodeOnAppendEntriesCb --> return to follower, pMsg->term:%lu, ths->pRaftStore->currentTerm:%lu, " - "ths->state:%d, logOK:%d", - pMsg->term, ths->pRaftStore->currentTerm, ths->state, logOK); - - syncNodeBecomeFollower(ths, "from candidate by append entries"); - - // ret or reply? - return ret; - } - - // accept request - if (pMsg->term == ths->pRaftStore->currentTerm && ths->state == TAOS_SYNC_STATE_FOLLOWER && logOK) { - // preIndex = -1, or has preIndex entry in local log - assert(pMsg->prevLogIndex <= ths->pLogStore->getLastIndex(ths->pLogStore)); - - // has extra entries (> preIndex) in local log - bool hasExtraEntries = pMsg->prevLogIndex < ths->pLogStore->getLastIndex(ths->pLogStore); - - // has entries in SyncAppendEntries msg - bool hasAppendEntries = pMsg->dataLen > 0; - - sTrace( - "syncNodeOnAppendEntriesCb --> accept, pMsg->term:%lu, ths->pRaftStore->currentTerm:%lu, ths->state:%d, " - "logOK:%d, hasExtraEntries:%d, hasAppendEntries:%d", - pMsg->term, ths->pRaftStore->currentTerm, ths->state, logOK, hasExtraEntries, hasAppendEntries); - - if (hasExtraEntries && hasAppendEntries) { - // not conflict by default - bool conflict = false; - - SyncIndex extraIndex = pMsg->prevLogIndex + 1; - SSyncRaftEntry* pExtraEntry = ths->pLogStore->getEntry(ths->pLogStore, extraIndex); - assert(pExtraEntry != NULL); - - SSyncRaftEntry* pAppendEntry = syncEntryDeserialize(pMsg->data, pMsg->dataLen); - assert(pAppendEntry != NULL); - - // log not match, conflict - assert(extraIndex == pAppendEntry->index); - if (pExtraEntry->term != pAppendEntry->term) { - conflict = true; - } - - if (conflict) { - // roll back - SyncIndex delBegin = ths->pLogStore->getLastIndex(ths->pLogStore); - SyncIndex delEnd = extraIndex; - - sTrace("syncNodeOnAppendEntriesCb --> conflict:%d, delBegin:%ld, delEnd:%ld", conflict, delBegin, delEnd); - - // notice! reverse roll back! - for (SyncIndex index = delEnd; index >= delBegin; --index) { - if (ths->pFsm->FpRollBackCb != NULL) { - SSyncRaftEntry* pRollBackEntry = ths->pLogStore->getEntry(ths->pLogStore, index); - assert(pRollBackEntry != NULL); - - // if (pRollBackEntry->msgType != TDMT_SYNC_NOOP) { - if (syncUtilUserRollback(pRollBackEntry->msgType)) { - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pRollBackEntry, &rpcMsg); - - SFsmCbMeta cbMeta; - cbMeta.index = pRollBackEntry->index; - cbMeta.isWeak = pRollBackEntry->isWeak; - cbMeta.code = 0; - cbMeta.state = ths->state; - cbMeta.seqNum = pRollBackEntry->seqNum; - ths->pFsm->FpRollBackCb(ths->pFsm, &rpcMsg, cbMeta); - rpcFreeCont(rpcMsg.pCont); - } - - syncEntryDestory(pRollBackEntry); - } - } - - // delete confict entries - ths->pLogStore->truncate(ths->pLogStore, extraIndex); - - // append new entries - ths->pLogStore->appendEntry(ths->pLogStore, pAppendEntry); - - // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pAppendEntry, &rpcMsg); - if (ths->pFsm != NULL) { - // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta; - cbMeta.index = pAppendEntry->index; - cbMeta.isWeak = pAppendEntry->isWeak; - cbMeta.code = 2; - cbMeta.state = ths->state; - cbMeta.seqNum = pAppendEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); - } - - // free memory - syncEntryDestory(pExtraEntry); - syncEntryDestory(pAppendEntry); - - } else if (hasExtraEntries && !hasAppendEntries) { - // do nothing - - } else if (!hasExtraEntries && hasAppendEntries) { - SSyncRaftEntry* pAppendEntry = syncEntryDeserialize(pMsg->data, pMsg->dataLen); - assert(pAppendEntry != NULL); - - // append new entries - ths->pLogStore->appendEntry(ths->pLogStore, pAppendEntry); - - // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pAppendEntry, &rpcMsg); - if (ths->pFsm != NULL) { - // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta; - cbMeta.index = pAppendEntry->index; - cbMeta.isWeak = pAppendEntry->isWeak; - cbMeta.code = 3; - cbMeta.state = ths->state; - cbMeta.seqNum = pAppendEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); - - // free memory - syncEntryDestory(pAppendEntry); - - } else if (!hasExtraEntries && !hasAppendEntries) { - // do nothing - - } else { - assert(0); - } - - SyncAppendEntriesReply* pReply = syncAppendEntriesReplyBuild(ths->vgId); - pReply->srcId = ths->myRaftId; - pReply->destId = pMsg->srcId; - pReply->term = ths->pRaftStore->currentTerm; - pReply->success = true; - - if (hasAppendEntries) { - pReply->matchIndex = pMsg->prevLogIndex + 1; - } else { - pReply->matchIndex = pMsg->prevLogIndex; - } - - SRpcMsg rpcMsg; - syncAppendEntriesReply2RpcMsg(pReply, &rpcMsg); - syncNodeSendMsgById(&pReply->destId, ths, &rpcMsg); - syncAppendEntriesReplyDestroy(pReply); - - // maybe update commit index from leader - if (pMsg->commitIndex > ths->commitIndex) { - // has commit entry in local - if (pMsg->commitIndex <= ths->pLogStore->getLastIndex(ths->pLogStore)) { - SyncIndex beginIndex = ths->commitIndex + 1; - SyncIndex endIndex = pMsg->commitIndex; - - // update commit index - ths->commitIndex = pMsg->commitIndex; - - // call back Wal - ths->pLogStore->updateCommitIndex(ths->pLogStore, ths->commitIndex); - - // execute fsm - if (ths->pFsm != NULL) { - for (SyncIndex i = beginIndex; i <= endIndex; ++i) { - if (i != SYNC_INDEX_INVALID) { - SSyncRaftEntry* pEntry = ths->pLogStore->getEntry(ths->pLogStore, i); - assert(pEntry != NULL); - - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pEntry, &rpcMsg); - - if (ths->pFsm->FpCommitCb != NULL && syncUtilUserCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta; - cbMeta.index = pEntry->index; - cbMeta.isWeak = pEntry->isWeak; - cbMeta.code = 0; - cbMeta.state = ths->state; - cbMeta.seqNum = pEntry->seqNum; - cbMeta.term = pEntry->term; - cbMeta.currentTerm = ths->pRaftStore->currentTerm; - cbMeta.flag = 0x11; - - SSnapshot snapshot; - ASSERT(ths->pFsm->FpGetSnapshot != NULL); - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); - - bool needExecute = true; - if (cbMeta.index <= snapshot.lastApplyIndex) { - needExecute = false; - } - - if (needExecute) { - ths->pFsm->FpCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - - // config change - if (pEntry->originalRpcType == TDMT_SYNC_CONFIG_CHANGE) { - SSyncCfg oldSyncCfg = ths->pRaftCfg->cfg; - - SSyncCfg newSyncCfg; - int32_t ret = syncCfgFromStr(rpcMsg.pCont, &newSyncCfg); - ASSERT(ret == 0); - - // update new config myIndex - bool hit = false; - for (int i = 0; i < newSyncCfg.replicaNum; ++i) { - if (strcmp(ths->myNodeInfo.nodeFqdn, (newSyncCfg.nodeInfo)[i].nodeFqdn) == 0 && - ths->myNodeInfo.nodePort == (newSyncCfg.nodeInfo)[i].nodePort) { - newSyncCfg.myIndex = i; - hit = true; - break; - } - } - - SReConfigCbMeta cbMeta = {0}; - bool isDrop; - - // I am in newConfig - if (hit) { - syncNodeUpdateConfig(ths, &newSyncCfg, pEntry->index, &isDrop); - - // change isStandBy to normal - if (!isDrop) { - if (ths->state == TAOS_SYNC_STATE_LEADER) { - syncNodeBecomeLeader(ths, "config change"); - } else { - syncNodeBecomeFollower(ths, "config change"); - } - } - - if (gRaftDetailLog) { - char* sOld = syncCfg2Str(&oldSyncCfg); - char* sNew = syncCfg2Str(&newSyncCfg); - sInfo("==config change== 0x11 old:%s new:%s isDrop:%d \n", sOld, sNew, isDrop); - taosMemoryFree(sOld); - taosMemoryFree(sNew); - } - } - - // always call FpReConfigCb - if (ths->pFsm->FpReConfigCb != NULL) { - cbMeta.code = 0; - cbMeta.currentTerm = ths->pRaftStore->currentTerm; - cbMeta.index = pEntry->index; - cbMeta.term = pEntry->term; - cbMeta.newCfg = newSyncCfg; - cbMeta.oldCfg = oldSyncCfg; - cbMeta.seqNum = pEntry->seqNum; - cbMeta.flag = 0x11; - cbMeta.isDrop = isDrop; - ths->pFsm->FpReConfigCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - - // restore finish - if (pEntry->index == ths->pLogStore->getLastIndex(ths->pLogStore)) { - if (ths->restoreFinish == false) { - if (ths->pFsm->FpRestoreFinishCb != NULL) { - ths->pFsm->FpRestoreFinishCb(ths->pFsm); - } - ths->restoreFinish = true; - sInfo("==syncNodeOnAppendEntriesCb== restoreFinish set true %p vgId:%d", ths, ths->vgId); - - /* - tsem_post(&ths->restoreSem); - sInfo("==syncNodeOnAppendEntriesCb== RestoreFinish tsem_post %p", ths); - */ - } - } - - rpcFreeCont(rpcMsg.pCont); - syncEntryDestory(pEntry); - } - } - } - } - } - } - - return ret; -} -#endif - static int32_t syncNodeMakeLogSame(SSyncNode* ths, SyncAppendEntries* pMsg) { int32_t code; @@ -696,8 +347,9 @@ static int32_t syncNodeMakeLogSame(SSyncNode* ths, SyncAppendEntries* pMsg) { SRpcMsg rpcMsg; syncEntry2OriginalRpc(pRollBackEntry, &rpcMsg); - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pRollBackEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pRollBackEntry->isWeak; cbMeta.code = 0; cbMeta.state = ths->state; @@ -713,8 +365,10 @@ static int32_t syncNodeMakeLogSame(SSyncNode* ths, SyncAppendEntries* pMsg) { // delete confict entries code = ths->pLogStore->syncLogTruncate(ths->pLogStore, delBegin); ASSERT(code == 0); - sDebug("vgId:%d sync event currentTerm:%lu log truncate, from %ld to %ld", ths->vgId, ths->pRaftStore->currentTerm, - delBegin, delEnd); + + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "log truncate, from %ld to %ld", delBegin, delEnd); + syncNodeEventLog(ths, eventLog); logStoreSimpleLog2("after syncNodeMakeLogSame", ths->pLogStore); return code; @@ -725,8 +379,9 @@ static int32_t syncNodePreCommit(SSyncNode* ths, SSyncRaftEntry* pEntry) { syncEntry2OriginalRpc(pEntry, &rpcMsg); if (ths->pFsm != NULL) { if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; cbMeta.code = 2; cbMeta.state = ths->state; @@ -842,7 +497,7 @@ int32_t syncNodeOnAppendEntriesSnapshotCb(SSyncNode* ths, SyncAppendEntries* pMs do { SyncIndex myLastIndex = syncNodeGetLastIndex(ths); SSnapshot snapshot; - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); + ths->pFsm->FpGetSnapshotInfo(ths->pFsm, &snapshot); bool condition0 = (pMsg->term == ths->pRaftStore->currentTerm) && (ths->state == TAOS_SYNC_STATE_FOLLOWER) && syncNodeHasSnapshot(ths); @@ -880,6 +535,72 @@ int32_t syncNodeOnAppendEntriesSnapshotCb(SSyncNode* ths, SyncAppendEntries* pMs } } while (0); + // fake match2 + // + // condition1: + // preIndex <= my commit index + // + // operation: + // if hasAppendEntries && pMsg->prevLogIndex == ths->commitIndex, append entry + // match my-commit-index or my-commit-index + 1 + // no operation on log + do { + bool condition = (pMsg->term == ths->pRaftStore->currentTerm) && (ths->state == TAOS_SYNC_STATE_FOLLOWER) && + (pMsg->prevLogIndex <= ths->commitIndex); + if (condition) { + sTrace("recv SyncAppendEntries, fake match2, msg-prevLogIndex:%ld, my-commitIndex:%ld", pMsg->prevLogIndex, + ths->commitIndex); + + SyncIndex matchIndex = ths->commitIndex; + bool hasAppendEntries = pMsg->dataLen > 0; + if (hasAppendEntries && pMsg->prevLogIndex == ths->commitIndex) { + // append entry + SSyncRaftEntry* pAppendEntry = syncEntryDeserialize(pMsg->data, pMsg->dataLen); + ASSERT(pAppendEntry != NULL); + + { + // has extra entries (> preIndex) in local log + SyncIndex logLastIndex = ths->pLogStore->syncLogLastIndex(ths->pLogStore); + bool hasExtraEntries = logLastIndex > pMsg->prevLogIndex; + + if (hasExtraEntries) { + // make log same, rollback deleted entries + code = syncNodeMakeLogSame(ths, pMsg); + ASSERT(code == 0); + } + } + + code = ths->pLogStore->syncLogAppendEntry(ths->pLogStore, pAppendEntry); + ASSERT(code == 0); + + // pre commit + code = syncNodePreCommit(ths, pAppendEntry); + ASSERT(code == 0); + + matchIndex = pMsg->prevLogIndex + 1; + + syncEntryDestory(pAppendEntry); + } + + // prepare response msg + SyncAppendEntriesReply* pReply = syncAppendEntriesReplyBuild(ths->vgId); + pReply->srcId = ths->myRaftId; + pReply->destId = pMsg->srcId; + pReply->term = ths->pRaftStore->currentTerm; + pReply->privateTerm = ths->pNewNodeReceiver->privateTerm; + pReply->success = true; + pReply->matchIndex = matchIndex; + + // send response + SRpcMsg rpcMsg; + syncAppendEntriesReply2RpcMsg(pReply, &rpcMsg); + syncNodeSendMsgById(&pReply->destId, ths, &rpcMsg); + syncAppendEntriesReplyDestroy(pReply); + + return ret; + } + } while (0); + // calculate logOK here, before will coredump, due to fake match bool logOK = syncNodeOnAppendEntriesLogOK(ths, pMsg); @@ -989,14 +710,16 @@ int32_t syncNodeOnAppendEntriesSnapshotCb(SSyncNode* ths, SyncAppendEntries* pMs if (pMsg->commitIndex <= ths->pLogStore->syncLogLastIndex(ths->pLogStore)) { // advance commit index to sanpshot first SSnapshot snapshot; - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); + ths->pFsm->FpGetSnapshotInfo(ths->pFsm, &snapshot); if (snapshot.lastApplyIndex >= 0 && snapshot.lastApplyIndex > ths->commitIndex) { SyncIndex commitBegin = ths->commitIndex; SyncIndex commitEnd = snapshot.lastApplyIndex; ths->commitIndex = snapshot.lastApplyIndex; - sDebug("vgId:%d sync event currentTerm:%lu commit by snapshot from index:%ld to index:%ld, %s", ths->vgId, - ths->pRaftStore->currentTerm, commitBegin, commitEnd, syncUtilState2String(ths->state)); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "commit by snapshot from index:%ld to index:%ld", commitBegin, + commitEnd); + syncNodeEventLog(ths, eventLog); } SyncIndex beginIndex = ths->commitIndex + 1; diff --git a/source/libs/sync/src/syncAppendEntriesReply.c b/source/libs/sync/src/syncAppendEntriesReply.c index 5290d7d28eb57728531bcc1dd63ba6a808d82563..47fd23baaef98add632cfbdd65d7ea73ef6efdfb 100644 --- a/source/libs/sync/src/syncAppendEntriesReply.c +++ b/source/libs/sync/src/syncAppendEntriesReply.c @@ -123,7 +123,7 @@ int32_t syncNodeOnAppendEntriesReplySnapshotCb(SSyncNode* ths, SyncAppendEntries syncIndexMgrLog2("recv SyncAppendEntriesReply, before pMatchIndex:", ths->pMatchIndex); if (gRaftDetailLog) { SSnapshot snapshot; - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); + ths->pFsm->FpGetSnapshotInfo(ths->pFsm, &snapshot); sTrace("recv SyncAppendEntriesReply, before snapshot.lastApplyIndex:%ld, snapshot.lastApplyTerm:%lu", snapshot.lastApplyIndex, snapshot.lastApplyTerm); } @@ -175,7 +175,7 @@ int32_t syncNodeOnAppendEntriesReplySnapshotCb(SSyncNode* ths, SyncAppendEntries ASSERT(pSender != NULL); bool hasSnapshot = syncNodeHasSnapshot(ths); SSnapshot snapshot; - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); + ths->pFsm->FpGetSnapshotInfo(ths->pFsm, &snapshot); // start sending snapshot first time // start here, stop by receiver @@ -183,27 +183,9 @@ int32_t syncNodeOnAppendEntriesReplySnapshotCb(SSyncNode* ths, SyncAppendEntries pMsg->privateTerm < pSender->privateTerm) { snapshotSenderStart(pSender); - char host[128]; - uint16_t port; - syncUtilU642Addr(pSender->pSyncNode->replicasId[pSender->replicaIndex].addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char* s = snapshotSender2Str(pSender); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d start sender first time, lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu " - "sender:%s", - ths->vgId, ths->pRaftStore->currentTerm, host, port, pSender->snapshot.lastApplyIndex, - pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, pSender->privateTerm, s); - taosMemoryFree(s); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d start sender first time, lastApplyIndex:%ld " - "lastApplyTerm:%lu lastConfigIndex:%ld privateTerm:%lu", - ths->vgId, ths->pRaftStore->currentTerm, host, port, pSender->snapshot.lastApplyIndex, - pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, pSender->privateTerm); - } + char* eventLog = snapshotSender2SimpleStr(pSender, "snapshot sender start"); + syncNodeEventLog(ths, eventLog); + taosMemoryFree(eventLog); } SyncIndex sentryIndex = pSender->snapshot.lastApplyIndex + 1; @@ -227,7 +209,7 @@ int32_t syncNodeOnAppendEntriesReplySnapshotCb(SSyncNode* ths, SyncAppendEntries syncIndexMgrLog2("recv SyncAppendEntriesReply, after pMatchIndex:", ths->pMatchIndex); if (gRaftDetailLog) { SSnapshot snapshot; - ths->pFsm->FpGetSnapshot(ths->pFsm, &snapshot); + ths->pFsm->FpGetSnapshotInfo(ths->pFsm, &snapshot); sTrace("recv SyncAppendEntriesReply, after snapshot.lastApplyIndex:%ld, snapshot.lastApplyTerm:%lu", snapshot.lastApplyIndex, snapshot.lastApplyTerm); } diff --git a/source/libs/sync/src/syncCommit.c b/source/libs/sync/src/syncCommit.c index efdb5b24ce6497890508c62881deccb6b408410c..ec3e1ab2baf94deb6153f3f0b69b067e05f33a92 100644 --- a/source/libs/sync/src/syncCommit.c +++ b/source/libs/sync/src/syncCommit.c @@ -50,15 +50,16 @@ void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) { // advance commit index to sanpshot first SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); if (snapshot.lastApplyIndex > 0 && snapshot.lastApplyIndex > pSyncNode->commitIndex) { SyncIndex commitBegin = pSyncNode->commitIndex; SyncIndex commitEnd = snapshot.lastApplyIndex; pSyncNode->commitIndex = snapshot.lastApplyIndex; - sDebug("vgId:%d sync event currentTerm:%lu commit by snapshot from index:%ld to index:%ld, %s", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, pSyncNode->commitIndex, snapshot.lastApplyIndex, - syncUtilState2String(pSyncNode->state)); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "commit by snapshot from index:%ld to index:%ld", pSyncNode->commitIndex, + snapshot.lastApplyIndex); + syncNodeEventLog(pSyncNode, eventLog); } // update commit index diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index 557342be1cf46ec656c515436404348ce835f740..599410510de48ad323707c5b0b495fd7b4adbcd2 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -178,6 +178,23 @@ int32_t syncSetStandby(int64_t rid) { return 0; } +bool syncNodeCheckNewConfig(SSyncNode* pSyncNode, const SSyncCfg* pNewCfg) { + bool IamInNew = syncNodeInConfig(pSyncNode, pNewCfg); + if (!IamInNew) { + return false; + } + + if (pNewCfg->replicaNum > pSyncNode->replicaNum + 1) { + return false; + } + + if (pNewCfg->replicaNum < pSyncNode->replicaNum - 1) { + return false; + } + + return true; +} + int32_t syncReconfigBuild(int64_t rid, const SSyncCfg* pNewCfg, SRpcMsg* pRpcMsg) { SSyncNode* pSyncNode = (SSyncNode*)taosAcquireRef(tsNodeRefId, rid); if (pSyncNode == NULL) { @@ -185,31 +202,12 @@ int32_t syncReconfigBuild(int64_t rid, const SSyncCfg* pNewCfg, SRpcMsg* pRpcMsg return -1; } ASSERT(rid == pSyncNode->rid); - int32_t ret = 0; - bool IamInNew = syncNodeInConfig(pSyncNode, pNewCfg); - -#if 0 - for (int i = 0; i < pNewCfg->replicaNum; ++i) { - if (strcmp((pNewCfg->nodeInfo)[i].nodeFqdn, pSyncNode->myNodeInfo.nodeFqdn) == 0 && - (pNewCfg->nodeInfo)[i].nodePort == pSyncNode->myNodeInfo.nodePort) { - IamInNew = true; - } - - /* - SRaftId newId; - newId.addr = syncUtilAddr2U64((pNewCfg->nodeInfo)[i].nodeFqdn, (pNewCfg->nodeInfo)[i].nodePort); - newId.vgId = pSyncNode->vgId; - if (syncUtilSameId(&(pSyncNode->myRaftId), &newId)) { - IamInNew = true; - } - */ - } -#endif - if (!IamInNew) { + if (!syncNodeCheckNewConfig(pSyncNode, pNewCfg)) { taosReleaseRef(tsNodeRefId, pSyncNode->rid); - terrno = TSDB_CODE_SYN_NOT_IN_NEW_CONFIG; + terrno = TSDB_CODE_SYN_NEW_CONFIG_ERROR; + sError("syncNodeCheckNewConfig error"); return -1; } @@ -233,33 +231,10 @@ int32_t syncReconfig(int64_t rid, const SSyncCfg* pNewCfg) { } ASSERT(rid == pSyncNode->rid); - bool IamInNew = syncNodeInConfig(pSyncNode, pNewCfg); - -#if 0 - for (int i = 0; i < pNewCfg->replicaNum; ++i) { - if (strcmp((pNewCfg->nodeInfo)[i].nodeFqdn, pSyncNode->myNodeInfo.nodeFqdn) == 0 && - (pNewCfg->nodeInfo)[i].nodePort == pSyncNode->myNodeInfo.nodePort) { - IamInNew = true; - } - - /* - // some problem in inet_addr - - SRaftId newId = EMPTY_RAFT_ID; - newId.addr = syncUtilAddr2U64((pNewCfg->nodeInfo)[i].nodeFqdn, (pNewCfg->nodeInfo)[i].nodePort); - newId.vgId = pSyncNode->vgId; - - if (syncUtilSameId(&(pSyncNode->myRaftId), &newId)) { - IamInNew = true; - } - */ - } -#endif - - if (!IamInNew) { - sError("sync reconfig error, not in new config"); + if (!syncNodeCheckNewConfig(pSyncNode, pNewCfg)) { taosReleaseRef(tsNodeRefId, pSyncNode->rid); - terrno = TSDB_CODE_SYN_NOT_IN_NEW_CONFIG; + terrno = TSDB_CODE_SYN_NEW_CONFIG_ERROR; + sError("syncNodeCheckNewConfig error"); return -1; } @@ -414,39 +389,78 @@ int32_t syncGetSnapshotMeta(int64_t rid, struct SSnapshotMeta* sMeta) { assert(rid == pSyncNode->rid); sMeta->lastConfigIndex = pSyncNode->pRaftCfg->lastConfigIndex; - sTrace("sync get snapshot meta: lastConfigIndex:%ld", pSyncNode->pRaftCfg->lastConfigIndex); + sTrace("vgId:%d, get snapshot meta, lastConfigIndex:%" PRId64, pSyncNode->vgId, pSyncNode->pRaftCfg->lastConfigIndex); taosReleaseRef(tsNodeRefId, pSyncNode->rid); return 0; } +int32_t syncGetSnapshotMetaByIndex(int64_t rid, SyncIndex snapshotIndex, struct SSnapshotMeta* sMeta) { + SSyncNode* pSyncNode = (SSyncNode*)taosAcquireRef(tsNodeRefId, rid); + if (pSyncNode == NULL) { + return -1; + } + assert(rid == pSyncNode->rid); + + ASSERT(pSyncNode->pRaftCfg->configIndexCount >= 1); + SyncIndex lastIndex = (pSyncNode->pRaftCfg->configIndexArr)[0]; + + for (int i = 0; i < pSyncNode->pRaftCfg->configIndexCount; ++i) { + if ((pSyncNode->pRaftCfg->configIndexArr)[i] > lastIndex && + (pSyncNode->pRaftCfg->configIndexArr)[i] <= snapshotIndex) { + lastIndex = (pSyncNode->pRaftCfg->configIndexArr)[i]; + } + } + sMeta->lastConfigIndex = lastIndex; + sTrace("vgId:%d, get snapshot meta by index:%" PRId64 " lastConfigIndex:%" PRId64, pSyncNode->vgId, snapshotIndex, + sMeta->lastConfigIndex); + + taosReleaseRef(tsNodeRefId, pSyncNode->rid); + return 0; +} + +SyncIndex syncNodeGetSnapshotConfigIndex(SSyncNode* pSyncNode, SyncIndex snapshotLastApplyIndex) { + ASSERT(pSyncNode->pRaftCfg->configIndexCount >= 1); + SyncIndex lastIndex = (pSyncNode->pRaftCfg->configIndexArr)[0]; + + for (int i = 0; i < pSyncNode->pRaftCfg->configIndexCount; ++i) { + if ((pSyncNode->pRaftCfg->configIndexArr)[i] > lastIndex && + (pSyncNode->pRaftCfg->configIndexArr)[i] <= snapshotLastApplyIndex) { + lastIndex = (pSyncNode->pRaftCfg->configIndexArr)[i]; + } + } + + sTrace("sync syncNodeGetSnapshotConfigIndex index:%ld lastConfigIndex:%ld", snapshotLastApplyIndex, lastIndex); + return lastIndex; +} + const char* syncGetMyRoleStr(int64_t rid) { const char* s = syncUtilState2String(syncGetMyRole(rid)); return s; } -int32_t syncGetVgId(int64_t rid) { +SyncTerm syncGetMyTerm(int64_t rid) { SSyncNode* pSyncNode = (SSyncNode*)taosAcquireRef(tsNodeRefId, rid); if (pSyncNode == NULL) { return TAOS_SYNC_STATE_ERROR; } assert(rid == pSyncNode->rid); - int32_t vgId = pSyncNode->vgId; + SyncTerm term = pSyncNode->pRaftStore->currentTerm; taosReleaseRef(tsNodeRefId, pSyncNode->rid); - return vgId; + return term; } -SyncTerm syncGetMyTerm(int64_t rid) { +SyncGroupId syncGetVgId(int64_t rid) { SSyncNode* pSyncNode = (SSyncNode*)taosAcquireRef(tsNodeRefId, rid); if (pSyncNode == NULL) { return TAOS_SYNC_STATE_ERROR; } assert(rid == pSyncNode->rid); - SyncTerm term = pSyncNode->pRaftStore->currentTerm; + SyncGroupId vgId = pSyncNode->vgId; taosReleaseRef(tsNodeRefId, pSyncNode->rid); - return term; + return vgId; } void syncGetEpSet(int64_t rid, SEpSet* pEpSet) { @@ -575,8 +589,6 @@ int32_t syncPropose(int64_t rid, const SRpcMsg* pMsg, bool isWeak) { return -1; } assert(rid == pSyncNode->rid); - sDebug("vgId:%d sync event currentTerm:%lu propose msgType:%s,%d", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, TMSG_INFO(pMsg->msgType), pMsg->msgType); ret = syncNodePropose(pSyncNode, pMsg, isWeak); taosReleaseRef(tsNodeRefId, pSyncNode->rid); @@ -585,10 +597,32 @@ int32_t syncPropose(int64_t rid, const SRpcMsg* pMsg, bool isWeak) { int32_t syncNodePropose(SSyncNode* pSyncNode, const SRpcMsg* pMsg, bool isWeak) { int32_t ret = 0; - sDebug("vgId:%d sync event currentTerm:%lu propose msgType:%s,%d", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, TMSG_INFO(pMsg->msgType), pMsg->msgType); + + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "propose type:%s,%d", TMSG_INFO(pMsg->msgType), pMsg->msgType); + syncNodeEventLog(pSyncNode, eventLog); if (pSyncNode->state == TAOS_SYNC_STATE_LEADER) { + if (pSyncNode->changing && pMsg->msgType != TDMT_SYNC_CONFIG_CHANGE_FINISH) { + ret = -1; + terrno = TSDB_CODE_SYN_PROPOSE_NOT_READY; + sError("sync propose not ready, type:%s,%d", TMSG_INFO(pMsg->msgType), pMsg->msgType); + goto _END; + } + + // config change + if (pMsg->msgType == TDMT_SYNC_CONFIG_CHANGE) { + if (!syncNodeCanChange(pSyncNode)) { + ret = -1; + terrno = TSDB_CODE_SYN_RECONFIG_NOT_READY; + sError("sync reconfig not ready, type:%s,%d", TMSG_INFO(pMsg->msgType), pMsg->msgType); + goto _END; + } + + ASSERT(!pSyncNode->changing); + pSyncNode->changing = true; + } + SRespStub stub; stub.createTime = taosGetTimestampMs(); stub.rpcMsg = *pMsg; @@ -606,12 +640,16 @@ int32_t syncNodePropose(SSyncNode* pSyncNode, const SRpcMsg* pMsg, bool isWeak) sError("syncPropose pSyncNode->FpEqMsg is NULL"); } syncClientRequestDestroy(pSyncMsg); + goto _END; + } else { ret = -1; terrno = TSDB_CODE_SYN_NOT_LEADER; sError("syncPropose not leader, %s", syncUtilState2String(pSyncNode->state)); + goto _END; } +_END: return ret; } @@ -809,9 +847,9 @@ SSyncNode* syncNodeOpen(const SSyncInfo* pOldSyncInfo) { pSyncNode->restoreFinish = false; // pSyncNode->pSnapshot = NULL; - // if (pSyncNode->pFsm->FpGetSnapshot != NULL) { + // if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { // pSyncNode->pSnapshot = taosMemoryMalloc(sizeof(SSnapshot)); - // pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, pSyncNode->pSnapshot); + // pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, pSyncNode->pSnapshot); // } // tsem_init(&(pSyncNode->restoreSem), 0, 0); @@ -825,14 +863,14 @@ SSyncNode* syncNodeOpen(const SSyncInfo* pOldSyncInfo) { // snapshot receivers pSyncNode->pNewNodeReceiver = snapshotReceiverCreate(pSyncNode, EMPTY_RAFT_ID); + // is config changing + pSyncNode->changing = false; + // start in syncNodeStart // start raft // syncNodeBecomeFollower(pSyncNode); - // snapshot meta - // pSyncNode->sMeta.lastConfigIndex = -1; - - sDebug("vgId:%d sync event currentTerm:%lu sync open", pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm); + syncNodeEventLog(pSyncNode, "sync open"); return pSyncNode; } @@ -844,15 +882,6 @@ void syncNodeStart(SSyncNode* pSyncNode) { syncNodeBecomeLeader(pSyncNode, "one replica start"); // Raft 3.6.2 Committing entries from previous terms - - // use this now - syncNodeAppendNoop(pSyncNode); - syncMaybeAdvanceCommitIndex(pSyncNode); // maybe only one replica - - if (gRaftDetailLog) { - syncNodeLog2("==state change become leader immediately==", pSyncNode); - } - return; } @@ -879,7 +908,7 @@ void syncNodeStartStandBy(SSyncNode* pSyncNode) { } void syncNodeClose(SSyncNode* pSyncNode) { - sDebug("vgId:%d sync event currentTerm:%lu sync close", pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm); + syncNodeEventLog(pSyncNode, "sync close"); int32_t ret; assert(pSyncNode != NULL); @@ -1263,11 +1292,44 @@ char* syncNode2Str(const SSyncNode* pSyncNode) { return serialized; } +void syncNodeEventLog(const SSyncNode* pSyncNode, char* str) { + int32_t userStrLen = strlen(str); + + SSnapshot snapshot = {.data = NULL, .lastApplyIndex = -1, .lastApplyTerm = 0}; + if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); + } + SyncIndex logLastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); + + if (userStrLen < 256) { + char logBuf[128 + 256]; + snprintf(logBuf, sizeof(logBuf), + "vgId:%d, sync %s %s, term:%lu, commit:%ld, lastlog:%ld, lastsnapshot:%ld, standby:%d, replica-num:%d, " + "lconfig:%ld, changing:%d", + pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm, + pSyncNode->commitIndex, logLastIndex, snapshot.lastApplyIndex, pSyncNode->pRaftCfg->isStandBy, + pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing); + sDebug("%s", logBuf); + + } else { + int len = 128 + userStrLen; + char* s = (char*)taosMemoryMalloc(len); + snprintf(s, len, + "vgId:%d, sync %s %s, term:%lu, commit:%ld, lastlog:%ld, lastsnapshot:%ld, standby:%d, replica-num:%d, " + "lconfig:%ld, changing:%d", + pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm, + pSyncNode->commitIndex, logLastIndex, snapshot.lastApplyIndex, pSyncNode->pRaftCfg->isStandBy, + pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing); + sDebug("%s", s); + taosMemoryFree(s); + } +} + char* syncNode2SimpleStr(const SSyncNode* pSyncNode) { int len = 256; char* s = (char*)taosMemoryMalloc(len); snprintf(s, len, - "syncNode: vgId:%d currentTerm:%lu, commitIndex:%ld, state:%d %s, isStandBy:%d, " + "syncNode: vgId:%d, currentTerm:%lu, commitIndex:%ld, state:%d %s, isStandBy:%d, " "electTimerLogicClock:%lu, " "electTimerLogicClockUser:%lu, " "electTimerMS:%d, replicaNum:%d", @@ -1304,150 +1366,188 @@ bool syncNodeInConfig(SSyncNode* pSyncNode, const SSyncCfg* config) { return b1; } -void syncNodeUpdateConfig(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncIndex lastConfigChangeIndex, bool* isDrop) { +void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncIndex lastConfigChangeIndex) { SSyncCfg oldConfig = pSyncNode->pRaftCfg->cfg; pSyncNode->pRaftCfg->cfg = *pNewConfig; pSyncNode->pRaftCfg->lastConfigIndex = lastConfigChangeIndex; - int32_t ret = 0; - - // save snapshot senders - int32_t oldReplicaNum = pSyncNode->replicaNum; - SRaftId oldReplicasId[TSDB_MAX_REPLICA]; - memcpy(oldReplicasId, pSyncNode->replicasId, sizeof(oldReplicasId)); - SSyncSnapshotSender* oldSenders[TSDB_MAX_REPLICA]; - for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { - oldSenders[i] = (pSyncNode->senders)[i]; - sDebug("vgId:%d sync event currentTerm:%lu save senders %d, %p, privateTerm:%lu", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, i, oldSenders[i], oldSenders[i]->privateTerm); - if (gRaftDetailLog) { - ; - } - } + bool IamInOld = syncNodeInConfig(pSyncNode, &oldConfig); + bool IamInNew = syncNodeInConfig(pSyncNode, pNewConfig); - // init internal - pSyncNode->myNodeInfo = pSyncNode->pRaftCfg->cfg.nodeInfo[pSyncNode->pRaftCfg->cfg.myIndex]; - syncUtilnodeInfo2raftId(&pSyncNode->myNodeInfo, pSyncNode->vgId, &pSyncNode->myRaftId); + bool isDrop = false; + bool isAdd = false; - // init peersNum, peers, peersId - pSyncNode->peersNum = pSyncNode->pRaftCfg->cfg.replicaNum - 1; - int j = 0; - for (int i = 0; i < pSyncNode->pRaftCfg->cfg.replicaNum; ++i) { - if (i != pSyncNode->pRaftCfg->cfg.myIndex) { - pSyncNode->peersNodeInfo[j] = pSyncNode->pRaftCfg->cfg.nodeInfo[i]; - j++; - } + if (IamInOld && !IamInNew) { + isDrop = true; + } else { + isDrop = false; } - for (int i = 0; i < pSyncNode->peersNum; ++i) { - syncUtilnodeInfo2raftId(&pSyncNode->peersNodeInfo[i], pSyncNode->vgId, &pSyncNode->peersId[i]); + + if (!IamInOld && IamInNew) { + isAdd = true; + } else { + isAdd = false; } - // init replicaNum, replicasId - pSyncNode->replicaNum = pSyncNode->pRaftCfg->cfg.replicaNum; - for (int i = 0; i < pSyncNode->pRaftCfg->cfg.replicaNum; ++i) { - syncUtilnodeInfo2raftId(&pSyncNode->pRaftCfg->cfg.nodeInfo[i], pSyncNode->vgId, &pSyncNode->replicasId[i]); + if (IamInNew) { + pSyncNode->pRaftCfg->isStandBy = 0; // change isStandBy to normal + } + if (isDrop) { + pSyncNode->pRaftCfg->isStandBy = 1; // set standby } - syncIndexMgrUpdate(pSyncNode->pNextIndex, pSyncNode); - syncIndexMgrUpdate(pSyncNode->pMatchIndex, pSyncNode); - voteGrantedUpdate(pSyncNode->pVotesGranted, pSyncNode); - votesRespondUpdate(pSyncNode->pVotesRespond, pSyncNode); + // persist last config index + raftCfgAddConfigIndex(pSyncNode->pRaftCfg, lastConfigChangeIndex); - pSyncNode->quorum = syncUtilQuorum(pSyncNode->pRaftCfg->cfg.replicaNum); + if (IamInNew) { + //----------------------------------------- + int32_t ret = 0; + + // save snapshot senders + int32_t oldReplicaNum = pSyncNode->replicaNum; + SRaftId oldReplicasId[TSDB_MAX_REPLICA]; + memcpy(oldReplicasId, pSyncNode->replicasId, sizeof(oldReplicasId)); + SSyncSnapshotSender* oldSenders[TSDB_MAX_REPLICA]; + for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { + oldSenders[i] = (pSyncNode->senders)[i]; - // reset snapshot senders + char* eventLog = snapshotSender2SimpleStr(oldSenders[i], "snapshot sender save old"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } - // clear new - for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { - (pSyncNode->senders)[i] = NULL; - } - - // reset new - for (int i = 0; i < pSyncNode->replicaNum; ++i) { - // reset sender - bool reset = false; - for (int j = 0; j < TSDB_MAX_REPLICA; ++j) { - if (syncUtilSameId(&(pSyncNode->replicasId)[i], &oldReplicasId[j])) { - char host[128]; - uint16_t port; - syncUtilU642Addr((pSyncNode->replicasId)[i].addr, host, sizeof(host), &port); - sDebug("vgId:%d sync event currentTerm:%lu reset sender for %lu, newIndex:%d, %s:%d, %p, privateTerm:%lu", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, (pSyncNode->replicasId)[i].addr, i, host, port, - oldSenders[j], oldSenders[j]->privateTerm); - (pSyncNode->senders)[i] = oldSenders[j]; - oldSenders[j] = NULL; - reset = true; - - // reset replicaIndex - int32_t oldreplicaIndex = (pSyncNode->senders)[i]->replicaIndex; - (pSyncNode->senders)[i]->replicaIndex = i; - sDebug("vgId:%d sync event currentTerm:%lu udpate replicaIndex from %d to %d, %s:%d, %p, reset:%d", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, oldreplicaIndex, i, host, port, - (pSyncNode->senders)[i], reset); + // init internal + pSyncNode->myNodeInfo = pSyncNode->pRaftCfg->cfg.nodeInfo[pSyncNode->pRaftCfg->cfg.myIndex]; + syncUtilnodeInfo2raftId(&pSyncNode->myNodeInfo, pSyncNode->vgId, &pSyncNode->myRaftId); + + // init peersNum, peers, peersId + pSyncNode->peersNum = pSyncNode->pRaftCfg->cfg.replicaNum - 1; + int j = 0; + for (int i = 0; i < pSyncNode->pRaftCfg->cfg.replicaNum; ++i) { + if (i != pSyncNode->pRaftCfg->cfg.myIndex) { + pSyncNode->peersNodeInfo[j] = pSyncNode->pRaftCfg->cfg.nodeInfo[i]; + j++; } } - } + for (int i = 0; i < pSyncNode->peersNum; ++i) { + syncUtilnodeInfo2raftId(&pSyncNode->peersNodeInfo[i], pSyncNode->vgId, &pSyncNode->peersId[i]); + } - // create new - for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { - if ((pSyncNode->senders)[i] == NULL) { - (pSyncNode->senders)[i] = snapshotSenderCreate(pSyncNode, i); - sDebug("vgId:%d sync event currentTerm:%lu create new sender %p replicaIndex:%d, privateTerm:%lu", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, (pSyncNode->senders)[i], i, - (pSyncNode->senders)[i]->privateTerm); + // init replicaNum, replicasId + pSyncNode->replicaNum = pSyncNode->pRaftCfg->cfg.replicaNum; + for (int i = 0; i < pSyncNode->pRaftCfg->cfg.replicaNum; ++i) { + syncUtilnodeInfo2raftId(&pSyncNode->pRaftCfg->cfg.nodeInfo[i], pSyncNode->vgId, &pSyncNode->replicasId[i]); } - } - // free old - for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { - if (oldSenders[i] != NULL) { - snapshotSenderDestroy(oldSenders[i]); - sDebug("vgId:%d sync event currentTerm:%lu delete old sender %p replicaIndex:%d", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, oldSenders[i], i); - oldSenders[i] = NULL; + syncIndexMgrUpdate(pSyncNode->pNextIndex, pSyncNode); + syncIndexMgrUpdate(pSyncNode->pMatchIndex, pSyncNode); + voteGrantedUpdate(pSyncNode->pVotesGranted, pSyncNode); + votesRespondUpdate(pSyncNode->pVotesRespond, pSyncNode); + + pSyncNode->quorum = syncUtilQuorum(pSyncNode->pRaftCfg->cfg.replicaNum); + + // reset snapshot senders + + // clear new + for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { + (pSyncNode->senders)[i] = NULL; } - } - bool IamInOld = syncNodeInConfig(pSyncNode, &oldConfig); - bool IamInNew = syncNodeInConfig(pSyncNode, pNewConfig); + // reset new + for (int i = 0; i < pSyncNode->replicaNum; ++i) { + // reset sender + bool reset = false; + for (int j = 0; j < TSDB_MAX_REPLICA; ++j) { + if (syncUtilSameId(&(pSyncNode->replicasId)[i], &oldReplicasId[j])) { + char host[128]; + uint16_t port; + syncUtilU642Addr((pSyncNode->replicasId)[i].addr, host, sizeof(host), &port); + + do { + char eventLog[256]; + snprintf(eventLog, sizeof(eventLog), "snapshot sender reset for %lu, newIndex:%d, %s:%d, %p", + (pSyncNode->replicasId)[i].addr, i, host, port, oldSenders[j]); + syncNodeEventLog(pSyncNode, eventLog); + } while (0); + + (pSyncNode->senders)[i] = oldSenders[j]; + oldSenders[j] = NULL; + reset = true; + + // reset replicaIndex + int32_t oldreplicaIndex = (pSyncNode->senders)[i]->replicaIndex; + (pSyncNode->senders)[i]->replicaIndex = i; + + do { + char eventLog[256]; + snprintf(eventLog, sizeof(eventLog), + "snapshot sender udpate replicaIndex from %d to %d, %s:%d, %p, reset:%d", oldreplicaIndex, i, host, + port, (pSyncNode->senders)[i], reset); + syncNodeEventLog(pSyncNode, eventLog); + } while (0); + } + } + } -#if 0 - for (int i = 0; i < oldConfig.replicaNum; ++i) { - if (strcmp((oldConfig.nodeInfo)[i].nodeFqdn, pSyncNode->myNodeInfo.nodeFqdn) == 0 && - (oldConfig.nodeInfo)[i].nodePort == pSyncNode->myNodeInfo.nodePort) { - IamInOld = true; - break; + // create new + for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { + if ((pSyncNode->senders)[i] == NULL) { + (pSyncNode->senders)[i] = snapshotSenderCreate(pSyncNode, i); + + char* eventLog = snapshotSender2SimpleStr((pSyncNode->senders)[i], "snapshot sender create new"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } } - } - for (int i = 0; i < newConfig->replicaNum; ++i) { - if (strcmp((newConfig->nodeInfo)[i].nodeFqdn, pSyncNode->myNodeInfo.nodeFqdn) == 0 && - (newConfig->nodeInfo)[i].nodePort == pSyncNode->myNodeInfo.nodePort) { - IamInNew = true; - break; + // free old + for (int i = 0; i < TSDB_MAX_REPLICA; ++i) { + if (oldSenders[i] != NULL) { + snapshotSenderDestroy(oldSenders[i]); + + do { + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "snapshot sender delete old %p replica-index:%d", oldSenders[i], i); + syncNodeEventLog(pSyncNode, eventLog); + } while (0); + + oldSenders[i] = NULL; + } } - } -#endif - *isDrop = true; - if (IamInOld && !IamInNew) { - *isDrop = true; - } else { - *isDrop = false; - } + // persist cfg + raftCfgPersist(pSyncNode->pRaftCfg); - // may be add me to a new raft group - if (IamInOld && IamInNew && oldConfig.replicaNum == 1) { - } + char tmpbuf[512]; + char* oldStr = syncCfg2SimpleStr(&oldConfig); + char* newStr = syncCfg2SimpleStr(pNewConfig); + snprintf(tmpbuf, sizeof(tmpbuf), "config change from %d to %d, index:%ld, %s --> %s", oldConfig.replicaNum, + pNewConfig->replicaNum, lastConfigChangeIndex, oldStr, newStr); + taosMemoryFree(oldStr); + taosMemoryFree(newStr); - if (IamInNew) { - pSyncNode->pRaftCfg->isStandBy = 0; // change isStandBy to normal - } - raftCfgPersist(pSyncNode->pRaftCfg); + // change isStandBy to normal (election timeout) + if (pSyncNode->state == TAOS_SYNC_STATE_LEADER) { + syncNodeBecomeLeader(pSyncNode, tmpbuf); + } else { + syncNodeBecomeFollower(pSyncNode, tmpbuf); + } + } else { + // persist cfg + raftCfgPersist(pSyncNode->pRaftCfg); - if (gRaftDetailLog) { - syncNodeLog2("==syncNodeUpdateConfig==", pSyncNode); + char tmpbuf[512]; + char* oldStr = syncCfg2SimpleStr(&oldConfig); + char* newStr = syncCfg2SimpleStr(pNewConfig); + snprintf(tmpbuf, sizeof(tmpbuf), "do not config change from %d to %d, index:%ld, %s --> %s", oldConfig.replicaNum, + pNewConfig->replicaNum, lastConfigChangeIndex, oldStr, newStr); + taosMemoryFree(oldStr); + taosMemoryFree(newStr); + syncNodeEventLog(pSyncNode, tmpbuf); } + +_END: + return; } SSyncNode* syncNodeAcquire(int64_t rid) { @@ -1473,12 +1573,6 @@ void syncNodeUpdateTerm(SSyncNode* pSyncNode, SyncTerm term) { } void syncNodeBecomeFollower(SSyncNode* pSyncNode, const char* debugStr) { - sDebug( - "vgId:%d sync event currentTerm:%lu become follower, isStandBy:%d, replicaNum:%d, " - "restoreFinish:%d, %s", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, pSyncNode->pRaftCfg->isStandBy, pSyncNode->replicaNum, - pSyncNode->restoreFinish, debugStr); - // maybe clear leader cache if (pSyncNode->state == TAOS_SYNC_STATE_LEADER) { pSyncNode->leaderCache = EMPTY_RAFT_ID; @@ -1490,6 +1584,21 @@ void syncNodeBecomeFollower(SSyncNode* pSyncNode, const char* debugStr) { // reset elect timer syncNodeResetElectTimer(pSyncNode); + + // trace log + do { + int32_t debugStrLen = strlen(debugStr); + if (debugStrLen < 256) { + char eventLog[256 + 64]; + snprintf(eventLog, sizeof(eventLog), "become follower %s", debugStr); + syncNodeEventLog(pSyncNode, eventLog); + } else { + char* eventLog = taosMemoryMalloc(debugStrLen + 64); + snprintf(eventLog, debugStrLen, "become follower %s", debugStr); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } + } while (0); } // TLA+ Spec @@ -1514,10 +1623,6 @@ void syncNodeBecomeLeader(SSyncNode* pSyncNode, const char* debugStr) { // reset restoreFinish pSyncNode->restoreFinish = false; - sDebug("vgId:%d sync event currentTerm:%lu become leader, isStandBy:%d, replicaNum:%d, restoreFinish:%d, %s", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, pSyncNode->pRaftCfg->isStandBy, pSyncNode->replicaNum, - pSyncNode->restoreFinish, debugStr); - // state change pSyncNode->state = TAOS_SYNC_STATE_LEADER; @@ -1563,6 +1668,25 @@ void syncNodeBecomeLeader(SSyncNode* pSyncNode, const char* debugStr) { // start heartbeat timer syncNodeStartHeartbeatTimer(pSyncNode); + + // append noop + syncNodeAppendNoop(pSyncNode); + syncMaybeAdvanceCommitIndex(pSyncNode); // maybe only one replica + + // trace log + do { + int32_t debugStrLen = strlen(debugStr); + if (debugStrLen < 256) { + char eventLog[256 + 64]; + snprintf(eventLog, sizeof(eventLog), "become leader %s", debugStr); + syncNodeEventLog(pSyncNode, eventLog); + } else { + char* eventLog = taosMemoryMalloc(debugStrLen + 64); + snprintf(eventLog, debugStrLen, "become leader %s", debugStr); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } + } while (0); } void syncNodeCandidate2Leader(SSyncNode* pSyncNode) { @@ -1574,10 +1698,6 @@ void syncNodeCandidate2Leader(SSyncNode* pSyncNode) { // Raft 3.6.2 Committing entries from previous terms - // use this now - syncNodeAppendNoop(pSyncNode); - syncMaybeAdvanceCommitIndex(pSyncNode); // maybe only one replica - // do not use this // syncNodeEqNoop(pSyncNode); } @@ -1633,8 +1753,8 @@ void syncNodeVoteForSelf(SSyncNode* pSyncNode) { bool syncNodeHasSnapshot(SSyncNode* pSyncNode) { bool ret = false; SSnapshot snapshot = {.data = NULL, .lastApplyIndex = -1, .lastApplyTerm = 0}; - if (pSyncNode->pFsm->FpGetSnapshot != NULL) { - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); if (snapshot.lastApplyIndex >= SYNC_INDEX_BEGIN) { ret = true; } @@ -1644,19 +1764,19 @@ bool syncNodeHasSnapshot(SSyncNode* pSyncNode) { bool syncNodeIsIndexInSnapshot(SSyncNode* pSyncNode, SyncIndex index) { ASSERT(syncNodeHasSnapshot(pSyncNode)); - ASSERT(pSyncNode->pFsm->FpGetSnapshot != NULL); + ASSERT(pSyncNode->pFsm->FpGetSnapshotInfo != NULL); ASSERT(index >= SYNC_INDEX_BEGIN); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); bool b = (index <= snapshot.lastApplyIndex); return b; } SyncIndex syncNodeGetLastIndex(SSyncNode* pSyncNode) { SSnapshot snapshot = {.data = NULL, .lastApplyIndex = -1, .lastApplyTerm = 0}; - if (pSyncNode->pFsm->FpGetSnapshot != NULL) { - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); } SyncIndex logLastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); @@ -1669,8 +1789,8 @@ SyncTerm syncNodeGetLastTerm(SSyncNode* pSyncNode) { if (syncNodeHasSnapshot(pSyncNode)) { // has snapshot SSnapshot snapshot = {.data = NULL, .lastApplyIndex = -1, .lastApplyTerm = 0}; - if (pSyncNode->pFsm->FpGetSnapshot != NULL) { - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); } SyncIndex logLastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); @@ -1722,8 +1842,8 @@ SyncTerm syncNodeGetPreTerm(SSyncNode* pSyncNode, SyncIndex index) { if (syncNodeHasSnapshot(pSyncNode)) { // has snapshot SSnapshot snapshot = {.data = NULL, .lastApplyIndex = -1, .lastApplyTerm = 0}; - if (pSyncNode->pFsm->FpGetSnapshot != NULL) { - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + if (pSyncNode->pFsm->FpGetSnapshotInfo != NULL) { + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); } if (index > snapshot.lastApplyIndex + 1) { @@ -1818,7 +1938,7 @@ static void syncNodeEqPingTimer(void* param, void* tmrId) { if (pSyncNode->FpEqMsg != NULL) { int32_t code = pSyncNode->FpEqMsg(pSyncNode->msgcb, &rpcMsg); if (code != 0) { - sError("vgId:%d sync enqueue ping msg error, code:%d", pSyncNode->vgId, code); + sError("vgId:%d, sync enqueue ping msg error, code:%d", pSyncNode->vgId, code); rpcFreeCont(rpcMsg.pCont); syncTimeoutDestroy(pSyncMsg); return; @@ -1852,7 +1972,7 @@ static void syncNodeEqElectTimer(void* param, void* tmrId) { if (pSyncNode->FpEqMsg != NULL) { int32_t code = pSyncNode->FpEqMsg(pSyncNode->msgcb, &rpcMsg); if (code != 0) { - sError("vgId:%d sync enqueue elect msg error, code:%d", pSyncNode->vgId, code); + sError("vgId:%d, sync enqueue elect msg error, code:%d", pSyncNode->vgId, code); rpcFreeCont(rpcMsg.pCont); syncTimeoutDestroy(pSyncMsg); return; @@ -1890,7 +2010,7 @@ static void syncNodeEqHeartbeatTimer(void* param, void* tmrId) { if (pSyncNode->FpEqMsg != NULL) { int32_t code = pSyncNode->FpEqMsg(pSyncNode->msgcb, &rpcMsg); if (code != 0) { - sError("vgId:%d sync enqueue timer msg error, code:%d", pSyncNode->vgId, code); + sError("vgId:%d, sync enqueue timer msg error, code:%d", pSyncNode->vgId, code); rpcFreeCont(rpcMsg.pCont); syncTimeoutDestroy(pSyncMsg); return; @@ -2028,8 +2148,9 @@ int32_t syncNodeOnClientRequestCb(SSyncNode* ths, SyncClientRequest* pMsg) { if (ths->pFsm != NULL) { // if (ths->pFsm->FpPreCommitCb != NULL && pEntry->originalRpcType != TDMT_SYNC_NOOP) { if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; cbMeta.code = 0; cbMeta.state = ths->state; @@ -2050,8 +2171,9 @@ int32_t syncNodeOnClientRequestCb(SSyncNode* ths, SyncClientRequest* pMsg) { if (ths->pFsm != NULL) { // if (ths->pFsm->FpPreCommitCb != NULL && pEntry->originalRpcType != TDMT_SYNC_NOOP) { if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; cbMeta.code = 1; cbMeta.state = ths->state; @@ -2090,34 +2212,33 @@ const char* syncStr(ESyncState state) { static int32_t syncDoLeaderTransfer(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* pEntry) { SyncLeaderTransfer* pSyncLeaderTransfer = syncLeaderTransferFromRpcMsg2(pRpcMsg); - sDebug("vgId:%d sync event currentTerm:%lu begin leader transfer", ths->vgId, ths->pRaftStore->currentTerm); + syncNodeEventLog(ths, "begin leader transfer"); - if (strcmp(pSyncLeaderTransfer->newNodeInfo.nodeFqdn, ths->myNodeInfo.nodeFqdn) == 0 && - pSyncLeaderTransfer->newNodeInfo.nodePort == ths->myNodeInfo.nodePort) { - sDebug("vgId:%d sync event currentTerm:%lu maybe leader transfer to %s:%d %lu", ths->vgId, - ths->pRaftStore->currentTerm, pSyncLeaderTransfer->newNodeInfo.nodeFqdn, - pSyncLeaderTransfer->newNodeInfo.nodePort, pSyncLeaderTransfer->newLeaderId.addr); + bool sameId = syncUtilSameId(&(pSyncLeaderTransfer->newLeaderId), &(ths->myRaftId)); + bool sameNodeInfo = strcmp(pSyncLeaderTransfer->newNodeInfo.nodeFqdn, ths->myNodeInfo.nodeFqdn) == 0 && + pSyncLeaderTransfer->newNodeInfo.nodePort == ths->myNodeInfo.nodePort; + bool same = sameId || sameNodeInfo; + if (same) { // reset elect timer now! int32_t electMS = 1; int32_t ret = syncNodeRestartElectTimer(ths, electMS); ASSERT(ret == 0); + + char eventLog[256]; + snprintf(eventLog, sizeof(eventLog), "maybe leader transfer to %s:%d %lu", + pSyncLeaderTransfer->newNodeInfo.nodeFqdn, pSyncLeaderTransfer->newNodeInfo.nodePort, + pSyncLeaderTransfer->newLeaderId.addr); + syncNodeEventLog(ths, eventLog); } - /* - if (syncUtilSameId(&(pSyncLeaderTransfer->newLeaderId), &(ths->myRaftId))) { - // reset elect timer now! - int32_t electMS = 1; - int32_t ret = syncNodeRestartElectTimer(ths, electMS); - ASSERT(ret == 0); - } - */ if (ths->pFsm->FpLeaderTransferCb != NULL) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.code = 0; cbMeta.currentTerm = ths->pRaftStore->currentTerm; cbMeta.flag = 0; cbMeta.index = pEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; cbMeta.seqNum = pEntry->seqNum; cbMeta.state = ths->state; @@ -2144,9 +2265,54 @@ int32_t syncNodeUpdateNewConfigIndex(SSyncNode* ths, SSyncCfg* pNewCfg) { return -1; } -static int32_t syncNodeConfigChange(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* pEntry) { +static int32_t syncNodeConfigChangeFinish(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* pEntry) { + SyncReconfigFinish* pFinish = syncReconfigFinishFromRpcMsg2(pRpcMsg); + ASSERT(pFinish); + + if (ths->pFsm->FpReConfigCb != NULL) { + SReConfigCbMeta cbMeta = {0}; + cbMeta.code = 0; + cbMeta.index = pEntry->index; + cbMeta.term = pEntry->term; + cbMeta.seqNum = pEntry->seqNum; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, pEntry->index); + cbMeta.state = ths->state; + cbMeta.currentTerm = ths->pRaftStore->currentTerm; + cbMeta.isWeak = pEntry->isWeak; + cbMeta.flag = 0; + + cbMeta.oldCfg = pFinish->oldCfg; + cbMeta.newCfg = pFinish->newCfg; + cbMeta.newCfgIndex = pFinish->newCfgIndex; + cbMeta.newCfgTerm = pFinish->newCfgTerm; + cbMeta.newCfgSeqNum = pFinish->newCfgSeqNum; + + ths->pFsm->FpReConfigCb(ths->pFsm, pRpcMsg, cbMeta); + } + + // update changing + ths->changing = false; + + char tmpbuf[512]; + char* oldStr = syncCfg2SimpleStr(&(pFinish->oldCfg)); + char* newStr = syncCfg2SimpleStr(&(pFinish->newCfg)); + snprintf(tmpbuf, sizeof(tmpbuf), "config change finish from %d to %d, index:%ld, %s --> %s", + pFinish->oldCfg.replicaNum, pFinish->newCfg.replicaNum, pFinish->newCfgIndex, oldStr, newStr); + taosMemoryFree(oldStr); + taosMemoryFree(newStr); + syncNodeEventLog(ths, tmpbuf); + + syncReconfigFinishDestroy(pFinish); + + return 0; +} + +static int32_t syncNodeConfigChange(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* pEntry, + SyncReconfigFinish* pFinish) { + // old config SSyncCfg oldSyncCfg = ths->pRaftCfg->cfg; + // new config SSyncCfg newSyncCfg; int32_t ret = syncCfgFromStr(pRpcMsg->pCont, &newSyncCfg); ASSERT(ret == 0); @@ -2154,72 +2320,38 @@ static int32_t syncNodeConfigChange(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftE // update new config myIndex syncNodeUpdateNewConfigIndex(ths, &newSyncCfg); - bool IamInNew = syncNodeInConfig(ths, &newSyncCfg); + // do config change + syncNodeDoConfigChange(ths, &newSyncCfg, pEntry->index); - /* - for (int i = 0; i < newSyncCfg.replicaNum; ++i) { - if (strcmp(ths->myNodeInfo.nodeFqdn, (newSyncCfg.nodeInfo)[i].nodeFqdn) == 0 && - ths->myNodeInfo.nodePort == (newSyncCfg.nodeInfo)[i].nodePort) { - newSyncCfg.myIndex = i; - IamInNew = true; - break; - } - } - */ + // set pFinish + pFinish->oldCfg = oldSyncCfg; + pFinish->newCfg = newSyncCfg; + pFinish->newCfgIndex = pEntry->index; + pFinish->newCfgTerm = pEntry->term; + pFinish->newCfgSeqNum = pEntry->seqNum; - bool isDrop; - - if (IamInNew) { - syncNodeUpdateConfig(ths, &newSyncCfg, pEntry->index, &isDrop); - - // change isStandBy to normal - if (!isDrop) { - char tmpbuf[128]; - snprintf(tmpbuf, sizeof(tmpbuf), "config change from %d to %d", oldSyncCfg.replicaNum, newSyncCfg.replicaNum); - if (ths->state == TAOS_SYNC_STATE_LEADER) { - syncNodeBecomeLeader(ths, tmpbuf); - } else { - syncNodeBecomeFollower(ths, tmpbuf); - } - } - } else { - char tmpbuf[128]; - snprintf(tmpbuf, sizeof(tmpbuf), "config change2 from %d to %d", oldSyncCfg.replicaNum, newSyncCfg.replicaNum); - syncNodeBecomeFollower(ths, tmpbuf); - } + return 0; +} - if (gRaftDetailLog) { - char* sOld = syncCfg2Str(&oldSyncCfg); - char* sNew = syncCfg2Str(&newSyncCfg); - sInfo("==config change== 0x11 old:%s new:%s isDrop:%d index:%ld IamInNew:%d \n", sOld, sNew, isDrop, pEntry->index, - IamInNew); - taosMemoryFree(sOld); - taosMemoryFree(sNew); - } +static int32_t syncNodeProposeConfigChangeFinish(SSyncNode* ths, SyncReconfigFinish* pFinish) { + SRpcMsg rpcMsg; + syncReconfigFinish2RpcMsg(pFinish, &rpcMsg); - // always call FpReConfigCb - if (ths->pFsm->FpReConfigCb != NULL) { - SReConfigCbMeta cbMeta = {0}; - cbMeta.code = 0; - cbMeta.currentTerm = ths->pRaftStore->currentTerm; - cbMeta.index = pEntry->index; - cbMeta.term = pEntry->term; - cbMeta.newCfg = newSyncCfg; - cbMeta.oldCfg = oldSyncCfg; - cbMeta.seqNum = pEntry->seqNum; - cbMeta.flag = 0x11; - cbMeta.isDrop = isDrop; - ths->pFsm->FpReConfigCb(ths->pFsm, pRpcMsg, cbMeta); + int32_t code = syncNodePropose(ths, &rpcMsg, false); + if (code != 0) { + sError("syncNodeProposeConfigChangeFinish error"); + ths->changing = false; } - return 0; } int32_t syncNodeCommit(SSyncNode* ths, SyncIndex beginIndex, SyncIndex endIndex, uint64_t flag) { int32_t code = 0; ESyncState state = flag; - sDebug("vgId:%d sync event currentTerm:%lu commit by wal from index:%" PRId64 " to index:%" PRId64 ", %s", ths->vgId, - ths->pRaftStore->currentTerm, beginIndex, endIndex, syncUtilState2String(state)); + + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "commit by wal from index:%ld to index:%ld", beginIndex, endIndex); + syncNodeEventLog(ths, eventLog); // execute fsm if (ths->pFsm != NULL) { @@ -2235,8 +2367,9 @@ int32_t syncNodeCommit(SSyncNode* ths, SyncIndex beginIndex, SyncIndex endIndex, // user commit if (ths->pFsm->FpCommitCb != NULL && syncUtilUserCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta; + SFsmCbMeta cbMeta = {0}; cbMeta.index = pEntry->index; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; cbMeta.code = 0; cbMeta.state = ths->state; @@ -2250,7 +2383,21 @@ int32_t syncNodeCommit(SSyncNode* ths, SyncIndex beginIndex, SyncIndex endIndex, // config change if (pEntry->originalRpcType == TDMT_SYNC_CONFIG_CHANGE) { - code = syncNodeConfigChange(ths, &rpcMsg, pEntry); + SyncReconfigFinish* pFinish = syncReconfigFinishBuild(ths->vgId); + ASSERT(pFinish != NULL); + + code = syncNodeConfigChange(ths, &rpcMsg, pEntry, pFinish); + ASSERT(code == 0); + + if (ths->state == TAOS_SYNC_STATE_LEADER) { + syncNodeProposeConfigChangeFinish(ths, pFinish); + } + syncReconfigFinishDestroy(pFinish); + } + + // config change finish + if (pEntry->originalRpcType == TDMT_SYNC_CONFIG_CHANGE_FINISH) { + code = syncNodeConfigChangeFinish(ths, &rpcMsg, pEntry); ASSERT(code == 0); } @@ -2261,14 +2408,17 @@ int32_t syncNodeCommit(SSyncNode* ths, SyncIndex beginIndex, SyncIndex endIndex, } // restore finish + // if only snapshot, a noop entry will be append, so syncLogLastIndex is always ok if (pEntry->index == ths->pLogStore->syncLogLastIndex(ths->pLogStore)) { if (ths->restoreFinish == false) { if (ths->pFsm->FpRestoreFinishCb != NULL) { ths->pFsm->FpRestoreFinishCb(ths->pFsm); } ths->restoreFinish = true; - sDebug("vgId:%d sync event currentTerm:%lu restore finish, %s, index:%ld", ths->vgId, - ths->pRaftStore->currentTerm, syncUtilState2String(ths->state), pEntry->index); + + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "restore finish, index:%ld", pEntry->index); + syncNodeEventLog(ths, eventLog); } } @@ -2298,3 +2448,28 @@ SSyncSnapshotSender* syncNodeGetSnapshotSender(SSyncNode* ths, SRaftId* pDestId) } return pSender; } + +bool syncNodeCanChange(SSyncNode* pSyncNode) { + if (pSyncNode->changing) { + sError("sync cannot change"); + return false; + } + + if ((pSyncNode->commitIndex >= SYNC_INDEX_BEGIN)) { + SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); + if (pSyncNode->commitIndex != lastIndex) { + sError("sync cannot change2"); + return false; + } + } + + for (int i = 0; i < pSyncNode->peersNum; ++i) { + SSyncSnapshotSender* pSender = syncNodeGetSnapshotSender(pSyncNode, &(pSyncNode->peersId)[i]); + if (pSender->start) { + sError("sync cannot change3"); + return false; + } + } + + return true; +} \ No newline at end of file diff --git a/source/libs/sync/src/syncMessage.c b/source/libs/sync/src/syncMessage.c index 23165f67901bfabacf594730d4aef9260b1a1aa0..454609009c8ecd3dfee235c5ea87ecd2a49c0fe2 100644 --- a/source/libs/sync/src/syncMessage.c +++ b/source/libs/sync/src/syncMessage.c @@ -2227,4 +2227,133 @@ void syncLeaderTransferLog2(char* s, const SyncLeaderTransfer* pMsg) { sTrace("syncLeaderTransferLog2 | len:%lu | %s | %s", strlen(serialized), s, serialized); taosMemoryFree(serialized); } +} + +// --------------------------------------------- +SyncReconfigFinish* syncReconfigFinishBuild(int32_t vgId) { + uint32_t bytes = sizeof(SyncReconfigFinish); + SyncReconfigFinish* pMsg = taosMemoryMalloc(bytes); + memset(pMsg, 0, bytes); + pMsg->bytes = bytes; + pMsg->vgId = vgId; + pMsg->msgType = TDMT_SYNC_CONFIG_CHANGE_FINISH; + return pMsg; +} + +void syncReconfigFinishDestroy(SyncReconfigFinish* pMsg) { + if (pMsg != NULL) { + taosMemoryFree(pMsg); + } +} + +void syncReconfigFinishSerialize(const SyncReconfigFinish* pMsg, char* buf, uint32_t bufLen) { + assert(pMsg->bytes <= bufLen); + memcpy(buf, pMsg, pMsg->bytes); +} + +void syncReconfigFinishDeserialize(const char* buf, uint32_t len, SyncReconfigFinish* pMsg) { + memcpy(pMsg, buf, len); + assert(len == pMsg->bytes); +} + +char* syncReconfigFinishSerialize2(const SyncReconfigFinish* pMsg, uint32_t* len) { + char* buf = taosMemoryMalloc(pMsg->bytes); + assert(buf != NULL); + syncReconfigFinishSerialize(pMsg, buf, pMsg->bytes); + if (len != NULL) { + *len = pMsg->bytes; + } + return buf; +} + +SyncReconfigFinish* syncReconfigFinishDeserialize2(const char* buf, uint32_t len) { + uint32_t bytes = *((uint32_t*)buf); + SyncReconfigFinish* pMsg = taosMemoryMalloc(bytes); + assert(pMsg != NULL); + syncReconfigFinishDeserialize(buf, len, pMsg); + assert(len == pMsg->bytes); + return pMsg; +} + +void syncReconfigFinish2RpcMsg(const SyncReconfigFinish* pMsg, SRpcMsg* pRpcMsg) { + memset(pRpcMsg, 0, sizeof(*pRpcMsg)); + pRpcMsg->msgType = pMsg->msgType; + pRpcMsg->contLen = pMsg->bytes; + pRpcMsg->pCont = rpcMallocCont(pRpcMsg->contLen); + syncReconfigFinishSerialize(pMsg, pRpcMsg->pCont, pRpcMsg->contLen); +} + +void syncReconfigFinishFromRpcMsg(const SRpcMsg* pRpcMsg, SyncReconfigFinish* pMsg) { + syncReconfigFinishDeserialize(pRpcMsg->pCont, pRpcMsg->contLen, pMsg); +} + +SyncReconfigFinish* syncReconfigFinishFromRpcMsg2(const SRpcMsg* pRpcMsg) { + SyncReconfigFinish* pMsg = syncReconfigFinishDeserialize2(pRpcMsg->pCont, pRpcMsg->contLen); + assert(pMsg != NULL); + return pMsg; +} + +cJSON* syncReconfigFinish2Json(const SyncReconfigFinish* pMsg) { + char u64buf[128]; + cJSON* pRoot = cJSON_CreateObject(); + + if (pMsg != NULL) { + cJSON_AddNumberToObject(pRoot, "bytes", pMsg->bytes); + cJSON_AddNumberToObject(pRoot, "vgId", pMsg->vgId); + cJSON_AddNumberToObject(pRoot, "msgType", pMsg->msgType); + + cJSON* pOldCfg = syncCfg2Json((SSyncCfg*)(&(pMsg->oldCfg))); + cJSON* pNewCfg = syncCfg2Json((SSyncCfg*)(&(pMsg->newCfg))); + cJSON_AddItemToObject(pRoot, "oldCfg", pOldCfg); + cJSON_AddItemToObject(pRoot, "newCfg", pNewCfg); + + snprintf(u64buf, sizeof(u64buf), "%ld", pMsg->newCfgIndex); + cJSON_AddStringToObject(pRoot, "newCfgIndex", u64buf); + + snprintf(u64buf, sizeof(u64buf), "%lu", pMsg->newCfgTerm); + cJSON_AddStringToObject(pRoot, "newCfgTerm", u64buf); + + snprintf(u64buf, sizeof(u64buf), "%lu", pMsg->newCfgSeqNum); + cJSON_AddStringToObject(pRoot, "newCfgSeqNum", u64buf); + } + + cJSON* pJson = cJSON_CreateObject(); + cJSON_AddItemToObject(pJson, "SyncReconfigFinish", pRoot); + return pJson; +} + +char* syncReconfigFinish2Str(const SyncReconfigFinish* pMsg) { + cJSON* pJson = syncReconfigFinish2Json(pMsg); + char* serialized = cJSON_Print(pJson); + cJSON_Delete(pJson); + return serialized; +} + +// for debug ---------------------- +void syncReconfigFinishPrint(const SyncReconfigFinish* pMsg) { + char* serialized = syncReconfigFinish2Str(pMsg); + printf("syncReconfigFinishPrint | len:%lu | %s \n", strlen(serialized), serialized); + fflush(NULL); + taosMemoryFree(serialized); +} + +void syncReconfigFinishPrint2(char* s, const SyncReconfigFinish* pMsg) { + char* serialized = syncReconfigFinish2Str(pMsg); + printf("syncReconfigFinishPrint2 | len:%lu | %s | %s \n", strlen(serialized), s, serialized); + fflush(NULL); + taosMemoryFree(serialized); +} + +void syncReconfigFinishLog(const SyncReconfigFinish* pMsg) { + char* serialized = syncReconfigFinish2Str(pMsg); + sTrace("syncReconfigFinishLog | len:%lu | %s", strlen(serialized), serialized); + taosMemoryFree(serialized); +} + +void syncReconfigFinishLog2(char* s, const SyncReconfigFinish* pMsg) { + if (gRaftDetailLog) { + char* serialized = syncReconfigFinish2Str(pMsg); + sTrace("syncReconfigFinishLog2 | len:%lu | %s | %s", strlen(serialized), s, serialized); + taosMemoryFree(serialized); + } } \ No newline at end of file diff --git a/source/libs/sync/src/syncRaftCfg.c b/source/libs/sync/src/syncRaftCfg.c index 45e00aca2c33be3a5f665d6eaaf7c43043c6a6e0..08c3e0126c143acb4196d9196ed8ee09f6678f43 100644 --- a/source/libs/sync/src/syncRaftCfg.c +++ b/source/libs/sync/src/syncRaftCfg.c @@ -66,6 +66,13 @@ int32_t raftCfgPersist(SRaftCfg *pRaftCfg) { return 0; } +int32_t raftCfgAddConfigIndex(SRaftCfg *pRaftCfg, SyncIndex configIndex) { + ASSERT(pRaftCfg->configIndexCount <= MAX_CONFIG_INDEX_COUNT); + (pRaftCfg->configIndexArr)[pRaftCfg->configIndexCount] = configIndex; + ++(pRaftCfg->configIndexCount); + return 0; +} + cJSON *syncCfg2Json(SSyncCfg *pSyncCfg) { char u64buf[128] = {0}; cJSON *pRoot = cJSON_CreateObject(); @@ -85,11 +92,6 @@ cJSON *syncCfg2Json(SSyncCfg *pSyncCfg) { } return pRoot; - /* - cJSON *pJson = cJSON_CreateObject(); - cJSON_AddItemToObject(pJson, "SSyncCfg", pRoot); - return pJson; - */ } char *syncCfg2Str(SSyncCfg *pSyncCfg) { @@ -99,6 +101,29 @@ char *syncCfg2Str(SSyncCfg *pSyncCfg) { return serialized; } +char *syncCfg2SimpleStr(SSyncCfg *pSyncCfg) { + int32_t len = 512; + char * s = taosMemoryMalloc(len); + memset(s, 0, len); + + snprintf(s, len, "{replica-num:%d, my-index:%d, ", pSyncCfg->replicaNum, pSyncCfg->myIndex); + char *p = s + strlen(s); + for (int i = 0; i < pSyncCfg->replicaNum; ++i) { + /* + if (p + 128 + 32 > s + len) { + break; + } + */ + char buf[128 + 32]; + snprintf(buf, sizeof(buf), "%s:%d, ", pSyncCfg->nodeInfo[i].nodeFqdn, pSyncCfg->nodeInfo[i].nodePort); + strncpy(p, buf, sizeof(buf)); + p = s + strlen(s); + } + strcpy(p - 2, "}"); + + return s; +} + int32_t syncCfgFromJson(const cJSON *pRoot, SSyncCfg *pSyncCfg) { memset(pSyncCfg, 0, sizeof(SSyncCfg)); // cJSON *pJson = cJSON_GetObjectItem(pRoot, "SSyncCfg"); @@ -154,6 +179,16 @@ cJSON *raftCfg2Json(SRaftCfg *pRaftCfg) { snprintf(buf64, sizeof(buf64), "%ld", pRaftCfg->lastConfigIndex); cJSON_AddStringToObject(pRoot, "lastConfigIndex", buf64); + cJSON_AddNumberToObject(pRoot, "configIndexCount", pRaftCfg->configIndexCount); + cJSON *pIndexArr = cJSON_CreateArray(); + cJSON_AddItemToObject(pRoot, "configIndexArr", pIndexArr); + for (int i = 0; i < pRaftCfg->configIndexCount; ++i) { + snprintf(buf64, sizeof(buf64), "%ld", (pRaftCfg->configIndexArr)[i]); + cJSON *pIndexObj = cJSON_CreateObject(); + cJSON_AddStringToObject(pIndexObj, "index", buf64); + cJSON_AddItemToArray(pIndexArr, pIndexObj); + } + cJSON *pJson = cJSON_CreateObject(); cJSON_AddItemToObject(pJson, "RaftCfg", pRoot); return pJson; @@ -177,6 +212,9 @@ int32_t raftCfgCreateFile(SSyncCfg *pCfg, SRaftCfgMeta meta, const char *path) { raftCfg.isStandBy = meta.isStandBy; raftCfg.snapshotEnable = meta.snapshotEnable; raftCfg.lastConfigIndex = meta.lastConfigIndex; + raftCfg.configIndexCount = 1; + memset(raftCfg.configIndexArr, 0, sizeof(raftCfg.configIndexArr)); + raftCfg.configIndexArr[0] = -1; char *s = raftCfg2Str(&raftCfg); char buf[CONFIG_FILE_LEN] = {0}; @@ -207,6 +245,23 @@ int32_t raftCfgFromJson(const cJSON *pRoot, SRaftCfg *pRaftCfg) { cJSON *pJsonLastConfigIndex = cJSON_GetObjectItem(pJson, "lastConfigIndex"); pRaftCfg->lastConfigIndex = atoll(cJSON_GetStringValue(pJsonLastConfigIndex)); + cJSON *pJsonConfigIndexCount = cJSON_GetObjectItem(pJson, "configIndexCount"); + pRaftCfg->configIndexCount = cJSON_GetNumberValue(pJsonConfigIndexCount); + + cJSON *pIndexArr = cJSON_GetObjectItem(pJson, "configIndexArr"); + int arraySize = cJSON_GetArraySize(pIndexArr); + assert(arraySize == pRaftCfg->configIndexCount); + + memset(pRaftCfg->configIndexArr, 0, sizeof(pRaftCfg->configIndexArr)); + for (int i = 0; i < arraySize; ++i) { + cJSON *pIndexObj = cJSON_GetArrayItem(pIndexArr, i); + assert(pIndexObj != NULL); + + cJSON *pIndex = cJSON_GetObjectItem(pIndexObj, "index"); + assert(cJSON_IsString(pIndex)); + (pRaftCfg->configIndexArr)[i] = atoll(pIndex->valuestring); + } + cJSON * pJsonSyncCfg = cJSON_GetObjectItem(pJson, "SSyncCfg"); int32_t code = syncCfgFromJson(pJsonSyncCfg, &(pRaftCfg->cfg)); ASSERT(code == 0); @@ -252,6 +307,12 @@ void syncCfgLog2(char *s, SSyncCfg *pCfg) { taosMemoryFree(serialized); } +void syncCfgLog3(char *s, SSyncCfg *pCfg) { + char *serialized = syncCfg2SimpleStr(pCfg); + sTrace("syncCfgLog3 | len:%lu | %s | %s", strlen(serialized), s, serialized); + taosMemoryFree(serialized); +} + void raftCfgPrint(SRaftCfg *pCfg) { char *serialized = raftCfg2Str(pCfg); printf("raftCfgPrint | len:%lu | %s \n", strlen(serialized), serialized); diff --git a/source/libs/sync/src/syncRaftLog.c b/source/libs/sync/src/syncRaftLog.c index 31ac44fa81b91a7bd13af621f1e291b5ac60ca7a..79d9b329c10bf8e2fc6db6ee7865bd31fbc764ae 100644 --- a/source/libs/sync/src/syncRaftLog.c +++ b/source/libs/sync/src/syncRaftLog.c @@ -163,10 +163,10 @@ static int32_t raftLogAppendEntry(struct SSyncLogStore* pLogStore, SSyncRaftEntr walFsync(pWal, true); - sDebug("vgId:%d sync event currentTerm:%lu write index:%ld, %s, isStandBy:%d, msgType:%s,%d, originalRpcType:%s,%d", - pData->pSyncNode->vgId, pData->pSyncNode->pRaftStore->currentTerm, pEntry->index, - syncUtilState2String(pData->pSyncNode->state), pData->pSyncNode->pRaftCfg->isStandBy, - TMSG_INFO(pEntry->msgType), pEntry->msgType, TMSG_INFO(pEntry->originalRpcType), pEntry->originalRpcType); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "write index:%ld, type:%s,%d, type2:%s,%d", pEntry->index, + TMSG_INFO(pEntry->msgType), pEntry->msgType, TMSG_INFO(pEntry->originalRpcType), pEntry->originalRpcType); + syncNodeEventLog(pData->pSyncNode, eventLog); return code; } @@ -318,15 +318,13 @@ int32_t logStoreAppendEntry(SSyncLogStore* pLogStore, SSyncRaftEntry* pEntry) { linuxErrMsg); ASSERT(0); } - // assert(code == 0); walFsync(pWal, true); - sDebug( - "vgId:%d sync event currentTerm:%lu old write index:%ld, %s, isStandBy:%d, msgType:%s,%d, originalRpcType:%s,%d", - pData->pSyncNode->vgId, pData->pSyncNode->pRaftStore->currentTerm, pEntry->index, - syncUtilState2String(pData->pSyncNode->state), pData->pSyncNode->pRaftCfg->isStandBy, TMSG_INFO(pEntry->msgType), - pEntry->msgType, TMSG_INFO(pEntry->originalRpcType), pEntry->originalRpcType); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "old write index:%ld, type:%s,%d, type2:%s,%d", pEntry->index, + TMSG_INFO(pEntry->msgType), pEntry->msgType, TMSG_INFO(pEntry->originalRpcType), pEntry->originalRpcType); + syncNodeEventLog(pData->pSyncNode, eventLog); return code; } diff --git a/source/libs/sync/src/syncReplication.c b/source/libs/sync/src/syncReplication.c index 08564f8293695c1eb2155191c3ae1b28396962e6..f044ae5733aa7714af208fe9c4208859206ac35e 100644 --- a/source/libs/sync/src/syncReplication.c +++ b/source/libs/sync/src/syncReplication.c @@ -124,7 +124,7 @@ int32_t syncNodeAppendEntriesPeersSnapshot(SSyncNode* pSyncNode) { logStoreSimpleLog2("begin append entries peers LogStore:", pSyncNode->pLogStore); if (gRaftDetailLog) { SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); sTrace("begin append entries peers, snapshot.lastApplyIndex:%ld, snapshot.lastApplyTerm:%lu", snapshot.lastApplyIndex, snapshot.lastApplyTerm); } diff --git a/source/libs/sync/src/syncRespMgr.c b/source/libs/sync/src/syncRespMgr.c index 5b127793d644a2387d13e53fc855455bb0266a07..354575d29e6be2b4abf5a88743572c869d3344f7 100644 --- a/source/libs/sync/src/syncRespMgr.c +++ b/source/libs/sync/src/syncRespMgr.c @@ -46,9 +46,11 @@ int64_t syncRespMgrAdd(SSyncRespMgr *pObj, SRespStub *pStub) { taosHashPut(pObj->pRespHash, &keyCode, sizeof(keyCode), pStub, sizeof(SRespStub)); SSyncNode *pSyncNode = pObj->data; - sDebug("vgId:%d sync event currentTerm:%lu resp mgr add, msgType:%s,%d seq:%lu handle:%p ahandle:%p", pSyncNode->vgId, - pSyncNode->pRaftStore->currentTerm, TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, keyCode, - pStub->rpcMsg.info.handle, pStub->rpcMsg.info.ahandle); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "resp mgr add, type:%s,%d, seq:%lu, handle:%p, ahandle:%p", + TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, keyCode, pStub->rpcMsg.info.handle, + pStub->rpcMsg.info.ahandle); + syncNodeEventLog(pSyncNode, eventLog); taosThreadMutexUnlock(&(pObj->mutex)); return keyCode; @@ -71,9 +73,11 @@ int32_t syncRespMgrGet(SSyncRespMgr *pObj, uint64_t index, SRespStub *pStub) { memcpy(pStub, pTmp, sizeof(SRespStub)); SSyncNode *pSyncNode = pObj->data; - sDebug("vgId:%d sync event currentTerm:%lu resp mgr get, msgType:%s,%d seq:%lu handle:%p ahandle:%p", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, - index, pStub->rpcMsg.info.handle, pStub->rpcMsg.info.ahandle); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "resp mgr get, type:%s,%d, seq:%lu, handle:%p, ahandle:%p", + TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, index, pStub->rpcMsg.info.handle, + pStub->rpcMsg.info.ahandle); + syncNodeEventLog(pSyncNode, eventLog); taosThreadMutexUnlock(&(pObj->mutex)); return 1; // get one object @@ -90,9 +94,11 @@ int32_t syncRespMgrGetAndDel(SSyncRespMgr *pObj, uint64_t index, SRespStub *pStu memcpy(pStub, pTmp, sizeof(SRespStub)); SSyncNode *pSyncNode = pObj->data; - sDebug("vgId:%d sync event currentTerm:%lu resp mgr get and del, msgType:%s,%d seq:%lu handle:%p ahandle:%p", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, - index, pStub->rpcMsg.info.handle, pStub->rpcMsg.info.ahandle); + char eventLog[128]; + snprintf(eventLog, sizeof(eventLog), "resp mgr get-and-del, type:%s,%d, seq:%lu, handle:%p, ahandle:%p", + TMSG_INFO(pStub->rpcMsg.msgType), pStub->rpcMsg.msgType, index, pStub->rpcMsg.info.handle, + pStub->rpcMsg.info.ahandle); + syncNodeEventLog(pSyncNode, eventLog); taosHashRemove(pObj->pRespHash, &index, sizeof(index)); taosThreadMutexUnlock(&(pObj->mutex)); diff --git a/source/libs/sync/src/syncSnapshot.c b/source/libs/sync/src/syncSnapshot.c index d3785060d4acae4f5a7f64b063fdbe9844cc85ed..c694a0b71539c410dd20c06c3085b2c10aebcbc0 100644 --- a/source/libs/sync/src/syncSnapshot.c +++ b/source/libs/sync/src/syncSnapshot.c @@ -21,7 +21,8 @@ #include "syncUtil.h" #include "wal.h" -static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SRaftId fromId); +static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, + SyncSnapshotSend *pBeginMsg); //---------------------------------- SSyncSnapshotSender *snapshotSenderCreate(SSyncNode *pSyncNode, int32_t replicaIndex) { @@ -45,11 +46,12 @@ SSyncSnapshotSender *snapshotSenderCreate(SSyncNode *pSyncNode, int32_t replicaI pSender->replicaIndex = replicaIndex; pSender->term = pSyncNode->pRaftStore->currentTerm; pSender->privateTerm = taosGetTimestampMs() + 100; - pSender->pSyncNode->pFsm->FpGetSnapshot(pSender->pSyncNode->pFsm, &(pSender->snapshot)); + pSender->pSyncNode->pFsm->FpGetSnapshotInfo(pSender->pSyncNode->pFsm, &(pSender->snapshot)); pSender->finish = false; } else { sError("snapshotSenderCreate cannot create sender"); } + return pSender; } @@ -83,7 +85,11 @@ void snapshotSenderStart(SSyncSnapshotSender *pSender) { pSender->blockLen = 0; // get current snapshot info - pSender->pSyncNode->pFsm->FpGetSnapshot(pSender->pSyncNode->pFsm, &(pSender->snapshot)); + pSender->pSyncNode->pFsm->FpGetSnapshotInfo(pSender->pSyncNode->pFsm, &(pSender->snapshot)); + + sTrace("snapshotSenderStart lastApplyIndex:%ld, lastApplyTerm:%lu, lastConfigIndex:%ld", + pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex); + if (pSender->snapshot.lastConfigIndex != SYNC_INDEX_INVALID) { /* SSyncRaftEntry *pEntry = NULL; @@ -134,86 +140,13 @@ void snapshotSenderStart(SSyncSnapshotSender *pSender) { syncSnapshotSend2RpcMsg(pMsg, &rpcMsg); syncNodeSendMsgById(&(pMsg->destId), pSender->pSyncNode, &rpcMsg); - char host[128]; - uint16_t port; - syncUtilU642Addr(pSender->pSyncNode->replicasId[pSender->replicaIndex].addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d begin seq:%d ack:%d lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu send " - "msg:%s", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, - pSender->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d begin seq:%d ack:%d lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, - pSender->privateTerm); - } + char *eventLog = snapshotSender2SimpleStr(pSender, "snapshot sender send"); + syncNodeEventLog(pSender->pSyncNode, eventLog); + taosMemoryFree(eventLog); syncSnapshotSendDestroy(pMsg); } -#if 0 -// when entry in snapshot, start sender -void snapshotSenderStart(SSyncSnapshotSender *pSender) { - if (!(pSender->start)) { - // start - snapshotSenderDoStart(pSender); - pSender->start = true; - } else { - // already start - ASSERT(pSender->pSyncNode->pRaftStore->currentTerm >= pSender->term); - - // if current term is higher, need start again - if (pSender->pSyncNode->pRaftStore->currentTerm > pSender->term) { - // force peer rollback - SyncSnapshotSend *pMsg = syncSnapshotSendBuild(0, pSender->pSyncNode->vgId); - pMsg->srcId = pSender->pSyncNode->myRaftId; - pMsg->destId = (pSender->pSyncNode->replicasId)[pSender->replicaIndex]; - pMsg->term = pSender->pSyncNode->pRaftStore->currentTerm; - pMsg->lastIndex = pSender->snapshot.lastApplyIndex; - pMsg->lastTerm = pSender->snapshot.lastApplyTerm; - pMsg->seq = SYNC_SNAPSHOT_SEQ_FORCE_CLOSE; - - SRpcMsg rpcMsg; - syncSnapshotSend2RpcMsg(pMsg, &rpcMsg); - syncNodeSendMsgById(&(pMsg->destId), pSender->pSyncNode, &rpcMsg); - - char *msgStr = syncSnapshotSend2Str(pMsg); - sTrace("snapshot send force close seq:%d ack:%d send msg:%s", pSender->seq, pSender->ack, msgStr); - taosMemoryFree(msgStr); - - syncSnapshotSendDestroy(pMsg); - - // close reader - int32_t ret = pSender->pSyncNode->pFsm->FpSnapshotStopRead(pSender->pSyncNode->pFsm, pSender->pReader); - ASSERT(ret == 0); - pSender->pReader = NULL; - - // start again - snapshotSenderDoStart(pSender); - pSender->start = true; - } else { - // current term, do nothing - ASSERT(pSender->pSyncNode->pRaftStore->currentTerm == pSender->term); - } - } - - char *s = snapshotSender2Str(pSender); - sInfo("snapshotSenderStart %s", s); - taosMemoryFree(s); -} -#endif - void snapshotSenderStop(SSyncSnapshotSender *pSender) { if (pSender->pReader != NULL) { int32_t ret = pSender->pSyncNode->pFsm->FpSnapshotStopRead(pSender->pSyncNode->pFsm, pSender->pReader); @@ -275,39 +208,15 @@ int32_t snapshotSend(SSyncSnapshotSender *pSender) { syncSnapshotSend2RpcMsg(pMsg, &rpcMsg); syncNodeSendMsgById(&(pMsg->destId), pSender->pSyncNode, &rpcMsg); - char host[128]; - uint16_t port; - syncUtilU642Addr(pSender->pSyncNode->replicasId[pSender->replicaIndex].addr, host, sizeof(host), &port); - if (pSender->seq == SYNC_SNAPSHOT_SEQ_END) { - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d finish seq:%d ack:%d lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu send " - "msg:%s", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, - pSender->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d finish seq:%d ack:%d lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, - pSender->privateTerm); - } + char *eventLog = snapshotSender2SimpleStr(pSender, "snapshot sender finish"); + syncNodeEventLog(pSender->pSyncNode, eventLog); + taosMemoryFree(eventLog); + } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d sending seq:%d ack:%d lastApplyIndex:%ld " - "lastApplyTerm:%lu " - "lastConfigIndex:%ld privateTerm:%lu", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, pSender->snapshot.lastConfigIndex, - pSender->privateTerm); + char *eventLog = snapshotSender2SimpleStr(pSender, "snapshot sender sending"); + syncNodeEventLog(pSender->pSyncNode, eventLog); + taosMemoryFree(eventLog); } syncSnapshotSendDestroy(pMsg); @@ -332,22 +241,9 @@ int32_t snapshotReSend(SSyncSnapshotSender *pSender) { syncSnapshotSend2RpcMsg(pMsg, &rpcMsg); syncNodeSendMsgById(&(pMsg->destId), pSender->pSyncNode, &rpcMsg); - char host[128]; - uint16_t port; - syncUtilU642Addr(pSender->pSyncNode->replicasId[pSender->replicaIndex].addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot send to %s:%d resend seq:%d ack:%d privateTerm:%lu send msg:%s", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, pSender->ack, - pSender->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug("vgId:%d sync event currentTerm:%lu snapshot send to %s:%d resend seq:%d ack:%d privateTerm:%lu", - pSender->pSyncNode->vgId, pSender->pSyncNode->pRaftStore->currentTerm, host, port, pSender->seq, - pSender->ack, pSender->privateTerm); - } + char *eventLog = snapshotSender2SimpleStr(pSender, "snapshot sender resend"); + syncNodeEventLog(pSender->pSyncNode, eventLog); + taosMemoryFree(eventLog); syncSnapshotSendDestroy(pMsg); } @@ -411,6 +307,23 @@ char *snapshotSender2Str(SSyncSnapshotSender *pSender) { return serialized; } +char *snapshotSender2SimpleStr(SSyncSnapshotSender *pSender, char *event) { + int32_t len = 256; + char *s = taosMemoryMalloc(len); + + SRaftId destId = pSender->pSyncNode->replicasId[pSender->replicaIndex]; + char host[128]; + uint16_t port; + syncUtilU642Addr(destId.addr, host, sizeof(host), &port); + + snprintf(s, len, "%s %p laindex:%ld laterm:%lu lcindex:%ld seq:%d ack:%d finish:%d pterm:%lu replica-index:%d %s:%d", + event, pSender, pSender->snapshot.lastApplyIndex, pSender->snapshot.lastApplyTerm, + pSender->snapshot.lastConfigIndex, pSender->seq, pSender->ack, pSender->finish, pSender->privateTerm, + pSender->replicaIndex, host, port); + + return s; +} + // ------------------------------------- SSyncSnapshotReceiver *snapshotReceiverCreate(SSyncNode *pSyncNode, SRaftId fromId) { bool condition = (pSyncNode->pFsm->FpSnapshotStartWrite != NULL) && (pSyncNode->pFsm->FpSnapshotStopWrite != NULL) && @@ -429,6 +342,10 @@ SSyncSnapshotReceiver *snapshotReceiverCreate(SSyncNode *pSyncNode, SRaftId from pReceiver->fromId = fromId; pReceiver->term = pSyncNode->pRaftStore->currentTerm; pReceiver->privateTerm = 0; + pReceiver->snapshot.data = NULL; + pReceiver->snapshot.lastApplyIndex = -1; + pReceiver->snapshot.lastApplyTerm = 0; + pReceiver->snapshot.lastConfigIndex = -1; } else { sInfo("snapshotReceiverCreate cannot create receiver"); @@ -446,11 +363,16 @@ void snapshotReceiverDestroy(SSyncSnapshotReceiver *pReceiver) { bool snapshotReceiverIsStart(SSyncSnapshotReceiver *pReceiver) { return pReceiver->start; } // begin receive snapshot msg (current term, seq begin) -static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SRaftId fromId) { +static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, + SyncSnapshotSend *pBeginMsg) { pReceiver->term = pReceiver->pSyncNode->pRaftStore->currentTerm; pReceiver->privateTerm = privateTerm; pReceiver->ack = SYNC_SNAPSHOT_SEQ_BEGIN; - pReceiver->fromId = fromId; + pReceiver->fromId = pBeginMsg->srcId; + + pReceiver->snapshot.lastApplyIndex = pBeginMsg->lastIndex; + pReceiver->snapshot.lastApplyTerm = pBeginMsg->lastTerm; + pReceiver->snapshot.lastConfigIndex = pBeginMsg->lastConfigIndex; ASSERT(pReceiver->pWriter == NULL); int32_t ret = pReceiver->pSyncNode->pFsm->FpSnapshotStartWrite(pReceiver->pSyncNode->pFsm, &(pReceiver->pWriter)); @@ -459,10 +381,10 @@ static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncTerm p // if receiver receive msg from seq = SYNC_SNAPSHOT_SEQ_BEGIN, start receiver // if already start, force close, start again -void snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SRaftId fromId) { +void snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTerm, SyncSnapshotSend *pBeginMsg) { if (!snapshotReceiverIsStart(pReceiver)) { // start - snapshotReceiverDoStart(pReceiver, privateTerm, fromId); + snapshotReceiverDoStart(pReceiver, privateTerm, pBeginMsg); pReceiver->start = true; } else { @@ -476,7 +398,7 @@ void snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncTerm privateTer pReceiver->pWriter = NULL; // start again - snapshotReceiverDoStart(pReceiver, privateTerm, fromId); + snapshotReceiverDoStart(pReceiver, privateTerm, pBeginMsg); pReceiver->start = true; } @@ -537,6 +459,15 @@ cJSON *snapshotReceiver2Json(SSyncSnapshotReceiver *pReceiver) { cJSON_AddNumberToObject(pFromId, "vgId", pReceiver->fromId.vgId); cJSON_AddItemToObject(pRoot, "fromId", pFromId); + snprintf(u64buf, sizeof(u64buf), "%lu", pReceiver->snapshot.lastApplyIndex); + cJSON_AddStringToObject(pRoot, "snapshot.lastApplyIndex", u64buf); + + snprintf(u64buf, sizeof(u64buf), "%lu", pReceiver->snapshot.lastApplyTerm); + cJSON_AddStringToObject(pRoot, "snapshot.lastApplyTerm", u64buf); + + snprintf(u64buf, sizeof(u64buf), "%lu", pReceiver->snapshot.lastConfigIndex); + cJSON_AddStringToObject(pRoot, "snapshot.lastConfigIndex", u64buf); + snprintf(u64buf, sizeof(u64buf), "%lu", pReceiver->term); cJSON_AddStringToObject(pRoot, "term", u64buf); @@ -556,6 +487,22 @@ char *snapshotReceiver2Str(SSyncSnapshotReceiver *pReceiver) { return serialized; } +char *snapshotReceiver2SimpleStr(SSyncSnapshotReceiver *pReceiver, char *event) { + int32_t len = 256; + char *s = taosMemoryMalloc(len); + + SRaftId fromId = pReceiver->fromId; + char host[128]; + uint16_t port; + syncUtilU642Addr(fromId.addr, host, sizeof(host), &port); + + snprintf(s, len, "%s %p start:%d ack:%d term:%lu pterm:%lu from:%s:%d laindex:%ld laterm:%lu lcindex:%ld", event, + pReceiver, pReceiver->start, pReceiver->ack, pReceiver->term, pReceiver->privateTerm, host, port, + pReceiver->snapshot.lastApplyIndex, pReceiver->snapshot.lastApplyTerm, pReceiver->snapshot.lastConfigIndex); + + return s; +} + // receiver do something int32_t syncNodeOnSnapshotSendCb(SSyncNode *pSyncNode, SyncSnapshotSend *pMsg) { // get receiver @@ -568,29 +515,13 @@ int32_t syncNodeOnSnapshotSendCb(SSyncNode *pSyncNode, SyncSnapshotSend *pMsg) { if (pMsg->term == pSyncNode->pRaftStore->currentTerm) { if (pMsg->seq == SYNC_SNAPSHOT_SEQ_BEGIN) { // begin - snapshotReceiverStart(pReceiver, pMsg->privateTerm, pMsg->srcId); + snapshotReceiverStart(pReceiver, pMsg->privateTerm, pMsg); pReceiver->ack = pMsg->seq; needRsp = true; - char host[128]; - uint16_t port; - syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d begin ack:%d, lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu, recv msg:%s", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, pMsg->lastIndex, - pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d begin ack:%d, lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld privateTerm:%lu", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, pMsg->lastIndex, - pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm); - } + char *eventLog = snapshotReceiver2SimpleStr(pReceiver, "snapshot receiver begin"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); } else if (pMsg->seq == SYNC_SNAPSHOT_SEQ_END) { // end, finish FSM @@ -602,130 +533,47 @@ int32_t syncNodeOnSnapshotSendCb(SSyncNode *pSyncNode, SyncSnapshotSend *pMsg) { // maybe update lastconfig if (pMsg->lastConfigIndex >= SYNC_INDEX_BEGIN) { - int32_t oldReplicaNum = pSyncNode->replicaNum; + // int32_t oldReplicaNum = pSyncNode->replicaNum; + SSyncCfg oldSyncCfg = pSyncNode->pRaftCfg->cfg; // update new config myIndex SSyncCfg newSyncCfg = pMsg->lastConfig; syncNodeUpdateNewConfigIndex(pSyncNode, &newSyncCfg); - bool IamInNew = syncNodeInConfig(pSyncNode, &newSyncCfg); -#if 0 - // update new config myIndex - bool IamInNew = false; - SSyncCfg newSyncCfg = pMsg->lastConfig; - for (int i = 0; i < newSyncCfg.replicaNum; ++i) { - if (strcmp(pSyncNode->myNodeInfo.nodeFqdn, (newSyncCfg.nodeInfo)[i].nodeFqdn) == 0 && - pSyncNode->myNodeInfo.nodePort == (newSyncCfg.nodeInfo)[i].nodePort) { - newSyncCfg.myIndex = i; - IamInNew = true; - break; - } - } -#endif - - bool isDrop; - if (IamInNew) { - sDebug( - "vgId:%d sync event currentTerm:%lu update config by snapshot, lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld ", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, pMsg->lastIndex, pMsg->lastTerm, - pMsg->lastConfigIndex); - syncNodeUpdateConfig(pSyncNode, &newSyncCfg, pMsg->lastConfigIndex, &isDrop); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu do not update config by snapshot, I am not in newCfg, " - "lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld ", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, pMsg->lastIndex, pMsg->lastTerm, - pMsg->lastConfigIndex); - } - - // change isStandBy to normal - if (!isDrop) { - char tmpbuf[128]; - snprintf(tmpbuf, sizeof(tmpbuf), "config change3 from %d to %d", oldReplicaNum, newSyncCfg.replicaNum); - if (pSyncNode->state == TAOS_SYNC_STATE_LEADER) { - syncNodeBecomeLeader(pSyncNode, tmpbuf); - } else { - syncNodeBecomeFollower(pSyncNode, tmpbuf); - } - } + // do config change + syncNodeDoConfigChange(pSyncNode, &newSyncCfg, pMsg->lastConfigIndex); } SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); - - char host[128]; - uint16_t port; - syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *logSimpleStr = logStoreSimple2Str(pSyncNode->pLogStore); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d finish, update log begin index:%ld, " - "snapshot.lastApplyIndex:%ld, " - "snapshot.lastApplyTerm:%lu, snapshot.lastConfigIndex:%ld, privateTerm:%lu, raft log:%s", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pMsg->lastIndex + 1, - snapshot.lastApplyIndex, snapshot.lastApplyTerm, snapshot.lastConfigIndex, pReceiver->privateTerm, - logSimpleStr); - taosMemoryFree(logSimpleStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d finish, update log begin index:%ld, " - "snapshot.lastApplyIndex:%ld, " - "snapshot.lastApplyTerm:%lu, snapshot.lastConfigIndex:%ld, privateTerm:%lu", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pMsg->lastIndex + 1, - snapshot.lastApplyIndex, snapshot.lastApplyTerm, snapshot.lastConfigIndex, pReceiver->privateTerm); - } + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); + + do { + char *eventLog = snapshotReceiver2SimpleStr(pReceiver, "snapshot receiver finish"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } while (0); pReceiver->pWriter = NULL; snapshotReceiverStop(pReceiver, true); pReceiver->ack = pMsg->seq; needRsp = true; - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d end ack:%d, lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu, recv msg:%s", - pReceiver->pSyncNode->vgId, pReceiver->pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, - pMsg->lastIndex, pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d end ack:%d, lastIndex:%ld, lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu", - pReceiver->pSyncNode->vgId, pReceiver->pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, - pMsg->lastIndex, pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm); - } + do { + char *eventLog = snapshotReceiver2SimpleStr(pReceiver, "snapshot receiver end"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } while (0); } else if (pMsg->seq == SYNC_SNAPSHOT_SEQ_FORCE_CLOSE) { pSyncNode->pFsm->FpSnapshotStopWrite(pSyncNode->pFsm, pReceiver->pWriter, false); snapshotReceiverStop(pReceiver, false); needRsp = false; - char host[128]; - uint16_t port; - syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d force close ack:%d, lastIndex:%ld, " - "lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu, recv " - "msg:%s", - pReceiver->pSyncNode->vgId, pReceiver->pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, - pMsg->lastIndex, pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d force close ack:%d, lastIndex:%ld, " - "lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu", - pReceiver->pSyncNode->vgId, pReceiver->pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, - pMsg->lastIndex, pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm); - } + do { + char *eventLog = snapshotReceiver2SimpleStr(pReceiver, "snapshot receiver force close"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } while (0); } else if (pMsg->seq > SYNC_SNAPSHOT_SEQ_BEGIN && pMsg->seq < SYNC_SNAPSHOT_SEQ_END) { // transfering @@ -737,27 +585,11 @@ int32_t syncNodeOnSnapshotSendCb(SSyncNode *pSyncNode, SyncSnapshotSend *pMsg) { } needRsp = true; - char host[128]; - uint16_t port; - syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port); - - if (gRaftDetailLog) { - char *msgStr = syncSnapshotSend2Str(pMsg); - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d receiving ack:%d, lastIndex:%ld, " - "lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu, recv msg:%s", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, pMsg->lastIndex, - pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm, msgStr); - taosMemoryFree(msgStr); - } else { - sDebug( - "vgId:%d sync event currentTerm:%lu snapshot recv from %s:%d receiving ack:%d, lastIndex:%ld, " - "lastTerm:%lu, " - "lastConfigIndex:%ld, privateTerm:%lu", - pSyncNode->vgId, pSyncNode->pRaftStore->currentTerm, host, port, pReceiver->ack, pMsg->lastIndex, - pMsg->lastTerm, pMsg->lastConfigIndex, pReceiver->privateTerm); - } + do { + char *eventLog = snapshotReceiver2SimpleStr(pReceiver, "snapshot receiver receiving"); + syncNodeEventLog(pSyncNode, eventLog); + taosMemoryFree(eventLog); + } while (0); } else { ASSERT(0); diff --git a/source/libs/sync/src/syncUtil.c b/source/libs/sync/src/syncUtil.c index 40265965481654736e8d8b7448c369bd1918e573..1d1ff7ae53682b3b47badc7e04084de89463b2f3 100644 --- a/source/libs/sync/src/syncUtil.c +++ b/source/libs/sync/src/syncUtil.c @@ -168,14 +168,26 @@ char* syncUtilRaftId2Str(const SRaftId* p) { } const char* syncUtilState2String(ESyncState state) { + /* + if (state == TAOS_SYNC_STATE_FOLLOWER) { + return "TAOS_SYNC_STATE_FOLLOWER"; + } else if (state == TAOS_SYNC_STATE_CANDIDATE) { + return "TAOS_SYNC_STATE_CANDIDATE"; + } else if (state == TAOS_SYNC_STATE_LEADER) { + return "TAOS_SYNC_STATE_LEADER"; + } else { + return "TAOS_SYNC_STATE_UNKNOWN"; + } + */ + if (state == TAOS_SYNC_STATE_FOLLOWER) { - return "TAOS_SYNC_STATE_FOLLOWER"; + return "follower"; } else if (state == TAOS_SYNC_STATE_CANDIDATE) { - return "TAOS_SYNC_STATE_CANDIDATE"; + return "candidate"; } else if (state == TAOS_SYNC_STATE_LEADER) { - return "TAOS_SYNC_STATE_LEADER"; + return "leader"; } else { - return "TAOS_SYNC_STATE_UNKNOWN"; + return "state_error"; } } @@ -249,23 +261,29 @@ bool syncUtilIsData(tmsg_t msgType) { #endif bool syncUtilUserPreCommit(tmsg_t msgType) { - if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_LEADER_TRANSFER) { + if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_CONFIG_CHANGE_FINISH && + msgType != TDMT_SYNC_LEADER_TRANSFER) { return true; } + return false; } bool syncUtilUserCommit(tmsg_t msgType) { - if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_LEADER_TRANSFER) { + if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_CONFIG_CHANGE_FINISH && + msgType != TDMT_SYNC_LEADER_TRANSFER) { return true; } + return false; } bool syncUtilUserRollback(tmsg_t msgType) { - if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_LEADER_TRANSFER) { + if (msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_CONFIG_CHANGE && msgType != TDMT_SYNC_CONFIG_CHANGE_FINISH && + msgType != TDMT_SYNC_LEADER_TRANSFER) { return true; } + return false; } diff --git a/source/libs/sync/test/CMakeLists.txt b/source/libs/sync/test/CMakeLists.txt index 725343e3735401a0d87e1124d1e2313f6159c993..2057aa23a4d07e492a70164017b5111b434218ff 100644 --- a/source/libs/sync/test/CMakeLists.txt +++ b/source/libs/sync/test/CMakeLists.txt @@ -49,6 +49,7 @@ add_executable(syncRaftLogTest "") add_executable(syncRaftLogTest2 "") add_executable(syncRaftLogTest3 "") add_executable(syncLeaderTransferTest "") +add_executable(syncReconfigFinishTest "") target_sources(syncTest @@ -255,6 +256,10 @@ target_sources(syncLeaderTransferTest PRIVATE "syncLeaderTransferTest.cpp" ) +target_sources(syncReconfigFinishTest + PRIVATE + "syncReconfigFinishTest.cpp" +) target_include_directories(syncTest @@ -512,6 +517,11 @@ target_include_directories(syncLeaderTransferTest "${TD_SOURCE_DIR}/include/libs/sync" "${CMAKE_CURRENT_SOURCE_DIR}/../inc" ) +target_include_directories(syncReconfigFinishTest + PUBLIC + "${TD_SOURCE_DIR}/include/libs/sync" + "${CMAKE_CURRENT_SOURCE_DIR}/../inc" +) target_link_libraries(syncTest @@ -718,6 +728,10 @@ target_link_libraries(syncLeaderTransferTest sync gtest_main ) +target_link_libraries(syncReconfigFinishTest + sync + gtest_main +) enable_testing() diff --git a/source/libs/sync/test/syncConfigChangeSnapshotTest.cpp b/source/libs/sync/test/syncConfigChangeSnapshotTest.cpp index 3ee2b458d6044fed8377d2f5a32d9e2c44351957..b4f173ff021e0d1dc7704bbef4c77e25fd24294c 100644 --- a/source/libs/sync/test/syncConfigChangeSnapshotTest.cpp +++ b/source/libs/sync/test/syncConfigChangeSnapshotTest.cpp @@ -36,9 +36,9 @@ void cleanup() { walCleanUp(); } void CommitCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SFsmCbMeta cbMeta) { SyncIndex beginIndex = SYNC_INDEX_INVALID; - if (pFsm->FpGetSnapshot != NULL) { + if (pFsm->FpGetSnapshotInfo != NULL) { SSnapshot snapshot; - pFsm->FpGetSnapshot(pFsm, &snapshot); + pFsm->FpGetSnapshotInfo(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } @@ -147,8 +147,8 @@ int32_t SnapshotDoWrite(struct SSyncFSM* pFsm, void* pWriter, void* pBuf, int32_ void RestoreFinishCb(struct SSyncFSM* pFsm) { sTrace("==callback== ==RestoreFinishCb=="); } void ReConfigCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SReConfigCbMeta cbMeta) { - sTrace("==callback== ==ReConfigCb== flag:0x%lX, isDrop:%d, index:%ld, code:%d, currentTerm:%lu, term:%lu", - cbMeta.flag, cbMeta.isDrop, cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term); + sTrace("==callback== ==ReConfigCb== flag:0x%lX, index:%ld, code:%d, currentTerm:%lu, term:%lu", cbMeta.flag, + cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term); } SSyncFSM* createFsm() { @@ -159,7 +159,7 @@ SSyncFSM* createFsm() { pFsm->FpPreCommitCb = PreCommitCb; pFsm->FpRollBackCb = RollBackCb; - pFsm->FpGetSnapshot = GetSnapshotCb; + pFsm->FpGetSnapshotInfo = GetSnapshotCb; pFsm->FpRestoreFinishCb = RestoreFinishCb; pFsm->FpSnapshotStartRead = SnapshotStartRead; pFsm->FpSnapshotStopRead = SnapshotStopRead; diff --git a/source/libs/sync/test/syncConfigChangeTest.cpp b/source/libs/sync/test/syncConfigChangeTest.cpp index b1dc53d80403e2e62707e9377a890874aceff9d4..c96e337378ca20323d37882e3bc73093e1d1cdb0 100644 --- a/source/libs/sync/test/syncConfigChangeTest.cpp +++ b/source/libs/sync/test/syncConfigChangeTest.cpp @@ -35,9 +35,9 @@ void cleanup() { walCleanUp(); } void CommitCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SFsmCbMeta cbMeta) { SyncIndex beginIndex = SYNC_INDEX_INVALID; - if (pFsm->FpGetSnapshot != NULL) { + if (pFsm->FpGetSnapshotInfo != NULL) { SSnapshot snapshot; - pFsm->FpGetSnapshot(pFsm, &snapshot); + pFsm->FpGetSnapshotInfo(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } @@ -78,8 +78,8 @@ int32_t GetSnapshotCb(struct SSyncFSM* pFsm, SSnapshot* pSnapshot) { void RestoreFinishCb(struct SSyncFSM* pFsm) { sTrace("==callback== ==RestoreFinishCb=="); } void ReConfigCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SReConfigCbMeta cbMeta) { - sTrace("==callback== ==ReConfigCb== flag:0x%lX, isDrop:%d, index:%ld, code:%d, currentTerm:%lu, term:%lu", - cbMeta.flag, cbMeta.isDrop, cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term); + sTrace("==callback== ==ReConfigCb== flag:0x%lX, index:%ld, code:%d, currentTerm:%lu, term:%lu", cbMeta.flag, + cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term); } SSyncFSM* createFsm() { @@ -90,7 +90,7 @@ SSyncFSM* createFsm() { pFsm->FpPreCommitCb = PreCommitCb; pFsm->FpRollBackCb = RollBackCb; - pFsm->FpGetSnapshot = GetSnapshotCb; + pFsm->FpGetSnapshotInfo = GetSnapshotCb; pFsm->FpRestoreFinishCb = RestoreFinishCb; pFsm->FpReConfigCb = ReConfigCb; diff --git a/source/libs/sync/test/syncRaftCfgTest.cpp b/source/libs/sync/test/syncRaftCfgTest.cpp index 8c6a704e2dcf9538058a75a086043e51f7eda080..0f111ef22c2575a38e07658eef60fe3b0656d06d 100644 --- a/source/libs/sync/test/syncRaftCfgTest.cpp +++ b/source/libs/sync/test/syncRaftCfgTest.cpp @@ -27,6 +27,14 @@ SRaftCfg* createRaftCfg() { } pCfg->isStandBy = taosGetTimestampSec() % 100; + pCfg->configIndexCount = 5; + for (int i = 0; i < MAX_CONFIG_INDEX_COUNT; ++i) { + (pCfg->configIndexArr)[i] = -1; + } + for (int i = 0; i < pCfg->configIndexCount; ++i) { + (pCfg->configIndexArr)[i] = i * 100; + } + return pCfg; } @@ -47,6 +55,8 @@ SSyncCfg* createSyncCfg() { void test1() { SSyncCfg* pCfg = createSyncCfg(); syncCfgLog2((char*)__FUNCTION__, pCfg); + syncCfgLog3((char*)__FUNCTION__, pCfg); + taosMemoryFree(pCfg); } @@ -100,6 +110,15 @@ void test5() { pCfg->isStandBy += 2; pCfg->snapshotEnable += 3; pCfg->lastConfigIndex += 1000; + + pCfg->configIndexCount = 5; + for (int i = 0; i < MAX_CONFIG_INDEX_COUNT; ++i) { + (pCfg->configIndexArr)[i] = -1; + } + for (int i = 0; i < pCfg->configIndexCount; ++i) { + (pCfg->configIndexArr)[i] = i * 100; + } + raftCfgPersist(pCfg); printf("%s update json file: %s myIndex->%d \n", (char*)__FUNCTION__, "./test3_raft_cfg.json", pCfg->cfg.myIndex); diff --git a/source/libs/sync/test/syncRaftLogTest2.cpp b/source/libs/sync/test/syncRaftLogTest2.cpp index 64e1da51a1ec33d2e632f36aa390efaefda1a960..32ff441a6f43fdaf8cce260473ebb89d635348c6 100644 --- a/source/libs/sync/test/syncRaftLogTest2.cpp +++ b/source/libs/sync/test/syncRaftLogTest2.cpp @@ -54,7 +54,7 @@ void init() { pSyncNode->pWal = pWal; pSyncNode->pFsm = (SSyncFSM*)taosMemoryMalloc(sizeof(SSyncFSM)); - pSyncNode->pFsm->FpGetSnapshot = GetSnapshotCb; + pSyncNode->pFsm->FpGetSnapshotInfo = GetSnapshotCb; } void cleanup() { diff --git a/source/libs/sync/test/syncRaftLogTest3.cpp b/source/libs/sync/test/syncRaftLogTest3.cpp index b47f8c96c512ee9f84a4a724bc94d66af87bdf67..c81a6d3ca9772f0ff3642062ab1854f1dd36342c 100644 --- a/source/libs/sync/test/syncRaftLogTest3.cpp +++ b/source/libs/sync/test/syncRaftLogTest3.cpp @@ -54,7 +54,7 @@ void init() { pSyncNode->pWal = pWal; pSyncNode->pFsm = (SSyncFSM*)taosMemoryMalloc(sizeof(SSyncFSM)); - pSyncNode->pFsm->FpGetSnapshot = GetSnapshotCb; + pSyncNode->pFsm->FpGetSnapshotInfo = GetSnapshotCb; } void cleanup() { @@ -80,7 +80,7 @@ void test1() { bool hasSnapshot = syncNodeHasSnapshot(pSyncNode); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); SyncTerm lastTerm = syncNodeGetLastTerm(pSyncNode); @@ -146,7 +146,7 @@ void test2() { bool hasSnapshot = syncNodeHasSnapshot(pSyncNode); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); SyncTerm lastTerm = syncNodeGetLastTerm(pSyncNode); @@ -203,7 +203,7 @@ void test3() { bool hasSnapshot = syncNodeHasSnapshot(pSyncNode); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); SyncTerm lastTerm = syncNodeGetLastTerm(pSyncNode); @@ -268,7 +268,7 @@ void test4() { bool hasSnapshot = syncNodeHasSnapshot(pSyncNode); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); SyncTerm lastTerm = syncNodeGetLastTerm(pSyncNode); @@ -335,7 +335,7 @@ void test5() { bool hasSnapshot = syncNodeHasSnapshot(pSyncNode); SSnapshot snapshot; - pSyncNode->pFsm->FpGetSnapshot(pSyncNode->pFsm, &snapshot); + pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot); SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode); SyncTerm lastTerm = syncNodeGetLastTerm(pSyncNode); diff --git a/source/libs/sync/test/syncReconfigFinishTest.cpp b/source/libs/sync/test/syncReconfigFinishTest.cpp new file mode 100644 index 0000000000000000000000000000000000000000..22e22bb5621d5942f78d49d320483667a13bbec1 --- /dev/null +++ b/source/libs/sync/test/syncReconfigFinishTest.cpp @@ -0,0 +1,135 @@ +#include +#include +#include "syncIO.h" +#include "syncInt.h" +#include "syncMessage.h" +#include "syncUtil.h" + +void logTest() { + sTrace("--- sync log test: trace"); + sDebug("--- sync log test: debug"); + sInfo("--- sync log test: info"); + sWarn("--- sync log test: warn"); + sError("--- sync log test: error"); + sFatal("--- sync log test: fatal"); +} + +SSyncCfg* createSyncOldCfg() { + SSyncCfg* pCfg = (SSyncCfg*)taosMemoryMalloc(sizeof(SSyncCfg)); + memset(pCfg, 0, sizeof(SSyncCfg)); + + pCfg->replicaNum = 3; + pCfg->myIndex = 1; + for (int i = 0; i < pCfg->replicaNum; ++i) { + ((pCfg->nodeInfo)[i]).nodePort = i * 100; + snprintf(((pCfg->nodeInfo)[i]).nodeFqdn, sizeof(((pCfg->nodeInfo)[i]).nodeFqdn), "100.200.300.%d", i); + } + + return pCfg; +} + +SSyncCfg* createSyncNewCfg() { + SSyncCfg* pCfg = (SSyncCfg*)taosMemoryMalloc(sizeof(SSyncCfg)); + memset(pCfg, 0, sizeof(SSyncCfg)); + + pCfg->replicaNum = 3; + pCfg->myIndex = 1; + for (int i = 0; i < pCfg->replicaNum; ++i) { + ((pCfg->nodeInfo)[i]).nodePort = i * 100; + snprintf(((pCfg->nodeInfo)[i]).nodeFqdn, sizeof(((pCfg->nodeInfo)[i]).nodeFqdn), "500.600.700.%d", i); + } + + return pCfg; +} + +SyncReconfigFinish *createMsg() { + SyncReconfigFinish *pMsg = syncReconfigFinishBuild(1234); + + SSyncCfg* pOld = createSyncOldCfg(); + SSyncCfg* pNew = createSyncNewCfg(); + pMsg->oldCfg = *pOld; + pMsg->newCfg = *pNew; + + pMsg->newCfgIndex = 11; + pMsg->newCfgTerm = 22; + pMsg->newCfgSeqNum = 33; + + taosMemoryFree(pOld); + taosMemoryFree(pNew); + + return pMsg; +} + + +void test1() { + SyncReconfigFinish *pMsg = createMsg(); + syncReconfigFinishLog2((char *)"test1:", pMsg); + syncReconfigFinishDestroy(pMsg); +} + + +void test2() { + SyncReconfigFinish *pMsg = createMsg(); + uint32_t len = pMsg->bytes; + char * serialized = (char *)taosMemoryMalloc(len); + syncReconfigFinishSerialize(pMsg, serialized, len); + SyncReconfigFinish *pMsg2 = syncReconfigFinishBuild(1000); + syncReconfigFinishDeserialize(serialized, len, pMsg2); + syncReconfigFinishLog2((char *)"test2: syncReconfigFinishSerialize -> syncReconfigFinishDeserialize ", pMsg2); + + taosMemoryFree(serialized); + syncReconfigFinishDestroy(pMsg); + syncReconfigFinishDestroy(pMsg2); +} + +void test3() { + SyncReconfigFinish *pMsg = createMsg(); + uint32_t len; + char * serialized = syncReconfigFinishSerialize2(pMsg, &len); + SyncReconfigFinish *pMsg2 = syncReconfigFinishDeserialize2(serialized, len); + syncReconfigFinishLog2((char *)"test3: SyncReconfigFinishSerialize2 -> syncReconfigFinishDeserialize2 ", pMsg2); + + taosMemoryFree(serialized); + syncReconfigFinishDestroy(pMsg); + syncReconfigFinishDestroy(pMsg2); +} + +void test4() { + SyncReconfigFinish *pMsg = createMsg(); + SRpcMsg rpcMsg; + syncReconfigFinish2RpcMsg(pMsg, &rpcMsg); + SyncReconfigFinish *pMsg2 = (SyncReconfigFinish *)taosMemoryMalloc(rpcMsg.contLen); + syncReconfigFinishFromRpcMsg(&rpcMsg, pMsg2); + syncReconfigFinishLog2((char *)"test4: syncReconfigFinish2RpcMsg -> syncReconfigFinishFromRpcMsg ", pMsg2); + + rpcFreeCont(rpcMsg.pCont); + syncReconfigFinishDestroy(pMsg); + syncReconfigFinishDestroy(pMsg2); +} + +void test5() { + SyncReconfigFinish *pMsg = createMsg(); + SRpcMsg rpcMsg; + syncReconfigFinish2RpcMsg(pMsg, &rpcMsg); + SyncReconfigFinish *pMsg2 = syncReconfigFinishFromRpcMsg2(&rpcMsg); + syncReconfigFinishLog2((char *)"test5: syncReconfigFinish2RpcMsg -> syncReconfigFinishFromRpcMsg2 ", pMsg2); + + rpcFreeCont(rpcMsg.pCont); + syncReconfigFinishDestroy(pMsg); + syncReconfigFinishDestroy(pMsg2); +} + +int main() { + gRaftDetailLog = true; + tsAsyncLog = 0; + sDebugFlag = DEBUG_TRACE + DEBUG_SCREEN + DEBUG_FILE; + logTest(); + + test1(); + test2(); + test3(); + test4(); + test5(); + + return 0; +} diff --git a/source/libs/sync/test/syncReplicateTest.cpp b/source/libs/sync/test/syncReplicateTest.cpp index d955e64f815ff9b739642c7bb06ea1fcd8eb61e8..66f347b6201ec377977445d5e55fc00e8dfaeb0e 100644 --- a/source/libs/sync/test/syncReplicateTest.cpp +++ b/source/libs/sync/test/syncReplicateTest.cpp @@ -32,9 +32,9 @@ void cleanup() { walCleanUp(); } void CommitCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SFsmCbMeta cbMeta) { SyncIndex beginIndex = SYNC_INDEX_INVALID; - if (pFsm->FpGetSnapshot != NULL) { + if (pFsm->FpGetSnapshotInfo != NULL) { SSnapshot snapshot; - pFsm->FpGetSnapshot(pFsm, &snapshot); + pFsm->FpGetSnapshotInfo(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } @@ -75,7 +75,7 @@ SSyncFSM* createFsm() { pFsm->FpCommitCb = CommitCb; pFsm->FpPreCommitCb = PreCommitCb; pFsm->FpRollBackCb = RollBackCb; - pFsm->FpGetSnapshot = GetSnapshotCb; + pFsm->FpGetSnapshotInfo = GetSnapshotCb; return pFsm; } diff --git a/source/libs/sync/test/syncSnapshotSenderTest.cpp b/source/libs/sync/test/syncSnapshotSenderTest.cpp index 404ba2acae2342eb51f581282b67dedc99cdf3ef..04a02460afca9133a41c80b419438fe400ee98e0 100644 --- a/source/libs/sync/test/syncSnapshotSenderTest.cpp +++ b/source/libs/sync/test/syncSnapshotSenderTest.cpp @@ -40,7 +40,7 @@ SSyncSnapshotSender* createSender() { pSyncNode->pFsm->FpSnapshotStartRead = SnapshotStartRead; pSyncNode->pFsm->FpSnapshotStopRead = SnapshotStopRead; pSyncNode->pFsm->FpSnapshotDoRead = SnapshotDoRead; - pSyncNode->pFsm->FpGetSnapshot = GetSnapshot; + pSyncNode->pFsm->FpGetSnapshotInfo = GetSnapshot; SSyncSnapshotSender* pSender = snapshotSenderCreate(pSyncNode, 2); pSender->start = true; diff --git a/source/libs/sync/test/syncSnapshotTest.cpp b/source/libs/sync/test/syncSnapshotTest.cpp index 820500e2d8f8b57427fec1f20741755a2ddc2d5c..954bdd8ea53687011aabc1edac653e2e8820ba27 100644 --- a/source/libs/sync/test/syncSnapshotTest.cpp +++ b/source/libs/sync/test/syncSnapshotTest.cpp @@ -35,9 +35,9 @@ const char *pWalDir = "./syncSnapshotTest_wal"; void CommitCb(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { SyncIndex beginIndex = SYNC_INDEX_INVALID; - if (pFsm->FpGetSnapshot != NULL) { + if (pFsm->FpGetSnapshotInfo != NULL) { SSnapshot snapshot; - pFsm->FpGetSnapshot(pFsm, &snapshot); + pFsm->FpGetSnapshotInfo(pFsm, &snapshot); beginIndex = snapshot.lastApplyIndex; } @@ -79,7 +79,7 @@ void initFsm() { pFsm->FpCommitCb = CommitCb; pFsm->FpPreCommitCb = PreCommitCb; pFsm->FpRollBackCb = RollBackCb; - pFsm->FpGetSnapshot = GetSnapshotCb; + pFsm->FpGetSnapshotInfo = GetSnapshotCb; } SSyncNode *syncNodeInit() { diff --git a/source/libs/sync/test/syncTestTool.cpp b/source/libs/sync/test/syncTestTool.cpp index 1eac372bd46781accf882ebeef794bea48703d8b..91a16cc0337bad490c5f19c59a4ca6f48acd3eea 100644 --- a/source/libs/sync/test/syncTestTool.cpp +++ b/source/libs/sync/test/syncTestTool.cpp @@ -148,8 +148,8 @@ void RestoreFinishCb(struct SSyncFSM* pFsm) { sTrace("==callback== ==RestoreFini void ReConfigCb(struct SSyncFSM* pFsm, const SRpcMsg* pMsg, SReConfigCbMeta cbMeta) { char* s = syncCfg2Str(&(cbMeta.newCfg)); - sTrace("==callback== ==ReConfigCb== flag:0x%lX, isDrop:%d, index:%ld, code:%d, currentTerm:%lu, term:%lu, newCfg:%s", - cbMeta.flag, cbMeta.isDrop, cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term, s); + sTrace("==callback== ==ReConfigCb== flag:0x%lX, index:%ld, code:%d, currentTerm:%lu, term:%lu, newCfg:%s", + cbMeta.flag, cbMeta.index, cbMeta.code, cbMeta.currentTerm, cbMeta.term, s); taosMemoryFree(s); } @@ -172,7 +172,7 @@ SSyncFSM* createFsm() { pFsm->FpRollBackCb = RollBackCb; pFsm->FpReConfigCb = ReConfigCb; - pFsm->FpGetSnapshot = GetSnapshotCb; + pFsm->FpGetSnapshotInfo = GetSnapshotCb; pFsm->FpRestoreFinishCb = RestoreFinishCb; pFsm->FpSnapshotStartRead = SnapshotStartRead; diff --git a/source/libs/tdb/inc/tdb.h b/source/libs/tdb/inc/tdb.h index 5912fd800c92805013cb79cb967040a8e73c8af2..628bf6d7aa594e8cb18d68ec3f36b6ce273a08ce 100644 --- a/source/libs/tdb/inc/tdb.h +++ b/source/libs/tdb/inc/tdb.h @@ -58,6 +58,7 @@ int32_t tdbTbcMoveToPrev(TBC *pTbc); int32_t tdbTbcGet(TBC *pTbc, const void **ppKey, int *pkLen, const void **ppVal, int *pvLen); int32_t tdbTbcDelete(TBC *pTbc); int32_t tdbTbcNext(TBC *pTbc, void **ppKey, int *kLen, void **ppVal, int *vLen); +int32_t tdbTbcPrev(TBC *pTbc, void **ppKey, int *kLen, void **ppVal, int *vLen); int32_t tdbTbcUpsert(TBC *pTbc, const void *pKey, int nKey, const void *pData, int nData, int insert); // TXN diff --git a/source/libs/tdb/src/db/tdbBtree.c b/source/libs/tdb/src/db/tdbBtree.c index fffda68731cb15da91669c72913f776cb67496ca..45e71f6c0dd706117549b9d0a54aa2a9be675be4 100644 --- a/source/libs/tdb/src/db/tdbBtree.c +++ b/source/libs/tdb/src/db/tdbBtree.c @@ -17,6 +17,7 @@ #define TDB_BTREE_ROOT 0x1 #define TDB_BTREE_LEAF 0x2 +#define TDB_BTREE_OVFL 0x4 struct SBTree { SPgno root; @@ -38,9 +39,11 @@ struct SBTree { #define TDB_BTREE_PAGE_SET_FLAGS(PAGE, flags) ((PAGE)->pData[0] = (flags)) #define TDB_BTREE_PAGE_IS_ROOT(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_ROOT) #define TDB_BTREE_PAGE_IS_LEAF(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_LEAF) +#define TDB_BTREE_PAGE_IS_OVFL(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_OVFL) #define TDB_BTREE_ASSERT_FLAG(flags) \ ASSERT(TDB_FLAG_IS(flags, TDB_BTREE_ROOT) || TDB_FLAG_IS(flags, TDB_BTREE_LEAF) || \ - TDB_FLAG_IS(flags, TDB_BTREE_ROOT | TDB_BTREE_LEAF) || TDB_FLAG_IS(flags, 0)) + TDB_FLAG_IS(flags, TDB_BTREE_ROOT | TDB_BTREE_LEAF) || TDB_FLAG_IS(flags, 0) || \ + TDB_FLAG_IS(flags, TDB_BTREE_OVFL)) #pragma pack(push, 1) typedef struct { @@ -62,10 +65,10 @@ static int tdbDefaultKeyCmprFn(const void *pKey1, int keyLen1, const void *pKey2 static int tdbBtreeOpenImpl(SBTree *pBt); static int tdbBtreeInitPage(SPage *pPage, void *arg, int init); static int tdbBtreeEncodeCell(SPage *pPage, const void *pKey, int kLen, const void *pVal, int vLen, SCell *pCell, - int *szCell); -static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pDecoder); + int *szCell, TXN *pTxn, SBTree *pBt); +static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pDecoder, TXN *pTxn, SBTree *pBt); static int tdbBtreeBalance(SBTC *pBtc); -static int tdbBtreeCellSize(const SPage *pPage, SCell *pCell); +static int tdbBtreeCellSize(const SPage *pPage, SCell *pCell, int dropOfp, TXN *pTxn, SBTree *pBt); static int tdbBtcMoveDownward(SBTC *pBtc); static int tdbBtcMoveUpward(SBTC *pBtc); @@ -255,7 +258,7 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL } pCell = tdbPageGetCell(btc.pPage, btc.idx); - tdbBtreeDecodeCell(btc.pPage, pCell, &cd); + tdbBtreeDecodeCell(btc.pPage, pCell, &cd, btc.pTxn, pBt); if (ppKey) { pTKey = tdbRealloc(*ppKey, cd.kLen); @@ -281,6 +284,14 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL memcpy(*ppVal, cd.pVal, cd.vLen); } + if (TDB_CELLDECODER_FREE_KEY(&cd)) { + tdbFree(cd.pKey); + } + + if (TDB_CELLDECODER_FREE_VAL(&cd)) { + tdbFree(cd.pVal); + } + tdbBtcClose(&btc); return 0; @@ -375,6 +386,11 @@ static int tdbBtreeInitPage(SPage *pPage, void *arg, int init) { pPage->vLen = pBt->valLen; pPage->maxLocal = pBt->maxLeaf; pPage->minLocal = pBt->minLeaf; + } else if (TDB_BTREE_PAGE_IS_OVFL(pPage)) { + pPage->kLen = pBt->keyLen; + pPage->vLen = pBt->valLen; + pPage->maxLocal = tdbPageCapacity(pBt->pageSize, sizeof(SIntHdr)); + pPage->minLocal = pBt->minLocal; } else { pPage->kLen = pBt->keyLen; pPage->vLen = sizeof(SPgno); @@ -499,7 +515,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx for (int i = 0; i < nOlds; i++) { if (sIdx + i < TDB_PAGE_TOTAL_CELLS(pParent)) { pCell = tdbPageGetCell(pParent, sIdx + i); - szDivCell[i] = tdbBtreeCellSize(pParent, pCell); + szDivCell[i] = tdbBtreeCellSize(pParent, pCell, 0, NULL, NULL); pDivCell[i] = tdbOsMalloc(szDivCell[i]); memcpy(pDivCell[i], pCell, szDivCell[i]); } @@ -524,7 +540,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx for (int i = 0; i < nOlds; i++) { nCells = TDB_PAGE_TOTAL_CELLS(pParent); if (sIdx < nCells) { - tdbPageDropCell(pParent, sIdx); + tdbPageDropCell(pParent, sIdx, pTxn, pBt); } else { ((SIntHdr *)pParent->pData)->pgno = 0; } @@ -582,7 +598,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx for (;;) { pCell = tdbPageGetCell(pOlds[infoNews[iNew - 1].iPage], infoNews[iNew - 1].oIdx); - szLCell = tdbBtreeCellSize(pOlds[infoNews[iNew - 1].iPage], pCell); + szLCell = tdbBtreeCellSize(pOlds[infoNews[iNew - 1].iPage], pCell, 0, NULL, NULL); if (!childNotLeaf) { szRCell = szLCell; } else { @@ -600,7 +616,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx } pCell = tdbPageGetCell(pPage, oIdx); - szRCell = tdbBtreeCellSize(pPage, pCell); + szRCell = tdbBtreeCellSize(pPage, pCell, 0, NULL, NULL); } ASSERT(infoNews[iNew - 1].cnt > 0); @@ -687,7 +703,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx for (int oIdx = 0; oIdx < TDB_PAGE_TOTAL_CELLS(pPage); oIdx++) { pCell = tdbPageGetCell(pPage, oIdx); - szCell = tdbBtreeCellSize(pPage, pCell); + szCell = tdbBtreeCellSize(pPage, pCell, 0, NULL, NULL); ASSERT(nNewCells <= infoNews[iNew].cnt); ASSERT(iNew < nNews); @@ -703,14 +719,14 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx if (iNew == nNews - 1 && pIntHdr->pgno == 0) { pIntHdr->pgno = TDB_PAGE_PGNO(pNews[iNew]); } else { - tdbBtreeDecodeCell(pPage, pCell, &cd); + tdbBtreeDecodeCell(pPage, pCell, &cd, pTxn, pBt); // TODO: pCell here may be inserted as an overflow cell, handle it SCell *pNewCell = tdbOsMalloc(cd.kLen + 9); int szNewCell; SPgno pgno; pgno = TDB_PAGE_PGNO(pNews[iNew]); - tdbBtreeEncodeCell(pParent, cd.pKey, cd.kLen, (void *)&pgno, sizeof(SPgno), pNewCell, &szNewCell); + tdbBtreeEncodeCell(pParent, cd.pKey, cd.kLen, (void *)&pgno, sizeof(SPgno), pNewCell, &szNewCell, pTxn, pBt); tdbPageInsertCell(pParent, sIdx++, pNewCell, szNewCell, 0); tdbOsFree(pNewCell); } @@ -846,13 +862,50 @@ static int tdbBtreeBalance(SBTC *pBtc) { } // TDB_BTREE_BALANCE +static int tdbFetchOvflPage(SPager *pPager, SPgno *pPgno, SPage **ppOfp, TXN *pTxn, SBTree *pBt) { + int ret = 0; + + *pPgno = 0; + SBtreeInitPageArg iArg; + iArg.pBt = pBt; + iArg.flags = TDB_FLAG_ADD(0, TDB_BTREE_OVFL); + ret = tdbPagerFetchPage(pPager, pPgno, ppOfp, tdbBtreeInitPage, &iArg, pTxn); + if (ret < 0) { + return -1; + } + + // mark dirty + ret = tdbPagerWrite(pPager, *ppOfp); + if (ret < 0) { + ASSERT(0); + return -1; + } + + return ret; +} + +static int tdbLoadOvflPage(SPager *pPager, SPgno *pPgno, SPage **ppOfp, TXN *pTxn, SBTree *pBt) { + int ret = 0; + + SBtreeInitPageArg iArg; + iArg.pBt = pBt; + iArg.flags = TDB_FLAG_ADD(0, TDB_BTREE_OVFL); + ret = tdbPagerFetchPage(pPager, pPgno, ppOfp, tdbBtreeInitPage, &iArg, pTxn); + if (ret < 0) { + return -1; + } + + return ret; +} + // TDB_BTREE_CELL ===================== static int tdbBtreeEncodePayload(SPage *pPage, SCell *pCell, int nHeader, const void *pKey, int kLen, const void *pVal, - int vLen, int *szPayload) { - int nPayload; + int vLen, int *szPayload, TXN *pTxn, SBTree *pBt) { + int ret = 0; + int nPayload = kLen + vLen; + int maxLocal = pPage->maxLocal; - nPayload = kLen + vLen; - if (nPayload + nHeader <= pPage->maxLocal) { + if (nPayload + nHeader <= maxLocal) { // no overflow page is needed memcpy(pCell + nHeader, pKey, kLen); if (pVal) { @@ -861,18 +914,190 @@ static int tdbBtreeEncodePayload(SPage *pPage, SCell *pCell, int nHeader, const *szPayload = nPayload; return 0; - } + } else { + // handle overflow case + // calc local storage size + int minLocal = pPage->minLocal; + int surplus = minLocal + (nPayload + nHeader - minLocal) % (maxLocal - sizeof(SPgno)); + int nLocal = surplus <= maxLocal ? surplus : minLocal; - { - // TODO: handle overflow case - ASSERT(0); + //int ofpCap = tdbPageCapacity(pBt->pageSize, sizeof(SIntHdr)); + + // fetch a new ofp and make it dirty + SPgno pgno = 0; + SPage *ofp, *nextOfp; + + ret = tdbFetchOvflPage(pPage->pPager, &pgno, &ofp, pTxn, pBt); + if (ret < 0) { + return -1; + } + + // local buffer for cell + SCell *pBuf = tdbRealloc(NULL, pBt->pageSize); + if (pBuf == NULL) { + return -1; + } + + int nLeft = nPayload; + int bytes; + int lastPage = 0; + if (nLocal >= kLen + 4) { + // pack key to local + memcpy(pCell + nHeader, pKey, kLen); + nLeft -= kLen; + // pack partial val to local if any space left + if (nLocal > kLen + 4) { + memcpy(pCell + nHeader + kLen, pVal, nLocal - kLen - sizeof(SPgno)); + nLeft -= nLocal - kLen - sizeof(SPgno); + } + + // pack nextPgno + memcpy(pCell + nHeader + nPayload - nLeft, &pgno, sizeof(pgno)); + + // pack left val data to ovpages + do { + lastPage = 0; + if (nLeft <= ofp->maxLocal - sizeof(SPgno)) { + bytes = nLeft; + lastPage = 1; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + // fetch next ofp if not last page + if (!lastPage) { + // fetch a new ofp and make it dirty + ret = tdbFetchOvflPage(pPage->pPager, &pgno, &nextOfp, pTxn, pBt); + if (ret < 0) { + tdbFree(pBuf); + return -1; + } + } else { + pgno = 0; + } + + memcpy(pBuf, ((SCell *)pVal) + vLen - nLeft, bytes); + memcpy(pBuf + bytes, &pgno, sizeof(pgno)); + + ret = tdbPageInsertCell(ofp, 0, pBuf, bytes + sizeof(pgno), 0); + if (ret < 0) { + tdbFree(pBuf); + return -1; + } + + ofp = nextOfp; + nLeft -= bytes; + } while (nLeft > 0); + } else { + int nLeftKey = kLen; + // pack partial key and nextPgno + memcpy(pCell + nHeader, pKey, nLocal - 4); + nLeft -= nLocal - 4; + nLeftKey -= nLocal -4; + + memcpy(pCell + nHeader + nLocal - 4, &pgno, sizeof(pgno)); + + int lastKeyPageSpace = 0; + // pack left key & val to ovpages + do { + // cal key to cpy + int lastKeyPage = 0; + if (nLeftKey <= ofp->maxLocal - sizeof(SPgno)) { + bytes = nLeftKey; + lastKeyPage = 1; + lastKeyPageSpace = ofp->maxLocal - sizeof(SPgno) - nLeftKey; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + // cpy key + memcpy(pBuf, ((SCell *)pKey) + kLen - nLeftKey, bytes); + + if (lastKeyPage) { + if (lastKeyPageSpace >= vLen) { + memcpy(pBuf + kLen -nLeftKey, pVal, vLen); + + nLeft -= vLen; + pgno = 0; + } else { + memcpy(pBuf + kLen -nLeftKey, pVal, lastKeyPageSpace); + nLeft -= lastKeyPageSpace; + + // fetch next ofp, a new ofp and make it dirty + ret = tdbFetchOvflPage(pPage->pPager, &pgno, &nextOfp, pTxn, pBt); + if (ret < 0) { + return -1; + } + } + } else { + // fetch next ofp, a new ofp and make it dirty + ret = tdbFetchOvflPage(pPage->pPager, &pgno, &nextOfp, pTxn, pBt); + if (ret < 0) { + return -1; + } + } + + memcpy(pBuf + kLen - nLeft, &pgno, sizeof(pgno)); + + ret = tdbPageInsertCell(ofp, 0, pBuf, bytes + sizeof(pgno), 0); + if (ret < 0) { + return -1; + } + + ofp = nextOfp; + nLeftKey -= bytes; + nLeft -= bytes; + } while (nLeftKey > 0); + + while (nLeft > 0) { + // pack left val data to ovpages + lastPage = 0; + if (nLeft <= maxLocal - sizeof(SPgno)) { + bytes = nLeft; + lastPage = 1; + } else { + bytes = maxLocal - sizeof(SPgno); + } + + // fetch next ofp if not last page + if (!lastPage) { + // fetch a new ofp and make it dirty + ret = tdbFetchOvflPage(pPage->pPager, &pgno, &nextOfp, pTxn, pBt); + if (ret < 0) { + tdbFree(pBuf); + return -1; + } + } else { + pgno = 0; + } + + memcpy(pBuf, ((SCell *)pVal) + vLen - nLeft, bytes); + memcpy(pBuf + bytes, &pgno, sizeof(pgno)); + + ret = tdbPageInsertCell(ofp, 0, pBuf, bytes + sizeof(pgno), 0); + if (ret < 0) { + tdbFree(pBuf); + return -1; + } + + ofp = nextOfp; + nLeft -= bytes; + } + } + + // free local buffer + tdbFree(pBuf); + + *szPayload = nLocal; + + // ASSERT(0); } return 0; } static int tdbBtreeEncodeCell(SPage *pPage, const void *pKey, int kLen, const void *pVal, int vLen, SCell *pCell, - int *szCell) { + int *szCell, TXN *pTxn, SBTree *pBt) { u8 leaf; int nHeader; int nPayload; @@ -911,7 +1136,7 @@ static int tdbBtreeEncodeCell(SPage *pPage, const void *pKey, int kLen, const vo vLen = 0; } - ret = tdbBtreeEncodePayload(pPage, pCell, nHeader, pKey, kLen, pVal, vLen, &nPayload); + ret = tdbBtreeEncodePayload(pPage, pCell, nHeader, pKey, kLen, pVal, vLen, &nPayload, pTxn, pBt); if (ret < 0) { // TODO ASSERT(0); @@ -922,8 +1147,13 @@ static int tdbBtreeEncodeCell(SPage *pPage, const void *pKey, int kLen, const vo return 0; } -static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader, SCellDecoder *pDecoder) { +static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader, SCellDecoder *pDecoder, TXN *pTxn, SBTree *pBt) { + int ret = 0; int nPayload; + int maxLocal = pPage->maxLocal; + + int kLen = pDecoder->kLen; + int vLen = pDecoder->vLen; if (pDecoder->pVal) { ASSERT(!TDB_BTREE_PAGE_IS_LEAF(pPage)); @@ -932,24 +1162,171 @@ static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader, nPayload = pDecoder->kLen + pDecoder->vLen; } - if (nHeader + nPayload <= pPage->maxLocal) { + if (nHeader + nPayload <= maxLocal) { // no over flow case - pDecoder->pKey = pCell + nHeader; + pDecoder->pKey = (SCell *)pCell + nHeader; if (pDecoder->pVal == NULL && pDecoder->vLen > 0) { - pDecoder->pVal = pCell + nHeader + pDecoder->kLen; + pDecoder->pVal = (SCell *)pCell + nHeader + pDecoder->kLen; } return 0; - } + } else { + // handle overflow case + // calc local storage size + int minLocal = pPage->minLocal; + int surplus = minLocal + (nPayload + nHeader - minLocal) % (maxLocal - sizeof(SPgno)); + int nLocal = surplus <= maxLocal ? surplus : minLocal; + + int nLeft = nPayload; + SPgno pgno = 0; + SPage *ofp; + SCell *ofpCell; + int bytes; + int lastPage = 0; + + if (nLocal >= pDecoder->kLen + 4) { + pDecoder->pKey = (SCell *)pCell + nHeader; + nLeft -= kLen; + if (nLocal > kLen + 4) { + // read partial val to local + pDecoder->pVal = tdbRealloc(pDecoder->pVal, vLen); + if (pDecoder->pVal == NULL) { + return -1; + } + TDB_CELLDECODER_SET_FREE_VAL(pDecoder); + + memcpy(pDecoder->pVal, pCell + nHeader + kLen, nLocal - kLen - sizeof(SPgno)); + + nLeft -= nLocal - kLen - sizeof(SPgno); + } - { - // TODO: handle overflow case - ASSERT(0); + memcpy(&pgno, pCell + nHeader + nPayload - nLeft, sizeof(pgno)); + + // unpack left val data from ovpages + while (pgno != 0) { + ret = tdbLoadOvflPage(pPage->pPager, &pgno, &ofp, pTxn, pBt); + if (ret < 0) { + return -1; + } + + ofpCell = tdbPageGetCell(ofp, 0); + + if (nLeft <= ofp->maxLocal - sizeof(SPgno)) { + bytes = nLeft; + lastPage = 1; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + memcpy(pDecoder->pVal + vLen - nLeft, ofpCell, bytes); + nLeft -= bytes; + + memcpy(&pgno, ofpCell + bytes, sizeof(pgno)); + } + } else { + int nLeftKey = kLen; + // load partial key and nextPgno + pDecoder->pKey = tdbRealloc(pDecoder->pKey, kLen); + if (pDecoder->pKey == NULL) { + return -1; + } + TDB_CELLDECODER_SET_FREE_KEY(pDecoder); + + memcpy(pDecoder->pKey, pCell + nHeader, nLocal - 4); + nLeft -= nLocal - 4; + nLeftKey -= nLocal -4; + + memcpy(&pgno, pCell + nHeader + nLocal - 4, sizeof(pgno)); + + int lastKeyPageSpace = 0; + // load left key & val to ovpages + while (pgno != 0) { + ret = tdbLoadOvflPage(pPage->pPager, &pgno, &ofp, pTxn, pBt); + if (ret < 0) { + return -1; + } + + ofpCell = tdbPageGetCell(ofp, 0); + + int lastKeyPage = 0; + if (nLeftKey <= maxLocal - sizeof(SPgno)) { + bytes = nLeftKey; + lastKeyPage = 1; + lastKeyPageSpace = ofp->maxLocal - sizeof(SPgno) - nLeftKey; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + // cpy key + memcpy(pDecoder->pKey + kLen - nLeftKey, ofpCell, bytes); + + if (lastKeyPage) { + if (lastKeyPageSpace >= vLen) { + pDecoder->pVal = ofpCell + kLen -nLeftKey; + + nLeft -= vLen; + pgno = 0; + } else { + // read partial val to local + pDecoder->pVal = tdbRealloc(pDecoder->pVal, vLen); + if (pDecoder->pVal == NULL) { + return -1; + } + TDB_CELLDECODER_SET_FREE_VAL(pDecoder); + + memcpy(pDecoder->pVal, ofpCell + kLen -nLeftKey, lastKeyPageSpace); + nLeft -= lastKeyPageSpace; + } + } + + memcpy(&pgno, ofpCell + bytes, sizeof(pgno)); + + nLeftKey -= bytes; + nLeft -= bytes; + } + + while (nLeft > 0) { + ret = tdbLoadOvflPage(pPage->pPager, &pgno, &ofp, pTxn, pBt); + if (ret < 0) { + return -1; + } + + ofpCell = tdbPageGetCell(ofp, 0); + + // load left val data to ovpages + lastPage = 0; + if (nLeft <= ofp->maxLocal - sizeof(SPgno)) { + bytes = nLeft; + lastPage = 1; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + if (lastPage) { + pgno = 0; + } + + if (!pDecoder->pVal) { + pDecoder->pVal = tdbRealloc(pDecoder->pVal, vLen); + if (pDecoder->pVal == NULL) { + return -1; + } + TDB_CELLDECODER_SET_FREE_VAL(pDecoder); + } + + memcpy(pDecoder->pVal, ofpCell + vLen - nLeft, bytes); + nLeft -= bytes; + + memcpy(&pgno, ofpCell + vLen - nLeft + bytes, sizeof(pgno)); + + nLeft -= bytes; + } + } } return 0; } -static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pDecoder) { +static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pDecoder, TXN *pTxn, SBTree *pBt) { u8 leaf; int nHeader; int ret; @@ -963,6 +1340,7 @@ static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pD pDecoder->vLen = -1; pDecoder->pVal = NULL; pDecoder->pgno = 0; + TDB_CELLDECODER_SET_FREE_NIL(pDecoder); // 1. Decode header part if (!leaf) { @@ -987,7 +1365,7 @@ static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pD } // 2. Decode payload part - ret = tdbBtreeDecodePayload(pPage, pCell, nHeader, pDecoder); + ret = tdbBtreeDecodePayload(pPage, pCell, nHeader, pDecoder, pTxn, pBt); if (ret < 0) { return -1; } @@ -995,41 +1373,71 @@ static int tdbBtreeDecodeCell(SPage *pPage, const SCell *pCell, SCellDecoder *pD return 0; } -static int tdbBtreeCellSize(const SPage *pPage, SCell *pCell) { +static int tdbBtreeCellSize(const SPage *pPage, SCell *pCell, int dropOfp, TXN *pTxn, SBTree *pBt) { u8 leaf; - int szCell; - int kLen = 0, vLen = 0; + int kLen = 0, vLen = 0, nHeader = 0; leaf = TDB_BTREE_PAGE_IS_LEAF(pPage); - szCell = 0; if (!leaf) { - szCell += sizeof(SPgno); + nHeader += sizeof(SPgno); } if (pPage->kLen == TDB_VARIANT_LEN) { - szCell += tdbGetVarInt(pCell + szCell, &kLen); + nHeader += tdbGetVarInt(pCell + nHeader, &kLen); } else { kLen = pPage->kLen; } if (pPage->vLen == TDB_VARIANT_LEN) { ASSERT(leaf); - szCell += tdbGetVarInt(pCell + szCell, &vLen); + nHeader += tdbGetVarInt(pCell + nHeader, &vLen); } else if (leaf) { vLen = pPage->vLen; } - szCell = szCell + kLen + vLen; + int nPayload = kLen + vLen; + if (nHeader + nPayload <= pPage->maxLocal) { + return nHeader + kLen + vLen; + } else { + int maxLocal = pPage->maxLocal; - if (szCell <= pPage->maxLocal) { - return szCell; - } + // calc local storage size + int minLocal = pPage->minLocal; + int surplus = minLocal + (nPayload + nHeader - minLocal) % (maxLocal - sizeof(SPgno)); + int nLocal = surplus <= maxLocal ? surplus : minLocal; - { - // TODO - ASSERT(0); - return 0; + // free ofp pages' cells + if (dropOfp) { + int ret = 0; + SPgno pgno = *(SPgno *) (pCell + nHeader + nLocal - sizeof(SPgno)); + int nLeft = nPayload - nLocal + sizeof(SPgno); + SPage *ofp; + int bytes; + + while (pgno != 0) { + ret = tdbLoadOvflPage(pPage->pPager, &pgno, &ofp, pTxn, pBt); + if (ret < 0) { + return -1; + } + + SCell *ofpCell = tdbPageGetCell(ofp, 0); + + if (nLeft <= ofp->maxLocal - sizeof(SPgno)) { + bytes = nLeft; + } else { + bytes = ofp->maxLocal - sizeof(SPgno); + } + + memcpy(&pgno, ofpCell + bytes, sizeof(pgno)); + + tdbPagerReturnPage(pPage->pPager, ofp, pTxn); + + nLeft -= bytes; + } + } + + return nHeader + nLocal; } } // TDB_BTREE_CELL @@ -1212,7 +1620,7 @@ int tdbBtreeNext(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen) { pCell = tdbPageGetCell(pBtc->pPage, pBtc->idx); - tdbBtreeDecodeCell(pBtc->pPage, pCell, &cd); + tdbBtreeDecodeCell(pBtc->pPage, pCell, &cd, pBtc->pTxn, pBtc->pBt); pKey = tdbRealloc(*ppKey, cd.kLen); if (pKey == NULL) { @@ -1245,6 +1653,52 @@ int tdbBtreeNext(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen) { return 0; } +int tdbBtreePrev(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen) { + SCell *pCell; + SCellDecoder cd; + void *pKey, *pVal; + int ret; + + // current cursor points to an invalid position + if (pBtc->idx < 0) { + return -1; + } + + pCell = tdbPageGetCell(pBtc->pPage, pBtc->idx); + + tdbBtreeDecodeCell(pBtc->pPage, pCell, &cd, pBtc->pTxn, pBtc->pBt); + + pKey = tdbRealloc(*ppKey, cd.kLen); + if (pKey == NULL) { + return -1; + } + + *ppKey = pKey; + *kLen = cd.kLen; + memcpy(pKey, cd.pKey, cd.kLen); + + if (ppVal) { + // TODO: vLen may be zero + pVal = tdbRealloc(*ppVal, cd.vLen); + if (pVal == NULL) { + tdbFree(pKey); + return -1; + } + + *ppVal = pVal; + *vLen = cd.vLen; + memcpy(pVal, cd.pVal, cd.vLen); + } + + ret = tdbBtcMoveToPrev(pBtc); + if (ret < 0) { + ASSERT(0); + return -1; + } + + return 0; +} + int tdbBtcMoveToNext(SBTC *pBtc) { int nCells; int ret; @@ -1381,7 +1835,7 @@ int tdbBtcGet(SBTC *pBtc, const void **ppKey, int *kLen, const void **ppVal, int } pCell = tdbPageGetCell(pBtc->pPage, pBtc->idx); - tdbBtreeDecodeCell(pBtc->pPage, pCell, &pBtc->coder); + tdbBtreeDecodeCell(pBtc->pPage, pCell, &pBtc->coder, pBtc->pTxn, pBtc->pBt); if (ppKey) { *ppKey = (void *)pBtc->coder.pKey; @@ -1418,7 +1872,7 @@ int tdbBtcDelete(SBTC *pBtc) { return -1; } - tdbPageDropCell(pBtc->pPage, idx); + tdbPageDropCell(pBtc->pPage, idx, pBtc->pTxn, pBtc->pBt); // update interior page or do balance if (idx == nCells - 1) { @@ -1442,9 +1896,9 @@ int tdbBtcDelete(SBTC *pBtc) { // update the cell with new key pCell = tdbOsMalloc(nKey + 9); - tdbBtreeEncodeCell(pPage, pKey, nKey, &pgno, sizeof(pgno), pCell, &szCell); + tdbBtreeEncodeCell(pPage, pKey, nKey, &pgno, sizeof(pgno), pCell, &szCell, pBtc->pTxn, pBtc->pBt); - ret = tdbPageUpdateCell(pPage, idx, pCell, szCell); + ret = tdbPageUpdateCell(pPage, idx, pCell, szCell, pBtc->pTxn, pBtc->pBt); if (ret < 0) { tdbOsFree(pCell); ASSERT(0); @@ -1483,7 +1937,7 @@ int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int // alloc space szBuf = kLen + nData + 14; - pBuf = tdbRealloc(pBtc->pBt->pBuf, pBtc->pBt->pageSize > szBuf ? szBuf : pBtc->pBt->pageSize); + pBuf = tdbRealloc(pBtc->pBt->pBuf, pBtc->pBt->pageSize > szBuf ? szBuf : pBtc->pBt->pageSize); if (pBuf == NULL) { ASSERT(0); return -1; @@ -1492,7 +1946,7 @@ int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int pCell = (SCell *)pBtc->pBt->pBuf; // encode cell - ret = tdbBtreeEncodeCell(pBtc->pPage, pKey, kLen, pData, nData, pCell, &szCell); + ret = tdbBtreeEncodeCell(pBtc->pPage, pKey, kLen, pData, nData, pCell, &szCell, pBtc->pTxn, pBtc->pBt); if (ret < 0) { ASSERT(0); return -1; @@ -1513,7 +1967,7 @@ int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int } else { ASSERT(pBtc->idx < nCells); - ret = tdbPageUpdateCell(pBtc->pPage, pBtc->idx, pCell, szCell); + ret = tdbPageUpdateCell(pBtc->pPage, pBtc->idx, pCell, szCell, pBtc->pTxn, pBtc->pBt); } if (ret < 0) { ASSERT(0); @@ -1574,7 +2028,7 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { // check if key <= current position if (idx < nCells) { pCell = tdbPageGetCell(pPage, idx); - tdbBtreeDecodeCell(pPage, pCell, &cd); + tdbBtreeDecodeCell(pPage, pCell, &cd, pBtc->pTxn, pBtc->pBt); c = pBt->kcmpr(pKey, kLen, cd.pKey, cd.kLen); if (c > 0) break; } @@ -1583,7 +2037,7 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { if (idx > 0) { pCell = tdbPageGetCell(pPage, idx - 1); tdbBtreeDecodeCell(pPage, pCell, &cd); - c = pBt->kcmpr(pKey, kLen, cd.pKey, cd.kLen); + c = pBt->kcmpr(pKey, kLen, cd.pKey, cd.kLen, pBtc->pTxn, pBtc->pBt); if (c <= 0) break; } } @@ -1723,4 +2177,4 @@ void tdbBtPageInfo(SPage *pPage, int idx) { pBtPageInfo->nOvfl = pPage->nOverflow; } #endif -// TDB_BTREE_DEBUG \ No newline at end of file +// TDB_BTREE_DEBUG diff --git a/source/libs/tdb/src/db/tdbPage.c b/source/libs/tdb/src/db/tdbPage.c index 78470b6256593165dc62fa4ec47f044a890198fc..7a70b621c6f65db2eb58dbc84c48ecbfe087c94a 100644 --- a/source/libs/tdb/src/db/tdbPage.c +++ b/source/libs/tdb/src/db/tdbPage.c @@ -82,7 +82,8 @@ int tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg) return 0; } -void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *)) { +void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, + TXN *, SBTree *pBt)) { pPage->pPageHdr = pPage->pData + szAmHdr; TDB_PAGE_NCELLS_SET(pPage, 0); TDB_PAGE_CCELLS_SET(pPage, pPage->pageSize - sizeof(SPageFtr)); @@ -98,7 +99,8 @@ void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell ASSERT((u8 *)pPage->pPageFtr == pPage->pFreeEnd); } -void tdbPageInit(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *)) { +void tdbPageInit(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, + TXN *, SBTree *pBt)) { pPage->pPageHdr = pPage->pData + szAmHdr; pPage->pCellIdx = pPage->pPageHdr + TDB_PAGE_HDR_SIZE(pPage); pPage->pFreeStart = pPage->pCellIdx + TDB_PAGE_OFFSET_SIZE(pPage) * TDB_PAGE_NCELLS(pPage); @@ -171,12 +173,12 @@ int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl return 0; } -int tdbPageUpdateCell(SPage *pPage, int idx, SCell *pCell, int szCell) { - tdbPageDropCell(pPage, idx); +int tdbPageUpdateCell(SPage *pPage, int idx, SCell *pCell, int szCell, TXN *pTxn, SBTree *pBt) { + tdbPageDropCell(pPage, idx, pTxn, pBt); return tdbPageInsertCell(pPage, idx, pCell, szCell, 0); } -int tdbPageDropCell(SPage *pPage, int idx) { +int tdbPageDropCell(SPage *pPage, int idx, TXN *pTxn, SBTree *pBt) { int lidx; SCell *pCell; int szCell; @@ -205,7 +207,7 @@ int tdbPageDropCell(SPage *pPage, int idx) { lidx = idx - iOvfl; pCell = TDB_PAGE_CELL_AT(pPage, lidx); - szCell = (*pPage->xCellSize)(pPage, pCell); + szCell = (*pPage->xCellSize)(pPage, pCell, 1, pTxn, pBt); tdbPageFree(pPage, lidx, pCell, szCell); TDB_PAGE_NCELLS_SET(pPage, nCells - 1); @@ -420,7 +422,7 @@ static int tdbPageDefragment(SPage *pPage) { ASSERT(pCell != NULL); - szCell = (*pPage->xCellSize)(pPage, pCell); + szCell = (*pPage->xCellSize)(pPage, pCell, 0, NULL, NULL); ASSERT(pCell + szCell <= pNextCell); if (pCell + szCell < pNextCell) { diff --git a/source/libs/tdb/src/db/tdbTable.c b/source/libs/tdb/src/db/tdbTable.c index 239aa5d7ef786b0941e857bf9e3f73a655f65d5a..1575f9f206d3dd6b77eec6e61a5840e47efedc4b 100644 --- a/source/libs/tdb/src/db/tdbTable.c +++ b/source/libs/tdb/src/db/tdbTable.c @@ -132,6 +132,10 @@ int tdbTbcNext(TBC *pTbc, void **ppKey, int *kLen, void **ppVal, int *vLen) { return tdbBtreeNext(&pTbc->btc, ppKey, kLen, ppVal, vLen); } +int tdbTbcPrev(TBC *pTbc, void **ppKey, int *kLen, void **ppVal, int *vLen) { + return tdbBtreePrev(&pTbc->btc, ppKey, kLen, ppVal, vLen); +} + int tdbTbcUpsert(TBC *pTbc, const void *pKey, int nKey, const void *pData, int nData, int insert) { return tdbBtcUpsert(&pTbc->btc, pKey, nKey, pData, nData, insert); } diff --git a/source/libs/tdb/src/inc/tdbInt.h b/source/libs/tdb/src/inc/tdbInt.h index 6524e3c9bcd873180378b5cfea2404b1a461ac7b..39b5584fab86f1b2ce1c15478c79eb71f924562e 100644 --- a/source/libs/tdb/src/inc/tdbInt.h +++ b/source/libs/tdb/src/inc/tdbInt.h @@ -116,13 +116,25 @@ typedef struct SBtInfo { int nData; } SBtInfo; +#define TDB_CELLD_F_NIL 0x0 +#define TDB_CELLD_F_KEY 0x1 +#define TDB_CELLD_F_VAL 0x2 + +#define TDB_CELLDECODER_SET_FREE_NIL(pCellDecoder) ((pCellDecoder)->freeKV = TDB_CELLD_F_NIL) +#define TDB_CELLDECODER_SET_FREE_KEY(pCellDecoder) ((pCellDecoder)->freeKV |= TDB_CELLD_F_KEY) +#define TDB_CELLDECODER_SET_FREE_VAL(pCellDecoder) ((pCellDecoder)->freeKV |= TDB_CELLD_F_VAL) + +#define TDB_CELLDECODER_FREE_KEY(pCellDecoder) ((pCellDecoder)->freeKV & TDB_CELLD_F_KEY) +#define TDB_CELLDECODER_FREE_VAL(pCellDecoder) ((pCellDecoder)->freeKV & TDB_CELLD_F_VAL) + typedef struct { int kLen; - const u8 *pKey; + u8 *pKey; int vLen; - const u8 *pVal; + u8 *pVal; SPgno pgno; u8 *pBuf; + u8 freeKV; } SCellDecoder; struct SBTC { @@ -156,6 +168,7 @@ int tdbBtcMoveToLast(SBTC *pBtc); int tdbBtcMoveToNext(SBTC *pBtc); int tdbBtcMoveToPrev(SBTC *pBtc); int tdbBtreeNext(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); +int tdbBtreePrev(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); int tdbBtcGet(SBTC *pBtc, const void **ppKey, int *kLen, const void **ppVal, int *vLen); int tdbBtcDelete(SBTC *pBtc); int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int nData, int insert); @@ -250,7 +263,7 @@ struct SPage { int vLen; // value length of the page, -1 for unknown int maxLocal; int minLocal; - int (*xCellSize)(const SPage *, SCell *); + int (*xCellSize)(const SPage *, SCell *, int, TXN *pTxn, SBTree *pBt); // Fields used by SPCache TDB_PCACHE_PAGE }; @@ -297,16 +310,18 @@ static inline int tdbTryLockPage(tdb_spinlock_t *pLock) { #define TDB_PAGE_USABLE_SIZE(pPage) ((u8 *)(pPage)->pPageFtr - (pPage)->pCellIdx) #define TDB_PAGE_FREE_SIZE(pPage) (*(pPage)->pPageMethods->getFreeBytes)(pPage) #define TDB_PAGE_PGNO(pPage) ((pPage)->pgid.pgno) -#define TDB_BYTES_CELL_TAKEN(pPage, pCell) ((*(pPage)->xCellSize)(pPage, pCell) + (pPage)->pPageMethods->szOffset) +#define TDB_BYTES_CELL_TAKEN(pPage, pCell) ((*(pPage)->xCellSize)(pPage, pCell, 0, NULL, NULL) + (pPage)->pPageMethods->szOffset) #define TDB_PAGE_OFFSET_SIZE(pPage) ((pPage)->pPageMethods->szOffset) int tdbPageCreate(int pageSize, SPage **ppPage, void *(*xMalloc)(void *, size_t), void *arg); int tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg); -void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *)); -void tdbPageInit(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *)); +void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, + TXN *, SBTree *pBt)); +void tdbPageInit(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, + TXN *, SBTree *pBt)); int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl); -int tdbPageDropCell(SPage *pPage, int idx); -int tdbPageUpdateCell(SPage *pPage, int idx, SCell *pCell, int szCell); +int tdbPageDropCell(SPage *pPage, int idx, TXN *pTxn, SBTree *pBt); +int tdbPageUpdateCell(SPage *pPage, int idx, SCell *pCell, int szCell, TXN *pTxn, SBTree *pBt); void tdbPageCopy(SPage *pFromPage, SPage *pToPage); int tdbPageCapacity(int pageSize, int amHdrSize); diff --git a/source/libs/tdb/test/CMakeLists.txt b/source/libs/tdb/test/CMakeLists.txt index b2c8aaf9bc9a8fa7cff472623d894c42340f0a0a..2621e02b02b9a9a28ec29c58b657202f8522a894 100644 --- a/source/libs/tdb/test/CMakeLists.txt +++ b/source/libs/tdb/test/CMakeLists.txt @@ -4,4 +4,9 @@ target_link_libraries(tdbTest tdb gtest gtest_main) # tdbUtilTest add_executable(tdbUtilTest "tdbUtilTest.cpp") -target_link_libraries(tdbUtilTest tdb gtest gtest_main) \ No newline at end of file +target_link_libraries(tdbUtilTest tdb gtest gtest_main) + +# tdbUtilTest +add_executable(tdbExOVFLTest "tdbExOVFLTest.cpp") +target_link_libraries(tdbExOVFLTest tdb gtest gtest_main) + diff --git a/source/libs/tdb/test/tdbExOVFLTest.cpp b/source/libs/tdb/test/tdbExOVFLTest.cpp new file mode 100644 index 0000000000000000000000000000000000000000..2d8d012a6ab831c47bba7024fe7d179f89e97167 --- /dev/null +++ b/source/libs/tdb/test/tdbExOVFLTest.cpp @@ -0,0 +1,469 @@ +#include + +#define ALLOW_FORBID_FUNC +#include "os.h" +#include "tdb.h" + +#include +#include +#include +#include + +typedef struct SPoolMem { + int64_t size; + struct SPoolMem *prev; + struct SPoolMem *next; +} SPoolMem; + +static SPoolMem *openPool() { + SPoolMem *pPool = (SPoolMem *)taosMemoryMalloc(sizeof(*pPool)); + + pPool->prev = pPool->next = pPool; + pPool->size = 0; + + return pPool; +} + +static void clearPool(SPoolMem *pPool) { + SPoolMem *pMem; + + do { + pMem = pPool->next; + + if (pMem == pPool) break; + + pMem->next->prev = pMem->prev; + pMem->prev->next = pMem->next; + pPool->size -= pMem->size; + + taosMemoryFree(pMem); + } while (1); + + assert(pPool->size == 0); +} + +static void closePool(SPoolMem *pPool) { + clearPool(pPool); + taosMemoryFree(pPool); +} + +static void *poolMalloc(void *arg, size_t size) { + void *ptr = NULL; + SPoolMem *pPool = (SPoolMem *)arg; + SPoolMem *pMem; + + pMem = (SPoolMem *)taosMemoryMalloc(sizeof(*pMem) + size); + if (pMem == NULL) { + assert(0); + } + + pMem->size = sizeof(*pMem) + size; + pMem->next = pPool->next; + pMem->prev = pPool; + + pPool->next->prev = pMem; + pPool->next = pMem; + pPool->size += pMem->size; + + ptr = (void *)(&pMem[1]); + return ptr; +} + +static void poolFree(void *arg, void *ptr) { + SPoolMem *pPool = (SPoolMem *)arg; + SPoolMem *pMem; + + pMem = &(((SPoolMem *)ptr)[-1]); + + pMem->next->prev = pMem->prev; + pMem->prev->next = pMem->next; + pPool->size -= pMem->size; + + taosMemoryFree(pMem); +} + +static int tKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kLen2) { + int k1, k2; + + std::string s1((char *)pKey1 + 3, kLen1 - 3); + std::string s2((char *)pKey2 + 3, kLen2 - 3); + k1 = stoi(s1); + k2 = stoi(s2); + + if (k1 < k2) { + return -1; + } else if (k1 > k2) { + return 1; + } else { + return 0; + } +} + +static int tDefaultKeyCmpr(const void *pKey1, int keyLen1, const void *pKey2, int keyLen2) { + int mlen; + int cret; + + ASSERT(keyLen1 > 0 && keyLen2 > 0 && pKey1 != NULL && pKey2 != NULL); + + mlen = keyLen1 < keyLen2 ? keyLen1 : keyLen2; + cret = memcmp(pKey1, pKey2, mlen); + if (cret == 0) { + if (keyLen1 < keyLen2) { + cret = -1; + } else if (keyLen1 > keyLen2) { + cret = 1; + } else { + cret = 0; + } + } + return cret; +} + +TEST(TdbOVFLPagesTest, TbUpsertTest) { + +} + +TEST(TdbOVFLPagesTest, TbPGetTest) { + +} + +static void generateBigVal(char *val, int valLen) { + for (int i = 0; i < valLen; ++i) { + char c = char(i & 0xff); + if (c == 0) { + c = 1; + } + val[i] = c; + } +} + +static TDB *openEnv(char const *envName, int const pageSize, int const pageNum) { + TDB *pEnv = NULL; + + int ret = tdbOpen(envName, pageSize, pageNum, &pEnv); + if (ret) { + pEnv = NULL; + } + + return pEnv; +} + +static void insertOfp(void) { + int ret = 0; + + taosRemoveDir("tdb"); + + // open Env + int const pageSize = 4096; + int const pageNum = 64; + TDB *pEnv = openEnv("tdb", pageSize, pageNum); + GTEST_ASSERT_NE(pEnv, nullptr); + + // open db + TTB *pDb = NULL; + tdb_cmpr_fn_t compFunc = tKeyCmpr; + ret = tdbTbOpen("ofp_insert.db", -1, -1, compFunc, pEnv, &pDb); + GTEST_ASSERT_EQ(ret, 0); + + // open the pool + SPoolMem *pPool = openPool(); + + // start a transaction + TXN txn; + int64_t txnid = 0; + ++txnid; + tdbTxnOpen(&txn, txnid, poolMalloc, poolFree, pPool, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED); + tdbBegin(pEnv, &txn); + + // generate value payload + char val[((4083 - 4 - 3 - 2)+1)*100]; // pSize(4096) - amSize(1) - pageHdr(8) - footerSize(4) + int valLen = sizeof(val) / sizeof(val[0]); + generateBigVal(val, valLen); + + // insert the generated big data + ret = tdbTbInsert(pDb, "key1", strlen("key1"), val, valLen, &txn); + GTEST_ASSERT_EQ(ret, 0); + + // commit current transaction + tdbCommit(pEnv, &txn); + tdbTxnClose(&txn); +} + +//TEST(TdbOVFLPagesTest, DISABLED_TbInsertTest) { +TEST(TdbOVFLPagesTest, TbInsertTest) { + insertOfp(); +} + +//TEST(TdbOVFLPagesTest, DISABLED_TbGetTest) { +TEST(TdbOVFLPagesTest, TbGetTest) { + insertOfp(); + + // open Env + int const pageSize = 4096; + int const pageNum = 64; + TDB *pEnv = openEnv("tdb", pageSize, pageNum); + GTEST_ASSERT_NE(pEnv, nullptr); + + // open db + TTB *pDb = NULL; + tdb_cmpr_fn_t compFunc = tKeyCmpr; + int ret = tdbTbOpen("ofp_insert.db", -1, -1, compFunc, pEnv, &pDb); + GTEST_ASSERT_EQ(ret, 0); + + // generate value payload + char val[((4083 - 4 - 3 - 2)+1)*100]; // pSize(4096) - amSize(1) - pageHdr(8) - footerSize(4) + int valLen = sizeof(val) / sizeof(val[0]); + generateBigVal(val, valLen); + + { // Query the data + void *pVal = NULL; + int vLen; + + ret = tdbTbGet(pDb, "key1", strlen("key1"), &pVal, &vLen); + ASSERT(ret == 0); + GTEST_ASSERT_EQ(ret, 0); + + GTEST_ASSERT_EQ(vLen, valLen); + GTEST_ASSERT_EQ(memcmp(val, pVal, vLen), 0); + + tdbFree(pVal); + } +} + +TEST(TdbOVFLPagesTest, TbDeleteTest) { + int ret = 0; + + taosRemoveDir("tdb"); + + // open Env + int const pageSize = 4096; + int const pageNum = 64; + TDB *pEnv = openEnv("tdb", pageSize, pageNum); + GTEST_ASSERT_NE(pEnv, nullptr); + + // open db + TTB *pDb = NULL; + tdb_cmpr_fn_t compFunc = tKeyCmpr; + ret = tdbTbOpen("ofp_insert.db", -1, -1, compFunc, pEnv, &pDb); + GTEST_ASSERT_EQ(ret, 0); + + // open the pool + SPoolMem *pPool = openPool(); + + // start a transaction + TXN txn; + int64_t txnid = 0; + ++txnid; + tdbTxnOpen(&txn, txnid, poolMalloc, poolFree, pPool, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED); + tdbBegin(pEnv, &txn); + + // generate value payload + char val[((4083 - 4 - 3 - 2)+1)*100]; // pSize(4096) - amSize(1) - pageHdr(8) - footerSize(4) + int valLen = sizeof(val) / sizeof(val[0]); + generateBigVal(val, valLen); + + { // insert the generated big data + ret = tdbTbInsert(pDb, "key1", strlen("key1"), val, valLen, &txn); + GTEST_ASSERT_EQ(ret, 0); + } + + { // query the data + void *pVal = NULL; + int vLen; + + ret = tdbTbGet(pDb, "key1", strlen("key1"), &pVal, &vLen); + ASSERT(ret == 0); + GTEST_ASSERT_EQ(ret, 0); + + GTEST_ASSERT_EQ(vLen, valLen); + GTEST_ASSERT_EQ(memcmp(val, pVal, vLen), 0); + + tdbFree(pVal); + } + /* open to debug committed file + tdbCommit(pEnv, &txn); + tdbTxnClose(&txn); + + ++txnid; + tdbTxnOpen(&txn, txnid, poolMalloc, poolFree, pPool, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED); + tdbBegin(pEnv, &txn); + */ + { // upsert the data + ret = tdbTbUpsert(pDb, "key1", strlen("key1"), "value1", strlen("value1"), &txn); + GTEST_ASSERT_EQ(ret, 0); + } + + { // query the upserted data + void *pVal = NULL; + int vLen; + + ret = tdbTbGet(pDb, "key1", strlen("key1"), &pVal, &vLen); + ASSERT(ret == 0); + GTEST_ASSERT_EQ(ret, 0); + + GTEST_ASSERT_EQ(vLen, strlen("value1")); + GTEST_ASSERT_EQ(memcmp("value1", pVal, vLen), 0); + + tdbFree(pVal); + } + + { // delete the data + ret = tdbTbDelete(pDb, "key1", strlen("key1"), &txn); + GTEST_ASSERT_EQ(ret, 0); + } + + { // query the deleted data + void *pVal = NULL; + int vLen = -1; + + ret = tdbTbGet(pDb, "key1", strlen("key1"), &pVal, &vLen); + ASSERT(ret == -1); + GTEST_ASSERT_EQ(ret, -1); + + GTEST_ASSERT_EQ(vLen, -1); + GTEST_ASSERT_EQ(pVal, nullptr); + + tdbFree(pVal); + } + + // commit current transaction + tdbCommit(pEnv, &txn); + tdbTxnClose(&txn); +} + +TEST(tdb_test, DISABLED_simple_insert1) { +//TEST(tdb_test, simple_insert1) { + int ret; + TDB *pEnv; + TTB *pDb; + tdb_cmpr_fn_t compFunc; + int nData = 1; + TXN txn; + int const pageSize = 4096; + + taosRemoveDir("tdb"); + + // Open Env + ret = tdbOpen("tdb", pageSize, 64, &pEnv); + GTEST_ASSERT_EQ(ret, 0); + + // Create a database + compFunc = tKeyCmpr; + ret = tdbTbOpen("db.db", -1, -1, compFunc, pEnv, &pDb); + GTEST_ASSERT_EQ(ret, 0); + + { + char key[64]; + //char val[(4083 - 4 - 3 - 2)]; // pSize(4096) - amSize(1) - pageHdr(8) - footerSize(4) + char val[(4083 - 4 - 3 - 2)+1]; // pSize(4096) - amSize(1) - pageHdr(8) - footerSize(4) + int64_t poolLimit = 4096; // 1M pool limit + int64_t txnid = 0; + SPoolMem *pPool; + + // open the pool + pPool = openPool(); + + // start a transaction + txnid++; + tdbTxnOpen(&txn, txnid, poolMalloc, poolFree, pPool, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED); + tdbBegin(pEnv, &txn); + + for (int iData = 1; iData <= nData; iData++) { + sprintf(key, "key0"); + sprintf(val, "value%d", iData); + + //ret = tdbTbInsert(pDb, key, strlen(key), val, strlen(val), &txn); + //GTEST_ASSERT_EQ(ret, 0); + + // generate value payload + int valLen = sizeof(val) / sizeof(val[0]); + for (int i = 6; i < valLen; ++i) { + char c = char(i & 0xff); + if (c == 0) { + c = 1; + } + val[i] = c; + } + + ret = tdbTbInsert(pDb, "key1", strlen("key1"), val, valLen, &txn); + GTEST_ASSERT_EQ(ret, 0); + + // if pool is full, commit the transaction and start a new one + if (pPool->size >= poolLimit) { + // commit current transaction + tdbCommit(pEnv, &txn); + tdbTxnClose(&txn); + + // start a new transaction + clearPool(pPool); + txnid++; + tdbTxnOpen(&txn, txnid, poolMalloc, poolFree, pPool, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED); + tdbBegin(pEnv, &txn); + } + } + + // commit the transaction + tdbCommit(pEnv, &txn); + tdbTxnClose(&txn); + + { // Query the data + void *pVal = NULL; + int vLen; + + for (int i = 1; i <= nData; i++) { + sprintf(key, "key%d", i); + sprintf(val, "value%d", i); + + ret = tdbTbGet(pDb, key, strlen(key), &pVal, &vLen); + ASSERT(ret == 0); + GTEST_ASSERT_EQ(ret, 0); + + GTEST_ASSERT_EQ(vLen, strlen(val)); + GTEST_ASSERT_EQ(memcmp(val, pVal, vLen), 0); + } + + tdbFree(pVal); + } + + { // Iterate to query the DB data + TBC *pDBC; + void *pKey = NULL; + void *pVal = NULL; + int vLen, kLen; + int count = 0; + + ret = tdbTbcOpen(pDb, &pDBC, NULL); + GTEST_ASSERT_EQ(ret, 0); + + tdbTbcMoveToFirst(pDBC); + + for (;;) { + ret = tdbTbcNext(pDBC, &pKey, &kLen, &pVal, &vLen); + if (ret < 0) break; + + // std::cout.write((char *)pKey, kLen) /* << " " << kLen */ << " "; + // std::cout.write((char *)pVal, vLen) /* << " " << vLen */; + // std::cout << std::endl; + + count++; + } + + GTEST_ASSERT_EQ(count, nData); + + tdbTbcClose(pDBC); + + tdbFree(pKey); + tdbFree(pVal); + } + } + + ret = tdbTbDrop(pDb); + GTEST_ASSERT_EQ(ret, 0); + + // Close a database + tdbTbClose(pDb); + + // Close Env + ret = tdbClose(pEnv); + GTEST_ASSERT_EQ(ret, 0); +} diff --git a/source/libs/tfs/src/tfs.c b/source/libs/tfs/src/tfs.c index 92beeffa0cb7246986bb10ab508fdc4d4688a562..2d3d41de64b7939c0a3f543811adf75715b59811 100644 --- a/source/libs/tfs/src/tfs.c +++ b/source/libs/tfs/src/tfs.c @@ -282,6 +282,8 @@ int32_t tfsMkdir(STfs *pTfs, const char *rname) { } int32_t tfsRmdir(STfs *pTfs, const char *rname) { + ASSERT(rname[0] != 0); + char aname[TMPNAME_LEN] = "\0"; for (int32_t level = 0; level < pTfs->nlevel; level++) { @@ -289,6 +291,7 @@ int32_t tfsRmdir(STfs *pTfs, const char *rname) { for (int32_t id = 0; id < pTier->ndisk; id++) { STfsDisk *pDisk = pTier->disks[id]; snprintf(aname, TMPNAME_LEN, "%s%s%s", pDisk->path, TD_DIRSEP, rname); + uInfo("====> tfs remove dir : path:%s aname:%s rname:[%s]", pDisk->path, aname, rname); taosRemoveDir(aname); } } diff --git a/source/libs/transport/inc/transComm.h b/source/libs/transport/inc/transComm.h index e680e3004283684b0d95c1fd0124f33e99d59d3b..146b127422345e2989a2c644e60aa2065b7ac8c5 100644 --- a/source/libs/transport/inc/transComm.h +++ b/source/libs/transport/inc/transComm.h @@ -26,6 +26,7 @@ extern "C" { #include "transLog.h" #include "transportInt.h" #include "trpc.h" +#include "ttrace.h" #include "tutil.h" typedef void* queue[2]; @@ -140,6 +141,7 @@ typedef struct { char spi : 2; char user[TSDB_UNI_LEN]; + STraceId traceId; uint64_t ahandle; // ahandle assigned by client uint32_t code; // del later uint32_t msgType; diff --git a/source/libs/transport/inc/transLog.h b/source/libs/transport/inc/transLog.h index ebf231cfcff11aea08158264da5b71e77a429692..9947ba803f4d22e3b2264338c487990c584c8eed 100644 --- a/source/libs/transport/inc/transLog.h +++ b/source/libs/transport/inc/transLog.h @@ -22,6 +22,7 @@ extern "C" { // clang-format off #include "tlog.h" +#include "ttrace.h" #define tFatal(...) do {if (rpcDebugFlag & DEBUG_FATAL){ taosPrintLog("RPC FATAL ", DEBUG_FATAL, rpcDebugFlag, __VA_ARGS__); }} while (0) #define tError(...)do { if (rpcDebugFlag & DEBUG_ERROR){ taosPrintLog("RPC ERROR ", DEBUG_ERROR, rpcDebugFlag, __VA_ARGS__); } } while(0) @@ -30,6 +31,10 @@ extern "C" { #define tDebug(...) do {if (rpcDebugFlag & DEBUG_DEBUG){ taosPrintLog("RPC ", DEBUG_DEBUG, rpcDebugFlag, __VA_ARGS__); }} while(0) #define tTrace(...) do {if (rpcDebugFlag & DEBUG_TRACE){ taosPrintLog("RPC ", DEBUG_TRACE, rpcDebugFlag, __VA_ARGS__); }} while(0) #define tDump(x, y) do {if (rpcDebugFlag & DEBUG_DUMP) { taosDumpData((unsigned char *)x, y); } } while(0) + +//#define tTR(param, ...) do { char buf[40] = {0};TRACE_TO_STR(trace, buf);tTrace("TRID: %s "param, buf, __VA_ARGS__);} while(0) +#define tGTrace(param, ...) do { char buf[40] = {0}; TRACE_TO_STR(trace, buf); tTrace(param ", GTID: %s", __VA_ARGS__, buf);} while(0) + // clang-format on #ifdef __cplusplus } diff --git a/source/libs/transport/src/trans.c b/source/libs/transport/src/trans.c index 925de2f3219672e40f270b92b754718a93f23f02..1ec96f4a7ae2710f11db44988f002569ac39e3bf 100644 --- a/source/libs/transport/src/trans.c +++ b/source/libs/transport/src/trans.c @@ -163,6 +163,11 @@ void rpcSetDefaultAddr(void* thandle, const char* ip, const char* fqdn) { transSetDefaultAddr(thandle, ip, fqdn); } +// void rpcSetMsgTraceId(SRpcMsg* pMsg, STraceId uid) { +// SRpcHandleInfo* pInfo = &pMsg->info; +// pInfo->traceId = uid; +//} + int32_t rpcInit() { // impl later return 0; diff --git a/source/libs/transport/src/transCli.c b/source/libs/transport/src/transCli.c index 3d570e08608b5908dc97541c741d8430d5480725..912c8b59c211413e4ee49ac5074585d734457278 100644 --- a/source/libs/transport/src/transCli.c +++ b/source/libs/transport/src/transCli.c @@ -48,9 +48,10 @@ typedef struct SCliMsg { STransConnCtx* ctx; STransMsg msg; queue q; - uint64_t st; STransMsgType type; - int sent; //(0: no send, 1: alread sent) + + uint64_t st; + int sent; //(0: no send, 1: alread sent) } SCliMsg; typedef struct SCliThrdObj { @@ -167,27 +168,27 @@ static void cliReleaseUnfinishedMsg(SCliConn* conn) { snprintf(key, sizeof(key), "%s:%d", ip, (int)port); \ } while (0) -#define CONN_HOST_THREAD_INDEX(conn) (conn ? ((SCliConn*)conn)->hThrdIdx : -1) -#define CONN_PERSIST_TIME(para) (para * 1000 * 10) -#define CONN_GET_HOST_THREAD(conn) (conn ? ((SCliConn*)conn)->hostThrd : NULL) -#define CONN_GET_INST_LABEL(conn) (((STrans*)(((SCliThrdObj*)(conn)->hostThrd)->pTransInst))->label) -#define CONN_SHOULD_RELEASE(conn, head) \ - do { \ - if ((head)->release == 1 && (head->msgLen) == sizeof(*head)) { \ - uint64_t ahandle = head->ahandle; \ - CONN_GET_MSGCTX_BY_AHANDLE(conn, ahandle); \ - conn->status = ConnRelease; \ - transClearBuffer(&conn->readBuf); \ - transFreeMsg(transContFromHead((char*)head)); \ - tDebug("cli conn %p receive release request, ref: %d", conn, T_REF_VAL_GET(conn)); \ - if (T_REF_VAL_GET(conn) > 1) { \ - transUnrefCliHandle(conn); \ - } \ - destroyCmsg(pMsg); \ - cliReleaseUnfinishedMsg(conn); \ - addConnToPool(((SCliThrdObj*)conn->hostThrd)->pool, conn); \ - return; \ - } \ +#define CONN_HOST_THREAD_IDX(conn) (conn ? ((SCliConn*)conn)->hThrdIdx : -1) +#define CONN_PERSIST_TIME(para) (para * 1000 * 10) +#define CONN_GET_HOST_THREAD(conn) (conn ? ((SCliConn*)conn)->hostThrd : NULL) +#define CONN_GET_INST_LABEL(conn) (((STrans*)(((SCliThrdObj*)(conn)->hostThrd)->pTransInst))->label) +#define CONN_SHOULD_RELEASE(conn, head) \ + do { \ + if ((head)->release == 1 && (head->msgLen) == sizeof(*head)) { \ + uint64_t ahandle = head->ahandle; \ + CONN_GET_MSGCTX_BY_AHANDLE(conn, ahandle); \ + conn->status = ConnRelease; \ + transClearBuffer(&conn->readBuf); \ + transFreeMsg(transContFromHead((char*)head)); \ + tDebug("%s conn %p receive release request, ref: %d", CONN_GET_INST_LABEL(conn), conn, T_REF_VAL_GET(conn)); \ + if (T_REF_VAL_GET(conn) > 1) { \ + transUnrefCliHandle(conn); \ + } \ + destroyCmsg(pMsg); \ + cliReleaseUnfinishedMsg(conn); \ + addConnToPool(((SCliThrdObj*)conn->hostThrd)->pool, conn); \ + return; \ + } \ } while (0) #define CONN_GET_MSGCTX_BY_AHANDLE(conn, ahandle) \ @@ -280,6 +281,7 @@ void cliHandleResp(SCliConn* conn) { transMsg.code = pHead->code; transMsg.msgType = pHead->msgType; transMsg.info.ahandle = NULL; + transMsg.info.traceId = pHead->traceId; SCliMsg* pMsg = NULL; STransConnCtx* pCtx = NULL; @@ -293,27 +295,28 @@ void cliHandleResp(SCliConn* conn) { if (transMsg.info.ahandle == NULL) { transMsg.info.ahandle = transCtxDumpBrokenlinkVal(&conn->ctx, (int32_t*)&(transMsg.msgType)); } - tDebug("cli conn %p construct ahandle %p, persist: 0", conn, transMsg.info.ahandle); + tDebug("%s conn %p construct ahandle %p, persist: 0", CONN_GET_INST_LABEL(conn), conn, transMsg.info.ahandle); } else { transMsg.info.ahandle = pCtx ? pCtx->ahandle : NULL; - tDebug("cli conn %p get ahandle %p, persist: 0", conn, transMsg.info.ahandle); + tDebug("%s conn %p get ahandle %p, persist: 0", CONN_GET_INST_LABEL(conn), conn, transMsg.info.ahandle); } } else { uint64_t ahandle = (uint64_t)pHead->ahandle; CONN_GET_MSGCTX_BY_AHANDLE(conn, ahandle); if (pMsg == NULL) { transMsg.info.ahandle = transCtxDumpVal(&conn->ctx, transMsg.msgType); - tDebug("cli conn %p construct ahandle %p by %s, persist: 1", conn, transMsg.info.ahandle, - TMSG_INFO(transMsg.msgType)); + tDebug("%s conn %p construct ahandle %p by %s, persist: 1", CONN_GET_INST_LABEL(conn), conn, + transMsg.info.ahandle, TMSG_INFO(transMsg.msgType)); if (!CONN_RELEASE_BY_SERVER(conn) && transMsg.info.ahandle == NULL) { transMsg.code = TSDB_CODE_RPC_NETWORK_UNAVAIL; transMsg.info.ahandle = transCtxDumpBrokenlinkVal(&conn->ctx, (int32_t*)&(transMsg.msgType)); - tDebug("cli conn %p construct ahandle %p due brokenlink, persist: 1", conn, transMsg.info.ahandle); + tDebug("%s conn %p construct ahandle %p due brokenlink, persist: 1", CONN_GET_INST_LABEL(conn), conn, + transMsg.info.ahandle); } } else { pCtx = pMsg ? pMsg->ctx : NULL; transMsg.info.ahandle = pCtx ? pCtx->ahandle : NULL; - tDebug("cli conn %p get ahandle %p, persist: 1", conn, transMsg.info.ahandle); + tDebug("%s conn %p get ahandle %p, persist: 1", CONN_GET_INST_LABEL(conn), conn, transMsg.info.ahandle); } } // buf's mem alread translated to transMsg.pCont @@ -321,26 +324,27 @@ void cliHandleResp(SCliConn* conn) { if (!CONN_NO_PERSIST_BY_APP(conn)) { transMsg.info.handle = conn; - tDebug("%s cli conn %p ref by app", CONN_GET_INST_LABEL(conn), conn); + tDebug("%s conn %p ref by app", CONN_GET_INST_LABEL(conn), conn); } - - tDebug("%s cli conn %p %s received from %s:%d, local info:%s:%d, msg size:%d code:0x%x", pTransInst->label, conn, - TMSG_INFO(pHead->msgType), taosInetNtoa(conn->addr.sin_addr), ntohs(conn->addr.sin_port), - taosInetNtoa(conn->localAddr.sin_addr), ntohs(conn->localAddr.sin_port), transMsg.contLen, transMsg.code); + // char buf[64] = {0}; + // TRACE_TO_STR(&transMsg.info.traceId, buf); + STraceId* trace = &transMsg.info.traceId; + tGTrace("conn %p %s received from %s:%d, local info: %s:%d, msg size: %d, code: %d", conn, TMSG_INFO(pHead->msgType), + taosInetNtoa(conn->addr.sin_addr), ntohs(conn->addr.sin_port), taosInetNtoa(conn->localAddr.sin_addr), + ntohs(conn->localAddr.sin_port), transMsg.contLen, transMsg.code); if (pCtx == NULL && CONN_NO_PERSIST_BY_APP(conn)) { - tTrace("except, server continue send while cli ignore it"); + tDebug("%s except, server continue send while cli ignore it", CONN_GET_INST_LABEL(conn)); // transUnrefCliHandle(conn); return; } if (CONN_RELEASE_BY_SERVER(conn) && transMsg.info.ahandle == NULL) { - tTrace("except, server continue send while cli ignore it"); + tDebug("%s except, server continue send while cli ignore it", CONN_GET_INST_LABEL(conn)); // transUnrefCliHandle(conn); return; } if (cliAppCb(conn, &transMsg, pMsg) != 0) { - tTrace("try to send req to next node"); return; } destroyCmsg(pMsg); @@ -363,7 +367,7 @@ void cliHandleResp(SCliConn* conn) { void cliHandleExcept(SCliConn* pConn) { if (transQueueEmpty(&pConn->cliMsgs)) { if (pConn->broken == true && CONN_NO_PERSIST_BY_APP(pConn)) { - tTrace("%s cli conn %p handle except, persist:0", CONN_GET_INST_LABEL(pConn), pConn); + tTrace("%s conn %p handle except, persist:0", CONN_GET_INST_LABEL(pConn), pConn); transUnrefCliHandle(pConn); return; } @@ -386,11 +390,11 @@ void cliHandleExcept(SCliConn* pConn) { if (pMsg == NULL && !CONN_NO_PERSIST_BY_APP(pConn)) { transMsg.info.ahandle = transCtxDumpVal(&pConn->ctx, transMsg.msgType); - tDebug("%s cli conn %p construct ahandle %p by %s", CONN_GET_INST_LABEL(pConn), pConn, transMsg.info.ahandle, + tDebug("%s conn %p construct ahandle %p by %s", CONN_GET_INST_LABEL(pConn), pConn, transMsg.info.ahandle, TMSG_INFO(transMsg.msgType)); if (transMsg.info.ahandle == NULL) { transMsg.info.ahandle = transCtxDumpBrokenlinkVal(&pConn->ctx, (int32_t*)&(transMsg.msgType)); - tDebug("%s cli conn %p construct ahandle %p due to brokenlink", CONN_GET_INST_LABEL(pConn), pConn, + tDebug("%s conn %p construct ahandle %p due to brokenlink", CONN_GET_INST_LABEL(pConn), pConn, transMsg.info.ahandle); } } else { @@ -404,11 +408,10 @@ void cliHandleExcept(SCliConn* pConn) { } } if (cliAppCb(pConn, &transMsg, pMsg) != 0) { - tTrace("try to send req to next node"); return; } destroyCmsg(pMsg); - tTrace("%s cli conn %p start to destroy", CONN_GET_INST_LABEL(pConn), pConn); + tTrace("%s conn %p start to destroy", CONN_GET_INST_LABEL(pConn), pConn); } while (!transQueueEmpty(&pConn->cliMsgs)); transUnrefCliHandle(pConn); } @@ -417,7 +420,7 @@ void cliTimeoutCb(uv_timer_t* handle) { SCliThrdObj* pThrd = handle->data; STrans* pTransInst = pThrd->pTransInst; int64_t currentTime = pThrd->nextTimeout; - tTrace("%s, cli conn timeout, try to remove expire conn from conn pool", pTransInst->label); + tTrace("%s conn timeout, try to remove expire conn from conn pool", pTransInst->label); SConnList* p = taosHashIterate((SHashObj*)pThrd->pool, NULL); while (p != NULL) { @@ -492,7 +495,7 @@ static void addConnToPool(void* pool, SCliConn* conn) { char key[128] = {0}; CONN_CONSTRUCT_HASH_KEY(key, conn->ip, conn->port); - tTrace("cli conn %p added to conn pool, read buf cap: %d", conn, conn->readBuf.cap); + tTrace("%s conn %p added to conn pool, read buf cap: %d", CONN_GET_INST_LABEL(conn), conn, conn->readBuf.cap); SConnList* plist = taosHashGet((SHashObj*)pool, key, strlen(key)); // list already create before @@ -515,10 +518,10 @@ static void cliRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) { if (nread > 0) { pBuf->len += nread; if (transReadComplete(pBuf)) { - tTrace("%s cli conn %p read complete", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p read complete", CONN_GET_INST_LABEL(conn), conn); cliHandleResp(conn); } else { - tTrace("%s cli conn %p read partial packet, continue to read", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p read partial packet, continue to read", CONN_GET_INST_LABEL(conn), conn); } return; } @@ -528,11 +531,11 @@ static void cliRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) { // ref http://docs.libuv.org/en/v1.x/stream.html?highlight=uv_read_start#c.uv_read_cb // nread might be 0, which does not indicate an error or EOF. This is equivalent to EAGAIN or EWOULDBLOCK under // read(2). - tTrace("%s cli conn %p read empty", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p read empty", CONN_GET_INST_LABEL(conn), conn); return; } if (nread < 0) { - tError("%s cli conn %p read error: %s", CONN_GET_INST_LABEL(conn), conn, uv_err_name(nread)); + tError("%s conn %p read error: %s", CONN_GET_INST_LABEL(conn), conn, uv_err_name(nread)); conn->broken = true; cliHandleExcept(conn); } @@ -557,7 +560,7 @@ static SCliConn* cliCreateConn(SCliThrdObj* pThrd) { return conn; } static void cliDestroyConn(SCliConn* conn, bool clear) { - tTrace("%s cli conn %p remove from conn pool", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p remove from conn pool", CONN_GET_INST_LABEL(conn), conn); QUEUE_REMOVE(&conn->conn); if (clear) { @@ -570,7 +573,7 @@ static void cliDestroy(uv_handle_t* handle) { taosMemoryFree(conn->stream); transCtxCleanup(&conn->ctx); transQueueDestroy(&conn->cliMsgs); - tTrace("%s cli conn %p destroy successfully", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p destroy successfully", CONN_GET_INST_LABEL(conn), conn); transDestroyBuffer(&conn->readBuf); taosMemoryFree(conn); } @@ -597,14 +600,14 @@ static void cliSendCb(uv_write_t* req, int status) { SCliConn* pConn = req->data; if (status == 0) { - tTrace("%s cli conn %p data already was written out", CONN_GET_INST_LABEL(pConn), pConn); + tTrace("%s conn %p data already was written out", CONN_GET_INST_LABEL(pConn), pConn); } else { - tError("%s cli conn %p failed to write: %s", CONN_GET_INST_LABEL(pConn), pConn, uv_err_name(status)); + tError("%s conn %p failed to write: %s", CONN_GET_INST_LABEL(pConn), pConn, uv_err_name(status)); cliHandleExcept(pConn); return; } if (cliHandleNoResp(pConn) == true) { - tTrace("%s cli conn %p no resp required", CONN_GET_INST_LABEL(pConn), pConn); + tTrace("%s conn %p no resp required", CONN_GET_INST_LABEL(pConn), pConn); return; } uv_read_start((uv_stream_t*)pConn->stream, cliAllocRecvBufferCb, cliRecvCb); @@ -640,11 +643,16 @@ void cliSend(SCliConn* pConn) { pHead->msgLen = (int32_t)htonl((uint32_t)msgLen); pHead->release = REQUEST_RELEASE_HANDLE(pCliMsg) ? 1 : 0; memcpy(pHead->user, pTransInst->user, strlen(pTransInst->user)); + pHead->traceId = pMsg->info.traceId; uv_buf_t wb = uv_buf_init((char*)pHead, msgLen); - tDebug("%s cli conn %p %s is send to %s:%d, local info %s:%d", CONN_GET_INST_LABEL(pConn), pConn, - TMSG_INFO(pHead->msgType), taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), - taosInetNtoa(pConn->localAddr.sin_addr), ntohs(pConn->localAddr.sin_port)); + + // char buf[64] = {0}; + // TRACE_TO_STR(&pMsg->info.traceId, buf); + STraceId* trace = &pMsg->info.traceId; + tGTrace("conn %p %s is sent to %s:%d, local info %s:%d", pConn, TMSG_INFO(pHead->msgType), + taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), taosInetNtoa(pConn->localAddr.sin_addr), + ntohs(pConn->localAddr.sin_port)); if (pHead->persist == 1) { CONN_SET_PERSIST_BY_APP(pConn); @@ -662,7 +670,7 @@ void cliConnCb(uv_connect_t* req, int status) { // impl later SCliConn* pConn = req->data; if (status != 0) { - tError("%s cli conn %p failed to connect server: %s", CONN_GET_INST_LABEL(pConn), pConn, uv_strerror(status)); + tError("%s conn %p failed to connect server: %s", CONN_GET_INST_LABEL(pConn), pConn, uv_strerror(status)); cliHandleExcept(pConn); return; } @@ -672,7 +680,7 @@ void cliConnCb(uv_connect_t* req, int status) { addrlen = sizeof(pConn->localAddr); uv_tcp_getsockname((uv_tcp_t*)pConn->stream, (struct sockaddr*)&pConn->localAddr, &addrlen); - tTrace("%s cli conn %p connect to server successfully", CONN_GET_INST_LABEL(pConn), pConn); + tTrace("%s conn %p connect to server successfully", CONN_GET_INST_LABEL(pConn), pConn); assert(pConn->stream == req->handle); cliSend(pConn); @@ -691,7 +699,7 @@ static void cliHandleQuit(SCliMsg* pMsg, SCliThrdObj* pThrd) { } static void cliHandleRelease(SCliMsg* pMsg, SCliThrdObj* pThrd) { SCliConn* conn = pMsg->msg.info.handle; - tDebug("%s cli conn %p start to release to inst", CONN_GET_INST_LABEL(conn), conn); + tDebug("%s conn %p start to release to inst", CONN_GET_INST_LABEL(conn), conn); if (T_REF_VAL_GET(conn) == 2) { transUnrefCliHandle(conn); @@ -716,15 +724,15 @@ SCliConn* cliGetConn(SCliMsg* pMsg, SCliThrdObj* pThrd) { if (pMsg->msg.info.handle != NULL) { conn = (SCliConn*)(pMsg->msg.info.handle); if (conn != NULL) { - tTrace("%s cli conn %p reused", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p reused", CONN_GET_INST_LABEL(conn), conn); } } else { STransConnCtx* pCtx = pMsg->ctx; conn = getConnFromPool(pThrd->pool, EPSET_GET_INUSE_IP(&pCtx->epSet), EPSET_GET_INUSE_PORT(&pCtx->epSet)); if (conn != NULL) { - tTrace("%s cli conn %p get from conn pool", CONN_GET_INST_LABEL(conn), conn); + tTrace("%s conn %p get from conn pool", CONN_GET_INST_LABEL(conn), conn); } else { - tTrace("not found conn in conn pool %p", pThrd->pool); + tTrace("%s not found conn in conn pool %p", ((STrans*)pThrd->pTransInst)->label, pThrd->pool); } } return conn; @@ -750,6 +758,8 @@ void cliHandleReq(SCliMsg* pMsg, SCliThrdObj* pThrd) { cliMayCvtFqdnToIp(&pCtx->epSet, &pThrd->cvtAddr); + transPrintEpSet(&pCtx->epSet); + SCliConn* conn = cliGetConn(pMsg, pThrd); if (conn != NULL) { conn->hThrdIdx = pCtx->hThrdIdx; @@ -768,11 +778,11 @@ void cliHandleReq(SCliMsg* pMsg, SCliThrdObj* pThrd) { int ret = transSetConnOption((uv_tcp_t*)conn->stream); if (ret) { - tError("%s cli conn %p failed to set conn option, errmsg %s", pTransInst->label, conn, uv_err_name(ret)); + tError("%s conn %p failed to set conn option, errmsg %s", pTransInst->label, conn, uv_err_name(ret)); } int fd = taosCreateSocketWithTimeOutOpt(TRANS_CONN_TIMEOUT); if (fd == -1) { - tTrace("%s cli conn %p failed to create socket", pTransInst->label, conn); + tTrace("%s conn %p failed to create socket", pTransInst->label, conn); cliHandleExcept(conn); return; } @@ -782,10 +792,10 @@ void cliHandleReq(SCliMsg* pMsg, SCliThrdObj* pThrd) { addr.sin_family = AF_INET; addr.sin_addr.s_addr = taosGetIpv4FromFqdn(conn->ip); addr.sin_port = (uint16_t)htons((uint16_t)conn->port); - tTrace("%s cli conn %p try to connect to %s:%d", pTransInst->label, conn, conn->ip, conn->port); + tTrace("%s conn %p try to connect to %s:%d", pTransInst->label, conn, conn->ip, conn->port); ret = uv_tcp_connect(&conn->connReq, (uv_tcp_t*)(conn->stream), (const struct sockaddr*)&addr, cliConnCb); if (ret != 0) { - tTrace("%s cli conn %p failed to connect to %s:%d, reason: %s", pTransInst->label, conn, conn->ip, conn->port, + tTrace("%s conn %p failed to connect to %s:%d, reason: %s", pTransInst->label, conn, conn->ip, conn->port, uv_err_name(ret)); cliHandleExcept(conn); return; @@ -944,14 +954,13 @@ int cliAppCb(SCliConn* pConn, STransMsg* pResp, SCliMsg* pMsg) { STrans* pTransInst = pThrd->pTransInst; if (pMsg == NULL || pMsg->ctx == NULL) { - tTrace("%s cli conn %p handle resp", pTransInst->label, pConn); + tTrace("%s conn %p handle resp", pTransInst->label, pConn); pTransInst->cfp(pTransInst->parent, pResp, NULL); return 0; } STransConnCtx* pCtx = pMsg->ctx; SEpSet* pEpSet = &pCtx->epSet; - transPrintEpSet(pEpSet); if (pCtx->retryCount == 0) { pCtx->origEpSet = pCtx->epSet; @@ -964,6 +973,7 @@ int cliAppCb(SCliConn* pConn, STransMsg* pResp, SCliMsg* pMsg) { (pResp->code == TSDB_CODE_RPC_NETWORK_UNAVAIL || pResp->code == TSDB_CODE_APP_NOT_READY || pResp->code == TSDB_CODE_NODE_NOT_DEPLOYED || pResp->code == TSDB_CODE_SYN_NOT_LEADER)) { pMsg->sent = 0; + tTrace("try to send req to next node"); pMsg->st = taosGetTimestampUs(); pCtx->retryCount += 1; if (pResp->code == TSDB_CODE_RPC_NETWORK_UNAVAIL) { @@ -974,22 +984,27 @@ int cliAppCb(SCliConn* pConn, STransMsg* pResp, SCliMsg* pMsg) { arg->param1 = pMsg; arg->param2 = pThrd; transDQSched(pThrd->delayQueue, doDelayTask, arg, TRANS_RETRY_INTERVAL); - tTrace("use local epset, current in use: %d, retry count:%d, limit: %d", pEpSet->inUse, pCtx->retryCount + 1, - pEpSet->numOfEps * 3); + transPrintEpSet(pEpSet); + tTrace("%s use local epset, inUse: %d, retry count:%d, limit: %d", pTransInst->label, pEpSet->inUse, + pCtx->retryCount + 1, pEpSet->numOfEps * 3); + transUnrefCliHandle(pConn); return -1; } } else if (pCtx->retryCount < TRANS_RETRY_COUNT_LIMIT) { if (pResp->contLen == 0) { pEpSet->inUse = (++pEpSet->inUse) % pEpSet->numOfEps; - tTrace("use local epset, current in use: %d, retry count:%d, limit: %d", pEpSet->inUse, pCtx->retryCount + 1, - TRANS_RETRY_COUNT_LIMIT); + transPrintEpSet(&pCtx->epSet); + tTrace("%s use local epset, inUse: %d, retry count:%d, limit: %d", pTransInst->label, pEpSet->inUse, + pCtx->retryCount + 1, TRANS_RETRY_COUNT_LIMIT); } else { SEpSet epSet = {0}; tDeserializeSEpSet(pResp->pCont, pResp->contLen, &epSet); pCtx->epSet = epSet; - tTrace("use remote epset, current in use: %d, retry count:%d, limit: %d", pEpSet->inUse, pCtx->retryCount + 1, - TRANS_RETRY_COUNT_LIMIT); + + transPrintEpSet(&pCtx->epSet); + tTrace("%s use remote epset, inUse: %d, retry count:%d, limit: %d", pTransInst->label, pEpSet->inUse, + pCtx->retryCount + 1, TRANS_RETRY_COUNT_LIMIT); } addConnToPool(pThrd->pool, pConn); @@ -1001,17 +1016,18 @@ int cliAppCb(SCliConn* pConn, STransMsg* pResp, SCliMsg* pMsg) { } } + STraceId* trace = &pResp->info.traceId; if (pCtx->pSem != NULL) { - tTrace("%s cli conn %p(sync) handle resp", pTransInst->label, pConn); + tGTrace("conn %p(sync) handle resp", pConn); if (pCtx->pRsp == NULL) { - tTrace("%s cli conn %p(sync) failed to resp, ignore", pTransInst->label, pConn); + tGTrace("conn %p(sync) failed to resp, ignore", pConn); } else { memcpy((char*)pCtx->pRsp, (char*)pResp, sizeof(*pResp)); } tsem_post(pCtx->pSem); pCtx->pRsp = NULL; } else { - tTrace("%s cli conn %p handle resp", pTransInst->label, pConn); + tGTrace("conn %p handle resp", pConn); if (pResp->code != 0 || pCtx->retryCount == 0 || transEpSetIsEqual(&pCtx->epSet, &pCtx->origEpSet)) { pTransInst->cfp(pTransInst->parent, pResp, NULL); } else { @@ -1039,6 +1055,7 @@ void transRefCliHandle(void* handle) { return; } int ref = T_REF_INC((SCliConn*)handle); + tTrace("%s conn %p ref %d", CONN_GET_INST_LABEL((SCliConn*)handle), handle, ref); UNUSED(ref); } void transUnrefCliHandle(void* handle) { @@ -1046,7 +1063,7 @@ void transUnrefCliHandle(void* handle) { return; } int ref = T_REF_DEC((SCliConn*)handle); - tDebug("%s cli conn %p ref %d", CONN_GET_INST_LABEL((SCliConn*)handle), handle, ref); + tTrace("%s conn %p ref %d", CONN_GET_INST_LABEL((SCliConn*)handle), handle, ref); if (ref == 0) { cliDestroyConn((SCliConn*)handle, true); } @@ -1067,16 +1084,17 @@ void transReleaseCliHandle(void* handle) { void transSendRequest(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, STransCtx* ctx) { STrans* pTransInst = (STrans*)shandle; - int index = CONN_HOST_THREAD_INDEX((SCliConn*)pReq->info.handle); - if (index == -1) { - index = cliRBChoseIdx(pTransInst); + int idx = CONN_HOST_THREAD_IDX((SCliConn*)pReq->info.handle); + if (idx == -1) { + idx = cliRBChoseIdx(pTransInst); } + TRACE_SET_MSGID(&pReq->info.traceId, tGenIdPI64()); STransConnCtx* pCtx = taosMemoryCalloc(1, sizeof(STransConnCtx)); pCtx->epSet = *pEpSet; pCtx->ahandle = pReq->info.ahandle; pCtx->msgType = pReq->msgType; - pCtx->hThrdIdx = index; + pCtx->hThrdIdx = idx; if (ctx != NULL) { pCtx->appCtx = *ctx; @@ -1089,27 +1107,30 @@ void transSendRequest(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, STra cliMsg->st = taosGetTimestampUs(); cliMsg->type = Normal; - SCliThrdObj* thrd = ((SCliObj*)pTransInst->tcphandle)->pThreadObj[index]; + SCliThrdObj* thrd = ((SCliObj*)pTransInst->tcphandle)->pThreadObj[idx]; - tDebug("send request at thread:%d, threadID: %08" PRId64 ", msg: %p, dst: %s:%d, app:%p", index, thrd->pid, pReq, - EPSET_GET_INUSE_IP(&pCtx->epSet), EPSET_GET_INUSE_PORT(&pCtx->epSet), pReq->info.ahandle); + STraceId* trace = &pReq->info.traceId; + tGTrace("%s send request at thread:%08" PRId64 ", dst: %s:%d, app:%p", pTransInst->label, thrd->pid, + EPSET_GET_INUSE_IP(&pCtx->epSet), EPSET_GET_INUSE_PORT(&pCtx->epSet), pReq->info.ahandle); ASSERT(transSendAsync(thrd->asyncPool, &(cliMsg->q)) == 0); } void transSendRecv(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, STransMsg* pRsp) { STrans* pTransInst = (STrans*)shandle; - int index = CONN_HOST_THREAD_INDEX(pReq->info.handle); - if (index == -1) { - index = cliRBChoseIdx(pTransInst); + int idx = CONN_HOST_THREAD_IDX(pReq->info.handle); + if (idx == -1) { + idx = cliRBChoseIdx(pTransInst); } tsem_t* sem = taosMemoryCalloc(1, sizeof(tsem_t)); tsem_init(sem, 0, 0); + TRACE_SET_MSGID(&pReq->info.traceId, tGenIdPI64()); + STransConnCtx* pCtx = taosMemoryCalloc(1, sizeof(STransConnCtx)); pCtx->epSet = *pEpSet; pCtx->ahandle = pReq->info.ahandle; pCtx->msgType = pReq->msgType; - pCtx->hThrdIdx = index; + pCtx->hThrdIdx = idx; pCtx->pSem = sem; pCtx->pRsp = pRsp; @@ -1119,16 +1140,17 @@ void transSendRecv(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, STransM cliMsg->st = taosGetTimestampUs(); cliMsg->type = Normal; - SCliThrdObj* thrd = ((SCliObj*)pTransInst->tcphandle)->pThreadObj[index]; - tDebug("send request at thread:%d, threadID:%08" PRId64 ", msg: %p, dst: %s:%d, app:%p", index, thrd->pid, pReq, - EPSET_GET_INUSE_IP(&pCtx->epSet), EPSET_GET_INUSE_PORT(&pCtx->epSet), pReq->info.ahandle); + SCliThrdObj* thrd = ((SCliObj*)pTransInst->tcphandle)->pThreadObj[idx]; + + STraceId* trace = &pReq->info.traceId; + tGTrace("%s send request at thread:%08" PRId64 ", dst: %s:%d, app:%p", pTransInst->label, thrd->pid, + EPSET_GET_INUSE_IP(&pCtx->epSet), EPSET_GET_INUSE_PORT(&pCtx->epSet), pReq->info.ahandle); transSendAsync(thrd->asyncPool, &(cliMsg->q)); tsem_wait(sem); tsem_destroy(sem); taosMemoryFree(sem); } - /* * **/ @@ -1151,7 +1173,7 @@ void transSetDefaultAddr(void* ahandle, const char* ip, const char* fqdn) { cliMsg->type = Update; SCliThrdObj* thrd = ((SCliObj*)pTransInst->tcphandle)->pThreadObj[i]; - tDebug("update epset at thread:%d, threadID:%08" PRId64 "", i, thrd->pid); + tDebug("%s update epset at thread:%08" PRId64 "", pTransInst->label, thrd->pid); transSendAsync(thrd->asyncPool, &(cliMsg->q)); } diff --git a/source/libs/transport/src/transComm.c b/source/libs/transport/src/transComm.c index a04e8b5fca05b7e02b58bbd4b5abfc528a5289e9..8cd7f9d827396fbb52b7e4c926f77ada47429ad5 100644 --- a/source/libs/transport/src/transComm.c +++ b/source/libs/transport/src/transComm.c @@ -376,17 +376,18 @@ static void transDQTimeout(uv_timer_t* timer) { SDelayQueue* queue = timer->data; tTrace("timer %p timeout", timer); uint64_t timeout = 0; + int64_t current = taosGetTimestampMs(); do { HeapNode* minNode = heapMin(queue->heap); if (minNode == NULL) break; SDelayTask* task = container_of(minNode, SDelayTask, node); - if (task->execTime <= taosGetTimestampMs()) { + if (task->execTime <= current) { heapRemove(queue->heap, minNode); task->func(task->arg); taosMemoryFree(task); timeout = 0; } else { - timeout = task->execTime - taosGetTimestampMs(); + timeout = task->execTime - current; break; } } while (1); @@ -442,7 +443,7 @@ int transDQSched(SDelayQueue* queue, void (*func)(void* arg), void* arg, uint64_ } } - tTrace("timer %p put task into queue, timeoutMs: %" PRIu64 "", queue->timer, timeoutMs); + tTrace("timer %p put task into delay queue, timeoutMs: %" PRIu64 "", queue->timer, timeoutMs); heapInsert(queue->heap, &task->node); uv_timer_start(queue->timer, transDQTimeout, timeoutMs, 0); return 0; @@ -453,11 +454,17 @@ void transPrintEpSet(SEpSet* pEpSet) { tTrace("NULL epset"); return; } - tTrace("epset begin inUse: %d", pEpSet->inUse); + char buf[512] = {0}; + int len = snprintf(buf, sizeof(buf), "epset { "); for (int i = 0; i < pEpSet->numOfEps; i++) { - tTrace("ip: %s, port: %d", pEpSet->eps[i].fqdn, pEpSet->eps[i].port); + if (i == pEpSet->numOfEps - 1) { + len += snprintf(buf + len, sizeof(buf) - len, "%d. %s:%d ", i, pEpSet->eps[i].fqdn, pEpSet->eps[i].port); + } else { + len += snprintf(buf + len, sizeof(buf) - len, "%d. %s:%d, ", i, pEpSet->eps[i].fqdn, pEpSet->eps[i].port); + } } - tTrace("epset end"); + len += snprintf(buf + len, sizeof(buf) - len, "}"); + tTrace("%s, inUse: %d", buf, pEpSet->inUse); } bool transEpSetIsEqual(SEpSet* a, SEpSet* b) { if (a->numOfEps != b->numOfEps || a->inUse != b->inUse) { diff --git a/source/libs/transport/src/transSvr.c b/source/libs/transport/src/transSvr.c index b6e3fd26765c42140132826a3f50b1adeed6e514..f9558d7252ceabfd6514f53665baefa60dd2ef23 100644 --- a/source/libs/transport/src/transSvr.c +++ b/source/libs/transport/src/transSvr.c @@ -169,7 +169,7 @@ static bool addHandleToAcceptloop(void* arg); conn->status = ConnRelease; \ transClearBuffer(&conn->readBuf); \ transFreeMsg(transContFromHead((char*)head)); \ - tTrace("server conn %p received release request", conn); \ + tTrace("conn %p received release request", conn); \ \ STransMsg tmsg = {.code = 0, .info.handle = (void*)conn, .info.ahandle = NULL}; \ SSvrMsg* srvMsg = taosMemoryCalloc(1, sizeof(SSvrMsg)); \ @@ -181,7 +181,7 @@ static bool addHandleToAcceptloop(void* arg); return; \ } \ if (conn->regArg.init) { \ - tTrace("server conn %p release, notify server app", conn); \ + tTrace("conn %p release, notify server app", conn); \ STrans* pTransInst = conn->pTransInst; \ (*pTransInst->cfp)(pTransInst->parent, &(conn->regArg.msg), NULL); \ memset(&conn->regArg, 0, sizeof(conn->regArg)); \ @@ -209,25 +209,25 @@ static bool addHandleToAcceptloop(void* arg); #define ASYNC_CHECK_HANDLE(exh1, refId) \ do { \ if (refId > 0) { \ - tTrace("server handle step1"); \ + tTrace("handle step1"); \ SExHandle* exh2 = transAcquireExHandle(refMgt, refId); \ if (exh2 == NULL || refId != exh2->refId) { \ - tTrace("server handle %p except, may already freed, ignore msg, ref1: %" PRIu64 ", ref2 : %" PRIu64 "", exh1, \ + tTrace("handle %p except, may already freed, ignore msg, ref1: %" PRIu64 ", ref2 : %" PRIu64 "", exh1, \ exh2 ? exh2->refId : 0, refId); \ goto _return1; \ } \ } else if (refId == 0) { \ - tTrace("server handle step2"); \ + tTrace("handle step2"); \ SExHandle* exh2 = transAcquireExHandle(refMgt, refId); \ if (exh2 == NULL || refId != exh2->refId) { \ - tTrace("server handle %p except, may already freed, ignore msg, ref1: %" PRIu64 ", ref2 : %" PRIu64 "", exh1, \ - refId, exh2 ? exh2->refId : 0); \ + tTrace("handle %p except, may already freed, ignore msg, ref1: %" PRIu64 ", ref2 : %" PRIu64 "", exh1, refId, \ + exh2 ? exh2->refId : 0); \ goto _return1; \ } else { \ refId = exh1->refId; \ } \ } else if (refId < 0) { \ - tTrace("server handle step3"); \ + tTrace("handle step3"); \ goto _return2; \ } \ } while (0) @@ -270,29 +270,28 @@ static void uvHandleReq(SSvrConn* pConn) { transMsg.msgType = pHead->msgType; transMsg.code = pHead->code; - transMsg.info.ahandle = (void*)pHead->ahandle; - transMsg.info.handle = NULL; - - // transDestroyBuffer(&pConn->readBuf); transClearBuffer(&pConn->readBuf); + pConn->inType = pHead->msgType; if (pConn->status == ConnNormal) { if (pHead->persist == 1) { pConn->status = ConnAcquire; transRefSrvHandle(pConn); - tDebug("server conn %p acquired by server app", pConn); + tDebug("conn %p acquired by server app", pConn); } } + STraceId* trace = &pHead->traceId; if (pConn->status == ConnNormal && pHead->noResp == 0) { transRefSrvHandle(pConn); - tDebug("server conn %p %s received from %s:%d, local info:%s:%d, msg size:%d code:0x%x", pConn, - TMSG_INFO(transMsg.msgType), taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), - taosInetNtoa(pConn->localAddr.sin_addr), ntohs(pConn->localAddr.sin_port), transMsg.contLen, transMsg.code); + + tGTrace("conn %p %s received from %s:%d, local info: %s:%d, msg size: %d", pConn, TMSG_INFO(transMsg.msgType), + taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), taosInetNtoa(pConn->localAddr.sin_addr), + ntohs(pConn->localAddr.sin_port), transMsg.contLen); } else { - tDebug("server conn %p %s received from %s:%d, local info:%s:%d, msg size:%d, resp:%d code:0x%x", pConn, - TMSG_INFO(transMsg.msgType), taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), - taosInetNtoa(pConn->localAddr.sin_addr), ntohs(pConn->localAddr.sin_port), transMsg.contLen, pHead->noResp, - transMsg.code); + tGTrace("conn %p %s received from %s:%d, local info: %s:%d, msg size: %d, resp:%d, code: %d", pConn, + TMSG_INFO(transMsg.msgType), taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), + taosInetNtoa(pConn->localAddr.sin_addr), ntohs(pConn->localAddr.sin_port), transMsg.contLen, pHead->noResp, + transMsg.code); // no ref here } @@ -300,17 +299,20 @@ static void uvHandleReq(SSvrConn* pConn) { // 1. server application should not send resp on handle // 2. once send out data, cli conn released to conn pool immediately // 3. not mixed with persist - + transMsg.info.ahandle = (void*)pHead->ahandle; transMsg.info.handle = (void*)transAcquireExHandle(refMgt, pConn->refId); transMsg.info.refId = pConn->refId; - tTrace("server handle %p conn: %p translated to app, refId: %" PRIu64 "", transMsg.info.handle, pConn, pConn->refId); + transMsg.info.traceId = pHead->traceId; + + tGTrace("handle %p conn: %p translated to app, refId: %" PRIu64 "", transMsg.info.handle, pConn, pConn->refId); assert(transMsg.info.handle != NULL); + if (pHead->noResp == 1) { transMsg.info.refId = -1; } // set up conn info - SRpcConnInfo* pConnInfo = &(transMsg.info.connInfo); + SRpcConnInfo* pConnInfo = &(transMsg.info.conn); pConnInfo->clientIp = (uint32_t)(pConn->addr.sin_addr.s_addr); pConnInfo->clientPort = ntohs(pConn->addr.sin_port); tstrncpy(pConnInfo->user, pConn->user, sizeof(pConnInfo->user)); @@ -328,12 +330,12 @@ void uvOnRecvCb(uv_stream_t* cli, ssize_t nread, const uv_buf_t* buf) { SConnBuffer* pBuf = &conn->readBuf; if (nread > 0) { pBuf->len += nread; - tTrace("server conn %p read summary, total read: %d, current read: %d", conn, pBuf->len, (int)nread); + tTrace("conn %p total read: %d, current read: %d", conn, pBuf->len, (int)nread); if (transReadComplete(pBuf)) { - tTrace("server conn %p alread read complete packet", conn); + tTrace("conn %p alread read complete packet", conn); uvHandleReq(conn); } else { - tTrace("server %p read partial packet, continue to read", conn); + tTrace("conn %p read partial packet, continue to read", conn); } return; } @@ -341,12 +343,12 @@ void uvOnRecvCb(uv_stream_t* cli, ssize_t nread, const uv_buf_t* buf) { return; } - tError("server conn %p read error: %s", conn, uv_err_name(nread)); + tError("conn %p read error: %s", conn, uv_err_name(nread)); if (nread < 0) { conn->broken = true; if (conn->status == ConnAcquire) { if (conn->regArg.init) { - tTrace("server conn %p broken, notify server app", conn); + tTrace("conn %p broken, notify server app", conn); STrans* pTransInst = conn->pTransInst; (*pTransInst->cfp)(pTransInst->parent, &(conn->regArg.msg), NULL); memset(&conn->regArg, 0, sizeof(conn->regArg)); @@ -363,14 +365,14 @@ void uvAllocConnBufferCb(uv_handle_t* handle, size_t suggested_size, uv_buf_t* b void uvOnTimeoutCb(uv_timer_t* handle) { // opt SSvrConn* pConn = handle->data; - tError("server conn %p time out", pConn); + tError("conn %p time out", pConn); } void uvOnSendCb(uv_write_t* req, int status) { SSvrConn* conn = req->data; // transClearBuffer(&conn->readBuf); if (status == 0) { - tTrace("server conn %p data already was written on stream", conn); + tTrace("conn %p data already was written on stream", conn); if (!transQueueEmpty(&conn->srvMsgs)) { SSvrMsg* msg = transQueuePop(&conn->srvMsgs); // if (msg->type == Release && conn->status != ConnNormal) { @@ -407,7 +409,7 @@ void uvOnSendCb(uv_write_t* req, int status) { } } } else { - tError("server conn %p failed to write data, %s", conn, uv_err_name(status)); + tError("conn %p failed to write data, %s", conn, uv_err_name(status)); conn->broken = true; transUnrefSrvHandle(conn); } @@ -424,8 +426,6 @@ static void uvOnPipeWriteCb(uv_write_t* req, int status) { } static void uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) { - tTrace("server conn %p prepare to send resp", smsg->pConn); - SSvrConn* pConn = smsg->pConn; STransMsg* pMsg = &smsg->msg; if (pMsg->pCont == 0) { @@ -434,6 +434,7 @@ static void uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) { } STransMsgHead* pHead = transHeadFromCont(pMsg->pCont); pHead->ahandle = (uint64_t)pMsg->info.ahandle; + pHead->traceId = pMsg->info.traceId; if (pConn->status == ConnNormal) { pHead->msgType = pConn->inType + 1; @@ -454,9 +455,11 @@ static void uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) { char* msg = (char*)pHead; int32_t len = transMsgLenFromCont(pMsg->contLen); - tDebug("server conn %p %s is sent to %s:%d, local info: %s:%d, msglen:%d", pConn, TMSG_INFO(pHead->msgType), - taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), taosInetNtoa(pConn->localAddr.sin_addr), - ntohs(pConn->localAddr.sin_port), len); + + STraceId* trace = &pMsg->info.traceId; + tGTrace("conn %p %s is sent to %s:%d, local info: %s:%d, msglen:%d", pConn, TMSG_INFO(pHead->msgType), + taosInetNtoa(pConn->addr.sin_addr), ntohs(pConn->addr.sin_port), taosInetNtoa(pConn->localAddr.sin_addr), + ntohs(pConn->localAddr.sin_port), len); pHead->msgLen = htonl(len); wb->base = msg; @@ -545,7 +548,7 @@ void uvWorkerAsyncCb(uv_async_t* handle) { int64_t refId = transMsg.info.refId; SExHandle* exh2 = transAcquireExHandle(refMgt, refId); if (exh2 == NULL || exh1 != exh2) { - tTrace("server handle except msg %p, ignore it", exh1); + tTrace("handle except msg %p, ignore it", exh1); transReleaseExHandle(refMgt, refId); destroySmsg(msg); continue; @@ -583,18 +586,18 @@ static void uvShutDownCb(uv_shutdown_t* req, int status) { static void uvWorkDoTask(uv_work_t* req) { // doing time-consumeing task // only auth conn currently, add more func later - tTrace("server conn %p start to be processed in BG Thread", req->data); + tTrace("conn %p start to be processed in BG Thread", req->data); return; } static void uvWorkAfterTask(uv_work_t* req, int status) { if (status != 0) { - tTrace("server conn %p failed to processed ", req->data); + tTrace("conn %p failed to processed ", req->data); } // Done time-consumeing task // add more func later // this func called in main loop - tTrace("server conn %p already processed ", req->data); + tTrace("conn %p already processed ", req->data); taosMemoryFree(req); } @@ -629,7 +632,7 @@ void uvOnAcceptCb(uv_stream_t* stream, int status) { } } void uvOnConnectionCb(uv_stream_t* q, ssize_t nread, const uv_buf_t* buf) { - tTrace("server connection coming"); + tTrace("connection coming"); if (nread < 0) { if (nread != UV_EOF) { tError("read error %s", uv_err_name(nread)); @@ -678,18 +681,18 @@ void uvOnConnectionCb(uv_stream_t* q, ssize_t nread, const uv_buf_t* buf) { if (uv_accept(q, (uv_stream_t*)(pConn->pTcp)) == 0) { uv_os_fd_t fd; uv_fileno((const uv_handle_t*)pConn->pTcp, &fd); - tTrace("server conn %p created, fd: %d", pConn, fd); + tTrace("conn %p created, fd: %d", pConn, fd); int addrlen = sizeof(pConn->addr); if (0 != uv_tcp_getpeername(pConn->pTcp, (struct sockaddr*)&pConn->addr, &addrlen)) { - tError("server conn %p failed to get peer info", pConn); + tError("conn %p failed to get peer info", pConn); transUnrefSrvHandle(pConn); return; } addrlen = sizeof(pConn->localAddr); if (0 != uv_tcp_getsockname(pConn->pTcp, (struct sockaddr*)&pConn->localAddr, &addrlen)) { - tError("server conn %p failed to get local info", pConn); + tError("conn %p failed to get local info", pConn); transUnrefSrvHandle(pConn); return; } @@ -798,7 +801,7 @@ static SSvrConn* createConn(void* hThrd) { pConn->refId = exh->refId; transRefSrvHandle(pConn); - tTrace("server handle %p, conn %p created, refId: %" PRId64 "", exh, pConn, pConn->refId); + tTrace("handle %p, conn %p created, refId: %" PRId64 "", exh, pConn, pConn->refId); return pConn; } @@ -809,7 +812,7 @@ static void destroyConn(SSvrConn* conn, bool clear) { transDestroyBuffer(&conn->readBuf); if (clear) { - tTrace("server conn %p to be destroyed", conn); + tTrace("conn %p to be destroyed", conn); // uv_shutdown_t* req = taosMemoryMalloc(sizeof(uv_shutdown_t)); uv_close((uv_handle_t*)conn->pTcp, uvDestroyConn); // uv_close(conn->pTcp) @@ -845,7 +848,7 @@ static void uvDestroyConn(uv_handle_t* handle) { transReleaseExHandle(refMgt, conn->refId); transRemoveExHandle(refMgt, conn->refId); - tDebug("server conn %p destroy", conn); + tDebug("conn %p destroy", conn); // uv_timer_stop(&conn->pTimer); transQueueDestroy(&conn->srvMsgs); @@ -974,18 +977,18 @@ void uvHandleRelease(SSvrMsg* msg, SWorkThrdObj* thrd) { uvStartSendRespInternal(msg); return; } else if (conn->status == ConnRelease || conn->status == ConnNormal) { - tDebug("server conn %p already released, ignore release-msg", conn); + tDebug("conn %p already released, ignore release-msg", conn); } destroySmsg(msg); } void uvHandleResp(SSvrMsg* msg, SWorkThrdObj* thrd) { // send msg to client - tDebug("server conn %p start to send resp (2/2)", msg->pConn); + tDebug("conn %p start to send resp (2/2)", msg->pConn); uvStartSendResp(msg); } void uvHandleRegister(SSvrMsg* msg, SWorkThrdObj* thrd) { SSvrConn* conn = msg->pConn; - tDebug("server conn %p register brokenlink callback", conn); + tDebug("conn %p register brokenlink callback", conn); if (conn->status == ConnAcquire) { if (!transQueuePush(&conn->srvMsgs, msg)) { return; @@ -994,7 +997,7 @@ void uvHandleRegister(SSvrMsg* msg, SWorkThrdObj* thrd) { conn->regArg.notifyCount = 0; conn->regArg.init = 1; conn->regArg.msg = msg->msg; - tDebug("server conn %p register brokenlink callback succ", conn); + tDebug("conn %p register brokenlink callback succ", conn); if (conn->broken) { STrans* pTransInst = conn->pTransInst; @@ -1062,7 +1065,7 @@ void transRefSrvHandle(void* handle) { return; } int ref = T_REF_INC((SSvrConn*)handle); - tDebug("server conn %p ref count: %d", handle, ref); + tDebug("conn %p ref count: %d", handle, ref); } void transUnrefSrvHandle(void* handle) { @@ -1070,7 +1073,7 @@ void transUnrefSrvHandle(void* handle) { return; } int ref = T_REF_DEC((SSvrConn*)handle); - tDebug("server conn %p ref count: %d", handle, ref); + tDebug("conn %p ref count: %d", handle, ref); if (ref == 0) { destroyConn((SSvrConn*)handle, true); } @@ -1091,16 +1094,16 @@ void transReleaseSrvHandle(void* handle) { m->msg = tmsg; m->type = Release; - tTrace("server conn %p start to release", exh->handle); + tTrace("conn %p start to release", exh->handle); transSendAsync(pThrd->asyncPool, &m->q); transReleaseExHandle(refMgt, refId); return; _return1: - tTrace("server handle %p failed to send to release handle", exh); + tTrace("handle %p failed to send to release handle", exh); transReleaseExHandle(refMgt, refId); return; _return2: - tTrace("server handle %p failed to send to release handle", exh); + tTrace("handle %p failed to send to release handle", exh); return; } void transSendResponse(const STransMsg* msg) { @@ -1118,17 +1121,19 @@ void transSendResponse(const STransMsg* msg) { SSvrMsg* m = taosMemoryCalloc(1, sizeof(SSvrMsg)); m->msg = tmsg; m->type = Normal; - tDebug("server conn %p start to send resp (1/2)", exh->handle); + + STraceId* trace = (STraceId*)&msg->info.traceId; + tGTrace("conn %p start to send resp (1/2)", exh->handle); transSendAsync(pThrd->asyncPool, &m->q); transReleaseExHandle(refMgt, refId); return; _return1: - tTrace("server handle %p failed to send resp", exh); + tTrace("handle %p failed to send resp", exh); rpcFreeCont(msg->pCont); transReleaseExHandle(refMgt, refId); return; _return2: - tTrace("server handle %p failed to send resp", exh); + tTrace("handle %p failed to send resp", exh); rpcFreeCont(msg->pCont); return; } @@ -1146,18 +1151,19 @@ void transRegisterMsg(const STransMsg* msg) { SSvrMsg* m = taosMemoryCalloc(1, sizeof(SSvrMsg)); m->msg = tmsg; m->type = Register; - tTrace("server conn %p start to register brokenlink callback", exh->handle); + + tTrace("conn %p start to register brokenlink callback", exh->handle); transSendAsync(pThrd->asyncPool, &m->q); transReleaseExHandle(refMgt, refId); return; _return1: - tTrace("server handle %p failed to send to register brokenlink", exh); + tTrace("handle %p failed to register brokenlink", exh); rpcFreeCont(msg->pCont); transReleaseExHandle(refMgt, refId); return; _return2: - tTrace("server handle %p failed to send to register brokenlink", exh); + tTrace("handle %p failed to register brokenlink", exh); rpcFreeCont(msg->pCont); } diff --git a/source/os/src/osSocket.c b/source/os/src/osSocket.c index 4a0d9e286629dfb4c788eb489ab41f9c6802d831..4d61e7036d695e4c0df34ef88d02a4609e6dcf1e 100644 --- a/source/os/src/osSocket.c +++ b/source/os/src/osSocket.c @@ -946,8 +946,7 @@ int32_t taosGetFqdn(char *fqdn) { #endif // __APPLE__ int32_t ret = getaddrinfo(hostname, NULL, &hints, &result); if (!result) { - // printf("failed to get fqdn, code:%d, reason:%s", ret, gai_strerror(ret)); - assert(0); + fprintf(stderr,"failed to get fqdn, code:%d, reason:%s", ret, gai_strerror(ret)); return -1; } diff --git a/source/util/src/tenv.c b/source/util/src/tenv.c index e717e82c5b27bb9911364571538cf8f6863098b8..4fc0542816397e780a962f9b6a52055114f97197 100644 --- a/source/util/src/tenv.c +++ b/source/util/src/tenv.c @@ -72,6 +72,9 @@ int32_t taosEnvToCfg(const char *envStr, char *cfgStr) { if (cfgNameLen > 0) { memcpy(cfgStr, buf, cfgNameLen); memset(&cfgStr[cfgNameLen], ' ', p - cfgStr - cfgNameLen + 1); + } else { + *cfgStr = '\0'; + return -1; } } return strlen(cfgStr); diff --git a/source/util/src/terror.c b/source/util/src/terror.c index 596aab0e09d08db07b7c47c5089838eb96c99171..6afb240ab91b52c603cd3009c57d8b3c708f7ad0 100644 --- a/source/util/src/terror.c +++ b/source/util/src/terror.c @@ -132,6 +132,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TSC_INVALID_INPUT, "Invalid tsc input") TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_API_ERROR, "Stmt API usage error") TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_TBNAME_ERROR, "Stmt table name not set") TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_CLAUSE_ERROR, "not supported stmt clause") +TAOS_DEFINE_ERROR(TSDB_CODE_TSC_QUERY_KILLED, "Query killed") +TAOS_DEFINE_ERROR(TSDB_CODE_TSC_NO_EXEC_NODE, "No available execution node") // mnode-common TAOS_DEFINE_ERROR(TSDB_CODE_MND_APP_ERROR, "Mnode internal error") @@ -276,9 +278,14 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_TOPIC, "Invalid topic") TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_TOPIC_QUERY, "Topic with invalid query") TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_TOPIC_OPTION, "Topic with invalid option") TAOS_DEFINE_ERROR(TSDB_CODE_MND_CONSUMER_NOT_EXIST, "Consumer not exist") -TAOS_DEFINE_ERROR(TSDB_CODE_MND_CGROUP_USED, "Consumer group being used by some consumer") +TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOPIC_OPTION_UNCHNAGED, "Consumer unchanged") +TAOS_DEFINE_ERROR(TSDB_CODE_MND_SUBSCRIBE_NOT_EXIST, "Subcribe not exist") +TAOS_DEFINE_ERROR(TSDB_CODE_MND_OFFSET_NOT_EXIST, "Offset not exist") +TAOS_DEFINE_ERROR(TSDB_CODE_MND_CONSUMER_NOT_READY, "Consumer not ready") TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOPIC_SUBSCRIBED, "Topic subscribed cannot be dropped") +TAOS_DEFINE_ERROR(TSDB_CODE_MND_CGROUP_USED, "Consumer group being used by some consumer") +// mnode-stream TAOS_DEFINE_ERROR(TSDB_CODE_MND_STREAM_ALREADY_EXIST, "Stream already exists") TAOS_DEFINE_ERROR(TSDB_CODE_MND_STREAM_NOT_EXIST, "Stream not exist") TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_STREAM_OPTION, "Invalid stream option") @@ -416,6 +423,9 @@ TAOS_DEFINE_ERROR(TSDB_CODE_SYN_IS_LEADER, "Sync is leader") TAOS_DEFINE_ERROR(TSDB_CODE_SYN_NOT_LEADER, "Sync not leader") TAOS_DEFINE_ERROR(TSDB_CODE_SYN_ONE_REPLICA, "Sync one replica") TAOS_DEFINE_ERROR(TSDB_CODE_SYN_NOT_IN_NEW_CONFIG, "Sync not in new config") +TAOS_DEFINE_ERROR(TSDB_CODE_SYN_NEW_CONFIG_ERROR, "Sync new config error") +TAOS_DEFINE_ERROR(TSDB_CODE_SYN_RECONFIG_NOT_READY, "Sync not ready for reconfig") +TAOS_DEFINE_ERROR(TSDB_CODE_SYN_PROPOSE_NOT_READY, "Sync not ready for propose") TAOS_DEFINE_ERROR(TSDB_CODE_SYN_INTERNAL_ERROR, "Sync internal error") // wal @@ -450,6 +460,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_CTG_VG_META_MISMATCH, "table meta and vgroup TAOS_DEFINE_ERROR(TSDB_CODE_SCH_STATUS_ERROR, "scheduler status error") TAOS_DEFINE_ERROR(TSDB_CODE_SCH_INTERNAL_ERROR, "scheduler internal error") TAOS_DEFINE_ERROR(TSDB_CODE_SCH_TIMEOUT_ERROR, "Task timeout") +TAOS_DEFINE_ERROR(TSDB_CODE_SCH_JOB_IS_DROPPING, "Job is dropping") TAOS_DEFINE_ERROR(TSDB_CODE_QW_MSG_ERROR, "Invalid msg order") // parser @@ -572,9 +583,14 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TSMA_NO_INDEX_IN_CACHE, "No tsma index in ca TAOS_DEFINE_ERROR(TSDB_CODE_RSMA_INVALID_ENV, "Invalid rsma env") TAOS_DEFINE_ERROR(TSDB_CODE_RSMA_INVALID_STAT, "Invalid rsma state") +//tq +TAOS_DEFINE_ERROR(TSDB_CODE_TQ_NO_COMMITTED_OFFSET, "No committed offset") + TAOS_DEFINE_ERROR(TSDB_CODE_INDEX_REBUILDING, "Index is rebuilding") +TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_INVALID_MSG, "Invalid message") + #ifdef TAOS_ERROR_C }; #endif diff --git a/source/util/src/thash.c b/source/util/src/thash.c index 8bda59bba7e709201729405e5ed0c59c88c8a856..98762d8fb0e4b48907910e6a7e154ed524f57262 100644 --- a/source/util/src/thash.c +++ b/source/util/src/thash.c @@ -56,7 +56,7 @@ typedef struct SHashEntry { } SHashEntry; struct SHashObj { - SHashEntry **hashList; + SHashEntry ** hashList; size_t capacity; // number of slots int64_t size; // number of elements in hash table _hash_fn_t hashFp; // hash function @@ -65,7 +65,7 @@ struct SHashObj { SRWLatch lock; // read-write spin lock SHashLockTypeE type; // lock type bool enableUpdate; // enable update - SArray *pMemBlock; // memory block allocated for SHashEntry + SArray * pMemBlock; // memory block allocated for SHashEntry _hash_before_fn_t callbackFp; // function invoked before return the value to caller }; @@ -633,7 +633,7 @@ void taosHashTableResize(SHashObj *pHashObj) { } int64_t st = taosGetTimestampUs(); - void *pNewEntryList = taosMemoryRealloc(pHashObj->hashList, sizeof(void *) * newCapacity); + void * pNewEntryList = taosMemoryRealloc(pHashObj->hashList, sizeof(void *) * newCapacity); if (pNewEntryList == NULL) { // uDebug("cache resize failed due to out of memory, capacity remain:%zu", pHashObj->capacity); return; @@ -642,7 +642,7 @@ void taosHashTableResize(SHashObj *pHashObj) { pHashObj->hashList = pNewEntryList; size_t inc = newCapacity - pHashObj->capacity; - void *p = taosMemoryCalloc(inc, sizeof(SHashEntry)); + void * p = taosMemoryCalloc(inc, sizeof(SHashEntry)); for (int32_t i = 0; i < inc; ++i) { pHashObj->hashList[i + pHashObj->capacity] = (void *)((char *)p + i * sizeof(SHashEntry)); @@ -653,9 +653,9 @@ void taosHashTableResize(SHashObj *pHashObj) { pHashObj->capacity = newCapacity; for (int32_t idx = 0; idx < pHashObj->capacity; ++idx) { SHashEntry *pe = pHashObj->hashList[idx]; - SHashNode *pNode; - SHashNode *pNext; - SHashNode *pPrev = NULL; + SHashNode * pNode; + SHashNode * pNext; + SHashNode * pPrev = NULL; if (pe->num == 0) { assert(pe->next == NULL); diff --git a/source/util/src/ttrace.c b/source/util/src/ttrace.c new file mode 100644 index 0000000000000000000000000000000000000000..f183fd58fd6acd7b4e111575de1414cc4776c9d0 --- /dev/null +++ b/source/util/src/ttrace.c @@ -0,0 +1,79 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +#include "ttrace.h" +#include "taos.h" +#include "thash.h" +#include "tuuid.h" + +// clang-format off +//static TdThreadOnce init = PTHREAD_ONCE_INIT; +//static void * ids = NULL; +//static TdThreadMutex mtx; +// +//void traceInit() { +// ids = taosHashInit(4096, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_ENTRY_LOCK); +// taosThreadMutexInit(&mtx, NULL); +//} +//void traceCreateEnv() { +// taosThreadOnce(&init, traceInit); +//} +//void traceDestroyEnv() { +// taosThreadMutexDestroy(&mtx); +// taosHashCleanup(ids); +//} +// +//STraceId traceInitId(STraceSubId *h, STraceSubId *l) { +// STraceId id = *h; +// id = ((id << 32) & 0xFFFFFFFF) | ((*l) & 0xFFFFFFFF); +// return id; +//} +//void traceId2Str(STraceId *id, char *buf) { +// int32_t f = (*id >> 32) & 0xFFFFFFFF; +// int32_t s = (*id) & 0xFFFFFFFF; +// sprintf(buf, "%d:%d", f, s); +//} +// +//void traceSetSubId(STraceId *id, STraceSubId *subId) { +// int32_t parent = ((*id >> 32) & 0xFFFFFFFF); +// taosThreadMutexLock(&mtx); +// taosHashPut(ids, subId, sizeof(*subId), &parent, sizeof(parent)); +// taosThreadMutexUnlock(&mtx); +//} +// +//STraceSubId traceGetParentId(STraceId *id) { +// int32_t parent = ((*id >> 32) & 0xFFFFFFFF); +// taosThreadMutexLock(&mtx); +// STraceSubId *p = taosHashGet(ids, (void *)&parent, sizeof(parent)); +// parent = *p; +// taosThreadMutexUnlock(&mtx); +// +// return parent; +//} +// +//STraceSubId traceGenSubId() { +// return tGenIdPI32(); +//} +//void traceSetRootId(STraceId *traceid, int64_t rootId) { +// traceId->rootId = rootId; +//} +//int64_t traceGetRootId(STraceId *traceId); +// +//void traceSetMsgId(STraceId *traceid, int64_t msgId); +//int64_t traceGetMsgId(STraceId *traceid); +// +//char *trace2Str(STraceId *id); +// +//void traceSetSubId(STraceId *id, int32_t *subId); +// clang-format on diff --git a/source/util/src/tuuid.c b/source/util/src/tuuid.c index 1f3cbd957321e81a2cc658c8456fbfee6098b72b..9101aec949873eb976517581c6790e1bc2dac3b9 100644 --- a/source/util/src/tuuid.c +++ b/source/util/src/tuuid.c @@ -49,7 +49,7 @@ int64_t tGenIdPI64(void) { } int64_t id; - + while (true) { int64_t ts = taosGetTimestampMs(); uint64_t pid = taosGetPId(); diff --git a/tests/pytest/util/common.py b/tests/pytest/util/common.py index 7b00e6f331f6053c96ce56b5a79219b6967c6ecd..e158f8edd2e30386afad73f688a25c0f7c56fc46 100644 --- a/tests/pytest/util/common.py +++ b/tests/pytest/util/common.py @@ -11,13 +11,20 @@ # -*- coding: utf-8 -*- +from collections import defaultdict import random import string -from util.sql import tdSql -from util.dnodes import tdDnodes import requests import time import socket + +import taos +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * +from util.common import * + class TDCom: def init(self, conn, logSql): tdSql.init(conn.cursor(), logSql) @@ -170,4 +177,122 @@ class TDCom: def close(self): self.cursor.close() + def create_database(self,tsql, dbName='test',dropFlag=1,precision="ms", **kwargs): + if dropFlag == 1: + tsql.execute("drop database if exists %s"%(dbName)) + ''' + vgroups replica precision strict wal fsync comp cachelast single_stable buffer pagesize pages minrows maxrows duration keep retentions + ''' + sqlString = f'create database if not exists {dbName} precision "{precision}" vgroups 4' + if len(kwargs) > 0: + dbParams = "" + for param, value in kwargs.items(): + dbParams += f'{param} {value} ' + sqlString += f'{dbParams}' + + tsql.execute(sqlString) + tdLog.debug("complete to create database %s"%(dbName)) + return + + def create_stable(self,tsql, dbName,stbName,columnDict,tagDict): + colSchema = '' + for i in range(columnDict['int']): + colSchema += ', c%d int'%i + tagSchema = '' + for i in range(tagDict['int']): + if i > 0: + tagSchema += ',' + tagSchema += 't%d int'%i + + tsql.execute("create table if not exists %s.%s (ts timestamp %s) tags(%s)"%(dbName, stbName, colSchema, tagSchema)) + tdLog.debug("complete to create %s.%s" %(dbName, stbName)) + return + + def create_ctables(self,tsql, dbName,stbName,ctbNum,tagDict): + tsql.execute("use %s" %dbName) + tagsValues = '' + for i in range(tagDict['int']): + if i > 0: + tagsValues += ',' + tagsValues += '%d'%i + + pre_create = "create table" + sql = pre_create + #tdLog.debug("doing create one stable %s and %d child table in %s ..." %(stbname, count ,dbname)) + for i in range(ctbNum): + sql += " %s_%d using %s tags(%s)"%(stbName,i,stbName,tagsValues) + if (i > 0) and (i%100 == 0): + tsql.execute(sql) + sql = pre_create + if sql != pre_create: + tsql.execute(sql) + + tdLog.debug("complete to create %d child tables in %s.%s" %(ctbNum, dbName, stbName)) + return + + def insert_data(self,tsql,dbName,stbName,ctbNum,rowsPerTbl,batchNum,startTs=0): + tdLog.debug("start to insert data ............") + tsql.execute("use %s" %dbName) + pre_insert = "insert into " + sql = pre_insert + if startTs == 0: + t = time.time() + startTs = int(round(t * 1000)) + #tdLog.debug("doing insert data into stable:%s rows:%d ..."%(stbName, allRows)) + for i in range(ctbNum): + sql += " %s_%d values "%(stbName,i) + for j in range(rowsPerTbl): + sql += "(%d, %d, %d)"%(startTs + j, j, j) + if (j > 0) and ((j%batchNum == 0) or (j == rowsPerTbl - 1)): + tsql.execute(sql) + if j < rowsPerTbl - 1: + sql = "insert into %s_%d values " %(stbName,i) + else: + sql = "insert into " + #end sql + if sql != pre_insert: + #print("insert sql:%s"%sql) + tsql.execute(sql) + tdLog.debug("insert data ............ [OK]") + return + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def getClientCfgPath(self): + buildPath = self.getBuildPath() + if (buildPath == ""): + tdLog.exit("taosd not found!") + else: + tdLog.info("taosd found in %s" % buildPath) + cfgPath = buildPath + "/../sim/psim/cfg" + tdLog.info("cfgPath: %s" % cfgPath) + return cfgPath + + def newcur(self,host='localhost',port=6030,user='root',password='taosdata'): + cfgPath = self.getClientCfgPath() + con=taos.connect(host=host, user=user, password=password, config=cfgPath, port=port) + cur=con.cursor() + print(cur) + return cur + + def newTdSql(self, host='localhost',port=6030,user='root',password='taosdata'): + newTdSql = TDSql() + cur = self.newcur(host=host,port=port,user=user,password=password) + newTdSql.init(cur, False) + return newTdSql + tdCom = TDCom() diff --git a/tests/script/jenkins/basic.txt b/tests/script/jenkins/basic.txt index 6fe1251ed7af715735be1bd708375f42640b1805..41bbfaa9388e810f2df6e70631aade4d06f8ee3f 100644 --- a/tests/script/jenkins/basic.txt +++ b/tests/script/jenkins/basic.txt @@ -72,13 +72,15 @@ ./test.sh -f tsim/stream/basic0.sim ./test.sh -f tsim/stream/basic1.sim ./test.sh -f tsim/stream/basic2.sim -# ./test.sh -f tsim/stream/distributeInterval0.sim +./test.sh -f tsim/stream/distributeInterval0.sim +# ./test.sh -f tsim/stream/distributesession0.sim # ./test.sh -f tsim/stream/session0.sim # ./test.sh -f tsim/stream/session1.sim # ./test.sh -f tsim/stream/state0.sim -# ./test.sh -f tsim/stream/triggerInterval0.sim +./test.sh -f tsim/stream/triggerInterval0.sim # ./test.sh -f tsim/stream/triggerSession0.sim -# ./test.sh -f tsim/stream/partitionby.sim +./test.sh -f tsim/stream/partitionby.sim +./test.sh -f tsim/stream/schedSnode.sim # ---- transaction @@ -132,7 +134,7 @@ #./test.sh -f tsim/mnode/basic1.sim -m # --- sma -./test.sh -f tsim/sma/tsmaCreateInsertData.sim +./test.sh -f tsim/sma/tsmaCreateInsertQuery.sim ./test.sh -f tsim/sma/rsmaCreateInsertQuery.sim # --- valgrind diff --git a/tests/script/sh/copy_udf.bat b/tests/script/sh/copy_udf.bat new file mode 100644 index 0000000000000000000000000000000000000000..5144cf8d25dba767774ba7457b77c5652b8eaa0b --- /dev/null +++ b/tests/script/sh/copy_udf.bat @@ -0,0 +1,23 @@ +@echo off + +echo Executing copy_udf.bat +set SCRIPT_DIR=%cd% +echo SCRIPT_DIR: %SCRIPT_DIR% + +cd ..\..\.. +set TAOS_DIR=%cd% +echo find udf library in %TAOS_DIR% +set UDF1_DIR=%TAOS_DIR%\debug\build\lib\udf1.dll +set UDF2_DIR=%TAOS_DIR%\debug\build\lib\udf2.dll + +echo %UDF1_DIR% +echo %UDF2_DIR% + +set UDF_TMP=C:\Windows\Temp\udf +rm -rf %UDF_TMP% +mkdir %UDF_TMP% + +echo Copy udf shared library files to %UDF_TMP% + +cp %UDF1_DIR% %UDF_TMP% +cp %UDF2_DIR% %UDF_TMP% diff --git a/tests/script/tsim/db/alter_option.sim b/tests/script/tsim/db/alter_option.sim index 4351ee5cb1415d20f74daee4fa18b97471bb9654..fede960a6ae1709f8fc267856b2a243f6dd349f4 100644 --- a/tests/script/tsim/db/alter_option.sim +++ b/tests/script/tsim/db/alter_option.sim @@ -92,7 +92,7 @@ endi if $data5_db != no_strict then # strict return -1 endi -if $data6_db != 345600 then # duration +if $data6_db != 345600m then # duration return -1 endi if $data7_db != 1440000m,1440000m,1440000m then # keep diff --git a/tests/script/tsim/db/basic6.sim b/tests/script/tsim/db/basic6.sim index 64103b5dac39530b6ba6c8ef61cff9421a3421cc..7525fe20870c7a25d64b0c77d60799a7c58c2001 100644 --- a/tests/script/tsim/db/basic6.sim +++ b/tests/script/tsim/db/basic6.sim @@ -34,7 +34,7 @@ endi if $data24 != 1 then return -1 endi -if $data26 != 2880 then +if $data26 != 2880m then return -1 endi if $data27 != 14400m,14400m,14400m then @@ -78,7 +78,7 @@ endi if $data24 != 1 then return -1 endi -if $data26 != 21600 then +if $data26 != 21600m then return -1 endi diff --git a/tests/script/tsim/db/create_all_options.sim b/tests/script/tsim/db/create_all_options.sim index 284875ee08489b3e745dcbf819fb9170ede49ccc..efe7ff99cfdd7bfb79c614dbaafd021b004cdfdc 100644 --- a/tests/script/tsim/db/create_all_options.sim +++ b/tests/script/tsim/db/create_all_options.sim @@ -113,7 +113,7 @@ endi if $data5_db != no_strict then # strict return -1 endi -if $data6_db != 14400 then # duration +if $data6_db != 14400m then # duration return -1 endi if $data7_db != 5256000m,5256000m,5256000m then # keep diff --git a/tests/script/tsim/dnode/redistribute_vgroup_replica3_v1_leader.sim b/tests/script/tsim/dnode/redistribute_vgroup_replica3_v1_leader.sim index 7b52b5130614f6780d20a37436f343fa6a46ba96..e9dd82cad9a09b8910218d116c3269578a8b04c2 100644 --- a/tests/script/tsim/dnode/redistribute_vgroup_replica3_v1_leader.sim +++ b/tests/script/tsim/dnode/redistribute_vgroup_replica3_v1_leader.sim @@ -53,31 +53,10 @@ endi if $data(4)[4] != ready then goto step1 endi -#if $data(5)[4] != ready then -# goto step1 -#endi print =============== step2: create db sql create database d1 vgroups 1 replica 3 -# dnode not exist -sql_error redistribute vgroup 3 dnode 6 dnode 3 dnode 4 -# vgroup not exist -sql_error redistribute vgroup 3 dnode 5 dnode 3 dnode 4 -# un changed -sql_error redistribute vgroup 2 dnode 2 dnode 3 dnode 4 -# no enought vnodes -sql_error redistribute vgroup 2 dnode 1 dnode 3 dnode 4 -# offline vnodes -sql_error redistribute vgroup 2 dnode 5 dnode 3 dnode 4 -# Invalid replica -sql_error redistribute vgroup 2 dnode 5 -sql_error redistribute vgroup 2 dnode 5 dnode 3 -sql_error redistribute vgroup 2 dnode 2 dnode 3 -sql_error redistribute vgroup 2 dnode 2 dnode 2 -sql_error redistribute vgroup 3 dnode 2 dnode 2 -sql_error redistribute vgroup 2 dnode 2 dnode 2 dnode 3 - system sh/exec.sh -n dnode5 -s start $x = 0 step2: diff --git a/tests/script/tsim/insert/update0.sim b/tests/script/tsim/insert/update0.sim index 0ba3e98c913e1c37c50c351bed7d7385a1cad0d3..ed74188bcb9b673f008e714877e9b9caeae88629 100644 --- a/tests/script/tsim/insert/update0.sim +++ b/tests/script/tsim/insert/update0.sim @@ -9,7 +9,7 @@ sql create database d0 keep 365000d,365000d,365000d sql use d0 print =============== create super table and register rsma -sql create table if not exists stb (ts timestamp, c1 int) tags (city binary(20),district binary(20)) rollup(min) file_factor 0.1; +sql create table if not exists stb (ts timestamp, c1 int) tags (city binary(20),district binary(20)) rollup(min); sql show stables if $rows != 1 then diff --git a/tests/script/tsim/mnode/basic5.sim b/tests/script/tsim/mnode/basic5.sim index 399dced16909586443a2bcfc151d5b525904c161..fc591aa25d3677ebfaf6e982555ab09512badad8 100644 --- a/tests/script/tsim/mnode/basic5.sim +++ b/tests/script/tsim/mnode/basic5.sim @@ -3,6 +3,10 @@ system sh/deploy.sh -n dnode1 -i 1 system sh/deploy.sh -n dnode2 -i 2 system sh/deploy.sh -n dnode3 -i 3 system sh/deploy.sh -n dnode4 -i 4 +system sh/cfg.sh -n dnode1 -c transPullupInterval -v 1 +system sh/cfg.sh -n dnode2 -c transPullupInterval -v 1 +system sh/cfg.sh -n dnode3 -c transPullupInterval -v 1 +system sh/cfg.sh -n dnode4 -c transPullupInterval -v 1 system sh/exec.sh -n dnode1 -s start sql connect @@ -260,7 +264,7 @@ $x = 0 step92: $x = $x + 1 sleep 1000 - if $x == 10 then + if $x == 20 then return -1 endi sql show mnodes diff --git a/tests/script/tsim/query/udf.sim b/tests/script/tsim/query/udf.sim index 93cae4e3912cab0b5c36e60d28743f0c10f1e45a..d9821a04955e6d4f5f80db84afa8531762432672 100644 --- a/tests/script/tsim/query/udf.sim +++ b/tests/script/tsim/query/udf.sim @@ -19,8 +19,14 @@ sql show databases; sql create table t (ts timestamp, f int); sql insert into t values(now, 1)(now+1s, 2); -sql create function udf1 as '/tmp/udf/libudf1.so' outputtype int bufSize 8; -sql create aggregate function udf2 as '/tmp/udf/libudf2.so' outputtype double bufSize 8; +system_content printf %OS% +if $system_content == Windows_NT then + sql create function udf1 as 'C:\\Windows\\Temp\\udf1.dll' outputtype int bufSize 8; + sql create aggregate function udf2 as 'C:\\Windows\\Temp\\udf2.dll' outputtype double bufSize 8; +else + sql create function udf1 as '/tmp/udf/libudf1.so' outputtype int bufSize 8; + sql create aggregate function udf2 as '/tmp/udf/libudf2.so' outputtype double bufSize 8; +endi sql show functions; if $rows != 2 then return -1 diff --git a/tests/script/tsim/show/basic.sim b/tests/script/tsim/show/basic.sim index 95201bc48e0db1b57471039c80169ccdf4e30094..f23d75d78b432d5f69391b18a5bb0318e93af08b 100644 --- a/tests/script/tsim/show/basic.sim +++ b/tests/script/tsim/show/basic.sim @@ -98,7 +98,7 @@ if $rows != 1 then endi #sql select * from information_schema.`streams` sql select * from information_schema.user_tables -if $rows != 28 then +if $rows != 29 then return -1 endi #sql select * from information_schema.user_table_distributed @@ -196,7 +196,7 @@ if $rows != 1 then endi #sql select * from performance_schema.`streams` sql select * from information_schema.user_tables -if $rows != 28 then +if $rows != 29 then return -1 endi #sql select * from information_schema.user_table_distributed @@ -210,4 +210,4 @@ if $rows != 3 then endi system sh/exec.sh -n dnode1 -s stop -x SIGINT -system sh/exec.sh -n dnode2 -s stop -x SIGINT \ No newline at end of file +system sh/exec.sh -n dnode2 -s stop -x SIGINT diff --git a/tests/script/tsim/sma/rsmaCreateInsertQuery.sim b/tests/script/tsim/sma/rsmaCreateInsertQuery.sim index f929dda18cb1b287c3ffe05487464624ff0eebc5..fb3503c84131f783485f50e1e420a91dd3e26ae5 100644 --- a/tests/script/tsim/sma/rsmaCreateInsertQuery.sim +++ b/tests/script/tsim/sma/rsmaCreateInsertQuery.sim @@ -5,11 +5,11 @@ sleep 50 sql connect print =============== create database with retentions -sql create database d0 retentions 15s:7d,1m:21d,15m:365d; +sql create database d0 retentions 5s:7d,10s:21d,15s:365d; sql use d0 print =============== create super table and register rsma -sql create table if not exists stb (ts timestamp, c1 int) tags (city binary(20),district binary(20)) rollup(min) file_factor 0.1; +sql create table if not exists stb (ts timestamp, c1 int) tags (city binary(20),district binary(20)) rollup(min); sql show stables if $rows != 1 then @@ -29,6 +29,8 @@ sql insert into ct1 values(now, 10); sql insert into ct1 values(now+1s, 1); sql insert into ct1 values(now+2s, 100); +print =============== wait maxdelay 15+1 seconds for results +sleep 16000 print =============== select * from retention level 2 from memory sql select * from ct1; diff --git a/tests/script/tsim/sma/tsmaCreateInsertData.sim b/tests/script/tsim/sma/tsmaCreateInsertData.sim deleted file mode 100644 index 7a8cc2a555e4c8c3c044d3674ab28b8cd97014d3..0000000000000000000000000000000000000000 --- a/tests/script/tsim/sma/tsmaCreateInsertData.sim +++ /dev/null @@ -1,56 +0,0 @@ -system sh/stop_dnodes.sh -system sh/deploy.sh -n dnode1 -i 1 -system sh/exec.sh -n dnode1 -s start -sleep 50 -sql connect - -print =============== create database -sql create database d1 vgroups 1 -sql use d1 - -print =============== create super table, include column type for count/sum/min/max/first -sql create table if not exists stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned) - -sql show stables -if $rows != 1 then - return -1 -endi - -print =============== create child table -sql create table ct1 using stb tags(1000) - -sql show tables -if $rows != 1 then - return -1 -endi - -print =============== insert data, mode1: one row one table in sql -sql insert into ct1 values(now+0s, 10, 2.0, 3.0) -sql insert into ct1 values(now+1s, 11, 2.1, 3.1)(now+2s, -12, -2.2, -3.2)(now+3s, -13, -2.3, -3.3) - - -print =============== create sma index from super table -sql create sma index sma_index_name1 on stb function(max(c1),max(c2),min(c1)) interval(5m,10s) sliding(2m) -print $data00 $data01 $data02 $data03 - -print =============== trigger stream to execute sma aggr task and insert sma data into sma store -sql insert into ct1 values(now+5s, 20, 20.0, 30.0) -#=================================================================== - -print =============== show streams ================================ -sql show streams; -print $data00 $data01 $data02 - -if $data00 != d1 then - return -1 -endi - -print =============== select * from ct1 from memory -sql select * from ct1; -print $data00 $data01 -if $rows != 5 then - print rows $rows != 5 - return -1 -endi - -system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/sma/tsmaCreateInsertQuery.sim b/tests/script/tsim/sma/tsmaCreateInsertQuery.sim new file mode 100644 index 0000000000000000000000000000000000000000..3a1ed62a7203566ae2e8c4c43df4bee11e5c9aa3 --- /dev/null +++ b/tests/script/tsim/sma/tsmaCreateInsertQuery.sim @@ -0,0 +1,181 @@ +system sh/stop_dnodes.sh +system sh/deploy.sh -n dnode1 -i 1 +system sh/exec.sh -n dnode1 -s start +sleep 50 +sql connect + +print =============== create database +sql create database d1 vgroups 1 +sql use d1 + +print =============== create super table, include column type for count/sum/min/max/first +sql create table if not exists stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned) + +sql show stables +if $rows != 1 then + return -1 +endi + +print =============== create child table +sql create table ct1 using stb tags(1000) + +sql show tables +if $rows != 1 then + return -1 +endi + +print =============== insert data, mode1: one row one table in sql +sql insert into ct1 values(now+0s, 10, 2.0, 3.0) +sql insert into ct1 values(now+1s, 11, 2.1, 3.1)(now+2s, -12, -2.2, -3.2)(now+3s, -13, -2.3, -3.3) + + +print =============== create sma index from super table +sql create sma index sma_index_name1 on stb function(max(c1),max(c2),min(c1)) interval(5m,10s) sliding(5m) +print $data00 $data01 $data02 $data03 + +print =============== trigger stream to execute sma aggr task and insert sma data into sma store +sql insert into ct1 values(now+5s, 20, 20.0, 30.0) +#=================================================================== + +print =============== show streams ================================ +sql show streams; +print $data00 $data01 $data02 + +if $data00 != d1 then + return -1 +endi + +print =============== select * from ct1 from memory +sql select * from ct1; +print $data00 $data01 $data02 $data03 +print $data10 $data11 $data12 $data13 +print $data20 $data21 $data22 $data23 +print $data30 $data31 $data32 $data33 +print $data40 $data41 $data42 $data43 +if $rows != 5 then + print rows $rows != 5 + return -1 +endi + +print =============== select * from stb from memory in designated vgroup +sql select _wstartts, _wendts, min(c1),max(c2),max(c1) from stb interval(5m,10s) sliding(5m); +print $data00 $data01 $data02 $data03 $data04 +if $rows != 1 then + print rows $rows != 1 + return -1 +endi + +if $data02 != -13 then + print data02 $data02 != -13 + return -1 +endi + +if $data03 != 20.00000 then + print data03 $data03 != 20.00000 + return -1 +endi + +if $data04 != 20 then + print data04 $data04 != 20 + return -1 +endi + +print =============== select * from stb from memory in common vgroups +sql select _wstartts, _wendts, min(c1),max(c2),max(c1),max(c3) from stb interval(5m,10s) sliding(5m); +print $data00 $data01 $data02 $data03 $data04 $data05 +if $rows != 1 then + print rows $rows != 1 + return -1 +endi + +if $data02 != -13 then + print data02 $data02 != -13 + return -1 +endi + +if $data03 != 20.00000 then + print data03 $data03 != 20.00000 + return -1 +endi + +if $data04 != 20 then + print data04 $data04 != 20 + return -1 +endi + +if $data05 != 30.000000000 then + print data05 $data05 != 30.000000000 + return -1 +endi + + + +system sh/exec.sh -n dnode1 -s stop -x SIGINT +system sh/exec.sh -n dnode1 -s start +sleep 50 + + +print =============== select * from ct1 from file +sql select * from ct1; +print $data00 $data01 $data02 $data03 +print $data10 $data11 $data12 $data13 +print $data20 $data21 $data22 $data23 +print $data30 $data31 $data32 $data33 +print $data40 $data41 $data42 $data43 +if $rows != 5 then + print rows $rows != 5 + return -1 +endi + +print =============== select * from stb from file in designated vgroup +sql select _wstartts, _wendts, min(c1),max(c2),max(c1) from stb interval(5m,10s) sliding(5m); +print $data00 $data01 $data02 $data03 $data04 +if $rows != 1 then + print rows $rows != 1 + return -1 +endi + +if $data02 != -13 then + print data02 $data02 != -13 + return -1 +endi + +if $data03 != 20.00000 then + print data03 $data03 != 20.00000 + return -1 +endi + +if $data04 != 20 then + print data04 $data04 != 20 + return -1 +endi + +print =============== select * from stb from file in common vgroups +sql select _wstartts, _wendts, min(c1),max(c2),max(c1),max(c3) from stb interval(5m,10s) sliding(5m); +print $data00 $data01 $data02 $data03 $data04 $data05 +if $rows != 1 then + print rows $rows != 1 + return -1 +endi + +if $data02 != -13 then + print data02 $data02 != -13 + return -1 +endi + +if $data03 != 20.00000 then + print data03 $data03 != 20.00000 + return -1 +endi + +if $data04 != 20 then + print data04 $data04 != 20 + return -1 +endi + +if $data05 != 30.000000000 then + print data05 $data05 != 30.000000000 + return -1 +endi + + diff --git a/tests/script/tsim/stable/alter_comment.sim b/tests/script/tsim/stable/alter_comment.sim index cfcbb9a1daa046c894bbfe47f4684ded5faf79a6..19f4858585662222efea2b339301a5afd3e721d6 100644 --- a/tests/script/tsim/stable/alter_comment.sim +++ b/tests/script/tsim/stable/alter_comment.sim @@ -159,7 +159,7 @@ sql alter table db.stb rename tag t1 tx print ========== alter common sql alter table db.stb comment 'abcde' ; -sql alter table db.stb ttl 10 ; +sql_error alter table db.stb ttl 10 ; sql show db.stables; if $data[0][6] != abcde then diff --git a/tests/script/tsim/stream/distributeInterval0.sim b/tests/script/tsim/stream/distributeInterval0.sim index b720272116a16f1486cde9d81f1e67ca51c09b2d..91ce49bc8c1e4dca2a51f7d0208bfc1724b06352 100644 --- a/tests/script/tsim/stream/distributeInterval0.sim +++ b/tests/script/tsim/stream/distributeInterval0.sim @@ -208,4 +208,4 @@ if $data11 != 2 then goto loop2 endi -system sh/stop_dnodes.sh \ No newline at end of file +system sh/stop_dnodes.sh diff --git a/tests/script/tsim/stream/distributesession0.sim b/tests/script/tsim/stream/distributesession0.sim new file mode 100644 index 0000000000000000000000000000000000000000..a165b86edd135ef17db466e216472eec603af207 --- /dev/null +++ b/tests/script/tsim/stream/distributesession0.sim @@ -0,0 +1,58 @@ +system sh/stop_dnodes.sh +system sh/deploy.sh -n dnode1 -i 1 +system sh/deploy.sh -n dnode2 -i 2 + +system sh/exec.sh -n dnode1 -s start +sleep 50 +sql connect + +sql create dnode $hostname2 port 7200 + +system sh/exec.sh -n dnode2 -s start + +sql create database test vgroups 4; +sql use test; +sql create stable st(ts timestamp,a int,b int,c int) tags(ta int,tb int,tc int); +sql create table ts1 using st tags(1,1,1); +sql create table ts2 using st tags(2,2,2); +sql create stream stream_t1 trigger at_once into streamtST as select _wstartts, count(*) c1, sum(a) c2 , max(b) c3 from st session(ts, 10s) ; + +sleep 1000 + +sql insert into ts1 values(1648791211000,1,1,1) (1648791211005,1,1,1); +sql insert into ts2 values(1648791221004,1,2,3) (1648791221008,2,2,3); +sql insert into ts1 values(1648791211005,1,1,1); +sql insert into ts2 values(1648791221006,5,5,5) (1648791221007,5,5,5); +sql insert into ts2 values(1648791221008,5,5,5) (1648791221008,5,5,5)(1648791221006,5,5,5); +sql insert into ts1 values(1648791231000,1,1,1) (1648791231002,1,1,1) (1648791231006,1,1,1); +sql insert into ts1 values(1648791211000,6,6,6) (1648791231002,2,2,2); +sql insert into ts1 values(1648791211002,7,7,7); +sql insert into ts1 values(1648791211002,7,7,7) ts2 values(1648791221008,5,5,5) ; + +$loop_count = 0 +loop1: +sql select * from streamtST; + +sleep 300 +$loop_count = $loop_count + 1 +if $loop_count == 10 then + return -1 +endi + +# row 0 +if $data01 != 10 then + print =====data01=$data01 + goto loop1 +endi + +if $data02 != 34 then + print =====data02=$data02 + goto loop1 +endi + +if $data03 != 7 then + print ======$data03 + return -1 +endi + +system sh/stop_dnodes.sh diff --git a/tests/script/tsim/stream/schedSnode.sim b/tests/script/tsim/stream/schedSnode.sim new file mode 100644 index 0000000000000000000000000000000000000000..dbf714a96fbabce40bccb30c2245b918b22c971d --- /dev/null +++ b/tests/script/tsim/stream/schedSnode.sim @@ -0,0 +1,173 @@ +system sh/stop_dnodes.sh +system sh/deploy.sh -n dnode1 -i 1 +system sh/deploy.sh -n dnode2 -i 2 + +system sh/exec.sh -n dnode1 -s start +sleep 50 +sql connect + +sql create snode on dnode 1 + +sql create database test vgroups 1; +sql create database target vgroups 1; +sql use test; +sql create stable st(ts timestamp, a int, b int , c int, d double) tags(ta int,tb int,tc int); +sql create table ts1 using st tags(1,1,1); +sql create table ts2 using st tags(2,2,2); +sql create table ts3 using st tags(3,2,2); +sql create table ts4 using st tags(4,2,2); +sql create stream stream_t1 trigger at_once into target.streamtST1 as select _wstartts, count(*) c1, count(d) c2 , sum(a) c3 , max(b) c4, min(c) c5 from st interval(10s); + +sleep 1000 + +sql insert into ts1 values(1648791213001,1,12,3,1.0); +sql insert into ts2 values(1648791213001,1,12,3,1.0); + +sql insert into ts3 values(1648791213001,1,12,3,1.0); +sql insert into ts4 values(1648791213001,1,12,3,1.0); + +sql insert into ts1 values(1648791213002,NULL,NULL,NULL,NULL); +sql insert into ts2 values(1648791213002,NULL,NULL,NULL,NULL); + +sql insert into ts3 values(1648791213002,NULL,NULL,NULL,NULL); +sql insert into ts4 values(1648791213002,NULL,NULL,NULL,NULL); + +sql insert into ts1 values(1648791223002,2,2,3,1.1); +sql insert into ts1 values(1648791233003,3,2,3,2.1); +sql insert into ts2 values(1648791243004,4,2,43,73.1); +sql insert into ts1 values(1648791213002,24,22,23,4.1); +sql insert into ts1 values(1648791243005,4,20,3,3.1); +sql insert into ts2 values(1648791243006,4,2,3,3.1) (1648791243007,4,2,3,3.1) ; +sql insert into ts1 values(1648791243008,4,2,30,3.1) (1648791243009,4,2,3,3.1) (1648791243010,4,2,3,3.1) ; +sql insert into ts2 values(1648791243011,4,2,3,3.1) (1648791243012,34,32,33,3.1) (1648791243013,4,2,3,3.1) (1648791243014,4,2,13,3.1); +sql insert into ts1 values(1648791243005,4,42,3,3.1) (1648791243003,4,2,33,3.1) (1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) ; +sql insert into ts2 values(1648791243005,4,42,3,3.1) (1648791243003,4,2,33,3.1) (1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) (1648791233004,13,12,13,2.1) ; +sql insert into ts1 values(1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) ; + +sql insert into ts3 values(1648791223002,2,2,3,1.1); +sql insert into ts4 values(1648791233003,3,2,3,2.1); +sql insert into ts3 values(1648791243004,4,2,43,73.1); +sql insert into ts4 values(1648791213002,24,22,23,4.1); +sql insert into ts3 values(1648791243005,4,20,3,3.1); +sql insert into ts4 values(1648791243006,4,2,3,3.1) (1648791243007,4,2,3,3.1) ; +sql insert into ts3 values(1648791243008,4,2,30,3.1) (1648791243009,4,2,3,3.1) (1648791243010,4,2,3,3.1) ; +sql insert into ts4 values(1648791243011,4,2,3,3.1) (1648791243012,34,32,33,3.1) (1648791243013,4,2,3,3.1) (1648791243014,4,2,13,3.1); +sql insert into ts3 values(1648791243005,4,42,3,3.1) (1648791243003,4,2,33,3.1) (1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) ; +sql insert into ts4 values(1648791243005,4,42,3,3.1) (1648791243003,4,2,33,3.1) (1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) (1648791233004,13,12,13,2.1) ; +sql insert into ts3 values(1648791243006,4,2,3,3.1) (1648791213001,1,52,13,1.0) (1648791223001,22,22,83,1.1) ; + +$loop_count = 0 +loop1: +sql select * from target.streamtST1; + +sleep 300 +$loop_count = $loop_count + 1 +if $loop_count == 10 then + return -1 +endi + +# row 0 +if $data01 != 8 then + print =====data01=$data01 + goto loop1 +endi + +if $data02 != 4 then + print =====data02=$data02 + goto loop1 +endi + +if $data03 != 4 then + print ======$data03 + return -1 +endi + +if $data04 != 52 then + print ======$data04 + return -1 +endi + +if $data05 != 13 then + print ======$data05 + return -1 +endi + +# row 1 +if $data11 != 6 then + print =====data11=$data11 + goto loop1 +endi + +if $data12 != 6 then + print =====data12=$data12 + goto loop1 +endi + +if $data13 != 92 then + print ======$data13 + return -1 +endi + +if $data14 != 22 then + print ======$data14 + return -1 +endi + +if $data15 != 3 then + print ======$data15 + return -1 +endi + +# row 2 +if $data21 != 4 then + print =====data21=$data21 + goto loop1 +endi + +if $data22 != 4 then + print =====data22=$data22 + goto loop1 +endi + +if $data23 != 32 then + print ======$data23 + return -1 +endi + +if $data24 != 12 then + print ======$data24 + return -1 +endi + +if $data25 != 3 then + print ======$data25 + return -1 +endi + +# row 3 +if $data31 != 30 then + print =====data31=$data31 + goto loop1 +endi + +if $data32 != 30 then + print =====data32=$data32 + goto loop1 +endi + +if $data33 != 180 then + print ======$data33 + return -1 +endi + +if $data34 != 42 then + print ======$data34 + return -1 +endi + +if $data35 != 3 then + print ======$data35 + return -1 +endi + +sql select _wstartts, count(*) c1, count(d) c2 , sum(a) c3 , max(b) c4, min(c) c5, avg(d) from st interval(10s); diff --git a/tests/script/tsim/stream/session0.sim b/tests/script/tsim/stream/session0.sim index 2a737578bca27fa1fc0087a0d517efb66767c117..24ffe76d2b34a5404c6fc354faddeff0c53c2ed8 100644 --- a/tests/script/tsim/stream/session0.sim +++ b/tests/script/tsim/stream/session0.sim @@ -231,9 +231,10 @@ sql use test3; sql create table t1(ts timestamp, a int, b int , c int, d double); sql create stream streams3 trigger at_once watermark 1d into streamt3 as select _wstartts, min(b), a,c from t1 session(ts,10s); sql create stream streams4 trigger at_once watermark 1d into streamt4 as select _wstartts, max(b), a,c from t1 session(ts,10s); -sql create stream streams5 trigger at_once watermark 1d into streamt5 as select _wstartts, top(b,3), a,c from t1 session(ts,10s); -sql create stream streams6 trigger at_once watermark 1d into streamt6 as select _wstartts, bottom(b,3), a,c from t1 session(ts,10s); -sql create stream streams7 trigger at_once watermark 1d into streamt7 as select _wstartts, spread(a), elapsed(ts), hyperloglog(a) from t1 session(ts,10s); +# sql create stream streams5 trigger at_once watermark 1d into streamt5 as select _wstartts, top(b,3), a,c from t1 session(ts,10s); +# sql create stream streams6 trigger at_once watermark 1d into streamt6 as select _wstartts, bottom(b,3), a,c from t1 session(ts,10s); +# sql create stream streams7 trigger at_once watermark 1d into streamt7 as select _wstartts, spread(a), elapsed(ts), hyperloglog(a) from t1 session(ts,10s); +sql create stream streams7 trigger at_once watermark 1d into streamt7 as select _wstartts, spread(a), hyperloglog(a) from t1 session(ts,10s); sql create stream streams8 trigger at_once watermark 1d into streamt8 as select _wstartts, histogram(a,"user_input", "[1,3,5,7]", 1), histogram(a,"user_input", "[1,3,5,7]", 0) from t1 session(ts,10s); sql insert into t1 values(1648791213001,1,1,1,1.0); sql insert into t1 values(1648791213002,2,3,2,3.4); @@ -269,13 +270,13 @@ if $rows == 0 then goto loop3 endi -sql select * from streamt5; +#sql select * from streamt5; if $rows == 0 then print ======$rows - goto loop3 + # goto loop3 endi -sql select * from streamt6; +# sql select * from streamt6; if $rows == 0 then print ======$rows goto loop3 diff --git a/tests/script/tsim/stream/triggerSession0.sim b/tests/script/tsim/stream/triggerSession0.sim index 69b3dc170459682a1112297a08c4a7a4a5fd5970..48827a64a274b34d935e8baea13a10f64baf617a 100644 --- a/tests/script/tsim/stream/triggerSession0.sim +++ b/tests/script/tsim/stream/triggerSession0.sim @@ -85,7 +85,7 @@ sql insert into t2 values(1648791243003,1,2,3,1.0) (1648791243002,1,2,3,1.0) (16 sleep 500 sql select * from streamt2; if $rows != 3 then - print ======$rows + print =====rows=$rows return -1 endi diff --git a/tests/script/tsim/testsuit.sim b/tests/script/tsim/testsuit.sim index 90210fd436f70ce7d6188b9fd78a0d970e1764c9..1636bb38c800350a2b39a26d5dcd9a25ab56dc23 100644 --- a/tests/script/tsim/testsuit.sim +++ b/tests/script/tsim/testsuit.sim @@ -1,35 +1,35 @@ -#run tsim/user/pass_alter.sim -#run tsim/user/basic1.sim -#run tsim/user/privilege2.sim -#run tsim/user/user_len.sim -#run tsim/user/privilege1.sim -#run tsim/user/pass_len.sim -#run tsim/table/basic1.sim -#run tsim/trans/lossdata1.sim -#run tsim/trans/create_db.sim -#run tsim/stable/alter_metrics.sim -#run tsim/stable/tag_modify.sim -#run tsim/stable/alter_comment.sim -#run tsim/stable/column_drop.sim -#run tsim/stable/column_modify.sim -#run tsim/stable/tag_rename.sim -#run tsim/stable/vnode3.sim -#run tsim/stable/metrics.sim -#run tsim/stable/alter_insert2.sim -#run tsim/stable/show.sim -#run tsim/stable/alter_import.sim -#run tsim/stable/tag_add.sim -#run tsim/stable/tag_drop.sim -#run tsim/stable/column_add.sim -#run tsim/stable/alter_count.sim -#run tsim/stable/values.sim +run tsim/user/pass_alter.sim +run tsim/user/basic1.sim +run tsim/user/privilege2.sim +run tsim/user/user_len.sim +run tsim/user/privilege1.sim +run tsim/user/pass_len.sim +run tsim/table/basic1.sim +run tsim/trans/lossdata1.sim +run tsim/trans/create_db.sim +run tsim/stable/alter_metrics.sim +run tsim/stable/tag_modify.sim +run tsim/stable/alter_comment.sim +run tsim/stable/column_drop.sim +run tsim/stable/column_modify.sim +run tsim/stable/tag_rename.sim +run tsim/stable/vnode3.sim +run tsim/stable/metrics.sim +run tsim/stable/alter_insert2.sim +run tsim/stable/show.sim +run tsim/stable/alter_import.sim +run tsim/stable/tag_add.sim +run tsim/stable/tag_drop.sim +run tsim/stable/column_add.sim +run tsim/stable/alter_count.sim +run tsim/stable/values.sim run tsim/stable/dnode3.sim -#run tsim/stable/alter_insert1.sim -#run tsim/stable/refcount.sim -#run tsim/stable/disk.sim +run tsim/stable/alter_insert1.sim +run tsim/stable/refcount.sim +run tsim/stable/disk.sim run tsim/db/basic1.sim run tsim/db/basic3.sim -#run tsim/db/basic7.sim +run tsim/db/basic7.sim run tsim/db/basic6.sim run tsim/db/create_all_options.sim run tsim/db/basic2.sim diff --git a/tests/system-test/1-insert/alter_stable.py b/tests/system-test/1-insert/alter_stable.py index c92efb403cf6c5bb7e283d176032907e32528607..cd64e3ddfe3b5d58292d98764df0e8b46033e27b 100644 --- a/tests/system-test/1-insert/alter_stable.py +++ b/tests/system-test/1-insert/alter_stable.py @@ -77,6 +77,7 @@ class TDTestCase: tdSql.error(f'alter stable {stbname} modify column c9 double') tdSql.error(f'alter stable {stbname} modify column c10 float') tdSql.error(f'alter stable {stbname} modify column c11 int') + tdSql.error(f'alter stable {stbname} drop tag t0') tdSql.execute(f'drop database {dbname}') def alter_stable_tag_check(self,dbname,stbname,tbname): @@ -126,6 +127,11 @@ class TDTestCase: for i in ['int','unsigned int','float','binary(10)','nchar(10)']: tdSql.error(f'alter stable {stbname} modify tag t8 {i}') tdSql.error(f'alter stable {stbname} modify tag t4 int') + tdSql.error(f'alter stable {stbname} drop column t0') + #!bug TD-16410 + # tdSql.error(f'alter stable {tbname} set tag t1=100 ') + # tdSql.execute(f'create table ntb (ts timestamp,c0 int)') + tdSql.error(f'alter stable ntb add column c2 ') tdSql.execute(f'drop database {dbname}') def run(self): diff --git a/tests/system-test/1-insert/alter_table.py b/tests/system-test/1-insert/alter_table.py index 3c0def86e4f053b941d0cc88144d0ac26db388de..a4e40d1b0b59f556c0307b740ed5333e9c0131e8 100644 --- a/tests/system-test/1-insert/alter_table.py +++ b/tests/system-test/1-insert/alter_table.py @@ -52,8 +52,6 @@ class TDTestCase: tdSql.execute(f'create database if not exists {dbname}') stbname = self.get_long_name(length=3, mode="letters") tbname = self.get_long_name(length=3, mode="letters") - tdLog.info('--------------------------child table tag check--------------------------------------') - tdLog.info(f'-----------------create stable {stbname} and child table {tbname}-------------------') tdSql.execute(f'create stable if not exists {dbname}.{stbname} (col_ts timestamp, c1 int) tags (tag_ts timestamp, t1 tinyint, t2 smallint, t3 int, \ t4 bigint, t5 tinyint unsigned, t6 smallint unsigned, t7 int unsigned, t8 bigint unsigned, t9 float, t10 double, t11 bool,t12 binary(20),t13 nchar(20))') tdSql.execute(f'create table if not exists {dbname}.{tbname} using {dbname}.{stbname} tags(now, 1, 2, 3, 4, 5, 6, 7, 8, 9.9, 10.1, True,"abc123","涛思数据")') @@ -90,13 +88,16 @@ class TDTestCase: tdSql.checkData(0,15,tag_nchar) # bug TD-16211 insert length more than setting binary and nchar - # tag_binary = self.get_long_name(length=21, mode="letters") - # tag_nchar = self.get_long_name(length=21, mode="letters") - # tdSql.error(f'alter table {dbname}.{tbname} set tag t12 = "{tag_binary}"') - # tdSql.error(f'alter table {dbname}.{tbname} set tag t13 = "{tag_nchar}"') - + # error_tag_binary = self.get_long_name(length=21, mode="letters") + # error_tag_nchar = self.get_long_name(length=21, mode="letters") + # tdSql.error(f'alter table {dbname}.{tbname} set tag t12 = "{error_tag_binary}"') + # tdSql.error(f'alter table {dbname}.{tbname} set tag t13 = "{error_tag_nchar}"') + error_tag_binary = self.get_long_name(length=25, mode="letters") + error_tag_nchar = self.get_long_name(length=25, mode="letters") + tdSql.error(f'alter table {dbname}.{tbname} set tag t12 = "{error_tag_binary}"') + tdSql.error(f'alter table {dbname}.{tbname} set tag t13 = "{error_tag_nchar}"') # bug TD-16210 modify binary to nchar - # tdSql.error(f'alter table {dbname}.{tbname} modify tag t12 nchar(10)') + tdSql.error(f'alter table {dbname}.{tbname} modify tag t12 nchar(10)') tdSql.execute(f"drop database {dbname}") def alter_ntb_column_check(self): ''' @@ -125,6 +126,20 @@ class TDTestCase: tdSql.execute(f'alter table {dbname}.{tbname} drop column `c15`') tdSql.query(f'describe {dbname}.{tbname}') tdSql.checkRows(14) + #! TD-16422 + # tdSql.execute(f'alter table {dbname}.{tbname} add column c16 binary(10)') + # tdSql.query(f'describe {dbname}.{tbname}') + # tdSql.checkRows(15) + # tdSql.checkEqual(tdSql.queryResult[14][2],10) + # tdSql.execute(f'alter table {dbname}.{tbname} drop column c16') + + # tdSql.execute(f'alter table {dbname}.{tbname} add column c16 nchar(10)') + # tdSql.query(f'describe {dbname}.{tbname}') + # tdSql.checkRows(15) + # tdSql.checkEqual(tdSql.queryResult[14][2],10) + # tdSql.execute(f'alter table {dbname}.{tbname} drop column c16') + + tdSql.execute(f'alter table {dbname}.{tbname} modify column c12 binary(30)') tdSql.query(f'describe {dbname}.{tbname}') tdSql.checkData(12,2,30) diff --git a/tests/system-test/1-insert/create_table_comment.py b/tests/system-test/1-insert/create_table_comment.py deleted file mode 100644 index 92ea083c5a52324dd812b676d99f493c63166564..0000000000000000000000000000000000000000 --- a/tests/system-test/1-insert/create_table_comment.py +++ /dev/null @@ -1,113 +0,0 @@ -################################################################### -# Copyright (c) 2016 by TAOS Technologies, Inc. -# All rights reserved. -# -# This file is proprietary and confidential to TAOS Technologies. -# No part of this file may be reproduced, stored, transmitted, -# disclosed or used in any form or by any means other than as -# expressly provided by the written permission from Jianhui Tao -# -################################################################### - -# -*- coding: utf-8 -*- - -import random -import string -from util.log import * -from util.cases import * -from util.sql import * - -class TDTestCase: - def init(self, conn, logSql): - tdLog.debug("start to execute %s" % __file__) - tdSql.init(conn.cursor()) - - def get_long_name(self, length, mode="mixed"): - """ - generate long name - mode could be numbers/letters/letters_mixed/mixed - """ - if mode == "numbers": - population = string.digits - elif mode == "letters": - population = string.ascii_letters.lower() - elif mode == "letters_mixed": - population = string.ascii_letters.upper() + string.ascii_letters.lower() - else: - population = string.ascii_letters.lower() + string.digits - return "".join(random.choices(population, k=length)) - - def __create_tb(self,dbname,stbname,tbname,comment): - tdSql.execute(f'create database if not exists {dbname}') - tdSql.execute(f'use {dbname}') - tdSql.execute( - f'create table {stbname} (ts timestamp,c0 int) tags(t0 int) ') - tdSql.execute( - f'create table {tbname} using {stbname} tags(1) comment "{comment}"') - def __create_normaltb(self,dbname,tbname,comment): - tdSql.execute(f'create database if not exists {dbname}') - tdSql.execute(f'use {dbname}') - tdSql.execute( - f'create table {tbname} (ts timestamp,c0 int) comment "{comment}"') - - def check_comment(self): - dbname = self.get_long_name(length=10, mode="letters") - ntbname = self.get_long_name(length=5, mode="letters") - - # create normal table with comment - comment = self.get_long_name(length=10, mode="letters") - self.__create_normaltb(dbname,ntbname,comment) - ntb_kv_list = tdSql.getResult("show tables") - print(ntb_kv_list) - tdSql.checkEqual(ntb_kv_list[0][8], comment) - tdSql.error('alter table {ntbname} comment "test1"') - tdSql.execute(f'drop database {dbname}') - - # max length(1024) - comment = self.get_long_name(length=1024, mode="letters") - self.__create_normaltb(dbname,ntbname,comment) - ntb_kv_list = tdSql.getResult("show tables") - tdSql.checkEqual(ntb_kv_list[0][8], comment) - tdSql.execute(f'drop database {dbname}') - - # error overlength - comment = self.get_long_name(length=1025, mode="letters") - tdSql.execute(f'create database if not exists {dbname}') - tdSql.execute(f'use {dbname}') - tdSql.error(f"create table ntb (ts timestamp,c0 int) comment '{comment}'") - tdSql.execute(f'drop database {dbname}') - - # create child table with comment - comment = self.get_long_name(length=10, mode="letters") - stbname = self.get_long_name(length=5, mode="letters") - tbname = self.get_long_name(length=3, mode="letters") - self.__create_tb(dbname,stbname,tbname,comment) - ntb_kv_list = tdSql.getResult("show tables") - tdSql.checkEqual(ntb_kv_list[0][8], comment) - tdSql.error(f'alter table {tbname} comment "test1"') - tdSql.execute(f'drop database {dbname}') - - # max length 1024 - comment = self.get_long_name(length=1024, mode="letters") - self.__create_tb(dbname,ntbname,comment) - ntb_kv_list = tdSql.getResult("show tables") - tdSql.checkEqual(ntb_kv_list[0][8], comment) - tdSql.execute(f'drop database {dbname}') - - # error overlength - comment = self.get_long_name(length=1025, mode="letters") - tdSql.execute(f'create database if not exists {dbname}') - tdSql.execute(f'use {dbname}') - tdSql.execute(f"create table stb (ts timestamp,c0 int) tags(t0 int)") - tdSql.error(f'create table stb_1 us stb tags(1) comment "{comment}"') - tdSql.execute(f'drop database {dbname}') - - def run(self): - self.check_comment() - - def stop(self): - tdSql.close() - tdLog.success("%s successfully executed" % __file__) - -tdCases.addWindows(__file__, TDTestCase()) -tdCases.addLinux(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/system-test/1-insert/insertWithMoreVgroup.py b/tests/system-test/1-insert/insertWithMoreVgroup.py index 7708ebb476ace3d8c5df779acb01ca4fe5b46295..29c293c6084c9edb7107e4289e194b320cd43207 100644 --- a/tests/system-test/1-insert/insertWithMoreVgroup.py +++ b/tests/system-test/1-insert/insertWithMoreVgroup.py @@ -119,7 +119,7 @@ class TDTestCase: # tdLog.debug("spent %.2fs to create 1 stable and %d table, create speed is %.2f table/s... [OK]"% (spendTime,count,speedCreate)) return - def mutiThread_create_tables(self,host,dbname,stbname,vgroups,threadNumbers,childrowcount): + def mutiThread_create_tables(self,host,dbname,stbname,vgroups,threadNumbers,childcount): buildPath = self.getBuildPath() config = buildPath+ "../sim/dnode1/cfg/" @@ -128,7 +128,7 @@ class TDTestCase: tsql.execute("drop database if exists %s"%dbname) tsql.execute("create database %s vgroups %d"%(dbname,vgroups)) tsql.execute("use %s" %dbname) - count=int(childrowcount) + count=int(childcount) threads = [] for i in range(threadNumbers): tsql.execute("create stable %s%d(ts timestamp, c1 int, c2 binary(10)) tags(t1 int)"%(stbname,i)) @@ -264,19 +264,88 @@ class TDTestCase: speedCreate=count/spendTime tdLog.debug("spent %.2fs to create 1 stable and %d table, create speed is %.2f table/s... [OK]"% (spendTime,count,speedCreate)) return + + def checkData(self,dbname,stbname,stableCount,CtableCount,rowsPerSTable,): + tdSql.execute("use %s"%dbname) + tdSql.query("show stables") + tdSql.checkRows(stableCount) + tdSql.query("show tables") + tdSql.checkRows(CtableCount) + for i in range(stableCount): + tdSql.query("select count(*) from %s%d"%(stbname,i)) + tdSql.checkData(0,0,rowsPerSTable) + return + + # test case1 base def test_case1(self): + #stableCount=threadNumbersCtb + parameterDict = {'vgroups': 1, \ + 'threadNumbersCtb': 5, \ + 'threadNumbersIda': 5, \ + 'stableCount': 5, \ + 'tablesPerStb': 50, \ + 'rowsPerTable': 10, \ + 'dbname': 'db', \ + 'stbname': 'stb', \ + 'host': 'localhost', \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + tdLog.debug("-----create database and muti-thread create tables test------- ") #host,dbname,stbname,vgroups,threadNumbers,tcountStart,tcountStop #host, dbname, stbname, threadNumbers, chilCount, ts_start, childrowcount - self.mutiThread_create_tables(host="localhost",dbname="db",stbname="stb", vgroups=1, threadNumbers=5, childrowcount=50) - self.mutiThread_insert_data(host="localhost",dbname="db",stbname="stb", threadNumbers=5,chilCount=50,ts_start=self.ts,childrowcount=10) - - return - - + self.mutiThread_create_tables( + host=parameterDict['host'], + dbname=parameterDict['dbname'], + stbname=parameterDict['stbname'], + vgroups=parameterDict['vgroups'], + threadNumbers=parameterDict['threadNumbersCtb'], + childcount=parameterDict['tablesPerStb']) + + self.mutiThread_insert_data( + host=parameterDict['host'], + dbname=parameterDict['dbname'], + stbname=parameterDict['stbname'], + threadNumbers=parameterDict['threadNumbersIda'], + chilCount=parameterDict['tablesPerStb'], + ts_start=parameterDict['startTs'], + childrowcount=parameterDict['rowsPerTable']) + + tableCount=parameterDict['threadNumbersCtb']*parameterDict['tablesPerStb'] + rowsPerStable=parameterDict['rowsPerTable']*parameterDict['tablesPerStb'] + + self.checkData(dbname=parameterDict['dbname'],stbname=parameterDict['stbname'], stableCount=parameterDict['threadNumbersCtb'],CtableCount=tableCount,rowsPerSTable=rowsPerStable) + def test_case3(self): - self.taosBenchCreate("127.0.0.1","no","db1", "stb1", 1, 8, 1*10) + #stableCount=threadNumbersCtb + parameterDict = {'vgroups': 1, \ + 'threadNumbersCtb': 8, \ + 'stableCount': 5, \ + 'tablesPerStb': 10, \ + 'rowsPerTable': 100, \ + 'dbname': 'db1', \ + 'stbname': 'stb1', \ + 'host': 'localhost', \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + + self.taosBenchCreate( + parameterDict['host'], + "no", + parameterDict['dbname'], + parameterDict['stbname'], + parameterDict['vgroups'], + parameterDict['threadNumbersCtb'], + parameterDict['tablesPerStb']) + tableCount=parameterDict['threadNumbersCtb']*parameterDict['tablesPerStb'] + rowsPerStable=parameterDict['rowsPerTable']*parameterDict['tablesPerStb'] + + self.checkData( + dbname=parameterDict['dbname'], + stbname=parameterDict['stbname'], + stableCount=parameterDict['threadNumbersCtb'], + CtableCount=tableCount, + rowsPerSTable=rowsPerStable) + # self.taosBenchCreate("test209","no","db2", "stb2", 1, 8, 1*10000) # self.taosBenchCreate("chenhaoran02","no","db1", "stb1", 1, 8, 1*10000) @@ -320,14 +389,6 @@ class TDTestCase: # tdSql.execute("create qnode on dnode %s"%dnodeId) - - # self.taosBenchCreate("test209","no","db2", "stb2", 1, 8, 1*10000) - - # self.taosBenchCreate("chenhaoran02","no","db1", "stb1", 1, 8, 1*10000) - - # self.taosBenchCreate("db1", "stb1", 4, 5, 100*10000) - # self.taosBenchCreate("db1", "stb1", 1, 5, 100*10000) - # run case def run(self): diff --git a/tests/system-test/1-insert/manyVgroups.json b/tests/system-test/1-insert/manyVgroups.json index 5dea41476c8cf7777b5a548f470577e03c576663..20ac3205523af96cec2e7c646c6245c53a55c7e8 100644 --- a/tests/system-test/1-insert/manyVgroups.json +++ b/tests/system-test/1-insert/manyVgroups.json @@ -11,7 +11,7 @@ "confirm_parameter_prompt": "no", "insert_interval": 0, "interlace_rows": 0, - "num_of_records_per_req": 100, + "num_of_records_per_req": 100000, "databases": [ { "dbinfo": { @@ -29,7 +29,7 @@ "batch_create_tbl_num": 50000, "data_source": "rand", "insert_mode": "taosc", - "insert_rows": 1, + "insert_rows": 100, "interlace_rows": 0, "insert_interval": 0, "max_sql_len": 10000000, @@ -45,28 +45,28 @@ }, { "type": "DOUBLE", - "count": 100 + "count": 1 }, { "type": "BINARY", - "len": 400, - "count": 10 + "len": 40, + "count": 1 }, { "type": "nchar", - "len": 200, - "count": 20 + "len": 20, + "count": 1 } ], "tags": [ { "type": "TINYINT", - "count": 2 + "count": 1 }, { "type": "BINARY", "len": 16, - "count": 2 + "count": 1 } ] } diff --git a/tests/system-test/1-insert/table_comment.py b/tests/system-test/1-insert/table_comment.py new file mode 100644 index 0000000000000000000000000000000000000000..5b85a3964f91a38b4d522e7750b851861c454c16 --- /dev/null +++ b/tests/system-test/1-insert/table_comment.py @@ -0,0 +1,134 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import random +import string + +from util.log import * +from util.cases import * +from util.sql import * +from util.common import * + +class TDTestCase: + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + # prepare data + self.ntbname = 'ntb' + self.stbname = 'stb' + self.column_dict = { + 'ts':'timestamp', + 'c1':'int', + 'c2':'float', + 'c3':'double', + 'c4':'timestamp' + } + self.tag_dict = { + 't0':'int' + } + self.comment_length = [0,1024] + self.error_comment_length = [1025] + self.table_type_list = ['normal_table','stable','child_table'] + self.comment_flag_list = [True,False] + + def __set_and_alter_comment(self,tb_type='',comment_flag= False): + + column_sql = '' + tag_sql = '' + for k,v in self.column_dict.items(): + column_sql += f"{k} {v}," + for k,v in self.tag_dict.items(): + tag_sql += f"{k} {v}," + if tb_type == 'normal_table' or tb_type == '': + if comment_flag == False: + tdSql.execute(f'create table {self.ntbname} ({column_sql[:-1]})') + self.check_comment_info() + self.alter_comment(self.ntbname) + tdSql.execute(f'drop table {self.ntbname}') + elif comment_flag == True: + for i in self.comment_length: + comment_info = tdCom.getLongName(i) + tdSql.execute(f'create table {self.ntbname} ({column_sql[:-1]}) comment "{comment_info}"') + self.check_comment_info(comment_info) + self.alter_comment(self.ntbname) + tdSql.execute(f'drop table {self.ntbname}') + for i in self.error_comment_length: + comment_info = tdCom.getLongName(i) + tdSql.error(f'create table {self.ntbname} ({column_sql[:-1]}) comment "{comment_info}"') + elif tb_type == 'stable': + for operation in ['table','stable']: + if comment_flag == False: + tdSql.execute(f'create {operation} {self.stbname} ({column_sql[:-1]}) tags({tag_sql[:-1]})') + self.check_comment_info(None,'stable') + self.alter_comment(self.stbname,'stable') + tdSql.execute(f'drop table {self.stbname}') + elif comment_flag == True: + for i in self.comment_length: + comment_info = tdCom.getLongName(i) + tdSql.execute(f'create {operation} {self.stbname} ({column_sql[:-1]}) tags({tag_sql[:-1]}) comment "{comment_info}"') + self.check_comment_info(comment_info,'stable') + self.alter_comment(self.stbname,'stable') + tdSql.execute(f'drop table {self.stbname}') + elif tb_type == 'child_table': + tdSql.execute(f'create table if not exists {self.stbname} ({column_sql[:-1]}) tags({tag_sql[:-1]})') + if comment_flag == False: + tdSql.execute(f'create table if not exists {self.stbname}_ctb using {self.stbname} tags(1)') + self.check_comment_info() + self.alter_comment(f'{self.stbname}_ctb') + tdSql.execute(f'drop table {self.stbname}_ctb') + elif comment_flag == True: + for j in self.comment_length: + comment_info = tdCom.getLongName(j) + tdSql.execute(f'create table if not exists {self.stbname}_ctb using {self.stbname} tags(1) comment "{comment_info}"') + self.check_comment_info(comment_info) + self.alter_comment(f'{self.stbname}_ctb') + tdSql.execute(f'drop table {self.stbname}_ctb') + tdSql.execute(f'drop table {self.stbname}') + def alter_comment(self,tbname,tb_type=''): + for i in self.comment_length: + comment_info = tdCom.getLongName(i) + print(comment_info) + tdSql.execute(f'alter table {tbname} comment "{comment_info}"') + self.check_comment_info(comment_info,tb_type) + for i in self.error_comment_length: + comment_info = tdCom.getLongName(i) + tdSql.error(f'alter table {tbname} comment "{comment_info}"') + def check_comment_info(self,comment_info=None,tb_type=''): + if tb_type == '' or tb_type == 'normal_table' or tb_type == 'child_table': + tdSql.query('show tables') + if comment_info == None: + tdSql.checkEqual(tdSql.queryResult[0][8],None) + else : + tdSql.checkEqual(tdSql.queryResult[0][8],comment_info) + elif tb_type == 'stable': + tdSql.query('show stables') + if comment_info == None: + tdSql.checkEqual(tdSql.queryResult[0][6],None) + else : + tdSql.checkEqual(tdSql.queryResult[0][6],comment_info) + def comment_check_case(self,table_type,comment_flag): + tdSql.prepare() + for tb in table_type: + for flag in comment_flag: + self.__set_and_alter_comment(tb,flag) + tdSql.execute('drop database db') + + def run(self): + self.comment_check_case(self.table_type_list,self.comment_flag_list) + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/system-test/2-query/Today.py b/tests/system-test/2-query/Today.py index 9eb06de9fb16731bbaa69460f24f4ecff78a2ff3..a9c3215a1283915e7c52702ef5ea80237c2d952a 100644 --- a/tests/system-test/2-query/Today.py +++ b/tests/system-test/2-query/Today.py @@ -4,394 +4,163 @@ from util.dnodes import * from util.log import * from util.sql import * from util.cases import * +import datetime class TDTestCase: - updatecfgDict = {'rpcDebugFlag': '143'} def init(self, conn, logSql): tdLog.debug(f"start to excute {__file__}") tdSql.init(conn.cursor()) - - def run(self): # sourcery skip: extract-duplicate-method - # for func now() , today(), timezone() - tdSql.prepare() - today_date = datetime.datetime.strptime( + self.today_date = datetime.datetime.strptime( datetime.datetime.now().strftime("%Y-%m-%d"), "%Y-%m-%d") + self.time_unit = ['b','u','a','s','m','h','d','w'] + self.error_param = ['1.5','abc','!@#','"abc"','today()'] + self.arithmetic_operators = ['+','-','*','/'] + self.relational_operator = ['<','<=','=','>=','>'] + # prepare data + self.ntbname = 'ntb' + self.stbname = 'stb' + self.column_dict = { + 'ts':'timestamp', + 'c1':'int', + 'c2':'float', + 'c3':'double', + 'c4':'timestamp' + } + self.tag_dict = { + 't0':'int' + } + self.tbnum = 2 + self.tag_values = [ + f'10', + f'100' + ] + self.values_list = [f'now,1,1.55,100.555555,today()', + f'now+1d,10,11.11,99.999999,now()', + f'today(),3,3.333,333.333333,now()', + f'today()-1d,10,11.11,99.999999,now()', + f'today()+1d,1,1.55,100.555555,today()'] + + def set_create_normaltable_sql(self, ntbname, column_dict): + column_sql = '' + for k, v in column_dict.items(): + column_sql += f"{k} {v}," + create_ntb_sql = f'create table {ntbname} ({column_sql[:-1]})' + return create_ntb_sql + def set_create_stable_sql(self,stbname,column_dict,tag_dict): + column_sql = '' + tag_sql = '' + for k,v in column_dict.items(): + column_sql += f"{k} {v}," + for k,v in tag_dict.items(): + tag_sql += f"{k} {v}," + create_stb_sql = f'create table {stbname} ({column_sql[:-1]}) tags({tag_sql[:-1]})' + return create_stb_sql + def data_check(self,column_dict={},tbname = '',values_list = [],tb_num = 1,tb = 'tb'): + for k,v in column_dict.items(): + num_up = 0 + num_down = 0 + num_same = 0 + if v.lower() == 'timestamp': + tdSql.query(f'select {k} from {tbname}') + for i in tdSql.queryResult: + if i[0] > self.today_date: + num_up += 1 + elif i[0] == self.today_date: + num_same += 1 + elif i[0] < self.today_date: + num_down += 1 + tdSql.query(f"select today() from {tbname}") + tdSql.checkRows(len(values_list)*tb_num) + tdSql.checkData(0, 0, str(self.today_date)) + tdSql.query(f"select * from {tbname} where {k}=today()") + if tb == 'tb': + tdSql.checkRows(num_same*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_same) + for i in [f'{tbname}',f'db.{tbname}']: + for unit in self.time_unit: + for symbol in ['+','-']: + tdSql.query(f"select today() {symbol}1{unit} from {i}") + tdSql.checkRows(len(values_list)*tb_num) + for unit in self.error_param: + for symbol in self.arithmetic_operators: + tdSql.error(f'select today() {symbol}{unit} from {i}') + for symbol in self.arithmetic_operators: + tdSql.query(f'select now(){symbol}null from {i}') + tdSql.checkData(0,0,None) + for symbol in self.relational_operator: + tdSql.query(f'select * from {i} where {k} {symbol} today()') + if symbol == '<' : + if tb == 'tb': + tdSql.checkRows(num_down*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_down) + elif symbol == '<=': + if tb == 'tb': + tdSql.checkRows((num_same+num_down)*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_same+num_down) + elif symbol == '=': + if tb == 'tb': + tdSql.checkRows(num_same*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_same) + elif symbol == '>=': + if tb == 'tb': + tdSql.checkRows((num_up + num_same)*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_up + num_same) + elif symbol == '>': + if tb == 'tb': + tdSql.checkRows(num_up*tb_num) + elif tb == 'stb': + tdSql.checkRows(num_up) + tdSql.query(f"select today()/0 from {tbname}") + tdSql.checkRows(len(values_list)*tb_num) + tdSql.checkData(0,0,None) + tdSql.query(f"select {k} from {tbname} where {k}=today()") + if tb == 'tb': + tdSql.checkRows(num_same*tb_num) + for i in range(num_same*tb_num): + tdSql.checkData(i, 0, str(self.today_date)) + elif tb == 'stb': + tdSql.checkRows(num_same) + for i in range(num_same): + tdSql.checkData(i, 0, str(self.today_date)) + def today_check_ntb(self): + tdSql.prepare() + tdSql.execute(self.set_create_normaltable_sql(self.ntbname,self.column_dict)) + for i in self.values_list: + tdSql.execute( + f'insert into {self.ntbname} values({i})') + self.data_check(self.column_dict,self.ntbname,self.values_list) + tdSql.execute('drop database db') + def today_check_stb_tb(self): + tdSql.prepare() + tdSql.execute(self.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict)) + for i in range(self.tbnum): + tdSql.execute(f'create table if not exists {self.stbname}_{i} using {self.stbname} tags({self.tag_values[i]})') + for j in self.values_list: + tdSql.execute(f'insert into {self.stbname}_{i} values ({j})') + # check child table + for i in range(self.tbnum): + self.data_check(self.column_dict,f'{self.stbname}_{i}',self.values_list) + # check stable + self.data_check(self.column_dict,self.stbname,self.values_list,self.tbnum,'stb') + tdSql.execute('drop database db') - tdLog.printNoPrefix("==========step1:create tables==========") - tdSql.execute( - '''create table if not exists ntb - (ts timestamp, c1 int, c2 float,c3 double,c4 timestamp) - ''' - ) - tdSql.execute( - '''create table if not exists stb - (ts timestamp, c1 int, c2 float,c3 double,c4 timestamp) tags(t0 int) - ''' - ) - tdSql.execute( - '''create table if not exists stb_1 using stb tags(100) - ''' - ) - tdLog.printNoPrefix("==========step2:insert data into ntb==========") - tdSql.execute( - 'insert into ntb values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())') - tdSql.execute( - 'insert into stb_1 values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())') - tdLog.printNoPrefix("==========step2:query test of ntb ==========") - - # test function today() - # normal table - tdSql.query("select today() from ntb") - tdSql.checkRows(3) - tdSql.checkData(0, 0, str(today_date)) - tdSql.query("select today() from db.ntb") - tdSql.checkRows(3) - tdSql.checkData(0, 0, str(today_date)) - tdSql.query("select today() +1w from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1w from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1d from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1d from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1h from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1h from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1m from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1m from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1s from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1s from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1a from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1a from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1u from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1u from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1b from ntb") - tdSql.checkRows(3) - tdSql.query("select today() +1b from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1w from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1w from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1d from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1d from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1h from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1h from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1m from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1m from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1s from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1s from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1a from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1a from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1u from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1u from db.ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1b from ntb") - tdSql.checkRows(3) - tdSql.query("select today() -1b from db.ntb") - tdSql.checkRows(3) - tdSql.query("select * from ntb where ts=today()") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 3) - tdSql.query("select * from ntb where ts<=today()") - tdSql.checkRows(2) - tdSql.checkData(0, 1, 10) - # for bug - # tdSql.query("select * from ntb where ts=today()") - tdSql.checkRows(2) - tdSql.checkData(0, 1, 3) - # tdSql.query("select * from ntb where ts>today()") - # tdSql.checkRows(1) - tdSql.query("select c4 from ntb where c4=today()") - tdSql.checkRows(1) - tdSql.checkData(0, 0, str(today_date)) - tdSql.query("select today() from ntb where ts=today()") - # tdSql.checkRows(2) - # tdSql.query("select * from ntb where ts>today()") - # tdSql.checkRows(1) - tdSql.query("select c4 from stb where c4=today()") - tdSql.checkRows(1) - tdSql.checkData(0, 0, str(today_date)) - tdSql.query("select today() from stb where ts=today()") - tdSql.checkRows(2) - tdSql.query("select * from ntb where ts>today()") - tdSql.checkRows(1) - tdSql.query("select c4 from stb_1 where c4=today()") - tdSql.checkRows(1) - tdSql.query("select today() from stb_1 where ts=2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit(f'This scene does not meet the requirements with {vgroups_num} vgroup!\n') + for i in range(self.rowNum): + for j in range(child_table_num): + tdSql.execute(f"insert into {stbname}_{j} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + for i in column_list: + tdSql.query(f'select bottom({i},2) from {stbname}') + tdSql.checkRows(2) + tdSql.checkEqual(tdSql.queryResult,[(1,),(1,)]) + for j in error_param_list: + tdSql.error(f'select bottom({i},{j}) from {stbname}') + for i in error_column_list: + tdSql.error(f'select bottom({i},10) from {stbname}') + + tdSql.execute(f'drop database {dbname}') + def run(self): - tdSql.error('select * from test where bottom(col2,1)=1') + self.bottom_check_base() + self.bottom_check_distribute() def stop(self): diff --git a/tests/system-test/2-query/distribute_agg_apercentile.py b/tests/system-test/2-query/distribute_agg_apercentile.py new file mode 100644 index 0000000000000000000000000000000000000000..fd1455ce16b8868de25d9800b73399ab0e88ec6f --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_apercentile.py @@ -0,0 +1,198 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def distribute_agg_query(self): + # basic filter + tdSql.query("select apercentile(c1 , 20) from stb1 where c1 is null") + tdSql.checkRows(0) + + tdSql.query("select apercentile(c1 , 20) from stb1 where t1=1") + tdSql.checkData(0,0,2.800000000) + + tdSql.query("select apercentile(c1+c2 ,100) from stb1 where c1 =1 ") + tdSql.checkData(0,0,11112.000000000) + + tdSql.query("select apercentile(c1 ,10 ) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,2.000000000) + + tdSql.query("select apercentile(c1,20) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select apercentile(c1,20) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select apercentile(c1,20) from stb1 union all select apercentile(c1,20) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,7.389181281) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select apercentile(tb1.c1,100), apercentile(tb2.c2,100) from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,9.000000000) + tdSql.checkData(0,0,9.000000000) + + # group by + tdSql.execute(" use testdb ") + tdSql.query(" select max(c1),c1 from stb1 group by t1 ") + tdSql.checkRows(20) + tdSql.query(" select max(c1),c1 from stb1 group by c1 ") + tdSql.checkRows(30) + tdSql.query(" select max(c1),c2 from stb1 group by c2 ") + tdSql.checkRows(31) + + # partition by tbname or partition by tag + tdSql.query("select apercentile(c1 ,10)from stb1 partition by tbname") + query_data = tdSql.queryResult + + # nest query for support max + tdSql.query("select apercentile(c2+2,10)+1 from (select max(c1) c2 from stb1)") + tdSql.checkData(0,0,31.000000000) + tdSql.query("select apercentile(c1+2,10)+1 as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,7.560701700) + tdSql.query("select apercentile(a+2,10)+1 as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,7.560701700) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3),spread(c1), apercentile(c1,10) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + tdSql.checkData(0,4,28.000000000) + tdSql.checkData(0,5,4.560701700) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/distribute_agg_count.py b/tests/system-test/2-query/distribute_agg_count.py new file mode 100644 index 0000000000000000000000000000000000000000..2ac9c86df0789830eafd088f22eadc621edaec63 --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_count.py @@ -0,0 +1,296 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + + def check_count_functions(self, tbname , col_name): + + max_sql = f"select count({col_name}) from {tbname};" + + same_sql = f"select sum(c) from (select {col_name} ,1 as c from {tbname} where {col_name} is not null) " + + tdSql.query(max_sql) + max_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if max_result !=same_result: + tdLog.exit(" count function work not as expected, sql : %s "% max_sql) + else: + tdLog.info(" count function work as expected, sql : %s "% max_sql) + + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def check_count_distribute_diff_vnode(self,col_name): + + vgroup_ids = [] + for k ,v in self.vnode_disbutes.items(): + if len(v)>=2: + vgroup_ids.append(k) + + distribute_tbnames = [] + + for vgroup_id in vgroup_ids: + vnode_tables = self.vnode_disbutes[vgroup_id] + distribute_tbnames.append(random.sample(vnode_tables,1)[0]) + tbname_ins = "" + for tbname in distribute_tbnames: + tbname_ins += "'%s' ,"%tbname + + tbname_filters = tbname_ins[:-1] + + max_sql = f"select count({col_name}) from stb1 where tbname in ({tbname_filters});" + + same_sql = f"select sum(c) from (select {col_name} ,1 as c from stb1 where tbname in ({tbname_filters}) and {col_name} is not null) " + + tdSql.query(max_sql) + max_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if max_result !=same_result: + tdLog.exit(" count function work not as expected, sql : %s "% max_sql) + else: + tdLog.info(" count function work as expected, sql : %s "% max_sql) + + def check_count_status(self): + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + + for tablename in tablenames: + for colname in colnames: + self.check_count_functions(tablename,colname) + + # check max function for different vnode + + for colname in colnames: + if colname.startswith("c"): + self.check_count_distribute_diff_vnode(colname) + else: + # self.check_count_distribute_diff_vnode(colname) # bug for tag + pass + + + def distribute_agg_query(self): + # basic filter + tdSql.query("select count(c1) from stb1 ") + tdSql.checkData(0,0,184) + + tdSql.query("select count(c1) from stb1 where t1=1") + tdSql.checkData(0,0,9) + + tdSql.query("select count(c1+c2) from stb1 where c1 =1 ") + tdSql.checkData(0,0,2) + + tdSql.query("select count(c1) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,9) + + tdSql.query("select count(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select count(c1) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select count(c1) from stb1 union all select count(c1) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,184) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select count(tb1.c1), count(tb2.c2) from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,10) + tdSql.checkData(0,1,10) + + # group by + tdSql.execute(" use testdb ") + + tdSql.query(" select count(*) from stb1 ") + tdSql.checkData(0,0,187) + tdSql.query(" select count(*) from stb1 group by t1 ") + tdSql.checkRows(20) + tdSql.query(" select count(*) from stb1 group by c1 ") + tdSql.checkRows(30) + tdSql.query(" select count(*) from stb1 group by c2 ") + tdSql.checkRows(31) + + # partition by tbname or partition by tag + tdSql.query("select max(c1),tbname from stb1 partition by tbname") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select max(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + tdSql.query("select max(c1),tbname from stb1 partition by t1") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select max(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + # nest query for support max + tdSql.query("select abs(c2+2)+1 from (select count(c1) c2 from stb1)") + tdSql.checkData(0,0,187.000000000) + tdSql.query("select count(c1+2) as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,184) + tdSql.query("select count(a+2) as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,184) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.check_count_status() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/distribute_agg_max.py b/tests/system-test/2-query/distribute_agg_max.py new file mode 100644 index 0000000000000000000000000000000000000000..ae0ab5aafab4b64ac9072b4de392d4359a4129de --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_max.py @@ -0,0 +1,293 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + + def check_max_functions(self, tbname , col_name): + + max_sql = f"select max({col_name}) from {tbname};" + + same_sql = f"select {col_name} from {tbname} order by {col_name} desc limit 1" + + tdSql.query(max_sql) + max_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if max_result !=same_result: + tdLog.exit(" max function work not as expected, sql : %s "% max_sql) + else: + tdLog.info(" max function work as expected, sql : %s "% max_sql) + + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def check_max_distribute_diff_vnode(self,col_name): + + vgroup_ids = [] + for k ,v in self.vnode_disbutes.items(): + if len(v)>=2: + vgroup_ids.append(k) + + distribute_tbnames = [] + + for vgroup_id in vgroup_ids: + vnode_tables = self.vnode_disbutes[vgroup_id] + distribute_tbnames.append(random.sample(vnode_tables,1)[0]) + tbname_ins = "" + for tbname in distribute_tbnames: + tbname_ins += "'%s' ,"%tbname + + tbname_filters = tbname_ins[:-1] + + max_sql = f"select max({col_name}) from stb1 where tbname in ({tbname_filters});" + + same_sql = f"select {col_name} from stb1 where tbname in ({tbname_filters}) order by {col_name} desc limit 1" + + tdSql.query(max_sql) + max_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if max_result !=same_result: + tdLog.exit(" max function work not as expected, sql : %s "% max_sql) + else: + tdLog.info(" max function work as expected, sql : %s "% max_sql) + + def check_max_status(self): + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + + for tablename in tablenames: + for colname in colnames: + self.check_max_functions(tablename,colname) + + # check max function for different vnode + + for colname in colnames: + if colname.startswith("c"): + self.check_max_distribute_diff_vnode(colname) + else: + # self.check_max_distribute_diff_vnode(colname) # bug for tag + pass + + + def distribute_agg_query(self): + # basic filter + tdSql.query("select max(c1) from stb1 where c1 is null") + tdSql.checkRows(0) + + tdSql.query("select max(c1) from stb1 where t1=1") + tdSql.checkData(0,0,10) + + tdSql.query("select max(c1+c2) from stb1 where c1 =1 ") + tdSql.checkData(0,0,11112.000000000) + + tdSql.query("select max(c1) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,10) + + tdSql.query("select max(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select max(c1) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select max(c1) from stb1 union all select max(c1) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,28) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select max(tb1.c1), tb2.c2 from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,9) + tdSql.checkData(0,0,9.00000) + + # group by + tdSql.execute(" use testdb ") + tdSql.query(" select max(c1),c1 from stb1 group by t1 ") + tdSql.checkRows(20) + tdSql.query(" select max(c1),c1 from stb1 group by c1 ") + tdSql.checkRows(30) + tdSql.query(" select max(c1),c2 from stb1 group by c2 ") + tdSql.checkRows(31) + + # partition by tbname or partition by tag + tdSql.query("select max(c1),tbname from stb1 partition by tbname") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select max(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + tdSql.query("select max(c1),tbname from stb1 partition by t1") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select max(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + # nest query for support max + tdSql.query("select abs(c2+2)+1 from (select max(c1) c2 from stb1)") + tdSql.checkData(0,0,31.000000000) + tdSql.query("select max(c1+2)+1 as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,31.000000000) + tdSql.query("select max(a+2)+1 as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,31.000000000) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.check_max_status() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/distribute_agg_min.py b/tests/system-test/2-query/distribute_agg_min.py new file mode 100644 index 0000000000000000000000000000000000000000..8a458c74df4f46929bb93abbebee2aad3dd7b76f --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_min.py @@ -0,0 +1,294 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + + def check_min_functions(self, tbname , col_name): + + min_sql = f"select min({col_name}) from {tbname};" + + same_sql = f"select {col_name} from {tbname} where {col_name} is not null order by {col_name} asc limit 1" + + tdSql.query(min_sql) + min_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if min_result !=same_result: + tdLog.exit(" min function work not as expected, sql : %s "% min_sql) + else: + tdLog.info(" min function work as expected, sql : %s "% min_sql) + + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def check_min_distribute_diff_vnode(self,col_name): + + vgroup_ids = [] + for k ,v in self.vnode_disbutes.items(): + if len(v)>=2: + vgroup_ids.append(k) + + distribute_tbnames = [] + + for vgroup_id in vgroup_ids: + vnode_tables = self.vnode_disbutes[vgroup_id] + distribute_tbnames.append(random.sample(vnode_tables,1)[0]) + tbname_ins = "" + for tbname in distribute_tbnames: + tbname_ins += "'%s' ,"%tbname + + tbname_filters = tbname_ins[:-1] + + min_sql = f"select min({col_name}) from stb1 where tbname in ({tbname_filters});" + + same_sql = f"select {col_name} from stb1 where tbname in ({tbname_filters}) and {col_name} is not null order by {col_name} asc limit 1" + + tdSql.query(min_sql) + min_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if min_result !=same_result: + tdLog.exit(" min function work not as expected, sql : %s "% min_sql) + else: + tdLog.info(" min function work as expected, sql : %s "% min_sql) + + def check_min_status(self): + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + + for tablename in tablenames: + for colname in colnames: + self.check_min_functions(tablename,colname) + + # check max function for different vnode + + for colname in colnames: + if colname.startswith("c"): + self.check_min_distribute_diff_vnode(colname) + else: + # self.check_min_distribute_diff_vnode(colname) # bug for tag + pass + + + def distribute_agg_query(self): + # basic filter + tdSql.query("select min(c1) from stb1 where c1 is null") + tdSql.checkRows(0) + + tdSql.query("select min(c1) from stb1 where t1=1") + tdSql.checkData(0,0,2) + + tdSql.query("select min(c1+c2) from stb1 where c1 =1 ") + tdSql.checkData(0,0,11112.000000000) + + tdSql.query("select min(c1) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,2) + + tdSql.query("select min(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select min(c1) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select min(c1) from stb1 union all select min(c1) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,0) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select min(tb1.c1), tb2.c2 from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,0) + tdSql.checkData(0,0,0.00000) + + # group by + tdSql.execute(" use testdb ") + tdSql.query(" select min(c1),c1 from stb1 group by t1 ") + tdSql.checkRows(20) + tdSql.query(" select min(c1),c1 from stb1 group by c1 ") + tdSql.checkRows(30) + tdSql.query(" select min(c1),c2 from stb1 group by c2 ") + tdSql.checkRows(31) + + # partition by tbname or partition by tag + tdSql.query("select min(c1),tbname from stb1 partition by tbname") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select min(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + tdSql.query("select min(c1),tbname from stb1 partition by t1") + query_data = tdSql.queryResult + + for row in query_data: + tbname = row[1] + tdSql.query(" select min(c1) from %s "%tbname) + tdSql.checkData(0,0,row[0]) + + # nest query for support max + tdSql.query("select abs(c2+2)+1 from (select min(c1) c2 from stb1)") + tdSql.checkData(0,0,3.000000000) + tdSql.query("select min(c1+2)+1 as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,3.000000000) + tdSql.query("select min(a+2)+1 as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,3.000000000) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3),min(c1) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + tdSql.checkData(0,4,0) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.check_min_status() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/distribute_agg_spread.py b/tests/system-test/2-query/distribute_agg_spread.py new file mode 100644 index 0000000000000000000000000000000000000000..94f1a61d7727d17d3f1bc9484ca8ed4d3155c886 --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_spread.py @@ -0,0 +1,281 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + + def check_spread_functions(self, tbname , col_name): + + spread_sql = f"select spread({col_name}) from {tbname};" + + same_sql = f"select max({col_name})-min({col_name}) from {tbname}" + + tdSql.query(spread_sql) + spread_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if spread_result !=same_result: + tdLog.exit(" max function work not as expected, sql : %s "% spread_sql) + else: + tdLog.info(" max function work as expected, sql : %s "% spread_sql) + + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def check_spread_distribute_diff_vnode(self,col_name): + + vgroup_ids = [] + for k ,v in self.vnode_disbutes.items(): + if len(v)>=2: + vgroup_ids.append(k) + + distribute_tbnames = [] + + for vgroup_id in vgroup_ids: + vnode_tables = self.vnode_disbutes[vgroup_id] + distribute_tbnames.append(random.sample(vnode_tables,1)[0]) + tbname_ins = "" + for tbname in distribute_tbnames: + tbname_ins += "'%s' ,"%tbname + + tbname_filters = tbname_ins[:-1] + + spread_sql = f"select spread({col_name}) from stb1 where tbname in ({tbname_filters})" + + same_sql = f"select max({col_name}) - min({col_name}) from stb1 where tbname in ({tbname_filters})" + + tdSql.query(spread_sql) + spread_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if spread_result !=same_result: + tdLog.exit(" spread function work not as expected, sql : %s "% spread_sql) + else: + tdLog.info(" spread function work as expected, sql : %s "% spread_sql) + + def check_spread_status(self): + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + + for tablename in tablenames: + for colname in colnames: + self.check_spread_functions(tablename,colname) + + # check max function for different vnode + + for colname in colnames: + if colname.startswith("c"): + self.check_spread_distribute_diff_vnode(colname) + else: + # self.check_spread_distribute_diff_vnode(colname) # bug for tag + pass + + + def distribute_agg_query(self): + # basic filter + tdSql.query("select spread(c1) from stb1 where c1 is null") + tdSql.checkRows(0) + + tdSql.query("select spread(c1) from stb1 where t1=1") + tdSql.checkData(0,0,8.000000000) + + tdSql.query("select spread(c1+c2) from stb1 where c1 =1 ") + tdSql.checkData(0,0,0.000000000) + + tdSql.query("select spread(c1) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,8.000000000) + + tdSql.query("select spread(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select spread(c1) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select spread(c1) from stb1 union all select max(c1)-min(c1) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,28.000000000) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select spread(tb1.c1), spread(tb2.c2) from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,9.000000000) + tdSql.checkData(0,0,9.00000) + + # group by + tdSql.execute(" use testdb ") + tdSql.query(" select max(c1),c1 from stb1 group by t1 ") + tdSql.checkRows(20) + tdSql.query(" select max(c1),c1 from stb1 group by c1 ") + tdSql.checkRows(30) + tdSql.query(" select max(c1),c2 from stb1 group by c2 ") + tdSql.checkRows(31) + + # partition by tbname or partition by tag + tdSql.query("select spread(c1) from stb1 partition by tbname") + query_data = tdSql.queryResult + + # nest query for support max + tdSql.query("select spread(c2+2)+1 from (select max(c1) c2 from stb1)") + tdSql.checkData(0,0,1.000000000) + tdSql.query("select spread(c1+2)+1 as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,29.000000000) + tdSql.query("select spread(a+2)+1 as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,29.000000000) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3),spread(c1) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + tdSql.checkData(0,4,28.000000000) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.check_spread_status() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/distribute_agg_sum.py b/tests/system-test/2-query/distribute_agg_sum.py new file mode 100644 index 0000000000000000000000000000000000000000..add4d75c6171194f6669d36f39134dbbde6ba846 --- /dev/null +++ b/tests/system-test/2-query/distribute_agg_sum.py @@ -0,0 +1,278 @@ +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import random ,os ,sys +import platform + + +class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.vnode_disbutes = None + self.ts = 1537146000000 + + + def check_sum_functions(self, tbname , col_name): + + sum_sql = f"select sum({col_name}) from {tbname};" + + same_sql = f"select {col_name} from {tbname} where {col_name} is not null " + + tdSql.query(same_sql) + pre_data = np.array(tdSql.queryResult)[np.array(tdSql.queryResult) != None] + if (platform.system().lower() == 'windows' and pre_data.dtype == 'int32'): + pre_data = np.array(pre_data, dtype = 'int64') + pre_sum = np.sum(pre_data) + + tdSql.query(sum_sql) + tdSql.checkData(0,0,pre_sum) + + def prepare_datas_of_distribute(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + def check_distribute_datas(self): + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + self.vnode_disbutes = vnode_tables + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + def check_sum_distribute_diff_vnode(self,col_name): + + vgroup_ids = [] + for k ,v in self.vnode_disbutes.items(): + if len(v)>=2: + vgroup_ids.append(k) + + distribute_tbnames = [] + + for vgroup_id in vgroup_ids: + vnode_tables = self.vnode_disbutes[vgroup_id] + distribute_tbnames.append(random.sample(vnode_tables,1)[0]) + tbname_ins = "" + for tbname in distribute_tbnames: + tbname_ins += "'%s' ,"%tbname + + tbname_filters = tbname_ins[:-1] + + sum_sql = f"select sum({col_name}) from stb1 where tbname in ({tbname_filters});" + + same_sql = f"select {col_name} from stb1 where tbname in ({tbname_filters}) and {col_name} is not null " + + tdSql.query(same_sql) + pre_data = np.array(tdSql.queryResult)[np.array(tdSql.queryResult) != None] + if (platform.system().lower() == 'windows' and pre_data.dtype == 'int32'): + pre_data = np.array(pre_data, dtype = 'int64') + pre_sum = np.sum(pre_data) + + tdSql.query(sum_sql) + tdSql.checkData(0,0,pre_sum) + + def check_sum_status(self): + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + + for tablename in tablenames: + for colname in colnames: + self.check_sum_functions(tablename,colname) + + # check max function for different vnode + + for colname in colnames: + if colname.startswith("c"): + self.check_sum_distribute_diff_vnode(colname) + else: + # self.check_sum_distribute_diff_vnode(colname) # bug for tag + pass + + + def distribute_agg_query(self): + # basic filter + tdSql.query(" select sum(c1) from stb1 ") + tdSql.checkData(0,0,2592) + + tdSql.query(" select sum(a) from (select sum(c1) a from stb1 partition by tbname) ") + tdSql.checkData(0,0,2592) + + tdSql.query(" select sum(c1) from stb1 where t1=1") + tdSql.checkData(0,0,54) + + tdSql.query("select sum(c1+c2) from stb1 where c1 =1 ") + tdSql.checkData(0,0,22224.000000000) + + tdSql.query("select sum(c1) from stb1 where tbname=\"ct2\"") + tdSql.checkData(0,0,54) + + tdSql.query("select sum(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + tdSql.query("select sum(c1) from stb1 where t1> 4 partition by tbname") + tdSql.checkRows(15) + + # union all + tdSql.query("select sum(c1) from stb1 union all select sum(c1) from stb1 ") + tdSql.checkRows(2) + tdSql.checkData(0,0,2592) + + tdSql.query("select sum(a) from (select sum(c1) a from stb1 union all select sum(c1) a from stb1)") + tdSql.checkRows(1) + tdSql.checkData(0,0,5184) + + # join + + tdSql.execute(" create database if not exists db ") + tdSql.execute(" use db ") + tdSql.execute(" create stable st (ts timestamp , c1 int ,c2 float) tags(t1 int) ") + tdSql.execute(" create table tb1 using st tags(1) ") + tdSql.execute(" create table tb2 using st tags(2) ") + + + for i in range(10): + ts = i*10 + self.ts + tdSql.execute(f" insert into tb1 values({ts},{i},{i}.0)") + tdSql.execute(f" insert into tb2 values({ts},{i},{i}.0)") + + tdSql.query("select sum(tb1.c1), sum(tb2.c2) from tb1, tb2 where tb1.ts=tb2.ts") + tdSql.checkRows(1) + tdSql.checkData(0,0,45) + tdSql.checkData(0,1,45.000000000) + + # group by + tdSql.execute(" use testdb ") + + # partition by tbname or partition by tag + tdSql.query("select sum(c1) from stb1 partition by tbname") + tdSql.checkRows(20) + + # nest query for support max + tdSql.query("select abs(c2+2)+1 from (select sum(c1) c2 from stb1)") + tdSql.checkData(0,0,2595.000000000) + tdSql.query("select sum(c1+2) as c2 from (select ts ,c1 ,c2 from stb1)") + tdSql.checkData(0,0,2960.000000000) + tdSql.query("select sum(a+2) as c2 from (select ts ,abs(c1) a ,c2 from stb1)") + tdSql.checkData(0,0,2960.000000000) + + # mixup with other functions + tdSql.query("select max(c1),count(c1),last(c2,c3),sum(c1+c2) from stb1") + tdSql.checkData(0,0,28) + tdSql.checkData(0,1,184) + tdSql.checkData(0,2,-99999) + tdSql.checkData(0,3,-999) + tdSql.checkData(0,4,28202310.000000000) + + def run(self): + + self.prepare_datas_of_distribute() + self.check_distribute_datas() + self.check_sum_status() + self.distribute_agg_query() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/first.py b/tests/system-test/2-query/first.py index 7227d1afb5e22f68af90fb9d2192eb7a4a088c96..e9a8cc950b109ca6a8ec5eff8e323c57ea42e2c2 100644 --- a/tests/system-test/2-query/first.py +++ b/tests/system-test/2-query/first.py @@ -11,8 +11,11 @@ # -*- coding: utf-8 -*- +import random +import string import sys import taos +from util.common import * from util.log import * from util.cases import * from util.sql import * @@ -25,124 +28,159 @@ class TDTestCase: tdSql.init(conn.cursor()) self.rowNum = 10 + self.tbnum = 20 self.ts = 1537146000000 - - def run(self): + self.binary_str = 'taosdata' + self.nchar_str = '涛思数据' + + def first_check_base(self): tdSql.prepare() - - tdSql.execute('''create table test(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned) tags(loc nchar(20))''') - tdSql.execute("create table test1 using test tags('beijing')") - tdSql.execute("insert into test1(ts) values(%d)" % (self.ts - 1)) - - # first verifacation - # bug TD-15957 - tdSql.query("select first(*) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, None) - - tdSql.query("select first(col1) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col2) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col3) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col4) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col11) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col12) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col13) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col14) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col5) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col6) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col7) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col8) from test1") - tdSql.checkRows(0) - - tdSql.query("select first(col9) from test1") - tdSql.checkRows(0) - + column_dict = { + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + tdSql.execute('''create table stb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + tdSql.execute("create table stb_1 using stb tags('beijing')") + tdSql.execute("insert into stb_1(ts) values(%d)" % (self.ts - 1)) + column_list = ['col1','col2','col3','col4','col5','col6','col7','col8','col9','col10','col11','col12','col13'] + for i in ['stb_1','db.stb_1','stb_1','db.stb_1']: + tdSql.query(f"select first(*) from {i}") + tdSql.checkRows(1) + tdSql.checkData(0, 1, None) + #!bug TD-16561 + # for i in ['stb','db.stb']: + # tdSql.query(f"select first(*) from {i}") + # tdSql.checkRows(1) + # tdSql.checkData(0, 1, None) + for i in column_list: + for j in ['stb_1','db.stb_1','stb_1','db.stb_1']: + tdSql.query(f"select first({i}) from {j}") + tdSql.checkRows(0) for i in range(self.rowNum): - tdSql.execute("insert into test1 values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) - - tdSql.query("select first(*) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 1) - - tdSql.query("select first(col1) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col2) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col3) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col4) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col11) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col12) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col13) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col14) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 1) - - tdSql.query("select first(col5) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 0.1) - - tdSql.query("select first(col6) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 0.1) - - tdSql.query("select first(col7) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, False) - - tdSql.query("select first(col8) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata1') - - tdSql.query("select first(col9) from test1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据1') + tdSql.execute(f"insert into stb_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + for k, v in column_dict.items(): + for j in ['stb_1', 'db.stb_1', 'stb', 'db.stb']: + tdSql.query(f"select first({k}) from {j}") + tdSql.checkRows(1) + # tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned + if v == 'tinyint' or v == 'smallint' or v == 'int' or v == 'bigint' or v == 'tinyint unsigned' or v == 'smallint unsigned'\ + or v == 'int unsigned' or v == 'bigint unsigned': + tdSql.checkData(0, 0, 1) + # float,double + elif v == 'float' or v == 'double': + tdSql.checkData(0, 0, 0.1) + # bool + elif v == 'bool': + tdSql.checkData(0, 0, False) + # binary + elif 'binary' in v: + tdSql.checkData(0, 0, f'{self.binary_str}1') + # nchar + elif 'nchar' in v: + tdSql.checkData(0, 0, f'{self.nchar_str}1') + #!bug TD-16569 + tdSql.query("select first(*),last(*) from stb where ts < 23 interval(1s)") + tdSql.checkRows(0) + tdSql.execute('drop database db') + def first_check_stb_distribute(self): + # prepare data for vgroup 4 + dbname = tdCom.getLongName(10, "letters") + stbname = tdCom.getLongName(5, "letters") + child_table_num = 20 + vgroup = 2 + column_dict = { + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + tdSql.execute(f"create database if not exists {dbname} vgroups {vgroup}") + tdSql.execute(f'use {dbname}') + # build 20 child tables,every table insert 10 rows + tdSql.execute(f'''create table {stbname}(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + for i in range(child_table_num): + tdSql.execute(f"create table {stbname}_{i} using {stbname} tags('beijing')") + tdSql.execute(f"insert into {stbname}_{i}(ts) values(%d)" % (self.ts - 1-i)) + #!bug TD-16561 + # for i in [f'{stbname}', f'{dbname}.{stbname}']: + # tdSql.query(f"select first(*) from {i}") + # tdSql.checkRows(1) + # tdSql.checkData(0, 1, None) + tdSql.query('show tables') + vgroup_list = [] + for i in range(len(tdSql.queryResult)): + vgroup_list.append(tdSql.queryResult[i][6]) + vgroup_list_set = set(vgroup_list) + # print(vgroup_list_set) + # print(vgroup_list) + for i in vgroup_list_set: + vgroups_num = vgroup_list.count(i) + if vgroups_num >=2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit('This scene does not meet the requirements with {vgroups_num} vgroup!\n') + for i in range(child_table_num): + for j in range(self.rowNum): + tdSql.execute(f"insert into {stbname}_{i} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + j + i, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 0.1, j + 0.1, j % 2, j + 1, j + 1)) + + for k, v in column_dict.items(): + for j in [f'{stbname}_{i}', f'{dbname}.{stbname}_{i}', f'{stbname}', f'{dbname}.{stbname}']: + tdSql.query(f"select first({k}) from {j}") + tdSql.checkRows(1) + # tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned + if v == 'tinyint' or v == 'smallint' or v == 'int' or v == 'bigint' or v == 'tinyint unsigned' or v == 'smallint unsigned'\ + or v == 'int unsigned' or v == 'bigint unsigned': + tdSql.checkData(0, 0, 1) + # float,double + elif v == 'float' or v == 'double': + tdSql.checkData(0, 0, 0.1) + # bool + elif v == 'bool': + tdSql.checkData(0, 0, False) + # binary + elif 'binary' in v: + tdSql.checkData(0, 0, f'{self.binary_str}1') + # nchar + elif 'nchar' in v: + tdSql.checkData(0, 0, f'{self.nchar_str}1') + #!bug TD-16569 + # tdSql.query(f"select first(*),last(*) from {stbname} where ts < 23 interval(1s)") + # tdSql.checkRows(0) + tdSql.execute(f'drop database {dbname}') + + + + pass + def run(self): + self.first_check_base() + self.first_check_stb_distribute() - tdSql.query("select first(*),last(*) from test1 where ts < 23 interval(1s)") - tdSql.checkRows(0) def stop(self): tdSql.close() diff --git a/tests/system-test/2-query/histogram.py b/tests/system-test/2-query/histogram.py index 0448952be867308e0a8404de9f3bf9b6b43036e4..bc061f8fb793ff716e35fe22fc278b25bca9e257 100644 --- a/tests/system-test/2-query/histogram.py +++ b/tests/system-test/2-query/histogram.py @@ -44,8 +44,7 @@ class TDTestCase: buildPath = root[:len(root) - len("/build/bin")] break return buildPath - - def run(self): + def histogram_check_base(self): print("running {}".format(__file__)) tdSql.execute("drop database if exists db") tdSql.execute("create database if not exists db") @@ -3183,6 +3182,72 @@ class TDTestCase: tdSql.execute('drop database db') + + def histogram_check_distribute(self): + dbname = "db" + stbname = "stb" + row_num = 10 + child_table_num = 20 + vgroups = 2 + user_input_json = "[1,3,5,7]" + ts = 1537146000000 + binary_str = 'taosdata' + nchar_str = '涛思数据' + column_dict = { + 'ts' : 'timestamp', + 'col1' : 'tinyint', + 'col2' : 'smallint', + 'col3' : 'int', + 'col4' : 'bigint', + 'col5' : 'tinyint unsigned', + 'col6' : 'smallint unsigned', + 'col7' : 'int unsigned', + 'col8' : 'bigint unsigned', + 'col9' : 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + tdSql.execute(f"create database if not exists {dbname} vgroups {vgroups}") + tdSql.execute(f'use {dbname}') + # build 20 child tables,every table insert 10 rows + tdSql.execute(f'''create table {stbname}(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + for i in range(child_table_num): + tdSql.execute(f"create table {stbname}_{i} using {stbname} tags('beijing')") + tdSql.query('show tables') + vgroup_list = [] + for i in range(len(tdSql.queryResult)): + vgroup_list.append(tdSql.queryResult[i][6]) + vgroup_list_set = set(vgroup_list) + for i in vgroup_list_set: + vgroups_num = vgroup_list.count(i) + if vgroups_num >=2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit(f'This scene does not meet the requirements with {vgroups_num} vgroup!\n') + for i in range(child_table_num): + for j in range(row_num): + tdSql.execute(f"insert into {stbname}_{i} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{binary_str}%d', '{nchar_str}%d')" + % (ts + j + i, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 0.1, j + 0.1, j % 2, j + 1, j + 1)) + # user_input + for k,v in column_dict.items(): + if v.lower() == 'tinyint' or v.lower() == 'smallint' or v.lower() == 'int' or v.lower() == 'bigint' or v.lower() =='float' or v.lower() =='double'\ + or v.lower() =='tinyint unsigned' or v.lower() =='smallint unsigned' or v.lower() =='int unsigned' or v.lower() =='bigint unsigned': + tdSql.query(f'select histogram({k}, "user_input", "{user_input_json}", 0) from {stbname}') + tdSql.checkRows(len(user_input_json[1:-1].split(','))-1) + elif 'binary' in v.lower() or 'nchar' in v.lower() or 'bool' == v.lower(): + tdSql.error(f'select histogram({k}, "user_input", "{user_input_json}", 0) from {stbname}') + + tdSql.execute(f'drop database {dbname}') + + + def run(self): + self.histogram_check_base() + self.histogram_check_distribute() + def stop(self): tdSql.close() tdLog.success("%s successfully executed" % __file__) diff --git a/tests/system-test/2-query/hyperloglog.py b/tests/system-test/2-query/hyperloglog.py index 337db140a167fa5c8dfae73d158b011f4a869189..8dd6bd2ddaef22698c66834cc26ec82a3ad16fea 100644 --- a/tests/system-test/2-query/hyperloglog.py +++ b/tests/system-test/2-query/hyperloglog.py @@ -214,6 +214,79 @@ class TDTestCase: for i in range(4): tdSql.execute(f'create table ct{i+1} using stb1 tags ( {i+1} )') { i % 32767 }, { i % 127}, { i * 1.11111 }, { i * 1000.1111 }, { i % 2} + def __create_stable(self,stbname='stb',column_dict={'ts':'timestamp','col1': 'tinyint','col2': 'smallint','col3': 'int', + 'col4': 'bigint','col5': 'tinyint unsigned','col6': 'smallint unsigned','col7': 'int unsigned', + 'col8': 'bigint unsigned','col9': 'float','col10': 'double','col11': 'bool','col12': 'binary(20)','col13': 'nchar(20)'}, + tag_dict={'ts_tag':'timestamp','t1': 'tinyint','t2': 'smallint','t3': 'int', + 't4': 'bigint','t5': 'tinyint unsigned','t6': 'smallint unsigned','t7': 'int unsigned', + 't8': 'bigint unsigned','t9': 'float','t10': 'double','t11': 'bool','t12': 'binary(20)','t13': 'nchar(20)'}): + column_sql = '' + tag_sql = '' + for k,v in column_dict.items(): + column_sql += f"{k} {v}," + for k,v in tag_dict.items(): + tag_sql += f"{k} {v}," + tdSql.execute(f'create table if not exists {stbname} ({column_sql[:-1]}) tags({tag_sql[:-1]})') + + def __insert_data(self): + + pass + + def __hyperloglog_check_distribute(self): + dbname = "dbtest" + stbname = "stb" + childtable_num = 20 + vgroups_num = 4 + row_num = 10 + ts = 1537146000000 + binary_str = 'taosdata' + nchar_str = '涛思数据' + column_dict = { + 'ts':'timestamp', + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + tag_dict = { + 'loc':'nchar(20)' + } + tdSql.execute(f"create database if not exists {dbname} vgroups {vgroups_num}") + tdSql.execute(f'use {dbname}') + self.__create_stable(stbname,column_dict,tag_dict) + for i in range(childtable_num): + tdSql.execute(f"create table {stbname}_{i} using {stbname} tags('beijing')") + tdSql.query('show tables') + vgroup_list = [] + for i in range(len(tdSql.queryResult)): + vgroup_list.append(tdSql.queryResult[i][6]) + vgroup_list_set = set(vgroup_list) + for i in vgroup_list_set: + vgroups_num = vgroup_list.count(i) + if vgroups_num >=2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit('This scene does not meet the requirements with {vgroups_num} vgroup!\n') + for i in range(row_num): + tdSql.execute(f"insert into stb_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{binary_str}%d', '{nchar_str}%d')" + % (ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + for k in column_dict.keys(): + tdSql.query(f"select hyperloglog({k}) from {stbname}") + tdSql.checkRows(1) + tdSql.query(f"select hyperloglog({k}) from {stbname} group by {k}") + + tdSql.execute(f'drop database {dbname}') + def __insert_data(self, rows): now_time = int(datetime.datetime.timestamp(datetime.datetime.now()) * 1000) @@ -311,6 +384,10 @@ class TDTestCase: tdLog.printNoPrefix("==========step4:after wal, all check again ") self.all_test() + tdLog.printNoPrefix("==========step5: distribute scene check") + self.__hyperloglog_check_distribute() + + def stop(self): tdSql.close() tdLog.success(f"{__file__} successfully executed") diff --git a/tests/system-test/2-query/last.py b/tests/system-test/2-query/last.py index 4ef13e9142f3a2ebc3ef55f6a2316fd6433908f3..ee65d22a22e28fa956d19ddecdcebe54e922ac71 100644 --- a/tests/system-test/2-query/last.py +++ b/tests/system-test/2-query/last.py @@ -1,6 +1,9 @@ +import random +import string from util.log import * from util.cases import * from util.sql import * +from util.common import * import numpy as np @@ -10,416 +13,258 @@ class TDTestCase: tdSql.init(conn.cursor()) self.rowNum = 10 + self.tbnum = 20 self.ts = 1537146000000 - - def run(self): - tdSql.prepare() + self.binary_str = 'taosdata' + self.nchar_str = '涛思数据' - tdSql.execute('''create table stb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned) tags(loc nchar(20))''') - tdSql.execute("create table stb_1 using stb tags('beijing')") - tdSql.execute("insert into stb_1(ts) values(%d)" % (self.ts - 1)) + def set_create_normaltable_sql(self, ntbname, column_dict): + column_sql = '' + for k, v in column_dict.items(): + column_sql += f"{k} {v}," + create_ntb_sql = f'create table {ntbname} (ts timestamp,{column_sql[:-1]})' + return create_ntb_sql - # last verifacation - tdSql.query("select last(*) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, None) - tdSql.query("select last(*) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, None) - tdSql.query("select last(col1) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col1) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col2) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col2) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col3) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col3) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col4) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col4) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col11) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col11) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col12) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col12) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col13) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col13) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col14) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col14) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col5) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col5) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col6) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col6) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col7) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col7) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col8) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col8) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col9) from stb_1") - tdSql.checkRows(0) - tdSql.query("select last(col9) from db.stb_1") - tdSql.checkRows(0) - tdSql.query("select count(col1) from stb_1 group by col7") - tdSql.checkRows(1) + def set_create_stable_sql(self,stbname,column_dict,tag_dict): + column_sql = '' + tag_sql = '' + for k,v in column_dict.items(): + column_sql += f"{k} {v}," + for k,v in tag_dict.items(): + tag_sql += f"{k} {v}," + create_stb_sql = f'create table {stbname} (ts timestamp,{column_sql[:-1]}) tags({tag_sql[:-1]})' + return create_stb_sql + + def last_check_stb_tb_base(self): + tdSql.prepare() + stbname = tdCom.getLongName(5, "letters") + column_dict = { + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + tag_dict = { + 'loc':'nchar(20)' + } + tdSql.execute(self.set_create_stable_sql(stbname,column_dict,tag_dict)) - for i in range(self.rowNum): - tdSql.execute("insert into stb_1 values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) + tdSql.execute(f"create table {stbname}_1 using {stbname} tags('beijing')") + tdSql.execute(f"insert into {stbname}_1(ts) values(%d)" % (self.ts - 1)) - tdSql.query("select last(*) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 10) - tdSql.query("select last(*) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 10) - tdSql.query("select last(col1) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col1) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from stb_1") + for i in [f'{stbname}_1', f'db.{stbname}_1']: + tdSql.query(f"select last(*) from {i}") + tdSql.checkRows(1) + tdSql.checkData(0, 1, None) + #!bug TD-16561 + # for i in ['stb','db.stb','stb','db.stb']: + # tdSql.query(f"select last(*) from {i}") + # tdSql.checkRows(1) + # tdSql.checkData(0, 1, None) + for i in column_dict.keys(): + for j in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']: + tdSql.query(f"select last({i}) from {j}") + tdSql.checkRows(0) + tdSql.query(f"select last({list(column_dict.keys())[0]}) from {stbname}_1 group by {list(column_dict.keys())[-1]}") tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col5) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col5) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col7) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col7) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col8) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col8) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col9) from stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col9) from db.stb_1") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col1,col2,col3) from stb_1") - tdSql.checkData(0,2,10) + for i in range(self.rowNum): + tdSql.execute(f"insert into {stbname}_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + for i in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']: + tdSql.query(f"select last(*) from {i}") + tdSql.checkRows(1) + tdSql.checkData(0, 1, 10) + for k, v in column_dict.items(): + for j in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']: + tdSql.query(f"select last({k}) from {j}") + tdSql.checkRows(1) + # tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned + if v.lower() == 'tinyint' or v.lower() == 'smallint' or v.lower() == 'int' or v.lower() == 'bigint' or v.lower() == 'tinyint unsigned' or v.lower() == 'smallint unsigned'\ + or v.lower() == 'int unsigned' or v.lower() == 'bigint unsigned': + tdSql.checkData(0, 0, 10) + # float,double + elif v.lower() == 'float' or v.lower() == 'double': + tdSql.checkData(0, 0, 9.1) + # bool + elif v.lower() == 'bool': + tdSql.checkData(0, 0, True) + # binary + elif 'binary' in v.lower(): + tdSql.checkData(0, 0, f'{self.binary_str}{self.rowNum}') + # nchar + elif 'nchar' in v.lower(): + tdSql.checkData(0, 0, f'{self.nchar_str}{self.rowNum}') + for i in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']: + tdSql.query(f"select last({list(column_dict.keys())[0]},{list(column_dict.keys())[1]},{list(column_dict.keys())[2]}) from {stbname}_1") + tdSql.checkData(0, 2, 10) - tdSql.query("select last(*) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 10) - tdSql.query("select last(*) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 1, 10) - tdSql.query("select last(col1) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col1) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col5) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col5) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col7) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col7) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col8) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col8) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col9) from stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col9) from db.stb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col1,col2,col3) from stb") - tdSql.checkData(0,2,10) - + tdSql.error(f"select {list(column_dict.keys())[0]} from {stbname} where last({list(column_dict.keys())[12]})='涛思数据10'") + tdSql.error(f"select {list(column_dict.keys())[0]} from {stbname}_1 where last({list(column_dict.keys())[12]})='涛思数据10'") + tdSql.execute('drop database db') - tdSql.execute('''create table ntb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned)''') - tdSql.execute("insert into ntb(ts) values(%d)" % (self.ts - 1)) - tdSql.query("select last(*) from ntb") + def last_check_ntb_base(self): + tdSql.prepare() + ntbname = tdCom.getLongName(5, "letters") + column_dict = { + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + create_ntb_sql = self.set_create_normaltable_sql(ntbname, column_dict) + tdSql.execute(create_ntb_sql) + tdSql.execute(f"insert into {ntbname}(ts) values(%d)" % (self.ts - 1)) + tdSql.query(f"select last(*) from {ntbname}") tdSql.checkRows(1) tdSql.checkData(0, 1, None) - tdSql.query("select last(*) from db.ntb") + tdSql.query(f"select last(*) from db.{ntbname}") tdSql.checkRows(1) tdSql.checkData(0, 1, None) - tdSql.query("select last(col1) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col1) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col2) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col2) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col3) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col3) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col4) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col4) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col11) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col11) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col12) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col12) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col13) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col13) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col14) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col14) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col5) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col5) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col6) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col6) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col7) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col7) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col8) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col8) from db.ntb") - tdSql.checkRows(0) - tdSql.query("select last(col9) from ntb") - tdSql.checkRows(0) - tdSql.query("select last(col9) from db.ntb") - tdSql.checkRows(0) - + for i in column_dict.keys(): + for j in [f'{ntbname}', f'db.{ntbname}']: + tdSql.query(f"select last({i}) from {j}") + tdSql.checkRows(0) for i in range(self.rowNum): - tdSql.execute("insert into ntb values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) - - tdSql.query("select last(*) from ntb") + tdSql.execute(f"insert into {ntbname} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + tdSql.query(f"select last(*) from {ntbname}") tdSql.checkRows(1) tdSql.checkData(0, 1, 10) - tdSql.query("select last(*) from db.ntb") + tdSql.query(f"select last(*) from db.{ntbname}") tdSql.checkRows(1) tdSql.checkData(0, 1, 10) - tdSql.query("select last(col1) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col1) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col2) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col3) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col4) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col11) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col12) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col13) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col14) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 10) - tdSql.query("select last(col5) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col5) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col6) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 9.1) - tdSql.query("select last(col7) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col7) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, True) - tdSql.query("select last(col8) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col8) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, 'taosdata10') - tdSql.query("select last(col9) from ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col9) from db.ntb") - tdSql.checkRows(1) - tdSql.checkData(0, 0, '涛思数据10') - tdSql.query("select last(col1,col2,col3) from ntb") - tdSql.checkData(0,2,10) + for k, v in column_dict.items(): + for j in [f'{ntbname}', f'db.{ntbname}']: + tdSql.query(f"select last({k}) from {j}") + tdSql.checkRows(1) + # tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned + if v.lower() == 'tinyint' or v.lower() == 'smallint' or v.lower() == 'int' or v.lower() == 'bigint' or v.lower() == 'tinyint unsigned' or v.lower() == 'smallint unsigned'\ + or v.lower() == 'int unsigned' or v.lower() == 'bigint unsigned': + tdSql.checkData(0, 0, 10) + # float,double + elif v.lower() == 'float' or v.lower() == 'double': + tdSql.checkData(0, 0, 9.1) + # bool + elif v.lower() == 'bool': + tdSql.checkData(0, 0, True) + # binary + elif 'binary' in v.lower(): + tdSql.checkData(0, 0, f'{self.binary_str}{self.rowNum}') + # nchar + elif 'nchar' in v.lower(): + tdSql.checkData(0, 0, f'{self.nchar_str}{self.rowNum}') + + tdSql.error( + f"select {list(column_dict.keys())[0]} from {ntbname} where last({list(column_dict.keys())[9]})='涛思数据10'") + + def last_check_stb_distribute(self): + # prepare data for vgroup 4 + dbname = tdCom.getLongName(10, "letters") + stbname = tdCom.getLongName(5, "letters") + vgroup_num = 4 + column_dict = { + 'col1': 'tinyint', + 'col2': 'smallint', + 'col3': 'int', + 'col4': 'bigint', + 'col5': 'tinyint unsigned', + 'col6': 'smallint unsigned', + 'col7': 'int unsigned', + 'col8': 'bigint unsigned', + 'col9': 'float', + 'col10': 'double', + 'col11': 'bool', + 'col12': 'binary(20)', + 'col13': 'nchar(20)' + } + + tdSql.execute( + f"create database if not exists {dbname} vgroups {vgroup_num}") + tdSql.execute(f'use {dbname}') + + # build 20 child tables,every table insert 10 rows + tdSql.execute(f'''create table {stbname}(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + for i in range(self.tbnum): + tdSql.execute( + f"create table {stbname}_{i} using {stbname} tags('beijing')") + tdSql.execute( + f"insert into {stbname}_{i}(ts) values(%d)" % (self.ts - 1-i)) + # for i in [f'{stbname}', f'{dbname}.{stbname}']: + # tdSql.query(f"select last(*) from {i}") + # tdSql.checkRows(1) + # tdSql.checkData(0, 1, None) + tdSql.query('show tables') + vgroup_list = [] + for i in range(len(tdSql.queryResult)): + vgroup_list.append(tdSql.queryResult[i][6]) + vgroup_list_set = set(vgroup_list) + for i in vgroup_list_set: + vgroups_num = vgroup_list.count(i) + if vgroups_num >= 2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit( + 'This scene does not meet the requirements with {vgroups_num} vgroup!\n') + + for i in range(self.tbnum): + for j in range(self.rowNum): + tdSql.execute(f"insert into {stbname}_{i} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + j + i, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 0.1, j + 0.1, j % 2, j + 1, j + 1)) + for i in [f'{stbname}', f'{dbname}.{stbname}']: + tdSql.query(f"select last(*) from {i}") + tdSql.checkRows(1) + tdSql.checkData(0, 1, 10) + for k, v in column_dict.items(): + for j in [f'{stbname}', f'{dbname}.{stbname}']: + tdSql.query(f"select last({k}) from {j}") + tdSql.checkRows(1) + # tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned + if v.lower() == 'tinyint' or v.lower() == 'smallint' or v.lower() == 'int' or v.lower() == 'bigint' or v.lower() == 'tinyint unsigned' or v.lower() == 'smallint unsigned'\ + or v.lower() == 'int unsigned' or v.lower() == 'bigint unsigned': + tdSql.checkData(0, 0, 10) + # float,double + elif v.lower() == 'float' or v.lower() == 'double': + tdSql.checkData(0, 0, 9.1) + # bool + elif v.lower() == 'bool': + tdSql.checkData(0, 0, True) + # binary + elif 'binary' in v.lower(): + tdSql.checkData(0, 0, f'{self.binary_str}{self.rowNum}') + # nchar + elif 'nchar' in v.lower(): + tdSql.checkData(0, 0, f'{self.nchar_str}{self.rowNum}') + tdSql.execute(f'drop database {dbname}') + + def run(self): + self.last_check_stb_tb_base() + self.last_check_ntb_base() + self.last_check_stb_distribute() - tdSql.error("select col1 from stb where last(col9)='涛思数据10'") - tdSql.error("select col1 from ntb where last(col9)='涛思数据10'") - tdSql.error("select col1 from stb_1 where last(col9)='涛思数据10'") def stop(self): tdSql.close() tdLog.success("%s successfully executed" % __file__) + tdCases.addWindows(__file__, TDTestCase()) -tdCases.addLinux(__file__, TDTestCase()) \ No newline at end of file +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/max.py b/tests/system-test/2-query/max.py index 5342c7d44988949f4754040b77ea416d20b79658..46426a5a9592fdc0e0b7d8695eecc92b24395c61 100644 --- a/tests/system-test/2-query/max.py +++ b/tests/system-test/2-query/max.py @@ -5,198 +5,213 @@ import numpy as np class TDTestCase: + updatecfgDict = {'debugFlag': 143 ,"cDebugFlag":143,"uDebugFlag":143 ,"rpcDebugFlag":143 , "tmrDebugFlag":143 , + "jniDebugFlag":143 ,"simDebugFlag":143,"dDebugFlag":143, "dDebugFlag":143,"vDebugFlag":143,"mDebugFlag":143,"qDebugFlag":143, + "wDebugFlag":143,"sDebugFlag":143,"tsdbDebugFlag":143,"tqDebugFlag":143 ,"fsDebugFlag":143 ,"fnDebugFlag":143, + "maxTablesPerVnode":2 ,"minTablesPerVnode":2,"tableIncStepPerVnode":2 } def init(self, conn, logSql): tdLog.debug("start to execute %s" % __file__) tdSql.init(conn.cursor()) self.rowNum = 10 self.ts = 1537146000000 - - def prepare_data(self): - - pass - def run(self): + self.binary_str = 'taosdata' + self.nchar_str = '涛思数据' + def max_check_stb_and_tb_base(self): tdSql.prepare() - intData = [] floatData = [] - - tdSql.execute('''create table stb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned) tags(loc nchar(20))''') + tdSql.execute('''create table stb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') tdSql.execute("create table stb_1 using stb tags('beijing')") - tdSql.execute('''create table ntb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned)''') for i in range(self.rowNum): - tdSql.execute("insert into ntb values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) + tdSql.execute(f"insert into stb_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) intData.append(i + 1) floatData.append(i + 0.1) + for i in ['ts','col11','col12','col13']: + for j in ['db.stb','stb','db.stb_1','stb_1']: + tdSql.error(f'select max({i} from {j} )') + + for i in range(1,11): + for j in ['db.stb','stb','db.stb_1','stb_1']: + tdSql.query(f"select max(col{i}) from {j}") + if i<9: + tdSql.checkData(0, 0, np.max(intData)) + elif i>=9: + tdSql.checkData(0, 0, np.max(floatData)) + tdSql.query("select max(col1) from stb_1 where col2<=5") + tdSql.checkData(0,0,5) + tdSql.query("select max(col1) from stb where col2<=5") + tdSql.checkData(0,0,5) + tdSql.execute('drop database db') + + def max_check_ntb_base(self): + tdSql.prepare() + intData = [] + floatData = [] + tdSql.execute('''create table ntb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20))''') for i in range(self.rowNum): - tdSql.execute("insert into stb_1 values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) + tdSql.execute(f"insert into ntb values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) intData.append(i + 1) - floatData.append(i + 0.1) - - # max verifacation - tdSql.error("select max(ts) from stb_1") - tdSql.error("select max(ts) from db.stb_1") - tdSql.error("select max(col7) from stb_1") - tdSql.error("select max(col7) from db.stb_1") - tdSql.error("select max(col8) from stb_1") - tdSql.error("select max(col8) from db.stb_1") - tdSql.error("select max(col9) from stb_1") - tdSql.error("select max(col9) from db.stb_1") - - tdSql.query("select max(col1) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col1) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from db.stb_1") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col5) from stb_1") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col5) from db.stb_1") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from stb_1") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from db.stb_1") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col1) from stb_1 where col2<=5") + floatData.append(i + 0.1) + for i in ['ts','col11','col12','col13']: + for j in ['db.ntb','ntb']: + tdSql.error(f'select max({i} from {j} )') + for i in range(1,11): + for j in ['db.ntb','ntb']: + tdSql.query(f"select max(col{i}) from {j}") + if i<9: + tdSql.checkData(0, 0, np.max(intData)) + elif i>=9: + tdSql.checkData(0, 0, np.max(floatData)) + tdSql.query("select max(col1) from ntb where col2<=5") tdSql.checkData(0,0,5) + tdSql.execute('drop database db') + + + def check_max_functions(self, tbname , col_name): + + max_sql = f"select max({col_name}) from {tbname};" + + same_sql = f"select {col_name} from {tbname} order by {col_name} desc limit 1" + + tdSql.query(max_sql) + max_result = tdSql.queryResult + + tdSql.query(same_sql) + same_result = tdSql.queryResult + + if max_result !=same_result: + tdLog.exit(" max function work not as expected, sql : %s "% max_sql) + else: + tdLog.info(" max function work as expected, sql : %s "% max_sql) + + + def support_distributed_aggregate(self): + + # prepate datas for 20 tables distributed at different vgroups + tdSql.execute("create database if not exists testdb keep 3650 duration 1000 vgroups 5") + tdSql.execute(" use testdb ") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t0 timestamp, t1 int, t2 bigint, t3 smallint, t4 tinyint, t5 float, t6 double, t7 bool, t8 binary(16),t9 nchar(32)) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(20): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( now(), {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, "binary{i}", "nchar{i}" )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + + for i in range(1,21): + if i ==1 or i == 4: + continue + else: + tbname = "ct"+f'{i}' + for j in range(9): + tdSql.execute( + f"insert into {tbname} values ( now()-{(i+j)*10}s, {1*(j+i)}, {11111*(j+i)}, {111*(j+i)}, {11*(j)}, {1.11*(j+i)}, {11.11*(j+i)}, {(j+i)%2}, 'binary{j}', 'nchar{j}', now()+{1*j}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdLog.info(" prepare data for distributed_aggregate done! ") + + # get vgroup_ids of all + tdSql.query("show vgroups ") + vgroups = tdSql.queryResult + + vnode_tables={} + + for vgroup_id in vgroups: + vnode_tables[vgroup_id[0]]=[] + + + # check sub_table of per vnode ,make sure sub_table has been distributed + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + vnode_tables[table_name[6]].append(table_name[0]) + + count = 0 + for k ,v in vnode_tables.items(): + if len(v)>=2: + count+=1 + if count < 2: + tdLog.exit(" the datas of all not satisfy sub_table has been distributed ") + + # check max function work status + + tdSql.query("show tables like 'ct%'") + table_names = tdSql.queryResult + tablenames = [] + for table_name in table_names: + tablenames.append(table_name[0]) + + tdSql.query("desc stb1") + col_names = tdSql.queryResult + + colnames = [] + for col_name in col_names: + if col_name[1] in ["INT" ,"BIGINT" ,"SMALLINT" ,"TINYINT" , "FLOAT" ,"DOUBLE"]: + colnames.append(col_name[0]) + for tablename in tablenames: + for colname in colnames: + self.check_max_functions(tablename,colname) + # max function with basic filter + print(vnode_tables) - tdSql.error("select max(ts) from stb") - tdSql.error("select max(ts) from db.stb") - tdSql.error("select max(col7) from stb") - tdSql.error("select max(col7) from db.stb") - tdSql.error("select max(col8) from stb") - tdSql.error("select max(col8) from db.stb") - tdSql.error("select max(col9) from stb") - tdSql.error("select max(col9) from db.stb") - - tdSql.query("select max(col1) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col1) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from db.stb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col5) from stb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col5) from db.stb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from stb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from db.stb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col1) from stb where col2<=5") - tdSql.checkData(0,0,5) + def run(self): + # max verifacation + self.max_check_stb_and_tb_base() + self.max_check_ntb_base() + + self.support_distributed_aggregate() - tdSql.error("select max(ts) from ntb") - tdSql.error("select max(ts) from db.ntb") - tdSql.error("select max(col7) from ntb") - tdSql.error("select max(col7) from db.ntb") - tdSql.error("select max(col8) from ntb") - tdSql.error("select max(col8) from db.ntb") - tdSql.error("select max(col9) from ntb") - tdSql.error("select max(col9) from db.ntb") - - tdSql.query("select max(col1) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col1) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col2) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col3) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col4) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col11) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col12) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col13) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col14) from db.ntb") - tdSql.checkData(0, 0, np.max(intData)) - tdSql.query("select max(col5) from ntb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col5) from db.ntb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from ntb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col6) from db.ntb") - tdSql.checkData(0, 0, np.max(floatData)) - tdSql.query("select max(col1) from stb_1 where col2<=5") - tdSql.checkData(0,0,5) def stop(self): tdSql.close() tdLog.success("%s successfully executed" % __file__) diff --git a/tests/system-test/2-query/tail.py b/tests/system-test/2-query/tail.py index 6039f3effa95f8014b31fd529138a872a6b9bedf..0e2110f9122c318dc0eff34a44dc453469c57bca 100644 --- a/tests/system-test/2-query/tail.py +++ b/tests/system-test/2-query/tail.py @@ -189,6 +189,7 @@ class TDTestCase: def check_tail_table(self , tbname , col_name , tail_rows , offset): tail_sql = f"select tail({col_name} , {tail_rows} , {offset}) from {tbname}" equal_sql = f"select {col_name} from (select ts , {col_name} from {tbname} order by ts desc limit {tail_rows} offset {offset}) order by ts" + #equal_sql = f"select {col_name} from {tbname} order by ts desc limit {tail_rows} offset {offset}" tdSql.query(tail_sql) tail_result = tdSql.queryResult @@ -293,22 +294,22 @@ class TDTestCase: tdSql.checkData(0, 0, None) tdSql.checkData(1, 0, None) - tdSql.query("select tail(c1,3,2) from ct4 where c1 >2 ") - tdSql.checkData(0, 0, 7) + tdSql.query("select tail(c1,3,2) from ct4 where c1 >2 order by 1") + tdSql.checkData(0, 0, 5) tdSql.checkData(1, 0, 6) - tdSql.checkData(2, 0, 5) + tdSql.checkData(2, 0, 7) - tdSql.query("select tail(c1,2,1) from ct4 where c2 between 0 and 99999") - tdSql.checkData(0, 0, 2) - tdSql.checkData(1, 0, 1) + tdSql.query("select tail(c1,2,1) from ct4 where c2 between 0 and 99999 order by 1") + tdSql.checkData(0, 0, 1) + tdSql.checkData(1, 0, 2) # tail with union all tdSql.query("select tail(c1,2,1) from ct4 union all select c1 from ct1") tdSql.checkRows(15) - tdSql.query("select tail(c1,2,1) from ct4 union all select c1 from ct2") + tdSql.query("select tail(c1,2,1) from ct4 union all select c1 from ct2 order by 1") tdSql.checkRows(2) - tdSql.checkData(0, 0, 1) - tdSql.checkData(1, 0, 0) + tdSql.checkData(0, 0, 0) + tdSql.checkData(1, 0, 1) tdSql.query("select tail(c2,2,1) from ct4 union all select abs(c2)/2 from ct4") tdSql.checkRows(14) @@ -334,18 +335,18 @@ class TDTestCase: tdSql.execute(f" insert into ttb1 values({ts_value} , {i})") tdSql.execute(f" insert into ttb2 values({ts_value} , {i})") - tdSql.query("select tail(tb2.num,3,2) from tb1, tb2 where tb1.ts=tb2.ts ") + tdSql.query("select tail(tb2.num,3,2) from tb1, tb2 where tb1.ts=tb2.ts order by 1 desc") tdSql.checkRows(3) - tdSql.checkData(0,0,5) + tdSql.checkData(0,0,7) tdSql.checkData(1,0,6) - tdSql.checkData(2,0,7) + tdSql.checkData(2,0,5) # nest query # tdSql.query("select tail(c1,2) from (select c1 from ct1)") - tdSql.query("select c1 from (select tail(c1,2) c1 from ct4)") + tdSql.query("select c1 from (select tail(c1,2) c1 from ct4) order by 1 nulls first") tdSql.checkRows(2) - tdSql.checkData(0, 0, 0) - tdSql.checkData(1, 0, None) + tdSql.checkData(0, 0, None) + tdSql.checkData(1, 0, 0) tdSql.query("select sum(c1) from (select tail(c1,2) c1 from ct1)") tdSql.checkRows(1) diff --git a/tests/system-test/2-query/top.py b/tests/system-test/2-query/top.py index 146bb349377ef539cfab4e294337621a807cfa27..83f535856ee8b9456d7759128063d5149df82ea1 100644 --- a/tests/system-test/2-query/top.py +++ b/tests/system-test/2-query/top.py @@ -11,6 +11,9 @@ # -*- coding: utf-8 -*- +import random +import string +from util.common import * from util.log import * from util.cases import * from util.sql import * @@ -22,82 +25,89 @@ class TDTestCase: tdSql.init(conn.cursor()) self.rowNum = 10 + self.tbnum = 20 self.ts = 1537146000000 - - def run(self): + self.binary_str = 'taosdata' + self.nchar_str = '涛思数据' + def top_check_base(self): tdSql.prepare() - - - - tdSql.execute('''create table test(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, - col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned) tags(loc nchar(20))''') - tdSql.execute("create table test1 using test tags('beijing')") + tdSql.execute('''create table stb(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + tdSql.execute("create table stb_1 using stb tags('beijing')") for i in range(self.rowNum): - tdSql.execute("insert into test1 values(%d, %d, %d, %d, %d, %f, %f, %d, 'taosdata%d', '涛思数据%d', %d, %d, %d, %d)" - % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1)) - - - # top verifacation - tdSql.error("select top(ts, 10) from test") - tdSql.error("select top(col1, 0) from test") - tdSql.error("select top(col1, 101) from test") - tdSql.error("select top(col2, 0) from test") - tdSql.error("select top(col2, 101) from test") - tdSql.error("select top(col3, 0) from test") - tdSql.error("select top(col3, 101) from test") - tdSql.error("select top(col4, 0) from test") - tdSql.error("select top(col4, 101) from test") - tdSql.error("select top(col5, 0) from test") - tdSql.error("select top(col5, 101) from test") - tdSql.error("select top(col6, 0) from test") - tdSql.error("select top(col6, 101) from test") - tdSql.error("select top(col7, 10) from test") - tdSql.error("select top(col8, 10) from test") - tdSql.error("select top(col9, 10) from test") - tdSql.error("select top(col11, 0) from test") - tdSql.error("select top(col11, 101) from test") - tdSql.error("select top(col12, 0) from test") - tdSql.error("select top(col12, 101) from test") - tdSql.error("select top(col13, 0) from test") - tdSql.error("select top(col13, 101) from test") - tdSql.error("select top(col14, 0) from test") - tdSql.error("select top(col14, 101) from test") - - tdSql.query("select top(col1, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col2, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col3, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col4, 2) from test") + tdSql.execute(f"insert into stb_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + column_list = ['col1','col2','col3','col4','col5','col6','col7','col8'] + error_column_list = ['col11','col12','col13'] + error_param_list = [0,101] + for i in column_list: + tdSql.query(f'select top({i},2) from stb_1') + tdSql.checkRows(2) + tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) + for j in error_param_list: + tdSql.error(f'select top({i},{j}) from stb_1') + for i in error_column_list: + tdSql.error(f'select top({i},10) from stb_1') + tdSql.query("select ts,top(col1, 2),ts from stb_1 group by tbname") tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col11, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col12, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col13, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select top(col14, 2) from test") - tdSql.checkRows(2) - tdSql.checkEqual(tdSql.queryResult,[(9,),(10,)]) - tdSql.query("select ts,top(col1, 2),ts from test1") - tdSql.checkRows(2) - tdSql.query("select top(col14, 100) from test") - tdSql.checkRows(10) - tdSql.query("select ts,top(col1, 2),ts from test group by tbname") - tdSql.checkRows(2) - tdSql.query('select top(col2,1) from test interval(1y) order by col2') + tdSql.query('select top(col2,1) from stb_1 interval(1y) order by col2') tdSql.checkData(0,0,10) + tdSql.error("select * from stb_1 where top(col2,1)=1") + tdSql.execute('drop database db') + def top_check_stb_distribute(self): + # prepare data for vgroup 4 + dbname = tdCom.getLongName(10, "letters") + stbname = tdCom.getLongName(5, "letters") + tdSql.execute(f"create database if not exists {dbname} vgroups 2") + tdSql.execute(f'use {dbname}') + # build 20 child tables,every table insert 10 rows + tdSql.execute(f'''create table {stbname}(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 tinyint unsigned, col6 smallint unsigned, + col7 int unsigned, col8 bigint unsigned, col9 float, col10 double, col11 bool, col12 binary(20), col13 nchar(20)) tags(loc nchar(20))''') + for i in range(self.tbnum): + tdSql.execute(f"create table {stbname}_{i} using {stbname} tags('beijing')") + tdSql.execute(f"insert into {stbname}_{i}(ts) values(%d)" % (self.ts - 1-i)) + column_list = ['col1','col2','col3','col4','col5','col6','col7','col8'] + for i in [f'{stbname}', f'{dbname}.{stbname}']: + for j in column_list: + tdSql.query(f"select top({j},1) from {i}") + tdSql.checkRows(0) + tdSql.query('show tables') + vgroup_list = [] + for i in range(len(tdSql.queryResult)): + vgroup_list.append(tdSql.queryResult[i][6]) + vgroup_list_set = set(vgroup_list) + + for i in vgroup_list_set: + vgroups_num = vgroup_list.count(i) + if vgroups_num >=2: + tdLog.info(f'This scene with {vgroups_num} vgroups is ok!') + continue + else: + tdLog.exit(f'This scene does not meet the requirements with {vgroups_num} vgroup!\n') + for i in range(self.rowNum): + for j in range(self.tbnum): + tdSql.execute(f"insert into {stbname}_{j} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')" + % (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1)) + + error_column_list = ['col11','col12','col13'] + error_param_list = [0,101] + for i in column_list: + tdSql.query(f'select top({i},2) from {stbname}') + tdSql.checkRows(2) + tdSql.checkEqual(tdSql.queryResult,[(10,),(10,)]) + for j in error_param_list: + tdSql.error(f'select top({i},{j}) from {stbname}') + for i in error_column_list: + tdSql.error(f'select top({i},10) from {stbname}') - tdSql.error("select * from test where bottom(col2,1)=1") - tdSql.error("select top(col14, 0) from test;") + tdSql.query(f"select ts,top(col1, 2),ts from {stbname} group by tbname") + tdSql.checkRows(2*self.tbnum) + tdSql.query(f'select top(col2,1) from {stbname} interval(1y) order by col2') + tdSql.checkData(0,0,10) + tdSql.error(f"select * from {stbname} where top(col2,1)=1") + def run(self): + self.top_check_base() + self.top_check_stb_distribute() def stop(self): tdSql.close() tdLog.success("%s successfully executed" % __file__) diff --git a/tests/system-test/2-query/ttl_comment.py b/tests/system-test/2-query/ttl_comment.py new file mode 100644 index 0000000000000000000000000000000000000000..8fc582c587dcdf23361cd48c140280363772df72 --- /dev/null +++ b/tests/system-test/2-query/ttl_comment.py @@ -0,0 +1,206 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, db_test.stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import taos +from util.log import tdLog +from util.cases import tdCases +from util.sql import tdSql + +class TDTestCase: + def caseDescription(self): + ''' + ttl/comment test + ''' + return + + def init(self, conn, logSql): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor(), logSql) + + def run(self): + tdSql.prepare() + + tdSql.error("create table ttl_table1(ts timestamp, i int) ttl 1.1") + tdSql.error("create table ttl_table2(ts timestamp, i int) ttl 1e1") + tdSql.error("create table ttl_table3(ts timestamp, i int) ttl -1") + + print("============== STEP 1 ===== test normal table") + + tdSql.execute("create table normal_table1(ts timestamp, i int)") + tdSql.execute("create table normal_table2(ts timestamp, i int) comment '' ttl 3") + tdSql.execute("create table normal_table3(ts timestamp, i int) ttl 2100000000020 comment 'hello'") + + tdSql.query("show tables like 'normal_table1'") + tdSql.checkData(0, 0, 'normal_table1') + tdSql.checkData(0, 7, 0) + tdSql.checkData(0, 8, None) + + + tdSql.query("show tables like 'normal_table2'") + tdSql.checkData(0, 0, 'normal_table2') + tdSql.checkData(0, 7, 3) + tdSql.checkData(0, 8, '') + + + tdSql.query("show tables like 'normal_table3'") + tdSql.checkData(0, 0, 'normal_table3') + tdSql.checkData(0, 7, 2147483647) + tdSql.checkData(0, 8, 'hello') + + tdSql.execute("alter table normal_table1 comment 'nihao'") + tdSql.query("show tables like 'normal_table1'") + tdSql.checkData(0, 0, 'normal_table1') + tdSql.checkData(0, 8, 'nihao') + + tdSql.execute("alter table normal_table1 comment ''") + tdSql.query("show tables like 'normal_table1'") + tdSql.checkData(0, 0, 'normal_table1') + tdSql.checkData(0, 8, '') + + tdSql.execute("alter table normal_table2 comment 'fly'") + tdSql.query("show tables like 'normal_table2'") + tdSql.checkData(0, 0, 'normal_table2') + tdSql.checkData(0, 8, 'fly') + + tdSql.execute("alter table normal_table3 comment 'fly'") + tdSql.query("show tables like 'normal_table3'") + tdSql.checkData(0, 0, 'normal_table3') + tdSql.checkData(0, 8, 'fly') + + tdSql.execute("alter table normal_table1 ttl 1") + tdSql.query("show tables like 'normal_table1'") + tdSql.checkData(0, 0, 'normal_table1') + tdSql.checkData(0, 7, 1) + + tdSql.execute("alter table normal_table3 ttl 0") + tdSql.query("show tables like 'normal_table3'") + tdSql.checkData(0, 0, 'normal_table3') + tdSql.checkData(0, 7, 0) + + + print("============== STEP 2 ===== test super table") + + tdSql.execute("create table super_table1(ts timestamp, i int) tags(t int)") + tdSql.execute("create table super_table2(ts timestamp, i int) tags(t int) comment ''") + tdSql.execute("create table super_table3(ts timestamp, i int) tags(t int) comment 'super'") + + tdSql.query("show stables like 'super_table1'") + tdSql.checkData(0, 0, 'super_table1') + tdSql.checkData(0, 6, None) + + + tdSql.query("show stables like 'super_table2'") + tdSql.checkData(0, 0, 'super_table2') + tdSql.checkData(0, 6, '') + + + tdSql.query("show stables like 'super_table3'") + tdSql.checkData(0, 0, 'super_table3') + tdSql.checkData(0, 6, 'super') + + + tdSql.execute("alter table super_table1 comment 'nihao'") + tdSql.query("show stables like 'super_table1'") + tdSql.checkData(0, 0, 'super_table1') + tdSql.checkData(0, 6, 'nihao') + + tdSql.execute("alter table super_table1 comment ''") + tdSql.query("show stables like 'super_table1'") + tdSql.checkData(0, 0, 'super_table1') + tdSql.checkData(0, 6, '') + + tdSql.execute("alter table super_table2 comment 'fly'") + tdSql.query("show stables like 'super_table2'") + tdSql.checkData(0, 0, 'super_table2') + tdSql.checkData(0, 6, 'fly') + + tdSql.execute("alter table super_table3 comment 'tdengine'") + tdSql.query("show stables like 'super_table3'") + tdSql.checkData(0, 0, 'super_table3') + tdSql.checkData(0, 6, 'tdengine') + + print("============== STEP 3 ===== test child table") + + tdSql.execute("create table child_table1 using super_table1 tags(1) ttl 10") + tdSql.execute("create table child_table2 using super_table1 tags(1) comment ''") + tdSql.execute("create table child_table3 using super_table1 tags(1) comment 'child'") + tdSql.execute("insert into child_table4 using super_table1 tags(1) values(now, 1)") + + + tdSql.query("show tables like 'child_table1'") + tdSql.checkData(0, 0, 'child_table1') + tdSql.checkData(0, 7, 10) + tdSql.checkData(0, 8, None) + + + tdSql.query("show tables like 'child_table2'") + tdSql.checkData(0, 0, 'child_table2') + tdSql.checkData(0, 7, 0) + tdSql.checkData(0, 8, '') + + + tdSql.query("show tables like 'child_table3'") + tdSql.checkData(0, 0, 'child_table3') + tdSql.checkData(0, 8, 'child') + + tdSql.query("show tables like 'child_table4'") + tdSql.checkData(0, 0, 'child_table4') + tdSql.checkData(0, 7, 0) + tdSql.checkData(0, 8, None) + + + tdSql.execute("alter table child_table1 comment 'nihao'") + tdSql.query("show tables like 'child_table1'") + tdSql.checkData(0, 0, 'child_table1') + tdSql.checkData(0, 8, 'nihao') + + tdSql.execute("alter table child_table1 comment ''") + tdSql.query("show tables like 'child_table1'") + tdSql.checkData(0, 0, 'child_table1') + tdSql.checkData(0, 8, '') + + tdSql.execute("alter table child_table2 comment 'fly'") + tdSql.query("show tables like 'child_table2'") + tdSql.checkData(0, 0, 'child_table2') + tdSql.checkData(0, 8, 'fly') + + tdSql.execute("alter table child_table3 comment 'tdengine'") + tdSql.query("show tables like 'child_table3'") + tdSql.checkData(0, 0, 'child_table3') + tdSql.checkData(0, 8, 'tdengine') + + + tdSql.execute("alter table child_table4 comment 'tdengine'") + tdSql.query("show tables like 'child_table4'") + tdSql.checkData(0, 0, 'child_table4') + tdSql.checkData(0, 8, 'tdengine') + + tdSql.execute("alter table child_table4 ttl 9") + tdSql.query("show tables like 'child_table4'") + tdSql.checkData(0, 0, 'child_table4') + tdSql.checkData(0, 7, 9) + + tdSql.execute("alter table child_table3 ttl 9") + tdSql.query("show tables like 'child_table3'") + tdSql.checkData(0, 0, 'child_table3') + tdSql.checkData(0, 7, 9) + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) + diff --git a/tests/system-test/2-query/unique.py b/tests/system-test/2-query/unique.py index 227efa6f9ceda24df73830bd46838fd657b67d48..f910ff14393395ec658522c1c0dad2a56ad7946a 100644 --- a/tests/system-test/2-query/unique.py +++ b/tests/system-test/2-query/unique.py @@ -93,8 +93,8 @@ class TDTestCase: "select unique(c1) , min(c1) from t1", "select unique(c1) , spread(c1) from t1", "select unique(c1) , diff(c1) from t1", - "select unique(c1) , abs(c1) from t1", - "select unique(c1) , c1 from t1", + #"select unique(c1) , abs(c1) from t1", # support + #"select unique(c1) , c1 from t1", "select unique from stb1 partition by tbname", "select unique(123--123)==1 from stb1 partition by tbname", "select unique(123) from stb1 partition by tbname", @@ -104,21 +104,21 @@ class TDTestCase: "select unique(c1 ,c2 ) from stb1 partition by tbname", "select unique(c1 ,NULL) from stb1 partition by tbname", "select unique(,) from stb1 partition by tbname;", - "select unique(floor(c1) ab from stb1 partition by tbname)", - "select unique(c1) as int from stb1 partition by tbname", + #"select unique(floor(c1) ab from stb1 partition by tbname)", # support + #"select unique(c1) as int from stb1 partition by tbname", "select unique('c1') from stb1 partition by tbname", "select unique(NULL) from stb1 partition by tbname", "select unique('') from stb1 partition by tbname", "select unique(c%) from stb1 partition by tbname", - #"select unique(t1) from stb1 partition by tbname", + #"select unique(t1) from stb1 partition by tbname", # support "select unique(True) from stb1 partition by tbname", "select unique(c1) , count(c1) from stb1 partition by tbname", "select unique(c1) , avg(c1) from stb1 partition by tbname", "select unique(c1) , min(c1) from stb1 partition by tbname", "select unique(c1) , spread(c1) from stb1 partition by tbname", "select unique(c1) , diff(c1) from stb1 partition by tbname", - "select unique(c1) , abs(c1) from stb1 partition by tbname", - "select unique(c1) , c1 from stb1 partition by tbname" + #"select unique(c1) , abs(c1) from stb1 partition by tbname", # support + #"select unique(c1) , c1 from stb1 partition by tbname" # support ] for error_sql in error_sql_lists: @@ -198,7 +198,7 @@ class TDTestCase: unique_datas = [] for elem in unique_result: unique_datas.append(elem[0]) - + unique_datas.sort(key=lambda x: (x is None, x)) tdSql.query(origin_sql) origin_result = tdSql.queryResult @@ -212,6 +212,7 @@ class TDTestCase: continue else: pre_unique.append(elem) + pre_unique.sort(key=lambda x: (x is None, x)) if pre_unique == unique_datas: tdLog.info(" unique query check pass , unique sql is: %s" %unique_sql) @@ -266,16 +267,16 @@ class TDTestCase: tdSql.checkRows(10) tdSql.error("select unique(c1),tbname from ct1") - tdSql.error("select unique(c1),t1 from ct1") + #tdSql.error("select unique(c1),t1 from ct1") #support # unique with common col - tdSql.error("select unique(c1) ,ts from ct1") - tdSql.error("select unique(c1) ,c1 from ct1") + #tdSql.error("select unique(c1) ,ts from ct1") + #tdSql.error("select unique(c1) ,c1 from ct1") # unique with scalar function - tdSql.error("select unique(c1) ,abs(c1) from ct1") + #tdSql.error("select unique(c1) ,abs(c1) from ct1") tdSql.error("select unique(c1) , unique(c2) from ct1") - tdSql.error("select unique(c1) , abs(c2)+2 from ct1") + #tdSql.error("select unique(c1) , abs(c2)+2 from ct1") # unique with aggregate function @@ -288,13 +289,13 @@ class TDTestCase: tdSql.query("select unique(c1) from ct4 where c1 is null") tdSql.checkData(0, 0, None) - tdSql.query("select unique(c1) from ct4 where c1 >2 ") - tdSql.checkData(0, 0, 8) - tdSql.checkData(1, 0, 7) - tdSql.checkData(2, 0, 6) - tdSql.checkData(5, 0, 3) + tdSql.query("select unique(c1) from ct4 where c1 >2 order by 1") + tdSql.checkData(0, 0, 3) + tdSql.checkData(1, 0, 4) + tdSql.checkData(2, 0, 5) + tdSql.checkData(5, 0, 8) - tdSql.query("select unique(c1) from ct4 where c2 between 0 and 99999") + tdSql.query("select unique(c1) from ct4 where c2 between 0 and 99999 order by 1 desc") tdSql.checkData(0, 0, 8) tdSql.checkData(1, 0, 7) tdSql.checkData(2, 0, 6) @@ -335,23 +336,23 @@ class TDTestCase: tdSql.execute(f" insert into ttb1 values({ts_value} , {i})") tdSql.execute(f" insert into ttb2 values({ts_value} , {i})") - tdSql.query("select unique(tb2.num) from tb1, tb2 where tb1.ts=tb2.ts ") + tdSql.query("select unique(tb2.num) from tb1, tb2 where tb1.ts=tb2.ts order by 1") tdSql.checkRows(10) tdSql.checkData(0,0,0) tdSql.checkData(1,0,1) tdSql.checkData(2,0,2) tdSql.checkData(9,0,9) - tdSql.query("select unique(tb2.num) from tb1, tb2 where tb1.ts=tb2.ts union all select unique(tb1.num) from tb1, tb2 where tb1.ts=tb2.ts ") + tdSql.query("select unique(tb2.num) from tb1, tb2 where tb1.ts=tb2.ts union all select unique(tb1.num) from tb1, tb2 where tb1.ts=tb2.ts order by 1") tdSql.checkRows(20) tdSql.checkData(0,0,0) - tdSql.checkData(1,0,1) - tdSql.checkData(2,0,2) - tdSql.checkData(9,0,9) + tdSql.checkData(2,0,1) + tdSql.checkData(4,0,2) + tdSql.checkData(18,0,9) # nest query # tdSql.query("select unique(c1) from (select c1 from ct1)") - tdSql.query("select c1 from (select unique(c1) c1 from ct4)") + tdSql.query("select c1 from (select unique(c1) c1 from ct4) order by 1 desc nulls first") tdSql.checkRows(10) tdSql.checkData(0, 0, None) tdSql.checkData(1, 0, 8) @@ -366,7 +367,7 @@ class TDTestCase: tdSql.checkData(0, 0, 45) tdSql.checkData(1, 0, 45) - tdSql.query("select 1-abs(c1) from (select unique(c1) c1 from ct4)") + tdSql.query("select 1-abs(c1) from (select unique(c1) c1 from ct4) order by 1 nulls first") tdSql.checkRows(10) tdSql.checkData(0, 0, None) tdSql.checkData(1, 0, -7.000000000) @@ -421,7 +422,7 @@ class TDTestCase: f"insert into sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" ) - tdSql.query("select unique(c2) from sub1_bound") + tdSql.query("select unique(c2) from sub1_bound order by 1 desc") tdSql.checkRows(5) tdSql.checkData(0,0,9223372036854775807) diff --git a/tests/system-test/6-cluster/5dnode3mnodeDrop.py b/tests/system-test/6-cluster/5dnode3mnodeDrop.py index f999a16b05ee654767f89f3627eb2664b0084bce..e81d5295f28bd0d3335ef3d2ed0c1017701c5efe 100644 --- a/tests/system-test/6-cluster/5dnode3mnodeDrop.py +++ b/tests/system-test/6-cluster/5dnode3mnodeDrop.py @@ -75,7 +75,7 @@ class TDTestCase: testCluster = False valgrind = 0 hostname = socket.gethostname() - print(hostname) + tdLog.debug(hostname) dnodes = [] start_port = 6030 start_port_sec = 6130 @@ -102,12 +102,12 @@ class TDTestCase: # create cluster for dnode in self.TDDnodes.dnodes[1:]: - # print(dnode.cfgDict) + # tdLog.debug(dnode.cfgDict) dnode_id = dnode.cfgDict["fqdn"] + ":" +dnode.cfgDict["serverPort"] dnode_first_host = dnode.cfgDict["firstEp"].split(":")[0] dnode_first_port = dnode.cfgDict["firstEp"].split(":")[-1] cmd = f" taos -h {dnode_first_host} -P {dnode_first_port} -s ' create dnode \"{dnode_id} \" ' ;" - print(cmd) + tdLog.debug(cmd) os.system(cmd) time.sleep(2) @@ -119,26 +119,26 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break count+=1 else: - print(tdSql.queryResult) - print("three mnodes is not ready in 10s ") + tdLog.debug(tdSql.queryResult) + tdLog.debug("three mnodes is not ready in 10s ") return -1 tdSql.query("show mnodes;") @@ -156,19 +156,19 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='offline' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break elif tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 1;") @@ -188,15 +188,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='offline': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 2;") @@ -218,15 +218,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[2][2]=='offline': if tdSql.queryResult[1][2]=='follower': - print("stop mnodes on dnode 3 successfully in 10s") + tdLog.debug("stop mnodes on dnode 3 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 3 failed in 10s") + tdLog.debug("stop mnodes on dnode 3 failed in 10s") return -1 tdSql.error("drop mnode on dnode 3;") tdSql.query("show mnodes;") @@ -268,8 +268,9 @@ class TDTestCase: tdSql.error("drop mnode on dnode 1") tdSql.query("show dnodes;") - print(tdSql.queryResult) - # drop and follower of mnode + tdLog.debug(tdSql.queryResult) + + # drop follower of mnode dropcount =0 while dropcount <= 10: for i in range(1,3): @@ -281,7 +282,7 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(2): - print("drop mnode %d successfully"%(i+1)) + tdLog.debug("drop mnode %d successfully"%(i+1)) break count+=1 tdLog.debug("create mnode on dnode %d"%(i+1)) @@ -291,7 +292,7 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3): - print("drop mnode %d successfully"%(i+1)) + tdLog.debug("drop mnode %d successfully"%(i+1)) break count+=1 dropcount+=1 @@ -306,7 +307,7 @@ class TDTestCase: def run(self): - # print(self.master_dnode.cfgDict) + # tdLog.debug(self.master_dnode.cfgDict) self.buildcluster(5) self.five_dnode_three_mnode() diff --git a/tests/system-test/6-cluster/5dnode3mnodeDropInsert.py b/tests/system-test/6-cluster/5dnode3mnodeDropInsert.py new file mode 100644 index 0000000000000000000000000000000000000000..cfa39206048b6579238ddf55d74cd10e351e0edc --- /dev/null +++ b/tests/system-test/6-cluster/5dnode3mnodeDropInsert.py @@ -0,0 +1,428 @@ +from ssl import ALERT_DESCRIPTION_CERTIFICATE_UNOBTAINABLE +import taos +import sys +import time +import os + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import TDDnodes +from util.dnodes import TDDnode +import time +import socket +import subprocess +from multiprocessing import Process +import threading +import time +import inspect +import ctypes +class MyDnodes(TDDnodes): + def __init__(self ,dnodes_lists): + super(MyDnodes,self).__init__() + self.dnodes = dnodes_lists # dnode must be TDDnode instance + self.simDeployed = False + + +class TDTestCase: + + def init(self,conn ,logSql): + tdLog.debug(f"start to excute {__file__}") + self.TDDnodes = None + self.ts = 1500000000000 + + def buildcluster(self,dnodenumber): + self.depoly_cluster(dnodenumber) + self.master_dnode = self.TDDnodes.dnodes[0] + self.host=self.master_dnode.cfgDict["fqdn"] + conn1 = taos.connect(self.master_dnode.cfgDict["fqdn"] , config=self.master_dnode.cfgDir) + tdSql.init(conn1.cursor()) + + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def _async_raise(self, tid, exctype): + """raises the exception, performs cleanup if needed""" + if not inspect.isclass(exctype): + exctype = type(exctype) + res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype)) + if res == 0: + raise ValueError("invalid thread id") + elif res != 1: + # """if it returns a number greater than one, you're in trouble, + # and you should call it again with exc=NULL to revert the effect""" + ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None) + raise SystemError("PyThreadState_SetAsyncExc failed") + + def stop_thread(self,thread): + self._async_raise(thread.ident, SystemExit) + + + def createDbTbale(self,dbcountStart,dbcountStop,stbname,chilCount): + # fisrt add data : db\stable\childtable\general table + + for couti in range(dbcountStart,dbcountStop): + tdLog.debug("drop database if exists db%d" %couti) + tdSql.execute("drop database if exists db%d" %couti) + print("create database if not exists db%d replica 1 duration 300" %couti) + tdSql.execute("create database if not exists db%d replica 1 duration 300" %couti) + tdSql.execute("use db%d" %couti) + tdSql.execute( + '''create table %s + (ts timestamp, c1 int, c2 bigint,c3 binary(16), c4 timestamp) + tags (t1 int) + '''%stbname + ) + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(chilCount): + tdSql.execute(f'create table {stbname}_{i+1} using {stbname} tags ( {i+1} )') + + def insertTabaleData(self,dbcountStart,dbcountStop,stbname,chilCount,ts_start,rowCount): + # insert data : create childtable and data + + for couti in range(dbcountStart,dbcountStop): + tdSql.execute("use db%d" %couti) + pre_insert = "insert into " + sql = pre_insert + chilCount=int(chilCount) + allRows=chilCount*rowCount + tdLog.debug("doing insert data into stable-index:%s rows:%d ..."%(stbname, allRows)) + exeStartTime=time.time() + for i in range(0,chilCount): + sql += " %s_%d values "%(stbname,i) + for j in range(rowCount): + sql += "(%d, %d, %d,'taos_%d',%d) "%(ts_start + j*1000, j, j, j, ts_start + j*1000) + if j >0 and j%4000 == 0: + # print(sql) + tdSql.execute(sql) + sql = "insert into %s_%d values " %(stbname,i) + # end sql + if sql != pre_insert: + # print(sql) + print(len(sql)) + tdSql.execute(sql) + exeEndTime=time.time() + spendTime=exeEndTime-exeStartTime + speedInsert=allRows/spendTime + tdLog.debug("spent %.2fs to INSERT %d rows into %s , insert rate is %.2f rows/s... [OK]"% (spendTime,allRows,stbname,speedInsert)) + + def checkData(self,dbname,stbname,stableCount,CtableCount,rowsPerSTable,): + tdSql.execute("use %s"%dbname) + tdSql.query("show stables") + tdSql.checkRows(stableCount) + tdSql.query("show tables") + tdSql.checkRows(CtableCount) + for i in range(stableCount): + tdSql.query("select count(*) from %s%d"%(stbname,i)) + tdSql.checkData(0,0,rowsPerSTable) + return + + + def depoly_cluster(self ,dnodes_nums): + + testCluster = False + valgrind = 0 + hostname = socket.gethostname() + dnodes = [] + start_port = 6030 + start_port_sec = 6130 + for num in range(1, dnodes_nums+1): + dnode = TDDnode(num) + dnode.addExtraCfg("firstEp", f"{hostname}:{start_port}") + dnode.addExtraCfg("fqdn", f"{hostname}") + dnode.addExtraCfg("serverPort", f"{start_port + (num-1)*100}") + dnode.addExtraCfg("monitorFqdn", hostname) + dnode.addExtraCfg("monitorPort", 7043) + dnode.addExtraCfg("secondEp", f"{hostname}:{start_port_sec}") + dnodes.append(dnode) + + self.TDDnodes = MyDnodes(dnodes) + self.TDDnodes.init("") + self.TDDnodes.setTestCluster(testCluster) + self.TDDnodes.setValgrind(valgrind) + self.TDDnodes.stopAll() + for dnode in self.TDDnodes.dnodes: + self.TDDnodes.deploy(dnode.index,{}) + + for dnode in self.TDDnodes.dnodes: + self.TDDnodes.starttaosd(dnode.index) + + # create cluster + for dnode in self.TDDnodes.dnodes[1:]: + # print(dnode.cfgDict) + dnode_id = dnode.cfgDict["fqdn"] + ":" +dnode.cfgDict["serverPort"] + dnode_first_host = dnode.cfgDict["firstEp"].split(":")[0] + dnode_first_port = dnode.cfgDict["firstEp"].split(":")[-1] + cmd = f" taos -h {dnode_first_host} -P {dnode_first_port} -s ' create dnode \"{dnode_id} \" ' ;" + print(cmd) + os.system(cmd) + + time.sleep(2) + tdLog.info(" create cluster with %d dnode done! " %dnodes_nums) + + def checkdnodes(self,dnodenumber): + count=0 + while count < 10: + time.sleep(1) + statusReadyBumber=0 + tdSql.query("show dnodes;") + if tdSql.checkRows(dnodenumber) : + print("dnode is %d nodes"%dnodenumber) + for i in range(dnodenumber): + if tdSql.queryResult[i][4] !='ready' : + status=tdSql.queryResult[i][4] + print("dnode:%d status is %s "%(i,status)) + break + else: + statusReadyBumber+=1 + print(statusReadyBumber) + if statusReadyBumber == dnodenumber : + print("all of %d mnodes is ready in 10s "%dnodenumber) + return True + break + count+=1 + else: + print("%d mnodes is not ready in 10s "%dnodenumber) + return False + + + def check3mnode(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='follower': + print("three mnodes is ready in 10s") + break + elif tdSql.queryResult[0][2]=='follower' : + if tdSql.queryResult[1][2]=='leader': + if tdSql.queryResult[2][2]=='follower': + print("three mnodes is ready in 10s") + break + elif tdSql.queryResult[0][2]=='follower' : + if tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='leader': + print("three mnodes is ready in 10s") + break + count+=1 + else: + print("three mnodes is not ready in 10s ") + return -1 + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,3,'ready') + + def check3mnode1off(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='offline' : + if tdSql.queryResult[1][2]=='leader': + if tdSql.queryResult[2][2]=='follower': + print("stop mnodes on dnode 2 successfully in 10s") + break + elif tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='leader': + print("stop mnodes on dnode 2 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 2 failed in 10s ") + return -1 + tdSql.error("drop mnode on dnode 1;") + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'offline') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,3,'ready') + + def check3mnode2off(self): + count=0 + while count < 40: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[1][2]=='offline': + if tdSql.queryResult[2][2]=='follower': + print("stop mnodes on dnode 2 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 2 failed in 10s ") + return -1 + tdSql.error("drop mnode on dnode 2;") + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,2,'offline') + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,2,'follower') + tdSql.checkData(2,3,'ready') + + def check3mnode3off(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[2][2]=='offline': + if tdSql.queryResult[1][2]=='follower': + print("stop mnodes on dnode 3 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 3 failed in 10s") + return -1 + tdSql.error("drop mnode on dnode 3;") + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,2,'follower') + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,2,'offline') + tdSql.checkData(2,3,'ready') + + def five_dnode_three_mnode(self,dnodenumber): + # testcase parameters + vgroups=1 + dbcountStart=0 + dbcountStop=1 + dbname="db" + stbname="stb" + tablesPerStb=1000 + rowsPerTable=100 + startTs=1640966400000 # 2022-01-01 00:00:00.000 + + tdSql.query("show dnodes;") + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(4,1,'%s:6430'%self.host) + tdSql.checkData(0,4,'ready') + tdSql.checkData(4,4,'ready') + tdSql.query("show mnodes;") + tdSql.checkRows(1) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + + # fisr add three mnodes; + tdSql.execute("create mnode on dnode 2") + tdSql.execute("create mnode on dnode 3") + + # fisrt check statut ready + self.check3mnode() + + tdSql.error("create mnode on dnode 2") + tdSql.query("show dnodes;") + print(tdSql.queryResult) + tdLog.debug("stop all of mnode ") + + # drop follower of mnode and insert data + self.createDbTbale(dbcountStart, dbcountStop,stbname,tablesPerStb) + #(method) insertTabaleData: (dbcountStart: Any, dbcountStop: Any, stbname: Any, chilCount: Any, ts_start: Any, rowCount: Any) -> None + threads=threading.Thread(target=self.insertTabaleData, args=( + dbcountStart, + dbcountStop, + stbname, + tablesPerStb, + startTs, + rowsPerTable)) + + threads.start() + dropcount =0 + while dropcount <= 10: + for i in range(1,3): + tdLog.debug("drop mnode on dnode %d"%(i+1)) + tdSql.execute("drop mnode on dnode %d"%(i+1)) + tdSql.query("show mnodes;") + count=0 + while count<10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(2): + print("drop mnode %d successfully"%(i+1)) + break + count+=1 + tdLog.debug("create mnode on dnode %d"%(i+1)) + tdSql.execute("create mnode on dnode %d"%(i+1)) + count=0 + while count<10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3): + print("drop mnode %d successfully"%(i+1)) + break + count+=1 + dropcount+=1 + threads.join() + self.check3mnode() + + + + def getConnection(self, dnode): + host = dnode.cfgDict["fqdn"] + port = dnode.cfgDict["serverPort"] + config_dir = dnode.cfgDir + return taos.connect(host=host, port=int(port), config=config_dir) + + + def run(self): + # print(self.master_dnode.cfgDict) + self.buildcluster(5) + self.five_dnode_three_mnode(5) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/system-test/6-cluster/5dnode3mnodeSeperate1VnodeStopInsert.py b/tests/system-test/6-cluster/5dnode3mnodeSeperate1VnodeStopInsert.py new file mode 100644 index 0000000000000000000000000000000000000000..1739db09af6f784eb22c7bfbe20aef4b9190720d --- /dev/null +++ b/tests/system-test/6-cluster/5dnode3mnodeSeperate1VnodeStopInsert.py @@ -0,0 +1,377 @@ +from ssl import ALERT_DESCRIPTION_CERTIFICATE_UNOBTAINABLE +import taos +import sys +import time +import os + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import TDDnodes +from util.dnodes import TDDnode +import time +import socket +import subprocess +from multiprocessing import Process +import threading +import time +import inspect +import ctypes +class MyDnodes(TDDnodes): + def __init__(self ,dnodes_lists): + super(MyDnodes,self).__init__() + self.dnodes = dnodes_lists # dnode must be TDDnode instance + self.simDeployed = False + + +class TDTestCase: + + def init(self,conn ,logSql): + tdLog.debug(f"start to excute {__file__}") + self.TDDnodes = None + + def buildcluster(self,dnodenumber): + self.depoly_cluster(dnodenumber) + self.master_dnode = self.TDDnodes.dnodes[0] + self.host=self.master_dnode.cfgDict["fqdn"] + conn1 = taos.connect(self.master_dnode.cfgDict["fqdn"] , config=self.master_dnode.cfgDir) + tdSql.init(conn1.cursor()) + + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def _async_raise(self, tid, exctype): + """raises the exception, performs cleanup if needed""" + if not inspect.isclass(exctype): + exctype = type(exctype) + res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype)) + if res == 0: + raise ValueError("invalid thread id") + elif res != 1: + # """if it returns a number greater than one, you're in trouble, + # and you should call it again with exc=NULL to revert the effect""" + ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None) + raise SystemError("PyThreadState_SetAsyncExc failed") + + def stop_thread(self,thread): + self._async_raise(thread.ident, SystemExit) + + + def insert_data(self,countstart,countstop): + # fisrt add data : db\stable\childtable\general table + + for couti in range(countstart,countstop): + tdLog.debug("drop database if exists db%d" %couti) + tdSql.execute("drop database if exists db%d" %couti) + print("create database if not exists db%d replica 1 duration 300" %couti) + tdSql.execute("create database if not exists db%d replica 1 duration 300" %couti) + tdSql.execute("use db%d" %couti) + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t1 int) + ''' + ) + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(4): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( {i+1} )') + + def checkData(self,dbname,stbname,stableCount,CtableCount,rowsPerSTable,): + tdSql.execute("use %s"%dbname) + tdSql.query("show stables") + tdSql.checkRows(stableCount) + tdSql.query("show tables") + tdSql.checkRows(CtableCount) + for i in range(stableCount): + tdSql.query("select count(*) from %s%d"%(stbname,i)) + tdSql.checkData(0,0,rowsPerSTable) + return + + def depoly_cluster(self ,dnodes_nums=5,independent=True): + + testCluster = False + valgrind = 0 + hostname = socket.gethostname() + dnodes = [] + start_port = 6030 + start_port_sec = 6130 + for num in range(1, dnodes_nums+1): + dnode = TDDnode(num) + dnode.addExtraCfg("firstEp", f"{hostname}:{start_port}") + dnode.addExtraCfg("fqdn", f"{hostname}") + dnode.addExtraCfg("serverPort", f"{start_port + (num-1)*100}") + dnode.addExtraCfg("monitorFqdn", hostname) + dnode.addExtraCfg("monitorPort", 7043) + dnode.addExtraCfg("secondEp", f"{hostname}:{start_port_sec}") + # configure three dnoe don't support vnodes + if independent and (num < 4): + dnode.addExtraCfg("supportVnodes", 0) + + dnodes.append(dnode) + + self.TDDnodes = MyDnodes(dnodes) + self.TDDnodes.init("") + self.TDDnodes.setTestCluster(testCluster) + self.TDDnodes.setValgrind(valgrind) + self.TDDnodes.stopAll() + for dnode in self.TDDnodes.dnodes: + self.TDDnodes.deploy(dnode.index,{}) + + for dnode in self.TDDnodes.dnodes: + self.TDDnodes.starttaosd(dnode.index) + + # create cluster + for dnode in self.TDDnodes.dnodes[1:]: + # print(dnode.cfgDict) + dnode_id = dnode.cfgDict["fqdn"] + ":" +dnode.cfgDict["serverPort"] + dnode_first_host = dnode.cfgDict["firstEp"].split(":")[0] + dnode_first_port = dnode.cfgDict["firstEp"].split(":")[-1] + cmd = f" taos -h {dnode_first_host} -P {dnode_first_port} -s ' create dnode \"{dnode_id} \" ' ;" + print(cmd) + os.system(cmd) + + time.sleep(2) + tdLog.info(" create cluster with %d dnode done! " %dnodes_nums) + + def checkdnodes(self,dnodenumber): + count=0 + while count < 100: + time.sleep(1) + statusReadyBumber=0 + tdSql.query("show dnodes;") + if tdSql.checkRows(dnodenumber) : + print("dnode is %d nodes"%dnodenumber) + for i in range(dnodenumber): + if tdSql.queryResult[i][4] !='ready' : + status=tdSql.queryResult[i][4] + print("dnode:%d status is %s "%(i,status)) + break + else: + statusReadyBumber+=1 + print(statusReadyBumber) + if statusReadyBumber == dnodenumber : + print("all of %d mnodes is ready in 10s "%dnodenumber) + return True + break + count+=1 + else: + print("%d mnodes is not ready in 10s "%dnodenumber) + return False + + + def check3mnode(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='follower': + print("three mnodes is ready in 10s") + break + elif tdSql.queryResult[0][2]=='follower' : + if tdSql.queryResult[1][2]=='leader': + if tdSql.queryResult[2][2]=='follower': + print("three mnodes is ready in 10s") + break + elif tdSql.queryResult[0][2]=='follower' : + if tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='leader': + print("three mnodes is ready in 10s") + break + count+=1 + else: + print("three mnodes is not ready in 10s ") + return -1 + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,3,'ready') + + def check3mnode1off(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='offline' : + if tdSql.queryResult[1][2]=='leader': + if tdSql.queryResult[2][2]=='follower': + print("stop mnodes on dnode 2 successfully in 10s") + break + elif tdSql.queryResult[1][2]=='follower': + if tdSql.queryResult[2][2]=='leader': + print("stop mnodes on dnode 2 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 2 failed in 10s ") + return -1 + tdSql.error("drop mnode on dnode 1;") + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'offline') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,3,'ready') + + def check3mnode2off(self): + count=0 + while count < 40: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[1][2]=='offline': + if tdSql.queryResult[2][2]=='follower': + print("stop mnodes on dnode 2 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 2 failed in 10s ") + return -1 + tdSql.error("drop mnode on dnode 2;") + + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,2,'offline') + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,2,'follower') + tdSql.checkData(2,3,'ready') + + def check3mnode3off(self): + count=0 + while count < 10: + time.sleep(1) + tdSql.query("show mnodes;") + if tdSql.checkRows(3) : + print("mnode is three nodes") + if tdSql.queryResult[0][2]=='leader' : + if tdSql.queryResult[2][2]=='offline': + if tdSql.queryResult[1][2]=='follower': + print("stop mnodes on dnode 3 successfully in 10s") + break + count+=1 + else: + print("stop mnodes on dnode 3 failed in 10s") + return -1 + tdSql.error("drop mnode on dnode 3;") + tdSql.query("show mnodes;") + tdSql.checkRows(3) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + tdSql.checkData(1,1,'%s:6130'%self.host) + tdSql.checkData(1,2,'follower') + tdSql.checkData(1,3,'ready') + tdSql.checkData(2,1,'%s:6230'%self.host) + tdSql.checkData(2,2,'offline') + tdSql.checkData(2,3,'ready') + + def five_dnode_three_mnode(self,dnodenumber): + tdSql.query("show dnodes;") + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(4,1,'%s:6430'%self.host) + tdSql.checkData(0,4,'ready') + tdSql.checkData(4,4,'ready') + tdSql.query("show mnodes;") + tdSql.checkRows(1) + tdSql.checkData(0,1,'%s:6030'%self.host) + tdSql.checkData(0,2,'leader') + tdSql.checkData(0,3,'ready') + + # fisr add three mnodes; + tdSql.execute("create mnode on dnode 2") + tdSql.execute("create mnode on dnode 3") + + # fisrt check statut ready + self.check3mnode() + + tdSql.error("create mnode on dnode 2") + tdSql.query("show dnodes;") + print(tdSql.queryResult) + tdLog.debug("stop all of mnode ") + + # seperate vnode and mnode in different dnodes. + # create database and stable + stopcount =0 + while stopcount < 2: + for i in range(dnodenumber): + # threads=[] + # threads = MyThreadFunc(self.insert_data(i*2,i*2+2)) + threads=threading.Thread(target=self.insert_data, args=(i,i+1)) + threads.start() + self.TDDnodes.stoptaosd(i+1) + self.TDDnodes.starttaosd(i+1) + + if self.checkdnodes(5): + print("123") + threads.join() + else: + print("456") + self.stop_thread(threads) + assert 1 == 2 ,"some dnode started failed" + return False + # self.check3mnode() + self.check3mnode() + + + stopcount+=1 + self.check3mnode() + + + def getConnection(self, dnode): + host = dnode.cfgDict["fqdn"] + port = dnode.cfgDict["serverPort"] + config_dir = dnode.cfgDir + return taos.connect(host=host, port=int(port), config=config_dir) + + + def run(self): + # print(self.master_dnode.cfgDict) + self.buildcluster(5) + self.five_dnode_three_mnode(5) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/system-test/6-cluster/5dnode3mnodeStop.py b/tests/system-test/6-cluster/5dnode3mnodeStop.py index e11f819263f8828f24977a9e0f247de14b8ede2a..654b27bfc078db0a1fb3e50a84037e5938271d23 100644 --- a/tests/system-test/6-cluster/5dnode3mnodeStop.py +++ b/tests/system-test/6-cluster/5dnode3mnodeStop.py @@ -99,12 +99,12 @@ class TDTestCase: # create cluster for dnode in self.TDDnodes.dnodes[1:]: - # print(dnode.cfgDict) + # tdLog.debug(dnode.cfgDict) dnode_id = dnode.cfgDict["fqdn"] + ":" +dnode.cfgDict["serverPort"] dnode_first_host = dnode.cfgDict["firstEp"].split(":")[0] dnode_first_port = dnode.cfgDict["firstEp"].split(":")[-1] cmd = f" taos -h {dnode_first_host} -P {dnode_first_port} -s ' create dnode \"{dnode_id} \" ' ;" - print(cmd) + tdLog.debug(cmd) os.system(cmd) time.sleep(2) @@ -116,25 +116,25 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break count+=1 else: - print("three mnodes is not ready in 10s ") + tdLog.debug("three mnodes is not ready in 10s ") return -1 tdSql.query("show mnodes;") @@ -152,19 +152,19 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='offline' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break elif tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 1;") @@ -184,15 +184,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='offline': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 2;") @@ -214,15 +214,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[2][2]=='offline': if tdSql.queryResult[1][2]=='follower': - print("stop mnodes on dnode 3 successfully in 10s") + tdLog.debug("stop mnodes on dnode 3 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 3 failed in 10s") + tdLog.debug("stop mnodes on dnode 3 failed in 10s") return -1 tdSql.error("drop mnode on dnode 3;") tdSql.query("show mnodes;") @@ -262,7 +262,7 @@ class TDTestCase: tdSql.error("create mnode on dnode 2") tdSql.query("show dnodes;") - print(tdSql.queryResult) + tdLog.debug(tdSql.queryResult) tdLog.debug("stop and follower of mnode") self.TDDnodes.stoptaosd(2) @@ -303,7 +303,7 @@ class TDTestCase: def run(self): - # print(self.master_dnode.cfgDict) + # tdLog.debug(self.master_dnode.cfgDict) self.buildcluster(5) self.five_dnode_three_mnode(5) diff --git a/tests/system-test/6-cluster/5dnode3mnodeStopInsert.py b/tests/system-test/6-cluster/5dnode3mnodeStopInsert.py index ce3f2c4093bb459b48552815b32629b76b5270b7..a53930faac92ffb2ce432a84dcaa4fc1d934d697 100644 --- a/tests/system-test/6-cluster/5dnode3mnodeStopInsert.py +++ b/tests/system-test/6-cluster/5dnode3mnodeStopInsert.py @@ -77,7 +77,7 @@ class TDTestCase: for couti in range(countstart,countstop): tdLog.debug("drop database if exists db%d" %couti) tdSql.execute("drop database if exists db%d" %couti) - print("create database if not exists db%d replica 1 duration 300" %couti) + tdLog.debug("create database if not exists db%d replica 1 duration 300" %couti) tdSql.execute("create database if not exists db%d replica 1 duration 300" %couti) tdSql.execute("use db%d" %couti) tdSql.execute( @@ -126,12 +126,12 @@ class TDTestCase: # create cluster for dnode in self.TDDnodes.dnodes[1:]: - # print(dnode.cfgDict) + # tdLog.debug(dnode.cfgDict) dnode_id = dnode.cfgDict["fqdn"] + ":" +dnode.cfgDict["serverPort"] dnode_first_host = dnode.cfgDict["firstEp"].split(":")[0] dnode_first_port = dnode.cfgDict["firstEp"].split(":")[-1] cmd = f" taos -h {dnode_first_host} -P {dnode_first_port} -s ' create dnode \"{dnode_id} \" ' ;" - print(cmd) + tdLog.debug(cmd) os.system(cmd) time.sleep(2) @@ -144,22 +144,22 @@ class TDTestCase: statusReadyBumber=0 tdSql.query("show dnodes;") if tdSql.checkRows(dnodenumber) : - print("dnode is %d nodes"%dnodenumber) + tdLog.debug("dnode is %d nodes"%dnodenumber) for i in range(dnodenumber): if tdSql.queryResult[i][4] !='ready' : status=tdSql.queryResult[i][4] - print("dnode:%d status is %s "%(i,status)) + tdLog.debug("dnode:%d status is %s "%(i,status)) break else: statusReadyBumber+=1 - print(statusReadyBumber) + tdLog.debug(statusReadyBumber) if statusReadyBumber == dnodenumber : - print("all of %d mnodes is ready in 10s "%dnodenumber) + tdLog.debug("all of %d mnodes is ready in 10s "%dnodenumber) return True break count+=1 else: - print("%d mnodes is not ready in 10s "%dnodenumber) + tdLog.debug("%d mnodes is not ready in 10s "%dnodenumber) return False @@ -169,25 +169,25 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break elif tdSql.queryResult[0][2]=='follower' : if tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("three mnodes is ready in 10s") + tdLog.debug("three mnodes is ready in 10s") break count+=1 else: - print("three mnodes is not ready in 10s ") + tdLog.debug("three mnodes is not ready in 10s ") return -1 tdSql.query("show mnodes;") @@ -205,19 +205,19 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='offline' : if tdSql.queryResult[1][2]=='leader': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break elif tdSql.queryResult[1][2]=='follower': if tdSql.queryResult[2][2]=='leader': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 1;") @@ -237,15 +237,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[1][2]=='offline': if tdSql.queryResult[2][2]=='follower': - print("stop mnodes on dnode 2 successfully in 10s") + tdLog.debug("stop mnodes on dnode 2 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 2 failed in 10s ") + tdLog.debug("stop mnodes on dnode 2 failed in 10s ") return -1 tdSql.error("drop mnode on dnode 2;") @@ -267,15 +267,15 @@ class TDTestCase: time.sleep(1) tdSql.query("show mnodes;") if tdSql.checkRows(3) : - print("mnode is three nodes") + tdLog.debug("mnode is three nodes") if tdSql.queryResult[0][2]=='leader' : if tdSql.queryResult[2][2]=='offline': if tdSql.queryResult[1][2]=='follower': - print("stop mnodes on dnode 3 successfully in 10s") + tdLog.debug("stop mnodes on dnode 3 successfully in 10s") break count+=1 else: - print("stop mnodes on dnode 3 failed in 10s") + tdLog.debug("stop mnodes on dnode 3 failed in 10s") return -1 tdSql.error("drop mnode on dnode 3;") tdSql.query("show mnodes;") @@ -311,7 +311,7 @@ class TDTestCase: tdSql.error("create mnode on dnode 2") tdSql.query("show dnodes;") - print(tdSql.queryResult) + tdLog.debug(tdSql.queryResult) tdLog.debug("stop all of mnode ") stopcount =0 @@ -325,10 +325,10 @@ class TDTestCase: self.TDDnodes.starttaosd(i+1) if self.checkdnodes(5): - print("123") + tdLog.debug("123") threads.join() else: - print("456") + tdLog.debug("456") self.stop_thread(threads) assert 1 == 2 ,"some dnode started failed" return False @@ -345,7 +345,7 @@ class TDTestCase: def run(self): - # print(self.master_dnode.cfgDict) + # tdLog.debug(self.master_dnode.cfgDict) self.buildcluster(5) self.five_dnode_three_mnode(5) diff --git a/tests/system-test/7-tmq/basic5.py b/tests/system-test/7-tmq/basic5.py index e44f327995facc02ca01aab6e92e9e41ecac36f4..4ed3be967eb1611b3cddd967b5d141ef770e19d3 100644 --- a/tests/system-test/7-tmq/basic5.py +++ b/tests/system-test/7-tmq/basic5.py @@ -192,7 +192,7 @@ class TDTestCase: time.sleep(1) tdLog.info("start consume processor") - pollDelay = 100 + pollDelay = 20 showMsg = 1 showRow = 1 @@ -208,7 +208,7 @@ class TDTestCase: os.system(shellCmd) # wait for data ready - prepareEnvThread.join() + # prepareEnvThread.join() tdLog.info("insert process end, and start to check consume result") while 1: diff --git a/tests/system-test/7-tmq/subscribeDb0.py b/tests/system-test/7-tmq/subscribeDb0.py index c9f256ed7495850bd1a88b9a6690f23a85cb68fa..4e8fb04517dc228329235994174c482d59868383 100644 --- a/tests/system-test/7-tmq/subscribeDb0.py +++ b/tests/system-test/7-tmq/subscribeDb0.py @@ -322,176 +322,6 @@ class TDTestCase: tdLog.printNoPrefix("======== test case 5 end ...... ") - def tmqCase6(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 6: Produce while one consumers to subscribe tow topic, Each contains one db") - tdLog.info("step 1: create database, stb, ctb and insert data") - # create and start thread - parameterDict = {'cfg': '', \ - 'dbName': 'db60', \ - 'vgroups': 4, \ - 'stbName': 'stb', \ - 'ctbNum': 10, \ - 'rowsPerTbl': 5000, \ - 'batchNum': 100, \ - 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 - parameterDict['cfg'] = cfgPath - - self.initConsumerTable() - - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) - - prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) - prepareEnvThread.start() - - parameterDict2 = {'cfg': '', \ - 'dbName': 'db61', \ - 'vgroups': 4, \ - 'stbName': 'stb2', \ - 'ctbNum': 10, \ - 'rowsPerTbl': 5000, \ - 'batchNum': 100, \ - 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 - parameterDict['cfg'] = cfgPath - - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict2['dbName'], parameterDict2['vgroups'])) - - prepareEnvThread2 = threading.Thread(target=self.prepareEnv, kwargs=parameterDict2) - prepareEnvThread2.start() - - tdLog.info("create topics from db") - topicName1 = 'topic_db60' - topicName2 = 'topic_db61' - - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) - tdSql.execute("create topic %s as database %s" %(topicName2, parameterDict2['dbName'])) - - consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + parameterDict2["rowsPerTbl"] * parameterDict2["ctbNum"] - topicList = topicName1 + ',' + topicName2 - ifcheckdata = 0 - ifManualCommit = 0 - keyList = 'group.id:cgrp1,\ - enable.auto.commit:false,\ - auto.commit.interval.ms:6000,\ - auto.offset.reset:earliest' - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - #consumerId = 1 - #self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - event.wait() - - tdLog.info("start consume processor") - pollDelay = 100 - showMsg = 1 - showRow = 1 - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - # wait for data ready - prepareEnvThread.join() - prepareEnvThread2.join() - - tdLog.info("insert process end, and start to check consume result") - expectRows = 1 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - if totalConsumeRows != expectrowcnt: - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - tdLog.exit("tmq consume rows error!") - - tdSql.query("drop topic %s"%topicName1) - tdSql.query("drop topic %s"%topicName2) - - tdLog.printNoPrefix("======== test case 6 end ...... ") - - def tmqCase7(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 7: Produce while two consumers to subscribe tow topic, Each contains one db") - tdLog.info("step 1: create database, stb, ctb and insert data") - # create and start thread - parameterDict = {'cfg': '', \ - 'dbName': 'db70', \ - 'vgroups': 4, \ - 'stbName': 'stb', \ - 'ctbNum': 10, \ - 'rowsPerTbl': 5000, \ - 'batchNum': 100, \ - 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 - parameterDict['cfg'] = cfgPath - - self.initConsumerTable() - - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) - - prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) - prepareEnvThread.start() - - parameterDict2 = {'cfg': '', \ - 'dbName': 'db71', \ - 'vgroups': 4, \ - 'stbName': 'stb2', \ - 'ctbNum': 10, \ - 'rowsPerTbl': 5000, \ - 'batchNum': 100, \ - 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 - parameterDict['cfg'] = cfgPath - - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict2['dbName'], parameterDict2['vgroups'])) - - prepareEnvThread2 = threading.Thread(target=self.prepareEnv, kwargs=parameterDict2) - prepareEnvThread2.start() - - tdLog.info("create topics from db") - topicName1 = 'topic_db60' - topicName2 = 'topic_db61' - - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) - tdSql.execute("create topic %s as database %s" %(topicName2, parameterDict2['dbName'])) - - consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + parameterDict2["rowsPerTbl"] * parameterDict2["ctbNum"] - topicList = topicName1 + ',' + topicName2 - ifcheckdata = 0 - ifManualCommit = 1 - keyList = 'group.id:cgrp1,\ - enable.auto.commit:false,\ - auto.commit.interval.ms:6000,\ - auto.offset.reset:earliest' - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - consumerId = 1 - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - event.wait() - - tdLog.info("start consume processor") - pollDelay = 100 - showMsg = 1 - showRow = 1 - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - # wait for data ready - prepareEnvThread.join() - prepareEnvThread2.join() - - tdLog.info("insert process end, and start to check consume result") - expectRows = 2 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - if totalConsumeRows != expectrowcnt: - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - tdLog.exit("tmq consume rows error!") - - tdSql.query("drop topic %s"%topicName1) - tdSql.query("drop topic %s"%topicName2) - - tdLog.printNoPrefix("======== test case 7 end ...... ") - def run(self): tdSql.prepare() @@ -505,8 +335,6 @@ class TDTestCase: self.tmqCase4(cfgPath, buildPath) self.tmqCase5(cfgPath, buildPath) - self.tmqCase6(cfgPath, buildPath) - self.tmqCase7(cfgPath, buildPath) def stop(self): diff --git a/tests/system-test/7-tmq/subscribeDb1.py b/tests/system-test/7-tmq/subscribeDb1.py index ed92a429aee5cc10cfc2b5964a215225f813c908..28a341f8f353cd16120420fa098ca46b1a458edb 100644 --- a/tests/system-test/7-tmq/subscribeDb1.py +++ b/tests/system-test/7-tmq/subscribeDb1.py @@ -72,10 +72,10 @@ class TDTestCase: if tdSql.getRows() == expectRows: break else: - time.sleep(5) - + time.sleep(5) + for i in range(expectRows): - tdLog.info ("ts: %s, consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 0), tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) + tdLog.info ("consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) resultList.append(tdSql.getData(i , 3)) return resultList @@ -85,7 +85,7 @@ class TDTestCase: logFile = cfgPath + '/../log/valgrind-tmq.log' shellCmd = 'nohup valgrind --log-file=' + logFile shellCmd += '--tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all --num-callers=20 -v --workaround-gcc296-bugs=yes ' - + if (platform.system().lower() == 'windows'): shellCmd = 'mintty -h never -w hide ' + buildPath + '\\build\\bin\\tmq_sim.exe -c ' + cfgPath shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) @@ -97,7 +97,7 @@ class TDTestCase: tdLog.info(shellCmd) os.system(shellCmd) - def create_tables(self,tsql, dbName,vgroups,stbName,ctbNum,rowsPerTbl): + def create_tables(self,tsql, dbName,vgroups,stbName,ctbNum): tsql.execute("create database if not exists %s vgroups %d"%(dbName, vgroups)) tsql.execute("use %s" %dbName) tsql.execute("create table if not exists %s (ts timestamp, c1 bigint, c2 binary(16)) tags(t1 int)"%stbName) @@ -151,8 +151,7 @@ class TDTestCase: parameterDict["dbName"],\ parameterDict["vgroups"],\ parameterDict["stbName"],\ - parameterDict["ctbNum"],\ - parameterDict["rowsPerTbl"]) + parameterDict["ctbNum"]) self.insert_data(tsql,\ parameterDict["dbName"],\ @@ -163,16 +162,16 @@ class TDTestCase: parameterDict["startTs"]) return - def tmqCase8(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 8: Produce while one consume to subscribe one db, inclue 1 stb") + def tmqCase6(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 6: Produce while one consumers to subscribe tow topic, Each contains one db") tdLog.info("step 1: create database, stb, ctb and insert data") # create and start thread parameterDict = {'cfg': '', \ - 'dbName': 'db8', \ + 'dbName': 'db60', \ 'vgroups': 4, \ 'stbName': 'stb', \ 'ctbNum': 10, \ - 'rowsPerTbl': 10000, \ + 'rowsPerTbl': 5000, \ 'batchNum': 100, \ 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 parameterDict['cfg'] = cfgPath @@ -183,100 +182,43 @@ class TDTestCase: prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) prepareEnvThread.start() - - tdLog.info("create topics from db") - topicName1 = 'topic_db1' - - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) - consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] / 2 - topicList = topicName1 - ifcheckdata = 0 - ifManualCommit = 0 - keyList = 'group.id:cgrp1,\ - enable.auto.commit:false,\ - auto.commit.interval.ms:6000,\ - auto.offset.reset:earliest' - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - event.wait() - tdLog.info("start consume processor") - pollDelay = 100 - showMsg = 1 - showRow = 1 - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - # wait for data ready - prepareEnvThread.join() - - tdLog.info("insert process end, and start to check consume result") - expectRows = 1 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - if totalConsumeRows != expectrowcnt: - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - tdLog.exit("tmq consume rows error!") - - - tdLog.info("again start consume processer") - self.initConsumerTable() - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - expectRows = 1 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - if totalConsumeRows != expectrowcnt: - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - tdLog.exit("tmq consume rows error!") - - tdSql.query("drop topic %s"%topicName1) - - tdLog.printNoPrefix("======== test case 8 end ...... ") - - def tmqCase9(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 9: Produce while one consume to subscribe one db, inclue 1 stb") - tdLog.info("step 1: create database, stb, ctb and insert data") - # create and start thread - parameterDict = {'cfg': '', \ - 'dbName': 'db9', \ + parameterDict2 = {'cfg': '', \ + 'dbName': 'db61', \ 'vgroups': 4, \ - 'stbName': 'stb', \ + 'stbName': 'stb2', \ 'ctbNum': 10, \ - 'rowsPerTbl': 10000, \ + 'rowsPerTbl': 5000, \ 'batchNum': 100, \ 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 parameterDict['cfg'] = cfgPath - self.initConsumerTable() + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict2['dbName'], parameterDict2['vgroups'])) - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + prepareEnvThread2 = threading.Thread(target=self.prepareEnv, kwargs=parameterDict2) + prepareEnvThread2.start() - prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) - prepareEnvThread.start() - tdLog.info("create topics from db") - topicName1 = 'topic_db1' + topicName1 = 'topic_db60' + topicName2 = 'topic_db61' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + tdSql.execute("create topic %s as database %s" %(topicName2, parameterDict2['dbName'])) - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] / 2 - topicList = topicName1 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + parameterDict2["rowsPerTbl"] * parameterDict2["ctbNum"] + topicList = topicName1 + ',' + topicName2 ifcheckdata = 0 - ifManualCommit = 1 + ifManualCommit = 0 keyList = 'group.id:cgrp1,\ enable.auto.commit:false,\ auto.commit.interval.ms:6000,\ auto.offset.reset:earliest' self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + #consumerId = 1 + #self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + event.wait() tdLog.info("start consume processor") @@ -286,7 +228,8 @@ class TDTestCase: self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) # wait for data ready - prepareEnvThread.join() + prepareEnvThread.join() + prepareEnvThread2.join() tdLog.info("insert process end, and start to check consume result") expectRows = 1 @@ -295,44 +238,25 @@ class TDTestCase: for i in range(expectRows): totalConsumeRows += resultList[i] - tdSql.query("select count(*) from %s.%s" %(parameterDict['dbName'], parameterDict['stbName'])) - countOfStb = tdSql.getData(0,0) - print ("====total rows of stb: %d"%countOfStb) - - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - if totalConsumeRows < expectrowcnt: - tdLog.exit("tmq consume rows error!") - - tdLog.info("again start consume processer") - self.initConsumerTable() - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - expectRows = 1 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows2 = 0 - for i in range(expectRows): - totalConsumeRows2 += resultList[i] - - tdLog.info("firstly act consume rows: %d"%(totalConsumeRows)) - tdLog.info("secondly act consume rows: %d, expect consume rows: %d"%(totalConsumeRows2, expectrowcnt)) - if totalConsumeRows + totalConsumeRows2 != expectrowcnt: + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) tdLog.exit("tmq consume rows error!") tdSql.query("drop topic %s"%topicName1) + tdSql.query("drop topic %s"%topicName2) - tdLog.printNoPrefix("======== test case 9 end ...... ") + tdLog.printNoPrefix("======== test case 6 end ...... ") - def tmqCase10(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 10: Produce while one consume to subscribe one db, inclue 1 stb") + def tmqCase7(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 7: Produce while two consumers to subscribe tow topic, Each contains one db") tdLog.info("step 1: create database, stb, ctb and insert data") # create and start thread parameterDict = {'cfg': '', \ - 'dbName': 'db10', \ + 'dbName': 'db70', \ 'vgroups': 4, \ 'stbName': 'stb', \ 'ctbNum': 10, \ - 'rowsPerTbl': 10000, \ + 'rowsPerTbl': 5000, \ 'batchNum': 100, \ 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 parameterDict['cfg'] = cfgPath @@ -343,135 +267,70 @@ class TDTestCase: prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) prepareEnvThread.start() - - tdLog.info("create topics from db") - topicName1 = 'topic_db1' - - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) - consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] - topicList = topicName1 - ifcheckdata = 0 - ifManualCommit = 1 - keyList = 'group.id:cgrp1,\ - enable.auto.commit:false,\ - auto.commit.interval.ms:6000,\ - auto.offset.reset:earliest' - self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - event.wait() - tdLog.info("start consume processor") - pollDelay = 100 - showMsg = 1 - showRow = 1 - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - time.sleep(2) - tdLog.info("pkill consume processor") - if (platform.system().lower() == 'windows'): - os.system("TASKKILL /F /IM tmq_sim.exe") - else: - os.system('pkill tmq_sim') - expectRows = 0 - resultList = self.selectConsumeResult(expectRows) - - # wait for data ready - prepareEnvThread.join() - tdLog.info("insert process end, and start to check consume result") - - tdLog.info("again start consume processer") - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - expectRows = 1 - resultList = self.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - if totalConsumeRows != expectrowcnt: - tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) - tdLog.exit("tmq consume rows error!") - - time.sleep(15) - tdSql.query("drop topic %s"%topicName1) - - tdLog.printNoPrefix("======== test case 10 end ...... ") - - def tmqCase11(self, cfgPath, buildPath): - tdLog.printNoPrefix("======== test case 11: Produce while one consume to subscribe one db, inclue 1 stb") - tdLog.info("step 1: create database, stb, ctb and insert data") - # create and start thread - parameterDict = {'cfg': '', \ - 'dbName': 'db11', \ + parameterDict2 = {'cfg': '', \ + 'dbName': 'db71', \ 'vgroups': 4, \ - 'stbName': 'stb', \ + 'stbName': 'stb2', \ 'ctbNum': 10, \ - 'rowsPerTbl': 10000, \ + 'rowsPerTbl': 5000, \ 'batchNum': 100, \ 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 parameterDict['cfg'] = cfgPath - self.initConsumerTable() + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict2['dbName'], parameterDict2['vgroups'])) - tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + prepareEnvThread2 = threading.Thread(target=self.prepareEnv, kwargs=parameterDict2) + prepareEnvThread2.start() - prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) - prepareEnvThread.start() - tdLog.info("create topics from db") - topicName1 = 'topic_db1' + topicName1 = 'topic_db60' + topicName2 = 'topic_db61' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + tdSql.execute("create topic %s as database %s" %(topicName2, parameterDict2['dbName'])) - tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) consumerId = 0 - expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] - topicList = topicName1 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + parameterDict2["rowsPerTbl"] * parameterDict2["ctbNum"] + topicList = topicName1 + ',' + topicName2 ifcheckdata = 0 ifManualCommit = 1 keyList = 'group.id:cgrp1,\ - enable.auto.commit:true,\ - auto.commit.interval.ms:1000,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ auto.offset.reset:earliest' self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + consumerId = 1 + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + event.wait() tdLog.info("start consume processor") - pollDelay = 20 + pollDelay = 100 showMsg = 1 showRow = 1 self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - time.sleep(3) - tdLog.info("pkill consume processor") - if (platform.system().lower() == 'windows'): - os.system("TASKKILL /F /IM tmq_sim.exe") - else: - os.system('pkill tmq_sim') - expectRows = 0 - resultList = self.selectConsumeResult(expectRows) - # wait for data ready prepareEnvThread.join() + prepareEnvThread2.join() + tdLog.info("insert process end, and start to check consume result") - - tdLog.info("again start consume processer") - self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) - - expectRows = 1 + expectRows = 2 resultList = self.selectConsumeResult(expectRows) totalConsumeRows = 0 for i in range(expectRows): totalConsumeRows += resultList[i] - if totalConsumeRows >= expectrowcnt or totalConsumeRows <= 0: - tdLog.info("act consume rows: %d, expect consume rows between %d and 0"%(totalConsumeRows, expectrowcnt)) + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) tdLog.exit("tmq consume rows error!") - time.sleep(15) tdSql.query("drop topic %s"%topicName1) + tdSql.query("drop topic %s"%topicName2) - tdLog.printNoPrefix("======== test case 11 end ...... ") + tdLog.printNoPrefix("======== test case 7 end ...... ") def run(self): tdSql.prepare() @@ -484,10 +343,9 @@ class TDTestCase: cfgPath = buildPath + "/../sim/psim/cfg" tdLog.info("cfgPath: %s" % cfgPath) - self.tmqCase8(cfgPath, buildPath) - self.tmqCase9(cfgPath, buildPath) - self.tmqCase10(cfgPath, buildPath) - self.tmqCase11(cfgPath, buildPath) + self.tmqCase6(cfgPath, buildPath) + self.tmqCase7(cfgPath, buildPath) + def stop(self): tdSql.close() diff --git a/tests/system-test/7-tmq/subscribeDb2.py b/tests/system-test/7-tmq/subscribeDb2.py new file mode 100644 index 0000000000000000000000000000000000000000..af31e802b31c6c682a1b120ba01b1e25cf6750f9 --- /dev/null +++ b/tests/system-test/7-tmq/subscribeDb2.py @@ -0,0 +1,347 @@ + +import taos +import sys +import time +import socket +import os +import threading + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * + +class TDTestCase: + hostname = socket.gethostname() + #rpcDebugFlagVal = '143' + #clientCfgDict = {'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + #clientCfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #updatecfgDict = {'clientCfg': {}, 'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + #updatecfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #print ("===================: ", updatecfgDict) + + def init(self, conn, logSql): + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + #tdSql.init(conn.cursor(), logSql) # output sql.txt file + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def newcur(self,cfg,host,port): + user = "root" + password = "taosdata" + con=taos.connect(host=host, user=user, password=password, config=cfg ,port=port) + cur=con.cursor() + print(cur) + return cur + + def initConsumerTable(self,cdbName='cdb'): + tdLog.info("create consume database, and consume info table, and consume result table") + tdSql.query("create database if not exists %s vgroups 1"%(cdbName)) + tdSql.query("drop table if exists %s.consumeinfo "%(cdbName)) + tdSql.query("drop table if exists %s.consumeresult "%(cdbName)) + + tdSql.query("create table %s.consumeinfo (ts timestamp, consumerid int, topiclist binary(1024), keylist binary(1024), expectmsgcnt bigint, ifcheckdata int, ifmanualcommit int)"%cdbName) + tdSql.query("create table %s.consumeresult (ts timestamp, consumerid int, consummsgcnt bigint, consumrowcnt bigint, checkresult int)"%cdbName) + + def insertConsumerInfo(self,consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifmanualcommit,cdbName='cdb'): + sql = "insert into %s.consumeinfo values "%cdbName + sql += "(now, %d, '%s', '%s', %d, %d, %d)"%(consumerId, topicList, keyList, expectrowcnt, ifcheckdata, ifmanualcommit) + tdLog.info("consume info sql: %s"%sql) + tdSql.query(sql) + + def selectConsumeResult(self,expectRows,cdbName='cdb'): + resultList=[] + while 1: + tdSql.query("select * from %s.consumeresult"%cdbName) + #tdLog.info("row: %d, %l64d, %l64d"%(tdSql.getData(0, 1),tdSql.getData(0, 2),tdSql.getData(0, 3)) + if tdSql.getRows() == expectRows: + break + else: + time.sleep(5) + + for i in range(expectRows): + tdLog.info ("ts: %s, consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 0), tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) + resultList.append(tdSql.getData(i , 3)) + + return resultList + + def startTmqSimProcess(self,buildPath,cfgPath,pollDelay,dbName,showMsg=1,showRow=1,cdbName='cdb',valgrind=0): + if valgrind == 1: + logFile = cfgPath + '/../log/valgrind-tmq.log' + shellCmd = 'nohup valgrind --log-file=' + logFile + shellCmd += '--tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all --num-callers=20 -v --workaround-gcc296-bugs=yes ' + + if (platform.system().lower() == 'windows'): + shellCmd = 'mintty -h never -w hide ' + buildPath + '\\build\\bin\\tmq_sim.exe -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> nul 2>&1 &" + else: + shellCmd = 'nohup ' + buildPath + '/build/bin/tmq_sim -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> /dev/null 2>&1 &" + tdLog.info(shellCmd) + os.system(shellCmd) + + def create_tables(self,tsql, dbName,vgroups,stbName,ctbNum,rowsPerTbl): + tsql.execute("create database if not exists %s vgroups %d"%(dbName, vgroups)) + tsql.execute("use %s" %dbName) + tsql.execute("create table if not exists %s (ts timestamp, c1 bigint, c2 binary(16)) tags(t1 int)"%stbName) + pre_create = "create table" + sql = pre_create + #tdLog.debug("doing create one stable %s and %d child table in %s ..." %(stbname, count ,dbname)) + for i in range(ctbNum): + sql += " %s_%d using %s tags(%d)"%(stbName,i,stbName,i+1) + if (i > 0) and (i%100 == 0): + tsql.execute(sql) + sql = pre_create + if sql != pre_create: + tsql.execute(sql) + + event.set() + tdLog.debug("complete to create database[%s], stable[%s] and %d child tables" %(dbName, stbName, ctbNum)) + return + + def insert_data(self,tsql,dbName,stbName,ctbNum,rowsPerTbl,batchNum,startTs): + tdLog.debug("start to insert data ............") + tsql.execute("use %s" %dbName) + pre_insert = "insert into " + sql = pre_insert + + t = time.time() + startTs = int(round(t * 1000)) + #tdLog.debug("doing insert data into stable:%s rows:%d ..."%(stbName, allRows)) + for i in range(ctbNum): + sql += " %s_%d values "%(stbName,i) + for j in range(rowsPerTbl): + sql += "(%d, %d, 'tmqrow_%d') "%(startTs + j, j, j) + if (j > 0) and ((j%batchNum == 0) or (j == rowsPerTbl - 1)): + tsql.execute(sql) + if j < rowsPerTbl - 1: + sql = "insert into %s_%d values " %(stbName,i) + else: + sql = "insert into " + #end sql + if sql != pre_insert: + #print("insert sql:%s"%sql) + tsql.execute(sql) + tdLog.debug("insert data ............ [OK]") + return + + def prepareEnv(self, **parameterDict): + print ("input parameters:") + print (parameterDict) + # create new connector for my thread + tsql=self.newcur(parameterDict['cfg'], 'localhost', 6030) + self.create_tables(tsql,\ + parameterDict["dbName"],\ + parameterDict["vgroups"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"],\ + parameterDict["rowsPerTbl"]) + + self.insert_data(tsql,\ + parameterDict["dbName"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"],\ + parameterDict["rowsPerTbl"],\ + parameterDict["batchNum"],\ + parameterDict["startTs"]) + return + + def tmqCase8(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 8: Produce while one consume to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db8', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 10000, \ + 'batchNum': 100, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] / 2 + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 0 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + # wait for data ready + prepareEnvThread.join() + + tdLog.info("insert process end, and start to check consume result") + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + + tdLog.info("again start consume processer") + self.initConsumerTable() + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 8 end ...... ") + + def tmqCase9(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 9: Produce while one consume to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db9', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 10000, \ + 'batchNum': 100, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] / 2 + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 1 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + # wait for data ready + prepareEnvThread.join() + + tdLog.info("insert process end, and start to check consume result") + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + tdSql.query("select count(*) from %s.%s" %(parameterDict['dbName'], parameterDict['stbName'])) + countOfStb = tdSql.getData(0,0) + print ("====total rows of stb: %d"%countOfStb) + + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + if totalConsumeRows < expectrowcnt: + tdLog.exit("tmq consume rows error!") + + tdLog.info("again start consume processer") + self.initConsumerTable() + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows2 = 0 + for i in range(expectRows): + totalConsumeRows2 += resultList[i] + + tdLog.info("firstly act consume rows: %d"%(totalConsumeRows)) + tdLog.info("secondly act consume rows: %d, expect consume rows: %d"%(totalConsumeRows2, expectrowcnt)) + if totalConsumeRows + totalConsumeRows2 != expectrowcnt: + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 9 end ...... ") + + def run(self): + tdSql.prepare() + + buildPath = self.getBuildPath() + if (buildPath == ""): + tdLog.exit("taosd not found!") + else: + tdLog.info("taosd found in %s" % buildPath) + cfgPath = buildPath + "/../sim/psim/cfg" + tdLog.info("cfgPath: %s" % cfgPath) + + self.tmqCase8(cfgPath, buildPath) + self.tmqCase9(cfgPath, buildPath) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +event = threading.Event() + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/subscribeDb3.py b/tests/system-test/7-tmq/subscribeDb3.py new file mode 100644 index 0000000000000000000000000000000000000000..6973f4c51fdc4db250898b15898f23782e613090 --- /dev/null +++ b/tests/system-test/7-tmq/subscribeDb3.py @@ -0,0 +1,337 @@ + +import taos +import sys +import time +import socket +import os +import threading + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * + +class TDTestCase: + hostname = socket.gethostname() + #rpcDebugFlagVal = '143' + #clientCfgDict = {'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + #clientCfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #updatecfgDict = {'clientCfg': {}, 'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + #updatecfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #print ("===================: ", updatecfgDict) + + def init(self, conn, logSql): + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + #tdSql.init(conn.cursor(), logSql) # output sql.txt file + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def newcur(self,cfg,host,port): + user = "root" + password = "taosdata" + con=taos.connect(host=host, user=user, password=password, config=cfg ,port=port) + cur=con.cursor() + print(cur) + return cur + + def initConsumerTable(self,cdbName='cdb'): + tdLog.info("create consume database, and consume info table, and consume result table") + tdSql.query("create database if not exists %s vgroups 1"%(cdbName)) + tdSql.query("drop table if exists %s.consumeinfo "%(cdbName)) + tdSql.query("drop table if exists %s.consumeresult "%(cdbName)) + + tdSql.query("create table %s.consumeinfo (ts timestamp, consumerid int, topiclist binary(1024), keylist binary(1024), expectmsgcnt bigint, ifcheckdata int, ifmanualcommit int)"%cdbName) + tdSql.query("create table %s.consumeresult (ts timestamp, consumerid int, consummsgcnt bigint, consumrowcnt bigint, checkresult int)"%cdbName) + + def insertConsumerInfo(self,consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifmanualcommit,cdbName='cdb'): + sql = "insert into %s.consumeinfo values "%cdbName + sql += "(now, %d, '%s', '%s', %d, %d, %d)"%(consumerId, topicList, keyList, expectrowcnt, ifcheckdata, ifmanualcommit) + tdLog.info("consume info sql: %s"%sql) + tdSql.query(sql) + + def selectConsumeResult(self,expectRows,cdbName='cdb'): + resultList=[] + while 1: + tdSql.query("select * from %s.consumeresult"%cdbName) + #tdLog.info("row: %d, %l64d, %l64d"%(tdSql.getData(0, 1),tdSql.getData(0, 2),tdSql.getData(0, 3)) + if tdSql.getRows() == expectRows: + break + else: + time.sleep(5) + + for i in range(expectRows): + tdLog.info ("ts: %s, consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 0), tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) + resultList.append(tdSql.getData(i , 3)) + + return resultList + + def startTmqSimProcess(self,buildPath,cfgPath,pollDelay,dbName,showMsg=1,showRow=1,cdbName='cdb',valgrind=0): + if valgrind == 1: + logFile = cfgPath + '/../log/valgrind-tmq.log' + shellCmd = 'nohup valgrind --log-file=' + logFile + shellCmd += '--tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all --num-callers=20 -v --workaround-gcc296-bugs=yes ' + + if (platform.system().lower() == 'windows'): + shellCmd = 'mintty -h never -w hide ' + buildPath + '\\build\\bin\\tmq_sim.exe -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> nul 2>&1 &" + else: + shellCmd = 'nohup ' + buildPath + '/build/bin/tmq_sim -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> /dev/null 2>&1 &" + tdLog.info(shellCmd) + os.system(shellCmd) + + def create_tables(self,tsql, dbName,vgroups,stbName,ctbNum,rowsPerTbl): + tsql.execute("create database if not exists %s vgroups %d"%(dbName, vgroups)) + tsql.execute("use %s" %dbName) + tsql.execute("create table if not exists %s (ts timestamp, c1 bigint, c2 binary(16)) tags(t1 int)"%stbName) + pre_create = "create table" + sql = pre_create + #tdLog.debug("doing create one stable %s and %d child table in %s ..." %(stbname, count ,dbname)) + for i in range(ctbNum): + sql += " %s_%d using %s tags(%d)"%(stbName,i,stbName,i+1) + if (i > 0) and (i%100 == 0): + tsql.execute(sql) + sql = pre_create + if sql != pre_create: + tsql.execute(sql) + + event.set() + tdLog.debug("complete to create database[%s], stable[%s] and %d child tables" %(dbName, stbName, ctbNum)) + return + + def insert_data(self,tsql,dbName,stbName,ctbNum,rowsPerTbl,batchNum,startTs): + tdLog.debug("start to insert data ............") + tsql.execute("use %s" %dbName) + pre_insert = "insert into " + sql = pre_insert + + t = time.time() + startTs = int(round(t * 1000)) + #tdLog.debug("doing insert data into stable:%s rows:%d ..."%(stbName, allRows)) + for i in range(ctbNum): + sql += " %s_%d values "%(stbName,i) + for j in range(rowsPerTbl): + sql += "(%d, %d, 'tmqrow_%d') "%(startTs + j, j, j) + if (j > 0) and ((j%batchNum == 0) or (j == rowsPerTbl - 1)): + tsql.execute(sql) + if j < rowsPerTbl - 1: + sql = "insert into %s_%d values " %(stbName,i) + else: + sql = "insert into " + #end sql + if sql != pre_insert: + #print("insert sql:%s"%sql) + tsql.execute(sql) + tdLog.debug("insert data ............ [OK]") + return + + def prepareEnv(self, **parameterDict): + print ("input parameters:") + print (parameterDict) + # create new connector for my thread + tsql=self.newcur(parameterDict['cfg'], 'localhost', 6030) + self.create_tables(tsql,\ + parameterDict["dbName"],\ + parameterDict["vgroups"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"],\ + parameterDict["rowsPerTbl"]) + + self.insert_data(tsql,\ + parameterDict["dbName"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"],\ + parameterDict["rowsPerTbl"],\ + parameterDict["batchNum"],\ + parameterDict["startTs"]) + return + + def tmqCase10(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 10: Produce while one consume to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db10', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 10000, \ + 'batchNum': 100, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 1 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + time.sleep(2) + tdLog.info("pkill consume processor") + if (platform.system().lower() == 'windows'): + os.system("TASKKILL /F /IM tmq_sim.exe") + else: + os.system('pkill tmq_sim') + expectRows = 0 + resultList = self.selectConsumeResult(expectRows) + + # wait for data ready + prepareEnvThread.join() + tdLog.info("insert process end, and start to check consume result") + + tdLog.info("again start consume processer") + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + time.sleep(15) + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 10 end ...... ") + + def tmqCase11(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 11: Produce while one consume to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db11', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 10000, \ + 'batchNum': 100, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d" %(parameterDict['dbName'], parameterDict['vgroups'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 1 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:true,\ + auto.commit.interval.ms:1000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 20 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + time.sleep(6) + tdLog.info("pkill consume processor") + if (platform.system().lower() == 'windows'): + os.system("TASKKILL /F /IM tmq_sim.exe") + else: + os.system('pkill tmq_sim') + expectRows = 0 + resultList = self.selectConsumeResult(expectRows) + + # wait for data ready + prepareEnvThread.join() + tdLog.info("insert process end, and start to check consume result") + + tdLog.info("again start consume processer") + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows >= expectrowcnt or totalConsumeRows <= 0: + tdLog.info("act consume rows: %d, expect consume rows between %d and 0"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + time.sleep(15) + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 11 end ...... ") + + def run(self): + tdSql.prepare() + + buildPath = self.getBuildPath() + if (buildPath == ""): + tdLog.exit("taosd not found!") + else: + tdLog.info("taosd found in %s" % buildPath) + cfgPath = buildPath + "/../sim/psim/cfg" + tdLog.info("cfgPath: %s" % cfgPath) + + self.tmqCase10(cfgPath, buildPath) + self.tmqCase11(cfgPath, buildPath) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +event = threading.Event() + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/subscribeDb4.py b/tests/system-test/7-tmq/subscribeDb4.py new file mode 100644 index 0000000000000000000000000000000000000000..de475803fe3297cc56143baaa3639a2e9e8d2553 --- /dev/null +++ b/tests/system-test/7-tmq/subscribeDb4.py @@ -0,0 +1,116 @@ +import sys +import time +import socket +import os +import threading + +import taos +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * +from util.common import * +sys.path.append("./7-tmq") +from tmqCommon import * + +class TDTestCase: + paraDict = {'dbName': 'db12', + 'dropFlag':1, + 'vgroups': 4, + 'precision': 'ms', + 'stbName': 'stb0', + 'ctbNum': 10, + 'rowsPerTbl': 10000, + 'batchNum': 10, + 'startTs': 0, # 1640966400000 ----> 2022-01-01 00:00:00.000 + 'event':'', + 'columnDict': {'int':2}, + 'tagDict': {'int':1} + } + + cdbName = 'cdb' + # some parameter to consumer processor + consumerId = 0 + expectrowcnt = 0 + topicList = '' + ifcheckdata = 0 + ifManualCommit = 1 + groupId = 'group.id:cgrp1' + autoCommit = 'enable.auto.commit:false' + autoCommitInterval = 'auto.commit.interval.ms:1000' + autoOffset = 'auto.offset.reset:earliest' + + pollDelay = 20 + showMsg = 1 + showRow = 1 + + hostname = socket.gethostname() + + def init(self, conn, logSql): + tdLog.debug(f"start to excute {__file__}") + logSql = False + tdSql.init(conn.cursor(), logSql) + + def tmqCase12(self): + tdLog.printNoPrefix("======== test case 12: ") + tdLog.info("step 1: create database, stb, ctb and insert data") + + tmqCom.initConsumerTable(self.cdbName) + + tdCom.create_database(tdSql,self.paraDict["dbName"],self.paraDict["dropFlag"], self.paraDict['precision']) + + self.paraDict["stbName"] = 'stb1' + tdCom.create_stable(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["columnDict"],self.paraDict["tagDict"]) + tdCom.create_ctables(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["ctbNum"],self.paraDict["tagDict"]) + tdCom.insert_data(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["ctbNum"],self.paraDict["rowsPerTbl"],self.paraDict["batchNum"]) + + self.paraDict["stbName"] = 'stb2' + tdCom.create_stable(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["columnDict"],self.paraDict["tagDict"]) + tdCom.create_ctables(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["ctbNum"],self.paraDict["tagDict"]) + tdCom.insert_data(tdSql,self.paraDict["dbName"],self.paraDict["stbName"],self.paraDict["ctbNum"],self.paraDict["rowsPerTbl"],self.paraDict["batchNum"]) + + tdLog.info("create topics from db") + topicName1 = 'topic_%s'%(self.paraDict['dbName']) + tdSql.execute("create topic %s as database %s" %(topicName1, self.paraDict['dbName'])) + + topicList = topicName1 + keyList = '%s,%s,%s,%s'%(self.groupId,self.autoCommit,self.autoCommitInterval,self.autoOffset) + self.expectrowcnt = self.paraDict["rowsPerTbl"] * self.paraDict["ctbNum"] * 2 + tmqCom.insertConsumerInfo(self.consumerId, self.expectrowcnt,topicList,keyList,self.ifcheckdata,self.ifManualCommit) + + tdLog.info("start consume processor") + tmqCom.startTmqSimProcess(self.pollDelay,self.paraDict["dbName"],self.showMsg, self.showRow,self.cdbName) + + tdLog.info("After waiting for a period of time, drop one stable") + time.sleep(10) + tdSql.execute("drop table %s.%s" %(self.paraDict['dbName'], self.paraDict['stbName'])) + + tdLog.info("wait result from consumer, then check it") + expectRows = 1 + resultList = tmqCom.selectConsumeResult(expectRows) + + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if not (totalConsumeRows >= self.expectrowcnt/2 and totalConsumeRows <= self.expectrowcnt): + tdLog.info("act consume rows: %d, expect consume rows: between %d and %d"%(totalConsumeRows, self.expectrowcnt/2, self.expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + time.sleep(15) + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 12 end ...... ") + + def run(self): + tdSql.prepare() + self.tmqCase12() + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +event = threading.Event() + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/subscribeStb4.py b/tests/system-test/7-tmq/subscribeStb4.py index d8a9ca95b086372fe581176f5dc7e220d1a71a98..f3982c8f1f7e2dcdd28f98100ed3817013901b74 100644 --- a/tests/system-test/7-tmq/subscribeStb4.py +++ b/tests/system-test/7-tmq/subscribeStb4.py @@ -196,16 +196,16 @@ class TDTestCase: auotCtbPrefix = 'autoCtb' # create and start thread - parameterDict = {'cfg': '', \ - 'actionType': 0, \ - 'dbName': 'db1', \ - 'dropFlag': 1, \ - 'vgroups': 4, \ - 'replica': 1, \ - 'stbName': 'stb1', \ - 'ctbNum': 10, \ - 'rowsPerTbl': 10000, \ - 'batchNum': 100, \ + parameterDict = {'cfg': '', + 'actionType': 0, + 'dbName': 'db1', + 'dropFlag': 1, + 'vgroups': 4, + 'replica': 1, + 'stbName': 'stb1', + 'ctbNum': 10, + 'rowsPerTbl': 10000, + 'batchNum': 100, 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 parameterDict['cfg'] = cfgPath @@ -349,7 +349,5 @@ class TDTestCase: tdSql.close() tdLog.success(f"{__file__} successfully executed") -event = threading.Event() - tdCases.addLinux(__file__, TDTestCase()) tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/tmqCommon.py b/tests/system-test/7-tmq/tmqCommon.py new file mode 100644 index 0000000000000000000000000000000000000000..68adaa412603e71413152257ad7d005d4017c753 --- /dev/null +++ b/tests/system-test/7-tmq/tmqCommon.py @@ -0,0 +1,118 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +from collections import defaultdict +import random +import string +import threading +import requests +import time +# import socketfrom + +import taos +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * +from util.common import * + +# class actionType(Enum): +# CREATE_DATABASE = 0 +# CREATE_STABLE = 1 +# CREATE_CTABLE = 2 +# INSERT_DATA = 3 + +class TMQCom: + def init(self, conn, logSql): + tdSql.init(conn.cursor()) + # tdSql.init(conn.cursor(), logSql) # output sql.txt file + + def initConsumerTable(self,cdbName='cdb'): + tdLog.info("create consume database, and consume info table, and consume result table") + tdSql.query("create database if not exists %s vgroups 1"%(cdbName)) + tdSql.query("drop table if exists %s.consumeinfo "%(cdbName)) + tdSql.query("drop table if exists %s.consumeresult "%(cdbName)) + + tdSql.query("create table %s.consumeinfo (ts timestamp, consumerid int, topiclist binary(1024), keylist binary(1024), expectmsgcnt bigint, ifcheckdata int, ifmanualcommit int)"%cdbName) + tdSql.query("create table %s.consumeresult (ts timestamp, consumerid int, consummsgcnt bigint, consumrowcnt bigint, checkresult int)"%cdbName) + + def initConsumerInfoTable(self,cdbName='cdb'): + tdLog.info("drop consumeinfo table") + tdSql.query("drop table if exists %s.consumeinfo "%(cdbName)) + tdSql.query("create table %s.consumeinfo (ts timestamp, consumerid int, topiclist binary(1024), keylist binary(1024), expectmsgcnt bigint, ifcheckdata int, ifmanualcommit int)"%cdbName) + + def insertConsumerInfo(self,consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifmanualcommit,cdbName='cdb'): + sql = "insert into %s.consumeinfo values "%cdbName + sql += "(now, %d, '%s', '%s', %d, %d, %d)"%(consumerId, topicList, keyList, expectrowcnt, ifcheckdata, ifmanualcommit) + tdLog.info("consume info sql: %s"%sql) + tdSql.query(sql) + + def selectConsumeResult(self,expectRows,cdbName='cdb'): + resultList=[] + while 1: + tdSql.query("select * from %s.consumeresult"%cdbName) + #tdLog.info("row: %d, %l64d, %l64d"%(tdSql.getData(0, 1),tdSql.getData(0, 2),tdSql.getData(0, 3)) + if tdSql.getRows() == expectRows: + break + else: + time.sleep(5) + + for i in range(expectRows): + tdLog.info ("consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) + resultList.append(tdSql.getData(i , 3)) + + return resultList + + def startTmqSimProcess(self,pollDelay,dbName,showMsg=1,showRow=1,cdbName='cdb',valgrind=0): + buildPath = tdCom.getBuildPath() + cfgPath = tdCom.getClientCfgPath() + if valgrind == 1: + logFile = cfgPath + '/../log/valgrind-tmq.log' + shellCmd = 'nohup valgrind --log-file=' + logFile + shellCmd += '--tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all --num-callers=20 -v --workaround-gcc296-bugs=yes ' + + if (platform.system().lower() == 'windows'): + shellCmd = 'mintty -h never -w hide ' + buildPath + '\\build\\bin\\tmq_sim.exe -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> nul 2>&1 &" + else: + shellCmd = 'nohup ' + buildPath + '/build/bin/tmq_sim -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> /dev/null 2>&1 &" + tdLog.info(shellCmd) + os.system(shellCmd) + + def syncCreateDbStbCtbInsertData(self, tsql, paraDict): + tdCom.create_database(tsql, paraDict["dbName"],paraDict["dropFlag"], paraDict['precision']) + tdCom.create_stable(tsql, paraDict["dbName"],paraDict["stbName"], paraDict["columnDict"], paraDict["tagDict"]) + tdCom.create_ctables(tsql, paraDict["dbName"],paraDict["stbName"],paraDict["ctbNum"],paraDict["tagDict"]) + if "event" in paraDict and type(paraDict['event']) == type(threading.Event()): + paraDict["event"].set() + tdCom.insert_data(tsql,paraDict["dbName"],paraDict["stbName"],paraDict["ctbNum"],paraDict["rowsPerTbl"],paraDict["batchNum"],paraDict["startTs"]) + return + + def threadFunction(self, **paraDict): + # create new connector for new tdSql instance in my thread + newTdSql = tdCom.newTdSql() + self.syncCreateDbStbCtbInsertData(self, newTdSql, paraDict) + return + + def asyncCreateDbStbCtbInsertData(self, paraDict): + pThread = threading.Thread(target=self.threadFunction, kwargs=paraDict) + pThread.start() + return pThread + + def close(self): + self.cursor.close() + +tmqCom = TMQCom() diff --git a/tests/system-test/fulltest.sh b/tests/system-test/fulltest.sh index 512fc396eea54c98d818c0cba1edd092d3eb6208..68daf5622758061114af8a7fba2e597e0979a97f 100755 --- a/tests/system-test/fulltest.sh +++ b/tests/system-test/fulltest.sh @@ -22,7 +22,7 @@ python3 ./test.py -f 1-insert/opentsdb_json_taosc_insert.py python3 ./test.py -f 1-insert/alter_stable.py python3 ./test.py -f 1-insert/alter_table.py python3 ./test.py -f 1-insert/insertWithMoreVgroup.py -# python3 ./test.py -f 1-inerst/create_table_comment.py +python3 ./test.py -f 1-insert/table_comment.py python3 ./test.py -f 2-query/between.py python3 ./test.py -f 2-query/distinct.py python3 ./test.py -f 2-query/varchar.py @@ -99,17 +99,26 @@ python3 ./test.py -f 2-query/stateduration.py python3 ./test.py -f 2-query/function_stateduration.py python3 ./test.py -f 2-query/statecount.py python3 ./test.py -f 2-query/tail.py +python3 ./test.py -f 2-query/ttl_comment.py +python3 ./test.py -f 2-query/distribute_agg_count.py +python3 ./test.py -f 2-query/distribute_agg_max.py +python3 ./test.py -f 2-query/distribute_agg_min.py +python3 ./test.py -f 2-query/distribute_agg_sum.py +python3 ./test.py -f 2-query/distribute_agg_spread.py +python3 ./test.py -f 2-query/distribute_agg_apercentile.py python3 ./test.py -f 6-cluster/5dnode1mnode.py python3 ./test.py -f 6-cluster/5dnode2mnode.py #python3 ./test.py -f 6-cluster/5dnode3mnodeStop.py -python3 ./test.py -f 6-cluster/5dnode3mnodeDrop.py +#python3 ./test.py -f 6-cluster/5dnode3mnodeDrop.py # BUG python3 ./test.py -f 6-cluster/5dnode3mnodeStopInsert.py python3 ./test.py -f 7-tmq/basic5.py python3 ./test.py -f 7-tmq/subscribeDb.py python3 ./test.py -f 7-tmq/subscribeDb0.py python3 ./test.py -f 7-tmq/subscribeDb1.py +python3 ./test.py -f 7-tmq/subscribeDb2.py +python3 ./test.py -f 7-tmq/subscribeDb3.py python3 ./test.py -f 7-tmq/subscribeStb.py python3 ./test.py -f 7-tmq/subscribeStb0.py python3 ./test.py -f 7-tmq/subscribeStb1.py diff --git a/tests/test/c/sdbDump.c b/tests/test/c/sdbDump.c index e5986cf4dddaa7191e7047e76b51305baefc55c1..a62c30660dfb603d18058c47e883b3308d30deb7 100644 --- a/tests/test/c/sdbDump.c +++ b/tests/test/c/sdbDump.c @@ -114,8 +114,10 @@ void dumpStb(SSdb *pSdb, SJson *json) { tjsonAddIntegerToObject(item, "tagVer", pObj->tagVer); tjsonAddIntegerToObject(item, "colVer", pObj->colVer); tjsonAddIntegerToObject(item, "nextColId", pObj->nextColId); - tjsonAddIntegerToObject(item, "xFilesFactor", pObj->xFilesFactor * 10000); - tjsonAddIntegerToObject(item, "delay", pObj->delay); + tjsonAddIntegerToObject(item, "watermark1", pObj->watermark[0]); + tjsonAddIntegerToObject(item, "watermark2", pObj->watermark[1]); + tjsonAddIntegerToObject(item, "maxdelay1", pObj->maxdelay[0]); + tjsonAddIntegerToObject(item, "maxdelay2", pObj->maxdelay[1]); tjsonAddIntegerToObject(item, "ttl", pObj->ttl); tjsonAddIntegerToObject(item, "numOfColumns", pObj->numOfColumns); tjsonAddIntegerToObject(item, "numOfTags", pObj->numOfTags); @@ -283,7 +285,8 @@ void dumpTrans(SSdb *pSdb, SJson *json) { tjsonAddIntegerToObject(item, "conflict", pObj->conflict); tjsonAddIntegerToObject(item, "exec", pObj->exec); tjsonAddStringToObject(item, "createdTime", i642str(pObj->createdTime)); - tjsonAddStringToObject(item, "dbname", pObj->dbname); + tjsonAddStringToObject(item, "dbname1", pObj->dbname1); + tjsonAddStringToObject(item, "dbname2", pObj->dbname2); tjsonAddIntegerToObject(item, "commitLogNum", taosArrayGetSize(pObj->commitActions)); tjsonAddIntegerToObject(item, "redoActionNum", taosArrayGetSize(pObj->redoActions)); tjsonAddIntegerToObject(item, "undoActionNum", taosArrayGetSize(pObj->undoActions)); @@ -294,8 +297,9 @@ void dumpTrans(SSdb *pSdb, SJson *json) { void dumpHeader(SSdb *pSdb, SJson *json) { tjsonAddIntegerToObject(json, "sver", 1); - tjsonAddStringToObject(json, "curVer", i642str(pSdb->curVer)); - tjsonAddStringToObject(json, "curTerm", i642str(pSdb->curTerm)); + tjsonAddStringToObject(json, "applyIndex", i642str(pSdb->applyIndex)); + tjsonAddStringToObject(json, "applyTerm", i642str(pSdb->applyTerm)); + tjsonAddStringToObject(json, "applyConfig", i642str(pSdb->applyConfig)); SJson *maxIdsJson = tjsonCreateObject(); tjsonAddItemToObject(json, "maxIds", maxIdsJson); diff --git a/tests/test/c/tmqDemo.c b/tests/test/c/tmqDemo.c index 5a972e071a5e28708cfb561ef1015d72ba830219..02fd3c13962eb5474deee298dca67dd3d07a7f49 100644 --- a/tests/test/c/tmqDemo.c +++ b/tests/test/c/tmqDemo.c @@ -353,8 +353,8 @@ tmq_list_t* build_topic_list() { void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) { static const int MIN_COMMIT_COUNT = 1000; - int msg_count = 0; - tmq_resp_err_t err; + int msg_count = 0; + int32_t err; if ((err = tmq_subscribe(tmq, topics))) { fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err)); @@ -379,7 +379,7 @@ void sync_consume_loop(tmq_t* tmq, tmq_list_t* topics) { } void perf_loop(tmq_t* tmq, tmq_list_t* topics, int32_t totalMsgs, int64_t walLogSize) { - tmq_resp_err_t err; + int32_t err; if ((err = tmq_subscribe(tmq, topics))) { fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(err)); diff --git a/tests/test/c/tmqSim.c b/tests/test/c/tmqSim.c index 8455bd9890533c6f3ee34c782f595db2b390209b..0f78a003d6acba40079a0d28c367f11b3966e2ae 100644 --- a/tests/test/c/tmqSim.c +++ b/tests/test/c/tmqSim.c @@ -62,7 +62,7 @@ typedef struct { tmq_t* tmq; tmq_list_t* topicList; - + int32_t numOfVgroups; int32_t rowsOfPerVgroups[MAX_VGROUP_CNT][2]; // [i][0]: vgroup id, [i][1]: rows of consume int64_t ts; @@ -74,7 +74,7 @@ typedef struct { char cdbName[32]; char dbName[32]; int32_t showMsgFlag; - int32_t showRowFlag; + int32_t showRowFlag; int32_t saveRowFlag; int32_t consumeDelay; // unit s int32_t numOfThread; @@ -108,26 +108,20 @@ static void printHelp() { } char* getCurrentTimeString(char* timeString) { - time_t tTime = taosGetTimestampSec(); + time_t tTime = taosGetTimestampSec(); struct tm tm = *taosLocalTime(&tTime, NULL); - sprintf(timeString, "%d-%02d-%02d %02d:%02d:%02d", - tm.tm_year + 1900, - tm.tm_mon + 1, - tm.tm_mday, - tm.tm_hour, - tm.tm_min, - tm.tm_sec); + sprintf(timeString, "%d-%02d-%02d %02d:%02d:%02d", tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, + tm.tm_min, tm.tm_sec); return timeString; } - void initLogFile() { char filename[256]; - char tmpString[128]; + char tmpString[128]; - sprintf(filename,"%s/../log/tmqlog_%s.txt", configDir, getCurrentTimeString(tmpString)); - //sprintf(filename, "%s/../log/tmqlog.txt", configDir); + sprintf(filename, "%s/../log/tmqlog_%s.txt", configDir, getCurrentTimeString(tmpString)); + // sprintf(filename, "%s/../log/tmqlog.txt", configDir); #ifdef WINDOWS for (int i = 2; i < sizeof(filename); i++) { if (filename[i] == ':') filename[i] = '-'; @@ -249,17 +243,18 @@ void addRowsToVgroupId(SThreadInfo* pInfo, int32_t vgroupId, int32_t rows) { for (i = 0; i < pInfo->numOfVgroups; i++) { if (vgroupId == pInfo->rowsOfPerVgroups[i][0]) { pInfo->rowsOfPerVgroups[i][1] += rows; - return; - } + return; + } } pInfo->rowsOfPerVgroups[pInfo->numOfVgroups][0] = vgroupId; pInfo->rowsOfPerVgroups[pInfo->numOfVgroups][1] += rows; pInfo->numOfVgroups++; - + taosFprintfFile(g_fp, "consume id %d, add one new vogroup id: %d\n", pInfo->consumerId, vgroupId); if (pInfo->numOfVgroups > MAX_VGROUP_CNT) { - taosFprintfFile(g_fp, "====consume id %d, vgroup num %d over than 32. new vgroupId: %d\n", pInfo->consumerId, pInfo->numOfVgroups, vgroupId); + taosFprintfFile(g_fp, "====consume id %d, vgroup num %d over than 32. new vgroupId: %d\n", pInfo->consumerId, + pInfo->numOfVgroups, vgroupId); taosCloseFile(&g_fp); exit(-1); } @@ -277,7 +272,8 @@ int32_t saveConsumeContentToTbl(SThreadInfo* pInfo, char* buf) { TAOS* pConn = taos_connect(NULL, "root", "taosdata", NULL, 0); assert(pConn != NULL); - sprintf(sqlStr, "insert into %s.content_%d values (%"PRId64", \'%s\')", g_stConfInfo.cdbName, pInfo->consumerId, pInfo->ts++, buf); + sprintf(sqlStr, "insert into %s.content_%d values (%" PRId64 ", \'%s\')", g_stConfInfo.cdbName, pInfo->consumerId, + pInfo->ts++, buf); TAOS_RES* pRes = taos_query(pConn, sqlStr); if (taos_errno(pRes) != 0) { pError("error in insert consume result, reason:%s\n", taos_errstr(pRes)); @@ -295,12 +291,13 @@ int32_t saveConsumeContentToTbl(SThreadInfo* pInfo, char* buf) { static int32_t msg_process(TAOS_RES* msg, SThreadInfo* pInfo, int32_t msgIndex) { char buf[1024]; int32_t totalRows = 0; - + // printf("topic: %s\n", tmq_get_topic_name(msg)); int32_t vgroupId = tmq_get_vgroup_id(msg); - + taosFprintfFile(g_fp, "msg index:%" PRId64 ", consumerId: %d\n", msgIndex, pInfo->consumerId); - //taosFprintfFile(g_fp, "topic: %s, vgroupId: %d, tableName: %s\n", tmq_get_topic_name(msg), vgroupId, tmq_get_table_name(msg)); + // taosFprintfFile(g_fp, "topic: %s, vgroupId: %d, tableName: %s\n", tmq_get_topic_name(msg), vgroupId, + // tmq_get_table_name(msg)); taosFprintfFile(g_fp, "topic: %s, vgroupId: %d\n", tmq_get_topic_name(msg), vgroupId); while (1) { @@ -316,9 +313,9 @@ static int32_t msg_process(TAOS_RES* msg, SThreadInfo* pInfo, int32_t msgIndex) const char* tbName = tmq_get_table_name(msg); if (0 != g_stConfInfo.showRowFlag) { - taosFprintfFile(g_fp, "tbname:%s, rows[%d]: %s\n", (tbName != NULL ? tbName:"null table"), totalRows, buf); - if (0 != g_stConfInfo.saveRowFlag) { - saveConsumeContentToTbl(pInfo, buf); + taosFprintfFile(g_fp, "tbname:%s, rows[%d]: %s\n", (tbName != NULL ? tbName : "null table"), totalRows, buf); + if (0 != g_stConfInfo.saveRowFlag) { + saveConsumeContentToTbl(pInfo, buf); } } @@ -326,7 +323,7 @@ static int32_t msg_process(TAOS_RES* msg, SThreadInfo* pInfo, int32_t msgIndex) } addRowsToVgroupId(pInfo, vgroupId, totalRows); - + return totalRows; } @@ -342,8 +339,8 @@ int queryDB(TAOS* taos, char* command) { return 0; } -static void tmq_commit_cb_print(tmq_t* tmq, tmq_resp_err_t resp, tmq_topic_vgroup_list_t* offsets, void* param) { - pError("tmq_commit_cb_print() commit %d\n", resp); +static void tmq_commit_cb_print(tmq_t* tmq, int32_t code, void* param) { + pError("tmq_commit_cb_print() commit %d\n", code); } void build_consumer(SThreadInfo* pInfo) { @@ -401,16 +398,11 @@ int32_t saveConsumeResult(SThreadInfo* pInfo) { int64_t now = taosGetTimestampMs(); // schema: ts timestamp, consumerid int, consummsgcnt bigint, checkresult int - sprintf(sqlStr, "insert into %s.consumeresult values (%"PRId64", %d, %" PRId64 ", %" PRId64 ", %d)", - g_stConfInfo.cdbName, - now, - pInfo->consumerId, - pInfo->consumeMsgCnt, - pInfo->consumeRowCnt, - pInfo->checkresult); + sprintf(sqlStr, "insert into %s.consumeresult values (%" PRId64 ", %d, %" PRId64 ", %" PRId64 ", %d)", + g_stConfInfo.cdbName, now, pInfo->consumerId, pInfo->consumeMsgCnt, pInfo->consumeRowCnt, pInfo->checkresult); char tmpString[128]; - taosFprintfFile(g_fp, "%s, consume id %d result: %s\n", getCurrentTimeString(tmpString), pInfo->consumerId ,sqlStr); + taosFprintfFile(g_fp, "%s, consume id %d result: %s\n", getCurrentTimeString(tmpString), pInfo->consumerId, sqlStr); TAOS_RES* pRes = taos_query(pConn, sqlStr); if (taos_errno(pRes) != 0) { @@ -421,7 +413,7 @@ int32_t saveConsumeResult(SThreadInfo* pInfo) { taos_free_result(pRes); - #if 0 +#if 0 // vgroups for (i = 0; i < pInfo->numOfVgroups; i++) { // schema: ts timestamp, consumerid int, consummsgcnt bigint, checkresult int @@ -445,19 +437,20 @@ int32_t saveConsumeResult(SThreadInfo* pInfo) { taos_free_result(pRes); } - #endif +#endif return 0; } void loop_consume(SThreadInfo* pInfo) { - tmq_resp_err_t err; + int32_t code; int64_t totalMsgs = 0; int64_t totalRows = 0; char tmpString[128]; - taosFprintfFile(g_fp, "%s consumer id %d start to loop pull msg\n", getCurrentTimeString(tmpString), pInfo->consumerId); + taosFprintfFile(g_fp, "%s consumer id %d start to loop pull msg\n", getCurrentTimeString(tmpString), + pInfo->consumerId); pInfo->ts = taosGetTimestampMs(); @@ -473,7 +466,7 @@ void loop_consume(SThreadInfo* pInfo) { totalMsgs++; if (totalRows >= pInfo->expectMsgCnt) { - char tmpString[128]; + char tmpString[128]; taosFprintfFile(g_fp, "%s over than expect rows, so break consume\n", getCurrentTimeString(tmpString)); break; } @@ -503,8 +496,8 @@ void* consumeThreadFunc(void* param) { return NULL; } - tmq_resp_err_t err = tmq_subscribe(pInfo->tmq, pInfo->topicList); - if (err) { + int32_t err = tmq_subscribe(pInfo->tmq, pInfo->topicList); + if (err != 0) { pError("tmq_subscribe() fail, reason: %s\n", tmq_err2str(err)); exit(-1); } @@ -519,17 +512,19 @@ void* consumeThreadFunc(void* param) { pPrint("tmq_commit() manual commit when consume end.\n"); /*tmq_commit(pInfo->tmq, NULL, 0);*/ tmq_commit_sync(pInfo->tmq, NULL); + taosFprintfFile(g_fp, "tmq_commit() manual commit over.\n"); + pPrint("tmq_commit() manual commit over.\n"); } err = tmq_unsubscribe(pInfo->tmq); - if (err) { + if (err != 0) { pError("tmq_unsubscribe() fail, reason: %s\n", tmq_err2str(err)); /*pInfo->consumeMsgCnt = -1;*/ /*return NULL;*/ } err = tmq_consumer_close(pInfo->tmq); - if (err) { + if (err != 0) { pError("tmq_consumer_close() fail, reason: %s\n", tmq_err2str(err)); /*exit(-1);*/ } diff --git a/tests/tsim/src/simExe.c b/tests/tsim/src/simExe.c index dbc3c2c4602f1424d997a07fb28ac92e1ea1ff0e..f97a13d2c5867ef35ceda35e716461400544371c 100644 --- a/tests/tsim/src/simExe.c +++ b/tests/tsim/src/simExe.c @@ -458,11 +458,17 @@ bool simExecuteSystemContentCmd(SScript *script, char *option) { char buf[4096] = {0}; char buf1[4096 + 512] = {0}; char filename[400] = {0}; - sprintf(filename, "%s/%s.tmp", simScriptDir, script->fileName); + sprintf(filename, "%s" TD_DIRSEP "%s.tmp", simScriptDir, script->fileName); +#ifdef WINDOWS + sprintf(buf, "cd %s && ", simScriptDir); + simVisuallizeOption(script, option, buf + strlen(buf)); + sprintf(buf1, "%s > %s 2>nul", buf, filename); +#else sprintf(buf, "cd %s; ", simScriptDir); simVisuallizeOption(script, option, buf + strlen(buf)); sprintf(buf1, "%s > %s 2>/dev/null", buf, filename); +#endif sprintf(script->system_exit_code, "%d", system(buf1)); simStoreSystemContentResult(script, filename); diff --git a/tests/tsim/src/simParse.c b/tests/tsim/src/simParse.c index 638c4a1ccb57da6b8d4f29521f931783cff13ba7..5b6dda4dae008364ed11c352e6fd7c8adfa0fc76 100644 --- a/tests/tsim/src/simParse.c +++ b/tests/tsim/src/simParse.c @@ -206,7 +206,7 @@ SScript *simParseScript(char *fileName) { for (int32_t i = 0; i < cmdlen; ++i) { if (buffer[i] == '\r' || buffer[i] == '\n') { - buffer[i] = ' '; + buffer[i] = '\0'; } } diff --git a/tools/shell/src/shellEngine.c b/tools/shell/src/shellEngine.c index 8ed0e9ddcf43a490c905ae9db5ada09f202d3899..1f29237d38b0441507bef05a39586d8a3b80ca56 100644 --- a/tools/shell/src/shellEngine.c +++ b/tools/shell/src/shellEngine.c @@ -855,8 +855,7 @@ void shellGetGrantInfo() { if (code == TSDB_CODE_OPS_NOT_SUPPORT) { fprintf(stdout, "Server is Community Edition, %s\n\n", sinfo); } else { - fprintf(stderr, "Failed to check Server Edition, Reason:0x%04x:%s\n\n", taos_errno(shell.conn), - taos_errstr(shell.conn)); + fprintf(stderr, "Failed to check Server Edition, Reason:0x%04x:%s\n\n", code, taos_errstr(tres)); } return; } diff --git a/tools/taos-tools b/tools/taos-tools index 3d5aa76f8c718dcffa100b45e4cbf313d499c356..28a49b447f71c4f014ebbac858b7215b897d57fd 160000 --- a/tools/taos-tools +++ b/tools/taos-tools @@ -1 +1 @@ -Subproject commit 3d5aa76f8c718dcffa100b45e4cbf313d499c356 +Subproject commit 28a49b447f71c4f014ebbac858b7215b897d57fd diff --git a/tools/taosadapter b/tools/taosadapter new file mode 160000 index 0000000000000000000000000000000000000000..29926478edd87533a043f91c1a9ed0e27671e626 --- /dev/null +++ b/tools/taosadapter @@ -0,0 +1 @@ +Subproject commit 29926478edd87533a043f91c1a9ed0e27671e626